Show HN: Feedback on Sketch Colourisation
github.comHi I am looking for some feedback on our new project "Sketch Colourisation". The envisioned UI and objectives are --
* An artist should have greater control on how to colour a sketch. While a text-to-image model lacks this fine-grained control, a per-pixel colourisation pipeline makes sketch colourisation a laborious process with high-entry barrier.
* What if an artist only draws a mask for a local region and specifies the colour palette for that local region? Then a neural network figures out how to colour the overall sketch -- while maintaining those local colour palette.
[I would really like a feedback if the above UI (i.e., mask and local colour palette) makes sense to users/designers. As researchers, we often have the wrong idea of what is desired by end-users.]
* On the exact implementation of the above concept, we designed a no-training based neural network framework -- and also make sure it runs on a Nvidia 4090. In other words, I will try to avoid any expensive training or inference -- which defeats the purpose of being useful to people (not just some research labs).
* Note, I am not so bothered about the exact implementation (or whether it is "novel") -- as long as it is useful.
* A shameless advertisement: The codebase (https://github.com/CHAITron/sketchdeco-code.git) is MIT License. It is no way near to being useful to people -- but I would really like to pursue this direction and your feedback/criticism will be immensely helpful.
Thanks
I think there's some demand in the space [1]. I've used the Pixiv colorizer in the past (Petalica Paint [0]) with some good results and some really not as good. Some of the 'errors' it makes are actually interesting and can be used to make the final result richer than without it. But the errors can be 'unfixable' and actually worsen the final result. So in the end it's hard to say if it's a time-saver.
Regarding the interface Petalica gets it right, although mask + palette makes sense.
A fully local model with support with different styles and a configurable respect for the line art (with default 100% respect) would definitely be a game changer. At their current level, the AI models are slightly lacking, so it feels like too much time investment compared to the quality of the results.
[0] https://petalica.com/index_en.html [1] https://github.com/lllyasviel/style2paints/issues/235
Nice to see any improvements in this space.
The workflow I've used in the past for this is using the fantastic StableDiffusion 1.5 ControlNet lineart model. See https://x.com/P_Galbraith/status/1716299002969469054 for example.
Probably the ideal workflow would use a reference image via something like IP-Adapter since a simple colour palette wouldn't really give enough control (see https://x.com/P_Galbraith/status/1716405163420963196 for example). Typically you have the character design done in a flat perspective. So it would be nice to have something like an IP-Adapter input as well as a detailed line drawing and a rough paintover.
You would also need to provide a way to give lighting control (i.e. control over light sources and direction) as well as multiple characters etc... for it to be useful in complex compositions.
The Invoke team released regional guidance using IP Adapter a few months ago, which can use color palettes + style transfer mode, along with text prompts and controlnets.
Would take a look at that for some inspiration -- The UI is Apache 2.0 and used by professional artists. I'd be curious how you think it performs relative to the workflow you've developed.
You're spot on that researchers don't always build the UI that end-users want to use. Always love to see people thinking about the creatives. Good work!