These Are Pixels Made of Wood! 🌲🧩

Dear Fellow Scholars, this is Two Minute 
Papers with Dr. Károly Zsolnai-Fehér. Everybody loves style transfer. 
This is a task typically done   with neural networks where we have 
two images, one for content, and one   for style, and the output is the content image 
reimagined with this new style. The cool thing   is that the style can be a different photo, 
a famous painting, or even, wooden patterns. Feast your eyes on these majestic images of 
this cat reimagined with wooden parquetry   with these previous methods. And now, look at 
the result of this new technique, that looks   way nicer. Everything is in order here, except one 
thing. And now, hold on to your papers, because   this is not style transfer. Not at all. This is 
not a synthetic photo made by a neural network,   this is a reproduction of this cat image 
by cutting wood slabs into tiny pieces   and putting them together carefully. This 
is computational parquetry. And here,   the key requirement is that if we look from afar, 
it looks like the target image, but if we zoom in,   it gets abundantly clear that the puzzle 
pieces here are indeed made of real wood.

And that is an excellent intuition for this 
work. It is kind of like image stylization,   but done in the real world. Now that is extremely challenging. Why is that? 
Well, first, there are lots of different kinds   of wood types. Second, if this piece was not a 
physical object but an image, this job would not   be that hard because we could add to it, clone it, 
and do all kinds of pixel magic to it. However,   these are real, physical pieces of wood, 
so we can do exactly none of that.

The only   thing we can do is take away from it, 
and we have limitations even on that,   because we have to design it in a way that 
a CNC device should be able to cut these   pieces. And third, you will see that 
initially, nothing seems to work well.   However, this technique does this with flying 
colors, so I wonder, how does this really work? First, we can take a photo of the wood panels that 
we have our disposal, decide how and where to cut,   give these instructions to the CNC 
machine to perform the cutting,   and now, we have to assemble them in a way that 
it resembles the target image. Well, still, that’s   easier said than done. For instance, imagine 
that we have this target image, and we have   these wood panels. This doesn’t look anything like 
that, so how could we possibly approximate it? If we try to match the colors of the two, we 
get something that is too much in the middle,   and the colors don’t resemble any 
of the original inputs.

See also  Fine Woodworking #290 – July/August 2021

Not good.   Instead, the authors opted to transform both 
of them to grayscale, and match not the colors,   but the intensities of the colors instead. This 
seems a little more usable…until we realize   that we still don’t know what 
pieces to use and where. Look. Here, on the left, you see how the image 
is being reproduced with the wood pieces,   but we have to mind the fact that as 
soon as we cut out one piece of wood,   it is not available anymore, so it has to be 
subtracted from our wood panel repository here.   As our resources are constrained, depending on 
what order we put the pieces together, we may   get a completely different result. But look. There 
is still a problem…the left part of the suit gets   a lot of detail, while the right part, not so 
much. I cannot judge which solution is better,   less or more detail, but it needs to be 
a little more consistent over the image.   Now you see that whatever we do, nothing 
seems to work well in the general case.

Now, we could get a much better solution 
if we would run the algorithm with   every possible starting point in the image, and 
with every possible ordering of the wood pieces,   but that would take longer 
than our lifetime to finish,   so what do we do? Well, the authors have two 
really cool heuristics to address this problem.   First, we can start from the middle, that usually 
gives us a reasonably good solution, since the   object of interest is often in the middle of the 
image and the good pieces are still available for   it. Or, even better, if that does not work 
too well, we can look for salient regions,   these are the places where there is a lot going 
on, and try to fill them in first. As you see,   both of these tricks seem to work quite well 
most of the time. Finally, something that works. And if you have been holding on to your papers, 
now squeeze that paper, because this technique   not only works, but provides us a great deal of 
artistic control over the results. Look at that!   And that’s not all, we can even 
control the resolution of the output,   or, we can create a hand-drawn geometry 
ourselves.

See also  Bowl Surgery, Episode 8 - Woodturning, Woodworking

I love how the authors took   a really challenging problem, 
where nothing really worked well,   and still, they didn’t stop until they 
absolutely nailed the solution. Congratulations! Thanks for watching and for your generous 
support, and I'll see you next time!.

As found on YouTube