Dear Fellow Scholars, this is Two Minute
Papers with Dr. Károly Zsolnai-Fehér. Everybody loves style transfer.
This is a task typically done with neural networks where we have
two images, one for content, and one for style, and the output is the content image
reimagined with this new style. The cool thing is that the style can be a different photo,
a famous painting, or even, wooden patterns. Feast your eyes on these majestic images of
this cat reimagined with wooden parquetry with these previous methods. And now, look at
the result of this new technique, that looks way nicer. Everything is in order here, except one
thing. And now, hold on to your papers, because this is not style transfer. Not at all. This is
not a synthetic photo made by a neural network, this is a reproduction of this cat image
by cutting wood slabs into tiny pieces and putting them together carefully. This
is computational parquetry. And here, the key requirement is that if we look from afar,
it looks like the target image, but if we zoom in, it gets abundantly clear that the puzzle
pieces here are indeed made of real wood.
And that is an excellent intuition for this
work. It is kind of like image stylization, but done in the real world. Now that is extremely challenging. Why is that?
Well, first, there are lots of different kinds of wood types. Second, if this piece was not a
physical object but an image, this job would not be that hard because we could add to it, clone it,
and do all kinds of pixel magic to it. However, these are real, physical pieces of wood,
so we can do exactly none of that.
The only thing we can do is take away from it,
and we have limitations even on that, because we have to design it in a way that
a CNC device should be able to cut these pieces. And third, you will see that
initially, nothing seems to work well. However, this technique does this with flying
colors, so I wonder, how does this really work? First, we can take a photo of the wood panels that
we have our disposal, decide how and where to cut, give these instructions to the CNC
machine to perform the cutting, and now, we have to assemble them in a way that
it resembles the target image. Well, still, that’s easier said than done. For instance, imagine
that we have this target image, and we have these wood panels. This doesn’t look anything like
that, so how could we possibly approximate it? If we try to match the colors of the two, we
get something that is too much in the middle, and the colors don’t resemble any
of the original inputs.
Not good. Instead, the authors opted to transform both
of them to grayscale, and match not the colors, but the intensities of the colors instead. This
seems a little more usable…until we realize that we still don’t know what
pieces to use and where. Look. Here, on the left, you see how the image
is being reproduced with the wood pieces, but we have to mind the fact that as
soon as we cut out one piece of wood, it is not available anymore, so it has to be
subtracted from our wood panel repository here. As our resources are constrained, depending on
what order we put the pieces together, we may get a completely different result. But look. There
is still a problem…the left part of the suit gets a lot of detail, while the right part, not so
much. I cannot judge which solution is better, less or more detail, but it needs to be
a little more consistent over the image. Now you see that whatever we do, nothing
seems to work well in the general case.
Now, we could get a much better solution
if we would run the algorithm with every possible starting point in the image, and
with every possible ordering of the wood pieces, but that would take longer
than our lifetime to finish, so what do we do? Well, the authors have two
really cool heuristics to address this problem. First, we can start from the middle, that usually
gives us a reasonably good solution, since the object of interest is often in the middle of the
image and the good pieces are still available for it. Or, even better, if that does not work
too well, we can look for salient regions, these are the places where there is a lot going
on, and try to fill them in first. As you see, both of these tricks seem to work quite well
most of the time. Finally, something that works. And if you have been holding on to your papers,
now squeeze that paper, because this technique not only works, but provides us a great deal of
artistic control over the results. Look at that! And that’s not all, we can even
control the resolution of the output, or, we can create a hand-drawn geometry
ourselves.
I love how the authors took a really challenging problem,
where nothing really worked well, and still, they didn’t stop until they
absolutely nailed the solution. Congratulations! Thanks for watching and for your generous
support, and I'll see you next time!.