Tinkering with EBSynth: Painting videos to look like cartoons

Recently I came across EbSynth, a program used to convert video footage into a motion painting by painting a single keyframe. I find it interesting that, although AI experts developed the program,  it doesn’t do any AI/Machine learning type stuff. Instead, it apparently uses ‘non-parametric example-based methods’. Which I understand to mean it makes no assumptions, it will just look at the single-frame painted by the artist and based on this single example alone it will paint what it knows for the rest of the video. It doesn’t attempt to guess anything more, so if something new pops into the video, which wasn’t in the frame, it will just not attempt to paint it. I guess this gives more control to the artist over how the final output looks. While the idea on the website is that you can use it to create ‘motion paintings’, you can create very crude cartoons. I quite like the idea that in a world of machine learning, it does not want the machine to do any learning. I’ve documented my playing with the tool here, where I paint a single keyframe to turn Boris into a half animated McDonald’s Monster.

Prepare

There are lots of different software you could do the steps in, I used software that was free to download, and had used before. I realise that I’ve actually used more tools than I need because Blender can not only export a sequence but can also be used to ‘stitch’ the images back together. I’m just not competent enough with Blender to do that. If you know what you are doing, you can replace some of this software with better ones that suit you. I never fully understand what I am doing; you might know better, so use your judgement.

Software you will need:

• Adobe Media Encoder – I believed this to be free, but it turns out that I might have access to it through a subscription I have. If you can’t use Media Encoder, you could try VLC, FFMpeg, Blender or Virtual Dub (but Virtual Dub didn’t work with my MP4 out of the box). Basically, you just need something to turn your video into a sequence of frames.
• EbSynth – To make a sequence of painted frames from one frame
• Virtual Dub – Or something to turn your painted sequence of frames back into a video. I would have used this to turn my video into a sequence, but it required some faffing to work with my video format.
• MS Paint – Or something more professional to do your painting in.

Make PNG sequence from video

First, you need to grab a video and turn it into a series of PNGs. Depending on your GPU, you might want to start with a short, low res video. I tried to use a video from my phone and my 2GB GTX 960, and I had difficulty both turning the video into frames in Adobe after effects, (but it was fine in adobe encoder) and using Ebsynth I got memory errors.

Remember that you are painting a single frame, so everything you want to paint needs to be mostly in the video at all times. E.g. the video will work better if the person or thing in the video doesn’t spin around. Ebsynth works from example frames that you colour in, so if there are things in the video that aren’t in the keyframe you paint, then you are in trouble (well actually, it is possible, but you’ll have to paint more keyframes).

1. Create a folder on your desktop and call it “tutorial” or something. Put your video in there. I used a video of Boris.
2. Then open Adobe Media Encoder and drag your video from the tutorial folder to the queue

Your queue should look like the image above.

3. Click the output type in Adobe Media Encoder so it says PNG so that it is a PNG image (and crop and resize the video if you want). Make sure the output files go in your tutorial folder, in a subfolder called ‘video’. Once you are ready press the play button so that your video directory looks like this:

Paint a Keyframe.

4. Create a new directory called ‘key’ in your tutorial directory, the one with your video in.

5. Copy one of your images created by adobe from the subfolder called video into the new key directory, keeping the filename the same. It needs to be a good frame where the eyes aren’t closed, and you can see the character you are painting well.

6. Open the single frame and paint it in MS paint and make sure to save the painted frame over itself in the “key” directory. I got bored halfway through colouring him in, but the plus point of getting bored is that it will give you a good understanding of what the program is doing in the final output because you can compare what I painted to what I didn’t.

To recap, you should now have a directory with your original video in. In this directory is two subdirectories:

• Video directory with all your video frames
• Keys directory with one single painted frame (using the same filename as it has in the video directory)

Get EbSynth to paint all the other video frames

Based on your single keyframe in your keys folder, Ebsynth is going to paint all your other frames

7. Open Ebysnth up. Drag the video directory to the keyframes section of the program and the key directory to the keyframes section of the program. Then press run all.

If you’ve had to paint multiple frames you are going to have to fill in the stop sections at the bottom of the program by clicking the + and adding more frames

Stitch the frames together to make a new video.

8. Open the Virtual Dub program. Locate your tutorial directory in Windows Explorer/finder and you should find that EbSynth has created a new directory full of painted frames.

9. Drag the first one into Virtual Dub and press play, it’s now stitching them together in a video. Export as a movie and you’re done. Here are the first few seconds as mine as a GIF: