Vpro Gids Gan Covers
10 May 2020Genrating VPRO Gids Covers with Generative Adversarial Networks (GAN)
Picture: A selection of generated VPRO Gids covers.
This post contains information about my StyleGAN application to VPRO Gids covers. I used StyleGAN, SRFBN and Super-SloMo to generate (moving) cover art for the 14th VPRO Gids edition of 2020. Also check out this page on the VPRO Gids website with additional information.
VPRO Gids
Picture: Example VPRO Gids covers from the training dataset.
VPRO Gids is the weekly television guide of Dutch broadcast station VPRO. VPRO has played a crucial part in Dutch design history with the guide being a focal point. I was asked to create an AI generated cover for the guide. This repository details the techniques I used to generate high-resolution VPRO Gids covers and latent space visualization videos.
Method
StyleGAN was used for image genration, SRFBN to create super-resolution images from the StyleGAN output and Super-SloMo to create slow motion versions of latent space walk videos.
1. StyleGAN
StyleGAN is a generative model capable of generating realistic images of people, characters, and more. More information can be found here.
In this application, StyleGAN was trained on a dataset of 9.5 years of historic VPRO covers. Because of training time constraints, a StyleGAN model with an output size of 512x512 was used. Since there are hardly any constant features across covers except for the logo, the expected output is artistic rather than realistic. Here is a PyTorch implementation of StyleGAN you can use to train your own model. Create your own dataset and follow the instructions to preprocess your data and create a generative model.
2. Feedback Network for Image Super-Resolution
Feedback Network for Image Super-Resolution (SRFBN) and some post-processing was used to create super-resolution images of the low-resolution StyleGAN output. SRFBN is an image super-resolution feedback network (SRFBN) to refine low-level representations with high-level information. SRFBN was applied to the StyleGAN output, with a magnification of x4, followed by a resizing and resampling of the output (using ImageMagick) to create images of size 285x215 mm with 300dpi.
3. Super-SloMo
To better visualize the generative capabilities of the StyleGAN model, a latent space walk was created to create videos that look like generated covers morphing into each other. To create a more detailed latent space visalization of the StyleGAN output in the time domain, Super-SloMo was used. Super-SloMo is an end-to-end convolutional neural network for variable-length multi-frame video interpolation, where the motion interpretation and occlusion reasoning are jointly modeled. For this project, Super-SloMo was applied to increase the number of frames by 4x. This PyTorch implementation is useful for running your own experiments.
Animated GIF Examples of latent space visualization:
Pictures: Animated GIF examples of latent space visualization of the StyleGAN model using SRFBN and Super-SloMo. Note: these are reduced in resolution and frames per second.
Video of latent space visalization:
VPRO Gids Artificial Intelligence Cover from VPRO Gids on Vimeo.
Resources
Related material is available via the following links:
StyleGAN:
- Original Code (NVIDIA Research)
- PyTorch implementation (Kim Seonghyeon)
- Karras, Tero, Samuli Laine, and Timo Aila. “A Style-Based Generator Architecture for Generative Adversarial Networks” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4401-4410. 2019.
Feedback Network for Image Super-Resolution
- Code: Feedback Network for Image Super-Resolution (Zhen Li)
- Paper: Li, Zhen, Jinglei Yang, Zheng Liu, Xiaomin Yang, Gwanggil Jeon, and Wei Wu. “Feedback network for image super-resolution.” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3867-3876. 2019.
Super-SloMo
- Code: Super-SloMo (Avinash Paliwal)
- Paper: Jiang, Huaizu, Deqing Sun, Varun Jampani, Ming-Hsuan Yang, Erik Learned-Miller, and Jan Kautz. “Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9000-9008. 2018.