Pastiche Master: Exemplar-Based High-Resolution Portrait Style Transfer

1. Introduction

    

    Artistic portraits are popular in our daily lives and especially in industries related to comics, animations, posters,and advertising.
In this paper, we focus on exemplar-based portrait style transfer, a core problem that aims to transfer the style of an exemplar artistic portrait onto a target face.
    Recent studies on StyleGAN (A style-based generator architecture for generative adversarial networks)show high performance on  artistic portrait generation by transfer learning with limited data. In this paper, we explore more challenging exemplar-based high-resolution portrait style transfer by introducing a novel DualStyleGAN with flexible control of dual styles of the original face domain and the extended artistic portrait domain. Different from StyleGAN, DualStyleGAN provides a natural way of style transfer by characterizing the content and style of a portrait with an intrinsic style path and a new extrinsic style path, respectively. The delicately designed extrinsic style path enables our model to modulate both the color and complex structural styles hierarchically to precisely pastiche the style example

2. Related Work

    Style-GAN synthesizes high-resolution face images with hierarchical style control. StyleGAN was fine tuned on limited cartoon data, and found it promising in generating plausible cartoon faces. The original model and fine-tuned model exhibit a reasonable degree of semantic alignment, However, the alignment gets weakened along with unconditional fine-tuning without valid supervision, eventually leading to a failure in layer swappin.

By comparison, our model has an explicit extrinsic style path that can be conditionally trained to characterize the structural styles. Moreover, supervision for learning diverse styles is provided via facial destylization.

3. Portrait Style Transfer via DualStyleGAN

    DualStyleGAN can be built  based on a pre-trained StyleGAN, which can be transferred to a new domain and characterize the styles of both the original and the extended domains. Unconditional fine-tuning translates the StyleGAN generative space as a whole, leading to the loss of diversity of the captured style. Our key idea is to seek valid supervision to learn diverse styles and to explicitly model the two kinds of styles with two individual style paths, We train DualStyleGAN with a principled progressive strategy for robust conditional fine-tuning.

    We train DualStyleGAN with a principled progressive strategy for

robust conditional fine-tuning, since the two domains might have a large appearance discrepancy, the challenge to balance between face realism and fidelity to the portrait appears. A possible solution for this problem could be multi-stage destylization:

• Stage I: Latent initialization. The artistic portrait S is first embedded   into the StyleGAN latent space by an encoder E
• Stage II: Latent optimization. In a face image is stylized by optimizing a latent code of g to reconstruct this image and applying this code to a fine-tuned model g′
• Stage III: Image embedding. The result has reasonable facial structures, providing valid supervision on how to deform and abstract the facial structures to imitate S

3.3. Progressive Fine-Tuning

  

 3.3. Progressive Fine-Tuning

    A progressive fine-tuning scheme is used to smoothly transform the generative space of DualStyleGAN towards the target domain. The scheme borrows the idea of curriculum learning to gradually increase the task difficulty in three stages:

• Stage I: Color transfer on source domain
• Stage II: Structure transfer on source domain.
• Stage III: Style transfer on target domain.

4. Conclusion

    

    We extend StyleGAN to accept style condition from new domains while preserving its style control in the original domain. This results in an interesting application of high-resolution exemplar-based portrait style transfer with a friendly data requirement. DualStyleGAN, with an additional style path to StyleGAN, can effectively model and modulate the intrinsic and extrinsic styles for flexible and diverse artistic portrait generation. We show that valid transfer learning on DualStyleGAN can be achieved with
a special architecture design and progressive training strategy. We believe our idea of model extension in terms of both architecture and data can be potentially applied to other tasks such as more general image-to-image translation and knowledge distillation.