DeepFaceLab: Integrated, flexible and extensible face-swapping framework

By team 2AT

  

What is this?

 

     DeepFaceLab is a “deep fake” framework, used for face-swapping. It’s a popular 

solution in this domain thanks to its performance and easy to use nature, compared

to other solutions. 

    Throughout the years, DeepFaceLab not only stirred up the interest of computer 

vision enthusiasts, but enjoyed its fair share of memes, generated by swapping faces 

of popular people.

    However, this new technology was also used to spread misinformation,

 manipulation, harassment and persuasion. Identifying deep fakes is a

 challenging task that requires considerable effort. The DeepFake 

Detection Challenge (DFDC), initiated by Facebook and Microsoft, tries

 to solve this issue. In order to train these models, you need highquality

 forgeries - and that’s where DeepFaceLab comes in. 

Implementation basics 

In DeepFaceLab (DFL for short), we can abstract the pipeline into three phases:

 extraction, training, and conversion. 

1. Extraction phase

    Steps for achieving face swap:

1. Face detection - done with S3FD

2. Face alignment - done by extracting facial landmarks

3. Face Segmentation - optional phase but keeps the training robust to hands, glasses

    or any other objects 

    

2. Training phase:

    This phase plays the most vital role in achieving photorealistic face-swapping results of

 DFL. There are two structures proposed for this step, DF and LIAE structure. DF structure

 can finish the face-swapping task but cannot inherit enough information from dst, such as

 lighting. To further enhance the problem of light consistency, we propose LIAE which is a

 more complex structure than DF.

3. Conversion phase: this phase is often the underdog of the 3 phases

1. src2dst: takes the original face and transposes it onto the target face 

2. blending: make the face seamlessly fit, using color transfer algorithms. Takes into

    account skin tones, face shapes, illumination etc 

3. sharpening: after blending, we sharpen the face, in order to preserve  the moles, 

   wrinkles and other minor details

The face manipulation technique behind DeepFaceLab is the Generative Adversarial

 Networks (GANs). The comparison between using GAN or not can be seen here:

Conclusions

    This is an easy to use and efficient solution to generating face-swapped content.

 DFL contributed to creating a state-of-the-art deep fake framework, having highly 

efficient tools for face swapping in videos, while keeping it open source since 2018.

 A few similar solutions are: Face2Face, Nirkin et al.’s automatic face swapping,

 DeepFakes, ZAO (mobile), FaceApp (mobile).

Bibliography

[1] https://arxiv.org/pdf/2005.05535v5.pdf

[2] https://github.com/iperov/DeepFaceLab