Use Machine Learning for Your Selfie-A-Day Series

Develop a small Python and OpenCV script to align all faces in a series of pictures automatically

Photo by Christopher Campbell on Unsplash

About a year ago, I saw this video:

Yes, I am not the first to notice this video and maybe I am a bit late to the party. Either way, I liked the idea. So I also started taking a selfie every day. After a few days, a question came to my mind. The eyes in the pictures are always in the same position. How can I achieve this?

Luckily, the friendly guy in the video got me covered:

Tutorial on how to centre pictures

After watching this video, I was terrified. “I don’t want to have so much manual labour with a little side project!” Luckily as programmer, I have the skills to outsource this task to a computer. Even better! I could just search on the internet if someone already solved this problem.

Surely there is someone how already built a program or a tutorial on how to do this.

At least this is what I thought. A bit of searching on the web showed there are a lot of tutorials on detecting eyes. E.g. this one https://datascienceplus.com/face-and-eye-detection-with-opencv/ (This website is just the first one to appear in google). But there was no “how-to” on aligning all eyes like it was done in the videos.

So I started writing a program. Today we will walk through the code and a few pitfalls you can avoid when you also want to write such a script.

TL;DR

The code is in GitHub, right here:

TerminatorMFX/Face-Alignment

Assumptions

For the script we need to make a few assumptions:

• At least one face is in each selfie
• At least two eyes are visible in each picture
• The face we want to centre is in near the middle
• The photos are taken as selfies, with one arm’s length distance

What we will learn

• Handling picture processing with OpenCV
• Picture transformations
• Use some simple geometric formulas

Let’s get started

Face and eye detection

First of all, we will have a look at the face detection. The OpenCV documentation inspires the algorithm. We are using a haar-cascade detection. The great thing is, openCV already comes with pertained classifiers. For eyes and for faces. We just have to load them.

First thing you may notice the image is converted to black and white (line 8). This happens since the classifier does not need the colour information. An aspect of the haar-cascade detection you can read about in the proposing paper is its possible speed. This would be great for real-time detection. But we don’t need a detection this fast. We have time, so we can analyze the pictures more precisely. The scale factor is set to a quite low value (line 9). This costs us a lot more computational power but also guarantees us better results.

Disclaimer for the used classification values. These values worked good for my pictures and their resolution. It is possible that you have to experiment a bit until you found the right parameters for your pictures.

In this snippet, we first load the classifier (lines 3–6). One for face and one for eye detection. To avoid false positives, we first look for the face. Then the rectangle with the face is analyzed and scanned for eyes.

Since we want one face and two eyes on each picture, an error is thrown if too many eyes are detected (lines 13, 22, 23, and 29).

As return, you can see two values.

• An image. Its sole purpose is debugging. It contains marks where the eyes and the face were detected. With this image, you can get start fine-tuning your detection.
• A list of coordinates, marking the midpoint of the eyes. These are important for our centring logic.

Centring logic

After we can detect eyes, the next goal is to centre the pictures. For this one, we need to go back to our school knowledge. We calculate the line between the two detected eyes. This straight should have a horizontal angle of 0°. The picture is rotated to achieve this angle:

Afterwards, the left eye is moved to always be on the same point. This happens via an affine transformation. Since we assume that every picture is taken from the same distance, these two transformations will do the trick and align both eyes.

Detection improvements

After a few test runs with the script, I recognized a problem. Often too many faces or eyes were found. Even after optimizing the classifier parameter, the problem did not disappear. To solve this problem, additional assumptions had to be made:

• The face for centring is near the middle of the picture
• The eyes are in the upper half of the image and also near the vertical axis of the image

With these two assumptions, I extended the code with an evaluation.

If more than one face is found, the one which is nearest to the centre of the picture is selected.

We are looking only for eyes in the upper half of the face. Additionally, if more than two eyes are found, only the two eyes nearest to the vertical middle of the picture are considered to be the eyes.

Batch processing

After we built a script which can align a single face, we need to build a batch processing. Since the image reading and writing seems to be quite trivial, I spare you the details. You can find my implementation right here:

TerminatorMFX/Face-Alignment

Possible improvements

The code is working just fine. But there is always the possibility for improvements.

• Currently, there are quite a lot of hardcoded values in the code.
• Pictures with sunglasses cannot be automatically aligned.
• Naming, you can always argue about naming.

Thanks for reading, and have fun with your own picture-a-day-series!

AI In Plain English

Enjoyed this article? If so, get more similar content by subscribing to our YouTube channel!