Find Out What Facebook Thinks Are In Your Photos

Facebook have some awesome in house tech and they are just one of many internet companies investing heavily in Artificial Intelligence.

Since April 2016, Facebook's AI Research Team (FAIR) have been using image recognition to automatically work out exactly what is in your uploaded photographs.  They have then been adding alt tags to those images which are populated with keywords representing the content of your images.

So whenever you upload a photograph to your account, Facebook will try to detect what objects are in the photo and generally try to make sense of it.

Whatever you are doing, whether you're eating at a restaurant, spending time with your little ones, or running a marathon, Facebook's algorithms will accurately figure out what's going on in your photograph and generate a caption to go with it.

By extracting data in this way, Facebook can learn a lot about you, as there is a suprising amount of data that can be extracted from a photograph.

So just from your photographs, Facebook can tell whether you have a dog, whether you have children, what your hobbies are and more. They could then use that information to show relevant ads.

This is not necessarily a bad thing and companies use targeted advertising routinely.

Currently though, Facebook are automatically generating these captions to make images more accessible to blind people, which I think is a fantastic idea.

So what do these captions look like? Well take a look at the image below.  Below this beautiful image, you can see in Chrome Dev tools that Facebook has labeled this image as 

"Image may contain: mountain, sky, cloud, house, outdoor, nature, and water"

I think this is an amazing description considering it has been generated automatically by a computer.

How to find out what Facebook thinks are in your photos!

Well if you're curious to find out what Facebook thinks of your photos, follow these steps:

1. Logon to Facebook and using Chrome head over to your photos.  You can do this in any modern browser by the way as they all have Dev tools, but I personally prefer the Dev tools in Chrome.
2. Choose a photo and click the thumbnail to view a large version of your chosen image.
3. Right click the image and choose "Inspect". This will open the Chrome Dev Tools.
4. Find the image tag (<img>) and look at the "alt" attribute of the image tag. This will contain the machine generated description you are looking for.

Facebook’s computer vision algorithms are able to detect multiple object classes in the scene (as in the example above) and can include food, objects (such as spectacles), facial expressions, sports, nature (i.e. sky and mountains) and much more.

But probably the spookiest part for me is that Facebook can even count the number of people in a group photograph, which I think is pretty mind blowing, and there's some serious neural network at work there.

If looking at the above code scares you, don't worry as there is a Google Chrome extension and a Firefox add-on created by a GitHub user which you can download and i will all the hard work for you and overlay the tags on your images.

If code doesn't make you run a mile in the opposite direction, then you can learn how to develop a deep convolutional neural network and implement this kind of system yourself.