Facebook's Recent Outage Shows How the Platform's AI Tags Photos

If you’re in the digital marketing space, you’re probably already aware of the Facebook and Instagram outage that occurred last week. 

According to Facebook, a “routine maintenance operation” triggered a bug that affected image viewing and uploading for both platforms. 

With photos down, users couldn’t help but notice that photos were replaced with identifying “Tags” assigned by Facebook’s machine vision systems. 

Oh yeah! I forgot Facebook uses machine learning to tag our photos with what it sees in the picture.

To be fair, "one person, beard" is pretty much a spot-on description of me. pic.twitter.com/fCpydUxtpz

— Zack Whittaker (@zackwhittaker) July 3, 2019

This provided a “peek behind the curtain” on how Facebook’s systems identify and categorize photos...and what data Facebook can pull from the photos you post. 

How does Facebook tag photos? 

Instead of seeing photos of your friends and family, users saw the photos replaced with this: 

Apparently, Facebook’s system has gotten pretty good at identifying elements from an image. From what’s in it (people, animals) to the setting (indoor, outdoor, beach) to other specifics (shoes, flower, cloud), Facebook can clearly pull a lot of information from user photos. 

Other users saw Facebook’s facial recognition systems picking up who exactly was in the photo, even if the users were not tagged. 

Not only is this interesting to see from the perspective of how far machine learning has come, but the outage also sheds light on how visually-disabled users experience the site with screen readers. 

(So, if you saw Facebook’s tags on your company page that were unclear, it might be time to update your tags manually to make the experience better for visually-impaired users engaging with your page via screen readers. 

For example, if you have information about an event with a URL or other information, and that’s not represented in the alt-text, visually impaired users can’t access that information. In order to be inclusive for all, it’s important to consider that any information that can be seen visually can also be relayed to those with impairments.) 

How much data does Facebook have on us? 

As you know, Facebook has come under fire several times for its data collection practices, specifically on third-party networks. 

However, even if you’ve locked down your online activity with Facebook’s Clear History feature or Firefox’s recent security update, these AI tags shed light on how Facebook can use these tags to learn more about you. 

It’s currently unclear if these photo tags are specifically used for ad targeting, but it is known that on-Facebook activity such as your occupation, age, location or interests are more or less fair game for ad targeting purposes. Thus, it’s not unlikely that the data Facebook’s AI pulls from your photos is also factored into ad targeting as well. 

It might seem small, but with these tags, Facebook can pull a lot of additional information about you that you might have otherwise thought to be private. Things like:

• If you have a pet 
• Any notable hobbies 
• How frequently you vacation and where you go
• Size of family 
• Significant life events (like getting married, having a baby, or buying a home). 

While the marketer in me thinks this is pretty cool from a targeting perspective, the Facebook user in me is frankly a little unsettled. 

At first glance, this peek behind the scenes may seem like just a small bug. But, looking a bit deeper, it’s very telling of how powerful machine learning has become in the digital space, and how much these AI systems can extract about you from just a simple photo. 

All of this strengthens Facebook’s ability to use data it collects on and off the platform to piece together its version of who “you” are as both a consumer and a person.

For brands, this shows that there is a lot more going on “under the hood” on these platforms than we may be aware of. Machine learning is quite literally growing more powerful each day as it gets new data to study. 

For that reason, social media offers marketers so much more than just a large audience for potential exposure. It’s advanced algorithms — creepy or not — can look at more than just who fits exactly in your demographic. They can understand and anticipate the wants, needs, and lifestyle of your target consumer.