Celebrity Deepfake Porn Cases Will Be Investigated by Meta Oversight Board

As AI tools become increasingly sophisticated and accessible, so too has one of its worst applications: non-consensual deepfake pornography. While much of this content is hosted on dedicated sites, more and more it’s finding its way onto social platforms. Today, the Meta Oversight Board announced that it was taking on cases that could force the company to reckon with how it deals with deepfake porn.

The board, which is an independent body that can issue both binding decisions and recommendations to Meta, will focus on two deepfake porn cases, both regarding celebrities who had their images altered to create explicit content. In one case about an unnamed American celebrity, deepfake porn depicting the celebrity was removed from Facebook after it had already been flagged elsewhere on the platform. The post was also added to Meta’s Media Matching Service Bank, an automated system that finds and removes images that have already been flagged as violating Meta’s policies, to keep it off the platform.

In the other case, a deepfake image of an unnamed Indian celebrity remained up on Instagram, even after users reported it for violating Meta’s policies on pornography. The deepfake of the Indian celebrity was removed once the board took up the case, according to the announcement.

In both cases, the images were removed for violating Meta’s policies on bullying and harassment, and did not fall under Meta’s policies on porn. Meta, however, prohibits “content that depicts, threatens or promotes sexual violence, sexual assault or sexual exploitation” and does not allow porn or sexually explicit ads on its platforms. In a blog post released in tandem with the announcement of the cases, Meta said it removed the posts for violating the “derogatory sexualized photoshops or drawings” portion of its bullying and harassment policy, and that it also “determined that it violated [Meta’s] adult nudity and sexual activity policy.”

The board hopes to use these cases to examine Meta’s policies and systems to detect and remove nonconsensual deepfake pornography, according to Julie Owono, an Oversight Board member. “I can tentatively already say that the main problem is probably detection,” she says. “Detection is not as perfect or at least is not as efficient as we would wish.”

Meta has also long faced criticism for its approach to moderating content outside the US and Western Europe. For this case, the board already voiced concerns that the American celebrity and Indian celebrity received different treatment in response to their deepfakes appearing on the platform.

“We know that Meta is quicker and more effective at moderating content in some markets and languages than others. By taking one case from the United States and one from India, we want to see if Meta is protecting all women globally in a fair way,” says Oversight Board cochair Helle Thorning-Schmidt. “It’s critical that this matter is addressed, and the board looks forward to exploring whether Meta’s policies and enforcement practices are effective at addressing this problem.”