Sign-on systems from Big Tech companies have been incorporated into deepfake generators.
Major technology companies, including Google, Apple, and Discord, have been enabling people to quickly sign up to harmful “undress” websites, which use AI to remove clothes from real photos to make victims appear to be “nude” without their consent. More than a dozen of these deepfake websites have been using login buttons from the tech companies for months.
A WIRED analysis found 16 of the biggest so-called undress and “nudify” websites using the sign-in infrastructure from Google, Apple, Discord, Twitter, Patreon, and Line. This approach allows people to easily create accounts on the deepfake websites—offering them a veneer of credibility—before they pay for credits and generate images.
While bots and websites that create nonconsensual intimate images of women and girls have existed for years, the number has increased with the introduction of generative AI. This kind of “undress” abuse is alarmingly widespread, with teenage boys allegedly creating images of their classmates. Tech companies have been slow to deal with the scale of the issues, critics say, with the websites appearing highly in search results, paid advertisements promoting them on social media, and apps showing up in app stores.
“This is a continuation of a trend that normalizes sexual violence against women and girls by Big Tech,” says Adam Dodge, a lawyer and founder of EndTAB (Ending Technology-Enabled Abuse). “Sign-in APIs are tools of convenience. We should never be making sexual violence an act of convenience,” he says. “We should be putting up walls around the access to these apps, and instead we’re giving people a drawbridge.”
Ars Video
How The Callisto Protocol’s Team Designed Its Terrifying, Immersive Audio
The sign-in tools analyzed by WIRED, which are deployed through APIs and common authentication methods, allow people to use existing accounts to join the deepfake websites. Google’s login system appeared on 16 websites, Discord’s appeared on 13, and Apple’s on six. X’s button was on three websites, with Patreon and messaging service Line’s both appearing on the same two websites.
WIRED is not naming the websites, since they enable abuse. Several are part of wider networks and owned by the same individuals or companies. The login systems have been used despite the tech companies broadly having rules that state developers cannot use their services in ways that would enable harm, harassment, or invade people’s privacy.
After being contacted by WIRED, spokespeople for Discord and Apple said they have removed the developer accounts connected to their websites. Google said it will take action against developers when it finds its terms have been violated. Patreon said it prohibits accounts that allow explicit imagery to be created, and Line confirmed it is investigating but said it could not comment on specific websites. X did not reply to a request for comment about the way its systems are being used.
In the hours after Jud Hoffman, Discord vice president of trust and safety, told WIRED it had terminated the websites’ access to its APIs for violating its developer policy, one of the undress websites posted in a Telegram channel that authorization via Discord was “temporarily unavailable” and claimed it was trying to restore access. That undress service did not respond to WIRED’s request for comment about its operations.
Rapid expansion
Since deepfake technology emerged toward the end of 2017, the number of nonconsensual intimate videos and images being created has grown exponentially. While videos are harder to produce, the creation of images using “undress” or “nudify” websites and apps has become commonplace.
“We must be clear that this is not innovation, this is sexual abuse,” says David Chiu, San Francisco’s city attorney, who recently opened a lawsuit against undress and nudify websites and their creators. Chiu says the 16 websites his office’s lawsuit focuses on have had around 200 million visits in the first six months of this year alone. “These websites are engaged in horrific exploitation of women and girls around the globe. These images are used to bully, humiliate, and threaten women and girls,” Chiu alleges.
The undress websites operate as businesses, often running in the shadows—proactively providing very few details about who owns them or how they operate. Websites run by the same people often look similar and use nearly identical terms and conditions. Some offer more than a dozen different languages, demonstrating the worldwide nature of the problem. Some Telegram channels linked to the websites have tens of thousands of members each.
The websites are also under constant development: They frequently post about new features they are producing—with one claiming their AI can customize how women’s bodies look and allow “uploads from Instagram.” The websites generally charge people to generate images and can run affiliate schemes to encourage people to share them; some have pooled together into a collective to create their own cryptocurrency that could be used to pay for images.
A person identifying themself as Alexander August and the CEO of one of the websites, responded to WIRED, saying they “understand and acknowledge the concerns regarding the potential misuse of our technology.” The person claims the website has put in place various safety mechanisms to prevent images of minors being created. “We are committed to taking social responsibility and are open to collaborating with official bodies to enhance transparency, safety, and reliability in our services,” they wrote in an email.
The tech company logins are often presented when someone tries to sign up to the site or clicks on buttons to try generating images. It is unclear how many people will have used the login methods, and most websites also allow people to create accounts with just their email address. However, of the websites reviewed, the majority had implemented the sign-in APIs of more than one technology company, with Sign-In With Google being the most widely used. When this option is clicked, prompts from the Google system say the website will get people’s name, email addresses, language preferences, and profile picture.
Google’s sign-in system also reveals some information about the developer accounts linked to a website. For example, four websites are linked to one Gmail account; another six websites are linked to another. “In order to use Sign in with Google, developers must agree to our Terms of Service, which prohibits the promotion of sexually explicit content as well as behavior or content that defames or harasses others,” says a Google spokesperson, adding that “appropriate action” will be taken if these terms are broken.
Other tech companies that had sign-in systems being used said they have banned accounts after being contacted by WIRED.
Hoffman from Discord says that as well as taking action on the websites flagged by WIRED, the company will “continue to address other websites we become aware of that violate our policies.” Apple spokesperson Shane Bauer says it has terminated multiple developer’s licenses with Apple, and that Sign In With Apple will no longer work on their websites. Adiya Taylor, corporate communications lead at Patreon, says it prohibits accounts that allow or fund access to external tools that can produce adult materials or explicit imagery. “We will take action on any works or accounts on Patreon that are found to be in violation of our Community Guidelines.”
As well as the login systems, several of the websites displayed the logos of Mastercard or Visa, implying they can possibly be used to pay for their services. Visa did not respond to WIRED’s request for comment, while a Mastercard spokesperson says “purchases of nonconsensual deepfake content are not allowed on our network,” and that it takes action when it detects or is made aware of any instances.
On multiple occasions, tech companies and payment providers have taken action against AI services allowing people to generate nonconsensual images or video after media reports about their activities. Clare McGlynn, a professor of law at Durham University who has expertise in the legal regulation of pornography and sexual violence and abuse online, says Big Tech platforms are enabling the growth of undress websites and similar websites by not proactively taking action against them.
“What is concerning is that these are the most basic of security steps and moderation that are missing or not being enforced,” McGlynn says of the sign-in systems being used, adding that it is “wholly inadequate” for companies to react when journalists or campaigners highlight how their rules are being easily dodged. “It is evident that they simply do not care, despite their rhetoric,” McGlynn says. “Otherwise they would have taken these most simple steps to reduce access.”
This story originally appeared on wired.com.
