Grok is undressing kids — can the legislation cease it?

Grok started 2026 because it started 2025: underneath fireplace for its AI-generated pictures.

Elon Musk’s chatbot has spent the final week flooding X with nonconsensual, sexualized deepfakes of adults and minors. Circulating screenshots present Grok complying with requests to place actual ladies in lingerie and make them unfold their legs, and to place babies in bikinis. Studies of pictures that had been later eliminated describe much more egregious contents. One X consumer confirmed in a dialog with The Verge that they got here throughout a number of pictures of minors with what the prompter dubbed “donut glaze” on their faces, which seem to have since been eliminated. At one level, Grok was producing about one nonconsensual sexualized picture per minute, in keeping with one estimate.

X’s phrases of service prohibit “the sexualization or exploitation of youngsters.” And on Saturday, the corporate acknowledged the platform would “take motion towards unlawful content material on X, together with Baby Sexual Abuse Materials (CSAM).” It seems to have taken down among the worst offenses. However general, it’s downplayed the incidents. Musk has mentioned that “Anybody utilizing Grok to make unlawful content material will undergo the identical penalties as in the event that they add unlawful content material,” however he’s made it clear by public X posts that he doesn’t imagine the overall undressing prompts are an issue, and he’s responded to the broader matter with laughing and fireplace emojis on X. The corporate’s tepid response has alarmed specialists who’ve spent years attempting to deal with AI-powered sexual harassment and abuse. A number of governments have mentioned they’re scrutinizing X. However even amid an unprecedented push for on-line regulation, the trail towards policing it or its chatbot’s creations isn’t clear.

xAI, creator of Grok, didn’t reply to a request for remark. Neither did Apple or Google when requested if the experiences violated their app retailer insurance policies.

Grok has at all times allowed, and Musk has brazenly inspired, extremely sexualized imagery. However over the previous week, the flexibility to ask Grok to edit pictures — by way of a brand new button that enables adjustments with out the unique poster’s permission — has gone viral for undressing ladies and minors. Enforcement of guardrails has been haphazard at greatest, and many of the supposed responses from X come from Grok itself, which implies they’re essentially thought up on the spot. The replies embrace stating that a few of its creations went “towards our tips for fictional content material solely” and, on the request of a consumer, a broadly reported apology — one thing xAI itself doesn’t seem to have issued.

One of many greatest questions right here is whether or not the pictures violate legal guidelines towards CSAM and nonconsensual intimate imagery (NCII) of adults, particularly within the US, the place X is headquartered. The US Division of Justice proscribes “digital or laptop generated pictures indistinguishable from an precise minor” that embrace sexual exercise or suggestive nudity. And the Take It Down Act, signed into legislation by President Donald Trump in Could 2025, prohibits nonconsensual AI-generated “intimate visible depictions” and requires sure platforms to quickly take away them.

Celebrities and influencers have described feeling violated by sexualized AI-generated pictures; in keeping with screenshots, Grok has produced footage of the singer Momo from TWICE, actress Millie Bobby Brown, actor Finn Wolfhard, and plenty of extra. Grok-generated pictures are additionally getting used particularly to assault ladies with political energy.

“It’s a instrument for expressing the underlying misogyny that pervades each nook of American society and most societies all over the world,” Riana Pfefferkorn, a coverage fellow on the Stanford Institute for Human-Centered Synthetic Intelligence (HAI), instructed The Verge. “It’s a privateness violation, it’s a violation of consent and of boundaries, this can be very intrusive, it’s a type of gendered violence in its method.” Maybe above all, specific pictures of minors — together with by devoted “nudify” apps — have turn out to be a rising drawback for legislation enforcement.

On Monday, the Client Federation of America (CFA), a gaggle of tons of of consumer-focused nonprofits, publicly known as for each state and federal motion towards xAI for “creating and distributing Baby Sexual Abuse Materials (CSAM) and different non-consensual intimate imagery (NCII) with Generative AI,” sending a letter signed by a handful of organizations to the Federal Commerce Fee and US attorneys normal.

But the specifics of what’s prohibited by US legislation are “fairly murky,” Mary Anne Franks, a professor in mental property, expertise, and civil rights legislation on the George Washington College Regulation College, mentioned. “A part of what I’ve not been ready to determine both is … whether or not that is truly crossing the road into precise nudity and sexual conditions.”

Utilizing AI to generate a picture of an identifiable minor in a bikini (or probably even bare) — although unequivocally unethical — is probably not unlawful underneath present CSAM legal guidelines within the US, specialists instructed The Verge. That mentioned, pictures like those that seem to incorporate semen might violate each preexisting CSAM legal guidelines and the Take It Down Act — and Franks suspects these aren’t the worst offenses on the market. “We will think about that no matter’s hitting the mainstream media, there’s in all probability 1,000,000 worse issues that persons are additionally producing … Each potential immediate you might consider might be arising,” Franks mentioned.

However regardless of these federal legal guidelines and a plethora of state-level ones, specialists say it’s troublesome to implement bans on AI-generated sexual imagery proper now — and even tougher to find out what accountability platforms might have. “In the end there are conflicting legal guidelines, and there’s no authorized precedent” for a lot of it, Shael Norris, founding govt director of SafeBAE, a corporation working to finish sexual violence, instructed The Verge.

John Langford, a visiting scientific affiliate professor of legislation at Yale Regulation College and counsel at Shield Democracy, mentioned the patchwork of sexual deepfake bans stays sparsely examined in courtroom. “All that is form of new — we’re simply now beginning to develop case legislation on what falls the place,” mentioned Langford. However there’s some yardstick, at the very least: For the Grok creations that do depict identifiable minors, we do now have “precedent [that] any computer-generated picture of an actual baby that’s sexually specific is unlawful,” Drew Davis, SafeBAE’s director of strategic initiatives, mentioned.

There are a handful of present federal prosecutions for creating or possessing AI-modified pictures of actual kids, and several other dozen on the state degree, mentioned Pfefferkorn. “Relating to whether or not the businesses themselves are liable, that’s the place we’re, I believe, in uncharted territory,” Pfefferkorn mentioned.

Davis added that we’re “coping with an advanced authorized panorama in terms of AI-generated pictures of minors.” That’s partly as a result of the grace interval for the “take it down” portion of the Take It Down Act, during which platforms should reply to such content material, stands till Could.

Additionally, Part 230 has lengthy shielded corporations from legal responsibility for content material different individuals posted. However as corporations flip to bots like Grok to permit customers to generate their very own pictures, it’s unclear what legal responsibility they bear. “This is the reason I’m so to see if there’s going to be … artistic prosecution right here,” Franks mentioned, including, “It’s about whether or not or not, by advantage of making these pictures, they’ve violated the prison provision.”

The caveat, a number of specialists instructed The Verge, is that just about the entire prison statutes dictate that the offender needed to put up the content material with the data that it was going to trigger hurt. Yale Regulation’s Langford mentioned that half introduces “actually exhausting questions on whether or not you might maintain Grok or xAI liable.” However, others say, personhood is attributed to firms in different conditions — why not this one? Musk’s frequent, unfiltered posting additionally presents an uncommon type of perception.

Pfefferkorn believes this might be a “pivotal 12 months by way of combating this drawback” and mentioned that she wouldn’t be shocked if class-action lawsuits surfaced.

However to complicate issues even additional, Musk and X have shut ties to the present administration — Musk’s ostensibly defunct Division of Authorities Effectivity (DOGE) was at one level working throughout the FTC itself, the company tasked with implementing the Take It Down Act. Past the US, the Trump administration has used commerce talks to discourage different nations from regulating American web platforms. Musk and Trump are publicly on good phrases, and any nation that makes an attempt to punish X might probably face the administration’s ire, on prime of doubtless noncompliance from X itself.

Even so, a world backlash is constructing. Members of the French authorities mentioned they might investigate the matter. The Indian IT ministry ordered xAI to submit a report about how it might stop additional materials that’s “obscene, pornographic, vulgar, indecent, sexually specific, pedophilic, or in any other case prohibited underneath legislation.” And the Malaysian authorities’s Communications and Multimedia Fee mentioned it had “taken word with critical concern” of complaints about misuse of AI on X, significantly the “digital manipulation of pictures of ladies and minors to provide indecent, grossly offensive, or in any other case dangerous content material.”

Grok has persistently gone off the rails in generally weird and incessantly sexual methods, from its antisemitic breakdown to permitting individuals to create partially nude pictures of Taylor Swift. Outdoors specialists have expressed considerations about its slapdash security efforts — after the July 2025 launch of Grok 4, it took greater than a month for the corporate to launch a mannequin card outlining issues like security options and take a look at outcomes, sometimes seen as a naked minimal within the business.

With out outdoors stress, Grok’s deepfakes drawback appears unlikely to finish anytime quickly. A few of the most egregious pictures appear to be taken down after the actual fact. However the bigger guardrails, that are detailed in Grok 4.1’s mannequin card with a quick point out of CSAM, clearly aren’t working in addition to deliberate. And Musk’s current feedback recommend he doesn’t see a lot fallacious with the present state of Grok. One of the vital puzzling issues about the entire saga, Pfefferkorn mentioned, isn’t that an AI platform will be induced to create potential CSAM — it’s that “we now have not essentially seen, to this point, a number of concern about whether or not they’re arising proper near that line.”