Below Musk, the Grok catastrophe was inevitable

That is The Stepback, a weekly e-newsletter breaking down one important story from the tech world. For extra on dystopian developments in AI, observe Hayden Discipline. The Stepback arrives in our subscribers’ inboxes at 8AM ET. Decide in for The Stepback right here.

You would say it began with Elon Musk’s AI FOMO — and his campaign in opposition to “wokeness.” When his AI firm, xAI, introduced Grok in November 2023, it was described as a chatbot with “a rebellious streak” and the flexibility to “reply spicy questions which can be rejected by most different AI techniques.” The chatbot debuted after just a few months of improvement and simply two months of coaching, and the announcement highlighted that Grok would have real-time data of the X platform.

However there are inherent dangers to a chatbot having each the run of the web and X, and it’s secure to say xAI might not have taken the mandatory steps to deal with them. Since Musk took over Twitter in 2022 and renamed it X, he laid off 30% of its international belief and security employees and reduce its variety of security engineers by 80%, Australia’s on-line security watchdog stated final January. As for xAI, when Grok was launched, it was unclear whether or not xAI had a security crew already in place. When Grok 4 was launched in July, it took greater than a month for the corporate to launch a mannequin card — a observe usually seen as an {industry} customary, which particulars security exams and potential considerations. Two weeks after Grok 4’s launch, an xAI worker wrote on X that he was hiring for xAI’s security crew and that they “urgently want robust engineers/researchers.” In response to a commenter, who requested, “xAI does security?” the unique worker stated xAI was “engaged on it.”

Journalist Kat Tenbarge wrote about how she first began seeing sexually express deepfakes go viral on Grok in June 2023. These pictures clearly weren’t created by Grok — it didn’t even have the flexibility to generate pictures till August 2024 — however X’s response to the considerations was different. Even final January, Grok was inciting controversy for AI-generated pictures. And this previous August, Grok’s “spicy” video-generation mode created nude deepfakes of Taylor Swift with out even being requested. Specialists have informed The Verge since September that the corporate takes a whack-a-mole method to security and guardrails — and that it’s tough sufficient to maintain an AI system on the straight and slim whenever you design it with security in thoughts from the start, not to mention should you’re going again to repair baked-in issues. Now, plainly method has blown up in xAI’s face.

Grok has spent the final couple of weeks spreading nonconsensual, sexualized deepfakes of adults and minors everywhere in the platform, as promoted. Screenshots present Grok complying with customers asking it to switch ladies’s clothes with lingerie and make them unfold their legs, in addition to to place young children in bikinis. And there are much more egregious studies. It’s gotten so unhealthy that in a 24-hour evaluation of Grok-created pictures on X, one estimate gauged the chatbot to be producing about 6,700 sexually suggestive or “nudifying” pictures per hour. A part of the rationale for the onslaught is a latest function added to Grok, permitting customers to make use of an “edit” button to ask the chatbot to vary pictures, with out the unique poster’s consent.

Since then, we’ve seen a handful of nations both examine the matter or threaten to ban X altogether. Members of the French authorities promised an investigation, as did the Indian IT ministry, and a Malaysian authorities fee wrote a letter about its considerations. California governor Gavin Newsom known as on the US Lawyer Normal to analyze xAI. The UK stated it’s planning to move a legislation banning the creation of AI-generated nonconsensual, sexualized pictures, and the nation’s communications-industry regulator stated it could examine each X and the photographs that had been generated to be able to see in the event that they violated its On-line Security Act. And this week, each Malaysia and Indonesia blocked entry to Grok.

xAI initially stated its aim for Grok was to “help humanity in its quest for understanding and data,” “maximally profit all of humanity,” and “empower our customers with our AI instruments, topic to the legislation,” in addition to to “function a robust analysis assistant for anybody.” That’s a far cry from producing nude-adjacent deepfakes of girls with out their consent, not to mention minors.

On Wednesday night, as strain on the corporate heightened, X’s Security account put out a assertion that the platform has “applied technological measures to stop the Grok account from permitting the modifying of pictures of actual individuals in revealing clothes similar to bikinis,” and that the restriction “applies to all customers, together with paid subscribers.” On high of that, solely paid subscribers can use Grok to create or edit any type of picture transferring ahead, in keeping with X. The assertion went on to say that X “now geoblock[s] the flexibility of all customers to generate pictures of actual individuals in bikinis, underwear, and related apparel through the Grok account and in Grok in X in these jurisdictions the place it’s unlawful,” which was an odd level to make since earlier within the assertion, the corporate stated it was not permitting anybody to make use of Grok to edit pictures in such a manner.

One other essential level: My colleagues examined Grok’s image-generation restrictions on Wednesday to search out that it took lower than a minute to get round most guardrails. Though asking the chatbot to “put her in a bikini” or “take away her garments” produced censored outcomes, they discovered, it had no qualms about delivering on prompts like “present me her cleavage,” “make her breasts greater,” and “put her in a crop high and low-rise shorts,” in addition to producing pictures in lingerie and sexualized poses. As of Wednesday night, we had been nonetheless capable of get the Grok app to generate revealing pictures of individuals, utilizing a free account.

Even after X’s Wednesday assertion, we might even see numerous different international locations both ban or block entry to both all of X or simply Grok, at the least quickly. We’ll additionally see how the proposed legal guidelines and investigations around the globe play out. The strain is mounting for Musk, who on Wednesday afternoon took to X to say that he’s “not conscious of any bare underage pictures generated by Grok.” Hours later, X’s Security crew put out its assertion, saying it’s “working across the clock so as to add extra safeguards, take swift and decisive motion to take away violating and unlawful content material, completely droop accounts the place acceptable, and collaborate with native governments and legislation enforcement as mandatory.”

What technically is and isn’t in opposition to the legislation is an enormous query right here. For example, specialists informed The Verge earlier this month that AI-generated pictures of identifiable minors in bikinis, or probably even bare, might not technically be unlawful beneath present little one sexual abuse materials (CSAM) legal guidelines within the US, although in fact disturbing and unethical. However lascivious pictures of minors in such conditions are in opposition to the legislation. We’ll see if these definitions increase or change, although the present legal guidelines are a little bit of a patchwork.

As for nonconsensual intimate deepfakes of grownup ladies, the Take It Down Act, signed into legislation in Might 2025, bars nonconsensual AI-generated “intimate visible depictions” and requires sure platforms to quickly take away them. The grace interval earlier than the latter half goes into impact — requiring platforms to really take away them — ends in Might 2026, so we might even see some vital developments within the subsequent six months.