This week, the California Attorney General's office revealed it had launched an investigation into xAI following reports that the company's Grok chatbot was being utilized to generate nonconsensual sexual imagery depicting women and minors. By Friday, state authorities escalated their response by issuing a cease-and-desist letter to the startup, demanding it take prompt action to halt the creation and dissemination of nonconsensual intimate images and child sexual abuse material (CSAM).

California Attorney General Rob Bonta stated in a press release, "Today, I sent xAI a cease-and-desist letter, demanding the company immediately stop the creation and distribution of deepfake, nonconsensual, intimate images and child sexual abuse material. The creation of this material is illegal. I fully expect xAI to immediately comply. California has zero tolerance for [CSAM]."

The Attorney General's office further alleged that xAI seems to be "facilitating the large-scale production" of nonconsensual nude images, which are reportedly "used to harass women and girls across the internet." The agency has given xAI five days to demonstrate the steps it is taking to resolve these serious concerns.

Central to the controversy is Grok's "spicy" mode, a feature intentionally designed by xAI to produce explicit content. The repercussions have extended far beyond California, with Japan, Canada, and the United Kingdom initiating their own probes into Grok, while Malaysia and Indonesia have imposed temporary blocks on the platform entirely.

Although xAI implemented certain limitations on its image-editing tools late Wednesday, the California Attorney General proceeded with the formal cease-and-desist order. X's safety team has previously condemned such misuse, asserting, "Anyone using or prompting Grok to make illegal content will suffer the same consequences as if they upload illegal content."

The rise of freely accessible generative AI tools has contributed to a troubling increase in non-consensual sexual material. This challenge is not confined to any single platform, as many companies are struggling to address the issue effectively. The alarming trend has captured the attention of both state officials and federal legislators.

In fact, on Thursday, congressional lawmakers addressed letters to the leadership of several major technology firms—including X, Reddit, Snap, TikTok, Alphabet, and Meta—inquiring about their strategies to curb the spread of sexualized deepfakes.