An OpenAI security analysis lead departed for Anthropic

Probably the most controversial points within the AI trade over the previous yr was what to do when a person shows indicators of psychological well being struggles in a chatbot dialog. OpenAI’s head of that kind of security analysis, Andrea Vallone, has now joined Anthropic.

”Over the previous yr, I led OpenAI’s analysis on a query with nearly no established precedents: how ought to fashions reply when confronted with indicators of emotional over-reliance or early indications of psychological well being misery?” Vallone wrote in a LinkedIn put up a few months in the past.

Vallone, who spent three years at OpenAI and constructed out the “mannequin coverage” analysis group there, labored on the way to finest deploy GPT-4, OpenAI’s reasoning fashions, and GPT-5, in addition to growing coaching processes for a few of the AI trade’s hottest security strategies, comparable to rule-based rewards. Now, she’s joined the alignment group at Anthropic, a bunch tasked with understanding AI fashions’ largest dangers and the way to handle them.

Vallone shall be working beneath Jan Leike, the OpenAI security analysis lead who departed the corporate in Might 2024 resulting from issues that OpenAI’s “security tradition and processes have taken a backseat to shiny merchandise.”

Main AI startups have more and more incited controversy over the previous yr over customers’ struggles with psychological well being, which may spiral deeper after confiding in AI chatbots, particularly since security guardrails have a tendency to interrupt down in longer conversations. Some teenagers have died by suicide, or adults have dedicated homicide, after confiding within the instruments. A number of households have filed wrongful loss of life fits, and there was at the very least one Senate subcommittee listening to on the matter. Security researchers have been tasked with addressing the issue.

Sam Bowman, a pacesetter on the alignment group, wrote in a LinkedIn put up that he was “pleased with how significantly Anthropic is taking the issue of determining how an AI system ought to behave.”

In a LinkedIn put up on Thursday, Vallone wrote that she’s “wanting to proceed my analysis at Anthropic, specializing in alignment and fine-tuning to form Claude’s conduct in novel contexts.”