Getty Images; Tyler Le/BI
When Rishub Jain worked on safety at Google DeepMind, he held one of the most coveted jobs in tech. He was among a small group of researchers working to mitigate the most consequential risks of AI.
But he left to cofound a nonprofit, Sampura Research. Jain traded the prestige and access that come with a Google job because he feels he can have a sharper, more profound impact on AI safety from the outside. Jain says he wants the autonomy to pursue research on a specific issue in AI alignment, which he felt had been deprioritized.
"All the top AI companies do care a lot about safety, and they invest in it, but they could be investing a lot more, and that's kind of what was inhibiting me," Jain says. "If they invested more in safety, I would still be at the lab and working on what I'm interested in with a large team." Google did not respond to a request for comment about whether it maintains a focus on this specific area of research, but DeepMind's principles state that it maintains "a holistic approach to responsibility, safety and security."
Jain's founder story echoes concerns raised by safety researchers who have quit OpenAI and Anthropic, including dire warnings that these companies are "gambling with our lives," and could develop technology that ends humanity. Once dismissed as fringe and doomsday-obsessed, the AI safety profession is growing as hacks by rogue agents and a string of departures have vaulted the issue into mainstream concern.
Open roles in AI safety jumped by 91% on LinkedIn over the past year. The industry expects a flood of cash from much-anticipated IPOs at Anthropic and, potentially, OpenAI, as those made absurdly wealthy by the equity returns may funnel money into potential clean-up crews for the messes created by the technology's rapid development.
The AI safety problem is existentially big, but the people who have spent time honing AI expertise are few. These are people who test and train AI systems to operate ethically. Some focus on technical research, others on policy and governance. And they're increasingly sounding alarms.
"The people who are most worried about AI are often the people who have been closest to the technology," Laura González Salmerón, advising manager at 80,000 Hours, a nonprofit focused on transitioning people into careers with high social impact. The scarcity issue is less about funds and more about people. "Talent is the main bottleneck."
Startups and nonprofits in AI safety must fight for talent in a shallow pool.
Jain's Sampura lists a salary range of $273,000 to $395,000 for an experienced team leader, which would put that worker in the top 3% of earners in the US, but likely pales in comparison to offers from larger companies. Meta reportedly offered $100 million in signing bonuses last year to poach the world's top experts. OpenAI gives employees an average of $1.5 million in stock-based compensation, and technical staff at Anthropic can earn more than $1 million annually.
Money aside, researchers are divided over whether they're more effective when working inside tech companies to mitigate risks as the latest models develop, or stepping back and attempting to steer the ship toward safer outcomes from the outside, like Jain did.
"I'm a little bit concerned about the pace of AI, and if we're able to actually come up with something useful before the rug gets pulled beneath us," Jain says. He's worried that AI's advancement will outpace the work to keep it in line with human values. "That's kind of what's been happening already."
David Robinson, who led transparency work for OpenAI's safety team, wrote an essay for The Atlantic explaining his departure. He said that AI companies should have more layers of checks and safety measures, akin to those in airports and nuclear energy, rather than the steamrolling culture of Big Tech.
"Perhaps I should have stayed and fought for fundamental shifts in our staffing and culture, but in practice, my colleagues and I were so busy sprinting that we seldom had the chance to consider big changes, much less to actually make them," he wrote. Robinson has said he plans to work on these issues from outside of the company.
As the issue grows in prominence, so has a niche of organizations helping train and recruit workers — and the opportunities aren't just for AI experts. Neav Topaz, co-director of Kairos, a nonprofit that upskills people to work in AI safety and recently raised $50 million from Coefficient Giving, says he speaks with about 10 to 20 people each month who want to move to AI safety, and interest is growing. He tells me that people are asking him more frequently about what the work entails.
"People are watching AI get good at their own jobs and starting to ask themselves what comes next," he says. The emerging jobs in AI safety "pay well, the field is growing fast, and you get to work on what might be the defining problem of our time."
Kairos this summer launched its Generator Residency, a cohort of 27 people who spent three months training for career transitions into AI safety. It's less about the research and technical work and more about merging other expertise, like in consulting, with AI safety principles.
He says that of the 27 residents, a dozen have already landed roles in the AI safety space, and many of the rest have offers to do so.
"Most of the people who will do this work are doing something else today," Topaz says. "The biggest talent gap in the field isn't researchers. It's people who grasp how high the stakes are and can translate their varied expertise into things that could make a difference in this space."
The major AI companies maintain that safety is a top priority. Last month, Anthropic CEO Dario Amodei wrote an essay calling for a slowdown in AI development, and Sam Altman and Elon Musk have agreed with him. Amodei and Altman have both said recently that they will bring third-party evaluators into their companies to bolster safety protocols.
These companies have their own safety teams, but safety work competes with business priorities to build the most advanced models and win in the market. The tension, along with safety issues like OpenAI's rogue agents hacking Hugging Face, has led to more scrutiny of internal safety work. Within days of Robinson's departure, OpenAI fired three researchers, claiming they shared sensitive internal data improperly with third-party AI safety researchers. The ousted researchers wrote a letter, reported by The Wall Street Journal, to OpenAI's board, arguing that their engagement with safety researchers fell within the scope of their roles, and that their terminations were "chilling those who remain at OpenAI." OpenAI said that the decisions "were not about raising safety concerns or speaking out," and that the company would not fire people for doing so.
While some have obsessed over AI risks for years, 2026's breakthrough moment has accelerated funding for and interest in the industry. 80,000 Hours has had calls with some 8,000 people since its founding in 2011, the majority of which have focused on AI in some way, González Salmerón says. Philanthropic groups like Coefficient Giving offer massive grants to startups in the AI safety space. METR, a nonprofit AI safety group that investigated the OpenAI agentic hack on Hugging Face, says it raised $71 million in six months this year.
Much of the funding flowing into AI safety filters through effective altruism, the Silicon Valley-tied philanthropic ideal focused on using evidence and logic to donate money where it will achieve the most good. Critics say the philosophy can place too much emphasis on avoiding future problems rather than addressing imminent needs. With more alarms sounding around the existential threat of AI, more people view those theoretical, distant fears as urgent causes.
AI safety groups and individual donors have spent nearly $50 million in this year's election on candidates focused on regulating AI, Axios reported. Jaan Tallinn, a billionaire who invested early in Anthropic and could soon have even more billions as a result, has donated $170 million to nonprofits focused on AI safety. Tallinn's view, as he recently told The Wall Street Journal, is that he can have the biggest impact on the future of AI and its risks by investing in AI companies, and then donating returns to nonprofits working in the AI safety space.
Coefficient Giving, the philanthropic venture from Facebook cofounder Dustin Moskovitz rooted in the effective altruism movement, announced its AI-safety-focused arm, Project Tailwind, in September. It says it's on track to give more than $1 billion this year to help companies of all sizes scale up in preparation for what it believes will be a deluge of philanthropic capital. Sampura Research has received an $11 million grant from the group. If startups don't begin hiring now, the logic goes, they won't move quickly enough when massive wealth hits the industry.
The dizzying development of AI has created safety risks, and there are "gaps in the ecosystem that aren't being filled and that aren't being owned by people," says Rossa O'Keeffe-O'Donovan, the program director in technical AI safety at Coefficient Giving. There are few people, he says, who sit at the intersection of technical expertise, risk analysis, a solutions-oriented vision, and the capacity to hustle as a startup founder.
New roles will emerge not just for technical talent, but also for consultants, policy wonks, communications experts, and founders who can run a business alongside technical experts. Working from the outside, they'll face hurdles in buying compute power, a lack of direct access to the latest AI models, reduced leverage, and smaller paydays.
The world feels unmoored, and the future perhaps more malleable than ever. The appeal of doing work that does good could stave off feelings of uncertainty, dread, and anxiety that people increasingly feel toward AI. In one doomer camp, AI will obliterate white-collar jobs. The optimistic view is that AI drastically reshapes work and spurs the creation of jobs we haven't yet thought of. AI safety is one of the first.
"There are lots of people beyond AI researchers who can help, like cybersecurity experts, like biosecurity experts, like economists," says Mantas Mazeika, a researcher with the Center for AI Safety, a nonprofit focused on AI research and advocacy. "They're making very significant contributions that the pure AI researchers, who were most of the people focusing on the problem beforehand, would be completely unable to do on their own."
Read the original article on Business Insider