Scale AI

Some of the most exciting uses of AI are where the impact can benefit millions of people. 🌍 Through our multi-year agreement with Qatar's Ministry of Communications and Information Technology, Scale is one of the partners helping advance AI adoption across government entities. From classrooms to legislative drafting, AI is being applied across public sector organizations with very different needs, workflows, and challenges. What does that look like in practice? Explore how AI is being used to better serve a nation: https://lnkd.in/eZucim2S

A great pianist can play almost anything, but mastering one piece is what makes a performance exceptional. The same goes for AI. With today's launch of Nemotron 3.5 Super VL, we’re collaborating with NVIDIA AI to combine their open post-training tools with Scale’s data, evaluation and applications building expertise. Together, we're turning general-purpose models into powerful, cost-effective specialists for enterprises everywhere. Welcome to the multi-model world. 🤝 https://lnkd.in/eHvyssjU

Today is World Mental Health Day. Millions of people struggle with mental illness. Some now turn to AI in their hardest moments. When they do, they deserve a response that is safe, compassionate and helpful. That’s why I’m so grateful to the clinicians who helped us understand what that response should look like. Mental health professionals show up when people need someone most. They wrote the crisis scenarios for our DistressBench research, and defined what a helpful response looks like. Their standards come from years of sitting with people through their most difficult moments. AI can’t replace human care. But it can learn from the people who provide it, so that anyone who reaches out is met with understanding and guided toward real support. You can read about this work in the TIME article https://lnkd.in/e29CxruN To every clinician and researcher who made this possible: thank you. And to anyone who’s struggling right now: you’re not alone, and you deserve support. https://lnkd.in/e29CxruN

Recognizing a mental health crisis isn’t the same as helping someone through it. NEWS: Today, Scale AI is introducing DistressBench, a clinician-authored benchmark that tests whether AI models provide the help called for when someone is facing a suicide or self-harm crisis. We evaluated 25 frontier models using 718 clinician-written conversations, each with specific criteria for what a helpful response needs to include. The findings show a gap between recognizing danger and responding appropriately: • In more than a third of conversations where models recognized a crisis, they still failed to provide help or resources. • On average, models left out nearly 3 in 10 required response steps. • When shown their own failed response alongside a better one, most models identified the better response but still declined to provide it. That last finding is especially striking. A model can recognize a more helpful answer and still fail to deliver it. Our hope is that benchmarks like this give researchers concrete evidence of where responses fall short and what needs to improve. People will increasingly turn to AI in their hardest moments - we need safety testing that keeps up with what people are asking of these systems.  https://lnkd.in/gSRNJFQ2

People can read between the lines: whether something will fit in a space, or what’s likely to happen next. For leading AI models, that remains a major challenge. We’re introducing Humanity’s Sixth Sense in partnership with Elorian AI, a new benchmark measuring how well AI can make sense of the world from visual context. Here’s what we learned: 💡 More thinking doesn’t always help. Giving models more time to reason doesn’t consistently improve performance and can sometimes make it worse. 💡 The challenge is understanding what they see. 94% of failures came from perception or latent inference, showing that models often miss the visual cue that matters or misinterpret who or what they’re looking at. 💡 Social intuition is especially difficult. Understanding roles, relationships, emotions, and unspoken dynamics was the weakest area for most models. As AI takes on more work in the physical world, it will need to understand the context people intuitively grasp. HSS gives us a way to measure this gap and a clearer path toward AI that can better understand and navigate the world around us. More on our findings: https://lnkd.in/eZWEKCMT

As AI agents take on more complex work, they need clear boundaries for how they operate. We’re bringing NVIDIA’s Open Agent Safety Platform technologies into Scale GenAI Portfolio to help build, train, deploy, run, and evaluate AI agents with key protections from the start: ☑️ Isolation to contain where agents operate ☑️ Security policies to govern what agents can access and do ☑️ Audit records to make agent actions traceable Together, these controls offer a foundation for building and deploying agents with clear operating boundaries, without having to assemble the underlying security infrastructure. More about our collaboration: https://lnkd.in/g7jnV79J

Our AI future should not be steered by speculative warnings. We have to be driven by evidence. That was the clearest takeaway from this week's AI conversations at the UN General Assembly. Governments need to get serious about testing frontier AI, and nowhere is that more urgent than in Washington. Getting serious means three things: → Clear responsibility for testing across agencies → Access to models before they're deployed → Funding for the experts and tools to do the work Scale AI's cyber research shows what testing can reveal. We've seen AI agents carry out an attacker's instructions while still finishing the user's task, so the user never knew anything went wrong. We've also seen models refuse to help people defend their own systems. Both are failures. Real testing has to measure the harm AI can cause and the help it fails to give. Government shouldn't have to rely solely on AI developers to explain these failures. It needs its own testing capacity, with the expertise to ask hard questions and the tools to answer them. Scale has worked with the U.S. government for years to test the risks and capabilities of AI models, and we're accelerating that work in the months ahead. The nation that can measure the frontier will be the one that steers it. Here's what we think America should do next: https://lnkd.in/eH9mEPPA