AIMode.newsSearch
Live

Section

AI

AI breakthroughs, model releases, research, and industry moves.

Subscribe to this section →
Advertisement
AI

Agentic reliability and evaluations : Enterprises that got burned by a bad eval are the most likely to remove humans from the loop, not the least

Across 108 enterprises, trust in automated agent evaluation rose sharply in July — and the failure rate it is supposed to predict did not move at all. The share of organizations that fully trust automated evaluation nearly tripled, from 5% in June to 13%, and the complaint that evaluations don’t match real-world outcomes fell 10 points. Yet the same share as last month — just under half — shipped an agent that passed its evals and then failed a customer. The reason is visible in the cross-tabs: the new trust belongs almost entirely to enterprises that have not yet been burned. Among those that have, 4% trust automated evaluation; among those that haven’t, 24% do. And getting burned does not slow the march to autonomy — it speeds it up. This is the second wave of the VentureBeat Pulse Research agent reliability tracker, and the first fielded on an instrument identical to the month before it. That makes July the first read on direction rather than position: what moved, what held, and what the movement means. What moved is confidence. In June, only 5% of enterprises said they fully trusted automated evaluation, and the most-cited limitation was that evaluations align poorly with real-world outcomes (29%). In July, 13% fully trust automated evaluation and the alignment complaint has fallen to 19%, no longer the leading objection. Both shifts are large enough to read as real rather than noise. What held is the failure. Just under half of organizations (49%) deployed an agent or LLM feature in the past year that passed internal evaluations and then caused a customer-facing failure — statistically indistinguishable from June’s 50% — and a quarter (24%) have seen it happen more than once. Confidence improved; correctness did not. That is the July gap: not between autonomy and trust, as in June, but between trust and the evidence for it. The cross-tabs explain where the new confidence comes from, and it is not from better evaluations. Trust is concentrated almost entirely among enterprises that have not experienced a false-confidence failure: 24% of them fully trust automated evaluation, against 4% of those that have. The trust curve is being lifted by inexperience. Meanwhile the enterprises that have been burned are not retreating from autonomy — 85% of them already allow zero-human deployment or are engineering toward it, against 61% of those that have not been burned. Overall the autonomy trajectory is flat at 67%, but the population inside it has shifted toward the organizations with the most direct evidence that evaluations miss things. The vendor market, by contrast, is finally showing signs of settling. The share of enterprises running no dedicated evaluation tooling fell from 17% to 12%; specialist platforms gained, with Braintrust nearly doubling to 15% and DeepEval reaching 17%; and switching intent cooled, with those planning no change rising from 36% to 44%. Selection criteria moved with it: ease of integration overtook cost as the top factor…

VentureBeat ·

AI

Agentic security: Enterprises enforce agent permissions two-thirds of the time — and isolate high-risk agents less than one in five

Across 116 enterprises, agents are in production and so are the incidents: A majority have already had a confirmed agent security event or a near-miss. Two-thirds of enterprises enforce scoped permissions at runtime. Barely one in five isolates its highest-risk agents, making containment the weakest layer in the stack precisely as autonomy scales. Credential sharing persists across nearly two-thirds of agent fleets, and 53% have already had a confirmed agent security event or near-miss, contributing to a growing lack of confidence in agentic security. Security stacks remain overwhelmingly borrowed from model providers and hyperscalers, and confidence has slipped. Today, as many enterprises now believe AI-armed attackers are ahead of their defenses as believe the reverse. This wave of VentureBeat Pulse Research examines how enterprises secure their AI agents: what tooling they run, how they manage agent identity and isolation, what has already gone wrong, how much they spend, and whether they believe their defenses are keeping pace with AI-enabled attackers. Only 18% of enterprises isolate their highest-risk AI agents, even as 65% of enterprises enforce scoped permissions at runtime and 56% monitor and log agent activity. The gap between what enterprises watch and what they contain is the central finding of this wave of VentureBeat Pulse Research. More than half of enterprises (53%) have agentic AI systems in production today, and another 27% are piloting or running a limited rollout. The agentic security incidents are arriving with them: 53% of organizations have already had an agent security event, with 19% confirming an incident and 38% having identified a near-miss that was caught before it caused harm. The central finding is a containment gap. Enterprises have built the controls that watch and permission agents but not the one that bounds the damage when those fail. Among enterprises describing their security posture, 65% enforce scoped identities and permissions at runtime and 56% observe and log agent activity, yet only 18% isolate high-risk agents in sandboxes. Even among enterprises running agents in production, isolation is enforced just 21% of the time, and just 8% pair enforcement with isolation. That ordering is backward from a defense-in-depth standpoint. From SOC teams to CISOs, security teams know that observation tells you what happened and enforcement tries to prevent it, but isolation is what limits the blast radius when prevention fails. Identity has improved without being solved. 49% of enterprises say each of their agents has its own scoped, managed identity, but 63% report credential sharing somewhere in the agent fleet, and only 29% describe a fleet with scoped identities and no sharing anywhere. The security stack doing this work remains overwhelmingly hyperscaler or model provider-native: OpenAI’s guardrails (44%), Microsoft Azure (42%), Anthropic’s managed-agent controls (37%), and Google Cloud (31%) lead, and 92% of en…

VentureBeat ·