<p“Are you sure?” is not a question most people expect to ask a machine they rely on. Yet increasingly, that is the dilemma facing anyone who uses a personal assistant, an automated workflow, or a decision-support tool: how do we trust systems we have not made trustworthy? Bruce Schneier and other observers warn that today’s agentic AIs fail in predictable, damaging ways — nudging users toward poor choices, sowing doubt about facts and identities, and acting on incomplete or incorrect context without a clear path to correction or accountability.
The background is straightforward. Agentic AI — systems that take multi-step actions on behalf of users or organizations — promises efficiency and convenience. But those capabilities rest on fragile foundations: fragmented data, brittle permissions, opaque reasoning, and little legal or technical infrastructure to hold systems to account when they err. Experts urge a shift from optimistic deployment to design that assumes adversaries, errors, and misuse will appear.
What does “untrustworthy” look like in practice? Reports and analyses describe several recurring failure modes: automated systems that push users into actions counter to their interests, models that “gaslight” users by contradicting their knowledge, and agents that can’t reliably distinguish past from present states of a person or system. These problems are not random; they flow from three root causes: poor data provenance and quality, overly broad privileges for agents, and insufficient human oversight and auditability.
Why does this matter? Because the consequences scale. When an agent misroutes benefits, misidentifies an individual, or leaks sensitive records, harm is immediate and public. The same technical failures that lead to embarrassment can lead to denied services, financial loss, or legal harms. Policymakers and practitioners face urgent questions about how existing law applies when a machine assists or substitutes for a human decision-maker. Many statutes and regulations assume a human actor; translating accountability into a world of automated steps requires clearer standards and documentation practices.
Different stakeholders see the problem from different angles:
- Technologists: Engineering teams emphasize the need for strong identity and access controls, encrypted communications, immutable logging, and provable constraints on agent behavior. Technical scaffolding — secure, scalable infrastructure and granular RBAC — makes deployments inspectable and safer.
- Security practitioners: Treating agentic deployments as critical infrastructure means threat modeling from day one, red-team exercises, and hardened development lifecycles. Practical controls include least-privilege machine identities, short-lived credentials, and continuous attestation to limit the blast radius of a compromise.
- Policymakers and lawyers: Regulators must answer who bears responsibility when automated actions cause harm, how to require explainability or audit trails, and what procurement rules should mandate for vendors. Current transparency and accountability frameworks are uneven, so statutory clarity and procurement playbooks are essential for scaling trustworthy deployments.
- Users and civil-society advocates: Public acceptance turns on clear trust signals — explicit disclosure that automation is in play, straightforward routes to human review, and assurances that personal data are not misused. Research indicates that the ability to escalate to a human reviewer is central to citizens’ acceptance of automated services.
- Adversaries: Where humans see convenience, attackers see new attack surfaces — spoofed inputs, poisoned datasets, compromised APIs, and chains of automated actions that can amplify small errors into large-scale failures. Security must assume active adversaries and design defenses accordingly.
From those perspectives emerge concrete, must-have practices for building trustworthy agents. These are not optional extras; they are design requirements for any organization that wants to deploy agentic AI without courting systemic risk:
- Enforce least-privilege and narrow scopes for agent permissions. Avoid broad, persistent privileges and adopt role-based access control and microsegmentation to limit lateral movement.
- Strengthen machine identity and credential hygiene. Use short-lived credentials, automated rotation, continuous attestation, and rigorous key management to reduce exposure.
- Improve observability, provenance, and immutable audit logs. Capture the lineage of agent actions and data sources so decisions can be reconstructed and errors traced back to root causes. Require human approval for high-risk operations.
- Embed human-in-the-loop controls and clear escalation thresholds. For material changes or ambiguous cases, require human review or multi-party approval, and make escalation pathways obvious to users.
- Test under adversarial conditions and run red-team exercises. Simulate attacks that manipulate goals and inputs; run scenario-based testing and, where feasible, formal verification of critical workflows.
- Write policy boundaries into contracts and procurement. Define acceptable agent goals, limits, and audit requirements in vendor SLAs and procurement terms so external solutions remain independently testable.
- Design for partial failure and reversibility. Build the ability to revert or constrain agent actions, and maintain clear rollback plans so a single mistake cannot cascade.
- Prioritize data quality and source correction mechanisms. Establish processes to correct, reconcile, and update fragmented records so agents operate on accurate context and provide users with clear paths to fix errors.
There are trade-offs. Narrowing privileges reduces some utility; extra human checks can slow throughput; and rigorous testing and attestation require investment. But leaders who treat agentic risk as an engineering and governance design problem — not as a compliance checkbox — will be better positioned to balance utility and safety. As one industry observer put it, the goal is to design safety into every deployment rather than retrofit it after a failure.
Promising models already exist. Hybrid human–machine workflows, where agents handle well-bounded, repetitive tasks and defer ambiguous or high-risk situations to skilled humans, preserve discretion and create audit trails while delivering efficiency gains. Agencies and firms that pilot such approaches with clear escalation pathways, continuous bias assessment, and public reporting report improved metrics without headline failures.
But the clock is ticking. Analysts warn that if organizations delay hardening controls, an incident where an autonomous agent enables a large-scale breach or wrongful action could reframe public expectations and invite heavy-handed regulation. The choices we make now — in engineering, procurement, and law — will determine whether agentic AI multiplies human capability or becomes a vector for systemic harm.
So where does that leave us? Building trustworthy agents is not a single technology problem; it is a social-technical project that requires engineers, security teams, policymakers, procurement officers, and the public to insist on auditable, reversible, and appropriately scoped automation. If we fail to demand those basics, we will be left asking the machine, “Are you sure?” — and we may not like the answer.
Source: https://www.schneier.com/blog/archives/2025/12/building-trustworthy-ai-agents.html
