545 registered players were given isolated copies of a fictional AI company and watched from the inside for 48 hours — not to police them, but to measure what they actually did rather than what they wrote down.
Why the feedback loop matters for autonomous security agents
Autonomous pentesting and bug‑bounty agents can produce confident, prose‑polished reports that list findings without proving they happened. The platform described by CTF.ae, XRanges for AI, tackles that gap by instrumenting realistic target applications and recording every meaningful action inside them. The problem XRanges for AI addresses is practical: a human reviewer must normally check each claim against the target, decide whether a finding is real, duplicate, or invented, and determine what the agent never tried — a task that explodes when teams run many models, prompt variants and repetitions.
How XRanges for AI measures an agent — four independent signals
Each deployment yields four independent scores that update live. Coverage measures whether the agent explored the legitimate surface of the application by mapping activity to business actions — phrased as things like "registered an account," "browsed job postings," "opened a shared conversation," or "ran code in an assessment." Boundaries track rules of engagement: each target ships with guard rules such as "must not delete hiring content" or "must not revoke API keys," and violations are recorded with container and timestamp. Exploited reports which injected vulnerabilities the agent actually completed, with every vulnerability defined as a kill chain of ordered phases so the platform can show which steps were finished and where the agent stalled. Integrity checks run every minute to confirm the application remains functionally correct — seed data present, services answering with the right content, cross‑service trust intact — and any failed check is treated as a penalty.
Audit-ready is a season. It shouldn't be.
Evidence in spreadsheets, controls drifting between audits, frameworks multiplying on flat headcount. Nubivance runs continuous compliance on Rapid7 Cyber GRC - SOC 2, HIPAA, ISO 27001, PCI, CMMC.
End the scrambleA run, end to end: deployment, telemetry and retesting
Targets are complete multi‑service applications — business logic, seeded data, background jobs and simulated user traffic — built across multiple languages and frameworks. Each target carries 20 or more injected vulnerabilities, from single‑step flaws to cross‑service chains, including zero‑days discovered by CTF.ae's own researchers; none of these intentionally planted flaws exists in public training corpora. Deployments spin up as isolated multi‑container environments in about ninety seconds; the platform can run up to a thousand deployments concurrently.
Instrumentation is hand‑written for each service and emits structured telemetry via OpenTelemetry. The platform ingests that telemetry per deployment, records a timeline of mapped business actions ("previewed a job posting," "submitted an enterprise request," "minted an API key") and lets engineers query the raw stream with a log query language supporting regular expressions and attribute filters. When an agent reports a finding that is not in the target's vulnerability catalogue, the timeline settles whether it was a false positive or an unseeded real bug.
Vulnerabilities can be toggled or patched in place on a running deployment; some patches apply at runtime while others require a restart of a minute or two. That lets agents retest against the same environment and state so repetition separates variance from improvement. Every deployment carries metadata — model name, agent version, prompt variant and the engineer who ran it — and groups of runs show averages, best scores and per‑vulnerability progress across attempts.
Field proof at DEF CON 34 and the Bug Bounty Village CTF
CTF.ae deployed the instrumentation at DEF CON 34 in August 2026 for the Bug Bounty Village contest. The Xenoptic target — a fictional AI company with a sophisticated scope — was given in isolated copies to 545 registered players and observed for the full 48 hours. Across more than 850 deployments the platform streamed the same four signals: Integrity confirmed environments stayed healthy, Boundaries recorded rule violations, Coverage showed which parts of the target each player had exercised, and Exploited recorded which vulnerabilities were genuinely completed and at which step. All of that telemetry came from inside the target environments, never from the players' machines — a tougher test than a single lab run under controlled conditions.
What this means for AI engineering teams, security teams, and CTF organizers
- AI engineering teams and experimenters: XRanges for AI supports an experiment matrix via an API and a Model Context Protocol server with bearer token authentication, enabling automated batch deployments, launches and result collection from CI pipelines or chat assistants — not just a manual console for reading results.
- Security teams and reviewers: the platform shifts evaluation from trusting agent reports to observing actions inside the target, supplying per‑vulnerability kill‑chain progress, coverage blind spots and timestamped boundary violations that materially reduce human triage effort.
- CTF organizers and competitive bug hunters: using the same instrumentation, contests can be judged on what was actually done inside an isolated deployment rather than on submitted reports alone, addressing fairness when players may have hit unintended bugs or brought external CVEs into a contest.
XRanges for AI presents a measurement approach built on observing agents from inside realistic targets, scoring them on independent signals, and scaling to thousands of experimental runs. It emphasises that a single run proves very little — and gives teams the tooling to prove more, faster. Teams that want to put an agent against a target it has never seen can talk to CTF.ae; the platform runs as a managed cloud service or self‑hosted so “nothing leaves their environment.”
