When a single fetcher hammered a site 39,000 times in a minute, the weak links in the internet’s infrastructure were exposed. Fastly’s August 2025 traffic analysis reads like an emergency dispatch: automated systems tied to large language and multimodal models are generating unprecedented volumes of bot traffic, shifting cost and complexity onto the very publishers that sustain the web.
AI crawlers strain origin servers and publishers
Fastly groups the activity into two behaviors: broad, persistent “crawlers” and intense, narrowly focused “fetchers.” Together these AI crawlers now represent roughly 80 percent of AI-related bot traffic, with fetchers making up the remaining 20 percent. The danger isn’t just volume; it’s the patterns: sustained sweeps across millions of pages and episodic spikes of tens of thousands of requests per minute aimed at single servers. Those patterns can overwhelm hosting stacks, inflate bandwidth bills, distort analytics, and in some cases force sites to throttle or block traffic — sometimes cutting off legitimate human readers in the process.
Major players show up prominently in Fastly’s telemetry. Traffic attributed to organizations such as Meta and OpenAI appears among the most active sources. That concentration raises immediate distributional questions: who benefits from harvesting public web content at scale, and who bears the costs?
Why this matters: the web was architected as a decentralized resource. Publishers, intermediaries, and users share the infrastructure and its costs. But large-scale automated harvesting flips that model: centralized AI training systems extract content and value, while origin sites — particularly small newsrooms, independent blogs, and niche publishers — absorb the operational and financial burden. Left unresolved, this dynamic risks technical degradation and market failure.
How AI crawlers operate and why they cause harm
“Crawlers” behave much like traditional indexing bots, traversing many sites to collect data. “Fetchers” are the surgical instruments: they target fewer endpoints but request massive volumes rapidly. Both behaviors can be legitimate components of model training and evaluation pipelines. However, neither is inherently benign. Poorly implemented crawling strategies, disregard for robots.txt, excessive concurrency, and repeated retries multiply load on origin servers — especially when slow responses lead bots to reattempt fetches, escalating the problem.
Technically, the drivers are clear. Foundation models require vast, diverse corpora to learn language and facts. The public web is abundant and inexpensive relative to curated, licensed datasets, so it is an economically attractive source. But that economic logic collides with operational realities: the marginal cost of an extra HTTP request is minimal for a well-provisioned cloud provider yet can be material for a small publisher paying per gigabyte or per request. The result is an extraction economy where centralized data consumers enjoy the upside while decentralized content hosts shoulder the downside.
Real-world impacts and security concerns
Publishers report disrupted services, unexpected bills, and contaminated analytics. A mid-sized independent newsroom described an automated surge that “effectively DDoSed” part of their site, forcing defensive measures that also blocked legitimate readers. Fastly’s examples are publicly available, but individual publisher experiences vary, and many incidents remain anecdotal and dispersed.
Security teams are also alarmed. Aggressive crawling can expose sensitive endpoints, stress misconfigured APIs, and obscure malicious activity behind legitimate-looking bot traffic. Since some AI crawlers mimic search engine bots or use shared infrastructure, simple IP blocking is often ineffective and can harm innocent actors. Effective defenses require coordination among CDNs, network operators, and content owners — and sometimes deeper changes to how bot identity and behavior are signaled.
Policy, technical fixes, and market responses
Regulators and industry actors are already discussing levers to rebalance incentives and reduce harm:
– Rate limiting and polite crawling standards: Enforceable crawl budgets, default limits respectful of origin capacity, and stronger adherence to robots.txt. Encouraging standardized polite behavior would reduce accidental overload.
– Authenticated data access and monetization: APIs or paid access models could let publishers control reuse, monetize content, or set technical parameters (rate limits, endpoints designed for bulk export) that protect infrastructure.
– Traffic attribution and transparency: Standardized bot identity tokens, registries, or attestation protocols would let site operators distinguish benign crawlers from abusive ones and make informed access decisions.
– Legal clarifications and compensation mechanisms: Existing copyright, database rights, and contract law shape permitted reuse. Policymakers could consider whether those frameworks adequately address large-scale model training or whether new obligations for provenance, attribution, or compensation are necessary.
Each approach has trade-offs. Technical standards need broad adoption and incentives. Paid APIs shift costs but could restrict access and entrench incumbents. Legal remedies face jurisdictional complexity and slow timelines. Implementation will be messy; actors on all sides have strong, sometimes opposing incentives.
Industry responses and the path forward
Some large AI developers publish papers and engineering notes describing data collection and filtering, asserting reliance on public-domain materials and arguing for the public benefits of improved tools. Yet transparency at the methodological level does not always translate into operational protections for origin servers. Practical protections — like crawling behavior that respects server load and clear mechanisms to request exclusion or negotiate access — are still inconsistent.
Sustainable solutions will likely combine technical protocols, industry standards, commercial arrangements, and regulatory guardrails. A mixed approach could protect small publishers, preserve innovation, and maintain the open web’s decentralization: rate limits and attribution tokens to prevent abuse; paid or authenticated endpoints to compensate content producers; and legal clarity to align incentives.
Conclusion: confronting the strain from AI crawlers
Fastly’s report is both a technical audit and an early warning. It documents scale and behavior without alleging nefarious intent from named organizations, but it makes clear the systemic question: how can a decentralized web survive and thrive when centralized AI systems consume content at scale? The future will be shaped by whether industry-led standards, improved tooling, and thoughtful regulation produce a sustainable equilibrium — or whether extraction-driven practices reshape the web, privileging scale over stewardship and leaving smaller publishers to shoulder the costs. Addressing the challenge of AI crawlers now will determine who pays, who benefits, and whether the web remains a shared commons.
