Why an AI Agent Can Visit Thousands of Sites to Pick a Camera
From Cloudflare CEO Matthew Prince: classic crawlers are not what bent the traffic curve—AI agents that research exhaustively are.
If internet bot traffic is surging, the first suspect many people name is the familiar cast: search crawlers, scrapers, attack traffic. Prince’s observation cuts the other way. Those older categories, he said, have not changed all that much. Googlebot-style indexing traffic has even declined as a share. What bent the curve steeply is AI-driven agents.
Five sites versus five thousand
His example is deliberately ordinary. Ask ChatGPT or Claude to help you pick a digital camera. To give a solid, well-compared answer, an agent may visit orders of magnitude more sites than you would. You might check five review sites and decide. An agent, trying not to miss information, might walk through five thousand related pages. Five thousand to five is the shape of a thousandfold gap—on a shopping task that still feels simple.
Scale that behavior to messier jobs—industry research, comparing insurance products, planning a trip—and the request volume becomes easier to picture. Agents do not get tired of opening the next tab. Humans do. That difference alone can rewrite how crowded the web feels from the server’s point of view.
More complete for you, heavier for the web
From the user’s side, the trade looks attractive: a broader answer than you would have gathered yourself. From the web’s side, it looks like inefficiency with a purpose—thousands of fetches to produce one reply. Prince’s point is not that agents are “wrong” to do this. It is that if the pattern continues, the internet faces demand growth that dwarfs the pandemic spike people still remember.
- More servers, bandwidth, CPUs, GPUs, memory, and storage to keep agents running.
- Higher load on ordinary sites that never planned for agent-scale browsing.
- Pressure to reinvent slow software stacks when traffic multiplies by orders of magnitude.
Agents, crawlers, bots—same substance
Prince also pushed back on word games. Call it an agent and it sounds futuristic; call it a crawler and it sounds slightly hostile. Underneath, both are machines fetching resources without a person clicking each step. What changed is not the existence of automation, but how ambitious and frequent that automation became once models could plan multi-step research.
So the camera story is less about shopping tips and more about a new traffic physics: one human intention can now spawn a swarm of machine requests. Understanding that multiplier is the first step to understanding why the open web suddenly feels more crowded—even when the number of people online has not jumped by the same factor.