AI agents are getting much better at reading the web. Better indexes, better ranking, cleaner structured output a model can use immediately – the improvements are real, and they are arriving quickly.
But the next phase of progress depends on a layer that gets far less attention. Not how well agents read the web, but how they reach it in the first place, and on what terms. That layer is unfinished in both directions: the engineering is genuinely hard and nowhere near done, and the rules governing it are being written right now. Both will decide whether agentic access to the web belongs to a handful of large companies or stays open to everyone.
The model itself is no longer the interesting variable. Stanford’s 2026 AI Index found that as of March 2026, six leading labs were clustered within 25 Elo points of each other on the Arena Leaderboard, and read that convergence as competition shifting toward cost and reliability rather than raw capability. When the frontier is that crowded, the differences that decide outcomes are the ones underneath it.
What changed in 2026
In early June, automated traffic passed human traffic online for the first time in the history of the internet, with bots making up around 57.5% of requests. By volume, the web is now more a place machines read than one people use.
That matters because of what the web’s economics assume. A site has always been paid for being read indirectly: a crawler took the content, a search engine ranked it, and visitors arrived. The model worked because reading and visiting were the same act. An agent separates them – it reads in order to answer, and the answer is the destination.
So the web started building the next model. Crawler defaults are tightening, automated visitors are being sorted by what they actually do, and access is becoming something granted rather than assumed. The infrastructure is learning to recognize its readers and set terms for them, which is what any system does once its assumptions stop describing reality.
The stack is being built for half the problem
Retrieval is not solved. Evaluations run in September 2026 show accuracy on hard, multi-hop browsing tasks spread from roughly 32% to 74% across providers on OpenAI's BrowseComp benchmark. But it is a crowded field, the gaps are narrowing, and it is where most of the attention and capital currently sit.
The part that is still wide open, and that far less money is chasing, is reliable, permissioned, provable access to the content in the first place. An index built on pages an agent can reach today but not tomorrow is not an asset that holds its value. The durable advantage in agentic search is moving quietly from “can we rank this well” to “can we reach this cleanly, and prove how we did.”
Access is an engineering problem before it is a policy one
It is tempting to treat access as a question of permissions – get the rules right and the data follows. That skips the part where someone has to build it.
The agentic shift raises the bar further. Agents arrive in bursts rather than on human schedules, need the page’s meaning rather than its markup, and have to fail loudly, because an agent that silently returns a stale price is worse than one that returns nothing. All of it has to work within the terms a site has set, which means the permission layer and the retrieval layer cannot be built separately – a system that can reach a page but cannot show it was allowed to is only half a system.
None of these are solved. They are being worked on, continuously, by people who treat access as a product rather than a prerequisite: maintaining the infrastructure, the ethically sourced networks, the compliance posture, and the engineering discipline that keep access reliable as both the web and the rules move. It is unglamorous next to model releases. It is also what everything else stands on.
Provenance stops being a chore
There is a second opportunity here, and the industry has spent too long treating it as a compliance task. Provenance – being able to show where a piece of content came from and on what terms you were allowed to have it – is becoming something that makes an agent more useful, not merely more defensible.
Most search systems were built for people, who need only a title, a link, and a summary to choose a result. Agents want the information itself, already organized, and they need to know where each piece came from. An index designed for agents carries the source and the date with every fact, so any answer can be traced back. An analyst asking for a competitor’s price needs to know whether the figure came from the vendor’s page this morning or a forum post from two years ago.
The same record that answers that question also answers the harder one about whether the page was permitted to be read at all. One piece of engineering, two problems solved – and as access rules tighten, the index that can account for how its content was obtained is the one that keeps working. Building provenance from the start is not overhead, but rather the cheapest trust an agent builder will ever buy.
Bigger than any one policy
The engineering only matters if the ground it is built on stays open. If access is decided one crawler and one contract at a time, the largest players win by default, because only they can negotiate licensing at scale. That is already visible in the early licensing market, where new marketplaces are being built to broker access between AI companies and publishers, with Microsoft among those launching one this year. Paying directly for what you use is a clearer arrangement than hoping attention finds its way back. But analysts have flagged that the market could rebuild the same gatekeeping the open web was meant to avoid, because outlets large enough to sign favorable deals do well while smaller ones get squeezed. The arriving market is good. Keeping it open to more than a handful of players is the work ahead.
So the work runs on two tracks. One is an open standard for declaring purpose – a shared, verifiable way for an agent to state what it is doing and why, that any site can read and any builder can implement. The other is the engineering that makes such a standard worth having: systems that reach the web reliably, carry their permissions with them, and keep working as both sides evolve. A standard without the infrastructure is a document. Infrastructure without the standard is a private arrangement available to perhaps a dozen companies.
That difference is not abstract. It decides whether the next useful thing built on the web comes from a team of four or only from a company large enough to have a licensing department – and whether the people who eventually use it get the better tool or merely the better-funded one. Access has always been the thing that determines who gets to build. Keeping it reliable is engineering. Keeping it open is everybody’s work.
