When AI Bots Turn Public Data Into a Server-Side Problem
The internet was built around the idea that public information should be accessible.
But what happens when thousands of automated systems access that information so aggressively that the cost of serving it starts becoming a problem?
That appears to be happening around the infrastructure used to distribute Linux kernel data.
Recent figures shared by Konstantin Ryabitsev, who is involved with the infrastructure behind git.kernel.org, show how much computing power can be consumed by modern automated crawlers.
The surprising part isn’t simply the amount of traffic.
It’s how that traffic is being generated.
Public Data Can Still Be Expensive
The Linux kernel is one of the world’s largest open-source projects.
Its source code, commits, and history are publicly available through Git. Anyone who needs the repository can download it using the tools designed for that purpose.
Yet automated AI-related traffic has been repeatedly requesting web-rendered versions of the same information.
That creates unnecessary server-side work.
Instead of simply transferring existing Git data, the server may have to process a request, generate an HTML page, and send that page back to the crawler.
One request isn’t a big deal.
Millions of similar requests are a completely different story.
Around One-Fifth of the Compute
The reported numbers put the scale into perspective.
Across five geographically separated machines with a combined capacity of about 90 CPU cores, roughly 14 to 16 cores are reportedly being occupied continuously by crawler activity.
That’s close to 20% of the available compute.
For comparison, legitimate human visitors represent only a small portion of the overall request activity.
So the problem isn’t necessarily that people are using the Linux infrastructure too much.
It’s that automated systems are repeatedly asking the servers to perform expensive work.
The Inefficient Way to Collect Data
The biggest issue is the difference between downloading information and repeatedly asking a server to reconstruct it.
A full clone of the Linux kernel repository has been estimated at around 200 CPU-seconds.
But collecting comparable information by repeatedly rendering commit pages through the web interface can require around 280 CPU-hours.
That’s an enormous difference.
And the numbers become even more dramatic when the same process is considered across the many forks of the repository.
An estimate involving 922 forks puts the total potential workload at roughly 258,160 CPU-hours.
The underlying data hasn’t suddenly become difficult to obtain.
The expensive part is the method being used to retrieve it.
Why Blocking Bots Isn’t Simple Anymore
Server administrators have several traditional ways to deal with abusive automated traffic.
They can block IP addresses.
They can restrict entire network providers.
They can use tools such as Fail2Ban.
They can also introduce challenges designed to make automated requests more expensive.
But sophisticated crawlers don’t necessarily stay still.
According to the reported experience from the kernel infrastructure, measures such as ASN blocking and Fail2Ban were not enough.
A system called Anubis, which uses proof-of-work challenges to discourage automated access, also worked for a period before crawlers adapted.
Some bots have reportedly progressed to solving more difficult challenges.
The IP Address Problem
There is another reason modern scraping is difficult to control.
A crawler operating from a recognizable cloud-provider IP is relatively easy to identify.
A crawler operating through residential or mobile proxy networks is much harder to distinguish from an ordinary internet user.
Reports indicate that some automated traffic has moved through residential and mobile IP infrastructure, including proxy networks connected to smart-TV software ecosystems.
At that point, simply blocking suspicious data-center IP ranges is no longer a complete solution.
The Strange Part: Nobody Is Hiding the Data
This is what makes the situation particularly interesting.
The Linux kernel isn’t a private database.
The information isn’t being hidden behind a subscription.
The repository is already available.
There are efficient ways to obtain the source and its history.
So the debate isn’t really about whether AI companies should be allowed to access public Linux data.
The bigger question is:
Who should pay the infrastructure cost when automated systems collect that data inefficiently?
If a company needs millions of records, downloading an appropriate dataset or repository is generally much more efficient than repeatedly forcing someone else’s servers to generate webpages.
Open Source Infrastructure Has Limits
Large technology companies have enormous computing resources.
Many open-source projects don’t.
A project can make its source code freely available while still operating its infrastructure with limited hardware, bandwidth, volunteers, donations, or carefully managed hosting budgets.
That creates an important distinction:
Free to access does not mean free to serve at unlimited scale.
Every request still consumes something.
CPU cycles.
Memory.
Bandwidth.
Storage I/O.
Network connections.
And ultimately, someone’s infrastructure budget.
What Can Infrastructure Maintainers Do?
The Linux kernel infrastructure team is reportedly considering ways to reduce the number of expensive URLs and operations available to automated crawlers.
That can include removing unnecessary features, limiting expensive requests, and making certain types of content harder to crawl at scale.
The goal doesn’t have to be shutting down access to the data.
A more practical approach is to encourage legitimate users and organizations to obtain the information through efficient mechanisms.
That benefits everyone.
Users get the data faster.
Servers do less work.
Infrastructure costs stay manageable.
A Warning for the AI Industry
This situation highlights a problem that is likely to become more common.
AI models require enormous quantities of information.
Open-source projects, documentation sites, code repositories, forums, and other public websites contain valuable material.
But large-scale collection changes the economics of the web.
A website that works perfectly for thousands of human visitors may behave very differently when automated systems begin making millions of requests.
The AI industry therefore has a responsibility beyond simply asking whether information is publicly accessible.
It should also consider how that information is being collected.
The Real Question
There is nothing unusual about collecting public data.
Search engines have been crawling the web for decades.
The difference today is scale.
Modern AI systems can create enormous demand for publicly available information, and automated crawlers can operate continuously without the natural limits imposed by human browsing.
That means responsible crawling needs to become part of the conversation.
The Linux kernel case is a useful reminder:
Public data may be free to download, but generating that data repeatedly can have a very real infrastructure cost.
And when that cost consumes a significant portion of a project’s computing capacity, the conversation stops being about data access and starts becoming a question of responsible use of shared infrastructure.
#AI #Linux #OpenSource #WebCrawling #WebScraping #LLM #ArtificialIntelligence #DevOps #Git #Technology
