TL;DR
Get privacy and security gear delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
A developer says traffic to his site rose and its performance worsened after he posted his feed-reader project on Hacker News. Reviewing NGINX logs, he reported that nearly half of the traffic came from one service that monitors websites for changes; the service was blocked, and he added rate limits and other infrastructure changes.
A developer says his website became slow after a post about his feed-reader project received about 100 upvotes on Hacker News, and he traced much of the resulting traffic to automated crawlers. In a report on his personal site, he said nearly half of the requests came from one service that monitors sites for changes and summarizes them in an AI-generated newsletter.
The developer, who has worked on the customizable feed reader sprinklz.io, initially thought the slowdown reflected a burst of genuine visitors. Instead, he said the site stayed slow and worsened over several days, while the server load seemed out of proportion to the number of active, logged-in users he observed.
He added logging for requests to each endpoint and reviewed aggregated NGINX logs. According to his account, most requests did not reach the endpoints used to retrieve news articles. Because the site is a single-page application, he said, many requests also did not run the JavaScript needed to render the page.
The developer said one service accounted for nearly half of the traffic and made requests many times per minute. He did not name it, saying it describes itself as a service that watches websites and turns changes into an AI newsletter. He said he blocked that source, added a robots.txt file intended to discourage some crawlers while allowing search-related traffic, and introduced rate limits. He also reported making infrastructure changes over the following weeks to improve performance.
When Crawlers Strain Small Sites
The report shows how automated requests can affect a small website even when they do not translate into a comparable number of human readers. Repeated page checks may consume server resources, and a site’s owner may need to distinguish useful indexing from traffic that contributes little value to visitors.
The case also highlights the limits of relying on a traffic total alone. The developer said his logged-in user activity did not explain the server load, while endpoint logs and request records helped reveal that automated clients were a major part of the demand. That distinction matters to site operators deciding whether to scale capacity, limit request rates, or block particular sources.
His account is a single-site report, not evidence of a general trend or proof that AI services broadly caused the slowdown. It does, however, provide a concrete example of the operational costs that can arise when monitoring and crawling tools repeatedly check a site without coordination with its owner.
website traffic monitoring tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
A Feed Reader Reached Hacker News
The developer said he posted sprinklz.io to Hacker News about a month before writing the report. The project lets users customize a news feed, including deprioritizing selected subjects such as AI and large language models. He said the post received around 100 upvotes and saw it as a useful demonstration of the product.
At first, he welcomed the attention and assumed the performance problem came from a temporary increase in visitors. But the slowdown persisted and grew worse after the post, while his available metrics did not match the level of server load. He then added endpoint-level logging and examined NGINX traffic records.
He also reported seeing recognized crawlers, including bots identifying themselves as Claude and OpenAI, as well as Google’s search indexer. His logs also included requests to common vulnerability-testing paths, such as a WordPress file location, though he said his site was not built with WordPress. These details are the developer’s observations; the report does not establish that every crawler caused a meaningful share of the slowdown.
“nearly half of our traffic was coming from one specific user”
— The developer, in his report
Traffic Counts and Attribution
The report does not provide total request counts, a precise measurement of server load, or a before-and-after performance benchmark. It also does not identify the service responsible for nearly half of the traffic, so readers cannot independently check its request behavior or response to being blocked.
It remains unclear how much of the slowdown came from that service versus other crawlers, vulnerability scans, or the increase in legitimate visitors. The developer listed several other automated clients but did not quantify each one’s contribution. The account describes a sequence of events after the Hacker News post, but does not establish that the post alone caused all of the traffic or the performance problems.
The report also does not specify which infrastructure changes were made, how much they improved response times, or whether the site continues to receive similar automated traffic. No statements from the unnamed service or the other crawler operators were included.
Performance Work Continues
The developer said he had made infrastructure improvements over several weeks that he expects to produce substantial performance gains for users. He did not give a schedule for further changes or publish benchmark results in the report.
For now, his account describes measures already taken: blocking the high-volume service, adding crawler guidance through robots.txt, and applying rate limits. Whether those steps will prevent future slowdowns, and how the site performs under renewed traffic, remains to be seen.
Key Questions
What happened to the developer’s website?
After a post about his feed-reader project received about 100 upvotes on Hacker News, the developer said the site became slow and stayed that way for several days. He later found substantial automated traffic in his logs.
Where did much of the traffic come from?
The developer said nearly half of the traffic came from one unnamed service that monitors websites for changes and summarizes them in an AI newsletter. He said it queried his site many times per minute.
What steps did he take?
He said he blocked the service, added a robots.txt file to discourage some crawlers while permitting search-related traffic, introduced rate limits, and made infrastructure changes aimed at improving performance.
Is AI bot traffic confirmed as the sole cause of the slowdown?
No. The developer reported that automated traffic was significant, but did not publish enough data to determine how much each crawler contributed or rule out other causes, including legitimate visitors.
Did the changes fix the performance problem?
The developer said he made infrastructure improvements that should produce performance gains, but his report did not include benchmarks or confirm how the site performed afterward.
Source: hn
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
