Security ·
Scrapers that never use the same address twice, and how we stop them without CAPTCHAs
Cet article est en anglais.
Updated 15 November, 14:53 UTC
- First version.
On 14 November, 48,658 different addresses opened a browser page on Blackout Map, and 41,519 of them opened exactly one page and never came back. That is not what people do. It is what a scraper looks like when it borrows a new home internet connection for every single request. This is how we spotted it, why blocking addresses doesn't work against it, and what we do instead of making everyone solve a puzzle.
What we noticed
Blackout Map has tens of thousands of pages: one per country, power grid, provider and website we check. That makes it a tempting target for anyone collecting data in bulk. Search engines are welcome, and so are AI crawlers that say who they are. The traffic in this post is different: it pretends to be a normal person in Chrome.
Three things gave it away:
- It never runs our JavaScript. A real browser on our pages makes small background requests: loading comments, checking a website again, drawing a chart. A program that only downloads the HTML never does. On 9 November, 99.4% of the browser-looking page requests we measured overnight came from addresses that never made a single one.
- It never arrives from anywhere. People come from a search engine, a link or another page on our site, so their browser says where they came from. These requests almost never did.
- Every address is new. One network sent 319 pages from 318 different addresses in an hour. Another, 305 pages from 304.
Browser pages opened per day
7–14 November 2026, UTCfrom addresses that opened just one page that dayeverything else
Why the addresses belong to ordinary people
The last point is the important one. Many of these addresses are not servers in a data centre. They belong to home internet providers: Cox Communications and RCN in the United States, UNINET in Mexico. That is the signature of a residential proxy network.
These networks sell access to real people's home connections. The people usually don't know: they installed a free VPN, a game or a browser extension that quietly rents out their spare bandwidth. A scraper pays the proxy company and gets millions of home addresses to send requests through. To a website, each request looks like a different person at home.
So the address in our log is almost never the scraper's. It is someone's living room. That rules out the usual answers:
- Blocking addresses does nothing. The scraper uses each one once, and tomorrow it may belong to a real reader.
- Limiting requests per address does nothing for the same reason. No address ever sends more than one or two pages.
- Publishing the addresses or reporting them to the provider would point at the people whose connections were borrowed, not at whoever is collecting the data.
What it costs
Downloading public pages isn't a crime, and most of the time this traffic is just noise. It stops being noise when it's big. On 12 November one scraper with a single fake Chrome identity made about 550,000 requests in a day, a third of all our traffic. On the evening of 13 November another one pushed our website pages from about 5,000 an hour to 24,000. Each of those pages starts a live check of the website it is about, so for about five hours every page on our server took more than five seconds to load. The people trying to find out whether something was down were the ones who waited.
One fake browser, day by day
requests with one exact Chrome 145 identityWebsite pages per hour on 13 November
UTCfrom 17:00 UTC: every page on our server took 5+ seconds
Obtenez Proton VPN dès aujourd’hui
What we do instead: a check, not a wall
We didn't want to do the obvious thing. A CAPTCHA or a "checking your browser" screen in front of every page would stop most scrapers, and it would also make every person who just wants to know whether their internet is down sit through a puzzle. So we built something smaller.
- Watch networks, not addresses. Every five minutes a small program looks at the last hour of visits grouped by network (the company that owns the addresses). When a network opens many pages while almost none of its visitors run JavaScript or arrive from anywhere, it is flagged and we get a message.
- Ask that network's visitors one quiet question. Visitors from a flagged network get a page that runs a line of JavaScript, remembers the answer for a day, and reloads. A real browser does this in a fraction of a second, so most people never notice it. A program that doesn't run JavaScript never gets past it and never receives a page.
- Let it expire. A flag ends by itself 24 hours after the scraping stops. Nothing is blocked permanently, and nobody's home address ends up on a list.
- Never check the honest ones. Search engines, the big platforms' own networks, and privacy services real people use, such as iCloud Private Relay and Cloudflare WARP, are never checked. Bots that say who they are follow our robots.txt and are left alone.
Eleven networks have been flagged since we switched it on on 9 November. The numbers are from the last hour each one matched:
| Network | Kind | Flagged | Pages / hour | Addresses | Ran JavaScript |
|---|---|---|---|---|---|
| GTT AS3257Backbone | Backbone | 9 Nov | 318 | 309 | 1.9% |
| CDNEXT AS212238Hosting | Hosting | 9 Nov | 384 | 370 | 0% |
| Web2Objects AS62874Hosting | Hosting | 9 Nov | 300 | 286 | 0% |
| RCN AS6079Home internet | Home internet | 9 Nov | 400 | 391 | 0% |
| Colocation America AS21769Hosting | Hosting | 9 Nov | 343 | 312 | 0% |
| Clouvider AS62240Hosting | Hosting | 9 Nov | 340 | 22 | 0% |
| Hostox AS218721Hosting | Hosting | 9 Nov | 378 | 8 | 0% |
| Alibaba Cloud (US) AS45102Cloud | Cloud | 9 Nov | 768 | 619 | 0% |
| UNINET (Telmex) AS8151Home internet | Home internet | 14 Nov | 312 | 285 | 0% |
| Oculus Networks AS398781Network provider | Network provider | 14 Nov | 319 | 316 | 0% |
| Cox Communications AS22773Home internet | Home internet | 14 Nov | 319 | 317 | 0.3% |
The two kinds look different. A hosting company like Clouvider sent 340 pages from 22 addresses: a few machines working hard. The home providers sent one page per address, hundreds of addresses an hour. Either way, almost none of their visitors ran our JavaScript.
For the scrapers that spread themselves so thin that no network ever looks suspicious, the same one-time check also applies to brand-new visitors on our deepest pages, unless they arrive from a search engine. The home page, the map, our feeds and our embeds are never checked.
What it doesn't stop
We'd rather be honest about the limits:
- Real browsers get through. Some scrapers drive a full copy of Chrome, which runs JavaScript like anyone else's. The one that slowed us down on 13 November did exactly that. For that one we changed our own pages instead: the costly live check now waits until someone has actually been on the page for a few seconds, and the number of checks running at the same time is capped.
- It is a moving target. Twice this week we only found a new scraper because the site got slower. Whatever we do, the next one will look a little different.
- We made one exception. The 12 November scraper used one exact fake browser identity and never sent a referrer, so requests with exactly that pattern are refused outright. Everything else gets a check, never a wall.
If you run a website
- Group your logs by network. Per address, residential proxy traffic is invisible. Grouped by network, one provider suddenly sends hundreds of one-page visitors an hour.
- Use what real browsers do anyway. Find a request your pages always make from JavaScript and see which visitors never make it. That is a better signal than anything in the user agent.
- Don't punish the address. It probably belongs to someone who has no idea their connection was rented out.
- Make expensive work wait for a person. Anything costly that a page starts by itself is a gift to scrapers. Delay it, or start it on a click.
Blackout Map is a live map of internet outages and shutdowns around the world. Open the map · Security watch · RSS · Follow us on X: @Blackoutmap_net























