BLACKOUT MAP

Security ·

Scrapers that never use the same address twice, and how we stop them without CAPTCHAs

Este artigo está em inglês.

Compartilhar
Updated 15 November, 14:53 UTC
  • First version.

On 14 November, 48,658 different addresses opened a browser page on Blackout Map, and 41,519 of them opened exactly one page and never came back. That is not what people do. It is what a scraper looks like when it borrows a new home internet connection for every single request. This is how we spotted it, why blocking addresses doesn't work against it, and what we do instead of making everyone solve a puzzle.

What we noticed

Blackout Map has tens of thousands of pages: one per country, power grid, provider and website we check. That makes it a tempting target for anyone collecting data in bulk. Search engines are welcome, and so are AI crawlers that say who they are. The traffic in this post is different: it pretends to be a normal person in Chrome.

Three things gave it away:

  • It never runs our JavaScript. A real browser on our pages makes small background requests: loading comments, checking a website again, drawing a chart. A program that only downloads the HTML never does. On 9 November, 99.4% of the browser-looking page requests we measured overnight came from addresses that never made a single one.
  • It never arrives from anywhere. People come from a search engine, a link or another page on our site, so their browser says where they came from. These requests almost never did.
  • Every address is new. One network sent 319 pages from 318 different addresses in an hour. Another, 305 pages from 304.

Browser pages opened per day

7–14 November 2026, UTC
489,084pages on 8 November
186,654addresses that opened one page that day
66,765pages on 9 November
7 Nov
8 Nov
9 Nov
10 Nov
11 Nov
12 Nov
13 Nov
14 Nov

from addresses that opened just one page that dayeverything else

Requests for our pages from anything that looked like a browser (search engines and bots that name themselves left out). The detector went on early on 9 November. Part of the drop is the check, part may be the scraper moving on: we can't separate the two.

Why the addresses belong to ordinary people

The last point is the important one. Many of these addresses are not servers in a data centre. They belong to home internet providers: Cox Communications and RCN in the United States, UNINET in Mexico. That is the signature of a residential proxy network.

These networks sell access to real people's home connections. The people usually don't know: they installed a free VPN, a game or a browser extension that quietly rents out their spare bandwidth. A scraper pays the proxy company and gets millions of home addresses to send requests through. To a website, each request looks like a different person at home.

So the address in our log is almost never the scraper's. It is someone's living room. That rules out the usual answers:

  • Blocking addresses does nothing. The scraper uses each one once, and tomorrow it may belong to a real reader.
  • Limiting requests per address does nothing for the same reason. No address ever sends more than one or two pages.
  • Publishing the addresses or reporting them to the provider would point at the people whose connections were borrowed, not at whoever is collecting the data.

What it costs

Downloading public pages isn't a crime, and most of the time this traffic is just noise. It stops being noise when it's big. On 12 November one scraper with a single fake Chrome identity made about 550,000 requests in a day, a third of all our traffic. On the evening of 13 November another one pushed our website pages from about 5,000 an hour to 24,000. Each of those pages starts a live check of the website it is about, so for about five hours every page on our server took more than five seconds to load. The people trying to find out whether something was down were the ones who waited.

One fake browser, day by day

requests with one exact Chrome 145 identity
548,564requests on 12 November
4,189the day after we refused it
7 Nov
8 Nov
9 Nov
10 Nov
11 Nov
12 Nov
13 Nov
14 Nov
Every request carried the same browser identity and no referrer. Our own pages always send one. From the evening of 12 November, requests with exactly that pattern are refused.

Website pages per hour on 13 November

UTC
~5,000a normal hour
24,103at 21:00 UTC
00:00
01:00
02:00
03:00
04:00
05:00
06:00
07:00
08:00
09:00
10:00
11:00
12:00
13:00
14:00
15:00
16:00
17:00
18:00
19:00
20:00
21:00
22:00
23:00

from 17:00 UTC: every page on our server took 5+ seconds

Requests for our "Is it down?" pages from anything that looked like a browser. This scraper ran real Chrome, so it passed the JavaScript check.

Anúncio Obtenha o Proton VPN hoje

What we do instead: a check, not a wall

We didn't want to do the obvious thing. A CAPTCHA or a "checking your browser" screen in front of every page would stop most scrapers, and it would also make every person who just wants to know whether their internet is down sit through a puzzle. So we built something smaller.

  1. Watch networks, not addresses. Every five minutes a small program looks at the last hour of visits grouped by network (the company that owns the addresses). When a network opens many pages while almost none of its visitors run JavaScript or arrive from anywhere, it is flagged and we get a message.
  2. Ask that network's visitors one quiet question. Visitors from a flagged network get a page that runs a line of JavaScript, remembers the answer for a day, and reloads. A real browser does this in a fraction of a second, so most people never notice it. A program that doesn't run JavaScript never gets past it and never receives a page.
  3. Let it expire. A flag ends by itself 24 hours after the scraping stops. Nothing is blocked permanently, and nobody's home address ends up on a list.
  4. Never check the honest ones. Search engines, the big platforms' own networks, and privacy services real people use, such as iCloud Private Relay and Cloudflare WARP, are never checked. Bots that say who they are follow our robots.txt and are left alone.

Eleven networks have been flagged since we switched it on on 9 November. The numbers are from the last hour each one matched:

NetworkKindFlaggedPages / hourAddressesRan JavaScript
GTT AS3257BackboneBackbone9 Nov3183091.9%
CDNEXT AS212238HostingHosting9 Nov3843700%
Web2Objects AS62874HostingHosting9 Nov3002860%
RCN AS6079Home internetHome internet9 Nov4003910%
Colocation America AS21769HostingHosting9 Nov3433120%
Clouvider AS62240HostingHosting9 Nov340220%
Hostox AS218721HostingHosting9 Nov37880%
Alibaba Cloud (US) AS45102CloudCloud9 Nov7686190%
UNINET (Telmex) AS8151Home internetHome internet14 Nov3122850%
Oculus Networks AS398781Network providerNetwork provider14 Nov3193160%
Cox Communications AS22773Home internetHome internet14 Nov3193170.3%

The two kinds look different. A hosting company like Clouvider sent 340 pages from 22 addresses: a few machines working hard. The home providers sent one page per address, hundreds of addresses an hour. Either way, almost none of their visitors ran our JavaScript.

For the scrapers that spread themselves so thin that no network ever looks suspicious, the same one-time check also applies to brand-new visitors on our deepest pages, unless they arrive from a search engine. The home page, the map, our feeds and our embeds are never checked.

What it doesn't stop

We'd rather be honest about the limits:

  • Real browsers get through. Some scrapers drive a full copy of Chrome, which runs JavaScript like anyone else's. The one that slowed us down on 13 November did exactly that. For that one we changed our own pages instead: the costly live check now waits until someone has actually been on the page for a few seconds, and the number of checks running at the same time is capped.
  • It is a moving target. Twice this week we only found a new scraper because the site got slower. Whatever we do, the next one will look a little different.
  • We made one exception. The 12 November scraper used one exact fake browser identity and never sent a referrer, so requests with exactly that pattern are refused outright. Everything else gets a check, never a wall.

If you run a website

  1. Group your logs by network. Per address, residential proxy traffic is invisible. Grouped by network, one provider suddenly sends hundreds of one-page visitors an hour.
  2. Use what real browsers do anyway. Find a request your pages always make from JavaScript and see which visitors never make it. That is a better signal than anything in the user agent.
  3. Don't punish the address. It probably belongs to someone who has no idea their connection was rented out.
  4. Make expensive work wait for a person. Anything costly that a page starts by itself is a gift to scrapers. Delay it, or start it on a click.
Compartilhe este artigo

Blackout Map is a live map of internet outages and shutdowns around the world. Open the map · Security watch · RSS · Follow us on X: @Blackoutmap_net

Todos os artigos

Continue lendo

Security · 6 Nov 2026 352How one asterisk opened a door in OpenSSH: the *sshole bug, explainedCVE-2026-123158 lets a 257-character password into accounts that have no password, on Alpine and other musl systems. How three sensible fixes built it, who is really affected, what some reports get wrong, and what to do. Security · 21 Oct 2026 16,535PHP2Shell: what we know about the two critical PHP flaws (now exploited)Two critical flaws in PHP, the language behind about 70% of websites, can let anyone run code on a server with one request. The fix is out and attacks are confirmed: how the attack works, what is confirmed, what to do, and what it could mean for the internet. Explainer · 21 Oct 2026 239Can a software flaw take the internet down?Ten years after the Mirai attack on Dyn: how security flaws turn into outages, the times they did, and what Blackout Map can and cannot see.

Mais do blog

Security · 17 Nov 2026Why hackers will never wipe out your debtOn 16 November malware ran on the websites of HSBC, Equifax and 176 other organisations, and the internet joked it got "everything but credit card debt and student loans". Why that is no accident: what the attack really did, and how banks keep copies of what you owe out of any hacker's reach. Security · 12 Nov 2026 135Apache's HTTP/2 flaw: found twice in eight days, fixed in 2.4.71CVE-2026-119618 is a race in Apache's mod_http2 that can let an attacker run code without logging in. Only servers with HTTP/2 on are affected. Who found it, who is at risk, the dispute over an exploit, and what to do. Outage · 10 Nov 2026Internet partial outage in Central African Republic: what we measuredBlackout Map measured an internet partial outage in Central African Republic from 20:45 UTC on 7 November 2026. What the data shows, and how to follow it live. Analysis · 6 Nov 2026 125Is 72 hours of supplies enough? What 93 nationwide power outages showCanada and the UK tell people to be ready for three days without power. We measured 93 nationwide outages since 2022: nine in ten ended within three days. The eight that didn't were all on islands, and local storm damage can take weeks to repair. Fact check · 5 Nov 2026 1,906Every "global blackout" date so far, and what really happenedA running list of viral global blackout dates: what was claimed, what was real, and why it can't happen that way. Includes 13, 17 and 18 November 2026, with the live status of the internet. Analysis · 5 Nov 2026Election shutdown watch: where the internet could be cut nextThe next elections to watch for internet shutdowns, starting with Cabo Verde, The Gambia, Guinea-Bissau and South Sudan, with each country's live internet status and what happened at this year's votes. Security · 3 Nov 2026 111Ivanti Connect Secure: three exploited flaws, and the first fix wasn't enoughAttackers chained three flaws to get root on fully updated Ivanti Connect Secure VPN appliances. All three are now on CISA's list of exploited vulnerabilities. Which versions fix it, what the attackers did, and what to check. Fact check · 31 Oct 2026 8,991Vai haver um apagão mundial em 17 de novembro? O que os dados mostramPublicações alertam para um apagão mundial em 17 de novembro: sem luz, sem internet por três meses e contas zeradas. De onde vem, por que não pode acontecer assim, o que está realmente previsto e a situação ao vivo de cada país. Analysis · 24 Oct 2026 144LaLiga's IP blocks: why half a million websites break in Spain on match daysTo fight pirate football streams, Spain's ISPs block shared IP addresses during LaLiga matches, and hundreds of thousands of unrelated websites go down with them. What OONI found, what we measured, and what you can do before Barcelona – Real Madrid. Security · 21 Oct 2026 993Node.js security release: ten fixes, one that can run codeOn the same day as PHP2Shell, Node.js fixed ten flaws in its 22, 24 and 26 lines, including a Brotli decoder bug that can allow remote code execution and several ways around TLS hostname checks. What is affected and what to do. Retrospective · Iran, February 2026 685Iran's wartime blackout: 88 days cut off from the worldFrom the first strikes on 28 February to 26 May 2026, Iran was almost completely cut off from the internet: 2,093 hours, the longest nationwide shutdown on record. What happened, how it worked and what it cost. Retrospective · West Africa, March 2024 180West Africa lost four undersea cables in one dayA suspected underwater landslide off Côte d'Ivoire broke four cables at once in March 2024. Ghana was in blackout for almost a week, Côte d'Ivoire fell to 1% of normal. Retrospective · Canada, July 2022 176How a maintenance update took down Canada's RogersIn July 2022 one update in Rogers' core network cut mobile, home internet, debit payments and emergency calls for millions of Canadians for most of a day. Retrospective · Tonga, January 2022 161Tonga: the volcano that cut a country offThe 2022 Hunga Tonga eruption shredded the only cable linking Tonga to the world. The country was almost completely offline for five weeks. Retrospective · Worldwide, October 2021 225The six hours Facebook disappeared from the internetIn October 2021 one routine command took Facebook, Instagram and WhatsApp off the internet for billions of people. No attack, no country outage. Retrospective · Myanmar, February 2021 173Myanmar 2021: a coup, and an internet curfewAfter the February 2021 coup, Myanmar's military cut the internet nationwide, then every night, then all mobile data: a shutdown shaped by the hour. Retrospective · Tigray, Ethiopia, November 2020 170Tigray: two years without a connectionFrom November 2020, the war in Ethiopia's Tigray region came with a phone and internet blackout that lasted about two years. Retrospective · Iran, November 2019 171Iran, November 2019: a country offline for a weekDuring the 2019 fuel protests Iran cut itself off from the internet almost completely, while keeping parts of its domestic network running. Retrospective · Jammu and Kashmir, India, August 2019 166Kashmir: 550 days without full internetThe shutdown that began in August 2019 became the longest ever imposed in a democracy. High-speed mobile internet only returned in February 2021. Retrospective · Egypt, January 2011 163Egypt 2011: the night a country switched off its internetIn January 2011, two days into mass protests, Egypt withdrew almost all its routes from the internet within minutes. It stayed dark for five days. Retrospective · Pakistan, February 2008 196The day Pakistan took YouTube offline for the whole worldIn 2008 a national block on YouTube leaked into the internet's routing system and made the site unreachable worldwide for about two hours.