Back to the archive

The encyclopedia · Software & IT · Technical decision · 2025

Cloudflare's 1.1.1.1 DNS resolver goes dark worldwide for 62 minutes

A dormant config line in a pre-production service took down the world's most-used public DNS resolver.

Cloudflare · 2025-07-14

What happened

On 14 July 2025, Cloudflare's public DNS resolver 1.1.1.1 was unavailable worldwide from 21:52 to 22:54 UTC — 62 minutes of full downtime for one of the internet's most-used public resolvers, with Gateway DNS also degraded in bursts.

The trigger was a config error, not an attack. On 6 June a change to a pre-production Data Localization Suite service had inadvertently included a reference to the 1.1.1.1 Resolver and its prefixes. It sat dormant for weeks. On 14 July a second, unrelated change to the same service triggered a global refresh of network configuration that withdrew all 1.1.1.1 Resolver prefixes from production data centres worldwide.

Cloudflare traced the underlying fault to error-prone legacy systems that hard-code explicit data-centre lists to prefixes and lack progressive, staged deployment. A coincidental BGP hijack by Tata Communications advertising 1.1.1.0/24 was spotted two minutes after the drop — exposed by the outage, not the cause of it.

Reverting the config restored traffic to about 77% of pre-incident levels, and routing was fully restored by 22:55 UTC. Cloudflare said it would deprecate the legacy prefix-mapping systems and move to staged addressing deployments with health monitoring.

Why it happened

  • A reference to 1.1.1.1 added to a pre-production config sat dormant for weeks, then went live on an unrelated refresh
  • Legacy systems that hard-code prefix lists and deploy without staging turned one config line into a global withdrawal
  • The world's most-used public DNS resolver had no graceful path when its prefixes were withdrawn
  • A coincidental BGP hijack made the outage look like an attack, but the damage was self-inflicted
What it cost62 minutes of global downtime for the 1.1.1.1 resolvercostly

The lesson

A config line that changes nothing today can take down the whole system when an unrelated change wakes it up — staging and health checks are the only guard.

Sources

spotted an error? The club wants to know.

Comments · 0

    Sign in to join the comments.

    More like this

    Somewhere, someone solved the problem this company failed at. 2nd Opinion →