Skip to main content

Cloudflare 1.1.1.1 Outage Report (July 14, 2025): Global DNS Disruption Root Cause Analysis

 

Cloudflare logo with colorful '1.1.1.1' text above the slogan 'The free app that makes your Internet safer' on a white background."

Key takeaways

  • Global DNS outage: Cloudflare's 1.1.1.1 resolver failed worldwide for 62 minutes on July 14, 2025, due to a configuration error in their service topology .
  • Root cause: A dormant misconfiguration from June 6 linked 1.1.1.1 to a non-production service. When activated, it withdrew critical IP prefixes globally .
  • Traffic impact: UDP/TCP/DoT queries dropped sharply, but DNS-over-HTTPS (DoH) via cloudflare-dns.com stayed stable thanks to separate IPs .
  • Unrelated hijack: Tata Communications (AS4755) advertised 1.1.1.0/24 during the outage, worsening routing issues for some users .
  • Resolution: Cloudflare restored services by 22:54 UTC after reverting configurations and manually re-announcing routes .

Why 1.1.1.1 matters for the internet

You might not think much about DNS resolvers, but they’re like the phonebooks of the internet. Cloudflare’s 1.1.1.1 launched back in 2018 as a faster, privacy-focused alternative to ISP-provided DNS. It quickly became one of the most used resolvers globally, handling billions of queries daily. The service uses anycast routing to direct traffic to the nearest data center, which usually means quick responses and reliability. But on July 14, that same design amplified a failure across every continent. For users relying solely on 1.1.1.1, the internet basically stopped working, websites wouldn’t load, apps froze, and confusion spread. Alot of folks didn’t realize how dependent they’d become on this single service until it vanished .

Timeline of the outage: When everything went dark

Here’s how the incident unfolded, minute by minute :

  • 21:48 UTC: A config change for Cloudflare’s Data Localization Suite (DLS) triggered a global refresh. This activated the dormant error from June 6.
  • 21:52 UTC: 1.1.1.1 prefixes began withdrawing from BGP tables. DNS traffic plummeted within minutes.
  • 21:54 UTC: Tata Communications (AS4755) started advertising 1.1.1.0/24, an unrelated hijack now visible due to Cloudflare’s withdrawal.
  • 22:01 UTC: Internal alerts fired. Incident declared.
  • 22:20 UTC: Fix deployed after reverting configurations.
  • 22:54 UTC: Full service restoration after routes stabilized.

Table: Affected IP ranges during the outage

Table displaying IP prefixes. IPv4 column lists four prefixes; IPv6 column lists corresponding prefixes, with one missing entry. Green header separates columns.

This 62-minute disruption showed how a small config error can cascade into global chaos. Engineers initially missed the June 6 mistake because it didn’t cause immediate problems, no alerts, no complaints. But when that second change hit, it all unraveled fast .

Technical breakdown: What actually broke

The core issue was a service topology misconfiguration. Cloudflare uses internal systems to map which IPs should be advertised where, especially for services like their Data Localization Suite (DLS) that restrict traffic to specific regions. On June 6, a config update accidentally tied 1.1.1.1’s prefixes to a non-production DLS service. Since that service wasn’t live yet, no one noticed .

Then, on July 14, an engineer attached a test location to that same DLS service. This triggered a global refresh of routing policies. Because of the earlier error, 1.1.1.1’s topology got reduced to one offline data center. Routers worldwide immediately withdrew announcements for its IP ranges. Traffic couldn’t reach Cloudflare’s DNS servers at all.

The legacy system managing these topologies lacked safeguards like canary deployments or staged rollouts. A peer-reviewed change still went global in one shot, no gradual testing, no kill switches. Cloudflare’s newer topology system avoids hardcoded IP lists, but migrating between systems created fragility. They’ve since acknowledged this "error-prone" approach needs retiring .

Why detection took 9 minutes: Monitoring gaps

Cloudflare’s internal alerts didn’t fire until 22:01 UTC, 9 minutes after traffic nosedived. Why the delay? A few reasons stand out:

  1. No immediate metric drops: The BGP withdrawal caused routing failure, not server crashes. Queries didn’t fail; they never arrived. Monitoring systems tuned for server errors missed this.
  2. Alert thresholds: Teams avoid overly sensitive alerts to prevent false alarms. As one Hacker News comment noted, operators often wait 5+ minutes before escalating to avoid "alert fatigue" .
  3. Legacy dependencies: Health checks relied on systems that themselves needed DNS resolution, creating blind spots during outages.

This lag highlights a tricky balance: catching failures fast without drowning teams in noise. Cloudflare’s post-mortem implies tighter BGP monitoring might help, but they haven’t detailed specific fixes yet .

The BGP hijack that wasn’t: Tata’s role

As Cloudflare’s routes vanished, something weird happened: Tata Communications (AS4755) started advertising 1.1.1.0/24. ThousandEyes observed this hijack propagating through some networks, worsening connectivity for users whose queries got routed to Tata .

Crucially, this wasn’t malicious. Tata likely advertised 1.1.1.0/24 due to old internal configurations, that prefix was used for testing long before Cloudflare claimed it. Once Cloudflare re-announced their routes, Tata withdrew the hijacked prefix. But for ~25 minutes, it added chaos. This incident underscores how fragile BGP can be when major routes vanish unexpectedly .

Impact analysis: Who felt the outage?

The outage hit hardest for users and apps relying exclusively on 1.1.1.1. But patterns emerged in the data :

  • Protocol differences:
    • UDP/TCP/DoT traffic dropped ~90% (these use IPs like 1.1.1.1 directly).
    • DoH (DNS-over-HTTPS) via cloudflare-dns.com stayed near normal. Its IPs weren’t tied to the faulty topology.
  • Backup resolver users: People using 1.1.1.1 with 1.0.0.1 or third-party DNS (e.g., 8.8.8.8) saw minimal disruption. Failovers kicked in.
  • Regional variances: Reports spiked in North America, Europe, and Asia. Cloudflare Radar confirmed global impact.

Table: Traffic recovery post-fix

"Traffic Restoration Timeline table shows three events from 22:20 to 22:54 UTC. Traffic restoration levels progress from 40% to 98% restored."

Ironically, the outage proved Cloudflare’s DoH resilience. By decoupling DNS from raw IPs, it avoided single points of failure. As one user noted, "DoH was working" when traditional DNS failed .

Lessons for the internet’s infrastructure

This outage wasn’t a cyberattack or hardware failure, it was process and system design flaws. Key takeaways for engineers :

  1. Staged rollouts save lives: Had Cloudflare used canary deployments for config changes, they’d have caught the error in one region first. Their new topology system supports this, but legacy tech didn’t.
  2. Validate dormant configs: "No impact" isn’t "safe." Systems must flag unused configurations that could activate later.
  3. Enforce resolver redundancy: Clients should always use multiple DNS resolvers (e.g., 1.1.1.1 + 8.8.8.8). Single-provider setups risk total outages.
  4. Monitor routing layer: Services need BGP/advertisement visibility, not just server health.

Cloudflare’s pledged to accelerate retiring legacy systems. But as they noted, "This was a humbling event." For the rest of us, it’s a reminder: even giants stumble, and backups matter .

FAQs about the Cloudflare 1.1.1.1 outage

Q: Could using 1.0.0.1 as a backup have helped?
A: Yes, but not completely. 1.0.0.1 shares infrastructure with 1.1.1.1, so both failed. Ideal backups use unrelated resolvers like Google’s 8.8.8.8 or Quad9 .

Q: Why did DNS-over-HTTPS (DoH) keep working?
A: DoH uses domain names (e.g., cloudflare-dns.com), not raw IPs. Those domains resolved via unaffected infrastructure. Always prefer DoH/DoT domains over IPs for resilience .

Q: Was this a BGP hijack?
A: Partially, but not by Cloudflare. Tata’s route advertisement was a side effect of Cloudflare’s withdrawal, not the cause. It amplified issues for some users though .

Q: How often does Cloudflare go down?
A: Rarely. In the last 30 days, 1.1.1.1 had 99.09% uptime vs. 99.99% for Google’s 8.8.8.8. This was an exception, not routine .

Q: Did the outage affect other Cloudflare services?
A: Mostly no. Core CDN, security, and dashboard services use different IPs and weren’t withdrawn. The 1.1.1.1 resolver was the primary casualty .

Comments

Post a Comment

Popular posts from this blog

Nvidia Networking Business Growth: NVLink InfiniBand Ethernet Revenue Surge in AI Data Centers | Underappreciated Segment Analysis & AI Infrastructure Boom

  Nvidia Networking Business Growth: NVLink InfiniBand Ethernet Revenue Surge in AI Data Centers | Underappreciated Segment Analysis & AI Infrastructure Boom Key Takeaways Nvidia's networking segment, though just 11% of total revenue, is growing at rocket-ship speeds while others sleep on it Real-world AI data centers are ditching old tech for Nvidia's InfiniBand because regular ethernet kinda chokes under pressure Analyst Ben Reitzes nailed it: this "underappreciated" business could quietly hit $10B+ as AI factories spread globally There's a catch though - Cisco's fighting dirty and copper cables might hold things back for a bit The Hidden Engine Behind AI's Growth Spurt When people talk Nvidia, they're fixated on GPUs. But the  real  magic happens when those GPUs actually talk to each other. That's where networking comes in, and honestly most folks dont even notice it. Nvidia's networking business (yep, the one making switches and cables)...
Post a Comment
Read more

Trump's 100% Semiconductor Tariff: Exemptions for US Manufacturing, Apple’s $100B Deal, Global Chip Industry Impact & Supply Chain Shifts

  Trump's 100% Semiconductor Tariff: Exemptions for US Manufacturing, Apple’s $100B Deal, Global Chip Industry Impact & Supply Chain Shifts Key Takeaways Policy Detail Key Information Tariff Rate 100% on imported semiconductors and chips Implementation Expected as soon as next week Exemption Criteria Companies building or committing to build in the US Exempt Companies Apple, Samsung, SK Hynix confirmed Target All semiconductors coming into the US Trade Impact Major disruption to global chip supply chains Investment Response Apple pledged additional $600 billion US investment Regional Exceptions South Korean firms get favorable treatment under existing trade deal Trump Announces Historic 100% Semiconductor Tariffs President Donald Trump announced a 100% tariff on chips and semiconductors built outside the United States during a White House press conference Wednesday. This ain't just another trade policy tweak - it's a complete overhaul of how America deals with ...
Post a Comment
Read more

Mount Vernon NY Retirement Hotspot: 25% Senior Surge & Affordable Homes Near NYC | GOBankingRates 2025

  Mount Vernon, NY: The Surprising Retirement Hotspot Nobody Saw Coming Key Takeaways Mount Vernon ranks #29 on GOBankingRates' list of fastest-growing retirement hotspots for 2025 with 18.1% of residents aged 65+  Senior population surged 25% between 2018-2023 - that's one in every five residents  Walk Score of 76 makes it "very walkable" with parks and transit accessible within 10 minutes  Average senior living costs $2,402 monthly, with some options starting at $1,367  Compact downtown feels more like a real community than a retirement bubble Why Mount Vernon's Suddenly Retirement Central (Not Some Fancy Hamptons Spot) When I first heard Mount Vernon was becoming a retirement hotspot, I almost spit out my coffee. I mean, this is the Bronx-adjacent town people used to drive through to get somewhere else! But check this: GOBankingRates just ranked it #29 on their 2025 fastest-growing retirement destinations list. And get this - 18.1% of residents are now 65 or ...
Post a Comment
Read more

ADP Jobs Preview: 104K Private Payroll Gain in July 2025 Signals Labor Market Resilience Before BLS Report

ADP Jobs Preview: 104K Private Payroll Gain in July 2025 Signals Labor Market Resilience Before BLS Report Key Takeaways Private payrolls surged by 104,000 in July, reversing June’s 23,000 loss . Leisure/hospitality (+46K) and financial activities (+28K) led gains; education/health services bled 38,000 jobs . Western states dominated hiring (+75K); the Northeast shed 18,000 positions . Wages held steady: job-stayers earned 4.4% more year-over-year; job-changers saw 7% bumps . The Fed faces pressure to delay rate cuts amid sticky wage growth and resilient labor demand . The Numbers Came In The ADP Research Institute dropped its July report. 104,000 private jobs materialized. Economists expected 76,000. June’s loss got revised too, only 23,000 jobs vanished, not 33,000 . The optimists grinned. The doomsayers shuffled their feet. Nela Richardson, ADP’s chief economist, called it a “healthy economy.” Employers believe consumers will keep spending . The six-month moving average? 67,000. The...
Post a Comment
Read more

Meta, Zuckerberg Settle $8B Facebook Investor Lawsuit over Facebook Privacy Litigation

  Key Takeaways Meta investors settled  an $8 billion lawsuit against Mark Zuckerberg and executives over privacy failures, ending a high-stakes trial . Cambridge Analytica scandal  triggered the lawsuit, where user data was harvested for political campaigns . Undisclosed settlement terms  mean no public accountability for Zuckerberg or the board, critics argue . FTC’s $5 billion fine  in 2019 was central to the case, but gaps in oversight remained . Caremark claims  are notoriously hard to prove, and this case sets no legal precedent . The $8 Billion Privacy Lawsuit Against Zuckerberg Ends Quietly Meta investors just settled a massive lawsuit against Mark Zuckerberg and ten other executives. They wanted $8 billion for privacy failures tied to the Cambridge Analytica mess. The trial started this week in Delaware’s Court of Chancery. But it ended fast, on day two. Judge Kathaleen McCormick got the news Thursday. Shareholders’ lawyer Sam Closic said the deal ...
Post a Comment
Read more

MicroStrategy (MSTR) Stock Surges 5% on S&P 500 Hopes as Bitcoin Hits Record Close

  Key Takeaways MicroStrategy qualifies  for S&P 500 inclusion after Bitcoin’s surge pushed its earnings past $11B over four quarters . STRK preferred shares  jumped 15% in a day, offering 6.6% yield as traders anticipate index inclusion . Coinbase surged 43% in June , fueled by stablecoin revenue growth and the GENIUS Act’s regulatory clarity . S&P inclusion isn’t guaranteed —the committee could reject MSTR over its Bitcoin-focused model . Analysts see 27% upside  for MSTR ($514 avg target), while COIN’s stablecoin income could overtake trading fees . Why MicroStrategy Might Enter the S&P 500 (And Why It’s Not Simple) Bitcoin’s rally to $107,750 in late June wasn’t just a win for crypto traders. For MicroStrategy, it meant clearing the final hurdle for S&P 500 eligibility: four straight quarters of net profits. See, accounting rules used to force companies like MSTR to report Bitcoin holdings at their lowest value ("impaired") even if prices recovere...
Post a Comment
Read more

Block Stock Soars 10% on S&P 500 Entry, Replaces Hess Effective July 23, 2025

  Key Takeaways S&P 500 Entry : Block (formerly Square) joins the S&P 500 on  July 23, 2025 , replacing Hess after its acquisition by Chevron . Market Reaction : Block’s stock surged  >10%  post-announcement as funds rebalanced portfolios to include it . Challenges Persist : Despite the boost, Block’s 2025 performance remains  down 14%  YTD due to weak Q1 results and tariff-related macro concerns . Strategic Significance : Entry validates Block’s pivot to blockchain/fintech and accelerates crypto’s mainstream adoption . Next Catalyst : Q2 earnings on  August 7  will test whether S&P-driven demand offsets economic headwinds . The Big News: Block Is Joining the S&P 500 Come July 23rd, Block, y’know, the company behind Square and Cash App, steps into the S&P 500. They’re takin’ Hess’s spot, which is exitin’ after Chevron wrapped up that $54 billion buyout. Hess had some juicy oil assets down in Guyana, but Chevron finally close...
Post a Comment
Read more