Cloudflare’s DNS resolver service, 1.1.1.1, experienced connectivity issues for some clients recently, resulting from a combination of BGP hijacking and route leaks.
The incident took place last week and impacted approximately 300 networks across 70 international locations. Despite the unclear figures, the corporation claimed that the campaign’s impact was “relatively modest,” with some overseas regions remaining largely oblivious to its existence.
Incident particulars
Cloudflare reports that at 18:51 UTC on June 27, Eletronet S.A. experienced an outage affecting their customers in Brazil. The incident lasted approximately 1 hour and 42 minutes, with Cloudflare noting that it was likely caused by a configuration issue on the customer’s end. The AS267613 network began sharing information about its 1.1.1.1/32 IP address arrangements with its acquaintances and upstream providers.
A number of networks and a leading Tier 1 provider initially accepted the flawed announcement, treating it as a Distant Triggered Black Hole (RTBH) route, which was processed accordingly.
The hijack likely stemmed from BGP routing preferences that inadvertently prioritized a specific route over others. The announcement by AS267613 of the 1.1.1.1/32 prefix proved more specific than Cloudflare’s 1.1.1.0/24, causing primary networks to misdirect website traffic to AS267613.
As a result of a technical issue, users attempting to access Cloudflare’s 1.1.1.1 DNS resolver were inadvertently blocked, leading to temporary unavailability of the service for certain customers.
At 18:52 UTC, a minute after the initial incident, Nova Rede de Telecomunicações Ltda’s (AS262504) unintended mistake led to the leak of 1.1.1.0/24 upstream to AS1031, which subsequently propagated globally, disrupting world routing.
A malicious leak altered the standard BGP routing paths, diverting traffic intended for 1.1.1.1, exacerbating the hijacking issue and introducing additional reachability and latency problems.
Cloudflare detected the problem around 20:00 UTC and successfully resolved the hijacking incident approximately two hours later. The route leak was successfully resolved at 02:28 UTC.
Remediation effort
Upon initial detection of the incident, Cloudflare promptly engaged with affected networks while also temporarily suspending peering relationships with implicated networks to contain the impact and prevent further dissemination of erroneous routing information.
The corporation clarifies that the incorrect bulletins had no impact on internal network routing, thanks to implementing the Resource Public Key Infrastructure (RPKI), thereby automatically rejecting invalid routes.
Cloudflare’s lengthy-term options introduced in its post-mortem report include:
- Enhance route leak detection methodologies through synergistic fusion of supplementary data streams and real-time contextual variables.
- Implementing robust Route Origin Validation (ROV) safeguards against malicious actors manipulating route announcements to disrupt internet traffic and undermines efforts to strengthen online security. By validating the origin of IP routes, ROV ensures that only authorized networks can announce routes, thereby preventing unwanted routing changes and preserving the integrity of the global network.
- Implementing the Mutually Agreed Norms for Routing Safety (MANRS) guidelines, a critical step in safeguarding global connectivity, involves two core measures: scrupulously discarding invalid prefix lengths and robustly deploying filtering mechanisms to ensure the integrity of routing data.
- Network operators are strongly urged to refuse IPv4 prefixes exceeding a length of /24 within the Default-Free Zone (DFZ), thereby promoting more efficient utilization of available address space.
- Implementing and deploying Autonomous System Path Authentication (ASPA) objects, currently in draft form with the Internet Engineering Task Force (IETF), is crucial to validate the authenticity of BGP announcements.
- Discover the untapped possibilities of harnessing RFC 9234 and Discard Origin Authorization (DOA) to revolutionize network security.
