A Little Bit of Egyptian Internet Twaddle

Pulling together various resources, I’m beginning to get a picture of what happened with the cut-off and restoration of the Internet in Egypt. First up, at least in some senses, it may be valid to say that the Internet played a role – Arbor Networks observed that traffic to and from Egyptian networks (and between them, in so far as any of them are customers of Arbor’s) had spiked dramatically, almost vertically, in the two hours before the cut-off and that the whole week up to the 28th of January had been one of unusually heavy traffic.

When the cut-off went into effect, at 5.20pm local time on the 27th, it was implemented by forcing all the networks that peer at the Telecom Egypt-controlled Internet exchange to drop their BGP peering sessions with the exception of AS20928, Noor Data Networks. Famously, this is the operator that serves the Central Bank and its payments settlement system. Essentially immediately, 2,576 networks announced by 26 Autonomous Systems became unreachable. The surviving 26 ASNs including, as well as the Central Bank, the Alexandria Library, and the national research & education network, which if it is at all like most NRENs has a lot of its own infrastructure.

On the 31st of January, there was a further wave of cut-offs which removed another 14 ASNs and 134 networks. The list of the last survivors is here – notably, someone had clearly realised that not cutting off the students, of all people, was a missed opportunity, as the NREN isn’t in there. However, one of the mobile operators (UAE incumbent Etisalat’s national opco) stayed online although they had been ordered to cut off the mobile service itself. Perhaps they provide service to the government’s mobile devices?

Interestingly, however, according to posts to NANOG, several of the .eg root DNS servers remained online (not surprisingly, as at least one is outside Egypt). Even more interestingly, even after the BGP sessions with the IX were pulled down, the lower layer equipment stayed active – Egyptian ISPs noticed that there was still link light on the fibre optic lines between their locations, and theoretically it would have been possible to cobble together static routing between their systems.

Similarly, the internal voice network remained operational and so did the international SS7 gateways that link it to other phone systems. As a result, some people found that they could still reach their ISP, whether by dial-up over the voice circuit or even sometimes on DSL. The question, though, was whether there were any routes beyond the ISP’s nearest point of presence. Several foreign ISPs offered free dial-up connectivity over international phone service (notably this French one).

And, it seems, Egyptian ISPs also tried to re-establish internal connectivity after the cut-off, when they noticed that the fibres were still lit up. However, the problem was more subtle than just pointing static routes at each other. Communicating with people outside Egypt wasn’t, after all, the primary need, and anyway, it required passing through the government-controlled exchange.

But the problem with Facebook, Twitter, Gmail or what have you is that unless they have data centres in your country, they’re international traffic. Depending on their internal architecture, even if they do, they might be dependent on international routes. An Egyptian engineer who posted to NANOG during the revolution made the interesting point that, although Egyptian ISPs are relatively well-interconnected among themselves, not that much traffic flows over the interdomain links as so much stuff goes out to the global Internet. It’s analogous to the old problem that the topological centre of the African Internet was 36 Tooley Street, London SE1 (the LINX headquarters), or 111 8th Avenue, New York, depending on whose version of the story you like better, although less pernicious as the infrastructure is there to solve it.

Sometimes this is useful – it’s harder to censor stuff hosted in another jurisdiction. But it’s also a problematic dependency. Back in the Egyptian NOC the New York Times was hosted on, they were struggling to find copies of key software packages to distribute, for example clients for Internet Relay Chat messaging, and also critical data files such as cached DNS zones, lists of domain names and their corresponding addresses. Some ISP engineers are now working on preparing emergency packages of software and data for use in an extreme emergency – for example, regular dumps of the root and local DNS zones, similar snapshots of the local routing table, not to mention PGP signing keys and contacts for as many other engineers as possible.

After all this, what were the government’s aims? The initial cut-off was probably motivated by a combination of wanting to black out sources of independent information and hoping that it would hinder the protestors’ organising. Some of its particular details – for example, leaving 20928 up and not trying to shut down interdomain links within Egypt – may have been an effort to keep some “normal service” going, as well as not preventing VIPs from transferring their money out of the country. It’s also possible that cutting off link light between all Egyptian ISPs without physically grubbing up the fibres was harder than it looked.

So then, why did they bring it back on the Tuesday of camels and thugs? One interpretation is that they were hoping people would go home and update their Facebook statuses, which would have been incredibly patronising. But the Egyptian elite patronised the hell out of the public every time it went on TV, so it can’t be ruled out. Another one is that they hoped to project an impression of returning normality, which didn’t really fit with thugs on horseback swinging knives, but then their response wasn’t characterised by coherence.

Another still is that they hoped it would help to get the government’s propaganda out there. This argument – Gamal Mubarak flipping through his copy of The Net Delusion in a curtained backroom of the palace – has the advantage that when the Internet and the mobile networks were reactivated, there was a rash of reports of loyalist trolls, and one of the first things that happened was that the government forced the mobile operators to send out threatening bulk SMS messages – spam as a weapon. But this was surely incredibly optimistic.

In fact, what did happen was that people started doing precisely what they had only been doing to a limited extent the week before. Twitter feeds from Egypt filled up with what the NANOG crew would term operational content – requests for more medical supplies, reports of a lost child, calls for more protestors to mass at a specific gate into Tahrir Square. This was the real thing – a tactical radio network for the mob – and ironically it was mostly running over SMS and going out to servers elsewhere in the world. And, of course, its major carrier was the much reviled Vodafone Egypt, unwilling deliverer of Central Security’s spam blitz.