So, the Huawei oversight report is out and it’s apparently terribly scary. If you’re interested in the content rather than the mood-music there’s a good key points summary here, and if you’re the kind of person who reads this blog, the report itself is here.
The point I would like to make, though, is that the report is a monument to everything we are losing. It’s a serious document adrift in a sea of fatuity. While its audience is obsessed either with Chinese 007, how much to suck up to the Donald, or just vague thinktank cool-hunting, the HCSEC and its oversight board is deeply concerned with reproducible software builds and version control.
In this post from a few months back I described the British approach with Huawei as a pragmatic solution that implemented a wider agenda of accommodating Chinese ambitions. Pulling focus, we might connect this right back to the decision to recognize both the PRC and reality in 1949.
Reading the report, we can see how the solution works in practice. Huawei discloses the source code of new releases to an organization, HCSEC, based in the UK and subject to the NCSC’s oversight. Its people are Huawei UK Ltd employees but they are subject to the British government’s security clearance process. This organization is responsible for conducting a security review of the source and reporting issues both back to Huawei R&D, and to the NCSC. It’s also responsible for verifying that the final product is in fact the code that passed the review process. This is the rub.
The distinction between source code as released by the developers and the package files distributed to customers is critical here and I for one suspect it is completely unappreciated by the report’s audience. There is a lot of stuff to do between signing off code, and generating something users can install on their machines. This is the so-called build process, which involves fetching the right version of the source, plus the appropriate sources for any other projects that are used in it, fetching assets like documentation, graphics, or data files, executing any automated tests, compiling the program code with the correct options for the hardware you intend it to run on, and finally packaging the whole thing in the file format your users expect, like a Linux .rpm or .deb or a Windows .msi, cryptographically signing it so they can check its integrity and authenticity, and shipping it out of the door. People spend a lot of time, money, and effort on this, and a lot of software exists to automate it. (The LinuxFromScratch project will give you an idea.)
From a security point of view the problem is that the checks operate at the time of reviewing the code, but a lot of things happen between then and the time of use. If the build process is off, the version in the package might not be the right one, code intended for testing might get shipped to users, random stuff left lying around could be included, the range of potential problems is enormous. The crucial point here is that this is a huge problem for really anyone who produces software that someone else is expected to use.
Ideally, you’d have a so-called reproducible build – the process would guarantee that if you ran it against the same version of the sources twice, you would get two packages that can be shown cryptographically to be identical. This practice was introduced in scale by the Debian Linux developers starting in 2014. Importantly, if this is so, anyone with the sources and the build scripts could run the build themselves and get an identical, working package they can trust.
The HCSEC tries to verify that the packages distributed to Huawei’s UK customers are in fact built from the source it reviewed, the whole source, and nothing but the source. This is where the problems arise, as it turns out that Huawei’s internal software engineering is pretty much as bad as everyone else’s. Very often, the wrong versions of third-party libraries are included, testing code and random waste are left in, and individual developers use wildly varying tools and processes. As an example, the HCSEC found 40 different versions of a foundational security library, OpenSSL, built into products it reviewed.
This, in fact, was what the NCSC and Huawei fell out about last year. The HCSEC was spending so much time manually building Huawei products from source so as to deliver good packages to the customers that it was detracting from its ability to review the source to begin with. The answer to this was for Huawei R&D to spend a lot of money trying to improve its own practices and not ship quite so much crap that needed fixing at the HCSEC. They focused on trying to get a reproducible build process going for one key product, the eNodeB or 4G base station, but this has turned out to be really difficult.
Another problem is that the variety of hardware and other options customers demand from Huawei means that each release of, say, the eNodeB or the Broadband Access Gateway involves dozens of sub-variants. (Something I didn’t know: product for British carriers’ foreign operations goes through the process, so the different regulatory options for all the countries where Vodafone operates need covering.)
The point that strikes me in the end is that Huawei’s builds are horrible, but unlike any other vendor, the HCSEC process means that we the public get to know about this. It would be fantastic if we had this kind of insight into, say, Facebook. No-one’s going to offer you that, though, are they?
So … Huawei’s software engineering processes are probably not much better or worse than anybody else’s, and there’s no sign of any of the issues mentioned being related to Chinese government interference?
They are terrible, but in a way that is really common, and the Oversight Board explicitly says they have found no evidence of any interference.