Over at the Pol, I’ve been trying to answer the question: how long until robots take Brussels lobbyists’ expense accounts? Software. It’s eating the world, they say. You may already have guessed that this is a reprise of Project Lobster, and you’d be right. The inputs are the European Transparency Register, which lists lobbyists and their clients, the Commissioners’ and staffers’ registers of matters (like so), and the survey I administered to readers here. 157 of you responded, for which I am truly thankful.
I used NetworkX, the network analysis library from the great folks at Los Alamos – is that more or less creepy/cool than CreepyDB, sorry, Gaffer? – and a lot of my own work. Having downloaded the transparency register, made it into a two-way hash table, and scraped all the data from the meetings register, I could create some NetworkX graphs and store them. I then wrote a command-line utility to read out key metrics from them.
The metrics I was particularly interested in are as follows:
Weighted network degree. This is just the count of meetings a given node (commissioner, staffer, lobbyist, or lobby) had, multiplied by any weighting applied to reflect the relative importance of meetings. I used the survey results, normalised as percentages of the average result, to weight the different hierarchical ranks (e.g. commissioner, vice-president, member of cabinet) and functional jobs (e.g. competition, transport). I further divided this weighting by the number of lobbies present at the meeting and applied it on a per-edge (NetworkX terminology for a link) basis.
Shortest-path betweenness centrality. This is defined as the fraction of all the most direct routes through the network, using the edge weightings as distances between nodes, that pass through a given node x. What it tells you is how central Mr X is in the network.
Gatekeepership. This is one I invented, and it’s the ratio of the average weighted network degree of people who met node x to the network-wide average – basically, it’s like Value Over Replacement Player for lobbyists, although I was unaware of VORP at the time. It tells you about the extra influence Mr X gives the people they meet relative to the typical node.
Greedy_Fragile. This is the really creepy one, developed by West Point to optimise drone targeting against terrorist networks. It arose from the insight that killing off the most central individuals tended to cause the network to become radically flatter and more decentralised, and therefore harder to target. Instead it might be better to whack lesser power-centres in the network, so as to render it more centralised and therefore, brittle.
It measures how much more or less centralised you make the whole network, being defined as the change in network-wide average centrality if node x is removed. This is why Al-Qa’ida No.3 is such a notoriously dangerous job. The intuition from it is whether someone tends to be an attractor in their own right, or a pillar of the overall structure.
After a lot of hacking painfully about and remembering how much Python I’d forgotten, I got results. Unfortunately, I’d made a seriously unwise assumption, that commissioners and staffers had unique names. Lobbyists do seem to be de-duplicated, but their targets are not. In fact they’re not even up to date or spelled consistently. This resulted in a race-condition: depending on who got scraped first, either all the DG Trade official Jonathan Hill’s meetings would be credited to the Financial Services Commissioner Jonathan Hill, or vice versa.
For a long sad moment I thought I’d discovered something funny. The less important Hill is massively lobbied because people think he’s the more important one! Then I realised the problem, and set about a completely new implementation. I was by this point back in practice, and it took me a day and a half to program it, re-run everything, repeat the data analysis, and rewrite the piece.
The results? Well, you can read them in the Pol. If you’re a masochist you can download all the code here. But if I was a lobbyist I might be thinking about learning to code. It’s the future. They said.
AIUI the register will also hold a list of whom (externally) met which which person in the EUP. Which provides foundation for another article. Not just the obvious one of who has the most meetings with the most influential people – but looking for patterns of low levels of meetings with just the right people.
This is what the gatekeepership, Dijkstra shortest path centrality, and GF metrics are meant to get at. Do you mean the European Parliament?