Routine maintenance

After this post and the outstanding response to it, I’ve just been working on the lobby project’s underpinnings, specifically to backport some data cleaning from the analyser script into the original scraper, and to fix the one-edge-per-row version of the scraper. As a result I’ve had to flush the datastore and also search out some URIs that have changed. So far we’ve recreated 931 out of 1,721 meetings, although we’re getting the dreaded “Execution status: run interrupted by a timeout”. Actually, we’ve got 1,233 meetings back, and we’ve got rid of some crap. Anyone wanting the dataset can get it from the Scraperwiki API here or here for linkwise rather than meetingwise (coming soon) as either json-dict or csv. A full SQL syntax is available.

With luck, there will also be some more data quite soon. On the analysis score, notably, this and also this seem useful. The first estimates the value of a node based on its edges, which is fundamentally what I’m trying to achieve, and the second finds the cliques in the network a given node belongs to.

Regarding visualisation issues, I think one of my mistakes last time out was to visualise the data as a multi-graph – i.e. a structure with zero or more links between each node, permitting the existence of multiple links between the same pair of nodes. This invariably means a lot of links. The nature of the data – multiple meetings are absolutely central to the whole project, and lobbies meet ministers at different times and on different issues – enforces an underlying multigraph structure. But it would be possible to condense it for visualisation purposes – if we rolled up all links between the same nodes into one, we could tot up their weights and perhaps show that in the visualisation, as a thicker line for example.