Bash scripting Hansard

I was recently discussing parliamentary attendance, and wanted to collect some data on the subject.  Unfortunately, MPs attending debates isn't recorded, so there's no way to measure this directly.

However, parliament.uk do publish speakers during debates, which should give some indication of MPs' involvement:

http://www.publications.parliament.uk/pa/cm201516/cmhansrd/cm151102/debindx/151102-x.htm

Taking this, I wrote a quick bash script which downloads this page, parses out lines which identify speakers parties:

Emily Thornberry (Islington South and Finsbury) (Lab):

From there, a quick regex will grab the party name.  (This isn't foolproof, but this is just the quick and dirty version.  It would be better to use a pre-set list of the party abbreviations.)

The important thing here is what these indicate - the record labels the party of each speaker the first time they speak in a debate, so if they speak several times, or have a long exchange, this will only be counted as one intervention.  We can then aggregate these together using the uniq command, and after a little bit of cleansing we have some figures broken down by party.  

Finally, I wrote a quick loop to download these records for every day since the start of the parliament (fortunately this is quite robust - requesting records for days parliament wasn't in session returns a 404 but doesn't alter the results, so this loop can be very dumb).  Putting the results into a spreadsheet, and comparing them with the numbers of MPs in each party (from here) then gives us some indication of the levels of activity for each MP.

These are the figures I get for November:

Party	Interventions	Intervention proportion	Seats	Seat proportion	Ratio
Con	1233	44.24%	330	51.16%	0.86
Lab	988	35.45%	232	35.97%	0.99
SNP	325	11.66%	54	8.37%	1.39
DUP	89	3.19%	8	1.24%	2.57
LD	51	1.83%	8	1.24%	1.48
SDLP	31	1.11%	3	0.47%	2.39
PC	24	0.86%	3	0.47%	1.85
UUP	19	0.68%	2	0.31%	2.20
Green	10	0.36%	1	0.16%	2.31
Ind	9	0.32%	3	0.47%	0.69
UKIP	8	0.29%	1	0.16%	1.85
TOTAL	2787		645

These are the overall figures for this parliamentary session so far:

Party	Interventions	Intervention proportion	Seats	Seat proportion	Ratio
Con	5502	44.58%	330	51.16%	0.87
Lab	4495	36.42%	232	35.97%	1.01
SNP	1404	11.38%	54	8.37%	1.36
DUP	337	2.73%	8	1.24%	2.20
LD	209	1.69%	8	1.24%	1.37
SDLP	121	0.98%	3	0.47%	2.11
PC	89	0.72%	3	0.47%	1.55
Green	62	0.50%	1	0.16%	3.24
UUP	56	0.45%	2	0.31%	1.46
Ind	39	0.32%	3	0.47%	0.68
UKIP	28	0.23%	1	0.16%	1.46
TOTAL	12342		645

In both the Conservatives are underrepresented - this is most likely because the record doesn't give party affiliations for ministers, so they will be missed here.  Other than this, the small parties all do quite well, with the Greens getting by far the highest ratio.  The Independents do particularly badly, and Labour score relatively low too.

It's important to keep in mind that there are lots of sources of error here, and that this is just one metric, and particularly favours a hit-and-run approach which probably isn't the ideal.  And of course, if you like this sort of thing you'll love TheyWorkForYou.