Bash scripting Hansard

I was recently discussing parliamentary attendance, and wanted to collect some data on the subject.  Unfortunately, MPs attending debates isn't recorded, so there's no way to measure this directly.

However, parliament.uk do publish speakers during debates, which should give some indication of MPs' involvement:

http://www.publications.parliament.uk/pa/cm201516/cmhansrd/cm151102/debindx/151102-x.htm

Taking this, I wrote a quick bash script which downloads this page, parses out lines which identify speakers parties:

Emily Thornberry (Islington South and Finsbury) (Lab):

From there, a quick regex will grab the party name.  (This isn't foolproof, but this is just the quick and dirty version.  It would be better to use a pre-set list of the party abbreviations.)

The important thing here is what these indicate - the record labels the party of each speaker the first time they speak in a debate, so if they speak several times, or have a long exchange, this will only be counted as one intervention.  We can then aggregate these together using the uniq command, and after a little bit of cleansing we have some figures broken down by party.  

Finally, I wrote a quick loop to download these records for every day since the start of the parliament (fortunately this is quite robust - requesting records for days parliament wasn't in session returns a 404 but doesn't alter the results, so this loop can be very dumb).  Putting the results into a spreadsheet, and comparing them with the numbers of MPs in each party (from here) then gives us some indication of the levels of activity for each MP.

These are the figures I get for November:

Party	Interventions	Intervention proportion	Seats	Seat proportion	Ratio
Con	1233	44.24%	330	51.16%	0.86
Lab	988	35.45%	232	35.97%	0.99
SNP	325	11.66%	54	8.37%	1.39
DUP	89	3.19%	8	1.24%	2.57
LD	51	1.83%	8	1.24%	1.48
SDLP	31	1.11%	3	0.47%	2.39
PC	24	0.86%	3	0.47%	1.85
UUP	19	0.68%	2	0.31%	2.20
Green	10	0.36%	1	0.16%	2.31
Ind	9	0.32%	3	0.47%	0.69
UKIP	8	0.29%	1	0.16%	1.85
TOTAL	2787		645

These are the overall figures for this parliamentary session so far:

Party	Interventions	Intervention proportion	Seats	Seat proportion	Ratio
Con	5502	44.58%	330	51.16%	0.87
Lab	4495	36.42%	232	35.97%	1.01
SNP	1404	11.38%	54	8.37%	1.36
DUP	337	2.73%	8	1.24%	2.20
LD	209	1.69%	8	1.24%	1.37
SDLP	121	0.98%	3	0.47%	2.11
PC	89	0.72%	3	0.47%	1.55
Green	62	0.50%	1	0.16%	3.24
UUP	56	0.45%	2	0.31%	1.46
Ind	39	0.32%	3	0.47%	0.68
UKIP	28	0.23%	1	0.16%	1.46
TOTAL	12342		645

In both the Conservatives are underrepresented - this is most likely because the record doesn't give party affiliations for ministers, so they will be missed here.  Other than this, the small parties all do quite well, with the Greens getting by far the highest ratio.  The Independents do particularly badly, and Labour score relatively low too.

It's important to keep in mind that there are lots of sources of error here, and that this is just one metric, and particularly favours a hit-and-run approach which probably isn't the ideal.  And of course, if you like this sort of thing you'll love TheyWorkForYou.

Configuration manager

Thing I made:
https://github.com/griffiths-hugh-git/configuration-manager

Details are included, but broadly it aims to make configuring settings inside Java archives robust and repeatable.

JSONPath and JSON Schema

JSONPath provides a JSON analogue to XPath.  JSONPaths come in one of several forms, including a JQuery-like pattern:

JSONPaths can be "definite" (i.e. single-valued):

$.id

or "indefinite" (i.e. multivalued):

$..id

They're cross-platform, but I've been using a Jayway library:

<dependency>
  <groupId>com.jayway.jsonpath</groupId>
  <artifactId>json-path</artifactId>
  <version>2.0.0</version>
</dependency>

This is based around a class JsonPath with straightforward static methods.  This adds a compilation overhead, so if you're using the same expression repeatedly you should look into the "compile" method, which gives a nonstatic version.

JSON schema is much less intuitive, and seems to have surprisingly little support in Java: http://json-schema.org/implementations.html
There's a useful online tool to generate a schema for a given JSON example:
http://jsonschema.net/#/

This seems to be the best publicized JSON schema library:
https://github.com/fge/json-schema-validator/

To build a schema object, you first need to convert it into Jackson's JSON types:

// Marshall schema
ObjectMapper mapper = new ObjectMapper();
JsonNode actualObj = mapper.readTree(config.getJsonSchema());

// Build schema object
final JsonSchemaFactory factory = JsonSchemaFactory.byDefault();
jsonSchema = factory.getJsonSchema(actualObj);

Once the schema object is built, validating documents against it requires a similar set of steps:

ObjectMapper mapper = new ObjectMapper();
JsonNode jn = mapper.readTree(entity.toString());
ProcessingReport report = jsonSchema.validate(jn);
if (!report.isSuccess()) {
  throw new IllegalArgumentException(
  "Entity did not conform to configured schema.");
}

Helpfully, json-schema-validator comes with a set of example classes, which show it in action.

JFreeChart

Allows creation of quick, simple Swing-based graphs.  I'm using this because I want to do some quick numerical modelling and validate the results.

Available in Maven:
http://mvnrepository.com/artifact/org.jfree/jfreechart/1.0.19

Vogella have a nice example:
http://www.vogella.com/tutorials/JFreeChart/article.html

Would recommend this for examples of specific charts:
http://www.java2s.com/Code/Java/Chart/CatalogChart.htm
http://www.java2s.com/Code/Java/Chart/JFreeChartLineChartDemo6.htm

In most applications you'd probably use web services and D3 or some other JS library.  It would probably be better to write your data to CSV and open it in Excel/OOffice.

Ruby on Rails

It all looks a lot like Angular, but comes with quite detailed templates.

Quickstart:
http://guides.rubyonrails.org/getting_started.html

CourseRA:
https://class.coursera.org/webapplications-002/

Built on good MVC principles.

HOME/app/views for the view pages. They're written as HTML fragments.
config/routes.rb contains routing instructions.

It has a concept of "resource", and will auto-set up CRUD methods for these for you.  This makes it a bit like a CMS.

Dynamic view content is via a JSP-like servlet syntax.

Database storage is done using automagical mapping and active records, which abstract away the database layer.
You define a model for your data objects, and Ruby writes a "migration" script to set it up in the DB.
Default is Sqlite, but others are avaiable: http://rubylearning.com/satishtalim/ruby_mysql_tutorial.html

Helpful environment divisions are set up in the database YML file, to allow updates to be applied to dev/preprod/prod.

Jira setup

Download from:
https://www.atlassian.com/software/jira/download
No problem running this on Linux.  It runs on a Tomcat server, with the usual file structure.
I installed it as a non-root user, so to start it up I need to execute $JIRA_HOME/bin/startup.sh
Best not to use port 8080 if you plan to have it running a lot.

Once it's up and running, you need to register a (free) account with Atlassian.  This can then go request a licence key automagically.

Atlassian's SourceTree, unfortunately, is only for Windows and Mac.

Session management in Spring

The HttpServletRequest contains an HttpSession object - this represents a user session.  Session state can be stored against this object, and will persist between HTTP calls.  It's contents are stored server-side, and are not visible to the client - Spring just gives the client cookie containing a JSESSIONID.

This blog entry gives an example of attaching a listener to session events:
http://www.mkyong.com/servlet/a-simple-httpsessionlistener-example-active-sessions-counter/
To hook in to session events, write a class implementing HttpSessionListener, and register it in web.xml as a "listener".  This allows you to trigger actions when sessions are created or destroyed.

The following in web.xml sets the session timeout (in minutes):
    <session-config>
        <session-timeout>30</session-timeout>
    </session-config>
The JSESSIONID cookie doesn't set this expiry time on the cookie.

Angular notes

Angular lets you build "MVW" webapps, so you can seperate view logic, data model and controller actions.

http://www.sitepoint.com/kickstart-your-angularjs-development-with-yeoman-grunt-and-bower/

Tooling

These are things you'll probably want by default:

Yeoman - scaffolding.  Like Maven archetypes.
Grunt - build automation.  Includes a Jetty-style webserver.
Bower - dependency management.

Grunt plays dumb for me - this seems to be to do with wanting Node modules installed locally rather than globally.  Don't just use sudo to make NPM work.  I used this:
https://github.com/glenpike/npm-g_nosudo
This is cleaner if you're starting fresh:
https://docs.npmjs.com/getting-started/fixing-npm-permissions

To build the Angular scaffold,
    npm install -g generator-angular
    yo angular

To view the website,
    grunt serve
The opens a webserver on port 9000.

This will hot-deploy changes to the code as you change them.

Bower

To add a Bower dependency,
    bower install angular-bootstrap --save
This finds, downloads and adds to the the project's bower.json.  It won't include it into in the HTML, you need to do that manually.

Karma tests

Unit tests for JS!  These are possible because Angular has separated the model and controller from the browser.  To run them,
    grunt test

Angular structure

In app/scripts/app.js is the module file - this is the Angular concept of a site.  It defines servlet mappings.
In app/scripts/controllers are the controllers.  Each controller consists of an anonymous function which acts on a $scope object.  The scope is the dependency-injected execution context.

Your data model should be stored in the $scope object, and the controller logic attached as functions to it.
Model data can be bound to the view using double braces, {{data}}.  More complex interactions use angular directives (attributes generally beginning "ng-").

That seems to be enough to get an Angular app up and running!