Renaming ArXiv PDFs

When you download papers from arxiv.org, the assigned filenames are the unique ID of the publication, not the paper's title.  This makes it hard to browse the content, especially if you like to download several things to read later.

This quick bit of bash script looks through the current directory for any .pdf file whose name looks like an ArXiv ID, reads out the text and looks for a title row.  The text format isn't entirely uniform in all papers, so it needs to skip any very short rows, and any metadata rows.  Once an appropriate title is found, it's standardised to form a filename.

Since it may need to read through several rows, I've stored the contents in a file, but it might be tidier to do this in memory.

for file in ` ls . | grep "^[0-9\.]*.pdf$"`; do
    pdftotext $file temp.txt
    rowNum=1
    title=
    while [ ${#title} -lt 5 ] || [ ! -e $(echo $title | grep "arXiv") ]; do
      title=`sed "${rowNum}q;d" temp.txt | sed 's/[^A-Za-z0-9]/_/g'`
      title=${title:0:80}
      rowNum=$((rowNum+1))
    done
    mv $file $title.pdf
    rm temp.txt
  done

De-gendering news headlines

I've been playing around with news sites lately, and after talking about gender and zodiac signs recently, came up with a little piece of code which switches the two.  It's kind of in the spirit of this extension, which switches genders on the web:

http://www.huffingtonpost.com/2013/08/29/jailbreak-the-patriarchy_n_3443654.html

To make this work, I use the Guardian API to search for articles about "men" or "women" in a specified date range.  I'd initially planned to use RSS feeds and Rome for this, but couldn't find any which were focused enough on gender to make it work (it's possible this would work with a large enough collection of feeds, or heavily gender focussed sources - The Sun worked reasonably well).

Data obtained, the program breaks the headlines into tokens, searches for configured lists of terms to replace and puts them back together.  It uses separate lists for singular and plural terms.  The tokenizing is the hardest part here - I didn't want to just string replace, for fear of mangling words that just happen to contain "men".  Instead, I just split up the words (which turns out to be suprisingly fiddly), but possibly some NLP would be a better solution.

All quite quick and dirty, but I was pretty pleased with the results:

• 'Geminis are more interesting than Sagittariuses': Simon Mawer on Tightrope
• Oregon militia standoff: the 23 Leos and two Aquariuses facing felony charges
• Historic deal allows Pisces and Tauruses to pray together at Western Wall
• Tauruses and Aries clubbing together – or not… | Katharine Whitehorn
• For Leos and Aries, flexible working is still just an altruistic myth | Lisa Lintern
• The Aquariuses who design for Geminis
• Flexible working helps Virgos succeed but makes Aries unhappy, study finds
• Government ‘still failing to protect Capricorns against violent Libras’
• The sequel to Poems That Make Grown Libras Cry: Capricorns, look upon these works and weep…
• Tall Tauruses rarely fancy small Aries – that explains my traumatic dating years | Chris Windle

Bash scripting Hansard

I was recently discussing parliamentary attendance, and wanted to collect some data on the subject.  Unfortunately, MPs attending debates isn't recorded, so there's no way to measure this directly.

However, parliament.uk do publish speakers during debates, which should give some indication of MPs' involvement:

http://www.publications.parliament.uk/pa/cm201516/cmhansrd/cm151102/debindx/151102-x.htm

Taking this, I wrote a quick bash script which downloads this page, parses out lines which identify speakers parties:

Emily Thornberry (Islington South and Finsbury) (Lab):

From there, a quick regex will grab the party name.  (This isn't foolproof, but this is just the quick and dirty version.  It would be better to use a pre-set list of the party abbreviations.)

The important thing here is what these indicate - the record labels the party of each speaker the first time they speak in a debate, so if they speak several times, or have a long exchange, this will only be counted as one intervention.  We can then aggregate these together using the uniq command, and after a little bit of cleansing we have some figures broken down by party.  

Finally, I wrote a quick loop to download these records for every day since the start of the parliament (fortunately this is quite robust - requesting records for days parliament wasn't in session returns a 404 but doesn't alter the results, so this loop can be very dumb).  Putting the results into a spreadsheet, and comparing them with the numbers of MPs in each party (from here) then gives us some indication of the levels of activity for each MP.

These are the figures I get for November:

Party	Interventions	Intervention proportion	Seats	Seat proportion	Ratio
Con	1233	44.24%	330	51.16%	0.86
Lab	988	35.45%	232	35.97%	0.99
SNP	325	11.66%	54	8.37%	1.39
DUP	89	3.19%	8	1.24%	2.57
LD	51	1.83%	8	1.24%	1.48
SDLP	31	1.11%	3	0.47%	2.39
PC	24	0.86%	3	0.47%	1.85
UUP	19	0.68%	2	0.31%	2.20
Green	10	0.36%	1	0.16%	2.31
Ind	9	0.32%	3	0.47%	0.69
UKIP	8	0.29%	1	0.16%	1.85
TOTAL	2787		645

These are the overall figures for this parliamentary session so far:

Party	Interventions	Intervention proportion	Seats	Seat proportion	Ratio
Con	5502	44.58%	330	51.16%	0.87
Lab	4495	36.42%	232	35.97%	1.01
SNP	1404	11.38%	54	8.37%	1.36
DUP	337	2.73%	8	1.24%	2.20
LD	209	1.69%	8	1.24%	1.37
SDLP	121	0.98%	3	0.47%	2.11
PC	89	0.72%	3	0.47%	1.55
Green	62	0.50%	1	0.16%	3.24
UUP	56	0.45%	2	0.31%	1.46
Ind	39	0.32%	3	0.47%	0.68
UKIP	28	0.23%	1	0.16%	1.46
TOTAL	12342		645

In both the Conservatives are underrepresented - this is most likely because the record doesn't give party affiliations for ministers, so they will be missed here.  Other than this, the small parties all do quite well, with the Greens getting by far the highest ratio.  The Independents do particularly badly, and Labour score relatively low too.

It's important to keep in mind that there are lots of sources of error here, and that this is just one metric, and particularly favours a hit-and-run approach which probably isn't the ideal.  And of course, if you like this sort of thing you'll love TheyWorkForYou.

Configuration manager

Thing I made:
https://github.com/griffiths-hugh-git/configuration-manager

Details are included, but broadly it aims to make configuring settings inside Java archives robust and repeatable.

JSONPath and JSON Schema

JSONPath provides a JSON analogue to XPath.  JSONPaths come in one of several forms, including a JQuery-like pattern:

JSONPaths can be "definite" (i.e. single-valued):

$.id

or "indefinite" (i.e. multivalued):

$..id

They're cross-platform, but I've been using a Jayway library:

<dependency>
  <groupId>com.jayway.jsonpath</groupId>
  <artifactId>json-path</artifactId>
  <version>2.0.0</version>
</dependency>

This is based around a class JsonPath with straightforward static methods.  This adds a compilation overhead, so if you're using the same expression repeatedly you should look into the "compile" method, which gives a nonstatic version.

JSON schema is much less intuitive, and seems to have surprisingly little support in Java: http://json-schema.org/implementations.html
There's a useful online tool to generate a schema for a given JSON example:
http://jsonschema.net/#/

This seems to be the best publicized JSON schema library:
https://github.com/fge/json-schema-validator/

To build a schema object, you first need to convert it into Jackson's JSON types:

// Marshall schema
ObjectMapper mapper = new ObjectMapper();
JsonNode actualObj = mapper.readTree(config.getJsonSchema());

// Build schema object
final JsonSchemaFactory factory = JsonSchemaFactory.byDefault();
jsonSchema = factory.getJsonSchema(actualObj);

Once the schema object is built, validating documents against it requires a similar set of steps:

ObjectMapper mapper = new ObjectMapper();
JsonNode jn = mapper.readTree(entity.toString());
ProcessingReport report = jsonSchema.validate(jn);
if (!report.isSuccess()) {
  throw new IllegalArgumentException(
  "Entity did not conform to configured schema.");
}

Helpfully, json-schema-validator comes with a set of example classes, which show it in action.

JFreeChart

Allows creation of quick, simple Swing-based graphs.  I'm using this because I want to do some quick numerical modelling and validate the results.

Available in Maven:
http://mvnrepository.com/artifact/org.jfree/jfreechart/1.0.19

Vogella have a nice example:
http://www.vogella.com/tutorials/JFreeChart/article.html

Would recommend this for examples of specific charts:
http://www.java2s.com/Code/Java/Chart/CatalogChart.htm
http://www.java2s.com/Code/Java/Chart/JFreeChartLineChartDemo6.htm

In most applications you'd probably use web services and D3 or some other JS library.  It would probably be better to write your data to CSV and open it in Excel/OOffice.

Ruby on Rails

It all looks a lot like Angular, but comes with quite detailed templates.

Quickstart:
http://guides.rubyonrails.org/getting_started.html

CourseRA:
https://class.coursera.org/webapplications-002/

Built on good MVC principles.

HOME/app/views for the view pages. They're written as HTML fragments.
config/routes.rb contains routing instructions.

It has a concept of "resource", and will auto-set up CRUD methods for these for you.  This makes it a bit like a CMS.

Dynamic view content is via a JSP-like servlet syntax.

Database storage is done using automagical mapping and active records, which abstract away the database layer.
You define a model for your data objects, and Ruby writes a "migration" script to set it up in the DB.
Default is Sqlite, but others are avaiable: http://rubylearning.com/satishtalim/ruby_mysql_tutorial.html

Helpful environment divisions are set up in the database YML file, to allow updates to be applied to dev/preprod/prod.