Hey! You! I don't like your bearfriend!

There was a post going around recently about making song lyrics gender-neutral by replacing all gendered pronouns with "bear", leading to the amazing line:

"Bear was a bear, bear was a bear, can I make it any more obvious?"

(Struggling to find the original post, but I think it's from this tumblr.)
I wanted to try this out, so I came up with a scheme to get hold of a corpus of film titles (always a useful thing to have!), and substitute "bear" for "boy" and "girl".

Turns out Wikipedia has a collection of lists of films, categorised by first letter. The URLs aren't entirely predictable - there's "J-K" and other groups to handle, but there's a handy table on each pge that can be used as a source.  Once you've got the URLs, they're are easy to download using wget.  Extracting the film titles can be done with some regex trickery after a bit of trial and error - there are lots of other links on the page, but only the film titles are italicised.
Once the links are extracted, there's a little data cleansing issue - the titles often contain disambiguation information in parentheses.  This can be free text, e.g. "(manga)" or "(2006 Swedish film)", so it's difficult to distinguish this from part of the title, e.g. "Beyond (The Animatrix)".  Looking through the data, it's fairly safe to remove any suffixed parantheses.

Having got hold of the film titles, it's a simple sed command to replace "boy" and "girl".  You could worry about word boundaries here, to make sure that you're only replacing whole words, but like "bearfriend" and "cowbears", some of the funniest results are when they're not separate words.
One final thing to watch out for - 27 of the movie titles actually do contain "bear" (including several Care Bears movies, and a few spurious "beards"), so you need to separate out the titles which are going to be modified first, or it will be hard to distinguish these later.

The results worked out pretty well - I would definitely like to see "Bad Bears II" and "Cowbear Bebop: The Movie"!  Here's the full list:

101 Rent Bears
4 Little Bears
5ive Bears
A Bear Named Charlie Brown
A Bear Named Sue
A Bear and a Dolphin
A Bear from Hunan
About a Bear
All the Bears Are Called Patrick
All the Bears Love Mandy Lane
All the Real Bears
American Bear: A Profile of Steven Prince
American Bear
American Bears
Assault Bears
Astro Bear
Attack on the Pin-Up Bears
Baby Bear
Bad Bear Bubby
Bad Bears II
Bad Bear
Bad Bears
Beastly Bearz
Beat Bear
Beautiful Bears
Bicycle Bear
Big Bears Don't Cry
Biker Bearz
Birthday Bear
Bear A
Bear Goes to Heaven
Bear, Bear
Bears
Bears and Bears
Bears Don't Cry
Bears' Night Out
Bears of the City
Bears on the Side
Bears Town
Bears Will Be Bears
Bearz n the Hood
Bubble Bear
Cabin Bear
Calendar Bears
Cannibal Bears
Career Bears
Chelsea Bears
City Bear
Cover Bear
Cowbear
Cowbear Bebop: The Movie
Cowbears & Aliens
Cuban Rebel Bears
Daddy's Little Bears
Dasepo Naughty Bears
DC Super Hero Bears: Hero of the Year
DC Super Hero Bears: Intergalactic Games
DC Super Hero Bears: Super Hero High
Devil Bear from Mars
Diary of a Lost Bear
Different for Bears
Dogtown and Z-Bears
Dr. Goldfoot and the Bear Bombs
Dragon Bears
Dreambears
Drugstore Cowbear
Drugstore bear
Earth Bears Are Easy
Elephant Bear
Even Cowbears Get the Blues
Every Bear Should Be Married
Factory Bear
Fanbears
Fat Bear
Fat Man and Little Bear
Five Bears
Flybears
Flying Bears
Follow Me, Bears!
For Colored Bears
Funny Bear
Gagambear
Georgy Bear
Ghosts of Bearfriends Past
Bear 6
Bear Happy
Bear in Gold Boots
Bear Most Likely
Bear on the Bridge
Bear Shy
Bear with a Pearl Earring
Bear, Interrupted
Bearfight
Bearfriend From Hell
Bearfriends
Bears Just Want to Have Fun
Bears Town
Bears und Panzer der Film
Bears Will Be Bears
Bears! Bears! Bears!
Golden Bear
Gone Bear
Good Bear!
Good Morning, Bears
Grandma's Bear
Gregory's Bear
Gregory's Two Bears
Hammerbear
Hellbear
Hellbear 2: The Golden Army
Hellbear: Blood and Iron
Hellbear: Sword of Storms
Hello, My Dolly Bearfriend
His Bear Friday
Inside the Bears
Invasion of the Bee Bears
Jersey Bears
Jersey Bear
Jewbear
Jimmy Neutron: Bear Genius
Julien Donkey-Bear
Just One of the Bears
Kiss the Bears
Kit Kittredge: An American Bear
Lars and the Real Bear
Lego DC Super Hero Bears: Brain Drain
Leningrad Cowbears Go America
Les Bears
Live Nude Bears
Lonesome Cowbears
Lost Bears: The Thirst
Lost Bears: The Tribe
Love That Bear
Mallbear
Marine Bear
Material Bears
Mean Bears
Midnight Cowbear
Modern Bear
Modern Bears
Mrs. Brown's Bears D'Movie
My Beautiful Bear, Mari
My Best Friend's Bear
My Bearfriend is Type B
My Bearfriend's Back
My Bear
My Bear 2
My Little Pony: Equestria Bears
My New Sassy Bear
My Sassy Bear
My Sassy Bear 2
My Scary Bear
My Super Ex-Bearfriend
New Waterford Bear
Nowhere Bear
Odd Bear Out
Oldbear
Once Upon a Bear
One Hundred Men and a Bear
Paperbears
Phat Bearz
PR Bears
Prayer of the Rollerbears
Queer Bears and Bears on the Shinkansen
Rally 'Round the Flag, Bears!
Reform School Bears
Resurrection of the Little Match Bear
Ride 'Em Cowbear
Riding in Cars with Bears
Riding in Vans with Bears
Run, Fatbear, Run
Samaritan Bear
Sex and the Single Bear
Shopbear
Show Bear
Showbear in Hollywood
Showbears
Sing Bear Sing
Soldier's Bear
Some Bears Do
Song For a Raggy Bear
Sonny Bear
Sorority Bears
Space Cowbears
Spy Bear
Steambear
Storm Bear
Stratosphere Bear
Suburban Bear
Sue, Mai & Sawa: Righting the Bear Ship
Superbear
Swamp Bear
Swing Bears
Tank Bear
Tell Them Willie Bear Is Here
The Adventures of Sharkbear and Lavabear in 3-D
The Bakery Bear of Monceau
The Bay Bear
The Bellbear
The Bellbear and the Playbears
The Bear and the Beast
The Bear in the Plastic Bubble
The Bear in the Striped Pajamas
The Bear Turns Man
The Bear Who Could Fly
The Bear Who Cried Werewolf
The Bears from Brazil
The Bears in Company C
The Bears in the Band
The Buffalo Bear
The Country Bear
The Cowbear Way
The Cowbears
The Dangerous Lives of Altar Bears
The Dead Bear
The Diary of a Teenage Bear
The Errand Bear
The Fabulous Baker Bears
The Flower Bear
The Geisha Bear
The Bear Can't Help It
The Bear from Monday
The Bear In The Park
The Bear in the Taxi
The Bear Who Kicked the Hornets' Nest
The Bear Who Knew Too Much
The Bear Who Leapt Through Time
The Bear Who Played with Fire
The Bearfriend Experience
The Bears of Pleasure Island
The Good Bear
The Goodbye Bear
The Gore Gore Bears
The Harvey Bears
The History Bears
The Incredibly True Adventures of Two Bears in Love
The Jerky Bears: The Movie
The Last Bear Scout
The Little Bear Who Lives Down the Lane
The Lost 15 Bears: The Big Adventure on Pirates' Island
The Lost Bears
The Lost Bears (franchise)
The Machine Bear
The Match Factory Bear
The Nasty Bear
The Newton Bears
The Other Boleyn Bear
The Patchwork Bear of Oz
The Poor Little Rich Bear
The Prince and the Showbear
The Rise of a Tombear
The Sunshine Bears
The Trouble with Bears
The Waterbear
The Whoopee Bears
The Wog Bear
The Young Bears of Rochefort
There's a Bear in My Soup
This Bear's Life
This Bear's Life
Three Smart Bears
Tombear
Tommy Bear
Trailer Park Bears: The Movie
Two English Bears
Two Bears and a Guy
Uptown Bears
Urban Cowbear
Valley Bear
Waterbears
Weather Bear
Weaving Bear
What a Bear Wants
What's a Nice Bear Like You Doing in a Place Like This?
When Bears Fly
Where the Bears Are
Whitebearz
Why Bears Love Sailors
WiseBears
Wonder Bears
Working Bear
Xiu Xiu: The Sent Down Bear
You Are My Sassy Bear
You're a Big Bear Now
Zenon: Bear of the 21st Century
Ziegfeld Bear

Renaming ArXiv PDFs

When you download papers from arxiv.org, the assigned filenames are the unique ID of the publication, not the paper's title.  This makes it hard to browse the content, especially if you like to download several things to read later.

This quick bit of bash script looks through the current directory for any .pdf file whose name looks like an ArXiv ID, reads out the text and looks for a title row.  The text format isn't entirely uniform in all papers, so it needs to skip any very short rows, and any metadata rows.  Once an appropriate title is found, it's standardised to form a filename.

Since it may need to read through several rows, I've stored the contents in a file, but it might be tidier to do this in memory.

for file in ` ls . | grep "^[0-9\.]*.pdf$"`; do
    pdftotext $file temp.txt
    rowNum=1
    title=
    while [ ${#title} -lt 5 ] || [ ! -e $(echo $title | grep "arXiv") ]; do
      title=`sed "${rowNum}q;d" temp.txt | sed 's/[^A-Za-z0-9]/_/g'`
      title=${title:0:80}
      rowNum=$((rowNum+1))
    done
    mv $file $title.pdf
    rm temp.txt
  done

De-gendering news headlines

I've been playing around with news sites lately, and after talking about gender and zodiac signs recently, came up with a little piece of code which switches the two.  It's kind of in the spirit of this extension, which switches genders on the web:

http://www.huffingtonpost.com/2013/08/29/jailbreak-the-patriarchy_n_3443654.html

To make this work, I use the Guardian API to search for articles about "men" or "women" in a specified date range.  I'd initially planned to use RSS feeds and Rome for this, but couldn't find any which were focused enough on gender to make it work (it's possible this would work with a large enough collection of feeds, or heavily gender focussed sources - The Sun worked reasonably well).

Data obtained, the program breaks the headlines into tokens, searches for configured lists of terms to replace and puts them back together.  It uses separate lists for singular and plural terms.  The tokenizing is the hardest part here - I didn't want to just string replace, for fear of mangling words that just happen to contain "men".  Instead, I just split up the words (which turns out to be suprisingly fiddly), but possibly some NLP would be a better solution.

All quite quick and dirty, but I was pretty pleased with the results:

• 'Geminis are more interesting than Sagittariuses': Simon Mawer on Tightrope
• Oregon militia standoff: the 23 Leos and two Aquariuses facing felony charges
• Historic deal allows Pisces and Tauruses to pray together at Western Wall
• Tauruses and Aries clubbing together – or not… | Katharine Whitehorn
• For Leos and Aries, flexible working is still just an altruistic myth | Lisa Lintern
• The Aquariuses who design for Geminis
• Flexible working helps Virgos succeed but makes Aries unhappy, study finds
• Government ‘still failing to protect Capricorns against violent Libras’
• The sequel to Poems That Make Grown Libras Cry: Capricorns, look upon these works and weep…
• Tall Tauruses rarely fancy small Aries – that explains my traumatic dating years | Chris Windle

Bash scripting Hansard

I was recently discussing parliamentary attendance, and wanted to collect some data on the subject.  Unfortunately, MPs attending debates isn't recorded, so there's no way to measure this directly.

However, parliament.uk do publish speakers during debates, which should give some indication of MPs' involvement:

http://www.publications.parliament.uk/pa/cm201516/cmhansrd/cm151102/debindx/151102-x.htm

Taking this, I wrote a quick bash script which downloads this page, parses out lines which identify speakers parties:

Emily Thornberry (Islington South and Finsbury) (Lab):

From there, a quick regex will grab the party name.  (This isn't foolproof, but this is just the quick and dirty version.  It would be better to use a pre-set list of the party abbreviations.)

The important thing here is what these indicate - the record labels the party of each speaker the first time they speak in a debate, so if they speak several times, or have a long exchange, this will only be counted as one intervention.  We can then aggregate these together using the uniq command, and after a little bit of cleansing we have some figures broken down by party.  

Finally, I wrote a quick loop to download these records for every day since the start of the parliament (fortunately this is quite robust - requesting records for days parliament wasn't in session returns a 404 but doesn't alter the results, so this loop can be very dumb).  Putting the results into a spreadsheet, and comparing them with the numbers of MPs in each party (from here) then gives us some indication of the levels of activity for each MP.

These are the figures I get for November:

Party	Interventions	Intervention proportion	Seats	Seat proportion	Ratio
Con	1233	44.24%	330	51.16%	0.86
Lab	988	35.45%	232	35.97%	0.99
SNP	325	11.66%	54	8.37%	1.39
DUP	89	3.19%	8	1.24%	2.57
LD	51	1.83%	8	1.24%	1.48
SDLP	31	1.11%	3	0.47%	2.39
PC	24	0.86%	3	0.47%	1.85
UUP	19	0.68%	2	0.31%	2.20
Green	10	0.36%	1	0.16%	2.31
Ind	9	0.32%	3	0.47%	0.69
UKIP	8	0.29%	1	0.16%	1.85
TOTAL	2787		645

These are the overall figures for this parliamentary session so far:

Party	Interventions	Intervention proportion	Seats	Seat proportion	Ratio
Con	5502	44.58%	330	51.16%	0.87
Lab	4495	36.42%	232	35.97%	1.01
SNP	1404	11.38%	54	8.37%	1.36
DUP	337	2.73%	8	1.24%	2.20
LD	209	1.69%	8	1.24%	1.37
SDLP	121	0.98%	3	0.47%	2.11
PC	89	0.72%	3	0.47%	1.55
Green	62	0.50%	1	0.16%	3.24
UUP	56	0.45%	2	0.31%	1.46
Ind	39	0.32%	3	0.47%	0.68
UKIP	28	0.23%	1	0.16%	1.46
TOTAL	12342		645

In both the Conservatives are underrepresented - this is most likely because the record doesn't give party affiliations for ministers, so they will be missed here.  Other than this, the small parties all do quite well, with the Greens getting by far the highest ratio.  The Independents do particularly badly, and Labour score relatively low too.

It's important to keep in mind that there are lots of sources of error here, and that this is just one metric, and particularly favours a hit-and-run approach which probably isn't the ideal.  And of course, if you like this sort of thing you'll love TheyWorkForYou.

Configuration manager

Thing I made:
https://github.com/griffiths-hugh-git/configuration-manager

Details are included, but broadly it aims to make configuring settings inside Java archives robust and repeatable.

JSONPath and JSON Schema

JSONPath provides a JSON analogue to XPath.  JSONPaths come in one of several forms, including a JQuery-like pattern:

JSONPaths can be "definite" (i.e. single-valued):

$.id

or "indefinite" (i.e. multivalued):

$..id

They're cross-platform, but I've been using a Jayway library:

<dependency>
  <groupId>com.jayway.jsonpath</groupId>
  <artifactId>json-path</artifactId>
  <version>2.0.0</version>
</dependency>

This is based around a class JsonPath with straightforward static methods.  This adds a compilation overhead, so if you're using the same expression repeatedly you should look into the "compile" method, which gives a nonstatic version.

JSON schema is much less intuitive, and seems to have surprisingly little support in Java: http://json-schema.org/implementations.html
There's a useful online tool to generate a schema for a given JSON example:
http://jsonschema.net/#/

This seems to be the best publicized JSON schema library:
https://github.com/fge/json-schema-validator/

To build a schema object, you first need to convert it into Jackson's JSON types:

// Marshall schema
ObjectMapper mapper = new ObjectMapper();
JsonNode actualObj = mapper.readTree(config.getJsonSchema());

// Build schema object
final JsonSchemaFactory factory = JsonSchemaFactory.byDefault();
jsonSchema = factory.getJsonSchema(actualObj);

Once the schema object is built, validating documents against it requires a similar set of steps:

ObjectMapper mapper = new ObjectMapper();
JsonNode jn = mapper.readTree(entity.toString());
ProcessingReport report = jsonSchema.validate(jn);
if (!report.isSuccess()) {
  throw new IllegalArgumentException(
  "Entity did not conform to configured schema.");
}

Helpfully, json-schema-validator comes with a set of example classes, which show it in action.

JFreeChart

Allows creation of quick, simple Swing-based graphs.  I'm using this because I want to do some quick numerical modelling and validate the results.

Available in Maven:
http://mvnrepository.com/artifact/org.jfree/jfreechart/1.0.19

Vogella have a nice example:
http://www.vogella.com/tutorials/JFreeChart/article.html

Would recommend this for examples of specific charts:
http://www.java2s.com/Code/Java/Chart/CatalogChart.htm
http://www.java2s.com/Code/Java/Chart/JFreeChartLineChartDemo6.htm

In most applications you'd probably use web services and D3 or some other JS library.  It would probably be better to write your data to CSV and open it in Excel/OOffice.