• Install MySQL using the 64-bit .DMG package installer from dev.mysql.com
• Install Ruby using RVM (that’s Ruby 1.9.2 or 1.9.1-p378 — at the time of writing the latest 1.9.1, p429, is buggy)
• Add these to lines to ~/.bash_login or ~/.bashrc:

export PATH="/usr/local/mysql/bin:$PATH"
export DYLD_LIBRARY_PATH="/usr/local/mysql/lib:$DYLD_LIBRARY_PATH"

• In a new shell (Terminal window), type gem install mysql as normal.

I’m posting this mainly as a record for myself, having wasted a lot of time in the past trying strange incantations from comments on various other blogs posts.

Some years ago, when working on a secure web app for a large organisation — let’s call them Secret Testing Ltd — I was keen that people shouldn’t choose hopelessly weak passwords. I was particularly concerned by my sysadmin colleague’s fondness for passwords of the form ‘p/\55w0rd’ or ‘S3cr3t-T35t|ng’.

I therefore wrote some simple Ruby code to try to catch very weak passwords:

class PasswordUtils
 
  def self.look_and_sound_alikes(original)
    look_or_sound_alikes = { \
      '0' => ['o'] ,
      '1' => ['i', 'l'],
      '2' => ['to', 'too'],
      '3' => ['e'],
      '4' => ['a', 'for'],
      '5' => ['s'],
      '6' => ['g'],
      '8' => ['b', 'ate'],
      '9' => ['g'],
      'i' => ['i'],
      '$' => ['s'],
      '|' => ['i', 'l', ''],
      '!' => ['i', 'l', ''],
      '@' => ['g', 'a'],
      '(' => ['c'],
      '[' => ['c', 'e'],
      '{' => ['e'],
      '&' => ['and'],
      '*' => ['star'],
      ')' => ['d'],
      '^' => ['a'],
      '/' => ['a', 'v'],
      '\\' => [''],
      '<' => ['k']
    } 
    # the penultimate two substitutions catch /\ (A) and \/ (V), albeit without
    # distinguishing them, and the last (in conjunction with the empty options 
    # for ! and |) catches !< and |< (K)
    versions = ['']
    original.downcase.each_char do |c|
      versions.collect! do |v|
        alikes = look_or_sound_alikes[c] || [c]
        alikes.collect { |c1| v + c1 }
      end
      versions.flatten!
    end
    versions
  end
 
  def self.obvious?(password, custom_banned_list = '')
    standard_banned_list = File.read("#{RAILS_ROOT}/config/banned_password_words.txt")
    banned_words = standard_banned_list.downcase.split(/\W+/) + 
      custom_banned_list.downcase.split(/\W+/).delete_if { |w| w.length < 4 }
    alike_words = [password.downcase] + PasswordUtils.look_and_sound_alikes(password)
    banned_words.any? do |banned_word|
      alike_words.any? { |alike_word| alike_word.include? banned_word }
    end
  end
 
end

The first method of the class undoes some common substitutions:

> PasswordUtils.look_and_sound_alikes('p/\55w0rd')
=> ["password", "pvssword"]
>  PasswordUtils.look_and_sound_alikes('S3cr3t-T35t|ng')
=> ["secret-testing", "secret-testlng", "secret-testng"]

And the second method uses this in conjunction with a list of banned words to check that an entered password isn’t hopelessly weak.

> PasswordUtils.obvious?('p/\55w0rd')
=> true
>  PasswordUtils.obvious?('S3cr3t-T35t|ng')
=> true
> PasswordUtils.obvious?('gkAsd76!o')
=> false

Obviously, you need a good banned words list — perhaps starting with some top N passwords lists, and including the name of your company, the web app, and words related to the area of business, local pubs, and so on.

It’s also a good idea to include custom banned words for individual users. For example:

custom_banned_list = "#{first_names} #{surname} #{department} #{organisation} #{email}"
PasswordUtils.password_obvious?(password, custom_banned_list)

Passing these tests isn’t a sufficient condition for ruling out a weak password, but it’s arguably a necessary one, and a good start.

We’re now sending about one million notifications a month, and it’s working beautifully. The obvious alternative — using SMS messages — would be costing us tens of thousands of pounds a month.

If you’re using or contemplating using Push Notifications, you may find these four points of interest:

1. Urban Airship offer a well-though-out and reliable service. But if you’re using them just as an intermediary in telling the APNS to beep users X, Y and Z right now — not making use of their scheduling or broadcast features, for example — it might well not be worth it. You’ll cut out a layer of indirection, and save money if you get big enough for it to matter, by talking to Apple directly.
2. There’s a surprising dearth of viable-looking open-source options for interfacing with the APNS. (APNS uses a binary protocol over a secure connection, and doesn’t like this connection torn down and reopened too often, so you can’t easily interface with it directly from a web app or cron job). But there is pyapns, an XML-RPC-speaking APNS provider built on Python’s Twisted networking framework, with client libraries for Python and Ruby. I’ve found this straightforward and rock-solid.
3. The APNS feedback service sometimes reports device tokens as inactive for no obvious reason, so make sure you’re prepared to hear from and reactivate users you’ve previously inactivated. (mappiness got bitten by this early on: a few dozen people were unable to authenticate to our server because they’d been reported as inactive and we’d assumed they were never coming back, despite that fact they had the app installed and Push Notifications switched on).
4. The app delegate method application:didRegisterForRemoteNotificationsWithDeviceToken: seems occasionally to return phantom zero-length device tokens — so don’t let your app or server get flummoxed by these. (We suffered from this on mappiness too: the bad device token got added to the front of a queue for sending from the app to our server; the server wouldn’t accept it; and this blocked all further data getting sent back until we fixed the server to silently ignore the bad tokens).

Many UK academics can access the data via institutional subscription to EDINA Digimap. I’m using it in my research into subjective wellbeing and environmental quality.

This post shows how to:

1. import the data files into a PostGIS database; and
2. de-normalise the data into a single table, where there’s a one-to-one mapping of postcodes to rows, and each row contains either all geographical locations covered by a postcode (as a single geometry column, of type multipolygon) or the reason why no such location is available

Why de-normalise?

Step 2 above is required for my purposes because Code-Point data is supplied in a number of separate files per postcode area:

• a .shp file (and associated .shx and .dbf) mapping postcodes and ‘vertical streets’ to the locations they cover;
• an accompanying text file mapping postcodes to ‘vertical streets’; and
• another text file listing postcodes with no associated locations, either because they represent PO boxes or because the data just isn’t available.

Vertical streets generally represent high-rise buildings, where one location in 2D space may be associated with multiple postcodes. Vertical streets are also a serious pain: not only may one vertical street be associated with many postcodes, but one postcode may be associated with many vertical streets and with non-vertical-street locations too.

Import the data files

Download and unzip the Code-Point with Polygons data, in ESRI .shp format, for the regions you want (where indicative query timings are provided later, these are for whole-UK data — 120 postcode areas — on a 2Ghz Intel iMac).

Dump everything in the same directory. For each postcode area XX you should have the following five files:

• XX.shp, XX.shx and XX.dbf
• XX_vstreet_lookup.txt
• XX_discard.txt

If you don’t already have one you want to use for this purpose, create a PostGIS-enabled PostgreSQL database (in this post, the database is named geo). If you’re not running PostgreSQL 8.4 or later you may need to fiddle with some fsm_ settings in the Postgres .conf file in order to cope with a data set this large.

Execute the following SQL to create the tables for discards and vertical streets (e.g. from within pgAdmin):

create table cpp_vertical_streets (
  postcode character varying(8),
  vstreet character varying(8)
);
 
create table cpp_discards (
  postcode character varying(8),
  reason character varying(7)
);

Next we need to create the table for the .shp-file polygons. We switch to Ruby for this — you can just enter the commands in an IRB terminal session:

path = '/path/to/CodePoint polygons'
`/usr/local/pgsql/bin/shp2pgsql -p -D -s 27700 "#{path}/ab.shp" cpp_polygons | /usr/local/pgsql/bin/psql -d geo -U postgres`

The -p flag to shp2pgsql just creates a table structure, and the -s 27700 gives it the EPSG code to tell it we’re using the OSGB36 datum. You can use any of the .shp files you’ve downloaded — it need not be the AB area one.

Now to do the import into the three tables. Still in Ruby, execute the following loop:

Dir["#{path}/*.shp"].each do |f|
  puts f
  `/usr/local/pgsql/bin/shp2pgsql -a -D -s 27700 "#{f}" cpp_polygons | /usr/local/pgsql/bin/psql -d geo -U postgres`
  `echo "copy cpp_vertical_streets from '#{f.sub(/\.shp$/, '_vstreet_lookup.txt')}' csv;" | /usr/local/pgsql/bin/psql geo postgres`
  `echo "copy cpp_discards from '#{f.sub(/\.shp$/, '_discard.txt')}' csv;" | /usr/local/pgsql/bin/psql geo postgres`
end

Back in SQL, add some boolean flags we’ll use later on, then create an index on the postcode column, which will save a lot of time later:

alter table cpp_polygons add column discard boolean;
alter table cpp_polygons add column vstreet boolean;
alter table cpp_polygons add column vstreet_and_std boolean;
 
create index pc_index on cpp_polygons (postcode);
 
vacuum analyze cpp_polygons;

If you like, you can confirm here that there’s only one row per postcode — this query should return nothing:

select * 
from cpp_polygons a 
inner join cpp_polygons b 
on a.postcode = b.postcode 
where a.gid <> b.gid;

So, we now have all the OS data held in three separate tables. In the rest of the post, we’ll be merging the data in the discards and vertical streets tables into the main table, cpp_polygons.

Discards

We’re going to create a new table, with the same structure as cpp_polygons, to hold the discarded postcode data. These postcodes will have a NULL geometry column, and a TRUE discard column (one of the booleans we added earlier). Later, we’ll insert the contents of this table into cpp_polygons.

Run the following SQL:

create sequence cpp_discard_seq start with 2000000;
 
create table cpp_discard_polys as (
select 
  nextval('cpp_discard_seq') as gid, 
  postcode,
  cast(null as character varying(20)) as upp, 
  substring(postcode from '^[A-Z][A-Z]?') as pc_area, 
  cast(null as geometry) as the_geom, 
  true as discard, 
  false as vstreet,
  false as vstreet_and_std
from cpp_discards
);

Vertical streets

Similarly, we’re now going to create a new table with the same structure as cpp_polygons to map vertical street postcodes to the right polygons. This requires a left join of the vertical streets data with some of the geometry data in cpp_polygons.

set enable_seqscan = false; 
-- (otherwise pg sometimes fails to use the index)
 
create sequence cpp_vstreet_seq start with 3000000;
 
create table cpp_vstreet_polys as (
  select 
    nextval('cpp_vstreet_seq') as gid, 
    max(v.postcode) as postcode, 
    cast(null as varchar(20)) as upp, 
    max(pc_area) as pc_area, 
    st_union(the_geom) as the_geom, 
    false as discard,
    true as vstreet,
    false as vstreet_and_std
  from cpp_vertical_streets v 
  left join cpp_polygons p 
  on v.vstreet = p.postcode
  group by v.postcode
);

Our last new table is for postcodes that are associated with both standard polygons and vertical street polygons.

create sequence cpp_vstreet_and_std_seq start with 4000000;
 
create table cpp_vstreet_and_std_polys as (
select 
  nextval('cpp_vstreet_and_std_seq') as gid, 
  p.postcode as postcode,
  cast(null as varchar(20)) as upp, 
  p.pc_area as pc_area,
  st_multi(st_union(p.the_geom, v.the_geom)) as the_geom, 
  false as discard, 
  false as vstreet,
  true as vstreet_and_std
from cpp_polygons p
inner join cpp_vstreet_polys v
on p.postcode = v.postcode
);

Cleaning and merging

Now we need to remove the vertical streets and polygons we just merged into a new table from their respective source tables.

delete from cpp_polygons where postcode in (select postcode from cpp_vstreet_and_std_polys);
-- the above could take around 25 mins
delete from cpp_vstreet_polys where postcode in (select postcode from cpp_vstreet_and_std_polys);

And, finally, to merge our three new tables into the main table.

insert into cpp_polygons select * from cpp_discard_polys;
insert into cpp_polygons select * from cpp_vstreet_polys;
insert into cpp_polygons select * from cpp_vstreet_and_std_polys;

In order to join my own data with this data, I find it easiest to format the postcodes on which the join is made with no spaces. So I add and index the following extra column.

alter table cpp_polygons add column postcode_no_sp character varying(8);
update cpp_polygons set postcode_no_sp = replace(postcode, ' ', ''); 
-- the above could take 10 - 15 mins
create index pcns_index on cpp_polygons (postcode_no_sp);
 
vacuum analyze cpp_polygons;

You might also want to remove the original vertical street rows, where the postcode column begins with a ‘V’, from the cpp_polygons table — I didn’t have any need for this.

This is somewhat annoying — the site’s original design has it communicating independently with Amazon (using Amazon’s XSLT API feature to transform their XML data into JSON), and that’s no longer possible with the use of a private key. But it’s not unfixable. The site now sends its API call first to my server, which returns a signed version, and then forwards the signed call on to Amazon.

I found most of what I needed for this on Chris Roos’ blog, but his version still wasn’t quite working for me (the two problems I recall are that Ruby’s CGI.escape doesn’t quite follow Amazon’s requirements, and that times need converting to GMT).

Anyway, in case you’re looking to do the same, here’s what I ended up with:

#!/usr/bin/env ruby
 
# Note: You need hmac.rb and hmac-sha2.rb from http://deisui.org/~ueno/ruby/hmac.html 
# somewhere in your require paths. ruby-hmac is currently broken under Ruby 1.9.
 
%w(rubygems cgi time hmac-sha2 base64).each { |lib| require lib }
 
ACCESS_IDENTIFIER = 'YOUR_PUBLIC_ID'
SECRET_IDENTIFIER = 'YOUR_PRIVATE_ID'
 
def aws_escape(s)
  s.gsub(/[^A-Za-z0-9_.~-]/) { |c| '%' + c[0].to_s(16).upcase }  
  # for 1.9, you'd replace [0] with .ord -- but ruby-hmac seems broken under 1.9
end
 
cgi = CGI.new
params = cgi.params.dup
 
amazon_endpoint = params.delete('amazon_endpoint')
amazon_path = params.delete('amazon_path')
js_callback = params.delete('js_callback')
 
signing_params = {
  'AWSAccessKeyId' => ACCESS_IDENTIFIER,
  'Timestamp' => Time.now.gmtime.iso8601
}
 
params.merge!(signing_params)
 
canonical_querystring = params.sort.collect do |key, value| 
  [aws_escape(key.to_s), aws_escape(value.to_s)].join('=') 
end.join('&')
 
string_to_sign = "GET\n#{amazon_endpoint}\n#{amazon_path}\n#{canonical_querystring}"
 
hmac = HMAC::SHA256.new(SECRET_IDENTIFIER)
hmac.update(string_to_sign)
signature = Base64.encode64(hmac.digest).chomp
 
params['Signature'] = signature
querystring = params.sort.collect do |key, value| 
  [aws_escape(key.to_s), aws_escape(value.to_s)].join('=') 
end.join('&')
 
signed_url = "http://#{amazon_endpoint}#{amazon_path}?#{querystring}"
 
cgi.out('type' => 'text/javascript') { "#{js_callback}('#{signed_url}');" }

You can test this locally by feeding key/value parameters to CGI, followed by Ctrl-D. These, for example:

amazon_endpoint=ecs.amazonaws.com
amazon_path=/onca/xml
js_callback=do_stuff
Service=AWSECommerceService
Version=2009-03-31
Operation=ItemSearch
SearchIndex=Books
Keywords=george+monbiot