Many UK academics can access the data via institutional subscription to EDINA Digimap. I’m using it in my research into subjective wellbeing and environmental quality.

This post shows how to:

1. import the data files into a PostGIS database; and
2. de-normalise the data into a single table, where there’s a one-to-one mapping of postcodes to rows, and each row contains either all geographical locations covered by a postcode (as a single geometry column, of type multipolygon) or the reason why no such location is available

Why de-normalise?

Step 2 above is required for my purposes because Code-Point data is supplied in a number of separate files per postcode area:

• a .shp file (and associated .shx and .dbf) mapping postcodes and ‘vertical streets’ to the locations they cover;
• an accompanying text file mapping postcodes to ‘vertical streets’; and
• another text file listing postcodes with no associated locations, either because they represent PO boxes or because the data just isn’t available.

Vertical streets generally represent high-rise buildings, where one location in 2D space may be associated with multiple postcodes. Vertical streets are also a serious pain: not only may one vertical street be associated with many postcodes, but one postcode may be associated with many vertical streets and with non-vertical-street locations too.

Import the data files

Download and unzip the Code-Point with Polygons data, in ESRI .shp format, for the regions you want (where indicative query timings are provided later, these are for whole-UK data — 120 postcode areas — on a 2Ghz Intel iMac).

Dump everything in the same directory. For each postcode area XX you should have the following five files:

• XX.shp, XX.shx and XX.dbf
• XX_vstreet_lookup.txt
• XX_discard.txt

If you don’t already have one you want to use for this purpose, create a PostGIS-enabled PostgreSQL database (in this post, the database is named geo). If you’re not running PostgreSQL 8.4 or later you may need to fiddle with some fsm_ settings in the Postgres .conf file in order to cope with a data set this large.

Execute the following SQL to create the tables for discards and vertical streets (e.g. from within pgAdmin):

create table cpp_vertical_streets (
  postcode character varying(8),
  vstreet character varying(8)
);
 
create table cpp_discards (
  postcode character varying(8),
  reason character varying(7)
);

Next we need to create the table for the .shp-file polygons. We switch to Ruby for this — you can just enter the commands in an IRB terminal session:

path = '/path/to/CodePoint polygons'
`/usr/local/pgsql/bin/shp2pgsql "#{path}/ab.shp" cpp_polygons -p -D -s 27700 | /usr/local/pgsql/bin/psql -d geo -U postgres`

The -p flag to shp2pgsql just creates a table structure, and the -s 27700 gives it the EPSG code to tell it we’re using the OSGB36 datum. You can use any of the .shp files you’ve downloaded — it need not be the AB area one.

Now to do the import into the three tables. Still in Ruby, execute the following loop:

Dir["#{path}/*.shp"].each do |f|
  puts f
  `/usr/local/pgsql/bin/shp2pgsql "#{f}" cpp_polygons -a -D -s 27700 | /usr/local/pgsql/bin/psql -d geo -U postgres`
  `echo "copy cpp_vertical_streets from '#{f.sub(/\.shp$/, '_vstreet_lookup.txt')}' csv;" | /usr/local/pgsql/bin/psql geo postgres`
  `echo "copy cpp_discards from '#{f.sub(/\.shp$/, '_discard.txt')}' csv;" | /usr/local/pgsql/bin/psql geo postgres`
end

Back in SQL, add some boolean flags we’ll use later on, then create an index on the postcode column, which will save a lot of time later:

alter table cpp_polygons add column discard boolean;
alter table cpp_polygons add column vstreet boolean;
alter table cpp_polygons add column vstreet_and_std boolean;
 
create index pc_index on cpp_polygons (postcode);
 
vacuum analyze cpp_polygons;

If you like, you can confirm here that there’s only one row per postcode — this query should return nothing:

select * 
from cpp_polygons a 
inner join cpp_polygons b 
on a.postcode = b.postcode 
where a.gid <> b.gid;

So, we now have all the OS data held in three separate tables. In the rest of the post, we’ll be merging the data in the discards and vertical streets tables into the main table, cpp_polygons.

Discards

We’re going to create a new table, with the same structure as cpp_polygons, to hold the discarded postcode data. These postcodes will have a NULL geometry column, and a TRUE discard column (one of the booleans we added earlier). Later, we’ll insert the contents of this table into cpp_polygons.

Run the following SQL:

create sequence cpp_discard_seq start with 2000000;
 
create table cpp_discard_polys as (
select 
  nextval('cpp_discard_seq') as gid, 
  postcode,
  cast(null as character varying(20)) as upp, 
  substring(postcode from '^[A-Z][A-Z]?') as pc_area, 
  cast(null as geometry) as the_geom, 
  true as discard, 
  false as vstreet,
  false as vstreet_and_std
from cpp_discards
);

Vertical streets

Similarly, we’re now going to create a new table with the same structure as cpp_polygons to map vertical street postcodes to the right polygons. This requires a left join of the vertical streets data with some of the geometry data in cpp_polygons.

set enable_seqscan = false; 
-- (otherwise pg sometimes fails to use the index)
 
create sequence cpp_vstreet_seq start with 3000000;
 
create table cpp_vstreet_polys as (
  select 
    nextval('cpp_vstreet_seq') as gid, 
    max(v.postcode) as postcode, 
    cast(null as varchar(20)) as upp, 
    max(pc_area) as pc_area, 
    st_union(the_geom) as the_geom, 
    false as discard,
    true as vstreet,
    false as vstreet_and_std
  from cpp_vertical_streets v 
  left join cpp_polygons p 
  on v.vstreet = p.postcode
  group by v.postcode
);

Our last new table is for postcodes that are associated with both standard polygons and vertical street polygons.

create sequence cpp_vstreet_and_std_seq start with 4000000;
 
create table cpp_vstreet_and_std_polys as (
select 
  nextval('cpp_vstreet_and_std_seq') as gid, 
  p.postcode as postcode,
  cast(null as varchar(20)) as upp, 
  p.pc_area as pc_area,
  st_multi(st_union(p.the_geom, v.the_geom)) as the_geom, 
  false as discard, 
  false as vstreet,
  true as vstreet_and_std
from cpp_polygons p
inner join cpp_vstreet_polys v
on p.postcode = v.postcode
);

Cleaning and merging

Now we need to remove the vertical streets and polygons we just merged into a new table from their respective source tables.

delete from cpp_polygons where postcode in (select postcode from cpp_vstreet_and_std_polys);
-- the above could take around 25 mins
delete from cpp_vstreet_polys where postcode in (select postcode from cpp_vstreet_and_std_polys);

And, finally, to merge our three new tables into the main table.

insert into cpp_polygons select * from cpp_discard_polys;
insert into cpp_polygons select * from cpp_vstreet_polys;
insert into cpp_polygons select * from cpp_vstreet_and_std_polys;

In order to join my own data with this data, I find it easiest to format the postcodes on which the join is made with no spaces. So I add and index the following extra column.

alter table cpp_polygons add column postcode_no_sp character varying(8);
update cpp_polygons set postcode_no_sp = replace(postcode, ' ', ''); 
-- the above could take 10 - 15 mins
create index pcns_index on cpp_polygons (postcode_no_sp);
 
vacuum analyze cpp_polygons;

You might also want to remove the original vertical street rows, where the postcode column begins with a ‘V’, from the cpp_polygons table — I didn’t have any need for this.

Step 1

The first step is to create an appropriate table, and pull everything in as character data.

create table nspd_2008_11 (
 postcode_7 char(7),
 postcode_8 char(8),
 postcode_egif char(8),
 intro_date_string char(6),
 termination_date_string char(6),
 county char(2),
 la char(2),
 ward char(2),
 large_user char(1),
 grid_easting char(6),
 grid_northing char(7),
 grid_ref_quality char(1),
 health_authority char(3),
 pan_sha char(3),
 country char(3),
 non_geo char(1),
 in_paf char(1),
 go_region char(1),
 ssr char(1),
 parl_const char(3),
 eer char(2),
 tecr char(3),
 ttwa char(3),
 pct char(5),
 nuts char(10),
 ed_1991_ogss char(8),
 ed_1991 char(6),
 ed_quality char(1),
 address_count char(4),
 delivery_point_count char(4),
 multiple_occupancy_count char(4),
 small_business_count char(4),
 previous_sha char(3),
 lea char(3),
 ha char(3),
 ward_1991 char(6),
 ward_1991_ogss char(6),
 ward_1998 char(6),
 stat_ward_2005 char(6),
 oa char(10),
 oa_indicator char(1),
 cas_ward char(6),
 national_park char(2),
 lsoa char(9),
 scottish_dzone char(9),
 msoa char(9),
 urban_rural_ew char(1),
 urban_rural_scot char(1),
 urban_rural_ni char(1),
 scottish_izone char(9),
 soa_ni char(8),
 oa_class char(3),
 old_pct char(5)
);
 
copy nspd_2008_11 from '/path/to/nspd/file.csv' csv;

Step 2

Since we’re going to the trouble of bringing it into PostGIS, likelihood is we’re going to want to use the spatial data. So let’s create a new table to hold the point geometries.

Notes:

• If necessary, we can join this table with our original table, on a postcode column, in future queries. We’ll index the relevant postcode columns in each table (as primary keys) to speed this up.
• We include OIDS, even though they are deprecated, as an easy way to permit access to the data for visualisation using qGIS.
• We exclude postcodes with the quality code ’9′, because this code means there is no grid reference available.
• We set the SRID of the point data to 27700, which is the appropriate value for the OSGB36 projection here.

create table nspd_2008_11_geometry with (oids) as (
  select postcode_7,
         grid_ref_quality,
         st_setsrid(st_makepoint(
           cast (grid_easting as integer),
           cast (grid_northing as integer)
         ), 27700) as point
  from nspd_2008_11
  where grid_ref_quality != '9'
);
 
alter table nspd_2008_11
add constraint postcode_7_key primary key (postcode_7);
 
alter table nspd_2008_11_geometry
add constraint postcode_7_geometry_key primary key (postcode_7);
 
vacuum analyze;

Now we can connect to the geometry table from qGIS to see the rough shape of Britain in postcode points. In a future post, I may look at creating polygons to represent postcode sectors using the data we’ve just imported.