George MacKerron: code blog » GIS

Polygons from PostGIS to Processing

George — Thu, 17 Nov 2011 11:28:43 +0000

There are plenty of ways to get spatial data from a PostGIS database into a Processing sketch.

You can export to CSV or SVG and load it from there; you can query the database directly; or, depending on context, you might choose to generate Processing commands directly, which is the route I went to display a background map of the UK in a recent visualization project.

First, I load outlines of the 4 constituent countries of the UK into my PostGIS database.

shp2pgsql -d -D -I -s 27700 "/pgdata/geodata/Nations/england.shp" eng \
  | sudo -u postgres psql mappiness
shp2pgsql -d -D -I -s 27700 "/pgdata/geodata/Nations/wales.shp" wal \
  | sudo -u postgres psql mappiness
shp2pgsql -d -D -I -s 27700 "/pgdata/geodata/Nations/scotland.shp" sco \
  | sudo -u postgres psql mappiness
shp2pgsql -d -D -I -s 27700 "/pgdata/geodata/Nations/nireland_ol_2001.shp" ni \
  | sudo -u postgres psql mappiness

Next, I combine these into one table, using st_simplifypreservetopology to discard a lot of unnecessary detail and st_dump to dissolve some English MULTIPOLYGONs into simple POLYGONS.

create table uk1km as (
        select st_simplifypreservetopology(the_geom, 1000) as the_geom from 
          (select (st_dump(the_geom)).geom as the_geom from eng) as eng_single
  union select st_simplifypreservetopology(the_geom, 1000) as the_geom from wal
  union select st_simplifypreservetopology(the_geom, 1000) as the_geom from sco
  union select st_simplifypreservetopology(the_geom, 1000) as the_geom from ni
);
alter table uk1km add column id serial primary key;
drop table eng; drop table wal; drop table sco; drop table ni;

Finally, I use string_agg to produce the vertex() commands required to reproduce my polygons in Processing. I change the sign on the y-coordinate (from northings to ‘southings’, to match the Processing coordinate space). I scale the OSGB36 coordinates down from metres to kilometres (I could otherwise have done this with a scale() command in Processing). And I discard a lot of small polygons (comprising 20 POINTs or fewer) I don’t need in my output.

select 'beginShape(); ' 
    || string_agg('vertex(' || st_x(geom) || ', ' || -st_y(geom) || '); ', '')
    || 'endShape(CLOSE); ' 
from (
  select id, (st_dumppoints(st_exteriorring(st_scale(the_geom, 0.001, 0.001)))).geom as geom 
  from uk1km 
  where st_numpoints(st_exteriorring(the_geom)) > 20
) as d
group by id;

Then, to draw the UK in a sketch I do this.

size(350, 620);
translate(10, height);
scale(0.5);
 
// paste in the output of the previous query

Which has this result (with thanks to EDINA for the data):

Approximating kernel-weighted proportions in PostGIS

George — Tue, 16 Aug 2011 13:27:23 +0000

Imagine you want compare various locations in terms of the availability of a certain type of environment, such as fresh water.

You might want to use a measure of the proximity of that environment — such as the nearest neighbour distance.

You might want to use a measure of the quantity of that environment in the vicinity — such as the proportion of land within a specific radius that is of that type.

Or you might ideally like a measure that combines both of these: one that incorporates the quantity of that environment, but gives greater weight to areas that are nearer, and lesser weight to those that are further away.

In that third case, what you probably want is a kernel-weighted proportion.

How it works

In this usage, a ‘kernel’ just means a distribution function, such as the normal (Gaussian), triangular, or Epanechnikov. The kernel is used to decide what weight should be attached to places at varying distances from the location under consideration.

I find it helpful to imagine the weight assigned to each point as a third dimension: the height at each point. I then imagine the kernel as a solid object positioned with its centre over the point I’m measuring from. For the normal or Epanechnikov kernels, this object is bell-shaped; for the triangular kernel, it’s a cone; for the (trivial) uniform kernel, it’s a cylinder.

The total volume of this object is the denominator of the kernel-weighted proportion. Now I imagine removing all parts of the object that don’t lie directly above the type of environment being measured. The volume of the remaining parts of the object — those directly overlying the environment of interest — is the numerator of the kernel-weighted proportion.

Implementation in PostGIS

The following SQL functions enable you to approximate kernel-weighted proportions in PostGIS. They do so by splitting the 3-dimensional kernel into horizontal slices of equal height, but with radii differing according to the kernel and kernel parameters specified. They calculate the summed volume of the intersections of the slices with the supplied polygons, and divide that by the total summed volume of the complete slices. (Assign each slice a height of 1 and these volumes are helpfully the same as the areas).

This approach is illustrated, for the normal kernel, at the top of this post.

-- kernel functions
 
create or replace function uniform_pdf
( double precision              -- $1 = x
, double precision default 1.0  -- $2 = bandwidth
, double precision default 0.0  -- $3 = centre
) returns double precision as $$
  select
    case
      when $1 > $3 - $2 and $1 < $3 + $2 then (
        select cast(0.5 as double precision)
      )
      else 0
    end;
$$ language sql immutable;
 
create or replace function triangular_pdf
( double precision              -- $1 = x
, double precision default 1.0  -- $2 = bandwidth
, double precision default 0.0  -- $3 = centre
) returns double precision as $$
  select
    case
      when $1 > $3 - $2 and $1 < $3 + $2 then (
        select 1 - abs(($1 - $3) / $2)
      )
      else 0
    end;
$$ language sql immutable;
 
create or replace function normal_pdf
( double precision              -- $1 = x
, double precision default 1.0  -- $2 = std dev (bandwidth)
, double precision default 0.0  -- $3 = mean (centre)
) returns double precision as $$
  select (1.0 / (sqrt(2.0 * pi() * pow($2, 2)))) 
       * exp(-pow($1 - $3, 2) / (2.0 * pow($2, 2)));
$$ language sql immutable;
 
create or replace function epanechnikov_pdf
( double precision              -- $1 = x
, double precision default 1.0  -- $2 = bandwidth
, double precision default 0.0  -- $3 = centre
) returns double precision as $$
  select
    case
      when $1 > $3 - $2 and $1 < $3 + $2 then (
        select 0.75 * (1 - pow(($1 - $3) / $2, 2))
      )
      else 0
    end;
$$ language sql immutable;
 
 
-- current kernel function 
-- (uncomment the kernel you want to use, and redefine: the normal is shown here)
 
create or replace function __current_kernel_pdf
( double precision              -- $1 = x
, double precision default 1.0  -- $2 = std dev/bandwidth
, double precision default 0.0  -- $3 = mean/centre
) returns double precision as $$
  select
    normal_pdf
    -- epanechnikov_pdf
    -- triangular_pdf
    -- uniform_pdf
    ($1, $2, $3);
$$ language sql immutable;
 
 
-- support functions
 
create or replace function __slice_height
( double precision  -- $1 = kernel std dev
, double precision  -- $2 = kernel radius at top of slice
, double precision  -- $3 = kernel radius at bottom of slice
) returns double precision as $$
  select __current_kernel_pdf($2, $1) - __current_kernel_pdf($3, $1);
$$ language sql immutable;
 
create or replace function __slice_radius
( double precision  -- $1 = kernel radius at top of slice
, double precision  -- $2 = kernel radius at bottom of slice
) returns double precision as $$
  select $1 + (($2 - $1) / 2);
$$ language sql immutable;
 
create or replace function __kernel_slice_volume
( double precision  -- $1 = kernel std dev
, double precision  -- $2 = kernel radius at top of slice
, double precision  -- $3 = kernel radius at bottom of slice
, int               -- $4 = buffer precision
) returns double precision as $$
  select
    coalesce(
      st_area(
        st_buffer(
          st_makepoint(0, 0),
          __slice_radius($2, $3),
          $4
        )
      ),
      0
    )
    * __slice_height($1, $2, $3);
$$ language sql immutable;
 
create or replace function __intersected_slice_volume
( geometry          -- $1 = area geometry
, geometry          -- $2 = kernel centre point geometry
, double precision  -- $3 = kernel std dev
, double precision  -- $4 = kernel radius at top of slice
, double precision  -- $5 = kernel radius at bottom of slice
, int               -- $6 = buffer precision
) returns double precision as $$
  select
    coalesce(
      st_area(
        st_intersection(
          $1,
          st_buffer(
            $2,
            __slice_radius($4, $5),
            $6
          )
        )
      ),
      0
    )
    * __slice_height($3, $4, $5);
$$ language sql immutable;
 
 
-- main function
 
create or replace function kernel_weighted_local_proportion
( geometry          -- $1 = area geometry
, geometry          -- $2 = kernel centre point geometry
, double precision  -- $3 = kernel std dev
, double precision  -- $4 = truncation bandwidth (for normal only -- for others, repeat $3)
, int               -- $5 = number of horizontal slices for approximation (for uniform, use 1)
, int               -- $6 = buffer precision (number of points per 1/4 circle)
) returns double precision as $$
  select
    sum(__intersected_slice_volume(
      $1, $2, $3,
      $4 * (cast(s as double precision) / $5),     -- kernel radius at top of slice
      $4 * (cast(s + 1 as double precision) / $5), -- kernel radius at bottom of slice
      $6
    ))
    /
    sum(__kernel_slice_volume(
      $3,
      $4 * (cast(s as double precision) / $5),     -- kernel radius at top of slice
      $4 * (cast(s + 1 as double precision) / $5), -- kernel radius at bottom of slice
      $6
    ))
  from generate_series(0, $5 - 1) s;
$$ language sql immutable;

Overlapping markers on your Google Map? Meet OverlappingMarkerSpiderfier

George — Wed, 22 Jun 2011 20:19:25 +0000

Ever noticed how, in Google Earth, marker pins that overlap each other spring apart gracefully when you click them, so you can pick the one you meant?

And ever noticed how, when using the Google Maps API, the exact same thing doesn’t happen?

This code makes Google Maps API version 3 map markers behave in that Google Earth way. Small numbers of markers (up to 8, configurable) spiderfy into a circle. Larger numbers fan out into a (more space-efficient) spiral.

Try it! (You can reload this page to re-randomise the marker positions).

The code is MIT licenced, has no dependencies, and is under 3K compiled (from CoffeeScript), minified and gzipped. I wrote it as part of the data download feature for Mappiness.

It’s on Github, where you can find out more, download or fork the code.

Simple PostGIS nearest neighbour function

George — Thu, 10 Mar 2011 11:31:59 +0000

Here’s a less generic and slightly different nearest-neighbour function based on Regina’s generic nearest-neighbour function at Boston GIS.

It follows the same basic idea of using series of enlarging search radii to restrict distance calculations to a manageable subset of things-that-might-be-near. The difference is that it uses a geometric progression of sizes (x, x * y, x * y^2, x * y^3, ...) instead of an arithmetic one (x, x + y, x + 2y, x + 3y, ...).

For some distributions of things-that-might-be-near, and tuned with the right parameters (x, y), this turns out substantially faster (I’ve used it to locate the nearest UK postcode to each mappiness response).

Note that it’s a lot simpler and less generic than the Boston GIS alternative: it returns only one nearest neighbour, without the associated distance, and is also less flexible about what it accepts as input.

create or replace function 
  nn(nearTo                   geometry
   , initialDistance          real
   , distanceMultiplier       real 
   , maxPower                 integer
   , nearThings               text
   , nearThingsIdField        text
   , nearThingsGeometryField  text)
returns integer as $$
declare 
  sql     text;
  result  integer;
begin
  sql := ' select ' || quote_ident(nearThingsIdField) 
      || ' from '   || quote_ident(nearThings)
      || ' where st_dwithin($1, ' 
      ||   quote_ident(nearThingsGeometryField) || ', $2 * ($3 ^ $4))'
      || ' order by st_distance($1, ' || quote_ident(nearThingsGeometryField) || ')'
      || ' limit 1';
  for i in 0..maxPower loop
    execute sql into result using nearTo              -- $1
                                , initialDistance     -- $2
                                , distanceMultiplier  -- $3
                                , i;                  -- $4
    if result is not null then return result; end if;
  end loop;
  return null;
end
$$ language 'plpgsql' stable;

Update, August 2011 — Looking at this again, I realise there was an error in the function originally posted above. Because it did only a simple bounding-box check at each step, it was possible for it to return a geometry that was not in fact the nearest, but could in the worst case be up to 41% (sqrt(2) times) further away than the nearest. The code has been updated to fix this by using st_dwithin. I’ve also tidied it up a bit.

as_kmldoc: easily visualise PostGIS queries as KML in Google Earth

George — Wed, 29 Sep 2010 13:58:33 +0000

PostGIS has an st_askml function. This turns geometries into fragments of KML, and thus takes you most of the way to easy visualisation of spatial queries using Google Earth. But not all the way: these fragments have then to be assembled into a complete document.

I’ve written some wrapper and aggregate functions to automate this. They’re probably deeply inefficient — I wouldn’t advocate building your next web service on them — but for one-off eyeballing of query results I find them really useful.

The key functions are called as_kmldoc; you could see them as the missing aggregate versions of st_askml.

How’s it used?

as_kmldoc has three flavours, differing only in the number of arguments (arity). There are versions for only a geometry, a geometry plus a textual field (which becomes an associated KML element), or a geometry plus two textual fields (which become the and in that order).

Example 1

I have a table of polygons corresponding to areas marked as parks in the OpenStreetMap data. I want to see some of these in Google Earth to check them against the satellite imagery. So I run:

select as_kmldoc(osgb36_polygon) from osm_green_spaces;

Then I paste the resulting KML string into Google Earth, and fiddle a little with the colour and transparency:

Example 2

I have a table of weather station locations, and associated names and codes, as used by Weather Underground. I can select all this data into a KML document by running:

select as_kmldoc(osgb_location, name, code) from wunderground_stations;

Which gives me:

Example 3

This one’s just showing off. Here I visualise the calculation of a kernel-weighted average — which I plan to post more about in future — using the Z dimension (height) to illustrate the relative weightings at different locations.

The complete query is a bit long and involved to reproduce here. But it makes use of one more new function, join_kmldocs, which enables you to combine any number of KML documents into one, by (slightly hackily) making each input document into a element of the output document.

How do I get it?

To add these functions to your PostGIS database, run the following query:

create or replace function xml_escape(text)
returns text as $$
  select replace(replace(replace($1, '&', '&'), '<', '<'), '>', '>');
$$ language sql immutable strict;
 
create or replace function kml_wrap(text)
returns text as $$
  select
    ''
    || $1 || '';
$$ language sql immutable strict;
 
create or replace function kml_concat(text, geometry)
returns text as $$
  select $1 || '' || st_askml($2) || '';
$$ language sql immutable strict;
 
create aggregate as_kmldoc(geometry) (
    sfunc = kml_concat,
    finalfunc = kml_wrap,
    stype = text,
    initcond = ''
);
 
create or replace function kml_concat(text, geometry, anyelement)
returns text as $$
  select $1 || '' || xml_escape(cast($3 as text)) || ''
  || st_askml($2) || '';
$$ language sql immutable strict;
 
create aggregate as_kmldoc(geometry, anyelement) (
    sfunc = kml_concat,
    finalfunc = kml_wrap,
    stype = text,
    initcond = ''
);
 
create or replace function kml_concat(text, geometry, anyelement, anyelement)
returns text as $$
  select $1 || '' || xml_escape(cast($3 as text))
  || '' || xml_escape(cast($4 as text))
  || '' || st_askml($2) || '';
$$ language sql immutable strict;
 
create aggregate as_kmldoc(geometry, anyelement, anyelement) (
    sfunc = kml_concat,
    finalfunc = kml_wrap,
    stype = text,
    initcond = ''
);
 
create or replace function join_kmldocs(variadic text[])
returns text as $$
  select kml_wrap(
    replace(
      replace(
        array_to_string($1, ''),
        '',
        ''
       ),
       '',
       ''
    )
  );
$$ language sql immutable strict;

pgAdmin hint

I work with PostGIS mostly via pgAdmin. If you’re using as_kmldoc via pgAdmin, you might find it useful to change pgAdmin’s ‘Result copy quoting’ to ‘None’ in Preferences > Query Tool. That way, you can copy and paste from pgAdmin straight into Google Earth.

Using OS Code-Point Polygons in PostGIS

George — Sat, 14 Nov 2009 12:13:43 +0000

Ordnance Survey’s Code-Point with Polygons “provides a precise geographical location for each postcode unit in Great Britain”. It’s available in various formats, including ESRI .shp files.

Many UK academics can access the data via institutional subscription to EDINA Digimap. I’m using it in my research into subjective wellbeing and environmental quality.

This post shows how to:

import the data files into a PostGIS database; and
de-normalise the data into a single table, where there’s a one-to-one mapping of postcodes to rows, and each row contains either all geographical locations covered by a postcode (as a single geometry column, of type multipolygon) or the reason why no such location is available

Why de-normalise?

Step 2 above is required for my purposes because Code-Point data is supplied in a number of separate files per postcode area:

a .shp file (and associated .shx and .dbf) mapping postcodes and ‘vertical streets’ to the locations they cover;
an accompanying text file mapping postcodes to ‘vertical streets’; and
another text file listing postcodes with no associated locations, either because they represent PO boxes or because the data just isn’t available.

Vertical streets generally represent high-rise buildings, where one location in 2D space may be associated with multiple postcodes. Vertical streets are also a serious pain: not only may one vertical street be associated with many postcodes, but one postcode may be associated with many vertical streets and with non-vertical-street locations too.

Import the data files

Download and unzip the Code-Point with Polygons data, in ESRI .shp format, for the regions you want (where indicative query timings are provided later, these are for whole-UK data — 120 postcode areas — on a 2Ghz Intel iMac).

Dump everything in the same directory. For each postcode area XX you should have the following five files:

XX.shp, XX.shx and XX.dbf
XX_vstreet_lookup.txt
XX_discard.txt

If you don’t already have one you want to use for this purpose, create a PostGIS-enabled PostgreSQL database (in this post, the database is named geo). If you’re not running PostgreSQL 8.4 or later you may need to fiddle with some fsm_ settings in the Postgres .conf file in order to cope with a data set this large.

Execute the following SQL to create the tables for discards and vertical streets (e.g. from within pgAdmin):

create table cpp_vertical_streets (
  postcode character varying(8),
  vstreet character varying(8)
);
 
create table cpp_discards (
  postcode character varying(8),
  reason character varying(7)
);

Next we need to create the table for the .shp-file polygons. We switch to Ruby for this — you can just enter the commands in an IRB terminal session:

path = '/path/to/CodePoint polygons'
`/usr/local/pgsql/bin/shp2pgsql -p -D -s 27700 "#{path}/ab.shp" cpp_polygons | /usr/local/pgsql/bin/psql -d geo -U postgres`

The -p flag to shp2pgsql just creates a table structure, and the -s 27700 gives it the EPSG code to tell it we’re using the OSGB36 datum. You can use any of the .shp files you’ve downloaded — it need not be the AB area one.

Now to do the import into the three tables. Still in Ruby, execute the following loop:

Dir["#{path}/*.shp"].each do |f|
  puts f
  `/usr/local/pgsql/bin/shp2pgsql -a -D -s 27700 "#{f}" cpp_polygons | /usr/local/pgsql/bin/psql -d geo -U postgres`
  `echo "copy cpp_vertical_streets from '#{f.sub(/\.shp$/, '_vstreet_lookup.txt')}' csv;" | /usr/local/pgsql/bin/psql geo postgres`
  `echo "copy cpp_discards from '#{f.sub(/\.shp$/, '_discard.txt')}' csv;" | /usr/local/pgsql/bin/psql geo postgres`
end

Back in SQL, add some boolean flags we’ll use later on, then create an index on the postcode column, which will save a lot of time later:

alter table cpp_polygons add column discard boolean;
alter table cpp_polygons add column vstreet boolean;
alter table cpp_polygons add column vstreet_and_std boolean;
 
create index pc_index on cpp_polygons (postcode);
 
vacuum analyze cpp_polygons;

If you like, you can confirm here that there’s only one row per postcode — this query should return nothing:

select * 
from cpp_polygons a 
inner join cpp_polygons b 
on a.postcode = b.postcode 
where a.gid <> b.gid;

So, we now have all the OS data held in three separate tables. In the rest of the post, we’ll be merging the data in the discards and vertical streets tables into the main table, cpp_polygons.

Discards

We’re going to create a new table, with the same structure as cpp_polygons, to hold the discarded postcode data. These postcodes will have a NULL geometry column, and a TRUE discard column (one of the booleans we added earlier). Later, we’ll insert the contents of this table into cpp_polygons.

Run the following SQL:

create sequence cpp_discard_seq start with 2000000;
 
create table cpp_discard_polys as (
select 
  nextval('cpp_discard_seq') as gid, 
  postcode,
  cast(null as character varying(20)) as upp, 
  substring(postcode from '^[A-Z][A-Z]?') as pc_area, 
  cast(null as geometry) as the_geom, 
  true as discard, 
  false as vstreet,
  false as vstreet_and_std
from cpp_discards
);

Vertical streets

Similarly, we’re now going to create a new table with the same structure as cpp_polygons to map vertical street postcodes to the right polygons. This requires a left join of the vertical streets data with some of the geometry data in cpp_polygons.

set enable_seqscan = false; 
-- (otherwise pg sometimes fails to use the index)
 
create sequence cpp_vstreet_seq start with 3000000;
 
create table cpp_vstreet_polys as (
  select 
    nextval('cpp_vstreet_seq') as gid, 
    max(v.postcode) as postcode, 
    cast(null as varchar(20)) as upp, 
    max(pc_area) as pc_area, 
    st_union(the_geom) as the_geom, 
    false as discard,
    true as vstreet,
    false as vstreet_and_std
  from cpp_vertical_streets v 
  left join cpp_polygons p 
  on v.vstreet = p.postcode
  group by v.postcode
);

Our last new table is for postcodes that are associated with both standard polygons and vertical street polygons.

create sequence cpp_vstreet_and_std_seq start with 4000000;
 
create table cpp_vstreet_and_std_polys as (
select 
  nextval('cpp_vstreet_and_std_seq') as gid, 
  p.postcode as postcode,
  cast(null as varchar(20)) as upp, 
  p.pc_area as pc_area,
  st_multi(st_union(p.the_geom, v.the_geom)) as the_geom, 
  false as discard, 
  false as vstreet,
  true as vstreet_and_std
from cpp_polygons p
inner join cpp_vstreet_polys v
on p.postcode = v.postcode
);

Cleaning and merging

Now we need to remove the vertical streets and polygons we just merged into a new table from their respective source tables.

delete from cpp_polygons where postcode in (select postcode from cpp_vstreet_and_std_polys);
-- the above could take around 25 mins
delete from cpp_vstreet_polys where postcode in (select postcode from cpp_vstreet_and_std_polys);

And, finally, to merge our three new tables into the main table.

insert into cpp_polygons select * from cpp_discard_polys;
insert into cpp_polygons select * from cpp_vstreet_polys;
insert into cpp_polygons select * from cpp_vstreet_and_std_polys;

In order to join my own data with this data, I find it easiest to format the postcodes on which the join is made with no spaces. So I add and index the following extra column.

alter table cpp_polygons add column postcode_no_sp character varying(8);
update cpp_polygons set postcode_no_sp = replace(postcode, ' ', ''); 
-- the above could take 10 - 15 mins
create index pcns_index on cpp_polygons (postcode_no_sp);
 
vacuum analyze cpp_polygons;

You might also want to remove the original vertical street rows, where the postcode column begins with a ‘V’, from the cpp_polygons table — I didn’t have any need for this.

Importing the National Statistics Postcode Directory into PostGIS

George — Tue, 05 May 2009 10:19:45 +0000

This isn’t a very difficult task, but creating the table is somewhat laborious, so perhaps you’ll find the following SQL helpful.

Step 1

The first step is to create an appropriate table, and pull everything in as character data.

create table nspd_2008_11 (
 postcode_7 char(7),
 postcode_8 char(8),
 postcode_egif char(8),
 intro_date_string char(6),
 termination_date_string char(6),
 county char(2),
 la char(2),
 ward char(2),
 large_user char(1),
 grid_easting char(6),
 grid_northing char(7),
 grid_ref_quality char(1),
 health_authority char(3),
 pan_sha char(3),
 country char(3),
 non_geo char(1),
 in_paf char(1),
 go_region char(1),
 ssr char(1),
 parl_const char(3),
 eer char(2),
 tecr char(3),
 ttwa char(3),
 pct char(5),
 nuts char(10),
 ed_1991_ogss char(8),
 ed_1991 char(6),
 ed_quality char(1),
 address_count char(4),
 delivery_point_count char(4),
 multiple_occupancy_count char(4),
 small_business_count char(4),
 previous_sha char(3),
 lea char(3),
 ha char(3),
 ward_1991 char(6),
 ward_1991_ogss char(6),
 ward_1998 char(6),
 stat_ward_2005 char(6),
 oa char(10),
 oa_indicator char(1),
 cas_ward char(6),
 national_park char(2),
 lsoa char(9),
 scottish_dzone char(9),
 msoa char(9),
 urban_rural_ew char(1),
 urban_rural_scot char(1),
 urban_rural_ni char(1),
 scottish_izone char(9),
 soa_ni char(8),
 oa_class char(3),
 old_pct char(5)
);
 
copy nspd_2008_11 from '/path/to/nspd/file.csv' csv;

Step 2

Since we’re going to the trouble of bringing it into PostGIS, likelihood is we’re going to want to use the spatial data. So let’s create a new table to hold the point geometries.

Notes:

If necessary, we can join this table with our original table, on a postcode column, in future queries. We’ll index the relevant postcode columns in each table (as primary keys) to speed this up.
We include OIDS, even though they are deprecated, as an easy way to permit access to the data for visualisation using qGIS.
We exclude postcodes with the quality code ’9′, because this code means there is no grid reference available.
We set the SRID of the point data to 27700, which is the appropriate value for the OSGB36 projection here.

create table nspd_2008_11_geometry with (oids) as (
  select postcode_7,
         grid_ref_quality,
         st_setsrid(st_makepoint(
           cast (grid_easting as integer),
           cast (grid_northing as integer)
         ), 27700) as point
  from nspd_2008_11
  where grid_ref_quality != '9'
);
 
alter table nspd_2008_11
add constraint postcode_7_key primary key (postcode_7);
 
alter table nspd_2008_11_geometry
add constraint postcode_7_geometry_key primary key (postcode_7);
 
vacuum analyze;

Now we can connect to the geometry table from qGIS to see the rough shape of Britain in postcode points. In a future post, I may look at creating polygons to represent postcode sectors using the data we’ve just imported.