George MacKerron: code blog

PostGIS has an st_askml function. This turns geometries into fragments of KML, and thus takes you most of the way to easy visualisation of spatial queries using Google Earth. But not all the way: these fragments have then to be assembled into a complete document.

I’ve written some wrapper and aggregate functions to automate this. They’re probably deeply inefficient — I wouldn’t advocate building your next web service on them — but for one-off eyeballing of query results I find them really useful.

The key functions are called as_kmldoc; you could see them as the missing aggregate versions of st_askml.

More »

My current iPhone app/research project, mappiness, is heavily reliant on Apple’s Push Notification Service (APNS).

We’re now sending about one million notifications a month, and it’s working beautifully. The obvious alternative — using SMS messages — would be costing us tens of thousands of pounds a month.

If you’re using or contemplating using Push Notifications, you may find these four points of interest:

1. Urban Airship offer a well-though-out and reliable service. But if you’re using them just as an intermediary in telling the APNS to beep users X, Y and Z right now — not making use of their scheduling or broadcast features, for example — it might well not be worth it. You’ll cut out a layer of indirection, and save money if you get big enough for it to matter, by talking to Apple directly.
2. There’s a surprising dearth of viable-looking open-source options for interfacing with the APNS. (APNS uses a binary protocol over a secure connection, and doesn’t like this connection torn down and reopened too often, so you can’t easily interface with it directly from a web app or cron job). But there is pyapns, an XML-RPC-speaking APNS provider built on Python’s Twisted networking framework, with client libraries for Python and Ruby. I’ve found this straightforward and rock-solid.
3. The APNS feedback service sometimes reports device tokens as inactive for no obvious reason, so make sure you’re prepared to hear from and reactivate users you’ve previously inactivated. (mappiness got bitten by this early on: a few dozen people were unable to authenticate to our server because they’d been reported as inactive and we’d assumed they were never coming back, despite that fact they had the app installed and Push Notifications switched on).
4. The app delegate method application:didRegisterForRemoteNotificationsWithDeviceToken: seems occasionally to return phantom zero-length device tokens — so don’t let your app or server get flummoxed by these. (We suffered from this on mappiness too: the bad device token got added to the front of a queue for sending from the app to our server; the server wouldn’t accept it; and this blocked all further data getting sent back until we fixed the server to silently ignore the bad tokens).

Perl, Ruby, and some other languages support a readable ‘extended’ regular expression syntax, in which literal whitespace is ignored and comments (starting with #) are available. They also support a multi-line mode where the . character matches anything, including a newline.

JavaScript does neither of these: it doesn’t recognise the extended syntax, and its version of multi-line only allows the ^ and $ characters to match the beginnings and ends of lines within a string (it will never allow the . to match a newline).

So I wrote the following function to convert extended and fully-multi-line RegExp source strings to the basic syntax that JavaScript understands.

More »

I recently switched to TextMate for editing Stata .do files: unlike Stata’s built-in editor on the Mac, it has syntax highlighting and other goodies via Timothy Beatty’s bundle (now hosted by Dan Byler).

One thing it doesn’t have, though, is tab completion. Or rather, it didn’t, until now.

More »

Ordnance Survey’s Code-Point with Polygons “provides a precise geographical location for each postcode unit in Great Britain”. It’s available in various formats, including ESRI .shp files.

Many UK academics can access the data via institutional subscription to EDINA Digimap. I’m using it in my research into subjective wellbeing and environmental quality.

This post shows how to:

1. import the data files into a PostGIS database; and
2. de-normalise the data into a single table, where there’s a one-to-one mapping of postcodes to rows, and each row contains either all geographical locations covered by a postcode (as a single geometry column, of type multipolygon) or the reason why no such location is available

More »

I have a simple site that generates covers for CDs I burn from iTunes purchases and so on (it pre-dates widespread use of JS libraries, and is in much need of prettifying). The site uses Amazon Product Advertising API calls to search and retrieve album cover art and track listings. Since earlier this month, such API calls have to be cryptographically signed.

This is somewhat annoying — the site’s original design has it communicating independently with Amazon (using Amazon’s XSLT API feature to transform their XML data into JSON), and that’s no longer possible with the use of a private key. But it’s not unfixable. The site now sends its API call first to my server, which returns a signed version, and then forwards the signed call on to Amazon.

I found most of what I needed for this on Chris Roos’ blog, but his version still wasn’t quite working for me (the two problems I recall are that Ruby’s CGI.escape doesn’t quite follow Amazon’s requirements, and that times need converting to GMT).

More »

Until Monday, O2 is offering a half-price USB dongle for pay-as-you-go mobile broadband. Buy it via Quidco, and you get not only the dongle but also a month’s access (or 3GB, whichever is the sooner) absolutely free.

So: free modem. No further commitment. And the opportunity to buy access a day at a time for £2, on a train journey, for example, or in an airport lounge, where the cheapest paid-for WiFi option is rarely less than £5, and often nearer £10. Handy.

Update. O2 is no longer doing the modem at half price, but the Quidco cashback has increased so that it still fully covers the modem and a £15 top-up.

More »

Between Time Machine and services like Dropbox, paranoid levels of backup are surprisingly painless to achieve on the Mac these days.

Still, just one more copy of your data, in just one more continent, surely can’t do any harm, right? One that won’t burn down with your house, but also isn’t just wafting vaguely in the Cloud at someone else’s whim. One that elevates your backup system from sensibly paranoid to borderline OCD. One, in this case, brought to you by rsync, find and Growl.

More »

I’ve been unhappy for a while having the SSH daemon on my web server VPSs with password authentication enabled. Of course, it’s on a non-standard port, blocks root logins, and takes a strong-ish password… but the risk of a successful dictionary attack has still felt too non-zero for comfort.

Equally, though, I’ve not wanted to give up the ability to log in to the servers from anywhere to fix things in an emergency, so I didn’t want to turn password authentication off and rely on public keys alone.

Until now, that is. I realised yesterday that, since I have iSSH on the iPhone, which does public key authentication, I can log into my servers from anywhere, even with password authentication turned off. Granted, doing anything serious on a tiny screen and slow connection is difficult. But all I actually need to be able to do from there is temporarily turn password authentication back on.

And to make this easy, I’ve put two ultra-simple scripts in ~/bin:

pwd_login_on.sh

#!/bin/bash
sudo sed -r -e 's/^PasswordAuthentication no$/PasswordAuthentication yes/' \
  -i.previous /etc/ssh/sshd_config
sudo /etc/init.d/ssh restart

pwd_login_off.sh

#!/bin/bash
sudo sed -r -e 's/^PasswordAuthentication yes$/PasswordAuthentication no/' \
  -i.previous /etc/ssh/sshd_config
sudo /etc/init.d/ssh restart

(These paths are suitable for Ubuntu 8.04).

Lovely.

Quick tip: you can get hold of some of Adobe’s very nice professional fonts for free when you download the InDesign CS4 trial (and possibly other CS4 apps too).

These include:

• Caslon Pro
• Chaparral Pro
• Garamond Pro
• Minion Pro
• Myriad Pro

If you don’t want to actually install InDesign, you can get to the fonts like so (if you’re a Mac user):

• Mount (double-click) the downloaded disk image
• Mount another disk image found on the newly mounted disk, at Adobe InDesign CS4/payloads/AdobeFontsAll/AdobeFontsAll.dmg
• The fonts are inside /Assets/contents on this second disk

Note that these fonts may well be covered by a very restrictive licence: I haven’t checked.