<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>George MacKerron: code blog</title>
	<atom:link href="http://blog.mackerron.com/feed/" rel="self" type="application/rss+xml" />
	<link>http://blog.mackerron.com</link>
	<description>GIS, software development, and other snippets</description>
	<lastBuildDate>Sun, 07 Feb 2010 15:16:31 +0000</lastBuildDate>
	<generator>http://wordpress.org/?v=2.9.2</generator>
	<language>en</language>
	<sy:updatePeriod>hourly</sy:updatePeriod>
	<sy:updateFrequency>1</sy:updateFrequency>
			<item>
		<title>Tab completion for Stata variables in TextMate</title>
		<link>http://blog.mackerron.com/2010/02/stata-textmate-tab-completion/</link>
		<comments>http://blog.mackerron.com/2010/02/stata-textmate-tab-completion/#comments</comments>
		<pubDate>Sat, 06 Feb 2010 16:59:33 +0000</pubDate>
		<dc:creator>George</dc:creator>
				<category><![CDATA[Mac]]></category>
		<category><![CDATA[Stata]]></category>

		<guid isPermaLink="false">http://blog.mackerron.com/?p=264</guid>
		<description><![CDATA[I recently switched to TextMate for editing Stata .do files: unlike Stata&#8217;s built-in editor on the Mac, it has syntax highlighting and other goodies via Timothy Beatty&#8217;s bundle. 

One thing it doesn&#8217;t have, though, is tab completion. Or rather, it didn&#8217;t, until now.



What does it do?

It&#8217;s the same thing you get in Stata&#8217;s Command window: [...]]]></description>
			<content:encoded><![CDATA[<p>I recently switched to <a href="http://macromates.com/">TextMate</a> for editing Stata .do files: unlike Stata&#8217;s built-in editor on the Mac, it has syntax highlighting and other goodies via <a href="http://www-users.york.ac.uk/~tb526/maconomics/state_of_the_dock_report.html">Timothy Beatty&#8217;s bundle</a>. </p>

<p>One thing it doesn&#8217;t have, though, is <a href="http://en.wikipedia.org/wiki/Command_line_completion">tab completion</a>. Or rather, it didn&#8217;t, until now.</p>

<p><span id="more-264"></span></p>

<h2>What does it do?</h2>

<p>It&#8217;s the same thing you get in Stata&#8217;s Command window: you type part of a variable name, press tab, and the name is completed as far as it can be without ambiguity. </p>

<p>For example, I type &#8212;</p>

<p><img src="/wp-content/uploads/2010/02/Screen-shot-2010-02-05-at-5-Feb-18.13.42.png" alt="tab fr" height="29" width="80" /></p>

<p>&#8212; I press tab &#8212;</p>

<p><img src="/wp-content/uploads/2010/02/Screen-shot-2010-02-05-at-5-Feb-18.13.51.png" alt="tab fruit_and_veg__q" height="29" width="177" /></p>

<p>&#8212; and the variable name is completed, leaving the cursor one space to the right and ready to keep typing. (Note that my use of Stata&#8217;s <code>tab[le]</code> command in these examples is an unfortunate coincidence and has nothing to do with tab completion! I could just as well have written <code>regress fr</code>. Sorry.)</p>

<p>In fact, <em>proper</em> tab completion (as found in Unixy shells, for file names, but not in Stata&#8217;s Command window) goes a little further than this: if you press tab at the end of an ambiguous name stem, it prompts you with all the possible endings.</p>

<p>And so it is with this TextMate bundle addition. I type &#8212;</p>

<p><img src="/wp-content/uploads/2010/02/Screen-shot-2010-02-05-at-5-Feb-18.14.11.png" alt="tab pol" height="29" width="88" /></p>

<p>&#8212; I press tab &#8212;</p>

<p><img src="/wp-content/uploads/2010/02/Screen-shot-2010-02-05-at-5-Feb-18.14.21.png" alt="tab politic" height="28" width="123" /></p>

<p>&#8212; and the name is completed up to the next ambiguity. So I press tab again &#8212;</p>

<p><img src="/wp-content/uploads/2010/02/Screen-shot-2010-02-05-at-5-Feb-18.14.33.png" alt="List of suggestions" height="104" width="310" /></p>

<p>&#8212; and I get a tooltip list of options. I add an &#8217;s&#8217; (just enough to pick one option out) &#8212;</p>

<p><img src="/wp-content/uploads/2010/02/Screen-shot-2010-02-05-at-5-Feb-18.15.57.png" alt="tab politics" height="30" width="134" /></p>

<p>&#8212; press tab again &#8212;</p>

<p><img src="/wp-content/uploads/2010/02/Screen-shot-2010-02-05-at-5-Feb-18.14.49.png" alt="tab politics_left_right__q" height="28" width="259" /></p>

<p>&#8212; and bingo!</p>

<h2>How does it work?</h2>

<p>Each time you press tab, the bundle asks Stata to run a simple .do file; this exports your current data set with an <code>if 0</code> condition, meaning that all the actual data is excluded, and we just get the variable names. (For detail, check the Ruby script).</p>

<p>A slightly unfortunate effect of this is that your Stata Results window (and log, if you&#8217;re keeping one) fills up with these <code>outsheet</code> commands. I hope that&#8217;s not a deal-breaker. If only Stata would provide some basic AppleScript support (please?) I might be able to do something about this.</p>

<h2>How do I get it?</h2>

<p>At some point I may roll this into a new bundle file to make installation easier (there are also a few fixes I&#8217;ve made to the original bundle). But in the meantime, you can add it to the bundle yourself.</p>

<p>Open the bundle editor (Bundles &gt; Bundle Editor &gt; Show Bundle Editor) and select the Stata bundle. Click the &#8216;+&#8217; button and select &#8216;New Command&#8217;. Give the command a name. Then, on the right, select Save: Nothing, Input: Selected Text or Word and Output: Replace Selected Text. Choose Activation: Key Equivalent: and a hotkey (I use Ctrl-Tab).</p>

<p>Then paste in the following Ruby script:</p>


<div class="wp_syntax"><div class="code"><pre class="ruby" style="font-family:monospace;"><span style="color:#008000; font-style:italic;">#!/usr/bin/env ruby</span>
&nbsp;
<span style="color:#CC0066; font-weight:bold;">require</span> ENV<span style="color:#006600; font-weight:bold;">&#91;</span><span style="color:#996600;">'TM_SUPPORT_PATH'</span><span style="color:#006600; font-weight:bold;">&#93;</span> <span style="color:#006600; font-weight:bold;">+</span> <span style="color:#996600;">'/lib/exit_codes.rb'</span>
dofile = <span style="color:#996600;">'/tmp/textmatestatavars.do'</span>
varfile = <span style="color:#996600;">'/tmp/textmatestatavars.tsv'</span>
&nbsp;
stem = ENV<span style="color:#006600; font-weight:bold;">&#91;</span><span style="color:#996600;">'TM_CURRENT_WORD'</span><span style="color:#006600; font-weight:bold;">&#93;</span>
stem = <span style="color:#996600;">''</span> <span style="color:#9966CC; font-weight:bold;">if</span> stem.<span style="color:#0000FF; font-weight:bold;">nil</span>? <span style="color:#006600; font-weight:bold;">||</span> stem.<span style="color:#9900CC;">match</span><span style="color:#006600; font-weight:bold;">&#40;</span><span style="color:#006600; font-weight:bold;">/</span>\s$<span style="color:#006600; font-weight:bold;">/</span><span style="color:#006600; font-weight:bold;">&#41;</span>
stem = stem.<span style="color:#9900CC;">lstrip</span>
&nbsp;
TextMate.<span style="color:#9900CC;">exit_discard</span> <span style="color:#9966CC; font-weight:bold;">if</span> stem.<span style="color:#9900CC;">match</span><span style="color:#006600; font-weight:bold;">&#40;</span><span style="color:#006600; font-weight:bold;">/</span><span style="color:#006600; font-weight:bold;">&#91;</span>^a<span style="color:#006600; font-weight:bold;">-</span>z0<span style="color:#006600; font-weight:bold;">-</span><span style="color:#006666;">9</span>_<span style="color:#006600; font-weight:bold;">&#93;</span><span style="color:#006600; font-weight:bold;">/</span>im<span style="color:#006600; font-weight:bold;">&#41;</span>
&nbsp;
<span style="color:#CC0066; font-weight:bold;">open</span><span style="color:#006600; font-weight:bold;">&#40;</span>dofile, <span style="color:#996600;">'w'</span><span style="color:#006600; font-weight:bold;">&#41;</span> <span style="color:#9966CC; font-weight:bold;">do</span> <span style="color:#006600; font-weight:bold;">|</span>f<span style="color:#006600; font-weight:bold;">|</span> 
  f.<span style="color:#CC0066; font-weight:bold;">print</span> <span style="color:#006600; font-weight:bold;">%</span><span style="color:#006600; font-weight:bold;">&#123;</span>set notifyuser off\noutsheet <span style="color:#008000; font-style:italic;">#{stem}* using &quot;#{varfile}&quot; if 0, noquote\n}</span>
<span style="color:#9966CC; font-weight:bold;">end</span>
&nbsp;
<span style="color:#CC00FF; font-weight:bold;">File</span>.<span style="color:#9900CC;">unlink</span><span style="color:#006600; font-weight:bold;">&#40;</span>varfile<span style="color:#006600; font-weight:bold;">&#41;</span> <span style="color:#9966CC; font-weight:bold;">if</span> <span style="color:#CC00FF; font-weight:bold;">File</span>.<span style="color:#9900CC;">exist</span>?<span style="color:#006600; font-weight:bold;">&#40;</span>varfile<span style="color:#006600; font-weight:bold;">&#41;</span>
<span style="color:#996600;">`osascript -e 'tell application &quot;Stata&quot; to open POSIX file &quot;#{dofile}&quot;'`</span>
&nbsp;
<span style="color:#9966CC; font-weight:bold;">unless</span> <span style="color:#CC00FF; font-weight:bold;">File</span>.<span style="color:#9900CC;">exist</span>?<span style="color:#006600; font-weight:bold;">&#40;</span>varfile<span style="color:#006600; font-weight:bold;">&#41;</span>
  <span style="color:#CC0066; font-weight:bold;">print</span> <span style="color:#996600;">'No variables found'</span>	
  TextMate.<span style="color:#9900CC;">exit_show_tool_tip</span>
<span style="color:#9966CC; font-weight:bold;">end</span>
&nbsp;
vars = <span style="color:#CC0066; font-weight:bold;">open</span><span style="color:#006600; font-weight:bold;">&#40;</span>varfile<span style="color:#006600; font-weight:bold;">&#41;</span> <span style="color:#006600; font-weight:bold;">&#123;</span> <span style="color:#006600; font-weight:bold;">|</span>f<span style="color:#006600; font-weight:bold;">|</span> f.<span style="color:#9900CC;">read</span>.<span style="color:#CC0066; font-weight:bold;">chomp</span>.<span style="color:#CC0066; font-weight:bold;">split</span><span style="color:#006600; font-weight:bold;">&#40;</span><span style="color:#996600;">&quot;<span style="color:#000099;">\t</span>&quot;</span><span style="color:#006600; font-weight:bold;">&#41;</span> <span style="color:#006600; font-weight:bold;">&#125;</span>
&nbsp;
<span style="color:#9966CC; font-weight:bold;">if</span> vars.<span style="color:#9900CC;">length</span> == <span style="color:#006666;">1</span>
  <span style="color:#CC0066; font-weight:bold;">print</span> vars.<span style="color:#9900CC;">first</span> <span style="color:#006600; font-weight:bold;">+</span> <span style="color:#996600;">' '</span>
  <span style="color:#CC0066; font-weight:bold;">exit</span>
<span style="color:#9966CC; font-weight:bold;">end</span>
&nbsp;
v1, v2 = vars.<span style="color:#9900CC;">sort</span>.<span style="color:#9900CC;">values_at</span><span style="color:#006600; font-weight:bold;">&#40;</span><span style="color:#006666;">0</span>, <span style="color:#006600; font-weight:bold;">-</span><span style="color:#006666;">1</span><span style="color:#006600; font-weight:bold;">&#41;</span>.<span style="color:#9900CC;">sort_by</span> <span style="color:#006600; font-weight:bold;">&#123;</span> <span style="color:#006600; font-weight:bold;">|</span>v<span style="color:#006600; font-weight:bold;">|</span> v.<span style="color:#9900CC;">length</span> <span style="color:#006600; font-weight:bold;">&#125;</span>
common_prefix = <span style="color:#006600; font-weight:bold;">&#40;</span>0...<span style="color:#006600; font-weight:bold;">&#40;</span>v1.<span style="color:#9900CC;">length</span><span style="color:#006600; font-weight:bold;">&#41;</span><span style="color:#006600; font-weight:bold;">&#41;</span>.<span style="color:#9900CC;">each</span> <span style="color:#9966CC; font-weight:bold;">do</span> <span style="color:#006600; font-weight:bold;">|</span>i<span style="color:#006600; font-weight:bold;">|</span>
  <span style="color:#9966CC; font-weight:bold;">break</span> v1<span style="color:#006600; font-weight:bold;">&#91;</span><span style="color:#006666;">0</span>, i<span style="color:#006600; font-weight:bold;">&#93;</span> <span style="color:#9966CC; font-weight:bold;">unless</span> v1<span style="color:#006600; font-weight:bold;">&#91;</span>i, <span style="color:#006666;">1</span><span style="color:#006600; font-weight:bold;">&#93;</span> == v2<span style="color:#006600; font-weight:bold;">&#91;</span>i, <span style="color:#006666;">1</span><span style="color:#006600; font-weight:bold;">&#93;</span>
<span style="color:#9966CC; font-weight:bold;">end</span>
common_prefix = v1 <span style="color:#9966CC; font-weight:bold;">unless</span> common_prefix.<span style="color:#9900CC;">is_a</span>?<span style="color:#006600; font-weight:bold;">&#40;</span><span style="color:#CC0066; font-weight:bold;">String</span><span style="color:#006600; font-weight:bold;">&#41;</span>
&nbsp;
<span style="color:#9966CC; font-weight:bold;">if</span> stem == common_prefix
  <span style="color:#CC0066; font-weight:bold;">print</span> vars.<span style="color:#9900CC;">join</span><span style="color:#006600; font-weight:bold;">&#40;</span><span style="color:#996600;">&quot;<span style="color:#000099;">\n</span>&quot;</span><span style="color:#006600; font-weight:bold;">&#41;</span>
  TextMate.<span style="color:#9900CC;">exit_show_tool_tip</span>
<span style="color:#9966CC; font-weight:bold;">else</span>
  <span style="color:#CC0066; font-weight:bold;">print</span> common_prefix
<span style="color:#9966CC; font-weight:bold;">end</span></pre></div></div>




<p>It should look like this:</p>

<p><img src="/wp-content/uploads/2010/02/Screen-shot-2010-02-06-at-16.52.53.png" alt="Screenshot" height="557" width="755" /></p>]]></content:encoded>
			<wfw:commentRss>http://blog.mackerron.com/2010/02/stata-textmate-tab-completion/feed/</wfw:commentRss>
		<slash:comments>2</slash:comments>
		</item>
		<item>
		<title>Using OS Code-Point Polygons in PostGIS</title>
		<link>http://blog.mackerron.com/2009/11/code-point-polygons-postgis/</link>
		<comments>http://blog.mackerron.com/2009/11/code-point-polygons-postgis/#comments</comments>
		<pubDate>Sat, 14 Nov 2009 12:13:43 +0000</pubDate>
		<dc:creator>George</dc:creator>
				<category><![CDATA[GIS]]></category>
		<category><![CDATA[PostGIS]]></category>
		<category><![CDATA[Ruby]]></category>
		<category><![CDATA[SQL]]></category>

		<guid isPermaLink="false">http://blog.mackerron.com/?p=193</guid>
		<description><![CDATA[Ordnance Survey&#8217;s Code-Point with Polygons &#8220;provides a precise geographical location for each postcode unit in Great Britain&#8221;. It&#8217;s available in various formats, including ESRI .shp files. 

Many UK academics can access the data via institutional subscription to EDINA Digimap. I&#8217;m using it in my research into subjective wellbeing and environmental quality.

This post shows how to:



import [...]]]></description>
			<content:encoded><![CDATA[<p>Ordnance Survey&#8217;s <a href="http://www.ordnancesurvey.co.uk/oswebsite/products/codepointpolygons/">Code-Point with Polygons</a> &#8220;provides a precise geographical location for each postcode unit in Great Britain&#8221;. It&#8217;s available in various formats, including <span class="caps">ESRI</span> .shp files. </p>

<p>Many UK academics can access the data via institutional subscription to <a href="http://edina.ac.uk/digimap/"><span class="caps">EDINA</span> Digimap</a>. I&#8217;m using it in <a href="http://personal.lse.ac.uk/MACKERRO/">my research into subjective wellbeing and environmental quality</a>.</p>

<p>This post shows how to:</p>


<ol>
<li><strong>import the data files</strong> into a <a href="http://postgis.refractions.net/">PostGIS</a> database; and</li>
<li><strong>de-normalise the data into a single table</strong>, where there&#8217;s a one-to-one mapping of postcodes to rows, and each row contains either all geographical locations covered by a postcode (as a single geometry column, of type multipolygon) or the reason why no such location is available</li>
</ol>



<p><span id="more-193"></span></p>

<h3>Why de-normalise?</h3>

<p>Step 2 above is required for my purposes because Code-Point data is supplied in a number of separate files per postcode area: </p>


<ul>
<li>a .shp file (and associated .shx and .dbf) mapping postcodes and &#8216;vertical streets&#8217; to the locations they cover;</li>
<li>an accompanying text file mapping postcodes to &#8216;vertical streets&#8217;; and</li>
<li>another text file listing postcodes with no associated locations, either because they represent PO boxes or because the data just isn&#8217;t available.</li>
</ul>



<p>Vertical streets generally represent high-rise buildings, where one location in 2D space may be associated with multiple postcodes. Vertical streets are also a serious pain: not only may one vertical street be associated with many postcodes, but one postcode may be associated with many vertical streets <em>and</em> with non-vertical-street locations too. </p>

<h2>Import the data files</h2>

<p>Download and unzip the Code-Point with Polygons data, in <span class="caps">ESRI</span> .shp format, for the regions you want (where indicative query timings are provided later, these are for whole-UK data &#8212; 120 postcode areas &#8212; on a 2Ghz Intel iMac).</p>

<p>Dump everything in the same directory. For each postcode area XX you should have the following five files:</p>


<ul>
<li><span class="caps">XX.</span>shp, <span class="caps">XX.</span>shx and <span class="caps">XX.</span>dbf</li>
<li>XX_vstreet_lookup.txt</li>
<li>XX_discard.txt</li>
</ul>



<p>If you don&#8217;t already have one you want to use for this purpose, create a PostGIS-enabled PostgreSQL database (in this post, the database is named <code>geo</code>). If you&#8217;re not running PostgreSQL 8.4 or later you may need to fiddle with some fsm_ settings in the Postgres .conf file in order to cope with a data set this large.</p>

<p>Execute the following <span class="caps">SQL </span>to create the tables for discards and vertical streets (e.g. from within pgAdmin):</p>


<div class="wp_syntax"><div class="code"><pre class="postgresql" style="font-family:monospace;"><span style="color: #1a008a; font-weight: bold;">create</span> <span style="color: #1a008a; font-weight: bold;">table</span> cpp_vertical_streets <span style="color: #66cc66;">&#40;</span>
  postcode <span style="color: #1a008a; font-weight: bold;">character</span> <span style="color: #1a008a; font-weight: bold;">varying</span><span style="color: #66cc66;">&#40;</span><span style="color: #cc66cc;">8</span><span style="color: #66cc66;">&#41;</span><span style="color: #66cc66;">,</span>
  vstreet <span style="color: #1a008a; font-weight: bold;">character</span> <span style="color: #1a008a; font-weight: bold;">varying</span><span style="color: #66cc66;">&#40;</span><span style="color: #cc66cc;">8</span><span style="color: #66cc66;">&#41;</span>
<span style="color: #66cc66;">&#41;</span>;
&nbsp;
<span style="color: #1a008a; font-weight: bold;">create</span> <span style="color: #1a008a; font-weight: bold;">table</span> cpp_discards <span style="color: #66cc66;">&#40;</span>
  postcode <span style="color: #1a008a; font-weight: bold;">character</span> <span style="color: #1a008a; font-weight: bold;">varying</span><span style="color: #66cc66;">&#40;</span><span style="color: #cc66cc;">8</span><span style="color: #66cc66;">&#41;</span><span style="color: #66cc66;">,</span>
  reason <span style="color: #1a008a; font-weight: bold;">character</span> <span style="color: #1a008a; font-weight: bold;">varying</span><span style="color: #66cc66;">&#40;</span><span style="color: #cc66cc;">7</span><span style="color: #66cc66;">&#41;</span>
<span style="color: #66cc66;">&#41;</span>;</pre></div></div>




<p>Next we need to create the table for the .shp-file polygons. We <strong>switch to Ruby</strong> for this &#8212; you can just enter the commands in an <span class="caps">IRB </span>terminal session:</p>


<div class="wp_syntax"><div class="code"><pre class="ruby" style="font-family:monospace;">path = <span style="color:#996600;">'/path/to/CodePoint polygons'</span>
<span style="color:#996600;">`/usr/local/pgsql/bin/shp2pgsql &quot;#{path}/ab.shp&quot; cpp_polygons -p -D -s 27700 | /usr/local/pgsql/bin/psql -d geo -U postgres`</span></pre></div></div>




<p>The -p flag to <code>shp2pgsql</code> just creates a table structure, and the -s 27700 gives it the <span class="caps">EPSG </span>code to tell it we&#8217;re using the <span class="caps">OSGB36 </span>datum. You can use any of the .shp files you&#8217;ve downloaded &#8212; it need not be the AB area one.</p>

<p><strong>Now to do the import</strong> into the three tables. Still in Ruby, execute the following loop:</p>


<div class="wp_syntax"><div class="code"><pre class="ruby" style="font-family:monospace;"><span style="color:#CC00FF; font-weight:bold;">Dir</span><span style="color:#006600; font-weight:bold;">&#91;</span><span style="color:#996600;">&quot;#{path}/*.shp&quot;</span><span style="color:#006600; font-weight:bold;">&#93;</span>.<span style="color:#9900CC;">each</span> <span style="color:#9966CC; font-weight:bold;">do</span> <span style="color:#006600; font-weight:bold;">|</span>f<span style="color:#006600; font-weight:bold;">|</span>
  <span style="color:#CC0066; font-weight:bold;">puts</span> f
  <span style="color:#996600;">`/usr/local/pgsql/bin/shp2pgsql &quot;#{f}&quot; cpp_polygons -a -D -s 27700 | /usr/local/pgsql/bin/psql -d geo -U postgres`</span>
  <span style="color:#996600;">`echo &quot;copy cpp_vertical_streets from '#{f.sub(/<span style="color:#000099;">\.</span>shp$/, '_vstreet_lookup.txt')}' csv;&quot; | /usr/local/pgsql/bin/psql geo postgres`</span>
  <span style="color:#996600;">`echo &quot;copy cpp_discards from '#{f.sub(/<span style="color:#000099;">\.</span>shp$/, '_discard.txt')}' csv;&quot; | /usr/local/pgsql/bin/psql geo postgres`</span>
<span style="color:#9966CC; font-weight:bold;">end</span></pre></div></div>




<p>Back in <span class="caps">SQL, </span>add some boolean flags we&#8217;ll use later on, then create an index on the postcode column, which will save a <strong>lot</strong> of time later:</p>


<div class="wp_syntax"><div class="code"><pre class="postgresql" style="font-family:monospace;"><span style="color: #1a008a; font-weight: bold;">alter</span> <span style="color: #1a008a; font-weight: bold;">table</span> cpp_polygons <span style="color: #1a008a; font-weight: bold;">add</span> <span style="color: #1a008a; font-weight: bold;">column</span> <span style="color: #1a008a; font-weight: bold;">discard</span> <span style="color: #1a008a; font-weight: bold;">boolean</span>;
<span style="color: #1a008a; font-weight: bold;">alter</span> <span style="color: #1a008a; font-weight: bold;">table</span> cpp_polygons <span style="color: #1a008a; font-weight: bold;">add</span> <span style="color: #1a008a; font-weight: bold;">column</span> vstreet <span style="color: #1a008a; font-weight: bold;">boolean</span>;
<span style="color: #1a008a; font-weight: bold;">alter</span> <span style="color: #1a008a; font-weight: bold;">table</span> cpp_polygons <span style="color: #1a008a; font-weight: bold;">add</span> <span style="color: #1a008a; font-weight: bold;">column</span> vstreet_and_std <span style="color: #1a008a; font-weight: bold;">boolean</span>;
&nbsp;
<span style="color: #1a008a; font-weight: bold;">create</span> <span style="color: #1a008a; font-weight: bold;">index</span> pc_index <span style="color: #1a008a; font-weight: bold;">on</span> cpp_polygons <span style="color: #66cc66;">&#40;</span>postcode<span style="color: #66cc66;">&#41;</span>;
&nbsp;
<span style="color: #1a008a; font-weight: bold;">vacuum</span> <span style="color: #1a008a; font-weight: bold;">analyze</span> cpp_polygons;</pre></div></div>




<p>If you like, you can confirm here that there&#8217;s only one row per postcode &#8212; this query <strong>should return nothing</strong>:</p>


<div class="wp_syntax"><div class="code"><pre class="postgresql" style="font-family:monospace;"><span style="color: #1a008a; font-weight: bold;">select</span> <span style="color: #66cc66;">*</span> 
<span style="color: #1a008a; font-weight: bold;">from</span> cpp_polygons <span style="color: #1a008a; font-weight: bold;">a</span> 
<span style="color: #1a008a; font-weight: bold;">inner</span> <span style="color: #1a008a; font-weight: bold;">join</span> cpp_polygons b 
<span style="color: #1a008a; font-weight: bold;">on</span> <span style="color: #1a008a; font-weight: bold;">a</span><span style="color: #66cc66;">.</span>postcode <span style="color: #66cc66;">=</span> b<span style="color: #66cc66;">.</span>postcode 
<span style="color: #1a008a; font-weight: bold;">where</span> <span style="color: #1a008a; font-weight: bold;">a</span><span style="color: #66cc66;">.</span>gid <span style="color: #66cc66;">&lt;&gt;</span> b<span style="color: #66cc66;">.</span>gid;</pre></div></div>




<p>So, we now have all the OS data held in three separate tables. In the rest of the post, we&#8217;ll be merging the data in the discards and vertical streets tables into the main table, cpp_polygons.</p>

<h2>Discards</h2>

<p>We&#8217;re going to create a new table, with the same structure as cpp_polygons, to hold the discarded postcode data. These postcodes will have a <span class="caps">NULL </span>geometry column, and a <span class="caps">TRUE </span>discard column (one of the booleans we added earlier). Later, we&#8217;ll insert the contents of this table into cpp_polygons.</p>

<p>Run the following <span class="caps">SQL</span>:</p>


<div class="wp_syntax"><div class="code"><pre class="postgresql" style="font-family:monospace;"><span style="color: #1a008a; font-weight: bold;">create</span> <span style="color: #1a008a; font-weight: bold;">sequence</span> cpp_discard_seq <span style="color: #1a008a; font-weight: bold;">start</span> <span style="color: #1a008a; font-weight: bold;">with</span> <span style="color: #cc66cc;">2000000</span>;
&nbsp;
<span style="color: #1a008a; font-weight: bold;">create</span> <span style="color: #1a008a; font-weight: bold;">table</span> cpp_discard_polys <span style="color: #1a008a; font-weight: bold;">as</span> <span style="color: #66cc66;">&#40;</span>
<span style="color: #1a008a; font-weight: bold;">select</span> 
  <span style="color: #1a008a;">nextval</span><span style="color: #66cc66;">&#40;</span><span style="color: #cc0000;">'cpp_discard_seq'</span><span style="color: #66cc66;">&#41;</span> <span style="color: #1a008a; font-weight: bold;">as</span> gid<span style="color: #66cc66;">,</span> 
  postcode<span style="color: #66cc66;">,</span>
  <span style="color: #1a008a; font-weight: bold;">cast</span><span style="color: #66cc66;">&#40;</span><span style="color: #1a008a; font-weight: bold;">null</span> <span style="color: #1a008a; font-weight: bold;">as</span> <span style="color: #1a008a; font-weight: bold;">character</span> <span style="color: #1a008a; font-weight: bold;">varying</span><span style="color: #66cc66;">&#40;</span><span style="color: #cc66cc;">20</span><span style="color: #66cc66;">&#41;</span><span style="color: #66cc66;">&#41;</span> <span style="color: #1a008a; font-weight: bold;">as</span> upp<span style="color: #66cc66;">,</span> 
  <span style="color: #1a008a; font-weight: bold;">substring</span><span style="color: #66cc66;">&#40;</span>postcode <span style="color: #1a008a; font-weight: bold;">from</span> <span style="color: #cc0000;">'^[A-Z][A-Z]?'</span><span style="color: #66cc66;">&#41;</span> <span style="color: #1a008a; font-weight: bold;">as</span> pc_area<span style="color: #66cc66;">,</span> 
  <span style="color: #1a008a; font-weight: bold;">cast</span><span style="color: #66cc66;">&#40;</span><span style="color: #1a008a; font-weight: bold;">null</span> <span style="color: #1a008a; font-weight: bold;">as</span> <span style="color: #1a008a; font-weight: bold;">geometry</span><span style="color: #66cc66;">&#41;</span> <span style="color: #1a008a; font-weight: bold;">as</span> the_geom<span style="color: #66cc66;">,</span> 
  <span style="color: #1a008a; font-weight: bold;">true</span> <span style="color: #1a008a; font-weight: bold;">as</span> <span style="color: #1a008a; font-weight: bold;">discard</span><span style="color: #66cc66;">,</span> 
  <span style="color: #1a008a; font-weight: bold;">false</span> <span style="color: #1a008a; font-weight: bold;">as</span> vstreet<span style="color: #66cc66;">,</span>
  <span style="color: #1a008a; font-weight: bold;">false</span> <span style="color: #1a008a; font-weight: bold;">as</span> vstreet_and_std
<span style="color: #1a008a; font-weight: bold;">from</span> cpp_discards
<span style="color: #66cc66;">&#41;</span>;</pre></div></div>




<h2>Vertical streets</h2>

<p>Similarly, we&#8217;re now going to create a new table with the same structure as cpp_polygons to map vertical street postcodes to the right polygons. This requires a left join of the vertical streets data with some of the geometry data in cpp_polygons.</p>


<div class="wp_syntax"><div class="code"><pre class="postgresql" style="font-family:monospace;"><span style="color: #1a008a; font-weight: bold;">set</span> enable_seqscan <span style="color: #66cc66;">=</span> <span style="color: #1a008a; font-weight: bold;">false</span>; 
<span style="color: #808080; font-style: italic;">-- (otherwise pg sometimes fails to use the index)</span>
&nbsp;
<span style="color: #1a008a; font-weight: bold;">create</span> <span style="color: #1a008a; font-weight: bold;">sequence</span> cpp_vstreet_seq <span style="color: #1a008a; font-weight: bold;">start</span> <span style="color: #1a008a; font-weight: bold;">with</span> <span style="color: #cc66cc;">3000000</span>;
&nbsp;
<span style="color: #1a008a; font-weight: bold;">create</span> <span style="color: #1a008a; font-weight: bold;">table</span> cpp_vstreet_polys <span style="color: #1a008a; font-weight: bold;">as</span> <span style="color: #66cc66;">&#40;</span>
  <span style="color: #1a008a; font-weight: bold;">select</span> 
    <span style="color: #1a008a;">nextval</span><span style="color: #66cc66;">&#40;</span><span style="color: #cc0000;">'cpp_vstreet_seq'</span><span style="color: #66cc66;">&#41;</span> <span style="color: #1a008a; font-weight: bold;">as</span> gid<span style="color: #66cc66;">,</span> 
    <span style="color: #1a008a; font-weight: bold;">max</span><span style="color: #66cc66;">&#40;</span>v<span style="color: #66cc66;">.</span>postcode<span style="color: #66cc66;">&#41;</span> <span style="color: #1a008a; font-weight: bold;">as</span> postcode<span style="color: #66cc66;">,</span> 
    <span style="color: #1a008a; font-weight: bold;">cast</span><span style="color: #66cc66;">&#40;</span><span style="color: #1a008a; font-weight: bold;">null</span> <span style="color: #1a008a; font-weight: bold;">as</span> <span style="color: #1a008a; font-weight: bold;">varchar</span><span style="color: #66cc66;">&#40;</span><span style="color: #cc66cc;">20</span><span style="color: #66cc66;">&#41;</span><span style="color: #66cc66;">&#41;</span> <span style="color: #1a008a; font-weight: bold;">as</span> upp<span style="color: #66cc66;">,</span> 
    <span style="color: #1a008a; font-weight: bold;">max</span><span style="color: #66cc66;">&#40;</span>pc_area<span style="color: #66cc66;">&#41;</span> <span style="color: #1a008a; font-weight: bold;">as</span> pc_area<span style="color: #66cc66;">,</span> 
    <span style="color: #1a008a;">st_union</span><span style="color: #66cc66;">&#40;</span>the_geom<span style="color: #66cc66;">&#41;</span> <span style="color: #1a008a; font-weight: bold;">as</span> the_geom<span style="color: #66cc66;">,</span> 
    <span style="color: #1a008a; font-weight: bold;">false</span> <span style="color: #1a008a; font-weight: bold;">as</span> <span style="color: #1a008a; font-weight: bold;">discard</span><span style="color: #66cc66;">,</span>
    <span style="color: #1a008a; font-weight: bold;">true</span> <span style="color: #1a008a; font-weight: bold;">as</span> vstreet<span style="color: #66cc66;">,</span>
    <span style="color: #1a008a; font-weight: bold;">false</span> <span style="color: #1a008a; font-weight: bold;">as</span> vstreet_and_std
  <span style="color: #1a008a; font-weight: bold;">from</span> cpp_vertical_streets v 
  <span style="color: #1a008a; font-weight: bold;">left</span> <span style="color: #1a008a; font-weight: bold;">join</span> cpp_polygons <span style="color: #1a008a; font-weight: bold;">p</span> 
  <span style="color: #1a008a; font-weight: bold;">on</span> v<span style="color: #66cc66;">.</span>vstreet <span style="color: #66cc66;">=</span> <span style="color: #1a008a; font-weight: bold;">p</span><span style="color: #66cc66;">.</span>postcode
  <span style="color: #1a008a; font-weight: bold;">group</span> <span style="color: #1a008a; font-weight: bold;">by</span> v<span style="color: #66cc66;">.</span>postcode
<span style="color: #66cc66;">&#41;</span>;</pre></div></div>




<p>Our last new table is for postcodes that are associated with both standard polygons <strong>and</strong> vertical street polygons.</p>


<div class="wp_syntax"><div class="code"><pre class="postgresql" style="font-family:monospace;"><span style="color: #1a008a; font-weight: bold;">create</span> <span style="color: #1a008a; font-weight: bold;">sequence</span> cpp_vstreet_and_std_seq <span style="color: #1a008a; font-weight: bold;">start</span> <span style="color: #1a008a; font-weight: bold;">with</span> <span style="color: #cc66cc;">4000000</span>;
&nbsp;
<span style="color: #1a008a; font-weight: bold;">create</span> <span style="color: #1a008a; font-weight: bold;">table</span> cpp_vstreet_and_std_polys <span style="color: #1a008a; font-weight: bold;">as</span> <span style="color: #66cc66;">&#40;</span>
<span style="color: #1a008a; font-weight: bold;">select</span> 
  <span style="color: #1a008a;">nextval</span><span style="color: #66cc66;">&#40;</span><span style="color: #cc0000;">'cpp_vstreet_and_std_seq'</span><span style="color: #66cc66;">&#41;</span> <span style="color: #1a008a; font-weight: bold;">as</span> gid<span style="color: #66cc66;">,</span> 
  <span style="color: #1a008a; font-weight: bold;">p</span><span style="color: #66cc66;">.</span>postcode <span style="color: #1a008a; font-weight: bold;">as</span> postcode<span style="color: #66cc66;">,</span>
  <span style="color: #1a008a; font-weight: bold;">cast</span><span style="color: #66cc66;">&#40;</span><span style="color: #1a008a; font-weight: bold;">null</span> <span style="color: #1a008a; font-weight: bold;">as</span> <span style="color: #1a008a; font-weight: bold;">varchar</span><span style="color: #66cc66;">&#40;</span><span style="color: #cc66cc;">20</span><span style="color: #66cc66;">&#41;</span><span style="color: #66cc66;">&#41;</span> <span style="color: #1a008a; font-weight: bold;">as</span> upp<span style="color: #66cc66;">,</span> 
  <span style="color: #1a008a; font-weight: bold;">p</span><span style="color: #66cc66;">.</span>pc_area <span style="color: #1a008a; font-weight: bold;">as</span> pc_area<span style="color: #66cc66;">,</span>
  <span style="color: #1a008a;">st_multi</span><span style="color: #66cc66;">&#40;</span><span style="color: #1a008a;">st_union</span><span style="color: #66cc66;">&#40;</span><span style="color: #1a008a; font-weight: bold;">p</span><span style="color: #66cc66;">.</span>the_geom<span style="color: #66cc66;">,</span> v<span style="color: #66cc66;">.</span>the_geom<span style="color: #66cc66;">&#41;</span><span style="color: #66cc66;">&#41;</span> <span style="color: #1a008a; font-weight: bold;">as</span> the_geom<span style="color: #66cc66;">,</span> 
  <span style="color: #1a008a; font-weight: bold;">false</span> <span style="color: #1a008a; font-weight: bold;">as</span> <span style="color: #1a008a; font-weight: bold;">discard</span><span style="color: #66cc66;">,</span> 
  <span style="color: #1a008a; font-weight: bold;">false</span> <span style="color: #1a008a; font-weight: bold;">as</span> vstreet<span style="color: #66cc66;">,</span>
  <span style="color: #1a008a; font-weight: bold;">true</span> <span style="color: #1a008a; font-weight: bold;">as</span> vstreet_and_std
<span style="color: #1a008a; font-weight: bold;">from</span> cpp_polygons <span style="color: #1a008a; font-weight: bold;">p</span>
<span style="color: #1a008a; font-weight: bold;">inner</span> <span style="color: #1a008a; font-weight: bold;">join</span> cpp_vstreet_polys v
<span style="color: #1a008a; font-weight: bold;">on</span> <span style="color: #1a008a; font-weight: bold;">p</span><span style="color: #66cc66;">.</span>postcode <span style="color: #66cc66;">=</span> v<span style="color: #66cc66;">.</span>postcode
<span style="color: #66cc66;">&#41;</span>;</pre></div></div>




<h2>Cleaning and merging</h2>

<p>Now we need to remove the vertical streets and polygons we just merged into a new table from their respective source tables.</p>


<div class="wp_syntax"><div class="code"><pre class="postgresql" style="font-family:monospace;"><span style="color: #1a008a; font-weight: bold;">delete</span> <span style="color: #1a008a; font-weight: bold;">from</span> cpp_polygons <span style="color: #1a008a; font-weight: bold;">where</span> postcode <span style="color: #1a008a; font-weight: bold;">in</span> <span style="color: #66cc66;">&#40;</span><span style="color: #1a008a; font-weight: bold;">select</span> postcode <span style="color: #1a008a; font-weight: bold;">from</span> cpp_vstreet_and_std_polys<span style="color: #66cc66;">&#41;</span>;
<span style="color: #808080; font-style: italic;">-- the above could take around 25 mins</span>
<span style="color: #1a008a; font-weight: bold;">delete</span> <span style="color: #1a008a; font-weight: bold;">from</span> cpp_vstreet_polys <span style="color: #1a008a; font-weight: bold;">where</span> postcode <span style="color: #1a008a; font-weight: bold;">in</span> <span style="color: #66cc66;">&#40;</span><span style="color: #1a008a; font-weight: bold;">select</span> postcode <span style="color: #1a008a; font-weight: bold;">from</span> cpp_vstreet_and_std_polys<span style="color: #66cc66;">&#41;</span>;</pre></div></div>




<p>And, finally, to merge our three new tables into the main table.</p>


<div class="wp_syntax"><div class="code"><pre class="postgresql" style="font-family:monospace;"><span style="color: #1a008a; font-weight: bold;">insert</span> <span style="color: #1a008a; font-weight: bold;">into</span> cpp_polygons <span style="color: #1a008a; font-weight: bold;">select</span> <span style="color: #66cc66;">*</span> <span style="color: #1a008a; font-weight: bold;">from</span> cpp_discard_polys;
<span style="color: #1a008a; font-weight: bold;">insert</span> <span style="color: #1a008a; font-weight: bold;">into</span> cpp_polygons <span style="color: #1a008a; font-weight: bold;">select</span> <span style="color: #66cc66;">*</span> <span style="color: #1a008a; font-weight: bold;">from</span> cpp_vstreet_polys;
<span style="color: #1a008a; font-weight: bold;">insert</span> <span style="color: #1a008a; font-weight: bold;">into</span> cpp_polygons <span style="color: #1a008a; font-weight: bold;">select</span> <span style="color: #66cc66;">*</span> <span style="color: #1a008a; font-weight: bold;">from</span> cpp_vstreet_and_std_polys;</pre></div></div>




<p>In order to join my own data with this data, I find it easiest to format the postcodes on which the join is made with no spaces. So I add and index the following extra column.</p>


<div class="wp_syntax"><div class="code"><pre class="postgresql" style="font-family:monospace;"><span style="color: #1a008a; font-weight: bold;">alter</span> <span style="color: #1a008a; font-weight: bold;">table</span> cpp_polygons <span style="color: #1a008a; font-weight: bold;">add</span> <span style="color: #1a008a; font-weight: bold;">column</span> postcode_no_sp <span style="color: #1a008a; font-weight: bold;">character</span> <span style="color: #1a008a; font-weight: bold;">varying</span><span style="color: #66cc66;">&#40;</span><span style="color: #cc66cc;">8</span><span style="color: #66cc66;">&#41;</span>;
<span style="color: #1a008a; font-weight: bold;">update</span> cpp_polygons <span style="color: #1a008a; font-weight: bold;">set</span> postcode_no_sp <span style="color: #66cc66;">=</span> <span style="color: #1a008a; font-weight: bold;">replace</span><span style="color: #66cc66;">&#40;</span>postcode<span style="color: #66cc66;">,</span> <span style="color: #cc0000;">' '</span><span style="color: #66cc66;">,</span> <span style="color: #cc0000;">''</span><span style="color: #66cc66;">&#41;</span>; 
<span style="color: #808080; font-style: italic;">-- the above could take 10 - 15 mins</span>
<span style="color: #1a008a; font-weight: bold;">create</span> <span style="color: #1a008a; font-weight: bold;">index</span> pcns_index <span style="color: #1a008a; font-weight: bold;">on</span> cpp_polygons <span style="color: #66cc66;">&#40;</span>postcode_no_sp<span style="color: #66cc66;">&#41;</span>;
&nbsp;
<span style="color: #1a008a; font-weight: bold;">vacuum</span> <span style="color: #1a008a; font-weight: bold;">analyze</span> cpp_polygons;</pre></div></div>




<p>You might also want to remove the original vertical street rows, where the postcode column begins with a &#8216;V&#8217;, from the cpp_polygons table &#8212; I didn&#8217;t have any need for this.</p>]]></content:encoded>
			<wfw:commentRss>http://blog.mackerron.com/2009/11/code-point-polygons-postgis/feed/</wfw:commentRss>
		<slash:comments>1</slash:comments>
		</item>
		<item>
		<title>Signing Amazon Product Advertising API calls in Ruby</title>
		<link>http://blog.mackerron.com/2009/08/sign-aws-api-in-ruby/</link>
		<comments>http://blog.mackerron.com/2009/08/sign-aws-api-in-ruby/#comments</comments>
		<pubDate>Sat, 22 Aug 2009 11:04:23 +0000</pubDate>
		<dc:creator>George</dc:creator>
				<category><![CDATA[JavaScript]]></category>
		<category><![CDATA[Ruby]]></category>
		<category><![CDATA[Web design]]></category>

		<guid isPermaLink="false">http://blog.mackerron.com/?p=182</guid>
		<description><![CDATA[I have a simple site that generates covers for CDs I burn from iTunes purchases and so on (it pre-dates widespread use of JS libraries, and is in much need of prettifying). The site uses Amazon Product Advertising API calls to search and retrieve album cover art and track listings. Since earlier this month, such [...]]]></description>
			<content:encoded><![CDATA[<p>I have <a href="http://mackerron.com/cdcovers/">a simple site</a> that generates covers for CDs I burn from iTunes purchases and so on (it pre-dates widespread use of JS libraries, and is in much need of prettifying). The site uses <a href="https://affiliate-program.amazon.com/gp/advertising/api/detail/main.html">Amazon Product Advertising <span class="caps">API</span></a> calls to search and retrieve album cover art and track listings. Since earlier this month, such <span class="caps">API </span>calls have to be cryptographically signed.</p>

<p>This is somewhat annoying &#8212; the site&#8217;s original design has it communicating independently with Amazon (using Amazon&#8217;s <span class="caps">XSLT API </span>feature to transform their <span class="caps">XML </span>data into <span class="caps">JSON</span>), and that&#8217;s no longer possible with the use of a private key. But it&#8217;s not unfixable. The site now sends its <span class="caps">API </span>call first to my server, which returns a signed version, and then forwards the signed call on to Amazon.</p>

<p>I found most of what I needed for this on <a href="http://chrisroos.co.uk/blog/2009-01-31-implementing-version-2-of-the-amazon-aws-http-request-signature-in-ruby">Chris Roos&#8217; blog</a>, but his version still wasn&#8217;t quite working for me (the two problems I recall are that Ruby&#8217;s <span class="caps">CGI.</span>escape doesn&#8217;t quite follow Amazon&#8217;s requirements, and that times need converting to <span class="caps">GMT</span>).</p>

<p><span id="more-182"></span></p>

<p>Anyway, in case you&#8217;re looking to do the same, here&#8217;s what I ended up with:</p>


<div class="wp_syntax"><div class="code"><pre class="ruby" style="font-family:monospace;"><span style="color:#008000; font-style:italic;">#!/usr/bin/env ruby</span>
&nbsp;
<span style="color:#008000; font-style:italic;"># Note: You need hmac.rb and hmac-sha2.rb from http://deisui.org/~ueno/ruby/hmac.html </span>
<span style="color:#008000; font-style:italic;"># somewhere in your require paths. ruby-hmac is currently broken under Ruby 1.9.</span>
&nbsp;
<span style="color:#006600; font-weight:bold;">%</span>w<span style="color:#006600; font-weight:bold;">&#40;</span>rubygems cgi time hmac<span style="color:#006600; font-weight:bold;">-</span>sha2 base64<span style="color:#006600; font-weight:bold;">&#41;</span>.<span style="color:#9900CC;">each</span> <span style="color:#006600; font-weight:bold;">&#123;</span> <span style="color:#006600; font-weight:bold;">|</span>lib<span style="color:#006600; font-weight:bold;">|</span> <span style="color:#CC0066; font-weight:bold;">require</span> lib <span style="color:#006600; font-weight:bold;">&#125;</span>
&nbsp;
ACCESS_IDENTIFIER = <span style="color:#996600;">'YOUR_PUBLIC_ID'</span>
SECRET_IDENTIFIER = <span style="color:#996600;">'YOUR_PRIVATE_ID'</span>
&nbsp;
<span style="color:#9966CC; font-weight:bold;">def</span> aws_escape<span style="color:#006600; font-weight:bold;">&#40;</span>s<span style="color:#006600; font-weight:bold;">&#41;</span>
  s.<span style="color:#CC0066; font-weight:bold;">gsub</span><span style="color:#006600; font-weight:bold;">&#40;</span><span style="color:#006600; font-weight:bold;">/</span><span style="color:#006600; font-weight:bold;">&#91;</span>^A<span style="color:#006600; font-weight:bold;">-</span>Za<span style="color:#006600; font-weight:bold;">-</span>z0<span style="color:#006600; font-weight:bold;">-</span><span style="color:#006666;">9</span>_.~<span style="color:#006600; font-weight:bold;">-</span><span style="color:#006600; font-weight:bold;">&#93;</span><span style="color:#006600; font-weight:bold;">/</span><span style="color:#006600; font-weight:bold;">&#41;</span> <span style="color:#006600; font-weight:bold;">&#123;</span> <span style="color:#006600; font-weight:bold;">|</span>c<span style="color:#006600; font-weight:bold;">|</span> <span style="color:#996600;">'%'</span> <span style="color:#006600; font-weight:bold;">+</span> c<span style="color:#006600; font-weight:bold;">&#91;</span><span style="color:#006666;">0</span><span style="color:#006600; font-weight:bold;">&#93;</span>.<span style="color:#9900CC;">to_s</span><span style="color:#006600; font-weight:bold;">&#40;</span><span style="color:#006666;">16</span><span style="color:#006600; font-weight:bold;">&#41;</span>.<span style="color:#9900CC;">upcase</span> <span style="color:#006600; font-weight:bold;">&#125;</span>  
  <span style="color:#008000; font-style:italic;"># for 1.9, you'd replace [0] with .ord -- but ruby-hmac seems broken under 1.9</span>
<span style="color:#9966CC; font-weight:bold;">end</span>
&nbsp;
cgi = <span style="color:#CC00FF; font-weight:bold;">CGI</span>.<span style="color:#9900CC;">new</span>
params = cgi.<span style="color:#9900CC;">params</span>.<span style="color:#9900CC;">dup</span>
&nbsp;
amazon_endpoint = params.<span style="color:#9900CC;">delete</span><span style="color:#006600; font-weight:bold;">&#40;</span><span style="color:#996600;">'amazon_endpoint'</span><span style="color:#006600; font-weight:bold;">&#41;</span>
amazon_path = params.<span style="color:#9900CC;">delete</span><span style="color:#006600; font-weight:bold;">&#40;</span><span style="color:#996600;">'amazon_path'</span><span style="color:#006600; font-weight:bold;">&#41;</span>
js_callback = params.<span style="color:#9900CC;">delete</span><span style="color:#006600; font-weight:bold;">&#40;</span><span style="color:#996600;">'js_callback'</span><span style="color:#006600; font-weight:bold;">&#41;</span>
&nbsp;
signing_params = <span style="color:#006600; font-weight:bold;">&#123;</span>
  <span style="color:#996600;">'AWSAccessKeyId'</span> <span style="color:#006600; font-weight:bold;">=&gt;</span> ACCESS_IDENTIFIER,
  <span style="color:#996600;">'Timestamp'</span> <span style="color:#006600; font-weight:bold;">=&gt;</span> <span style="color:#CC00FF; font-weight:bold;">Time</span>.<span style="color:#9900CC;">now</span>.<span style="color:#9900CC;">gmtime</span>.<span style="color:#9900CC;">iso8601</span>
<span style="color:#006600; font-weight:bold;">&#125;</span>
&nbsp;
params.<span style="color:#9900CC;">merge</span>!<span style="color:#006600; font-weight:bold;">&#40;</span>signing_params<span style="color:#006600; font-weight:bold;">&#41;</span>
&nbsp;
canonical_querystring = params.<span style="color:#9900CC;">sort</span>.<span style="color:#9900CC;">collect</span> <span style="color:#9966CC; font-weight:bold;">do</span> <span style="color:#006600; font-weight:bold;">|</span>key, value<span style="color:#006600; font-weight:bold;">|</span> 
  <span style="color:#006600; font-weight:bold;">&#91;</span>aws_escape<span style="color:#006600; font-weight:bold;">&#40;</span>key.<span style="color:#9900CC;">to_s</span><span style="color:#006600; font-weight:bold;">&#41;</span>, aws_escape<span style="color:#006600; font-weight:bold;">&#40;</span>value.<span style="color:#9900CC;">to_s</span><span style="color:#006600; font-weight:bold;">&#41;</span><span style="color:#006600; font-weight:bold;">&#93;</span>.<span style="color:#9900CC;">join</span><span style="color:#006600; font-weight:bold;">&#40;</span><span style="color:#996600;">'='</span><span style="color:#006600; font-weight:bold;">&#41;</span> 
<span style="color:#9966CC; font-weight:bold;">end</span>.<span style="color:#9900CC;">join</span><span style="color:#006600; font-weight:bold;">&#40;</span><span style="color:#996600;">'&amp;'</span><span style="color:#006600; font-weight:bold;">&#41;</span>
&nbsp;
string_to_sign = <span style="color:#996600;">&quot;GET<span style="color:#000099;">\n</span>#{amazon_endpoint}<span style="color:#000099;">\n</span>#{amazon_path}<span style="color:#000099;">\n</span>#{canonical_querystring}&quot;</span>
&nbsp;
hmac = <span style="color:#6666ff; font-weight:bold;">HMAC::SHA256</span>.<span style="color:#9900CC;">new</span><span style="color:#006600; font-weight:bold;">&#40;</span>SECRET_IDENTIFIER<span style="color:#006600; font-weight:bold;">&#41;</span>
hmac.<span style="color:#9900CC;">update</span><span style="color:#006600; font-weight:bold;">&#40;</span>string_to_sign<span style="color:#006600; font-weight:bold;">&#41;</span>
signature = <span style="color:#CC00FF; font-weight:bold;">Base64</span>.<span style="color:#9900CC;">encode64</span><span style="color:#006600; font-weight:bold;">&#40;</span>hmac.<span style="color:#9900CC;">digest</span><span style="color:#006600; font-weight:bold;">&#41;</span>.<span style="color:#CC0066; font-weight:bold;">chomp</span>
&nbsp;
params<span style="color:#006600; font-weight:bold;">&#91;</span><span style="color:#996600;">'Signature'</span><span style="color:#006600; font-weight:bold;">&#93;</span> = signature
querystring = params.<span style="color:#9900CC;">sort</span>.<span style="color:#9900CC;">collect</span> <span style="color:#9966CC; font-weight:bold;">do</span> <span style="color:#006600; font-weight:bold;">|</span>key, value<span style="color:#006600; font-weight:bold;">|</span> 
  <span style="color:#006600; font-weight:bold;">&#91;</span>aws_escape<span style="color:#006600; font-weight:bold;">&#40;</span>key.<span style="color:#9900CC;">to_s</span><span style="color:#006600; font-weight:bold;">&#41;</span>, aws_escape<span style="color:#006600; font-weight:bold;">&#40;</span>value.<span style="color:#9900CC;">to_s</span><span style="color:#006600; font-weight:bold;">&#41;</span><span style="color:#006600; font-weight:bold;">&#93;</span>.<span style="color:#9900CC;">join</span><span style="color:#006600; font-weight:bold;">&#40;</span><span style="color:#996600;">'='</span><span style="color:#006600; font-weight:bold;">&#41;</span> 
<span style="color:#9966CC; font-weight:bold;">end</span>.<span style="color:#9900CC;">join</span><span style="color:#006600; font-weight:bold;">&#40;</span><span style="color:#996600;">'&amp;'</span><span style="color:#006600; font-weight:bold;">&#41;</span>
&nbsp;
signed_url = <span style="color:#996600;">&quot;http://#{amazon_endpoint}#{amazon_path}?#{querystring}&quot;</span>
&nbsp;
cgi.<span style="color:#9900CC;">out</span><span style="color:#006600; font-weight:bold;">&#40;</span><span style="color:#996600;">'type'</span> <span style="color:#006600; font-weight:bold;">=&gt;</span> <span style="color:#996600;">'text/javascript'</span><span style="color:#006600; font-weight:bold;">&#41;</span> <span style="color:#006600; font-weight:bold;">&#123;</span> <span style="color:#996600;">&quot;#{js_callback}('#{signed_url}');&quot;</span> <span style="color:#006600; font-weight:bold;">&#125;</span></pre></div></div>




<p>You can test this locally by feeding key/value parameters to <span class="caps">CGI, </span>followed by Ctrl-D. These, for example:</p>



<pre>amazon_endpoint=ecs.amazonaws.com
amazon_path=/onca/xml
js_callback=do_stuff
Service=AWSECommerceService
Version=2009-03-31
Operation=ItemSearch
SearchIndex=Books
Keywords=george+monbiot</pre>]]></content:encoded>
			<wfw:commentRss>http://blog.mackerron.com/2009/08/sign-aws-api-in-ruby/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>O2 mobile broadband on a Mac: tips for you, suggestion for O2</title>
		<link>http://blog.mackerron.com/2009/08/o2-mobile-broadband-on-a-mac-tips-for-you-suggestion-for-o2/</link>
		<comments>http://blog.mackerron.com/2009/08/o2-mobile-broadband-on-a-mac-tips-for-you-suggestion-for-o2/#comments</comments>
		<pubDate>Sat, 01 Aug 2009 12:52:50 +0000</pubDate>
		<dc:creator>George</dc:creator>
				<category><![CDATA[Mac]]></category>
		<category><![CDATA[iPhone]]></category>

		<guid isPermaLink="false">http://blog.mackerron.com/?p=158</guid>
		<description><![CDATA[Until Monday, O2 is offering a half-price USB dongle for pay-as-you-go mobile broadband. Buy it via Quidco, and you get not only the dongle but also a month&#8217;s access (or 3GB, whichever is the sooner) absolutely free.

So: free modem. No further commitment. And the opportunity to buy access a day at a time for &#163;2, [...]]]></description>
			<content:encoded><![CDATA[<p>Until Monday, O2 is offering a <a href="http://shop.o2.co.uk/promo/o2mobilebroadband/tab/Pay_and_Go">half-price <span class="caps">USB </span>dongle</a> for pay-as-you-go mobile broadband. <a href="http://www.quidco.com/o2">Buy it via Quidco</a>, and you get not only the dongle but also a month&#8217;s access (or 3GB, whichever is the sooner) absolutely free.</p>

<p>So: free modem. No further commitment. And the opportunity to buy access a day at a time for &#163;2, on a train journey, for example, or in an airport lounge, where the cheapest paid-for WiFi option is rarely less than &#163;5, and often nearer &#163;10. Handy.</p>

<p><strong>Update. O2 is no longer doing the modem at half price, but the Quidco cashback has increased so that it still fully covers the modem and a &#163;15 top-up.</strong></p>

<p><span id="more-158"></span></p>

<h3>The dongle</h3>

<p>The dongle, a Huawei <span class="caps">E160, </span>is a fairly pretty white thing, and it supports my Mac. So far so good. Unfortunately, the software and instructions have <a href="http://www.olpcnews.com/software/operating_system/ivan_krstic_xo_crippled_by_cra.html">crack-smoking hobos</a> written all over them. Huawei &#8220;Mobile Connect&#8221; is ugly, crash-prone, writes logs to the console like there&#8217;s no tomorrow (literally <a href="http://forum.huawei.com/jive4/thread.jspa?threadID=327290&amp;tstart=200&amp;orderStr=8">thousands of messages per minute</a>), and has never yet successfully connected me.</p>

<p>Luckily, you don&#8217;t need to use Mobile Connect: once the drivers are installed, you can connect using the menu item for the modem. Or rather, once the drivers are installed and you&#8217;ve taken the following counter-intuitive and undocumented step you can.</p>

<h3>Making it work</h3>

<p>When it installed, my dongle was configured as Vendor: Other, Model: Huawei Mobile Connect &#8211; 3G modem (this is under System Preferences &gt; Network &gt; <span class="caps">HUAWEI </span>mobile &gt; Advanced&#8230;). This sounds good, but it doesn&#8217;t work: there&#8217;s nowhere to enter your <span class="caps">APN.</span></p>

<p><img src="http://blog.mackerron.com/wp-content/uploads/2009/08/Picture-7.png" alt="System Preferences screenshot (before" /></p>

<p>Instead, after much fiddling, I found I had to switch Vendor to Generic and Model to <span class="caps">GPRS </span>(GSM/3G). Although confusingly less right, this does seem to do the trick.</p>

<p><img src="http://blog.mackerron.com/wp-content/uploads/2009/08/Picture-9.png" alt="System Preferences screenshot (after" /></p>

<p>I also ended up downloading <a href="http://www.huawei.com/mobileweb/en/doc/list.do?type=-1&amp;id=736">new drivers from the Huawei site</a>. (Not for the <span class="caps">E160.</span> That would be too simple. The &#8216;Mac&#8217; drivers for the <span class="caps">E160 </span>turn out to be a Windows .exe file. The drivers I&#8217;m using are supposedly for the <span class="caps">E220</span>). I&#8217;m not sure if this step was necessary or even well-advised, but it might be worth trying if you&#8217;re not having any luck.</p>

<h3>Listen up, O2</h3>

<p>Which brings us to the suggestion to <span class="caps">O2.</span> O2: I have an iPhone. It&#8217;s on your network. It supports tethering. I&#8217;m not about to <a href="http://shop.o2.co.uk/update/internet.html">pay you an extra &#163;15 a month</a> for the once or twice a month I want to use it. Let me use it for pay-as-you-go mobile broadband!</p>

<p>In this case you&#8217;d have saved yourself the full cost of a modem dongle &#8212; and generally, you&#8217;d be saving yourself the support costs and return costs associated with everyone who can&#8217;t get the horrible thing working. You&#8217;d have saved the world the cost of the resources and pollution required in manufacturing it. And you&#8217;d have saved me the pain of several hours fiddling with System Preferences, the inconvenience of another gadget to pack for every trip, and probable future kernel panics associated with Huawei&#8217;s flaky drivers.</p>]]></content:encoded>
			<wfw:commentRss>http://blog.mackerron.com/2009/08/o2-mobile-broadband-on-a-mac-tips-for-you-suggestion-for-o2/feed/</wfw:commentRss>
		<slash:comments>6</slash:comments>
		</item>
		<item>
		<title>Growling Mac backups with rsync</title>
		<link>http://blog.mackerron.com/2009/08/growling-backups/</link>
		<comments>http://blog.mackerron.com/2009/08/growling-backups/#comments</comments>
		<pubDate>Sat, 01 Aug 2009 11:40:01 +0000</pubDate>
		<dc:creator>George</dc:creator>
				<category><![CDATA[Mac]]></category>
		<category><![CDATA[System admin]]></category>

		<guid isPermaLink="false">http://blog.mackerron.com/?p=68</guid>
		<description><![CDATA[Between Time Machine and services like Dropbox, paranoid levels of backup are surprisingly painless to achieve on the Mac these days. 

Still, just one more copy of your data, in just one more continent, surely can&#8217;t do any harm, right? One that won&#8217;t burn down with your house, but also isn&#8217;t just wafting vaguely in [...]]]></description>
			<content:encoded><![CDATA[<p>Between <a href="http://www.apple.com/macosx/what-is-macosx/time-machine.html">Time Machine</a> and services like <a href="http://www.getdropbox.com/">Dropbox</a>, paranoid levels of backup are surprisingly painless to achieve on the Mac these days. </p>

<p>Still, just one more copy of your data, in just one more continent, surely can&#8217;t do any harm, right? One that won&#8217;t burn down with your house, but also isn&#8217;t just wafting vaguely in the Cloud at someone else&#8217;s whim. One that elevates your backup system from sensibly paranoid to borderline <span class="caps">OCD.</span> One, in this case, brought to you by rsync, find and Growl.</p>

<p><span id="more-68"></span></p>

<h3>What it does</h3>

<p>This script does the following:</p>


<ul>
<li>recursive, incremental, remote backup of the contents of folders you choose</li>
<li>safely archiving anything that changed since last time</li>
<li>and excluding things by name or folder (all this is just <code>rsync</code> so far)</li>
<li>and also excluding anything unfeasibly big (this is down to <code>find</code>)</li>
<li>all the while using Growl to keep you apprised of the situation, as exemplified below (thanks to <code>growlnotify</code>).</li>
</ul>



<p><img src="http://blog.mackerron.com/wp-content/uploads/2009/07/Picture-24.png" alt="Growling backup error screenshot" /></p>

<p>On the remote server, you end up with a file structure as below, with your current files under <em>Current</em> and any files that have moved, changed, or been deleted in a dated folder under <em>Archive</em>:</p>

<p><img src="http://blog.mackerron.com/wp-content/uploads/2009/07/Picture-26.png" alt="Backups screenshot" /></p>

<p><img src="http://blog.mackerron.com/wp-content/uploads/2009/07/Picture-25.png" alt="Backups screenshot" /></p>

<h3>What you&#8217;ll need</h3>

<p>You&#8217;ll need:</p>


<ul>
<li>a remote backup server you have <code>rsync</code> access to, with (passwordless) public key authentication set up.</li>
<li>Growl installed, obviously, and also <code>growlnotify</code> (by running <code>./install.sh</code> in the Extras folder on the Growl installation .dmg).</li>
<li>a way to schedule the script to run &#8212; iCal, <code>cron</code>, or <code>launchd</code> (for the management of which last I recommend <a href="http://tuppis.com/lingon/">Lingon</a>).</li>
</ul>



<h3>The script</h3>

<p>There&#8217;s some rather superstitious quoting. Even then, it&#8217;ll probably still bite you in case of spaces.</p>


<div class="wp_syntax"><div class="code"><pre class="bash" style="font-family:monospace;"><span style="color: #666666; font-style: italic;">#!/bin/bash</span>
&nbsp;
<span style="color: #666666; font-style: italic;"># --- Set backup parameters ---</span>
&nbsp;
<span style="color: #666666; font-style: italic;"># What local files/folders are we backing up?</span>
<span style="color: #007800;">SRC</span>=<span style="color: #ff0000;">&quot;/Users/gjm06/Development /Users/gjm06/Documents /Users/gjm06/Sites&quot;</span>
&nbsp;
<span style="color: #666666; font-style: italic;"># What server are we backing up to?</span>
<span style="color: #007800;">DESTSERVER</span>=<span style="color: #ff0000;">&quot;george.example.com&quot;</span>
&nbsp;
<span style="color: #666666; font-style: italic;"># What's our user name on the server?</span>
<span style="color: #007800;">SSHUSER</span>=<span style="color: #ff0000;">&quot;george&quot;</span>
&nbsp;
<span style="color: #666666; font-style: italic;"># What's the remote shell command for this server </span>
<span style="color: #666666; font-style: italic;"># (slight obscurity/security by avoiding port 22)</span>
<span style="color: #007800;">RSH</span>=<span style="color: #ff0000;">&quot;ssh -p 2244&quot;</span>
&nbsp;
<span style="color: #666666; font-style: italic;"># What location on the server shall we back up to?</span>
<span style="color: #007800;">DESTPATH</span>=<span style="color: #ff0000;">&quot;/home/george/backups&quot;</span>
&nbsp;
<span style="color: #666666; font-style: italic;"># Any non-valuable stuff to exclude by name/location?</span>
<span style="color: #007800;">EXCLUDE</span>=<span style="color: #ff0000;">&quot;.DS_Store,.svn/,.Trash/,tmp/,log/,vendor/&quot;</span>
&nbsp;
<span style="color: #666666; font-style: italic;"># Exclude things that are *how* big? </span>
<span style="color: #666666; font-style: italic;"># e.g. I don't want GBs of virtual machines going over the wire</span>
<span style="color: #666666; font-style: italic;"># (NB. this is in bytes)</span>
<span style="color: #007800;">MAXFILESIZE</span>=<span style="color: #000000;">100000</span>
&nbsp;
&nbsp;
<span style="color: #666666; font-style: italic;"># --- Set Growl parameters ---</span>
&nbsp;
<span style="color: #666666; font-style: italic;"># What will this script be called in the Growl prefpane?</span>
<span style="color: #007800;">GROWLNAME</span>=<span style="color: #ff0000;">&quot;Remote backup script&quot;</span>
&nbsp;
<span style="color: #666666; font-style: italic;"># And what app's icon will appear in Growl notifications?</span>
<span style="color: #007800;">APPICON</span>=<span style="color: #ff0000;">&quot;Time Machine&quot;</span>
&nbsp;
&nbsp;
<span style="color: #666666; font-style: italic;"># --- Do it! ---</span>
&nbsp;
<span style="color: #7a0874; font-weight: bold;">echo</span> Checking not already running 
&nbsp;
<span style="color: #007800;">PROCS</span>=<span style="color: #000000; font-weight: bold;">`</span><span style="color: #c20cb9; font-weight: bold;">ps</span> <span style="color: #660033;">-A</span> <span style="color: #660033;">-o</span> <span style="color: #ff0000;">&quot;pid=,command=&quot;</span><span style="color: #000000; font-weight: bold;">`</span>
<span style="color: #007800;">MYNAME</span>=<span style="color: #ff0000;">&quot;$0&quot;</span>
<span style="color: #007800;">MYBASENAME</span>=<span style="color: #000000; font-weight: bold;">`</span><span style="color: #c20cb9; font-weight: bold;">basename</span> <span style="color: #007800;">$MYNAME</span><span style="color: #000000; font-weight: bold;">`</span>
<span style="color: #007800;">MYPID</span>=<span style="color: #007800;">$$</span>
&nbsp;
<span style="color: #666666; font-style: italic;"># The next line works like so:</span>
<span style="color: #666666; font-style: italic;"># * take the process list (for all users),</span>
<span style="color: #666666; font-style: italic;"># * filter *in* processes named like this script (making sure we're on word boundaries),</span>
<span style="color: #666666; font-style: italic;"># * filter *out* (-v) the one that *is* this script (by PID), and finally</span>
<span style="color: #666666; font-style: italic;"># * filter *out* the grep commands themselves.</span>
&nbsp;
<span style="color: #007800;">MERUNNING</span>=<span style="color: #000000; font-weight: bold;">`</span><span style="color: #7a0874; font-weight: bold;">echo</span> <span style="color: #ff0000;">&quot;<span style="color: #007800;">$PROCS</span>&quot;</span> <span style="color: #000000; font-weight: bold;">|</span> <span style="color: #c20cb9; font-weight: bold;">grep</span> <span style="color: #660033;">-E</span> <span style="color: #660033;">-e</span> <span style="color: #ff0000;">&quot;\b<span style="color: #007800;">$MYBASENAME</span>\b&quot;</span> \
  <span style="color: #000000; font-weight: bold;">|</span> <span style="color: #c20cb9; font-weight: bold;">grep</span> <span style="color: #660033;">-E</span> <span style="color: #660033;">-v</span> <span style="color: #ff0000;">&quot;\b<span style="color: #007800;">$MYPID</span>\b&quot;</span> <span style="color: #000000; font-weight: bold;">|</span> <span style="color: #c20cb9; font-weight: bold;">grep</span> <span style="color: #660033;">-v</span> <span style="color: #c20cb9; font-weight: bold;">grep</span><span style="color: #000000; font-weight: bold;">`</span>
&nbsp;
<span style="color: #666666; font-style: italic;"># Then, if anything's left (i.e. MERUNNING isn't a zero-length string...)</span>
&nbsp;
<span style="color: #000000; font-weight: bold;">if</span> <span style="color: #7a0874; font-weight: bold;">&#91;</span> <span style="color: #000000; font-weight: bold;">!</span> <span style="color: #660033;">-z</span> <span style="color: #ff0000;">&quot;<span style="color: #007800;">$MERUNNING</span>&quot;</span> <span style="color: #7a0874; font-weight: bold;">&#93;</span>; <span style="color: #000000; font-weight: bold;">then</span>
  <span style="color: #000000; font-weight: bold;">/</span>usr<span style="color: #000000; font-weight: bold;">/</span>local<span style="color: #000000; font-weight: bold;">/</span>bin<span style="color: #000000; font-weight: bold;">/</span>growlnotify <span style="color: #660033;">--name</span> <span style="color: #ff0000;">&quot;<span style="color: #007800;">$GROWLNAME</span>&quot;</span> <span style="color: #660033;">--appIcon</span> <span style="color: #ff0000;">&quot;<span style="color: #007800;">$APPICON</span>&quot;</span> \
    <span style="color: #660033;">--message</span> <span style="color: #ff0000;">&quot;Another backup seems to be in progress&quot;</span> \
    <span style="color: #ff0000;">&quot;Ignoring scheduled backup&quot;</span>
  <span style="color: #7a0874; font-weight: bold;">exit</span> <span style="color: #000000;">1</span>
<span style="color: #000000; font-weight: bold;">fi</span>
&nbsp;
&nbsp;
<span style="color: #7a0874; font-weight: bold;">echo</span> Finding large files to exclude...
&nbsp;
<span style="color: #007800;">OVERSIZELIST</span>=<span style="color: #ff0000;">&quot;/tmp/files_over_backup_limit&quot;</span>
<span style="color: #c20cb9; font-weight: bold;">find</span> <span style="color: #007800;">$SRC</span> <span style="color: #660033;">-size</span> +<span style="color: #007800;">$MAXFILESIZE</span> <span style="color: #000000; font-weight: bold;">&gt;</span> <span style="color: #007800;">$OVERSIZELIST</span>
&nbsp;
<span style="color: #666666; font-style: italic;"># If you ever want to know what's excluded -- and you should! -- just do</span>
<span style="color: #666666; font-style: italic;"># a 'cat /tmp/files_over_backup_limit' in the Terminal</span>
&nbsp;
<span style="color: #666666; font-style: italic;"># Next, 'wc -l' counts the *lines* in our oversize list, and we extract </span>
<span style="color: #666666; font-style: italic;"># just the number from the output</span>
&nbsp;
<span style="color: #007800;">NUMOVERSIZE</span>=<span style="color: #000000; font-weight: bold;">`</span><span style="color: #c20cb9; font-weight: bold;">wc</span> <span style="color: #660033;">-l</span> <span style="color: #ff0000;">&quot;<span style="color: #007800;">$OVERSIZELIST</span>&quot;</span> <span style="color: #000000; font-weight: bold;">|</span> <span style="color: #c20cb9; font-weight: bold;">grep</span> <span style="color: #660033;">-o</span> <span style="color: #660033;">-E</span> <span style="color: #7a0874; font-weight: bold;">&#91;</span>0123456789<span style="color: #7a0874; font-weight: bold;">&#93;</span>+<span style="color: #000000; font-weight: bold;">`</span>
&nbsp;
&nbsp;
<span style="color: #7a0874; font-weight: bold;">echo</span> Rsyncing...
&nbsp;
<span style="color: #666666; font-style: italic;"># date makes up the name for our archive folders, while</span>
<span style="color: #666666; font-style: italic;"># eval is used to invoke shell expansion on the list of file names to exclude,</span>
<span style="color: #666666; font-style: italic;"># turning them into a list of --exclude= statements</span>
&nbsp;
<span style="color: #007800;">BDATE</span>=<span style="color: #000000; font-weight: bold;">`</span><span style="color: #c20cb9; font-weight: bold;">date</span> <span style="color: #ff0000;">&quot;+%y_%m_%d-%H.%M.%S&quot;</span><span style="color: #000000; font-weight: bold;">`</span>
<span style="color: #007800;">EXPEXCLUDES</span>=<span style="color: #000000; font-weight: bold;">`</span><span style="color: #7a0874; font-weight: bold;">eval</span> <span style="color: #ff0000;">&quot;echo --exclude={<span style="color: #007800;">$EXCLUDE</span>} &quot;</span><span style="color: #000000; font-weight: bold;">`</span>
<span style="color: #007800;">ERRORS</span>=<span style="color: #ff0000;">&quot;/tmp/rsync_backup_errors&quot;</span>
&nbsp;
<span style="color: #666666; font-style: italic;"># The &amp;&amp; and || operators give us a simple way to respond to zero and </span>
<span style="color: #666666; font-style: italic;"># non-zero (= error) rsync exit codes, thanks to lazy evaluation by the shell</span>
&nbsp;
rsync <span style="color: #660033;">--rsh</span>=<span style="color: #ff0000;">&quot;<span style="color: #007800;">$RSH</span>&quot;</span> \
  <span style="color: #660033;">--recursive</span> <span style="color: #660033;">--partial</span> <span style="color: #660033;">--delete</span> <span style="color: #660033;">--delete-excluded</span> \
  <span style="color: #660033;">--links</span> <span style="color: #660033;">--times</span> <span style="color: #660033;">--relative</span> <span style="color: #660033;">--compress</span> \
  <span style="color: #660033;">--backup</span> <span style="color: #660033;">--backup-dir</span>=<span style="color: #ff0000;">&quot;<span style="color: #007800;">$DESTPATH</span>/Archive/<span style="color: #007800;">$BDATE</span>/&quot;</span> \
  <span style="color: #660033;">--exclude-from</span>=<span style="color: #ff0000;">&quot;<span style="color: #007800;">$OVERSIZELIST</span>&quot;</span> \
  <span style="color: #007800;">$EXPEXCLUDES</span> \
  <span style="color: #007800;">$SRC</span> \
  <span style="color: #007800;">$SSHUSER</span><span style="color: #000000; font-weight: bold;">@</span><span style="color: #007800;">$DESTSERVER</span>:<span style="color: #ff0000;">&quot;<span style="color: #007800;">$DESTPATH</span>/Current/&quot;</span> \
  <span style="color: #000000;">2</span><span style="color: #000000; font-weight: bold;">&gt;</span><span style="color: #ff0000;">&quot;<span style="color: #007800;">$ERRORS</span>&quot;</span> \
<span style="color: #000000; font-weight: bold;">&amp;&amp;</span> 
<span style="color: #000000; font-weight: bold;">/</span>usr<span style="color: #000000; font-weight: bold;">/</span>local<span style="color: #000000; font-weight: bold;">/</span>bin<span style="color: #000000; font-weight: bold;">/</span>growlnotify <span style="color: #660033;">--name</span> <span style="color: #ff0000;">&quot;<span style="color: #007800;">$GROWLNAME</span>&quot;</span> <span style="color: #660033;">--appIcon</span> <span style="color: #ff0000;">&quot;<span style="color: #007800;">$APPICON</span>&quot;</span> \
  <span style="color: #660033;">--message</span> <span style="color: #ff0000;">&quot;Backed up to <span style="color: #007800;">$DESTSERVER</span>, <span style="color: #007800;">$NUMOVERSIZE</span> oversized files excluded&quot;</span> \
  <span style="color: #ff0000;">&quot;Completed backup&quot;</span> \
<span style="color: #000000; font-weight: bold;">||</span> \
<span style="color: #000000; font-weight: bold;">/</span>usr<span style="color: #000000; font-weight: bold;">/</span>local<span style="color: #000000; font-weight: bold;">/</span>bin<span style="color: #000000; font-weight: bold;">/</span>growlnotify <span style="color: #660033;">--name</span> <span style="color: #ff0000;">&quot;<span style="color: #007800;">$GROWLNAME</span>&quot;</span> <span style="color: #660033;">--appIcon</span> <span style="color: #ff0000;">&quot;<span style="color: #007800;">$APPICON</span>&quot;</span> <span style="color: #660033;">-s</span> \
  <span style="color: #660033;">--message</span> <span style="color: #ff0000;">&quot;<span style="color: #780078;">`cat &quot;$ERRORS&quot;`</span>&quot;</span> \
  <span style="color: #ff0000;">&quot;Failed backup&quot;</span></pre></div></div>




<p>Hope it&#8217;s useful in your own paranoid backup endeavours. And, by the way, if you ever hear me say &#8220;surely a simple Bash script couldn&#8217;t take <em>that</em> long&#8221;, please don&#8217;t stop reminding me just how much I hate Bash until I change my mind.</p>]]></content:encoded>
			<wfw:commentRss>http://blog.mackerron.com/2009/08/growling-backups/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>iPhone + public key SSH authentication: lovely</title>
		<link>http://blog.mackerron.com/2009/05/iphone-public-key-ssh/</link>
		<comments>http://blog.mackerron.com/2009/05/iphone-public-key-ssh/#comments</comments>
		<pubDate>Thu, 28 May 2009 13:03:20 +0000</pubDate>
		<dc:creator>George</dc:creator>
				<category><![CDATA[System admin]]></category>

		<guid isPermaLink="false">http://blog.mackerron.com/?p=53</guid>
		<description><![CDATA[I&#8217;ve been unhappy for a while having the SSH daemon on my web server VPSs with password authentication enabled. Of course, it&#8217;s on a non-standard port, blocks root logins, and takes a strong-ish password&#8230; but the risk of a successful dictionary attack has still felt too non-zero for comfort.

Equally, though, I&#8217;ve not wanted to give [...]]]></description>
			<content:encoded><![CDATA[<p>I&#8217;ve been unhappy for a while having the <span class="caps">SSH </span>daemon on my web server <span class="caps">VPS</span>s with password authentication enabled. Of course, it&#8217;s on a non-standard port, blocks root logins, and takes a strong-ish password&#8230; but the risk of a successful dictionary attack has still felt too non-zero for comfort.</p>

<p>Equally, though, I&#8217;ve not wanted to give up the ability to log in to the servers from anywhere to fix things in an emergency, so I didn&#8217;t want to turn password authentication off and rely on public keys alone.</p>

<p>Until now, that is. I realised yesterday that, since I have iSSH on the iPhone, which does public key authentication, I <em>can</em> log into my servers from anywhere, even with password authentication turned off. Granted, doing anything serious on a tiny screen and slow connection is difficult. But all I actually need to be able to do from there is temporarily turn password authentication back on.</p>

<p>And to make this easy, I&#8217;ve put two ultra-simple scripts in ~/bin:</p>

<p><strong>pwd_login_on.sh</strong></p>


<div class="wp_syntax"><div class="code"><pre class="bash" style="font-family:monospace;"><span style="color: #666666; font-style: italic;">#!/bin/bash</span>
<span style="color: #c20cb9; font-weight: bold;">sudo</span> <span style="color: #c20cb9; font-weight: bold;">sed</span> <span style="color: #660033;">-r</span> <span style="color: #660033;">-e</span> <span style="color: #ff0000;">'s/^PasswordAuthentication no$/PasswordAuthentication yes/'</span> \
  -i.previous <span style="color: #000000; font-weight: bold;">/</span>etc<span style="color: #000000; font-weight: bold;">/</span>ssh<span style="color: #000000; font-weight: bold;">/</span>sshd_config
<span style="color: #c20cb9; font-weight: bold;">sudo</span> <span style="color: #000000; font-weight: bold;">/</span>etc<span style="color: #000000; font-weight: bold;">/</span>init.d<span style="color: #000000; font-weight: bold;">/</span><span style="color: #c20cb9; font-weight: bold;">ssh</span> restart</pre></div></div>




<p><strong>pwd_login_off.sh</strong></p>


<div class="wp_syntax"><div class="code"><pre class="bash" style="font-family:monospace;"><span style="color: #666666; font-style: italic;">#!/bin/bash</span>
<span style="color: #c20cb9; font-weight: bold;">sudo</span> <span style="color: #c20cb9; font-weight: bold;">sed</span> <span style="color: #660033;">-r</span> <span style="color: #660033;">-e</span> <span style="color: #ff0000;">'s/^PasswordAuthentication yes$/PasswordAuthentication no/'</span> \
  -i.previous <span style="color: #000000; font-weight: bold;">/</span>etc<span style="color: #000000; font-weight: bold;">/</span>ssh<span style="color: #000000; font-weight: bold;">/</span>sshd_config
<span style="color: #c20cb9; font-weight: bold;">sudo</span> <span style="color: #000000; font-weight: bold;">/</span>etc<span style="color: #000000; font-weight: bold;">/</span>init.d<span style="color: #000000; font-weight: bold;">/</span><span style="color: #c20cb9; font-weight: bold;">ssh</span> restart</pre></div></div>




<p>(These paths are suitable for Ubuntu 8.04).</p>

<p>Lovely.</p>]]></content:encoded>
			<wfw:commentRss>http://blog.mackerron.com/2009/05/iphone-public-key-ssh/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Free Adobe fonts</title>
		<link>http://blog.mackerron.com/2009/05/free-adobe-fonts/</link>
		<comments>http://blog.mackerron.com/2009/05/free-adobe-fonts/#comments</comments>
		<pubDate>Thu, 28 May 2009 12:17:47 +0000</pubDate>
		<dc:creator>George</dc:creator>
				<category><![CDATA[Mac]]></category>
		<category><![CDATA[Web design]]></category>

		<guid isPermaLink="false">http://blog.mackerron.com/?p=25</guid>
		<description><![CDATA[Quick tip: you can get hold of some of Adobe&#8217;s very nice professional fonts for free when you download the InDesign CS4 trial (and possibly other CS4 apps too).

These include:



Caslon Pro
Chaparral Pro
Garamond Pro
Minion Pro
Myriad Pro




If you don&#8217;t want to actually install InDesign, you can get to the fonts like so (if you&#8217;re a Mac user):



Mount [...]]]></description>
			<content:encoded><![CDATA[<p>Quick tip: you can get hold of some of Adobe&#8217;s very nice professional fonts for free when you download the InDesign <span class="caps">CS4 </span>trial (and possibly other <span class="caps">CS4 </span>apps too).</p>

<p>These include:</p>


<ul>
<li>Caslon Pro</li>
<li>Chaparral Pro</li>
<li>Garamond Pro</li>
<li>Minion Pro</li>
<li>Myriad Pro</li>
</ul>



<p>If you don&#8217;t want to actually install InDesign, you can get to the fonts like so (if you&#8217;re a Mac user):</p>


<ul>
<li>Mount (double-click) the downloaded disk image</li>
<li>Mount another disk image found on the newly mounted disk, at Adobe InDesign <span class="caps">CS4</span>/payloads/AdobeFontsAll/AdobeFontsAll.dmg</li>
<li>The fonts are inside /Assets/contents on this second disk</li>
</ul>



<p><strong>Note that these fonts may well be covered by a very restrictive licence: I haven&#8217;t checked.</strong></p>]]></content:encoded>
			<wfw:commentRss>http://blog.mackerron.com/2009/05/free-adobe-fonts/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Testing in Internet Explorer for VMWare Fusion users</title>
		<link>http://blog.mackerron.com/2009/05/testing-in-ie-for-fusion-users/</link>
		<comments>http://blog.mackerron.com/2009/05/testing-in-ie-for-fusion-users/#comments</comments>
		<pubDate>Thu, 28 May 2009 09:25:43 +0000</pubDate>
		<dc:creator>George</dc:creator>
				<category><![CDATA[Mac]]></category>
		<category><![CDATA[Web design]]></category>

		<guid isPermaLink="false">http://blog.mackerron.com/?p=39</guid>
		<description><![CDATA[You might have noticed that Microsoft has lately started making available time-limited Virtual PC images of Windows installations with (separately) Internet Explorer versions 6, 7 and 8. 

This is brilliant for Mac users who need to test websites in IE, because you get a real installation of each browser in its home environment, with the [...]]]></description>
			<content:encoded><![CDATA[<p>You might have noticed that Microsoft has lately started making available time-limited Virtual PC images of Windows installations with (separately) Internet Explorer versions 6, 7 and 8. </p>

<p>This is brilliant for Mac users who need to test websites in <span class="caps">IE, </span>because you get a real installation of each browser in its home environment, with the right JavaScript engine and working conditional comments, plug-ins, Windows Media Player, and so on. With the alternatives, such as running under Wine or using <a href="http://tredosoft.com/Multiple_IE">Multiple IEs</a> in a single Windows virtual machine, one or more of these things tends to be missing.</p>

<p><span id="more-39"></span></p>

<p>Unfortunately, the images are supplied as self-extracting .exe files, which of course are Windows programs. But happily, the Mac ports of Wine, the open-source Windows emulation software, can run them.</p>


<ul>
<li>Get hold of <a href="http://www.microsoft.com/downloads/details.aspx?FamilyId=21EABB90-958F-4B64-B5F1-73D0A413C8EF&amp;displaylang=en">the self-extracting archives from Microsoft</a>. I find XP+IE6 and XP+IE8 a useful combination.</li>
<li>Install <a href="http://www.kronenberg.org/darwine/">Darwine</a>.</li>
<li>Open /Applications/Darwine/WineHelper, choose File &gt; Open&#8230; and pick one of the .exe files you downloaded in the last step.</li>
<li>Ignore any Darwine messages about needing to install Mozilla, and click Next/Agree/etc. as necessary in the installer to extract the .vhd file. (I understand the free <a href="http://www.virtualbox.org/">VirtualBox</a> will take such a .vhd directly, but I&#8217;ve not tried this myself).</li>
<li>Download <a href="http://www.kju-app.org/">Q</a>. This is a largely useless program for Intel Mac users, since its purpose is to emulate the <span class="caps">CPU </span>you already have. But bear with me: it includes a very handy utility.</li>
<li>Open /Applications/Utilities/Terminal, and paste in the following command, substituting in the correct path to your .vhd file and a path of your choice for the .vmdk file it will be converted to:</li>
</ul>




<div class="wp_syntax"><div class="code"><pre class="bash" style="font-family:monospace;"><span style="color: #000000; font-weight: bold;">/</span>Applications<span style="color: #000000; font-weight: bold;">/</span>Q.app<span style="color: #000000; font-weight: bold;">/</span>Contents<span style="color: #000000; font-weight: bold;">/</span>MacOS<span style="color: #000000; font-weight: bold;">/</span>qemu-img convert \
  <span style="color: #660033;">-f</span> vpc <span style="color: #660033;">-O</span> vmdk path_to_old.vhd path_to_new.vmdk</pre></div></div>





<ul>
<li>Nothing will appear to be happening for a few minutes, but don&#8217;t worry. Wait for the prompt to reappear.</li>
<li>Create a new Virtual Machine in Fusion, based on the new .vmdk you just created. Agree to Fusion&#8217;s request to convert the format of the disk.</li>
<li>Before starting up the new machine, remove its <span class="caps">USB </span>controller, sound card and serial port, and disable 3D graphics (this is all optional, but it reduces the number of requests for drivers you&#8217;ll have to deny later).</li>
<li>Start the virtual machine, canceling any requests for driver disks or Windows Update attempts.</li>
<li>Install <span class="caps">VMW</span>are Tools. I find everything works except the mouse, so at this point I choose the custom installation and deselect the mouse driver.</li>
<li>Reboot and you&#8217;re done.</li>
<li>Don&#8217;t forget to move the original .exe and .vhd files to the Trash: they&#8217;re big and completely useless now.</li>
</ul>

]]></content:encoded>
			<wfw:commentRss>http://blog.mackerron.com/2009/05/testing-in-ie-for-fusion-users/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Paste unformatted keyboard shortcut in Office 2008 for Mac</title>
		<link>http://blog.mackerron.com/2009/05/paste-unformatted-keyboard-shortcut-office-2008-mac/</link>
		<comments>http://blog.mackerron.com/2009/05/paste-unformatted-keyboard-shortcut-office-2008-mac/#comments</comments>
		<pubDate>Tue, 19 May 2009 15:35:57 +0000</pubDate>
		<dc:creator>George</dc:creator>
				<category><![CDATA[Mac]]></category>

		<guid isPermaLink="false">http://blog.mackerron.com/?p=30</guid>
		<description><![CDATA[iWork and some other Mac apps provide the keyboard shortcut Apple-Alt-Shift-V for the extremely useful command Edit &#62; Paste and match style. But Office 2008 provides no such equivalent for its equivalent command, Edit &#62; Paste &#62; Unformatted Text &#62; OK. Annoying.

Happily, you can fix this with a strategically named and placed AppleScript. Open /Applications/AppleScript/Script [...]]]></description>
			<content:encoded><![CDATA[<p>iWork and some other Mac apps provide the keyboard shortcut Apple-Alt-Shift-V for the extremely useful command <em>Edit &gt; Paste and match style</em>. But Office 2008 provides no such equivalent for its equivalent command, <em>Edit &gt; Paste &gt; Unformatted Text &gt; OK</em>. Annoying.</p>

<p>Happily, you can fix this with a strategically named and placed AppleScript. Open /Applications/AppleScript/Script Editor, and paste in the following lines:</p>


<div class="wp_syntax"><div class="code"><pre class="applescript" style="font-family:monospace;"><span style="color: #ff0033; font-weight: bold;">tell</span> <span style="color: #0066ff;">application</span> <span style="color: #009900;">&quot;Microsoft Word&quot;</span> 
  paste special <span style="color: #000000;">&#40;</span><span style="color: #0066ff;">text</span> object <span style="color: #ff0033; font-weight: bold;">of</span> <span style="color: #0066ff;">selection</span><span style="color: #000000;">&#41;</span> data type paste <span style="color: #0066ff;">text</span>
<span style="color: #ff0033; font-weight: bold;">end</span> <span style="color: #ff0033; font-weight: bold;">tell</span></pre></div></div>




<p>Save in /Users/YourUserName/Documents/Microsoft User Data/Word Script Menu Items as <strong>Paste unformatted\mosV.scpt</strong> (the backslashed bit at the end of the filename provides the shortcut). Restart Word and you can now paste without importing extraneous styles with Apple-Alt-Shift-V.</p>

<h3>Update</h3>

<p>You might also try this alternative approach (via <a href="http://www.betalogue.com/2008/01/28/word-2008-tip-creating-a-direct-command-for-pasting-unformatted-text/#comment-7949">a handy comment elsewhere</a>):</p>


<div class="wp_syntax"><div class="code"><pre class="applescript" style="font-family:monospace;"><span style="color: #ff0033; font-weight: bold;">try</span>
  <span style="color: #ff0033; font-weight: bold;">set</span> theClip <span style="color: #ff0033; font-weight: bold;">to</span> Unicode <span style="color: #0066ff;">text</span> <span style="color: #ff0033; font-weight: bold;">of</span> <span style="color: #000000;">&#40;</span><span style="color: #0066ff;">the clipboard</span> <span style="color: #ff0033;">as</span> record<span style="color: #000000;">&#41;</span>
  <span style="color: #ff0033; font-weight: bold;">tell</span> <span style="color: #0066ff;">application</span> <span style="color: #009900;">&quot;Microsoft Word&quot;</span> <span style="color: #ff0033; font-weight: bold;">to</span> <span style="color: #ff0033; font-weight: bold;">tell</span> <span style="color: #0066ff;">selection</span> <span style="color: #ff0033; font-weight: bold;">to</span> type <span style="color: #0066ff;">text</span> <span style="color: #0066ff;">text</span> theClip
<span style="color: #ff0033; font-weight: bold;">end</span> <span style="color: #ff0033; font-weight: bold;">try</span></pre></div></div>




<p>This has the advantage of leaving the cursor where you expect, at the end of the pasted text &#8212; the original script fragment leaves it at the beginning. And no, <code>type text text</code> is sadly not a typo: that&#8217;s really how Word likes to be addressed.</p>]]></content:encoded>
			<wfw:commentRss>http://blog.mackerron.com/2009/05/paste-unformatted-keyboard-shortcut-office-2008-mac/feed/</wfw:commentRss>
		<slash:comments>26</slash:comments>
		</item>
		<item>
		<title>Getting a Firefox extension&#8217;s directory from within the extension</title>
		<link>http://blog.mackerron.com/2009/05/getting-firefox-extension-directory/</link>
		<comments>http://blog.mackerron.com/2009/05/getting-firefox-extension-directory/#comments</comments>
		<pubDate>Thu, 07 May 2009 12:23:07 +0000</pubDate>
		<dc:creator>George</dc:creator>
				<category><![CDATA[JavaScript]]></category>

		<guid isPermaLink="false">http://blog.mackerron.com/?p=8</guid>
		<description><![CDATA[When creating a Firefox extension recently, I was surprised how much pain is involved simply in finding out, from within an extension, where that extension&#8217;s files are installed.

You have to create a &#8216;component&#8217;, which entails a fair chunk of unintelligible boilerplate code (well, unintelligible unless you&#8217;re much better acquainted with Firefox&#8217;s innards than developing a [...]]]></description>
			<content:encoded><![CDATA[<p>When creating a <a href="http://mackerron.com/zot2bib/">Firefox extension</a> recently, I was surprised how much pain is involved simply in finding out, from within an extension, where that extension&#8217;s files are installed.</p>

<p>You have to create a &#8216;component&#8217;, which entails a fair chunk of unintelligible boilerplate code (well, unintelligible unless you&#8217;re much better acquainted with Firefox&#8217;s innards than developing a basic extension generally requires you to be).</p>

<p>Plenty of places will tell you <em>roughly</em> how to do it. But, after some experimentation, let me show you <em>exactly</em> how.</p>


<ul>
<li>Copy <a href="http://www.mackerron.com/svn/zot2bib/trunk/components/get_ext_dir.js">this JavaScript file</a> into your extension&#8217;s /components directory (if that directory doesn&#8217;t exist yet, create it).</li>
<li>Use it from elsewhere in your extension like so:</li>
</ul>




<div class="wp_syntax"><div class="code"><pre class="javascript" style="font-family:monospace;">own_path <span style="color: #339933;">=</span> Components.<span style="color: #660066;">classes</span><span style="color: #009900;">&#91;</span><span style="color: #3366CC;">&quot;@mackerron.com/get_ext_dir;1&quot;</span><span style="color: #009900;">&#93;</span>
  .<span style="color: #660066;">createInstance</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span>.<span style="color: #660066;">wrappedJSObject</span>.<span style="color: #660066;">get_ext_dir</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span></pre></div></div>




<p>And that&#8217;s it. Let me know if it helps you out.</p>]]></content:encoded>
			<wfw:commentRss>http://blog.mackerron.com/2009/05/getting-firefox-extension-directory/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
	</channel>
</rss>
