We’ve been inspired in part by Ruairi Glynn‘s amazing work here at UCL, and Martin has been happily experimenting with the OpenKinect bindings for Processing.

Meanwhile, I recently got to grips with the excellent Three.js, which makes WebGL — aka 3D graphics in modern browsers — as easy as falling off a log. I’m also a big fan of making things accessible over the web. And so I began to investigate prospects for working with Kinect data in HTML5.

There’s DepthJS, an extension for Chrome and Safari, but this requires a locally-connected Kinect and isn’t very clear on Windows support. There’s also Intrael, which serves the depth data as JPEG files and provides some simple scene recognition output as JSON. But it’s closed-source and not terribly flexible.

The depthcam

So I decided to roll my own. I give you: the depthcam!

Click here or on the screenshot to connect.

It’s a live-streaming 3D point-cloud, carried over a binary WebSocket. It responds to movement in the scene by panning the (virtual) camera, and you can also pan and zoom around with the mouse.

Currently you’ll need Google Chrome to try it, and the number of people who can tune in at once is limited for reasons of bandwidth. If you can’t connect, or nothing much is happening, try this short video on YouTube instead.

It might be the future of video-conferencing. It could also be the start of a new wave of web-based movement-powered games.

How it works

The code is on GitHub, and is in three parts:

1. A short Python script wiring up the OpenKinect Python wrapper to the Autobahn WebSockets library. Depending on the arguments it’s run with, this can either serve a web browser directly, or it can push the depth data up to…
2. A simple node.js server that gets us round the UCL firewall. This accepts depth data pushed from the Python script, and broadcasts it onwards (still using binary WebSockets) to any connected web browsers, which are running…
3. The web-based client, written in CoffeeScript. This connects to the node.js server, receives the depth data, and visualises it as a particle system using Three.js and WebGL.

The incoming data from the Kinect is pretty heavy, at 18 MB/s (640 × 480 × 2 bytes per pixel x 30Hz). This is more than we can expect to (or afford to!) push over the Internet. So the Python script does some basic video compression to cut this down by several orders of magnitude, to 30 – 100 KB/s. It follows this three-step recipe:

1. Reduce the amount of data by down-sampling and quantizing to 160 × 120 × 1 byte per pixel.
2. Increase the data’s compressibility. First, reduce noise — which also looks bad — by making each transmitted depth value (i.e. pixel) the median of the values received in the last three frames. Then express each value as its difference from the previous value, with just an occasional absolute-value keyframe to allow new viewers to pick up the stream.
3. Compress the data using LZMA, which gets better compression ratios than GZIP or BZIP2, and decompression times somewhere between the two.

Feel free to fork the code and do something great with it.

You can export to CSV or SVG and load it from there; you can query the database directly; or, depending on context, you might choose to generate Processing commands directly, which is the route I went to display a background map of the UK in a recent visualization project.

First, I load outlines of the 4 constituent countries of the UK into my PostGIS database.

shp2pgsql -d -D -I -s 27700 "/pgdata/geodata/Nations/england.shp" eng \
  | sudo -u postgres psql mappiness
shp2pgsql -d -D -I -s 27700 "/pgdata/geodata/Nations/wales.shp" wal \
  | sudo -u postgres psql mappiness
shp2pgsql -d -D -I -s 27700 "/pgdata/geodata/Nations/scotland.shp" sco \
  | sudo -u postgres psql mappiness
shp2pgsql -d -D -I -s 27700 "/pgdata/geodata/Nations/nireland_ol_2001.shp" ni \
  | sudo -u postgres psql mappiness

Next, I combine these into one table, using st_simplifypreservetopology to discard a lot of unnecessary detail and st_dump to dissolve some English MULTIPOLYGONs into simple POLYGONS.

create table uk1km as (
        select st_simplifypreservetopology(the_geom, 1000) as the_geom from 
          (select (st_dump(the_geom)).geom as the_geom from eng) as eng_single
  union select st_simplifypreservetopology(the_geom, 1000) as the_geom from wal
  union select st_simplifypreservetopology(the_geom, 1000) as the_geom from sco
  union select st_simplifypreservetopology(the_geom, 1000) as the_geom from ni
);
alter table uk1km add column id serial primary key;
drop table eng; drop table wal; drop table sco; drop table ni;

Finally, I use string_agg to produce the vertex() commands required to reproduce my polygons in Processing. I change the sign on the y-coordinate (from northings to ‘southings’, to match the Processing coordinate space). I scale the OSGB36 coordinates down from metres to kilometres (I could otherwise have done this with a scale() command in Processing). And I discard a lot of small polygons (comprising 20 POINTs or fewer) I don’t need in my output.

select 'beginShape(); ' 
    || string_agg('vertex(' || st_x(geom) || ', ' || -st_y(geom) || '); ', '')
    || 'endShape(CLOSE); ' 
from (
  select id, (st_dumppoints(st_exteriorring(st_scale(the_geom, 0.001, 0.001)))).geom as geom 
  from uk1km 
  where st_numpoints(st_exteriorring(the_geom)) > 20
) as d
group by id;

Then, to draw the UK in a sketch I do this.

size(350, 620);
translate(10, height);
scale(0.5);
 
// paste in the output of the previous query

Which has this result (with thanks to EDINA for the data):

Then you may find my MIT-licensed passcode view controller — as seen in the mappiness app and in the short screencast below — of use.

See Github for the code and (scant) documentation.

Imagine you want compare various locations in terms of the availability of a certain type of environment, such as fresh water.

You might want to use a measure of the proximity of that environment — such as the nearest neighbour distance.

You might want to use a measure of the quantity of that environment in the vicinity — such as the proportion of land within a specific radius that is of that type.

Or you might ideally like a measure that combines both of these: one that incorporates the quantity of that environment, but gives greater weight to areas that are nearer, and lesser weight to those that are further away.

In that third case, what you probably want is a kernel-weighted proportion.

How it works

In this usage, a ‘kernel’ just means a distribution function, such as the normal (Gaussian), triangular, or Epanechnikov. The kernel is used to decide what weight should be attached to places at varying distances from the location under consideration.

I find it helpful to imagine the weight assigned to each point as a third dimension: the height at each point. I then imagine the kernel as a solid object positioned with its centre over the point I’m measuring from. For the normal or Epanechnikov kernels, this object is bell-shaped; for the triangular kernel, it’s a cone; for the (trivial) uniform kernel, it’s a cylinder.

The total volume of this object is the denominator of the kernel-weighted proportion. Now I imagine removing all parts of the object that don’t lie directly above the type of environment being measured. The volume of the remaining parts of the object — those directly overlying the environment of interest — is the numerator of the kernel-weighted proportion.

Implementation in PostGIS

The following SQL functions enable you to approximate kernel-weighted proportions in PostGIS. They do so by splitting the 3-dimensional kernel into horizontal slices of equal height, but with radii differing according to the kernel and kernel parameters specified. They calculate the summed volume of the intersections of the slices with the supplied polygons, and divide that by the total summed volume of the complete slices. (Assign each slice a height of 1 and these volumes are helpfully the same as the areas).

This approach is illustrated, for the normal kernel, at the top of this post.

-- kernel functions
 
create or replace function uniform_pdf
( double precision              -- $1 = x
, double precision default 1.0  -- $2 = bandwidth
, double precision default 0.0  -- $3 = centre
) returns double precision as $$
  select
    case
      when $1 > $3 - $2 and $1 < $3 + $2 then (
        select cast(0.5 as double precision)
      )
      else 0
    end;
$$ language sql immutable;
 
create or replace function triangular_pdf
( double precision              -- $1 = x
, double precision default 1.0  -- $2 = bandwidth
, double precision default 0.0  -- $3 = centre
) returns double precision as $$
  select
    case
      when $1 > $3 - $2 and $1 < $3 + $2 then (
        select 1 - abs(($1 - $3) / $2)
      )
      else 0
    end;
$$ language sql immutable;
 
create or replace function normal_pdf
( double precision              -- $1 = x
, double precision default 1.0  -- $2 = std dev (bandwidth)
, double precision default 0.0  -- $3 = mean (centre)
) returns double precision as $$
  select (1.0 / (sqrt(2.0 * pi() * pow($2, 2)))) 
       * exp(-pow($1 - $3, 2) / (2.0 * pow($2, 2)));
$$ language sql immutable;
 
create or replace function epanechnikov_pdf
( double precision              -- $1 = x
, double precision default 1.0  -- $2 = bandwidth
, double precision default 0.0  -- $3 = centre
) returns double precision as $$
  select
    case
      when $1 > $3 - $2 and $1 < $3 + $2 then (
        select 0.75 * (1 - pow(($1 - $3) / $2, 2))
      )
      else 0
    end;
$$ language sql immutable;
 
 
-- current kernel function 
-- (uncomment the kernel you want to use, and redefine: the normal is shown here)
 
create or replace function __current_kernel_pdf
( double precision              -- $1 = x
, double precision default 1.0  -- $2 = std dev/bandwidth
, double precision default 0.0  -- $3 = mean/centre
) returns double precision as $$
  select
    normal_pdf
    -- epanechnikov_pdf
    -- triangular_pdf
    -- uniform_pdf
    ($1, $2, $3);
$$ language sql immutable;
 
 
-- support functions
 
create or replace function __slice_height
( double precision  -- $1 = kernel std dev
, double precision  -- $2 = kernel radius at top of slice
, double precision  -- $3 = kernel radius at bottom of slice
) returns double precision as $$
  select __current_kernel_pdf($2, $1) - __current_kernel_pdf($3, $1);
$$ language sql immutable;
 
create or replace function __slice_radius
( double precision  -- $1 = kernel radius at top of slice
, double precision  -- $2 = kernel radius at bottom of slice
) returns double precision as $$
  select $1 + (($2 - $1) / 2);
$$ language sql immutable;
 
create or replace function __kernel_slice_volume
( double precision  -- $1 = kernel std dev
, double precision  -- $2 = kernel radius at top of slice
, double precision  -- $3 = kernel radius at bottom of slice
, int               -- $4 = buffer precision
) returns double precision as $$
  select
    coalesce(
      st_area(
        st_buffer(
          st_makepoint(0, 0),
          __slice_radius($2, $3),
          $4
        )
      ),
      0
    )
    * __slice_height($1, $2, $3);
$$ language sql immutable;
 
create or replace function __intersected_slice_volume
( geometry          -- $1 = area geometry
, geometry          -- $2 = kernel centre point geometry
, double precision  -- $3 = kernel std dev
, double precision  -- $4 = kernel radius at top of slice
, double precision  -- $5 = kernel radius at bottom of slice
, int               -- $6 = buffer precision
) returns double precision as $$
  select
    coalesce(
      st_area(
        st_intersection(
          $1,
          st_buffer(
            $2,
            __slice_radius($4, $5),
            $6
          )
        )
      ),
      0
    )
    * __slice_height($3, $4, $5);
$$ language sql immutable;
 
 
-- main function
 
create or replace function kernel_weighted_local_proportion
( geometry          -- $1 = area geometry
, geometry          -- $2 = kernel centre point geometry
, double precision  -- $3 = kernel std dev
, double precision  -- $4 = truncation bandwidth (for normal only -- for others, repeat $3)
, int               -- $5 = number of horizontal slices for approximation (for uniform, use 1)
, int               -- $6 = buffer precision (number of points per 1/4 circle)
) returns double precision as $$
  select
    sum(__intersected_slice_volume(
      $1, $2, $3,
      $4 * (cast(s as double precision) / $5),     -- kernel radius at top of slice
      $4 * (cast(s + 1 as double precision) / $5), -- kernel radius at bottom of slice
      $6
    ))
    /
    sum(__kernel_slice_volume(
      $3,
      $4 * (cast(s as double precision) / $5),     -- kernel radius at top of slice
      $4 * (cast(s + 1 as double precision) / $5), -- kernel radius at bottom of slice
      $6
    ))
  from generate_series(0, $5 - 1) s;
$$ language sql immutable;

• Install MySQL using the 64-bit .DMG package installer from dev.mysql.com
• Install Ruby using RVM (that’s Ruby 1.9.2 or 1.9.1-p378 — at the time of writing the latest 1.9.1, p429, is buggy)
• Add these to lines to ~/.bash_login or ~/.bashrc:

export PATH="/usr/local/mysql/bin:$PATH"
export DYLD_LIBRARY_PATH="/usr/local/mysql/lib:$DYLD_LIBRARY_PATH"

• In a new shell (Terminal window), type gem install mysql as normal.

I’m posting this mainly as a record for myself, having wasted a lot of time in the past trying strange incantations from comments on various other blogs posts.

And ever noticed how, when using the Google Maps API, the exact same thing doesn’t happen?

This code makes Google Maps API version 3 map markers behave in that Google Earth way. Small numbers of markers (up to 8, configurable) spiderfy into a circle. Larger numbers fan out into a (more space-efficient) spiral.

Try it! (You can reload this page to re-randomise the marker positions).

The code is MIT licenced, has no dependencies, and is under 3K compiled (from CoffeeScript), minified and gzipped. I wrote it as part of the data download feature for Mappiness.

It’s on Github, where you can find out more, download or fork the code.

This recipe makes wonderful bread: crusty, open-textured, moist, and beautiful.

It needs no kneading, but this doesn’t imply any sort of trade-off. It’s a recipe for a perfect loaf which happens to be effortless, and an effortless recipe which happens to make a perfect loaf.

It’s based on Jim Lahey’s recipe from the New York Times in 2006 (also the subject of an article and accompanying video, and now a full-length book). But it’s even easier, as there’s no awkward middle stage with linen cloths.

In the eight months since we started using it, we’ve made virtually all our own bread.

What you need

Ingredients

• 400g strong white bread flour
• 400ml water
• One third of a teaspoon dried yeast
• 2 teaspoons salt

And:

• Up to another 100g flour

Equipment

• A lidded cast-iron casserole pot, around 21cm diameter. We’ve been told that a ceramic pot also works, but we haven’t tried this.
• The bread-making usual: scales (preferably electronic, for accuracy), mixing bowl, measuring jug, wooden spoon, wire rack, timer, oven

If you need any of this equipment, the above pictures link to Amazon (who give us a referral fee if you buy something).

What you do

There are three short steps separated by two risings.

Step 1: Mix

Measure the 400g of flour into the bowl. Add the salt and yeast, and stir briefly to mix in.

Pour in the water, all at once. Stir until fully mixed.

The result will be much stickier and wetter than normal dough, with a consistency more like cake mixture. You couldn’t knead it if you wanted to.

Notes:

• In summer, cold water (i.e. room/tap temperature) is fine. In winter, use one part boiling to three or four parts cold to avoid slowing the risings down.
• The yeast quantity isn’t a mistake, though it’s a lot less than a standard bread recipe (and the back of the yeast packet) calls for. There’s also no need to reactivate it in the water first.
• Too much salt is bad for you, of course, but under-salted bread tastes really disappointing. We use two average-sized teaspoons, slightly over level, of sea salt crystals (which I suspect provide less salt in any given volume than fine table salt would).

Rising 1

Cover the bowl (e.g. with a large plate) and leave for 12 – 24 hours. Room temperature is fine in summer. In winter, somewhere a bit warmer — such as near a stove, boiler or hot water tank — works better.

Step 2: Add flour

The dough will have increased in size and spread out across the bowl. Its level top will dotted with bubbles, and it will be even more wet and gooey than before.

Stir in more flour, probably 50 – 100g, a handful at a time, until the mixture has just about formed itself into a dough: still stretchy, still sticking to the bowl, but half-capable of holding its shape.

Rising 2

Cover the bowl again and leave for another 2 – 12 hours.

Half an hour from the end of this rising, put the lidded casserole pot in the oven and start pre-heating to 220 degrees C.

Step 3: Bake

Take the now-super-hot pot from the oven and remove the lid.

Tip and scrape the re-risen dough into the pot, doing your best not to knock out all the air. It should look rough, ugly, and unpromising! Throw a bit of flour over the top for decorative effect.

Replace the lid and return the pot to the oven.

After half an hour, remove the lid.

After a further 15 minutes, take the pot from the oven. Tip the loaf out and leave it to cool on a wire rack. Admire!

Note:

• In summary, it’s: 30 minutes pre-heating the pot, 30 minutes baking with the lid on, and 15 minutes baking with the lid off. A timer/alarm is handy.
• A rounded flexible scraper (ours is cut out of the bottom of a plastic take-away carton) is good for smoothly transferring the dough from bowl to pot.
• Some oven temperature knobs stop marking temperatures at a point below 220 degrees. In this case you may need to experiment with how far beyond the end of the markings you need to go.
• The fully-baked loaf shouldn’t stick, so there should be no need to flour or grease the pot. If it does stick, it might be that the pot wasn’t hot enough — try turning up the oven a bit, and be sure to give the pot plenty of pre-heating time.

Variations

We rarely keep it quite this simple: various combinations of flours and seeds make a more interesting loaf.

Our favourites are:

Poppy seed

Carpet the bottom of the casserole pot with poppy seeds just before you tip the dough in, and carpet the top of the dough with more seeds straight after (mere sprinklings will disappoint). The seeds should stick pretty tight and toast just the right amount during baking.

Optionally, also replace half the plain white bread flour with white spelt flour. This isn’t stocked that widely — we get Doves Farm stuff, from Planet Organic — but it adds lightness and really enhances the flavour, making a loaf that’s lovely for simple hot-buttered-toasting… and pretty much everything else.

Sesame and barley

Carpet the pot-bottom and loaf-top as above, this time with plenty of sesame seeds. Also replace around 100g of the normal flour with barley flour. Barley gives a brilliant nutty flavour, and the sesame seeds make for a delicious, crispy, toasty crust.

Barley flour is also not always easy to find. We usually get ours from an independent health food shop (Coopers, Lower Marsh, London SE1). If you can’t find it, some fine oatmeal is also nice in this loaf.

Caraway and rye

Replace up to 150g of the wheat flour with rye — which makes a tastier but heavier bread — and add a good teaspoon of caraway seeds to the dry ingredients at step 1. This loaf comes out looking gorgeous and artisanal if dusted with semolina flour just before baking.

Six seed

Obviously you can experiment with adding different seeds and/or multi-seed flours. Bacheldre Mill do a malted 5 seed flour, but the fennel seeds in it are a bit overpowering. Wessex Mill’s six seed flour is much nicer. We replace 150g of the plain flour with this, and add a good handful of extra linseeds. Semolina flour on the top and bottom looks great on this one too.

Gluten-free

For some coeliac friends we’ve made the poppy seed recipe, but replacing all the flour with the Doves Farm gluten-free variety, with surprisingly acceptable results (pictured right).

It’s a little bit chewy, but if eaten the same day it’s still really tasty, crusty and moist.

The results of our experiments with adding xanthan gum — which is sometimes used to give gluten-free doughs a bit of bounce — have been disastrously sticky and dense, though.

Schedule

The recipe might sound demanding, but we find the steps quick and the pauses flexible enough that they’re easy to fit around our schedule.

In the week, we typically do step 1 in the evening after work, step 2 the next morning before work, and step 3 (baking) that evening. One loaf lasts us two days, so we start the next loaf the next evening.

Since the second rising can be done in two hours or so, you can also do the whole thing in one day if you do step 1 first thing and steps 2 and 3 the same evening.

Saving money (but maybe not the world)

Even organic flour only costs 50p for 500g, and the yeast is pretty nearly free. I estimate the gas or electricity to keep the oven at 220 degrees for 75 minutes could cost 50 – 100% of that again (anyone know more exactly?), but you’re still looking at a pound or less a loaf.

A domestic oven baking one loaf is almost certainly less energy-efficient than a commercial oven baking many, so making your own bread is probably not entirely green. Then again, if your oven’s electric you could always switch to 100% renewable electricity.

It follows the same basic idea of using series of enlarging search radii to restrict distance calculations to a manageable subset of things-that-might-be-near. The difference is that it uses a geometric progression of sizes (x, x * y, x * y^2, x * y^3, ...) instead of an arithmetic one (x, x + y, x + 2y, x + 3y, ...).

For some distributions of things-that-might-be-near, and tuned with the right parameters (x, y), this turns out substantially faster (I’ve used it to locate the nearest UK postcode to each mappiness response).

Note that it’s a lot simpler and less generic than the Boston GIS alternative: it returns only one nearest neighbour, without the associated distance, and is also less flexible about what it accepts as input.

create or replace function 
  nn(nearTo                   geometry
   , initialDistance          real
   , distanceMultiplier       real 
   , maxPower                 integer
   , nearThings               text
   , nearThingsIdField        text
   , nearThingsGeometryField  text)
returns integer as $$
declare 
  sql     text;
  result  integer;
begin
  sql := ' select ' || quote_ident(nearThingsIdField) 
      || ' from '   || quote_ident(nearThings)
      || ' where st_dwithin($1, ' 
      ||   quote_ident(nearThingsGeometryField) || ', $2 * ($3 ^ $4))'
      || ' order by st_distance($1, ' || quote_ident(nearThingsGeometryField) || ')'
      || ' limit 1';
  for i in 0..maxPower loop
    execute sql into result using nearTo              -- $1
                                , initialDistance     -- $2
                                , distanceMultiplier  -- $3
                                , i;                  -- $4
    if result is not null then return result; end if;
  end loop;
  return null;
end
$$ language 'plpgsql' stable;

Update, August 2011 — Looking at this again, I realise there was an error in the function originally posted above. Because it did only a simple bounding-box check at each step, it was possible for it to return a geometry that was not in fact the nearest, but could in the worst case be up to 41% (sqrt(2) times) further away than the nearest. The code has been updated to fix this by using st_dwithin. I’ve also tidied it up a bit.

We decided it would be nice to allow respondents to interact with this CDF after providing some key values, and implemented this in the web browser with <canvas> (falling back on FlashCanvas for IE).

In the process, we implemented three kinds of cubic spline calculation in the ever-wonderful CoffeeScript: natural, clamped and (what we actually needed) monotonic cubic splines.

You can play with some examples below: click-and-drag the round handles, or double-click to enter values directly. Feel free to borrow the code (ideally with attribution) if it’s of use to you.

The spline code

class MonotonicCubicSpline
 
# by George MacKerron, mackerron.com
 
# adapted from: 
# http://sourceforge.net/mailarchive/forum.php?thread_name=
# EC90C5C6-C982-4F49-8D46-A64F270C5247%40gmail.com&forum_name=matplotlib-users
# (easier to read at http://old.nabble.com/%22Piecewise-Cubic-Hermite-Interpolating-
# Polynomial%22-in-python-td25204843.html)
 
# with help from:
# F N Fritsch & R E Carlson (1980) 'Monotone Piecewise Cubic Interpolation', 
#   SIAM Journal of Numerical Analysis 17(2), 238 - 246.
# http://en.wikipedia.org/wiki/Monotone_cubic_interpolation
# http://en.wikipedia.org/wiki/Cubic_Hermite_spline
 
  constructor: (x, y) ->
    n = x.length
    delta = []; m = []; alpha = []; beta = []; dist = []; tau = []
    for i in [0...(n - 1)]
      delta[i] = (y[i + 1] - y[i]) / (x[i + 1] - x[i])
      m[i] = (delta[i - 1] + delta[i]) / 2 if i > 0
    m[0] = delta[0]
    m[n - 1] = delta[n - 2]
    to_fix = []
    for i in [0...(n - 1)]
      to_fix.push(i) if delta[i] == 0
    for i in to_fix
      m[i] = m[i + 1] = 0
    for i in [0...(n - 1)]
      alpha[i] = m[i] / delta[i]
      beta[i]  = m[i + 1] / delta[i] 
      dist[i]  = Math.pow(alpha[i], 2) + Math.pow(beta[i], 2)
      tau[i]   = 3 / Math.sqrt(dist[i])
    to_fix = []
    for i in [0...(n - 1)]
      to_fix.push(i) if dist[i] > 9
    for i in to_fix
      m[i]     = tau[i] * alpha[i] * delta[i]
      m[i + 1] = tau[i] * beta[i]  * delta[i]
    @x = x[0...n]  # copy
    @y = y[0...n]  # copy
    @m = m
 
  interpolate: (x) ->
    for i in [(@x.length - 2)..0]
      break if @x[i] <= x
    h = @x[i + 1] - @x[i]
    t = (x - @x[i]) / h
    t2 = Math.pow(t, 2)
    t3 = Math.pow(t, 3)
    h00 =  2 * t3 - 3 * t2 + 1
    h10 =      t3 - 2 * t2 + t
    h01 = -2 * t3 + 3 * t2
    h11 =      t3  -    t2
    y = h00 * @y[i] + 
        h10 * h * @m[i] + 
        h01 * @y[i + 1] + 
        h11 * h * @m[i + 1]
    y
 
 
class CubicSpline  # natural or clamped
 
# by George MacKerron, mackerron.com
 
# adapted from:
# http://www.michonline.com/ryan/csc/m510/splinepresent.html
 
  constructor: (x, a, d0, dn) ->
    return unless x? and a?
    clamped = d0? and dn?
    n = x.length - 1
    h = []; y = []; l = []; u = []; z = []; c = []; b = []; d = []; k = []; s = []
    for i in [0...n]
      h[i] = x[i + 1] - x[i]
      k[i] = a[i + 1] - a[i]
      s[i] = k[i] / h[i]
    if clamped
      y[0] = 3 * (a[1] - a[0]) / h[0] - 3 * d0
      y[n] = 3 * dn - 3 * (a[n] - a[n - 1]) / h[n - 1]
    for i in [1...n]
      y[i] = 3 / h[i] * (a[i + 1] - a[i]) - 3 / h[i - 1] * (a[i] - a[i - 1])
    if clamped
      l[0] = 2 * h[0]
      u[0] = 0.5
      z[0] = y[0] / l[0]
    else
      l[0] = 1
      u[0] = 0
      z[0] = 0
    for i in [1...n]
      l[i] = 2 * (x[i + 1] - x[i - 1]) - h[i - 1] * u[i - 1]
      u[i] = h[i] / l[i]
      z[i] = (y[i] - h[i - 1] * z[i - 1]) / l[i]
    if clamped
      l[n] = h[n - 1] * (2 - u[n - 1])
      z[n] = (y[n] - h[n - 1] * z[n - 1]) / l[n]
      c[n] = z[n]
    else
      l[n] = 1
      z[n] = 0
      c[n] = 0
    for i in [(n - 1)..0]
      c[i] = z[i] - u[i] * c[i + 1]
      b[i] = (a[i + 1] - a[i]) / h[i] - h[i] * (c[i + 1] + 2 * c[i]) / 3
      d[i] = (c[i + 1] - c[i]) / (3 * h[i])
    @x = x[0..n]  # copy
    @a = a[0...n]
    @b = b
    @c = c[0...n]
    @d = d
 
  derivative: ->
    s = new this.constructor()
    s.x = @x[0...@x.length]  # copy
    s.a = @b[0...@b.length]  # copy
    s.b = 2 * c for c in @c
    s.c = 3 * d for d in @d
    s.d = 0 for x in [0...@d.length]
    s
 
  interpolate: (x) ->
    for i in [(@x.length - 1)..0]
      break if @x[i] <= x
    deltaX = x - @x[i]
    y = @a[i] + 
        @b[i] * deltaX + 
        @c[i] * Math.pow(deltaX, 2) + 
        @d[i] * Math.pow(deltaX, 3)
    y

Download:

• the above CoffeeScript
• the compiled JavaScript
• the minified compiled JavaScript

Some years ago, when working on a secure web app for a large organisation — let’s call them Secret Testing Ltd — I was keen that people shouldn’t choose hopelessly weak passwords. I was particularly concerned by my sysadmin colleague’s fondness for passwords of the form ‘p/\55w0rd’ or ‘S3cr3t-T35t|ng’.

I therefore wrote some simple Ruby code to try to catch very weak passwords:

class PasswordUtils
 
  def self.look_and_sound_alikes(original)
    look_or_sound_alikes = { \
      '0' => ['o'] ,
      '1' => ['i', 'l'],
      '2' => ['to', 'too'],
      '3' => ['e'],
      '4' => ['a', 'for'],
      '5' => ['s'],
      '6' => ['g'],
      '8' => ['b', 'ate'],
      '9' => ['g'],
      'i' => ['i'],
      '$' => ['s'],
      '|' => ['i', 'l', ''],
      '!' => ['i', 'l', ''],
      '@' => ['g', 'a'],
      '(' => ['c'],
      '[' => ['c', 'e'],
      '{' => ['e'],
      '&' => ['and'],
      '*' => ['star'],
      ')' => ['d'],
      '^' => ['a'],
      '/' => ['a', 'v'],
      '\\' => [''],
      '<' => ['k']
    } 
    # the penultimate two substitutions catch /\ (A) and \/ (V), albeit without
    # distinguishing them, and the last (in conjunction with the empty options 
    # for ! and |) catches !< and |< (K)
    versions = ['']
    original.downcase.each_char do |c|
      versions.collect! do |v|
        alikes = look_or_sound_alikes[c] || [c]
        alikes.collect { |c1| v + c1 }
      end
      versions.flatten!
    end
    versions
  end
 
  def self.obvious?(password, custom_banned_list = '')
    standard_banned_list = File.read("#{RAILS_ROOT}/config/banned_password_words.txt")
    banned_words = standard_banned_list.downcase.split(/\W+/) + 
      custom_banned_list.downcase.split(/\W+/).delete_if { |w| w.length < 4 }
    alike_words = [password.downcase] + PasswordUtils.look_and_sound_alikes(password)
    banned_words.any? do |banned_word|
      alike_words.any? { |alike_word| alike_word.include? banned_word }
    end
  end
 
end

The first method of the class undoes some common substitutions:

> PasswordUtils.look_and_sound_alikes('p/\55w0rd')
=> ["password", "pvssword"]
>  PasswordUtils.look_and_sound_alikes('S3cr3t-T35t|ng')
=> ["secret-testing", "secret-testlng", "secret-testng"]

And the second method uses this in conjunction with a list of banned words to check that an entered password isn’t hopelessly weak.

> PasswordUtils.obvious?('p/\55w0rd')
=> true
>  PasswordUtils.obvious?('S3cr3t-T35t|ng')
=> true
> PasswordUtils.obvious?('gkAsd76!o')
=> false

Obviously, you need a good banned words list — perhaps starting with some top N passwords lists, and including the name of your company, the web app, and words related to the area of business, local pubs, and so on.

It’s also a good idea to include custom banned words for individual users. For example:

custom_banned_list = "#{first_names} #{surname} #{department} #{organisation} #{email}"
PasswordUtils.password_obvious?(password, custom_banned_list)

Passing these tests isn’t a sufficient condition for ruling out a weak password, but it’s arguably a necessary one, and a good start.