George MacKerron: code blog

GIS, software development, and other snippets

A depthcam? A webkinect? Introducing a new kind of webcam

Update, 10 February: Sorry for some serious reliability issues over the last few days. The streaming server is now hosted in-house at CASA, which should be a lot more robust.

At CASA we’ve been looking at using a Kinect or three in our forthcoming ANALOGIES (Analogues of Cities) conference + exhibition.

We’ve been inspired in part by Ruairi Glynn‘s amazing work here at UCL, and Martin has been happily experimenting with the OpenKinect bindings for Processing.

Meanwhile, I recently got to grips with the excellent Three.js, which makes WebGL — aka 3D graphics in modern browsers — as easy as falling off a log. I’m also a big fan of making things accessible over the web. And so I began to investigate prospects for working with Kinect data in HTML5.

There’s DepthJS, an extension for Chrome and Safari, but this requires a locally-connected Kinect and isn’t very clear on Windows support. There’s also Intrael, which serves the depth data as JPEG files and provides some simple scene recognition output as JSON. But it’s closed-source and not terribly flexible.

The depthcam

So I decided to roll my own. I give you: the depthcam!

Screenshot

Click here or on the screenshot to connect.

It’s a live-streaming 3D point-cloud, carried over a binary WebSocket. It responds to movement in the scene by panning the (virtual) camera, and you can also pan and zoom around with the mouse.

Currently you’ll need Google Chrome to try it, and the number of people who can tune in at once is limited for reasons of bandwidth. If you can’t connect, or nothing much is happening, try this short video on YouTube instead.

It might be the future of video-conferencing. It could also be the start of a new wave of web-based movement-powered games.

How it works

The code is on GitHub, and is in three parts:

  1. A short Python script wiring up the OpenKinect Python wrapper to the Autobahn WebSockets library. Depending on the arguments it’s run with, this can either serve a web browser directly, or it can push the depth data up to…
  2. A simple node.js server that gets us round the UCL firewall. This accepts depth data pushed from the Python script, and broadcasts it onwards (still using binary WebSockets) to any connected web browsers, which are running…
  3. The web-based client, written in CoffeeScript. This connects to the node.js server, receives the depth data, and visualises it as a particle system using Three.js and WebGL.

The incoming data from the Kinect is pretty heavy, at 18 MB/s (640 × 480 × 2 bytes per pixel x 30Hz). This is more than we can expect to (or afford to!) push over the Internet. So the Python script does some basic video compression to cut this down by several orders of magnitude, to 30 – 100 KB/s. It follows this three-step recipe:

  1. Reduce the amount of data by down-sampling and quantizing to 160 × 120 × 1 byte per pixel.
  2. Increase the data’s compressibility. First, reduce noise — which also looks bad — by making each transmitted depth value (i.e. pixel) the median of the values received in the last three frames. Then express each value as its difference from the previous value, with just an occasional absolute-value keyframe to allow new viewers to pick up the stream.
  3. Compress the data using LZMA, which gets better compression ratios than GZIP or BZIP2, and decompression times somewhere between the two.

Feel free to fork the code and do something great with it.

Share

Written by George

February 3rd, 2012 at 7:21 pm

  • Email

    Wow..Wow..Wow…. Kinect, Python, node.js, WebSockets, CoffeeScript… genius .. Wow.

  • http://twitter.com/dead_pony Christopher Weir

    That is AMAZING!!! Great job!

  • Optimus Prime

    ImportError: cannot import name WebClientContextFactory, even though I have autobahn and twisted installed. Ideas?

  • http://www.facebook.com/peter.lenagh Peter Lenagh

    Ha that’s awesome! Nice one :)

  • Anonymous

    I’m no Python guru, but is Autobahn definitely properly installed, and does Python know where to look for it (i.e. is it on your PYTHONPATH)? Perhaps it would be worth checking out the documentation/support on the Autobahn site: http://www.tavendo.de/autobahn/serverframework.html

  • http://www.facebook.com/amin.asadi Amin Asadi

    cool

  • Anonymous

    Thought you might like to know, this also works on the current FireFox nightly (13.0a1 [2012-02-07])

  • Alvaro

    And Firefox 11 beta

  • Anonymous

     Thanks both — that’s good to know.

  • Lcardona

    Hello, i’m getting error:
    freenect.c:2447: error: too many arguments to function ‘freenect_init’
    when compiling, any ideas??? AMAZING JOB!

  • Julentxu

    No binaries right now?

  • Anonymous

    None of my code is compiled — it’s all Python or CoffeeScript/JS — so there are no binaries.

    Perhaps you mean no installer? If anyone fancies packaging up the Kinect server side in an installer for Mac/Windows/Linux, be my guest: pull requests gratefully accepted.

  • Anonymous

    I assume this is during ‘brew install libfreenect’? Probably best to look for help from the libfreenect/OpenKinect people.

  • curious

    Hello!
    I am trying to setup your wonderful code within a LAN. I have managed
    to successful setup python script and node.js server thanks to your
    INSTALL instructions. However I am not getting any data stream once
    connected with a Chrome client (can see flat grid wall). You mention
    python script can serve webpage directly, perhaps this implementation
    is more simple. What are the argument parameters for ‘func’ and
    ‘url’? Thanks!

  • Anonymous

    You’re right: if you’re using it locally the direct serving option is easier, and you can ignore the node.js stuff. In this case, launch the Python script like so:

    ./websocket-kinect.py server ws://localhost:9000

    Then point the Chrome client in the right direction by adding a ‘ws’ parameter to the url:

    file://path/to/index.html?ws=ws://localhost:9000

  • Andrew

    Well. This is pretty amazing work. Thank you for sharing the code.

  • http://twitter.com/pashka_xell xeLL

    Hi everyone! Can you please answer for what? What is the ultimate goal?

  • Wizgrav

    Hi, I’m the author of intrael. It is and always has been open source(GPLv3). The latest version(2.0) resides in the SVN repo and has some js glue the simplifies use. Please update your description. You can find a webgl point cloud viewer here http://www.youtube.com/watch?v=fBNN8cKFla4. It’s adapted from a demo by mrdoob(just 3 lines of code actually) and uses the MJPEG output from intrael.

  • jawj

    Thanks for clarifying — I’ve struck out the offending sentence. Can Intrael send over WebSockets now?

  • Wizgrav

    Thanks :) For the JSON output, I opted for Server Sent Events which are basically one way websockets. They were much easier to implement in the general flow of the server but they have one caveat. There was a mixup between the spec writers and implementors and they forgot(Yeah I know lol) to add cors support so trying to retrieve data this way won’t generally work. One could always disable checking(in chrome is the –disable-web-security flag) but FF 11+ has support for it now and patches are in for webkit as well. 

  • http://www.broadband-expert.co.uk/ BroadbandBlogger

    Great details you have here. depthcam is good.