George MacKerron: code blog » JavaScript

Extended and properly-multi-line Regular Expressions for JavaScript

George — Sun, 08 Aug 2010 18:44:46 +0000

I did a bit of work a while back on extending HotRuby. While none of that has yet seen the light of day, I thought the following snippet of it could be worth sharing.

Perl, Ruby, and some other languages support a much more readable ‘extended’ regular expression syntax, in which literal whitespace is ignored and comments (starting with #) are available. They also support a multi-line mode where the . character matches anything, including a newline.

JavaScript does neither of these: it doesn’t recognise the extended syntax, and its version of multi-line only allows the ^ and $ characters to match the beginnings and ends of lines within a string (it will never allow the . to match a newline).

So I wrote the following function to convert extended and fully-multi-line RegExp source strings to the basic syntax that JavaScript understands.

function convertRegExpSource(source, ext, multi) {  // string, boolean, boolean
  if (! ext && ! multi) return source;
  var convertedSource = '', len = source.length;
  var inCharClass = false, inComment = false, justBackslashed = false;
  for (var i = 0; i < len; i ++) {
    var c = source.charAt(i);
    if (justBackslashed) {
      if (! inComment) convertedSource += c;
      justBackslashed = false;
      continue;
    }
    if (c == '\\') {
      if (! inComment) convertedSource += c;
      justBackslashed = true;
      continue;
    }
    if (inCharClass) {
      convertedSource += c;
      if (c == ']') inCharClass = false;
      continue;
    }
    if (inComment) {
      if (c == "\n" || c == "\r") inComment = false;
      continue;
    }
    if (c == '[') {
      convertedSource += c;
      inCharClass = true;
      continue;
    }
    if (ext && c == '#') {
      inComment = true;
      continue;
    }
    if (multi && c == '.') {
      convertedSource += '[\\s\\S]';
      continue;
    }
    if (! ext || ! c.match(/\s/)) convertedSource += c;
  }
  return convertedSource;
}

Use it

The starting example below is the Daring Fireball monster URL regex.

Extended RegExp source (editable)

\b ( # Capture 1: entire matched URL (?: [a-z][\w-]+: # URL protocol and colon (?: /{1,3} # 1-3 slashes | # or [a-z0-9%] # Single letter or digit or '%' # (Trying not to match e.g. "URI::Escape") ) | # or www\d{0,3}[.] # "www.", "www1.", "www2." … "www999." | # or [a-z0-9.\-]+[.][a-z]{2,4}/ # looks like domain name followed by a slash ) (?: # One or more: [^\s()<>]+ # Run of non-space, non-()<> | # or $([^\s()<>]+|(\([^\s()<>]+$))*\) # balanced parens, up to 2 levels )+ (?: # End with: $([^\s()<>]+|(\([^\s()<>]+$))*\) # balanced parens, up to 2 levels | # or [^\s`!()\[\]{};:'".,<>?«»“”‘’] # not a space or one of these punct chars ) )

Basic RegExp source output

Let me know if you find any bugs, or if you know a nicer way to do this (I know nothing about parsing, so there surely is one).

I think I may suggest extended RegExp support as a possible CoffeeScript enhancement.

Signing Amazon Product Advertising API calls in Ruby

George — Sat, 22 Aug 2009 11:04:23 +0000

I have a simple site that generates covers for CDs I burn from iTunes purchases and so on (it pre-dates widespread use of JS libraries, and is in much need of prettifying). The site uses Amazon Product Advertising API calls to search and retrieve album cover art and track listings. Since earlier this month, such API calls have to be cryptographically signed.

This is somewhat annoying — the site’s original design has it communicating independently with Amazon (using Amazon’s XSLT API feature to transform their XML data into JSON), and that’s no longer possible with the use of a private key. But it’s not unfixable. The site now sends its API call first to my server, which returns a signed version, and then forwards the signed call on to Amazon.

I found most of what I needed for this on Chris Roos’ blog, but his version still wasn’t quite working for me (the two problems I recall are that Ruby’s CGI.escape doesn’t quite follow Amazon’s requirements, and that times need converting to GMT).

Anyway, in case you’re looking to do the same, here’s what I ended up with:

#!/usr/bin/env ruby
 
# Note: You need hmac.rb and hmac-sha2.rb from http://deisui.org/~ueno/ruby/hmac.html 
# somewhere in your require paths. ruby-hmac is currently broken under Ruby 1.9.
 
%w(rubygems cgi time hmac-sha2 base64).each { |lib| require lib }
 
ACCESS_IDENTIFIER = 'YOUR_PUBLIC_ID'
SECRET_IDENTIFIER = 'YOUR_PRIVATE_ID'
 
def aws_escape(s)
  s.gsub(/[^A-Za-z0-9_.~-]/) { |c| '%' + c[0].to_s(16).upcase }  
  # for 1.9, you'd replace [0] with .ord -- but ruby-hmac seems broken under 1.9
end
 
cgi = CGI.new
params = cgi.params.dup
 
amazon_endpoint = params.delete('amazon_endpoint')
amazon_path = params.delete('amazon_path')
js_callback = params.delete('js_callback')
 
signing_params = {
  'AWSAccessKeyId' => ACCESS_IDENTIFIER,
  'Timestamp' => Time.now.gmtime.iso8601
}
 
params.merge!(signing_params)
 
canonical_querystring = params.sort.collect do |key, value| 
  [aws_escape(key.to_s), aws_escape(value.to_s)].join('=') 
end.join('&')
 
string_to_sign = "GET\n#{amazon_endpoint}\n#{amazon_path}\n#{canonical_querystring}"
 
hmac = HMAC::SHA256.new(SECRET_IDENTIFIER)
hmac.update(string_to_sign)
signature = Base64.encode64(hmac.digest).chomp
 
params['Signature'] = signature
querystring = params.sort.collect do |key, value| 
  [aws_escape(key.to_s), aws_escape(value.to_s)].join('=') 
end.join('&')
 
signed_url = "http://#{amazon_endpoint}#{amazon_path}?#{querystring}"
 
cgi.out('type' => 'text/javascript') { "#{js_callback}('#{signed_url}');" }

You can test this locally by feeding key/value parameters to CGI, followed by Ctrl-D. These, for example:

amazon_endpoint=ecs.amazonaws.com
amazon_path=/onca/xml
js_callback=do_stuff
Service=AWSECommerceService
Version=2009-03-31
Operation=ItemSearch
SearchIndex=Books
Keywords=george+monbiot

Getting a Firefox extension’s directory from within the extension

George — Thu, 07 May 2009 12:23:07 +0000

When creating a Firefox extension recently, I was surprised how much pain is involved simply in finding out, from within an extension, where that extension’s files are installed.

You have to create a ‘component’, which entails a fair chunk of unintelligible boilerplate code (well, unintelligible unless you’re much better acquainted with Firefox’s innards than developing a basic extension generally requires you to be).

Plenty of places will tell you roughly how to do it. But, after some experimentation, let me show you exactly how.

Copy this JavaScript file into your extension’s /components directory (if that directory doesn’t exist yet, create it).
Use it from elsewhere in your extension like so:

own_path = Components.classes["@mackerron.com/get_ext_dir;1"]
  .createInstance().wrappedJSObject.get_ext_dir();

And that’s it. Let me know if it helps you out.