37signals' Chalk Dissected

Update 2010-11-05: I dove into the JavaScript a little and explained most of it. Sam Stephenson tweeted that Chalk is written in CoffeeScript and compiled on the fly when served using Brochure. That's hot! (for those unaware Sam Stephenson works at 37signals, and is also the man behind Prototype.)

37signals recently released a blackboard web app for iPad called Chalk.

It includes Thomas Fuchs new mobile JS framework Zepto, a few images, iOS SpringBoard icon, and of course HTML, CSS, and JavaScript. It weighs in at about 244k including 216k of images. HTML, CSS, and JavaScript are not minified (except Zepto), but they are gzipped. Because the image-to-text ratio is high gzip can only shave off 12k. There is absolutely nothing there that isn't required though. The code and resources are very tight, readable, and beautiful.

The manifest is a nice summary of the contents, and allows browsers to cache the app for offline use. Combine this with mobile Safari's "Add to Home Screen" button and you have yourself a free chalkboard app that works offline.

CACHE MANIFEST

/
/zepto.min.js
/chalk.js
/images/background.jpg
/images/chalk.png
/images/chalk-sprites.png
/images/chalk-tile-erase.jpg
/images/chalk-tile-red.png
/images/chalk-tile-white.png
/stylesheets/chalk.css

Not much there, just 10 requests to fetch the whole thing. 11 including the manifest. In we go.

 

HTML

2k, 61 lines. 10 of which are Google Analytics JavaScript. Let's glance at some of it.

Standard html5 doctype, and a manifest for application caching.

The rest of the HTML is mainly structural. There is not a single text node in the entire tree (excluding whitespace). The chalkboard is a canvas element and an image element used to render the canvas contents as an image for sharing. The other elements are just sprites and buttons. There are div elements for the light switch and shade (a dimmer on each side), share button, instructions on sharing, close button, ledge, chalk, eraser and corresponding indicators. Phew, that was a mouthful. (oblig: "that's what she said!")

The interesting thing about the HTML is that without any JavaScript or CSS the document would be a completely blank white page (except for a strange looking share button w/ no title). Talk about progressive enhancement. Here's a look at the HTML:

Onward.

 

Zepto

Zepto is a tiny, modern JS framework for mobile WebKit browsers such as those found on iPhone and Android handsets. I'm not going to cover it here but I'll mention that it's similar in feel to jQuery. In fact it tries to mimic jQuery very closely to make migrations from Zepto to jQuery easy, and vice versa. The reason it weighs in at just under 6k (2k gzipped) is that it doesn't overreach or have to support legacy crap like IE6. It was started by Thomas Fuchs so you know it's good.

 

Display (CSS & Images)

6.6k, 385 lines. This is basically half of the text portion, excluding Zepto. There are 6 images including one called chalk-sprites.png. Interesting. Let's look at the background first though.

 

Background

 

background.jpg 1024x946px

The background is the blackboard itself, and is almost square at 1024x946. The cork border and light switch are there too. This is set as the background-image of the html element and is positioned at a negative x or y in order to centre it properly. CSS media queries are used to detect the screen's orientation. This way the same image is used for both orientations, clever.

 

Chalkboard

Just a canvas element positioned over the chalkboard using media queries. There's also an image element called "output" used to render an image for sharing.

 

Sprites

 

chalk-sprites.png

Sprites are used for all the other elements: ledge, chalk, eraser, tool indicator, share button, instructions, and close button (to leave the sharing mode). Positioned using CSS, standard stuff. There is white text alongside those green arrows. If you want to see it we'll have to change the background to black.

 

Light Switch & Shade

When you touch the light switch on the left side of the chalkboard - only visible in landscape orientation - the cork border dims and the ledge and share button disappear, leaving the chalkboard under the spotlight all classy like. The shade consists of two "dimmer" div elements inside a shade div, which is hidden by default.

The dimmers background color is black at 67% opacity. The shade element fades in using -webkit-transition: on its visibility property while the dimmers use CSS3 transitions on their background. The dimmers are positioned using media queries as well, one on each side of the board. Interestingly their parent shade has a height and width of 0. Rather than each having a unique id they just have the class "dim" and the :nth-child pseudo-class selector is used to position them independently.

If you took a look at the HTML before you'll have noticed there's no shade class defined on the body element. Looks like they're using JavaScript to add the shade class to body, triggering the transitions to the visible shades and setting the dimmers backgrounds to black at the same time, causing the fading effect. The shade fades in while the ledge and share button fade out.

The light switch itself is displayed only in landscape orientation, again using a media query.

 

Tools

There are 2 layers to the tools on the ledge. There are the images of the tools and their indicators, but also an anchor element for each tool that acts as targets to select them. When tools are select the indicators fade in and out using CSS3 transitions on opacity by adding and removing the class "active" on the tool.

There are pattern images for each colour of chalk, and one for the the eraser. The eraser "pattern" is the entire blackboard so erasing it doesn't look ugly. I love that kind of attention to detail.

 

Sharing

The shade effect that happens when you hit the share button is similar to the shade effect used for the light switch. It's a bit more complex as the sharing instructions are positioned differently in portrait and landscape orientations, but there's nothing really new in there (that I can see).

The rest of the CSS is largely presentational stuff like removing margins and padding, and positioning using lots of media queries. You can see it all at chalk.37signals.com/stylesheets/chalk.css.

 

JavaScript (and CoffeeScript)

5.5k in about 170 lines. That's just half the size of the CSS.

Sam Stephenson shared the original CoffeeScript source with us. It's about 150 lines, and is a bit easier to read as CS is far cleaner than JS.

The bulk of the magic is done w/ hardware accelerated CSS3 rather than slow JS animation using setInterval and setTimeout to change properties. That sort of thing isn't novel anymore anyway. The fact that JS is really only used for drawing and toggling CSS classes is pretty awesome!

The entire contents of the JS reside inside the DOMContentLoaded event handler attached to window.

 

Initialization

 

First we get a handle on all the elements and the canvas' 2d drawing context. I almost want to say views and controls as it really feels just like hooking up a controller and view in a desktop GUI app. Sometimes the line between dynamic web page and web app are blurred, not so here. Chalk is 100% app.

The canvas' dimensions and pen are initialized in lines 13 - 19, and then the chalkboard background is drawn onto the canvas using the drawImage() method.

The canvas offsets are cached for calculations, and are updated when the window fires the "orientationChange" event. Next up tools (a.k.a. pens) are created and initialized.

 

Tools

 

createPattern(name, callback) loads one of the pattern images, chalk-tile-*, and then creates a pattern in the drawing context and passes it to the given callback.

setStroke(pattern, width) effectively sets the pen used for drawing, described as a pattern & stroke width. The patterns are initialized and the white pen is passed to setStroke since it's the default tool.

The last part defines the 3 tools, note that the active tool "white_chalk" is at the end. Also note that the tool names are the ids of the target elements in the ledge. activateTool(tool) accepts a tool name. The tool to activate is moved to the end of the tools array on lines 31-32, activeTool is set to the given tool as well on line 32. The reason for moving the active tool to the end of the array is revealed in the for loop on line 34, the order of the tools array determines their z-index ordering (highest number is in front). Then the 'active' CSS class is added to the active tool to show the indicator, and then the pen is set by assigning a pen to the context's strokeStyle property.

Finally the white_chalk tool is activated and the click event for the tool targets is setup.

 

Drawing

 

Drawing is done by listening for touch events on the canvas element. An array of points to draw is initialized to a 1-element array containing null. Null values make the draw function break up the line being drawn by skipping the next point in the array. x and y coords are initialized in touchstart, points are appended to the points array in touchmove, and the touchend handler appends two points and null to the points array to end the line. I'm not sure why [x, y] is used as the points in the touchend handler rather than coords from the event. Please leave a comment if you know why!

The draw function is called for each point in the points array at 30ms intervals. A line is started by calling context.beginPath(), each point is drawn, and then the line is ended with context.stroke(). The 2nd condition of the while loop ensures that we don't draw for too long, as bad things would happen if the function were executed a 2nd time while it was already running.

Sam Stephenson was kind enough to clarify these points. See his comment below the post for clarification on using [x, y] in the touchend handler and the 10ms limit when drawing points.

 

Light Switch & Shade

 

When the light switch is touched (or clicked) the shade class on the body element is toggled. Nothing to it.

 

Sharing

 

The share window is opened after a 10ms delay, just enough time for any drawing to be completed before rendering the image. The image is created by assigning the result of canvas' toDataURL() method to the output image element's src attribute.

When the share window is closed the output image element gets its src set to the sprites image. I'm not sure why that was done. As Sam mentions in his comment below, this is done to reclaim the memory used by the rendered image.

The rest of the code there just sets up event handlers and toggles CSS classes.

 

That's it!

That about covers it. Don't have an iPad? Play around with it anyway, but be warned that you can't draw anything. You can select chalk and the eraser and hit the light switch. I instinctively tried touching my MacBook's display but alas it doesn't magically respond to touches, lame.

Have fun drawing. Thanks to 37signals for a beautiful (and useful) example of a few modern web technologies.

ScrobbleShark is alive!

Thanks to a kind fellow from Last.fm a new web app of mine called ScrobbleShark appears to be working. If you are one of the 100 people on earth who uses both Grooveshark and Last.fm - and one of the 10 who also want your Grooveshark plays scrobbled to Last.fm - then check it out. It supports automatic and manual scrobbling so you can try it out at first, just be sure to click 'pause scrobbling' after setting it up because automatic is the default option.

It's really easy to use. You login with your Google account, authenticate with Last.fm, enter your Grooveshark username, and you are up and running. I need some beta testers so if you only use Grooveshark, why not join Last.fm and give it a shot?

There are apps to visualize your listening history and other fun stuff. You can also meet people who have similar tastes in music, if you want to. It's really cool! Grooveshark is a great player and Last.fm is great for the community and data mining possibilities. ScrobbleShark helps you enjoy the best of both services.

Let me know what you think if you try it.

A preview of Mach-O file generation

This month I got back into an x86 compiler I started last May. It lives on github.

The code is a bit of a mess but it mostly works. It generates Mach object files that are linked with gcc to produce executable binaries.

The Big Refactoring of January 2010 has come to an end and the tests pass again, even if printing is broken it prints something, and more importantly compiles test/test_huge.code into something that works.

After print is fixed I can clean up the code before implementing anything new. I wasn't sure if I'd get back into this or not and am pretty excited about it. I'm learning a lot from this project.

If you are following the Mach-O posts you might want to look at asm/machofile.rb, a library for creating Mach-O files. Using it is quite straightforward, an example is in asm/binary.rb, in the #output method.

Definitely time for bed now!

Basics of the Mach-O file format

This post is part of a series on generating basic x86 Mach-O files with Ruby. The first post introduced CStruct, a Ruby class used to serialize simple struct-like objects.

Please note that the best way to learn about Mach-O properly is to read Apple's documentation on Mach-O, which is pretty good combined with the comments in /usr/include/mach-o/*.h. These posts will only cover the basics necessary to generate a simple object file for linking with ld or gcc, and are not meant to be comprehensive.

Mach-O File Format Overview

A Mach-O file consists of 2 main pieces: the header and the data. The header is basically a map of the file describing what it contains and the position of everything contained in it. The data comes directly after the header and consists of a number of binary blobs of data, one after the other.

The header contains 3 types of records: the Mach header, segments, and sections. Each binary blob is described by a named section in the header. Sections are grouped into one or more named segments. The Mach header is just one part of the header and should not be confused with the entire header. It contains information about the file as a whole, and specifies the number of segments as well.

Take a quick look at Figure 1 in Apple's Mach-O overview, which illustrates this quite nicely.

A very basic Mach object file consists of a header followed by single blob of machine code. That blob could be described by a single section named __text, inside a single nameless segment. Here's a diagram showing the layout of such a file:

            ,---------------------------,
  Header    |  Mach header              |
            |    Segment 1              |
            |      Section 1 (__text)   | --,
            |---------------------------|   | 
  Data      |           blob            | <-'
            '---------------------------'      

The Mach Header

The Mach header contains the architecture (cpu type), the type of file (object in our case), and the number of segments. There is more to it but that's about all we care about. To see exactly what's in a Mach header fire up a shell and type otool -h /bin/zsh (on a Mac).

Using CStruct we define the Mach header like so:

Segments

Segments, or segment commands, specify where in memory the segment should be loaded by the OS, and the number of bytes to allocate for that segment. They also specify which bytes inside the file are part of that segment, and how many sections it contains.

One benefit to generating an object file rather than an executable is that we let the linker worry about some details. One of those details is where in memory segments will ultimately end up.

Names are optional and can be arbitrary, but the convention is to name segments with uppercase letters preceded by two underscores, e.g. __DATA or __TEXT

The code exposes some more details about segment commands, but should be easy enough to follow.

Sections

All sections within a segment are described one after the other directly after each segment command. Sections define their name, address in memory, size, offset of section data within the file, and segment name. The segment name might seem redundant but in the next post we'll see why this is useful information to have in the section header.

Sections can optionally specify a map to addresses within their binary blob, called a relocation table. This is used by the linker. Since we're letting the linker work out where to place everything in memory the addresses inside our machine code will need to be updated.

By convention segments are named with lowercase letters preceded by two underscores, e.g. __bss or __text

Finally, the Ruby code describing section structs:

macho.rb

As much of the Mach-O format as we need is defined in asm/macho.rb. The Mach header, Segment commands, sections, relocation tables, and symbol table structs are all there, with a few constants as well.

I'll cover symbol tables and relocation tables in my next post.

Looking at real Mach-O files

To see the segments and sections of an object file, run otool -l /usr/lib/crt1.o. -l is for load commands. If you want to see why we stick to generating object files instead of executables run otool -l /bin/zsh. They are complicated beasts.

If you want to see the actual data for a section otool provides a couple of ways to do this. The first is to use otool -d <segment> <section> for an arbitrary section. To see the contents of a well-known section, such as __text in the __TEXT segment, use otool -t /usr/bin/true. You can also disassemble the __text section with otool -tv /usr/bin/true.

You'll get to know otool quite well if you work with Mach-O.

Take a break!

That was probably a lot to digest, and to make real sense of it you might need to read some of the official documentation.

We're close to being able to describe a minimal Mach object file that can be linked, and the resulting binary executed. By the end of the next post we'll be there.

(You can almost do that with what we know now. If you create a Mach file with a Mach header (ncmds=1), a single unnamed segment (nsects=1), and then a section named __text with a segment name of __TEXT, and some x86 machine code as the section data, you would almost have a useful Mach object file.)

Till next time, happy hacking!

Working with C-style structs in Ruby

This is the beginning of a series on generating Mach-O object files in Ruby. We start small by introducing some Ruby tools that are useful when working with binary data. Subsequent articles will cover a subset of the Mach-O file format, then generating Mach object files suitable for linking with ld or gcc to produce working executables. A basic knowledge of Ruby and C are assumed. You can likely wing it on the Ruby side of things if you know any similar languages.

First we need to read and write structured binary files with Ruby. Array#pack and String#unpack get the job done at a low level, but every time I use them I have to look up the documentation. It would also be nice to encapsulate serializing and deserializing into classes describing the various binary data structures. The built-in Struct class sounds promising but did not meet my needs, nor was it easily extended to meet them.

Meet CStruct, a class that you can use to describe a binary structure, somewhat similar to how you would do it in C. Subclassing CStruct results in a class whose instances can be serialized, and unserialized, with little effort. You can subclass descendants of CStruct to extend them with additional members. CStruct does not implement much more than is necessary for the compiler. For example there is no support for floating point. If you want to use this for more general purpose tasks be warned that it may require some work. Anything supported by Array#pack is fairly easy to add though.

First a quick example and then we'll get into the CStruct class itself. In C you may write the following to have one struct "inherit" from another:



With CStruct in Ruby that translates to:



CStructs act like Ruby's built-in Struct to a certain extent. They are instantiated the same way, by passing values to #new in the same order they are defined in the class. You can find out the size (in bytes) of a CStruct instance using the #bytesize method, or of any member using #sizeof(name).

The most important method (for us) is #serialize, which returns a binary string representing the contents of the CStruct.

(I know that CStruct.new_from_bin should be called CStruct.unserialize, you can see where my focus was when I wrote it.)

CStruct#serialize automatically creates a "pack pattern", which is an array of strings used to pack each member in turn. The pack pattern is mapped to the result of calling Array#pack on each corresponding member, and then the resulting strings are joined together. Serializing strings complicates matters so we cannot build up a pack pattern string and then serialize it in one go, but conceptually it's quite similar.

Unserializing is the same process in reverse, and was mainly added for completeness and testing purposes.

That's about all you need to know to use CStruct. The code needs some work but I decided to just go with what I have already so I can get on with the more interesting and fun tasks.

Next in this series: Basics of the Mach-O file format

Using Emacs to develop Mojo apps for webOS

The latest technology I've been learning is Palm's SDK for webOS, Mojo.  My first impression is that it's a great platform and Palm could do a great job of 2.0 if they cut down on some of the verbosity of gluing together the UI.  I have learned to like JavaScript over the years as I learned that despite its warts there are good parts too.  If you squint just right you can see that it's scheme with Algol syntax.  HTML and CSS are what they are, but with WebKit running the show and only a single engine to target it's not that bad.  I've gone from Eclipse to Emacs for the coding itself and highly recommend Emacs for Mojo development.  There is nothing that I miss from the Eclipse or Komodo Edit thanks to the fact that Mojo uses open languages and standards.

As far as actual development goes the Mojo documentation steers you towards a combination of Eclipse, Palm's Mojo plugin for Eclipse, and the Aptana Studio plugin.  My editor of choice is Emacs but I decided to give it a spin just to get started quickly, how bad could it be? I'm not going to get into details but I will say that I don't think I'll ever use Eclipse for anything; it's far too sluggish and provides no compelling features for the languages that I use.  I tried Komodo Edit and it was significantly better but still not for me.  Emacs is great for editing HTML, JavaScript, and CSS so all I really missed from the IDEs were the shortcuts to package, install, and launch apps in the emulator.  I headed over to the Emacs Wiki and downloaded Jonathan Arkell's Mojo support for Emacs which provided a great base to get started with.  There are wrappers around (all?) of the Palm SDK commands but it needed a bit of work to make it just do what I wanted with as little input and thought as possible.

A couple of of Lisp hacking sessions later and I'm happy enough with mojo.el to bump the version to v0.9.  I've checked off what I feel are the most important checkpoints on the webOS Internals comparison of editors and the framework is in place to make implementing most of the remaining commands very trivial.  I might take a bit of time today to flesh things out just to check more points off so people feel more confident that it's a fully featured environment, because it certainly is.

It now requires json.el in order to parse appinfo.json. json.el might be included with Emacs if you have a very recent version, otherwise you can google for it or get it from my config file repo on github where you can also find my latest version of mojo.el.  You still just (require 'mojo) in your .emacs file.

The wrappers around Palm SDK commands now search upwards for the Mojo project root directory (from the default-directory for current-buffer) and parse appinfo.json to give you sane defaults for mojo-package, mojo-install, mojo-launch, mojo-delete, and mojo-inspect.  You can list installed apps and when entering app ids there is completion and history, as you have come to expect in Emacs.  The most useful command for development is mojo-package-install-and-inspect which does exactly what it says: packages, installs, and launches the application for inspection.  No interaction is required as long as you are editing a buffer inside your Mojo project.

If you read the install instructions in mojo.el and decide to setup some keybindings then you will have single-task commands for packaging, installing, launching, or all three steps at once.

Please give me some feedback if you try this out.  I've developed it on Mac OS X and Jonathan on Windows so please try it on Linux and send me a patch or even better a pull request on github if it needs some work. There is room for improvement. The next feature on my radar before I would consider it worthy of a v1.0 tag is intelligent switching to corresponding buffers, e.g. mojo-switch-to-view, mojo-switch-to-assistant, things like that. Basically things I miss from the Rails package for Emacs.

Happy hacking!

Yet another compiler posted to github, and first post

Tonight I pushed a pet compiler project I have been working on to github. It's my first code on github and I hadn't really intended on making it public. I'll be quite surprised if anyone is interested in it at the moment, the README doesn't even explain where to start.  Oh well, I will fix it up for public consumption later.  Release early and often.

Every programmer thinks he knows what his perfect language looks like, I am no different. Right now it's little more than turing complete, but it parses a small language and outputs x86 machine code. Only Mach-O binaries on OS X work at the moment but ELF support on Linux is imminent. The assembler is homegrown and done in Ruby but only assembles the subset of x86 needed to compile ... whatever I might call this language in the future.

I'm not sure what the end result will be at all but traits I think a modern language should include are:

• Immutable data structures
• Literal notation for hashes, arrays, closures, etc.
• Multiple block support (possibly io style)
• Optional OO
• AOT and JIT compilation to machine code
• Platform independent, widely ported
• FFI
• GC (need I mention this anymore?)

That's it off the top of my head.  Factor, Clojure, Ruby, JavaScript, and C# are languages worth stealing from.  These may overlap but also io, SmallTalk, Python, Java, C, Lisp, Scheme.  I hope to write up some on exactly what I like from each of these languages in the future.  Till then.