CommonJS effort sets JavaScript on path for world domination

This has been a big year for JavaScript. New, fast engines have tested their legs. Libraries have matured. With the ECMAScript 5 draft proposal, the language is growing. However, the language remains largely in exile, to only be used in Web browsers. This year has marked a resurgence of efforts to make JavaScript useful outside the browser. This was patently obvious at this year’s first European JavaScript conference, jsconf.eu.

There have been a variety of JavaScript platforms for programming outside the browser since 1996, starting with Netscape’s server-side offerings. A system called Helma (best known in Austria) has been around almost as long. In 2007, AppJet provided a service (now discontinued) for creating and hosting server-side JavaScript applications. Aptana offers an IDE for front-to-back JavaScript Web applications called Jaxer. Through each generation of Web development, JavaScript did not measure up against Perl, PHP, Python, Ruby, Java, or other alternatives for general-purpose programming.

Several things have changed. The creation, proliferation, and discovery of JavaScript-driven HTTP requests gave rise to highly interactive websites and applications. This drove the need for faster JavaScript engines, which have since become available to every serious denizen of the Web and most of their unsuspecting extended family. Furthermore, these highly-interactive applications usually require significant programming both on the client and the server, much of which now overlaps. Particularly, the technique of Progressive Enhancement often requires a feature to be implemented both on the server—for initial rendering and the baseline user-experience—and on the client, for an enhanced experience.

JavaScript has a couple more things going for it. JavaScript is the only programming language that has ever shipped on every consumer computer, and it has done so for the last 10 years. In these post-QBasic times, HTML, CSS, and JavaScript have become for every ungainly, bespectacled troglodyte the gateway from social exile to deeper social exile. More noteworthy is that a growing number of graphic designers who would never confess that they knew how to program will readily hack with jQuery. There are a lot of JavaScript programmers. Also, despite its deplorable shortcomings, JavaScript is cool and people like it.

So, what is holding JavaScript back from world domination?

• JavaScript has no module system. To compose JavaScript scripts, they must be either managed in HTML, concatenated, injected, or manually fetched and evaluated. There is no native facility for scope isolation or dependency management.
• JavaScript has no standard library. It has a browser API, dates, and math, but no file system API, much less an IO stream API or primordial types for binary data.
• JavaScript has no standard interfaces for things like Web servers or databases.
• JavaScript has no package management system that manages dependencies and automatically installs them, except JSAN (not to be confused with JSON), which falls short for scope isolation.

CommonJS (n�e ServerJS) is an initiative that began in January with a post by Kevin Dangoor to his blog and the founding of a mailing list. In his blog, Kevin outlined the above problems and called for server-side JavaScript aficionados to band together, write some specs, and support interoperability on their respective platforms.

“[This] is not a technical problem,” he wrote. “It’s a matter of people getting together and making a decision to step forward and start building up something bigger and cooler together.”

The call was answered.

One week later, with 224 members, and 653 messages posted, we knew Kevin had struck a nerve. It probably helped that Kevin was already known for creating the Python Turbo Gears framework, Paver, and is presently working on Mozilla’s Bespin. The people behind Helma, Hannes Wallnofer and Chris Zumbrunn, showed up. Kris Zyp, of JSON extension fame, showed up with his JavaScript persistent-object store project, Persevere. Ondrej Zara brought v8cgi to the drawing board. Ihab Awad brought Google’s Caja. Wes Garland brought a Mozilla SpiderMonkey-based implementation to the table.

One month later, we had converged on a specification for module systems and interoperable modules. Now, there are over a dozen compliant implementations of CommonJS module loaders, and hundreds of compliant modules.

CommonJS is a growing collection of standards, including

• Modules
• Binary strings and buffers
• Charset encodings
• Binary, buffered, and textual input and output (io) streams
• System process arguments, environment, and streams
• File system interface
• Socket streams
• Unit test assertions, running, and reporting
• Web server gateway interface, JSGI
• Local and remote packages and package management

So far, CommonJS has converged on specifications for modules (1.1), a system module (1.0), and a unit testing API (1.0). The intent is to drive for a binary API next; both ECMA TC39 (the technical committee responsible for ECMAScript) and the W3C (the standards body responsible for browser standards) are considering a joint pursuit.

Modules are a big deal. CommonJS modules represent the first time a JavaScript community has converged on a module system that is as good as (if not better than) the best of Python and Ruby’s—supporting not only dependency management, but also scope isolation and relative module identifiers. Module systems supporting CommonJS can be written for servers, browsers, desktop applications, and secure sandboxes. Browser deployments can vary from XHR+eval (for easy debugging) to script injection (for production or debugging) either with a build step or dynamic, server-side support. This means that modules targeting CommonJS compliance are truly interoperable. The CommonJS module standard separates the concern of how your module system works from how modules should be written.

And the best part is that it’s real. At jsconf.eu, several the CommonJS founder projects showcased CommonJS compliant module systems, a couple projects announced they were using CommonJS, and many others expressed interest in bringing CommonJS to their platforms.

NodeJS, a server-side JavaScript platform built on Google’s lightening-fast V8 engine by Ryan Dahl, showcases purely asynchronous IO and event loop concurrency. Dahl spoke passionately about the pitfalls of threading and synchronous IO. “Threaded concurrency is a leaky abstraction,” he claimed, and further, “Doing the right thing should be easy; doing the wrong thing should be hard.” In most programming contexts, there are pervasive synchronous IO libraries and system routines. It can be hard to know whether an API will block, which makes it impossible to reliably write non-blocking code. Anyone who has tried to write a concurrent socket server knows that the road is littered with the bones of those who came before. Dahl’s talk won him a standing ovation. Node is targeting a degree of CommonJS interoperability and presently supports the CommonJS module system. 

Narwhal is a cross-platform, multi-engine general purpose JavaScript platform brought to you by Tom Robinson and 280 North, the makers of Cappuccino, Objective-J, 280 Slides, and 280 Atlas. Narwhal presently supports Mozilla’s Rhino, V8, Apple’s JavaScriptCore, and desktop applications in Mozilla’s XULRunner. Narwhal can also host its modules for use in webpages.

Narwhal supports CommonJS modules and has prototype implementations of IO, file-system support, and binary data. Narwhal also comes with a wealth of standard library modules collected and ported from every end of the Web for everything from command line option parsing to secure hashing. Narwhal works out of the box and includes a package manager, “tusk,” that gives you instant access to literally dozens of installable packages. In all seriousness, the catalog is growing quickly because making a package is as simple as adding a package.json file to the root of a project on Github and as flexible as putting a Zip archive with a package.json anywhere on the Web.

Tom Robinson also ported Ruby’s Rack Web application gateway and middleware framework to Narwhal, dubbing it Jack. Jack tracks the CommonJS JSGI specification. Jack applications can be run locally on Rhino with the Java Simple framework or Jetty, on JavaScriptCore with Tom’s own Jill framework (comes with the narwhal-jsc package), and can be deployed on Google’s AppEngine.

Francisco Tolmasky from 280 North also presented on 280 Atlas and Cappuccino, a Web application IDE and framework modeled after Apple’s developer tools. 280 Atlas and the Cappuccino build tools are built on Narwhal with its JavaScriptCore engine adapters. Particularly, Francisco ported Ruby’s Rake to Narwhal, dubbing it Jake. Jake can be installed by running tusk install jake. 

Julian Viereck and Ajaxian’s very own Dion Almear gave a presentation on Bespin. The experimental Web-based IDE is in the midst of a refactor that will make it much easier to develop plugins, and will include server-side JavaScript components built on SproutCore, a server-side JavaScript framework that supports CommonJS modules.