When Grunt first came around, it was an undeniable breath of fresh air: finally, a build tool with a common “task” interface for the variety of front-end jobs we’d been piecing together with a mishmash of ad-hoc shell scripts and slow Rhino-based solutions (remember the Dojo build system?). Better even, it was written in JavaScript using NodeJS, in the native language of the Web, so that Web people could easily understand and extend it.

Grunt made our lives easier and everyone was happy. The new freedom was exhilarating. Front-end devs highfived each other in the corridors.

But as time went on, we kept using it for more and more complex projects and Gruntfiles seemed to grow out of control, even though a lot of the tasks that were actually performed remained pretty much the same (transpile languages, minimise, etc). Setting up even a simple project today involves a fair amount of boilerplate code.

To ease the barrier to entry, scaffolding tools like Yeoman were introduced to abstract this process into a single command. The result, though, is simply that the boilerplate code has been generated for you (yo webapp generates a Gruntfile over 400 lines long). It doesn’t make the boilerplate code any more maintainable or readable.

A Plumber for your asset pipeline

A few months ago, I started writing Plumber (formerly Luigi), a build tool focused on the processing of web assets using declarative pipelines of operations. This is only a subset of what people use Grunt for, but a key subset and one I felt wasn’t served as well as it could be.

I’m going to write a series of blog posts to describe the problems I set out to solve, the rationale behind the solution I came to and what it looks like for anyone who wants to use it.

Please note: I started this project before I was aware of similar projects like Gulp or James. Whilst these are all interesting in their own right, the design differences will hopefully become apparent in this post and the coming ones. Actually, Plumber might be closer to the work-in-progress node-task spec.

A simple Grunt setup

To illustrate the problems I mentioned above, let’s take a simple imaginary web project: some JavaScript files, a jQuery dependency (as a Bower component) and some stylesheets written in LESS. See the plumber-examples repository for the complete code.

Following good front-end practices, we optimise the JavaScript by concatenating the application files together, minimising them and hashing their filename to cache-bust their URL based on their contents.

Here’s the corresponding Gruntfile to do that work:

module.exports = function(grunt) {

    grunt.initConfig({
        concat: {
            app: {
                src: 'src/js/**/*.js',
                dest: 'build/compiled/app.js'
            }
        },
        uglify: {
            app: {
                src: 'build/compiled/app.js',
                dest: 'build/minimised/app.min.js'
            },
            // For this example, we minimise jQuery ourselves
            // rather than using any pre-minimised file.
            jquery: {
                src: 'bower_components/jquery/jquery.js',
                dest: 'build/minimised/jquery.min.js'
            }
        },
        hash: {
            options: {
                mapping: 'dist-grunt/assets-mapping.json'
            },
            files: {
                src: 'build/minimised/*.js',
                dest: 'dist-grunt'
            }
        },
        less: {
            'dist-grunt/main.css': 'src/stylesheets/main.less'
        }
    });

    grunt.loadNpmTasks('grunt-contrib-concat');
    grunt.loadNpmTasks('grunt-contrib-uglify');
    grunt.loadNpmTasks('grunt-contrib-less');
    grunt.loadNpmTasks('grunt-hash');

    grunt.registerTask('javascript', ['concat', 'uglify', 'hash']);
    grunt.registerTask('css', ['less']);

    grunt.registerTask('default', ['javascript', 'css']);

};

Because Grunt treats independent tasks as the primary unit, there is no way to connect them, or to pass the output of one as the input of another. The three tasks registered at the end merely declare what tasks to run and in what order. At each step, we must write out files to disk and in order to pick them up at the next.

This explicit chaining is particularly brittle, as changes in one step might require adapting another, and it clearly contributes to the boilerplate in this config. It also clutters our working directory with intermediate files we’re not interested in.

Most tasks now conform to the multi-task and src-dest standard pattern, but not all. As an example, notice how the less configuration differs from the other ones. With new tasks, you often need to learn a new way of passing input/output paths (e.g. RequireJS, CoffeeScript, Karma, etc.), which again contributes to making Gruntfiles more complex.

Some tasks are also guilty of taking liberties with the principle of “doing one thing well” and take on multiple responsibilities. For instance, RequireJS and LESS tasks are often used both to compile and minify sources, which blur the clarity of the process (what if you also want to minify another plain JS or CSS file?). While this isn’t intrinsically a Grunt issue, it is certainly exacerbated by the complexity of connecting simpler tasks together.

Finally, even though Grunt and Bower are both part of the Yeoman initiative, there is nothing to help you connect the two and pass files from Bower components to Grunt tasks; once again, explicit file paths are expected.

Declarative pipelines

Deconstructing the non-linear Grunt configuration, the tasks could be described as the following two declarative pipelines:

Take all the JavaScript sources concatenated as ‘app.js’
and the library from the jQuery Bower component,
Then minimise them,
Then hash their filenames,
Then output all the resulting files in the ‘dist’ folder

Take all the LESS sources,
Then compile them to CSS,
Then concatenate them as ‘styles.css’,
Then output the resulting file in the ‘dist’ folder

As it turns out, this is pretty much exactly what a Plumbing file looks like:

// Load all the operations we need
var all      = require('plumber-all');
var write    = require('plumber-write');
var glob     = require('plumber-glob');
var bower    = require('plumber-bower');
var concat   = require('plumber-concat');
var uglifyjs = require('plumber-uglifyjs');
var hash     = require('plumber-hash');
var less     = require('plumber-less');

module.exports = function(pipelines) {

    pipelines['javascript'] = [
        all(
            [glob('src/js/**/*.js'), concat('app')],
            bower('jquery')
        ),
        uglifyjs(),
        hash(),
        write('dist-plumber')
    ];

    pipelines['css'] = [
        glob('src/stylesheets/main.less'),
        less(),
        write('dist-plumber')
    ];

};

The first thing to notice is the shift from Grunt’s imperative syntax (write how to do the work) to Plumber’s declarative one (describe what you want).

The emphasis is no longer on individual tasks, but instead on arrays of operations, or “sequential pipelines”. Typically, files are injected at the start and passed through a series of operations before being written out again. The chaining of operations happens automatically, without the need for manually specified intermediate files.

For a more visual representation, the pipelines are drawn in this diagram:

Following the principle of each operation doing only one thing and doing it well, even the sourcing of files is performed as an operation (glob). This creates a uniform way to address files, rather than each operation specifying its own format. It also allows other operations to source files, such as the bower operation above which gets the main file of the jquery local Bower component automatically.

Because all operations conform to the same interface of receiving a list of resources as input and producing a Promise of a list of resources as output, they can be composed easily.

The all operation augments the sequential nature of the pipelines by executing all the sub-pipelines it receives as argument in parallel, and piping the output to the rest of the pipeline. In the example above, both the concatenated all.js file and the jQuery library are passed to the next uglifyjs step.

This architecture leads to many advantages:

Scalability

If you add a new library to your project, you only need to do it in one place, not at every step.

For example, to add the masonry Bower component to your build, you simply add it to your sources below jquery and it will get uglified and hashed with the other files:

pipelines['javascript'] = [
    all(
        [glob('src/js/**/*.js'), concat('app')],
        bower('jquery'),
        bower('masonry'),
    ),
    uglifyjs(),
    hash(),
    write('dist-plumber')
];

Since you don’t have to change the configuration any of the rest of the steps, the complexity of your Plumbing file remains linear with the number of inputs, rather than the product of the number of inputs and the number of steps as it would in a Gruntfile.

Flexibility

The declarative nature of the pipeline makes it easy to move things around.

Bored of minimising jQuery yourself? You could use the minified version directly and simply move the uglifyjs operation one step up:

pipelines['javascript'] = [
    all(
        [glob('src/js/**/*.js'), concat('app'), uglifyjs()],
        // find jquery.min.js in jQuery Bower component directory
        bower('jquery', 'jquery.min.js')
    ),
    hash(),
    write('dist-plumber')
];

Composition and expressiveness

Operations are just plain JavaScript functions, so you can simply use variables to make the code more readable and avoid repetitions:

module.exports = function(pipelines) {
    var sources = glob.within('src');
    var jsFiles = sources('js/**/*.js');
    var lessFiles = sources('stylesheets/main.less');

    var toDist = write('dist-plumber');

    pipelines['javascript'] = [
        all(
            [jsFiles, concat('app')],
            bower('jquery')
        ),
        uglifyjs(),
        hash(),
        toDist
    ];

    pipelines['jshint'] = [
        jsFiles,
        jshint()
    ];

    pipelines['css'] = [
        lessFiles,
        less(),
        toDist
    ];

};

Sensible defaults

Plumber applies the principle of sensible defaults. For instance, if you’re hashing filenames, you’re almost certain to want to know the mapping of original to hashed filename, so it is outputted to the pipeline by the hash operation. In our example, it is also written to the dist-plumber directory.

To conclude this first post, I’d like to emphasise that Plumber is still in its early days and that none of this is to be taken as an attack against the principles behind Grunt (or Gulp, or any other build tools). There is a reason why these tools have become so popular, and I’m merely interested in probing ways to make it simpler and better to create for the Web. I hope sharing this rationale can help draft the node-task spec and encourage a discussion about what we all want from our build tools.

You’ll find the code for the simple example above in the plumber-examples repository, and Plumber itself in the Plumber repository.

Follow me on Twitter (@theefer) if you’d like to be notified of the next posts in this series. I’ll be talking about source maps and smart watching of changes.

Thanks to Oliver Ash for proof-reading this blog post!