Grunt made our lives easier and everyone was happy. The new freedom was exhilarating. Front-end devs highfived each other in the corridors.
But as time went on, we kept using it for more and more complex projects and Gruntfiles seemed to grow out of control, even though a lot of the tasks that were actually performed remained pretty much the same (transpile languages, minimise, etc). Setting up even a simple project today involves a fair amount of boilerplate code.
To ease the barrier to entry, scaffolding tools like Yeoman were introduced to abstract this process into a single command. The result, though, is simply that the boilerplate code has been generated for you (
yo webapp generates a Gruntfile over 400 lines long). It doesn’t make the boilerplate code any more maintainable or readable.
A Plumber for your asset pipeline
A few months ago, I started writing Plumber (formerly Luigi), a build tool focused on the processing of web assets using declarative pipelines of operations. This is only a subset of what people use Grunt for, but a key subset and one I felt wasn’t served as well as it could be.
I’m going to write a series of blog posts to describe the problems I set out to solve, the rationale behind the solution I came to and what it looks like for anyone who wants to use it.
Please note: I started this project before I was aware of similar projects like Gulp or James. Whilst these are all interesting in their own right, the design differences will hopefully become apparent in this post and the coming ones. Actually, Plumber might be closer to the work-in-progress node-task spec.
A simple Grunt setup
Here’s the corresponding Gruntfile to do that work:
Because Grunt treats independent tasks as the primary unit, there is no way to connect them, or to pass the output of one as the input of another. The three tasks registered at the end merely declare what tasks to run and in what order. At each step, we must write out files to disk and in order to pick them up at the next.
This explicit chaining is particularly brittle, as changes in one step might require adapting another, and it clearly contributes to the boilerplate in this config. It also clutters our working directory with intermediate files we’re not interested in.
Most tasks now conform to the multi-task and src-dest standard pattern, but not all. As an example, notice how the
less configuration differs from the other ones. With new tasks, you often need to learn a new way of passing input/output paths (e.g. RequireJS, CoffeeScript, Karma, etc.), which again contributes to making Gruntfiles more complex.
Some tasks are also guilty of taking liberties with the principle of “doing one thing well” and take on multiple responsibilities. For instance, RequireJS and LESS tasks are often used both to compile and minify sources, which blur the clarity of the process (what if you also want to minify another plain JS or CSS file?). While this isn’t intrinsically a Grunt issue, it is certainly exacerbated by the complexity of connecting simpler tasks together.
Finally, even though Grunt and Bower are both part of the Yeoman initiative, there is nothing to help you connect the two and pass files from Bower components to Grunt tasks; once again, explicit file paths are expected.
Deconstructing the non-linear Grunt configuration, the tasks could be described as the following two declarative pipelines:
and the library from the jQuery Bower component,
Then minimise them,
Then hash their filenames,
Then output all the resulting files in the ‘dist’ folder
Take all the LESS sources,
Then compile them to CSS,
Then concatenate them as ‘styles.css’,
Then output the resulting file in the ‘dist’ folder
As it turns out, this is pretty much exactly what a Plumbing file looks like:
The first thing to notice is the shift from Grunt’s imperative syntax (write how to do the work) to Plumber’s declarative one (describe what you want).
The emphasis is no longer on individual tasks, but instead on arrays of operations, or “sequential pipelines”. Typically, files are injected at the start and passed through a series of operations before being written out again. The chaining of operations happens automatically, without the need for manually specified intermediate files.
For a more visual representation, the pipelines are drawn in this diagram:
Following the principle of each operation doing only one thing and doing it well, even the sourcing of files is performed as an operation (
glob). This creates a uniform way to address files, rather than each operation specifying its own format. It also allows other operations to source files, such as the
bower operation above which gets the main file of the
jquery local Bower component automatically.
Because all operations conform to the same interface of receiving a list of resources as input and producing a Promise of a list of resources as output, they can be composed easily.
all operation augments the sequential nature of the pipelines by executing all the sub-pipelines it receives as argument in parallel, and piping the output to the rest of the pipeline. In the example above, both the concatenated
all.js file and the jQuery library are passed to the next
This architecture leads to many advantages:
If you add a new library to your project, you only need to do it in one place, not at every step.
For example, to add the
masonry Bower component to your build, you simply add it to your sources below
jquery and it will get uglified and hashed with the other files:
Since you don’t have to change the configuration any of the rest of the steps, the complexity of your Plumbing file remains linear with the number of inputs, rather than the product of the number of inputs and the number of steps as it would in a Gruntfile.
The declarative nature of the pipeline makes it easy to move things around.
Bored of minimising jQuery yourself? You could use the minified version directly and simply move the
uglifyjs operation one step up:
Composition and expressiveness
Plumber applies the principle of sensible defaults. For instance, if you’re hashing filenames, you’re almost certain to want to know the mapping of original to hashed filename, so it is outputted to the pipeline by the
hash operation. In our example, it is also written to the
To conclude this first post, I’d like to emphasise that Plumber is still in its early days and that none of this is to be taken as an attack against the principles behind Grunt (or Gulp, or any other build tools). There is a reason why these tools have become so popular, and I’m merely interested in probing ways to make it simpler and better to create for the Web. I hope sharing this rationale can help draft the node-task spec and encourage a discussion about what we all want from our build tools.
You’ll find the code for the simple example above in the plumber-examples repository, and Plumber itself in the Plumber repository.
Follow me on Twitter (@theefer) if you’d like to be notified of the next posts in this series. I’ll be talking about source maps and smart watching of changes.
Thanks to Oliver Ash for proof-reading this blog post!
[…] we saw in the last post about Plumber, sequencing transformations in Grunt isn’t particularly elegant as you have to manually […]