Intro to sbt-web

Inside sbt-web: Part I – Using sbt-web

Introduction

sbt-web is an sbt-based web infrastructure for handling client-side files. It serves as the new foundation for the Playframework and potentially any other sbt-based web project. sbt-web defines a basic directory structure (overriden in Play), and it offers an API for processing sources, static files, and reporting errors.

Motivation

Why do we need a new web infrastructure, why was the old one not good enough?

When Play 2.0 was conceived in 2011, the developers chose to integrate several client-side technologies: CoffeeScript, LESS, Google Closure, and later RequireJS. These were tightly integrated into Play, so everyone who wanted to use something different (e.g. SASS or Dust) had to first learn sbt, write a plugin, and somehow deal with integrating that framework into Play’s build mechanism. This obviously wasn’t a flexible enough solution.

Additionally, no-one could foresee the activity and speed of innovation in the JavaScript world1. Every week there’s a new framework coming out, and existing ones make steady progress. The Node.js community alone is around 10x bigger than the Play community, so keeping CoffeeScript and LESS inside Play up-to-date was a daunting task.

It became more and more difficult to catch up. At some point it wasn’t even possible to update Twitter Bootstrap because of newer LESS features that weren’t supported by Play. And the Play team simply didn’t have the resources to update client-side dependencies every couple of weeks.

So with Play 2.3 the decision was made to factor out any client-side concerns into a new independent project called sbt-web. sbt-web introduces the notion of assets to sbt. Taken from the Rails world, assets are simply files that are served to and displayed or otherwise processed by the web browser.

sbt-web is organized via sbt plugins that use sbt-web’s infrastructure to make integrating and updating the latest tools easier.

Project Layout

The basic project structure2 defined by sbt-web adapts to the standard sbt (or Maven) conventions and essentially adds the assets folder to the mix.

The input folder is src/main/assets for source files that are potentially converted (e.g. from CoffeeScript to JavaScript) or otherwise processed both while developing and in production. src/main/public is for static assets. When preparing for production-readiness (staging), the two will be merged and processed.

Please note that the subfolder organization below assets is only an example, sbt-web doesn’t require you to use a specific naming scheme (or any subfolder at all for that matter).

There is also a src/main/test/assets folder indicating that sbt-web also supports testing client-side code.

+ src
--+ main
----+ assets
------+ js
----+ public
------+ css
------+ images
------+ js
--+ test
----+ assets
------+ js
----+ public
------+ css
------+ images
------+ js

The target folder contains all sbt output files, sbt-web’s output is organized under web. No matter where the input comes from, it will be stored in public/main plus their original relative path. The subfolders are again only examples.

+ target
--+ web
----+ public
------+ main
--------+ css
--------+ images
--------+ js
------+ test
--------+ css
--------+ images
--------+ js

And finally the stage directory that contains the complete, distributable version of your assets when staging:

+ target
--+ web
----+ stage

Note that Play overrides src/main to app, and src/test to test.

Processing Assets

Assets can be sources that need processing or analysis by a source task. All assets, no matter if converted sources or static, will run through the asset pipeline.

Source Tasks

Source tasks process files in src/main/assets.

Some plugins in this category will convert source files to a file that can be processed by a web browser. The output goes to target/web/public/main or target/web/resource-managed/main. Examples are CoffeeScript to JavaScript, or LESS to CSS.

Plugins in the analysis category don’t produce any output but only check your source files and report errors or other results, for example static code analysis such as JSHint, or server-side testing such as Mocha.

Most source plugins use the JavaScript infrastructure that we’ll cover later.

sbt-web-dev

Try out source tasks by running sbt web-assets:assets and check the target/web folder. Errors should be displayed in the console.

Asset Pipeline

The asset pipeline is also a concept from Rails. In Rails however, it describes all asset processing. In sbt-web, it only describes processing for staging output, that is when you want to deploy your app to a server. The asset pipeline consists of stages that are processed in sequential order. Each stage takes the previous stage’s set of files3 and passes it to the next stage, possibly adding generated files or removing no longer needed ones.

An asset pipeline plugin typically provides one stage. You have to explicitly add it to the pipeline in your build file. Examples are gzipping your assets, or adding a hash of the file to its name. As you can imagine, order matters here. You should hash first, then gzip.

sbt-web-staging

To try the pipeline out, run sbt web-pipeline and check the target/web folder. To prepare for staging, run sbt web-stage and check target/web/stage/public. The latter should contain all your assets while the former only contains products from source tasks and the asset pipeline.

JavaScript Tooling

On top of sbt-web there are two additional frameworks.

js-engine allows you to execute DOM-less JavaScript using either the JVM-based Trireme engine, Rhino, or Node.js itself4. In your projects you’d add the sbt wrapper sbt-js-engine which also incorporates sbt-js-task that makes it easy to run and process JavaScript code in your build.

webdriver-sbt provides the ability to execute DOM-based JavaScript, which is interesting for both testing and server-side rendering.

Both will be covered in future blog posts.

Client-Side Dependency Management

Not a part of sbt-web, but deeply integrated, are WebJars. WebJars are client-side dependencies packaged in JARs so you can declare them as normal dependencies in your build file. sbt-web extracts WebJars to target/web/web-modules/main/webjars/lib. WebJars are also used to export assets from sub-projects.

In Play you can refer to them via lib. For example @routes.Assets.at("lib/bootstrap/css/bootstrap.css").

Work is also underway to resolve dependencies from Node Package Manager (npm) package.json files.

Using sbt-web in Play

So far Play is the only framework using sbt-web, but in theory you could also use it to manage your personal homepage. sbt-web provides a couple of plug-ins for various web languages, preprocessors, and frameworks.

If you haven’t seen sbt-web in action yet, please watch this excellent and comprehensive talk by James Ward.

sbt-web and Play use sbt’s auto-plugin architecture, so as soon as you add an sbt-web plug-in to plugins.sbt, it is enabled automatically.

Here are the official source plugins:

  • sbt-coffeescript CoffeeScript to JavaScript conversion
  • sbt-jshint JavaScript linting
  • sbt-stylus CSS preprocessor
  • sbt-less CSS preprocessor
  • sbt-mocha Server-side JavaScript testing
  • sbt-scalajs ScalaJS to JavaScript conversion (work in progress)

And the official asset pipeline plugins:

  • sbt-digest Fingerprinting (adds a MD5 hash to filename to avoid browser caching issues)
  • sbt-gzip Gzip assets (compresses assets using Gzip to reduce load times)
  • sbt-rjs RequireJS integration, JavaScript minification (thus also a source task), automatic WebJars CDN mapping
  • sbt-uglify Minify/uglify JavaScript (without RequireJS)

Just add them to your plugins.sbt via addSbtPlugin("com.typesafe.sbt" % sbt-<plugin>" % "1.0.x"). Asset pipeline plugins also have to be added to pipelineStages in your build.sbt:

pipelineStages := Seq(rjs, digest, gzip)

Summary

sbt-web gives us flexible and powerful asset handling in Play or any other sbt project. It is now easier than before to integrate modern tooling without resorting to seperate build systems for front-end concerns. The community now depends less on the framework’s developers’ time. Since it is easier to contribute plugins, and sbt-web can also be used in other frameworks, the community can potentially become larger.

In the upcoming posts we’ll look at understanding sbt and sbt-web on a deeper level, writing our own plug-ins, and using the JavaScript engine.


  1. The JavaScript drinking game goes as follows: Enter an arbitrary noun or verb into Google, append "JS", and if there’s a JavaScript framework with that name, you have to drink.

  2. Quoted directly from the sbt-web GitHub page.

  3. Actually mappings of base directory to relative path, but we’ll cover that in the next post in more detail.

  4. For performance reasons it is recommended to use Node. Java 8’s implementation is also very fast, but not yet supported.

Posted in Scala & Playframwork

Leave a Reply

Your email address will not be published. Required fields are marked *

*

Time limit is exhausted. Please reload CAPTCHA.