Understanding sbt and sbt-web settings

Inside sbt-web: Part II – Understanding sbt and sbt-web’s settings

Just enough sbt

If we want to talk about sbt-web, we must first discuess a few concepts related to sbt itself1. sbt is the Scala Build Tool, often critized for being overly complex. It however enables a lot of interesting use cases, so one might argue its not incidental but domain-related complexity2.

Let’s first look at the abstractions sbt provides and then go on to understand how sbt-web builds on them.

Builds

A build definition corresponds to the project you’re building. Each build can consist of multiple sub-builds, with one being the root project, or root build. Each project contains a top-level build file, build.sbt, that corresponds to the root build. .sbt files allow a subset of Scala syntax, and are compiled by sbt to Scala files. Alternatively you can write Scala build files directly (all build-related Scala files go into the project folder), but with the recent additions to sbt, this is rarely needed.

If you don’t provide a build file, sbt will infer a standard build for you.

A typical project might look like this:

my-project/
  build.sbt             < -- root build file
  project/                <-- definitions for all sub-projects
    build.properties      <-- define sbt version
    plugins.sbt           <-- sbt plugins
    Common.scala          <-- common build code
  my-domain-model      <-- sub-project
    build.sbt             <-- sub-project build file
    src/main/scala/...    <-- sources
  my-play-web-ui
    build.sbt
    app
    conf
  my-play-admin-web-ui
    build.sbt
    app
    conf

You will always run sbt in the root project, never in one of the sub-projects. In order to switch to a sub-project, enter project <project-name>. You also have the option of running a task directly inside a sub-project via <SubProject>/<task>, but depending on the task it might just run in your current project3.

~ $ cd /.../my-project
my-project $ sbt
[info] Loading project definition from /.../my-project/project
[info] Set current project to my-project (in build file://.../my-project/)
&gt; projects
[info] In file: /.../my-project/
[info]    my-domain-model
[info]    my-play-web-ui
[info]    my-play-admin-web-ui
[info]  * my-project
&gt; project my-play-web-ui
[my-play-web-ui] $

Settings

A build file is essentially a collection of key-value pairs with its basic type Setting[T], the key being a String and the value being T. In contrast to most other build systems, sbt differentiates between setting definition and assignment (or initialization in sbt lingo).

We define a setting by defining a key:

val myProjectDir = SettingKey[java.io.File]("my-project-dir", "Project Directory")
// shorthand macro that infers the key from the val's name
val myOtherProjectDir = settingKey[java.io.File]("Project Directory")

my-project-dir is the setting’s key. By defining it in hyphen-style, you can refer to it on the console via hyphen-style or camelCase. The shorthand macro only supports camelCase.

As mentioned above, this only declares a setting key. It doesn’t have a value yet. You set it to a concrete value by using the Pascal-like assignment operator that returns the setting.

myProjectDir := new java.io.File(".")

This can be set in a top build, and then overriden by sub-builds. Technically each build is immutable, so assigning a value means creating a copy of the build with the changed value.

Tasks

Tasks are just like settings, except while settings are set once, tasks are recomputed each time they are executed. They have the type Setting[Task[T]], and their keys are declared as follows:

val clearBuildDir = TaskKey[Unit]("clear-build-dir", "Delete Build directory")
// or macro
val clearBuildDir = taskKey[Unit]("Delete Build directory")

Typing to Unit emphasizes that a task is side-effecting, however many tasks are typed to values, for example the directory that was cleared by the above task.

val clearBuildDir = taskKey[java.io.File]("Delete Build directory")

clearBuildDir := {
  val buildDirectory = ...
  buildDirectory.files.delete()
  buildDirectory
}

Setting and Task Hierarchies

Settings and tasks can refer to other settings via their .value property4. From these calls, sbt traces the hierarchy of settings. Here’s an example of how sbt-web defines the location of the staging directory:

// sbt default
val target := "target"

// sbt-web
val webTarget := target.value / "web"

val stagingDirectory := webTarget.value / "stage"

You can access a setting’s value only in assignments.

Scopes

Settings can be scoped. By scoping, you can assign different values to the same setting depending on context. You can scope by project, by configuration, and by task. These are called the three scope axes.

Scoping by project simply refers to whether the setting applies to the root build, a sub-build, or the entire build.

// Setting a version for an entire build so you don't have to define it in each sub-build
version in ThisBuild := "1.0.1"

Configurations are just containers. For example, you differentiate between sources for compilation and for testing, but both are compiled in the same way. So we have a single setting, sourceDirectory5, scoped to the configurations Compile and Test.

baseDirectory := "."

sourceDirectory := baseDirectory.value / "src"

sourceDirectory in Compile := baseDirectory.value / sourceDirectory.value / "main"

sourceDirectory in Test := baseDirectory.value / sourceDirectory.value / "test"

Scoping by task sets a setting’s value only for that given task.

// Scope to task
outputFile in packageSrc := target.value / name.value + "-" + version.value + "_src.zip"
outputFile in packageBin := target.value / name.value + "-" + version.value + ".jar"
// Scope to configuration and task
outputFile in (Debian, packageBin) := target.value / name.value + "-" + version.value + ".deb"

Instead of defining your own settings, first check if sbt already offers a similiar setting, and reuse that setting by scoping it. Compile and Test are provided by sbt, but feel free to create your own configuration to avoid conflicts. The default configuration is Global.

Inspecting settings

When you write a build file, you want to inspect the settings you wrote. Start sbt on the command line in your project directory, and use the show command. Refer to scoped settings via <configuration?>:<task?>::<setting>.

$ sbt
> show sourceDirectory
[info] /Users/UserName/MyProject/src
> show compile:sourceDirectory
[info] /Users/UserName/MyProject/src/main
> show test:sourceDirectory
[info] /Users/UserName/MyProject/src/test

Keep in mind that when referring to settings or scopes, you don’t use the val’s name, but its key (except if you use the macro, which sbt-web doesn’t).

Some settings are not defined in the Global configuration, for example compile, but we can still call sbt compile instead of sbt compile:compile. This is because sbt looks for a key in Global, Compile, and Test (in that order, unless you override configurations in a project). This becomes clearer when using the inspect command, which lists a setting’s dependencies, the fallback search path (Delegates) and other scopes it is defined in (the Related section).

> inspect compile
[info] Task: sbt.inc.Analysis
[info] Description:
[info]  Compiles sources.
[info] Defined at:
[info]  (sbt.Defaults) Defaults.scala:247
[info] Dependencies:
...
[info]  compile:compile::compileInputs
...
[info] Delegates:
[info]  compile:compile
[info]  *:compile
...
[info] Related:
[info]  web-assets:compile
[info]  web-assets-test:compile
[info]  test:compile

The output is exactly the same when inspecting compile:compile. When inspecting test:compile, we can see that all its dependencies are scoped to test as well.

> inspect test:compile
...
[info] Dependencies:
...
[info]  test:compile::compileInputs
...

Plugins

Finally, if we want to reuse build code across projects, we can pack it into sbt plugins. Plugins can define their own settings that are added to a build, and also assign default values.

We’ll take a closer look at plugins in the next blog post when we write an sbt-web plugin!

sbt-web settings

sbt-web project structure keys

Let’s apply this newly acquired knowledge to understand how sbt-web is organized. If you want to follow along in the source, open SbtWeb.scala. sbt-web uses a common convention to store its settings in a companion object called Import (see com.typesafe.sbt.web.Import).

As mentioned in part 1, sbt-web adds the notion of assets to sbt. This is done via the configurations Assets (key web-assets), and TestAssets (key web-assets-test)6.

The only top-level setting is pipelineStages, these are the steps that process assets in the asset pipeline. We’ll take a closer look at this setting the next blog post.

The remaining settings and tasks are defined in WebKeys. Most of the settings define the web project layout. We cannot infer from the names of the settings if they are input or output settings, so here’s an overview.

Source settings (used by source tasks) are simply sbt settings scoped to Assets and TestAssets.

+ src
--+ main
----+ assets .....sourceDirectory in Assets
----+ public .....resourceDirectory in Assets
--+ test
----+ assets .....sourceDirectory in TestAssets
----+ public .....resourceDirectory in TestAssets

We have seen the target folder already, but it actually contains a lot more. The distinction between public, assets-managed, and resources-managed is blurry. Source tasks would typically output to resourceManaged in Assets, whereas pipelines could output to resourceManaged in Assets or public in Assets.

Also of note are the module folders for resolved NodeJS and WebJars modules.

+ target
--+ web ............webTarget
----+ public
------+ main ...........public in Assets
------+ test ...........public in TestAssets
----+ assets-managed
------+ main ...........sourceManaged in Assets
------+ test ...........sourceManaged in TestAssets
----+ resources-managed
------+ main ...........resourceManaged in Assets
------+ test ...........resourceManaged in TestAssets
----+ node-modules
------+ main ...........nodeModuleDirectory in Assets
------+ test ...........nodeModuleDirectory in TestAssets
----+ web-modules
------+ main ...........webModuleDirectory in Assets
------+ test ...........webModuleDirectory in TestAssets
----+ stage  .........stagingDirectory

Every target setting depends on webTarget.

sbt-web tasks

Remember that tasks are a special category of settings used for computed values and side-effects.

  • assets copies all assets from input to output. It also executes source tasks and copies resolved WebJars. Its value becomes public.
  • pipeline (key web-pipeline) runs the asset pipeline.
  • stage runs all sbt-web tasks to produce the final, deployment-ready output. Its value becomes stagingDirectory.

You might wonder why pipeline is typed to PathMapping, that is a mapping from a base directory to a relative path. When sbt-web reads from an input, it takes the base directory, for example src/main/assets, and the relative paths, for example /js/main.js. When producing an output, the base directory will be changed to an output directory, and the relative path will stay the same.

There are more tasks, but they are mostly for internal use, for example to resolve WebJars, deduplicate mappings, and more.

Interacting with the console

To play around with sbt-web keys, add sbt-web to your plugins.

// plugins.sbt
addSbtPlugin("com.typesafe.sbt" % "sbt-web" % "1.0.2")

// build.sbt
lazy val root = project.in(file(".")).enablePlugins(SbtWeb)

Include some assets in your project, run sbt, and try the sbt-web tasks. Alternatively you can check out this screencast where I set up a small example project.

Summary

We now have a better understanding of sbt and sbt-web’s organization. This will help us write our first asset pipeline stage in the next post.


  1. The same goes for Play IMHO. If you want to use Play, you have to master sbt.

  2. Or put differently, simpler build tools often stay simple when you follow their conventions to the point, but require workarounds when your build requirements grow more complex.

  3. Here’s an example of a multi project build, incorporating both sbt sub-builds and Play sub-modules: https://github.com/mariussoutier/play-multi-multi-project

  4. Since settings are static and tasks are dynamic, settings cannot refer to tasks.

  5. Or sources to refer to all files in sourceDirectory.

  6. There is also the Plugin configuration which is internal to sbt-web.

Posted in Scala & Playframwork
One comment on “Understanding sbt and sbt-web settings
  1. lls says:

    Hi! Thanks for the great summary on sbt.
    I think there might be a type in “Setting and Task Hierarchies”.
    Instead of

    // sbt default
    val target := “target”

    // sbt-web
    val webTarget := target.value / “web”

    val stagingDirectory := webTarget.value / “stage”

    shouldn’t it rather say

    // sbt default
    target := “target”

    // sbt-web
    webTarget := target.value / “web”

    stagingDirectory := webTarget.value / “stage”

Leave a Reply

Your email address will not be published. Required fields are marked *

*

Time limit is exhausted. Please reload CAPTCHA.