When should I move from scala-cli to sbt?

When a project needs several modules, a published artifact, or build logic beyond running and testing. scala-cli handles scripts, single-file tools and prototypes well; sbt is the standard for libraries and services, and `scala-cli --power export` can generate the initial build for you.

What does %% mean in an sbt dependency?

It appends the Scala binary version to the artifact name, so `org %% "lib" % "1.0"` resolves `lib_3` on Scala 3. A single `%` means the artifact has no Scala version — use it for pure Java libraries.

What is an assembly JAR and when do I need one?

A single JAR containing your code and all its dependencies, built by sbt-assembly. You need it when the runtime cannot resolve dependencies for you — `spark-submit`, a bare `java -jar`, or a container with no build tool.

Why mark Spark dependencies as Provided?

Because the cluster already has them. Bundling Spark makes the JAR hundreds of megabytes and risks a version conflict with the cluster's own copy. `Provided` puts them on the compile and test classpath but leaves them out of the assembly.

sbt, Packaging, and Shipping a JAR

Everything so far ran with scala-cli run. That is the right tool for a file; a service or a library needs a build definition, dependency management across modules, and an artifact someone else can run.

From scala-cli to sbt

scala-cli --power export --sbt --output bookshop .

Exporting to sbt project in bookshop
Exported to: /home/you/bookshop

Or start from scratch. The layout sbt expects:

bookshop/
├── build.sbt
├── project/
│   ├── build.properties        sbt version
│   └── plugins.sbt             build plugins
└── src/
    ├── main/scala/             production code
    ├── main/resources/         config, bundled into the JAR
    └── test/scala/             tests

// build.sbt
ThisBuild / scalaVersion := "3.6.2"
ThisBuild / organization := "com.example"
ThisBuild / version      := "0.1.0-SNAPSHOT"

lazy val root = (project in file("."))
  .settings(
    name := "bookshop",
    libraryDependencies ++= Seq(
      "com.lihaoyi"   %% "upickle" % "4.0.2",
      "org.scalameta" %% "munit"   % "1.0.3" % Test,
    ),
    scalacOptions ++= Seq("-deprecation", "-feature", "-Wunused:all", "-Werror"),
  )

// project/build.properties
sbt.version=1.10.7

sbt compile

[info] welcome to sbt 1.10.7 (Eclipse Adoptium Java 21.0.5)
[info] loading project definition from /home/you/bookshop/project
[info] loading settings for project root from build.sbt ...
[info] compiling 3 Scala sources to /home/you/bookshop/target/scala-3.6.2/classes ...
[success] Total time: 4 s, completed 9 Sep 2026, 16:02:11

%% appends the Scala binary version — upickle_3 here. A single % is for Java libraries, which have no Scala version. % Test puts a dependency on the test classpath only.

-Werror is worth turning on early. Combined with -Wunused:all and the exhaustivity warnings from lesson 3, it turns a category of latent bug into a build failure.

The commands you will actually use

sbt test

[info] compiling 1 Scala source to .../test-classes ...
ShippingSuite:
  + light parcels cost the base rate 0.021s
  + heavy standard parcels ship free 0.001s
[info] Passed: Total 4, Failed 0, Errors 0, Passed 4
[success] Total time: 3 s

sbt

sbt:bookshop> ~testQuick
[info] Compiling 1 Scala source ...
[info] Passed: Total 4, Failed 0, Errors 0, Passed 4
1. Waiting for source changes in project bookshop... (press enter to interrupt)

The interactive shell is the difference between sbt being slow and being fast: one JVM startup instead of one per command. ~ re-runs on file change, and testQuick runs only the tests affected by what changed.

sbt "runMain com.example.bookshop.Ingest --date 2026-01-04"
sbt dependencyTree
sbt "show libraryDependencies"
sbt clean

Multiple modules

// build.sbt
ThisBuild / scalaVersion := "3.6.2"

lazy val core = (project in file("core"))
  .settings(
    name := "bookshop-core",
    libraryDependencies += "com.lihaoyi" %% "upickle" % "4.0.2",
  )

lazy val ingest = (project in file("ingest"))
  .dependsOn(core)
  .settings(
    name := "bookshop-ingest",
    libraryDependencies += "org.apache.spark" %% "spark-sql" % "3.5.4" % Provided,
  )

lazy val root = (project in file("."))
  .aggregate(core, ingest)
  .settings(publish / skip := true)

sbt compile

[info] compiling 4 Scala sources to /home/you/bookshop/core/target/scala-3.6.2/classes ...
[info] compiling 2 Scala sources to /home/you/bookshop/ingest/target/scala-3.6.2/classes ...
[success] Total time: 7 s

dependsOn makes core a compile dependency of ingest; aggregate makes a command on root run in both. Modules keep a Spark job’s dependencies out of a library that does not need them — which matters, because Spark drags in a large and opinionated classpath.

Building a runnable JAR

// project/plugins.sbt
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "2.3.1")

lazy val ingest = (project in file("ingest"))
  .dependsOn(core)
  .settings(
    libraryDependencies += "org.apache.spark" %% "spark-sql" % "3.5.4" % Provided,
    assembly / mainClass := Some("com.example.bookshop.Ingest"),
    assembly / assemblyJarName := "bookshop-ingest.jar",
    assembly / assemblyMergeStrategy := {
      case PathList("META-INF", "services", _*) => MergeStrategy.concat
      case PathList("META-INF", _*)             => MergeStrategy.discard
      case "reference.conf"                     => MergeStrategy.concat
      case _                                    => MergeStrategy.first
    },
  )

sbt ingest/assembly

[info] Strategy 'concat' was applied to 2 files
[info] Strategy 'discard' was applied to 184 files
[info] Strategy 'first' was applied to 41 files (Run the task at debug level to see details)
[info] Built: /home/you/bookshop/ingest/target/scala-3.6.2/bookshop-ingest.jar
[success] Total time: 22 s

ls -lh ingest/target/scala-3.6.2/bookshop-ingest.jar

-rw-r--r-- 1 you you 8.4M Sep  9 16:14 bookshop-ingest.jar

8.4 MB, because Spark is Provided and excluded. Remove % Provided and the same JAR is over 200 MB and risks a version clash with the cluster’s own Spark.

The merge strategy matters more than it looks. Two dependencies both shipping META-INF/services/... need concat, or a service loader silently finds only one implementation — the usual cause of “works locally, no such driver on the cluster”.

Running it on Spark

// ingest/src/main/scala/com/example/bookshop/Ingest.scala
package com.example.bookshop

import org.apache.spark.sql.SparkSession

object Ingest:
  def main(args: Array[String]): Unit =
    val date = args.sliding(2).collectFirst { case Array("--date", d) => d }
      .getOrElse(sys.error("--date is required"))

    val spark = SparkSession.builder().appName("bookshop-ingest").getOrCreate()
    try
      import spark.implicits.*
      val orders = spark.read.option("header", "true").csv(s"s3://bookshop/landing/$date/")
      val cleaned = orders
        .filter($"status" =!= "pending")
        .withColumn("amount", $"amount".cast("decimal(10,2)"))

      println(s"writing ${cleaned.count()} rows for $date")
      cleaned.write.mode("append").saveAsTable("bookshop.silver.orders")
    finally spark.stop()

spark-submit \
  --class com.example.bookshop.Ingest \
  --master yarn --deploy-mode cluster \
  ingest/target/scala-3.6.2/bookshop-ingest.jar --date 2026-01-04

25/09/09 16:20:02 INFO Client: Submitting application application_1789459200000_0042
25/09/09 16:20:14 INFO Client: Application report for application_1789459200000_0042 (state: RUNNING)
writing 9788 rows for 2026-01-04
25/09/09 16:22:41 INFO Client: Application report for application_1789459200000_0042 (state: FINISHED)
	 final status: SUCCEEDED

The try/finally around spark.stop() is not decoration — without it a failure leaves the application hanging until the cluster times it out.

Cross-building

ThisBuild / crossScalaVersions := Seq("2.13.15", "3.6.2")

sbt +test

[info] Setting Scala version to 2.13.15 ...
[info] Passed: Total 4, Failed 0, Errors 0, Passed 4
[info] Setting Scala version to 3.6.2 ...
[info] Passed: Total 4, Failed 0, Errors 0, Passed 4
[success] Total time: 31 s

The + prefix runs a task for every cross version. Libraries need this; applications generally do not. Note that Spark 3.5 is published for Scala 2.13, so a Scala 3 Spark job relies on binary compatibility between the two — usually fine, occasionally the explanation for a strange NoSuchMethodError.

Publishing

ThisBuild / organization := "com.example"
ThisBuild / licenses := Seq("MIT" -> url("https://opensource.org/licenses/MIT"))
ThisBuild / homepage := Some(url("https://github.com/example/bookshop"))

publishTo := Some("releases" at "https://nexus.internal/repository/maven-releases/")
credentials += Credentials(Path.userHome / ".sbt" / ".credentials")

sbt publish

[info] published bookshop-core_3 to https://nexus.internal/.../bookshop-core_3/0.1.0/bookshop-core_3-0.1.0.pom
[info] published bookshop-core_3 to https://nexus.internal/.../bookshop-core_3-0.1.0.jar
[success] Total time: 6 s

publishLocal puts the artifact in ~/.ivy2/local for another project on the same machine to resolve — the fastest way to test a library change end to end without a release.

Keeping scala-cli around

sbt for the project does not mean sbt for everything:

scala-cli run scripts/backfill.scala -- --from 2026-01-01 --to 2026-01-31

Compiling project (Scala 3.6.2, JVM (21))
backfilling 2026-01-01 .. 2026-01-31 (31 days)
done

scala-cli package scripts/report.scala -o report --assembly
./report

Wrote /home/you/bookshop/report, run it with
  ./report

One-off scripts, migrations and ad-hoc reports are faster to write and run this way, and they still get the compiler and the same dependencies.

Practice

1. Add a dependency and check where it came from.

sbt "whatDependsOn com.fasterxml.jackson.core jackson-databind 2.17.2"

[info] com.fasterxml.jackson.core:jackson-databind:2.17.2
[info]   +-org.apache.spark:spark-core_2.13:3.5.4
[info]     +-com.example:bookshop-ingest_3:0.1.0-SNAPSHOT

Transitive dependencies are where version conflicts come from. This is the first command to run when an assembly JAR misbehaves at runtime but compiles cleanly.

2. Build an assembly with and without Provided.

-rw-r--r-- 1 you you 8.4M bookshop-ingest.jar     # spark Provided
-rw-r--r-- 1 you you 241M bookshop-ingest.jar     # spark bundled

241 MB uploaded to the cluster on every submit, plus a real chance of a classpath conflict with the cluster’s own Spark. Provided is not an optimisation, it is the correct setting.

3. Use ~testQuick while editing.

1. Waiting for source changes in project bookshop... (press enter to interrupt)
[info] Compiling 1 Scala source ...
[info] Passed: Total 2, Failed 0, Errors 0, Passed 2

Only the tests touching the changed code ran. In a warm sbt shell the loop is a second or two, against ten for sbt test from a cold start each time.

4. Omit a merge strategy and build an assembly.

[error] deduplicate: different file contents found in the following:
[error] /home/you/.cache/coursier/.../jackson-core-2.17.2.jar:META-INF/versions/9/module-info.class
[error] /home/you/.cache/coursier/.../jackson-databind-2.17.2.jar:META-INF/versions/9/module-info.class
[error] (assembly) deduplicate: different file contents found

Two JARs disagreeing about one file. Discarding META-INF fixes it — but concat the services entries first, or a ServiceLoader will silently see only one implementation at runtime.

That closes the Scala track. The thread through all ten lessons: the compiler is the tool — immutable values, exhaustive matches, types that distinguish an order id from a customer id, and givens resolved before the program ever runs.