Working with Scala Futures

I recently finished writing a Twitter bot. It helps development teams using Rally (a proprietary agile planning tool) to celebrate their wins by tweeting about recently completed stories. It has a simple 3-stage process:

1. retrieve status from Rally
2. compare this status update with previous status update to see what's new
3. tweet about anything that is newly completed.

The main technologies used are: Spray HTTP Client, Typesafe config, json4s, scala-pickling and twitter4j. You can view the source of the bot, or see it in action @farnsworthtalks. The configuration is externalised, so you can run a variant of your own, if that's useful to you.

In this project I have used Futures throughout. In my experience, Futures become natural after a brief, initial period of confusion. Only after I reacquaint myself with them do things begin to flow. Proficiency and fluency with Futures is a good thing, because once you have a Future you need to keep modelling your application as Future operations, or suffer the penalty of blocking for a result. For me at least, I need to practice, practice and practice some more until they are second nature.

In my bot, the first step is to retrieve an HTTP payload. Using Spray HTTP Client this is returned in a Future. Once you have a Future you are propelled forward within this monadic context to find and merge in more and more Futures. Getting used to thinking in Futures can be difficult, but as I have found it is worthwhile as it imbues parallelism throughout the application code.

Retrieving a remote HTTP payload is seamless, and results in our first Future. This is mapped through json4s parsing logic that converts the String payload into workable case classes:

val byRecordType: Seq[Future[Set[Story]]] = Seq("story", "defect").map{recordType =>
  val svc = url(urlFor(recordType, teamName)).as(user, pass)
  val json: Future[JValue] = Http(svc OK as.String)
    .map(parse(_)).map{ _ \ "QueryResult" \ "Results" }
  json.map(_.children.map{result: JValue => result.extract[Story]}.toSet)
}

The result type is a temporally current collection with Futures within. In other words a Seq that exists right now, filled with Sets that will exist in the future. A common transformation on such a structure is to sequence these inner futures into a single future collection. This is an important act of simplification, as I am not interested in the collection until all its members are actualised. Whilst I have a Seq[Future[T]], what I really need is a Future[Seq[T]].

In the Coursera Principles of Reactive Programming course students are asked to write this Future warping logic themselves. It's a lot of fun, but thankfully the standard library provides the necessary generic functionality in the Future.sequence method.

With a set of Stories (that will be parsed in the future) in hand, the next step is to compare this to the stories that were found in previous runs. This is done using scala-pickling, using a synchronous API. As the application logic is already in the land of the future, it makes sense to explicitly wrap this synchronous operation in a Future, so that we can enjoy parallelism between this function and the ones that follow.

The unpickling is enabled simply with this code:

Future { Source.fromFile(storiesFile).mkString.unpickle[Map[String, Set[Story]]] }

Finding what stories are newly completed is a set complement operation on the values within the Future. Using a for-comprehension the complement can also be queued up to complete in the Future.

val newStories: FutureStories = for {
  all <- allStories
  fromBefore <- storiesFromLastRun
} yield all.map{ case (k, v) => (k, v -- fromBefore.getOrElse(k, Set.empty))}

Tweeting these new stories is trivial using twitter4j. (This blog post explains the process well). After ensuring the TwitterFactory is configured with the correct secret credentials, we need only call updateStatus. Again, this is synchronous, so it has been manually wrapped in a Future. This is not entirely necessary, as blocking the tweeting operations would simplify the exit condition for the process. However, I included it for what I believe is a necessary sense of aesthetic symmetry.

Future { twitter.updateStatus(message) }

At the end of the process, there are two values to be awaited before the process can terminate. Firstly the tweeting, and secondly the serialisation to file of the latest known list of completed stories. Either operation may finish before the other in theory, so both future results of these operations are sequenced to a single Future and the process awaits its completion (or blows the stack if something went wrong).

val finalValue = Future.sequence(Seq(storiesFile, statuses))
finalValue.onFailure{case t: Throwable => throw t}
Await.ready(finalValue, 1 minute)

The most complex aspect of working with futures was the situation where I needed to transform a Map[A, Future[B]] into a Future[Map[A, B]]. This is another take on the sequencing of a collection of futures, but there is no standard library method to make this easy for maps. The essence of the solution is to migrate the A into the Future, by transforming the Future[B] into a Future[(A,B)], discarding the original A and sequencing the resulting Iterable[Future[(A,B)]] into a Future[Iterable[(A,B)]], then mapping these future tuples into a Map. The transformation looks like this:

type A = Int; type B = Int
val m: Map[A, Future[B]] = Map(1 -> Future(2))
val iterFutureTuples: Iterable[Future[(A,B)]] = m.map{case (a,b) => b.map(bb => (a,bb))}
val futureIterTuples: Future[Iterable[(A,B)]] = Future.sequence(iterFutureTuples)
val futureMap: Future[Map[A,B]] = futureIterTuples.map(_.toMap)

Or more succinctly as:

Future.sequence(myMap.map{case (a,b) => b.map(bb => (a,bb))}).map(_.toMap)

In my experience the initial barrier to working with Futures is similar to that experience when first encountering Options and other monadic types. This feels more complex again because of the tendency to think in terms of time. But by putting aside temporal concerns and considering Future as just another monadic context, they begin to feel natural.

What is your experience programming using Scala Futures? Do you have any insights to share? I would appreciate any thoughts or feedback.