Eliding Actor Messages

One of my Scala applications is a motion control system that I use to move around a robotic arm. I had a minor timing issue which I handled in a generic way using an additional actor.

Contents

• The Situation
• The Problem
• The Solution
• Implementation - Basics
• Implementation - Application
• Implementation - Manager

The Situation

The user tells the application to move the arm to the desired position. The application breaks up the motion into little steps and sends them off to the robot arm controller at a steady pace so that the arm moves at a uniform speed to the target position. The arm controller sends back motion events as the arm moves. Those motion events are fed into a simulation of the robotic arm, and a 3D image of the arm is rendered to the screen so that you can watch the image of the robotic arm move as the real arm moves.

The motion events are fired (published) by a ListenerManager to each listener, one of which is the simulation of the arm. Since the ListenerManager is running in the thread of the controller, the listener should spend minimal time processing the event. The listener thus simply sends the motion event as a message to an actor that handles all of the processing to update the simulated arm and do the rendering. The listener is actually just a closure, a function that takes one argument which is a motion event (BotPartMovedEvent) and returns nothing (Unit). Given the controller instance in the variable bot, setting up the simulation listener in the above architecture looks something like this:

    bot.addListener((ev:BotPartMovedEvent)=> renderWorker ! ev)

renderWorker is an actor that might look something like this:

    private val renderWorker = new Actor {
        def act() {
            loop {
                react {
                    case ev:BotPartMovedEvent => updateAndRender(ev)
                        //updateAndRender does the real rendering work
                }
            }
        }
    }

If the events are coming in at a slow enough rate, the simple solution described above is sufficient. I can hook these pieces together and have an architecture that looks something like this:

• The user issues a command to move to a particular point, which sends a message to the actor that processes the command.
• That actor uses a timer to send regularly spaced commands to the robot controller to move in small steps.
• The controller sends back motion events as it moves.
• The listener (registered by the simulator on the controller) puts each motion event into a message and sends it to the actor that handles the rendering.
• The rendering actor updates the simulation model and renders it to the screen.

But this doesn't work.

The Problem

In order to move the arm as smoothly as possible, the application sends commands to the robotic arm controller every 10ms. The arm controller sends back motion events when it moves, so it is sending back events at about 10ms intervals as well. The simulator listens to those events, so it knows that every 10ms the 3D scene has changed.

But rendering the 3D image takes a lot longer than 10ms; more on the order of 100ms. I can't just do the rendering every time I get a motion event, or I would quickly build up a large backlog of motion events and the simulated image would lag far behind reality.

I need to slow down the rendering so that I am not attempting to render every 10ms. In other words, I have to ignore most of those motion events, or at least not use most of them to trigger rendering.

The Solution

Fortunately, I don't really need to render the screen every 10ms, I can render at a slower rate. In this case, I chose to render every 250ms. That rate, 4 frames per second, is fast enough to provide reasonable visual matching with the real arm, but slow enough so as not to use up an inordinate amount of processing time.

I don't want to clutter my motion event publishing or my render processing with a bunch of code to handle timing, and I don't want to worry about motion events piling up in the inbox of the rendering actor, so I factored the timing concerns out into a separate actor with just one task: throwing away most of the requests. I can do this because I don't really need the information in each motion event, as I can directly query the controller to get its position information.

With this approach my publisher (the controller) can continue to publish all motion events, and my rendering thread doesn't have to make any decisions about whether or not to render, it simply renders every time it gets an event. The new actor decides which of the motion events get discarded and which get sent on to the renderer.

The last step in the basic architecture above (where the listener sends the message to the actor handling the rendering) is thus replaced by two steps:

• The listener (registered by the simulator on the controller) puts the motion event into a message and sends it to the actor that handles the timing issues.
• The timing actor decides when to forward a message and when to discard it, and forwards the selected messages to the actor doing the rendering.

Implementation - Basics

We start with the assumption that our worker actor that is doing the real work (the rendering actor in the example above) accepts messages of a particular type, and we help the developer of that actor remember that assumption by defining a trait (ActionWorker) that must be implemented by the worker actor in order for the timing actor (the manager) to recognize it:

trait ActionWorker[T] extends Actor

Our timing actor (ActionManager) is an Actor that is parameterized by the same type and accepts a constructor argument that is the actor to be controlled:

class ActionManager[T](worker:ActionWorker[T]) extends Actor ...

The ActionManager class is described in more detail below.

Lastly we have four message types:

• ActionRequest wraps the real message and is sent to the ActionManager.
• ActionCommand is sent from the ActionManager to the ActionWorker.
• ActionCompleted is sent from the ActionWorker back to the ActionManager when the worker has completed its task.
• ActionTimerExpired is used as part of the timeout mechanism that determines which messages to forward to the worker.

abstract case class ActionMessage

/** Clients send us an action request when they want the worker
 * to do something for them.
 * @param request The message to be passed to our worker.
 * @param delay Wait at least this long after the previous request
 *        before executing this request.
 */
case class ActionRequest[T](request:T, delayMillis:long) extends ActionMessage {
    def this(request:T) = this(request, 0)
}

/** The manager sends the worker an ActionCommand
 * to tell it to work on something. */
case class ActionCommand[T](manager:ActionManager[T],request:T)
    extends ActionMessage

/** The worker sends an ActionComplete to the manager when it is done with a
 * command and is ready to receive another command. */
case class ActionCompleted[T](request:T) extends ActionMessage

/** The manager sends itself an ActionTimerExpired
 * when its timer time is up. */
case class ActionTimerExpired() extends ActionMessage

You can think of the ActionMessage classes as being a wrapper for the original event class, and you can think of the ActionManager class as being a wrapper for the original worker actor (the renderer), as each wrapper takes as a constructor argument an instance of the wrapped object. Where before our listener sent the published event (BotPartMovedEvent) on to the worker actor (renderWorker), now we send a wrapped event (BotPartMovedEvent in an ActionRequest) to our wrapped actor (an ActionWorker in a ActionManager).

Implementation - Application

We need to specify somewhere the desired delay between the events that we pass along to the action worker. I chose to include this in the ActionRequest rather than in the ActionManager. Here is the updated code to register the listener, with changes in bold.

    val renderDelayMillis = 250
    bot.addListener((ev:BotPartMovedEvent)=>
             renderManager ! ActionRequest(ev,renderDelayMillis))

where renderManager is defined like this:

    private val renderManager = new ActionManager[BotPartMovedEvent](renderActor)

We modify our renderWorker actor to deal with our new classes:

    private val renderActor = new ActionWorker[BotPartMovedEvent] {
        def act() {
            loop {
                react {
                    case ActionCommand(mgr,ev) => updateAndRender(ev)
                        //updateAndRender does the real rendering work
                }
            }
        }
    }

We have a couple of other changes we will be making to renderActor below.

With these classes, here is the general flow for the motion events:

1. The controller generates a motion event.
2. The listener wraps the motion event in an ActionRequest and sends it to the ActionManager.
3. If the ActionManager decides to forward the event, it sends the original motion event in an ActionCommand to the ActionWorker.
4. When the ActionWorker is done with its task, it sends an ActionComplete message back to the ActionManager.

To satisfy the above flow, we modify our worker actor to send an ActionCompleted message back to the ActionManager when we are done with our task:

    private val renderActor = new ActionWorker[BotPartMovedEvent] {
        def act() {
            loop {
                react {
                    case ActionCommand(mgr,ev) =>
                        updateAndRender(ev)
                        mgr ! new ActionCompleted(ev)
                }
            }
        }
    }

There is one more change to make to the worker actor: make it run at a lower priority, so that the motion controller can have priority in order to keep the robot motion smooth. I did this by using receive instead of react, then explicitly changing the thread priority before entering the loop:

    private val renderActor = new ActionWorker[BotPartMovedEvent] {
        def act() {
            val th = Thread.currentThread
            th.setPriority(th.getPriority - 3)
            loop {
                receive {
                    case ActionCommand(mgr,ev) =>
                        updateAndRender(ev)
                        mgr ! new ActionCompleted(ev)
                }
            }
        }
    }

Implementation - Manager

Finally we get to the meat, the ActionManager that actually does the work of deciding which events to forward and which to throw away. The requirement given above of not forwarding a message more often than a specified interval was incomplete. That requirement is refined here and two other requirements are added:

• A message should not be forwarded to the worker until a specified minimum interval has passed since the previous message was forwarded.
• A message should not be forwarded to the worker while the worker is busy.
• The last message received while messages are not being forwarded should be forwarded to the worker at the next available time.

The above requirements ensure that the last message in any sequence of messages is always forwarded to the worker. If we receive a message that we can't immediately forward due to one of the previous two requirements, we hang on to it as the most recently received message until both of our restrictions are no longer true (i.e. the minimum interval has elapsed and the worker is idle). If we receive a message while waiting, we discard the previously saved most recent message and replace it with the newly received message. Whatever message we have when the minimum delay is up and the worker is no longer busy will be forwarded to the worker. The three pieces of state the manager uses are thus a timer, a worker busy flag, and the most recently received message.

With this approach, given a sequence of messages sent to the manager (where a sequence in this case is defined as a set of messages all of which are separated in time by less than either the minimum interval or the worker processing time, whichever is greater) we are guaranteed that, at a minimum, the first and last messages of the sequence will be forwarded to the worker. The only messages we will be discarding are messages in the middle of the sequence - thus we are eliding messages from the sequence.

Here is the basic structure of the manager class, including the declarations of the variables described above and the basic actor flow with the set of messages we recognize.

import java.util.Timer
import java.util.TimerTask

import scala.actors.Actor
import scala.actors.Actor.loop

class ActionManager[T](worker:ActionWorker[T]) extends Actor {
    //The next command to be sent to the worker; None when no request pending
    private var nextRequest:Option[T] = None

    //True when our worker is busy processing a command from us
    private var workerIsActive:boolean = false

    //The time that we last send on an action to our worker
    private var lastActionTime:long = 0
    private val timer = new Timer()
    private var timerTask:Option[TimerTask] = None

    def act() {
        loop {
            react {
                case ActionRequest(req,delay) => ...
                case ActionCompleted(req) => ...
                case ActionTimerExpired() => ...
            }
        }
    }
}

We know we will need to start and stop a timer, and that when the timer runs out we will need to send ourselves a signal. Here are the timer support methods that do that.

    private def stopTimer() {
        if (timerTask.isDefined) {
            timerTask.get.cancel()
            timerTask = None
        }
    }

    private val timerExpired = new ActionTimerExpired()
    private def startTimer(waitMillis:long) {
        stopTimer
        val task = new TimerTask() {
            override def run() { ActionManager.this ! timerExpired }
        }
        timer.schedule(task,waitMillis)
        timerTask = Some(task)
    }

When our timer expires, we will want to forward the next message to the worker if we have one; but if we don't have one, we will want to note that the worker is no longer busy and return to a quiescent state. Here are a few simple methods to handle that:

    private def sendNextOrDone() {
        if (nextRequest.isDefined)
            sendNextToTarget()
        else
            sequenceDone()
    }

    /** Send the pending request along to the target and update our vars. */
    private def sendNextToTarget() {
        val cmd = new ActionCommand(this,nextRequest.get)
        target ! cmd        //send the command to the target
        nextRequest = None
        targetIsActive = true
    }

    /** Here when the target has become inactive.
     * This is a good place to put in debugging hooks. */
    private def sequenceDone() {
        targetIsActive = false      //no longer active
    }

Now we can examine the cases in the main act loop.

The client sends an ActionRequest to the manager for each request, as implemented in our listener above. In our example, the req is a BotPartMovedEvent. The delay is the minimum time to wait since the previously forwarded message before forwarding this message. The manager saves only the most recent message, then decides whether to send on that message based on the two factors we discussed above (minimum interval and worker idle). If it has not been long enough since the last message was forwarded, we start a timer that will tell us when that time is up. If there was already a timer from the previous message, we stop that timer, so that we are always doing our calculations using the current delay parameter. Here's the code for that part of our manager:

    def act() {
        ...
            case ActionRequest(req,delay) =>
                //Save only the latest message.
                nextRequest = Some(req.asInstanceOf[T])

                //See how long it has been since the last message forwarded.
                val now = System.currentTimeMillis()
                val thisActionTime = lastActionTime + delay
                if (thisActionTime>now) {
                    //Need to wait a while before executing this action
                    startTimer(thisActionTime - now)  //wait this long
                } else {
                    //We are past the delay time for this action
                    if (!targetIsActive) {
                        //If the target is not active, we can send along
                        // the action request immediately.
                        sendNextToTarget()
                    }
                }
        ...
    }

When the worker finishes processing a message it sends an ActionComplete message back to the manager, which then sends on the next message if there is one and if the minimum time has elapsed.

    def act() {
        ...
            case ActionCompleted(req) =>
                targetIsActive = false
                if (timerTask.isEmpty) {
                    sendNextOrDone()
                }
        ...
    }

Finally, when the timer expires it sends the manager an ActionTimerExpired message. If the manager has a message to forward to the worker, and the worker is not busy, it forward it.

    def act() {
        ...
            case ActionTimerExpired() =>
                if (!targetIsActive) {
                    sendNextOrDone()
                }
                timerTask = None
        ...
    }

That's it. Put all the pieces together and you have an actor that you can slip into a stream of messages of any type, with just a couple of minor changes to the original message sender and receiver, to throttle the message stream by discarding messages down to a specified rate, and only send a message to the receiver when it is not busy processing a previous message.

My actual implementation includes a few other details, such as counters to see how many messages were received, discarded and forwarded, and some messages for the purpose of retrieving and printing those stats at the desired times, none of which make a significant impact on the above code.

Lift Without Maven

It is possible to run Lift without Maven. It's actually pretty easy. It's just not supported. I'll tell you how to do it.

• Background
• With Maven
• Without Maven
• Ant Build File

Background

You can tell from my blog posts that I like Scala. I have wanted to try out Liftweb for a while, but was turned off by the apparent requirement of using Maven. I figure that if I do like Lift, it will be enough work to convince my company that they should switch to Scala and Lift without also having to convince them at the same time that they should switch to Maven.

The members of the Lift community seem pleased with Maven and are not interested in supporting a non-Maven workflow. It's not that Lift requires Maven - in fact, David really wants people to know that Lift does not require Maven - but at the moment it's simply not a priority for them to support a non-Maven download. Although you can't really fault a newcomer for thinking that Maven is required, given that the Getting Started page currently states that Maven 2 is a "prerequisite". If you start asking about downloading and using Lift without Maven you tend to get a lot of responses about how great Maven is and how you should really try it. This is perhaps unfortunate, because many people, myself included, are not looking for a discussion about Maven and really want to minimize the number of new tools and workflow changes necessary just to try out a piece of software. Maven may be great, but that's not what I want to hear about right now. So I decided to figure out for myself just what it would take to download, compile and run Liftweb without Maven.

I started by using the standard Maven-based approach to get the files onto my machine, then hacked until I had what I wanted. You can do the same thing (it was actually pretty easy), or if you are having trouble with Maven or you simply don't want to touch it yet, you can use the directions I give below on how to download and set up the Liftweb demo without it. So although the semi-official answer (on the Lift wiki) to how to use Lift without Maven is that you at least need to use Maven to download the initial lift project, I'm here to tell you that you can download, compile and run the demo Lift app without using Maven at all.

With Maven

The download page on the Lift web site is interesting. There is a short paragraph for each of Mac, Windows and Linux downloads. The Mac and Windows paragraphs each include a link to a download package. The Linux paragraph just says, if you have trouble, ask someone on the mailing list.

I scratched my head a bit at the lack of a download link on the Download page, but finally went back to the main page, saw the link to the Getting Started page, opened the HTML document there, and started reading how to install Liftweb.

Step 1 is to run the mvn archetype:generate command, which downloads the Lift templates that Maven uses to build and update Lift. Unfortunately, those Lift templates require version 2.0.9 of Maven, and the newest version available as a standard package on my system (Fedora 9) is Maven 2.0.4. Rather than upgrading my OS, I went to the Maven website, downloaded the latest version of Maven (2.1.0), and installed it in an alternate location on my system. I then updated my $PATH to include that alternate location so that I would be using a new enough version of Maven to work on the Lift demo.

After downloading and installing the updated version of Maven, I was able to run the mvn archetype command listed on the Getting Started page. Since I had not used Maven before on this computer, it downloaded a bunch of files - about 5MB worth. The Getting Started page does say Maven will output "several page's worth of text", so this was not entirely a surprise.

Step 2 is to run the mvn jetty:run command. This command finishes building the demo Liftweb app and starts it using Jetty listening on port 8080. When I ran this second Maven command, it downloaded another big batch of files from the Internet. The Getting Started page says that after the first step, "you've now successfully created your first project", and it just says the second step "should produce more output", implying that there is not much work to be done in that second step. However, the second step took at least 5 times as long and downloaded another 30 MB of files; that was a bit of a surprise. In total Maven took about four minutes and downloaded 35MB of files - 710 files, including 82 jar files - into my new $HOME/.m2 directory (the Maven cache), including older versions of Scala and Ant than I already had on my machine. That's not really all that much time or space for a modern computer, but it was perhaps a bit more than I was expecting would be required for a "hello world" program. And in fact, as you will see below, only a small part of those files are necessary.

Unfortunately, after downloading half the Internet and finishing the build of the Liftweb demo, it failed to run on my system. Ah, but of course: I am already running Tomcat, which is using port 8080, the same port the Liftweb demo was trying to use. No problem, I thought, I will just change the port that the Liftweb demo uses. Unfortunately, my quick attempt was unsuccessful. I searched through all the files for 8080, found it in one place, changed it to 8181, then ran mvn jetty:run again. No luck - after crunching for about ten seconds it still tried to open port 8080. Among the many messages that Maven spit out was the line "Nothing to compile - all classes are up to date", so apparently it did not notice that I had edited one of the source files. Well, no big deal, I just stopped Tomcat for a while and tried it again, and I was able to access the demo web page at localhost:8080.

So getting the Liftweb demo running using the Maven approach was pretty easy:

1. Download and install a new version of Maven.
2. Stop Tomcat on my machine.
3. Run the two maven commands on the Liftweb Getting Started page.
4. Open my web browser to http://localhost:8080 and view the hello world page.

Without Maven

As you can see from the above, getting Lift downloaded and working with Maven was, for me, no problem. But what I really wanted to do was to build a war file that I could drop into my already-running Tomcat. I started with a simple ant script to compile HelloWorld.scala, and kept adding bits and pieces to it until I was able to build a working war file. Below are the details about the final setup I used to create a war file that worked under Tomcat.

I am using a grand total of 11 files:

• Two scala files: Boot.scala and HelloWorld.scala
• Two html files: index.html and templates-hidden/default.html
• Two xml files: web.xml and build.xml
• Five jar files: lift-util.jar, lift-webkit.jar, scala-library.jar, log4j.jar, and commons-fileupload.jar

Of the above files, both scala files, both html files, web.xml and the two lift jar files came unmodified out of the Liftweb demo, the Scala library came from the standard Scala download, the other two jar files were standard downloads from Apache, and the one file I created myself was the ant build file build.xml.

My directory hierarchy looks like this:

liftdemo/
    build/              All generated files go here
    build.xml           Ant build file
    src/
        scala/
            bootstrap/liftweb/Boot.scala
            demo/helloworld/snippet/HelloWorld.scala
        webapp/
            index.html
            web.xml
            templates-hidden/default.html

The five library jar files are stored elsewhere in my system and are copied in to the build directory when the war file is being created. You could create a lib directory in the liftdemo directory and drop the libraries in there if you wanted to. If you used Maven to download Lift, you can pull all of these libraries from your $HOME/.m2 Maven cache, else you can download them from their internet homes if you don't already have them somewhere on your system.

The war file liftdemo.war looks like this:

META-INF/MANIFEST.MF
WEB-INF/
    web.xml
    classes/
        bootstrap/liftweb/
            Boot$$anonfun$1.class
            Boot.class
        demo/helloworld/snippet/HelloWorld.class
    lib/
        commons-fileupload-1.2.1.jar
        lift-util-1.0.jar
        lift-webkit-1.0.jar
        log4j-1.2.14.jar
        scala-library.jar
index.html
templates-hidden/default.html

You can download files directly from the Liftweb Maven repository without using Maven: use these direct download links for the demo sources packed in a jar file, lift-util-1.0.jar, and lift-webkit-1.0.jar. (Note that these are all version 1.0 files, so if you are reading this article well past the posting date there are probably newer versions - poke around the repository to find out what the latest version is.) Unpack the demo source jar to a temporary location, then copy the five files mentioned above into the appropriate places in the hierarchy. Download scala, log4j and commons-fileupload (if you haven't already), drop in a buld.xml file (see below), and modify the properties to fit your environment. At that point you should be able to run a simple ant command to create a war file in the build directory. Copy that war file into the webapps directory, and you should be able to see the demo page in the standard way from your web browser.

After that, you are on your own. Obviously this very simple setup does not handle dependencies like Maven does, so when those dependencies get updated, you will have to manually manage that process. And when you build up your code and start using other features, you will have to manually add additional jar files to build.xml. But for some of us, that's just what we want.

The size of the src directory listed above is under 60KB, the two liftweb jar files are under 3MB together, and the scala and other two libraries are, combined, just over 4MB. The war file weighs in at 6.3MB. The original liftweb demo download includes some tests that I did not include in my war file, so it could be that the tests use a few more of those 82 jar files that Maven downloaded.

The build file below is pretty simple. There are really only two commands necessary to build the webapp: first compile the two scala files, then build the war file. I hope this post has made it clear that you don't need Maven at all to use Lift, and that Lift webapps are satisfactorily small and simple to build and run.

Ant Build File

Here is my sample buid.xml file for building the Liftweb demo without Maven. Remember to review the properties and change them as appropriate for your environment. In particular, you will almost certainly want to change the home.dir property, and probably the jar file locations as well. If you create a lib directory in your liftdemo directory and put the five libraries there, you can simplify the build file by getting rid of home.dir and a number of the properties used to define the locations of the libraries.

<project name="liftdemo" default="war">

    <target name="init" description="Initialize all properties and paths">

 <property name="home.dir" value="/u/jimmc"/>

 <property name="build.dir" value="build"/>
 <property name="class.dir" value="${build.dir}/class"/>
 <property name="scala.src.dir" value="src/scala"/>
 <property name="war.build.dir" value="${build.dir}/war"/>
 <property name="war.lib.dir" value="${war.build.dir}/WEB-INF/lib"/>
 <property name="war.file" value="build/liftdemo.war"/>
 <property name="web.xml" value="src/webapp/web.xml"/>

 <!-- Location of scala files -->
 <property name="scala.dir"
  value="${home.dir}/net/scala/scala-current"/>
 <property name="scala-compiler.jar"
  value="${scala.dir}/lib/scala-compiler.jar"/>
 <property name="scala-library.jar"
  value="${scala.dir}/lib/scala-library.jar"/>

 <!-- Location of Liftweb files -->
 <property name="liftweb.dir" value="${home.dir}/net/liftweb/1.0"/>
 <property name="lift-util.jar"
  value="${liftweb.dir}/lift-util-1.0.jar"/>
 <property name="lift-webkit.jar"
  value="${liftweb.dir}/lift-webkit-1.0.jar"/>

 <!-- A couple of other libraries are required -->
 <property name="log4j.jar"
  value="${home.dir}/net/log4j/log4j-1.2.14.jar"/>
 <property name="fileupload.jar"
       value="${home.dir}/net/fileupload/commons-fileupload-1.2.1.jar"/>

 <path id="scala.classpath">
     <pathelement location="${scala-compiler.jar}"/>
     <pathelement location="${scala-library.jar}"/>
 </path>

 <path id="compile.classpath">
     <path refid="scala.classpath"/>
     <pathelement location="${lift-util.jar}"/>
     <pathelement location="${lift-webkit.jar}"/>
 </path>

 <!-- define the "scalac" ant task -->
 <taskdef resource="scala/tools/ant/antlib.xml">
     <classpath refid="scala.classpath"/>
 </taskdef>
    </target>

    <target name="clean" depends="init">
 <delete dir="${build.dir}"/>
    </target>

    <target name="mkdirs" depends="init">
 <mkdir dir="${build.dir}"/>
 <mkdir dir="${class.dir}"/>
    </target>

    <target name="compile" depends="init,mkdirs"
     description="Compile scala files">
 <scalac
     srcdir="${scala.src.dir}"
     destdir="${class.dir}"
     addparams="-g:vars"
     classpathref="compile.classpath"
     force="changed"
     deprecation="on"
 >
     <include name="**/*.scala"/>
 </scalac>
    </target>

    <target name="war" depends="init,compile">
 <!-- First make an image directory in war.build.dir -->
 <copy todir="${war.build.dir}">
     <fileset dir="src/webapp" includes="**/*.html"/>
 </copy>
 <mkdir dir="${war.build.dir}/WEB-INF"/>

 <copy todir="${war.lib.dir}" file="${scala-library.jar}"/>
 <copy todir="${war.lib.dir}" file="${lift-util.jar}"/>
 <copy todir="${war.lib.dir}" file="${lift-webkit.jar}"/>
 <copy todir="${war.lib.dir}" file="${log4j.jar}"/>
 <copy todir="${war.lib.dir}" file="${fileupload.jar}"/>

 <copy todir="${war.build.dir}/WEB-INF/classes">
     <fileset dir="${class.dir}" includes="**/*.class"/>
 </copy>

 <!-- Put everything from our war directory into a war file -->
 <war
     destfile="${war.file}"
     webxml="${web.xml}"
 >
     <fileset dir="${war.build.dir}" includes="**/*"/>
 </war>
    </target>

</project>

Scala Functions vs Methods

Scala has both functions and methods. Most of the time we can ignore this distinction, but sometimes we have to deal with the fact that they are not quite the same thing.

In my Scala Syntax Primer I mention that I use the terms method and function interchangeably in the discussion. This is a simplification. In many situations, you can ignore the difference between functions and methods and just think of them as the same thing, but occasionally you may run into a situation in which the difference matters. This is analogous to how most of us treat mass and weight. In our daily lives on the surface of planet Earth, we treat them as interchangeable units, with 1Kg being the same as 2.2 pounds. But they are not quite the same thing: when an astronaut walks on the surface of the moon, his mass (kilograms) has not changed but his weight (pounds) is only about one sixth of what it was on Earth.

In contrast to Kg vs pounds, you are rather more likely to come across a situation in which the distinction between Scala functions and methods is important than you are to be walking on the surface of the moon. So when can you ignore the difference between functions and methods, and when do you need to pay attention to it? You can answer that question once you understand the difference.

A Scala method, as in Java, is a part of a class. It has a name, a signature, optionally some annotations, and some bytecode.

A function in Scala is a complete object. There are a series of traits in Scala to represent functions with various numbers of arguments: Function0, Function1, Function2, etc. As an instance of a class that implements one of these traits, a function object has methods. One of these methods is the apply method, which contains the code that implements the body of the function. Scala has special "apply" syntax: if you write a symbol name followed by an argument list in parentheses (or just a pair of parentheses for an empty argument list), Scala converts that into a call to the apply method for the named object. When we create a variable whose value is a function object and we then reference that variable followed by parentheses, that gets converted into a call to the apply method of the function object.

When we treat a method as a function, such as by assigning it to a variable, Scala actually creates a function object whose apply method calls the original method, and that is the object that gets assigned to the variable. Functions consume more memory than their functionally equivalent methods because they include not only the code that implements the function, but all of the surrounding code for the accompanying methods and class structure. Thus you would not want every method to be a function; but functions give you a great deal of power that is not available with just methods, and in those situations that power is well worth the additional memory used.

Let's look at some details of this mechanism. Create a file test.scala with this in it:

class test {
    def m1(x:Int) = x+3
    val f1 = (x:Int) => x+3
}

Compile that file with scalac and list the resulting files. Scala creates two files: test.class and test$$anonfun$1.class. That strange extra class file is the anonymous class for the function object that Scala created in response to the function expression assigned to f1. If you use more than one function expression in your test class, there will be more than one anonymous class file, even if you write the same function expression over again.

If you run javap on the test class, you will see this:

Compiled from "test.scala"
public class test extends java.lang.Object implements scala.ScalaObject{
    public test();
    public scala.Function1 f1();
    public int m1(int);
    public int $tag()       throws java.rmi.RemoteException;
}

Running javap on the function class test$$anonfun$1 yields this:

Compiled from "test.scala"
public final class test$$anonfun$1 extends java.lang.Object implements scala.Function1,scala.ScalaObject{
    public test$$anonfun$1(test);
    public final java.lang.Object apply(java.lang.Object);
    public final int apply(int);
    public int $tag()       throws java.rmi.RemoteException;
    public scala.Function1 andThen(scala.Function1);
    public scala.Function1 compose(scala.Function1);
    public java.lang.String toString();
}

Because this class implements the Function1 interface we know it is a function of one argument. You can see that it contains a handful of methods, including the apply method.

You can also define a function in terms of an existing method by referencing that method name followed by a space and an underscore. Modify test.scala to add another line that does this:

class test {
    def m1(x:Int) = x+3
    val f1 = (x:Int) => x+3
    val f2 = m1 _
}

The m1 _ syntax tells Scala to treat m1 as a function rather than taking the value generated by a call to that method. Alternatively, you can explicitly declare the type of f2, in which case you don't need to include the trailing underscore:

    val f2 : (Int) => Int = m1

In general, if Scala expects a function type, you can pass it a method name and have it automatically converted to a function. For example, if you are calling a method that accepts a function as one of its parameters, you can supply as that argument a method of the appropriate signature without having to include the trailing underscore.

Back to our test file, now when you compile test.scala there will be two anonymous class files, one for the f1 class and one for the f2 class. You can use either definition for f2, they generate identical class files.

If you use the -c option to javap to get the code of the second anonymous class, you can see the call to the m1 method of the test class in the apply method:

public final int apply(int);
  Code:
   0:   aload_0
   1:   getfield        #17; //Field $outer:Ltest;
   4:   astore_2
   5:   aload_0
   6:   getfield        #17; //Field $outer:Ltest;
   9:   iload_1
   10:  invokevirtual   #51; //Method test.m1:(I)I
   13:  ireturn

Let's fire up the scala interpreter and see how this works. In the following examples, input text is shown in bold and output text in regular weight.

scala> def m1(x:Int) = x+3
m1: (Int)Int

scala> val f1 = (x:Int) => x+3
f1: (Int) => Int = <function>

scala> val f2 = m1 _
f2: (Int) => Int = <function>

scala> m1(2)
res0: Int = 5

scala> f1(2)
res1: Int = 5

scala> f2(2)
res2: Int = 5

Note the difference in the signatures between m1 and f1: the signature (Int)Int is for a method that takes one Int argument and returns an Int value, whereas the signature (Int) => Int is for a function that takes one Int argument and returns an Int value.

At this point we seem to have a method m1 and two functions f1 and f2 that all do the same thing. But f1 and f2 are actually variables that contain an instance of a generated class that implements the Function1 interface, and that object instance has methods that m1 does not have.

scala> f1.toString
res3: java.lang.String = <function>

scala> (f1 andThen f2)(2)
res4: Int = 8

Because m1 is itself a method, unlike f1 you can't call methods on it:

scala> m1.toString
<console>:6: error: missing arguments for method m1 in object $iw;
follow this method with `_' if you want to treat it as a partially applied function
       m1.toString
       ^

Note that each time you separately reference a method as a function Scala will create a separate object.

scala> val f3 = m1 _
f3: (Int) => Int = <function>

scala> f2 == f3
res6: Boolean = false

Even though f2 and f3 both refer to m1, and both do the same thing, they are not considered equal by Scala because function objects inherit the default equality method that compares identity, and these are two different objects. If you want two function values to be equal, you must ensure that they refer to the same instance of function object:

scala> val f4 = f2
f4: (Int) => Int = <function>

scala> f2 == f4
res7: Boolean = true

Here are a few other examples showing that the expression m1 _ is in fact a function object:

scala> m1 _
res8: (Int) => Int = <function>

scala> (m1 _).toString
res9: java.lang.String = <function>

scala> (m1 _).apply(3)
res10: Int = 6

As of Scala version 2.8.0, the expression (m1 _)(3) will also return the same value (there is a bug in previous versions that causes this syntax to give a type mismatch error).

There are some other differences between methods and functions. A method can be type-parameterized, but an anonymous function can not:

scala> def m2[T](x:T) = x.toString.substring(0,4)
m2: [T](T)java.lang.String

scala> m2("abcdefg")
res11: java.lang.String = abcd

scala> m2(1234567)
res12: java.lang.String = 1234

However, if you are willing to define an explicit class for your function, then you can type-parameterize it similarly:

scala> class myfunc[T] extends Function1[T,String] {
     |     def apply(x:T) = x.toString.substring(0,4)
     | }
defined class myfunc

scala> val f5 = new myfunc[String]
f5: myfunc[String] = <function>

scala> f5("abcdefg")
res13: java.lang.String = abcd

scala> val f6 = new myfunc[Int]
f6: myfunc[Int] = <function>

scala> f6(1234567)
res14: java.lang.String = 1234

So go ahead and keep converting pounds to kilograms by dividing by 2.2 (unless you are an astronaut), but when you start mixing functions and methods in Scala, keep in mind that they are not quite the same thing.

Fun With Feedback

When I was in high school I participated in an interesting exercise to demonstrate the importance of feedback. You might want to try this out as a party game. It is easy to laugh at others when they do poorly at this seemingly simple exercise, but it is in fact a rather difficult task.

Get a set of children's building blocks, such as a set of hardwood unit blocks, with a variety of shapes and sizes. From the original set of blocks, create two identical sets. Each set should have at least ten blocks in it with at least four different shapes. More variety is better. Having blocks within each set that are similar but not identical, such as two different kinds of triangles, makes it more difficult. Given these two identical sets, anything you can build with one set can be built with the other set.

Arrange two desks or tables, with their chairs, for two people such that there is a level work space in front of each person in which that person can build, and each person can hear the other, but neither can see the other person or the other blocks. On each of the two work spaces, place one of the two sets of blocks you created above. The two work spaces should be as close to identical as is practical to ensure that any structure built on one work space can be built on the other work space. One possible arrangement is to place a large vertical separator in the middle of a table such that it separates the two ends of the table, then have each person sit on an opposite end of the table facing each other. The separator must be large enough to prevent the two people from seeing each other (and stable enough not to fall over). This allows onlookers to stand on the sides of the table and easily see both of the builders and constructions.

Select two test subjects and sit them down in front of their respective work spaces and blocks. Designate one as the master and the other as the apprentice (or one as the speaker and the other as the listener, if you prefer). The master builds a structure with his blocks (any structure of his choice containing a least a designated minimum number of blocks), and instructs the apprentice as to how to build exactly the same structure. The apprentice is not allowed to ask questions or provide any other kind of feedback to the master. The apprentice must build the same structure as the master based solely on the verbal description given by the master. The master, in turn, must provide his verbal description of his structure without any knowledge of what the apprentice is doing.

It turns out that this is surprisingly difficult to do well. In most cases, the apprentice's construction will diverge from the master's construction after only a few blocks, and by the time the master is placing his tenth block and describing it to the apprentice, the apprentice is completely lost and unable to place the block in any location remotely resembling the master's description. At some point along the way, the apprentice becomes aware that something is wrong, but can't do anything about it - and the master, blithely unaware of the apprentice's difficulty, inexorably continues to build and describe something that the apprentice no longer understands. An alert master may be able to tell from the reactions of the viewers that the apprentice is having a problem, but without any feedback will not be able to know quite what the problem is, so will be unable to correct it and will have to forge on.

I watched a number of other pairs who attempted this exercise fail to build structures of any significant size, and I was determined to do a better job. When it was my turn to be master, I very carefully described each block I selected, how I oriented it, and how I placed it next to other previously placed blocks. I was describing everything at an excruciating level of detail, far more than I would in normal circumstances. And, as I learned at the end of my turn, for a while this was working quite well, and my apprentice was able to construct significantly more than had anyone else.

But there was a snag: at one point I selected a small triangular block, which happened to be a right equilateral triangle. When placing it, I instructed my apprentice to orient the triangle by pointing the corner which was a right angle in a certain direction. Unfortunately, math was not a strong point for my apprentice, and for a while she was unable to recall what a right angle was. As a consequence, she did not understand what I meant concerning the orientation of the block, but because I was unaware of her problem and continued my building and describing, she had to guess about how to place the block, and she guessed wrong. Of course, errors cascaded from there because further steps which relied on the placement of that block could not be done properly.

Had she been able to ask me a couple of simple questions, I could easily and quickly have clarified for her what I meant. Without any feedback at all, the failure to understand one small detail meant the game was over.

You can try the same exercise, but this time let the apprentice ask questions. It is amazing how much difference it makes. Without feedback, almost nobody can succeed on this task, and when they do it requires an extraordinary amount of redundancy in the master's descriptions. With verbal feedback, most people can do quite a good job, and much faster than without feedback, since the master has the liberty to be much sloppier in his initial description, confident in the knowledge that the apprentice will ask for more details on any part he does not understand. With visual feedback, virtually every apprentice can quite speedily reproduce the master's design. The speed and accuracy of communication improves dramatically as the feedback bandwidth increases.

I highly recommend this exercise for teaching the value of precise communication (especially when there is no or little feedback available), the power of a high-bandwidth feedback channel (especially when there is any complexity involved) and the recognition that some tasks are far harder than they appear (especially when you have not directly attempted the task yourself).

Scala Listener Manager

Java uses the Listener pattern, a version of Publish-Subscribe, to decouple the processing of events from their generation. This is a nice pattern, but I was annoyed by the amount of boilerplate I had to add to my Java code every time I had a publishing class that needed to manage listeners.

As a subscriber, the Java code for adding a listener is not bad: just call the publisher's addListener method, typically passing an anonymous inner class that defines the callback method. But on the publisher side, you have to implement addListener, removeListener and fireEvent methods for the published event.

In Scala, all of that publisher boilerplate to manage listeners can be replaced by three words. Sweet! And the inner class boilerplate in the subscriber disappears also.

I implemented a Scala trait called ListenerManager to be used by a publisher. This trait defines the methods to add and remove listeners, and to fire off the published events:

trait ListenerManager[E] {
    type L = (E) => Unit
    private var listeners: List[L] = Nil
    private var listenersLock = new Object()

    /** True if the listener is already in our list. */
    private def isListener(listener:L) = listeners.exists(_==listener)

    /** Add a listener to our list if it is not already there. */
    def addListener(listener:L) = listenersLock.synchronized {
        if (!isListener(listener))
            listeners = listener :: listeners
    }

    /** Remove a listener from our list.  If not in the list, ignored. */
    def removeListener(listener:L):Unit = listenersLock.synchronized {
        listeners = listeners.filter(_!=listener)
    }

    /** Apply the given closure to all of the listeners in the list. */
    protected def fireEvent(event:E) = listenersLock.synchronized {
        listeners.foreach(_.apply(event))
    }
}

To add the ability to a class to publish an event, just add with ListenerManager[MyEvent] (these are the three words mentioned above) to the defining class line for that class. This will define the addListener and removeListener methods for subscribers to call, and the fireEvent method for the publisher to call.

Let's take a look at the above code in more detail.

The type [E] in the ListenerManager trait declaration is the type of the event to be published. The publisher class would typically define a one-line case class for that event type that lists the data of interest. For example:

case class ButtonEvent(source:Button, action:Integer)

The first line of the body of ListenerManager defines the type of the listeners we are managing, which is a function that takes an instance of our event and does not return a value. In the subscriber, the code to add a listener passes in a function of this type, which can be specified in-line as a closure:

    publisher.addListener((x:ButtonEvent)=>(println("Button info:"+x)))

The call to println could be replaced by a call to a local method that does more complicated processing of the event.

Next we define a list of listeners and the add and remove methods to manipulate that list. Our private isListener method, used to ensure that a listener does not get into the list more than once, is only used in one place so could have been done in-line, but has been written separately to add a bit of clarity to the code.

Last is the fireEvent method that publishes the event. For each listener in our list, it calls that listener's apply method, which is how to invoke a function when given it as an object.

The addListener, removeListener and fireEvent methods are synchronized in case the application is using separate threads for the publisher and each subscriber. We can't lock on the listeners list because we recreate that each time a listener is added and removed, and to minimize locking contention we would rather not lock on the publishing object, so we add another object (listenersLock) just for locking the methods that access the listener list, and we synchronize on that object.

Synchronizing the fireEvent method means that any subscriber calling addListener or removeListener will block until all of the callbacks are done. We could choose not to synchronize fireEvent to prevent that blocking, and that method will still work because once it picks up the listeners list and starts iterating through it, it will continue to use that list even if someone else calls addListener or removeListener. However, doing so would mean it would be possible for a listener to be called after the subscriber has called removeListener, which is probably more of an issue than blocking when adding or removing a listener while callbacks are happening. We will assume that listeners will follow the standard guideline that they should keep processing during the callback to a minimum, and that it they need to do a lot of work based on a callback, they will implement a mechanism to do that work in another thread so as not to use up a lot of time on the publisher's thread.

For the subscriber, using ListenerManager saves the boilerplate of an inner class. For the publisher, using ListenerManager saves a lot more, making it possible to set up publication just by defining a case class for the published event, adding the with declaration to the class definition, and adding calls to the fireEvent method at publication points. Pretty easy.

I could have implemented the ListenerManager trait to take an instance of a callback class with an action method, as is done in Java. Or, to take more of a Scala approach, I could have switched from using the Listener pattern as is used in Java to an Actor-based Publish/Subscribe implementation. For this particular case I chose a simple path down the middle, keeping the basic Java Listener concept but eliminating the boilerplate.