Binary websockets with Play 2.0 and Scala

In a recent article I showed how you can use webrtc, canvas and websockets together to create a face detection application whose frontend runs completely in the browser, without the need for plugins. In that article I used a Jetty based backend to handle the image analysis using OpenCV through the JavaCV wrapper.

When I almost finished the article, I noticed that websockets is also supported from Play 2.0. I really like developping in Play and in Scala so as an experiment I rewrote the backend part from a Jetty/Java/JavaCV stack to a Play2.0/Scala/JavaCV stack. If you want to do this for yourself, make sure you start with the frontend code from here. Since the frontend code hasn’t changed except the location where the websockets are listening.

Setting up the Play 2.0 environment

I’m not going to talk too much about how to start a Play 2.0/Scala project. you can find the details in some of my other posts should you need more information. What we do need to do, is setup the dependencies for JavaCV so that they can be used from Play 2. I’ve manually added them to my local ivy repository, so that I can reference them from the sbt configuration like any other dependency. For my example I created the following directory layout for the JavaCV libraries:

./play-2.0-RC2/repository/cache/javacv
./play-2.0-RC2/repository/cache/javacv/javacpp
./play-2.0-RC2/repository/cache/javacv/javacpp/ivy-2.3.1.xml
./play-2.0-RC2/repository/cache/javacv/javacpp/ivydata-2.3.1.properties
./play-2.0-RC2/repository/cache/javacv/javacv
./play-2.0-RC2/repository/cache/javacv/javacv/ivy-2.3.1.xml
./play-2.0-RC2/repository/cache/javacv/javacv/ivydata-2.3.1.properties
./play-2.0-RC2/repository/cache/javacv/javacv-macosx-x86_64
./play-2.0-RC2/repository/cache/javacv/javacv-macosx-x86_64/ivy-2.3.1.xml
./play-2.0-RC2/repository/cache/javacv/javacv-macosx-x86_64/ivydata-2.3.1.properties
./play-2.0-RC2/repository/local/javacv
./play-2.0-RC2/repository/local/javacv/javacpp
./play-2.0-RC2/repository/local/javacv/javacpp/2.3.1
./play-2.0-RC2/repository/local/javacv/javacpp/2.3.1/ivys
./play-2.0-RC2/repository/local/javacv/javacpp/2.3.1/ivys/ivy.xml
./play-2.0-RC2/repository/local/javacv/javacpp/2.3.1/jars
./play-2.0-RC2/repository/local/javacv/javacpp/2.3.1/jars/javacpp.jar
./play-2.0-RC2/repository/local/javacv/javacv
./play-2.0-RC2/repository/local/javacv/javacv/2.3.1
./play-2.0-RC2/repository/local/javacv/javacv/2.3.1/ivys
./play-2.0-RC2/repository/local/javacv/javacv/2.3.1/ivys/ivy.xml
./play-2.0-RC2/repository/local/javacv/javacv/2.3.1/jars
./play-2.0-RC2/repository/local/javacv/javacv/2.3.1/jars/javacv.jar
./play-2.0-RC2/repository/local/javacv/javacv-macosx-x86_64
./play-2.0-RC2/repository/local/javacv/javacv-macosx-x86_64/2.3.1
./play-2.0-RC2/repository/local/javacv/javacv-macosx-x86_64/2.3.1/ivys
./play-2.0-RC2/repository/local/javacv/javacv-macosx-x86_64/2.3.1/ivys/ivy.xml
./play-2.0-RC2/repository/local/javacv/javacv-macosx-x86_64/2.3.1/jars
./play-2.0-RC2/repository/local/javacv/javacv-macosx-x86_64/2.3.1/jars/javacv-macosx-x86_64.jar

As you can see from this listing, I just added the three javacv supplied jars to my local repository. I also added a minimal ivy.xml so that they can be used from ivy and sbt. This minimal ivy.xml looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<ivy-module version="2.0" xmlns:m="http://ant.apache.org/ivy/maven">
    <info
        organisation="javacv"
        module="javacv-macosx-x86_64"
        revision="2.3.1"
        status="release">
        </info>

        <publications>
            <artifact type="jar"/>
        </publications>
</ivy-module>

With these files added to my repository I can setup the dependencies for the Play 2.0 project in the Build.scala file.

import sbt._
import Keys._
import PlayProject._

object ApplicationBuild extends Build {

    val appName         = "PlayWebsocketJavaCV"
    val appVersion      = "1.0-SNAPSHOT"

    val appDependencies = Seq(
      "javacv" % "javacv" % "2.3.1",
      "javacv" % "javacpp" % "2.3.1",
      "javacv" % "javacv-macosx-x86_64" % "2.3.1"
    )

    val main = PlayProject(appName, appVersion, appDependencies, mainLang = SCALA).settings(
      // Add your own project settings here      
    )
}

Now run “play update” and “play eclipsify” to update the dependencies and your Eclipse configuation (if you’re using Eclipse that is).

Configure websockets in Play

Using websockets in Play 2.0 is very easy. The first thing you need to do is add the URL to your routes configuration in the conf directory.

GET     /wsrequest                  controllers.Application.wsrequest

And, of course, you need to implement the action this route points to:

/**
 * Simple websocket listener configured in play 2. This uses a synchronous model, where
 * the same channel is used to send the response. For this usecase this is useful, if
 * we want async processing we could have used Akka actors together with play 2.0 async
 * support.
 */
def wsrequest = WebSocket.using[Array[Byte]] { request =>

  // Create the outbound value that is called for each
  val out = Enumerator.imperative[Array[Byte]]();

 val in = Iteratee.foreach[Array[Byte]](content => {
    out.push(FaceDetect.detect(content));
  })

  // tie the in and out values to each other
  (in, out)
}

In this code we configure an input channel (in), and an output channel (out) and connect them to the socket. Whenever the HTML5 client sends a request over the websocket our “in” method is called, and when we want to send something to the client we can use the “out” channel. The “in” channel needs to be defined as an Iteratee (more info see these Play docs). What this does is, that for each input message we receive we run the specifici method. In this case we run the FaceDetect.detect operation (more on this later) and the result from this operation is pushed back to the client using the “out” channel. This “out” channel itself is defined as an Enumerator (see these play docs). We can attach different listeners if we want to this enumerator, but in this case we don’t do anything with the message, just pass it along to the client.

Using JavaCV from scala

The last step is the code of the FaceDetect.detect function. The java version, see earlier mentioned article, is very easily converted to a scala one.

package javacv

import com.googlecode.javacv.cpp.opencv_core._
import com.googlecode.javacv.cpp.opencv_imgproc._
import com.googlecode.javacv.cpp.opencv_highgui._
import com.googlecode.javacv.cpp.opencv_objdetect._
import com.googlecode.javacpp.BytePointer
import java.nio.ByteBuffer
import javax.imageio.ImageIO
import java.io.ByteArrayOutputStream
import scala.tools.nsc.io.VirtualFile

object FaceDetect {

 var CASCADE_FILE =PATH_TO_CASCADE_FILE;
   var minsize = 20;
 var group = 0;
 var scale = 1.1;

  def detect(imageData:Array[Byte]) : Array[Byte] = {

    // we need to wrap the input array, since BytePointer doesn't accept
    // a bytearray as input. It accepts a byte varargs, but Array[Byte]
    // doesn't convert automatically
    var wrappedData = ByteBuffer.wrap(imageData);
    var originalImage = cvDecodeImage(cvMat(1, imageData.length,CV_8UC1, new BytePointer(wrappedData)));

    // convert to grayscale for easy detection
    var grayImage = IplImage.create(originalImage.width(), originalImage.height(), IPL_DEPTH_8U, 1);
    cvCvtColor(originalImage, grayImage, CV_BGR2GRAY);

    // storage is needed to store information during detection
 var storage = CvMemStorage.create();

 // load and run the cascade
 var cascade = new CvHaarClassifierCascade(cvLoad(CASCADE_FILE));
 var faces = cvHaarDetectObjects(grayImage, cascade, storage, scale, group, minsize);

 // draw a rectangle for the detected faces
 for (i <- 0 until faces.total) {
   var r = new CvRect(cvGetSeqElem(faces, i));
   cvRectangle(originalImage, cvPoint(r.x, r.y),
     cvPoint(r.x + r.width(), r.y + r.height),
     CvScalar.YELLOW, 1, CV_AA, 0);
 }

 // convert to bytearray and return
 var bout = new ByteArrayOutputStream();
 var imgb = originalImage.getBufferedImage();
 ImageIO.write(imgb, "png", bout);

    bout.toByteArray()
  }
}

The only issue I ran into was with the BytePointer constructor. One of the signatures accepts a varargs of the type byte. In java this allows me to just supply this constructor with a byte[], in Scala however this doesn’t work. Luckily though, a BytePointer can also be created using a ByteBuffer. For the rest this is a one-to-one conversion of Java to Scala.

Running the code

And that’s almost it. By default play listens on port 9000, in the Jetty based example we had the server running on 9999 and listening to the root context. To work with our Play2/Scala based server we just need to point the browser to the correct websocket server url.

 ws = new WebSocket("ws://127.0.0.1:9000/wsrequest");

And now, when we run it, we use Play 2 as our server and run the JavaCV code using scal. And more importantly it still works:

Reference: Binary websockets with Play 2.0 and Scala (and a bit op JavaCV/OpenCV) from our JCG partner Jos Dirksen at the Smart Java blog.