Protect a REST service using HMAC (Play 2.0)

When you talk about security for REST based APIs, people often point to HTTPS. With HTTPS you can easily protect your services from prying eyes using methods everybody is familiar with. When you, however, require an additional level of security, or HTTPS just isn’t available, you need an alternative. For instance you might need to track the usage of your API for each customer, or need to know exactly who is makking all these calls. You could use HTTPS together with client authentication, but that would require setting up a complete PKI Infrastructure and a secure way to identify your customers and exchange private keys. And in contrast with WS-Security for SOAP based service, there isn’t a standard we can use for REST.

A common way to solve this (Microsoft, Amazon, Google and Yahoo take this approach), is by signing your message based on a shared secret between the client and the service. Note that with this approach we only sign the data, we don’t encrypt it. The signature we’re talking about in this case, is something that is usually called a Hash-based Message Authentication Code (or HMAC for short). With an HMAC we create a message authentication code (MAC) for a request based on a secret key we’ve exchanged.

In this article I’ll show you how you can implement this algorithm for a Play 2.0 based REST service. If you use a different technology the steps will be pretty much the same way though.

HMAC Scenario

For the client part, I’ll just use a simple HTTPClient based application. To implement this we’ll have to take the following steps:

1. First, we need to exchange a shared secret with out client. Often this is sent by the API provider to the client using an email message, or the provider has a website where you can lookup the shared secret. Note that this secret is shared only between you and service, each client will have a different shared secret. This isn’t something you share like a public key, 
2. To make sure the client and the service calculate the signature over the same content, we need to normalize the request that is to be signed. If we don’t do this, the server might interpret whitespace in a different manner as the client did and conclude that the signature is invalid. 
3. Based on this normalized message, the client creates an HMAC value using the shared secret. 
4. Now the client is ready to sent the request to the service. He adds the HMAC value to the headers, and also something that identifies him as user. For instance a username or some other public value. 
5. When the service receives the request it extracts the username and the HMAC value from the headers.
6. Based on the username, the service knows which shared secret should have been used to sign the message. The service would, for instance, retrieve this from a datastore somewhere. 
7. The service now normalizes the request in the same manner as the client did, and calculates the HMAC value for itself. 
8. If the HMAC from the client matches the calculated HMAC from the server, you know that the message’s integrity is guaranteed, and that the client is who he says he is. If either the wrong username was supplied, or an incorrect secret was used to calculate the headers, the HMAC values wouldn’t match.

What do we need to do, to implement HMAC? In the following section we’ll at the following subjects.

• Determine the fields to use for input. 
• Create client code that can calculate this HMAC and add the corresponding headers 
• Create Play 2.0 based interceptor that checks the HMAC headers

Determine the input fields

The first thing we need to do is determine the input for our HMAC calculation. The following table describes the elements we’ll include:

What we’ll include is pretty much the following information from a request:

PUT /example/resource/1
Content-Md5: uf+Fg2jkrCZgzDcznsdwLg==
Content-Type: text/plain; charset=UTF-8
Date: Tue, 26 Apr 2011 19:59:03 CEST

Client code that can be used to create an HMAC signature

Below you can see the client code with which we’ll make the calls to the HMAC protected service. This is just a quick HTTPClient based client with which we can test our Service.

public class HMACClient {
 
 private final static String DATE_FORMAT = "EEE, d MMM yyyy HH:mm:ss z";
 private final static String HMAC_SHA1_ALGORITHM = "HmacSHA1";
 
 private final static String SECRET = "secretsecret";
 private final static String USERNAME = "jos";
 
 private static final Logger LOG = LoggerFactory.getLogger(HMACClient.class);
 
 public static void main(String[] args) throws HttpException, IOException, NoSuchAlgorithmException {
  HMACClient client = new HMACClient();
  client.makeHTTPCallUsingHMAC(USERNAME);
 }
 
 public void makeHTTPCallUsingHMAC(String username) throws HttpException, IOException, NoSuchAlgorithmException {
  String contentToEncode = "{\"comment\" : {\"message\":\"blaat\" , \"from\":\"blaat\" , \"commentFor\":123}}";
  String contentType = "application/vnd.geo.comment+json";
  //String contentType = "text/plain";
  String currentDate = new SimpleDateFormat(DATE_FORMAT).format(new Date());
 
  HttpPost post = new HttpPost("http://localhost:9000/resources/rest/geo/comment");
  StringEntity data = new StringEntity(contentToEncode,contentType,"UTF-8");
  post.setEntity(data);
 
  String verb = post.getMethod();
  String contentMd5 = calculateMD5(contentToEncode);
  String toSign = verb + "\n" + contentMd5 + "\n"
    + data.getContentType().getValue() + "\n" + currentDate + "\n"
    + post.getURI().getPath();
 
  String hmac = calculateHMAC(SECRET, toSign);
 
  post.addHeader("hmac", username + ":" + hmac);
  post.addHeader("Date", currentDate);
  post.addHeader("Content-Md5", contentMd5);
 
  HttpClient client = new DefaultHttpClient();
  HttpResponse response = client.execute(post);
 
  System.out.println("client response:" + response.getStatusLine().getStatusCode());
 }
 
 private String calculateHMAC(String secret, String data) {
  try {
   SecretKeySpec signingKey = new SecretKeySpec(secret.getBytes(), HMAC_SHA1_ALGORITHM);
   Mac mac = Mac.getInstance(HMAC_SHA1_ALGORITHM);
   mac.init(signingKey);
   byte[] rawHmac = mac.doFinal(data.getBytes());
   String result = new String(Base64.encodeBase64(rawHmac));
   return result;
  } catch (GeneralSecurityException e) {
   LOG.warn("Unexpected error while creating hash: " + e.getMessage(), e);
   throw new IllegalArgumentException();
  }
 }
 
 private String calculateMD5(String contentToEncode) throws NoSuchAlgorithmException {
  MessageDigest digest = MessageDigest.getInstance("MD5");
  digest.update(contentToEncode.getBytes());
  String result = new String(Base64.encodeBase64(digest.digest()));
  return result;
 }
}

And then use the HMAC algorithm to create a signature based on a shared secret.

private String calculateHMAC(String secret, String data) {
  try {
   SecretKeySpec signingKey = new SecretKeySpec(secret.getBytes(), HMAC_SHA1_ALGORITHM);
   Mac mac = Mac.getInstance(HMAC_SHA1_ALGORITHM);
   mac.init(signingKey);
   byte[] rawHmac = mac.doFinal(data.getBytes());
   String result = new String(Base64.encodeBase64(rawHmac));
   return result;
  } catch (GeneralSecurityException e) {
   LOG.warn("Unexpected error while creating hash: " + e.getMessage(), e);
   throw new IllegalArgumentException();
  }
 }

After we’ve calculcated the HMAC value, we need to send it to the server. We do this by providing a custom header:

post.addHeader("hmac", username + ":" + hmac);

As you can see, we also add our username. This is needed by the server to determine which secret to use to calculate the HMAC value on the server side. When we now run this code, a simple POST operation will be executed that sends the following request to the server:

POST /resources/rest/geo/comment HTTP/1.1[\r][\n]
hmac: jos:+9tn0CLfxXFbzPmbYwq/KYuUSUI=[\r][\n]
Date: Mon, 26 Mar 2012 21:34:33 CEST[\r][\n]
Content-Md5: r52FDQv6V2GHN4neZBvXLQ==[\r][\n]
Content-Length: 69[\r][\n]
Content-Type: application/vnd.geo.comment+json; charset=UTF-8[\r][\n]
Host: localhost:9000[\r][\n]
Connection: Keep-Alive[\r][\n]
User-Agent: Apache-HttpClient/4.1.3 (java 1.5)[\r][\n]
[\r][\n]
{"comment" : {"message":"blaat" , "from":"blaat" , "commentFor":123}}

Implementing in Scala / Play

So far we’ve seen what the client needs to do to provide us with the correct headers. Service providers often offer specific libraries, in multiple languages, that handle the details of signing the message. But as you can see, doing it by hand, isn’t that difficult. Now, let’s look at the server side, where we use scala together with the Play 2.0 framework to check whether the supplied header contains the correct information. For more information on setting up the correct scala environment to test this code look at my previous post on scala (http://www.smartjava.org/content/play-20-akka-rest-json-and-dependencies).

The first thing to do is setup the correct routes to support this POST operation. We do this in the conf/routes file

POST /resources/rest/geo/comment   controllers.Application.addComment

This is basic Play functionality. All POST calls to the /resource/rest/geo/comment URL will be passed on to the specified controller. Let’s look at what this operation looks like:

def addComment() = Authenticated {
    (user, request) => {
     // convert the supplied json to a comment object
     val comment = Json.parse(request.body.asInstanceOf[String]).as[Comment]
 
     // pass the comment object to a service for processing
     commentService.storeComment(comment)
     println(Json.toJson(comment))
        Status(201)
      }
  }

Now it gets a bit more complicated. As you can see in the listing above, we’ve defined an addComment operation. But, instead of directly defining an Action like this:

def processGetAllRequest() = Action {
    val result = service.processGetAllRequest;
    Ok(result).as("application/json");
  }

We, instead, define it like this:

def addComment() = Authenticated {
    (user, request) => {

What we do here is create a composite action http://www.playframework.org/documentation/2.0/ScalaActionsComposition). We can easily do this, since Scala is a functional language. The ‘Authenticated’ reference you see here is just a simple reference to a simple function, that takes another function as its argument. In the ‘Authenticated’ function we’ll check the HMAC signature. You can read this as using annotations, but now without the need for any special constructs. So, what does our HMAC check look like.

import play.api.mvc.Action
import play.api.Logger
import play.api.mvc.RequestHeader
import play.api.mvc.Request
import play.api.mvc.AnyContent
import play.api.mvc.Result
import controllers.Application._
import java.security.MessageDigest
import javax.crypto.spec.SecretKeySpec
import javax.crypto.Mac
import org.apache.commons.codec.binary.Base64
import play.api.mvc.RawBuffer
import play.api.mvc.Codec
 
/**
 * Obejct contains security actions that can be applied to a specific action called from
 * a controller.
 */
object SecurityActions {
 
  val HMAC_HEADER = "hmac"
  val CONTENT_TYPE_HEADER = "content-type"
  val DATE_HEADER = "Date"
 
  val MD5 = "MD5"
  val HMACSHA1 = "HmacSHA1"
 
  /**
   * Function authenticated is defined as a function that takes as parameter
   * a function. This function takes as argumens a user and a request. The authenticated
   * function itself, returns a result.
   *
   * This Authenticated function will extract information from the request and calculate
   * an HMAC value.
   *
   *
   */
  def Authenticated(f: (User, Request[Any]) => Result) = {
    // we parse this as tolerant text, since our content type
    // is application/vnd.geo.comment+json, which isn't picked
    // up by the default body parsers. Alternative would be
    // to parse the RawBuffer manually
    Action(parse.tolerantText) {
 
      request =>
        {
          // get the header we're working with
          val sendHmac = request.headers.get(HMAC_HEADER);
 
          // Check whether we've recevied an hmac header
          sendHmac match {
 
            // if we've got a value that looks like our header 
            case Some(x) if x.contains(":") && x.split(":").length == 2 => {
 
              // first part is username, second part is hash
              val headerParts = x.split(":");
              val userInfo = User.find(headerParts(0))
 
              // Retrieve all the headers we're going to use, we parse the complete 
              // content-type header, since our client also does this
              val input = List(
                request.method,
                calculateMD5(request.body),
                request.headers.get(CONTENT_TYPE_HEADER),
                request.headers.get(DATE_HEADER),
                request.path)
 
              // create the string that we'll have to sign
              val toSign = input.map(
                a => {
                  a match {
                    case None => ""
                    case a: Option[Any] => a.asInstanceOf[Option[Any]].get
                    case _ => a
                  }
                }).mkString("\n")
 
              // use the input to calculate the hmac
              val calculatedHMAC = calculateHMAC(userInfo.secret, toSign)
 
              // if the supplied value and the received values are equal
              // return the response from the delegate action, else return
              // unauthorized
              if (calculatedHMAC == headerParts(1)) {
                f(userinfo, request)
              } else {
                 Unauthorized
              }
            }
 
            // All the other possibilities return to 401 
            case _ => Unauthorized
          }
        }
    }
  }
 
  /**
   * Calculate the MD5 hash for the specified content
   */
  private def calculateMD5(content: String): String = {
    val digest = MessageDigest.getInstance(MD5)
    digest.update(content.getBytes())
    new String(Base64.encodeBase64(digest.digest()))
  }
 
  /**
   * Calculate the HMAC for the specified data and the supplied secret
   */
  private def calculateHMAC(secret: String, toEncode: String): String = {
    val signingKey = new SecretKeySpec(secret.getBytes(), HMACSHA1)
    val mac = Mac.getInstance(HMACSHA1)
    mac.init(signingKey)
    val rawHmac = mac.doFinal(toEncode.getBytes())
    new String(Base64.encodeBase64(rawHmac))
  }
}

That’s a lot of code, but most of it will be pretty easy to understand. The ‘calculateHMAC’ and the ‘calculateMD5’ methods are just basic scala wrappers around Java functionality. The documentation inside this class should be enough to understand what is happening. I do, however, want to highlight a couple of interesting concepts in this code. The first thing is the method signatue:

def Authenticated(f: (User, Request[Any]) => Result) = {

What this means is that the Authenticated method itself, takes as it’s arguments another method (or function if you want to call it that). If you look back at the target of our route, you can see that we do just that:

def addComment() = Authenticated {
    (user, request) => ...

Now what happens when this ‘Authenticated’ method is called? The first thing we do, is check whether the HMAC header exists and is in the correct format:

val sendHmac = request.headers.get(HMAC_HEADER);
  sendHmac match {
 
            // if we've got a value that looks like our header 
            case Some(x) if x.contains(":") && x.split(":").length == 2 => {
            ...
            }
 
            // All the other possibilities return to 401 
            case _ => Unauthorized

We do this by using a match against the HMAC header. If it contains a value that is of the correct format, we process the header and calculate the HMAC value in the same manner as our client did. If not we return a 401. If the HMAC value is correct we delegate to the provided function using this code:

if (calculatedHMAC == headerParts(1)) {
                f(userInfo, request)
              } else {
                 Unauthorized
              }

And that pretty much is it. With this code you can easily use an HMAC to check whether the message has changed in transit, and whether your client is really known to you. Very easy as you can see. Just a small sidenote on JSON usage from Play 2.0. If you look at the action code, you can see I use the standard JSON functionality:

def addComment() = Authenticated {
    (user, request) => {
     // convert the supplied json to a comment object
     val comment = Json.parse(request.body.asInstanceOf[String]).as[Comment]
 
     // pass the comment object to a service for processing
     commentService.storeComment(comment)
     println(Json.toJson(comment))
        Status(201)
      }
  }

First we parse the received JSON using ‘json.parse’ to a ‘comment’ class, then store the comment, and convert the command object back to a string value. Not the most useful code, but it does nicely demonstrate some of the JSON functionality provided by Play 2.0. To convert from JSON to an object and back again, something called “Implicit Conversions” is used. I won’t dive too much in the details, but a good explanation can be found here: http://www.codecommit.com/blog/ruby/implicit-conversions-more-powerful-t…. What happens here is that the JSON.parse and the Json.toJson method look for a specific method on the Comment class. And if it can’t find it there, it looks for the specific operation in its scope. To see how this works for the JSON parsing let’s look a the Comment class and its companion object:

import play.api.libs.json.Format
import play.api.libs.json.JsValue
import play.api.libs.json.JsObject
import play.api.libs.json.JsString
import play.api.libs.json.JsNumber
import play.api.libs.json.JsArray
 
object Comment {
 
  implicit object CommentFormat extends Format[Comment] {
 
    def reads(json: JsValue): Comment = {
      val root = (json \ "comment")
 
      Comment(
        (root \ "message").as[String],
        (root \ "from").as[String],
        (root \ "commentFor").as[Long])
 
    }
 
    def writes(comment: Comment): JsValue = {
      JsObject(List("comment" ->
        JsObject(Seq(
          "message" -> JsString(comment.message),
          "from" -> JsString(comment.message),
          "commentFor" -> JsNumber(comment.commentFor)))))
    }
  }
 
}
 
case class Comment(message: String, from: String, commentFor: Long) {}

What you see here is that in the companion object we create a new ‘Format’ object. The ‘reads’ and ‘writes’ operations in this object will now be used by the JSON operation to convert from and to JSON when working with the ‘Comment’ class. Very powerful stuff, even though it’s a bit magic ;-) For more information on the Scala/Play environment I used for this example see my previous posts:
http://www.smartjava.org/content/play-20-akka-rest-json-and-dependencies
http://www.smartjava.org/content/using-querulous-scala-postgresql

Reference: Protect a REST service using HMAC (Play 2.0) from our JCG partner Jos Dirksen at the Smart Java blog.