Use Custom Packet Framing for Microservices Messaging

In my previous article, you looked at why forcing communication between microservices, using REST JSON endpoints, is not always the best approach. In this article, I will outline an example custom framing solution that you can extend for your own projects.

Due to the complexity of creating useful framing, implementing the frame will be covered in my next article.

Example introduction

The problem this article will aim to address will be handling communication securely with a custom data store. The language used to implement this frame will be Elixir. Elixir provides an exceptionally flexible binary matching and manipulation syntax, which will enable this article to be more succinct. However, you can implement the frame detailed in this article using almost any language.

To highlight the example thoroughly, the framing you’ll create will not only rely on the HTTP transport protocol. Indeed, it will not even utilize transmission control protocol (TCP). Instead, it will be based on the user datagram protocol (UDP) and will implement some of TCP’s features. This will ensure that packets are absolutely minimal in size.

Packet protocol features

The framing you’ll create will include several important features, but will remain flexible enough for you to extend at length with many of your own, should you wish. The features will include:

• the packet identification signature
• the packet class, method and attributes
• the length parameter
• the transaction identifier
• security using a message integrity system

Packet signatures

Packet signatures, called the “Magic Cookie” in some protocols, are a string of bytes found near the beginning of a packet that can be used to distinguish it from another packet format. The signature should not be long, but should be sufficient enough to be unique. For this example, you’ll use the simple string SHIP. This is a 32 bit value (4 characters of 8 bytes), which is a common size for many protocols.

Packet class, method and attributes

The packets class, method and attributes are hierarchical and identify the purpose of the packet.

Class

The class is the primary packet purpose. This will differentiate the packet as either a request message, a response message or an error message. Since you’ll not be using TCP, you will also need an ack message, which is short for acknowledgement. ack messaging will be covered in detail in the next article.

Method

The method will be the command you wish to invoke on the server. Since this is for a data store, the available methods will include values such as get, set and delete.

Attributes

Attributes cover pretty much everything else you wish to send, or receive, from your server. This will include the username and password necessary for authenticating, the servers realm needed to identify a specific server instance and any request, response or error data.

Packet length

Defining the packet length is important to determine the boundaries of a packet. Typically, a packet header size is fixed, but its data is not. By placing the length value at a set position in the header, the entire length of the packet can be specified and adhered to. This is particularly important if you start chunking your packets into smaller messages that need to be reconstructed on the receiving platform.

Packet transaction identifier

Transaction identifiers are required by the server to determine packet ordering and to acknowledge receipt. Again, since TCP will not be used, it will be important to be able to identify if packets have been dropped. Using a sequential transaction id is sufficient for this purpose.

Packet integrity

The integrity of a packet is simply the authenticity of the packet. This includes both its authentication and whether it has been tampered with. If the packet integrity is off or the user does not have permissions to make the request, the packet will result in an error response.

The packet header structure

As mentioned above, transport protocol packets will usually have a fixed header. The following diagram outlines the structure of our example;

0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                         Magic Cookie                          |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|1 1|     SHIP Message Type     |        Message Length         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
|                     Transaction ID (96 bits)                  |
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Magic cookie

As you can see, the packet begins with the Magic Cookie or simply the string “SHIP” which identifies the packet.

Infix

Next, there are two bits called the infix, which is simply padding and allows the header to be byte aligned. The bits are both set to 1 and also double as packet identification, since if they are anything but ‘1, 1‘, the packet will not be recognized.

Message type

The message type is an interpolation of the packet class and method types. They are of the format:

0                   1
 0 1 2 3 4 5 6 7 8 9 0 1 2 3
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|    m0   |c|  m1 |c|   m2  |
+---------+-+-----+-+-------+

When parsing, the class bits and method bits are combined into two separate binary values, creating a 2-bit class identifier and a 14-bit method identifier. This means there can only be four class types (request, response, error and ack), but a possible 16,384 method types!

Length

The length is the number of bytes (not bits) of the entire packet. A length number of 16-bits means the packet size can be up to 65535 bytes in length or 65.535 kilobytes. Certainly big enough for a single packet.

Transaction identifier

The transaction id, or identifier, will be a sequential numerical value that increments with each subsequent packet. Typically, it is a good idea for the first selected transaction id to be randomly generated between 0 and 2,147,483,646. As packets are dispatched, should the transaction id reach its maximum value, it will simply loop back to 0 and continue to be incremented.

Within the server, a packet is typically identified by a combination of the transaction id and the senders IP address and port number. If the server dispatches messages to a client that are not a form of response (such as push messaging), then the transaction id should be unique across all clients. This will not be a factor in the example used in this article series, however.

Transaction id’s need only be unique for a given timeout period, necessary to determine if a packet should be resent. A typical timeout period may be 20 to 30 seconds.

Putting it in code

That’s quite a lot of in-depth discussion so far. Let’s put this into action with some code:

defmodule CodeshipDB.Pkt do
  use Bitwise

  @pkt_magic_cookie "SHIP"
  @infix 3

  defstruct class: nil,
            method: nil,
            transactionid: nil,
            integrity: false,
            key: nil,
            attrs: %{}
end

First, the structure of the packet is detailed. The defstruct lists the values that make up the packet:

• The integrity field simply states whether an integrity check is included in the packet. This allows you to skip integrity checks if you choose.
• The key field will hold the unique string that is used to build the integrity check. This will be explained in the next article.
• The attrs is currently an empty hash-map. This will be populated with further fields needed to make up the body of the packet.

Above the defstruct are module attributes. These are values that won’t change, but are easier to read and refer to by stating them at the beginning of the module. The @infix is set to 3, which is the equivalent to 11 in binary.

Packet attributes

Now that you have the packet structure, you will need to define your classes, methods and attributes.

def classes(),
    do: [
      :error,
      :request,
      :response,
      :ack
    ]
  def methods(),
    do: [
      :bucket_exists,
      :destroy,
      :get,
      :set,
      :update,
      :del,
      :del_all
    ]
  def attrs(),
    do: [
      {:bucket, :value},
      {:json, :value},
      {:data, :value},
      {:username, :value},
      {:password, :value},
      {:realm, :value},
      {:message_integrity, :value},
      {:error_code, :error_attribute}
    ]

Place these functions within the above module. As you can see, they define the four packet types and the commands you may want to execute against your data store. You can add to the method list as new functionality is built into your data store. However, ensure you add to the end of the list, as adding values within or changing the existing order will break backwards compatibility.

The attrs is the list of attributes used when marshalling the packets. They can be identified as:

Attributes can be defined as one of two types; a simple value or an error_attribute, which is a value pair, defined as a tuple. To convert from bytes to attribute data and vice-versa, you’ll need encoders and decoders:

# loop through all attributes and create a decoder function
  # and an encoder function
  for {{name, type}, byte} <- attrs() |> Enum.with_index() do
    case type do
      :value ->
        defp decode_attribute(unquote(byte), value, _),
          do: {unquote(name), value}

        defp encode_attribute(unquote(name), value, _),
          do: {unquote(byte), value}

      :error_attribute ->
        defp decode_attribute(unquote(byte), value, _tid),
          do: {unquote(name), decode_attr_err(value)}

        defp encode_attribute(unquote(name), value, _),
          do: {unquote(byte), encode_attr_err(value)}
    end
  end

  # handle unknown attribute decoding
  defp decode_attribute(byte, value, _) do
    {byte, value}
  end

  # handle unknown attribute encoding
  defp encode_attribute(other, value, _) do
    {other, value}
  end

  # loop through all methods and create a decoder function
  # and an encoder function
  for {name, id} <- methods() |> Enum.with_index() do
    defp get_method(<<unquote(id)::size(12)>>),
      do: unquote(name)

    defp get_method_id(unquote(name)),
      do: unquote(id)
  end

  # handle unknown method decoding
  defp get_method(<<o::size(12)>>),
    do: o

  # handle unknown method encoding
  defp get_method_id(o),
    do: o

  # loop through all classes and create a decoder function
  # and an encoder function
  for {name, id} <- classes() |> Enum.with_index() do
    defp get_class(<<unquote(id)::size(2)>>),
      do: unquote(name)

    defp get_class_id(unquote(name)),
      do: <<unquote(id)::2>>
  end

Enter the above into the module. It will exist outside of a function, which means the Elixir compiler will execute it at compile time and it will generate a function pair (encoder and decoder) for every class, method and attribute listed. You might notice that handlers exist for unknown attributes and methods, but not for classes. This is simply because all possible class types will be catered for. A 2-bit class identifier can only support four class types and you’ve specified four in your class list.

Finally, you’ll need some helpers to iterate through lists of attributes to encode or decode.

# Converts a given binary encoded list of attributes into an Elixir list of tuples
  defp decode_attrs(pkt, len, tid, attrs \\ %{})

  defp decode_attrs(<<>>, _len, _, attrs), do: attrs # an empty attribute

  defp decode_attrs(<<type::size(16), item_length::size(16), bin::binary>>, len, tid, attrs) do
    whole_pkt? = item_length == byte_size(bin)

    padding_length =
      case rem(item_length, 4) do
        0 -> 0
        _ when whole_pkt? -> 0
        other -> 4 - other
      end

    <<value::binary-size(item_length), _::binary-size(padding_length), rest::binary>> = bin
    {t, v} = decode_attribute(type, value, tid)
    new_length = len - (2 + 2 + item_length + padding_length)
    decode_attrs(rest, new_length, tid, Map.put(attrs, t, v))
  end

  # Converts a given binary encoded error into an Elixir tuple
  defp decode_attr_err(<<_mbz::size(20), class::size(4), number::size(8), reason::binary>>),
    do: {class * 100 + number, reason}

  # Encodes an attribute tuple into its specific encoded binary
  defp encode_bin({_, nil}), do: <<>> # an empty attribute

  defp encode_bin({t, v}) do
    l = byte_size(v)

    padding_length =
      case rem(l, 4) do
        0 -> 0
        other -> (4 - other) * 8
      end

    <<t::16, l::16, v::binary-size(l), 0::size(padding_length)>>
  end

  # Encodes a error tuple into its binary representation
  defp encode_attr_err({error_code, reason}) do
    class = div(error_code, 100)
    number = rem(error_code, 100)
    <<0::size(20), class::size(4), number::size(8), reason::binary>>
  end

There are some interesting points to note above. The error decoder and encoder converts an error code, such as 404 or 500, into a 12-bit binary. It does this by storing the tens into a byte (since a byte can store a number up to 255, while the tens will only ever be a maximum of 99) and the hundreds digit into a 4-bit value. The primary reason for this is that HTTP codes are grouped by the hundreds digit, whereby 2xx codes are a success code, 3xx mean redirection, 4xx are client errors and 5xx are server errors. By simply storing the hundreds digit into a separate block, error messages can be matched and sorted more quickly.

Another point to identify is the larger encoding and decoding functions that deal with padded binaries. Packet framing often attempts to ensure that the contained data is byte-aligned (bit-multiples of 8), which allows for more efficient handling and manipulation of packets.

Message integrity

The final stage before writing the outer encoder and decoder functions is the message integrity. This involves executing a hmac sha1 function over the entirety of the packet and appending it to the end. However, since applying the integrity to the end of the packet changes the packet length, the length parameter in the header needs to be updated before the integrity is calculated.

# full check of integrity
  defp check_integrity(pkt_binary, nil), do: {false, pkt_binary}

  defp check_integrity(pkt_binary, key) when byte_size(pkt_binary) > 20 + 24 do
    with s <- byte_size(pkt_binary) - 24,
         <<message::binary-size(s), 0x00::size(8), 0x08::size(8), 0x00::size(8), 0x14::size(8),
           integrity::binary-size(20)>> <- pkt_binary,
         ^integrity <- hmac_sha1(message, key) do
      <<h::size(16), old_size::size(16), payload::binary>> = message
      new_size = old_size - 24
      {true, <<h::size(16), new_size::size(16), payload::binary>>}
    else
      _ ->
        {false, pkt_binary}
    end
  end

  # Inserts a valid integrity marker and value to the end of a binary
  defp insert_integrity(pkt_binary, nil),
    do: pkt_binary

  defp insert_integrity(pkt_binary, key) do
    <<0::2, type::14, len::16, magic::32, trid::96, attrs::binary>> = pkt_binary
    nlen = len + 4 + 20
    value = <<0::2, type::14, nlen::16, magic::32, trid::96, attrs::binary>>
    integrity = hmac_sha1(value, key)

    <<0::2, type::14, nlen::16, magic::32, trid::96, attrs::binary, 0x00::size(8), 0x08::size(8),
      0x00::size(8), 0x14::size(8), integrity::binary-size(20)>>
  end

  defp hmac_sha1(msg, hash) when is_binary(msg) and is_binary(hash) do
    key = :crypto.hash(:md5, to_charlist(hash))
    :crypto.hmac(:sha, key, msg)
  end

Using hmac sha1 is typically not used in new packet formats, these days, as it is now possible to decypher sha1 encodings with some effort. Therefore, you might want to research updating this to something more current. However, hmac sha1 is still very much used for security throughout the internet, so it is not a horrible choice.

The finishing touch

The final part of the equation is the encoder and decoder functions. These will be the functions you call, external to the module, to convert from a binary string to a decoded packet and back again.

def decode(pkt_binary, key \\ nil) do
  {integrity, pkt_binary} = check_integrity(pkt_binary, key)

  <<@pkt_magic_cookie, @infix::2, m0::5, c0::1, m1::3, c1::1, m2::4, length::16,
    transactionid::96, rest::binary>> = pkt_binary

  method = get_method(<<m0::5, m1::3, m2::4>>)
  class = get_class(<<c0::1, c1::1>>)
  attrs = decode_attrs(rest, length, transactionid)

  {:ok,
   %__MODULE__{
     class: class,
     method: method,
     integrity: integrity,
     key: key,
     transactionid: transactionid,
     attrs: attrs
   }}
end

def encode(%__MODULE__{} = config, nkey \\ nil) do
  m = get_method_id(config.method)
  <<m0::5, m1::3, m2::4>> = <<m::12>>
  <<c0::1, c1::1>> = get_class_id(config.class)

  bin_attrs =
    for {t, v} <- config.attrs,
        into: "",
        do: encode_bin(encode_attribute(t, v, config.transactionid))

  length = byte_size(bin_attrs)

  pkt_binary_0 =
    <<@pkt_magic_cookie, @infix::2, m0::5, c0::1, m1::3, c1::1, m2::4, length::16,
      config.transactionid::96, bin_attrs::binary>>

  case config.integrity do
    false -> pkt_binary_0
    true -> insert_integrity(pkt_binary_0, nkey)
  end
end

The key value in the function signatures is optional and will depend on whether you wish to use the message integrity functionality. The functions merely construct or deconstruct the binary representation of the packet, passing the attributes through their unique functions in order to build or deduce the data in the packet.

Kicking it all off

In the next article, you’ll see how this packet format can be used with a simple data store application. However, for now, you can try this module out by booting up the interactive Elixir console and entering the following code:

pkt = %CodeshipDB.Pkt{
  class: :request,
  method: :set,
  transactionid: 1,
  attrs: %{
    json: "{\"key\":\"my_key\",\"data\":\"abcde\"}"
  }
}

payload = CodeshipDB.Pkt.encode(pkt)

pkt == CodeshipDB.Pkt.decode(payload)

You can boot the Elixir console, providing you have it installed, by entering iex in the command line, followed by the Enter key. If you followed this article precisely, the code above should result in a simple true response, meaning the packet data was encoded and then decoded to its original state.

If you cannot wait for the next article, a simple, albeit unfinished, example of the CodeShip data store can be found here.

Do you want to know how to develop your skillset to become a Java Rockstar?

Subscribe to our newsletter to start Rocking right now!

To get you started we give you our best selling eBooks for FREE!

1. JPA Mini Book

2. JVM Troubleshooting Guide

3. JUnit Tutorial for Unit Testing

4. Java Annotations Tutorial

5. Java Interview Questions

6. Spring Interview Questions

7. Android UI Design

and many more ....

I agree to the Terms and Privacy Policy

Sign up

Thank you!

We will contact you soon.