The Go language is a really great fit for the increasingly popular service-orientated architecture.
In the last few years, many good practices have emerged to help with the problems that come with microservices. These practices are important if you don’t want to end up with a hard-to-maintain, hard-to-operate infrastructure. By combining them with some of the more overlooked features of Go, you can make development and the operation of your services a lot easier.
In this blog post, I explain these features of Go and offer best practices focused on developing, building, and configuring your services. These practices are easy to apply, and as a result, new developers can get on board faster and cooperate better; you can deploy, understand, and debug your services more rapidly; and in general, the development will be easier.
- The source code of the example service is available on GitHub: https://github.com/gulyasm/goservice
Keep It Simple
At enbrite.ly, we reveal fraudulent activities in online advertising. As a result, we have developed and operate an infrastructure composed of services. Most of these services are implemented as Go services.
We chose Go because it’s easy to learn, simple, and productive. In the last year, we’ve written more than 20 services with a development team of six.
At the beginning, we used magic frameworks and libraries to ease the development process by using what others have written. We started to use Martini and later Negroni to add middlewares and logging.
I’ve read many great Go developers going with the standard HTTP library, and I thought, “Boy, that’s hardcore.” But as you learn more and more about the standard HTTP library, the more you realize that it’s pure genius and perfect for services.
Check out this example about how to create a minimal service that can be called with a domain and return the IP addresses:
type IPMessage struct {
IPs []net.IP
}
type ErrorMessage struct {
Error string
}
func IPHandler(rw http.ResponseWriter, r *http.Request) {
domain := r.URL.Query().Get("domain")
if len(domain) == 0 {
json.NewEncoder(rw).Encode(ErrorMessage{"No domain parameter"})
return
}
ips, err := net.LookupIP(domain)
if err != nil {
json.NewEncoder(rw).Encode(ErrorMessage{"Invalid domain address."})
return
}
json.NewEncoder(rw).Encode(IPMessage{ips})
}
func main() {
address := ":8090"
r := http.NewServeMux()
r.HandleFunc("/service/ip", IPHandler)
log.Println("IP service request at: " + address)
log.Println(http.ListenAndServe(address, r))
}We define the IPHandler function which is an http.HandlerFunc. In the main function, we add this to our router.
It’s better to always explicitly get a router than implicitly use the default; this way we can swap with another router whenever we want. If we decide we need path parameters, we can easily switch to the Gorilla router by changing http.NewServeMux() to mux.NewRouter(). Only one line of change. Gorilla router is actually the only third-party library that we sometimes use in our services. It’s well written and perfect for REST services.
We’ve also started the good practice of logging the listening address with the port; it’s easier for other developers to see in the logs that the service did in fact start and which address it is listening on. The return value of the ListenAndServe function should be logged also so that we get a log message if there is any error and the function returns instead of blocking.
The next thing we might need is a middleware. Creating middlewares is easy with the standard library itself. All we have to do is define an http.Handler that calls our base handler and add that to the ListenAndServe function instead of the base handler. Again, no need for third-party libraries; a simple and elegant solution.
The middleware itself:
type M struct {
handler http.Handler
}
func (m M) ServeHTTP(rw http.ResponseWriter, r *http.Request) {
start := time.Now()
m.handler.ServeHTTP(rw, r)
log.Printf("%s served in %s\n", r.URL, time.Since(start))
}
func NewM(h http.Handler) http.Handler {
return M{h}
}If we want to add anything to our middleware, we can do it in the ServeHTTP method. To make our code cleaner and more readable, we can extract the building of our base handler into a separate function.
func createBaseHandler() http.Handler {
r := http.NewServeMux()
r.HandleFunc("/service/ip", IPHandler)
return NewM(r)
}After defining the base handler, we add it to the ListenAndServe function.
log.Println("Service request at: " + address)
log.Println(http.ListenAndServe(address, createBaseHandler()))Logging is another area like middlewares where people turn to third-party libraries. One feature that comes up as a deficiency of the standard library logging package is log levels.
While it’s true that sometimes it comes in handy to switch between verbose and production logging, we follow a simple rule. You should log an event or state if it’s important and carries valuable information:
- If it has no valuable information, don’t log it.
- If you used that log message for debugging purposes, remove it in production.
- If it’s an error, log it with Fatal or Panic, according to the nature of the error.
- Any other log goes with
log.Println.
With this little rule, you don’t really need log levels other than the ones in the standard logging library.
Another best practice we follow with error logging is to log the error itself if there is one. Don’t just log a generic “Error happened,” log the content of the error returned with as much information as you see fit. It can be really frustrating when an error happened, but the error message doesn’t tell you anything specific. A good example is a template error. When the template parsing fails, the returned error instance contains the exact problem. Without it, it’s hard to debug the problem.
By providing more detailed error message, you help debugging later on. Believe me, your team will thank you later!
You can also set a logging prefix with log.SetPrefix to note which log message comes from which service. If you add host information to the prefix, you can even identify which host logged the message, which can be pretty useful in a distributed environment.
With this simple and easy-to-follow rule, our logging contains only valuable information. This reduces the noise but ensures that we have every valuable piece of information that we need for debugging, telemetry, or post-mortems.
To set the prefix, add the following to your main function before any logging statement:
log.SetPrefix("[service] ")The Build Process
The building and deployment of an application has to be easy and automatic. After trying different approaches, we ended up using Makefiles.
Combining Makefiles with the tools provided by the go command makes a full-featured build tool. Building, testing, test coverage, static analysis, and formatting are all part of the go command. By making them part of the build process, the developers don’t have to think about these steps. If every service uses the same way for building, it’s easy for every developer to build any of them.
With only a make command, we vet, format, test, and build our service. Let’s walk through an example of how to add a Makefile to our service:
default: build
build:
go fmt
go vet
go build
test: build
go test
coverage-test:
go test -coverprofile=coverage.out
go tool cover -func=coverage.out
go tool cover -html=coverage.out
rm coverage.outWith this unified Makefile, every team member checks, tests, and formats the code the same way. It helps tremendously with code reviews. No need to discuss formatting. Go’s vet hasve already checked the code for common mistakes. If the build process passed, you can be 100 percent sure there are no unused variables or imports.
I personally really like the compiler feature that fails when encountering an unused variable. We don’t have to argue about unused variables and guarantees that there are no loose ends.
To analyze test coverage, use the make coverage-test command. It runs tests and displays the results in HTML in your browser, so you can see what your tests cover and what’s missing. Super easy test coverage, provided by the standard Go tools.
Always Add a UI. No Exceptions
I have a hard rule: Every service has to provide a UI on the root URL /. The UI should show the internal state and the history of whatever the particular service does.
The trust that every service can be reached and checked on http://<service address>/ gives developers confidence and a starting point when they have to debug, develop, or simply get to know a new service.
With Go, it’s very easy to add a UI to any service. Templates offer an easy way of generating HTML. Add Bootstrap CSS, and you have an operational UI. Start it with a goroutine, and you don’t even have to handle parallelism. Running an admin site parallel to the service itself is astonishingly easy with Go.
First, create a basic HTML template with some bootstrapping CSS, a table for the queries, and the datatables javascript module which adds features like pagination, search, and sorting to ordinary HTML tables.
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title></title>
<link rel="stylesheet" href="http://d2fq26gzgmfal8.cloudfront.net/bootstrap.min.css" media="screen">
<link rel="stylesheet" type="text/css" href="https://cdn.datatables.net/r/bs/dt-1.10.9/datatables.min.css" />
</head>
<body>
<div class="container">
<div class="row">
<h1>Domain checker</h1>
</div>
<div class="row">
<table class="table table-bordered" id="queries">
<thead>
<tr>
<th>Domain</th>
<th>IPs</th>
</tr>
</thead>
<tbody>
{{ range .Queries }}
<tr>
<td><a href="http://{{ .Domain }}">{{ .Domain }} </a>
</td>
<td>{{ .IPs }}</td>
</tr>
{{ end }}
</tbody>
</table>
</div>
</div>
<script src="http://d2fq26gzgmfal8.cloudfront.net/jquery-1.10.2.min.js"></script>
<script type="text/javascript" src="https://cdn.datatables.net/r/bs/dt-1.10.9/datatables.min.js"></script>
<script src="http://d2fq26gzgmfal8.cloudfront.net/bootstrap.min.js"></script>
<script>
$(document).ready(function()
{
$('#queries').DataTable();
});
</script>
</body>
</html>We have to expose another endpoint on / that parses, executes, and displays this template.
func uiHandler(rw http.ResponseWriter, r *http.Request) {
tmpl, err := template.ParseFiles("templates/index.html")
if err != nil {
log.Panic("Error occured parsing the template", err)
}
page := PageData{
Queries: queries,
}
if err = tmpl.Execute(rw, page); err != nil {
log.Panic("Failed to write template", err)
}
}To wire it up, we have to add the UI endpoint to our router, define the structs we use and add any incoming domain queries to the queries list so we can display it on the UI.
We use one struct to register the queries and their results. The other is for the template that describes the UI.
// The Query type represents a query against our service.
type Query struct {
Domain string
IPs []net.IP
}
// The data struct for our basic UI
type PageData struct {
Queries []Query
}Where we defined our previous handler in our createBaseHandler function, we can add the UI endpoint.
r.HandleFunc("/", uiHandler)And at last, we append every query we complete in our IPHandler to a global variable. The variable:
var queries = []Query{}In the IPHandler function:
queries = append(queries, Query{domain, ips})Every team member loves that we provide UI for our services. This policy helps a lot when new team members try to understand what a specific service does. The UI gives them a functional view of what the service is used for. It’s also easier to see what’s happening with a service when you can check it in your browser and don’t have to query a DB.
A surprising side effect was that client operations and sales also started to use them; the easily accessible UI made our services available outside of the development team.
Uniform Heartbeat
We have a simple set of metrics we have to know about our running services:
- Is it running at all?
- If it’s running, which build is it running?
And we want to provide this information for every service we write and operate. With Go, it couldn’t be easier.
We provide a library that runs a webservice on a given port. When called, it returns the SHA-1 code of the commit used to build the binary, the uptime, and the status. To use this library, all we have to is add go heartbeat.RunHeartbeatService(portnum) to our main function.
With just one line, we added a heartbeat to our service that can be used by any third party to check the status of our service. We use Consul for service discovery and configuration management. Our Consul cluster uses these heartbeats to know if the service instance is healthy.
For DNS, we utilize AWS Route53; it also uses this heartbeat signal to see which IP addresses are valid for a given DNS record.
To set the commit SHA-1 code when building our service, we change the go build command in our Makefile to:
go build --ldflags="-X github.com/enbritely/heartbeat-golang.CommitHash`git rev-parse HEAD"
It will add the SHA-1 hash to the heartbeat library, that will return it when http://<service-address/heartbeat url is called.
Another simple solution, and again, it’s easy for anyone to remember that every service has this heartbeat. It can be used to get the running version or to check the status. Not a planned advantage, but it also helped a lot later with automation. When we introduced automatic failover, where one service is replaced with another in case of failure, we used this heartbeat feature to check for the health of the service.
Another example of later advantage was when one of our teammates created a website where all the deployed services were listed. Because we had this heartbeat service, we could just query all the services and display the page with the results.
To enable the heartbeat service listening on the address configured in the HEARTBEAT_ADDRESS environmental variable, add the following two lines to your main function.
hAddress := os.Getenv(“HEARTBEAT_ADDRESS”) go heartbeat.RunHeartbeatService(hAddress)
To install the heartbeat service, you have to go get-it.
go get github.com/enbritely/heartbeat-golang
Configuration
I like to keep the configuration simple too. We follow the 12 Factor Application technique that describes that configuration should come from the environment. That means environmental variables.
The simplicity of it makes it very flexible. It’s easy to set different configurations in production than testing, staging, or development. We use our service startup scripts to set these variables in production, so they’re available when the service starts.
Environmental variables are easy to set up on development machines: You can define them on the command line or export them in your .bashrc or .profile file. We made it even easier.
Every developer defines a .dotenv file in the root folder of the project. It contains our environmental variables; sourcing them will set the development configuration. It’s fast, easy, and every developer can adjust it.
For example, if you don’t want to run a DB on your machine, you can set the DB address to point to any location you want. Some of us use Docker to run local DB, some of us use AWS RDS for testing. With environmental variables and the .dotenv file, it’s not a problem.
Conclusion
Software development should be fun. It’s in the whole team’s best interest that we go for it if there are parts that we can make fun. Making code reviews, deployment, testing, or monitoring easier is a great place to start.
Go has great support for all of that. Making our development environment uniform across each project helps existing or new team members to get on board easier. Making services accessible helps debugging. The UI creates a human-readable interface for our services. All of these practices help our team focus on things that matter.
What’s your own experience with Go? Do you have other best practices? Please share it in the comments!
| Reference: | Utilizing the Simplicity of Go for Easy Development from our JCG partner Mate Gulyas at the Codeship Blog blog. |
