Dockerized Development and Production Environment For Go (GoLang)

Why should we Dockerize?

Dockerizing an application is the process of configuring the application and its environment to package in a single Docker image and to run it within Docker containers.

Even though Go (GoLang) applications (mostly) compile to a single binary, applications often depend on other supporting files (templates, configurations, etc). When there are a lot of supporting files in an application, it is common to have problems because of out of sync codebase. Turning your Go application into a Docker image is a great way to distribute & deploy your application and be assured that it will run on other machines exactly like it did in development environment regardless of any customizations that machine might have.

How to Dockerize?

Here we will learn how to Dockerize a Go web application for development as well as production environments. While dockerizing web applications is a pretty straight-forward task, there are a few challenges unique to every language/technology that needs to be overcome.

Let's start by creating a simple web application which:

• Listens for HTTP requests on a port. (let's say 18770)
• Handles a GET request /get with a name parameter.
• On receiving a request, responds back with a "Hello" message and a lucky number.

Before we begin, it is imperative to have a working knowledge of Go (its build, test and run processes) and Docker. https://golang.org/doc/ and https://docs.docker.com/ can be good places to begin.

Project Directory Structure

Let us start by creating the requisite directory structure first. Let's name our project my_go_project and have the following file and folder structure.

my_go_project/
    ├── src/
    │     └── my_go_project/
    │             ├── main.go
    │             └── main_test.go
    ├── docker-compose.yml
    ├── Dockerfile
    └── Makefile

main.go

In main.go, lets write our simple web application as follows:

package main

import (
	"flag"
	"fmt"
	"log"
	"net/http"
)

func main() {
	http.HandleFunc("/get", func(w http.ResponseWriter, r *http.Request) {

		name := r.URL.Query()["name"][0]
		fmt.Fprintf(w, "Hello, %q\nYour lucky number is: %d", name, generateLuckyNumber(name))
	})

	err := http.ListenAndServe(":18770", nil) // Note: Not "localhost:18770" but ":18770"
	log.Fatal(err)
}

// Function to return a lucky number for a given name.
func generateLuckyNumber(name string) int {
	// Not really generating a lucky number here, just returning length of the string.
	return len(name)
}

Here we have written a simple HTTP server which listens on port 18770 for a /get GET request and expects a parameter name. We are generating a lucky number (not really :p) and returning it in the response text.

Note: In the above code, we are only making use of Go's internal packages. In the final solution, we will also be using an external package glog to demonstrate external dependencies management.

main_test.go

Even the most brilliant coders are incapable of writing codes that always work exactly as intended. Therefore testing is an important part of the software development process. Test-driven development helps in keeping the quality of your code high and protects from regressions.

Let's update our main_test.go to contain our test cases as follows:

package main

  import (
  	"testing"
  )

  // Unit test for generateLuckyNumber() using test tables.
  func TestGenerateLuckyNumber(t *testing.T) {
  	testCases := []struct {
  		name string
  		want int
  	}{
  		{
  			name: "surender",
  			want: 8,
  		},
  		{
  			name: "thakran",
  			want: 7,
  		},
  	}
  	for _, test := range testCases {
  		result := generateLuckyNumber(test.name)

  		if result != test.want {
  			t.Errorf("generateLuckyNumber(%s) -> %d want %d", test.name, result, test.want)
  		}
  	}
  }

This concludes our basic web application which we will now be dockerizing.

Container Directory Structure

Before we continue on to dockerizing our application we need to decide on the directory structure we intend inside a docker container.

The following directory structure should serve our purposes.

/workspace/
    ├── src/
    │     ├── my_go_project/
    │     │        ├── main.go
    │     │        └── main_test.go
    │    ...                            // External dependencies.
    ├── bin/
    │     ├── my_go_project             // Final project binary.
    │    ...                            // Other binaries (if any).
    ├── pkg/                            // Go package objects.
    ├── docker-compose.yml
    ├── Dockerfile
    └── Makefile

Note that we are placing our project in the filesystem root's /workspace directory. It is a personal choice to use any directory you want because we usually load only one project in a container.

Dockerfile

Every dockerized application must have a Dockerfile which is usually located in the root of the application. A Dockerfile is a text document that contains all the commands a user could call on the command line to assemble an image.

Let's create a Dockerfile which has the following content:

FROM surenderthakran/go:1.7.3

MAINTAINER https://github.com/surenderthakran

ENV GOPATH /workspace
ENV PATH "$PATH:$GOPATH/bin"

COPY ./ $GOPATH

WORKDIR $GOPATH

RUN make --no-print-directory install

CMD make --no-print-directory run

I usually prefer to use my own base images while dockerizing my applications so that I have more control over what goes into my projects (just because I can :p). We are free to use surenderthakran/go:1.7.3 or choose an Official Docker Image.

First we need to update a couple of environment variables: $GOPATH and $PATH. We set our $GOPATH as /workspace. This environment variable specifies the location of our workspace and this where Go will look for /src, /bin and /pkg directories.

We also added our $GOPATH/bin directory to the $PATH environment variable.

Next we copied our project's code base to our $GOPATH directory inside the container and change the present working directory to it. Dockerfiles best practices suggests using COPY instead of ADD to move local files in a docker container.

The RUN and CMD instructions, in their simplest form, are used to build our image and to run our containers respectively. RUN instruction is executed while building our docker image and CMD instruction is executed at runtime when we launch the container.

The --no-print-directory option in the make commands is to prevent make from printing directory change messages and cluttering our log screen.

Makefile

I (and a lot of other developers) prefer to keep the actual image building and run commands out of the Dockerfile because it provides much more scripting liberty than using the Dockerfile. I personally prefer to use make.

Let's update our Makefile to have the following content:

GO_PROJECT_NAME := my_go_project

# Go rules
go_format:
	@echo "\n....Formatting $(GO_PROJECT_NAME)'s go files...."
	gofmt -w $(GOPATH)/src/$(GO_PROJECT_NAME)

go_prep_install:
	@echo "\n....Preparing installation environment for $(GO_PROJECT_NAME)...."
	mkdir -p $(GOPATH)/bin $(GOPATH)/pkg
	go get github.com/cespare/reflex

go_dep_install:
	@echo "\n....Installing dependencies for $(GO_PROJECT_NAME)...."
	go get -t ./...

go_install:
	@echo "\n....Compiling $(GO_PROJECT_NAME)...."
	go install $(GO_PROJECT_NAME)

go_test:
	@echo "\n....Running tests for $(GO_PROJECT_NAME)...."
	go test ./src/$(GO_PROJECT_NAME)/...

go_run:
	@echo "\n....Running $(GO_PROJECT_NAME)...."
	$(GOPATH)/bin/$(GO_PROJECT_NAME)


# Project rules
install:
	$(MAKE) go_prep_install
	$(MAKE) go_dep_install
	$(MAKE) go_install

run:
ifeq ($(CODE_ENV), dev)
	reflex -r '\.go$\' -s make restart
else
	$(MAKE) go_run
endif

restart:
	@$(MAKE) go_format
	@$(MAKE) go_install
	@$(MAKE) go_test
	@$(MAKE) go_run

.PHONY: go_format go_prep_install go_dep_install go_install go_run install run restart

Most of what is written above should be easy to grasp for someone who has used Makefile's before. If you have not I strongly recommend to atleast skim over the Official GNU Make manual.

While creating our docker image, we first need to prepare our Go workspace for our application by creating the required /bin and /pkg directories in our $GOPATH. Next we install all our application's external dependencies by running go get -t ./.... With the -t flag any separate dependencies in test files are also installed.

Finally, we install our application by running go install $(GO_PROJECT_NAME) where GO_PROJECT_NAME=my_go_project as defined at the top of our Makefile.

Once our application is installed, a Go binary called $GO_PROJECT_NAME is created in our application's $GOPATH/bin directory which we will be using to run our application. Once done, a docker image is also created on the host machine which we can use to launch docker containers running our application. As defined in the Dockerfile, make run is executed when a docker container is being launched which in turn executes our application's Go binary by running $(GOPATH)/bin/$(GO_PROJECT_NAME).

We now have a working production environment for our application but we also need to take care of our application in the development environment. We differentiate between development and production environment using the value of CODE_ENV environment variable in the docker container. In a development environment its value would be dev.

In the development environment, we need to keep track of any code changes, run tests on the changed code, reinstall the application and then re-run it.

After a bit of research I decided on reflex as the current best solution for monitoring file system changes and running reload commands for Go.

When a docker image is being built we also installed reflex as a dependency using go get github.com/cespare/reflex. When a container is launched in a development environment then on make run it runs reflex which recursively checks for any changes in any .go files.

On any change in the codebase, apart from re-installing and re-running the application it also reformats the application's Go source files and runs tests using gofmt and go test.

But there are still a few unanswered questions:

• How do the code changes occur in a docker container which is suppossed to be completely separated and independent of the host machine?
• How does an application running inside a container serve requests coming from outside?

docker-compose.yml

Although, Docker Compose is a docker tool for configuring and running multi-container applications, it can also help us manage our development environment.

It allows us to configure applications in a docker-compose.yml file usually located in the root of our project.

Let's update our docker-compose.yml file to the following:

version: '2'
services:
  app:
    build: .
    command: make --no-print-directory run
    volumes:
      - ./src/my_go_project:/workspace/src/my_go_project
    environment:
      CODE_ENV: dev
    ports:
      - "18770:18770"
    logging:
      driver: 'json-file'

We are using version 2 of the Compose file format and you can refer to Compose file version 2 reference to grasp most of the above code. I would however like to draw attention to few points in the above file:

For a running docker container to receive code changes from the host machine, we mount our project's Go source directory over the project's source directory inside the container. We do this with the volumes option by specifying the mount paths like this: ./src/my_go_project:/workspace/src/my_go_project.

As I mentioned earlier, we differentiate between the development and production environments using the CODE_ENV environment variable whose value is dev in development environment. We can set environment variables inside docker container using the environment option like we did above.

Also for our application, which runs on port 18770, to be accessible outside the container we need to expose the port outside the container and map it to an unused port on the host machine. We are free to choose any ephemeral port on the host machine. In our example, I choose to use the same port in container and on host machine.

How to use?

In Production

We can build a docker image for our go application by running:

docker build --no-cache=true -t go_app .

The --no-cache=true flag askes docker to not use existing images from its cache while building intermediate images. Though docker has a pretty robust cache leveraging mechanism, for sensitive production images it is best to be safe in case any unwanted change in previously cached images trickled into your final image.

To launch a docker container for the image created above we run:

docker run -it -d --name go_app_1 --net=host go_app

With --net=host the container's port 18770 is mapped to host's port 18770. We can also map our application's port inside the container to a different port (ex: 80) by launching our container like this:

docker run -it -d --name go_app_1 -p 80:18770 go_app

In Development

Running our application in development is a very easy process thanks to the docker-compose command. To build our application's docker image we run:

docker-compose build

And to launch a docker container, we run:

docker-compose up

The Final Solution

Find the complete Go dockerized solution which we discussed in the article at surenderthakran/go_docker_env from which I believe all team and project sizes can benefit.