These are just my opinions. It is the current state of my brain when it comes to organizing a Go project. I reserve the right to change these opinions later ;) Also, it’s totally fine if you prefer something different. I’d love to hear or read about why you feel differently.

If you’d like to do some homework, here is some light pre-reading on the topic:

• The one-and-only, must-have, eternal Go project layout

• Go best practices, six years in

• I’ll take pkg over internal

• Organizing Go code

• Go Project Layout

The TL;DR of this entire post is that I make use of a cmd directory for housing my main application packages and a pkg directory to contain my library code. That’s it. Simple and to the point. Until the project evolves, this seems like a great base to start from.

Below you can see the general layout of a project directory. We’ll go through each of these files and directories and describe their uses.

├── .dockerignore
├── .editorconfig
├── .gitignore
├── .gitlab-ci.yml
├── .golangci.yml
├── Makefile
├── README.md
├── cmd
│   └── main.go
├── deployments
│   ├── Dockerfile
│   └── docker-compose.yml
├── go.mod
├── go.sum
└── pkg
    ├── config
    │   └── config.go
    ├── db
    │   ├── db.go
    │   └── db_test.go
    ├── handler
    │   ├── handler.go
    │   └── handler_test.go
    ├── mailer
    │   ├── mailer.go
    │   └── mailer_test.go
    ├── router
    │   ├── router.go
    │   └── router_test.go
    ├── submission
    │   └── submission.go
    └── templs
        ├── layout.templ
        ├── layout_templ.go
        └── types.go

Root-level Files

.dockerignore

This file contains references to anything I do not want copied into containers when using the COPY command inside a Dockerfile

Typically this includes secrets, local/development-only configuration, random binaries, and any other files not directly responsible for building the app into a container.

.editorconfig

EditorConfig is helpful for maintaining consistent code style across different editors and IDEs.

.gitignore

There’s no strict guide here, but essentially, avoid checking in binaries, zips, certs, locally generated things that you are not specifically vendoring, code editor or IDE-specific directories and configuration files (think .vscode or .idea). There are of course exceptions to these loosely defined rules, but until exceptions present themselves, these are decent guidelines to follow.

Generally, it is GitHub’s Go.gitignore template and adding a few things like:

.vscode
.idea

/bin
coverage.html
/test-reports
*.tar
*.zip
.DS_Store

.gitlab-ci.yml

This is configuration for GitLab CI/CD pipelines. Typically I will want to run tests, lint code, build binaries and/or containers, and push an image on each pushed commit to GitLab.

Swap this with your CI/CD automation of choice.

.golangci.yml

I use golangci-lint to aggregate the various linters that are used to check the code.

The general configuration I use:

run:
  modules-download-mode: readonly
  skip-files:
    - ".*\\.pb\\.go$"
linters:
  fast: true
  enable:
  - errcheck
  - godot
  - gosimple
  - govet
  - ineffassign
  - staticcheck
  - typecheck
  - unused
  - whitespace
  - revive

I usually ignore linting files generated by external tools as I’ve found that some generators don’t create code that necessarily aligns with how strict I like to be for commenting or handling errors. There are sometimes other quirks when applying these broad rules on code that is generated by external tooling.

Makefile

The Makefile is made up of targets that you can think of as individual development tasks that are made up of commands that might have many arguments or flags, or be dependent upon previously run commands. Think the Makefile as a list of more convenient and easy to remember commands.

For example, think about building a binary. In local development, we may be able to get away with simply running go build main.go, but can we do the same in a CI job? Are there additional compile-time flags you want to use? Would your development binary be able to run in a container or someone else’s development machine? Perhaps, without additional input.

You may want to start with a simple run target defined like so:

.PHONY: run
run:
    go run ./cmd

A more complex example of a build target:

BRANCH?=$(shell git rev-parse --abbrev-ref HEAD)
SHA:=$(shell git rev-parse --short HEAD)

GOOS?=$(shell go env GOOS)
GOARCH?=$(shell go env GOARCH)

BINDIR:=./bin
BINARY_PATH ?= ${BINDIR}/${GOOS}-${GOARCH}
BINARY_NAME ?= app

AWS_ACCT ?= XXXXXXXXXX
AWS_REGION ?= us-east-1
IMG_NAME ?= mybinary
DOCKER_REPO ?= $(AWS_ACCT).dkr.ecr.$(AWS_REGION).amazonaws.com/$(IMG_NAME)
DOCKER_BRANCH = $(subst /,-,$(BRANCH))
DOCKER_TAG ?= $(DOCKER_BRANCH)-$(SHA)
DOCKER_IMG ?= $(DOCKER_REPO):$(DOCKER_TAG)

.PHONY: build
build: ## Build the go binaries
	GOOS=${GOOS} GOARCH=${GOARCH} go build -o $(BINARY_PATH)/$(BINARY_NAME) -ldflags "-X 'main.version=${DOCKER_TAG}' -X 'main.desc=branch:${BRANCH},commit:${SHA}'" ./cmd
endif

With this make build command, I have a lot more control over how the binary is made, mainly via environment variables. I have sensible defaults for ensuring the build command works as expected in local development. It builds a binary for our local OS and architecture and puts it in our ./bin directory at the root of the project.

In CI, I am probably going to want to be explicit about the OS/architecture for the resulting binary. In this case I simply define the appropriate environment variables and can use the same make build that I would locally.

This is obviously a very involved example. Make it super simple until otherwise needed.

The same sort of ideas carry over into other tasks as well. Running tests, generating code from protobuf files, linting code, building and pushing application containers to a repository - the list goes on.

README.md

The README should contain a high-level overview of the project, instructions on fully setting up a development environment, and any other information about how to interact with the code repository. In other words, it’s an instruction manual for someone who is brand new to the repo. It should contain everything they need to know about the development, build, and deploy experience.

go.mod / go.sum

Do people write Go without modules anymore? These should be straightforward, but if not:

The go.mod file defines the module import path, the minimum version of Go required to create the module, and dependency requirements with version locks.

The go.sum file contains the checksum of the dependencies along with the version. It’s used to confirm that none of the dependencies have been modified.

Root-level Directories

bin

The bin directory is where local builds will be output. The majority of the time I simply have a single binary there after running my go build... command.

A more complex example would be using additional directories that are named by the OS and architecture of the binary files that they contain.

In the example below, you can see that there are two binaries. One is for MacOS on Apple Silicon and the other is for 64-bit Linux.

./bin
├── darwin-arm64
│   └── app
└── linux-amd64
    └── app

I like to include the bin directory itself in Git, but not its contents.

cmd

The cmd directory contains the main application(s) for this project.

For simple projects, I’ll probably just have a single file in there called main.go.

In more complex situations where I have more than one binary that is built within a project, I use a separate directory for each application. The name should match the name of the executable you want to have.

In the example below, I have a server and a cli application.

./cmd
└── server
│   └── main.go
└── cli
    └── main.go

deployments

I tend to keep any IaaS, PaaS, system and container orchestration related files, deployment configurations and templates, and build/deploy scripts here. Basically anything outside of the code itself that would be needed to run the code elsewhere.

I’ve worked on a bunch of different projects that are deployed and configured in many different ways. Generally, a root-level deployments directory would contain my Dockerfiles, bash scripts (useful when Make isn’t enough, container entrypoints, or there are complex one-off tasks), configuration files for deployments, sidecar containers, etc.

Sometimes I keep environment-specific configuration here. Sometimes it’s defined elsewhere. Depends on the project and what the infrastructure is like.

No secrets, though. Never, never, ever commit secrets to your repo. The thought sends chills down my spine. Please let me sleep at night and take proper care of your important information. Your SREs appreciate it as well.

pkg

Normally, a pkg directory would only contain library code that is OK to be used by external applications and anything private to this repository would live in an internal directory.

In order to simplify the project structure, I ususally start by only using the lone pkg directory to group Go library code in one place. It helps to not clutter our root directory and really limits the places you need to look around for code you’re importing in other places.

Again, simple until it needs to be otherwise.

Post-Viewing

Did you think you were done? If you’ve made it this far, you’re either quite passionate about making software, slightly out of your mind, quite bored, or some combination of all of the above. Either way, if you really want to press on, there are a ton of others who have opinions on the topic as well.

• Best Practices for Industrial Programming from GopherCon EU 2018

• GopherCon 2018: Kat Zien - How Do You Structure Your Go Apps

• GoLab 2018 - Massimiliano Pippi - Project layout patterns in Go.

Conclusion

To me, this is the minimal amount of ceremony to be productive and organized in a small-to-medium sized Go project. I don’t take a hard line on any of these opinions. If I’m working with others, chances are they also have opinions and I’d rather get things done than die on the mole hill I’ve described above.

Also, I get this isn’t the sexiest topic in the world, but I appreciate you sticking around and reading through. I’m hoping that by forcing myself to write about some of the things I find myself doing on a daily basis, it will help me realize whether or not I’m doing them for good reasons.

Charles as described on its site:

Charles is an HTTP proxy / HTTP monitor / Reverse Proxy that enables a developer to view all of the HTTP and SSL / HTTPS traffic between their machine and the Internet. This includes requests, responses and the HTTP headers (which contain the cookies and caching information).

Lately, however, I’ve been doing a lot more work on Android apps than I have on server-side applications. I figured it would be similarly helpful to debug some of the requests we were making from our apps without logging out to Logcat and cluttering our app code.

The requests I wanted to check happened to be protected by SSL, so in addition to adding the proxy information to my device’s networking configuration, I also needed to install Charles’ root certificate. It turns out that this is possible, but requires a bit of work. Luckily I have now done this enough times to feel good about writing it up.

First off, this setup will not work on an emulator without an additional step (see below). I admittedly haven’t needed to do any proxy debugging without a test device, though.

Also, as of this post, we are targeting Android SDK version 25 in our applications, but this specific setup should work with all versions. If not, feel free to correct me so that I can update this post.

Now on with the show.

Setting Up Your Project

If your test device is running Android 7.0 or newer, there are a few extra steps to ensure that you are able to trust the Charles CA Certificate. To ensure Android trusts the certificate, you’ll need to specify a Network Security Configuration for your application. There are a few different ways of adding your certificate to the application’s trusted configuration, but I found that the easiest was not to include Charles’ certificate within my application code, but to simply trust the user-installed CA certificates (like previous versions of Android used to do).

Specifying theandroid:networkSecurityConfig property within your AndroidManifest.xml file:

1<application
2  android:allowBackup="true"
3  android:icon="@mipmap/ic_launcher"
4  android:label="@string/app_name"
5  android:roundIcon="@mipmap/ic_launcher_round"
6  android:supportsRtl="true"
7  android:theme="@style/AppTheme"
8  android:networkSecurityConfig="@xml/network_security_config">

Specifying the res/xml/network_security_config.xml file:

1<?xml version="1.0" encoding="utf-8"?>
2<network-security-config>
3  <debug-overrides>
4    <trust-anchors>
5      <!-- Trust user added CAs while debuggable only -->
6      <certificates src="user" />
7    </trust-anchors>
8  </debug-overrides>
9</network-security-config>

Setting Up Charles

Open up Charles on your computer and then open Proxy > Proxy Settings from the toolbar. You’ll need to make note of the HTTP Proxy Port that is specified.

Then open up Proxy > SSL Proxying Settings from the toolbar and add the appropriate Locations (host/port) for the requests you wish to debug. I usually just add a wildcard route like *.myapidomain.com but specific domains like whatever-app.herokuapp.com work fine as well.

Setting Up Your Android device

First, make note of your computer’s IP address within your local network. You’ll need it in the next step.

Also make sure Charles is open and that the computer is on the same wifi network as the Android device you’ll be setting up.

Go to your device’s settings and wifi configuration.

Press and hold on the network you’re going to use and select Modify network from the alert modal.

Then check Advanced Options and scroll until you see Proxy.

Tap the Proxy dropdown and select Manual.

For the Proxy hostname input your computer’s IP address that you noted above.

Then for the Proxy port use the HTTP Proxy Port that you noted from Charles’ Proxy Settings from above.

Tap Save.

After saving, you may notice a pop-up on your computer like this:

You should now click Allow to let your Android device connect to Charles via the proxy it has set up.

If you don’t see this prompt, you can ensure that your device can use the proxy from within Charles by going to Proxy > Access Control Settings and adding your Android device’s IP address on the wifi network.

One more step!

Now on your Android device, open up the browser and go to the following URL chls.pro/ssl. You can also see these instructions from within the Charles app - go to Help > SSL Proxying > Install Charles Root Certificate on a Mobile Device or Remote Browser.

The browser should download the cert and once finished, tap to open the file. Then Android should prompt you to install the file.

After you’ve named the certificate and tapped “OK”, you will either be prompted to input your device’s pin/password or prompted to set up your device’s pin/password. Once that process is complete, you should be ready to debug some requests.

Note Until you remove the manual proxy from this device’s wifi configuration, whenever you use this network again you will need to have Charles open and running on the same IP address. You can always set the network configuration back to “Proxy: None” once you’re done debugging.

Using An Emulator

According to Charles’ site you can start the emulator with a flag in order to use the proxy.

In the Android emulator run configuration add an Additional Emulator Command Line Option: -http-proxy http://LOCAL_IP:8888

Where LOCAL_IP is the IP address of your computer, not 127.0.0.1 as that is the IP address of the emulated phone.

EDIT: Thanks to Brandon Davis for pointing out that the full command would be

emulator -avd EMULATOR_NAME -http-proxy IP_ADDRESS:PORT

In addition, Android Studio 3.0 will now allow managing proxy settings for your emulator images. Read all about the new features here and if you can’t wait for a stable release, 3.0 is currently available in the dev channel.

Debugging requests

Now that you’re set up, you should see Charles recording traffic from your device. As long as you’re configured properly, you should see those SSL request details from the domains you specified in Charles’ SSL Proxying Settings.

Feel free to discuss your results and let me know if I missed anything obvious.

I wasn’t sure what to expect out of Nashville, but when we arrived we were almost immediately greeted with a buffet of the finest BBQ I’ve ever had. Our hosts at Overdog, who we have worked closely with over the past two years, brought us in to talk shop and meet the team face-to-face (some for the first time). We love collaborating with their team and I enjoyed getting to see how they work and hang out around town.

Nodevember’s format seems to be of the “go big or go home” sort. There was a lot of variety in the talks and speakers and it was a bit overwhelming to choose where to go next with four tracks. The talks were all a half hour long, so there wasn’t really too much time to dig deep, but there seemed to be a good balance between intro-level and higher-level content, which is always great. I love going into a talk and having the feeling like the concepts are way over my head. I like knowing what I don’t know.

I also liked the venue, Libscomb University. All of the facilities seemed new and it was refreshing to be able to walk outside between talks. We stayed in a nice Airbnb about a mile from the campus, which was itself only about 5 miles to downtown Nashville, so everything was a short drive away.

The Talks

Some of the standouts from day one, for me, were David Neal showing everyone how to get started writing “Cross-Platform Desktop Apps with Electron”, Ben Ilegbodu priming us on the new hotness with “Sugar and Spice and everything nice about ES6”, and Wyatt Preul giving an overview of how easy it can be to start building microservices with “The proof is in the pudding - seneca+hapi”.

I had already started using some of the new ES6 features with a React app I built recently, but I’m especially excited about some of the things Ben shared about destructuring, rest/spread parameters, function defaults, and using for…of to iterate over iterable objects. This will definitely help with removing a lot of cruft from future code.

It was also great to meet Wyatt after his talk about seneca and hapi. We just so happen to use seneca at Odd Networks and one of the very best things about going to industry conferences is being able to connect with the folks that collaborate and work on some of the everyday tech that you use. We’re pretty excited for the future of seneca and can’t wait to help contribute.

The closing keynote from Day 1 was a talk by Ed Finkler regarding “Open Sourcing Mental Illness”. I really admire Ed for getting in front of a room of his peers to tackle a tough subject and share his own personal struggles. His call to action was that we should all be working toward being open and accepting of the issues that affect more people than we probably realize (1 in 5 Americans, according to Ed) and making sure our employers do the same. I hope others will consider making a donation to Ed’s Indiegogo campaign to bring awareness to the mental health issues in our industry and what we can do about it. He’s doing great work!

After Hours

After filling our brains with much needed sustinence, it was time to see what else Nashville had to offer. The nightlife downtown was electric with a seemingly endless array of bars with a large crowd and a live band. A quick bite and a walkabout was it for me, though.

Day 2

The final day of Nodevember had an equally packed schedule and I really enjoyed the five talks I went to.

Things got kicked off for me listening to Kyle Simpson introduce everyone to “Blocking Across The Wire”. He explained CSP (Communicating Sequential Processes) and his experiments essentially using a websocket channel to coordinate blocking between the browser and server. This talk was refreshing and challenging and opened my eyes to a new approach to async programming. I suggest you check out Kyle’s A Tale Of Three Lists if you’re interested in seeing a comparison of async patterns.

Next up was Mike Glukhovsky with a whirlwind demo titled “RRR - React, RethinkDB, Raspberry Pi”. I’ve been interested in RethinkDB for some time and it was cool to see how changefeeds were used to display and tweet the photos taken by the hardware Mike had whipped up for Rethink HQ using a Raspberry Pi and a webcam. He gave everyone a broad overview of the three title technologies and I’m pretty sure I heard quite a few people in the audience get psyched about RethinkDB and the possibilities of being able to react to the way your data changes in real-time.

Another interesting talk, albiet a bit unexpected, was “Reactive Composition: TodoMVC Tutorial/Lab” from Scott Southworth. I have to admit that I totally thought this talk would be about RxJS and ReactiveX. Instead, Scott walked everyone through a history of the many tools of web development past and posited that maybe we should be doing more visual programming. He also demoed a very interesting real-time data analysis tool that he and his team built for monitoring an enterprise health network’s data flow. Definitely inspiring and worth a thought.

Laurie Voss’ talk, “npm and SOA: how we use SOA, and why node is good at SOA”, reinforced a lot of the ideas that the Odd Networks team has in regards to our microservice architecture. It was especially funny to see the “less simple version” of npm’s architecture, entitled “shitshow”. We’ve already begun simplifying our architecure diagrams for new hires and interviewees because of this reminder of just how crazy things can be.

The final talk I was able to attend was “React Native in Production” from Adam Miskiewicz. His talk and also getting the chance to pick the brains of a few of his coworkers at Chalk + Chisel after his talk gave us some great insight into the state of React Native for both iOS and Android. I definitely left feeling more optimistic about React Native than I had been.

Closing Thoughts

I had a lot of fun and feel like what I learned and who I got to meet was well worth the trip. I would definitely go back to Nodevember again and highly recommend it for anyone using Node.js or even looking to get a feel for the language and the thriving community around it.

You can catch all videos of the talks as the Nodevember YouTube playlist gets updated.

Wrappers, libraries, SDKs, gems — there are probably more benefits to making a simple abstraction than you realize. Ridding your application of non-business related functionality can help you focus on what matters, give back to your community, and provide an excellent way to introduce you or your company to a new audience.

A few months back MadGlory began a project that requires internet connected devices to be in real-time communication with a web application. While MG was responsible for the web application, another team was responsible for the device’s hardware, firmware, and software. Since early development on any application can involve countless changes due to domain discovery and other factors, it was decided that both the web application and the devices would use Device Cloud by Etherios as an intermediary of sorts.

There is a very large feature set, but from our point of view Device Cloud was a service that provided a common, stable, and robust API to manage our internet connected devices and get access to their data. It stored our devices’ files (logs, images, event data, etc.), provided an interface to real-time status of a device, exposed the devices’ internal API, and even allowed us to provision and update the devices remotely.

One useful feature was the Device Cloud’s virtual filesystem. The devices we were working with pushed files (JSON data) to the Device Cloud, which were in turn sent our web application as push notifications (HTTP PUT), containing the file data and some other meta information, whenever a new file event occurred. There are a TON of different events for which you can set up monitors to receive push notifications using Device Cloud’s Monitor feature, so we figured out what needed to be monitored and got to work.

Eventually, we ended up with a working example of sending and receiving data from our devices. While our code worked, it was very tightly coupled to the inner workings of the Device Cloud API. We had an entire model namespace dedicated to interacting with the Device Cloud. There were separate classes for requests, responses, modeling API objects (like Monitor or PushNotification) mixed in with our business logic of how our devices use those things. It was evident that our application was splitting its domain logic between both business needs of working with our device data and a Ruby wrapper for the Device Cloud API. Something smelled funny.

We needed to model how our application uses the Device Cloud, not how everyone else interacts with it using Ruby. In an effort to decouple that Device Cloud wrapper from our business logic, we started extracting the API models to a separate gem, and the device_cloud repository was born.

This technique is not new. Many third-party services provide SDKs, gems, libraries, or simple example code to show how to interact with their API from within a certain language. I think this is a good practice not only to provide a sort of hands-on documentation of interacting with a third-party, but also to keep your application’s code organized and relevant.

This simple change drastically helped form how I think about interacting with third-party services and what is really important to an application. First and foremost, it made our application’s code more concise. By abstracting the code that defined how the Device Cloud worked, we were left with exactly what we needed — our data. It also had a couple of interesting side-effects.

Creating the device_cloud gem allowed us to contribute some reusable code to the open-source community. This does two very powerful things.

First, it gives the contributors an avenue to publicly demonstrate their abilities. MadGlory is a consultancy and we are hired to be expert engineers. That said, it can sometimes be tough to give detailed examples of the work you or your company is capable of due to non-disclosure agreements (NDAs). Releasing your abstracted wrappers or gems can provide concrete examples. Its no secret that more and more companies are using GitHub as a resume of sorts when searching for talented employees (don’t get me started on cold-call recruiting emails). Why not leverage your company’s GitHub organization as a sort of digital portfolio?

Lastly, and maybe most importantly, you are helping your community grow. Producing simple, well-documented tools is crucial to cultivating a thriving tech community. Programming novices are encouraged with quick wins and easy-to-grok code, as DHH (video) and Farrah Bostik (video) alluded to this year at RailsConf 2014 in their keynotes. In addition, the more knowledge we can share with one another, the easier it is for everyone to build on top of those common tools and push the community forward, which is what Yehuda Katz (video) was talking about in his RailsConf keynote.

Putting aside the intangible benefits of community building and showcasing work, abstracting small bits of non-business related code has the real benefit of helping you focus your energy on the task at hand, and that is enough for me.

Email: erik@limeleaf.io