Importing a dashboard with Grafana

What I'm about to write will probably be out-of-date quite soon, as they are based on Grafana version 4.6.3, and version 5.0 is rolling out (already at the alpha stage as of this writing). In version 5.0 there will be a different, way more sensible approach of provisioning data sources and dashboards. Nevertheless, I still plan to write this down, for there is still a large user base of Grafana v4, and I do have a poor memory!

Thanks to Grafana, you can choose from tons of open source dashboards for your targets of monitoring (my personal favorite in terms of Docker host and container monitoring), instead of building everything from scratch on your own. On your Grafana UI you can import the dashboard as simply as inputting its ID and selecting its data source.

Neat, right? But what if you need to bootstrap your Grafana programmatically? UI is clearly not an option. The official way, as of Grafana v4, is through its HTTP API. Throughout the API document, however, there is only mentioning about creating/updating dashboard, not importing it, and creating a dashboard with downloaded json file did run into errors (complaining alert configuration error).

I kept digging until I found this comment, which sort of tells how the import API should be fabricated. I tried it and succeeded, but still wish it were properly documented.

Readings on error code design of RESTful services

Having written serious RESTful services to some extent, you'll end up thinking about your error codes. Chances are that you had adopted a few HTTP error codes (404, 500, etc.) in your service but realized at some point that these are just not enough to describe all possible erroneous scenarios. Because HTTP error code is essentially designed for web pages and browsers, not specifically for web services.

http://blog.restcase.com/rest-api-error-codes-101/ is a nice piece of writing in this regard. It compares two different perspectives on how we treat HTTP error codes and how we integrate them with customized error codes in the services. It also proposed a practical approach of error code design. Key aspects:

• Use only a small, well-known subset (less than 10) of HTTP status codes: 404 is good while 418 only makes sense to a handful of people.
• Return HTTP status code 2XX only if everything works out OK.
• Use HTTP status code where appropriate: 401 if authentication is needed, 404 if you couldn't find a resource (not just the API itself doesn't exist), etc.
• Consider if the client should re-submit the request without any modification after this error when choosing a HTTP error code.
• If an erroneous situation doesn't fall into any category that your chosen set of HTTP status codes could represent, return a general code (400 for client side error, 500 for server side error) in the HTTP header, and include a customized, more meaningful error code in the HTTP body.

Refer also to the following FAQs.

https://softwareengineering.stackexchange.com/questions/203492/when-to-use-http-status-code-404-in-an-api

https://stackoverflow.com/questions/942951/rest-api-error-return-good-practices

Container cluster monitoring

Prometheus would come as a very strong option when you have a cluster full of dockerized services to monitor. One nice perk that is not mentioned much is that it supports monitoring of both individual docker containers as well as a whole kubernetes environment, which allows you to go onboard when you start with scattered docker containers, and have plans to move toward k8s'ing them, which is what we are facing today.

This blog post shares some pretty comprehensive experience of Prometheus based monitoring, explaining things from various production level aspects.

Libraries and vendoring in golang

Golang has this nice vendoring feature where you could keep your import paths canonical while storing actual libraries in different places ($GOPATH and vendor). But while coding your way, you may have come across the dilemma where vendored libraries in different levels, even though being the same in actuality, are recognized as different packages by the compiler.

Say, you have a library called common, in which you imported and vendored in github.com/golang/mock. In your program prog you imported common, but also you imported and vendored in github.com/golang/mock. At this point if you go and build prog, bam, you'd get your compiler complaining about the two gomock being different packages.

The solution, long story short, is to flatten all vendored libraries into the topmost vendor folder. That is, removing vendors from common, and having github.com/golang/mock, as well as other vendors, both by common and prog, all in the vendor folder of prog.

Another similar example - https://github.com/mattfarina/golang-broken-vendor

To make things easier, this manner of practice have even got some tooling support:

https://stackoverflow.com/questions/38597046/how-to-handle-nested-vendor-directories-in-go-packages
https://github.com/kardianos/govendor/blob/master/doc/faq.md#q-how-do-i-prevent-vendor-source-from-being-checked-in
https://github.com/golang/dep/issues/985

Refer to the following discussions for more details:

https://groups.google.com/forum/#!msg/golang-nuts/AnMr9NL6dtc/UnyUUKcMCAAJ
https://www.reddit.com/r/golang/comments/4cptba/best_practice_for_vendoring_in_libraries/

A deceptive implication in using short circuit AND/OR

Take a look at two snippets of code first.

[[ ${code} -ne 0 ]] && fatal "${msg}"

and

[[ ${code} -eq 0 ]] || fatal "${msg}"

Apart from the use of short circuit AND and short circuit OR, they do pretty much the same thing, that compares the value of code and if it does not equal to 0 calls the fatal function with the value of msg.

But if this is the last line in a function, it will affect the return code of the function. Let's say the value of code indeed is 0, the first line would return a non-zero value, because the equality comparison would fail. Being equal to the conventional success value, code usually means whatever before this line succeeds. The first usage would, however, result in the function returning a failure value, which would hardly be what we want.

We could of course add something like exit 0 after this line. But being a minimalist as I am, you'd want to keep your code as simple as possible. Then use the second form. Simple as that.

Package name VS. directory name in Golang

So conventionally a golang package should be named after the directory where its code resides. What if I break the convention?

Well, you'd be frowned upon, for starters. And your code would be less readable. But it's not totally forbidden.

From a technical standpoint:

A Go package has both a name and a path. The package name is specified in the package statement of its source files; client code uses it as the prefix for the package's exported names. Client code uses the package path when importing the package. By convention, the last element of the package path is the package name.

So say you have a package that resides in the directory samplepkg, but the package name is anotherpkg. You should import from samplepkg, and refer to its exported types/functions with anotherpkg.

package main

import (
        "fmt"
        "github.com/tonyyang132/hello-world/samplepkg"
)

func main() {
        fmt.Println(anotherpkg.Msg())
}

Govendor explained

With progressing in programming in Golang, chances are that you'll deal with go package dependencies. Govendor does a pretty good job in this regard and https://zerokspot.com/weblog/2017/04/23/getting-started-with-govendor/ explains common scenarios where govender could serve with practical examples.

Golang project structures

If you are, like me, a new gopher, you'll probably be wondering how am I supposed to structure a golang project. There are packages, there are directories, there are $GOPATH, etc. Well, if that's the case, the following readings are right for you.

- https://talks.golang.org/2014/organizeio.slide includes theories as well as tricks about not only project structures but how to make your golang programming more efficiently in general, like CDPATH.

- https://dave.cheney.net/2014/12/01/five-suggestions-for-setting-up-a-go-project is more of a explainatory style, covering all aspects about code structures in detail.

Golang closures

http://keshavabharadwaj.com/2016/03/31/closure_golang/ - explains golang closure comprehensively, from anonymous functions all the way down.

Unlike https://tour.golang.org/moretypes/25 and https://gobyexample.com/closures which ranks better in google but only explains stuff with very simple examples.

HTTPS/SSL/TLS explained

A comprehensive and graceful blog article about HTTPS/SSL/TLS, with a fascinating name - The First Few Milliseconds of an HTTPS Connection

A great rabbitmq tutorial

Chances are that you have heard rabbitmq in a lot of occasions but never tried it seriously, like me. If you need help being practical, here is one great series of tutorials - https://www.rabbitmq.com/tutorials/tutorial-one-python.html.

Docker multi-host networking

If you ever have a need to run dockerized workload at scale of more than one host, but not large enough to bother with stuff like Kubernetes, chances are you'll be dealing with multi-host networking, i.e. containers situated on one host need to be able to talk with one on another host.

Of course the official document is where you want to start. But the part where they introduce Machine seems a bit extravagant, as if they were trying to promote one of their products here. Also if you already have your docker infrastructure installed, you'll be facing the necessity of modifying your setup and adding a key-value store, instead of wiping out what you've done to start all over again with the fancy Machine.

What they did with Machine in the document was simply installing docker - what you probably already have - and configuring it to talk to a key-value store. The configuration part was just adding --cluster-store and --cluster-advertise options when running dockerd.

If you are on Ubuntu, you'll want to modify /etc/default/docker and make sure the following line is enabled.

DOCKER_OPTS="--cluster-store=consul://172.16.10.31:8500 --cluster-advertise=eth0:2376"

It goes without saying that the detailed values need to be adjusted according to your own environment.

Restart dockerd to activate the new configurations. Repeat these on every docker host and you are all set to create an overlay network across your whole infrastructure.

Update: from Ubuntu 16.04 onward you may want to check /etc/docker/daemon.json instead of /etc/default/docker for docker daemon options.

Dockerfile instruction clarifications

When composing a Dockerfile, chances are that you'll face instructions that seemingly do almost same things. The following 3 pieces of great readings kindly clarify things up.

ADD vs COPY

ENTRYPOINT vs CMD

EXPOSE, -p, -P, --link

Installing macvim through Homebrew

Both are fairly common among the Mac community, so I'd assume this to be quite a legitimate scenario.

If the following is the path you go through:

1. Install Homebrew on a clean macOS with the one liner mentioned in https://brew.sh/;

2. The command line tools is installed along with Step 1;

3. Install macvim through brew (not the brew cask version);

4. An error occurs in Step 3 complaining about not able to compile without a full installation of Xcode;

5. Install Xcode through App Store;

6. Install macvim again.

Chances are that you will end up with an error like the following:

xcode-select: error: tool 'xcodebuild' requires Xcode, but active developer directory '/Library/Developer/CommandLineTools' is a command line tools instance

The root cause is pretty straightforward - besides having Xcode installed, you need to point your active developer directory to it. So what you've been missing here are:

1. Launch Xcode once, accept the license, then quit it (this is really tricky and easily neglected if you are not an Apple developer, so let's underline it);

2. Point your active developer directory to Xcode, either:
    sudo xcode-select --reset or if this doesn't help
    sudo xcode-select --switch /Applications/Xcode.app/Contents/Developer


By the way the active developer directory could be viewed with
    xcode-select --print-path

Or better yet, if you have read this note or similar readings before you even have your Homebrew installed, install Xcode first, which will (supposedly, haven't really tried) save you from the trouble of installing the command line tools.

CPU Pinning and NUMA Topology Awareness

A very thorough article about CPU pinning and NUMA topology awareness, which is what I've been doing lately:

http://redhatstackblog.redhat.com/2015/05/05/cpu-pinning-and-numa-topology-awareness-in-openstack-compute/

Also some background information about tickless kernel:

http://www.breakage.org/2013/11/15/nohz_fullgodmode/