Importing a dashboard with Grafana

What I'm about to write will probably be out-of-date quite soon, as they are based on Grafana version 4.6.3, and version 5.0 is rolling out (already at the alpha stage as of this writing). In version 5.0 there will be a different, way more sensible approach of provisioning data sources and dashboards. Nevertheless, I still plan to write this down, for there is still a large user base of Grafana v4, and I do have a poor memory!

Thanks to Grafana, you can choose from tons of open source dashboards for your targets of monitoring (my personal favorite in terms of Docker host and container monitoring), instead of building everything from scratch on your own. On your Grafana UI you can import the dashboard as simply as inputting its ID and selecting its data source.

Neat, right? But what if you need to bootstrap your Grafana programmatically? UI is clearly not an option. The official way, as of Grafana v4, is through its HTTP API. Throughout the API document, however, there is only mentioning about creating/updating dashboard, not importing it, and creating a dashboard with downloaded json file did run into errors (complaining alert configuration error).

I kept digging until I found this comment, which sort of tells how the import API should be fabricated. I tried it and succeeded, but still wish it were properly documented.

Readings on error code design of RESTful services

Having written serious RESTful services to some extent, you'll end up thinking about your error codes. Chances are that you had adopted a few HTTP error codes (404, 500, etc.) in your service but realized at some point that these are just not enough to describe all possible erroneous scenarios. Because HTTP error code is essentially designed for web pages and browsers, not specifically for web services.

http://blog.restcase.com/rest-api-error-codes-101/ is a nice piece of writing in this regard. It compares two different perspectives on how we treat HTTP error codes and how we integrate them with customized error codes in the services. It also proposed a practical approach of error code design. Key aspects:

• Use only a small, well-known subset (less than 10) of HTTP status codes: 404 is good while 418 only makes sense to a handful of people.
• Return HTTP status code 2XX only if everything works out OK.
• Use HTTP status code where appropriate: 401 if authentication is needed, 404 if you couldn't find a resource (not just the API itself doesn't exist), etc.
• Consider if the client should re-submit the request without any modification after this error when choosing a HTTP error code.
• If an erroneous situation doesn't fall into any category that your chosen set of HTTP status codes could represent, return a general code (400 for client side error, 500 for server side error) in the HTTP header, and include a customized, more meaningful error code in the HTTP body.

Refer also to the following FAQs.

https://softwareengineering.stackexchange.com/questions/203492/when-to-use-http-status-code-404-in-an-api

https://stackoverflow.com/questions/942951/rest-api-error-return-good-practices

Container cluster monitoring

Prometheus would come as a very strong option when you have a cluster full of dockerized services to monitor. One nice perk that is not mentioned much is that it supports monitoring of both individual docker containers as well as a whole kubernetes environment, which allows you to go onboard when you start with scattered docker containers, and have plans to move toward k8s'ing them, which is what we are facing today.

This blog post shares some pretty comprehensive experience of Prometheus based monitoring, explaining things from various production level aspects.

Libraries and vendoring in golang

Golang has this nice vendoring feature where you could keep your import paths canonical while storing actual libraries in different places ($GOPATH and vendor). But while coding your way, you may have come across the dilemma where vendored libraries in different levels, even though being the same in actuality, are recognized as different packages by the compiler.

Say, you have a library called common, in which you imported and vendored in github.com/golang/mock. In your program prog you imported common, but also you imported and vendored in github.com/golang/mock. At this point if you go and build prog, bam, you'd get your compiler complaining about the two gomock being different packages.

The solution, long story short, is to flatten all vendored libraries into the topmost vendor folder. That is, removing vendors from common, and having github.com/golang/mock, as well as other vendors, both by common and prog, all in the vendor folder of prog.

Another similar example - https://github.com/mattfarina/golang-broken-vendor

To make things easier, this manner of practice have even got some tooling support:

https://stackoverflow.com/questions/38597046/how-to-handle-nested-vendor-directories-in-go-packages
https://github.com/kardianos/govendor/blob/master/doc/faq.md#q-how-do-i-prevent-vendor-source-from-being-checked-in
https://github.com/golang/dep/issues/985

Refer to the following discussions for more details:

https://groups.google.com/forum/#!msg/golang-nuts/AnMr9NL6dtc/UnyUUKcMCAAJ
https://www.reddit.com/r/golang/comments/4cptba/best_practice_for_vendoring_in_libraries/

A deceptive implication in using short circuit AND/OR

Take a look at two snippets of code first.

[[ ${code} -ne 0 ]] && fatal "${msg}"

and

[[ ${code} -eq 0 ]] || fatal "${msg}"

Apart from the use of short circuit AND and short circuit OR, they do pretty much the same thing, that compares the value of code and if it does not equal to 0 calls the fatal function with the value of msg.

But if this is the last line in a function, it will affect the return code of the function. Let's say the value of code indeed is 0, the first line would return a non-zero value, because the equality comparison would fail. Being equal to the conventional success value, code usually means whatever before this line succeeds. The first usage would, however, result in the function returning a failure value, which would hardly be what we want.

We could of course add something like exit 0 after this line. But being a minimalist as I am, you'd want to keep your code as simple as possible. Then use the second form. Simple as that.