2018年6月18日星期一

Number of forks causing Ansible playbook to fail

TL;DR - The defaults.forks settings from /etc/ansible/ansible.cfg affects the parallelism of a playbook running. If a play that involves complicated wait_for logic hangs when there is a relatively high number of hosts that the play runs against while running fine when the number is low, it could be that the number of forks is exhausted and the wait_for condition could not be reached.

Rationale -

We have this Ansible role to install a private IPFS cluster, in which we wait_for the bootstrapping peer to have fully started before installing other peers. The detailed sequence is -


  1. All other peers wait while the bootstrapping peer (the first one in the cluster in our case) is being deployed - configuration files generated, docker container started and service started.
  2. After service started, the bootstrapping peer prints its ID to a file.
  3. All other peers' wait is unlocked by the bootstrapping peer's ID file, their entry point script (in which the bootstrapping peer's ID is referred to by ipfs bootstrap add) generated along with other configuration files, their containers and services started.


The default value of defaults.forks is 5. If we have, say, 7 hosts that IPFS needs to run on, chances are that all 5 forks would block on the wait_for part. There would then be no more fork for the bootstrapping peer to finish its tasks, and then wait_for would not unlock until the default timeout of 300 seconds is reached and the playbook fails.

Increasing the defaults.forks value to 7 solved our problem.

P.S. It was by reading the official document for the second time that I realized exhaustion of defaults.forks was causing our IPFS installation failure. So many thanks to the Ansible team and hey, do read!

2018年6月15日星期五

Dump HTTP request with golang

httputil.DumpRequest works like a charm if you would like to dump an HTTP request to its string representation in order to, say, tell precisely what information a front-end application sends to a back-end application. It takes into consideration regenerating an exact same body if it's instructed to dump the body (the second parameter), keeping subsequent operations intact.

2018年6月7日星期四

How to wait_for a remote file with Ansible

I've recently realized that the wati_for module of Ansible is a bit tricky when it comes to waiting for a remote file, i.e. a file that's supposed to be present or absent on a host other than the one that's running the Ansible task.

If you check the doc, there is this host attribute which is rather intriguing. But when you do put it on a test drive, it won't work. My understanding is host is meant for paring with port, together checking the readiness of, say, a restful service.

The correct way then to wait for a remote file is through delegate_to, without further ado.

2018年6月3日星期日