Docker Volumes on the cheap

Docker volumes plugins have an API. When creating a volume plugin, in addition to implement the API, you will also have to implement the volume API. On the other hand, there are a large number of fuse-based implementations for remote storages of various sorts. This post is about leveraging these implementations as pseudo-volumes, while performing the mount in a container. This comes at the cost and complexity of sharing a well-known directory on the host between the container that will perform the mount, with the container(s) that use the mount.

Capabilities

The first step to manage and understand is how to give away enough rights to the container that will perform the mount so that other processes on the host (outside that container) will be able to access the files and directories. This can be achieved through using the following options for docker run:

  • Give the fuse device to your container through --device /dev/fuse
  • Raise its capabilities to bypass some of the security layers through: --cap-add SYS_ADMIN and --security-opt "apparmor=unconfined"
  • Mount the local host file system into the container in a way that it can be shared back with other processes and containers using rshared.

Unmounting

You also want to properly unmount when the container gracefully terminates. This is achieved through a combination of tini and proper trap in the shell. The reason for this is that your are likely to interface the fuse implementation through some sort of entrypoint

tini

Use tini as the main entrypoint in your Dockerfile and arrange to give it the -g option so that it properly propagates signals to the entire mounting system in the container. tini exists as an Alpine package which can be helpful in keeping down the size of the image.

Cleanup

In order to clean properly, you can arrange for a shell to be called at the end of your entrypoint implementation, perhaps after having checked that the mount performed as it should. This shell will trap the INT and TERM signals, unmount the volume (lazily) and propagate these to the mount process. Without tini, you would never have received these signals, by Docker construction and design.

Examples

I have made available two example images following these principles:

  • webdav-client mounts WebDAV resources and leverages the full potential of davfs2, being able to leverage all the options supported by davfs2. This is in contrast to the davfs volume which also uses davfs2 under the hood, but misses some of the configuration options.
  • s3fs mounts a remote S3 bucket using the fuse implementation with the same name. It supports all the official versions of the original fuse projects through tags

2 min read

All tags of a Docker (Hub) image

The registry behind the Docker hub has an API. You can use this API to list out all the tags for a given existing image using code similar to the following. This heavily deviates from the original to enable the use of curl (preferred) or wget and without using jq, but rather relying on standard linux command-line tooling.

# This is the image that you wish to list the tags for
im="abiosoft/caddy"

if [ -z "$(echo "$im" | grep -o '/')" ]; then
    hub="https://registry.hub.docker.com/v2/repositories/library/$im/tags/"
else
    hub="https://registry.hub.docker.com/v2/repositories/$im/tags/"
fi

# Get number of pages
if [ -z "$(command -v curl)" ]; then
    first=$(wget -q -O - $hub)
else
    first=$(curl -sL $hub)
fi
count=$(echo $first | sed -E 's/\{\s*"count":\s*([0-9]+).*/\1/')
pagination=$(echo $first | grep -Eo '"name":\s*"[a-zA-Z0-9_.-]+"' | wc -l)
pages=$(expr $count / $pagination + 1)

# Get all tags one page after the other
tags=
i=0
while [ $i -le $pages ] ;
do
    i=$(expr $i + 1)
    if [ -z "$(command -v curl)" ]; then
        page=$(wget -q -O - "$hub?page=$i")
    else
        page=$(curl -sL "$hub?page=$i")
    fi
    ptags=$(echo $page | grep -Eo '"name":\s*"[a-zA-Z0-9_.-]+"' | sed -E 's/"name":\s*"([a-zA-Z0-9_.-]+)"/\1/')
    tags="${ptags} $tags"
done

# Once here, the variable tags should contain the list of all tags for the image.

This code can be used in hooks to write complex build/push instructions. Such instructions can be used to automatically enhanced a standard image with additional features in a future-proof way: for every new version of the original image, watching it will ensure that your enhanced version will get built.

1 min read

Docker Secrets in Grafana

As my PR has now been merged, the latest official Grafana Docker image (still in beta) has now support for Docker secrets. Using this version, you should be able to write stack files similar to the following (shortened) one, provided you have the password for the main administation user stored in the file at config/grafana/admin.pwd.

1 min read

Solving Dependencies in Docker Swarm

When/If moving from Docker Compose files to Stack files in Docker Swarm, you might have problems solving dependencies and especially starting order as depends_on is not supported when deploying a stack in swarm mode. One solution is to arrange for your services to implement a network server and wait for this port to be opened and respond before going on with running a given service.

2 min read

Accessing Containers from Remote

Sometimes (oftentimes?), when debugging Docker based setup, you wish you could access a service running in a container from the outside. Typically, these would be services that are internal to the cluster architecture, but should not be exposed for remote access, e.g. a database, a pub/sub queue, etc. Luckily, the only thing you really need is access to a remote SSH server and accept to (temporarily) add an SSH client to your container. Here is how, documented for an alpine container, but steps will be the same for containers based on other distributions.

2 min read

Git LFS on Ubuntu

Git LFS enables the storage of large files out of the main repository, replacing them by an index and reference to remote storage instead.

1 min read

concocter v1.0

concocter has just reached an official v1.0 release. concocter is my own take on the init process in containers. It offers features already found in other solutions such as supervisord or docker-gen with enough twists for justifying the effort of writing yet another tool in a similar vein.

1 min read

Plexgear AC600 Nano

I recently bought a Plexgear AC600 Nano USB wifi dongle to bring some life to an old Intel NUC that was lying around unused. Unfortunately, and unlike as advertised, it does not work under the latest version of Ubuntu. The linux drivers provided by the vendor do not work with the latest version of the kernel.

1 min read

Docker API in Tcl

The main goal of my Docker API implementation in Tcl is to cover most of the official API while providing a programming interface that feels Tcl-ish. To that end, it builds upon the Tk-syle of programming that creates a context object and then creates a command with the same name as the object to perform most further operations.

1 min read