Docker is the nice tool for almost every use case in my sphere. It’s easy to use, it’s fast to build and deploy. Docker can be used with miscellaneous storage drivers such as btrfs, datamapper, overlayfs, aufs. A long time I was using docker with btrfs backend and everything seems to be nice, but when load on this server increased, corrupted layers are began to appear.

It looks like build process in the next RUN step unable to find files from previous step. I had no desire to dig into it because of time, so I decided to change storage backend. It may looks like the choice is big enough, in fact every variant is full of surprises. Some of them is still not in the linux kernel, some of them have serious bugs, and others is just slow.

Some time ago I heard abount LVM thin provisioning. It’s relatively new feature and I had no use cases to use it. Now I have it, because docker can work with it. You can create Virtual Group directly on block device and pass it to docker daemon. It works fast enough to hold many parallel docker builds in a single moment in time. So, let’s try it.

Creating Virtual Group docker:

pvcreate /dev/sdb
vgcreate docker /dev/sdb

Now we shall create a volume for metadata. man 8 lvcreate gives us a formula for calculating a size for metadata volume.

(Pool_LV_size / Pool_LV_chunk_size * 64b)

Default chink size is 64KiB, now lets find desired metadata size.

echo $[799057190584.31995 / 65536 * 64]
  780329287.67999995
echo $[780329287.67999995/1024/1024]
  744.17999999999995

780329287.67999995 - The VG size in bytes (744GiB) 65536 - chunk size in bytes (64KiB)

Now creating metadata volume with calculated size.

lvcreate -n docker-pool-meta -L 744M docker
  Logical volume "docker-pool-meta" created.

Next, we need to create a volume for data. In my situations I just use the rest size of virtual group. First, I was trying to create volume this way:

lvcreate -n docker-pool-data -l 100%FREE docker
  Logical volume "docker-pool-data" created.

It’s successfully created, but It fails in the next step when we need to convert this two volumes to the thin pool.

lvconvert --type thin-pool --poolmetadata docker/docker-pool-meta docker/docker-pool-data
  WARNING: Converting logical volume docker/docker-pool-data and docker/docker-pool-meta to pool's data and metadata volumes.
  THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
Do you really want to convert docker/docker-pool-data and docker/docker-pool-meta? [y/n]: y
  Volume group "docker" has insufficient free space (0 extents): 38102 required.

If in your case everything is fine, my congratulations, you can skip this part.

Looks like when pool is being creating lvm needs some extra space, in my situation it’s 38102 extents. Okay, so I need to create volume with size smaller on 38102 extents. Lets find the virtual group size in extents.

vgs -o +vg_free_count,vg_extent_count
  VG     #PV #LV #SN Attr   VSize   VFree  Free #Ext
  docker   1   2   0 wz--n- 744,18g     0     0 190511

Fine, now lets see how many extents uses metadata volume.

lvs -o +vg_extent_count
  LV               VG     Attr       LSize    #Ext
  docker-pool-meta docker twi-aotz-- 744,0m   185

Now, it’s easy to find desired data volume size.

echo $[190511-38102-185]
  152224

lvremove docker/docker-pool-data
  Logical volume "docker-pool-data" successfully removed

lvcreate -n docker-pool-data -l 152224 docker
  Logical volume "docker-pool-data" created.

Time to convert it.

lvconvert --type thin-pool --poolmetadata docker/docker-pool-meta docker/docker-pool-data
  WARNING: Converting logical volume docker/docker-pool-data and docker/docker-pool-meta to pool's data and metadata volumes.
  THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)

/usr/bin/docker daemon -H unix:///run/docker.sock -H tcp://0.0.0.0:2375 --dns=172.17.0.1 --storage-driver=devicemapper --storage-opt dm.thinpooldev=/dev/mapper/docker-docker--pool--data --storage-opt=dm.use_deferred_removal=true --storage-opt=dm.mountopt=noatime --storage-opt=dm.fs=xfs

At this time I’m actively testing this solution with thin pools so this article may be updated from time to time If I’ll find some new problems or features. In fact as I think, this is the most acceptable solution for now in ratio speed/safety. Maybe soon we will see something more interesting, I hope so.