Docker: docker-compose in swarm mode give ro filesystem

snap
1

I was testing out docker swarm using 3 ubuntu servers with the Docker snap. I was using the guide here:

https://docs.docker.com/engine/swarm/stack-deploy/

Is there any way to configure docker-compose or the ‘docker stack deploy’ to use a different location, or open it up to be read-write when using docker from the snap?

docker version
Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.4
 Git commit:        e68fc7a
 Built:             Tue May  7 17:57:34 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.4
  Git commit:       e68fc7a
  Built:            Tue May  7 17:57:34 2019
  OS/Arch:          linux/amd64
  Experimental:     false


docker stack deploy --compose-file docker-compose.yml stackdemo
Ignoring unsupported options: build

Creating network stackdemo_default
Creating service stackdemo_web
Creating service stackdemo_redis

docker service ps stackdemo_web --no-trunc
ID                          NAME                  IMAGE                                                                                                        NODE                DESIRED STATE       CURRENT STATE                     ERROR                                            PORTS
56ncig1i2i12xrfm8plapn8pf   stackdemo_web.1       swarm-master:5000/stackdemo:latest@sha256:b4fff10b4ec76f703aeb0550023af9d531f49a85e9246460c0ccbe91fac1486b   swarm-node1         Ready               Assigned less than a second ago                                                    
j4czjqey1w5ge3qi6tqz9h6zp    \_ stackdemo_web.1   swarm-master:5000/stackdemo:latest@sha256:b4fff10b4ec76f703aeb0550023af9d531f49a85e9246460c0ccbe91fac1486b   swarm-master        Shutdown            Rejected 4 seconds ago            "mkdir /var/lib/docker: read-only file system"   
nta5hfxpzumq1gfq4k9k22tpd    \_ stackdemo_web.1   swarm-master:5000/stackdemo:latest@sha256:b4fff10b4ec76f703aeb0550023af9d531f49a85e9246460c0ccbe91fac1486b   swarm-master        Shutdown            Rejected 9 seconds ago            "mkdir /var/lib/docker: read-only file system"   
r2x8zvd6g40h8mnyj8mp2i0kw    \_ stackdemo_web.1   swarm-master:5000/stackdemo:latest@sha256:b4fff10b4ec76f703aeb0550023af9d531f49a85e9246460c0ccbe91fac1486b   swarm-master        Shutdown            Rejected 15 seconds ago           "mkdir /var/lib/docker: read-only file system"   
vqh5djso41pys9k71x81wuuge    \_ stackdemo_web.1   swarm-master:5000/stackdemo:latest@sha256:b4fff10b4ec76f703aeb0550023af9d531f49a85e9246460c0ccbe91fac1486b   swarm-master        Shutdown            Rejected 19 seconds ago           "mkdir /var/lib/docker: read-only file system"   
vir9h0gl7ruruv0ex1d2afks8    \_ stackdemo_web.1   swarm-master:5000/stackdemo:latest@sha256:b4fff10b4ec76f703aeb0550023af9d531f49a85e9246460c0ccbe91fac1486b   swarm-master        Shutdown            Rejected 25 seconds ago           "mkdir /var/lib/docker: read-only file system"
2

Hi,

This seems to be the same issue as Call for testing of the docker snap and I’m still unaware of where docker gets the /var/lib/docker path from and why it’s not trying to use $SNAP_COMMON/var-lib-docker/ instead which is writable.

3

Hello @ijohnson,

I’d be willing to help, but I am unfamiliar with how to get the Docker snap code. Would you be willing to help me help the project? If you could point me in the direction on how to download the code for review, I can go through the documentation on how to build the snap for testing.

I found https://github.com/docker/docker-snap and https://code.launchpad.net/~docker/+git/snap but neither of these look to be under current development.

4

The latter link on launchpad is actually still under development on the master branch, though we had to release a fix overwriting that code due to an unexpected issue. If you are unfamiliar with using launchpad, checkout my comment here Call for testing of the docker snap

5

Thanks.

I’ll take a look and see what I can find.

6

I know this is an old topic, but I’m back at this 20.04 machine and am having issues with swarm configs again.

I am using this swarm file.

---
version: "3"
services:
  bookstack:
    image: linuxserver/bookstack
    container_name: bookstack
    environment:
      - PUID=1000
      - PGID=1000
      - DB_HOST=bookstack_db
      - DB_USER=bookstack
      - DB_PASS=yourdbpass
      - DB_DATABASE=bookstackapp
    volumes:
      - /path/to/data:/config
    ports:
      - 6875:80
    restart: unless-stopped
    depends_on:
      - bookstack_db
  bookstack_db:
    image: linuxserver/mariadb
    container_name: bookstack_db
    environment:
      - PUID=1000
      - PGID=1000
      - MYSQL_ROOT_PASSWORD=yourdbpass
      - TZ=Europe/London
      - MYSQL_DATABASE=bookstackapp
      - MYSQL_USER=bookstack
      - MYSQL_PASSWORD=yourdbpass
    volumes:
      - /path/to/data:/config
    restart: unless-stopped

with the volumes being bind mounts to my home directory, and as volumes in portainer.

In both cases I get

mkdir /var/lib/docker read-only file system

I’m at a loss to know where to go from here without having to rebuild a heap of stuff.

If I do have no remove the snap version and start again, is there a way to disable snap system wide… It doesn’t work with lsd either so is of no use to me at all.

7

I still don’t know why this happens, I looked again and my best guess is (still) that there is a specific bug in docker swarm that makes it not observe the --data-root argument we use to launch dockerd.

I was able to work-around the issue by removing --data-root argument from the command line we launch dockerd with, and adding a layout on /var/lib/docker to the snap:

layout:
  /etc/docker:
    bind: $SNAP_DATA/etc/docker
  /var/lib/docker:
    bind: $SNAP_COMMON/var-lib-docker

I did not do much more testing of the docker snap with this, perhaps that doesn’t work for other reasons but it at least works to run some basic containers and avoids this problem.

@tianon, WDYT of the proposed patch above ? It seems to fix a long standing bug with docker swarm with the docker snap

1 Like
8

@ijohnson, I realize this post is somewhat old. But I hope the following could help resolving this or similar issues. With that said, it seems unlikely that the bug is in docker swarm. I extracted docker_576.snap into a temp folder and executed dockerd in foreground using the same command snap uses.

$ mkdir docker_576 && cd docker_576
$ sudo snap disable docker
docker disabled
$ sudo cp /var/lib/snapd/snaps/docker_576.snap ./
$ unsquashfs ./docker_576.snap

The parameters for the command were captured by “ps -aux | grep docker” while dockerd from snap was running.

$ sudo su -
# ./squashfs-root/bin/dockerd --group docker --exec-root=/run/snap.docker --data-root=/var/snap/docker/common/var-lib-docker --pidfile=/run/snap.docker/docker.pid --config-file=/var/snap/docker/576/config/daemon.json

The process running in foreground and shows the stdout. In a new terminal I can list the processes:

ps -aux | grep -vE "vscode|grep" | grep docker
root      376703  2.5  0.4 2922408 159236 pts/2  Sl+  04:14   0:53 ./dockerd --group docker --exec-root=/run/snap.docker --data-root=/var/snap/docker/common/var-lib-docker --pidfile=/run/snap.docker/docker.pid --config-file=/var/snap/docker/576/config/daemon.json
root      376724  0.5  0.1 3037244 49820 ?       Ssl  04:14   0:11 containerd --config /run/snap.docker/containerd/containerd.toml --log-level error

Then I initialized swarm and executed my test stack.

$ ./squashfs-root/bin/docker swarm init
Swarm initialized: current node (nnrv94zhg8kz4e4lt2tk4mir1) is now a manager.
[..omitted..]
$ ./squashfs-root/bin/docker stack deploy --compose-file ./docker-compose.yaml dms
[..omitted..]

Stack came up with some errors, but all errors related to issues in my docker-compose file.

./squashfs-root/bin/docker stack ls
NAME                SERVICES            ORCHESTRATOR
dms                 10                  Swarm

./squashfs-root/bin/docker stack services dms
ID                  NAME                          MODE                REPLICAS            IMAGE                                              PORTS
7gsrvyzvn26j        dms_zabbix-server             replicated          1/1                 zabbix/zabbix-server-mysql:ubuntu-5.0-latest       *:10051->10051/tcp
av0cm3jcnzvv        dms_zabbix-snmptraps          replicated          1/1                 zabbix/zabbix-snmptraps:ubuntu-5.0-latest          *:162->1162/udp
[..omitted..]

./squashfs-root/bin/docker stack ps dms
ID                  NAME                                         IMAGE                                              NODE                DESIRED STATE       CURRENT STATE             ERROR                       PORTS
vu1rh6n3sd78        dms_zabbix-agent.h2u25r9ix81va827x41ztigzr   zabbix/zabbix-agent:ubuntu-5.0-latest              eth256              Running             Running 47 seconds ago
[..omitted..]

After seeing stack come up, I checked /var/lib/docker. This was original error; however, it does not exist. So, I assume dockerd did honor the --data-root argument.

$ sudo ls -ld /var/lib/docker
ls: cannot access '/var/lib/docker': No such file or directory

Going back to the snap. I removed the stack and left the swarm.

$ ./squashfs-root/bin/docker stack rm dms
[..omitted..]
$ ./squashfs-root/bin/docker swarm leave -f
Node left the swarm.

Killed the foreground process. The ps command shows blank output, hence wc for visual.

$ ps -aux | grep -vE "vscode|grep" | grep docker | wc
      0       0       0

Enabled docker snap, initialized swarm, and deployed stack.

$ sudo snap enable docker 
docker enabled
$ which docker
/snap/bin/docker

$ docker swarm init 
Swarm initialized: current node (hn7ob8317t5f2jxosd8inkcp5) is now a manager.
[..omitted..]
docker stack deploy --compose-file ./docker-compose.yaml dms
[..omitted..]

$ docker stack ls
NAME                SERVICES            ORCHESTRATOR
dms                 10                  Swarm

$ docker stack services dms
ID                  NAME                          MODE                REPLICAS            IMAGE                                              PORTS
0u13t6rwz59q        dms_zabbix-web-apache-mysql   replicated          0/1                 zabbix/zabbix-web-apache-mysql:ubuntu-5.0-latest   *:80->8080/tcp, *:443->8443/tcp
2ema42zlnjng        dms_mysql-server              replicated          0/1                 mysql:8.0                                          
4u3lfblwgh48        dms_zabbix-agent              global              0/1                 zabbix/zabbix-agent:ubuntu-5.0-latest              *:10050->10050/tcp
9sfmrb9yxsse        dms_zabbix-proxy-mysql        replicated          0/1                 zabbix/zabbix-proxy-mysql:ubuntu-5.0-latest        *:10071->10051/tcp
jecmnd2bmzpc        dms_zabbix-proxy-sqlite3      replicated          0/1                 zabbix/zabbix-proxy-sqlite3:ubuntu-5.0-latest      *:10061->10051/tcp
lqadk0en4kxt        dms_zabbix-web-nginx-mysql    replicated          0/1                 zabbix/zabbix-web-nginx-mysql:ubuntu-5.0-latest    *:8081->8080/tcp, *:8443->8443/tcp
r7qwiifynm70        dms_zabbix-snmptraps          replicated          0/1                 zabbix/zabbix-snmptraps:ubuntu-5.0-latest          *:162->1162/udp
u3qqlb1y64um        dms_zabbix-server             replicated          0/1                 zabbix/zabbix-server-mysql:ubuntu-5.0-latest       *:10051->10051/tcp
uhfybrbptwep        dms_db_data_mysql             replicated          0/1                 busybox:latest                                     
zltoa796hd54        dms_zabbix-java-gateway       replicated          0/1                 zabbix/zabbix-java-gateway:ubuntu-5.0-latest       *:10052->10052/tcp

$ docker stack ps dms
ID                  NAME                                             IMAGE                                              NODE                DESIRED STATE       CURRENT STATE                     ERROR                              PORTS
wck8q917vxrd        dms_zabbix-agent.hn7ob8317t5f2jxosd8inkcp5       zabbix/zabbix-agent:ubuntu-5.0-latest              eth256              Ready               Rejected less than a second ago   "mkdir /var/lib/docker: read-o…"   
9z5vpev29qvj         \_ dms_zabbix-agent.hn7ob8317t5f2jxosd8inkcp5   zabbix/zabbix-agent:ubuntu-5.0-latest              eth256              Shutdown            Rejected 5 seconds ago            "mkdir /var/lib/docker: read-o…"   
[..omitted..]

$ docker inspect 0u13t6rwz59q --format='{{json .Status}}' | jq
{
  "Timestamp": "2020-10-20T02:17:33.653518547Z",
  "State": "rejected",
  "Message": "preparing",
  "Err": "mkdir /var/lib/docker: read-only file system", <----------
  "ContainerStatus": {
    "ContainerID": "",
    "PID": 0,
    "ExitCode": 0
  },
  "PortStatus": {}
}

Got back to the same error. All of this was ran on snap docker_576.

$ snap list docker
Name    Version   Rev  Tracking     Publisher   Notes
docker  19.03.13  576  latest/edge  canonical✓  -

Obviously in the above example dockerd had full access to /var/lib/ (ran as root) and choose not to run mkdir /var/lib/docker and honor --data-root argument. Which makes me think that the issue is somehow related to snap itself. Unless there is a flaw in my logic or I made an error somewhere above. Please let me know what you think. Looking forward to hearing from you.

9

Could you check that all the same environment variables were defined as are defined in the wrapper that the snap uses for dockerd? See docker-snap/bin/dockerd-wrapper at main · canonical/docker-snap · GitHub