Docker Outside of Docker
I’ve been working with Docker a ton lately, and recently decided to deploy Jenkins on Alpine within a compose of other private repositories/registries/etc.
However, as soon as I wanted to use Docker within a Jenkins Pipeline, things went south really fast.
So, after getting the container running (which was easy), as soon as I tried any Docker commands on either CLI or via Jenkins, I was met with the following error:
ash: docker: not found
Duh, of course! The interwebs led me down the road of the obvious of needing Docker installed in the container. But I didn’t want to maintain a docker daemon to deploy Jenkins, which also had a docker version inside that needed to progress as my builds progress.
But it also says you can mount the docker binary and docker.sock, and should be gravy.
At this point I have the following:
Docker Run Equivalent
docker run -it –restart always -v /var/run/docker.sock:/var/run/docker.sock -v /usr/bin/docker:/usr/bin/docker jenkins:2.32.2-alpine
jenkins: image: jenkins:2.32.2-alpine restart: always ports: - 8080:8080 volumes: - /usr/bin/docker:/usr/bin/docker - /var/run/docker.sock:/var/run/docker.sock
But when attempting another Docker operation, I’m still met with the same error.
I can see the binary and socket inside the container, and as expected they are owned by root – but I’d expect an error about permissions, not “not found” errors.
It just didn’t make sense, so suddenly I remembered the ldd command, which will show static and dynamically linked object dependencies:
$ ldd /usr/bin/docker
linux-vdso.so.1 => (0x00007fff6691a000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fd2bd893000)
libltdl.so.7 => /usr/lib/x86_64-linux-gnu/libltdl.so.7 (0x00007fd2bd689000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd2bd2bf000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fd2bd0bb000)
As you can see, some are linked directly to a memory address, while some indicate a file! Ah-ha! So let’s update our orchestration:
Docker Run Equivalent
docker run -it –restart always -v /lib/x86_64-linux-gnu/libc.so.6:/lib/x86_64-linux-gnu/libc.so.6 -v /lib/x86_64-linux-gnu/libdl.so.2:/lib/x86_64-linux-gnu/libdl.so.2 -v /lib/x86_64-linux-gnu/libpthread.so.0:/lib/x86_64-linux-gnu/libpthread.so.0 -v /lib64/ld-linux-x86-64.so.2:/lib64/ld-linux-x86-64.so.2 -v /usr/bin/docker:/usr/bin/docker -v /usr/lib/x86_64-linux-gnu/libltdl.so.7:/usr/lib/x86_64-linux-gnu/libltdl.so.7 -v /var/run/docker.sock:/var/run/docker.sock jenkins:2.32.2-alpine
jenkins: image: jenkins:2.32.2-alpine restart: always ports: - 8080:8080 volumes: - /lib/x86_64-linux-gnu/libc.so.6:/lib/x86_64-linux-gnu/libc.so.6 - /lib/x86_64-linux-gnu/libdl.so.2:/lib/x86_64-linux-gnu/libdl.so.2 - /lib/x86_64-linux-gnu/libpthread.so.0:/lib/x86_64-linux-gnu/libpthread.so.0 - /lib64/ld-linux-x86-64.so.2:/lib64/ld-linux-x86-64.so.2 - /usr/bin/docker:/usr/bin/docker - /usr/lib/x86_64-linux-gnu/libltdl.so.7:/usr/lib/x86_64-linux-gnu/libltdl.so.7 - /var/run/docker.sock:/var/run/docker.sock
Not Quite There
Once more, running the pipeline results in an error, but this time it’s the error we expect:
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.25/containers/json: dial unix /var/run/docker.sock: connect: permission denied
In case you weren’t aware, the Jenkins container runs as a jenkins user, which clearly doesn’t have access to the socket.
Stumbling more around the interwebs, I found an article indicating their trials and tribulations on this exact topic; but their solution was to add sudo to the image. Oh, and it’s also where I got the phrase “Docker Outside of Docker.”
There’s 2 issues I have with this:
- They blindly give the jenkins user full sudo, which pretty much avoids the use of a non-suid user. Sure, it CAN be tweaked for ONLY docker commands, but…
- I have been using the practice of not using sudo on commands unless 100000% necessary for as long as I can remember; I’d guess well over 15 years.
- Also, I should state that the Jenkins Freestyle Pipeline does not operate with sudo, and I wasn’t ready to delve into proper Jenkinsfile construction for advanced pipelines just yet. SO MUCH TO DO!
At this point, the only thing I can think of is adding a GUID inside the container that matches the docker GUID in the host.
I’ll save you the details, but I came across this StackOverflow post that conveniently had a script for setting up this matching for us, which I’ve heavily modified here. Use it with the configurations listed above.
The code should be very simple to follow, but essentially it:
- ensures the docker socket exists and is actually a socket.
- ensures the docker binary exists and is functional.
- Remember to use ldd $(which docker) to see if you have any dynamically linked deps
- ensures a group with GID of the docker socket exists inside the container, and creates a docker group should it not exist.
- ensures the desired docker user exists in the aforementioned group GID, and of course adds it if not.
BUT WAIT, THERE’S MORE!
Yeah so… What had happened was… It’s not gonna work. Not yet.
You need to restart the docker daemon for it to accept the new user access. THEN you can use docker inside your container.
This entry was posted by killerspaz on March 30, 2017 at 11:26 pm, and is filed under Containers, Docker, Uncategorized. Follow any responses to this post through RSS 2.0. You can leave a response or trackback from your own site.
I experienced a similar problem with Jenkins 2.60.1, with docker in the Jenkins container not being able to find libltdl. I found that downgrading to Jenkins 2.50 fixed it, and I didn’t have to add all the shared libraries