Downloading Docker Images from Docker Hub without using Docker

32

18

I want to manually download a Docker Image from Docker Hub. More specifically, I want to download a Docker Image from Docker Hub on a machine in a restricted environment which does not (and cannot) have the Docker client software installed. I would have thought that this would be possible using the official API, but this does not appear to be the case - see the following discussion:

Is it really the case that the API doesn't support downloading images? Is there a way to work around this?


UPDATE 1:

I came across the following ServerFault post:

The accepted solution uses the docker save command, which doesn't help in my situation. But another solution posted there cites the following StackOverflow post:

One of the solutions there refers to a command-line tool called docker-registry-debug which, among other things, can generate a curl command for downloading an image. Here is what I got:

user@host:~$ docker-registry-debug curlme docker ubuntu

# Reading user/passwd from env var "USER_CREDS"
# No password provided, disabling auth
# Getting token from https://index.docker.io
# Got registry endpoint from the server: https://registry-1.docker.io
# Got token: signature=1234567890abcde1234567890abcde1234567890,repository="library/docker",access=read
curl -i --location-trusted -I -X GET -H "Authorization: Token signature=1234567890abcde1234567890abcde1234567890,repository="library/docker",access=read" https://registry-1.docker.io/v1/images/ubuntu/layer

user@host:~$ curl \
-i --location-trusted -I -X GET \
-H "Authorization: Token signature=1234567890abcde1234567890abcde1234567890,repository="library/docker",access=read" 

https://registry-1.docker.io/v1/images/ubuntu/layer
HTTP/1.1 404 NOT FOUND
Server: gunicorn/18.0
Date: Wed, 29 Nov 2017 01:00:00 GMT
Expires: -1
Content-Type: application/json
Pragma: no-cache
Cache-Control: no-cache
Content-Length: 29
X-Docker-Registry-Version: 0.8.15
X-Docker-Registry-Config: common
Strict-Transport-Security: max-age=31536000

So unfortunately it looks like the curl command generated does not work.


UPDATE 2:

It looks like I'm able to download layer blobs from Docker Hub. Here is how I'm currently going about it.

Get an authorization token:

user@host:~$ export TOKEN=\
"$(curl \
--silent \
--header 'GET' \
"https://auth.docker.io/token?service=registry.docker.io&scope=repository:library/ubuntu:pull" \
| jq -r '.token' \
)"

Pull an image manifest:

user@host:~$ curl \
--silent \
--request 'GET' \
--header "Authorization: Bearer ${TOKEN}" \
'https://registry-1.docker.io/v2/library/ubuntu/manifests/latest' \
| jq '.'

Pull an image manifest and extract the blob sums:

user@host:~$ curl \
--silent \
--request 'GET' \
--header "Authorization: Bearer ${TOKEN}" \
'https://registry-1.docker.io/v2/library/ubuntu/manifests/latest' \
| jq -r '.fsLayers[].blobSum'

sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
sha256:be588e74bd348ce48bb7161350f4b9d783c331f37a853a80b0b4abc0a33c569e
sha256:e4ce6c3651b3a090bb43688f512f687ea6e3e533132bcbc4a83fb97e7046cea3
sha256:421e436b5f80d876128b74139531693be9b4e59e4f1081c9a3c379c95094e375
sha256:4c7380416e7816a5ab1f840482c9c3ca8de58c6f3ee7f95e55ad299abbfe599f
sha256:660c48dd555dcbfdfe19c80a30f557ac57a15f595250e67bfad1e5663c1725bb

Download a single layer blob and write it to a file:

user@host:~$ BLOBSUM=\
"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"

user@host:~$ curl \
--silent \
--location \
--request GET \
--header "Authorization: Bearer ${TOKEN}" \
"https://registry-1.docker.io/v2/library/ubuntu/blobs/${BLOBSUM}" \
> "${BLOBSUM/*:/}.gz"

Write all of the blob sums to a file:

user@host:~$ curl \
--silent \
--request 'GET' \
--header "Authorization: Bearer ${TOKEN}" \
'https://registry-1.docker.io/v2/library/ubuntu/manifests/latest' \
| jq -r '.fsLayers[].blobSum' > ubuntu-blobsums.txt

Download all of the layer blobs from the manifest:

user@host:~$ while read BLOBSUM; do
curl \
--silent \
--location \
--request 'GET' \
--header "Authorization: Bearer ${TOKEN}" \
"https://registry-1.docker.io/v2/library/ubuntu/blobs/${BLOBSUM}" \
> "${BLOBSUM/*:/}.gz"; \
done < blobsums.txt

Now I have a bunch of layer blobs and I need to recombine them into an image - I think.


Related Links:

igal

Posted 2017-11-29T19:23:46.820

Reputation: 680

"More specifically, I want to download a Docker Image from Docker Hub on a machine in a restricted environment which does not (and cannot) have the Docker client software installed." => What's the point to have the images on this machine then ? (easier workaround is using a pivot host, one where you acn docker pull from dockerhub and docker save/ docker push to an internal registry after)Tensibai 2017-11-30T11:06:03.160

@Tensibai To copy it to another machine that does have Docker but doesn't have internet access.igal 2017-11-30T13:46:04.853

Did you had a look at the docker pull code? It sounds the way to go to build something like this from basic http callsTensibai 2017-11-30T19:04:00.370

@Tensibai I think I figured it out. I also think I got a solution from the Docker community. I'll come back and post the solution later today.igal 2017-11-30T20:57:39.843

@Tensibai I posted a solution with a shell script that solves the problem.

igal 2017-12-03T15:04:26.180

Perhaps it would be easier to give the docker host Internet access by using the other as a proxyMarged 2019-03-23T12:43:17.253

@Marged That wasn't an option. The server was configured in such a way that it could not easily be used as a proxy.igal 2019-03-23T13:00:10.437

Answers

23

So it turns out that the Moby Project has a shell script on the Moby Github which can download images from Docker Hub in a format that can be imported into Docker:

The usage syntax for the script is given by the following:

download-frozen-image-v2.sh target_dir image[:tag][@digest] ...

The image can then be imported with tar and docker load:

tar -cC 'target_dir' . | docker load

To verify that the script works as expected, I downloaded an Ubuntu image from Docker Hub and loaded it into Docker:

user@host:~$ bash download-frozen-image-v2.sh ubuntu ubuntu:latest
user@host:~$ tar -cC 'ubuntu' . | docker load
user@host:~$ docker run --rm -ti ubuntu bash
root@1dd5e62113b9:/#

In practice I would have to first copy the data from the internet client (which does not have Docker installed) to the target/destination machine (which does have Docker installed):

user@nodocker:~$ bash download-frozen-image-v2.sh ubuntu ubuntu:latest
user@nodocker:~$ tar -C 'ubuntu' -cf 'ubuntu.tar' .
user@nodocker:~$ scp ubuntu.tar user@hasdocker:~

and then load and use the image on the target host:

user@hasdocker:~ docker load ubuntu.tar
user@hasdocker:~ docker run --rm -ti ubuntu bash
root@1dd5e62113b9:/#

igal

Posted 2017-11-29T19:23:46.820

Reputation: 680

The machine with internet connectivity does not and cannot have Docker installed. but you apply docker load030 2017-12-03T15:11:37.627

@030 Just to test/demonstrate that the script works and that the download image data can be imported into Docker. In practice I would first have to copy the data to a machine with Docker installed.igal 2017-12-03T15:13:05.310

Perhaps you could add that part for clarification030 2017-12-03T15:13:41.220

2@030 I added an example session illustrating what the workflow would look like in practice.igal 2017-12-03T15:20:27.787

7

There is a tool called Skopeo which can retrieve Docker images from a repository and save them in several formats.

For example:

  1. Download the image and save the layers as a tarball: skopeo copy docker://ubuntu docker-archive:/tmp/ubuntu.tar:ubuntu

  2. Transfer /tmp/ubuntu.tar to another machine if desired.

  3. Load the image on a Docker instance which does not have internet connection: docker load --input /tmp/ubuntu.tar

It is available in CentOS 7 repo with the package name skopeo. There are no Debian or Ubuntu packages at this time (but it is easy to compile).

snap

Posted 2017-11-29T19:23:46.820

Reputation: 171

3

thanks for motivation. I made a powershell version of it. Check it out... With it you can move in dockerhub containers to a restricted docker networks with a windows desktop and an ssh-scp tool to docker machine without root or administrator rights

https://gitlab.com/Jancsoj78/dockerless_docker_downloader a new hacker tool :)

$image = "ubuntu"
$tag = "latest"
$imageuri = "https://auth.docker.io/token?service=registry.docker.io&scope=repository:library/"+$image+":pull"
$taguri = "https://registry-1.docker.io/v2/library/"+$image+"/manifests/"+$tag
$bloburi = "https://registry-1.docker.io/v2/library/"+$image+"/blobs/sha256:"

#token request
$token = Invoke-WebRequest -Uri $imageuri | ConvertFrom-Json | Select -expand token

#pull image manifest
$blobs = $($(Invoke-Webrequest -Headers @{Authorization="Bearer $token"} -Method GET -Uri $taguri | ConvertFrom-Json | Select -expand fsLayers ) -replace "sha256:" -replace "@{blobSum=" -replace "}")

#download blobs
for ($i=0; $i -lt $blobs.length; $i++) {
    $blobelement =$blobs[$i]

    Invoke-Webrequest -Headers @{Authorization="Bearer $token"} -Method GET -Uri $bloburi$blobelement -OutFile blobtmp

    $source = "blobtmp"
    $newfile = "$blobelement.gz"

#overwrite
Copy-Item $source $newfile -Force -Recurse
#source blobs
ls *.gz
}
#postprocess
echo "copy these .gz to your docker machine"
echo "docker import .gz backward one by one"
echo "lastone with ubuntu:latest"
echo "after docker export and reimport to make a simple layer image"

Jancsó József

Posted 2017-11-29T19:23:46.820

Reputation: 31

1

To me it is not completely clear what you are trying to achieve and why the attempts are not a solution for the problem. If I would need to solve this issue I would like @Tensibai and other Q&As indicated, do a docker pull first on a system with internet connectivity, save the docker image, copy it to the machine without internet connectivity, load the image and run it.

Demonstration

There are no images on system A:

userA@systemA ~ $ docker images
REPOSITORY        TAG               IMAGE ID          CREATED             SIZE
userA@systemA ~ $

Pull an image from dockerhub:

userA@systemA ~ $
docker pull nginx
Using default tag: latest
latest: Pulling from library/nginx
bc95e04b23c0: Pull complete 
f3186e650f4e: Pull complete 
9ac7d6621708: Pull complete 
Digest: sha256:b81f317384d7388708a498555c28a7cce778a8f291d90021208b3eba3fe74887
Status: Downloaded newer image for nginx:latest
userA@systemA ~ $ docker images
REPOSITORY        TAG               IMAGE ID            CREATED             SIZE
nginx             latest            9e7424e5dbae        10 days ago         108MB

Save docker image:

userA@systemA ~ $ docker save nginx -o nginx.tar

Copy docker image to systemB and load it.

userB@systemB ~ $ docker load -i nginx.tar
cec7521cdf36: Loading layer  58.44MB/58.44MB
350d50e58b6c: Loading layer  53.76MB/53.76MB
63c39cd4a775: Loading layer  3.584kB/3.584kB
Loaded image: nginx:latest
userB@systemB ~ $ docker images
REPOSITORY        TAG               IMAGE ID            CREATED             SIZE
nginx             latest            9e7424e5dbae        10 days ago         108MB

030

Posted 2017-11-29T19:23:46.820

Reputation: 9 808

1

The machine with internet connectivity does not and cannot have Docker installed. The question is asking for a way to download an image without using the Docker client. See my solution.

igal 2017-12-03T15:03:17.830

0

Here is an adapted python script, thus having an OS independant solution: docker-drag

Use it like that, and it will create a TAR archive that you will be able to import using docker load :

python docker_pull.py hello-world
python docker_pull.py alpine:3.9
python docker_pull.py kalilinux/kali-linux-docker

Dutfaz

Posted 2017-11-29T19:23:46.820

Reputation: 1

1If you close your github account there's nothing left, if you're ok with sharing it, keep the link but paste the script here also please. You can [edit] your answer, paste your code and then select it and type ctrl+K or the {} (code) button in the editor top bar to format it.Tensibai 2019-04-09T13:16:06.527

I would love to paste the code here but it is 100 lines long and I don't think it will be readable. Nevertheless, you can fork the code to save your own copy of the script.Dutfaz 2019-04-09T13:19:30.570

That's not for me, that to have an answer which is self sustaining, if the link break, do you really think this may help someone reading this answer in a few months ? (btw the maximum size for an answer is 30k characters)Tensibai 2019-04-09T13:39:44.230