Docker registry垃圾回收

通过：

docker run -p 5000:5000 -v /netdata/xxxx/registry:/var/lib/registry registry.cn-hangzhou.aliyuncs.com/term/registry-gc:2.7 registry garbage-collect /var/lib/registry/config.yml

搭建好registry服务后，docker push alpine:3.8镜像至registry服务。其中alpine的dockerfile内容如下：

FROM scratch
ADD rootfs.tar.xz /
CMD ["/bin/sh"]

registry.cn-hangzhou.aliyuncs.com/term/registry-gc:2.7的dockerfile如下:

FROM alpine:3.4

RUN set -ex 
    && apk add --no-cache ca-certificates apache2-utils

COPY registry /bin/registry
COPY config.yml /etc/docker/registry/config.yml

VOLUME ["/var/lib/registry"]
EXPOSE 5000

COPY docker-entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]

CMD ["/etc/docker/registry/config.yml"]

config.yml内容如下:

version: 0.1
log:
  fields:
    service: registry
storage:
  cache:
    blobdescriptor: inmemory
  filesystem:
    rootdirectory: /var/lib/registry
  delete:
    enabled: true
  maintenance:
    readonly:
      enabled: false
http:
  addr: :5000
  headers:
    X-Content-Type-Options: [nosniff]
health:
  storagedriver:
    enabled: true
    interval: 10s
    threshold: 3

后端存储

镜像推送至registry后，可查看registry后端的本地存储目录结构如下：

docker
`-- registry
    `-- v2
        |-- blobs
        |   `-- sha256
        |       |-- 01
        |       |   `-- 0107b3d497f0786d2d586d6093015ac5c64140794798fe140728f78c21af7bdc
        |       |       `-- data
        |       |-- 1e
        |       |   `-- 1e7481c6b78ad4b425f1985e3324117075ab3ebe2ef99d54a578d0f8c1fe0667
        |       |       `-- data
        |       `-- 5a
        |           `-- 5a83255105eaf8d24ad660befe32d7b9d449889ffde99b3ec59caf8edee59a3f
        |               `-- data
        `-- repositories
            `-- alpine
                |-- _layers
                |   `-- sha256
                |       |-- 0107b3d497f0786d2d586d6093015ac5c64140794798fe140728f78c21af7bdc
                |       |   `-- link
                |       `-- 5a83255105eaf8d24ad660befe32d7b9d449889ffde99b3ec59caf8edee59a3f
                |           `-- link
                |-- _manifests
                |   |-- revisions
                |   |   `-- sha256
                |   |       `-- 1e7481c6b78ad4b425f1985e3324117075ab3ebe2ef99d54a578d0f8c1fe0667
                |   |           `-- link
                |   `-- tags
                |       `-- 3.8
                |           |-- current
                |           |   `-- link
                |           `-- index
                |               `-- sha256
                |                   `-- 1e7481c6b78ad4b425f1985e3324117075ab3ebe2ef99d54a578d0f8c1fe0667
                |                       `-- link
                `-- _uploads

查看Manifest

查看alpine:3.8的manifest文件内容：

curl -i -H "Accept:application/vnd.docker.distribution.manifest.v2+json" http://127.0.0.1:5000/v2/alpine/manifests/3.8

Response Header:

HTTP/1.1 200 OK
Content-Length: 526
Content-Type: application/vnd.docker.distribution.manifest.v2+json
Docker-Content-Digest: sha256:1e7481c6b78ad4b425f1985e3324117075ab3ebe2ef99d54a578d0f8c1fe0667
Docker-Distribution-Api-Version: registry/2.0
Etag: "sha256:1e7481c6b78ad4b425f1985e3324117075ab3ebe2ef99d54a578d0f8c1fe0667"
X-Content-Type-Options: nosniff
Date: Tue, 04 Sep 2018 11:59:14 GMT

Response body:

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 1512,
      "digest": "sha256:0107b3d497f0786d2d586d6093015ac5c64140794798fe140728f78c21af7bdc"
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 12456,
         "digest": "sha256:5a83255105eaf8d24ad660befe32d7b9d449889ffde99b3ec59caf8edee59a3f"
      }
   ]
}

执行垃圾回收

sudo docker run --rm --name gctest -v /home/term/registry:/var/lib/registry registry:2.5.1 garbage-collect  -d /var/lib/registry/config.yml

得到的结果如下：

alpine
alpine: marking manifest sha256:1e7481c6b78ad4b425f1985e3324117075ab3ebe2ef99d54a578d0f8c1fe0667
alpine: marking blob sha256:5a83255105eaf8d24ad660befe32d7b9d449889ffde99b3ec59caf8edee59a3f
alpine: marking configuration sha256:0107b3d497f0786d2d586d6093015ac5c64140794798fe140728f78c21af7bdc

3 blobs marked, 0 blobs eligible for deletion

发现所有的blobs都被mark, 无blobs可清理

删除Manifest

curl -X DELETE http://127.0.0.1:5000/v2/alpine/manifests/sha256:1e7481c6b78ad4b425f1985e3324117075ab3ebe2ef99d54a578d0f8c1fe0667

sha256:1e7481c6b78ad4b425f1985e3324117075ab3ebe2ef99d54a578d0f8c1fe0667从获取Manifest时的Header: Docker-Content-Digest里获取。

调用manifest API删除manifest文件后，再次查看registry后端存储目录结构:

docker
`-- registry
    `-- v2
        |-- blobs
        |   `-- sha256
        |       |-- 01
        |       |   `-- 0107b3d497f0786d2d586d6093015ac5c64140794798fe140728f78c21af7bdc
        |       |       `-- data
        |       |-- 1e
        |       |   `-- 1e7481c6b78ad4b425f1985e3324117075ab3ebe2ef99d54a578d0f8c1fe0667
        |       |       `-- data
        |       |-- 5a
        |       |   `-- 5a83255105eaf8d24ad660befe32d7b9d449889ffde99b3ec59caf8edee59a3f
        |       |       `-- data
        |       `-- a3
        |           `-- a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
        |               `-- data
        `-- repositories
            `-- alpine
                |-- _layers
                |   `-- sha256
                |       |-- 0107b3d497f0786d2d586d6093015ac5c64140794798fe140728f78c21af7bdc
                |       |   `-- link
                |       |-- 5a83255105eaf8d24ad660befe32d7b9d449889ffde99b3ec59caf8edee59a3f
                |       |   `-- link
                |       `-- a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
                |           `-- link
                |-- _manifests
                |   |-- revisions
                |   |   `-- sha256
                |   |       `-- 1e7481c6b78ad4b425f1985e3324117075ab3ebe2ef99d54a578d0f8c1fe0667
                |   `-- tags
                `-- _uploads

对比之前的目录，发现alpine下的tags目录内容已被删

再次垃圾回收

再次执行垃圾回收：

sudo docker run --rm --name gctest -v /home/term/registry:/var/lib/registry registry:2.5.1 garbage-collect  -d /var/lib/registry/config.yml

得到结果如下：

alpine

0 blobs marked, 4 blobs eligible for deletion
blob eligible for deletion: sha256:0107b3d497f0786d2d586d6093015ac5c64140794798fe140728f78c21af7bdc
blob eligible for deletion: sha256:1e7481c6b78ad4b425f1985e3324117075ab3ebe2ef99d54a578d0f8c1fe0667
blob eligible for deletion: sha256:5a83255105eaf8d24ad660befe32d7b9d449889ffde99b3ec59caf8edee59a3f
blob eligible for deletion: sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4

以上为dry-run方式执行。

真正执行gc时，日志如下:

time="2018-09-05T01:50:32Z" level=info msg="Deleting blob: /docker/registry/v2/blobs/sha256/08/0873c923e00e0fd2ba78041bfb64a105e1ecb7678916d1f7776311e45bf5634b" go.version=go1.6.3 instance.id=ccafa0cc-4335-4535-b16a-9bff7886c896
time="2018-09-05T01:50:32Z" level=info msg="Deleting blob: /docker/registry/v2/blobs/sha256/11/11cd0b38bc3ceb958ffb2f9bd70be3fb317ce7d255c8a4c3f4af30e298aa1aab" go.version=go1.6.3 instance.id=ccafa0cc-4335-4535-b16a-9bff7886c896
time="2018-09-05T01:50:32Z" level=info msg="Deleting blob: /docker/registry/v2/blobs/sha256/1e/1e7481c6b78ad4b425f1985e3324117075ab3ebe2ef99d54a578d0f8c1fe0667" go.version=go1.6.3 instance.id=ccafa0cc-4335-4535-b16a-9bff7886c896
time="2018-09-05T01:50:32Z" level=info msg="Deleting blob: /docker/registry/v2/blobs/sha256/5a/5a83255105eaf8d24ad660befe32d7b9d449889ffde99b3ec59caf8edee59a3f" go.version=go1.6.3 instance.id=ccafa0cc-4335-4535-b16a-9bff7886c896
time="2018-09-05T01:50:32Z" level=info msg="Deleting blob: /docker/registry/v2/blobs/sha256/8e/8e3ba11ec2a2b39ab372c60c16b421536e50e5ce64a0bc81765c2e38381bcff6" go.version=go1.6.3 instance.id=ccafa0cc-4335-4535-b16a-9bff7886c896
time="2018-09-05T01:50:32Z" level=info msg="Deleting blob: /docker/registry/v2/blobs/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" go.version=go1.6.3 instance.id=ccafa0cc-4335-4535-b16a-9bff7886c896
time="2018-09-05T01:50:32Z" level=info msg="Deleting blob: /docker/registry/v2/blobs/sha256/01/0107b3d497f0786d2d586d6093015ac5c64140794798fe140728f78c21af7bdc" go.version=go1.6.3 instance.id=ccafa0cc-4335-4535-b16a-9bff7886c896

gc只清除blobs下的数据，但不会清除repositories里的内容及blobs下对应的目录。

docker
`-- registry
    `-- v2
        |-- blobs
        |   `-- sha256
        |       |-- 01
        |       |-- 08
        |       |-- 11
        |       |-- 1e
        |       |-- 5a
        |       |-- 8e
        |       `-- a3
        `-- repositories
            `-- alpine
                |-- _layers
                |   `-- sha256
                |       |-- 0107b3d497f0786d2d586d6093015ac5c64140794798fe140728f78c21af7bdc
                |       |   `-- link
                |       |-- 11cd0b38bc3ceb958ffb2f9bd70be3fb317ce7d255c8a4c3f4af30e298aa1aab
                |       |   `-- link
                |       |-- 5a83255105eaf8d24ad660befe32d7b9d449889ffde99b3ec59caf8edee59a3f
                |       |-- 8e3ba11ec2a2b39ab372c60c16b421536e50e5ce64a0bc81765c2e38381bcff6
                |       `-- a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
                |           `-- link
                |-- _manifests
                |   |-- revisions
                |   |   `-- sha256
                |   |       |-- 0873c923e00e0fd2ba78041bfb64a105e1ecb7678916d1f7776311e45bf5634b
                |   |       `-- 1e7481c6b78ad4b425f1985e3324117075ab3ebe2ef99d54a578d0f8c1fe0667
                |   `-- tags
                `-- _uploads

删除blobs

调用registry api删除blob：

curl -i -X DELETE http://127.0.0.1:5000/v2/alpine/blobs/sha256:5a83255105eaf8d24ad660befe32d7b9d449889ffde99b3ec59caf8edee59a3f

Response Header:

HTTP/1.1 202 Accepted
Content-Length: 0
Docker-Distribution-Api-Version: registry/2.0
X-Content-Type-Options: nosniff
Date: Wed, 05 Sep 2018 01:05:20 GMT
Content-Type: text/plain; charset=utf-8

Reponse body:

docker
`-- registry
    `-- v2
        |-- blobs
        |   `-- sha256
        |       |-- 01
        |       |   `-- 0107b3d497f0786d2d586d6093015ac5c64140794798fe140728f78c21af7bdc
        |       |       `-- data
        |       |-- 1e
        |       |   `-- 1e7481c6b78ad4b425f1985e3324117075ab3ebe2ef99d54a578d0f8c1fe0667
        |       |       `-- data
        |       |-- 5a
        |       |   `-- 5a83255105eaf8d24ad660befe32d7b9d449889ffde99b3ec59caf8edee59a3f
        |       |       `-- data
        |       `-- a3
        |           `-- a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
        |               `-- data
        `-- repositories
            `-- alpine
                |-- _layers
                |   `-- sha256
                |       |-- 0107b3d497f0786d2d586d6093015ac5c64140794798fe140728f78c21af7bdc
                |       |   `-- link
                |       |-- 5a83255105eaf8d24ad660befe32d7b9d449889ffde99b3ec59caf8edee59a3f
                |       `-- a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
                |           `-- link
                |-- _manifests
                |   |-- revisions
                |   |   `-- sha256
                |   |       `-- 1e7481c6b78ad4b425f1985e3324117075ab3ebe2ef99d54a578d0f8c1fe0667
                |   `-- tags
                `-- _uploads

对比之前目录结构，发现:docker/registry/v2/repositoires/<name>/_layers/<algorithm>/<digest>/link文件被删。

Config内容

{
    "architecture": "amd64",
    "config": {
        "ArgsEscaped": true,
        "AttachStderr": false,
        "AttachStdin": false,
        "AttachStdout": false,
        "Cmd": [
            "/bin/sh"
        ],
        "Domainname": "",
        "Entrypoint": null,
        "Env": [
            "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
        ],
        "Hostname": "",
        "Image": "sha256:80c02d9956b63482b0e4344c56cc029d8c330e01a3f6093516b0a60b83118b9c",
        "Labels": null,
        "OnBuild": null,
        "OpenStdin": false,
        "StdinOnce": false,
        "Tty": false,
        "User": "",
        "Volumes": null,
        "WorkingDir": ""
    },
    "container": "bf174046ba695f7f6c1d841dd2894b2c34a96d73bb832c102b3ba2b6a0d20120",
    "container_config": {
        "ArgsEscaped": true,
        "AttachStderr": false,
        "AttachStdin": false,
        "AttachStdout": false,
        "Cmd": [
            "/bin/sh",
            "-c",
            "#(nop) ",
            "CMD ["/bin/sh"]"
        ],
        "Domainname": "",
        "Entrypoint": null,
        "Env": [
            "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
        ],
        "Hostname": "bf174046ba69",
        "Image": "sha256:80c02d9956b63482b0e4344c56cc029d8c330e01a3f6093516b0a60b83118b9c",
        "Labels": {},
        "OnBuild": null,
        "OpenStdin": false,
        "StdinOnce": false,
        "Tty": false,
        "User": "",
        "Volumes": null,
        "WorkingDir": ""
    },
    "created": "2018-09-04T09:32:26.252138866Z",
    "docker_version": "17.06.2-ce",
    "history": [
        {
            "created": "2018-09-04T09:32:25.454551817Z",
            "created_by": "/bin/sh -c #(nop) ADD file:babb191309280a09293bff66ca4aead1ecad6119d9add3a803d489fa866d5282 in / "
        },
        {
            "created": "2018-09-04T09:32:26.252138866Z",
            "created_by": "/bin/sh -c #(nop)  CMD ["/bin/sh"]",
            "empty_layer": true
        }
    ],
    "os": "linux",
    "rootfs": {
        "diff_ids": [
            "sha256:2a7f9859cf97588d6533a9ac63c988b34beff0cc2758182b70e7a8cc8383d7f3"
        ],
        "type": "layers"
    }
}

结论

要想调用registry API，请打开deletable=true的开关，可通过环境变量: REGISTRY_STORAGE_DELETE_ENABLED=true指定, 也可在config.yml里配置
镜像清除，删除manifest即可，无需调用API删除layer，真正Blob清理需执行garbage-collect.
执行garbage-collect前，请将regsitry的readonly=true开关打开，重启registry，保证gc时，registry只读。此时不允许docker push镜像，否则会清理掉gc过程中新push的blobs。清理完成后，将readonly开关关闭,再次重启registry。
目前registry暂无GC API，只能通过脚本执行GC工作。

坑点

目前registry不支持通过环境变量设置readonly=true，在registry:2.5.1版本试过，会导致registry起不来，报错:

goroutine 1 [running]:
panic(0xc2a220, 0xc820438620)
	/usr/local/go/src/runtime/panic.go:481 +0x3e6
github.com/docker/distribution/registry/handlers.NewApp(0x7fd101c70ad8, 0xc8203ba780, 0xc82000d600, 0x7fd101c70ad8)
	/go/src/github.com/docker/distribution/registry/handlers/app.go:135 +0xc44
github.com/docker/distribution/registry.NewRegistry(0x7fd101c70b18, 0xc8203ba780, 0xc82000d600, 0xc82000d600, 0x0, 0x0)
	/go/src/github.com/docker/distribution/registry/registry.go:86 +0x2b0
github.com/docker/distribution/registry.glob.func1(0x14cc660, 0xc820347f50, 0x1, 0x1)
	/go/src/github.com/docker/distribution/registry/registry.go:55 +0x2a9
github.com/docker/distribution/vendor/github.com/spf13/cobra.(*Command).execute(0x14cc660, 0xc820347f00, 0x1, 0x1, 0x0, 0x0)
	/go/src/github.com/docker/distribution/vendor/github.com/spf13/cobra/command.go:495 +0x6d4
github.com/docker/distribution/vendor/github.com/spf13/cobra.(*Command).Execute(0x14cc800, 0x0, 0x0)
	/go/src/github.com/docker/distribution/vendor/github.com/spf13/cobra/command.go:560 +0x180
main.main()

须通过额外挂载配置文件解决

相关阅读:
HDU 3416 Marriage Match IV（SPFA+最大流）
asp.net一些很酷很实用的.Net技巧
 asp.net部分控件使用和开发技巧总结
 ASP_NET Global_asax详解
 asp.net 多字段模糊查询代码
 Asp.net中防止用户多次登录的方法
 SQL Server 事务、异常和游标
 有关Cookie
asp.net 连接sql server 2005 用户 'sa' 登录失败。asp.net开发第一步连接的细节问题
 asp.net生成高质量缩略图通用函数(c#代码),支持多种生成方式
原文地址：https://www.cnblogs.com/blogabc/p/10024765.html