Why Use Zot Instead of Distribution

  • OCI Standard: Zot is based on the OCI standard and is maintained by the community.
  • Active Issues: The distribution has many pending issues that are unlikely to be resolved soon, some of which are quite complex.
  • Lightweight Setup: Zot is lightweight and easy to set up; it’s a single Golang binary command accompanied by a simple JSON config file.
  • Multiple Upstream Mirrors: Zot supports multiple upstream mirrors with a single service, while distribution can only serve one upstream.

Zot Problems

While Zot is a powerful tool, there are some challenges to keep in mind:

Problem Result Issue Mitigation Hack
GC/dedup lock granularity too high Pull during GC/dedup is very slow Issue 2968 Adjust GC interval Limit GC run to Sunday 2-5 AM
Blob will download even if already downloaded in S3 Slow pull Issue 2661 Use Nginx cache upfront -
Image downloads on all platforms even if only one is needed Slow pull Issue 2997 - Limit downloads to linux/amd64
Service restart checks all cached images on S3 Slow start/restart Issue 2413 HA or restart when idle Skip check S3 if set env var

These issues are waiting for resolution, so I have implemented some hacks, available here. You can build and use my hacked version of Zot.

Deployment

The architecture resembles:

podman/docker/containers ----> nginx -----> zot ----> s3

Example: Caching docker.io images with self signed cert

step 1: Configure Zot

Below is a sample JSON configuration for Zot.

{
    "http": {
        "address": "0.0.0.0",
        "port": "8187",
        "compat": [
            "docker2s2"
        ]
    },
    "log": {
        "level": "debug"
    },
    "extensions": {
        "sync": {
            "enable": true,
            "credentialsFile": "/etc/zot/config/sync-auth.json",
            "registries": [
                {
                    "urls": [
                        "https://mirror.ccs.tencentyun.com"
                    ],
                    "content": [
                        {
                            "prefix": "**",
                            "destination": "/docker-images"
                        }
                    ],
                    "onDemand": true,
                    "tlsVerify": true,
                    "maxRetries": 3,
                    "retryDelay": "30s",
                    "preserveDigest": true
                },
                {
                    "urls": [
                        "https://ghcr.io"
                    ],
                    "content": [
                        {
                            "prefix": "**",
                            "destination": "/ghcr-images"
                        }
                    ],
                    "onDemand": true,
                    "tlsVerify": true,
                    "maxRetries": 6,
                    "retryDelay": "1m",
                    "preserveDigest": true
                }
            ],
            "downloadDir": "/var/lib/zot/download/"
        }
    }
}

Here, we map docker.io images to localhost/docker-images. For instance, pulling docker.io/library/debian:bookworm actually pulls localhost/docker-images/library/debian:bookworm.

step 2: Nginx Proxy And Cache Configuration

You can choose from two options for path mapping.

  • Option 1: Implement a transparent proxy using a self-signed root CA for certificate trust and DNS redirection. This makes the process seamless for clients.
  • Option 2: Use container engines like podman/containerd that support registry rewrite for mapping image paths(eg podman), this requiring client-side configuration.

Each option has its pros and cons, with Option 1 being more transparent and Option 2 being more secure without certificate hacks.

The main function of nginx is to redirect requests to Zot based on the path mappings, while also caching the immutable image layers to reduce S3 access and improve pull speed. Zot only serves image layers after completely downloading them locally and then uploading them to S3. Serving images also involves local downloads and distribution, which can be slow if not cached.

  • share/mirror.conf
        proxy_set_header Host $host;
        proxy_pass http://zot_registry_backends;
        proxy_connect_timeout       300;
        proxy_send_timeout          300;
        proxy_read_timeout          1200;
        send_timeout                300;
        proxy_hide_header Www-Authenticate;
        add_header Www-Authenticate $rewritten_www_authenticate_header always;
        if ($request_uri ~* "^/v2/(.+)$") {
            rewrite ^/v2/(.+)$ $rewritten_v2_uri break;
        }
  • oci_mirror.conf
  map $host $mirror_subpath {
      hostnames;

      ghcr.io ghcr-images;
      ,*.docker.io docker-images;
      # add more upstream as you wish
      default docker-images;
  }

  map $args $rewritten_scope_args {
      ~^(?<prefix2>.*scope=repository%3A)(?<suffix2>.*)$     ${prefix2}$mirror_subpath%2F${suffix2};
  }

  map $upstream_http_www_authenticate $rewritten_www_authenticate_header {
      ~^(?<prefix1>.*http://).*(?<suffix1>/service/token.*)$     $prefix1$host$suffix1;
  }

  map $uri $rewritten_v2_uri {
      ~^/v2/(.+)$ /v2/$mirror_subpath/$1;
  }

  # for debug
  # log_format oci_mirror_log '$remote_addr "$time_local" $host "$request" $status $bytes_sent "$http_referer" "$http_user_agent" subpath: $mirror_subpath origin_blob_p: $origin_blob rewrite_v2_result: $rewritten_v2_uri uri: $uri';
  log_format oci_mirror_log '$remote_addr "$time_local" $host "$request" $status $bytes_sent "$http_referer" "$http_user_agent" subpath:$mirror_subpath $upstream_cache_status';

  upstream zot_registry_backends {
      hash $remote_addr;
      server zot_backend:8187 weight=10 max_fails=10 fail_timeout=30s;
  }

  proxy_cache_path /nginx/cache/dcr/mirror_cache/ levels=1:2 keys_zone=dcr_mirror_cache:50m max_size=50g inactive=72h use_temp_path=off;

  server {
      listen 80;
      listen 443 ssl;
      http2 on;
      # add server_name you signed by your root cert
      server_name docker.io registry-1.docker.io
                  ghcr.io;

      ssl_certificate         /etc/ssl/my.cer;
      ssl_certificate_key     /etc/ssl/my.key;

      ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

      proxy_cache_lock on;
      proxy_cache_lock_timeout 30s;

      location /v2/ {
          include /etc/nginx/vhosts/share/mirror.conf;
          proxy_cache_valid 200 302 70h;
          proxy_cache dcr_mirror_cache;
      }

      # don't cache mutable entity /v2/<name>/manifests/<reference> (unless the reference is a digest)
      location ~ ^/v2/.+?/manifests/([^:/]+)$ {
          proxy_cache off;
          include /etc/nginx/vhosts/share/mirror.conf;
      }

      # don't cache mutable entity /v2/<name>/tags/list
      location ~ ^/v2/.+?/tags/list {
          proxy_cache off;
          include /etc/nginx/vhosts/share/mirror.conf;
      }

      # don't cache mutable entity /v2/_catalog
      location ~ ^/v2/_catalog$ {
          include /etc/nginx/vhosts/share/mirror.conf;
          proxy_cache off;
      }

      location / {
          auth_basic off;
          proxy_set_header Host $host;
          proxy_set_header X-Real-IP $remote_addr;
          proxy_set_header X-Forwarded-For  $proxy_add_x_forwarded_for;
          proxy_set_header X-Forwarded-Proto $scheme;
          proxy_read_timeout 3600;
          add_header X-Upstream  $upstream_addr;
          proxy_cache_valid 200 302 24h;
          proxy_cache dcr_mirror_cache;
          proxy_pass http://zot_registry_backends;

          if ($args ~* ^scope=repository%3A) {
              set $args $rewritten_scope_args;
          }
      }

      access_log /var/log/nginx/oci_registry_mirror.access_log oci_mirror_log;
      error_log /var/log/nginx/oci_registry_mirror.error_log;
  }

step 3: Start Zot Service

Below is a bash script to start the Zot service.

#!/bin/bash

ulimit -n 131072

exec 2>&1

export HTTPS_PROXY=http://yourproxyserver:8008
export NO_PROXY="localhost,127.0.0.1,.myqcloud.com,.tencentyun.com,.qcloud.com"
# zot will use HOME to check $HOME/.config/containers/xx.config
export HOME=/var/lib/zot/

# ref https://github.com/dispensable/zot/commit/03398d35a3e3ae5e04881260ebae4d2a98a72b62
# orginal issue: https://github.com/project-zot/zot/issues/2413
export ZOT_DOUBAN_FAST_START=YES

# clean unexpected sync files to prevent unexpected disk usage
# this op only happens when zot been closed unexpectly like kill -9
find /var/lib/zot/download/ -wholename '*/.sync/*' -delete

exec setuidgid zot /usr/bin/zot serve /etc/zot/config/config.json

step 4: Add Hosts and Pull Images For test

Finally, update your /etc/hosts to point to your Nginx IP.

echo "<your nginx ip> docker.io registry-1.docker.io ghcr.io" >> /etc/hosts
podman pull docker.io/library/debian:bookworm

Summary

In this blog, we explored using Zot as a Docker/OCI registry cache. Zot offers some advantages over traditional distributions, such as being lightweight and community-driven. While there are some known issues, we also covered deployment strategies, including how to configure Zot and set up Nginx as a proxy layer. By following these steps, you can effectively cache Docker images and optimize your workflow.