In this article, we will be exploring and walking you through how to set up your private docker registry on a Kubernetes cluster. While we walk you through the steps, we will also be in contact with the components in play, their functionalities and why we are using them. A docker registry is nothing but a repository for docker images. Click here to learn Docker with Kubernetes.
root@master1:~# docker -v Docker version 19.03.12, build 48a66213fe root@master1:~# kubectl version --short Client Version: v1.18.5 Server Version: v1.18.5 root@master1:~# kubectl get nodes NAME STATUS ROLES AGE VERSION master1 Ready master 65d v1.18.5 worker1 Ready <none> 65d v1.18.5 worker2 Ready <none> 65d v1.18.5 worker3 Ready <none> 65d v1.18.5 root@master1:~#
The most popular example being Dockerhub which is a hosted docker registry where you can share and store container images and is undeniably the largest public store for open source docker images. While there are thousands of docker images available publicly as well as from your peers and within your organization, enterprises would want to have a private repository with adequate access control and security. Here comes the need for self-hosting or purchasing a SaaS registry.
Every public cloud provider has come up with its own solution. Popular examples being AWS Elastic Container Registry, Azure’s Container Registry and Google Container Registry, to name some. Additionally, repository management solutions like Sonatype’s Nexus and JFrog’s Artifactory are popularly used tools for systems that prefer staying cloud agnostic. While there are plenty of solutions in the market, one can always choose to self-host their registry service for better control and ease of integration. Let us see how!
Having your own private repository for docker images gives absolute control over storage options, tightening access control and implementing authentication practices, all customized per your need.
A Kubernetes docker registry is a docker registry running as a Kubernetes pod. Like any storage service, there will be a volume attached to the pod that stores your private docker images, and you can set up access controls for the registry via access controls of the Kubernetes pod. Having your private docker registry directly in your Kubernetes cluster will give you greater speeds because the pulls will not be network calls. To sum it up, you will get better speed, low latency, higher availability, and absolute access control over the registry. All of these make the little pain of hosting and maintaining your registry service worth it.
Click here to find DevOps course that will help accelerate your journey.
Let us get started:
Go through this step-by-step process of creating the cluster components. It is best understood if you implement it alongside. If you do not have a public cloud cluster handy, spin up a single node cluster on your local machine using minikube and get rolling.
Let us start by addressing the elephant in the room: Security. In this section, we will build two important pillars by creating self-signed certificates and implementing user authentication.
We will start with creating a TLS certificate using openssl and authenticate the users using htpasswd.
root@master1:~# mkdir -p /registry && cd "$_" root@master1:/registry# mkdir certs root@master1:/registry# openssl req -x509 -newkey rsa:4096 -days 365 -nodes -sha256 -keyout certs/tls.key -out certs/tls.crt -subj "/CN=docker-registry" -addext "subjectAltName = DNS:docker-registry" Generating a RSA private key ...........................................................................................................................................++++ .............................................................++++ writing new private key to 'certs/tls.key' ----- root@master1:/registry#
We will now use htpasswd to add user authentication so they can access our docker registry.
root@master1:/registry# mkdir auth root@master1:/registry# docker run --rm --entrypoint htpasswd registry:2.6.2 -Bbn myuser mypasswd > auth/htpasswd Unable to find image 'registry:2.6.2' locally 2.6.2: Pulling from library/registry 486039affc0a: Pulling fs layer ba51a3b098e6: Pulling fs layer 470e22cd431a: Pulling fs layer 1048a0cdabb0: Pulling fs layer ca5aa9d06321: Pulling fs layer 1048a0cdabb0: Waiting ca5aa9d06321: Waiting ba51a3b098e6: Verifying Checksum ba51a3b098e6: Download complete 486039affc0a: Verifying Checksum 486039affc0a: Pull complete 470e22cd431a: Verifying Checksum 470e22cd431a: Download complete ba51a3b098e6: Pull complete 1048a0cdabb0: Verifying Checksum 1048a0cdabb0: Download complete ca5aa9d06321: Verifying Checksum ca5aa9d06321: Download complete 470e22cd431a: Pull complete 1048a0cdabb0: Pull complete ca5aa9d06321: Pull complete Digest: sha256:c4bdca23bab136d5b9ce7c06895ba54892ae6db0ebfc3a2f1ac413a470b17e47 Status: Downloaded newer image for registry:2.6.2 root@master1:/registry#
A Secret, as explained in the official Kubernetes documents, is an object that contains a small amount of sensitive data such as a password, a token, or a key. Such information might otherwise be put in a Pod specification or in a container image. Using a Secret means that you do not need to include confidential data in your application code.
We will create a TLS type of secret and a Generic type of secret to mount our certificate and password, respectively.
root@master1:/# kubectl create secret tls certs-secret --cert=/registry/certs/tls.crt --key=/registry/certs/tls.key secret/certs-secret created root@master1:/#
The TLS type of Secret to mount the private and public certificates we had created in the previous steps is ready.
root@master1:/# kubectl create secret generic auth-secret --from-file=/registry/auth/htpasswd secret/auth-secret created root@master1:/#
This Generic type of Secret will now store our htpasswd file values.
A Persistent Volume is a piece of volume and a cluster resource that you provision just like the nodes of the cluster.
A Persistent Volume Claim is a request for storage by a user in the cluster. Just like a Pod consumes node resources, PVCs request and consume PV resources.
Click here to read more about PV and PVC.
We will use this config file to create the Persistent Volume: registry-volume.yaml
apiVersion: v1 kind: PersistentVolume metadata: name: docker-repo-pv spec: capacity: storage: 1Gi accessModes: - ReadWriteOnce hostPath: path: /tmp/repository --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: docker-repo-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi
We will now apply these configurations to our cluster
root@master1:/# kubectl create -f registry-volume.yaml persistentvolume/docker-repo-pv created persistentvolumeclaim/docker-repo-pvc created root@master1:/#
Now, we have a PV and a PVC to use the volume resources from. Let us start using this volume.
In this step, we will create the pod and service using which the docker registry will be hosted and accessible on your cluster.
We will use the publicly available image called ‘registry’ available on DockerHub.
Since we want to access the registry, we will be creating a Kubernetes Service to allow the Pod to be accessible inside the cluster over port 5000.
apiVersion: v1 kind: Pod metadata: name: docker-registry-pod labels: app: registry spec: containers: - name: registry image: registry:2.6.2 volumeMounts: - name: repo-vol mountPath: "/var/lib/registry" - name: certs-vol mountPath: "/certs" readOnly: true - name: auth-vol mountPath: "/auth" readOnly: true env: - name: REGISTRY_AUTH value: "htpasswd" - name: REGISTRY_AUTH_HTPASSWD_REALM value: "Registry Realm" - name: REGISTRY_AUTH_HTPASSWD_PATH value: "/auth/htpasswd" - name: REGISTRY_HTTP_TLS_CERTIFICATE value: "/certs/tls.crt" - name: REGISTRY_HTTP_TLS_KEY value: "/certs/tls.key" volumes: - name: repo-vol persistentVolumeClaim: claimName: docker-repo-pvc - name: certs-vol secret: secretName: certs-secret - name: auth-vol secret: secretName: auth-secret --- apiVersion: v1 kind: Service metadata: name: docker-registry spec: selector: app: registry ports: - port: 5000 targetPort: 5000
Save the above configurations in a file like: docker-registry-pod.yaml and apply it to the cluster as we do below.
root@master1:/# kubectl create -f docker-registry-pod.yaml pod/docker-registry-pod created service/docker-registry created root@master1:/# kubectl get all NAME READY STATUS RESTARTS AGE pod/docker-registry-pod 1/1 Running 0 36sNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/docker-registry ClusterIP 10.107.59.73 <none> 5000/TCP 36s service/Kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 65d root@master1:/#
Stitching all the Kubernetes components and resources:
The docker registry we have created will be our central repository for all docker images the pod deployments will be using to spawn containers in it. Hence, it should be accessible from all points in the cluster.
Saving registry information as environment variables.
root@master1:/# export REGISTRY_NAME="docker-registry" root@master1:/# export REGISTRY_IP="10.107.59.73" root@master1:/#
Copying these environment variable values to the /etc/hosts file to enable all nodes in the cluster to resolve the service-name.
root@master1:/# for x in $(kubectl get nodes -o jsonpath='{ $.items[*].status.addresses[?(@.type=="InternalIP")].address }'); do ssh root@$x "echo '$REGISTRY_IP $REGISTRY_NAME' >> /etc/hosts"; done
root@master1:/# Copying the certificates to appropriate locations
root@master1:/# for x in $(kubectl get nodes -o jsonpath='{ $.items[*].status.addresses[?(@.type=="InternalIP")].address }'); do ssh root@$x "rm -rf /etc/docker/certs.d/$REGISTRY_NAME:5000;mkdir -p /etc/docker/certs.d/$REGISTRY_NAME:5000"; done
root@master1:/#
root@master1:/# for x in $(kubectl get nodes -o jsonpath='{ $.items[*].status.addresses[?(@.type=="InternalIP")].address }'); do scp /registry/certs/tls.crt root@$x:/etc/docker/certs.d/$REGISTRY_NAME:5000/ca.crt; done
tls.crt 100% 1822 2.7MB/s 00:00
tls.crt 100% 1822 2.4MB/s 00:00
tls.crt 100% 1822 1.6MB/s 00:00
tls.crt 100% 1822 2.1MB/s 00:00
root@master1:/# Without much delay, let us try to login to our registry.
root@master1:/# docker login docker-registry:5000 -u myuser -p mypasswd WARNING! Using --password via the CLI is insecure. Use --password-stdin. WARNING! Your password will be stored unencrypted in /root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-storeLogin Succeeded root@master1:/#
Congratulations on setting up your own hosted docker repository with absolute control. Now, we will make some updates to make it accessible through any node on the cluster.
root@master1:/# kubectl create secret docker-registry reg-cred-secret --docker-server=$REGISTRY_NAME:5000 --docker-username=myuser --docker-password=mypasswd secret/reg-cred-secret created root@master1:/#
This created a Secret type of docker-registry that allows all nodes in our cluster to access the registry with the credentials we had created on our first step.
To check if everything is setup correctly, we will attempt to push a custom image to our private docker registry.
root@master1:/# docker pull nginx Using default tag: latest latest: Pulling from library/nginx bf5952930446: Pull complete cb9a6de05e5a: Pull complete 9513ea0afb93: Pull complete b49ea07d2e93: Pull complete a5e4a503d449: Pull complete Digest: sha256:b0ad43f7ee5edbc0effbc14645ae7055e21bc1973aee5150745632a24a752661 Status: Downloaded newer image for nginx:latest docker.io/library/nginx:latest root@master1:/# root@master1:/# docker tag nginx:latest docker-registry:5000/mynginx:v1 root@master1:/# root@master1:/# docker push docker-registry:5000/mynginx:v1 The push refers to repository [docker-registry:5000/mynginx] 550333325e31: Layer already exists 22ea89b1a816: Layer already exists a4d893caa5c9: Layer already exists 0338db614b95: Layer already exists d0f104dc0a1f: Layer already exists v1: digest: sha256:179412c42fe3336e7cdc253ad4a2e03d32f50e3037a860cf5edbeb1aaddb915c size: 1362 root@master1:/# root@master1:/# kubectl exec docker-registry-pod -it -- sh / # ls /var/lib/registry/docker/registry/v2/repositories/ mynginx / #
The image exists!
You can run a docker images ls to get more information.
As a final step, we will try to launch a Pod with our recently pushed image.
root@master1:/# kubectl run nginx-pod --image=docker-registry:5000/mynginx:v1 --overrides='{ "apiVersion": "v1", "spec": { "imagePullSecrets": [{"name": "reg-cred-secret"}] } }'
pod/nginx-pod created
root@master1:/#
root@master1:/# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
docker-registry-pod 1/1 Running 0 46m 10.38.0.1 worker2 <none> <none>
nginx-pod 1/1 Running 0 38s 10.38.0.2 worker2 <none> <none>
root@master1:/#
root@master1:/# curl 10.38.0.2:80
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p><p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p><p><em>Thank you for using nginx.</em></p>
</body>
</html>
root@master1:/# Viola!
The containerization sphere keeps expanding everywhere, making it extremely essential to keep up with the latest products, tools, and newer functionalities. Here are some courses you will find relevant and helpful to not only help you get started on your journey but also get a wholesome hands-on learning experience with Docker and Kubernetes-like technologies.
We have set up a docker registry for your cluster that will be storing images inside the cluster and is accessible from anywhere in the cluster. While the setup is simple, there are areas like access control and security that we need to be mindful of when scaling our Kubernetes ecosystem. The best use case is when you are developing multiple applications and frequently making changes to each of those and testing locally, having the in-cluster registry is a great latency reducer. One thing to note here is that all images are only accessible to this cluster. For scenarios where applications will be deployed across multiple stacks and each cluster will reside on a separate Kubernetes cluster, hosting your docker registry in your cluster will pose some problems. Cross-cluster access for clusters hosting different stacks is a security breach and, in this case, it is best to pull out your registry to a central zone and have all stacks pull from it.
A registry of docker container images hosted inside a Kubernetes cluster.
No. Kubernetes is an open-source container orchestration tool that allows you to host applications in it. You can host your docker registry by running it as a Kubernetes pod.
Docker registries can be broadly classified into two types: public & private, based on their extent of accessibility.
Dockerhub is the default docker registry. You can find millions of images publicly hosted on https://hub.docker.com
| Name | Date | Fee | Know more |
|---|
