Private Docker Registry on Kubernetes: Steps to Set Up

In this article, we will be exploring and walking you through how to set up your private docker registry on a Kubernetes cluster. While we walk you through the steps, we will also be in contact with the components in play, their functionalities and why we are using them. A docker registry is nothing but a repository for docker images. Click here to learn Docker with Kubernetes. 

root@master1:~# docker -v 
Docker version 19.03.12, build 48a66213fe 
root@master1:~# kubectl version --short 
Client Version: v1.18.5 
Server Version: v1.18.5 
root@master1:~# kubectl get nodes 
NAME STATUS ROLES AGE VERSION 
master1 Ready master 65d v1.18.5 
worker1 Ready <none> 65d v1.18.5 
worker2 Ready <none> 65d v1.18.5 
worker3 Ready <none> 65d v1.18.5 
root@master1:~# 

The most popular example being Dockerhub which is a hosted docker registry where you can share and store container images and is undeniably the largest public store for open source docker images. While there are thousands of docker images available publicly as well as from your peers and within your organization, enterprises would want to have a private repository with adequate access control and security. Here comes the need for self-hosting or purchasing a SaaS registry.
Every public cloud provider has come up with its own solution. Popular examples being AWS Elastic Container Registry, Azure’s Container Registry and Google Container Registry, to name some. Additionally, repository management solutions like Sonatype’s Nexus and JFrog’s Artifactory are popularly used tools for systems that prefer staying cloud agnostic. While there are plenty of solutions in the market, one can always choose to self-host their registry service for better control and ease of integration. Let us see how! 

What is Kubernetes Private Docker Registry?  

Having your own private repository for docker images gives absolute control over storage options, tightening access control and implementing authentication practices, all customized per your need.

A Kubernetes docker registry is a docker registry running as a Kubernetes pod. Like any storage service, there will be a volume attached to the pod that stores your private docker images, and you can set up access controls for the registry via access controls of the Kubernetes pod. Having your private docker registry directly in your Kubernetes cluster will give you greater speeds because the pulls will not be network calls. To sum it up, you will get better speed, low latency, higher availability, and absolute access control over the registry. All of these make the little pain of hosting and maintaining your registry service worth it.
Click here to find DevOps course that will help accelerate your journey. 

Let us get started: 

Go through this step-by-step process of creating the cluster components. It is best understood if you implement it alongside. If you do not have a public cloud cluster handy, spin up a single node cluster on your local machine using minikube and get rolling. 

Creating Files for Authentication  

Let us start by addressing the elephant in the room: Security. In this section, we will build two important pillars by creating self-signed certificates and implementing user authentication.
We will start with creating a TLS certificate using openssl and authenticate the users using htpasswd.

root@master1:~# mkdir -p /registry && cd "$_"
root@master1:/registry# mkdir certs
root@master1:/registry# openssl req -x509 -newkey rsa:4096 -days 365 -nodes -sha256 -keyout certs/tls.key -out certs/tls.crt -subj "/CN=docker-registry" -addext "subjectAltName = DNS:docker-registry"
Generating a RSA private key
...........................................................................................................................................++++
.............................................................++++
writing new private key to 'certs/tls.key'
-----
root@master1:/registry# 

We will now use htpasswd to add user authentication so they can access our docker registry. 

root@master1:/registry# mkdir auth
root@master1:/registry# docker run --rm --entrypoint htpasswd registry:2.6.2 -Bbn myuser mypasswd > auth/htpasswd
Unable to find image 'registry:2.6.2' locally
2.6.2: Pulling from library/registry
486039affc0a: Pulling fs layer
ba51a3b098e6: Pulling fs layer
470e22cd431a: Pulling fs layer
1048a0cdabb0: Pulling fs layer
ca5aa9d06321: Pulling fs layer
1048a0cdabb0: Waiting
ca5aa9d06321: Waiting
ba51a3b098e6: Verifying Checksum
ba51a3b098e6: Download complete
486039affc0a: Verifying Checksum
486039affc0a: Pull complete
470e22cd431a: Verifying Checksum
470e22cd431a: Download complete
ba51a3b098e6: Pull complete
1048a0cdabb0: Verifying Checksum
1048a0cdabb0: Download complete
ca5aa9d06321: Verifying Checksum
ca5aa9d06321: Download complete
470e22cd431a: Pull complete
1048a0cdabb0: Pull complete
ca5aa9d06321: Pull complete
Digest: sha256:c4bdca23bab136d5b9ce7c06895ba54892ae6db0ebfc3a2f1ac413a470b17e47
Status: Downloaded newer image for registry:2.6.2
root@master1:/registry# 

Using Secrets to mount the certificates  

A Secret, as explained in the official Kubernetes documents, is an object that contains a small amount of sensitive data such as a password, a token, or a key. Such information might otherwise be put in a Pod specification or in a container image. Using a Secret means that you do not need to include confidential data in your application code. 

We will create a TLS type of secret and a Generic type of secret to mount our certificate and password, respectively. 

root@master1:/# kubectl create secret tls certs-secret --cert=/registry/certs/tls.crt --key=/registry/certs/tls.key
secret/certs-secret created
root@master1:/# 

The TLS type of Secret to mount the private and public certificates we had created in the previous steps is ready. 

root@master1:/# kubectl create secret generic auth-secret --from-file=/registry/auth/htpasswd
secret/auth-secret created
root@master1:/# 

This Generic type of Secret will now store our htpasswd file values. 

Creating Persistent Volumes and Claims for repository storage  

A Persistent Volume is a piece of volume and a cluster resource that you provision just like the nodes of the cluster. 

A Persistent Volume Claim is a request for storage by a user in the cluster. Just like a Pod consumes node resources, PVCs request and consume PV resources. 

Click here to read more about PV and PVC. 

We will use this config file to create the Persistent Volume: registry-volume.yaml 

apiVersion: v1
kind: PersistentVolume
metadata:
name: docker-repo-pv
spec:
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
hostPath:
path: /tmp/repository
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: docker-repo-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi 

We will now apply these configurations to our cluster 

root@master1:/# kubectl create -f registry-volume.yaml
persistentvolume/docker-repo-pv created
persistentvolumeclaim/docker-repo-pvc created
root@master1:/# 

Now, we have a PV and a PVC to use the volume resources from. Let us start using this volume. 

Creating the registry pod  

In this step, we will create the pod and service using which the docker registry will be hosted and accessible on your cluster.

We will use the publicly available image called ‘registry’ available on DockerHub. 

Since we want to access the registry, we will be creating a Kubernetes Service to allow the Pod to be accessible inside the cluster over port 5000. 

apiVersion: v1
kind: Pod
metadata:
name: docker-registry-pod
labels:
app: registry
spec:
containers:
- name: registry
image: registry:2.6.2
volumeMounts:
- name: repo-vol
mountPath: "/var/lib/registry"
- name: certs-vol
mountPath: "/certs"
readOnly: true
- name: auth-vol
mountPath: "/auth"
readOnly: true
env:
- name: REGISTRY_AUTH
value: "htpasswd"
- name: REGISTRY_AUTH_HTPASSWD_REALM
value: "Registry Realm"
- name: REGISTRY_AUTH_HTPASSWD_PATH
value: "/auth/htpasswd"
- name: REGISTRY_HTTP_TLS_CERTIFICATE
value: "/certs/tls.crt"
- name: REGISTRY_HTTP_TLS_KEY
value: "/certs/tls.key"
volumes:
- name: repo-vol
persistentVolumeClaim:
claimName: docker-repo-pvc
- name: certs-vol
secret:
secretName: certs-secret
- name: auth-vol
secret:
secretName: auth-secret
---
apiVersion: v1
kind: Service
metadata:
name: docker-registry
spec:
selector:
app: registry
ports:
- port: 5000
targetPort: 5000 

Save the above configurations in a file like: docker-registry-pod.yaml and apply it to the cluster as we do below. 

root@master1:/# kubectl create -f docker-registry-pod.yaml
pod/docker-registry-pod created
service/docker-registry created
root@master1:/# kubectl get all
NAME READY STATUS RESTARTS AGE
pod/docker-registry-pod 1/1 Running 0 36sNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/docker-registry ClusterIP 10.107.59.73 <none> 5000/TCP 36s
service/Kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 65d
root@master1:/# 

Stitching all the Kubernetes components and resources: 

• All images for your cluster will be now saved and pulled from this pod.
• These images will be stored in the persistent volume claim 
• The persistent volume claim will be attached to the registry pod 
• We will use user authentication and certificate verification for securing access to our docker binaries in the registry. 

Accessing a Docker registry from a Kubernetes Cluster  

The docker registry we have created will be our central repository for all docker images the pod deployments will be using to spawn containers in it. Hence, it should be accessible from all points in the cluster.

Saving registry information as environment variables. 

root@master1:/# export REGISTRY_NAME="docker-registry"
root@master1:/# export REGISTRY_IP="10.107.59.73"
root@master1:/# 

Copying these environment variable values to the /etc/hosts file to enable all nodes in the cluster to resolve the service-name. 

root@master1:/# for x in $(kubectl get nodes -o jsonpath='{ $.items[*].status.addresses[?(@.type=="InternalIP")].address }'); do ssh root@$x "echo '$REGISTRY_IP $REGISTRY_NAME' >> /etc/hosts"; done
root@master1:/# 

Copying the certificates to appropriate locations 

root@master1:/# for x in $(kubectl get nodes -o jsonpath='{ $.items[*].status.addresses[?(@.type=="InternalIP")].address }'); do ssh root@$x "rm -rf /etc/docker/certs.d/$REGISTRY_NAME:5000;mkdir -p /etc/docker/certs.d/$REGISTRY_NAME:5000"; done
root@master1:/#
root@master1:/# for x in $(kubectl get nodes -o jsonpath='{ $.items[*].status.addresses[?(@.type=="InternalIP")].address }'); do scp /registry/certs/tls.crt root@$x:/etc/docker/certs.d/$REGISTRY_NAME:5000/ca.crt; done
tls.crt 100% 1822 2.7MB/s 00:00
tls.crt 100% 1822 2.4MB/s 00:00
tls.crt 100% 1822 1.6MB/s 00:00
tls.crt 100% 1822 2.1MB/s 00:00
root@master1:/# 

Testing your Private Docker Registry  

Without much delay, let us try to login to our registry. 

root@master1:/# docker login docker-registry:5000 -u myuser -p mypasswd
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-storeLogin Succeeded
root@master1:/# 

Congratulations on setting up your own hosted docker repository with absolute control. Now, we will make some updates to make it accessible through any node on the cluster. 

root@master1:/# kubectl create secret docker-registry reg-cred-secret --docker-server=$REGISTRY_NAME:5000 --docker-username=myuser --docker-password=mypasswd
secret/reg-cred-secret created
root@master1:/# 

This created a Secret type of docker-registry that allows all nodes in our cluster to access the registry with the credentials we had created on our first step. 

To check if everything is setup correctly, we will attempt to push a custom image to our private docker registry. 

root@master1:/# docker pull nginx
Using default tag: latest
latest: Pulling from library/nginx
bf5952930446: Pull complete
cb9a6de05e5a: Pull complete
9513ea0afb93: Pull complete
b49ea07d2e93: Pull complete
a5e4a503d449: Pull complete
Digest: sha256:b0ad43f7ee5edbc0effbc14645ae7055e21bc1973aee5150745632a24a752661
Status: Downloaded newer image for nginx:latest
docker.io/library/nginx:latest
root@master1:/#
root@master1:/# docker tag nginx:latest docker-registry:5000/mynginx:v1
root@master1:/#
root@master1:/# docker push docker-registry:5000/mynginx:v1
The push refers to repository [docker-registry:5000/mynginx]
550333325e31: Layer already exists
22ea89b1a816: Layer already exists
a4d893caa5c9: Layer already exists
0338db614b95: Layer already exists
d0f104dc0a1f: Layer already exists
v1: digest: sha256:179412c42fe3336e7cdc253ad4a2e03d32f50e3037a860cf5edbeb1aaddb915c size: 1362
root@master1:/# 
root@master1:/# kubectl exec docker-registry-pod -it -- sh
/ # ls /var/lib/registry/docker/registry/v2/repositories/
mynginx
/ # 

The image exists! 

You can run a docker images ls to get more information. 

As a final step, we will try to launch a Pod with our recently pushed image. 

root@master1:/# kubectl run nginx-pod --image=docker-registry:5000/mynginx:v1 --overrides='{ "apiVersion": "v1", "spec": { "imagePullSecrets": [{"name": "reg-cred-secret"}] } }'
pod/nginx-pod created
root@master1:/#
root@master1:/# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
docker-registry-pod 1/1 Running 0 46m 10.38.0.1 worker2 <none> <none>
nginx-pod 1/1 Running 0 38s 10.38.0.2 worker2 <none> <none>
root@master1:/#
root@master1:/# curl 10.38.0.2:80
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;

}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p><p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p><p><em>Thank you for using nginx.</em></p>
</body>
</html>
root@master1:/# 

Viola! 

Docker and Kubernetes skills to master next  

The containerization sphere keeps expanding everywhere, making it extremely essential to keep up with the latest products, tools, and newer functionalities. Here are some courses you will find relevant and helpful to not only help you get started on your journey but also get a wholesome hands-on learning experience with Docker and Kubernetes-like technologies. 

• Learn how to use Docker on your Windows machine: https://www.knowledgehut.com/blog/devops/run-docker-on-windows 
• Up your Docker game by using this detailed course: https://www.knowledgehut.com/devops/docker-training 
• Start with learning or warming up your Docker-Kubernetes skills: https://www.knowledgehut.com/devops/docker-with-Kubernetes-training 
• Learn all that is there to become a Kubernetes Administrator: https://www.knowledgehut.com/devops/Kubernetes-training 
• Understand what a ‘service mesh’ is and why do we need it: https://linkerd.io/2017/04/25/whats-a-service-mesh-and-why-do-i-need-one/ 

Conclusion  

We have set up a docker registry for your cluster that will be storing images inside the cluster and is accessible from anywhere in the cluster. While the setup is simple, there are areas like access control and security that we need to be mindful of when scaling our Kubernetes ecosystem. The best use case is when you are developing multiple applications and frequently making changes to each of those and testing locally, having the in-cluster registry is a great latency reducer. One thing to note here is that all images are only accessible to this cluster. For scenarios where applications will be deployed across multiple stacks and each cluster will reside on a separate Kubernetes cluster, hosting your docker registry in your cluster will pose some problems. Cross-cluster access for clusters hosting different stacks is a security breach and, in this case, it is best to pull out your registry to a central zone and have all stacks pull from it.