Learn Kubernetes Secrets Management with Examples

The majority of Kubernetes-deployed apps need access to databases, services, and other resources that are hosted outside of their network. Using Kubernetes secrets to handle the login details required to access such resources is the simplest method. Secrets assist in cluster organization and distribution of sensitive data. You will discover what Kubernetes secrets are and how to build and use them in your cluster in this blog. You can also refer to CKA training programs to learn  

About Kubernetes Secrets

A Kubernetes secret is an object used to store confidential information like usernames, passwords, tokens, and keys. Secrets are produced by the system after the installation of an app or by users whenever they need to save confidential information and make it accessible to a pod. 

Passwords, tokens, or keys might be unintentionally revealed during Kubernetes operations if they were only included in a pod specification or container image. Therefore, the secret's primary purpose is to keep the information it contains from being accidentally discovered while also keeping it accessible to the user wherever they are. 

Secret Management and Kubernetes

Proper secret management is a major priority on the Kubernetes platform. For those who use GitOps, Sebastiaan notes that this is especially difficult. The configuration of Kubernetes deployments using GitOps is done according to specifications kept in a Git repository. One source of truth can be found in this collection.

The deployment of Kubernetes does not use the design of encrypted configuration secrets or procedures. It's much like retaining your password in a simple textual content document, that's a horrible idea. With GitOps, you may require a way of operating competently with secrets and techniques. 

Creating a Secret

A Kubernetes secret can be made using one of the following techniques: 

• For a command-line-based method, use kubectl. 
• Make the secret configuration file. 
• To produce the secret, use a generator like Kustomize. 

Making Secrets with Kubectl

1. Create the files needed to store the private data before you begin creating secrets with kubectl:

echo -n '[username]' > [file1]
echo -n '[password]' > [file2] 
Copy Code

The -n switch instructs echo not to insert a new line after the string. Because the new line is also considered a character, it would be encoded along with the other characters, resulting in a different encoded value. 

2. Utilizing the files from the previous step, build a secret using kubectl secrets right now. To construct a secret that is opaque, use the general subcommand. Additionally, for each file you want to include, add the —from-file option:

kubectl create secret generic [secret-name] \ 
--from-file = [file1] \ 
--from-file = [ file2] 
Copy Code

The result attests to the secret's creation: 

3. To provide keys for values stored in the secret, use the following syntax:

kubectl create secret generic [secret-name] \ 
--from-file=[key1]=[file1] \ 
--from-file=[key2]=[file2] 

4. Try typing: to see if the secret was successfully created.

kubectl get secrets 

The command displays a list of the accessible secrets together with information about their names, types, the quantity of data values they hold, and their age:

Create Secrets in a Configuration File

1. Starting with encoding the values you wish to save, give the necessary information in a configuration file to generate a secret: 

echo -n '[value1]' | base64 
echo -n '[value2]' | base64 

2. Now use a text editor to generate a yaml file. The file should appear as follows: 

apiVersion: v1 
kind: Secret 
metadata: 
name: newsecret 
type: Opaque 
data: 
username: dXNlcg== 
password: NTRmNDFkMTJlOGZh 

3. To create the secret, save the file and execute the kubectl apply command: 

kubectl apply -f [file] 

Create Kubernetes Secret with Generators

Quick secret generation is made possible by tools like Kustomize. 

1. To create a secret with Kustomize, make a file called kustomization.yaml and format it as shown below: 

secretGenerator: 
- name: db-credentials 
files: 
- username.txt 
- password.txt 

2. Alternately, include the literals section with the key-value pairs you intend to save to deliver the data values in their unencrypted, literal form: 

secretGenerator: 
- name: db-credentials 
literals: 
- username=user 
- password=54f41d12e8fa 

3. In the location where kustomization.yaml is placed, use the command after saving the file: 

kubectl apply -k . 

The result attests to the secret's creation: 

Editing a Secret

We can now get on with the real editing of the secret. You must change the secret value to the appropriate format since it must be saved as a base64 string. Using the Linux/macOS command line once more is one method to accomplish this: 

$ echo -n "my-new-password" | base64 
bXktbmV3LXBhc3N3b3Jk 

To prevent the base64 executable from receiving a newline character at the end of the echo output, we must use the -n parameter. Your app may experience problems if you don't use the -n parameter because the secret will end with the character n! Edit the secret with kubectl after copying the encoded value: 

$ kubectl edit secret db-credentials -n my-app 

Your default text editor launches, showing the current secret manifest. You're done when you swap out the secret data with the new value, save your work, and quit the editor. 

It's also feasible to let Kubernetes handle the base64 conversion if your secret data is something that fits well within a YAML file, like a password. To add a new stringData field to the YAML file holding all the secret values that you wish to update, use the same command as previously to enter the editor. 

stringData: 
 password: my-new-password 

Automatic merging and necessary transformations are done by Kubernetes between the stringData field and the data field. If the same keys are defined in both fields, stringData will take precedence, so you don't need to remove any existing keys from the data field. 

After you've saved and shut off the editor, you can check to see if the value has been changed. The edit will be cancelled if you close the window without making any changes. 

Using a Secret

It is necessary for the pod to use the secret to make a reference to it when it is created. Allowing a pod access to a secret. 

• The secret should be mounted as a file in a disc that any number of pod containers can access. 
• Import the secret into a container as an environment variable. 
• Utilize the imagePullSecrets field and kubelet. 

The steps for creating, decoding, and accessing Kubernetes secrets are covered in the sections that follow. 

Using Secrets as files from a Pod

One way Kubernetes can help you access data from a Secret in a Pod is by making the value of that Secret available as a file inside the filesystem of one or more of the Pod's containers. 

• Use a new secret or an existing one to customize that. The same secret may appear in multiple Pods. 
• Add a volume to the.spec.volumes[ section of your Pod definition. The volume can be given any name, and the value of the.spec.volumes[].secret.secretName field should match the name of the Secret object. 
• Each container that requires the secret should have a.spec.containers[].volumeMounts[] added to it. Set the parameters.spec.containers[].volumeMounts[].mountPath to the name of an empty directory where you want the secrets to be stored and.spec.containers[].volumeMounts[].readOnly = true. 
• Make changes to your image or command line to direct the software to look in that location for files. Each secret data map key corresponds to a different filename under mountPath. 

Here's an illustration of a pod that mounts a secret with the name mysecret in a volume: 

apiVersion: v1 
kind: Pod 
metadata: 
name: mypod 
spec: 
containers: 
- name: mypod 
image: redis 
volumeMounts: 
- name: foo 
mountPath: "/etc/foo" 
readOnly: true 
volumes: 
- name: foo 
secret: 
secretName: mysecret 
optional: false # default setting; "mysecret" must exist 

You must provide a reference to each Secret you intend to utilise in.spec.volumes. 

If the Pod contains more than one container, each container needs its own volume. 

mounts one block, but not more. 

Secret requires spec.volumes. 

Using Secrets as Environment Variables

To use a Secret in a Pod's environment variable: 

Make a Secret (or use an existing one). The same Secret may appear in multiple Pod references. 

You must include an environment variable for each secret key you want to consume to your pod definition in each container where you want to consume the value of a secret key. The secret's name and key should be filled out in env[ by the environment variable that uses the secret key. valueFrom.secretKeyRef. 

Make changes to your picture and/or command line to direct the programme to seek values in the designated environment variables. 

An illustration of a Pod using an environment variable to access a Secret is as follows:

apiVersion: v1 
kind: Pod 
metadata: 
name: secret-env-pod 
spec: 
containers: 
- name: mycontainer 
image: redis 
env: 
- name: SECRET_USERNAME 
valueFrom: 
secretKeyRef: 
name: mysecret 
key: username 
optional: false # same as default; "mysecret" must exist 
# and include a key named "username" 
- name: SECRET_PASSWORD 
valueFrom: 
secretKeyRef: 
name: mysecret 
key: password 
optional: false # same as default; "mysecret" must exist 
# and include a key named "password" 
restartPolicy: Never 

Container image pull secrets

The kubelet on each node needs a mechanism to authenticate to the private repository if you wish to fetch container images from it. This can be accomplished by configuring image pull secrets. Pod configuration determines how these secrets are used. 

A list of Secrets in the same namespace as the Pod is referenced in the imagePullSecrets field of a pod. Credentials for accessing the image registry can be passed to the kubelet via an imagePullSecret. A private image is pulled for your Pod by the kubelet using this information. For additional details on the imagePullSecrets field, refer to PodSpec in the Pod API reference. 

Using Secrets with static Pods  

ConfigMaps and Secrets are incompatible with static pods. 

Opaque secrets

If no secret configuration file is present, opaque is the default Secret type. When creating a Secret with kubectl, you will provide an Opaque Secret type by using the generic subcommand. An empty Secret of type Opaque is created, for instance, with the following command. 

kubectl create secret generic empty-secret 
kubectl get secret empty-secret 

The result appears to be: 

NAME TYPE DATA AGE 
empty-secret Opaque 0 2m6s 

The number of data objects contained in the Secret is indicated in the DATA field. If the value is 0, you have made an empty Secret. 

Service account token Secrets

The credential for a token that identifies a service account is kept in a kubernetes.io/service-account-token kind of Secret.

Since version 1.22, this kind of Secret object is no longer utilized to mount credentials into pods; instead, it is advised to use the TokenRequest API to get tokens instead of service account token Secret objects. Since they have a limited lifespan and cannot be read by other API clients, tokens retrieved through the TokenRequest API are more secure than those kept in Secret objects. Getting a token using the TokenRequest API is possible with the kubectl to generate the token command. 

If you can't get a token using the TokenRequest API and you're okay with exposing your service account token credential in a viewable API object for security reasons, then you should only construct a service account token Secret object. 

When utilizing this Secret type, make sure the kubernetes.io/service-account.name annotation is set to an actual service account name. The ServiceAccount object should be created first if you're making both it and the Secret object. 

The kubernetes.io/service-account.uid annotation and the token key in the data field, which is filled with an authentication token, are two fields that a Kubernetes controller fills in after creating the Secret. 

A service account token is declared in the setup example below Secret: 

apiVersion: v1 
kind: Secret 
metadata: 
name: secret-sa-sample 
annotations: 
kubernetes.io/service-account.name: "sa-name" 
type: kubernetes.io/service-account-token 
data:
# You can include additional key value pairs as you do with Opaque Secrets 
extra: YmFyCg== 

Once the Secret has been created, watch for Kubernetes to fill the data field with the token key. 

Docker config Secrets

To build a Secret to hold the login information for a container image registry, select one of the type values listed below: 

• kubernetes.io/dockercfg 
• kubernetes.io/dockerconfigjson 

The serialized version of the classic /.dockercfg file, which is used to set up the Docker command line, can be stored in the kubernetes.io/dockercfg type. When utilising this Secret type, you must make sure the Secret data field has a.dockercfg key whose value is the content of a.dockercfg file encoded in base64 format. 

The kubernetes.io/dockerconfigjson type was created to store serialised JSON that adheres to the same formatting standards as the new /.docker/config.json format for /.dockercfg. When utilising this Secret type, the Secret object's data field needs to include a.dockerconfigjson key that contains a base64-encoded string containing the contents of the.docker.config.json file. 

Here is an illustration of a Secret of the kubernetes.io/dockercfg type: 

apiVersion: v1 
kind: Secret 
metadata: 
name: secret-dockercfg 
type: kubernetes.io/dockercfg 
data: 
.dockercfg: | 
"<base64 encoded ~/.dockercfg file>" 

The API server checks whether the requested key is present in the data field when you create these kinds of Secrets using a manifest, and it confirms that the provided value can be interpreted as a valid JSON. The JSON is not verified by the API server as a Docker configuration file. 

When you don't have a Docker configuration file or want to use kubectl to generate a Secret for accessing a container registry, you can: 

kubectl create secret docker-registry secret-tiger-docker \ 
--docker-email=tiger@acme.example \ 
--docker-username=tiger \ 
--docker-password=pass1234 \ 
--docker-server=my-registry.example:5000 

By using that command, a Secret of the kubernetes.io/dockerconfigjson type is created. If you take the new Secret's.data.dockerconfigjson field and decode it from base64: 

kubectl get secret secret-tiger-docker -o jsonpath='{.data.*}' | base64 –d 

following which this JSON document serves as the output: 

{ 
"auths": { 
"my-registry.example:5000": { 
"username": "tiger", 
"password": "pass1234", 
"email": "tiger@acme.example", 
"auth": "dGlnZXI6cGFzczEyMzQ=" 
} 
} 
} 

Basic authentication Secret

For storing the credentials required for basic authentication, a type called kubernetes.io/basic-auth is available. One of the following two keys must be present in the Secret's data field if this Secret type is to be used: 

• username: the username for authentication 
• password: the password or token for authentication 

The two keys above have base64-encoded strings as their values in both cases. Naturally, using the stringData for Secret generation, you can supply the clear text content. 

Basic authentication is demonstrated by the following manifest: Secret: 

apiVersion: v1 
kind: Secret 
metadata: 
name: secret-basic-auth 
type: kubernetes.io/basic-auth 
stringData: 
username: admin # required field for kubernetes.io/basic-auth 
password: t0p-Secret # required field for kubernetes.io/basic-auth 

The secret kind is simply offered for convenience. For credentials used for fundamental authentication, you can define an opaque type. However, utilizing the defined and open-source Secret type (kubernetes.io/basic-auth) makes your Secret's purpose clear to others and establishes a convention for what key names to anticipate. For a Secret of this type, the Kubernetes API confirms that the necessary keys are configured. 

SSH Authentication Secrets

To store the information required for SSH authentication, the built-in type kubernetes.io/ssh-auth is available. When utilising this Secret type, you must enter an ssh-privatekey key-value pair as the SSH credential to use in the data (or stringData) field. 

An illustration of a secret for SSH public/private key authentication is the following manifest: 

apiVersion: v1 
kind: Secret 
metadata: 
name: secret-ssh-auth 
type: kubernetes.io/ssh-auth 
data: 
# the data is abbreviated in this example 
ssh-privatekey: | 
MIIEpQIBAAKCAQEAulqb/Y... 

Only for the user's convenience is the SSH authentication Secret type available. For SSH authentication credentials, you might rather establish an Opaque type Secret. However, utilizing the defined and open-source Secret type (kubernetes.io/ssh-auth) clarifies the purpose of your Secret and establishes a standard for key names that other users can rely on. in addition to checking to see if the necessary keys are supplied in a Secret configuration by the API server. 

TLS secrets

A built-in Secret type called kubernetes.io/tls is available in Kubernetes for storing a certificate and the key that goes with it, which are commonly used for TLS. 

TLS secrets can be used with other resources or directly in your workload, but one typical application is configuring encryption in transit for an Ingress. The data (or stringData) field of the Secret configuration must contain the tls.key and the tls.crt key when using this form of secret, even though the API server does not actually validate the values for each key. 

An example configuration for a TLS Secret is included in the following YAML: 

apiVersion: v1 
kind: Secret 
metadata: 
name: secret-tls 
type: kubernetes.io/tls 
data: 
# the data is abbreviated in this example 
tls.crt: | 
MIIC2DCCAcCgAwIBAgIBATANBgkqh... 
tls.key: | 
MIIEpgIBAAKCAQEA7yn3bRHQ5FHMQ... 

For the convenience of the user, the TLS Secret type is offered. For credentials used by TLS server and/or client, you can create an opaque. The API server does validate if the necessary keys are provided in a Secret configuration, therefore using the built-in Secret type helps assure the consistency of Secret format throughout your project. 

The following example demonstrates how to utilize the tls subcommand when establishing a TLS Secret using kubectl: 

kubectl create secret tls my-tls-secret \ 
--cert=path/to/cert/file \ 
--key=path/to/key/file 

Bootstrap Token Secrets

An improvised token Bootstrap.kubernetes.io/token can be used to create Secret by specifically defining the Secret type. This kind of Secret is intended for usage with the node bootstrap tokens. It holds the signature tokens for well-known ConfigMaps. 

Unstable token the name of a secret is often bootstrap-token-token-id>, where token-id> is a 6-character string representing the token ID. Secrets are typically created in the kube-system namespace. 

A bootstrap token Secret might appear as the following in a Kubernetes manifest: 

apiVersion: v1 
kind: Secret 
metadata: 
name: bootstrap-token-5emitj 
namespace: kube-system 
type: bootstrap.kubernetes.io/token 
data: 
auth-extra-groups: c3lzdGVtOmJvb3RzdHJhcHBlcnM6a3ViZWFkbTpkZWZhdWx0LW5vZGUtdG9rZW4= 
expiration: MjAyMC0wOS0xM1QwNDozOToxMFo= 
token-id: NWVtaXRq 
token-secret: a3E0Z2lodnN6emduMXAwcg== 
usage-bootstrap-authentication: dHJ1ZQ== 
usage-bootstrap-signing: dHJ1ZQ== 

Marking a Secret as Immutable

If the immutable field is set to true, you can build an immutable Secret. For instance, 

apiVersion: v1 
kind: Secret 
metadata: 
... 
data: 
... 
immutable: true 

Safely storing Kubernetes secrets

When you use kubectl create -f secret.yaml to generate a Secret, Kubernetes stores it in etcd. Unless you provide an encryption provider, etcd stores secrets in clear. 

Use cases of Secret Management in Kubernetes 

Use case: As container environment variables 

Create a secret 

apiVersion: v1 
kind: Secret 
metadata: 
name: mysecret 
type: Opaque 
data: 
USER_NAME: YWRtaW4= 
PASSWORD: MWYyZDFlMmU2N2Rm 

Create the Secret: 

kubectl apply -f mysecret.yaml 

To declare all of the data in the Secret as container environment variables, use envFrom. The name of the environment variable in the Pod is changed to the key from the Secret. 

apiVersion: v1 
kind: Pod 
metadata: 
name: secret-test-pod 
spec: 
containers: 
- name: test-container 
image: k8s.gcr.io/busybox 
command: [ "/bin/sh", "-c", "env" ] 
envFrom: 
- secretRef: 
name: mysecret 
restartPolicy: Never 

Use case: Pod with SSH keys 

Make a secret with certain SSH keys in it: 

kubectl create secret generic ssh-key-secret --from-file=ssh-privatekey=/path/to/.ssh/id_rsa --from-file=ssh-publickey=/path/to/.ssh/id_rsa.pub 

The result is comparable to: 

secret "ssh-key-secret" created 

The secret can now be consumed in a volume by a pod that references it using the SSH key: 

apiVersion: v1 
kind: Pod 
metadata: 
name: secret-test-pod 
labels: 
name: secret-test 
spec: 
volumes: 
- name: secret-volume 
secret: 
secretName: ssh-key-secret 
containers: 
- name: ssh-test-container 
image: mySshImage 
volumeMounts: 
- name: secret-volume 
readOnly: true 
mountPath: "/etc/secret-volume" 

The key parts will be accessible in: after the container's command has completed. 

/etc/secret-volume/ssh-publickey 
/etc/secret-volume/ssh-privatekey 

The secret information can then be utilized by the container to create an SSH connection. 

Use case: Pods with prod / test credentials

In this example, two pods—one consuming a secret with credentials for the production environment and the other consuming a secret with credentials for the test environment—are shown. 

You can create a kustomization. 

yaml with a secretGenerator field or run kubectl create secret. 

kubectl create secret generic prod-db-secret --from-literal=username=produser --from-literal=password=Y4nys7f11 

The result is comparable to: 

secret "prod-db-secret" created 

For testing environment credentials, you can also make a secret. 

kubectl create secret generic test-db-secret --from-literal=username=testuser --from-literal=password=iluvtests 

The result is comparable to: 

secret "test-db-secret" created 

Create the Pods now: 

cat <<EOF > pod.yaml 
apiVersion: v1 
kind: List 
items: 
- kind: Pod 
apiVersion: v1 
metadata: 
name: prod-db-client-pod 
labels: 
name: prod-db-client 
spec: 
volumes: 
- name: secret-volume 
secret: 
secretName: prod-db-secret 
containers: 
- name: db-client-container 
image: myClientImage 
volumeMounts: 
- name: secret-volume 
readOnly: true 
mountPath: "/etc/secret-volume" 
- kind: Pod 
apiVersion: v1 
metadata: 
name: test-db-client-pod 
labels: 
name: test-db-client 
spec: 
volumes: 
- name: secret-volume 
secret: 
secretName: test-db-secret 
containers: 
- name: db-client-container 
image: myClientImage 
volumeMounts: 
- name: secret-volume 
readOnly: true 
mountPath: "/etc/secret-volume" 
EOF 

Use case: dotfiles in a secret volume

By specifying a key that starts with a dot, you can make your data "hidden." A dotfile or "hidden" file is represented by this key. When the following secret is mounted into a volume, for instance, secret-volume 

apiVersion: v1 
kind: Secret 
metadata: 
name: dotfile-secret 
data: 
.secret-file: dmFsdWUtMg0KDQo= 
--- 
apiVersion: v1 
kind: Pod 
metadata: 
name: secret-dotfiles-pod 
spec: 
volumes: 
- name: secret-volume 
secret: 
secretName: dotfile-secret 
containers: 
- name: dotfile-test-container 
image: k8s.gcr.io/busybox 
command: 
- ls 
- "-l" 
- "/etc/secret-volume" 
volumeMounts: 
- name: secret-volume 
readOnly: true 
mountPath: "/etc/secret-volume" 

A single file with the name.secret-file will be present on the volume and accessible from the dotfile-test-container at the path /etc/secret-volume/.secret-file. 

Use case: Secret visible to one container in a Pod 

Consider a programme that must manage HTTP requests, execute intricate business logic, and then sign some messages using an HMAC. Due to the complicated application logic, it's possible that the server contains an undiscovered remote file reading vulnerability that might allow an attacker to access the private key. 

This might be split into two processes running in two containers: a signer container that can see the private key and answers to simple signing requests from the frontend, and a frontend container that manages user interaction and business logic (for example, over localhost networking). 

With this partitioned technique, an attacker must now persuade the application server to do an unauthorized action, which may be more difficult than convincing it to read a file. 

Conclusion

This blog should have taught you what Kubernetes secrets are, what kinds there are, and how to create one. Aspects of how to obtain secrets were also covered in the tutorial. If you want to dig deeper, you can take up the following - advanced DevOps course and undergo the following training - Docker Kubernetes training.