Self-Hosted Kubernetes: When Your Private Registry Lies to You (ImagePullBackOff Field Guide)

Symptom: Pending / ImagePullBackOff on a single Deployment — what the operator sees first.

As a homelab operator running k3s or k8s at home, you've likely encountered the ImagePullBackOff error at least once. This error is a common occurrence when Kubernetes is unable to pull an image from the specified registry. However, in a self-hosted environment, the issue often manifests differently from what you might see in a cloud-based setup. The most common symptom is a Pending status for a single Deployment, accompanied by the ImagePullBackOff condition.

For example, you might see an event like this in your Kubernetes event stream:

Events:
  Type     Reason     Age   From               Message
  ----     ------     ----  ----               -------
  Normal   Scheduled  5m    default-scheduler  Successfully assigned default/my-deployment-7c8b9f6d9f-4z789 to node1
  Warning  BackOff    4m    kubelet            Back-off pulling image "<registry-host>/my-image:latest"
  Normal   Pulled     4m    kubelet            Successfully pulled image "<registry-host>/my-image:latest" in 1.234s
  Warning  Failed     4m    kubelet            Error: ErrImagePull: image "<registry-host>/my-image:latest" not found

In this example, the Deployment is stuck in a Pending state because the image <registry-host>/my-image:latest is not found in the registry. This is a clear indication that the issue is related to the image not being available in the registry.

Event strings cheat sheet: not found vs unauthorized vs tls: internal error — quick triage matrix.

When troubleshooting ImagePullBackOff, it's essential to understand the different error messages you might encounter. Here's a quick triage matrix to help you identify the root cause:

• not found: This error occurs when Kubernetes is unable to find the specified image in the registry. It's a common issue when the image tag is incorrect or the image has not been pushed to the registry yet.
• unauthorized: This error occurs when Kubernetes is unable to authenticate with the registry. It's a common issue when the registry is configured with authentication and the kubelet is not able to provide the necessary credentials.
• tls: internal error: This error occurs when Kubernetes is unable to establish a TLS connection with the registry. It's a common issue when the registry is configured with a self-signed certificate or the kubelet is unable to verify the certificate.

By understanding these error messages, you can quickly identify the root cause of the ImagePullBackOff error and take appropriate action to resolve it.

Case study: Docs site depending on an image that was never pushed to the internal registry.

One of the most common scenarios that lead to ImagePullBackOff is when a Kubernetes Deployment depends on an image that has not been pushed to the internal registry. This can happen for a variety of reasons, such as a typo in the image tag, a network issue, or a human error.

For example, let's say you have a Kubernetes Deployment that depends on an image called my-docs-site:latest. However, you forgot to push the image to the internal registry. When you try to deploy the Deployment, Kubernetes will encounter an ImagePullBackOff error because it cannot find the image in the registry.

To resolve this issue, you need to push the image to the internal registry. Here's how you can do it:

1. Build the image on your build host:

   docker build -t my-docs-site:latest .
   

2. Push the image to the internal registry using skopeo:

   skopeo copy docker-daemon:my-docs-site:latest docker://<registry-host>/my-docs-site:latest
   

3. Update the Deployment manifest to use the correct image tag:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: my-docs-site
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: my-docs-site
     template:
       metadata:
         labels:
           app: my-docs-site
       spec:
         containers:
         - name: my-docs-site
           image: <registry-host>/my-docs-site:latest
           ports:
           - containerPort: 80
   

4. Apply the updated Deployment manifest:

   kubectl apply -f my-docs-site-deployment.yaml
   

Once you've pushed the image to the internal registry and updated the Deployment manifest, Kubernetes should be able to pull the image and successfully deploy the Deployment.

Pattern A: Temporary nginx bootstrap (keep health probes intact while the real image is missing).

In some cases, you might not have the image available in the internal registry yet, but you still want to deploy the Deployment. In this scenario, you can use a temporary nginx bootstrap to keep the health probes intact while the real image is missing.

Here's how you can do it:

1. Create a temporary nginx Deployment:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: my-docs-site-temp
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: my-docs-site-temp
     template:
       metadata:
         labels:
           app: my-docs-site-temp
       spec:
         containers:
         - name: my-docs-site-temp
           image: nginx:latest
           ports:
           - containerPort: 80
   

2. Apply the temporary nginx Deployment:

   kubectl apply -f my-docs-site-temp-deployment.yaml
   

3. Update the Deployment manifest to use the correct image tag once the image is available:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: my-docs-site
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: my-docs-site
     template:
       metadata:
         labels:
           app: my-docs-site
       spec:
         containers:
         - name: my-docs-site
           image: <registry-host>/my-docs-site:latest
           ports:
           - containerPort: 80
   

4. Apply the updated Deployment manifest:

   kubectl apply -f my-docs-site-deployment.yaml
   

Once you've updated the Deployment manifest to use the correct image tag, Kubernetes should be able to pull the image and successfully deploy the Deployment.

Pattern B: Build off-cluster → push via skopeo with correct DNS to registry VIP / <registry-host>.

In some cases, you might not have the image available in the internal registry yet, but you still want to deploy the Deployment. In this scenario, you can build the image off-cluster and push it via skopeo with the correct DNS to the registry VIP or <registry-host>.

Here's how you can do it:

1. Build the image on your build host:

   docker build -t my-docs-site:latest .
   

2. Push the image to the internal registry using skopeo with the correct DNS:

   skopeo copy docker-daemon:my-docs-site:latest docker://<registry-host>/my-docs-site:latest
   

3. Update the Deployment manifest to use the correct image tag:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: my-docs-site
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: my-docs-site
     template:
       metadata:
         labels:
           app: my-docs-site
       spec:
         containers:
         - name: my-docs-site
           image: <registry-host>/my-docs-site:latest
           ports:
           - containerPort: 80
   

4. Apply the updated Deployment manifest:

   kubectl apply -f my-docs-site-deployment.yaml
   

Once you've pushed the image to the internal registry and updated the Deployment manifest, Kubernetes should be able to pull the image and successfully deploy the Deployment.

Rollout: Switch to the "real" Deployment manifest once the image tag exists.

Once the image tag exists in the internal registry, you can switch to the "real" Deployment manifest. This will ensure that Kubernetes is able to pull the image and successfully deploy the Deployment.

Here's how you can do it:

1. Update the Deployment manifest to use the correct image tag:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: my-docs-site
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: my-docs-site
     template:
       metadata:
         labels:
           app: my-docs-site
       spec:
         containers:
         - name: my-docs-site
           image: <registry-host>/my-docs-site:latest
           ports:
           - containerPort: 80
   

2. Apply the updated Deployment manifest:

   kubectl apply -f my-docs-site-deployment.yaml
   

Once you've updated the Deployment manifest to use the correct image tag, Kubernetes should be able to pull the image and successfully deploy the Deployment.

Prevention: Sentinel / CI smoke pull; don't point prod Deployment at :latest until push succeeds.

To prevent ImagePullBackOff in the future, you can use a sentinel or CI smoke pull to ensure that the image is available in the internal registry before deploying the Production Deployment. This will ensure that the Production Deployment is always able to pull the image and successfully deploy the Deployment.

Here's how you can do it:

1. Create a sentinel or CI smoke pull to ensure that the image is available in the internal registry:

   docker pull <registry-host>/my-docs-site:latest
   

2. If the image is available in the internal registry, deploy the Production Deployment:

   kubectl apply -f my-docs-site-deployment.yaml
   

3. If the image is not available in the internal registry, deploy the Temporary nginx bootstrap:

   kubectl apply -f my-docs-site-temp-deployment.yaml
   

By using a sentinel or CI smoke pull, you can ensure that the Production Deployment is always able to pull the image and successfully deploy the Deployment. This will prevent ImagePullBackOff in the future and ensure that your self-hosted Kubernetes cluster is always running smoothly.