Self-Hosted Kubernetes: When Your Private Registry Lies to You (ImagePullBackOff Field Guide)

Jun 24, 2026

Symptom: Pending / ImagePullBackOff on a single Deployment — what the operator sees first.

As a homelab operator running k3s or k8s at home, you've likely encountered the ImagePullBackOff error at least once. This error is a common occurrence when Kubernetes is unable to pull an image from the specified registry. However, in a self-hosted environment, the issue often manifests differently from what you might see in a cloud-based setup. The most common symptom is a Pending status for a single Deployment, accompanied by the ImagePullBackOff condition.

For example, you might see an event like this in your Kubernetes event stream:

Events:
  Type     Reason     Age   From               Message
  ----     ------     ----  ----               -------
  Normal   Scheduled  5m    default-scheduler  Successfully assigned default/my-deployment-7c8b9f6d9f-4z789 to node1
  Warning  BackOff    4m    kubelet            Back-off pulling image "<registry-host>/my-image:latest"
  Normal   Pulled     4m    kubelet            Successfully pulled image "<registry-host>/my-image:latest" in 1.234s
  Warning  Failed     4m    kubelet            Error: ErrImagePull: image "<registry-host>/my-image:latest" not found

In this example, the Deployment is stuck in a Pending state because the image <registry-host>/my-image:latest is not found in the registry. This is a clear indication that the issue is related to the image not being available in the registry.

Event strings cheat sheet: `not found` vs `unauthorized` vs `tls: internal error` — quick triage matrix.

When troubleshooting ImagePullBackOff, it's essential to understand the different error messages you might encounter. Here's a quick triage matrix to help you identify the root cause:

not found: This error occurs when Kubernetes is unable to find the specified image in the registry. It's a common issue when the image tag is incorrect or the image has not been pushed to the registry yet.
unauthorized: This error occurs when Kubernetes is unable to authenticate with the registry. It's a common issue when the registry is configured with authentication and the kubelet is not able to provide the necessary credentials.
tls: internal error: This error occurs when Kubernetes is unable to establish a TLS connection with the registry. It's a common issue when the registry is configured with a self-signed certificate or the kubelet is unable to verify the certificate.

By understanding these error messages, you can quickly identify the root cause of the ImagePullBackOff error and take appropriate action to resolve it.

Case study: Docs site depending on an image that was never pushed to the internal registry.

One of the most common scenarios that lead to ImagePullBackOff is when a Kubernetes Deployment depends on an image that has not been pushed to the internal registry. This can happen for a variety of reasons, such as a typo in the image tag, a network issue, or a human error.

For example, let's say you have a Kubernetes Deployment that depends on an image called my-docs-site:latest. However, you forgot to push the image to the internal registry. When you try to deploy the Deployment, Kubernetes will encounter an ImagePullBackOff error because it cannot find the image in the registry.

To resolve this issue, you need to push the image to the internal registry. Here's how you can do it:

Build the image on your build host:
```
docker build -t my-docs-site:latest .
```

Push the image to the internal registry using skopeo:

skopeo copy docker-daemon:my-docs-site:latest docker://<registry-host>/my-docs-site:latest

Update the Deployment manifest to use the correct image tag:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-docs-site
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-docs-site
  template:
    metadata:
      labels:
        app: my-docs-site
    spec:
      containers:
      - name: my-docs-site
        image: <registry-host>/my-docs-site:latest
        ports:
        - containerPort: 80

Apply the updated Deployment manifest:

kubectl apply -f my-docs-site-deployment.yaml

Once you've pushed the image to the internal registry and updated the Deployment manifest, Kubernetes should be able to pull the image and successfully deploy the Deployment.

Pattern A: Temporary nginx bootstrap (keep health probes intact while the real image is missing).

In some cases, you might not have the image available in the internal registry yet, but you still want to deploy the Deployment. In this scenario, you can use a temporary nginx bootstrap to keep the health probes intact while the real image is missing.

Here's how you can do it:

Create a temporary nginx Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-docs-site-temp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-docs-site-temp
  template:
    metadata:
      labels:
        app: my-docs-site-temp
    spec:
      containers:
      - name: my-docs-site-temp
        image: nginx:latest
        ports:
        - containerPort: 80

Apply the temporary nginx Deployment:

kubectl apply -f my-docs-site-temp-deployment.yaml

Update the Deployment manifest to use the correct image tag once the image is available:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-docs-site
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-docs-site
  template:
    metadata:
      labels:
        app: my-docs-site
    spec:
      containers:
      - name: my-docs-site
        image: <registry-host>/my-docs-site:latest
        ports:
        - containerPort: 80

Apply the updated Deployment manifest:

kubectl apply -f my-docs-site-deployment.yaml

Once you've updated the Deployment manifest to use the correct image tag, Kubernetes should be able to pull the image and successfully deploy the Deployment.

Pattern B: Build off-cluster → push via skopeo with correct DNS to registry VIP / `<registry-host>`.

In some cases, you might not have the image available in the internal registry yet, but you still want to deploy the Deployment. In this scenario, you can build the image off-cluster and push it via skopeo with the correct DNS to the registry VIP or <registry-host>.

Here's how you can do it:

Build the image on your build host:
```
docker build -t my-docs-site:latest .
```

Push the image to the internal registry using skopeo with the correct DNS:

skopeo copy docker-daemon:my-docs-site:latest docker://<registry-host>/my-docs-site:latest

Update the Deployment manifest to use the correct image tag:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-docs-site
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-docs-site
  template:
    metadata:
      labels:
        app: my-docs-site
    spec:
      containers:
      - name: my-docs-site
        image: <registry-host>/my-docs-site:latest
        ports:
        - containerPort: 80

Apply the updated Deployment manifest:

kubectl apply -f my-docs-site-deployment.yaml

Once you've pushed the image to the internal registry and updated the Deployment manifest, Kubernetes should be able to pull the image and successfully deploy the Deployment.

Rollout: Switch to the "real" Deployment manifest once the image tag exists.

Once the image tag exists in the internal registry, you can switch to the "real" Deployment manifest. This will ensure that Kubernetes is able to pull the image and successfully deploy the Deployment.

Here's how you can do it:

Update the Deployment manifest to use the correct image tag:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-docs-site
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-docs-site
  template:
    metadata:
      labels:
        app: my-docs-site
    spec:
      containers:
      - name: my-docs-site
        image: <registry-host>/my-docs-site:latest
        ports:
        - containerPort: 80

Apply the updated Deployment manifest:

kubectl apply -f my-docs-site-deployment.yaml

Once you've updated the Deployment manifest to use the correct image tag, Kubernetes should be able to pull the image and successfully deploy the Deployment.

Prevention: Sentinel / CI smoke pull; don't point prod Deployment at `:latest` until push succeeds.

To prevent ImagePullBackOff in the future, you can use a sentinel or CI smoke pull to ensure that the image is available in the internal registry before deploying the Production Deployment. This will ensure that the Production Deployment is always able to pull the image and successfully deploy the Deployment.

Here's how you can do it:

Create a sentinel or CI smoke pull to ensure that the image is available in the internal registry:
```
docker pull <registry-host>/my-docs-site:latest
```
If the image is available in the internal registry, deploy the Production Deployment:
```
kubectl apply -f my-docs-site-deployment.yaml
```
If the image is not available in the internal registry, deploy the Temporary nginx bootstrap:
```
kubectl apply -f my-docs-site-temp-deployment.yaml
```

By using a sentinel or CI smoke pull, you can ensure that the Production Deployment is always able to pull the image and successfully deploy the Deployment. This will prevent ImagePullBackOff in the future and ensure that your self-hosted Kubernetes cluster is always running smoothly.

自我托管Kubernetes：當您的私有Registry騙了你（ImagePullBackOff手冊）

2026年6月24日

症狀：單一 Deployment 的 Pending / ImagePullBackOff — 操作員首見之狀。

在家中運行 k3s 或 k8s 的家庭實驗室操作員，很可能至少遇到過一次 ImagePullBackOff 錯誤。當 Kubernetes 無法從指定的registry拉取鏡像時，會出現這種錯誤。然而，在自托管環境中，問題往往表現得與雲端環境不同。最常見的症狀是一個單一的 Deployment 处於 Pending 狀態，並伴有 ImagePullBackOff 狀態。

例如，您可能會在 Kubernetes 事件流中看到類似以下的事件：

事件：
  型別     原因     年齡   發出者               訊息
  ----     ------     ----  ----               -------
  正常   賈贖     5分    default-scheduler  成功分配 default/my-deployment-7c8b9f6d9f-4z789 至 node1
  警告  回避     4分    kubelet            回避拉取鏡像 "<registry-host>/my-image:latest"
  正常   拉取     4分    kubelet            成功在 1.234s 時間內拉取鏡像 "<registry-host>/my-image:latest"
  警告  失敗     4分    kubelet            錯誤：ErrImagePull: 鏡像 "<registry-host>/my-image:latest" 未找到

在這個例子中，Deployment 因為鏡像 <registry-host>/my-image:latest 在 registry 中未找到而處於 Pending 狀態。這是一個明確的指示，問題是鏡像未在 registry 中可用。

Event 字串快速參考表：`not found`、`unauthorized`、`tls: internal error` — 快速診斷矩陣。

在排除 ImagePullBackOff 問題時，了解可能會遇到的不同錯誤訊息非常重要。以下是一份快速診斷矩陣，幫助您找出根本原因：

not found：當 Kubernetes 无法在 registry 中找到指定的影像時，會出現此錯誤。這通常是因為影像標籤錯誤或影像尚未推送到 registry。
unauthorized：當 Kubernetes 无法與 registry 進行身份驗證時，會出現此錯誤。這通常是因為 registry 已配置身份驗證，而 kubelet 无法提供必要的憑證。
tls: internal error：當 Kubernetes 无法與 registry 建立 TLS 連接時，會出現此錯誤。這通常是因為 registry 已配置自-signed 证书，或 kubelet 无法驗證該证书。

了解這些錯誤訊息，您可以快速找出 ImagePullBackOff 錯誤的根本原因，並採取適當措施解決問題。

案例研究：文件站依賴未推送到內部registry的圖像

導致 ImagePullBackOff 的最常見情況之一是，Kubernetes Deployment 依賴未推送到內部registry的圖像。這可能是由於圖像標籤中的拼字錯誤、網絡問題或人為錯誤等原因。

例如，假設您有一個依賴於名為 my-docs-site:latest 的圖像的Kubernetes Deployment。然而，您忘記將圖像推送到內部registry。當您嘗試部署Deployment時，Kubernetes 將遇到 ImagePullBackOff 錯誤，因為它找不到registry中的圖像。

要解決此問題，您需要將圖像推送到內部registry。以下是操作步驟：

在建置主機上建立圖像：
```
docker build -t my-docs-site:latest .
```

使用 skopeo 將圖像推送到內部registry：

skopeo copy docker-daemon:my-docs-site:latest docker://<registry-host>/my-docs-site:latest

更新Deployment的定義文件以使用正確的圖像標籤：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-docs-site
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-docs-site
  template:
    metadata:
      labels:
        app: my-docs-site
    spec:
      containers:
      - name: my-docs-site
        image: <registry-host>/my-docs-site:latest
        ports:
        - containerPort: 80

應用更新的Deployment定義文件：

kubectl apply -f my-docs-site-deployment.yaml

一旦將圖像推送到內部registry並更新了Deployment定義文件，Kubernetes 就應該能夠拉取圖像並成功部署Deployment。

模式 A：臨時 nginx 初始化（在真實影像缺失時保持健康探測器完好）。

在某些情況下，你可能還沒有將影像存儲在內部 registry 中，但仍然希望部署 Deployment。在此情況下，你可以使用臨時 nginx 初始化來在真實影像缺失時保持健康探測器完好。

以下是具體操作步驟：

創建臨時 nginx Deployment：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-docs-site-temp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-docs-site-temp
  template:
    metadata:
      labels:
        app: my-docs-site-temp
    spec:
      containers:
      - name: my-docs-site-temp
        image: nginx:latest
        ports:
        - containerPort: 80

執行臨時 nginx Deployment：

kubectl apply -f my-docs-site-temp-deployment.yaml

更新 Deployment 定義文件，使用正確的影像標籤，一旦影像可用：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-docs-site
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-docs-site
  template:
    metadata:
      labels:
        app: my-docs-site
    spec:
      containers:
      - name: my-docs-site
        image: <registry-host>/my-docs-site:latest
        ports:
        - containerPort: 80

執行更新的 Deployment 定義文件：

kubectl apply -f my-docs-site-deployment.yaml

一旦你將 Deployment 定義文件更新為使用正確的影像標籤，Kubernetes 就應該能夠拉取影像並成功部署 Deployment。

模式 B：在群集外建立影像 → 使用 skopeo 推送至 registy VIP / `<registry-host>`，並使用正確的 DNS。

在某些情況下，您可能尚未在內部 registry 中找到所需的影像，但仍希望部署 Deployment。在這種情況下，您可以將影像在群集外建立，然後使用 skopeo 和正確的 DNS 推送到 registry VIP 或 <registry-host>。

以下是操作步驟：

在建置主機上建立影像：
```
docker build -t my-docs-site:latest .
```

使用 skopeo 和正確的 DNS 將影像推送到內部 registry：

skopeo copy docker-daemon:my-docs-site:latest docker://<registry-host>/my-docs-site:latest

更新 Deployment 的 manifest 以使用正確的影像標籤：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-docs-site
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-docs-site
  template:
    metadata:
      labels:
        app: my-docs-site
    spec:
      containers:
      - name: my-docs-site
        image: <registry-host>/my-docs-site:latest
        ports:
        - containerPort: 80

應用更新後的 Deployment manifest：

kubectl apply -f my-docs-site-deployment.yaml

一旦將影像推送到內部 registry，並更新 Deployment 的 manifest，Kubernetes 就應該能夠拉取影像並成功部署 Deployment。

滚動更新：一旦內部註冊表中存在影像标签，即可切換到「真實」的 Deployment 程式表。

一旦內部註冊表中存在影像标签，就可以切換到「真實」的 Deployment 程式表。這樣可以確保 Kubernetes 能夠拉取影像並成功部署 Deployment。

以下是操作步驟：

更新 Deployment 程式表，使用正確的影像标签：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-docs-site
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-docs-site
  template:
    metadata:
      labels:
        app: my-docs-site
    spec:
      containers:
      - name: my-docs-site
        image: <registry-host>/my-docs-site:latest
        ports:
        - containerPort: 80

應用更新的 Deployment 程式表：

kubectl apply -f my-docs-site-deployment.yaml

一旦更新 Deployment 程式表以使用正確的影像标签，Kubernetes 就應該能夠拉取影像並成功部署 Deployment。

預防：哨兵 / CI 煙熏拉取；在推送成功之前，不要將 Production Deployment 設定指向 `:latest`。

為防止未來發生 ImagePullBackOff，您可以使用哨兵或 CI 煙熏拉取，確保在部署 Production Deployment 之前，圖像已在內部 registery 中可用。這樣可以確保 Production Deployment 始終能夠拉取圖像並成功部署。

以下是操作步驟：

使用哨兵或 CI 煙熏拉取確保圖像已在內部 registery 中可用：
```
docker pull <registry-host>/my-docs-site:latest
```
如果圖像已在內部 registery 中可用，則部署 Production Deployment：
```
kubectl apply -f my-docs-site-deployment.yaml
```
如果圖像未在內部 registery 中可用，則部署臨時的 nginx 起點：
```
kubectl apply -f my-docs-site-temp-deployment.yaml
```

藉由使用哨兵或 CI 煙熏拉取，可以確保 Production Deployment 始終能夠拉取圖像並成功部署。這樣可以防止未來發生 ImagePullBackOff，並確保您的自托管 Kubernetes 集群始終運行順利。

Self-Hosted Kubernetes: When Your Private Registry Lies to You (ImagePullBackOff Field Guide)

Symptom: Pending / ImagePullBackOff on a single Deployment — what the operator sees first.

Event strings cheat sheet: not found vs unauthorized vs tls: internal error — quick triage matrix.

Case study: Docs site depending on an image that was never pushed to the internal registry.

Pattern A: Temporary nginx bootstrap (keep health probes intact while the real image is missing).

Pattern B: Build off-cluster → push via skopeo with correct DNS to registry VIP / <registry-host>.

Rollout: Switch to the "real" Deployment manifest once the image tag exists.

Prevention: Sentinel / CI smoke pull; don't point prod Deployment at :latest until push succeeds.

自我托管Kubernetes：當您的私有Registry騙了你（ImagePullBackOff手冊）

症狀：單一 Deployment 的 Pending / ImagePullBackOff — 操作員首見之狀。

Event 字串快速參考表：not found、unauthorized、tls: internal error — 快速診斷矩陣。

案例研究：文件站依賴未推送到內部registry的圖像

模式 A：臨時 nginx 初始化（在真實影像缺失時保持健康探測器完好）。

模式 B：在群集外建立影像 → 使用 skopeo 推送至 registy VIP / <registry-host>，並使用正確的 DNS。

滚動更新：一旦內部註冊表中存在影像标签，即可切換到「真實」的 Deployment 程式表。

預防：哨兵 / CI 煙熏拉取；在推送成功之前，不要將 Production Deployment 設定指向 :latest。

Event strings cheat sheet: `not found` vs `unauthorized` vs `tls: internal error` — quick triage matrix.

Pattern B: Build off-cluster → push via skopeo with correct DNS to registry VIP / `<registry-host>`.

Prevention: Sentinel / CI smoke pull; don't point prod Deployment at `:latest` until push succeeds.

Event 字串快速參考表：`not found`、`unauthorized`、`tls: internal error` — 快速診斷矩陣。

模式 B：在群集外建立影像 → 使用 skopeo 推送至 registy VIP / `<registry-host>`，並使用正確的 DNS。

預防：哨兵 / CI 煙熏拉取；在推送成功之前，不要將 Production Deployment 設定指向 `:latest`。