blog.3q.fi

Tekton on a Homelab: What GitHub Actions Won't Tell You

Jun 26, 2026

Article cover: Tekton on a Homelab: What GitHub Actions Won't Tell You

1. Why Homelab CI ≠ Cloud CI

1. Why Homelab CI ≠ Cloud CI
1. Why Homelab CI ≠ Cloud CI

The allure of CI/CD is universal, but the journey from cloud to homelab can be fraught with challenges. For those running Kubernetes in a personal or small-scale environment, the traditional cloud-based CI solutions like GitHub Actions can seem like a distant dream. The primary reasons for this are the cost of cloud runners, the complexity of hairpin NAT, the need for internal registry DNS, and the operational burden on a single administrator.

Cost of Cloud Runners

Cloud runners, while convenient, come at a cost. For small projects, the recurring charges can add up quickly. In a homelab setting, where resources are limited and budget is often non-existent, cloud runners are often out of the question. Moreover, the cost of cloud runners can vary significantly depending on the provider and the region, making it difficult to predict or budget for.

Hairpin NAT and Internal Registry DNS

Hairpin NAT is a common issue in homelab environments. When a pod needs to access an internal registry, it often requires a workaround due to the way Kubernetes handles network traffic. This can be particularly challenging when trying to set up a webhook to push images to an internal registry. The lack of a proper DNS resolution for internal services can lead to connectivity issues, making it difficult to establish reliable CI pipelines.

Single-Admin Ops Burden

In a homelab, the burden of managing CI/CD pipelines falls on a single administrator. This means that the solution needs to be robust, yet straightforward enough to manage without the need for extensive technical expertise. The complexity of setting up and maintaining CI/CD pipelines in a cloud environment can be overwhelming, and the need for a single point of contact can make troubleshooting and maintenance a significant challenge.

Example Scenario

Consider a scenario where you have a small Kubernetes cluster running in your home lab. You want to set up a CI pipeline for a series of applications, but you don't want to spend money on cloud runners. You also need to ensure that your internal registry is accessible from the CI pipeline. The lack of proper DNS resolution and the complexity of hairpin NAT can make this a daunting task.

# Example of a problematic setup
kubectl apply -f https://example.com/tekton-crd.yaml
kubectl apply -f https://example.com/tekton-pipelines.yaml

2. Tekton vs ARC: When Actions Runner Controller is Enough vs When Tekton Pipelines + Triggers Fit a Multi-Repo Homelab

GitHub Actions Runner Controller (ARC) is a popular choice for CI in homelab environments because it is simple to set up and manage. However, for more complex scenarios, such as multi-repo environments, Tekton pipelines with Triggers can provide a more robust solution. Let's explore the differences between these two approaches.

GitHub Actions Runner Controller (ARC)

ARC is a lightweight solution that allows you to run GitHub Actions on your Kubernetes cluster. It is easy to set up and manage, making it a popular choice for small-scale projects. However, it has limitations when it comes to more complex CI/CD workflows.

# Example of ARC setup
kubectl apply -f https://github.com/actions/runner-controller/releases/download/v1.0.12/runner-controller.yaml

Tekton Pipelines + Triggers

Tekton pipelines offer more flexibility and power for complex CI/CD workflows. They allow you to define custom pipelines and triggers, making it easier to manage multi-repo environments. However, setting up Tekton can be more complex and requires a deeper understanding of Kubernetes and CI/CD concepts.

# Example of Tekton setup
kubectl apply -f https://github.com/tektoncd/pipeline/releases/download/v0.27.0/tekton-pipelines.yaml

When to Use ARC

ARC is a good choice when you have a simple CI/CD workflow and don't need the advanced features of Tekton. It is easy to set up and manage, making it a good fit for small-scale projects.

When to Use Tekton

Tekton is a better choice when you have a multi-repo environment and need more advanced CI/CD workflows. It allows you to define custom pipelines and triggers, making it easier to manage complex workflows.

Example Scenario

Consider a scenario where you have a multi-repo environment with multiple developers contributing to different projects. You want to set up a CI pipeline that triggers builds and tests for each repository. In this case, Tekton pipelines with Triggers would be a better choice than ARC.

# Example Tekton pipeline
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: multi-repo-ci
spec:
  workspaces:
  - name: source
  tasks:
  - name: build-and-test
    taskRef:
      name: build-and-test-task
    workspaces:
    - name: source
      workspace: source

3. Layout: Namespaces (ci-triggers, ci-tenant-*), EventListener, GPU Node Avoidance for CI Pods

To set up a reliable CI pipeline in a homelab environment, it is essential to organize your resources and avoid common pitfalls. This section will cover the layout of your Kubernetes cluster, including namespaces, EventListener, and GPU node avoidance.

Namespaces

Namespaces are a powerful feature in Kubernetes that allow you to organize your resources. In a CI/CD environment, it is a good practice to use separate namespaces for different components. This makes it easier to manage and troubleshoot your pipelines.

# Create namespaces
kubectl create namespace ci-triggers
kubectl create namespace ci-tenant-infra-bootstrap

EventListener

EventListener is a Kubernetes resource that watches for events and triggers actions based on those events. In a CI/CD environment, you can use EventListener to trigger builds and tests when changes are pushed to your repositories.

# Example EventListener
apiVersion: eventing.knative.dev/v1
kind: EventListener
metadata:
  name: ci-trigger
  namespace: ci-triggers
spec:
  subscriber:
    ref:
      apiVersion: serving.knative.dev/v1
      kind: Service
      name: ci-trigger-subscriber

GPU Node Avoidance

In a homelab environment, it is common to have a mix of CPU and GPU nodes. However, CI/CD pods should be run on CPU nodes to avoid any potential issues with GPU drivers or conflicts. To avoid running CI/CD pods on GPU nodes, you can use node selectors or affinity rules.

# Example node selector
apiVersion: batch/v1
kind: Job
metadata:
  name: ci-pod
spec:
  template:
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/arch
                operator: NotIn
                values:
                - nvidia
      containers:
      - name: ci-pod
        image: <image:tag>

Example Scenario

Consider a scenario where you have a mix of CPU and GPU nodes in your Kubernetes cluster. You want to ensure that your CI/CD pods are run on CPU nodes to avoid any potential issues. In this case, you can use node selectors or affinity rules to avoid running CI/CD pods on GPU nodes.

# Example node selector
apiVersion: batch/v1
kind: Job
metadata:
  name: ci-pod
spec:
  template:
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/arch
                operator: NotIn
                values:
                - nvidia
      containers:
      - name: ci-pod
        image: <image:tag>

4. Kaniko / Build → registry-internal.*: Push Pattern, Tag Discipline, Tie-In to ImagePullBackOff Prevention

Kaniko is a powerful tool for building container images directly on your Kubernetes cluster. In a homelab environment, it is essential to use Kaniko to build images and push them to an internal registry. This section will cover the push pattern, tag discipline, and how to tie it in with ImagePullBackOff prevention.

Push Pattern

The push pattern involves building images using Kaniko and pushing them to an internal registry. This pattern is essential in a homelab environment because it allows you to avoid the overhead of cloud runners and ensures that your images are stored locally.

# Example Kaniko build
kubectl run -it --rm --image=gcr.io/kaniko-project/executor:latest --restart=Never \
  --env=REGISTRY=<registry-host> \
  --env=IMAGE=<image:tag> \
  -- /bin/sh -c "export REGISTRY_PASSWORD=$(kubectl get secret -n ci-tenant-infra-bootstrap registry-credentials -o jsonpath='{.data.password}' | base64 --decode); \
  export REGISTRY_USERNAME=$(kubectl get secret -n ci-tenant-infra-bootstrap registry-credentials -o jsonpath='{.data.username}' | base64 --decode); \
  /kaniko/executor --dockerfile=Dockerfile --destination=<registry-host>/<image:tag>"

Tag Discipline

Tag discipline is essential in a CI/CD environment because it ensures that your images are always up-to-date and can be easily rolled back. In a homelab environment, it is a good practice to use semantic versioning for your tags.

# Example tag discipline
apiVersion: batch/v1
kind: Job
metadata:
  name: ci-pod
spec:
  template:
    spec:
      containers:
      - name: ci-pod
        image: <registry-host>/<image:tag>:<version>

Tie-In to ImagePullBackOff Prevention

ImagePullBackOff is a common issue in Kubernetes when a pod cannot pull an image from a registry. To prevent this issue, you can use a combination of tag discipline and a custom webhook to ensure that the correct image is always available.

# Example webhook
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: ci-webhook
webhooks:
- name: ci-webhook.example.com
  rules:
  - operations: ["CREATE", "UPDATE"]
    apiGroups: ["batch"]
    apiVersions: ["v1"]
    resources: ["jobs"]
  clientConfig:
    service:
      namespace: ci-triggers
      name: ci-webhook-service
    caBundle: <base64-encoded-certificate>
  admissionReviewVersions: ["v1", "v1beta1"]
  sideEffects: None
  failurePolicy: Ignore
  sideEffects: None
  timeoutSeconds: 30
  matchPolicy: Exact

Example Scenario

Consider a scenario where you have a CI/CD pipeline that builds and pushes images to an internal registry. You want to ensure that the correct image is always available and prevent ImagePullBackOff issues. In this case, you can use a combination of tag discipline and a custom webhook to ensure that the correct image is always available.

# Example webhook
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: ci-webhook
webhooks:
- name: ci-webhook.example.com
  rules:
  - operations: ["CREATE", "UPDATE"]
    apiGroups: ["batch"]
    apiVersions: ["v1"]
    resources: ["jobs"]
  clientConfig:
    service:
      namespace: ci-triggers
      name: ci-webhook-service
    caBundle: <base64-encoded-certificate>
  admissionReviewVersions: ["v1", "v1beta1"]
  sideEffects: None
  failurePolicy: Ignore
  sideEffects: None
  timeoutSeconds: 30
  matchPolicy: Exact

5. L0 Before Webhooks: run-pilot-l0.sh Smoke on Current HEAD — Why Manual PipelineRun Beats Debugging GitHub Delivery First

5. L0 Before Webhooks: `run-pilot-l0.sh` Smoke on Current `HEAD` — Why Manual PipelineRun Beats Debugging GitHub Delivery First
5. L0 Before Webhooks: `run-pilot-l0.sh` Smoke on Current `HEAD` — Why Manual PipelineRun Beats Debugging GitHub Delivery First

Before setting up webhooks, it is essential to perform a manual smoke test to ensure that your CI/CD pipeline is working correctly. This section will cover the run-pilot-l0.sh script and why a manual PipelineRun beats debugging GitHub delivery first.

run-pilot-l0.sh Script

The run-pilot-l0.sh script is a simple script that runs a manual PipelineRun to ensure that your CI/CD pipeline is working correctly. This script is particularly useful in a homelab environment because it allows you to test your pipeline without relying on external triggers.

# Example run-pilot-l0.sh
#!/bin/bash

# Set environment variables
export REGISTRY=<registry-host>
export IMAGE=<image:tag>

# Run PipelineRun
kubectl apply -f - <<EOF
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  name: pilot-l0
spec:
  pipelineRef:
    name: ci-pipeline
  workspaces:
  - name: source
    workspace: source
  params:
  - name: registry
    value: $REGISTRY
  - name: image
    value: $IMAGE
EOF

Why Manual PipelineRun Beats Debugging GitHub Delivery First

Debugging GitHub delivery can be a complex and time-consuming process. In a homelab environment, it is often easier to perform a manual PipelineRun to ensure that your pipeline is working correctly. This approach allows you to test your pipeline without relying on external triggers, making it easier to identify and fix any issues.

Example Scenario

Consider a scenario where you have a CI/CD pipeline that builds and pushes images to an internal registry. You want to ensure that the pipeline is working correctly before setting up webhooks. In this case, you can use the run-pilot-l0.sh script to perform a manual PipelineRun and ensure that the pipeline is working correctly.

# Example run-pilot-l0.sh
#!/bin/bash

# Set environment variables
export REGISTRY=<registry-host>
export IMAGE=<image:tag>

# Run PipelineRun
kubectl apply -f - <<EOF
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  name: pilot-l0
spec:
  pipelineRef:
    name: ci-pipeline
  workspaces:
  - name: source
    workspace: source
  params:
  - name: registry
    value: $REGISTRY
  - name: image
    value: $IMAGE
EOF

6. Failure Modes: Webhook Signature, Ingress Port 8080 Mismatch, Task Pod Stuck Init on Bad runc Nodes, Status Token RBAC

In a CI/CD environment, failure modes can arise from various sources. This section will cover common failure modes such as webhook signature issues, ingress port mismatches, task pod stuck in Init state, and RBAC issues.

Webhook Signature

Webhook signatures are used to ensure that webhooks are coming from a trusted source. In a homelab environment, it is essential to ensure that webhook signatures are correctly configured to prevent unauthorized access.

# Example webhook signature
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: ci-webhook
webhooks:
- name: ci-webhook.example.com
  rules:
  - operations: ["CREATE", "UPDATE"]
    apiGroups: ["batch"]
    apiVersions: ["v1"]
    resources: ["jobs"]
  clientConfig:
    service:
      namespace: ci-triggers
      name: ci-webhook-service
    caBundle: <base64-encoded-certificate>
  admissionReviewVersions: ["v1", "v1beta1"]
  sideEffects: None
  failurePolicy: Ignore
  sideEffects: None
  timeoutSeconds: 30
  matchPolicy: Exact
  webhookClientConfig:
    service:
      namespace: ci-triggers
      name: ci-webhook-service
    caBundle: <base64-encoded-certificate>

Ingress Port 8080 Mismatch

Ingress port mismatches can cause issues with webhooks. In a homelab environment, it is essential to ensure that the ingress port is correctly configured to prevent any issues with webhooks.

# Example ingress port mismatch
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ci-ingress
spec:
  rules:
  - host: <hostname>
    http:
      paths:
      - path: /webhook
        pathType: Prefix
        backend:
          service:
            name: ci-webhook
            port:
              number: 8080

Task Pod Stuck Init on Bad runc Nodes

Task pods can get stuck in the Init state on bad runc nodes. In a homelab environment, it is essential to ensure that your runc nodes are healthy and up-to-date to prevent any issues with task pods.

# Example runc node health
apiVersion: node.k8s.io/v1
kind: Node
metadata:
  name: <node-name>
spec:
  taints:
  - key: "runc-node"
    effect: "NoSchedule"
    value: "true"

Status Token RBAC

RBAC (Role-Based Access Control) is essential in a CI/CD environment to ensure that only authorized users can access resources. In a homelab environment, it is essential to ensure that RBAC is correctly configured to prevent any unauthorized access.

# Example RBAC
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ci-webhook
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["batch"]
  resources: ["jobs"]
  verbs: ["create", "update"]

Example Scenario

Consider a scenario where you have a CI/CD pipeline that uses webhooks to trigger builds and tests. You want to ensure that the pipeline is working correctly and prevent any issues with webhook signatures, ingress port mismatches, task pod stuck in Init state, and RBAC issues. In this case, you can use the above examples to configure your webhook, ingress, runc nodes, and RBAC to prevent any issues.

# Example webhook signature
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: ci-webhook
webhooks:
- name: ci-webhook.example.com
  rules:
  - operations: ["CREATE", "UPDATE"]
    apiGroups: ["batch"]
    apiVersions: ["v1"]
    resources: ["jobs"]
  clientConfig:
    service:
      namespace: ci-triggers
      name: ci-webhook-service
    caBundle: <base64-encoded-certificate>
  admissionReviewVersions: ["v1", "v1beta1"]
  sideEffects: None
  failurePolicy: Ignore
  sideEffects: None
  timeoutSeconds: 30
  matchPolicy: Exact
  webhookClientConfig:
    service:
      namespace: ci-triggers
      name: ci-webhook-service
    caBundle: <base64-encoded-certificate>

7. Runbook Pointer

If you encounter any issues with your CI/CD pipeline, you can refer to the runbook for detailed instructions on how to troubleshoot and resolve common issues. The runbook is available at the following URL:

Runbook URL: https://example.com/runbook/tekton-homelab-ci

By following the runbook, you can ensure that your CI/CD pipeline is working correctly and prevent any issues from arising.

Example Runbook

# Runbook: Tekton on a Homelab

## Troubleshooting

- **Webhook Signature Issues:** Ensure that webhook signatures are correctly configured.
- **Ingress Port Mismatch:** Check the ingress port configuration.
- **Task Pod Stuck Init:** Ensure that runc nodes are healthy and up-to-date.
- **RBAC Issues:** Ensure that RBAC is correctly configured.

## Next Steps

<figure class="figure">
<img src="/media/articles/tekton-homelab-ci/fig-03-next-steps.svg" alt="Next Steps" loading="lazy" />
<figcaption>Next Steps</figcaption>
</figure>

- Perform a manual PipelineRun using `run-pilot-l0.sh` to ensure that your pipeline is working correctly.
- Set up webhooks to trigger builds and tests.
- Monitor your pipeline for any issues and refer to the runbook for troubleshooting.

By following the above steps and referring to the runbook, you can ensure that your CI/CD pipeline is working correctly and prevent any issues from arising.

Related internal runbook: tekton-homelab-ci (VPN)