Prerequisite: Complete TASK-F-GITLAB-CI-ONLY-GUIDE.md first. Your CI pipeline must be green — image in the registry, all stages passing — before starting this guide.
This guide adds the CD (Continuous Delivery) layer. When complete, every push to develop automatically updates the dev environment. Pushes to release/* branches update UAT. Production is promoted manually via a merge request.
By the end of this guide you will have:
- Two new CI stages:
update-devandupdate-uatthat commit image tags to your CD repo - Argo CD watching your CD repo and automatically deploying to
healthpulse-dev - Manual-sync gates for UAT and prod
- A prod promotion workflow via merge request — no pipeline ever writes to prod directly
GitOps means Git is the source of truth for what is deployed. Instead of CI running kubectl apply directly, CI writes to Git and a separate tool (Argo CD) applies what Git says.
CI Repo (source code) CD Repo (kubernetes manifests)
──────────────────── ──────────────────────────────
src/ kubernetes/
.gitlab-ci.yml app-manifests/
Dockerfile deployment.yml
service.yml
On every push to develop: kustomization.yaml
1. CI builds the image ├── overlays/
2. CI pushes to registry │ ├── dev/kustomization.yaml ← CI updates newTag here
3. CI clones CD repo │ ├── uat/kustomization.yaml ← CI updates on release/*
4. CI runs kustomize edit │ └── prod/kustomization.yaml ← PR only, never by CI
5. CI commits + pushes └── argocd/
healthpulse-dev.yml
healthpulse-uat.yml
healthpulse-prod.yml
Why two repos?
| Reason | Detail |
|---|---|
| Audit trail | Every deployment is a commit in the CD repo — who deployed what, when, and why |
| Rollback | Revert the CD repo commit to roll back a deployment |
| Access control | Developers commit to the CI repo; only ops/leads can merge to the CD repo's prod overlay |
| No drift | Argo CD continuously reconciles — if someone does a manual kubectl apply, Argo CD reverts it |
Adding CD extends the pipeline with two new stages at the end:
install
└── npm ci
test (parallel)
├── lint
└── unit-tests
scan (parallel)
├── gitleaks
├── sonarqube
└── snyk-security
build
└── build-app
publish (parallel)
├── docker-publish
└── artifactory-upload
scan-image
└── image-scan
update-dev ← NEW: runs on 'develop' branch only
└── update-dev-manifest — clones CD repo, updates dev overlay, commits
update-uat ← NEW: runs on 'release/*' branches only
└── update-uat-manifest — clones CD repo, updates uat overlay, commits
Argo CD then picks up the commit and deploys:
CD repo commit (by CI)
│
▼
Argo CD detects new image tag in overlays/dev/kustomization.yaml
│
▼
healthpulse-dev: AUTO-SYNC → deploys immediately
healthpulse-uat: MANUAL-SYNC → Argo CD shows OutOfSync, operator clicks Sync
healthpulse-prod: MANUAL-SYNC + PR gate → no commit without a merged PR
- Go to GitLab → New project → Create blank project
- Name it
healthpulse-cd - Set visibility to Private
- Do not initialise with a README
The capstone provides the full Kubernetes manifest structure. Clone your CD repo and copy the kubernetes/ directory into it:
git clone https://gitlab.com/<your-group>/healthpulse-cd.git
cd healthpulse-cd
# Copy the kubernetes/ directory from the capstone into this repo
cp -r /path/to/healthpulse-capstone/kubernetes .
git add kubernetes/
git commit -m "feat: add kubernetes manifests and kustomize overlays"
git push origin mainYour CD repo should now look like:
healthpulse-cd/
└── kubernetes/
├── app-manifests/
│ ├── deployment.yml
│ ├── service.yml
│ ├── hpa.yml
│ └── kustomization.yaml
├── overlays/
│ ├── dev/
│ │ ├── kustomization.yaml ← CI will update newTag here
│ │ ├── namespace.yml
│ │ └── ingress.yml
│ ├── uat/
│ │ ├── kustomization.yaml ← CI will update newTag here on release/*
│ │ ├── namespace.yml
│ │ └── ingress.yml
│ └── prod/
│ ├── kustomization.yaml ← NEVER updated by CI
│ ├── namespace.yml
│ └── ingress.yml
└── argocd/
├── healthpulse-dev.yml
├── healthpulse-uat.yml
└── healthpulse-prod.yml
Each overlay's kustomization.yaml has a <REGISTRY> placeholder. Replace it with your actual $CI_REGISTRY_IMAGE value.
Find your registry image path: go to your CI repo → Deploy → Container Registry — the path shown is your CI_REGISTRY_IMAGE value (e.g. registry.gitlab.com/your-group/devops-projects).
kubernetes/overlays/dev/kustomization.yaml — update the newName line:
images:
- name: healthpulse-portal
newName: registry.gitlab.com/your-group/devops-projects/healthpulse-portal
newTag: latestDo the same for overlays/uat/kustomization.yaml and overlays/prod/kustomization.yaml.
Commit and push:
git add kubernetes/overlays/
git commit -m "fix: set registry image path in kustomize overlays"
git push origin mainKustomize is a Kubernetes configuration tool built into kubectl. It lets you define a base set of manifests and overlays per environment — without duplicating YAML or using templates.
kubernetes/app-manifests/ ← base: shared across all environments
deployment.yml Defines the Deployment, Service, HPA
service.yml Does NOT set namespace or image tag
hpa.yml
kustomization.yaml
kubernetes/overlays/dev/ ← overlay: dev-specific overrides only
kustomization.yaml Sets namespace + image tag for dev
namespace.yml Creates the healthpulse-dev namespace
ingress.yml Dev-specific hostname
The CI job runs one command:
kustomize edit set image healthpulse-portal=$CI_REGISTRY_IMAGE/$APP_NAME:$BUILD_VERSIONThis updates newTag: in overlays/dev/kustomization.yaml from:
images:
- name: healthpulse-portal
newName: registry.gitlab.com/your-group/devops-projects/healthpulse-portal
newTag: latest ← beforeto:
images:
- name: healthpulse-portal
newName: registry.gitlab.com/your-group/devops-projects/healthpulse-portal
newTag: 11-5dd92339 ← after (the new version)Argo CD detects this file change in Git and applies the updated image tag to the cluster — pulling the new image and rolling out a new deployment.
You can preview what Kustomize generates before applying:
# Preview what the dev overlay produces
kustomize build kubernetes/overlays/dev
# Apply directly to the cluster
kustomize build kubernetes/overlays/dev | kubectl apply -f -Argo CD runs the equivalent of kustomize build + kubectl apply automatically.
These variables go in your CI repo — they are used by the CI pipeline to authenticate with and push to the CD repo.
Go to your CI repo → Settings → CI/CD → Variables.
CI_JOB_TOKEN only allows pushing to the repo the pipeline is running in. To push to the CD repo you need a separate token.
- Go to GitLab → your avatar (top right) → Edit profile → Access tokens
- Click Add new token
- Name:
gitlab-ci-cd-push - Expiry: set an appropriate date
- Scope: tick write_repository
- Click Create personal access token
- Copy the token — you will not see it again
- In your CI repo Variables, add:
| Key | Value | Protected | Masked |
|---|---|---|---|
CD_REPO_TOKEN | The token you just copied | Yes | Yes |
CD_REPO_URL | gitlab.com/your-group/healthpulse-cd | No | No |
CD_REPO_URLformat: Do NOT includehttps://— the job prepends it. Just the domain and path:gitlab.com/your-group/healthpulse-cd
Add two new stages and two new jobs to your existing .gitlab-ci.yml in the CI repo.
stages:
- install
- test
- scan
- build
- publish
- scan-image
- update-dev # ← NEW: updates dev overlay on develop branch
- update-uat # ← NEW: updates uat overlay on release/* branches# =============================================================
# STAGE: UPDATE-DEV
# Runs ONLY on the 'develop' branch, NEVER on CI-generated commits.
#
# What it does:
# 1. Clones the CD repo
# 2. Runs kustomize edit set image — updates newTag in overlays/dev/
# 3. Commits and pushes with [skip ci] to prevent an infinite loop
#
# After this job, Argo CD detects the new tag in Git and auto-deploys
# to the healthpulse-dev namespace.
# =============================================================
update-dev-manifest:
stage: update-dev
image: alpine/git:v2.52.0
needs: [docker-publish]
rules:
- if: '$CI_COMMIT_MESSAGE =~ /\[(skip ci|ci skip)\]/i'
when: never # Never run on CI-generated commits
- if: '$CI_COMMIT_BRANCH == "develop"' # Only run on the develop branch
variables:
BUILD_VERSION: "${CI_PIPELINE_IID}-${CI_COMMIT_SHORT_SHA}"
before_script:
# Install kustomize v5.8.1 (latest stable) via direct GitHub release download.
# Pinned version avoids breakage from silent upstream changes.
# Do NOT use the hack/install_kustomize.sh script — it parses the GitHub API
# and has a history of intermittent failures when the API response format changes.
- apk add --no-cache curl
- curl -Lo kustomize.tar.gz https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2Fv5.8.1/kustomize_v5.8.1_linux_amd64.tar.gz
- tar -xzf kustomize.tar.gz
- mv kustomize /usr/local/bin/
- kustomize version
script:
# Clone the CD repo using the personal access token
- git clone "https://oauth2:${CD_REPO_TOKEN}@${CD_REPO_URL}.git" cd-repo
- cd cd-repo
- git config user.email "gitlab-ci@healthpulse.com"
- git config user.name "GitLab CI"
# Update the image tag in the dev overlay
- cd kubernetes/overlays/dev
- kustomize edit set image healthpulse-portal=${CI_REGISTRY_IMAGE}/${APP_NAME}:${BUILD_VERSION}
- git add kustomization.yaml
# [skip ci] prevents this commit from triggering a new pipeline in the CD repo
- git commit -m "ci: deploy dev -> ${BUILD_VERSION} [skip ci]"
- git push
environment:
name: development# =============================================================
# STAGE: UPDATE-UAT
# Runs ONLY on 'release/*' branches, NEVER on CI-generated commits.
#
# After this job, Argo CD shows healthpulse-uat as OutOfSync.
# The operator logs into Argo CD and clicks Sync to approve the deploy.
# Nothing deploys to UAT without a human action.
# =============================================================
update-uat-manifest:
stage: update-uat
image: alpine/git:v2.52.0
needs: [docker-publish]
rules:
- if: '$CI_COMMIT_MESSAGE =~ /\[skip ci\]/'
when: never
- if: '$CI_COMMIT_BRANCH =~ /^release\//' # Only on release/x.y.z branches
variables:
BUILD_VERSION: "${CI_PIPELINE_IID}-${CI_COMMIT_SHORT_SHA}"
before_script:
- apk add --no-cache curl
- curl -Lo kustomize.tar.gz https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2Fv5.8.1/kustomize_v5.8.1_linux_amd64.tar.gz
- tar -xzf kustomize.tar.gz
- mv kustomize /usr/local/bin/
script:
- git clone "https://oauth2:${CD_REPO_TOKEN}@${CD_REPO_URL}.git" cd-repo
- cd cd-repo
- git config user.email "gitlab-ci@healthpulse.com"
- git config user.name "GitLab CI"
- cd kubernetes/overlays/uat
- kustomize edit set image healthpulse-portal=${CI_REGISTRY_IMAGE}/${APP_NAME}:${BUILD_VERSION}
- git add kustomization.yaml
- git commit -m "ci: deploy uat -> ${BUILD_VERSION} [skip ci]"
- git push
environment:
name: uatWithout [skip ci], the pipeline would loop forever:
Developer pushes to develop
│
▼
CI builds image, updates CD repo
│
▼
CD repo push triggers a new pipeline ← LOOP
│
▼
New pipeline builds new image... ← INFINITE
Two safeguards stop this:
Safeguard 1 — [skip ci] in the commit message: GitLab reads the commit message natively. If it contains [skip ci] or [ci skip] (case-insensitive), GitLab skips the pipeline entirely for that commit. This is a built-in GitLab feature — no configuration required.
git commit -m "ci: deploy dev -> 12-abc1234 [skip ci]"Safeguard 2 — rules: when: never: The jobs have this as their first rule:
- if: '$CI_COMMIT_MESSAGE =~ /\[(skip ci|ci skip)\]/i'
when: neverThe i flag makes the match case-insensitive. Covering both [skip ci] and [ci skip] ensures the guard works regardless of which variant is used. Even if a pipeline somehow starts on one of these commits, these jobs will not run.
The
when: neverrule must be first in the rules list. Rules are evaluated in order — the first match wins.
Modern alternative — git push -o ci.skip: Git push options (Git 2.10+) offer a cleaner approach that doesn't embed text in the commit message:
git push -o ci.skipFor this guide's CI-generated commits, we use [skip ci] in the message because it is more portable — it works across all Git client versions and also skips merge request pipelines, which push options do not.
Argo CD Application CRDs tell Argo CD which Git repo and path to watch, and which Kubernetes cluster to deploy to.
Before applying, update the repoURL in each Argo CD Application file in your CD repo. Replace the GitHub placeholder with your actual GitLab CD repo URL.
kubernetes/argocd/healthpulse-dev.yml:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: healthpulse-dev
namespace: argocd
spec:
project: default
source:
repoURL: https://gitlab.com/your-group/healthpulse-cd.git # ← your CD repo
targetRevision: main
path: kubernetes/overlays/dev
destination:
server: https://kubernetes.default.svc
namespace: healthpulse-dev
syncPolicy:
automated:
prune: true # Remove resources deleted from Git
selfHeal: true # Revert manual kubectl changes (drift protection)
syncOptions:
- CreateNamespace=truekubernetes/argocd/healthpulse-uat.yml:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: healthpulse-uat
namespace: argocd
spec:
project: default
source:
repoURL: https://gitlab.com/your-group/healthpulse-cd.git # ← your CD repo
targetRevision: main
path: kubernetes/overlays/uat
destination:
server: https://kubernetes.default.svc
namespace: healthpulse-uat
syncPolicy:
syncOptions:
- CreateNamespace=true
# No 'automated' block — operator must click Sync in Argo CD UIkubernetes/argocd/healthpulse-prod.yml:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: healthpulse-prod
namespace: argocd
spec:
project: default
source:
repoURL: https://gitlab.com/your-group/healthpulse-cd.git # ← your CD repo
targetRevision: main
path: kubernetes/overlays/prod
destination:
server: https://kubernetes.default.svc
namespace: healthpulse-prod
syncPolicy:
syncOptions:
- CreateNamespace=true
# No 'automated' block — manual sync required
# Prod overlay is NEVER updated by CI — only via merged PRArgo CD needs to pull from your private GitLab CD repo. Add the repo credentials in the Argo CD UI:
- Open the Argo CD UI → Settings → Repositories
- Click Connect Repo
- Fill in:
- Connection method: HTTPS
- Project: leave as
default - Repository URL:
https://gitlab.com/your-group/healthpulse-cd.git - Username: your GitLab username (or the token name if using a Deploy Token)
- Password: a token with
read_repositoryscope (see token options below)
- Click Connect — verify it shows Successful
Which GitLab token to use:
| Token type | Where to create | Recommended? |
|---|---|---|
| Project Deploy Token | CD repo → Settings → Repository → Deploy tokens | ✅ Best — scoped to this repo only, not tied to a user account |
| Project Access Token | CD repo → Settings → Access Tokens | ✅ Good — repo-scoped, survives user account changes |
| Personal Access Token | GitLab → User Settings → Access Tokens |
For a Deploy Token: create it with read_repository scope, use the token username as the Username field and the token value as the Password field.
Argo CD version note: This guide is validated against Argo CD 3.x (latest stable). The Application CRD uses
apiVersion: argoproj.io/v1alpha1— this is still the only available version and has not changed. If you are upgrading from Argo CD 2.x, review the 2.14 → 3.0 upgrade guide for RBAC breaking changes before applying these manifests.
Run these once from a machine with kubectl access to the cluster where Argo CD is installed:
kubectl apply -f kubernetes/argocd/healthpulse-dev.yml
kubectl apply -f kubernetes/argocd/healthpulse-uat.yml
kubectl apply -f kubernetes/argocd/healthpulse-prod.ymlVerify they appear in Argo CD:
kubectl get applications -n argocd
# Expected:
# NAME SYNC STATUS HEALTH STATUS
# healthpulse-dev Synced Healthy
# healthpulse-uat Unknown Missing
# healthpulse-prod Unknown Missinghealthpulse-dev should go Synced / Healthy immediately (it reads whatever newTag is already in the overlay). UAT and prod will show Missing until a sync is triggered.
git checkout develop
# make any small change (e.g. update a comment)
git add .
git commit -m "feat: test full CI/CD pipeline"
git push origin developExpected pipeline run order:
install → lint + unit-tests → scan → build-app →
docker-publish + artifactory-upload → image-scan →
update-dev-manifest
Go to your CD repo on GitLab → Code → Commits. You should see a new commit:
ci: deploy dev -> 12-abc1234 [skip ci]
Click the commit → view kubernetes/overlays/dev/kustomization.yaml. The newTag should be updated to the new version.
Open the Argo CD UI. The healthpulse-dev application should show:
Status: Syncing... → Synced
Health: Progressing... → Healthy
This usually takes 1-3 minutes. Click on the application to see the sync history — the most recent entry will show ci: deploy dev -> 12-abc1234 [skip ci].
kubectl get pods -n healthpulse-dev
# Expected: pods Running
kubectl describe pod <pod-name> -n healthpulse-dev | grep Image
# Expected: Image: registry.gitlab.com/your-group/devops-projects/healthpulse-portal:12-abc1234git checkout develop
git checkout -b release/1.0.0
git push origin release/1.0.0Expected pipeline: update-uat-manifest runs instead of update-dev-manifest. The job commits the new image tag to overlays/uat/kustomization.yaml.
Go to Argo CD UI → healthpulse-uat application. It shows OutOfSync — the Git state has a new image tag that is not yet deployed.
Click Sync → Synchronize to approve the deployment. Argo CD pulls the new image and rolls it out to healthpulse-uat.
kubectl get pods -n healthpulse-uat
# Expected: pods Running with the new image tagWhy manual sync for UAT? Dev is auto-deploy because it is the feedback loop for developers — every push should be immediately visible. UAT is a gate — someone has decided this build is ready for testing and actively approves it going to UAT. The Sync click is that approval.
Production is never updated by CI directly. No stage in the pipeline writes to overlays/prod/. Prod is promoted manually through a merge request:
1. UAT has been tested and signed off
2. Open a MR in the CD repo:
- Copy the newTag value from overlays/uat/kustomization.yaml
- Paste it into overlays/prod/kustomization.yaml
- Title: "promote: prod -> 12-abc1234"
3. Team lead reviews and merges the MR
4. Operator goes to Argo CD → healthpulse-prod → clicks Sync
5. Production deploys
# kubernetes/overlays/prod/kustomization.yaml
images:
- name: healthpulse-portal
newName: registry.gitlab.com/your-group/devops-projects/healthpulse-portal
- newTag: 10-5dd92339 ← current prod
+ newTag: 12-abc1234 ← promoting from UAT| Concern | How this addresses it |
|---|---|
| Accidental deploy | A human must open a PR, another must review and merge |
| Audit trail | The MR records who promoted, when, and links to the pipeline that produced the image |
| Rollback | Revert the MR to roll back prod to the previous tag |
| Separation of duties | Developers push code; leads control what goes to prod |
Before marking this step complete, verify every item:
- CD repo created (
healthpulse-cd) withkubernetes/directory committed -
<REGISTRY>placeholders replaced with actual registry path in all three overlaykustomization.yamlfiles -
CD_REPO_TOKENvariable added to CI repo (masked, protected,write_repositoryscope) -
CD_REPO_URLvariable added to CI repo (nohttps://prefix) -
update-devandupdate-uatstages added to.gitlab-ci.yml -
update-dev-manifestandupdate-uat-manifestjobs added with correctrules: - Argo CD connected to the CD repo (private repo credentials added in Settings → Repositories)
- Argo CD Application CRDs applied:
healthpulse-dev,healthpulse-uat,healthpulse-prod - Push to
developtriggers full pipeline —update-dev-manifestcommits new tag to CD repo - No infinite loop — the
[skip ci]commit does not trigger another pipeline - Argo CD auto-syncs
healthpulse-devwithin 3 minutes of CD repo commit -
kubectl get pods -n healthpulse-devshows pods running the new image tag - Push to
release/1.0.0—update-uat-manifestcommits tag tooverlays/uat/ - Argo CD shows
healthpulse-uatas OutOfSync after the commit - Operator clicks Sync — UAT deploys successfully
- Prod promoted via MR in CD repo — team lead reviews, merges, operator syncs in Argo CD
remote: HTTP Basic: Access denied
fatal: Authentication failed for 'https://gitlab.com/your-group/healthpulse-cd.git/'
Check three things:
CD_REPO_TOKENscope — the token must havewrite_repositoryscope. Aread_repository-only token can clone but cannot push.CD_REPO_URLformat — must NOT includehttps://. The job prepends it:# Correct: CD_REPO_URL: gitlab.com/your-group/healthpulse-cd # Wrong — will produce https://https://...: CD_REPO_URL: https://gitlab.com/your-group/healthpulse-cd
- Token expiry — if the token has expired, generate a new one and update the variable.
/bin/sh: kustomize: not found
The before_script install failed. Check the full job log — look for the curl or tar step that errored.
The guide uses a direct pinned release download (v5.8.1). If this fails, check:
Wrong architecture — the guide uses
linux_amd64. If your runner is ARM64:- curl -Lo kustomize.tar.gz https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2Fv5.8.1/kustomize_v5.8.1_linux_arm64.tar.gzVersion no longer available — check the kustomize releases page for the latest version and update the URL.
Network issue in CI — if the GitHub releases URL is blocked, mirror the binary into Artifactory and download from there.
Do not use
hack/install_kustomize.sh— this script parses the GitHub API to find the latest release binary and has a documented history of intermittent failures (GitHub issues #4769, #3571) when the API response format changes. The direct download used in this guide is reliable and reproducible.
The [skip ci] tag is missing from the commit message, or the rules: are in the wrong order.
Check 1 — commit message: Go to the CD repo → Commits. Does the commit from CI contain [skip ci]?
Check 2 — rules order: The when: never rule must be first:
rules:
- if: '$CI_COMMIT_MESSAGE =~ /\[(skip ci|ci skip)\]/i'
when: never # ← MUST be first
- if: '$CI_COMMIT_BRANCH == "develop"'If the branch rule comes first, GitLab matches it on the [skip ci] commit and runs the job anyway.
Auto-sync is configured in healthpulse-dev.yml via the automated: block. Check that:
- The Application CRD was applied with the
automated:block present:syncPolicy: automated: prune: true selfHeal: true
- Re-apply if needed:
kubectl apply -f kubernetes/argocd/healthpulse-dev.yml - Check Argo CD has repo access — go to Settings → Repositories, verify the connection is Successful
Go to Argo CD UI → Settings → Repositories. If the CD repo shows a failed connection:
- Delete the existing repo connection
- Re-add it with a fresh Personal Access Token (minimum scope:
read_repository) - Verify: the status should show Successful
This usually means the name: in the images: block of kustomization.yaml does not match what you passed to kustomize edit set image.
The command:
kustomize edit set image healthpulse-portal=${CI_REGISTRY_IMAGE}/${APP_NAME}:${BUILD_VERSION}Updates the entry where name: healthpulse-portal. If your kustomization.yaml has a different name:
images:
- name: my-app ← must match the name in the kustomize command
newName: registry.../healthpulse-portal
newTag: latestFix: either update the name: in kustomization.yaml to healthpulse-portal, or update the kustomize command to use the name that matches.
kubectl describe pod <pod-name> -n healthpulse-dev | grep Image
# Shows old image tagCheck that the imagePullPolicy in deployment.yml allows pulling a new image:
Always— always pulls (use forlatestor mutable tags)IfNotPresent— only pulls if not cached (safe for immutable version tags like12-abc1234)
Since the pipeline uses immutable tags (pipeline-sha), IfNotPresent is correct. If pods are not updating, it may mean the new tag is not actually different from w