Halo! Saya Ask Owen. Tanyakan tentang proyek, pengalaman, skill, pendidikan, atau cara menghubungi Owen.
Automation
Automating TLS Certificate Rotation with Ansible, Vault, and Kubernetes
A practical, public-safe design for turning certificate retrieval, Kubernetes updates, and verification into one controlled workflow.
Certificate rotation looks simple when it is written as “replace the certificate.” In practice, the change crosses a secret store, an automation runner, a Kubernetes Secret, an ingress layer, and the endpoint that users actually reach.
The useful unit of automation is therefore not the update command. It is the complete path from retrieving the intended certificate to proving that the intended endpoint serves it.
Start with the state transition
Before writing a playbook, I describe the states that matter:
- The certificate exists in the approved secret store.
- The automation runner can access only the required material.
- The target Kubernetes Secret contains the new certificate and key.
- The ingress layer has reloaded the change when a reload is required.
- The externally served certificate matches the expected fingerprint and validity window.
This turns an ambiguous maintenance task into a sequence with observable checkpoints.
Keep secret handling narrow
The runner should retrieve only the certificate material needed for the target change. Sensitive values should stay out of logs, command output, and persisted temporary files.
- name: Read certificate material
ansible.builtin.set_fact:
tls_certificate: "{{ secret_response.certificate }}"
tls_private_key: "{{ secret_response.private_key }}"
no_log: trueno_log is not a complete security model, but it closes an easy leak path. Access policy, runner isolation, and the lifetime of temporary values still need to be designed explicitly.
Separate preflight from mutation
A safe workflow should fail before it changes the cluster when the inputs are incomplete or inconsistent. My preflight checklist includes:
- certificate and private key are both present;
- the certificate is not already expired;
- the key matches the certificate;
- the target context and namespace are explicit;
- the expected resource exists;
- the operator can see what will change.
The mutation step can then be kept small: update the Secret, request the necessary reload, and wait for readiness.
kubectl create secret tls public-tls \
--cert=certificate.pem \
--key=private-key.pem \
--dry-run=client -o yamlGenerating the manifest first creates a review point. The final application remains a separate, deliberate action.
Verification is part of the change
A successful Kubernetes update only proves that the API accepted a resource. It does not prove that the public endpoint is serving the new certificate.
I use two checks:
- compare the expected fingerprint with the certificate stored in the cluster;
- connect to the served endpoint and compare the certificate presented there.
That distinction catches stale ingress state, an unexpected load-balancer path, or a rollout that never reached the component terminating TLS.
Make restarts controlled and visible
Some ingress configurations detect Secret changes automatically. Others need an explicit rollout or reload. The workflow should not assume either behavior.
When a restart is required, I treat it as its own guarded step:
- Record the current workload generation.
- Trigger the narrowest available restart.
- Wait for readiness with a bounded timeout.
- Stop and report the last known state if readiness fails.
- Re-run endpoint verification.
This makes failure easier to locate. A rotation can fail during secret retrieval, resource update, rollout, or endpoint verification—and each failure needs a different next action.
Design idempotency around the fingerprint
Rotation automation should be safe to run twice. If the current and desired fingerprints match, the workflow can report “already current” and skip the mutation and restart steps.
That small decision reduces unnecessary changes and makes retries less risky. It also produces a cleaner audit trail: the playbook can explain whether it changed the system, verified an existing state, or stopped on a failed precondition.
What a complete run should leave behind
A useful run result is compact and operational:
- target environment and resource, without secret values;
- old and new certificate fingerprints;
- validity dates;
- whether the Secret changed;
- whether a reload occurred;
- endpoint verification status;
- the exact failed checkpoint when the run stops.
The main lesson is straightforward: automation should not hide the workflow. It should make the workflow, its permissions, and its evidence easier to inspect.
