Triton health probes don’t just tell you if a service is alive; they fundamentally shape how Kubernetes decides when and where to send traffic.
Let’s see Triton’s readiness and liveness probes in action. Imagine we have a simple web service running in Kubernetes.
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-web-app
spec:
replicas: 2
selector:
matchLabels:
app: my-web-app
template:
metadata:
labels:
app: my-web-app
spec:
containers:
- name: web-app
image: nginx:latest # Replace with your actual image
ports:
- containerPort: 80
readinessProbe:
httpGet:
path: /healthz # A custom endpoint that checks app health
port: 80
initialDelaySeconds: 5
periodSeconds: 5
livenessProbe:
httpGet:
path: /healthz # Or a separate endpoint for just liveness
port: 80
initialDelaySeconds: 15
periodSeconds: 20
Here, nginx:latest is a placeholder. In a real scenario, this would be your application’s container image. The readinessProbe is configured to hit /healthz on port 80 every 5 seconds, with a 5-second initial delay. The livenessProbe does the same but with a 15-second initial delay and checks every 20 seconds.
This setup is crucial for managing traffic flow. When a pod starts, Kubernetes first waits for initialDelaySeconds to pass. Then, it starts executing the readinessProbe at the periodSeconds interval. If the probe returns an HTTP status code between 200 and 399 (inclusive), the pod is considered "ready." Only then does Kubernetes add the pod’s IP address to the Service’s endpoints and start sending it live traffic. If the readiness probe fails, the pod is removed from the Service’s endpoints, effectively taking it out of rotation without restarting it.
The livenessProbe acts as a watchdog. If the liveness probe fails repeatedly (after a certain number of consecutive failures, often 3 by default), Kubernetes assumes the container is unhealthy and restarts the pod. This is for situations where the application might be running but stuck in a dead loop or unresponsive, and a simple restart is the best recovery action.
The core problem these probes solve is gracefully handling application lifecycle events and transient failures. Without readiness probes, Kubernetes might send traffic to a pod that’s still initializing or in the process of starting up, leading to user-facing errors. Without liveness probes, a hung application would continue to receive traffic indefinitely until manually intervened with.
Think of readinessProbe as saying, "I’m ready to serve new requests right now." It controls whether the pod is available for new traffic. livenessProbe is more like, "Am I still alive and functioning correctly?" It controls whether the pod should be restarted if it’s not.
The most surprising thing about these probes is their impact on distributed system resilience. A well-tuned readiness probe can prevent cascading failures by ensuring that traffic is only directed to healthy, fully initialized instances. Conversely, a poorly configured probe, especially one that’s too aggressive or checks the wrong things, can cause instability. For instance, a readiness probe that fails because of a temporary, external dependency (like a database being slow to respond) will cause the pod to be removed from service, potentially overwhelming the remaining healthy instances.
The exact levers you control are the path, port, initialDelaySeconds, periodSeconds, timeoutSeconds, successThreshold, and failureThreshold. initialDelaySeconds is your friend during startup to give your app time to spin up. periodSeconds is how often you check. timeoutSeconds is how long you wait for a response before considering it a failure. successThreshold (default 1) is how many consecutive successes are needed after failures to mark it as healthy again. failureThreshold (default 3) is how many consecutive failures trigger a restart (for liveness) or removal from endpoints (for readiness).
The next concept you’ll likely grapple with is how to design probe endpoints that accurately reflect your application’s true health, especially in complex microservice architectures.