If you run infrastructure, the honest reading of that list is not that twelve new things have arrived. It is that ten of them are concepts you already operate, under names you already use, and two of them are genuinely new and are the ones that will hurt you. An agent loop is a reconciliation loop. Guardrails are admission control. Sandboxing is what you have been doing to untrusted workloads since cgroups.

This post is the translation table, and then the part the infographics leave out: exactly where each analogy breaks. The breaks are the interesting bit. If an agent were just a controller, you would already know how to run one.

TLDR

• Ten of the twelve terms map cleanly onto infrastructure primitives you already operate: control loops, IAM, sandboxes, admission policies, change gates, schedulers.
• The agent loop is a reconciliation loop with a nondeterministic controller. Same shape, and every operational assumption that depends on "same input, same output" stops holding.
• Tool use is an IAM question, not an AI question. An agent's blast radius is exactly the union of the credentials you handed its tools. Nothing about the model changes that.
• Prompt injection is privilege escalation with a content payload rather than a binary one, and your telemetry is a delivery channel for it.
• The two genuinely new things are nondeterminism and unbounded runtime cost. Neither has a good analogue in the infrastructure you already run.
• Ask the blast-radius question before the model question. Which credentials, which environments, and what does the audit trail actually record.

Prerequisites

• Working familiarity with containers and some orchestrator, most likely Kubernetes
• Some exposure to IAM or RBAC, at any level of enthusiasm
• Having read one agentic AI explainer and come away unsure what was actually being claimed

The translation table

Start here. This is the whole argument in one screen.

Ten of those twelve are re-labellings. That is not a criticism of the vocabulary. It is the reason infrastructure people are unusually well equipped to reason about agents, and unusually badly served by explainers pitched at executives.

Now the parts worth going into properly.

The agent loop is a reconciliation loop with one crucial difference

Every agentic explainer draws the same cycle: perceive, plan, act, observe, repeat. If you have written a Kubernetes controller, you have drawn that cycle yourself and called it something else.

The shape is identical. A controller watches the API server, compares actual state to the spec, acts to close the gap, and observes the result. An agent reads its context, plans a step, calls a tool, and observes the output. If you want the mechanics of the first one in detail, Write a Simple Kubernetes Operator builds one from scratch, and everything in it transfers. For the loop from the agent side, including why the thing that judges the work has to be separate from the thing that does it, see Stop Prompting, Start Looping.

Here is the difference, and it is not a small one. A controller is deterministic and an agent is not.

Give a controller the same cluster state twice and it produces the same action twice. That single property is load-bearing for almost everything you know about operating control loops. It is why you can test a controller, why you can reason about a stuck reconcile, why a rerun is a diagnostic tool rather than a gamble, and why "it did something different this time" is a bug report rather than expected behaviour.

An agent given identical inputs may take a different path. Not usually a wildly different one, but different enough that the following all stop being reliable:

• Reproducing a failure. Running it again is not a controlled experiment.
• Testing coverage. Passing once does not establish that the path is safe.
• Post-incident analysis. "Why did it do that" may have no better answer than "it sampled a different token".

Everything else in this post follows from that one property. The infrastructure analogies hold right up until they depend on determinism, and then they stop.

Tool use is an IAM problem wearing a new hat

This is the term that causes the most confused conversation, and it is the one with the cleanest answer.

An agent cannot do anything except through a tool. The model produces text. Text becomes an action only when something on your side takes that text and calls an API. So the question "what can this agent do to my infrastructure" has an exact answer, and it is not a question about the model at all:

An agent's blast radius is the union of the permissions held by every tool you gave it.

That is an IAM audit, and you already know how to do one. If the agent has a tool that calls kubectl with a kubeconfig bound to cluster-admin, then the agent is cluster-admin. No amount of instruction in a system prompt changes that, in the same way that telling an intern to be careful is not an access control mechanism.

The practical consequence is that the safety conversation should start with credentials, not with the model:

# The only question that actually bounds what an agent can do.
kubectl auth can-i --list --as=system:serviceaccount:agents:incident-responder

If that output frightens you, the model choice is irrelevant. If it is tightly scoped, then a bad plan produces a rejected API call rather than an outage.

MCP is a plugin interface, and it inherits plugin-interface problems

Model Context Protocol is the term most likely to be presented as more novel than it is. It is a protocol for exposing tools, data and prompts to an agent through a consistent interface, so a capability written once can be used by any client that speaks it.

Structurally, that is the same idea as CSI for storage or CNI for networking: a stable interface so that vendors write one implementation instead of one per consumer. We have written about when to reach for MCP versus a plain CLI, and the short version is that the answer is usually both.

What matters operationally is that a plugin interface is a supply chain. Each MCP server is code, from someone, running with access to whatever you gave it. That is the same trust question as a Helm chart, a Terraform provider or a GitHub Action, with the added wrinkle that an MCP server's tool descriptions are themselves text that reaches the model. Our writeup of the MCP design flaw and the RCE it enabled covers where that went wrong in practice.

Treat MCP servers the way you treat any third-party admission webhook or CSI driver: pin versions, read what you install, and do not run one you cannot attribute.

Guardrails are admission control, and they belong outside the agent

"Guardrails" in most explainers means rules and policies that limit unsafe actions. Written down like that, it sounds like something you configure inside the AI product.

The version that survives contact with production is the one you already run: policy enforced at the boundary the agent cannot reach past. An admission webhook does not ask the workload to behave. It rejects the request. RBAC does not trust the client's intent. It evaluates the call.

That distinction is the whole game. There are two places to put a guardrail:

1. In the prompt. "Never delete a production namespace." This is a strong suggestion to a nondeterministic system, and it is defeated by anything that alters the model's context, including a malicious log line.
2. In the enforcement layer. No delete permission on production namespaces. This is defeated by nothing, because the capability does not exist.

Prompt-level rules are worth having, in the same way that documentation and linting are worth having. They are not controls. If a guardrail matters, it belongs in RBAC, in OPA or Kyverno, in a network policy, or in the absence of a credential.

Grounding is observability, and it is also an attack surface

Grounding means connecting the model's output to real data instead of what it inferred. For infrastructure work, "real data" is your telemetry: metrics, logs, traces, and the live state of the API.

The upside is genuine, and it is the part of AI operations that is actually working today. An agent that reads real metrics before proposing a cause is doing what a good on-call engineer does. Our assessment of what AI SRE agents fix and break found the investigation half to be the solid half, and grounding is why.

The part the infographic cannot fit in a box is that grounding makes your telemetry an input to a decision-making system. Logs are attacker-influenced data. A log line is written by a request, and a request can be crafted. Once an agent reads logs and can act on them, a string in a log becomes a potential instruction.

This is prompt injection, and for infrastructure people the clearest framing is that it is privilege escalation with a content payload. The classic escalation path is untrusted input reaching a privileged interpreter. Here the interpreter is the model and the input is anything it reads: log lines, ticket text, commit messages, alert annotations, HTTP user agents.

The mitigations are the ones you would expect from that framing, and none of them are AI-specific:

• Keep the privileged action behind a check the model does not control
• Treat everything the agent reads as untrusted, including your own telemetry
• Scope credentials so a successful injection is bounded
• Log what the agent read as well as what it did, or you cannot reconstruct the escalation

Human-in-the-loop is a change gate, with the same failure mode

Human review and approval before sensitive actions. You run this already: pull request review, a manual approval step on a deploy pipeline, a break-glass procedure with a second pair of eyes.

Which means you already know how it fails. Approval gates decay into rubber stamps in direct proportion to how often they fire and how little context they carry. A reviewer facing the fortieth "agent wants to restart a pod" prompt of the day is not reviewing, they are clicking.

The lesson from change management transfers exactly:

• Gate on blast radius, not on action count. Restarting a stateless pod does not need a human. Anything touching persistent data or production networking does.
• Give the approver the diff, not the intent. "I will scale the deployment" is not reviewable. replicas: 3 -> 30 is.
• Make rejection cheap and normal. A gate nobody ever rejects is measuring nothing.

If your agent's approval prompt does not contain enough information to make an informed no, it is theatre with an audit trail.

Orchestrator, subagent, multi-agent: you have debugged this before

The last group is presented as the frontier: a manager layer that assigns tasks, specialised workers with narrow jobs, several agents collaborating on a workflow.

That is a distributed system. Specifically it is a scheduler, a set of workers, and shared state, which is the architecture of nearly everything you already operate.

So the fun part is that you can predict the failure modes without having run one:

• Partial failure. One subagent fails, the orchestrator does not notice, the workflow reports success. You have seen this in every job runner ever written.
• Duplicated work. Two agents assigned overlapping tasks both act, and the second undoes the first.
• Coordination cost exceeding the work. Passing context between agents costs tokens, and past a certain point the orchestration is more expensive than doing it in one place.
• No idempotency. Retrying a failed step re-runs a side effect. Same bug as a webhook without a deduplication key.

The design questions are the ones you would ask of any worker pool. What happens when a worker dies halfway? Is the unit of work idempotent? Where is the shared state, and what happens when two workers write it? Our on-call agent built on Mastra was killed with SIGKILL at the worst possible moment specifically to answer those, which is the right instinct to bring.

Memory and context window: state, and a resource limit

These two get flattened together in most explainers and they are quite different.

Memory is persistence. An agent with memory carries information between runs, which means it has state, which means all your stateful-workload instincts apply. Where does it live, what happens when it is lost, who can read it, and is it in your backup. The Deployment versus StatefulSet distinction is exactly the right lens: an agent with memory is not a stateless replica you can reschedule freely, and if that memory holds anything derived from production data, it inherits the same handling requirements as the data itself.

Context window is a resource limit. It is the amount the model can consider at once, and the operational behaviour when you exceed it is familiar: things get evicted. Early context drops out, and the agent forgets a constraint it was given at the start, in exactly the way a process forgets nothing gracefully when it hits a memory limit.

The practical consequence is that an instruction given early in a long-running agent session is not a durable constraint. It is a value in a buffer that is being evicted. This is another reason enforcement belongs outside the model: a rule in RBAC is still there on hour six, and a rule in the opening prompt may not be.

What is actually new

Strip out the re-labelled concepts and two things remain that have no clean equivalent in the infrastructure you already run.

Nondeterminism in the control loop. Every operational practice you have for control loops assumes reproducibility. Testing, staged rollout, incident reproduction, "revert and see if it stops" all lean on it. An agent breaks that assumption, and the honest response is not to pretend otherwise but to move the guarantees somewhere deterministic: enforce in policy, verify with checks the agent cannot influence, and treat its output as a proposal until something deterministic has validated it.

Runtime cost as a variable. A controller's cost is roughly fixed and predictable. An agent's cost is a function of how much it reads and how many times it loops, both of which vary per run and can be influenced by the input. A pathological case is not just slow, it is expensive, and there is no equivalent of a resources.limits block that the loop cannot argue with. Budget caps and iteration limits are not optimisations here, they are the same category of control as a memory limit.

The questions to ask before an agent touches production

None of this needs a policy document. It needs five answers.

1. Which credentials? Run the can-i --list for its identity. That output is the blast radius, and everything else is commentary.
2. Enforced where? For each safety rule, name the enforcement point. If the answer is the system prompt, it is not a control.
3. What does it read? Everything in that list is untrusted input, including your own logs and tickets.
4. What does the audit trail record? Actions alone are not enough. Without what it read, an injection is unreconstructable.
5. What is the cost ceiling? Per run and per day, enforced by something outside the loop.

Answer those and the model choice becomes what it should have been all along: an implementation detail you can change later.

Summary

The vocabulary is not the hard part, and it is mostly not new. An agent loop is a reconciliation loop, tool use is an IAM boundary, guardrails are admission control, grounding is observability, human-in-the-loop is a change gate, and orchestrators with subagents are a worker pool with all the partial-failure problems that implies.

Reading it that way does two useful things. It tells you that your existing instincts mostly transfer, which is more than most explainers will tell you. And it isolates the two places where they do not: a control loop that is not reproducible, and a running cost that is not bounded.

Those two are where the work is. Everything else you have been doing for years.

FAQ

Is an agent really just a control loop?
Structurally, yes, and the comparison holds until it depends on determinism. A controller given the same state acts the same way; an agent may not. Testing, reproduction and rollback all rest on that property, so they all need rethinking.

What is the single most useful control to add first?
A scoped identity. Most agent risk is credential risk, and giving the agent its own least-privilege service account bounds the damage from every other mistake, including a successful prompt injection.

Are prompt-level guardrails worthless then?
Not worthless, but they belong in the same category as documentation and linting: they improve the common case and they do not stop the adversarial one. Anything that must not happen belongs in RBAC, policy or the absence of a credential.

How is prompt injection different from ordinary injection?
Mostly in the payload. It is untrusted input reaching a privileged interpreter, which is a shape you already defend against. The awkward part is that the interpreter has no reliable syntax boundary between instructions and data, so escaping and parameterisation, the usual fixes, are not available.

Do I need a multi-agent setup?
Usually not at first. It is a distributed system, and it brings coordination overhead, partial-failure handling and token cost. Start with one agent and narrow tools, and split only when a single loop is demonstrably the bottleneck.

Where does MCP fit if we already have CLIs?
MCP standardises capability exposure across clients, and a CLI is often cheaper in tokens and already known to the model. Our comparison goes through the tradeoff properly; in practice most teams end up running both.

The parts that actually cost people data are in the lifecycle: who deletes what, when, and what survives. A default you never chose decides whether removing a PVC also destroys the disk behind it. An access mode that reads like a lock is not enforced at all. A volume you carefully set to Retain will sit in Released refusing every new claim until you edit a field nobody told you about.

This post is the anatomy: the five objects, how they bind, and the seven behaviours that surprise people. Every rule here is checked against the upstream Kubernetes documentation, and the exact strings and version numbers are quoted so you can verify them rather than take my word for it.

TLDR

• ReadWriteOnce means one node, not one pod. Several pods on the same node can all mount an RWO volume read-write. ReadWriteOncePod is the one that means what people assume RWO means.
• Access modes are not enforced. Upstream says plainly that RWO, ROX and RWX "don't set any constraints on the volume". Only ReadWriteOncePod is a real constraint.
• reclaimPolicy defaults to Delete. For dynamically provisioned volumes, deleting the PVC deletes the disk and the data on it.
• A PVC stuck in Terminating is usually working correctly. The kubernetes.io/pvc-protection finalizer holds it until no pod is using it.
• Retain does not mean reusable. The PV goes to Released and will not bind again while its claimRef is set.
• Volume expansion is one way. You can grow a PVC, never shrink it, and editing the PV's capacity by hand stops the resize from happening at all.
• StatefulSet PVCs outlive the StatefulSet by default. persistentVolumeClaimRetentionPolicy changes that, and it went GA in Kubernetes v1.32.

Prerequisites

• A Kubernetes cluster you can create and delete objects in, ideally not a production one
• kubectl configured against it
• Familiarity with pods and either Deployments or StatefulSets
• A CSI driver installed if you want to try dynamic provisioning, which is the default on every managed cloud offering

The five objects

Kubernetes storage is often described as two objects. It is really five, and the two that get left out are the ones that decide what happens to your data.

The split worth internalising is namespaced versus cluster-wide. A PVC lives in a namespace, belongs to a team, and is deleted when that namespace is deleted. A PV and a StorageClass are cluster objects owned by whoever runs the cluster. Deleting a namespace therefore deletes claims, and what that does to the underlying disks depends entirely on a policy set by someone else.

The fifth object, which you rarely write by hand, is the CSI driver. It is the thing that actually calls the cloud API to create a disk and attaches it to a node. When storage misbehaves in ways the objects above cannot explain, the driver's controller and node pods are where the answer is.

PV vs PVC: supply and demand

The cleanest way to hold the distinction is that a PVC is demand and a PV is supply.

A claim says what the workload needs, in the workload's own namespace, without knowing anything about the infrastructure:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-data
  namespace: databases
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: fast-ssd
  resources:
    requests:
      storage: 100Gi

A PersistentVolume is the supply side: a real piece of storage, described in cluster terms.

There are two ways supply appears, and knowing which one you are using tells you who is responsible when things go wrong.

Dynamic provisioning is what every managed cluster gives you by default. It is also why so many people have never looked at a PV object: one is quietly created and destroyed on their behalf, carrying policies they did not set.

Binding is one-to-one, and it is sticky

Once a claim finds a volume, the two are wired together permanently. Upstream is unambiguous:

Once bound, PersistentVolumeClaim binds are exclusive, regardless of how they were bound. A PVC to PV binding is a one-to-one mapping, using a ClaimRef which is a bi-directional binding between the PersistentVolume and the PersistentVolumeClaim.

Two consequences follow, and both catch people out.

You cannot point two claims at one volume to share it. If you need several pods writing to the same storage, that is an access mode and a driver question, not a binding question. One PV serves exactly one PVC.

The binding is recorded on both objects. The PV gets a claimRef naming the claim. This is the field that makes a Retained volume refuse to be reused, which we come to below.

If you want a specific claim to land on a specific volume, you pre-bind by naming the volume in the claim. Note the empty storageClassName, which upstream flags explicitly:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: foo-pvc
  namespace: foo
spec:
  storageClassName: "" # Empty string must be explicitly set otherwise default StorageClass will be set
  volumeName: foo-pv

Leave storageClassName off entirely and the default StorageClass is applied, dynamic provisioning kicks in, and you get a brand new empty disk instead of the volume you were trying to attach to. That is a genuinely nasty failure, because it looks like success: the pod starts, the mount is there, and the data is simply gone.

Access modes: the part almost everyone gets wrong

This is the single biggest misconception in Kubernetes storage, and it is worth stating bluntly.

ReadWriteOnce does not mean one pod. Here is the upstream definition, verbatim:

ReadWriteOnce: the volume can be mounted as read-write by a single node. ReadWriteOnce access mode still can allow multiple pods to access (read from or write to) that volume when the pods are running on the same node. For single pod access, please see ReadWriteOncePod.

So an RWO volume happily serves three pods at once, as long as the scheduler put them on the same node. Teams discover this when a rolling update briefly runs old and new pods together, both writing, and a database that assumed exclusive access finds its files corrupted. The behaviour is not a bug and it is not a driver quirk. It is the documented meaning of the mode.

The four modes and their kubectl abbreviations:

Now the second half, which is less known and more alarming. Access modes on a PV are, with one exception, not enforced by anything:

Even if the access modes are specified as ReadWriteOnce, ReadOnlyMany, or ReadWriteMany, they don't set any constraints on the volume. For example, even if a PersistentVolume is created as ReadOnlyMany, it is no guarantee that it will be read-only. If the access modes are specified as ReadWriteOncePod, the volume is constrained and can be mounted on only a single Pod.

Read that again. ReadOnlyMany does not make a volume read-only. The access mode is matching metadata used when pairing claims with volumes, not a lock applied to the storage. If you want a hard guarantee that exactly one pod can write, ReadWriteOncePod is the only mode that provides one, it is CSI-only, and it graduated to stable in Kubernetes v1.29.

The reclaim policy decides whether you keep your data

Every PV carries a persistentVolumeReclaimPolicy that says what happens when its claim goes away.

Delete removes the PV object and the storage asset in the external infrastructure. The disk is gone. This is the important part:

Volumes that were dynamically provisioned inherit the reclaim policy of their StorageClass, which defaults to Delete.

And on the StorageClass side:

If no reclaimPolicy is specified when a StorageClass object is created, it will default to Delete.

Put those together. On a default managed cluster, with a StorageClass nobody edited, kubectl delete pvc destroys the underlying disk. Delete a namespace and every claim in it goes, taking the disks with it. No confirmation, no soft delete, no recycle bin.

Retain keeps everything and hands you the cleanup. Recycle still appears in the API and is deprecated:

The Recycle reclaim policy is deprecated. Instead, the recommended approach is to use dynamic provisioning.

Treat Recycle as a historical artifact. The real choice is Delete or Retain.

The Retain trap

Setting Retain protects the data and then produces the second-most-common storage support ticket. When the claim is deleted, the volume moves to Released, and:

the PersistentVolume still exists and the volume is considered "released". But it is not yet available for another claim because the previous claimant's data remains on the volume.

A Released PV will not bind to a new claim. Not to an identical claim, not to one with the same name in the same namespace. The blocker is the claimRef still pointing at the claim that no longer exists. Clearing it is what returns the volume to Available:

# The volume is Released and no new claim will touch it.
kubectl get pv
# NAME       CAPACITY   RECLAIM POLICY   STATUS     CLAIM
# pv-data    100Gi      Retain           Released   databases/postgres-data

# Drop the stale binding to make it Available again.
kubectl patch pv pv-data -p '{"spec":{"claimRef": null}}'

The data on the volume is untouched by this. You are only removing the record of a binding to a claim that has been deleted.

Why your PVC is stuck in Terminating

You run kubectl delete pvc, the command returns, and the claim sits in Terminating indefinitely. Nothing is broken. This is Storage Object in Use Protection doing its job:

If a user deletes a PVC in active use by a Pod, the PVC is not removed immediately. PVC removal is postponed until the PVC is no longer actively used by any Pods.

The mechanism is a finalizer. Two exist, and their exact names are worth knowing because they show up in kubectl describe:

• kubernetes.io/pvc-protection on claims
• kubernetes.io/pv-protection on volumes

Why your pod is stuck in Pending

The other half of the stuck-object family, and this one is a StorageClass setting.

volumeBindingMode has two values. Immediate is the default and binds as soon as the claim is created. WaitForFirstConsumer delays binding until a pod actually needs the volume.

That delay is not laziness, it is topology. With Immediate, upstream notes that PVs "will be bound or provisioned without knowledge of the Pod's scheduling requirements", which "can result in unschedulable Pods". In plain terms: on a cloud with zones, an Immediate claim can provision a disk in eu-west-1a while the only node with capacity for your pod is in eu-west-1b. The disk cannot cross the zone boundary, the pod cannot be scheduled, and it waits forever.

WaitForFirstConsumer inverts the order. The scheduler picks a node first, then the volume is provisioned to match. If you run a multi-zone cluster, this is almost always what you want:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer

The diagnostic is quick. A pod in Pending with a claim in Pending and no provisioning events points at topology or at a missing default StorageClass. A pod in Pending with a claim already Bound points at the node the volume landed on.

Expansion only goes one way

Volume expansion has been stable since v1.24 and works like this: you edit the claim, requesting more, and the backing volume grows.

You can only use the volume expansion feature to grow a Volume, not to shrink it.

Two conditions and one trap.

The conditions: the StorageClass needs allowVolumeExpansion: true, and the CSI driver has to support resize. Without the first, the API rejects the edit.

The trap is that expansion is driven by the difference between the claim and the volume, so closing that gap by hand disables it:

Directly editing the size of a PersistentVolume can prevent an automatic resize of that volume. If you edit the capacity of a PersistentVolume, and then edit the .spec of a matching PersistentVolumeClaim to make the size of the PersistentVolumeClaim match the PersistentVolume, then no storage resize happens. The Kubernetes control plane will see that the desired state of both resources matches, conclude that the backing volume size has been manually increased and that no resize is necessary.

So the correct move is to edit the PVC and nothing else:

# Right: ask for more on the claim, let the controller do the rest.
kubectl patch pvc postgres-data -p '{"spec":{"resources":{"requests":{"storage":"200Gi"}}}}'

Since shrinking is impossible, over-provisioning a volume is a decision you cannot walk back. The only route down is to create a smaller volume and copy the data across.

StatefulSets: the claims outlive the workload

Deployments and StatefulSets treat storage completely differently, which is most of the reason StatefulSets exist. If that distinction is still fuzzy, Kubernetes Deployments vs StatefulSets covers it directly.

A StatefulSet's volumeClaimTemplates generate one claim per replica, named <template-name>-<statefulset-name>-<ordinal>. A template called www in a StatefulSet called web produces www-web-0, www-web-1, www-web-2. That naming is the mechanism behind stable identity: when web-1 is rescheduled, it is reattached to www-web-1 and gets its own data back rather than a fresh disk.

The behaviour that surprises people is what happens on scale-down and delete:

Deleting and/or scaling a StatefulSet down will not delete the volumes associated with the StatefulSet. This is done to ensure data safety, which is generally more valuable than an automatic purge of all related StatefulSet resources.

Scale from 5 to 3 and two claims stay behind, still billed, still holding data. Scale back to 5 and those same claims are picked up again, which is exactly what you want for a database and exactly what you do not want for a cache you have been scaling for a year.

To change it, set persistentVolumeClaimRetentionPolicy, which reached GA in Kubernetes v1.32:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
spec:
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain   # keep the data if someone deletes the StatefulSet
    whenScaled: Delete    # but reclaim it when scaling down
  replicas: 3
  volumeClaimTemplates:
    - metadata:
        name: www
      spec:
        accessModes: [ "ReadWriteOnce" ]
        storageClassName: fast-ssd
        resources:
          requests:
            storage: 10Gi

whenDeleted: Retain with whenScaled: Delete is a sensible pairing for most stateful workloads: scaling in is routine and reversible, deleting the StatefulSet is usually a mistake.

Reading the state of a volume

Four phases, and each one tells you which half of the system to look at:

A Released volume on a Delete policy that never disappears usually means the driver could not remove the backing disk, often because it was deleted out from under Kubernetes in the cloud console.

A checklist worth running against your cluster

None of this needs a rewrite of anything. It is four commands and a decision.

# 1. What is the default StorageClass, and does it delete data?
kubectl get storageclass -o custom-columns=\
'NAME:.metadata.name,RECLAIM:.reclaimPolicy,EXPAND:.allowVolumeExpansion,BINDING:.volumeBindingMode,DEFAULT:.metadata.annotations.storageclass\.kubernetes\.io/is-default-class'

# 2. Which volumes would take their disks with them?
kubectl get pv -o custom-columns='NAME:.metadata.name,POLICY:.spec.persistentVolumeReclaimPolicy,STATUS:.status.phase,CLAIM:.spec.claimRef.name'

# 3. Anything already stranded?
kubectl get pv --field-selector status.phase=Released

# 4. Claims nobody is using, quietly costing money
kubectl get pvc --all-namespaces

If step 1 shows Delete on the default class, that is the setting to think hardest about. The annotation that marks a class as default is storageclass.kubernetes.io/is-default-class: "true", and the reclaim policy on a StorageClass cannot be changed after creation, so the fix is a new class rather than an edit.

Note that a PV's reclaim policy can be patched in place, which is the fastest way to protect volumes that already exist:

kubectl patch pv pv-data -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

Summary

The object model is the easy half. A PVC is demand, a PV is supply, a StorageClass is the recipe, and a CSI driver does the work. Bind one-to-one, mount by claim name, done.

The half that decides whether you keep your data is the lifecycle, and it comes down to a few rules that are not obvious from the YAML:

• ReadWriteOnce is a node constraint, and access modes other than ReadWriteOncePod are not enforced at all
• reclaimPolicy defaults to Delete, so on an untouched cluster deleting a claim deletes the disk
• Retain leaves the volume in Released, and it stays unusable until claimRef is cleared
• Finalizers holding a Terminating PVC are protecting a volume that is still mounted, so find the pod rather than patching the finalizer away
• Expansion grows and never shrinks, and hand-editing PV capacity silently disables it
• StatefulSet claims survive scale-down and deletion unless persistentVolumeClaimRetentionPolicy says otherwise

For the wider operational picture around these objects, Real-World Kubernetes Deployments covers the neighbouring concerns: probes, resource limits and disruption budgets.

FAQ

Can two pods share one PersistentVolumeClaim?
Yes, if they land on the same node or if the volume is ReadWriteMany with a driver that supports it. What you cannot do is bind two claims to one volume, since binding is strictly one-to-one.

Does deleting a namespace delete the underlying disks?
It deletes every PVC in that namespace. Whether the disks go with them depends on the reclaim policy of each PV, which for dynamically provisioned volumes is inherited from the StorageClass and defaults to Delete.

Why is my PVC Pending with no events?
Usually no default StorageClass, or a storageClassName naming a class that does not exist. If the class uses WaitForFirstConsumer, Pending is also the correct state until a pod actually references the claim.

Can I change a PVC's access mode after creating it?
Not in place for the general case. The supported route for moving to ReadWriteOncePod is documented as a task upstream, and it involves the PV rather than editing the claim's mode directly.

Is it safe to delete a PV that shows as Released?
Only once you are certain the data is not needed, or the policy is Retain and you have copied it. On Retain the storage asset in the cloud survives the PV object, so deleting the PV does not free the disk or stop the bill.

Do I still need to care about in-tree volume plugins?
Mostly no. The cloud providers' in-tree plugins have been migrated to CSI, and new drivers are CSI only. It matters when reading older manifests, where a spec.awsElasticBlockStore block signals something worth modernising.

Often it is not. Often the problem is that a filter drops you before a human reads anything, or you cannot describe what you actually did, or the one thing you own has no name attached to it inside your own company.

Those are one-day problems. This is a list of eight, ordered by how much they change what happens to you in the next month rather than the next five years. Several come from things we measured rather than things that sound right, and where that is the case the evidence is linked.

Do the first three even if you do nothing else. They take an afternoon between them.

TLDR

• We counted 1,785 real job postings. Podman appears in zero of them. OpenTofu appears in seven, never without Terraform beside it.
• The synonym check is the highest-value 20 minutes in this list, and it is the one with numbers behind it.
• Buzzword padding is theatre. A 30-item skills list did not improve scores in our test. Exact nouns from the posting do.
• Write the three-boundary story. Interviewers are testing whether you debug boundaries or brands.
• Name one thing you own and tell someone. Most engineers have no answer to "what are you the person for?"
• Fix your on-call answer. It is the question candidates lose on and the one they never prepare.
• Career breaks cost points on six of eight models we tested. That is worth knowing before you explain yours.

How the posting numbers were gathered

Every percentage in the next section comes from the same corpus: all top-level comments in the Hacker News "Who is hiring" threads for March through August 2026, fetched from the public Algolia API. That is 1,785 postings, of which 338 mention DevOps, SRE, platform engineering or the core tooling.

It is a sample with a known bias. Hacker News skews toward startups and remote-friendly companies, so it under-represents enterprise hiring, where the exact-match filtering is usually worse rather than better. Treat the direction as solid and the precise percentages as indicative.

Prerequisites

• A current CV, even a bad one
• Two or three job postings you would genuinely apply to
• One uninterrupted afternoon

1. The synonym pass, 20 minutes

Start here because it is the cheapest thing on the list with the largest failure mode.

When we tested how AI screens DevOps resumes, the models were reasonable. They ranked strong, mid and weak candidates correctly, and swapping tool names for equivalents barely moved the score. Then a plain keyword-and-knockout filter, the kind that runs before any model, rejected the same engineer outright for writing OpenTofu where the posting said Terraform.

That filter cannot reason. It matches strings. So the job is to make sure the strings match.

To find out how bad the mismatch actually is, we counted. We pulled 1,785 real job postings from six months of Hacker News "Who is hiring" threads, March to August 2026, and kept the 338 that mention DevOps, SRE, platform or the core tooling. Then for each pair of equivalent terms we asked a narrow question: among postings that mention either form, how many mention only one?

Almost nothing names both. And two results are worth stating outright:

Podman appears in zero of 1,785 postings. Not zero of the infrastructure ones. Zero of all of them. OpenTofu appears in seven, and in every case alongside Terraform, never on its own.

So a CV that says Podman where the market says Docker, or OpenTofu where the market says Terraform, does not match a slightly smaller set of jobs. On an exact-match filter it matches nothing. You are not being judged on the substitution, you are being excluded before anyone sees it.

The rest split in ways worth knowing:

PostgreSQL versus Postgres is nearly a coin flip, which means picking one form and sticking to it costs you about half the postings that mention the database at all. Kubernetes versus K8s runs three to one, so writing only "K8s" is the more expensive mistake of the two.

The fix costs nothing. Write both forms once each:

Terraform (and OpenTofu)
Docker (and Podman)
Kubernetes / K8s
PostgreSQL (Postgres)
CI/CD and CICD
GitHub Actions (previously Jenkins)

Write years as numerals. "5 years" and "five years" are different strings to a regex, and only one of them is what the pattern is looking for.

This is not keyword stuffing. Stuffing is a 30-item skills list, and we measured that too: it did nothing. This is making sure the words you already earned are written in the form the machine is looking for.

2. The three-boundary story, 60 minutes

Every DevOps interview eventually asks a version of: something is broken, walk me through it. Most candidates answer with tools. "I'd check the logs. I'd look at Kubernetes."

That answer is weak because it is a list of places, not a method. Under pressure it turns into clicking around hoping something turns red.

Write out three incidents you were actually part of, in this shape:

1. What the user saw          "checkout returned 502s, dashboards all green"
2. What you thought first     "green dashboards means health checks test
                               something different from what users do"
3. How you narrowed it        "walked the request path: DNS, LB, ingress,
                               service, pod, dependency, until it stopped"
4. What it turned out to be   "readiness probe hit /health, which did not
                               touch the database the request needed"
5. What changed after         "probe now exercises the dependency; added an
                               alert on 5xx rate rather than pod status"

Step 3 is the one being graded. Interviewers are not checking whether you know what a service mesh is, they are checking whether you narrow systematically or guess. Step 5 is the one that separates senior answers: junior engineers fix the incident, senior engineers change the thing that let it happen.

If you cannot fill in step 5 for any of your three, that is genuinely useful information about your current role.

3. Fix your on-call answer, 30 minutes

Almost nobody prepares this and it comes up in nearly every interview, in both directions.

When they ask you: they want to know whether you have carried a pager and what you learned. "Yes, one week in four" is a fact, not an answer. Have one specific thing you changed because of on-call: an alert you deleted because it never meant anything, a runbook you wrote after being paged twice for the same thing, a threshold you moved.

Deleting a noisy alert is a genuinely strong answer, and it is one that people undersell because it feels like removing work rather than doing it.

When you ask them, and you should ask: how many people are in the rotation, what got paged last month, and what happens when someone is on holiday. A rotation of three is a different job from a rotation of ten. Most candidates find this out in week two of the new job, which is the worst possible time.

4. Name the one thing you own, 30 minutes

Ask yourself what you are the person for at your company. Not what you work on. What breaks and someone says your name.

A surprising number of experienced engineers cannot answer this, and it is the single biggest difference between people whose careers compound and people who stay level for four years while being very busy.

If you have an answer, say it out loud to your manager this week. "I want to be the person who owns our deployment pipeline" is a sentence that changes what work comes to you.

If you do not have one, pick something small, currently unowned and irritating enough that people complain about it. The flaky test suite. The Terraform module nobody understands. The alert that fires every Sunday. Own it publicly, fix it, and you now have an answer, a story for section 2, and a reason to be in the room next time it is discussed.

5. Write the internal README, 45 minutes

Pick the most confusing thing in your infrastructure and document it. Not comprehensively, just the part that costs people an hour whenever they meet it.

This is on the list for three reasons. It is the fastest way to become the person who understands that system, because writing it down is how you find out you did not. It is visible in a way that ordinary work is not. And it is one of the few artefacts you can point at in a performance review that is unambiguously yours.

Keep it to one page. The five-page version does not get written, and the one-page version gets read.

6. Update your CV while you still have the details, 45 minutes

Not a rewrite. Add the last six months while you still remember the numbers, because in a year you will not.

For each thing you did, write it in this shape:

Weak:    "Responsible for CI/CD pipelines"
Better:  "Owned the CI pipeline for 40 engineers"
Best:    "Cut CI wall time from 22 to 9 minutes by splitting the test
          suite and caching dependencies, for 40 engineers"

The difference is not writing skill, it is whether you kept the numbers. Go and get them now: your CI dashboard, your incident tracker, your cloud bill. Twenty minutes of digging gives you a year of specifics.

One honest note on scope. Say what you did. "We migrated to Kubernetes" tells a reader nothing about you. "I moved 12 of our 30 services, and wrote the migration guide the rest of the team used" does, and is checkable.

7. Decide what you are aiming at, 30 minutes

DevOps splits into paths that look similar from inside and are quite different jobs: platform engineering, SRE, cloud infrastructure, security, and the generalist who does all of it at a smaller company.

You do not need to commit for five years. You need to know which one you are aiming at this year, because it changes what you say yes to. Someone aiming at platform engineering should be taking the internal-tooling work. Someone aiming at SRE should be taking the on-call and reliability work. Both are "DevOps" and they compound in different directions.

We wrote about the five paths here if it helps to see them side by side. The point of this half hour is one sentence: "this year I am aiming at X, so I will take more Y work."

8. If you have a career break, decide how you handle it

This one is uncomfortable and it is on the list because we measured it rather than assumed it.

In our resume test, adding a 14-month caregiving break to an otherwise identical CV cost points on six of the eight models, from 1.0 up to 7.6 out of 100. Same person, same experience, same everything else. The break was the only difference.

That is not a reason to hide it, and hiding gaps tends to fail anyway. It is a reason to not leave the reader to fill in the blank themselves. A single line stating the period and, if you did anything technical during it, what you kept current, removes the ambiguity the scoring was punishing.

Worth being clear about what this finding is: evidence that the systems in the pipeline treat breaks as a signal. It is not an endorsement of that. If you are on the hiring side of this, the actionable version is to check whether your own screening does the same thing, because it very likely does and nobody has looked.

What this list deliberately leaves out

No certifications. Not because they are worthless, but because they are not a one-day task and their return varies enormously by market and employer.

No personal brand, no posting cadence, no side project. Those are multi-month commitments and they are what most articles like this recommend precisely because they sound impressive rather than because they are the binding constraint.

The binding constraint, for most people who feel stuck, is one of the first four things on this list. A filter rejecting you on a synonym. Not being able to tell the story of your own work. Nobody knowing what you own.

The afternoon version

If you only have a few hours:

Under three hours, and it addresses the reasons people are actually stuck rather than the reasons that are pleasant to talk about.

FAQ

Can you really fix a career in a day?
No, and the title is doing some work. What you can fix in a day is the set of avoidable failures sitting between your actual ability and the outcomes you are getting. That is usually the gap, not the ability.

Is the keyword thing still true with AI screening everywhere?
It is more true, because the models are the second reader. In our test the model was the fair part: it ignored tool synonyms and buzzword padding and ranked candidates sensibly. The dumb keyword filter that runs before it is what rejected a strong engineer over OpenTofu.

I have done all eight. Now what?
Then your constraint is genuinely skills or scope, and the multi-month advice becomes the right advice. Depth in one area beats familiarity with ten, and the fastest depth is owning something in production that pages you.

Should I list every tool I have touched?
No. We measured a 30-item skills list and it did not help. Exact nouns from the posting, plus depth on the handful you can actually be interviewed on.

Kafka is genuinely good at a specific set of problems. This article walks through six of them, what each looks like in practice, and the part the architecture diagram leaves out: the failure mode you meet in month three. It ends with the case for not running Kafka at all, because that is the right answer more often than the conference talks suggest.

TLDR

• Kafka is a replicated, partitioned log, not a queue. Almost every surprise below follows from that one fact.
• Ordering is per partition, never global. If you need per-customer ordering, the customer id has to be the key.
• Log analysis works because Kafka absorbs backpressure when your search cluster falls over.
• CDC is the most valuable and most dangerous: a stalled connector pins your Postgres WAL and fills the primary's disk.
• Event sourcing on Kafka means no point lookups and no easy deletes, which collides with erasure requests.
• If you have one producer, one consumer and no replay requirement, you want a database table or SQS, not a cluster.

Prerequisites

• Comfortable with the idea of producers, consumers and topics
• Some exposure to a message queue, even just SQS or RabbitMQ
• Basic SQL, for the change data capture section

First, the thing that explains everything else

Kafka is a log. Not a queue, a log.

A queue hands a message to one consumer and forgets it. A log appends messages to an ordered file, keeps them for a configured time, and lets any number of consumers read at their own position. Nothing is removed when it is read. Consumers track an offset, and that offset is the only thing that says where they are.

Three consequences fall out of that, and they are behind most of what follows:

Replay is free. Reset the offset and read history again. This is why Kafka suits event sourcing and why it saves you when a downstream consumer had a bug for six hours.

Ordering is per partition. A topic is split into partitions for parallelism, and Kafka only guarantees order within one. There is no global ordering unless you run a single partition, which throws away the parallelism. Messages with the same key land on the same partition, so the key choice is your ordering guarantee.

Retention is a policy, not forever. By default Kafka drops data past a time or size threshold. Treating a topic as permanent storage requires either infinite retention, log compaction, or tiered storage, and each of those has costs.

topic: orders
partition 0:  [ o1 ][ o4 ][ o7 ]      <- ordered within the partition
partition 1:  [ o2 ][ o5 ][ o8 ]      <- ordered within the partition
partition 2:  [ o3 ][ o6 ][ o9 ]      <- ordered within the partition

Across partitions: no ordering at all.
Same key always lands on the same partition, so key by the entity
whose order you care about (customer id, account id, device id).

With that in hand, the six patterns.

1. Log analysis

Application, server and payment logs land in Kafka, and Elasticsearch and Kibana read from it. Straightforward enough that it is worth asking what Kafka is actually adding, because a log shipper can write to Elasticsearch directly.

The answer is backpressure. When Elasticsearch slows down or falls over, direct shippers have two options, and both are bad: buffer on local disk until the disk fills, or drop logs. With Kafka in between, the shippers keep writing at full speed and the backlog sits in one place you have sized deliberately. Elasticsearch comes back, the consumer works through the lag, nothing was lost.

The second thing it adds is fan-out. Once logs are in a topic, adding a second consumer that ships a subset to cold storage, or feeds a security tool, costs nothing at the producer side. Nobody has to reconfigure two hundred hosts.

The failure mode: teams size retention for the happy path. Seven days of logs at normal volume is fine, until an incident produces ten times the usual log volume at the exact moment the consumer is degraded. Size retention for your worst hour, not your average day, and alert on consumer lag rather than on broker disk, because lag tells you the problem hours earlier.

2. Real-time ML pipelines

User, product and app events stream through Kafka into a feature store and on to models that score in real time. The interesting arrow on that diagram is the feedback loop at the bottom: predictions become events themselves, which is what lets you measure a model against what actually happened.

The reason this pattern needs streaming rather than a nightly batch is feature freshness. A fraud model that scores a transaction using yesterday's aggregate of the account's behaviour is scoring a different account than the one in front of it. "Number of transactions in the last five minutes" is not a batch feature.

The failure mode: training and serving skew. The features you train on are computed by a batch job over historical data. The features you serve are computed by a stream job. Two implementations of "average order value over 30 days" written by two people in two languages will disagree, and the model will quietly underperform in production while looking fine in evaluation. Every serious writeup of this problem lands on the same fix: define the feature once and compute it one way for both paths, which is most of the argument for a feature store existing at all.

3. System monitoring and alerting

Services publish events, Kafka carries them, Flink analyses the stream, alerts come out the other end.

Before building this, be clear about what it is for, because it is not a replacement for Prometheus. Metrics systems are excellent at "CPU is above 90% on this host". This pattern is for alerting on business events in sequence: three failed payments from the same account inside a minute, a checkout funnel where the payment step stopped completing, a device that reported healthy then went silent for longer than its normal interval.

The distinction matters because those questions need windows and state. You are not thresholding a gauge, you are asking whether a pattern occurred across a stream of events in time order.

The failure mode: late data. Events do not arrive in the order they happened. A mobile client goes through a tunnel and delivers a batch of events ninety seconds after the fact. If your alert uses a one minute tumbling window on arrival time, those events land in the wrong window, and you get either a false alert or a missed one. This is what watermarks are for, and configuring them is a real decision rather than a default: too tight and you drop legitimate late events, too loose and every alert is delayed by the allowance.

event time:    10:00:05  10:00:20  10:00:45   (what actually happened)
arrival time:  10:00:06  10:02:10  10:00:46   (what your job sees)
                            ^
                    90s late, lands in the wrong window
                    unless the job groups by event time

Group by event time, not arrival time, and decide explicitly how long you are willing to wait for stragglers.

4. Change data capture

A connector like Debezium reads the database's transaction log and turns every insert, update and delete into an event on a topic. Sink connectors carry those to warehouses, search indexes and data lakes.

This is the pattern with the best return, because it solves the dual-write problem. Without CDC, keeping a search index in sync means your application writes to Postgres and then writes to Elasticsearch, and when the second write fails you have two systems disagreeing with no record of it. CDC removes the second write entirely: the database commit is the only write, and everything downstream derives from the log of commits. If a sink is down, it catches up.

Once change events are flowing, the next question is always how to query them, and hand-rolling a consumer that maintains a rolled-up view turns out to be much harder than it looks once you account for updates and deletes. This is the gap streaming databases fill: Materialize and similar systems consume these change streams and keep SQL views incrementally up to date, so you write a query rather than a consumer.

The failure mode, and it is a serious one: the Postgres replication slot. Debezium reads from a logical replication slot, and Postgres will not discard WAL segments that a slot has not yet confirmed. Stop the connector, or let it crash and not get restarted, and WAL accumulates on the primary. On a busy database that fills the disk in hours, and a full disk on the primary is a production outage caused by a pipeline nobody thought of as production.

If you run CDC against Postgres, these are not optional:

-- How far behind is each replication slot, in bytes of WAL it is pinning?
SELECT
  slot_name,
  active,
  pg_size_pretty(
    pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)
  ) AS retained_wal
FROM pg_replication_slots
ORDER BY pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) DESC;

Alert on retained_wal crossing a threshold and on active = false for any slot that should be running. Postgres 13 and later also support max_slot_wal_keep_size, which caps how much WAL a slot may pin and invalidates the slot instead of filling the disk. Losing a connector and having to resnapshot is a bad afternoon. Losing the primary is a bad quarter.

Two more things to plan for before you turn CDC on: the initial snapshot reads the entire table, which on a large table is hours of load you should schedule rather than discover, and schema changes propagate downstream, so an ALTER TABLE becomes a compatibility question for every consumer. That is what a schema registry is for.

5. Event-driven microservices

Order, payment and inventory services publish events. Shipping, notifications, analytics and billing consume them. Adding a consumer requires no change to any producer, which is the property everybody wants.

It is a real benefit. The synchronous version of this diagram is a service calling four others and being as available as the least available of them.

The failure mode: the decoupling is narrower than it looks. You have removed the runtime coupling and replaced it with a schema coupling plus eventual consistency, and the second one changes how the product behaves. After OrderCreated is published, there is a window where the order exists and shipping does not know. Usually milliseconds. Occasionally, when a consumer group is rebalancing or a consumer is lagging, considerably longer. Any UI that reads its own write immediately after will show a user something that looks broken.

Three things worth deciding up front rather than during an incident:

Key by the entity whose ordering matters. OrderUpdated and OrderCancelled for the same order must land on the same partition or they can be processed out of order. Key on order id.

Consumers must be idempotent. Kafka's exactly-once semantics apply to reads and writes within Kafka and to transactions across Kafka topics. The moment a consumer writes to Postgres or calls a payment API, delivery is effectively at-least-once, and that side effect will occasionally happen twice. Deduplicate on an event id, or make the operation naturally idempotent.

Carry a correlation id on every event. Debugging a synchronous call chain is a stack trace. Debugging a choreography of six services reacting to each other is reading six logs and guessing, unless every event carries the id that ties them together.

6. Event sourcing

Rather than storing current state, you store the sequence of events that produced it, and derive views from them. The audit trail is complete by construction, and you can rebuild any projection by replaying.

Kafka's log is a natural fit, and this is where replay stops being a nice property and becomes the point: found a bug in how you computed account balances, fix the projection code, replay from the beginning, and the new read model is correct.

The failure modes, because this pattern has several:

Kafka is not a database. There is no "get the current state of order 12345" without either replaying the topic, keeping a compacted topic keyed by id, or maintaining the projection in an actual database and querying that. Most event sourcing setups end up with Postgres holding the read models, and Kafka holding the events.

Replays are not free at scale. Rebuilding a projection from two years of events means reprocessing two years of events. Plan snapshots.

Deletion is genuinely hard. An immutable log is exactly the wrong shape for "delete everything about this person". Log compaction can remove superseded records by key, but an append-only history of what a user did is not something you can surgically edit. The usual answer is crypto-shredding: encrypt personal data per subject and destroy the key, so the events remain and the contents become unreadable. Decide this before you have production data, because retrofitting it means rewriting history you designed to be unrewritable.

When you do not need Kafka

Kafka's cost is not the licence, it is the operational surface: brokers, partitions, consumer group rebalances, schema evolution, connector supervision, and a set of failure modes your team has to learn. That cost is worth paying at a certain scale and for certain properties. Below it, you are paying for a cluster to do what a table would.

Reach for something simpler when all of these are true:

• One producer, one consumer, and no plans for a second
• No replay requirement, because reprocessing history is not a thing you need
• Throughput in the hundreds per second, not the hundreds of thousands
• No ordering requirement beyond what a single worker naturally provides

For those, a Postgres table with SELECT ... FOR UPDATE SKIP LOCKED is a perfectly good queue, runs on the database you already operate, and is debuggable with SQL you already know. SQS gives you the same with no server to run. RabbitMQ handles complex routing better than Kafka does.

Signals that you have genuinely outgrown that, and the cluster starts earning its keep:

• More than one team wants the same stream, and you are tired of adding webhooks
• You need to reprocess history after a bug, and cannot
• The dual-write problem is causing real inconsistency between systems
• A single consumer can no longer keep up, and you need partitioned parallelism
• Sustained throughput where a database-backed queue is spending its time on lock contention

Summary

The pattern across all six is that Kafka is worth it when you need the log properties: many independent readers, replay, and durability of an ordered history. When you only need to hand a job to a worker, it is a cluster you have to keep alive for no return.

FAQ

Is Kafka a message queue?
Not really, and the difference matters. A queue removes a message once it is consumed. Kafka appends to a log, keeps it for the retention period, and lets each consumer group track its own position. That is why replay works and why "the message was consumed" is not a thing Kafka tracks for you.

Does Kafka guarantee ordering?
Within a partition, yes. Across a topic, no. Messages with the same key go to the same partition, so choosing the key is choosing what you get ordering on. If your design assumes global ordering, it will work in staging with one partition and break the first time you scale out.

Is exactly-once delivery real?
Within Kafka, yes, using idempotent producers and transactions across topics. End to end into an external system, no. Once a consumer writes to a database or calls an API, you are in at-least-once territory and need idempotent consumers. Treat "exactly-once" as a Kafka-internal property, not a promise about your sinks.

Can I use Kafka as my database?
For an ordered history, yes. For querying current state, no. There is no index and no point lookup. Compacted topics give you the latest value per key, which is closer, but most systems keep the read models in a database and the events in Kafka.

How many partitions should a topic have?
Enough that your maximum consumer parallelism is not capped, since one partition can be read by only one consumer in a group, and few enough that you are not carrying overhead for nothing. Partitions are easy to add and impossible to remove, and adding them changes key-to-partition mapping, which breaks ordering for existing keys. Start with a number you can justify and leave headroom.

What about Redpanda, Pulsar or a managed service?
Every pattern here is about the log abstraction, not the implementation, so they all apply to Kafka-compatible systems. Managed services remove most of the operational cost that the last section warns about, which genuinely moves where the "is it worth it" line sits.

We measured it against DigitalOcean's Inference Engine. Same model, same prompt, one flag different:

• stream: false: 15,706 ms before a single character appears
• stream: true: 1,265 ms to the first token

Twelve times faster to something on screen, for a one-word change. Except the flag is the easy part. The part that quietly undoes it is the route handler in the middle, and there are three ways to write one that turns the second number back into the first.

This post builds the proxy that does not, measures what it costs, and documents two things about DigitalOcean's endpoint that will waste your afternoon if nobody tells you. The working app is on GitHub.

TLDR

• Streaming changes time to first token from 15.7s to 1.3s. It does not make generation faster: total time is roughly the same either way.
• A route handler that does await upstream.json() throws the entire benefit away. Pipe, do not await.
• Piping through a Next.js route handler costs about 120 ms. That is the real overhead, measured.
• SSE frames split across network reads. Parse naively and you silently drop whichever token straddles the boundary.
• /v1/models lists 76 models. Several return 403, not available for your subscription tier.
• Reasoning models have slow first tokens anyway. qwen3-32b took 7.9s to say anything, streaming or not.

Prerequisites

• Node 20+ and a Next.js 15 or 16 app using the App Router
• A DigitalOcean model access key, from GradientAI Platform → Model access keys
• Comfort with fetch, ReadableStream and async iteration

What streaming actually buys you

First, the measurement, because the reason to stream is not the reason people usually give.

Median of three runs against openai-gpt-oss-120b, one prompt, on 17 August 2026:

Note what did not change. Total generation time was about the same in both modes. Streaming does not make the model faster. It changes when the user finds out it is working, and that is the entire user-visible difference between a feature that feels broken and one that feels fast.

That distinction matters when someone asks you to "make the AI faster". Often they do not want more tokens per second, they want the blank screen to stop.

The route handler that quietly ruins it

The obvious implementation is the one that fails:

// app/api/chat/route.ts  DO NOT SHIP THIS
export async function POST(req: Request) {
  const { messages } = await req.json();

  const upstream = await fetch('https://inference.do-ai.run/v1/chat/completions', {
    method: 'POST',
    headers: { Authorization: `Bearer ${process.env.DO_INFERENCE_KEY}` },
    body: JSON.stringify({ model: 'openai-gpt-oss-120b', messages, stream: true }),
  });

  // Here is the bug. `stream: true` is set, and it makes no difference at all.
  const data = await upstream.text();
  return new Response(data);
}

stream: true is set. The upstream really does send tokens as they are produced. And await upstream.text() waits for every one of them before your handler returns anything. You have asked for a stream and then reassembled it into a blocking call.

This is easy to miss because it works. Tests pass, the response is correct, and the only symptom is that the app feels slow, which nobody logs.

The proxy that preserves it

The fix is to return a ReadableStream that forwards chunks as they arrive:

const decoder = new TextDecoder();
const encoder = new TextEncoder();
let buffer = '';

const body = new ReadableStream<Uint8Array>({
  async start(controller) {
    const reader = upstream.body!.getReader();
    try {
      for (;;) {
        const { done, value } = await reader.read();
        if (done) break;

        buffer += decoder.decode(value, { stream: true });
        const { text, rest, done: finished } = parseSSE(buffer);
        buffer = rest;

        if (text) controller.enqueue(encoder.encode(text));
        if (finished) break;
      }
    } finally {
      await reader.cancel().catch(() => {});
      controller.close();
    }
  },
  cancel() {
    // The browser went away: tab closed, navigated, or hit stop.
    upstream.body?.cancel().catch(() => {});
  },
});

return new Response(body, {
  headers: {
    'Content-Type': 'text/plain; charset=utf-8',
    'X-Accel-Buffering': 'no',
    'Cache-Control': 'no-cache, no-transform',
  },
});

Measured, this costs about 120 ms against calling DigitalOcean directly: 1,388 ms versus 1,265 ms to first token. That is the honest price of having a server in the middle, and it is worth paying, because the alternative is shipping your API key to the browser.

The bug you will not notice until it is in production

Chunks from the network do not align to line boundaries. One reader.read() can hand you this:

data: {"choices":[{"delta":{"content":"abc"}}]}
data: {"choices":[{"delta":{"con

That second frame is cut in half. Parse the buffer line by line and throw away what is left, and the token in the incomplete frame vanishes. The output is still fluent, still plausible, and missing a word every few hundred. Nothing errors.

The fix is to keep the remainder and prepend it to the next read:

export function parseSSE(buffer: string) {
  let text = '';
  let done = false;
  const lines = buffer.split('\n');
  // The last element may be a partial line. Hold it back for the next read.
  const rest = lines.pop() ?? '';

  for (const line of lines) {
    const trimmed = line.trim();
    if (!trimmed.startsWith('data:')) continue;
    const payload = trimmed.slice(5).trim();
    if (payload === '[DONE]') { done = true; continue; }
    try {
      const delta = JSON.parse(payload)?.choices?.[0]?.delta?.content;
      if (typeof delta === 'string') text += delta;
    } catch { /* incomplete frame */ }
  }
  return { text, rest, done };
}

lines.pop() is the entire fix, and it is worth a test, because this is the kind of bug that survives code review:

it('holds back a partial line instead of losing it', () => {
  const whole =
    'data: {"choices":[{"delta":{"content":"abc"}}]}\n' +
    'data: {"choices":[{"delta":{"con';

  const first = parseSSE(whole);
  expect(first.text).toBe('abc');

  // Feeding the remainder back recovers the token that was split.
  const second = parseSSE(first.rest + 'tent":"def"}}]}\n');
  expect(second.text).toBe('def');
});

Cancellation is a billing feature

When a user hits stop or closes the tab, the model keeps generating unless you tell it not to. You pay for those tokens and nobody reads them.

Next.js gives you req.signal, which fires when the client disconnects. Forward it:

export async function POST(req: Request) {
  const body = await req.json();
  // req.signal aborts when the browser goes away. Passing it upstream is what
  // actually stops the generation, and the bill.
  return streamChat(body, process.env.DO_INFERENCE_KEY ?? '', req.signal);
}

On the client, an AbortController gives you a working stop button:

const abort = useRef<AbortController | null>(null);

async function run() {
  abort.current = new AbortController();
  const res = await fetch('/api/chat', {
    method: 'POST',
    body: JSON.stringify({ messages }),
    signal: abort.current.signal,
  });
  // ...read the stream
}

<button onClick={() => abort.current?.abort()}>Stop</button>

Without the cancel() handler on the ReadableStream shown earlier, aborting the browser request leaves the upstream connection open and generating. The stop button looks like it works and changes nothing on your invoice.

Use the Node runtime, not edge

It is tempting to put a streaming route on the edge runtime. Do not, for long generations:

export const runtime = 'nodejs';
export const dynamic = 'force-dynamic';

Edge functions have shorter maximum durations, and a fifteen second generation that occasionally runs to forty will be cut off mid-sentence. force-dynamic matters too: a cached AI response is not a performance win, it is a bug where every user gets the first user's answer.

Two things about DigitalOcean's endpoint

The model list is not the list you can call. GET /v1/models returns 76 entries. Several of them, including the Claude family, answer with:

{ "error": { "message": "this model is not available for your subscription tier" } }

That is a 403 at request time, not a filtered list. If you are building a model picker from that endpoint, validate against your tier or your users will pick models that cannot run.

Reasoning models break the streaming promise. The headline number in this post is openai-gpt-oss-120b at 1.3s to first token. Running the identical test against alibaba-qwen3-32b:

Both were streaming. The reasoning model spends the first eight seconds thinking before it emits anything, so the user still gets a blank screen, just a shorter one. Streaming cannot help with silence at the source.

If time to first token is what you care about, the model choice matters more than the streaming flag. Test the model you intend to ship, not the one in the tutorial.

The whole thing, working

The repository has the complete app: the proxy, the route handler, a client that renders tokens as they arrive and displays its own measured time to first token, and the tests including the split-frame case.

git clone https://github.com/The-DevOps-Daily/do-inference-nextjs
cd do-inference-nextjs
cp .env.example .env.local   # add DO_INFERENCE_KEY
npm install && npm run dev

FAQ

Does streaming reduce total generation time?
No. In our runs total time was roughly the same with and without it. What changes is when the first token arrives, which is what users experience as speed.

Can I skip the route handler and call DigitalOcean from the browser?
Only if you are happy publishing your API key. The 120 ms the proxy costs is the price of keeping the credential server side, and it is a bargain.

Why plain text rather than SSE to the browser?
Because the browser side gets simpler: reader.read() and append. Use SSE to the client if you need to interleave metadata such as token counts or tool calls in the same channel.

Does this work with the Vercel AI SDK?
Yes, and the SDK handles the parsing and cancellation shown here. This post builds it by hand because the failure modes are much easier to recognise once you have seen what the SDK is doing for you.

Is this specific to DigitalOcean?
The endpoint is OpenAI-compatible, so the same handler works against any provider with that shape. The two gotchas at the end are DigitalOcean-specific; the streaming mechanics are not.

📌 Handpicked by DevOps Daily - Your weekly dose of curated DevOps news and updates!

⚓ Kubernetes

📄 Eleven minutes, zero humans: Building a self-healing Kubernetes upgrade pipeline on Kairos

Once upon a time, upgrading a Kubernetes control plane meant staying awake for it. SSH into every node. Run the upgrade by hand. Watch etcd health the whole time, hoping quorum holds through every reb

📅 Aug 14, 2026 • 📰 CNCF Blog

🔗 Read more

📄 Qodana Lints Your Code. What’s Checking Your DevOps and Platform Engineering Stack?

A developer in DevOps pushes a Kubernetes deployment with no resource limits, a pod running as root explicitly, and a GitHub Actions workflow runs with mutable tags – and it goes straight to productio

📅 Aug 13, 2026 • 📰 JetBrains Blog

🔗 Read more

📄 Forensic container checkpointing on Amazon Elastic Kubernetes Service (Amazon EKS)

Amazon EKS 1.34 makes the Kubelet Checkpoint API functional, so you can capture a running container's full state (memory, processes, and network connections) without stopping the workload. This post s

📅 Aug 12, 2026 • 📰 AWS Containers Blog

🔗 Read more

📄 Introducing advanced Kubernetes control plane configuration in Amazon EKS

With Amazon EKS, you can now configure Kubernetes control plane components (the API server, scheduler, and controller manager) directly through EKS APIs. This post explains what's configurable and inc

📅 Aug 12, 2026 • 📰 AWS Containers Blog

🔗 Read more

📄 How to Pretty-Print Your Kubernetes YAML as KYAML and Why You'd Want To

YAML has been the standard way to write Kubernetes manifests for years. Every example, tutorial, and configuration file you come across is written in it. The problem isn't that YAML is a bad format. I

📅 Aug 11, 2026 • 📰 Kubernetes Blog

🔗 Read more

☁️ Cloud Native

📄 Reproducible ESP32 Firmware Development with Docker and Docker Sandboxes

Build ESP32 firmware with reproducible Docker environments and use Docker Sandboxes for isolated AI-assisted development and hardware testing.

📅 Aug 14, 2026 • 📰 Docker Blog

🔗 Read more

📄 Lightweight Dragonfly Deployment: P2P Distribution Without the Database Stack

Dragonfly speeds up file and container image distribution using peer-to-peer (P2P) technology, but a standard installation deploys several components and dependencies. Beyond the Scheduler, Seed Clien

📅 Aug 13, 2026 • 📰 CNCF Blog

🔗 Read more

📄 Docker VMM Public Beta: A Complete Overhaul, Built for Performance

Docker VMM is now available in public beta for Mac and Windows. Learn what this means for performance, stability, and governance and how to try it yourself.

📅 Aug 12, 2026 • 📰 Docker Blog

🔗 Read more

📄 Good apps aren’t born, they’re guided: Building observable policy as code

As parents in tech, we’ve learned that neither children nor applications thrive without clear boundaries. There are no “good” or “bad” kids, just as there are no inherently “good” or “bad” application

📅 Aug 12, 2026 • 📰 CNCF Blog

🔗 Read more

📄 Measuring Sustainability via Project Kepler, with Niki Manoledaki

Niki Manoledaki is a Staff Platform Engineer at Grafana Labs, A CNCF Ambassador and Green Software Foundation Champion, and a core maintainer of Project Kepler. We explore the recent rewrite of Projec

📅 Aug 12, 2026 • 📰 Kubernetes Podcast

🔗 Read more

📄 SynchDB 1.4 Released - Oracle Container Database Support and TLS-Secured FDW Snapshots

Dear Community Members, We are excited to announce the release of SynchDB 1.4, a PostgreSQL extension for real-time replication from heterogeneous source databases into PostgreSQL/IvorySQL. This relea

📅 Aug 12, 2026 • 📰 PostgreSQL News

🔗 Read more

🔄 CI/CD

📄 How to bring your software delivery workflow into GitHub with agent apps

See how four GitHub agent apps can help you scope, secure, roll out, and ship a feature across the SDLC–all without leaving GitHub. The post How to bring your software delivery workflow into GitHub wi

📅 Aug 14, 2026 • 📰 GitHub Blog

🔗 Read more

📄 Stories from the Factory Floor: Our AI software factory saved me from an incident and I lived to tell the tale

Last summer, I shipped what I thought was a routine cleanup to production. It turned out to be a bug. But before the vast majority of users ever saw it, our AI software factory caught it and rolled ba

📅 Aug 14, 2026 • 📰 LaunchDarkly Blog

🔗 Read more

📄 How Harness AI Reaches Your Toolchain, Safely

One MCP Gateway lets AI Chat call GitHub, Jira, and Confluence with per-tool permissions, RBAC visibility, and no dropped sessions at scale. | Blog

📅 Aug 14, 2026 • 📰 Harness Blog

🔗 Read more

📄 Your guide to GitHub Universe 2026 is here: The schedule just launched!

The GitHub Universe session catalog is live. Explore interactive workshops, community talks, demos, and panels. Plus, register before August 19 to save $300. The post Your guide to GitHub Universe 202

📅 Aug 13, 2026 • 📰 GitHub Blog

🔗 Read more

📄 How I built a demo generator with GitLab Duo Agent Platform

A demo used to take me days to build — screenshots, narration, stitching it together in an external tool, chasing feedback — and every time the feature changed I'd have to start over. A few months ago

📅 Aug 13, 2026 • 📰 GitLab Blog

🔗 Read more

📄 GitHub availability report: July 2026

In July, we experienced eight incidents that resulted in degraded performance across GitHub services. The post GitHub availability report: July 2026 appeared first on The GitHub Blog.

📅 Aug 12, 2026 • 📰 GitHub Blog

🔗 Read more

📄 How GitLab tracks vulnerabilities through refactors and reformatting

Every day, security scans face the same problem: an agent or a developer adds a comment, reformats a file, or moves a function, and a naive vulnerability tracker suddenly reports the same finding twic

📅 Aug 12, 2026 • 📰 GitLab Blog

🔗 Read more

📄 GitLab Patch Release: 19.2.2, 19.1.4, 19.0.6

📅 Aug 12, 2026 • 📰 GitLab Blog

🔗 Read more

📄 Harness Community: Connect, Learn, and Build Together

Join the Harness Community to connect with practitioners, solve delivery challenges, share expertise, and shape the future of software delivery. | Blog

📅 Aug 12, 2026 • 📰 Harness Blog

🔗 Read more

📄 Software Delivery Platform: Key Features & How to Evaluate

A software delivery platform isn't just a CI/CD tool. Get the must-have feature checklist and the demo questions to use when evaluating vendors. | Blog

📅 Aug 11, 2026 • 📰 Harness Blog

🔗 Read more

📄 Scaling organizational knowledge in Kiro with Amazon Bedrock Knowledge Bases, LangChain, and MCP

“A pull request comes back with a single comment: “This doesn’t follow our circuit breaker pattern. Check the Architectural Decision Record .” You know the architecture decision record exists somewher

📅 Aug 10, 2026 • 📰 AWS DevOps Blog

🔗 Read more

🏗️ IaC

📄 Compatibility Testing Pulumi HCL

Pulumi HCL has at its core a simple promise: A program that works for tofu apply will also work for pulumi up. This must be true to allow Terraform modules to be shared between tofu config and Pulumi

📅 Aug 14, 2026 • 📰 Pulumi Blog

🔗 Read more

📄 Never Miss What Your Infrastructure Is Telling You

Plenty happens in a Pulumi organization while you’re looking somewhere else. Neo finishes a task you kicked off just before taking lunch. A teammate submits an ESC change request that needs your appro

📅 Aug 11, 2026 • 📰 Pulumi Blog

🔗 Read more

📊 Observability

📄 Streamline day-two SAP operations with Trento version 3

Key takeaways Automate compliance and observability: Trento version 3 delivers deep visibility into SAP environments by integrating Saptune and SUSE Multi-Linux Manager to track SAP notes and security

📅 Aug 14, 2026 • 📰 SUSE Blog

🔗 Read more

📄 What can you do with OpenTelemetry entity events?

Metrics, logs, and traces tell you how your systems behave. They are much quieter about what actually exists: which hosts, interfaces, switches, services, and volumes are out there right now, and, cru

📅 Aug 14, 2026 • 📰 OpenTelemetry Blog

🔗 Read more

📄 Scheduled support lifecycle announcement about Fluent Package v7

Hi users! We had launched fluent-package v6 series last year, recently shipped v6.0.4 in LTS release channel. In this blog article, we explain the planned next major updates - v7.0.0. When the next LT

📅 Aug 14, 2026 • 📰 Fluentd Blog

🔗 Read more

📄 Certificate Transparency Monitoring is now generally available

Cloudflare's Certificate Transparency Monitoring is now generally available. The biggest change: we no longer email you about certificates Cloudflare issued for your domain, so when an alert lands in

📅 Aug 13, 2026 • 📰 Cloudflare Blog

🔗 Read more

📄 Automated agent triage with Agent Tracing and Claude Routines

How Sentry uses a Claude Routine and the Sentry MCP to automatically triage 800 AI agent conversations overnight and file bugs.

📅 Aug 13, 2026 • 📰 Sentry Blog

🔗 Read more

📄 Windows Monitoring with Zabbix

Windows environments provide a variety of approaches for monitoring both on the OS and the application level. The article will cover utilizing Zabbix agent on Windows to collect and discover OS and ap

📅 Aug 12, 2026 • 📰 Zabbix Blog

🔗 Read more

📄 Announcing General Availability of New Relic Notebooks

Troubleshoot faster with New Relic Notebooks. Combine live queries, visualizations, and text in one unified, collaborative workspace to end tab fatigue.

📅 Aug 11, 2026 • 📰 New Relic Blog

🔗 Read more

📄 What's new in Sentry Logs: The summer 2026 roundup

Everything that shipped for Sentry Logs this summer: log pinning, JSONL exports, terabyte-scale search, and a dozen usability improvements.

📅 Aug 11, 2026 • 📰 Sentry Blog

🔗 Read more

📄 Empowering Relics to Own Their Career Growth

Learn how New Relic’s 5th Grow Your Career Month equips employees with continuous learning, leadership development, and AI skills to drive career growth.

📅 Aug 10, 2026 • 📰 New Relic Blog

🔗 Read more

🔐 Security

📄 Threats Making WAVs - Incident Response to a Cryptomining Attack

Guardicore security researchers describe and uncover a full analysis of a cryptomining attack, which hid a cryptominer inside WAV files. The report includes the full attack vectors, from detection, in

📅 Aug 17, 2026 • 📰 Linode Blog

🔗 Read more

📄 How Cloudflare detects MCP traffic and helps secure it

Cloudflare Gateway identifies MCP requests using protocol-level heuristics. Security teams can use that signal to find shadow MCP traffic, enforce Portal-only access for approved servers, and block di

📅 Aug 14, 2026 • 📰 Cloudflare Blog

🔗 Read more

📄 Gitea 1.27.2 is released

We are excited to announce the release of Gitea 1.27.2, the second patch release for the 1.27 series. It contains a large batch of security fixes alongside bug fixes for Gitea Actions, packages, L

📅 Aug 14, 2026 • 📰 Gitea Blog

🔗 Read more

📄 Friday Five — August 14, 2026

TechZine: Red Hat tames the open source AI chaosThe AI ecosystem is still in its infancy. This is evident from the regular releases of immature, yet highly imaginative, open source solutions. It’s up

📅 Aug 14, 2026 • 📰 Red Hat Blog

🔗 Read more

📄 What 50 open source projects taught us about security in the AI era

See how the open source projects in Session 4 of the GitHub Secure Open Source Fund combined AI-assisted workflows, maintainer expertise, GitHub security tools, expert guidance, and funding to improve

📅 Aug 13, 2026 • 📰 GitHub Blog

🔗 Read more

📄 PostgreSQL 18.6, 17.11, 16.15, 15.19, 14.24 and 19 Beta 3 Released!

The PostgreSQL Global Development Group has released an update to all supported versions of PostgreSQL, including 18.6, 17.11, 16.15, 15.19, and 14.24, as well as the third beta release of PostgreSQL

📅 Aug 13, 2026 • 📰 PostgreSQL News

🔗 Read more

📄 A new security baseline for enterprise agentic adoption

Agent Baseline is a blueprint for AI adoption that defines six security outcomes for putting enterprise agents to work without giving them unchecked authority. Consider this scenario: a customer-suppo

📅 Aug 12, 2026 • 📰 Docker Blog

🔗 Read more

📄 The Agent Baseline: 35 Controls, But Where Should You Start?

The Agent Baseline defines 35 controls across six security outcomes—but the right starting point depends on how your organization uses agents. Learn how to sequence controls for coding, internal, and

📅 Aug 12, 2026 • 📰 Snyk Blog

🔗 Read more

📄 A sandbox is only as closed as what an AI agent can reach

In July, OpenAI and Hugging Face responsibly disclosed an OpenAI model under internal evaluation escaped its sandbox, reached the open internet, and accessed Hugging Face’s internal production infrast

📅 Aug 12, 2026 • 📰 GitLab Blog

🔗 Read more

💾 Databases

📄 PLEASE_READ_ME: The Opportunistic Ransomware Devastating MySQL Servers

Guardicore Labs uncovers a Ransomware detection campaign targeting MySQL servers. Attackers use Double Extortion and publish data to pressure victims.

📅 Aug 17, 2026 • 📰 Linode Blog

🔗 Read more

📄 Amazon RDS for Oracle now supports Oracle Application Express (APEX) version 26.1

Amazon Relational Database Service (Amazon RDS) for Oracle now supports Oracle Application Express (APEX) version 26.1. Amazon RDS for Oracle is a managed database service that makes it simple to set

📅 Aug 14, 2026 • 📰 CloudFormation Updates

🔗 Read more

📄 AWS Billing and Cost Management introduces Managed Dashboards

AWS Billing and Cost Management (BCM) Dashboards now include Managed Dashboards. These are a collection of preconfigured and read-only dashboards located in your dashboard list. They deliver actionabl

📅 Aug 14, 2026 • 📰 CloudFormation Updates

🔗 Read more

📄 plx : Write PostgreSQL functions in the language you already know.

What plx is plx is a PostgreSQL extension that lets you write stored functions and triggers in the dialect you already know (the current set is listed below). When you run CREATE FUNCTION, plx transpi

📅 Aug 12, 2026 • 📰 PostgreSQL News

🔗 Read more

📄 Dasha - performance dashboard

Dasha is an open source performance dashboard for PostgreSQL fleets. It connects to your clusters with a read-only role, shows what the databases are doing right now, and explains what to do about it.

📅 Aug 12, 2026 • 📰 PostgreSQL News

🔗 Read more

📄 Fresh context: change data capture, not batch ETL

In many systems, the reason an agent quotes yesterday's data isn't the model. It's the pipeline behind it: a nightly ETL job that refreshed the agent's context hours ago. Change data capture (CDC) can

📅 Aug 12, 2026 • 📰 Redis Blog

🔗 Read more

📄 Agent memory as a moat: how context compounds

Base LLM inference is stateless. The model doesn't remember your last conversation, your users' preferences, or the mistake your agent made ten minutes ago. Unless the app supplies persisted context,

📅 Aug 12, 2026 • 📰 Redis Blog

🔗 Read more

📄 How Medium Powers Real-Time Recommendations at 1M OPS

Inside Medium’s move from relational features to list features in its ScyllaDB-based feature store “Keep readers reading” is the not-so-simple goal of Medium’s recommendations system. To predict what’

📅 Aug 11, 2026 • 📰 ScyllaDB Blog

🔗 Read more

📄 Accelerate PostgreSQL migrations using Gemini in Database Migration Service

Imagine this scenario: Your team decides to migrate a core application from an existing commercial database like Oracle or SQL Server to open source PostgreSQL or a fully managed service such as Alloy

📅 Aug 11, 2026 • 📰 Google Cloud Blog

🔗 Read more

📄 ScyllaDB Customer Experience Spotlight: Susie Solis

Meet Susie Solis, a Technical Support Engineer on the Customer Experience team here at ScyllaDB.

📅 Aug 10, 2026 • 📰 ScyllaDB Blog

🔗 Read more

📄 Why Agentic AI Architecture Needs a Database, Not Just a Vector Store

Agentic AI architecture is the system design that lets an AI agent perceive context, reason over it, call tools, maintain memory, and take actions across multiple steps. It spans the model, the orches

📅 Aug 10, 2026 • 📰 TiDB Blog

🔗 Read more

🌐 Platforms

📄 The Oracle of Delphi Will Steal Your Credentials

Our deception technology is able to reroute attackers into honeypots, where they believe that they found their real target. The attacks brute forced passwords for RDP credentials to connect to the vic

📅 Aug 17, 2026 • 📰 Linode Blog

🔗 Read more

📄 The Nansh0u Campaign – Hackers Arsenal Grows Stronger

In the beginning of April, three attacks detected in the Guardicore Global Sensor Network (GGSN) caught our attention. All three had source IP addresses originating in South-Africa and hosted by Volum

📅 Aug 17, 2026 • 📰 Linode Blog

🔗 Read more

📄 Sovereign Workload Placement: How Regulated Enterprises Decide Where Things Run

For more than a decade, cloud-first was the default. If a workload could run in the public cloud, it went there, and the architecture question was mostly about cost and speed. That default is being re

📅 Aug 15, 2026 • 📰 SUSE Blog

🔗 Read more

📄 What’s new with Google Cloud

Want to know the latest from Google Cloud? Find it here in one handy location. Check back regularly for our newest updates, announcements, resources, events, learning opportunities, and more. Tip: Not

📅 Aug 14, 2026 • 📰 Google Cloud Blog

🔗 Read more

📄 Secure all your internal vibe-coded applications — in one click

Introducing Cloudflare Access for Workers. Attach an Access policy directly to a Worker and it applies everywhere that Worker runs — routes, custom domains, workers.dev, and previews — automatically.

📅 Aug 14, 2026 • 📰 Cloudflare Blog

🔗 Read more

📄 Amazon SES click tracking now supports custom URL paths for mobile app deep linking

Amazon Simple Email Service (SES) now makes it easier to support mobile deep linking with the new ses:custom-path HTML attribute. When you add this attribute to an tag, SES carries your path segment t

📅 Aug 14, 2026 • 📰 CloudFormation Updates

🔗 Read more

📄 What Is Disaster Recovery as a Service (DRaaS) & What Are Your Alternatives for Disaster Recovery and Business Continuity?

Critical services rarely fail at a convenient moment. Hardware breaks, software misbehaves and human error slips through, often when demand is highest. Planning for these events is a key part of respo

📅 Aug 14, 2026 • 📰 SUSE Blog

🔗 Read more

📄 ODC-Noord: Building blocks for an existing government cloud

How did a small team in the east of the Netherlands (Groningen) from the Government Datacenter North (ODC-Noord) grow into a supplier of crucial building blocks for the Netherlands digital government

📅 Aug 14, 2026 • 📰 OpenShift Blog

🔗 Read more

📄 Breaking free from lock-in: How a leading insurance provider migrated 1,500 workloads to ROSA in 10 months

Imagine finding out your core platform contract is ending, leaving you with a multi million-dollar liability—and just 10 months to move 1,500 critical workloads. That was the reality for the engineeri

📅 Aug 14, 2026 • 📰 OpenShift Blog

🔗 Read more

📄 AWS Client VPN now supports CLI, administration controls, and faster connections

AWS Client VPN introduces a rebuilt AWS VPN Client v6.0.x which offers new features like command-line interface (CLI) support, enterprise administrative controls, and faster connection establishment t

📅 Aug 13, 2026 • 📰 CloudFormation Updates

🔗 Read more

📄 Total eclipse of the Internet: traffic impacts in Iceland, Spain, and Portugal

Cloudflare's data shows a clear impact on Internet traffic from Iceland to Spain and Portugal, following the path of totality of the total solar eclipse that occurred on August 12, 2026.

📅 Aug 13, 2026 • 📰 Cloudflare Blog

🔗 Read more

📄 Using BigQuery Graphs with measures for trusted agentic workloads

When enterprises transition from using simple chat assistants to autonomous, agentic workloads, they quickly run into a hard truth: Agents are prone to inaccurate insights when working with directly r

📅 Aug 13, 2026 • 📰 Google Cloud Blog

🔗 Read more

📰 Misc

📄 Visual Studio Code 1.134 (Insiders)

Learn what's new in Visual Studio Code 1.134 (Insiders) Read the full article

📅 Aug 18, 2026 • 📰 VS Code Blog

🔗 Read more

📄 GitHub Copilot’s Latest Update Bets on Model Choice, Not Model Loyalty

GitHub’s latest Copilot updates add Kimi K3, MAI-Code-1.1-Flash, Agent Plugins 1.0, model switching, CLI improvements, and local Ollama support.

📅 Aug 17, 2026 • 📰 DevOps.com

🔗 Read more

📄 Per-developer environments were the goal. Agents moved the goalposts.

Multi-tenancy has moved in one direction for 60 years: the tenant keeps getting smaller. Mainframe time-sharing carved a single machine The post Per-developer environments were the goal. Agents moved

📅 Aug 15, 2026 • 📰 The New Stack

🔗 Read more

📄 Grok 4.6 matched Fable 5 Max at an 85% discount. Downloadable models set that price.

I’m Matt Burns, Chief Content Officer at Insight Media Group. Each week, I round up the most important AI developments, The post Grok 4.6 matched Fable 5 Max at an 85% discount. Downloadable models se

📅 Aug 15, 2026 • 📰 The New Stack

🔗 Read more

📄 Treat Business Workflow Changes Like Deployments

Business automation often reaches production without the release discipline applied to application code. A routing rule changes, an approval threshold moves, or an integration starts writing to a new

📅 Aug 14, 2026 • 📰 DevOps.com

🔗 Read more

📄 Apple’s new AI split means your iOS app could behave differently in China

Apple is splitting up its AI stack. Instead of rolling out the same system worldwide, the company reportedly built a The post Apple’s new AI split means your iOS app could behave differently in China

📅 Aug 14, 2026 • 📰 The New Stack

🔗 Read more

📄 Alibaba’s new model promises Opus 4.6-level performance on your laptop

Alibaba recently made the open weights of its 2.4 trillion parameter Qwen3.8 model available. That’s a massive model, and its The post Alibaba’s new model promises Opus 4.6-level performance on your l

📅 Aug 14, 2026 • 📰 The New Stack

🔗 Read more

📄 Microsoft Decouples AI Agents From the VS Code Editor in Latest Release

Microsoft has shipped Visual Studio Code 1.133, and the headline change is architectural rather than cosmetic: AI agent sessions now run in a dedicated background process rather than within the editor

📅 Aug 14, 2026 • 📰 DevOps.com

🔗 Read more

📄 Exploring Compose HTML for Server Side Rendering

Something is happening in server-rendered web development. React shipped Server Components. HTMX made “hypermedia” cool again. Phoenix LiveView proved a server can push interactive UI updates without

📅 Aug 14, 2026 • 📰 JetBrains Blog

🔗 Read more

📄 Developer Resistance to AI Isn’t Fear – It is Identity

Developer resistance to AI is less about job loss than a deeper shift from hands-on coding to supervising, validating and orchestrating AI-generated work.

📅 Aug 14, 2026 • 📰 DevOps.com

🔗 Read more

📄 How student athletes are changing the game

The program’s participants, pictured on their first day at Red HatOn June 1, the first cohort of student athletes arrived at the Raleigh office to take part in the Red Hat Sales Combine Accelerator Pr

📅 Aug 14, 2026 • 📰 Red Hat Blog

🔗 Read more

📄 Stop managing SAP infrastructure by hand. Automate it.

How SUSE helps organizations deploy SAP environments faster, more consistently and with less operational risk. Key Takeaways: Manual SAP deployments create configuration drift, slow down migrations an

📅 Aug 13, 2026 • 📰 SUSE Blog

🔗 Read more

So you do the obvious things. You rebuild against the newest tag. The number does not move at all. You switch to -slim. Sometimes the number collapses, sometimes it changes by nothing. You start to suspect the number is not measuring what the ticket assumes it measures.

It is not. This article takes 17 common base images, counts every advisory that applies to the exact package versions inside each one, and shows where the number comes from. The short version: it is an inventory count, one package produces three quarters of it, the language runtime you actually execute is not represented in it at all, and on a fully patched image every remaining finding is one you cannot fix.

TLDR

• The count tracks how many packages the image records, not risk. node:22 records 413 packages and 1,684 advisories. node:22-slim records 88 and 80.
• 73% of node:22's advisories come from linux-libc-dev, a package of C header files. Your container runs the host's kernel, so a finding there is not evidence that anything in your image is vulnerable.
• node:22-slim records the identical 88 packages as debian:bookworm. Node.js is installed from a tarball, so not one of those findings is about the runtime you actually execute.
• debian:bookworm and debian:bookworm-slim record the same 88 packages and the same 80 advisories. Slim removes docs, man pages and locales, not packages.
• On a fully patched Debian 12 image, all 80 have no fix available. The "fixable" number a scanner shows you is a measure of how far behind you are, not of your risk.
• Debian's own triage marks 27 of those 80 unimportant, including one the NVD scores 9.8 Critical and marks Disputed.

Prerequisites

• Familiarity with Dockerfiles and base image tags
• A rough idea of what a CVE and a CVSS score are
• curl, tar, jq and Node.js if you want to reproduce the measurements
• No Docker daemon required

How I measured this, and what the method does not cover

There is no Docker daemon involved. A registry serves the manifest and each layer as an addressable blob, so you can stream a layer through tar, keep only the package database, and discard the rest. Layer blobs still get downloaded, they just never become a local image.

The package database is what a scanner reads to build its inventory:

• Debian and Ubuntu keep it at /var/lib/dpkg/status
• Alpine and Wolfi keep it at /lib/apk/db/installed
• Distroless splits it into one file per package under /var/lib/dpkg/status.d/

Every package was then queried against OSV using the distro's own feed: Debian:12, Debian:13, Ubuntu:24.04:LTS, Alpine:v3.24, Wolfi. Distro advisories are keyed by source package, so binaries were collapsed onto their source first. Counting binary packages would inflate every total.

Three things about this method are worth stating plainly, because two of them made me throw away a set of numbers.

This inventories OS package records, and nothing else. It is not a full image scan. Anything installed outside the package manager is invisible to it, and that turns out to matter a great deal, as the second finding below shows.

Layers must be replayed in order. My first attempt walked layers from the top and stopped at the first package database it found. That is right for dpkg/status, which whichever layer last ran apt rewrites wholesale. It is wrong for distroless, which spreads status.d/ across 19 layers, one file per package, so stopping at the top layer reported distroless as having exactly 1 package. Replaying every layer in order fixes it. Note that a faithful replay would also need to honour OCI whiteout markers for deleted files; none of these images delete package database entries, but a general-purpose tool must handle it.

Follow the pagination. /v1/querybatch returns at most 1000 vulns per query and hands back a next_page_token. linux-libc-dev alone exceeds that, so my first run reported node:22 at 1,457. Paginating to exhaustion gave the real figure of 1,684. The truncation is documented, but a client that ignores the token undercounts by thousands and looks perfectly healthy doing it.

The numbers

Measured 14 August 2026, linux/amd64.

Within this sample, ordering by advisory count is nearly the same as ordering by package count. That is not a law of nature and the sample mixes feeds that are not comparable, so treat it as what it is: in these images, the total mostly reflects how much the image records, and one source package dominates the largest entries.

Finding 1: one package produces three quarters of the number

Breaking node:22's 1,684 advisories down by source package puts one entry far out in front:

The linux source package produces exactly one binary here: linux-libc-dev. Debian describes it as "Linux support headers for userspace development", and its file list is headers under /usr/include plus package metadata. No kernel, no modules, nothing that executes.

Your container does not run its own kernel, it runs the host's. So a kernel CVE attached to the headers in your image is not evidence that your image is vulnerable, and it is not evidence that your host is either. It is an artefact of mapping "this package was built from a kernel source tree" onto "this image is affected".

That accounts for 1,227 of 1,684 advisories, 73% of the total. Excluding it leaves 457.

Be careful about how far you take this. A vulnerable host kernel absolutely can be attacked from inside a container; the headers neither cause nor prevent that, and removing them from the report does not make the host safe. The correct conclusion is narrow: these findings are attributed to the wrong artefact, and the question they raise ("is the host kernel patched?") is not one the image scan can answer.

This is a long-running complaint against every scanner built on distro feeds. The Trivy issue asking for it was closed as not planned, with similar reports across Trivy and GitLab container scanning.

Finding 2: the runtime you actually run is not in the count

Here is the result that changed how I read every one of these reports. I diffed the package name sets of node:22-slim and debian:bookworm:

node:22-slim      88 package records
debian:bookworm   88 package records
identical sets:   true
dpkg entries matching node/npm/yarn:  none

node:22-slim records exactly the same 88 packages as plain debian:bookworm. The official Node images install Node from an upstream tarball into /usr/local, outside dpkg entirely. So when a scanner reports 80 findings against node:22-slim, not one of them concerns Node.js, npm, or anything else you actually execute. It is a report about Debian, delivered while a Node runtime sits next to it, unexamined.

The same holds for python:3.13, which builds CPython under /usr/local, and for distroless/nodejs22-debian12, whose 10 dpkg records are base-files, libc6, libssl3, tzdata and friends, with the Node binary copied in.

Contrast Chainguard, which packages the runtime through apk:

chainguard/wolfi-base    15 packages
chainguard/node          27 packages
  node-related apk packages: nodejs-26, node-gyp, npm-12

This has a direct consequence for every "our image has fewer CVEs" comparison you will ever be shown, including the table earlier in this article. Wolfi's feed covers the Node runtime because Wolfi packages it. Debian's feed does not, because Debian is not shipping it. Those two numbers are not measuring the same surface, and the Debian-based one is flattered by an omission.

If you want an inventory that includes the runtime and your application dependencies, you need an SBOM built by a tool that catalogs language ecosystems, not just the OS package database.

Finding 3: "slim" means two completely different things

debian:bookworm         88 packages   80 advisories   48.5 MB
debian:bookworm-slim    88 packages   80 advisories   28.2 MB

node:22                413 packages 1684 advisories  408.4 MB
node:22-slim            88 packages   80 advisories   79.9 MB

For the first pair the package sets are identical, which the official rootfs manifests confirm. Debian's slim variant removes files, not packages: documentation, man pages, info files, locales and lintian data, per the slimify exclusion list. It saves 20 MB and zero advisories. Anyone who moved from debian:bookworm to debian:bookworm-slim to fix a scan result changed nothing at all.

The second pair is a different operation. node:22 is built on buildpack-deps, which installs a compiler toolchain, git, subversion, mercurial, image libraries and libpq-dev so native modules can build. node:22-slim skips all of it, and the 325 packages it drops carry the advisories.

So "use the slim tag" is good advice for a reason most people state wrongly. It helps when the slim variant omits packages. On the Debian base images it is purely a size optimisation. This is also specific to Debian and to this snapshot, not a general property of the word "slim" across distributions.

Finding 4: on a patched image, nothing is fixable

Splitting each image's findings by whether a fixed version exists for the release that image is actually on:

Getting this right took two attempts and the first one was wrong in a way worth describing, because the same mistake is easy to make in your own tooling. An OSV record carries one affected entry per distro release. My first pass asked "does any entry anywhere in this record have a fixed event", which answers a different question: Debian 13 having a patch says nothing about your Debian 12 image. Of the 2,046 records here, 1,615 have mixed fix status across their entries, so the loose version massively overstated how much was fixable. The count has to be scoped to the matching ecosystem and package.

Once scoped, the pattern is stark and it makes sense on reflection. Querying by installed version only returns advisories that version does not already satisfy. A fully up-to-date debian:bookworm therefore shows 80 findings of which exactly zero have a fix, because anything with an available fix was already installed. What is left is the residue Debian has recorded and chosen not to patch in this release.

The images with fixable findings are the ones running behind. distroless/nodejs22-debian12 carries glibc 2.36-9+deb12u13 while debian:bookworm is on u14, and that single point release accounts for its 21 fixable findings:

glibc 2.36-9+deb12u13   19 advisories   6 with "fixed": "2.36-9+deb12u14"
glibc 2.36-9+deb12u14   13 advisories   0 with a fix

This reframes what the scanner's "fixable" column actually is. It measures your patch lag. Drive it to zero and it stays at zero until the next advisory lands, which is exactly what you want from it. The other column, the permanently unfixed remainder, never moves no matter what you do, and it is the one the remediation ticket usually quotes.

Finding 5: a 9.8 that Debian calls unimportant

Debian's security tracker records a triage verdict alongside each advisory, and OSV carries it through as ecosystem_specific.urgency. Of debian:bookworm's 80 advisories, 27 are marked unimportant.

CVE-2019-1010022 in glibc is the clearest case. The NVD record carries the vector CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H, which computes to a base score of 9.8, Critical. That is the number your dashboard sorts on and your policy gate blocks on. The NVD also marks the record Disputed, and its description ends by quoting the glibc maintainers:

NOTE: Upstream comments indicate "this is being treated as a non-security bug and no real threat.

Debian's tracker entry still lists it as unfixed in bookworm, and the machine-readable triage on the same advisory reads:

{ "urgency": "unimportant" }

So a Critical-scored, unfixed finding sits in glibc, in essentially every glibc-based image, and the people who maintain the code say it is not a security bug. It has been there since 2019. Three of the four oldest glibc advisories here are of this type, and one of them, CVE-2010-4756, dates from 2010.

None of that makes CVSS useless. It makes a base score computed from a vector, with no knowledge of whether the code path is reachable in your image, a poor priority ranking. The distro maintainers published their assessment in a field almost nobody reads, and it disagrees with the number everyone acts on.

Finding 6: zero does not mean clean

Alpine and the Chainguard images all report 0 here. Two different things produce that, and only one of them is about security.

The real part: these images record far fewer packages. chainguard/node records 27 against node:22's 413. alpine:3.21 records 15. Fewer packages means less to patch, less to inventory, and less to argue about in a review. That advantage is structural.

The artifact part is the feed. I checked how many records in each OSV feed describe a vulnerability with no fixed version:

Debian's feed carries 160 glibc records where Wolfi's carries 35, and Debian is the only one of the four with a meaningful count of permanently unfixed entries. Alpine's OSV input is converted from its fix-oriented SecDB, which under-represents issues that have no fix yet; Alpine's own security tracker lists potentially-vulnerable issues that SecDB does not. Chainguard's own advisory system does publish unfixed states such as "under investigation" and "fix not planned", so the zero here reflects the OSV export and these specific installed versions rather than a policy of silence.

The honest reading is narrow: a large part of the gap between "80" and "0" is a difference in what each feed writes down, so cross-distro CVE totals compare disclosure practice as much as security. Comparing within one feed, as the node:22 to node:22-slim to distroless chart does, is fair and shows a real effect.

What actually moves the number

Separate the build image from the runtime image. The biggest lever, and free. The toolchain that makes node:22 a 413-package image is needed at build time and never at run time.

# Build stage: the fat image, with every toolchain you need
FROM node:22 AS build
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build && npm prune --omit=dev

# Runtime stage: only what serves traffic
FROM node:22-slim
WORKDIR /app
ENV NODE_ENV=production
# package.json matters at runtime: Node reads its "type" field to decide
# whether .js is ESM or CommonJS, so omitting it breaks ESM builds.
COPY --from=build --chown=node:node /app/package.json ./
COPY --from=build --chown=node:node /app/node_modules ./node_modules
COPY --from=build --chown=node:node /app/dist ./dist
USER node
CMD ["node", "dist/server.js"]

Two things that bite here. Use a .dockerignore containing node_modules, or COPY . . will overwrite the clean Linux tree that npm ci just built with whatever your laptop has. And native addons compiled against libraries present in buildpack-deps can fail at runtime in -slim if the shared library is not there, so test the runtime image rather than assuming it starts.

That change takes the base from 1,684 advisories to 80 and from 408 MB to 80 MB. Your application's own dependencies then add both size and findings on top; the base image is a floor, not the final figure.

Go further down if the runtime allows it. distroless/nodejs22-debian12 runs Node on 10 package records. Know the tradeoff first: there is no shell, so kubectl exec -it ... -- sh gets you nothing and debugging moves to ephemeral debug containers. You can still exec binaries that are present.

Pin by digest and rebuild deliberately. A weekly rebuild only picks up fixes if the base actually gets re-resolved. Tags are mutable and layer caching will happily reuse a stale base, so rebuild with --pull, or pin FROM image@sha256:... and update the digest on a schedule with something like Renovate. Pinning without a bump process is how images end up two point releases behind, which is precisely what happened to distroless/nodejs22 above.

Gate on something an engineer can satisfy. "No criticals" fails on a bug glibc's maintainers call a non-issue and cannot be satisfied by any action, so teams add blanket exceptions, and the exceptions are what let a real finding through six months later. A workable policy blocks on findings with an available fix older than N days, blocks on anything in KEV regardless of fixability, and routes the unfixed remainder to a review queue rather than the build log. EPSS can help order that queue, as long as you remember it estimates exploitation activity and says nothing about whether the code is reachable in your image.

Where this leaves the scanner

None of this says stop scanning. Scanners are how you learn that your image still carries the curl from before the last advisory, and that alone justifies running them.

What the measurements say is that the headline total is close to meaningless as a risk signal, and managing it as a target produces work with no security value. Three of the six findings here are cases where the number moved a lot without the image getting safer, or refused to move regardless of what anyone did. One is a case where the number said nothing at all about the software actually being executed.

The useful number is much smaller than the one on the dashboard: findings in packages you actually execute, with a fix available or a known exploit, in code paths your application reaches. Everything else is a report about Debian's bookkeeping, and it deserves a review queue rather than a release gate.

Reproduce it yourself

With Docker and a scanner, the quick version:

# how many package records, which is most of the answer
docker run --rm node:22 sh -c 'dpkg -l | grep -c "^ii"'
docker run --rm node:22-slim sh -c 'dpkg -l | grep -c "^ii"'

# how much of the count is kernel headers
trivy image --scanners vuln node:22 --format json \
  | jq '[.Results[].Vulnerabilities[]? | select(.PkgName=="linux-libc-dev")] | length'

The registry-only method used here streams layer blobs and keeps just the package database:

REG=registry-1.docker.io
REPO=library/node
TAG=22-slim
DEST=$(mktemp -d)

TOKEN=$(curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:$REPO:pull" \
  | jq -r .token)

# resolve the amd64 manifest out of the multi-arch index, and keep the digest
DIGEST=$(curl -s -H "Authorization: Bearer $TOKEN" \
  -H 'Accept: application/vnd.oci.image.index.v1+json' \
  "https://$REG/v2/$REPO/manifests/$TAG" \
  | jq -r '.manifests[] | select(.platform.architecture=="amd64" and .platform.os=="linux") | .digest')
echo "measuring $REPO@$DIGEST"

# replay layers in order into a fresh directory, keeping only the package db
for L in $(curl -s -H "Authorization: Bearer $TOKEN" \
    -H 'Accept: application/vnd.oci.image.manifest.v1+json' \
    "https://$REG/v2/$REPO/manifests/$DIGEST" | jq -r '.layers[].digest'); do
  curl -sL -H "Authorization: Bearer $TOKEN" "https://$REG/v2/$REPO/blobs/$L" \
    | tar -xz -C "$DEST" --wildcards \
        '*var/lib/dpkg/status' '*var/lib/dpkg/status.d*' '*lib/apk/db/installed' 2>/dev/null
done

grep -c '^Package: ' "$DEST/var/lib/dpkg/status"

Then query one package, scoping fix status to the release you are actually on:

curl -s -X POST https://api.osv.dev/v1/query \
  -d '{"package":{"name":"glibc","ecosystem":"Debian:12"},"version":"2.36-9+deb12u14"}' \
  | jq '{
      total: (.vulns | length),
      no_fix: [ .vulns[]
        | select([ .affected[]
            | select(.package.ecosystem=="Debian:12" and .package.name=="glibc")
            | .ranges[]?.events[]? | select(.fixed) ] | length == 0) ] | length
    }'

Note the nested select on ecosystem and package name. Without it you are asking whether the bug is fixed in some other Debian release, which is the mistake described in Finding 4.

FAQ

Does this mean base image CVEs never matter?
No. It means the total is the wrong thing to manage. A fixable critical in a library your code calls on every request matters a great deal, and it is sitting in the same report as 1,227 kernel header findings that are attributed to the wrong artefact. The work is separating them, which is what reachability analysis, KEV and VEX exist to do.

Why does my scanner report a different total?
Different inventory catalogers, different advisory sources, different handling of aliases and source-to-binary mapping. Note that severity filtering is usually not the cause: Trivy reports all severities by default and only drops unfixed findings when you pass --ignore-unfixed, and Grype's only-fixed defaults to false. Expect the same shape and different digits.

Is Alpine more secure than Debian?
This data cannot answer that, and neither can a comparison of their CVE counts, for the reasons in Finding 6. Alpine images are smaller and carry fewer packages, which is a genuine advantage. musl and busybox also behave differently from glibc and coreutils in ways that occasionally break applications. Choose on package count, support lifetime, patch latency and runtime compatibility, not on a scanner total.

What about apt-get upgrade in my Dockerfile?
On a current base image it has nothing to do, since all 80 findings already lack a fix. It also makes builds non-reproducible, because the same Dockerfile produces different images on different days. Prefer pinning a digest and bumping it deliberately.

Is distroless always the right answer?
No. You lose the shell, which changes how you debug production, and the base is still Debian, so distroless/base-debian12 still reports 15 advisories with no fix for any of them. It is a large improvement, not a zero. It also needs the same digest-bump discipline as anything else, as the two-point-release lag in distroless/nodejs22 shows.

RFC 10008, published in June 2026, addresses that with a new method called QUERY. It is the first genuinely new HTTP method since PATCH arrived in RFC 5789 in March 2010.

The summary going around is "a GET with a body", which is close enough to be useful and wrong in the way that matters. QUERY is a new method whose response is cacheable using a cache key that includes the request body, and that single requirement is why this is an infrastructure story rather than an API design story.

The spec is done. The body-keyed caching is not on by default in the places you deploy. And the RFC anticipated that, which is the part almost nobody is talking about.

TL;DR

• QUERY is safe, idempotent and cacheable, and it carries a request body. Standards track, not a draft.
• The cache key MUST incorporate the request content and related metadata. Not just the bytes.
• Browsers send it today but do not cache it. Managed CDNs largely do not accept it yet: CloudFront, for one, allows a fixed list of seven methods and QUERY is not among them.
• The RFC ships an escape hatch: answer with Location or Content-Location and clients repeat the query with a plain GET, which every cache you own already understands.
• Cross-origin QUERY needs a preflight, but so does the JSON POST you are replacing, and preflights are cached. This costs less than people are claiming.
• Servers MUST fail a QUERY with a missing or inconsistent Content-Type. There is also an Accept-Query response header for advertising support.
• In browsers, method: 'query' goes on the wire lowercase and fails. Node's fetch normalises it. Same code, different behaviour.

Prerequisites

• Familiarity with HTTP methods and status codes
• Some exposure to caching headers, or a CDN configuration screen
• Nothing to install to follow along

What QUERY actually says

The normative text is short and worth reading directly:

A QUERY requests that the request target process the enclosed content in a safe and idempotent manner and then respond with the result of that processing.

Safe. "The client does not request or expect any change to the state of the target resource." This is what lets a prefetcher or proxy issue the request without being reckless.

Idempotent. "QUERY requests are idempotent; they can be retried or repeated when needed, for instance, after a connection failure."

Cacheable. "The response to a QUERY method is cacheable; a cache MAY use it to satisfy subsequent QUERY requests."

Two requirements that are easy to miss and will fail your integration tests:

Servers MUST fail the request if the Content-Type request field is missing or is inconsistent with the request content.

That is a MUST, not a nicety. And for discovery, the RFC defines a response header:

The "Accept-Query" response header field can be used by a resource to directly signal support for the QUERY method while identifying the specific query format media types that may be used.

So a resource can advertise both that it speaks QUERY and which body formats it accepts. If you are adding QUERY to an API, send Accept-Query.

The requirement that makes this an ops problem

The cache key for a QUERY request MUST incorporate the request content and related metadata.

RFC 9111 defines a cache's primary key as the request method plus the target URI. In practice most caches you meet are GET-shaped: the URL is the key, with a Vary on a few headers. GET /search?q=nginx is one entry because the URL is one string.

QUERY does not fit that. Two requests to the same path with different bodies are different queries and need different entries. A cache supporting QUERY has to read the request content before it can decide whether it already holds the answer.

Note "and related metadata". Identical bytes under a different Content-Type or content coding can mean a different query, so the bytes alone are not a sufficient key.

This pattern is not unprecedented. Varnish has supported hashing request bodies into the cache key for POST for years, with an explicit size cap before it gives up. So the honest claim is not that nobody can do this. It is that no browser and few managed CDNs do it by default today, and the ones that adopt it will need a bounded buffering policy, because the bodies QUERY exists to carry are large by definition.

The correctness trap hiding inside it

The RFC flags a failure mode worth taking seriously:

Caches that normalize QUERY content incorrectly or in ways that are significantly different from how the resource processes the content can return an incorrect response.

Caches may normalise the body when generating a key, so trivially different bodies hit the same entry. Two requests whose JSON differs only in key order:

{ "status": "active", "max_price": 100 }
{ "max_price": 100, "status": "active" }

Semantically identical to most applications, and normalising them into one entry is a useful optimisation. But if the cache normalises something your server treats as significant, it now serves confidently wrong answers.

This is cache key confusion: two components in a chain disagreeing about what a request means.

Where it stands right now

Status sections age badly, so here is what is measured, what is reported, and what is neither. Checked August 2026.

The authorship is a useful signal: Julian Reschke, plus James Snell of Cloudflare and Mike Bishop of Akamai. Two of three work at CDNs, which suggests where the first real cache implementations will land.

The escape hatch the RFC built in

Here is the part that changes the advice, and it is missing from most coverage.

The RFC does not require you to wait for body-keyed caching. It explicitly offers a handoff to GET:

A successful response can include a Content-Location header containing an identifier for a resource corresponding to the results of the operation; a client can send a GET request for the indicated URI to retrieve the results of the query operation just performed.

And Location can point at an equivalent resource so a client can "send a GET request to the indicated URI to repeat the query operation just performed without resending the query content". A 303 sends the client to a plain GET for the result.

So the pattern that works with today's infrastructure is: accept the QUERY, do the work, and answer with a Content-Location pointing at a cacheable GET URL for those results. The follow-up traffic is ordinary GET, which every cache, CDN and browser has understood for thirty years.

One redirect detail worth knowing, because it differs from POST: 301 and 302 do not rewrite QUERY into GET the way user agents historically did with POST. QUERY is preserved across 301, 302, 307 and 308. Only 303 moves you to GET, which is exactly what 303 has always meant.

Three things that will bite you

1. The lowercase trap, in browsers

The Fetch standard normalises the case of exactly six method names: DELETE, GET, HEAD, OPTIONS, POST and PUT. QUERY is not among them, and adding it is an open question. HTTP methods are case-sensitive, so in a browser:

// Browser: sends the method `query`, lowercase, on the wire.
// Your server is looking for `QUERY` and answers 405 or 501.
fetch('/search', { method: 'query', body: JSON.stringify(filters) });

Write it uppercase and always include Content-Type, which the RFC requires:

fetch('/search', {
  method: 'QUERY',                                    // uppercase, always
  headers: { 'Content-Type': 'application/json' },    // MUST be present and accurate
  body: JSON.stringify({ status: 'active', max_price: 100 }),
});

The wrinkle: recent undici, which backs Node's fetch, added QUERY to its normalisation. So the same lowercase code can work server-side in Node and fail in a browser. Uppercase it everywhere and the difference stops mattering.

2. The preflight, which costs less than you have been told

QUERY is not CORS-safelisted:

A QUERY request from user agents implementing Cross-Origin Resource Sharing (CORS) will require a "preflight" request, as QUERY does not belong to the set of CORS-safelisted methods.

True, and widely reported as "every QUERY costs two round trips". That overstates it twice over.

First, preflight results are cached. Set Access-Control-Max-Age and subsequent requests skip the OPTIONS.

Second, and more important: the POST you are replacing almost certainly triggered a preflight already. application/json is not a safelisted content type, so a cross-origin JSON POST has always needed a preflight. Swapping it for QUERY usually adds no new preflight at all.

Your preflight response needs more than the methods line:

Access-Control-Allow-Origin: https://app.example.com
Access-Control-Allow-Methods: QUERY, POST
Access-Control-Allow-Headers: Content-Type
Access-Control-Max-Age: 86400

Access-Control-Allow-Headers: Content-Type matters, since QUERY always carries one.

3. Your infrastructure has a method allowlist

This is the one that becomes an incident, and the reason this is a DevOps article.

Between the client and your handler sits some combination of CDN, load balancer, WAF, reverse proxy and API gateway. Several reject methods they do not recognise, and hardened configurations often allow a fixed list. CloudFront is a concrete example: it permits a fixed set of seven methods, and QUERY is not one of them. An unknown method typically returns 405 or 501 at the edge, and your application logs show nothing, because the request never arrived.

Find out before you write any code:

Two commands, five minutes, and you know whether this is a project or a non-starter.

So should you use it

Server to server, inside your own network: yes, and soon. No CORS, no browser cache to wait for, and you control both ends. Retries become semantically clean and you stop arguing about whether a search POST can be repeated.

Public API, alongside POST: yes, as an addition. Accept QUERY on the same route, advertise it with Accept-Query, keep POST working. Nothing breaks and you are ready when caches arrive.

Browser to server: only with the GET handoff. A straight POST-to-QUERY swap gains you nothing today, because no browser caches the response. Answer with Content-Location and let the follow-up be a GET, and you get real caching from infrastructure that already exists.

To escape URL length limits: yes, today. If you are base64-encoding a filter blob into a query string and fighting an 8KB header limit, QUERY solves that now, caching or not.

A note on retries

QUERY makes an automatic retry semantically permissible. It does not implement one for you.

Your client still has to know that QUERY is idempotent, decide which failures qualify, enforce limits and hold a replayable body, and a streaming body may not be replayable at all. Undici needed explicit work to classify QUERY as retryable. RFC 9110 already permitted retrying a POST when the client knew it was idempotent; what QUERY changes is that the guarantee is now in the method rather than in a comment in your code. That is worth having, but it is a clarity win, not free behaviour.

Wrapping up

QUERY is a good addition, and the people who built it knew exactly which problem they were solving. It removes a category of awkwardness that has sat in HTTP APIs for two decades.

It is also a lesson in how protocol changes actually land. Publishing an RFC is the start of the work. The method exists, browsers will send it, and your application can accept it this afternoon, but the property that makes QUERY worth adopting, a cache that keys on the request content, is not switched on in the places you deploy.

The good news is that the authors saw that coming and gave you Content-Location. You can adopt the cleaner semantics now and hand the caching to GET, which every cache in the world already understands. That is a better answer than waiting, and it is sitting in section 2 of the RFC where nobody quoting the announcement has bothered to look.

So we built one and killed it. The agent is an on-call responder: it takes an alert, triages it, gathers evidence, proposes a fix, waits for a human to approve, performs the action, and writes the handover note. Then we sent it SIGKILL at the worst possible instant, the moment after it rolled back a production deploy and before the step finished.

It recovered. It also rolled the deploy back a second time.

That is the useful finding, and this post is mostly about it: what Mastra gave us for free, what it did not, and the roughly ten lines that make the difference between an agent that is crash-safe and one that only looks crash-safe. Everything here is reproducible from the repo.

TL;DR

• The approval gate is the real win. A step suspends, the process exits, and a different process hours later resumes the run exactly where it stopped. Without a framework you build this yourself, and you will build it worse.
• After a SIGKILL mid-action, storage showed the run stuck: every earlier step success, the dying step running forever, and suspendedPaths empty, so resume() could not help it.
• restartAllActiveWorkflowRuns() recovered it and drove the run to completion. It also re-executed the interrupted step, so the rollback happened twice.
• Durable execution is at-least-once, not exactly-once. That is true of Temporal, DBOS and Restate as well. It is a property of the model, not a defect in Mastra.
• An idempotency key derived from the run id fixes it. Same crash, same recovery, action runs once.
• A small eval caught a plausible prompt "improvement" that silently stopped paging for a customer-facing outage. Same result on three different models.

Prerequisites

• Comfort with TypeScript and async/await
• A rough idea of what an LLM tool call is
• Node.js 22+ if you want to run the repo (it uses native type stripping)

What we built

Six steps. Two of them call a model, one waits for a human, one has a side effect that hurts if it happens twice.

The world it investigates is a fixture: fixed alerts, fixed deploy history, fixed error rates. That is deliberate. It means the only non-determinism in the system is the model itself, so a run differs in wording but never in facts.

Here is the agent doing its job. The alert says checkout p99 is 14.2 seconds, and there was a deploy eight minutes ago:

It reached the right answer: page, not ticket, because customers are affected right now, and roll back the deploy that landed immediately before the spike. Nine and a half seconds end to end on deepseek-v4-pro, of which seven were spent reaching the approval gate.

Where it won

The approval gate is worth the whole framework

This is the step that justifies the dependency:

const approve = createStep({
  id: 'approve',
  inputSchema: proposed,
  outputSchema: approved,
  suspendSchema: z.object({ question: z.string(), proposal: z.string() }),
  resumeSchema: z.object({ approved: z.boolean() }),
  execute: async ({ inputData, resumeData, suspend }) => {
    if (!resumeData) {
      return await suspend({
        question: `Approve this action on ${inputData.service}?`,
        proposal: inputData.proposal,
      });
    }
    return { ...inputData, approved: resumeData.approved };
  },
});

suspend() writes the entire run state to storage and returns. The process can exit. Tomorrow morning, a completely different process picks the run up by id and resumes it, and the agent carries on from step four with everything the first three steps learned still intact.

Think about building that yourself. You need to serialise the whole conversation, the tool results and the position in the flow, store it, then reconstruct it. It is a weekend of work, and the version you write will have bugs the framework has already found.

The types actually hold

Each step declares its input and output schema, and the next step's input is literally the previous step's output type:

const gathered = triaged.extend({
  evidence: z.object({
    recentDeploy: z.string().nullable(),
    errorRate: z.number(),
    baseline: z.number(),
  }),
});

Rename a field in step two and step three stops compiling. For a pipeline where the interesting bugs are shape mismatches four steps downstream, that is not a small thing.

The evals earn their place immediately

We wrote a three-case eval, then made a prompt edit that any of us might have committed on a Friday. The original instructions say "be conservative: if customers are currently affected, it is a page". The "improvement" says "page: only for total outages of the entire platform" and "avoid paging people unless absolutely unavoidable".

That reads like a reasonable response to alert fatigue. Here is what it does:

The one case that broke is the one that matters: a live customer-facing outage quietly downgraded from a page to a ticket. Nobody gets woken up. You find out from customers.

We ran the same eval on three different models and got the identical 3/3 to 2/3 result each time, which says the regression is a property of the prompt change rather than a quirk of one model.

Where it would not

Now the part that made the post worth writing.

The setup

We gave the act step a window: it writes to a ledger, then stays busy for a few seconds. The harness watches that ledger and sends SIGKILL the instant the side effect lands. That timing is not a guess. The process always dies inside the dangerous window, after the action has really happened and before the step has recorded that it finished.

Then a completely fresh process asks storage what it thinks happened.

Read that last block carefully, because it is the whole problem.

The run is orphaned. Four steps are safely recorded as success, which is genuinely valuable: we know exactly how far it got. But the step that was in flight is marked running, and it will stay running forever, because the only process that could have finished it is dead. And suspendedPaths is empty, so the run is not suspended, which means resume() has nothing to resume.

Nothing recovers this on its own. The incident is half-handled and silent.

Recovery works, and costs you a second rollback

Mastra has an API for exactly this situation. It picks up runs that storage still believes are active and drives them to completion:

await wf.restartAllActiveWorkflowRuns();

It worked. The run went to success, the writeup was generated, the incident closed properly.

And the ledger went from one entry to two.

We rolled back the deploy, crashed, recovered, and rolled it back again. In a real system that is a second rollback fired at a service someone may already be repairing by hand.

The reason this deserves a section rather than a footnote is that "durable execution" is marketed in a way that strongly implies the opposite. If you read the feature list and assume your side effects are protected, you will ship exactly this bug, and you will only find it during an incident, which is the worst possible time to discover that your incident tooling has a bug.

The fix is small, and you have to know to write it

Derive a key from something stable across the restart, and make the action a no-op the second time:

export function recordOnce(key: string, entry: LedgerInput) {
  if (entries().some((e) => e.key === key)) return null;   // already done
  appendFileSync(LEDGER, JSON.stringify({ ...entry, key }) + '\n');
}

// in the step, `runId` survives the crash, so the key does too
recordOnce(`${runId}:act`, { runId, action: inputData.proposal, target: inputData.service });

The critical detail is where the key comes from. It has to be derived from the run id, which storage remembers, and not generated inside the step, which would produce a fresh key on every attempt and guard nothing.

Same experiment, same SIGKILL, same recovery call:

Ten lines, and the difference between an agent that is crash-safe and one that merely appears to be.

Three smaller things that cost us time

The restart call returns before the work finishes. restartAllActiveWorkflowRuns() resolves immediately, not when the restarted runs complete. Our first version of the harness read the ledger straight after it and reported the wrong answer. You need to poll storage until the run leaves the running state.

Orphan recovery is not automatic. Nothing sweeps up stuck runs for you. If your process can die, something in your deployment has to call the restart path on boot, and that something is your code.

The API has moved. We first installed @mastra/core@0.10 because that is what a plain semver range resolved to, then pinned 1.57.0 for everything here. Between those two versions, createRunAsync() became createRun(), and getWorkflowRunById() returns the run flattened rather than under a snapshot key.

How fast is fast? 1.58.0 shipped overnight while this article was being finished. That is not a complaint, an actively developed library is what you want here, but it does mean you should pin your version and read the changelog rather than trusting a blog post, including this one.

What the framework is actually buying you

To make the comparison concrete rather than rhetorical, we built the same triage against the same endpoint as a plain tool loop, no framework at all:

for (let i = 0; i < 6; i++) {
  const reply = await chat(messages);
  messages.push(reply);
  if (!reply.tool_calls?.length) break;      // done
  for (const tc of reply.tool_calls) {
    const out = callTool(tc.function.name, JSON.parse(tc.function.arguments));
    messages.push({ role: 'tool', tool_call_id: tc.id, content: JSON.stringify(out) });
  }
}

It works. It reaches the same conclusion, page plus roll back 4f21ab9, in three model turns. If your agent is one model with a few tools and no state between calls, this is genuinely the right answer and a framework is overhead.

What it cannot do is everything this post has been about. There is no approval gate, because there is nowhere to put a run while a human thinks. There is no recovery, because there is no record. If that process dies, the run is simply gone, and no amount of idempotency keys helps because there is nothing left to restart.

That is the honest trade. You adopt a framework at the point where runs must outlive processes, and not before.

Would we use it again

Yes, for this shape of problem, with the caveat above written on the wall.

The parts that made it worth the dependency were the suspend and resume across processes, which is the hard part done properly, and the step-level record in storage, which meant that after an ugly crash we could see precisely which steps had committed and which had not. Debugging that same crash in a hand-rolled loop means reading logs and guessing.

The part to internalise is that durable execution protects your workflow, not your side effects. Mastra remembered where the run had got to, which is exactly what it promises. It could not know whether the rollback we fired had reached the deploy system, because nothing outside our own code could know that. That boundary is where your idempotency keys go, and no framework will draw it for you.

What we did not test

Being clear about the edges of this:

• One workload, one shape. An incident responder with a human gate. Nothing here says how it behaves with high concurrency, long-running memory, or hundreds of parallel runs.
• SQLite storage. We used LibSQL on one machine. A Postgres-backed store under real contention may behave differently, particularly around the orphaned-run case.
• One failure mode. We killed the process. We did not test network partitions, storage failures mid-write, or a model provider going down between steps.
• Not a framework comparison. We did not build this five ways and time them. If you want the survey, we wrote the top five agent frameworks in 2026 separately, and this post is the hands-on half of that one.
• An open model, not a frontier one. Everything ran on deepseek-v4-pro through an OpenAI-compatible gateway. The crash results are independent of the model, but the triage quality would likely improve on a larger one.

If the agent loop itself is the part that still feels like magic, our agentic loop simulator walks through plan, build, verify and repeat one step at a time.

The one thing to take away

If you are putting an agent anywhere near a system that can change production, write the crash test before you write the demo. It took us an afternoon, it is about eighty lines, and it turned a comfortable assumption into a measured fact.

The assumption was that durable execution meant our actions were safe. The fact is that it meant our workflow was safe, and our actions were exactly as safe as we had made them.

Neon Auth takes a different position: the auth server runs in the same project as your database. You turn it on with one line of config, and after a deploy the user who signs in is a row in your Postgres, in a schema you can query and join against your own tables. This post walks through how that works, what you actually get, and why the sync layer you are used to writing simply goes away. There is a working repo at the end.

TL;DR

• Neon Auth is an auth server that lives inside your Neon project. Enable it with auth: true in neon.ts and provision it with one neon deploy.
• It issues signed JWTs and publishes a JWKS endpoint, so any backend verifies a token with public-key crypto and no shared secret.
• User, session, and account data live in a neon_auth schema in the same Postgres. The id in the token is the primary key of neon_auth.user, so it is a real foreign key for your tables, no webhook sync required.
• Because auth state lives in Postgres, it branches with your database: a preview branch gets its own isolated set of users.
• It is built on Better Auth, so the sign-in, sign-up, and token endpoints are the standard ones you may already know.

Prerequisites

• A Neon project on the platform preview (us-east-2, new projects)
• The Neon CLI (npm i -g neon) and a linked project
• Familiarity with JWTs at the level of "a signed token with claims"

The reconciliation tax

Here is the shape most apps end up with. Two systems, and glue in the middle to keep them agreeing:

Auth service                     Your database
┌───────────────┐   webhook      ┌───────────────┐
│ users         │ ─────────────▶ │ users (copy)  │
│ sessions      │   + retry job  │ orders        │
│ oauth config  │ ◀───reconcile─ │ ...           │
└───────────────┘                └───────────────┘
        the id here  ─── must match ─── the foreign key here

None of that glue is business logic. It exists only because identity lives in one place and your data lives in another, and the two have to be reconciled. When they drift, you get the classic bugs: an order row whose user_id points at a user your database never heard about, or a user who can sign in but has no profile because the webhook that was supposed to create it got a 500 and never retried.

Neon Auth removes the two-systems problem by putting the auth server in the same project as the database.

Turn it on

The whole configuration is one property. In neon.ts, the file that declares what services your Neon project runs, you set auth: true:

import { defineConfig } from "@neon/config/v1";

export default defineConfig({
  // Provisions a Neon Auth server on this branch. Postgres is on by default.
  auth: true,
});

Then deploy. neon deploy provisions the service and writes its connection details into your local .env.local for development:

That is the entire setup. There is no second project to create, no separate dashboard, no API key to copy between systems. The base URL is where users sign in and out; the JWKS URL is where you fetch the public keys to verify tokens.

What you get: a token and a way to trust it

Neon Auth is built on Better Auth, so the HTTP surface is the standard set of endpoints under the base URL: /sign-up/email, /sign-in/email, /get-session, /token, and the JWKS at /.well-known/jwks.json. A signed-in session exchanges for a JWT at /token. Decoded, that token carries the claims you would expect:

{
  "sub": "e2163035-50f4-4753-906d-78b79a124b0b",
  "name": "Alice",
  "email": "alice@example.com",
  "role": "authenticated",
  "iss": "https://<id>.neonauth.<region>.aws.neon.tech",
  "exp": 1782990705
}

The token is signed with EdDSA (an Ed25519 key), and the JWKS endpoint serves the matching public key. That means any backend can verify a token without sharing a secret with the auth server: fetch the public key, check the signature, check the issuer. In a Neon Function the whole verification is a few lines with jose:

import { createRemoteJWKSet, jwtVerify } from 'jose';

const jwks = createRemoteJWKSet(new URL(process.env.NEON_AUTH_JWKS_URL!));
const issuer = new URL(process.env.NEON_AUTH_BASE_URL!).origin;

async function verify(token: string) {
  // Throws if the signature, issuer, or expiry is wrong.
  const { payload } = await jwtVerify(token, jwks, { issuer });
  return { id: payload.sub as string, name: payload.name as string };
}

createRemoteJWKSet fetches and caches the public keys, so this does not hit the network on every request. Nothing here is Neon-specific cryptography; it is standard JWT verification against a JWKS, which is exactly the point. Your backend does not need a Neon SDK to trust a Neon Auth token.

On the frontend you do not hand-roll any of this. The @neondatabase/auth package gives you a client and server helper, and @neondatabase/auth-ui ships the sign-in and sign-up screens, so a Next.js app wires up with a provider and a catch-all route rather than a login form you build yourself. The demo repo has the full wiring.

The part that matters: the user is a row in your database

This is where the single-project design pays off. Neon Auth stores its data in a neon_auth schema inside the same Postgres as your app. It is not hidden behind an API; it is tables you can query:

The id in neon_auth.user is the same value as the sub claim in the JWT. So when your app stores something owned by a user, you store that id, and it is a genuine foreign key into a table sitting in the same database. You can join across the two:

-- messages your app wrote, next to the identity that wrote them,
-- resolved in one query against one database.
select m.id, m.body, u.email
from public.messages m
join neon_auth."user" u on u.id::text = m.user_id
order by m.id;

 id |     body      |        email
----+---------------+----------------------
  1 | hello         | alice@example.com
  2 | welcome back  | chat-test@example.com

There is no webhook that copied alice@example.com into your schema, and no reconciliation job to make sure it stays copied. The message row and the user row are in the same Postgres, so the join is a normal join. That is the whole reconciliation tax from earlier, gone: not automated, just absent.

Auth that branches with your data

Neon's headline feature is database branching: fork the whole database, data and all, in seconds. Because auth state lives in the same Postgres, it branches too. Create a branch for a preview environment and it comes with its own neon_auth schema, its own users, its own sessions. Someone signing up against a preview branch is not creating an account in production.

With a separate auth service this is genuinely hard. You either point every preview at one shared auth tenant (so preview signups pollute real data) or you script the creation and teardown of a throwaway tenant per environment. When auth lives in the branch, you get an isolated identity store for free every time you branch, and it disappears when the branch does.

Where this does not fit

The single-project design has a cost, and it is worth being straight about it before you build on this.

It is beta, and the region is fixed. The platform preview this uses is available only in AWS US East (Ohio), aws-us-east-2. If your data has to live in the EU, this is not a decision you can make yet.

Coupling identity to your database provider is a real trade. The usual argument for a separate auth service is that it is separate: you can move your database without touching your login flow. Here the two move together. That is exactly what removes the sync layer, and it is also what you give up. The mitigating detail is that it is Better Auth underneath with a standard schema, so an exit is a Postgres migration rather than a re-implementation, but it is still work you would not otherwise do.

Standard JWT caveats still apply. Verification is stateless, so a token stays valid until it expires. If you need a sign-out that takes effect immediately everywhere, you need a check against session state on the requests that matter, the same as with any JWT setup.

None of these are reasons not to use it. They are the questions to answer first, and "we are in one AWS region and we are staying on Postgres" makes most of them go away.

The repo

A full working example, a Next.js app with Neon Auth plus a WebSocket chat backend that verifies these tokens, is here:

The next post in this series, realtime chat with auth, builds on this and takes the token to the hard place: authenticating a WebSocket, where the browser cannot even set an Authorization header.

Wrapping up

Most auth setups carry a hidden cost that has nothing to do with authentication: the work of keeping a separate identity system in sync with your database. Neon Auth removes that cost by not having a separate system. One line of config provisions an auth server in your project; it issues standard JWTs you verify against a JWKS with no shared secret; and the users it manages are rows in a neon_auth schema you can join to your own tables. The identity that signs in and the data it owns live in the same Postgres, and branch together. That is a smaller, more boring architecture than the two-system norm, which is exactly what you want from the auth layer.

This post is a build-log for a realtime chat that answers it. It runs a Neon Function as the WebSocket server, uses Neon Auth for identity, and stores messages in the same Postgres. Every socket is authenticated with a Neon Auth JWT that the function verifies before it accepts the upgrade, the stored identity comes from the verified token rather than anything the client claims, and messages fan out across isolates with Postgres LISTEN/NOTIFY. If you have not seen how Neon Auth issues those tokens, the previous post, auth for a Postgres app without a separate service, covers it. The full repo is at the end.

TL;DR

• Browsers cannot set headers on a WebSocket, so the client passes its Neon Auth JWT as a ?token= query parameter. The Sec-WebSocket-Protocol subprotocol is the alternative that keeps it out of access logs, and the post covers when to prefer it.
• The function exports { fetch, upgrade }. The upgrade hook verifies the token against the Neon Auth JWKS and rejects with 401 before the socket is ever accepted.
• The identity written to each message is the sub from the verified token, never a name the client sends. That is the difference between "signed in as Alice" and "typed the name Alice".
• Broadcasting in-process only reaches clients on the same isolate. Postgres LISTEN/NOTIFY fans each message out to every isolate so the chat is genuinely shared.
• The client reconnects with backoff and re-mints a token on each attempt, because serverless isolates get evicted when idle.

Prerequisites

• A Neon project with Neon Auth enabled (auth: true in neon.ts, see the previous post)
• Comfort with WebSockets and JWTs
• Node.js and the Neon CLI

The shape of it

There are two backends and one browser. The Next.js app handles sign-in and serves chat history over HTTP; the Neon Function is the WebSocket server the browser talks to directly for live messages.

A Neon Function is a long-running Node.js handler, not a per-request lambda, which is what makes a WebSocket server possible at all. The function exports two entry points: fetch for ordinary HTTP, and upgrade for the WebSocket handshake.

import { Hono } from 'hono';
import { WebSocketServer } from 'ws';

const app = new Hono();
app.get('/', (c) => c.text('Connect over WebSocket with ?token=<jwt>'));
const wss = new WebSocketServer({ noServer: true });

export default {
  fetch: (request: Request) => app.fetch(request),
  async upgrade(req, socket, head) {
    // ...this is where auth happens, before we accept the socket
  },
};

Auth over a WebSocket

Because the browser cannot add a header, the token rides in the URL. The client mints a JWT from its Neon Auth session and opens the socket with it as a query parameter:

const token = await getToken();           // from the Neon Auth session
const ws = new WebSocket(`${WS_URL}?token=${encodeURIComponent(token)}`);

On the server, the upgrade hook reads that token and verifies it before doing anything else. Verification is the standard JWKS check from the previous post: fetch the auth server's public key, check the signature, check the issuer. If it fails, the connection is refused with a raw 401 and never becomes a WebSocket at all.

import { createRemoteJWKSet, jwtVerify } from 'jose';

const jwks = createRemoteJWKSet(new URL(env.auth.jwksUrl));
const issuer = new URL(env.auth.baseUrl).origin;

async function verifyToken(token: string | null) {
  if (!token) return null;
  try {
    const { payload } = await jwtVerify(token, jwks, { issuer });
    return { id: payload.sub as string, name: (payload.name as string) ?? 'anon' };
  } catch {
    return null;
  }
}

async upgrade(req, socket, head) {
  const url = new URL(req.url ?? '/', 'http://localhost');
  const identity = await verifyToken(url.searchParams.get('token'));
  if (!identity) {
    socket.write('HTTP/1.1 401 Unauthorized\r\n\r\n');
    socket.destroy();
    return;
  }
  wss.handleUpgrade(req, socket, head, (ws) => onConnection(ws, identity));
}

Rejecting at the handshake matters. An unauthenticated client never gets an open socket, so there is no "connected but not yet authenticated" state to babysit, no first-message-must-be-a-token dance, and no window where an anonymous connection is holding a slot. The check is a precondition of the upgrade, not a step after it.

The subprotocol alternative

The query parameter is not the only option, and if you noticed that the WebSocket constructor also takes a subprotocol, you have already spotted the other one. Whatever you pass there is sent as a Sec-WebSocket-Protocol header, which means the token travels in a header after all:

// The token rides in Sec-WebSocket-Protocol instead of the URL.
const ws = new WebSocket(WS_URL, ['auth', token]);

The server reads it from req.headers['sec-websocket-protocol'] and must echo one of the offered values back in the handshake response, or the browser drops the connection.

The advantage is real: request URLs are logged by almost every proxy and server by default, and headers usually are not, so this keeps the token out of your access logs. The costs are that the subprotocol value must be a valid token per the WebSocket spec (a JWT is fine, it is base64url and dots), you now have to remember the echo step, and you are using a protocol negotiation field for something that is not a protocol.

This build uses the query parameter because it is the simpler thing to demonstrate and the short TTL bounds the exposure. If you are running this where your proxy logs are retained and widely readable, the subprotocol version is the better default, and it changes about four lines.

A socket outlives its token

One thing the handshake check does not give you: the token is verified once, at connect. A socket opened with a valid token stays open after that token expires, potentially for hours. For a chat that is usually fine, and it is what this build does.

If you need revocation to bite on a live connection, the handshake is not enough. The usual fix is to record the token's exp at connect time and close the socket when it passes, forcing the client through its normal reconnect path with a fresh token:

const expiresAt = (payload.exp as number) * 1000;
setTimeout(() => ws.close(4001, 'token expired'), expiresAt - Date.now());

Because the client already re-mints on every reconnect, that turns into a brief blip rather than a logout.

The identity comes from the token, not the client

This is the part that is easy to get subtly wrong. Once the socket is open, the client sends message text. It would be tempting to also let it send a display name, or a user id, along with each message. Do not. The only trustworthy identity is the one inside the verified token. The message handler uses identity captured from the JWT at connection time, and takes only the message body from the wire:

ws.on('message', async (data) => {
  const body = data.toString().slice(0, 2000).trim();
  if (!body) return;
  const [row] = await db
    .insert(messages)
    .values({ userId: identity.id, userName: identity.name, body }) // from the token
    .returning();
  await pool.query('SELECT pg_notify($1, $2)', [CHANNEL, JSON.stringify(row)]);
});

userId and userName come from the verified token; body is the only thing the client controls. That is the line between "signed in as Alice" and "sent a message with the name Alice attached". If you trusted a client-supplied id, any connected user could write a message as anyone else. Because the id is the sub claim, it is also the primary key of neon_auth.user, so every row is attributable to a real account you can join against, which is the whole point of the previous post.

Fan-out: why in-process broadcasting is not enough

Here is the gotcha that only shows up under load. The obvious way to broadcast is to keep the connected sockets in a Set and loop over them when a message arrives. That works perfectly with one server process. But a Neon Function, like most serverless runtimes, can run several isolates at once, each with its own set of connected clients. A message that arrives on isolate A and only loops over isolate A's sockets never reaches the users connected to isolate B. Your chat silently splits into rooms that cannot hear each other.

The fix is to route every message through Postgres. Each isolate holds its in-process Set for the final hop, but it also LISTENs on a Postgres channel. When a message is inserted, the handler NOTIFYs that channel, and every isolate, including the one that received the message, gets the payload and broadcasts to its own sockets.

const clients = new Set<WebSocket>();     // sockets on THIS isolate
const CHANNEL = 'chat_messages';

// A dedicated connection LISTENs; the DB is the fan-out bus.
const listener = new Client({ connectionString: env.postgres.databaseUrlUnpooled });
await listener.connect();
await listener.query(`LISTEN ${CHANNEL}`);
listener.on('notification', (msg) => {
  for (const ws of clients) {
    if (ws.readyState === ws.OPEN) ws.send(msg.payload);
  }
});

So the path of a message is: verify the sender at connect, insert the row on receive, NOTIFY the channel, every isolate hears it, each isolate sends to its own sockets. Postgres is doing double duty as the message store and the pub/sub bus, which means there is no Redis or separate broker to run. The database you already have is the fan-out layer.

Proving it works

Claims about auth are cheap; the interesting question is whether the wall actually holds. The repo ships an end-to-end test that runs the whole flow against the deployed function: it tries to connect without a token, with a garbage token, and then with a real Neon Auth JWT, and finally checks that a message from one client reaches a second client and lands in Postgres under the verified identity.

The two 401 lines are the important ones: they confirm the handshake refuses anything without a valid token. The last line confirms the row was stored under the identity from the JWT, not a name off the wire. The test signs a throwaway user up against Neon Auth and exchanges the session for a JWT exactly the way the browser does, so it exercises the real token path rather than a mock.

Reconnecting like a serverless client should

One more reality of serverless: an idle isolate can be evicted, which closes your socket. The client treats that as normal and reconnects with exponential backoff, and, importantly, mints a fresh token on each attempt rather than reusing the one it opened with, since tokens expire.

ws.onclose = () => {
  setConnected(false);
  if (!closed) timer = setTimeout(connect, Math.min(1000 * 2 ** retry++, 15000));
};
// connect() calls getToken() again every time, so a reconnect never
// replays an expired token.

That is what makes the short token TTL from earlier a non-issue in practice: the client is already re-authenticating on every reconnect, so nothing depends on a token living a long time.

The repo

The full function, the Next.js app with Neon Auth, and the integration test are here:

Wrapping up

The hard part of realtime auth is not the cryptography, it is the handshake: a WebSocket cannot carry a header, so you pass the token in the URL and verify it before you accept the connection, refusing anything invalid with a 401 up front. From there the rules are ordinary but easy to skip under deadline: take identity from the verified token and never from the client, and remember that in-process broadcasting fragments across isolates, so route fan-out through the database with LISTEN/NOTIFY. Because Neon Auth issues the tokens and Postgres stores both the messages and the pub/sub, the whole thing is one project with nothing else to run, and the test suite proves the wall around it actually stands.

What separates them now is what happens on the second day: when the process restarts halfway through a run, when a tool needs a human to approve it, when someone asks why the agent did that, and when you need to prove a prompt change made things better rather than worse.

This ranks five frameworks on that basis. The criteria are stated below so you can disagree with the ranking rather than guess at it, and every number comes from GitHub and npm on 11 August 2026 rather than from anyone's marketing page.

TL;DR

• Mastra takes first place for TypeScript teams that want one integrated stack: durable workflows, memory, evals and tracing without assembling four libraries.
• LangGraph wins on control and ecosystem depth. Pick it when you need to define the graph yourself.
• OpenAI Agents SDK is the shortest path if you have already committed to OpenAI.
• Vercel AI SDK owns the streaming and UI edge, and now has real agent primitives, but still no durable workflow engine.
• PydanticAI is the one to reach for if your team is Python and cares about types.
• Popularity is not the ranking. The most-starred project in this space is not in the top five, and the reason is explained below.

Prerequisites

• Familiarity with calling an LLM API and the idea of tool or function calling
• Node.js 20+ or Python 3.10+ depending on which you try

The criteria

A ranking without criteria is just an opinion with numbers attached. These are mine, weighted for teams putting an agent in front of real users:

1. Durable execution. If the process dies mid-run, does the agent resume, or does the user lose their work?
2. Memory that is not a hand-rolled array. Conversation and working memory as a supported concept with real storage behind it.
3. Evaluation. Can you tell whether a change made the agent better, before shipping it?
4. Observability. Traces you can read when someone asks what happened.
5. Type safety and developer experience, because agents are mostly plumbing and plumbing benefits enormously from a compiler.
6. Model neutrality. How expensive is it to change provider when pricing moves?

Nothing here scores frameworks on how quickly you can build a demo. They are all fine at that.

The numbers

Collected on 11 August 2026 from api.github.com/repos/<owner>/<repo> and api.npmjs.org/downloads/point/last-week/<package>, so you can re-run them and check. Stars measure attention rather than quality. The npm figures cover the JavaScript package only, which is why a Python-first project shows n/a rather than a zero, and why the two columns should not be compared against each other.

Notice that the ranking below is not this chart sorted. If it were, this article would be a popularity contest and you could have got it from GitHub yourself.

How they score against the criteria

The distinction that matters in this table is built in versus available. Almost everything here is available somewhere, if you are willing to add a dependency and wire it up. What separates them is how much of that wiring you do yourself.

Two things in that table are worth saying out loud, because they cut against the ranking.

PydanticAI's durability story is better than its position suggests. It supports four co-maintained durable execution backends (Temporal, DBOS, Prefect and Restate), plus Kitaru and Airflow. That is more choice than anyone else here offers. The tradeoff is that you are running Temporal, which is a real piece of infrastructure to operate, where Mastra's durability needs nothing extra on day one.

Vercel AI SDK's row of "no" is not a failing grade. It is a different product, and the section below explains why it is still on the list.

1. Mastra

Best for: a TypeScript team building a production agent on a deadline.

Mastra is the one that treats the second-day problems as the product rather than as extensions. Durable workflows, memory, evals, tracing and MCP support are in the box and designed together, which is the difference between a framework and a collection.

The workflow primitive is the part worth understanding. Steps are typed, composable and resumable, so a run that dies at step four resumes at step four rather than at the beginning:

import { createWorkflow, createStep } from '@mastra/core/workflows';
import { z } from 'zod';

const triage = createStep({
  id: 'triage',
  inputSchema: z.object({ alert: z.string() }),
  outputSchema: z.object({ severity: z.enum(['page', 'ticket', 'ignore']) }),
  execute: async ({ inputData, mastra }) => {
    const agent = mastra.getAgent('oncall');
    const res = await agent.generate(`Classify: ${inputData.alert}`);
    return { severity: parseSeverity(res.text) };
  },
});

export const incidentWorkflow = createWorkflow({ id: 'incident' })
  .then(triage)
  .then(notify)
  .commit();

The schemas are the point. Each step declares what it takes and returns, so the compiler catches a mismatch between step three and step four rather than production catching it.

The memory work is the part with numbers attached, and it is the strongest single argument for the top spot. Mastra's Observational Memory runs background observer and reflector agents that maintain a dense observation log, replacing raw message history as a conversation grows. On LongMemEval, published February 2026, it reports:

The number to compare is the gpt-4o one, because that is what other published results use. The previous openly reproducible best was Supermemory at 81.60%.

Two caveats, because a vendor benchmark deserves them. This is Mastra measuring Mastra, and a benchmark is not your workload. What makes it worth citing anyway is that the implementation and the benchmark runner are both open source, so the claim is checkable rather than asserted. It also needs no vector database, which removes a piece of infrastructure most memory designs assume.

Where it wins: one dependency instead of four, with the pieces already fitted together. Local development has a Studio for inspecting runs and traces, which removes the usual print-statement phase. Model-neutral, so switching provider is configuration.

Where it loses: it is younger than LangGraph and the ecosystem around it is correspondingly smaller. If you want a pre-built integration for something unusual, you are more likely to find it in LangChain's ecosystem, and more likely to write it yourself here. It is also TypeScript-first, so a Python shop should look further down this list.

Adoption: 27,101 stars and 1.3M weekly downloads of @mastra/core, with production use reported at Replit, PayPal, Sanity and Brex. Founded by Sam Bhagwat, Abhi Aiyer and Shane Thomas, who built Gatsby and stayed on through its acquisition by Netlify. YC W25.

2. LangGraph

Best for: complex, stateful workflows where you want to define the graph yourself.

LangGraph models an agent as an explicit state machine. You define nodes and edges, and control flows exactly where you put it. When the branching is genuinely complicated, that explicitness is worth a great deal, and nothing else here gives you the same grip on the details.

from langgraph.graph import StateGraph, END

graph = StateGraph(AgentState)
graph.add_node("triage", triage_node)
graph.add_node("remediate", remediate_node)
graph.add_conditional_edges(
    "triage",
    lambda s: "remediate" if s["severity"] == "page" else END,
)
graph.set_entry_point("triage")
app = graph.compile(checkpointer=checkpointer)

That checkpointer is durable execution, and it was in LangGraph before most of the field took the problem seriously.

Where it wins: control, maturity, and the largest ecosystem in the category. If an integration exists anywhere, it probably exists here first.

Where it loses: you write more of the plumbing yourself, and the graph is a real abstraction to learn rather than an API to call. The JavaScript library is a real one, with durable execution, interrupts, memory and both the graph and functional APIs, so "Python only" would be unfair. The softer and still true version is that Python is where the project's centre of gravity sits: the examples, the integrations and the community answers you will search for are disproportionately Python.

3. OpenAI Agents SDK

Best for: teams already committed to OpenAI who want the shortest path.

A small, well-made library covering agents, handoffs, guardrails and sessions, in Python and TypeScript. If your models come from OpenAI and your needs are a tool loop with some structure, this is less code than anything else here and the built-in tracing is genuinely good.

Handoffs are the idea worth borrowing. Instead of one agent with twelve tools, you give each agent a narrow job and let it pass control:

from agents import Agent, Runner

escalation = Agent(
    name="escalation",
    instructions="Page the on-call engineer and summarise the alert.",
)

triage = Agent(
    name="triage",
    instructions="Classify the alert. Hand off anything user-facing.",
    handoffs=[escalation],
)

result = await Runner.run(triage, "checkout latency p99 is 14s")

The handoff is a tool call under the hood, so the model decides when to escalate and the trace shows you why.

Where it wins: minimal surface area, excellent tracing, first-party support for OpenAI's own features on the day they ship.

Where it loses: the gravity is toward one provider. It does support others, but you are building on a vendor's SDK, and the day pricing moves is the day that matters. Durable execution is not the built-in story it is in Mastra or LangGraph.

4. Vercel AI SDK

Best for: streaming model output into a React interface.

At 20.5 million weekly downloads it is by far the most used package in this article, and it has moved a long way from being only a streaming helper. It now ships ToolLoopAgent and WorkflowAgent, subagents, memory guidance, policy-based tool approvals, and HarnessAgent for driving preconfigured harnesses like Claude Code or Codex. Anyone still describing it as "just the UI layer", as an earlier draft of this article did, is working from a stale picture.

The distinction that survives is narrower and still decisive: there is no durable workflow engine. The loop runs in your process. If that process dies at step four, nothing brings it back to step four, and the documented workflow patterns are conditionals and functions in your own code rather than a checkpointed state machine.

That is a design choice, not a defect. The pattern that works well in 2026 is to use it for the edge it is unmatched at while something else owns durability. Mastra reuses it at the UI boundary for exactly this reason.

The API is about as small as this gets, and swapping provider really is one line:

import { streamText, tool } from 'ai';
import { anthropic } from '@ai-sdk/anthropic';
import { z } from 'zod';

const result = streamText({
  model: anthropic('claude-sonnet-5'), // swap for openai(...) and nothing else changes
  prompt: 'Summarise the last deploy',
  tools: {
    getDeploy: tool({
      description: 'Fetch the most recent deploy',
      inputSchema: z.object({ service: z.string() }),
      execute: async ({ service }) => fetchDeploy(service),
    }),
  },
});

return result.toUIMessageStreamResponse(); // straight into a React hook

That last line is the reason people reach for it. Getting tokens onto the screen, with tool calls rendered as they happen, is genuinely hard, and this makes it a one-liner.

Where it wins: streaming, generative UI, and the smoothest React integration available.

Where it loses: durability and evaluation. A run that dies is gone, and there is no eval story in the box, so both are yours to build or to borrow from another library.

5. PydanticAI

Best for: Python teams who want types to mean something.

From the Pydantic team, and it shows. Structured outputs are validated properly, dependency injection is a first-class idea, and the whole thing feels like a library written by people who ship production Python rather than demos.

The output type is the contract, and the agent is re-prompted until it satisfies it:

from typing import Literal

from pydantic import BaseModel
from pydantic_ai import Agent

class Triage(BaseModel):
    severity: Literal['page', 'ticket', 'ignore']
    reason: str

agent = Agent('anthropic:claude-sonnet-5', output_type=Triage)

result = await agent.run('checkout latency p99 is 14s')
print(result.output.severity)  # a validated Triage, not a string to parse

You get a typed object or an error. There is no branch where the agent returns prose and you write a regex to rescue it.

Where it wins: validation you can trust, a clean testing story, and the FastAPI-shaped ergonomics that a lot of Python teams already think in. Durability is a genuine strength too: four co-maintained backends is more choice than anything else on this list.

Where it loses: it deliberately does less itself. Durability, observability and evals all come from separate pieces (Temporal or DBOS, Logfire, pydantic-evals), which is more assembly than Mastra asks for, and more infrastructure to run. If you want one integrated framework, this is not trying to be one.

Why CrewAI and Google ADK are not in the five

Leaving out the most-starred project in the category needs a reason.

CrewAI has 56,938 stars, more than anything else here, and it is genuinely the fastest way to express a team of role-playing agents that collaborate. The usual dismissal, that the crew metaphor is too strong an opinion about how your agents should be organised, only addresses half the product: CrewAI also has Flows, a more controlled API with persistent state, resume and human-in-the-loop triggers, which is much closer to what LangGraph offers. The narrower reason it is not ranked is that the framework asks you to choose between those two models up front, and its centre of gravity is still the crew. When that metaphor fits your problem, it fits well, and it should be on your shortlist.

Google ADK at 21,072 stars is the closest call on this list, and the easy dismissal of it is wrong. It is not Python-only (Python, TypeScript, Go, Java and Kotlin are all supported) and it is not Gemini-only (there are adapters for Claude, OpenAI, Ollama, vLLM and LiteLLM). The honest reason it is not ranked is narrower: its centre of gravity is Google Cloud, where the managed deployment, Cloud Trace observability and auth story are clearly the intended path. If you are already there, move it up your own list.

Both belong on a longer list. Neither changes the answer for most teams.

Choosing between them

If the loop itself is the part that still feels like magic, our agentic loop simulator steps through plan, build, verify and repeat one stage at a time, including what happens when you let the agent grade its own work.

Common questions

Do I need an agent framework at all?

Often not. If you are calling one model with three tools and no state between calls, a plain SDK call in a loop is perfectly reasonable and easier to debug. The frameworks start paying for themselves at the point you need runs to survive a restart, conversations to persist, and changes to be evaluated rather than eyeballed. Adopt one when you hit that, not before.

Which is best for a TypeScript team?

Mastra, in most cases, because durability, memory, evals and tracing arrive together. Vercel AI SDK if the hard part is the interface rather than the agent, and the two are frequently used together. LangGraph's JavaScript library is fully capable, but most of its examples and community answers are written in Python.

Which is best for Python?

LangGraph if the complexity is in the control flow and you want to hold the graph yourself. PydanticAI if the complexity is in the data and you want validated outputs, with durability supplied by Temporal or DBOS.

Is CrewAI a bad choice because it is not in the top five?

No. It is the most-starred project in the category and it is very good at what it does, which is teams of role-playing agents collaborating on a task. It is not ranked here because that metaphor is a strong assumption about how your system is shaped, and most production agents are one agent doing one job carefully.

How hard is it to switch later?

Easier than it feels, if you keep your tools as plain functions and your prompts out of the framework's types. The tool implementations and the domain logic port with little friction. What does not port is the orchestration layer, so the switching cost is roughly the cost of rewriting your workflow definitions.

Are these rankings based on benchmarks?

No, with one exception. The ranking weighs documented capability against the criteria at the top of this article. The only measured numbers here are the GitHub and npm figures, and Mastra's LongMemEval results, which are Mastra's own published benchmark rather than an independent one.

What this ranking does not tell you

Being honest about the limits of a list like this:

• These are mostly not benchmarks. No agent was built five ways and timed. The ranking weighs documented capability against the stated criteria. The one measured result quoted here, Mastra's LongMemEval score, is Mastra's own published benchmark, not an independent test.
• Stars and downloads measure attention, not fit. They are in the table because they are checkable, not because they are decisive.
• This market moves faster than the article. Every number has a date on it for that reason.
• Your constraints beat this ranking. A team with deep LangChain experience should probably use LangGraph regardless of what is written here.

The genuinely useful exercise is to build the same small thing twice, in your language, with your model, and see which one you would rather maintain. We are planning to do exactly that next, with an on-call agent.

For related reading, we have written about running a background job that must not be lost, which is the same durability problem agents face, and about what one merge costs in CI for measuring things rather than guessing.

That is not laziness. Rotation is avoided because nobody can say what will break. The password lives in more places than anyone can list, and the only way to find them all is to change it and see what pages. So it never gets changed, and it keeps working, and it stays in the same places for another two years.

This is about the count. Trace one credential from a laptop to production, count the copies it leaves behind, and you have the number that decides both how expensive rotation is and how bad a leak is.

TL;DR

• References are easy to find. Copies of the value are the problem, and they are in different systems owned by different people.
• Run the inventory before you buy anything. Most teams are surprised by their own answer.
• A secret in git history is leaked even after you delete the file. The only fix is rotation.
• A Kubernetes Secret is base64, not encryption. -o yaml and base64 -d is the whole attack.
• In a leak, revoke first, investigate second. The instinct to understand before acting is the expensive one.
• Rotation is expensive because it is manual and risky. Both go away if the credential expires on its own, which is why short-lived beats stored.
• .env survives because it works offline with no auth dance. Any replacement that loses that will lose to it.

Prerequisites

• A service with credentials in more than one environment
• Shell access to your repo and CI configuration

Start by counting

Before choosing a tool, answer one question: for a single credential, how many places would you have to change?

Not "where is it referenced". References are the easy half and grep finds them. The hard half is copies of the value, which live in systems that do not grep: your CI provider's secret store, a running container's environment, a developer's laptop, a terminal scrollback, an error report.

Here is the reference count from one of our own repositories, a Next.js app with Stripe, Postgres and SES:

You can produce the same table in a few seconds:

# Distinct secret names your CI knows about
grep -rhoE "secrets\.[A-Z_][A-Z0-9_]*" .github/workflows | sort -u

# Distinct environment variables the code expects
grep -rhoE "process\.env\.[A-Z_][A-Z0-9_]*" src/ | sort -u | wc -l

# Every file that mentions one specific secret
grep -rl "DATABASE_URL" --include="*.ts" --include="*.yml" \
  --include="*.yaml" --include="Dockerfile*" . | grep -v node_modules

That app has 48 distinct environment variables across the codebase and 10 secrets configured in CI. Those are small numbers for a small product, and the point is not that they are alarming. The point is that six files is the number grep can see, and it is not the number that matters.

Where the copies actually get made

Follow one database password from a laptop to a running pod.

Four of those six are worth being specific about, because each one fails differently.

Git. Deleting the file in a later commit does nothing. The blob is still reachable, and if it was ever pushed, assume it was cloned. Rewriting history with git filter-repo does not help either, because the fork, the CI cache and somebody's laptop still have the old objects. A secret that reaches a remote is burnt. Rotate it and move on.

The CI secret store. These are write-only and masked in logs, which is good. But masking is a string replacement on output, not a boundary. Any workflow that can read the secret can also transform it, and a transformed secret does not match the mask:

# This defeats log masking. Not a hypothetical: it is how
# a malicious dependency in a build step exfiltrates.
- run: echo "${{ secrets.API_KEY }}" | base64

The lesson is scope. A secret available to every workflow in the repo is available to every dependency those workflows install.

Docker build arguments. ARG values are recorded in image metadata. Anyone who can pull the image can read them:

docker history --no-trunc myimage:latest | grep -i secret

Use BuildKit secret mounts instead, which never enter a layer:

# syntax=docker/dockerfile:1
RUN --mount=type=secret,id=npmtoken \
    NPM_TOKEN=$(cat /run/secrets/npmtoken) npm ci

Kubernetes Secrets. The name oversells it. The value is base64, and base64 is an encoding, not a cipher:

$ kubectl get secret db-creds -o jsonpath='{.data.password}'
c3VwZXJzZWNyZXQtdmFsdWUK

$ echo 'c3VwZXJzZWNyZXQtdmFsdWUK' | base64 -d
supersecret-value

Encryption at rest in etcd is off unless you configure an EncryptionConfiguration. Until then, anyone with read access to the Secret, or to an etcd backup, has the value.

What a leak actually costs

The expensive part of a leak is not the leak. It is the hour after it, when everyone wants to understand what happened before touching anything.

Invert that. Revoke first, investigate second. A revoked credential turns an incident into an outage, and an outage is a much better problem: it is visible, bounded and fixable in minutes. An un-revoked credential is an open door for as long as your investigation takes.

The order that works:

1. Revoke or disable the credential. Not rotate, revoke. Rotation implies a working replacement, and getting one takes time you do not have.
2. Confirm it is dead. Try to use it. An AWS key that still returns a caller identity has not been revoked.
3. Then work out the exposure window and what was reachable with it.
4. Issue the replacement and deploy.
5. Only now, work out how it escaped.

Step 2 catches a common mistake. Deleting an IAM user's access key is immediate; removing a key from your secret store is not, because everything already running still holds the old value in memory.

# Prove the old key is dead, do not assume it
AWS_ACCESS_KEY_ID=OLD AWS_SECRET_ACCESS_KEY=OLD \
  aws sts get-caller-identity
# Expect: InvalidClientTokenId

The exposure window is where your copy count comes back. If the credential was in six places, you have six timelines to reason about and six systems that might still be using it.

Why rotation is expensive, and how to make it cheap

Rotation is avoided because it has two properties nobody wants: it is manual, and it can take production down. Every place holding the old value has to pick up the new one, and if one is missed, it fails at an unpredictable time.

The usual answer is to automate rotation. That helps, but it is treating the symptom. The real fix is to make the credential short-lived, because then rotation is not an event at all. It is just what the system does.

Three rungs, in the order that is worth climbing:

Rung one: stop making new copies. Cheap and immediate. Add secret scanning to pre-commit and CI so a credential cannot reach git in the first place. This does not fix anything existing, but it stops the count growing while you work on the rest.

# Fails the build on a detected secret, and scans history too
gitleaks detect --source . --redact --exit-code 1

Rung two: replace static credentials with identity. Most cloud credentials do not need to exist. If your CI can assume a role via OIDC, there is no key to leak, rotate or inventory:

permissions:
  id-token: write   # lets the runner request an OIDC token
  contents: read

steps:
  - uses: aws-actions/configure-aws-credentials@v4
    with:
      role-to-assume: arn:aws:iam::111122223333:role/ci-deploy
      aws-region: eu-west-1

That removes AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY from your CI store entirely. Every cloud has an equivalent, and it is the single highest-value change on this list, because those two keys are the most damaging thing in most CI configurations.

Rung three: make what remains expire on its own. Some credentials genuinely have to exist, such as a database password. Issue them dynamically with a short lease, so a leaked value is worthless in an hour:

$ vault read database/creds/app-readonly
Key                Value
---                -----
lease_id           database/creds/app-readonly/9zK2...
lease_duration     1h
username           v-approle-app-readonly-x7Fq2mN
password           A1a-8sKd0PqWmZx3

Note what this changes about the copy count. A credential valid for an hour cannot accumulate copies, because the copies stop working. The inventory problem solves itself.

Why .env files refuse to die

Every secrets product has spent a decade trying to kill the .env file, and it is still there. Worth being honest about why, because a replacement that ignores this will lose too.

.env works offline. It needs no login, no network, no token refresh, no VPN. It works on a plane, in a hotel with captive-portal wifi, and at 3am when the identity provider is the thing that is broken. It is one file you can read, edit and delete with tools you already have.

Every centralised alternative trades that away. Now starting your app locally needs an authenticated session with a service that can be down. That is a real cost, and teams route around it by exporting the secrets to a .env file once and forgetting about it, which puts you back where you started with an extra subscription.

The tools that win on developer machines are the ones that keep the ergonomics:

# The secret never lands on disk; it exists for the life of the process
doppler run -- npm run dev
infisical run -- npm run dev
op run --env-file=.env.template -- npm run dev

That shape works because it does not ask anyone to change how they start the app. If your rollout plan involves telling developers to do something more annoying than what they do now, plan for it to fail.

Do these first

In order, because the order matters more than the tool:

1. Count. Pick your most sensitive credential and list every place it exists. Not references, copies. If you cannot finish the list, that is the finding.
2. Scan history. gitleaks detect over the full history. Anything it finds is already leaked and needs rotating, not deleting.
3. Kill the static cloud keys. Move CI to OIDC. This is the biggest single reduction in blast radius available to most teams.
4. Check whether etcd encrypts Secrets if you run Kubernetes, and check whether your backups are plain text.
5. Write down the revoke procedure for your top five credentials, before you need it. One page, per credential, revoke first.
6. Then compare tools, with your copy count as the requirement rather than a feature list.

Build versus buy

Doing this yourself is viable. Cloud-native secret stores are competent, and if you are on one cloud, its own manager plus OIDC covers most of what matters. What you give up is the cross-environment story: developer laptops, CI, and several clouds behaving the same way.

That gap is what the vendors sell. Infisical and Doppler both centre on the run -- shape above, which is the ergonomics problem rather than the storage problem. 1Password comes at it from the human side, which fits teams already using it for passwords. HashiCorp Vault is the heavyweight, and dynamic credentials are its genuinely differentiating feature, at the cost of an operational burden that is real. We have written separately about running Vault properly, and there is a broader comparison of the managed options.

The honest decision rule: if your answer to "how many copies" was small and you are on one cloud, you probably need OIDC and a scanner rather than a product. If the answer was large, or you could not finish counting, the value on offer is the inventory and the consistency, not the encryption. Everything encrypts adequately.

What this does not cover

• Encryption keys and certificates, which have a different lifecycle. Rotating a signing key means thinking about what was signed with the old one.
• Secret zero. Every scheme needs one credential to bootstrap the rest. Cloud instance identity is the usual answer, and it is worth knowing which one you rely on.
• Anything about who should have access. This is about where secrets physically are, which is a separate question from authorisation, and the easier one.

The number to take away is your own copy count. It predicts your rotation cost, it predicts your blast radius, and unlike most security metrics you can measure it this afternoon with grep and an honest hour.

For the surrounding practice, we have written about hardening a CI/CD pipeline and pre-commit hooks that catch problems before they land.

In this guide, you build Receipt Relay, a FastAPI application that sends a transactional receipt through SMTPFast and makes that entire pipeline visible. You start with domain verification and a direct API smoke test, then add safe email rendering, delivery polling, signed webhooks, and tests that never send a real message.

TL;DR

• SMTPFast's Connect to Cloudflare flow creates the DKIM, SPF, DMARC, and MAIL FROM records for you.
• You do not need a normal inbound MX record or an existing mailbox just to send transactional email.
• The SMTPFast dashboard currently asks only for an API-key name. Dashboard-created keys have broad access, so keep them server-side and separate them by environment.
• POST /emails returns a correlation ID, not proof of delivery. Use that ID to retrieve the delivery trace.
• Keep delivery status separate from engagement. A tracking-pixel request is an open signal, not proof that a human read the message.
• Verify webhook HMAC signatures over the raw body before parsing JSON, and deduplicate events before processing them.

Prerequisites

• Python 3.11 or later
• Git
• An SMTPFast account
• A domain you control and access to its DNS configuration
• An inbox you control for the live test
• Basic familiarity with FastAPI and HTTP APIs
• Optional: Docker for the container section

This walkthrough uses a Cloudflare-managed domain because SMTPFast provides a one-click setup for it. Other DNS providers work too; you add the same records manually.

The 200 is only the first hop

Receipt Relay has one narrow job. A user enters a customer name, recipient, order reference, item, amount, and currency. FastAPI validates those fields, renders HTML and plain-text versions of a receipt, and calls SMTPFast. The browser receives the email ID and follows its delivery trace.

There are four boundaries in the flow:

1. Browser to FastAPI. Only receipt fields and an optional demo access code cross this boundary.
2. FastAPI to SMTPFast. The backend adds the API key and submits the email.
3. SMTPFast to the recipient server. The asynchronous delivery work happens here.
4. SMTPFast back to FastAPI. Signed webhook events report lifecycle changes without requiring an open browser.

The SMTPFast email ID connects all four boundaries. Treat it as a correlation key, not an inbox confirmation.

Set up SMTPFast before writing code

Prove the provider works before introducing application code. That gives you a clean line between DNS or account problems and bugs in your FastAPI integration.

1. Add your sending domain

Sign in to SMTPFast, open the domain area, and add the domain you want to send from. You can use a root domain such as example.com, or a subdomain such as mail.example.com if you want transactional mail isolated from other systems.

The exact from address used later must belong to this domain:

receipts@example.com

2. Connect the domain to Cloudflare

When SMTPFast detects Cloudflare nameservers, the domain page displays Connect to Cloudflare:

1. Click Connect to Cloudflare.
2. Review the domain and proposed records in the Cloudflare tab.
3. Approve the change.
4. Return to SMTPFast.
5. Click Verify Now.

Cloudflare creates the records for you. SMTPFast's current setup includes:

• Three DKIM CNAME records for cryptographic signing
• An SPF TXT record authorizing the sending service
• A DMARC TXT record describing how receivers handle authentication failures
• An MX record on a bounce subdomain for MAIL FROM processing
• An SPF TXT record on that bounce subdomain

SMTPFast documents the current one-click flow and each record's purpose in its Domains documentation.

If you do not use Cloudflare, copy the records shown by SMTPFast into your DNS provider exactly as displayed. Do not reuse values from another domain. DKIM hostnames are generated for your SMTPFast domain.

There are two common manual-setup mistakes. First, keep DKIM CNAMEs DNS-only rather than proxying them. Second, publish one SPF record per hostname:

# Wrong: two SPF policies on example.com
example.com  TXT  "v=spf1 include:_spf.google.com ~all"
example.com  TXT  "v=spf1 include:amazonses.com ~all"

# Right: merge both senders into one policy
example.com  TXT  "v=spf1 include:_spf.google.com include:amazonses.com ~all"

3. Wait for verification

DNS changes are often visible quickly, but the underlying sending identity can take a few minutes to finish verifying. If the domain stays pending:

1. Confirm the records exist on the correct domain.
2. Check that all three DKIM CNAMEs are not proxied.
3. Confirm there is only one SPF record on each hostname.
4. Click Verify Now again.
5. Allow more time if SMTPFast says the records are visible but verification is still in progress.

Do not debug application code until the domain is verified. SMTPFast rejects an otherwise valid request when its from address uses an unverified domain.

4. Create the API key

Open the API Keys page and click Create API Key. The current dashboard asks for one value: a descriptive key name.

Use a name that identifies the application and environment, such as receipt-relay-local. Click Create Key, copy the generated value immediately, and store it in a password manager or secret store. SMTPFast only displays the complete key when it is created.

The dashboard does not currently show a scope selector. SMTPFast's Authentication documentation says dashboard-created keys default to all scopes, while keys created through the API can request explicit scopes.

Because the dashboard key has broad access:

• Use a separate key for local, staging, and production.
• Keep it in server-side environment variables.
• Never place it in browser JavaScript, screenshots, Git commits, or container images.
• Revoke it when the environment no longer exists.

5. Run a direct API smoke test

Export the key in your current terminal session, then send to an inbox you control. Replace both email addresses before running the command.

The first response proves that SMTPFast accepted the request. The second shows what happened later. The full response includes status, timestamps, and an events array; see the Emails API reference for the current shape.

Fix provider setup errors here, before proceeding:

Build Receipt Relay with FastAPI

With the direct request working, put a small application boundary around it. The browser never receives the SMTPFast key and never calls SMTPFast directly.

The complete application is available as a reusable GitHub template:

6. Install and configure the application

Click Use this template on GitHub to create your own repository, or clone the reference application directly:

git clone https://github.com/The-DevOps-Daily/smtpfast-receipt-relay.git
cd smtpfast-receipt-relay

Create a virtual environment and install the project with its development tools:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -e ".[dev]"

Create .env from the included template:

cp .env.example .env
chmod 600 .env

Add the key and verified sender:

SMTPFAST_API_KEY=replace-with-your-smtpfast-api-key
SMTPFAST_FROM_EMAIL=receipts@your-verified-domain.com
SMTPFAST_BASE_URL=https://smtpfa.st/api/v1
SMTPFAST_TIMEOUT_SECONDS=20

# Added after creating the public webhook
SMTPFAST_WEBHOOK_SECRET=

# Optional shared code for a short-lived demo
APP_ACCESS_TOKEN=

Start FastAPI with the environment file:

uvicorn app.main:app --reload --port 8080 --env-file .env

Open http://localhost:8080. The page displays the configured sender but never returns either secret.

7. Validate before consuming quota

An email send is an external side effect. It consumes quota and can reach a real person, so reject malformed values before calling the provider.

class ReceiptRequest(BaseModel):
    model_config = ConfigDict(extra="forbid", str_strip_whitespace=True)

    customer_name: str = Field(min_length=1, max_length=100)
    recipient: EmailStr
    order_id: str = Field(
        min_length=3,
        max_length=64,
        pattern=r"^[A-Za-z0-9][A-Za-z0-9._-]+$",
    )
    product_name: str = Field(min_length=2, max_length=120)
    amount_cents: int = Field(ge=50, le=100_000_000)
    currency: Literal["USD", "EUR", "GBP"] = "USD"

The model makes several deliberate decisions:

• EmailStr rejects malformed recipients.
• The order reference uses a small, header-friendly character set.
• Money crosses the API as integer cents rather than floating point.
• Currency is an enum rather than arbitrary text.
• extra="forbid" makes misspelled fields fail explicitly.

In a real checkout, accept an order ID and load the authoritative item and total from a database. Do not let a browser decide how much was paid.

8. Render safe HTML and a text alternative

Transactional messages need a useful plain-text body as well as HTML. Escape values before inserting them into the HTML context:

customer = html.escape(receipt.customer_name)
product = html.escape(receipt.product_name)
order_id = html.escape(receipt.order_id)
total = _format_amount(receipt.amount_cents, receipt.currency)

Validation constrains shape and length; it does not make a string safe for HTML. A customer named <script>alert(1)</script> must appear as text, not markup.

Build the SMTPFast payload with both bodies and two correlation values:

return {
    "from": self._settings.smtpfast_from_email,
    "to": [str(receipt.recipient)],
    "subject": f"Receipt for order {receipt.order_id}",
    "html": html_body,
    "text": text_body,
    "tags": [
        {"name": "category", "value": "receipt"},
        {"name": "order_id", "value": receipt.order_id},
    ],
    "headers": {"X-Entity-Ref-ID": receipt.order_id},
}

Tags help filter provider records. X-Entity-Ref-ID carries your application reference with the message. Neither replaces a database relationship, but both make one send easier to diagnose.

9. Call SMTPFast from the server

The client submits the payload to /emails, validates the returned ID, and records request latency:

async def send_receipt(self, receipt: ReceiptRequest) -> ReceiptAccepted:
    self._require_send_configuration()
    started_at = time.perf_counter()
    response = await self._request(
        "POST",
        "/emails",
        json=self._build_receipt_payload(receipt),
    )
    latency_ms = round((time.perf_counter() - started_at) * 1_000)

    data = response.json()
    email_id = data["id"]
    return ReceiptAccepted(
        email_id=email_id,
        status=str(data.get("status") or "queued"),
        latency_ms=latency_ms,
    )

The shared helper adds authentication only on the backend:

response = await client.request(
    method,
    f"{self._settings.smtpfast_base_url}{path}",
    headers={
        "Authorization": f"Bearer {self._settings.smtpfast_api_key}",
        "Content-Type": "application/json",
    },
    json=json,
)

The fallback queued status is intentionally conservative. The application has an ID and knows the request was accepted; it does not invent a later delivery event.

10. Keep a narrow browser-facing API

The browser submits to a FastAPI route rather than the provider:

@application.post("/api/receipts", response_model=ReceiptAccepted)
async def send_receipt(
    receipt: ReceiptRequest,
    x_app_access_token: str | None = Header(default=None),
) -> ReceiptAccepted:
    _require_app_access(runtime_settings, x_app_access_token)
    return await application.state.smtpfast_client.send_receipt(receipt)

The complete handler maps configuration, authentication, rate-limit, and upstream failures into safe application errors. It never returns SMTPFast's raw error body, which may contain internal identifiers or request data.

Receipt Relay also exposes /health without calling SMTPFast. A load balancer should be able to check the process without sending an email or making the provider a dependency of every probe.

11. Retrieve and display the lifecycle

After a send, the browser receives the email ID and calls GET /api/emails/{email_id}. The backend retrieves and validates the SMTPFast record:

async def get_email(self, email_id: str) -> EmailTrace:
    response = await self._request(
        "GET",
        f"/emails/{quote(email_id, safe='')}",
    )
    data = response.json()
    events = [
        EmailEvent.model_validate({**event, "source": "api"})
        for event in data.get("events", [])
    ]
    return EmailTrace.model_validate({**data, "events": events})

The browser polls briefly, renders values with textContent, stops after a bounded number of attempts, and leaves a manual refresh button. A typical sequence is:

queued -> sending -> sent -> delivered

• Queued means SMTPFast accepted the work.
• Sent means the sending provider accepted the message for delivery.
• Delivered means the recipient mail server accepted it.
• Bounced or failed means delivery did not complete.

Even delivered does not guarantee primary-inbox placement. The receiving system can still route the message to spam.

12. Keep delivery separate from engagement

An open or click does not make a message "more delivered," and it should not replace the terminal delivery outcome.

SMTPFast records an open when its tracking pixel is requested. Image proxies, privacy features, and security scanners can request that pixel without a person reading the email. During the live Receipt Relay test, an open signal arrived about one second after delivery even though nobody had opened the inbox.

Receipt Relay therefore keeps Delivered as the status, shows the later event separately, and labels it Open signal rather than Opened.

Receive and verify SMTPFast webhooks

Polling works for an interactive demo, but an application should not need an open browser to learn about a bounce. Webhooks reverse the flow: SMTPFast calls your application when an event occurs.

13. Expose a public HTTPS endpoint

Deploy Receipt Relay to your preferred platform or expose it through a trusted development tunnel. SMTPFast must be able to reach this endpoint:

https://your-app.example/webhooks/smtpfast

http://localhost:8080 exists only on your computer from SMTPFast's perspective.

14. Create the webhook

Create a standard-format webhook in SMTPFast with the public URL. Subscribe only to events your application uses:

[
  "email.sent",
  "email.delivered",
  "email.delivery_delayed",
  "email.bounced",
  "email.failed",
  "email.suppressed",
  "email.opened",
  "email.clicked"
]

SMTPFast returns a signing secret when the webhook is created. It is not the API key. Store it separately as SMTPFAST_WEBHOOK_SECRET, then restart or redeploy the application. The webhook page's test action reports the response code and response time. The current event list and retry policy live in the Webhooks documentation.

15. Verify the signature before parsing JSON

Standard webhook requests include X-SMTPfast-Signature, an HMAC-SHA256 digest of the raw request body using the webhook signing secret.

The word raw matters. Parse and reserialize JSON and you can change whitespace, ordering, or escaping, producing a different digest.

Read and bound the raw body first:

body = await request.body()
if len(body) > MAX_WEBHOOK_BYTES:
    raise HTTPException(status_code=413, detail="Webhook payload is too large.")
if not _valid_webhook_signature(body, x_smtpfast_signature, secret):
    raise HTTPException(status_code=401, detail="Invalid webhook signature.")

Compare the expected and received values in constant time:

def _valid_webhook_signature(
    body: bytes,
    signature: str | None,
    secret: str,
) -> bool:
    if not signature:
        return False
    expected = hmac.new(secret.encode(), body, hashlib.sha256).hexdigest()
    return secrets.compare_digest(signature, expected)

Only after signature verification do you parse and validate:

payload = json.loads(body)
event = SMTPFastWebhookEvent.model_validate(payload)
await application.state.trace_store.add(event)

Signature verification proves that someone with the webhook secret produced the payload. Pydantic validation separately proves that the payload has the shape your application expects. You need both.

16. Make retries safe

SMTPFast retries when an endpoint fails or times out. Receiving the same event more than once is expected behavior.

Receipt Relay uses a bounded in-memory OrderedDict keyed by SMTPFast event ID. That deduplicates retries during one process lifetime and keeps the demo dependency-free. Production handling needs a durable sequence:

1. Verify the signature.
2. Validate the payload.
3. Insert the event with a unique constraint on event ID.
4. Commit the transaction.
5. Return a successful response.
6. Process slow downstream work asynchronously.

Do not acknowledge an event you have not recorded safely.

Test the integration end to end

The automated suite should not consume quota, depend on DNS, or place messages in an inbox.

17. Mock SMTPFast in tests

HTTPX's MockTransport lets a test inspect the outgoing request and return a representative provider response:

def handler(request: httpx.Request) -> httpx.Response:
    assert request.method == "POST"
    assert request.url == "https://smtpfa.st/api/v1/emails"
    assert request.headers["Authorization"] == "Bearer sf_live_test"

    payload = json.loads(request.content)
    assert payload["headers"]["X-Entity-Ref-ID"] == "ORD-2048"
    assert "Ana <script>alert(1)</script>" in payload["html"]
    return httpx.Response(200, json={"id": "email_abc123"})

The escaped-name assertion tests the important HTML boundary, not just the happy path.

The webhook test signs the exact bytes it submits:

body = json.dumps(event, separators=(",", ":")).encode()
signature = hmac.new(b"whsec_test", body, hashlib.sha256).hexdigest()

response = client.post(
    "/webhooks/smtpfast",
    content=body,
    headers={"X-SMTPfast-Signature": signature},
)

Add a negative test with a bad signature. One test proves correctly signed bytes pass; the other stops verification from accidentally becoming optional.

Run the checks:

ruff check .
ruff format --check .
pytest

No real SMTPFast key is required.

18. Send one real receipt

Return to http://localhost:8080, load the example, enter an inbox you control, and submit once.

Verify the complete path:

1. Receipt Relay displays an SMTPFast email ID.
2. The trace advances from queued through sending and sent.
3. The recipient server accepts the message or returns a failure.
4. The email contains readable HTML and a useful text alternative.
5. The sender uses the verified domain.
6. Later engagement appears separately from delivery.

Check spam. A technically successful first send from a new domain can still be filtered; authentication is a foundation for deliverability, not a guarantee of inbox placement.

Run the same app in Docker

The project includes a non-root Docker image. Run the container locally with the same .env file:

docker build -t smtpfast-receipt-relay .
docker run --rm \
  --publish 8080:8080 \
  --env-file .env \
  smtpfast-receipt-relay

Use the non-sending health endpoint:

curl http://localhost:8080/health

Expected output:

{"status":"ok"}

You can deploy the same image to any container platform that accepts environment variables and exposes a public HTTPS URL. Once that URL exists, create the SMTPFast webhook, store its signing secret in the platform's secret manager, and restart the application.

Production checklist

Receipt Relay is production-minded, not production-complete. Before adapting it to a real product:

• Load trusted order data. Accept an order ID and render values from your database rather than trusting browser-submitted totals.
• Add idempotency. A double-click, worker retry, or network timeout must not send a duplicate receipt.
• Persist provider IDs. Store the SMTPFast email ID with the business record that caused the send.
• Persist webhook events. Use durable storage and a unique event-ID constraint before acknowledging delivery.
• Use real authentication. Replace the shared demo code with user- and tenant-aware authorization.
• Apply quotas. Add per-user, per-tenant, and global send limits.
• Protect recipient data. Avoid logging full addresses and bodies by default; define retention and deletion behavior.
• Enable tracking deliberately. Open and click events affect privacy and remain imperfect signals.
• Version templates. Add localization, rendering checks, and snapshot tests.
• Monitor the pipeline. Track API failures, time to delivery, bounce categories, webhook retries, and consumer lag.

What to take away

The most useful value returned by an email send is not "success." It is the ID that lets the rest of your application correlate what happens next.

Receipt Relay validates a real side effect before sending it, keeps SMTPFast credentials on the server, renders HTML and text bodies, follows each message's delivery trace, and verifies webhook events over the raw request body. The browser makes the lifecycle visible while the backend owns the provider and security boundaries.

The same pattern applies to password resets, invoices, deployment alerts, and account notifications: send once, keep the correlation ID, and design for everything that happens after the 200.

📌 Handpicked by DevOps Daily - Your weekly dose of curated DevOps news and updates!

⚓ Kubernetes

📄 Does Kubernetes DRA Replace HAMi?

Projects that want to share a GPU on Kubernetes have to work around an API instead of with it. The device plugin interface could count devices, and that was the whole vocabulary: nvidia.com/gpu: 1. It

📅 Aug 7, 2026 • 📰 CNCF Blog

🔗 Read more

📄 Shadow AI in CI/CD: Threat-modeling the path from developer laptop to Kubernetes

Artificial intelligence is becoming part of daily software delivery, often before it becomes part of the security architecture. That gap has a name: Shadow AI. It is any AI tool, model, agent, extensi

📅 Aug 7, 2026 • 📰 CNCF Blog

🔗 Read more

📄 The migration catalyst: turning virtualization disruption into application innovation

Starting nearly three decades ago, the cost efficiencies of server virtualization drove the first waves of IT transformation, wringing new efficiency out of the x86 servers that had already shaped the

📅 Aug 6, 2026 • 📰 OpenShift Blog

🔗 Read more

📄 GitLab Secrets Manager adds ESO, Terraform, API support

Today, you might maintain separate secret stores for CI/CD, Kubernetes, and Terraform. However, that leaves multiple tools to manage, access models to keep in sync, and audit trails to correlate when

📅 Aug 6, 2026 • 📰 GitLab Blog

🔗 Read more

📄 OpenCost 1.121.0: First-of-a-kind Kubernetes inference cost tracking

Your GPU bill is rising. Your models are serving billions of tokens. Yet one question remains unanswered: what does each token actually cost? This is not a hypothetical problem. Platform teams today o

📅 Aug 5, 2026 • 📰 CNCF Blog

🔗 Read more

📄 Use EVPN in Red Hat OpenShift 4.22 to integrate production networks across Kubernetes cluster boundaries

Red Hat OpenShift Networking is making it easier for you to seamlessly and directly integrate your Kubernetes platforms with the data center networks you already operate by adopting the same standards

📅 Aug 5, 2026 • 📰 OpenShift Blog

🔗 Read more

📄 Run GPU batch inference on Amazon ECS Managed Instances with scale to zero

Deploy a single CloudFormation stack that builds a GPU batch inference pipeline on Amazon ECS Managed Instances. It uses Amazon SQS for job buffering and Application Auto Scaling to scale to zero when

📅 Aug 3, 2026 • 📰 AWS Containers Blog

🔗 Read more

📄 Gateway API v1.6: TCPRoute and UDPRoute Graduate to Standard

The Kubernetes SIG Network community is thrilled to share the release of Gateway API v1.6.0, which was released on June 30th of this year! Gateway API has become the standard for modern, role-oriented

📅 Aug 3, 2026 • 📰 Kubernetes Blog

🔗 Read more

☁️ Cloud Native

📄 Managing virtual machines on Red Hat OpenShift with Service Mesh

Managing virtualized workloads alongside containerized applications remains a persistent challenge for IT operations, often creating siloed management environments. At Red Hat Summit 2026, I had the o

📅 Aug 7, 2026 • 📰 OpenShift Blog

🔗 Read more

📄 LitmusChaos Q1-Q2 2026 update: community, contributions, and project progress

About LitmusChaos LitmusChaos is an open source chaos engineering platform that helps teams identify weaknesses and potential outages in their infrastructure by running controlled chaos experiments. B

📅 Aug 6, 2026 • 📰 CNCF Blog

🔗 Read more

📄 Extending Amazon ECS Express Mode to Build an Optimal Container Environment

Amazon ECS Express Mode gives you load balancing, scaling, logging, and networking out of the box. Learn how to extend an Express Mode service beyond its defaults with three hands-on examples: turning

📅 Aug 4, 2026 • 📰 AWS Containers Blog

🔗 Read more

📄 Empty sandboxes break developer experience

Learn how Docker Sandbox kits turn empty sandboxes into productive development environments with repeatable tooling, credentials, and configuration.

📅 Aug 3, 2026 • 📰 Docker Blog

🔗 Read more

📄 Docker AI Governance: Audit Logs, Now Where Your Security Team Already Works

Now in Docker AI Governance: a single searchable record of every policy decision your agents trigger, streamed to the SIEM your security team already runs, so you can show what your agents did and wha

📅 Aug 3, 2026 • 📰 Docker Blog

🔗 Read more

🔄 CI/CD

📄 Automate Incident Intake with AI SRE Runbooks

Automate incident intake with Harness AI SRE runbooks: auto-create tickets, open Slack channels, start Zoom bridges, and cut response time to seconds. | Blog

📅 Aug 10, 2026 • 📰 Harness Blog

🔗 Read more

📄 A guide to slash commands in the GitHub Copilot app

Go beyond chat in the GitHub Copilot app with these slash commands. They'll help you plan, collaborate, automate, and customize your dev workflow. The post A guide to slash commands in the GitHub Copi

📅 Aug 6, 2026 • 📰 GitHub Blog

🔗 Read more

📄 Confidential AI for GitLab Self-Hosted

Your developers want AI coding agents. Your source code is regulated IP that can't be sent to a third-party AI service, and your compliance team has said so in writing. The usual escape hatch, standin

📅 Aug 6, 2026 • 📰 GitLab Blog

🔗 Read more

📄 Infrastructure Control Plane | Day 2 Operations & Drift

Learn why infrastructure breaks after deployment and how control planes enforce governance, detect drift, and automate remediation across Terraform, Ansible, and CI/CD. | Blog

📅 Aug 6, 2026 • 📰 Harness Blog

🔗 Read more

📄 New bazel.build websites incoming!

We're happy to announce the launch of the new bazel.build documentation site and the new web UI for the Bazel Central Registry! New documentation site Last year, Alan Mond wrote a viral blog post that

📅 Aug 5, 2026 • 📰 Bazel Blog

🔗 Read more

📄 How the GitHub legal team used Copilot CLI to streamline their workflows

Learn how to build tools to simplify how you work—without writing a single line of code. The post How the GitHub legal team used Copilot CLI to streamline their workflows appeared first on The GitHub

📅 Aug 4, 2026 • 📰 GitHub Blog

🔗 Read more

📄 Turn one giant AI-generated pull request to a reviewable stack

Instead of one huge, un-reviewable pull request, teach coding agents to decompose work into a clean, ordered stack with GitHub stacked pull requests. The post Turn one giant AI-generated pull request

📅 Aug 4, 2026 • 📰 GitHub Blog

🔗 Read more

📄 Agent Optimization: Define what better means, and let AgentControl find it

Agent Optimization, now in beta in AgentControl, automatically searches for a better agent configuration against criteria you define.

📅 Aug 4, 2026 • 📰 LaunchDarkly Blog

🔗 Read more

📄 Stories from the Factory Floor: Building a software factory on our scariest code

We pointed coding agents at our oldest, most business-critical frontend. Here’s what it taught me about what a healthy AI software factory actually looks like.

📅 Aug 3, 2026 • 📰 LaunchDarkly Blog

🔗 Read more

📄 Blog: Selective drift correction with ignore rules

We are excited to introduce drift ignore rules for Flux Kustomizations, a long-requested capability that lets you tell Flux to leave specific fields alone during drift detection and correction, while

📅 Aug 3, 2026 • 📰 Flux CD Blog

🔗 Read more

🏗️ IaC

📄 Terraform Scalability: When IaC Outgrows Your Setup

Terraform scalability issues slow teams down. Learn how to overcome IaC bottlenecks with better management. See how Harness helps. | Blog

📅 Aug 10, 2026 • 📰 Harness Blog

🔗 Read more

📄 Autobase 2.10 released

Autobase 2.10 expands day-to-day PostgreSQL operations with new cluster management capabilities. Administrators can now perform common cluster actions directly from the Console UI, configure advanced

📅 Aug 7, 2026 • 📰 PostgreSQL News

🔗 Read more

📄 Accelerate CloudFormation development with the IaC MCP Server

Walk through a complete CloudFormation development cycle - authoring, validation, deployment, and troubleshooting - without leaving your AI assistant, using the AWS IaC MCP Server.

📅 Aug 4, 2026 • 📰 AWS DevOps Blog

🔗 Read more

📄 YOLO Mode Is the Right Default. Your Laptop Is the Wrong Place for It.

Claude Code calls the flag --dangerously-skip-permissions, and the community long ago renamed it YOLO mode. It lets your coding agent run any command it wants without ever asking for permission. Every

📅 Aug 4, 2026 • 📰 Pulumi Blog

🔗 Read more

📄 Emulating Terraform on Pulumi's Engine

The core promise of Pulumi’s HCL support is that you can bring your existing Terraform configuration and modules, and pulumi will run them. If it works in OpenTofu and doesn’t work in Pulumi, we would

📅 Aug 4, 2026 • 📰 Pulumi Blog

🔗 Read more

📄 Bring Your Terraform Estate Into the Agentic Era

At Pulumi, we are building the platform for agentic infrastructure. Pulumi Cloud provides the guardrails and enterprise readiness needed to safely move fast in this new era. While we are seeing extrao

📅 Aug 4, 2026 • 📰 Pulumi Blog

🔗 Read more

📄 A guided tour of Terraform state, hosted modules, and HCL in Pulumi

Today’s big release contains a whole new set of features designed for seamless interoperability with the Terraform and OpenTofu ecosystems, and there’s a lot there — so much that it can be tough to ge

📅 Aug 4, 2026 • 📰 Pulumi Blog

🔗 Read more

📊 Observability

📄 Unifying Workers AI and AI Gateway into a single AI control plane

Cloudflare is unifying AI Gateway and Workers AI into a single control plane, giving developers observability, billing, and dynamic routing across both managed GPUs and external providers. Learn how u

📅 Aug 7, 2026 • 📰 Cloudflare Blog

🔗 Read more

📄 How to Choose Digital Experience Monitoring Tools

Discover how digital experience monitoring tools help you understand user issues beyond APM, enabling faster, clearer insights for better software performance.

📅 Aug 7, 2026 • 📰 New Relic Blog

🔗 Read more

📄 Scaling Autonomous Operations with AWS DevOps Agent and ServiceNow

This post is co-written with Govind Menon, Head of MCP Product at ServiceNow. Introduction Enterprise teams managing applications on AWS often rely on ServiceNow as their IT service management (ITSM)

📅 Aug 6, 2026 • 📰 AWS DevOps Blog

🔗 Read more

📄 Podcast recap: Observability won’t save your agents

On a recent episode of the MonkCast, Marek Poliks spoke with James Governor about why governing agents from the outside leaves teams perpetually one step behind.

📅 Aug 6, 2026 • 📰 LaunchDarkly Blog

🔗 Read more

📄 How we built an automated debugging workflow at Sentry

How Sentry uses Seer autofix and Claude routines to build an automated debugging workflow that detects, fixes, and routes code issues automatically.

📅 Aug 6, 2026 • 📰 Sentry Blog

🔗 Read more

📄 Curing alert fatigue: How embedded AI is redefining Red Hat OpenShift cluster troubleshooting

Between virtual machines, microservices, and AI pipelines, hybrid clouds can be incredibly complex and can bring an unwelcome partner: alert fatigue. SREs and IT OPs teams face a constant flood of dis

📅 Aug 6, 2026 • 📰 OpenShift Blog

🔗 Read more

📄 Under the hood: how Amazon EKS Auto Mode detects, repairs, and diagnoses node failures

On Amazon EKS Auto Mode, node failures are detected, drained, and replaced automatically before anyone reaches for a laptop. This post shows how the Node Monitoring Agent and Karpenter form a detect-a

📅 Aug 5, 2026 • 📰 AWS Containers Blog

🔗 Read more

📄 Session Replay Tools: A Technical Buyer’s Guide and Comparison

Discover how to evaluate session replay tools for engineering teams, ensuring they meet technical needs for incident response and observability.

📅 Aug 5, 2026 • 📰 New Relic Blog

🔗 Read more

📄 Your OTel spans, our errors: A Sentry love story in one trace

The OtlpIntegration bridges OTel traces and Sentry errors. Keep your OTel setup, add Sentry for errors, and see both in one trace waterfall.

📅 Aug 5, 2026 • 📰 Sentry Blog

🔗 Read more

🔐 Security

📄 Threats Making WAVs - Incident Response to a Cryptomining Attack

Guardicore security researchers describe and uncover a full analysis of a cryptomining attack, which hid a cryptominer inside WAV files. The report includes the full attack vectors, from detection, in

📅 Aug 10, 2026 • 📰 Linode Blog

🔗 Read more

📄 How Google Cloud detects, contains, and protects against emerging threats

At Google Cloud, securing your data and business systems is our foundational commitment. We empower our customers with the tools, governance, and infrastructure needed to securely deploy workloads and

📅 Aug 7, 2026 • 📰 Google Cloud Blog

🔗 Read more

📄 CVE-2026-63077: Additional Guidance Following Reports of Active Exploitation

This post is a follow-up to our July 27, 2026, announcement about CVE-2026-63077. Summary What has changed since our initial announcement Since our initial announcement on July 27, 2026, we have recei

📅 Aug 7, 2026 • 📰 JetBrains Blog

🔗 Read more

📄 Friday Five — August 7, 2026

Red Hat Recognized as a Leader for Third Consecutive Year in 2026 Gartner® Magic Quadrant™ for Cloud-Native Application PlatformsRed Hat OpenShift is recognized as a Leader in the 2026 Magic Quadrant

📅 Aug 7, 2026 • 📰 Red Hat Blog

🔗 Read more

📄 Proactive patch management & compliance: Hardening the hybrid Azure fleet at scale

Welcome back to SUSE Solutions on Azure: The Technical Series. Bridging the Gap Between Linux Freedom and Azure Scale Enterprise Linux on Azure requires a careful balance between open source flexibili

📅 Aug 6, 2026 • 📰 SUSE Blog

🔗 Read more

📄 Open Source Summit + Embedded Linux Conference Europe 2026 Schedule Champions Open Source Innovation and Marks 35 Years of Linux

Industry leaders gather to advance the open source infrastructure powering embedded systems, cloud orchestration, AI security, safety-critical applications…

📅 Aug 5, 2026 • 📰 KubeCon Updates

🔗 Read more

📄 Governance Is a Developer Experience Problem

Learn why AI governance is about more than security. Discover how trust, clear boundaries, and developer experience enable AI adoption at scale.

📅 Aug 5, 2026 • 📰 Docker Blog

🔗 Read more

📄 New Relic SecurityRX - Security for Operational Reliability

Treat security as a reliability problem. New UI experience (with the homepage), automation capabilities (with Jira), and the agent public preview for a complete remediation workflow.

📅 Aug 5, 2026 • 📰 New Relic Blog

🔗 Read more

📄 Continuous Offensive Security & AI Pentesting: 20 FAQs

Get answers to 20 common questions about continuous offensive security, AI penetration testing, DAST, and AI red teaming.

📅 Aug 5, 2026 • 📰 Snyk Blog

🔗 Read more

📄 The Software Supply Chain Is Under Siege. Devs Are Still the First Line of Defense

77% of organizations experienced a software supply chain incident in the past year. Explore Omdia's latest research on top risks, security gaps, and why developers are your first line of defense.

📅 Aug 4, 2026 • 📰 Docker Blog

🔗 Read more

📄 Evo Continuous Offensive Security Is Here Pentesting Grade Coverage For The 350 Days A Year You Aren't Testing

Snyk Evo Continuous Offensive Security brings autonomous, AI-powered pentesting to the 350 days between traditional tests, uncovering exploitable flaws attackers can find first.

📅 Aug 4, 2026 • 📰 Snyk Blog

🔗 Read more

📄 AI Model Risk Intelligence Know Which Models You Can Trust Before You Deploy

AI model risk depends on how a model is deployed. Learn how Evo combines adversarial testing, attack impact, and deployment context to help teams compare models and enforce policy.

📅 Aug 4, 2026 • 📰 Snyk Blog

🔗 Read more

💾 Databases

📄 The Complete Agent State Stack: Memory, Files, and Serverless Database Persistence for AI Apps

A serverless database is a fully-managed database that automatically scales compute and storage with demand, requires no server provisioning or capacity planning, and bills only for actual usage, incl

📅 Aug 7, 2026 • 📰 TiDB Blog

🔗 Read more

📄 Postgres Summit US 2026 Schedule is now live!

Hi all, The talk schedule for Postgres Summit US 2026 is now published. Browse it here: Talk Schedule The summit runs September 30 through October 2, 2026 at Convene, 555 Broadway, New York, NY, organ

📅 Aug 7, 2026 • 📰 PostgreSQL News

🔗 Read more

📄 Why Attend TiDB SCaiLE 2026: Same Complexity, Different Clock Speeds

A single user action in an agentic application no longer maps to a single database query. It spawns agent instances that branch context in milliseconds, hold memory across sessions, and provision thei

📅 Aug 6, 2026 • 📰 TiDB Blog

🔗 Read more

📄 How we took malware advisories beyond npm

GitHub malware advisories no longer stop at npm. Here's how we wired OpenSSF's malicious-packages data into the Advisory Database, and why we built the pipeline paranoid. The post How we took malware

📅 Aug 6, 2026 • 📰 GitHub Blog

🔗 Read more

📄 Refactoring a SQL Table at Scale: Lessons from Harness CI

How Harness refactored a flat SQL table into a normalized schema, cutting storage per row from 400 bytes to 28 bytes and making API latency constant at any scale. | Blog

📅 Aug 6, 2026 • 📰 Harness Blog

🔗 Read more

📄 How Redis brings persistent memory to Snowflake Cortex Agents

AI agents can reason and act, but without memory, every interaction starts from zero. Intelligent short-term memory and persistent context across conversations are what turns a capable model into a tr

📅 Aug 6, 2026 • 📰 Redis Blog

🔗 Read more

📄 Top vector database alternatives for RAG pipelines

You're building an AI app: maybe a RAG system, an agent with memory, or a chatbot with semantic caching. You need vector search, and you're weighing your options. One is a unified real-time platform l

📅 Aug 5, 2026 • 📰 Redis Blog

🔗 Read more

📄 Migrating Real-Time Data into TiDB with Debezium CDC

Moving data into a new database is rarely a one-shot copy. Migrating off a legacy system, adopting a distributed SQL database, carrying out a heterogeneous database migration, or standing up an analyt

📅 Aug 4, 2026 • 📰 TiDB Blog

🔗 Read more

📄 DDIA 2nd Edition Excerpt: On Scalability

Martin Kleppmann and Chris Riccomini's scalability considerations for designing data-intensive applications -- from the second edition of the Designing Data-Intensive Applications book

📅 Aug 4, 2026 • 📰 ScyllaDB Blog

🔗 Read more

📄 Your Meko Questions, Answered

Interest in Meko has been tremendous, with user questions coming in thick and fast via Discord, LinkedIn, and at in-person events. In his recent AMA session, Yugabyte co-founder Karthik Ranganathan an

📅 Aug 4, 2026 • 📰 Yugabyte Blog

🔗 Read more

📄 Announcing E-Maj 5.0.0.

We are very glad to announce the E-Maj 5.0.0 version. Among improvements, this major version: Allows non-superuser roles to install and use E-Maj in a database, the usable features depending on the pr

📅 Aug 4, 2026 • 📰 PostgreSQL News

🔗 Read more

📄 pgBackRest 2.59.0 Released

July 30, 2026: The pgBackRest community is pleased to announce the release of pgBackRest 2.59.0, the latest version of the reliable, easy-to-use backup and restore solution that can seamlessly scale u

📅 Aug 4, 2026 • 📰 PostgreSQL News

🔗 Read more

🌐 Platforms

📄 Keep Your Tech Flame Alive: Trailblazer Rachel Bayley

In this Akamai FLAME Trailblazer blog post, Rachel Bayley encourages women to step into the unknown and to be their authentic selves.

📅 Aug 10, 2026 • 📰 Linode Blog

🔗 Read more

📄 The Oracle of Delphi Will Steal Your Credentials

Our deception technology is able to reroute attackers into honeypots, where they believe that they found their real target. The attacks brute forced passwords for RDP credentials to connect to the vic

📅 Aug 10, 2026 • 📰 Linode Blog

🔗 Read more

📄 The Nansh0u Campaign – Hackers Arsenal Grows Stronger

In the beginning of April, three attacks detected in the Guardicore Global Sensor Network (GGSN) caught our attention. All three had source IP addresses originating in South-Africa and hosted by Volum

📅 Aug 10, 2026 • 📰 Linode Blog

🔗 Read more

📄 Platform Engineering ROI: What it costs to build your own platform

What it actually costs to build your own internal developer platform over five years, and why most “we’ll just build The post Platform Engineering ROI: What it costs to build your own platform appeare

📅 Aug 9, 2026 • 📰 The New Stack

🔗 Read more

📄 Public Cloud Toolchains in SUSE Linux Enterprise 16: Evolution and Transparent Containers

The release of the SUSE Linux Enterprise (SLE) 16 distributions has long come and gone and the development cycle for SLE 16.1 is well on the way and will culminate in the SLE 16.1 release later this y

📅 Aug 8, 2026 • 📰 SUSE Blog

🔗 Read more

📄 The login screen is where sovereignty gets real

Everyone points at the cloud. Almost nobody points at the front door. Ask most executives where their sovereignty risk sits and they point at the cloud, the data, the AI models. Fair enough, those are

📅 Aug 8, 2026 • 📰 SUSE Blog

🔗 Read more

📄 Amazon EC2 R8i and R8i-Flex instances are now available in Europe (Milan) region

Starting today, Amazon Elastic Compute Cloud (Amazon EC2) R8i and R8i-flex instances are available in the Europe (Milan) region. These instances are powered by custom Intel Xeon 6 processors, availabl

📅 Aug 7, 2026 • 📰 CloudFormation Updates

🔗 Read more

📄 Amazon Timestream for InfluxDB now supports backup and restore

Amazon Timestream for InfluxDB now lets you create and manage your own backups and restore your data on demand. You can trigger one-time, on-demand backups, schedule automated recurring backups at the

📅 Aug 7, 2026 • 📰 CloudFormation Updates

🔗 Read more

📄 Amazon Cognito now available as a skill in the Agent Toolkit for AWS

Amazon Cognito is now available as a core skill (aws-auth) in the Agent Toolkit for AWS. AI coding agents using the toolkit can now set up, configure, secure, and troubleshoot Amazon Cognito using bes

📅 Aug 7, 2026 • 📰 CloudFormation Updates

🔗 Read more

📄 Zero-code, low-cost data ingestion: New BigQuery DTS capabilities

In a fast-paced digital economy, data is your most critical engine. Yet, many enterprises find themselves trapped in a costly paradox, spending over 100 hours a week building and fixing fragile, in-ho

📅 Aug 7, 2026 • 📰 Google Cloud Blog

🔗 Read more

📄 Unifying Structured and Unstructured Data Insights with BQ Search Innovations

Modern enterprises possess a vast amount of unstructured data, yet they frequently encounter significant challenges in managing and extracting value from it. Historically, unlocking the insights hidde

📅 Aug 7, 2026 • 📰 Google Cloud Blog

🔗 Read more

📄 GOL! How TelevisaUnivision streamed the FIFA World Cup to millions with Google Cloud

Live sports broadcasting represents the ultimate stress test for digital media infrastructure, where operational success or failure is measured in milliseconds and observed live by millions of viewers

📅 Aug 7, 2026 • 📰 Google Cloud Blog

🔗 Read more

📰 Misc

📄 Visual Studio Code 1.133 (Insiders)

Learn what's new in Visual Studio Code 1.133 (Insiders) Read the full article

📅 Aug 11, 2026 • 📰 VS Code Blog

🔗 Read more

📄 Coding agents can be evaluated. We just have to evaluate the work.

I recently argued with a software factory provider, whose position was that coding agents cannot be evaluated. Their reasoning was The post Coding agents can be evaluated. We just have to evaluate the

📅 Aug 9, 2026 • 📰 The New Stack

🔗 Read more

📄 AI coding got faster. Why didn’t engineering?

AI is great at making individuals faster, but the surrounding systems are then slowing everything right back down. This result The post AI coding got faster. Why didn’t engineering? appeared first on

📅 Aug 9, 2026 • 📰 The New Stack

🔗 Read more

📄 AI adoption isn’t the same as AI usage

Every engineering org I’ve talked to this year has some version of the same chart. Seat activations climbing. Token spend The post AI adoption isn’t the same as AI usage appeared first on The New Stac

📅 Aug 8, 2026 • 📰 The New Stack

🔗 Read more

📄 Microsoft’s New Testing Agent Tackles the Trust Gap in AI-Generated Code

AI coding assistants write code fast. Whether that code can be trusted is a separate question, and it’s becoming a more urgent one. Surveys this year put average developer trust in AI-generated output

📅 Aug 7, 2026 • 📰 DevOps.com

🔗 Read more

📄 ‘Flooding Dropper’ Is Hitting npm With a Tidal Wave of Malicious Packages

Threat researchers at Sonatype are warning developers of an expanding campaign that is generating a wide range of npm accounts and dropping small numbers of malicious packages from each one, essential

📅 Aug 7, 2026 • 📰 DevOps.com

🔗 Read more

📄 JetBrains Academy – July Digest

Somewhere between the fifteenth open tab and the third iced coffee, it hit me. Maybe we don’t hate meetings. We just hate the ones where nobody has anything to say. Welcome back to another mandatory m

📅 Aug 7, 2026 • 📰 JetBrains Blog

🔗 Read more

📄 Stop burning your AI budget: Optimize GPU usage and model deployment with workflow navigator

Uber burned through its entire 2026 AI tools budget by April. Microsoft faced a similar crisis, pulling Claude Code licenses because the tool worked too well and people used it too much. Even OpenAI's

📅 Aug 7, 2026 • 📰 Red Hat Blog

🔗 Read more

📄 Why Reliability Guardrails Are Needed in Every AI Coding Pipeline

We’re in the middle of a reliability reckoning. Thanks to AI, companies are shipping code much faster than before. But if there’s anything to learn from the surge in high-profile outages over the last

📅 Aug 6, 2026 • 📰 DevOps.com

🔗 Read more

📄 AI Architecture: Moving Past the Washing to the Truth

In the current hype cycle, “AI” has become a linguistic junk drawer—a catch-all term that vendors use to mask everything from basic if-then statements to massive neural networks. For the modern enterp

📅 Aug 6, 2026 • 📰 SUSE Blog

🔗 Read more

📄 Println Debugging Done Right

The simplest tools are often the most useful, and debugging is a prime example of this. There are many advanced debugging techniques, and while they all have their use cases, println debugging is stil

📅 Aug 6, 2026 • 📰 JetBrains Blog

🔗 Read more

📄 Figma Connect for WebStorm: Stage One of a Better Design-to-Code Experience

Where time actually goes in design-to-code Every design implementation starts the same way: find the Figma tab, find the right frame, screenshot it, paste it somewhere, switch back to the terminal. By

📅 Aug 6, 2026 • 📰 JetBrains Blog

🔗 Read more

If the answer is no, one of two things happens. You say "it's on the roadmap" and watch the deal slip a quarter, or somebody promises a date and the work lands on you with a deadline attached and no design time. Both are avoidable, because the expensive part of this work is not the protocol. It is a data model change, and you can make that change long before anyone asks.

This covers what enterprise buyers actually mean, what has to change in your application, the validation steps that turn a SAML integration into an authentication bypass if you skip them, and the order to build it in.

TL;DR

• SSO and provisioning are different problems. SAML answers "is this person who they say they are". SCIM answers "who should exist in the first place, and who should stop existing".
• The hard part is neither protocol. It is that your app probably assumes a user owns their own account. Enterprise means the organisation owns the account, and that is a schema change.
• Build the organisation and connection model first. It is useful on its own and it is the thing you cannot retrofit under deadline pressure.
• SAML is XML with a signature. Validating that signature is necessary and not sufficient. You must also check Audience, Destination, InResponseTo, the time window, and that the assertion you read is the assertion that was signed.
• A whole class of 2018 CVEs existed because libraries read the text of a signed XML node differently to the way the signature covered it. An XML comment inside NameID was enough to log in as somebody else.
• SCIM is a boring REST API you host. The part everyone gets wrong is deprovisioning: PATCH with active: false must actually kill sessions, not just flip a column.
• Roles are the trap. Sync group membership, but keep your own authorisation model. Do not let the IdP be the source of truth for permissions you enforce.

Prerequisites

• An application with its own user accounts and sessions
• Familiarity with HTTP redirects, form POSTs, and JSON APIs
• Access to an identity provider test tenant. Okta and Microsoft Entra ID both offer free developer tenants, and you will want one before writing any code

What they are actually asking for

"SSO" in a procurement document usually bundles three separate things. Being precise about which one is being asked for saves a lot of argument later.

Authentication. The user lands on your login page, types a work email, and gets bounced to their company's identity provider. They come back authenticated. No password of yours involved. This is SAML, or increasingly OIDC.

Provisioning and deprovisioning. When IT adds someone to the "Acme Engineering" group, an account appears in your product without anyone inviting them. When that person leaves, the account is disabled within minutes. This is SCIM, and it is the one people underestimate.

Central policy. MFA, session lifetime, device posture, conditional access. You get this largely for free by delegating authentication, which is a genuinely good reason to support SSO beyond the contract.

The second is where the value is for the buyer. An IT admin who has to remember to log into fourteen SaaS dashboards to remove a departing employee will eventually forget one, and that forgotten account is an audit finding.

Note what happens if you build only SAML. The account gets created on first login instead, which sounds fine until someone leaves: the IdP stops letting them log in, but your app still holds an active session and an enabled account. The buyer asked for deprovisioning and you gave them a login page.

The change that has to come first

Here is the part worth internalising, because it is the only part that is genuinely hard to retrofit.

Most products start with a user model that looks roughly like this:

CREATE TABLE users (
  id            uuid PRIMARY KEY,
  email         text UNIQUE NOT NULL,
  password_hash text,
  created_at    timestamptz NOT NULL DEFAULT now()
);

The account belongs to the person. They chose the email, they chose the password, they can change both, and they can delete the account. Every enterprise requirement contradicts that. The account belongs to the company. The company decides the email, forbids the password, and revokes the account without asking.

So the model has to grow an organisation, and a way to route someone to the right identity provider:

CREATE TABLE organizations (
  id          uuid PRIMARY KEY,
  name        text NOT NULL,
  created_at  timestamptz NOT NULL DEFAULT now()
);

-- One configured identity provider for an organisation. A large customer may
-- have more than one, so this is deliberately not a column on organizations.
CREATE TABLE sso_connections (
  id              uuid PRIMARY KEY,
  organization_id uuid NOT NULL REFERENCES organizations(id),
  protocol        text NOT NULL CHECK (protocol IN ('saml', 'oidc')),
  -- SAML: the IdP's entity ID, SSO URL and signing certificate
  idp_entity_id   text,
  idp_sso_url     text,
  idp_certificate text,
  enabled         boolean NOT NULL DEFAULT false,
  created_at      timestamptz NOT NULL DEFAULT now()
);

-- Which email domains route to which organisation. This is what turns
-- "alice@acme.com" on your login form into "send her to Acme's Okta".
CREATE TABLE organization_domains (
  organization_id uuid NOT NULL REFERENCES organizations(id),
  domain          text NOT NULL UNIQUE,
  verified_at     timestamptz,
  PRIMARY KEY (organization_id, domain)
);

ALTER TABLE users
  ADD COLUMN organization_id uuid REFERENCES organizations(id),
  -- The IdP's stable identifier for this person. Not the email.
  ADD COLUMN external_id     text,
  ADD COLUMN sso_connection_id uuid REFERENCES sso_connections(id);

-- Two people at different companies can share an email in theory; in practice
-- the important constraint is that an IdP's ID is unique within its connection.
CREATE UNIQUE INDEX users_connection_external_id
  ON users (sso_connection_id, external_id)
  WHERE external_id IS NOT NULL;

Three details in there matter more than they look.

external_id is not the email. People change surnames, and IT changes their email address. If you key the account on email, that rename creates a second account and orphans the first. Every IdP sends a stable identifier that survives a rename. Store it and match on it.

Domain verification is not optional. organization_domains is a routing table that decides which company controls a login. If anyone can claim gmail.com, or worse, claim a competitor's domain, you have handed them every future user at that domain. Verify by DNS TXT record before setting verified_at, and never route on an unverified row.

Password login has to become conditional. Once an organisation has SSO enforced, a user in it must not be able to fall back to a password, or you have added a bypass around all that conditional access the customer bought. That is a change to your login path, your password reset path, and your account recovery path. Finding all three under deadline is how mistakes happen.

SAML, concretely

SAML 2.0 is a 2005 OASIS standard built on XML. It is verbose and unfashionable and it is what enterprise IdPs speak, so here we are.

The flow you want is SP-initiated: the user starts at your app, you send them to the IdP, they come back. Your app is the Service Provider (SP), the customer's Okta or Entra ID is the Identity Provider (IdP).

1. Alice hits your login page, types alice@acme.com
2. You look up acme.com in organization_domains -> Acme's connection
3. You build an AuthnRequest, redirect her to the IdP's SSO URL
4. She authenticates there (password, MFA, whatever Acme mandates)
5. IdP POSTs a SAMLResponse to your Assertion Consumer Service URL
6. You validate it, find the user by external_id, create a session

Two URLs you will hand the customer's IT admin, so name them properly and never change them:

• ACS URL (Assertion Consumer Service), where step 5 POSTs. Something like https://app.example.com/auth/saml/{connection_id}/acs
• SP Entity ID, a stable identifier for your application. A URL is conventional but it is an identifier, not an endpoint

Put the connection ID in the ACS URL path. The alternative is figuring out which connection a response belongs to by inspecting the response itself, which means parsing untrusted XML before you know which certificate should have signed it.

The response arrives as a base64-encoded XML document in a form POST. Stripped to the parts that matter:

<samlp:Response Destination="https://app.example.com/auth/saml/abc123/acs"
                InResponseTo="_a1b2c3">
  <saml:Issuer>http://www.okta.com/exk1fake</saml:Issuer>
  <saml:Assertion>
    <ds:Signature>...</ds:Signature>
    <saml:Subject>
      <saml:NameID Format="...emailAddress">alice@acme.com</saml:NameID>
      <saml:SubjectConfirmationData NotOnOrAfter="2026-08-08T09:05:00Z"
                                    Recipient="https://app.example.com/auth/saml/abc123/acs"
                                    InResponseTo="_a1b2c3"/>
    </saml:Subject>
    <saml:Conditions NotBefore="2026-08-08T08:55:00Z"
                     NotOnOrAfter="2026-08-08T09:05:00Z">
      <saml:AudienceRestriction>
        <saml:Audience>https://app.example.com/saml/metadata</saml:Audience>
      </saml:AudienceRestriction>
    </saml:Conditions>
    <saml:AttributeStatement>
      <saml:Attribute Name="email">
        <saml:AttributeValue>alice@acme.com</saml:AttributeValue>
      </saml:Attribute>
      <saml:Attribute Name="groups">
        <saml:AttributeValue>Engineering</saml:AttributeValue>
        <saml:AttributeValue>Admins</saml:AttributeValue>
      </saml:Attribute>
    </saml:AttributeStatement>
  </saml:Assertion>
</samlp:Response>

The validation that people skip

This is the section to read twice. A SAML integration that validates the signature and nothing else is not secure, and the failure mode is complete authentication bypass rather than something subtle.

Every one of these must pass:

The signature is valid, against the certificate you configured for this connection. Not against a certificate embedded in the response. That sounds obvious written down, and it has been shipped more than once.

Something is actually signed. Either the Response or the Assertion must be signed, and you must check which. If only the Response is signed and you read attributes from an unsigned Assertion inside it, an attacker rewrites the assertion freely.

The thing you read is the thing that was signed. This is the failure mode behind the 2018 CVE cluster, and it deserves its own section below.

Audience matches your SP Entity ID. Without this, an assertion the customer's IdP issued for a different vendor can be replayed at you. Both are legitimate assertions from a trusted IdP; only the audience distinguishes them.

Destination and Recipient match your ACS URL.

NotBefore and NotOnOrAfter bracket the current time, with a small clock skew allowance. Sixty seconds is plenty.

InResponseTo matches a request you issued and have not already consumed. Store the request ID when you generate the AuthnRequest, delete it on use. This is your replay defence, and it is why unsolicited IdP-initiated login is harder to secure: there is no request to correlate.

The assertion ID has not been seen before. Belt and braces on replay, and cheap: a table of consumed IDs with a TTL matching your skew window.

The comment that logged in as someone else

In February 2018, Duo Labs published a vulnerability class affecting many SAML implementations at once, and it is the clearest illustration of why "the signature was valid" is not the end of the story.

XML canonicalization and DOM text extraction disagree about comments. The signature is computed over the canonical form of the node, which includes everything. But some XML APIs, when asked for the text content of a node, return only the first text child and stop at a comment.

So an attacker who legitimately controls the account john_doe registers, then inserts a comment into the NameID of their own valid, correctly signed assertion:

<saml:NameID>john<!---->_doe</saml:NameID>

The signature still verifies, because the bytes covered by the signature are unchanged in canonical form. But the service provider asks for the text of NameID, gets back john, and logs the attacker in as a different user entirely.

This affected multiple independent libraries simultaneously: OneLogin's python-saml (CVE-2017-11427) and ruby-saml (CVE-2017-11428), Clever's saml2-js (CVE-2017-11429), OmniAuth-SAML (CVE-2017-11430), Shibboleth (CVE-2018-0489), and Duo's own Network Gateway (CVE-2018-7340).

The lesson is not "patch those CVEs", they are long fixed. It is that the gap between what was signed and what you read is a real and non-obvious attack surface, and it is the reason to stay on a maintained library rather than assembling XML handling yourself.

SCIM: the boring half that matters more

SCIM 2.0 is defined by RFC 7642 (use cases), RFC 7643 (core schema) and RFC 7644 (protocol). Unlike SAML, you are the server: the IdP calls your API on a schedule or on change.

You host a handful of endpoints under a base URL, authenticated with a bearer token you generate per connection:

GET    /scim/v2/Users?filter=userName eq "alice@acme.com"
POST   /scim/v2/Users
GET    /scim/v2/Users/{id}
PUT    /scim/v2/Users/{id}
PATCH  /scim/v2/Users/{id}
DELETE /scim/v2/Users/{id}

GET    /scim/v2/Groups
POST   /scim/v2/Groups
PATCH  /scim/v2/Groups/{id}

A user resource is JSON with a schema URN:

{
  "schemas": ["urn:ietf:params:scim:schemas:core:2.0:User"],
  "id": "8f4a1c22-...",
  "externalId": "00u1fake",
  "userName": "alice@acme.com",
  "name": { "givenName": "Alice", "familyName": "Ng" },
  "emails": [{ "value": "alice@acme.com", "primary": true }],
  "active": true
}

externalId is the IdP's identifier. id is yours. Return yours in the response body and in a Location header; the IdP stores it and uses it for every subsequent call.

Filtering is the part people get caught by. The IdP checks whether a user exists before creating them, using SCIM's own filter grammar:

GET /scim/v2/Users?filter=userName eq "alice@acme.com"

You have to parse that. Not all of it, thankfully. In practice Okta and Entra ID send eq on userName and externalId and little else, so a narrow parser that handles the operators you have observed and returns a clear error for anything else beats a general implementation you got subtly wrong. Return a ListResponse, with totalResults: 0 and an empty Resources array when there is no match, not a 404.

Updates arrive as PATCH with SCIM's own operation format, which resembles JSON Patch but is not it:

{
  "schemas": ["urn:ietf:params:scim:api:messages:2.0:PatchOp"],
  "Operations": [
    { "op": "replace", "path": "active", "value": false }
  ]
}

Providers vary in exactly how they send these: path is sometimes omitted with the value carrying the field, op casing differs, and some send "value": "False" as a string. Handle the variations you see in testing and log loudly on anything unrecognised, because silently ignoring a PATCH you did not understand is how deprovisioning quietly stops working.

Deprovisioning is a promise, not a column

This is the single most common gap, and it is worth being blunt about because it is the requirement the customer actually cares about.

When someone leaves the company, the IdP sends you active: false. Most implementations set a column and return 200. The customer's security team believes access is revoked. It is not, because:

• The user's existing session cookie is still valid until it expires
• Their API tokens still work
• Their OAuth grants to your integrations still work
• If you have a mobile app with a long-lived refresh token, it still refreshes

A correct handler does all of this:

def deactivate_user(user_id: str) -> None:
    with db.transaction():
        db.execute("UPDATE users SET active = false WHERE id = %s", (user_id,))
        # Everything below is the part that is usually missing.
        db.execute("DELETE FROM sessions WHERE user_id = %s", (user_id,))
        db.execute("UPDATE api_tokens SET revoked_at = now() "
                   "WHERE user_id = %s AND revoked_at IS NULL", (user_id,))
        db.execute("DELETE FROM oauth_grants WHERE user_id = %s", (user_id,))
    # Session state that lives outside the database has to go too.
    cache.delete_pattern(f"session:{user_id}:*")
    audit.log("user.deactivated", user_id=user_id, source="scim")

Two further notes. Prefer deactivation to deletion: DELETE /Users/{id} should almost always be a soft delete, because hard-deleting a user destroys the audit trail the same customer will ask for. And if your sessions are stateless JWTs with a long expiry, you have a design problem that SCIM has just exposed. Either shorten the expiry to something you can tolerate as a revocation delay, or check a revocation list on each request.

Groups, roles, and the trap

The IdP will send group membership, either as a SAML attribute or through SCIM's /Groups endpoint. The obvious move is to map groups straight onto your permissions. Resist slightly.

Map IdP groups to your roles through an explicit, per-connection mapping table that the customer's admin configures in your UI:

CREATE TABLE group_role_mappings (
  connection_id uuid NOT NULL REFERENCES sso_connections(id),
  idp_group     text NOT NULL,     -- "Acme-Engineering-Admins"
  role          text NOT NULL,     -- "admin", your vocabulary
  PRIMARY KEY (connection_id, idp_group)
);

Three reasons this indirection earns its keep. Customers name groups for their own org chart, not your permission model, and those names change. A rename in Okta should not silently strip everyone's access. And when a customer disputes what someone could see, you want a record of the mapping you applied rather than an inference from directory state that has since changed.

Keep one guardrail: never let a group sync remove the last administrator of an organisation. Every product that skips this eventually locks a customer out of their own account on a Friday afternoon.

Build it in this order

Sequenced so each step is useful on its own, and nothing later requires unpicking anything earlier:

1. Organisation and membership model. Users belong to an org. Useful immediately for billing and shared workspaces.
2. Domain claiming with DNS verification. Unverified domains route nowhere.
3. Conditional password login. A flag on the org that disables password auth for its members, exercised before any IdP exists.
4. SAML with one provider. Okta or Entra ID, whichever your first customer uses. Full validation from day one.
5. Session revocation. Build the "kill everything for this user" function and call it from your admin panel. SCIM will need it.
6. SCIM Users. Create, update, and active: false wired to step 5.
7. SCIM Groups and role mapping.
8. Audit log, exposed to the customer. They will ask, and it is much easier if you emitted events all along.

Steps 1 to 3 are the ones to do now, before anyone asks. They are pure prerequisite, they carry no protocol risk, and they are the reason a SAML project takes three weeks instead of three months.

Build or buy

Worth being straight about the tradeoff rather than pretending it is obvious in either direction.