Why a run is not moving

A workflow queued behind a mutex looks exactly like a workflow waiting for a node with enough memory, which looks exactly like a workflow whose image is still pulling. Three unrelated problems, one grey Pending badge, and no obvious next move.

dagweave separates two of the three by name. A Pending step says either which lock it is queued behind or that the scheduler will not place its pod, and how long it has been at it. Anything else Argo said about the step sits on the row beside it, verbatim and unclassified. This page is what you can get and what to do with each answer.

Waiting on a lock

If the step or the workflow declares a mutex or a semaphore and something else holds it, the row names both sides:

◷ Waiting on mutex deploy-lock for 12m14s, held by deploy-prod-x7k2q.

Semaphores read the same way. The lock is written as Argo scopes it, so a mutex in the workflow’s own namespace is bare and one outside it is prefixed with its namespace. Only one holder is named even where a semaphore admits several, and nothing here claims a position in a queue, because Argo publishes the holders and never the order.

A run held by dagweave’s own queue is a different thing, and that one does say where it stands: how many of your team’s runs are ahead of it, and why it is waiting. If a run waited in our queue and then waited again on a lock, you see both, in the order they happened. They are never merged into one line, because a queue wait means capacity and a lock wait means another workflow is holding something, and those have different fixes.

This comes out of Argo’s own synchronization status on the Workflow object, which the connector reads with everything else. There is no metrics pipeline behind it and nothing to install.

What to do. Probably nothing. A lock doing its job looks precisely like this, and you asked for it. The case worth acting on is a lock held by a run that has already finished, which means the holder died without releasing. Open the named run. If it is genuinely gone, deleting the holding Workflow releases the lock.

If the queue is the problem rather than a bug, the answer is upstream of the run: a semaphore with more slots, or a narrower lock so that two things which never actually conflict stop taking turns. Both are fields on the step and on the workflow, in the inspector’s concurrency section.

Waiting for a node

The pod exists and the scheduler cannot place it anywhere:

◷ No node can take this pod for 12m14s.

Under it sits the sentence the scheduler wrote, which Argo copies onto the node and dagweave passes through untouched. It might read 0/12 nodes are available: 12 Insufficient nvidia.com/gpu, or whatever yours says. It is worth reading closely, because it names the exact resource that ran out.

What to do. One of three things is true. The step asks for more than any node has, in which case fix the request in the node’s resource settings. The step asks for a node that does not exist, in which case check its nodeSelector, tolerations and affinity against the nodes you actually run. Or the cluster is full and a scale-up is in flight, in which case the step clears on its own and the number in the message tells you how close you are.

A step that will never schedule waits forever by default, which is rarely what you want at three in the morning. pendingTimeout caps how long a step’s pod may sit Pending before Argo fails it, and you set it per step in the inspector. A step that cannot get a GPU then fails in ten minutes with a clear reason instead of holding the run open until someone notices.

Waiting for an image

dagweave does not recognise this one. A pod stuck on ImagePullBackOff or ErrImagePull gets no named wait, because the connector classifies lock waits and unschedulable pods and nothing else. What you get is the step’s Pending phase and whatever message Argo copied off the pod, which is where the registry error will be if there is one.

So read the message. A slow pull on a cold node clears by itself; a backoff does not, and the tag is wrong, the registry needs credentials the namespace does not have, or the image was never pushed. Check the tag first.

Running, but nothing is running

A run sitting in Running while nothing underneath it advances is more likely a cap than a fault. dagweave names none of the three below as a wait reason, so this section is a matter of knowing where to look.

The workflow’s own parallelism. A workflow with parallelism set runs that many pods at once and holds the rest, which is why a fan-out of five hundred crawls through in tens. It is in the inspector next to the lock fields.
The controller’s namespace parallelism. Your cluster operator caps how many workflows run at once per namespace, and your run is queued behind other people’s. Nothing inside your workflow will move it and nothing in dagweave reports it; ask whoever runs the controller.
A step that is not a step. An approval node compiles to an Argo suspend, and a suspended run is waiting for a person rather than for the cluster. dagweave has no approve button. Resume the workflow with the Argo CLI or the Argo UI and the run carries on.

Nothing at all is updating

If every run went quiet at the same moment, suspect the connector before the cluster. dagweave marks a connector offline after 300 seconds without a heartbeat and says so on its row in the connectors list. With failure notifications on, the same channels tell you the first time one goes dark, once per outage.

Your runs are unaffected by any of this. Argo keeps running them; dagweave has simply stopped hearing about them. When the connector reconnects it reports current state and the runs catch up to where they really are. The connector’s own pod logs are readable from its row in the dagweave UI if you administer the team, and a dispatch that never arrived is nearly always explained in them.

Deciding it has waited long enough

Three controls, at three different scopes, and it is worth knowing which one you want:

pendingTimeout caps the wait before a pod starts. Use it where waiting longer will not help, which is scheduling and image problems.
A step timeout caps how long a step runs once it has started, so use it for work that hangs rather than work that never begins.
A workflow deadline caps the whole run regardless of which step is to blame.

All three are per-node or per-workflow fields in the inspector, and all three compile to ordinary Argo fields, so a workflow you exported keeps them.