Workflow Orchestration

TL;DR: A workflow is a YAML definition in services/workflows/<id>/ that moves through stages of steps. A run is a durable instance of one definition version, stored in Postgres, that can wait for days on a person or an event and survive restarts. Agent steps run as headless Claude Code sessions on a separate runner role, governed by the same hooks and gateway as an interactive session. People answer decisions routed to roles. Writes to Jira, Salesforce or GitHub happen only in a write step, after an approval, exactly once.

The model

Concept	What it is	Where it lives
Definition	Stages, steps, roles, triggers, limits. One version per content hash (YAML + prompts + schemas)	services/workflows/<id>/, synced to workflow_definitions
Run	One instance, pinned to the definition hash it started on	workflow_runs
Step attempt	One execution of one step; an agent attempt names the core agent run that executes it	workflow_steps (agent_run_id → agent_runs)
Decision	A human task routed to a workflow role, answered once	workflow_decisions
Artefact	A step's handoff object: proposed → approved → executed	workflow_artifacts
Effect	One write-back, keyed (run, step, effect_key)	workflow_effects
Event	One inbound webhook, unique per (source, external_id)	workflow_events
History	Append-only trajectory of every transition and actor	workflow_history

A run moves through created → running → waiting_decision | waiting_event | waiting_timer → running → completed | failed | cancelled | suspended. The tick (workflow_tick job, every 5 s, one replica at a time) plans one transition per run under a compare-and-set on the run's version. The tick never starts a process: it queues each agent step as a core agent run (agent_runs, origin workflow_step) for the runner, in the same transaction.

Editing a definition creates a new version. Runs in flight stay on the hash they started with; new runs take the active version. /admin/sync shows the declared and applied hashes for the workflows plane.

Definition reference

workflow:
  id: core-dev                  # must equal the directory name
  name: Core Dev flow
  domain: core                  # optional grouping label
  description: …                # optional
  enabled: true                 # default true
  repo: https://github.com/…    # optional: agent steps get a shallow clone as their working directory
  permission_mode: dontAsk      # optional: Claude Code permission mode for agent steps (default dontAsk)
  variants:                     # dotted-path overrides applied when a run loads its pinned definition
    default: {}
    india: { stages.design.steps.design_review.role: lead_dev_in }
  context:
    required: [jira_key]        # a start without these keys is refused (422)
    scope: [project]
  triggers:
    - { kind: manual, roles: [developer] }               # console, API, MCP tool
    - { kind: event, source: jira, match: { event: issue_transitioned, to: Ready } }
    - { kind: schedule, cron: "0 0 7 * * 1" }            # six-field cron, seconds first
  run_as: example@service.astounddigital.com  # required with any schedule or event trigger
  roles:                        # workflow role → platform roles, groups, or project membership
    product_owner: { roles: [project_manager], groups: [business-users] }
    lead_dev: { roles: [platform_admin] }
  stages:
    - key: requirements
      entry: { requires: [context.jira_key] }            # paths that must be non-null to enter
      steps: [ … ]
  limits: { max_attempts: 3, max_loop: 2, wall_clock: 2h }
  on_failure: { default: suspend, connector_unavailable: retry_backoff }

Every key is checked: an unknown key fails the load, and the validator lists every problem with its dotted path (stages.review.steps.check.loop_max).

A definition with a schedule or event trigger must name run_as, the email of the service user that owns the runs nobody starts by hand; without it the definition fails validation. That user is initiated_by on every scheduled and event-started run, and its identity authenticates the run's write-backs. The email is resolved when the trigger fires, not at load: if no active user has it, the trigger is skipped and logged. If the user is deactivated after a run started, that run's writes are refused as missing_data. Manual runs are always owned by the person who started them.

Steps

All steps take key (unique across the workflow), kind and an optional name. The other fields depend on the kind:

Kind	Required	Optional	What happens
agent	prompt	agent, output (JSON Schema file), tools, model, budget_usd, gate, loop_to + loop_max	Queued for the runner; a headless Claude Code session produces structured output, stored as a proposed artefact
decision	role	artifact, outcomes, on, due, escalate_to	Opens a decision for the role; the run waits
manual	—	role	A person completes the step with an artefact; no model call
event	source	match	Waits for a matching inbound event
gate	gate	escalate_to	Evaluates an expression over run state; refusal follows on_failure.gate_failed
parallel	parallel (agent branches), join: all | any	—	Fans out branches (<key>.<branch>), joins their outputs under <key>
write	system, tool	args_from, effect_key, requires_decision	Calls one MCP tool after an approval; recorded once in the effects ledger

• tools: Claude Code tool names (Read, Grep, Edit, Bash(git diff:*)) and mcp:<server> for a server from services/mcp/. A step gets only the servers it lists.
• gate: findings.blocking == 0, exists(path), len(path) > 0, over the step's own output, steps.<key>, context.* and state.*. With loop_to and loop_max, a refused gate loops back to an earlier step at most loop_max times.
• on: per-outcome jump, usually request_changes: <earlier step>. A decision answers approve (advance), request_changes (redo, at most limits.max_loop times), reject (the run completes with outcome rejected) or cancel (the run is cancelled).
• args_from: <step>.output.<path> naming an object of tool arguments. The approved (or reviser-edited) artefact is used when there is one.
• args (write only): values set over the args_from object, so an identity the engine already holds is never taken from the model. Strings interpolate {{args.*}}, {{context.*}}, {{state.*}} (e.g. state.branch, the run's workflow/<run_id> branch) and {{run_id}}; any other root fails the load, and a referenced value that is not set refuses the write rather than sending it empty. core-dev opens its PR with args: { head: "{{state.branch}}", base: next }.
• effect_key: interpolates {{args.*}}, {{context.*}}, {{state.*}} and {{run_id}}. Two writes with the same key in one run are one effect.
• due / escalate_to: an unanswered decision past due is reminded once; with escalate_to set it gets a second window and is then escalated once to that role as an escalation task. The run keeps waiting.
• variants: each key is a dotted path. context.<key> sets a context default (the run's own value wins); anything else walks the definition, naming list elements by key (stages.design.steps.design_review.role). A path that names nothing is refused when the definition loads.
• requires_decision: the decision that must be approve before the write; without it, the nearest preceding decision. A write with no preceding decision fails validation.

Failure handling

on_failure maps a failure class to an action. Classes: default, connector_unavailable, budget_exceeded, output_invalid, gate_failed, timeout, missing_data. Actions: retry_backoff (a new attempt after an exponential delay, up to limits.max_attempts; when they run out the run is suspended with failure_reason: attempts_exhausted: …, its work kept, so an operator's retry can finish it), suspend, escalate (opens a decision for escalate_to, default admin), fail. Anything unmapped suspends.

What an agent step receives

An agent step is one agent run of core's systemprompt-agent-runs. Its request is written when the step is queued: the prompt (the step's prompt file followed by the run context, the run's branch, the outputs of earlier steps and any reviewer feedback), the output schema (without $schema and $id), the tools, only the MCP servers the step lists, the model, the budget and the wall clock. Core's Claude Code adapter runs claude -p with --json-schema, --strict-mcp-config and --max-budget-usd, and records the session's events in agent_run_events; the run detail page reads them from there. Write prompts as contracts: what inputs exist, what artefact to hand off, and that the step does not write to external systems.

Each session runs as the run's initiating user with credentials issued by core's RunCredentialService:

• a short-lived personal access token and gateway session, revoked when the run ends;
• the governance hooks (PreToolUse → /api/public/hooks/govern) under the plugin id workflow:<definition>, so every tool call passes the four-stage chain and is audited;
• the attribution header x-systemprompt-scope-workflow: <run> (and the project), so cost and requests roll up per run;
• a cleared environment and a private HOME: no host credentials are inherited.

A retried step resumes the failed attempt's session, and so does an agent run core reclaims after its runner died.

Deploying the runner

The runner is server.role: runner: it serves only health and metrics and, once the server answers /readyz, runs core's agent-run claim loop over workflow_step runs (FOR UPDATE SKIP LOCKED, a 90 s lease renewed every 30 s). Each claimed run gets the run's workspace (a clone on workflow/<run> when the definition names a repo), is executed by core and settles its step. A replica whose lease lapses loses its runs to another replica; the tick settles any run that ended with its step still open.

Helm (deploy/helm/systemprompt-astound/values.yaml):

Value	Default	Meaning
runner.enabled	false	Deploy the runner Deployment
runner.replicas	1	Replicas
runner.image	<image>:<tag>-runner	The runner Dockerfile stage: the server plus Node and Claude Code
runner.maxConcurrentSteps	4	Concurrent sessions per replica
runner.workdirSize	20Gi	emptyDir for agent-run directories and clones
runner.autoscaling.*	off	HPA on workflow_steps_queued (needs an external-metrics adapter)

Environment (read by the runner at start):

Variable	Default
WORKFLOWS_RUNNER_ENABLED	true
WORKFLOWS_RUNNER_MAX_CONCURRENT_STEPS	4
WORKFLOWS_RUNNER_GATEWAY_URL	the profile's internal API URL
WORKFLOWS_RUNNER_HOOKS_URL	the gateway URL

On Kubernetes, point the gateway URL at the gateway Service and the hooks URL at the admin Service. The runner image puts claude on PATH.

Metrics: workflow_runs{state}, workflow_steps_queued, workflow_step_duration_seconds{kind}, workflow_step_cost_microdollars_total, workflow_decisions_waiting.

Starting runs and answering decisions

Surface	Start	Decide
Console	/admin/workflows → Start	/admin/workflows/inbox
JSON API	POST /api/public/admin/workflows/definitions/{id}/runs with {context, project_id?, variant?}	POST /api/public/admin/workflows/decisions/{id}/{approve|request_changes|reject|cancel} with {comment?, revised_artifact?}
MCP (agent-farm server)	run_start with origin: workflow	workflow_inbox, workflow_decide

The MCP tools are run_start(origin: workflow, definition, context_json, variant, project), run_list(scope, workflow_run, definition), run_facts(run_id), workflow_decide(decision_id, outcome, comment) and workflow_inbox(limit). They apply the same role and project rules as the API and ship on the user-scoped agent-farm server, reachable by developers and project managers through the astound-engineering workspace; see Agent runs from Claude Code.

Who may act:

• Start: a holder of one of the manual trigger's roles, and a member of the project the run is filed under.
• Decide: a holder of the decision's workflow role (resolved through roles:; an unbound name is read as a platform role) who can see the run. Anyone else receives 403. A decision already answered returns 409.
• See a run: administrators, the initiator and members of the run's project. A run's artefacts are only read through its own id.

revised_artifact on an approval replaces the step's proposed artefact; the revision is what a later write sends.

Inbound events

POST /api/public/workflows/events/{source} accepts events from the sources declared in services/workflows/sources.yaml:

sources:
  - id: jira
    kind: webhook
    adapter: jira          # jira | salesforce | generic
    secret_ref: jira_webhook_secret

• Authentication: a bearer PAT, or for a source with secret_ref an HMAC-SHA256 of the raw body under that profile secret, sent as sha256=<hex> in x-hub-signature-256 (or signature_header).
• Normalisation: the adapter turns the body into {event, external_id, keys, payload}.
• Idempotency: (source, external_id) is unique. A redelivery answers 200 {"status":"duplicate"} and changes nothing.
• Matching: an event resumes a run waiting in an event step whose match it satisfies ($context.<key> compares against the run context), and starts a run of every definition whose event trigger matches, with trigger_key = source:external_id, so it starts once.

Operations

Action	API	What it repeats
Suspend	POST …/runs/{id}/suspend	Nothing; the run stops being planned
Resume	POST …/runs/{id}/resume	Nothing; planning continues from the cursor
Retry step	POST …/runs/{id}/steps/{key}/retry (suspended runs, or failed runs whose class is attempts_exhausted, connector_unavailable or timeout; failed agent or write steps)	That step only: an agent step resumes its session, a write re-arms its effect
Restart	POST …/runs/{id}/restart	A new run with the same context; the old one is linked as parent
Cancel	POST …/runs/{id}/cancel	Nothing; open steps are cancelled

• Write-backs are exactly once. Before a call the effect row is written pending; after it, succeeded with the record reference or failed. A rerun finds a succeeded row and does not call again. A pending row left by a crash is in doubt: the run waits for an operator rather than risking a duplicate.
• Decided steps leave waiting when the answer is consumed: approve and request_changes succeed the decision step, reject and cancel fail it.
• Reviewer feedback. A step re-run by request_changes gets the comment, the reviser's artefact if any, and its previous output under a "Reviewer feedback" heading in its prompt.
• Leases. A runner that dies stops renewing; once its agent run's lease lapses, core queues the run again and another replica resumes the session (three attempts, then the step fails as connector_unavailable).
• Cost. A step costs what the gateway metered for its run's session; when the run completes every step's cost is read again, so requests recorded late are counted.
• Schedules start at most one run per cron slot, and skip slots older than ten minutes after an outage.
• Back-links. A Jira write gets a comment linking back to the run.

Shipped definitions

Id	Shape	Roles	Write
core-dev	story analysis → product owner → design → lead dev → parallel backend/frontend → review loop (≤ 2) → lead dev → PR	product_owner, lead_dev, lead_dev_in (variant india)	github.create_pull_request
growth-proposal	deal, solution and estimate finders in parallel → proposal → presales	presales, presales_uk (variant uk), deal_desk	salesforce.create_record (Note)
pmo-review	weekly schedule → findings → PMO	pmo, pmo_lead	atlassian.createJiraIssue

Each is driven through five scenarios (happy path, rejection, missing context, connector failure, isolation of concurrent runs) by tests/integration/admin-core/src/workflows_scenarios.rs.

Limits and what is not covered

• There is no github MCP server in this installation yet, so core-dev's pull-request write fails as connector_unavailable until one is registered.
• The Salesforce record-create tool name in growth-proposal must be checked against the tools the org's hosted server lists.
• A write step performs one effect. Fanning out one record per item in a list (one Jira issue per PMO action) is not supported; pmo-review files one tracking issue that lists the actions.
• Scheduled and event-started runs are initiated by a system actor with no connector tokens, so their writes to Atlassian or Salesforce fail. Start those runs by hand when they must write.
• The MCP workflow tools are reachable only by administrators (see above).