Observability

Every applied resource produces one event. That event is recorded to a session store, translated into Prometheus counters, and aggregated into a summary at the end of the run. Health checks run alongside and feed the agent’s remediation. Together these are how a CCM run is observed after the fact and monitored while it runs.

Where it lives

internal/session: the session stores. internal/metrics: the Prometheus collectors. internal/healthcheck: the goss and nagios runners. model/transaction.go and model/healthcheck.go: the event and health-check types. Key files: internal/session/directory.go, internal/session/util.go, internal/metrics/metrics.go.

The event flow

One event per resource, recorded once, then fanned out to counters and the run summary.

Sessions and events

StartSession records a start event, each applied resource records a TransactionEvent, and StopSession builds a SessionSummary from all events. The memory store keeps events in a slice and is the default. The directory store writes one JSON file per event, named by the event’s KSUID.

Load-bearing decision

Event IDs are KSUIDs, and that choice does double duty. KSUIDs are timestamp-prefixed and lexicographically sortable, so sorting the .event filenames reconstructs chronological order without reading timestamps. Parsing the ID also whitelists it to base62 with no path separators, which is the directory store’s defense against path traversal, since the ID becomes a filename.

The directory store is append-only and crash-friendly. Each record is an independent file write with no shared index to corrupt, and replay is a directory scan that reads each event’s protocol field to pick the concrete type before unmarshaling.

Metrics

Collectors live under the choria namespace and ccm subsystem. updateMetrics (internal/session/util.go:12) runs from both stores’ RecordEvent. It increments a total and exactly one outcome counter, chosen by a priority order: noop, then changed, skipped, refreshed, failed, error, and stable.

Load-bearing decision

Noop is checked first in that priority order. A noop run has Changed=true on its event, but it counts as noop, not changed, so the counters stay mutually consistent: each event increments the total plus exactly one state counter.

Durations are recorded with timers around the work: manifest apply, resource apply, fact gather, and per-check health-check time. RegisterMetrics registers the collectors and ListenAndServe serves them at /metrics when a port is set.

Health checks

Health checks run for both apply and health-check-only modes. Each check dispatches by format.

nagios: Runs a command through the manager's runner and maps its exit code to a status: 0 is OK, 1 Warning, 2 Critical, anything else Unknown. Retries up to tries, sleeping between attempts, stopping on OK.
goss: Renders the goss rules through the template engine, writes a temp spec, validates it, and reports Critical when any check failed, else OK. It never emits Warning or Unknown.
Status: HealthCheckStatus values equal the nagios exit codes, so the plugin format maps directly and goss reuses the same enum.

A non-OK result or an execution error marks the event failed. In the agent, a critical result increments a remediation counter and queues a priority apply, so a failing check drives a corrective run. Several Agent* metrics are declared here but recorded at their call sites in the agent, and AgentHealthCheckTime is currently registered without a call site.

Continue to the Reference and Map for the CLI surface, the source map, and a glossary.