# Observability
Every applied resource produces one event. That event is recorded to a session store,
translated into Prometheus counters, and aggregated into a summary at the end of the run.
Health checks run alongside and feed the agent's remediation. Together these are how a CCM run
is observed after the fact and monitored while it runs.
{{% notice style="note" title="Where it lives" %}}
`internal/session`: the session stores. `internal/metrics`: the Prometheus collectors.
`internal/healthcheck`: the goss and nagios runners. `model/transaction.go` and
`model/healthcheck.go`: the event and health-check types. Key files: `internal/session/directory.go`,
`internal/session/util.go`, `internal/metrics/metrics.go`.
{{% /notice %}}
## The event flow
One event per resource, recorded once, then fanned out to counters and the run summary.
## Sessions and events
`StartSession` records a start event, each applied resource records a `TransactionEvent`, and
`StopSession` builds a `SessionSummary` from all events. The memory store keeps events in a
slice and is the default. The directory store writes one JSON file per event, named by the
event's KSUID.
{{% notice style="warning" title="Load-bearing decision" %}}
Event IDs are KSUIDs, and that choice does double duty. KSUIDs are timestamp-prefixed and
lexicographically sortable, so sorting the `.event` filenames reconstructs chronological order
without reading timestamps. Parsing the ID also whitelists it to base62 with no path
separators, which is the directory store's defense against path traversal, since the ID becomes
a filename.
{{% /notice %}}
The directory store is append-only and crash-friendly. Each record is an independent file
write with no shared index to corrupt, and replay is a directory scan that reads each event's
`protocol` field to pick the concrete type before unmarshaling.
## Metrics
Collectors live under the `choria` namespace and `ccm` subsystem. `updateMetrics`
(`internal/session/util.go:12`) runs from both stores' `RecordEvent`. It increments a total and
exactly one outcome counter, chosen by a priority order: noop, then changed, skipped, refreshed,
failed, error, and stable.
{{% notice style="warning" title="Load-bearing decision" %}}
Noop is checked first in that priority order. A noop run has `Changed=true` on its event, but it
counts as noop, not changed, so the counters stay mutually consistent: each event increments the
total plus exactly one state counter.
{{% /notice %}}
Durations are recorded with timers around the work: manifest apply, resource apply, fact gather,
and per-check health-check time. `RegisterMetrics` registers the collectors and `ListenAndServe`
serves them at `/metrics` when a port is set.
## Health checks
Health checks run for both apply and health-check-only modes. Each check dispatches by format.
nagios
Runs a command through the manager's runner and maps its exit code to a status: 0 is OK, 1 Warning, 2 Critical, anything else Unknown. Retries up to tries, sleeping between attempts, stopping on OK.
goss
Renders the goss rules through the template engine, writes a temp spec, validates it, and reports Critical when any check failed, else OK. It never emits Warning or Unknown.
Status
HealthCheckStatus values equal the nagios exit codes, so the plugin format maps directly and goss reuses the same enum.
A non-OK result or an execution error marks the event failed. In the agent, a critical result
increments a remediation counter and queues a priority apply, so a failing check drives a
corrective run. Several `Agent*` metrics are declared here but recorded at their call sites in
the agent, and `AgentHealthCheckTime` is currently registered without a call site.
{{% notice style="tip" title="Next" %}}
Continue to the [Reference and Map]({{% relref "reference" %}}) for the CLI surface, the source
map, and a glossary.
{{% /notice %}}