Architecture

CCM is built around a strict dependency inversion. The model package defines contracts and almost no behavior. Concrete resource types and providers depend on those contracts and on a global registry, never on the orchestrator. The manager package is the orchestrator that satisfies the central contract and wires in facts, data, sessions, NATS, and logging. That inversion is why a provider can self-register without the manager knowing it exists, and why the same resource logic runs from the CLI, a manifest, or the agent.

The layers

Read the stack from the outside in. Entry points build a manager and hand it work. The apply engine turns a manifest into resources. Each resource embeds a shared base that owns the apply flow. The base calls into the concrete type, which decides what to change and calls a provider. Providers do the platform work. Every layer below the manager programs against model interfaces only.

The contracts

The model package is deliberately thin. It holds interfaces and shared value structs, and defers behavior to the packages that implement them.

Resource

The runtime behavior of a managed thing: Type, Name, ResourceId, Provider, Properties, plus Apply, Healthcheck, and Info. Implemented by the per-type Type structs under resources/*/type.go. Defined at model/resource.go:26.

ResourceProperties

The declarative desired state: CommonProperties(), Validate(), ResolveTemplates(), ResolveDeferredTemplates(), ToYamlManifest(). Every per-type properties struct embeds CommonResourceProperties. Defined at model/resource.go:43.

Provider

Intentionally tiny: only Name() string (model/provider.go:8). Each resource package widens it into a capability interface, so the registry can stay type-agnostic while types get a rich contract.

ProviderFactory

The registration unit: TypeName(), Name(), IsManageable(facts, props), and New(Logger, CommandRunner). Defined at model/provider.go:12.

Manager

The large seam the whole engine programs against: facts, data, logging, execution, sessions and events, registration, templating, and NATS. Defined at model/manager.go:25.

The registry as a plugin bus

internal/registry is a global directory keyed by resource type then provider name. Each provider package exposes a Register() that calls registry.MustRegister(&factory{}) from an init(). The manager triggers all of these through a single blank import, _ "github.com/choria-io/ccm/resources" at manager/manager.go:27, so it never names a concrete provider. Duplicate provider names within a type are rejected with ErrDuplicateProvider.

Resolution happens when a resource applies. FindSuitableProvider (internal/registry/registry.go:168) probes every factory for the type with IsManageable(facts, props), keeps the ones that report they can manage the resource, sorts them ascending by the returned priority integer, and takes the first.

The manager

The CCM struct (manager/manager.go:36) is the concrete model.Manager. It is built with NewManager(log, userLogger, opts...) and functional options from manager/opts.go (WithNatsContext, WithSessionDirectory, WithRegistrationDestination, WithNoop, WithEnvironmentData, and others). It defaults to an in-memory session store and a no-op registration publisher, so a bare manager is safe to run offline.

Facts

Facts() caches gathered facts; SystemFacts() always re-gathers with a 2s default deadline; SetFacts and MergeFacts override or overlay the cache.

Data

SetData deep-merges resolved data with an external overlay that always wins; Data() returns a copy.

Sessions

StartSession, RecordEvent, SessionSummary, plus ShouldRefresh and IsResourceFailed, which read the last recorded event for a resource to drive subscribe and require.

Templating

TemplateEnvironment(ctx) assembles the render environment and injects the registration lookup and KV-get closures.

Noop

NoopMode() and SetNoopMode gate every mutating branch in the resource types.

Cross-manager safety

Some platform tooling is not safe to run in parallel. CCM guards it with process-wide mutexes, PackageGlobalLock and ServiceGlobalLock (model/global_locks.go:14), that serialize all package and service operations even across concurrent managers and manifests. The stated reason is that concurrent dpkg/apt or systemd operations would corrupt their databases.

Two type-dispatch switches are hand-maintained and carry a matching TODO: NewResourcePropertiesFromYaml (model/resource.go:158) and ResourceInfo (manager/manager.go:397) both map type names to constructors by hand, and the intent is to make the registry carry that mapping so new types register once. The SelectProvider boilerplate is likewise duplicated across every resources/*/type.go with a // TODO: move to base.

The Resource-Provider Model

Every resource type in CCM is built from three parts. A Type decides whether the system already matches the desired state. A Provider makes the platform-specific change. A shared base.Base owns the flow that connects them: gates, requirements, health checks, and turning the outcome into an event. The split is deliberate. The type stays portable and unit-testable against a mock provider, and the flow is written once for all seven resource types.

The inversion

base.Base owns the apply flow but does not know any resource type. It calls back into the concrete type through a small interface, base.EmbeddedResource (resources/base/base.go:27): ApplyResource, SelectProvider, NewTransactionEvent, and Type. Each type embeds *base.Base and, in its New constructor, sets Base.Resource = t so the base can call back into it. That back-pointer is what lets one copy of the control-gating, require-checking, and event-mapping code serve every type.

Type

Embeds *base.Base; holds the typed properties, the manager, and the resolved provider. Its ApplyResource reads status, compares against the desired state, calls a provider verb, and re-verifies.

Provider interface

Widens model.Provider per type. FileProvider adds CreateDirectory, Store, SetAttributes, Remove, Status (resources/file/file.go:18). PackageProvider and ServiceProvider add their own verbs.

Concrete provider

The only place OS mutation lives. The posix file provider is the sole caller of os.MkdirAll, os.Rename, os.Chown, and os.RemoveAll (resources/file/posix/posix.go).

Applying one resource

The flow below is base.applyOrHealthCheck (resources/base/base.go:74). The type-specific decisions happen inside the type’s ApplyResource.

1. Select the provider The type resolves through registry.FindSuitableProvider, which filters providers by IsManageable(facts, props) and picks the lowest priority number. The result is cached.
2. Gate on control checkControl evaluates control.if and control.unless expressions. If they say do not manage, the event is marked Skipped and returned.
3. Resolve deferred templates File overrides ResolveDeferredTemplates so content and source render only after the control gate, letting unless protect against template errors on a resource that will be skipped.
4. Check requirements For each require entry, Manager.IsResourceFailed reads the last recorded event. Any failed or unmet requirement marks this event Skipped with UnmetRequirements and returns.
5. Read status and decide provider.Status reads the actual state, and isDesiredState compares it against the properties, returning a human-readable mismatch reason.
6. Act, or describe A switch calls the provider verb. Every mutating branch is guarded by if !noop. In noop mode the verb is skipped and a noopMessage describes what would have happened.
7. Verify convergence After a real change, the type re-reads status and re-runs isDesiredState. If the system did not converge it returns ErrDesiredStateFailed with the reason.
8. Finalize and record FinalizeState writes the flags onto the state, base maps them onto the event, and the caller logs and records it.

Ensure states

Ensure semantics are per type, and deliberately loose where the platform is unreliable.

Two invariants worth internalizing

Convergence is verified, not assumed. After a non-noop change the type re-reads status and re-checks the desired state, failing with the specific mismatch reason if the system did not converge. This is why isDesiredState returns a reason string, not just a bool.

Subscribe and refresh

Only the service type uses refresh. Inside its ApplyResource it calls ShouldRefresh(properties.Subscribe), which asks the manager whether the last recorded event for each subscribed resource had Changed == true. A refresh is suppressed unless the service is ensured running, and skipped when the service is currently stopped, since starting it already covers the change. When it fires, the provider restarts the service and the event is marked Refreshed.

The Apply Engine

The apply engine turns a manifest into a sequence of applied resources. It resolves the manifest from a file, an HTTP tarball, or a NATS object store; layers Hiera data and overrides over it; builds each resource; and runs them in declaration order, recording an event after each so later resources can see what earlier ones did.

Resolving a manifest

ResolveManifestReader (resources/apply/apply.go:395) is the core resolver. It runs before any resource executes, and its ordering is precise: data is resolved and templates are expanded before the schema is checked, so a runtime template does not fail a structural validation.

1. Resolve the source ResolveManifestUrl dispatches on scheme: obj:// to the object store, http(s) to a tarball fetch, empty scheme to a local file. Archive paths untar, find manifest.yaml, and set the working directory.
2. Parse the manifest Unmarshal the top-level data, hierarchy, and overrides, plus the ccm block with pre_message, post_message, fail_on_error, resources, and resources_jet_file.
3. Resolve Hiera hiera.ResolveYaml consumes hierarchy.order, merge, and overrides, returning the resolved data and validation rules. Overriding data is deep-merged on top, then the rules are enforced.
4. Publish data mgr.SetData stores the resolved data and the template environment is built from it, so resource fields can reference Data.
5. Produce the resource list Either the inline ccm.resources, or, if resources_jet_file is set, the rendered output of a Jet template. Multi-name blocks are flattened in place, preserving order.
6. Parse and template each resource NewValidatedResourcePropertiesFromYaml builds typed properties per type, resolves templates, and validates.
7. Validate against the schema Last, the resolved payload is checked against schemas/manifest.json, substituting placeholders for still-deferred template fields. NO_SCHEMA_VALIDATION=1 bypasses this.

Execution and ordering

Execute (resources/apply/apply.go:603) opens a session, then iterates the resources. For each it builds the concrete resource through the ResourceFactory, calls Apply or Healthcheck, logs the result, records the event, and publishes any register_when_stable entries. When fail_on_error is set, a failed resource stops the run after the current entry.

Cross-resource behavior is stateful through the session. Because Execute records each event immediately, IsResourceFailed and ShouldRefresh inspect the last event for a reference, so a later resource sees an earlier one’s change. This is again why declaration order matters.

Generating resources with Jet

When resources_jet_file is set instead of inline resources, jetParseManifestResources (resources/apply/jet.go:19) renders a Jet template with the delimiters [[ and ]], chosen so they do not collide with the {{ }} templating used for scalar fields. The resolved Hiera Data is in scope, so a template can loop over Data.packages to emit a resource per item. The rendered YAML is then parsed exactly like inline resources. Jet generates the resource list; ordinary templating fills in individual fields.

Nested applies

The apply resource type lets one manifest apply another. Its sole provider, ccmmanifest.Provider (resources/applyresource/ccmmanifest/ccmmanifest.go:31), snapshots the manager’s noop, data, and working directory, runs the child inside the parent’s session so events are shared, and restores the manager afterward.

Recursion is bounded by DefaultMaxRecursionDepth = 10. Note that the depth guard is not fully threaded through the apply-resource boundary today: Type.ApplyResource calls ApplyManifest with a hardcoded depth of 0, so the guard is effectively aspirational for deeply nested manifests. Two TODOs in the executor also flag that a resource-factory error currently aborts the whole run rather than recording a failed event, and that resource dispatch should move into the registry.

The Agent

The agent runs CCM continuously. It resolves facts and external data, fetches manifests from one or more sources, applies them on an interval, and runs health checks that can trigger a remediating apply between intervals. It is a single scheduler driving many per-source workers.

One scheduler, many workers

The Agent (agent/agent.go:30) creates one worker per manifest. Each worker only watches its own source and requests an apply; it never schedules one. A single applyTicker in Run drives every scheduled apply, which serializes fact refreshes and prevents two manifests from applying at once. Workers signal the loop through a buffered-size-one applyTrigger channel, and the send is coalesced so a burst of source changes collapses into one priority apply.

DefaultInterval

5 minutes. The apply cadence, floored at MinInterval of 30 seconds.

MinFactUpdateInterval

2 minutes. Facts are not re-gathered more often than this, independent of the apply interval.

applyTrigger

Buffered channel of size one. A worker whose source changed pushes a priority apply that runs even inside the interval window.

Sources

Dispatched by scheme in worker.cacheManifest: obj:// to a JetStream object store watcher, http(s) to a conditional-GET fetcher, empty scheme to a local file.

The loop

Facts, data, and resilience

Facts and data resolution sit behind a mutex and a retry policy. getFacts skips entirely when the cached facts are younger than the 2-minute floor. Otherwise it retries under jittered backoff, and after a configured number of failures it falls back to the last good facts rather than blocking the loop. getData resolves external data through Hiera on each cycle and falls back the same way. This is why a transient NATS or HTTP outage does not stall applies: the agent keeps running on the last known-good inputs.

Health checks are deliberately independent of applies. runHealthChecks runs each worker in health-check-only mode and does not refresh facts or data. A worker reporting a critical result increments a remediation counter and queues a priority apply, but the queued applies fire only after every check completes, so applies and checks never interleave.

Metrics and shutdown

When a monitor port is configured, the agent registers Prometheus collectors and serves /metrics. It exposes apply and health-check durations, remediation counts, manifest fetch counts and errors, and facts and data resolve timing. Shutdown is graceful: Run returns on context cancellation after waiting for the workers, and Stop closes the manager. Workers use a cancel-with-cause context, so a manifest deleted from an object store propagates a readable reason.

Two items are reserved rather than active. AgentHealthCheckTime is registered but never observed, so the health-check duration series stays empty. A TODO in agent.go notes the intent to watch the KV for external data and only re-fetch on change, rather than re-resolving every cycle.

Data, Facts, and Templates

Three cooperating packages feed every run. facts gathers what is true about the host. hiera layers a hierarchy of data blocks into one resolved map, driven by those facts. templates renders that data into resource fields. The manager wires them together and hands the result to the apply engine.

Facts

facts.Gather (facts/facts.go:17) builds a map from the built-in families, host, network, partition, cpu, and memory, each backed by gopsutil and each skippable with a config flag. It then merges file-based facts on top: for the system config directory and then the user directory, it reads facts.json, facts.yaml, and a sorted facts.d/ directory, deep-merging each in order so later sources win.

Runtime caching lives in the manager, not the facts package. Facts() gathers once and memoizes; SystemFacts() always re-gathers with a 2-second default deadline; SetFacts and MergeFacts let callers override or overlay, which is how the agent reuses its last good facts when a gather fails.

Hiera

Resolve (hiera/resolver.go:98) starts from the data: block as the base, then walks hierarchy.order. Each level is a template like os:{{ lookup('facts.host.info.platformFamily') }}. A level counts as matched only when its embedded expression produces a non-empty value, so a level whose fact is missing is skipped. The matched string is looked up in overrides: and merged into the base.

merge: first

The default. The first matching level wins; its top-level keys replace the base, and resolution returns immediately.

merge: deep

Every matching level accumulates. Maps merge recursively and slices concatenate, in hierarchy order.

Sources

ResolveUrl dispatches by scheme: a local YAML or JSON file, an http(s) URL with basic auth, or a NATS JetStream KV document at kv://Bucket/Key.

Validation

YAML comments @require and @validate <expr> become rules. Resolution returns the rules so a multi-source caller validates once after merging.

Templates

{{ expr }} and ${ expr } are aliases. Both route through the same expr-lang evaluator, so ${ Data.package_name } and {{ Data.package_name }} are identical. A separate jet(...) renderer handles Jet templates with [[ ]] delimiters, used mainly to generate resource lists.

Field access

Inside an expression, use Go field names: Data.app_name, Facts.os, Environ.X.

lookup(key, default)

Takes a lowercased, dotted path rooted at the environment: lookup('data.nested.key'), lookup('facts.env'). It marshals the environment to JSON and queries it with gjson.

Type preservation

When the whole string is a single expression, its native typed value is returned, so a templated port stays an integer. Mixed strings are stringified and concatenated.

Deferred fields

Struct fields tagged template:"deferred", such as file content, render in a second pass after the control gate, so an unless can protect a resource from a template error.

One syntax is reserved but not implemented. The Resolve doc comment and a test reference Puppet-style env:%{env} placeholders, but only {{ }} and ${ } are recognized. A %{...} entry is treated as a literal key with no matched expression, so it is skipped. Real hierarchies use env:{{ lookup('facts.env') }}.

Registration and Discovery

Registration is service discovery without a discovery daemon. When a managed resource reaches a stable state, it publishes an entry into a NATS JetStream stream. Other nodes read those entries, usually from a template, to build configuration such as a load-balancer upstream or a Prometheus target list. Publishing is a side effect of apply; lookup is a template function or a CLI query.

The publish gate

A resource carries a register_when_stable list. After the resource applies, publishRegistration (resources/apply/apply.go:736) decides whether to announce it. The gate is conservative: it publishes only when the resource has entries, the manager is not in noop mode, the event did not fail, and every health check returned OK. A resource with no health checks passes that last test.

A KV store built from a stream

Each entry maps to a single subject. The address contributes its IP with dots turned into underscores so it is one token, and an InstanceId FNV hash of the full tuple forms the final token, so every unique cluster, service, protocol, address, and port maps to one stable subject. The stream captures the whole namespace with MaxMsgsPerSubject: 1 and AllowRollup, and reliable publishes carry a Nats-Rollup: sub header.

Expiry, lookup, and watch

Expiry is two-tiered. A per-entry Nats-TTL header sits under the stream-wide MaxAge, and on removal the server writes a subject delete marker so watchers see the removal rather than a silent disappearance. The stream denies hard deletes but allows purge, so JetStreamRemove can retire a single entry.

Lookup

JetStreamLookup builds a wildcard filter subject, fetches the last message per matching subject, drops server marker messages, and returns entries sorted by address then port.

Watch

JetStreamWatch streams WatchEvents: normal messages become Register events, marker messages become Remove events with the entry reconstructed from the subject.

From a template

The manager exposes lookup as registrations(cluster, protocol, service, ip), so a rendered config file discovers its peers directly.

Prometheus

RegistrationEntries.PrometheusFileSD() converts entries into file_sd JSON, grouped by cluster, service, and protocol.

The Priority and Annotations fields travel on every entry but, within this subsystem, are only consumed by the Prometheus conversion. Priority is validated and sorted around but not otherwise acted on here, reserved for downstream consumers such as SRV-style weighting.

Observability

Every applied resource produces one event. That event is recorded to a session store, translated into Prometheus counters, and aggregated into a summary at the end of the run. Health checks run alongside and feed the agent’s remediation. Together these are how a CCM run is observed after the fact and monitored while it runs.

The event flow

Sessions and events

StartSession records a start event, each applied resource records a TransactionEvent, and StopSession builds a SessionSummary from all events. The memory store keeps events in a slice and is the default. The directory store writes one JSON file per event, named by the event’s KSUID.

The directory store is append-only and crash-friendly. Each record is an independent file write with no shared index to corrupt, and replay is a directory scan that reads each event’s protocol field to pick the concrete type before unmarshaling.

Metrics

Collectors live under the choria namespace and ccm subsystem. updateMetrics (internal/session/util.go:12) runs from both stores’ RecordEvent. It increments a total and exactly one outcome counter, chosen by a priority order: noop, then changed, skipped, refreshed, failed, error, and stable.

Durations are recorded with timers around the work: manifest apply, resource apply, fact gather, and per-check health-check time. RegisterMetrics registers the collectors and ListenAndServe serves them at /metrics when a port is set.

Health checks

Health checks run for both apply and health-check-only modes. Each check dispatches by format.

nagios

Runs a command through the manager's runner and maps its exit code to a status: 0 is OK, 1 Warning, 2 Critical, anything else Unknown. Retries up to tries, sleeping between attempts, stopping on OK.

goss

Renders the goss rules through the template engine, writes a temp spec, validates it, and reports Critical when any check failed, else OK. It never emits Warning or Unknown.

Status

HealthCheckStatus values equal the nagios exit codes, so the plugin format maps directly and goss reuses the same enum.

A non-OK result or an execution error marks the event failed. In the agent, a critical result increments a remediation counter and queues a priority apply, so a failing check drives a corrective run. Several Agent* metrics are declared here but recorded at their call sites in the agent, and AgentHealthCheckTime is currently registered without a call site.

Reference and Map

This page is the index to the codebase: the commands a user can run, the packages that back them, and the types that recur across the subsystem pages.

Command surface

Every ccm command builds a manager and drives a subsystem covered elsewhere in this map. The global flags --debug and --info set log verbosity.

Every ensure subcommand and the piped API funnel through one factory, resources.NewResourceFromProperties, so the CLI, manifests, and the wire API run identical resource logic. The status command’s valid types are generated at run time from the registry, so it tracks whatever providers are registered.

Source map

Key types

Glossary

Resource

A system component to manage: a package, file, service, exec, archive, or scaffold. Declares a desired state.

Provider

The platform-specific implementation for a resource type, such as apt, dnf, systemd, or posix. Selected at run time by facts.

Ensure

The desired state of a resource, such as present, absent, running, or a package version.

Manifest

A YAML document of data, a hierarchy, and a list of resources, applied as a unit.

Facts

Gathered truths about the host, used to select providers and drive the data hierarchy.

Hiera

The hierarchical data resolver that merges layered data blocks, chosen by facts, into one resolved map.

Session

The record of a run: a sequence of events, stored in memory or on disk, that drives require and subscribe.

Noop

Dry-run mode. No change is made, but the event still reports what would have changed.

Require

A fail-gate: a resource is skipped if a named prior resource failed. Not a scheduler.

Subscribe / refresh

A resource, in practice a service, restarts when a named prior resource changed.

register_when_stable

A resource property that publishes a discovery entry once the resource is stable and healthy.