Resources

Resources describe the desired state of your infrastructure. Each resource represents something to manage and is backed by a provider that implements platform-specific management logic.

Every resource has a type, a unique name, and resource-specific properties.

Resource types

• Apply: Compose manifests from smaller reusable manifests
• Archive: Download, extract and copy files from tar.gz and zip archives
• Exec: Execute commands idempotently
• File: Manage files content, ownership and more
• Package: Install, upgrade, downgrade and remove packages using OS native packagers
• Scaffold: Render template directories into target directories with synchronization and purge support
• Service: Enable, disable, start, stop and restart services using OS native service managers

Common properties

All resources support the following common properties:

Conditional resource execution

Resources can be conditionally executed using a control section and expressions that should resolve to boolean values.

package:
  name: zsh
  ensure: 5.9
  control:
    if: lookup("facts.host.info.os") == "linux"
    unless: lookup("facts.host.info.virtualizationSystem") == "docker"

ccm ensure package zsh 5.9 \
    --if "lookup('facts.host.info.os') == 'linux'"
    --unless "lookup('facts.host.info.virtualizationSystem') == 'docker'"

{
  "protocol": "io.choria.ccm.v1.resource.ensure.request",
  "type": "package",
  "properties": {
    "name": "zsh",
    "ensure": "5.9",
    "control": {
      "if": "lookup(\"facts.host.info.os\") == \"linux\"",
      "unless": "lookup(\"facts.host.info.virtualizationSystem\") == \"docker\""
    }
  }
}

This installs zsh on all linux machines unless they are running inside a docker container.

The following table shows how the two conditions interact:

Hierarchical Data

The Choria Hierarchical Data Resolver is a small data resolver inspired by Hiera. It evaluates a YAML or JSON document alongside a set of facts to produce a final data map.

The resolver supports first and deep merge strategies and relies on expression-based string interpolation for hierarchy entries. It is optimized for single files that hold both hierarchy and data, rather than the multi-file approach common in Hiera.

Key features:

• Lookup expressions based on the Expr Language
• Type-preserving lookups (returns typed data, not just strings)
• Command-line tool with built-in system facts
• Go library for embedding

Usage

An annotated example:

hierarchy:
    # Lookup and override order - facts are resolved in expressions
    # Use GJSON path syntax for nested facts: ${ lookup('facts.host.info.hostname') }
    order:
     - env:${ lookup('facts.env') }
     - role:${ lookup('facts.role') }
     - host:${ lookup('facts.hostname') }
    merge: deep  # "deep" merges all matches; "first" stops at first match

# Base data - hierarchy results are merged into this
data:
   log_level: INFO
   packages:
     - ca-certificates
   web:
     listen_port: 80
     tls: false

# Override sections keyed by hierarchy order entries
overrides:
    env:prod:
      log_level: WARN

    role:web:
      packages:
        - nginx
      web:
        listen_port: 443
        tls: true

    host:web01:
      log_level: TRACE

The templating here is identical to that in the Template documentation, except only the lookup() function is available (no file access functions).

CLI example

The ccm hiera command resolves hierarchy files with facts. It is designed to be a generally usable tool, with flexible options for providing facts.

Given the input file data.json:

{
    "hierarchy": {
        "order": [
            "fqdn:${ lookup('facts.fqdn') }"
        ]
    },
    "data": {
        "test": "value"
    },
    "overrides": {
        "fqdn:my.fqdn.com": {
            "test": "override"
        }
    }
}

Resolve with facts provided on the command line:

$ ccm hiera parse data.json fqdn=my.fqdn.com
{
  "test": "override"
}

$ ccm hiera parse data.json fqdn=other.fqdn.com
{
  "test": "value"
}

Output formats:

# YAML output
$ ccm hiera parse data.json fqdn=other.fqdn.com --yaml
test: value

# Environment variable output
$ ccm hiera parse data.json fqdn=other.fqdn.com --env
HIERA_TEST=value

Fact sources

Facts can come from multiple sources, which are merged together.

System facts

Use -S or --system-facts:

# View system facts
$ ccm hiera facts -S

# Resolve using system facts
$ ccm hiera parse data.json -S

Environment variables as facts

Use -E or --env-facts:

ccm hiera parse data.json -E

Facts file

Use --facts FILE:

ccm hiera parse data.json --facts facts.yaml

Command-line facts

Pass key=value pairs as positional arguments:

ccm hiera parse data.json env=prod role=web

All fact sources can be combined. Command-line facts take highest precedence.

Data in NATS

NATS is a lightweight messaging system that supports Key-Value stores. Hierarchy data can be stored in NATS and used with ccm ensure and ccm hiera commands.

To use NATS as a hierarchy store, configure a NATS context for authentication:

nats context add ccm --server nats.example.org --user ccm --password s3cret \
  --description "CCM Configuration Store"

Create a KV store and add your hierarchy data:

$ nats kv add CCM --replicas 3 --context ccm
$ nats kv put CCM data "$(cat hiera.yaml)"

Resolve the hierarchy using the KV store:

ccm hiera parse kv://CCM/data --context ccm -S

Data on web servers

Hierarchy data can also be stored on a web server and fetched via HTTP or HTTPS.

ccm hiera parse https://example.net/site.yaml -S

HTTP Basic Auth is supported via URL credentials:

ccm hiera parse https://user:pass@example.net/site.yaml -S

Data annotations

YAML comments in the data: section can carry annotation directives that validate resolved values. Annotations are always active; there is no opt-in flag.

Two directives are supported:

@required is accepted as an alias for @require.

Example

data:
  # The username to run the process as
  # @require
  user: bob

  # @validate isShellSafe(value)
  command: "/usr/bin/thing"

  # @require
  # @validate isIPv4(value)
  listen_address: 10.0.0.1

  # No annotations, no validation
  port: 8080
  debug: false

Multiple annotations per key are supported. @require is checked first; when it fails on a nil value, @validate is skipped for that key.

Validation expressions

The @validate directive accepts Expr Language expressions evaluated by the choria-io/validator library. The value being validated is available as value in the expression.

Available expressions:

Multiple expressions can be combined using Expr operators: isIPv4(value) || isIPv6(value).

Non-string values are converted to their string representation before validation.

Behavior

Annotations are extracted from the data: section only. Comments in overrides: are ignored.

Validation runs against the final merged data. A base value of user: "" with @require passes when an override provides a non-empty value.

@validate on map values is skipped. Maps are validated by annotating their nested keys individually. @validate on array values is skipped.

Unrecognized directives starting with @ produce a warning, catching typos like @requiired.

JSON data sources do not support annotations because JSON has no comment syntax.

Templates

Applications need data to vary their behavior and configure their environments. Configuration management tools are no different.

Data can be used for:

• Configuring resource names that differ between operating systems (e.g., httpd vs apache2)
• Setting different configuration values depending on environment, role, or other dimensions
• Deciding whether environments should have something installed (e.g., development vs production)

CCM supports various data sources:

• System Facts - Operating system, networking, and disk configuration
• Custom Facts - From facts.{yaml,json} and facts.d/*.{yaml,json} in system and user config directories
• Environment - Variables from the shell environment and ./.env files
• Hiera Data - Hierarchical data with overrides based on facts

Accessing data

Expressions like {{ lookup('facts.host.info.platformFamily') }} or ${ lookup('facts.host.info.platformFamily') } use the Expr Language.

Available variables

In templates, you have direct access to:

Available functions

GJSON path examples

The lookup() function uses GJSON path syntax for nested access:

lookup("facts.host.info.platformFamily")     # Simple nested path
lookup("facts.network.interfaces.0.name")    # Array index
lookup("data.packages.#")                    # Array length

CLI usage

These expressions work on the CLI:

$ ccm ensure package '${ lookup("data.package_name", "httpd") }'

It might be easier to avoid using $ on the CLI so the alternative syntax is helpful:

$ ccm ensure package '{{ lookup("data.package_name", "httpd") }}'

This fetches package_name from the data and defaults to httpd if not found.

Facts

CCM includes a built-in fact resolver that gathers system information. To see available facts:

$ ccm facts                              # All facts as JSON
$ ccm facts host                         # Query specific path
$ ccm facts --yaml                       # Output as YAML

Access facts in expressions using ${ Facts.host.info.platformFamily } or ${ lookup('facts.host.info. platformFamily') }.

Custom facts

Custom facts are loaded from the system (/etc/choria/ccm/) and user (~/.config/choria/ccm/) configuration directories. Within each directory, facts are loaded in this order:

1. facts.json
2. facts.yaml
3. facts.d/*.{json,yaml} - files sorted by filename

Later sources override earlier ones, and user directory facts override system directory facts. This makes facts.d/ useful for modular or drop-in facts from other tools.

Files in facts.d/ are processed in lexicographic filename order, so 01-base.json is loaded before 02-override.json. Only files with .json or .yaml extensions are read; all other files (including .yml) are ignored.

Security

Facts directories are subject to the following security constraints:

• Absolute paths only - configuration directories must be absolute paths; relative paths are rejected
• Path cleaning - paths are normalized to remove traversal components (e.g., /../)
• Symlinks ignored - symlinked files, symlinked facts.d/ directories, and symlinked entries within facts.d/ are all skipped

Hiera data for CLI

Hiera data is resolved using the Choria Hierarchical Data Resolver. By default, data is read from ./.hiera, or you can specify a file with --hiera.

Hiera data sources

Hiera data can be loaded from:

• Local file: ./.hiera or path specified with --hiera
• Key-Value store: --hiera kv://BUCKET/key (requires --context for NATS)
• HTTP(S): --hiera https://example.com/data.yaml (supports Basic Auth via URL credentials)

Merge strategies

The hierarchy.merge setting controls how overrides are applied:

• first (default): Stops at the first matching override
• deep: Deep merges all matching overrides in order

Example

Given this file stored in ./.hiera:

hierarchy:
  order:
    - os:${ lookup('facts.host.info.platformFamily') }
  merge: first

data:
  package_name: ""

overrides:
  os:debian:
    package_name: apache2

  os:rhel:
    package_name: httpd

Running ccm ensure package '${ lookup("data.package_name") }' installs httpd on RHEL-based systems and apache2 on Debian-based systems.

Environment

The shell environment and variables defined in ./.env can be accessed in two ways:

# Direct access
home_dir: "${ Environ.HOME }"

# Via lookup (with default value)
my_var: "${ lookup('environ.MY_VAR', 'default') }"

The .env file uses standard KEY=value format, one variable per line.

Shell Usage

CCM is designed as a CLI-first tool. Each resource type has its own subcommand under ccm ensure, with required inputs as arguments and optional settings as flags.

The ccm ensure commands are idempotent, making them safe to run multiple times in shell scripts.

Use ccm --help and ccm <command> --help to explore available commands and options.

Managing a single resource

Managing a single resource is straightforward.

ccm ensure package zsh 5.8

This ensures the package zsh is installed at version 5.8.

To view the current state of a resource:

ccm status package zsh

Managing multiple resources

When managing multiple resources in a script, create a session first. The session records the outcome of each resource, enabling features like refreshing a service when a file changes.

#!/bin/bash

eval "$(ccm session new)"

ccm ensure package httpd
ccm ensure file /etc/httpd/conf/httpd.conf .... --require package#httpd
ccm ensure service httpd --subscribe file#/etc/httpd/conf/httpd.conf
ccm session report --remove

This creates a temporary directory for session state. If the file resource changes, the service restarts automatically.

To avoid eval, use: export CCM_SESSION_STORE=$(mktemp -d)

Data in the CLI

As covered in the templates section, commands automatically read ./.env and ./.hiera files, merging that data into the session.

Use the --hiera flag or CCM_HIERA_DATA environment variable to specify a different data file.

With data loaded, you can access:

• {{ lookup("data.my_data_key") }} or ${ lookup("data.my_data_key") } for Hiera data
• {{ lookup("env.MY_ENV_VAR") }} or ${ lookup("env.MY_ENV_VAR") } for environment variables
• {{ lookup("facts.host.info.platformFamily") }} or ${ lookup("facts.host.info.platformFamily") }} for system facts

Example using Hiera data:

ccm ensure package '{{ lookup("data.package") }}' '{{ lookup("data.version") }}'

Expressions use the Expr Language, enabling complex logic:

$ ccm ensure package '{{ lookup("facts.host.info.platformFamily") == "rhel" ? "httpd" : "apache2" }}'
WARN  package#httpd changed ensure=present runtime=14.509s provider=dnf

For complex conditional logic, use Hiera data with hierarchy overrides instead.

Applying manifests

For non-shell script usage use YAML manifests with ccm apply:

See YAML Manifests for manifest format details.

Viewing system facts

CCM gathers system facts that can be used in templates and conditions:

# Show all facts as JSON
$ ccm facts

# Show facts as YAML
$ ccm facts --yaml

# Query specific facts using gjson syntax
$ ccm facts host.info.platformFamily

Resolving Hiera data

The ccm hiera command helps debug and test Hiera data resolution:

# Resolve a Hiera file with system facts
$ ccm hiera parse data.yaml -S

# Resolve with custom facts
$ ccm hiera parse data.yaml os=linux env=production

# Query a specific key from the result
$ ccm hiera parse data.yaml --query packages

YAML Manifests

A manifest is a YAML file that combines data, hierarchy configuration, and resources in a single file.

Manifests support template expressions but not procedural logic. Think of them as declarative configuration similar to multi-resource shell scripts.

A visual editor for manifests is available at CCM Studio.

Manifest structure

A manifest contains these top-level sections:

The manifest is resolved using the Choria Hierarchical Data Resolver.

Full example

Input data

Define input data like parameters for a module:

data:
  package_name: "httpd"

Resources

Define resources to manage in the ccm section:

ccm:
  resources:
    - package:
        - "${ lookup('data.package_name') }":
            ensure: latest

Configure hierarchy

Configure a hierarchy to vary data by dimensions like OS platform:

hierarchy:
  order:
    - os:${ lookup('facts.host.info.platformFamily') }

This looks for overrides in os:rhel, os:debian, etc.

Overrides

Provide platform-specific data overrides:

overrides:
  os:debian:
    package_name: apache2

Applying manifests

The complete manifest:

data:
  package_name: "httpd"

ccm:
  resources:
    - package:
        - "${ lookup('data.package_name') }":
             ensure: latest

hierarchy:
  order:
    - os:${ lookup('facts.host.info.platformFamily') }

overrides:
  os:debian:
    package_name: apache2

Apply the manifest with ccm apply. The first run makes changes; subsequent runs are stable:

$ ccm apply manifest.yaml
INFO  Creating new session record resources=1
WARN  package#httpd changed ensure=latest runtime=3.560699287s provider=dnf

$ ccm apply manifest.yaml
INFO  Creating new session record resources=1
INFO  package#httpd stable ensure=latest runtime=293.448824ms provider=dnf

To preview the fully resolved manifest without applying it:

$ ccm apply manifest.yaml --render
data:
  package_name: apache2
resources:
- package:
    - apache2:
        ensure: latest

Pre and post messages

Display messages before and after manifest execution:

ccm:
  pre_message: |
    Starting configuration update...
  post_message: |
    Configuration complete.
  resources:
    - ...

Both are optional.

Overriding data

Override or augment manifest data with an external Hiera source:

ccm apply manifest.yaml --hiera kv://CCM/common

Supported sources include local files, KV stores (kv://), and HTTP(S) URLs.

Setting defaults

Reduce repetition by setting defaults for resources of the same type:

ccm:
  resources:
    - file:
        - defaults:
            owner: app
            group: app
            mode: "0644"
            ensure: present
        - /app/config/file.conf:
            source: file.conf
        - /app/bin/app:
            source: app.bin
            mode: "0700"

    - file:
        - /etc/motd:
            ensure: present
            source: motd.txt
            owner: root
            group: root
            mode: "0644"

The first two files inherit the defaults values. The /app/bin/app file overrides just the mode. The /etc/motd file is a separate resource block, so defaults do not apply.

Templating

Manifests support template expressions like ${ lookup("key") } for adjusting values. These expressions cannot generate new resources; they only modify values in valid YAML.

Available variables

Templates have access to:

Generating resources with Jet templates

To dynamically generate resources from data, use Jet Templates.

Given this data:

data:
  common_packages:
    - zsh
    - vim-enhanced
    - nagios-plugins-all
    - tar
    - nmap-ncat

Reference a Jet template file instead of inline resources:

ccm:
  resources_jet_file: resources.jet

Create the template file:

{* resources.jet *}
[[ range Data.common_packages ]]
- package:
    - [[ . ]]:
        ensure: present
[[ end ]]

The template receives the fully resolved Hiera data, plus Facts and Environ.

Failure handling

By default, if a resource fails, the apply continues to the next resource.

Fail on error

To stop execution on the first failure, set fail_on_error:

ccm:
  fail_on_error: true
  resources:
    - exec:
        - /usr/bin/false: {}
        - /usr/bin/true: {}

The second resource is never executed:

$ ccm apply manifest.yaml
INFO  Executing manifest manifest=manifest.yaml resources=2
ERROR exec#/usr/bin/false failed ensure=present runtime=3ms provider=posix errors=failed to reach desired state exit code 1
WARN  Terminating manifest execution due to failed resource

Resource dependencies

Use the require property to ensure a resource only runs after its dependencies succeed:

ccm:
  resources:
    - package:
        - httpd:
            ensure: present
    - file:
        - /etc/httpd/conf.d/custom.conf:
            ensure: present
            content: "Listen 8080"
            owner: root
            group: root
            mode: "0644"
            require:
              - package#httpd

If the required resource fails, the dependent resource is skipped.

Dry run (noop mode)

Preview changes without applying them:

ccm apply manifest.yaml --noop

Health check only mode

Run only health checks without applying resources:

ccm apply manifest.yaml --monitor-only

This is useful for verifying system state without making changes.

Manifests in NATS object store

Manifests can be stored in NATS Object Stores, avoiding the need to distribute files locally.

Configure a NATS context for authentication:

nats context add ccm --server nats.example.org --user ccm --password s3cret \
  --description "CCM Configuration Store"

Create an Object Store:

nats obj add CCM --replicas 3 --context ccm

Create a manifest with supporting files:

$ mkdir /tmp/manifest
$ echo 'CCM Managed' > /tmp/manifest/motd
$ cat > /tmp/manifest/manifest.yaml << 'EOF'
ccm:
  resources:
    - file:
        - /etc/motd:
            ensure: present
            source: motd
            owner: root
            group: root
            mode: "0644"
EOF

Package and upload to the Object Store:

$ tar -C /tmp/manifest/ -cvzf /tmp/manifest.tgz .
$ nats obj put CCM manifest.tgz --context ccm

Apply the manifest:

$ ccm apply obj://CCM/manifest.tgz --context ccm
INFO  Using manifest from Object Store in temporary directory bucket=CCM file=manifest.tgz
INFO  file#/etc/motd stable ensure=present runtime=0s provider=posix

Manifests on web servers

Store gzipped tar archives on a web server and apply them directly:

$ ccm apply https://example.net/manifest.tar.gz
INFO  Executing manifest manifest=https://example.net/manifest.tar.gz resources=1
INFO  file#/etc/motd stable ensure=present runtime=0s provider=posix

HTTP Basic Auth is supported via URL credentials:

ccm apply https://user:pass@example.net/manifest.tar.gz

Additional facts

Provide additional facts from the command line or a file:

# Individual facts
$ ccm apply manifest.yaml --fact env=production --fact region=us-east

# Facts from a file
$ ccm apply manifest.yaml --facts custom-facts.yaml

Facts from these sources are merged with system facts, with command-line facts taking precedence.

Agent

Some configurations benefit from running YAML Manifests continuously. For example, dotfiles might be left unmanaged (allowing local modifications), while Docker should always be up to date.

The CCM Agent runs manifests continuously, loading them from local files, Object Storage, or HTTP(S) URLs, with Key-Value data overlaid.

The Agent also supports Registration, a service discovery system where resources that reach a stable state publish their details to NATS. Other nodes can query the registry in templates to build configurations dynamically.

Run modes

The agent supports two modes of operation that combine to be efficient and fast-reacting:

• Full manifest apply: Manages the complete state of every resource
• Health check mode: Runs only Monitoring checks, which can trigger a full manifest apply as remediation

By enabling both modes, you can run health checks very frequently (even at 10- or 20-second intervals) while keeping full Configuration Management runs less frequent (every few hours).

Enabling both modes is optional but recommended. Adding health checks to key resources is also recommended.

Supported manifest and data sources

Manifest sources

Manifests can be loaded from:

• Local file: /path/to/manifest.yaml
• Object Storage: obj://bucket/key.tar.gz
• HTTP(S): https://example.com/manifest.tar.gz (supports Basic Auth via URL credentials)

Remote sources (Object Storage and HTTP) must be .tar.gz archives containing a manifest.yaml file, templates and file sources.

External data sources

For Hiera data resolution, the agent supports:

• Local file: file:///path/to/data.yaml
• Key-Value store: kv://bucket/key
• HTTP(S): https://example.com/data.yaml

Logical flow

The agent continuously runs and manages manifests as follows:

1. At startup, the agent fetches data and gathers facts
2. Starts a worker for each manifest source
   1. Each worker starts watchers to download and manage the manifest (polling every 30 seconds for remote sources)
3. Triggers workers at the configured interval for a full apply
   1. Each run updates facts (minimum 2-minute interval) and data
   2. Applies each manifest serially
4. Triggers workers at the configured health check interval
   1. Health check runs do not update facts or data
   2. Runs health checks for each manifest serially
   3. If any health checks are critical (not warning), the agent triggers a full apply for that worker

In the background, object stores and HTTP sources are watched for changes. Updates trigger immediate apply runs with exponential backoff retry on failures.

Prometheus metrics

When monitor_port is configured, the agent exposes Prometheus metrics on /metrics. These metrics can be used to monitor agent health, track resource states and events, and observe health check statuses.

Agent metrics

Resource metrics

Health check metrics

Facts metrics

Configuration

The agent is included in the ccm binary. To use it, create a configuration file and enable the systemd service.

The configuration file is located at /etc/choria/ccm/config.yaml:

# CCM Agent Configuration Example

# Time between scheduled manifest apply runs.
# Must be at least 30s. Defaults to 5m.
interval: 5m

# Time between health check runs.
# When set, health checks run independently of apply runs and can trigger
# remediation applies when critical issues are detected.
# Omit to disable periodic health checks.
health_check_interval: 1m

# List of manifest sources to apply. Each source creates a separate worker.
# Supported formats:
#   - Local file: /path/to/manifest.yaml
#   - Object store: obj://bucket/key.tar.gz
#   - HTTP(S): https://example.com/manifest.tar.gz
#   - HTTP with Basic Auth: https://user:pass@example.com/manifest.tar.gz
# Remote sources must be .tar.gz archives containing a manifest.yaml file.
manifests:
  - /etc/choria/ccm/manifests/base.yaml
  - obj://ccm-manifests/app.tar.gz
  - https://config.example.com/manifests/web.tar.gz

# Logging level: debug, info, warn, error
log_level: info

# NATS context for authentication. Defaults to 'CCM'.
nats_context: CCM

# Optional URL for external Hiera data resolution.
# Supported formats: file://, kv://, http(s)://
# The resolved data is merged into the manifest data context.
external_data_url: kv://ccm-data/common

# Directory for caching remote manifest sources.
# Defaults to /etc/choria/ccm/source.
cache_dir: /etc/choria/ccm/source

# Port for Prometheus metrics endpoint (/metrics).
# Set to 0 or omit to disable.
monitor_port: 9100

# Registration destination for service discovery.
# Valid values: "nats" (fire-and-forget) or "jetstream" (reliable with rollup).
# Omit to disable registration. See the Registration section for details.
# registration: jetstream

After configuring, start the service:

ccm ensure service ccm-agent running --enable

Registration

The registration system publishes service discovery entries to NATS when managed resources reach a stable state. Resources that pass all health checks and are not in a failed state announce themselves to a shared registry. Other nodes discover these services dynamically through template lookups.

Use cases

• Dynamic load balancer configuration - Web server resources register themselves on successful deploy, and a file resource on the load balancer node uses template lookups to generate an upstream configuration
• Service mesh discovery - Database and cache resources register their addresses and ports, allowing application nodes to build connection strings from live registry data
• Canary deployments - New service instances register with lower priority values, allowing consumers to prefer established instances while gradually shifting traffic

Resource configuration

Any resource type can publish registration entries by adding register_when_stable to its properties. Each entry describes a service endpoint to advertise.

resources:
  - service:
      nginx:
        ensure: running
        enable: true
        health_checks:
          - check: exec
            command: curl -sf http://localhost:80/health
        register_when_stable:
          - cluster: production
            service: web
            protocol: http
            address: "{{ Facts.network.primaryIP }}"
            port: 80
            priority: 10
            ttl: 10m
            annotations:
              hostname: "{{ Facts.host.info.hostname }}"
              version: "1.2.3"

This manifest ensures Nginx is running, verifies it responds to health checks, and then registers the node as a web service in the production cluster.

Entry properties

The cluster, address, port, and annotations fields support template expressions for dynamic values resolved at apply time.

How it works

Registration entries are published after a resource is applied. The following conditions must all be met:

1. The resource has one or more register_when_stable entries configured
2. The resource is not in noop mode
3. The resource apply did not fail
4. All health checks (if any) passed with OK status

Each entry is published to a NATS subject with the structure and may be persisted in a NATS stream.

Prerequisites

NATS stream

When using the jetstream destination, a JetStream stream must exist before registration entries can be published. Use ccm registration init to create or update the stream:

ccm registration init --replicas 3 --max-age 5m

This creates a stream named REGISTRATION (or updates it if it already exists) with the correct subject filter, rollup, and TTL settings. Use --registration (-R) to specify a different stream name and --context to select a NATS context.

The --max-age value should exceed the agent’s apply or health check interval to prevent entries from expiring between runs. It also controls how long deletion markers for expired entries are retained in the stream.

Agent configuration

Add the registration field to the agent configuration file at /etc/choria/ccm/config.yaml:

registration: jetstream

nats_context: CCM
interval: 5m
manifests:
  - /etc/choria/ccm/manifests/base.yaml

Valid values for registration are nats (core NATS, fire-and-forget) and jetstream (reliable delivery). Omitting the field disables registration.

Template lookups

Other resources can query the registration registry using the registrations() function in templates. This enables dynamic configuration based on what services are currently registered.

The function takes four string arguments and returns an array of matching registration entries. Each entry exposes the struct fields listed in the Template Key column of the Entry Properties table (e.g., Address, Port, Priority).

registrations(cluster, protocol, service, address)

Any argument can be "*" to wildcard that position.

The registrations() function is available in all three template engines.

registrations('production', 'http', 'web', '*')

[[ range _, entry := registrations("production", "http", "web", "*") ]]
  server [[ entry.Address ]]:[[ entry.Port ]] weight [[ 256 - entry.Priority ]]
[[ end ]]

{{ range $entry := registrations "production" "http" "web" "*" }}
  server {{ $entry.Address }}:{{ $entry.Port }}
{{ end }}

Format transformers

The result of registrations() supports transformation methods that convert entries into formats consumed by other systems. These methods are callable from all three template engines.

Prometheus file service discovery

The PrometheusFileSD() method converts registration entries into the Prometheus file-based service discovery JSON format. Entries are grouped by cluster, service, and protocol into target groups.

Entries without a port are excluded from the output.

For entries where the service or protocol is prometheus, entries are included by default and only excluded if the prometheus.io/scrape annotation is explicitly set to something other than "true". For all other entries, the prometheus.io/scrape annotation must be explicitly set to "true" for the entry to be included.

Labels on each target group include cluster, service, protocol, and annotations that have non-empty values, do not start with __, and whose keys match [a-zA-Z_][a-zA-Z0-9_]*. Other annotations are silently skipped.

registrations('production', 'http', 'web', '*').PrometheusFileSD()

[[ registrations("production", "http", "web", "*").PrometheusFileSD() ]]

{{ $r := registrations "production" "http" "web" "*" }}{{ $r.PrometheusFileSD }}

Example output

Given two registered web servers in the production cluster with a prometheus.io/scrape: "true" annotation:

[
  {
    "labels": {
      "cluster": "production",
      "service": "web",
      "protocol": "http"
    },
    "targets": [
      "10.0.0.1:8080",
      "10.0.0.2:8080"
    ]
  }
]

CLI usage

The ccm apply and ccm ensure file commands can query the registration registry by passing the --registration flag with the name of the JetStream stream holding registration data.

Apply a manifest that uses registrations() in its templates:

ccm apply manifest.yaml --registration REGISTRATIONS

Create a file whose content is generated from a template that queries registered services:

ccm ensure file /etc/haproxy/backends.cfg \
  --content-file backends.cfg.templ \
  --registration REGISTRATIONS

A resource can delegate to a template file that calls registrations():

resources:
  - file:
      /etc/haproxy/backends.cfg:
        ensure: present
        content: |
          {{ template("backends.cfg.jet") }}

Creating entries

The create subcommand publishes a single registration entry to the JetStream stream. This is useful for manually registering services or for integration with external provisioning tools.

ccm registration create \
  --cluster production \
  --service web \
  --protocol http \
  --address 10.0.0.5 \
  --port 8080

All entry properties are specified as flags:

Add annotations by repeating the -A flag:

ccm registration create \
  --cluster production \
  --service web \
  --protocol http \
  --address 10.0.0.5 \
  --port 8080 \
  --priority 10 \
  --ttl 5m \
  -A version=1.2.3 \
  -A hostname=web-05

Removing entries

The rm subcommand (alias delete) purges a registration entry from the JetStream stream. All identifying fields must be specified to match the entry to remove.

ccm registration rm \
  --cluster production \
  --service web \
  --protocol http \
  --address 10.0.0.5 \
  --port 8080

The command prompts for confirmation before removing the entry. Use --force to skip the confirmation prompt:

ccm registration rm \
  --cluster production \
  --service web \
  --protocol http \
  --address 10.0.0.5 \
  --port 8080 \
  --force

Querying and watching

The ccm registration command provides tools for inspecting and monitoring registration data from the command line.

Both subcommands accept the same positional arguments for filtering: cluster, protocol, service, and address. All default to * (wildcard). The --context flag (env: NATS_CONTEXT) controls NATS authentication and defaults to CCM. The --registration (-R) flag sets the JetStream stream name and defaults to REGISTRATION.

Query

The query subcommand performs a point-in-time lookup of registered entries.

ccm registration query

This returns all entries across all clusters, protocols, services, and addresses. Filter by positional arguments:

ccm registration query production http web

Output is a human-readable listing grouped by service and cluster. Machine-readable formats are available with --json or --yaml:

ccm registration query production --json
ccm registration query production http web "*" --yaml

Results are sorted by cluster, then protocol, then service.

Watch

The watch subcommand subscribes to the registration stream and displays changes in real time. It runs until interrupted.

ccm registration watch

New and updated entries are logged as info-level messages. Entries removed by TTL expiry, deletion, or purge are logged as warnings including the removal reason.

Filter the watch to specific entries using the same positional arguments:

ccm registration watch production http web

The --json flag outputs each event as a JSON object for integration with other tools.

Full example

This example shows two nodes working together. A web server registers itself, and a load balancer discovers all web servers to generate its configuration.

Web server node

data:
  web_port: 8080

ccm:
  resources:
    - service:
        my-app:
          ensure: running
          enable: true
          health_checks:
            - check: exec
              command: curl -sf http://localhost:8080/health
          register_when_stable:
            - cluster: production
              service: web
              protocol: http
              address: "{{ Facts.network.primaryIP }}"
              port: "{{ Data.web_port }}"
              priority: 10
              ttl: 10m

Load balancer node

ccm:
  resources:
    - file:
        /etc/haproxy/backends.cfg:
          ensure: present
          content: |
            {{ template("backends.cfg.jet") }}

    - exec:
        reload-haproxy:
          command: systemctl reload haproxy
          refresh_only: true
          subscribe:
            - "file#/etc/haproxy/backends.cfg"

With a backends.cfg.jet template:

backend web_servers
  balance roundrobin
[[ range _, entry := registrations("production", "http", "web", "*") ]]
  server [[ entry.Address ]] [[ entry.Address ]]:[[ entry.Port ]] check weight [[ 256 - entry.Priority ]]
[[ end ]]

Each time the agent runs on the load balancer, it queries the registry for all web services and regenerates the HAProxy configuration. The exec resource reloads HAProxy when the configuration file changes.

Monitoring

By default, CCM verifies resource health using the resource’s native state. For example, if a service should be running and systemd reports it as running, CCM considers it healthy.

For deeper validation, all resources support custom health checks. These checks run after a resource is managed and can verify that the resource is functioning correctly, not just present.

Health check properties

Each health check must specify either command or goss_rules – they are mutually exclusive. The format is auto-detected based on which field is set (nagios for command, goss for goss_rules), but can be overridden explicitly.

Nagios format

Health checks using command follow Nagios plugin conventions for exit codes:

Example

- service:
    - httpd:
        ensure: running
        enable: true
        health_checks:
          - name: check_http
            command: |
              /usr/lib64/nagios/plugins/check_http -H localhost -p 80 --expect "Acme Inc"
            tries: 5
            try_sleep: 1s
            timeout: 10s

ccm ensure service httpd running \
    --check '/usr/lib64/nagios/plugins/check_http -H localhost -p 80' \
    --check-tries 5 \
    --check-sleep 1s

This example verifies that the web server responds with content containing “Acme Inc”. If the check fails, it retries up to 5 times with 1 second between attempts.

Goss format

Health checks using goss_rules embed Goss validation rules directly in the manifest. This validates system state, including running services, listening ports, file contents, and HTTP responses, without external check scripts.

The check result is OK when all Goss rules pass, or CRITICAL when any rule fails. See the Goss documentation for the full list of supported resource types and matchers.

Example

- service:
    - httpd:
        ensure: running
        enable: true
        health_checks:
          - name: httpd_health
            goss_rules:
              service:
                httpd:
                  running: true
                  enabled: true
              port:
                tcp:80:
                  listening: true
              http:
                http://localhost:
                  status: 200
                  body:
                    - Acme Inc
            tries: 5
            try_sleep: 1s

This validates that the httpd service is running and enabled, port 80 is listening, and the web server responds with a 200 status containing “Acme Inc”.

Templates in Goss rules

Goss rules are processed through CCM’s own template engine before evaluation. This means you can use the standard {{ }} expression syntax with access to Facts, Data, and Environ, as well as the lookup(), template(), and jet() functions. See the Data section for full details on template syntax.

health_checks:
  - name: app_check
    goss_rules:
      http:
        "http://localhost:{{ Data.app_port }}/health":
          status: 200
          body:
            - "{{ lookup('data.expected_response', 'ok') }}"
      file:
        "{{ Data.config_path }}":
          exists: true

For more complex rules, you can use the template() function with Jet templates:

health_checks:
  - name: app_check
    goss_rules:
      http:
        "http://localhost:{{ template('check_url.jet') }}":
          status: 200

Agent integration

When running the Agent with health_check_interval configured, health checks run independently of full manifest applies. If any health check returns a CRITICAL status, the agent triggers a remediation apply for that manifest.

Design Documents

Design documents provide detailed implementation guidance for CCM’s resource types, providers, and internal components. They are intended for developers contributing to CCM or those seeking to understand specific implementation details.

For end-user documentation on how to use resources, see Resources.

Contents

Each design document covers:

• Purpose and scope: What the component does and its responsibilities
• Architecture: How the component fits into CCM’s overall design
• Implementation details: Key data structures, interfaces, and algorithms
• Provider contracts: Requirements for implementing new providers
• Testing considerations: How to test the component

Available Documents

• Archive Type: Archive resource for downloading and extracting archives
• Apply Type: Apply resource for composing manifests from reusable parts
• Exec Type: Exec resource for command execution
• File Type: File resource for managing files and directories
• Package Type: Package resource for system package management
• Scaffold Type: Scaffold resource for template directory rendering
• Service Type: Service resource for system service management
• Adding a Type: How to add a new resource type to CCM
• Docs Style Guide: Define writing conventions for CCM documentation