Introduction

CCM is a small-scale Configuration Management system designed to meet users where they are - enabling experimentation, R&D, and exploration without the overhead of full-system management while still following sound Configuration Management principles.

We focus on great UX, immediate feedback, and interactive use with minimal friction.

  • Small and Focused

    Embraces the popular package-config-service style of Configuration Management.

    Focused on the needs of a single application or unit of software.

    Minimal design for easy adoption, experimentation, and integration with tools like LLMs.

  • Loves Snowflakes

    Brings true Configuration Management principles to ad-hoc systems, enabling use cases where traditional CM fails.

    Use management bundles in an à la carte fashion to quickly bring experimental infrastructure to a known state.

  • External Data

    Rich, Hierarchical Data and Facts accessible in the command line, scripts, and manifests.

    Democratizes and opens the data that drives systems management to the rest of your stack.

  • No Dependencies

    Zero-dependency binaries, statically linked and fast.

    Easy deployment in any environment without the overhead-cost of traditional CM.

    Embraces a Just Works philosophy with no, or minimal, configuration.

  • Optional Networking

    Optional Network infrastructure needed only when your needs expand.

    Choose from simple webservers to clustered, reliable, Object and Key-Value stores using technology you already know.

  • Everywhere

    Great at the CLI, shell scripts, YAML manifests, Choria Agents, and Go applications.

    Scales from a single IoT Raspberry Pi 3 to millions of nodes.

    Integrates easily with other software via SDK or Unix-like APIs.

Shell Example

Here we do a package-config-service style deployment using a shell script. The script is safe to run multiple times as the CCM commands are all idempotent.

#!/bin/bash

eval $(ccm session new)
ccm ensure package httpd
ccm ensure file /etc/httpd/conf.d/listen.conf content="Listen 8080"
ccm ensure service httpd --subscribe file#/etc/httpd/conf.d/listen.conf
ccm session report --remove

When run, this will create a session in a temporary directory and manage the resources. If the file resource changes after initial deployment, the service will restart.

We support dynamic data on the CLI, ccm will read .env files and .hiera files and feed that into the runtime data. Using this even shell scripts can easily gain access to rich data.

$ cat .env
package_name="httpd"
$ ccm ensure package '{{ Data.package_name }}'
 INFO  package#httpd stable ensure=present runtime=8ms provider=dnf

Manifest Example

Taking the example above, here is what it looks like in a manifest, complete with multi-OS support using Hierarchical Data and System Facts:

data:
  package_name: httpd
  service_name: httpd

ccm:
  resources:
    - package:
        - "{{ Data.package_name }}":
            ensure: present
    - file: 
        - "/etc/httpd/conf.d/listen.conf":
            ensure: present
            content: |
                Listen 8080
    - service:
        - "{{ Data.service_name }}":
            ensure: running
            enable: true
            subscribe: 
              - file#/etc/httpd/conf.d/listen.conf

hierarchy:
  order:
    - os:{{ lookup('facts.host.info.platformFamily') }}

overrides:
  os:debian:
    package_name: apache2
    service_name: apache2

Here we define the inputs in data and the Hiera hierarchy along with OS-specific overrides. The data is referenced in the manifest using the {{ Data.package_name }} syntax.

Status

This is a new project seeking contributors and early adopters.

Currently, only archive, exec, file, service, scaffold and package resources are implemented, with support limited to dnf, apt and systemd.

The CLI and shell interaction have reached a mature state. Next, we’re exploring network-related features and deeper monitoring integration.

Subsections of Introduction

Resources

Resources describe the desired state of your infrastructure. Each resource represents something to manage and is backed by a provider that implements platform-specific management logic.

Every resource has a type, a unique name, and resource-specific properties.

Resource Types

  • Archive - Download, extract and copy files from tar.gz and zip archives
  • Exec - Idempotent command execution
  • File - Manage files content, ownership and more
  • Package - Install, upgrade, downgrade and remove packages using OS Native packagers
  • Scaffold - Render template directories into target directories with synchronization and purge support
  • Service - Enable, disable, start, stop and restart services using OS Native service managers

Common Properties

All resources support the following common properties:

PropertyDescription
nameUnique identifier for the resource
ensureDesired state (values vary by resource type)
aliasAlternative name for use in subscribe, require, and logging
providerForce a specific provider
requireList of resources (type#name or type#alias) that must succeed first
health_checksHealth checks to run after applying (see Monitoring)
controlConditional execution rules (see below)

Conditional Resource Execution

Resources can be conditionally executed using a control section and expressions that should resolve to boolean values.

package:
  name: zsh
  ensure: 5.9
  control:
    if: lookup("facts.host.info.os") == "linux"
    unless: lookup("facts.host.info.virtualizationSystem") == "docker"
ccm ensure package zsh 5.9 \
    --if "lookup('facts.host.info.os') == 'linux'"
    --unless "lookup('facts.host.info.virtualizationSystem') == 'docker'"
{
  "protocol": "io.choria.ccm.v1.resource.ensure.request",
  "type": "package",
  "properties": {
    "name": "zsh",
    "ensure": "5.9",
    "control": {
      "if": "lookup(\"facts.host.info.os\") == \"linux\"",
      "unless": "lookup(\"facts.host.info.virtualizationSystem\") == \"docker\""
    }
  }
}

Here we install zsh on all linux machines unless they are running inside a docker container.

The following table shows how the two conditions interact:

ifunlessResource Managed?
(not set)(not set)Yes
true(not set)Yes
false(not set)No
(not set)trueNo
(not set)falseYes
truetrueNo
truefalseYes
falsetrueNo
falsefalseNo

About Names

Resources can be specified like:

/etc/motd:
  ensure: present

This sets name to /etc/motd, in the following paragraphs we will refer to this as name.

Subsections of Resources

Archive

The archive resource downloads and extracts archives from HTTP/HTTPS URLs. It supports tar.gz, tgz, tar, and zip formats.

Note

The archive file path (name) must have the same archive type extension as the URL. For example, if the URL ends in .tar.gz, the name must also end in .tar.gz.

- archive:
    - /opt/downloads/app-v1.2.3.tar.gz:
        url: https://releases.example.com/app/v1.2.3/app-v1.2.3.tar.gz
        checksum: "a1b2c3d4e5f6..."
        extract_parent: /opt/app
        creates: /opt/app/bin/app
        owner: app
        group: app
        cleanup: true
ccm ensure archive /opt/downloads/app.tar.gz \
    --url https://releases.example.com/app.tar.gz \
    --extract-parent /opt/app \
    --creates /opt/app/bin/app \
    --owner root --group root
{
  "protocol": "io.choria.ccm.v1.resource.ensure.request",
  "type": "archive",
  "properties": {
    "name": "/opt/downloads/app-v1.2.3.tar.gz",
    "url": "https://releases.example.com/app/v1.2.3/app-v1.2.3.tar.gz",
    "checksum": "a1b2c3d4e5f6...",
    "extract_parent": "/opt/app",
    "creates": "/opt/app/bin/app",
    "owner": "app",
    "group": "app",
    "cleanup": true
  }
}

This downloads the archive, extracts it to /opt/app, and removes the archive file after extraction. Future runs skip the download if /opt/app/bin/app exists.

Ensure Values

ValueDescription
presentThe archive must be downloaded
absentThe archive file must not exist

Properties

PropertyDescription
nameAbsolute path where the archive will be saved
urlHTTP/HTTPS URL to download the archive from
checksumExpected SHA256 checksum of the downloaded file
extract_parentDirectory to extract the archive contents into
createsFile path; if this file exists, the archive is not downloaded or extracted
cleanupRemove the archive file after successful extraction (requires extract_parent and creates)
ownerOwner of the downloaded archive file (username)
groupGroup of the downloaded archive file (group name)
usernameUsername for HTTP Basic Authentication
passwordPassword for HTTP Basic Authentication
headersAdditional HTTP headers to send with the request (map of header name to value)
providerForce a specific provider (http only)

Authentication

The archive resource supports two authentication methods:

Basic Authentication:

- archive:
    - /opt/downloads/private-app.tar.gz:
        url: https://private.example.com/app.tar.gz
        username: deploy
        password: "{{ lookup('data.deploy_password') }}"
        extract_parent: /opt/app
        owner: root
        group: root
ccm ensure archive /opt/downloads/private-app.tar.gz \
    --url https://private.example.com/app.tar.gz \
    --username deploy \
    --password "{{ lookup('data.deploy_password') }}"
{
  "protocol": "io.choria.ccm.v1.resource.ensure.request",
  "type": "archive",
  "properties": {
    "name": "/opt/downloads/private-app.tar.gz",
    "url": "https://private.example.com/app.tar.gz",
    "username": "deploy",
    "password": "secret",
    "extract_parent": "/opt/app",
    "owner": "root",
    "group": "root"
  }
}

Custom Headers:

- archive:
    - /opt/downloads/app.tar.gz:
        url: https://api.example.com/releases/app.tar.gz
        headers:
          Authorization: "Bearer {{ lookup('data.api_token') }}"
          X-Custom-Header: custom-value
        extract_parent: /opt/app
        owner: root
        group: root
ccm ensure archive /opt/downloads/app.tar.gz \
    --url https://api.example.com/releases/app.tar.gz \
    --headers "Authorization:{{ lookup('data.api_token') }}"
{
  "protocol": "io.choria.ccm.v1.resource.ensure.request",
  "type": "archive",
  "properties": {
    "name": "/opt/downloads/app.tar.gz",
    "url": "https://api.example.com/releases/app.tar.gz",
    "headers": {
      "Authorization": "Bearer mytoken",
      "X-Custom-Header": "custom-value"
    },
    "extract_parent": "/opt/app",
    "owner": "root",
    "group": "root"
  }
}

Idempotency

The archive resource is idempotent through multiple mechanisms:

  1. Checksum verification: If a checksum is provided and the existing file matches, no download occurs.
  2. Creates file: If creates is specified and that file exists, neither download nor extraction occurs.
  3. File existence: If the archive file exists with matching checksum and owner/group, no changes are made.

For best idempotency, always specify either checksum or creates (or both).

Cleanup Behavior

When cleanup: true is set:

  • The archive file is deleted after successful extraction
  • The extract_parent property is required
  • The creates property is required to track extraction state across runs

Supported Archive Formats

ExtensionExtraction Tool
.tar.gz, .tgztar -xzf
.tartar -xf
.zipunzip
Note

The extraction tools (tar, unzip) must be available in the system PATH.

Exec

The exec resource executes commands to bring the system into the desired state. It is idempotent when used with the creates property or refreshonly mode.

Warning

Specify commands with their full path, or use the path property to set the search path.

- exec:
    - /usr/bin/touch /tmp/hello:
        creates: /tmp/hello
        timeout: 30s
        cwd: /tmp

Alternatively for long commands or to improve UX for referencing execs in require or subscribe:

- exec:
    - touch_hello:
        command: /usr/bin/touch /tmp/hello
        creates: /tmp/hello
        timeout: 30s
        cwd: /tmp
ccm ensure exec "/usr/bin/touch /tmp/hello" --creates /tmp/hello --timeout 30s
{
  "protocol": "io.choria.ccm.v1.resource.ensure.request",
  "type": "exec",
  "properties": {
    "name": "/usr/bin/touch /tmp/hello",
    "creates": "/tmp/hello",
    "timeout": "30s",
    "cwd": "/tmp"
  }
}

The command runs only if /tmp/hello does not exist.

Providers

The exec resource supports two providers:

ProviderDescription
posixDefault. Executes commands directly without a shell. Arguments are parsed and passed to the executable.
shellExecutes commands via /bin/sh -c "...". Use this for shell features like pipes, redirections, and builtins.

The posix provider is the default and is suitable for most commands. Use the shell provider when you need shell features:

- exec:
    - cleanup-logs:
        command: find /var/log -name '*.log' -mtime +30 -delete && echo "Done"
        provider: shell

    - check-service:
        command: systemctl is-active httpd || systemctl start httpd
        provider: shell

    - process-data:
        command: cat /tmp/input.txt | grep -v '^#' | sort | uniq > /tmp/output.txt
        provider: shell
Note

The shell provider passes the entire command string to /bin/sh -c, so shell quoting rules apply. The posix provider parses arguments using shell-like quoting but does not invoke a shell.

Properties

PropertyDescription
nameThe command to execute (used as the resource identifier)
commandAlternative command to run instead of name
cwdWorking directory for command execution
environment (array)Environment variables in KEY=VALUE format
pathSearch path for executables as a colon-separated list (e.g., /usr/bin:/bin)
returns (array)Exit codes indicating success (default: [0])
timeoutMaximum execution time (e.g., 30s, 5m); command is killed if exceeded
createsFile path; if this file exists, the command does not run
refreshonly (boolean)Only run when notified by a subscribed resource
subscribe (array)Resources to subscribe to for refresh notifications (type#name or type#alias)
logoutput (boolean)Log the command output
providerForce a specific provider (posix or shell)

File

The file resource manages files and directories, including their content, ownership, and permissions.

Warning

Use absolute file paths and primary group names.

- file:
    - /etc/motd:
        ensure: present
        content: |
          Managed by CCM {{ now() }}
        owner: root
        group: root
        mode: "0644"
ccm ensure file /etc/motd --source /tmp/ccm/motd --owner root --group root --mode 0644
{
  "protocol": "io.choria.ccm.v1.resource.ensure.request",
  "type": "file",
  "properties": {
    "name": "/etc/motd",
    "ensure": "present",
    "content": "Managed by CCM\n",
    "owner": "root",
    "group": "root",
    "mode": "0644"
  }
}

This copies the contents of /tmp/ccm/motd to /etc/motd verbatim and sets ownership.

Use --content or --content-file to parse content through the template engine before writing.

Ensure Values

ValueDescription
presentThe file must exist
absentThe file must not exist
directoryThe path must be a directory

Properties

PropertyDescription
nameAbsolute path to the file
ensureDesired state (present, absent, directory)
contentFile contents, parsed through the template engine
sourceCopy contents from another local file
ownerFile owner (username)
groupFile group (group name)
modeFile permissions in octal notation (e.g., "0644")
providerForce a specific provider (posix only)

Package

The package resource manages system packages. Specify whether the package should be present, absent, at the latest version, or at a specific version.

Warning

Use real package names, not virtual names, aliases, or group names.

- package:
    - zsh:
        ensure: "5.9"
ccm ensure package zsh 5.9
{
  "protocol": "io.choria.ccm.v1.resource.ensure.request",
  "type": "package",
  "properties": {
    "name": "zsh",
    "ensure": "5.9"
  }
}

Ensure Values

ValueDescription
presentThe package must be installed
latestThe package must be installed at latest version
absentThe package must not be installed
<version>The package must be installed at this version

Properties

PropertyDescription
namePackage name
ensureDesired state or version
providerForce a specific provider (dnf, apt)

Provider Notes

APT (Debian/Ubuntu)

The APT provider preserves existing configuration files during package installation and upgrades. When a package is upgraded and the maintainer has provided a new version of a configuration file, the existing file is kept (--force-confold behavior).

Packages in a partially installed or config-files state (removed but configuration remains) are treated as absent. Reinstalling such packages will preserve the existing configuration files.

Note

The provider will not run apt update before installing a package. Use an exec resource to update the package index if necessary.

The provider runs non-interactively and suppresses prompts from apt-listbugs and apt-listchanges.

Scaffold

The scaffold resource renders files from a source template directory to a target directory. Templates have access to facts and Hiera data, enabling dynamic configuration generation from directory structures.

Warning

Target paths must be absolute and canonical (no . or .. components).

- scaffold:
    - /etc/app:
        ensure: present
        source: templates/app
        engine: jet
        purge: true
ccm ensure scaffold /etc/app templates/app --engine jet --purge
{
  "protocol": "io.choria.ccm.v1.resource.ensure.request",
  "type": "scaffold",
  "properties": {
    "name": "/etc/app",
    "ensure": "present",
    "source": "templates/app",
    "engine": "jet",
    "purge": true
  }
}

This renders templates from the templates/app directory into /etc/app using the Jet template engine, removing any files in the target not present in the source.

Tip

This is implemented using the github.com/choria-io/scaffold Go library, you can use this in your own projects or use the included scaffold CLI tool.

Ensure Values

ValueDescription
presentTarget directory must exist with rendered template files
absentManaged files must be removed; target directory removed if empty

Properties

PropertyDescription
nameAbsolute path to the target directory
sourceSource template directory path (relative to working directory or absolute)
engineTemplate engine: go or jet (default: jet)
skip_emptyDo not create empty files in rendered output
left_delimiterCustom left template delimiter
right_delimiterCustom right template delimiter
purgeRemove files in target not present in source
postPost-processing commands: glob pattern to command mapping
providerForce a specific provider (choria only)

Template Engines

Two template engines are supported:

EngineLibraryDefault DelimitersDescription
goGo text/template{{ / }}Standard Go templates
jetJet templating[[ / ]]Jet template language

The engine defaults to jet if not specified. Delimiters can be customized via left_delimiter and right_delimiter.

Custom Delimiters

- scaffold:
    - /etc/myservice:
        ensure: present
        source: templates/myservice
        engine: go
        left_delimiter: "<<"
        right_delimiter: ">>"
ccm ensure scaffold /etc/myservice templates/myservice \
    --engine go --left-delimiter "<<" --right-delimiter ">>"
{
  "protocol": "io.choria.ccm.v1.resource.ensure.request",
  "type": "scaffold",
  "properties": {
    "name": "/etc/myservice",
    "ensure": "present",
    "source": "templates/myservice",
    "engine": "go",
    "left_delimiter": "<<",
    "right_delimiter": ">>"
  }
}

Post-Processing

The post property defines commands to run on rendered files. Each entry is a map where the key is a glob pattern matched against the file’s basename and the value is a command to execute. Use {} in the command as a placeholder for the file’s full path; if omitted, the path is appended as the last argument.

- scaffold:
    - /opt/app:
        ensure: present
        source: templates/app
        post:
          - "*.go": "go fmt {}"
          - "*.sh": "chmod +x {}"
ccm ensure scaffold /opt/app templates/app \
    --post "*.go"="go fmt {}" --post "*.sh"="chmod +x {}"
{
  "protocol": "io.choria.ccm.v1.resource.ensure.request",
  "type": "scaffold",
  "properties": {
    "name": "/opt/app",
    "ensure": "present",
    "source": "templates/app",
    "post": [
      {"*.go": "go fmt {}"},
      {"*.sh": "chmod +x {}"}
    ]
  }
}

Post-processing runs immediately after each file is rendered. Files skipped due to skip_empty are not post-processed.

Purge Behavior

When purge: true is set, files in the target directory that are not present in the source template directory are deleted during rendering. In noop mode, these deletions are logged but not performed.

When purge is disabled (the default), files not present in the source are tracked but not removed. They do not affect idempotency checks for ensure: present, meaning the resource is considered stable even if extra files exist in the target.

Removal Behavior

When ensure: absent, only managed files (changed and stable) are removed. Files not belonging to the scaffold (purged files) are left untouched. After removing managed files and empty subdirectories, the target directory itself is removed on a best-effort basis — it is only deleted if empty. If unrelated files remain, the directory is preserved and no error is raised.

Idempotency

The scaffold resource determines idempotency by rendering templates in noop mode and comparing results against the target directory.

For ensure: present:

  • Changed files: Files that would be created or modified. Any changed files make the resource unstable.
  • Stable files: Files whose content matches the rendered output. At least one stable file must exist for the resource to be considered stable.
  • Purged files: Files in the target not present in the source. These only affect stability when purge is enabled.

For ensure: absent, the status check filters Changed and Stable lists to only include files that actually exist on disk. This means after a successful removal, the scaffold is considered absent even if the target directory still exists with unrelated files. Purged files never affect the absent stability check.

Source Resolution

The source property is resolved relative to the manager’s working directory when it is a relative path. URL sources (with a scheme) are passed through unchanged. This allows manifests bundled with template directories to use relative paths.

Template Environment

Templates receive the full template environment, which provides access to:

  • facts - System facts for the managed node
  • data - Hiera-resolved configuration data
  • Template helper functions

Creating Scaffolds

A scaffold source is a directory tree where every file is a template. The directory structure is mirrored directly into the target, so the source layout becomes the output layout.

Source Directory Structure

templates/app/
├── _partials/
│   └── header.conf
├── config.yaml
├── scripts/
│   └── setup.sh
└── README.md

This renders into:

/etc/app/
├── config.yaml
├── scripts/
│   └── setup.sh
└── README.md

The _partials directory is special — its contents are available to templates but are never copied to the target.

Template Syntax

Every file in the source directory is processed as a template. The syntax depends on the engine selected.

Jet engine (default, [[ / ]] delimiters):

# config.yaml
hostname: [[ facts.hostname ]]
environment: [[ data.environment ]]
workers: [[ data.worker_count ]]

Go engine ({{ / }} delimiters):

# config.yaml
hostname: {{ .facts.hostname }}
environment: {{ .data.environment }}
workers: {{ .data.worker_count }}

The Jet engine is the default because its [[ / ]] delimiters avoid conflicts with configuration files that use curly braces (YAML, JSON, systemd units). Use the Go engine when you need access to Sprig functions.

Partials

Files inside a _partials directory are reusable template fragments. They are rendered on demand using the render function but are excluded from the output.

This is useful for shared headers, repeated configuration blocks, or any content used across multiple files.

Jet:

[[ render("_partials/header.conf", .) ]]

server {
    listen [[ data.port ]];
}

Go:

{{ render "_partials/header.conf" . }}

server {
    listen {{ .data.port }};
}

Built-in Functions

Two functions are available in both template engines:

render evaluates another template file from the source directory and returns its output as a string. The partial is rendered using the same engine and data as the calling template.

[[ render("_partials/database.conf", .) ]]
{{ render "_partials/database.conf" . }}

write creates an additional file in the target directory from within a template. This is useful for dynamically generating files based on data — for example, creating one configuration file per service.

[[ write("extra.conf", "generated content") ]]
{{ write "extra.conf" "generated content" }}

Sprig Functions

When using the Go template engine, all Sprig template functions are available. These provide string manipulation, math, date formatting, list operations, and more:

# Go engine example with Sprig functions
hostname: {{ .facts.hostname | upper }}
packages: {{ join ", " .data.packages }}
generated: {{ now | date "2006-01-02" }}

Example Scaffold

A complete scaffold for an application configuration:

Source structure:

templates/myapp/
├── _partials/
│   └── logging.conf
├── myapp.conf
└── scripts/
    └── healthcheck.sh

_partials/logging.conf (Jet):

log_level = [[ data.log_level ]]
log_file = /var/log/myapp/[[ facts.hostname ]].log

myapp.conf (Jet):

[[ render("_partials/logging.conf", .) ]]

[server]
bind = 0.0.0.0
port = [[ data.port ]]
workers = [[ data.workers ]]

scripts/healthcheck.sh (Jet):

#!/bin/bash
curl -sf http://localhost:[[ data.port ]]/health || exit 1

Manifest using this scaffold:

- scaffold:
    - /etc/myapp:
        ensure: present
        source: templates/myapp
        purge: true
        post:
          - "*.sh": "chmod +x {}"

With facts {"hostname": "web01"} and data {"port": 8080, "workers": 4, "log_level": "info"}, this renders:

/etc/myapp/
├── myapp.conf
└── scripts/
    └── healthcheck.sh

Where myapp.conf contains:

log_level = info
log_file = /var/log/myapp/web01.log

[server]
bind = 0.0.0.0
port = 8080
workers = 4

Service

The service resource manages system services. Services have two independent properties: whether they are running and whether they are enabled to start at boot.

Warning

Use real service names, not virtual names or aliases.

Services can subscribe to other resources and restart when those resources change.

- service:
    - httpd:
        ensure: running
        enable: true
        subscribe:
          - package#httpd
ccm ensure service httpd running --enable --subscribe package#httpd
{
  "protocol": "io.choria.ccm.v1.resource.ensure.request",
  "type": "service",
  "properties": {
    "name": "httpd",
    "ensure": "running",
    "enable": true,
    "subscribe": ["package#httpd"]
  }
}

Ensure Values

ValueDescription
runningThe service must be running
stoppedThe service must be stopped

If ensure is not specified, it defaults to running.

Properties

PropertyDescription
nameService name
ensureDesired state (running or stopped; default: running)
enable (boolean)Enable the service to start at boot
subscribe (array)Resources to watch; restart the service when they change (type#name or type#alias)
providerForce a specific provider (systemd only)

Hierarchical Data

The Choria Hierarchical Data Resolver is a small data resolver inspired by Hiera. It evaluates a YAML or JSON document alongside a set of facts to produce a final data map.

The resolver supports first and deep merge strategies and relies on expression-based string interpolation for hierarchy entries. It is optimized for single files that hold both hierarchy and data, rather than the multi-file approach common in Hiera.

Key features:

  • Lookup expressions based on the Expr Language
  • Type-preserving lookups (returns typed data, not just strings)
  • Command-line tool with built-in system facts
  • Go library for embedding

Usage

Here’s an annotated example:

hierarchy:
    # Lookup and override order - facts are resolved in expressions
    # Use GJSON path syntax for nested facts: {{ lookup('facts.host.info.hostname') }}
    order:
     - env:{{ lookup('facts.env') }}
     - role:{{ lookup('facts.role') }}
     - host:{{ lookup('facts.hostname') }}
    merge: deep  # "deep" merges all matches; "first" stops at first match

# Base data - hierarchy results are merged into this
data:
   log_level: INFO
   packages:
     - ca-certificates
   web:
     listen_port: 80
     tls: false

# Override sections keyed by hierarchy order entries
overrides:
    env:prod:
      log_level: WARN

    role:web:
      packages:
        - nginx
      web:
        listen_port: 443
        tls: true

    host:web01:
      log_level: TRACE

The templating here is identical to that in the Template documentation, except only the lookup() function is available (no file access functions).

Default Hierarchy

If no hierarchy section is provided, the resolver uses a default hierarchy of ["default"].

CLI Example

The ccm hiera command resolves hierarchy files with facts. It is designed to be a generally usable tool, with flexible options for providing facts.

Given the input file data.json:

{
    "hierarchy": {
        "order": [
            "fqdn:{{ lookup('facts.fqdn') }}"
        ]
    },
    "data": {
        "test": "value"
    },
    "overrides": {
        "fqdn:my.fqdn.com": {
            "test": "override"
        }
    }
}

Resolve with facts provided on the command line:

$ ccm hiera parse data.json fqdn=my.fqdn.com
{
  "test": "override"
}

$ ccm hiera parse data.json fqdn=other.fqdn.com
{
  "test": "value"
}

Output formats:

# YAML output
$ ccm hiera parse data.json fqdn=other.fqdn.com --yaml
test: value

# Environment variable output
$ ccm hiera parse data.json fqdn=other.fqdn.com --env
HIERA_TEST=value

Fact Sources

Facts can come from multiple sources, which are merged together:

System facts (-S or --system-facts):

# View system facts
$ ccm hiera facts -S

# Resolve using system facts
$ ccm hiera parse data.json -S

Environment variables as facts (-E or --env-facts):

ccm hiera parse data.json -E

Facts file (--facts FILE):

ccm hiera parse data.json --facts facts.yaml

Command-line facts (key=value pairs):

ccm hiera parse data.json env=prod role=web

All fact sources can be combined. Command-line facts take highest precedence.

Data in NATS

NATS is a lightweight messaging system that supports Key-Value stores. Hierarchy data can be stored in NATS and used with ccm ensure and ccm hiera commands.

To use NATS as a hierarchy store, configure a NATS context for authentication:

nats context add ccm --server nats.example.org --user ccm --password s3cret \
  --description "CCM Configuration Store"

Create a KV store and add your hierarchy data:

$ nats kv add CCM --replicas 3 --context ccm
$ nats kv put CCM data "$(cat hiera.yaml)"

Resolve the hierarchy using the KV store:

ccm hiera parse kv://CCM/data --context ccm -S

Data on Web Servers

Hierarchy data can also be stored on a web server and fetched via HTTP or HTTPS.

ccm hiera parse https://example.net/site.yaml -S

HTTP Basic Auth is supported via URL credentials:

ccm hiera parse https://user:pass@example.net/site.yaml -S

Data

Just like applications need data to vary their behavior and configure their environments, so do Configuration Management tools.

For this reason, I wrote extlookup and hiera years ago for the Puppet community. Despite CCM’s minimal focus, we still need data to vary behavior.

Data can be used for:

  • Configuring resource names that differ between operating systems (e.g., httpd vs apache2)
  • Setting different configuration values depending on environment, role, or other dimensions
  • Deciding whether environments should have something installed (e.g., development vs production)

CCM supports various data sources:

  • System Facts - Operating system, networking, and disk configuration
  • Custom Facts - From /etc/choria/ccm/facts.{yaml,json} and ~/.config/choria/ccm/facts.{yaml,json}
  • Environment - Variables from the shell environment and ./.env files
  • Hiera Data - Hierarchical data with overrides based on facts

Accessing Data

Expressions like {{ lookup('facts.host.info.platformFamily') }} use the Expr Language.

Available Variables

In templates, you have direct access to:

VariableDescription
FactsSystem facts (e.g., Facts.host.info.platformFamily)
DataResolved Hiera data (e.g., Data.package_name)
EnvironEnvironment variables (e.g., Environ.HOME)

Available Functions

FunctionDescription
lookup(key, default)Lookup data using GJSON Path Syntax. Example: lookup("facts.host.info.os", "linux")
readFile(path), file(path)Read a file into a string (files must be in the working directory)
template(f)Parse f using templates. If f ends in .templ, reads the file first, if it ends in .jet calls the jet() function
jet(f), jet(f, "[[", "]]")Parse f using Jet templates with optional custom delimiters. If f ends in .jet, reads the file first

GJSON Path Examples

The lookup() function uses GJSON path syntax for nested access:

lookup("facts.host.info.platformFamily")     # Simple nested path
lookup("facts.network.interfaces.0.name")    # Array index
lookup("data.packages.#")                    # Array length

CLI Usage

These expressions work on the CLI:

$ ccm ensure package '{{ lookup("data.package_name", "httpd") }}'

This fetches package_name from the data and defaults to httpd if not found.

Facts

CCM includes a built-in fact resolver that gathers system information. To see available facts:

$ ccm facts                              # All facts as JSON
$ ccm facts host                         # Query specific path
$ ccm facts --yaml                       # Output as YAML

Access facts in expressions using {{ Facts.host.info.platformFamily }} or {{ lookup('facts.host.info.platformFamily') }}.

Hiera Data for CLI

Hiera data is resolved using the Choria Hierarchical Data Resolver. By default, data is read from ./.hiera, or you can specify a file with --hiera.

Note

This applies to ccm ensure commands. The ccm apply command uses manifests that contain their own Hiera data.

Hiera Data Sources

Hiera data can be loaded from:

  • Local file: ./.hiera or path specified with --hiera
  • Key-Value store: --hiera kv://BUCKET/key (requires --context for NATS)
  • HTTP(S): --hiera https://example.com/data.yaml (supports Basic Auth via URL credentials)

Merge Strategies

The hierarchy.merge setting controls how overrides are applied:

  • first (default): Stops at the first matching override
  • deep: Deep merges all matching overrides in order

Example

Given this file stored in ./.hiera:

hierarchy:
  order:
    - os:{{ lookup('facts.host.info.platformFamily') }}
  merge: first

data:
  package_name: ""

overrides:
  os:debian:
    package_name: apache2

  os:rhel:
    package_name: httpd

Running ccm ensure package '{{ lookup("data.package_name") }}' installs httpd on RHEL-based systems and apache2 on Debian-based systems.

Note

See the Hiera section for details on configuring Hiera data in NATS.

Environment

The shell environment and variables defined in ./.env can be accessed in two ways:

# Direct access
home_dir: "{{ Environ.HOME }}"

# Via lookup (with default value)
my_var: "{{ lookup('environ.MY_VAR', 'default') }}"

The .env file uses standard KEY=value format, one variable per line.

Shell Usage

CCM is designed as a CLI-first tool. Each resource type has its own subcommand under ccm ensure, with required inputs as arguments and optional settings as flags.

The ccm ensure commands are idempotent, making them safe to run multiple times in shell scripts.

Use ccm --help and ccm <command> --help to explore available commands and options.

Managing a Single Resource

Managing a single resource is straightforward.

ccm ensure package zsh 5.8

This ensures the package zsh is installed at version 5.8.

To view the current state of a resource:

ccm status package zsh

Managing Multiple Resources

When managing multiple resources in a script, create a session first. The session records the outcome of each resource, enabling features like refreshing a service when a file changes.

#!/bin/bash

eval "$(ccm session new)"

ccm ensure package httpd
ccm ensure file /etc/httpd/conf/httpd.conf .... --require package#httpd
ccm ensure service httpd --subscribe file#/etc/httpd/conf/httpd.conf
ccm session report --remove

This creates a temporary directory for session state. If the file resource changes, the service restarts automatically.

If you prefer not to eval command output, use: export CCM_SESSION_STORE=$(mktemp -d)

Data in the CLI

As covered in the templates section, commands automatically read ./.env and ./.hiera files, merging that data into the session.

Use the --hiera flag or CCM_HIERA_DATA environment variable to specify a different data file.

With data loaded, you can access:

  • {{ lookup("data.my_data_key") }} for Hiera data
  • {{ lookup("env.MY_ENV_VAR") }} for environment variables
  • {{ lookup("facts.host.info.platformFamily") }} for system facts

Example using Hiera data:

ccm ensure package '{{ lookup("data.package") }}' '{{ lookup("data.version") }}'

Expressions use the Expr Language, enabling complex logic:

$ ccm ensure package '{{ lookup("facts.host.info.platformFamily") == "rhel" ? "httpd" : "apache2" }}'
WARN  package#httpd changed ensure=present runtime=14.509s provider=dnf

For complex conditional logic, we recommend using Hiera data with hierarchy overrides instead.

Applying Manifests

For non-shell script usage use YAML manifests with ccm apply:

See YAML Manifests for manifest format details.

Viewing System Facts

CCM gathers system facts that can be used in templates and conditions:

# Show all facts as JSON
$ ccm facts

# Show facts as YAML
$ ccm facts --yaml

# Query specific facts using gjson syntax
$ ccm facts host.info.platformFamily

Resolving Hiera Data

The ccm hiera command helps debug and test Hiera data resolution:

# Resolve a Hiera file with system facts
$ ccm hiera parse data.yaml -S

# Resolve with custom facts
$ ccm hiera parse data.yaml os=linux env=production

# Query a specific key from the result
$ ccm hiera parse data.yaml --query packages

Subsections of Shell Usage

JSON API

CCM provides a STDIN/STDOUT API for managing resources programmatically. This enables integration with external languages, allowing you to build DSLs in Ruby, Perl, Python, or any language that can execute processes and handle JSON or YAML.

Overview

The API uses a simple request/response pattern:

  1. Send a request to ccm ensure api pipe via STDIN
  2. Receive a response on STDOUT

Both JSON and YAML formats are supported for requests. The response format is always JSON, but can be explicitly set to YAML using --yaml.

Command

ccm ensure api pipe [--yaml] [--noop] [--facts <file>] [--data <file>]
FlagDescription
--yamlOutput response in YAML format instead of JSON
--noopDry-run mode; report what would change without making changes
--facts <file>Load additional facts from a YAML file
--data <file>Load Hiera-style data from a YAML file

Request Format

Requests must include a protocol identifier, resource type, and properties.

Note

JSON Schemas for these requests and responses are available at resource_ensure_request.json and resource_ensure_response.json.

JSON Request

{
  "protocol": "io.choria.ccm.v1.resource.ensure.request",
  "type": "package",
  "properties": {
    "name": "nginx",
    "ensure": "present"
  }
}

YAML Request

protocol: io.choria.ccm.v1.resource.ensure.request
type: package
properties:
  name: nginx
  ensure: present

Request Fields

FieldRequiredDescription
protocolYesMust be io.choria.ccm.v1.resource.ensure.request
typeYesResource type example package
propertiesYesResource properties (varies by type)

Response Format

Responses include a protocol identifier and either a state object (on success) or an error message.

For the full response structure, see ResourceEnsureApiResponse in the Go documentation. The state field contains a TransactionEvent.

Successful Response

{
  "protocol": "io.choria.ccm.v1.resource.ensure.response",
  "state": {
    "protocol": "io.choria.ccm.v1.transaction.event",
    "event_id": "2abc123def456",
    "timestamp": "2026-01-28T10:30:00Z",
    "type": "package",
    "provider": "dnf",
    "name": "nginx",
    "requested_ensure": "present",
    "final_ensure": "1.24.0-1.el9",
    "duration": 1500000000,
    "changed": true,
    "failed": false
  }
}

Error Response

{
  "protocol": "io.choria.ccm.v1.resource.ensure.response",
  "error": "invalid protocol \"wrong.protocol\""
}

Examples

Install a Package

echo '{
  "protocol": "io.choria.ccm.v1.resource.ensure.request",
  "type": "package",
  "properties": {
    "name": "htop",
    "ensure": "present"
  }
}' | ccm ensure api pipe

Manage a Service

cat <<EOF | ccm ensure api pipe
protocol: io.choria.ccm.v1.resource.ensure.request
type: service
properties:
  name: nginx
  ensure: running
EOF

Create a File

echo '{
  "protocol": "io.choria.ccm.v1.resource.ensure.request",
  "type": "file",
  "properties": {
    "name": "/etc/motd",
    "ensure": "present",
    "content": "Welcome to this server\n",
    "owner": "root",
    "group": "root",
    "mode": "0644"
  }
}' | ccm ensure api pipe

Dry-Run Mode

echo '{
  "protocol": "io.choria.ccm.v1.resource.ensure.request",
  "type": "package",
  "properties": {
    "name": "vim",
    "ensure": "absent"
  }
}' | ccm ensure api pipe --noop

Resource Types

For detailed information about each resource type and its properties, see the Resource Documentation

CLI Plugins

CCM supports extending the CLI with custom commands using App Builder. This allows you to create organization-specific workflows that integrate with the ccm command.

Plugin Locations

CCM searches for plugins in two directories:

LocationPurpose
/etc/choria/ccm/plugins/System-wide plugins
$XDG_CONFIG_HOME/choria/ccm/plugins/User plugins (typically ~/.config/choria/ccm/plugins/)

Plugins in the user directory override system plugins with the same name.

Plugin File Format

Plugin files must be named <command>-plugin.yaml. The filename determines the command name:

deploy-plugin.yaml    → ccm deploy
backup-plugin.yaml    → ccm backup
myapp-plugin.yaml     → ccm myapp

Basic Plugin Structure

Plugins use App Builder’s YAML definition format. Here’s a minimal example:

# deploy-plugin.yaml
name: deploy
description: Deploy application configuration
commands:
  - name: web
    description: Deploy web server configuration
    type: exec
    command: |
      ccm apply /etc/ccm/manifests/webserver.yaml

This creates ccm deploy web which applies a manifest.

Passing Data to Manifests

Plugins can pass data to manifests through environment variables or facts. Both are accessible in manifest templates.

Using Environment Variables

Environment variables set in the plugin are available in manifests via {{ lookup("env.VAR_NAME") }}:

# deploy-plugin.yaml
name: deploy
description: Deploy with environment
commands:
  - name: production
    description: Deploy to production
    type: exec
    environment:
      - "DEPLOY_ENV=production"
      - "LOG_LEVEL=warn"
    command: |
      ccm apply /etc/ccm/manifests/app.yaml

In the manifest:

resources:
  - type: file
    name: /etc/app/config.yaml
    content: |
      environment: {{ lookup("env.DEPLOY_ENV") }}
      log_level: {{ lookup("env.LOG_LEVEL") }}

Using Facts

Pass additional facts using the --fact flag. Facts are available via {{ lookup("facts.KEY") }}:

# deploy-plugin.yaml
name: deploy
description: Deploy with custom facts
commands:
  - name: app
    description: Deploy application
    type: exec
    arguments:
      - name: version
        description: Application version to deploy
        required: true
    command: |
      ccm apply /etc/ccm/manifests/app.yaml --fact app_version={{ .Arguments.version }}

In the manifest:

resources:
  - type: package
    name: myapp
    ensure: present
    version: "{{ lookup('facts.app_version') }}"

Further Reading

For complete App Builder documentation including all command types, templating features, and advanced options, see the App Builder documentation.

Since version 0.13.0 of App Builder it has a transform and a command that can invoke CCM Manifests, this combines well with flags, arguments and form wizards to create custom UI’s that manage your infrastructure.

YAML Manifests

A manifest is a YAML file that combines data, hierarchy configuration, and resources in a single file.

Manifests support template expressions but not procedural logic. Think of them as declarative configuration similar to multi-resource shell scripts.

CCM Studio

An experimental visual editor for manifests is available at CCM Studio.

Manifest Structure

A manifest contains these top-level sections:

SectionDescription
dataInput data (like module parameters)
hierarchyLookup order and merge strategy for overrides
overridesData overrides keyed by hierarchy entries
ccmResource definitions and execution options
Note

A JSON Schema for manifests is available at https://choria-cm.dev/schemas/ccm/v1/manifest.json. Configure your editor to use this schema for completion and validation.

The manifest is resolved using the Choria Hierarchical Data Resolver.

Full Example

Input Data

Define input data like parameters for a module:

data:
  package_name: "httpd"

Resources

Define resources to manage in the ccm section:

ccm:
  resources:
    - package:
        - "{{ lookup('data.package_name') }}":
            ensure: latest

Configure Hierarchy

Configure a hierarchy to vary data by dimensions like OS platform:

hierarchy:
  order:
    - os:{{ lookup('facts.host.info.platformFamily') }}

This looks for overrides in os:rhel, os:debian, etc.

Overrides

Provide platform-specific data overrides:

overrides:
  os:debian:
    package_name: apache2

Applying Manifests

The complete manifest:

data:
  package_name: "httpd"

ccm:
  resources:
    - package:
        - "{{ lookup('data.package_name') }}":
             ensure: latest

hierarchy:
  order:
    - os:{{ lookup('facts.host.info.platformFamily') }}

overrides:
  os:debian:
    package_name: apache2

Apply the manifest with ccm apply. The first run makes changes; subsequent runs are stable:

$ ccm apply manifest.yaml
INFO  Creating new session record resources=1
WARN  package#httpd changed ensure=latest runtime=3.560699287s provider=dnf

$ ccm apply manifest.yaml
INFO  Creating new session record resources=1
INFO  package#httpd stable ensure=latest runtime=293.448824ms provider=dnf

To preview the fully resolved manifest without applying it:

$ ccm apply manifest.yaml --render
data:
  package_name: apache2
resources:
- package:
    - apache2:
        ensure: latest

Pre and Post Messages

Display messages before and after manifest execution:

ccm:
  pre_message: |
    Starting configuration update...
  post_message: |
    Configuration complete.
  resources:
    - ...

Both are optional.

Overriding Data

Override or augment manifest data with an external Hiera source:

ccm apply manifest.yaml --hiera kv://CCM/common

Supported sources include local files, KV stores (kv://), and HTTP(S) URLs.

Setting Defaults

Reduce repetition by setting defaults for resources of the same type:

ccm:
  resources:
    - file:
        - defaults:
            owner: app
            group: app
            mode: "0644"
            ensure: present
        - /app/config/file.conf:
            source: file.conf
        - /app/bin/app:
            source: app.bin
            mode: "0700"

    - file:
        - /etc/motd:
            ensure: present
            source: motd.txt
            owner: root
            group: root
            mode: "0644"

The first two files inherit the defaults values. The /app/bin/app file overrides just the mode. The /etc/motd file is a separate resource block, so defaults do not apply.

Templating

Manifests support template expressions like {{ lookup("key") }} for adjusting values. These expressions cannot generate new resources; they only modify values in valid YAML.

Available Variables

Templates have access to:

VariableDescription
FactsSystem facts
DataResolved Hiera data
EnvironEnvironment variables

Generating Resources with Jet Templates

To dynamically generate resources from data, use Jet Templates.

Given this data:

data:
  common_packages:
    - zsh
    - vim-enhanced
    - nagios-plugins-all
    - tar
    - nmap-ncat

Reference a Jet template file instead of inline resources:

ccm:
  resources_jet_file: resources.jet

Create the template file:

{* resources.jet *}
[[ range Data.common_packages ]]
- package:
    - [[ . ]]:
        ensure: present
[[ end ]]
Note

The template file must be in the same directory as the manifest.

The template receives the fully resolved Hiera data, plus Facts and Environ.

Failure Handling

By default, if a resource fails, the apply continues to the next resource.

Fail on Error

To stop execution on the first failure, set fail_on_error:

ccm:
  fail_on_error: true
  resources:
    - exec:
        - /usr/bin/false: {}
        - /usr/bin/true: {}

The second resource is never executed:

$ ccm apply manifest.yaml
INFO  Executing manifest manifest=manifest.yaml resources=2
ERROR exec#/usr/bin/false failed ensure=present runtime=3ms provider=posix errors=failed to reach desired state exit code 1
WARN  Terminating manifest execution due to failed resource

Resource Dependencies

Use the require property to ensure a resource only runs after its dependencies succeed:

ccm:
  resources:
    - package:
        - httpd:
            ensure: present
    - file:
        - /etc/httpd/conf.d/custom.conf:
            ensure: present
            content: "Listen 8080"
            owner: root
            group: root
            mode: "0644"
            require:
              - package#httpd

If the required resource fails, the dependent resource is skipped.

Dry Run (Noop Mode)

Preview changes without applying them:

ccm apply manifest.yaml --noop
Note

Noop mode cannot always detect cascading effects. If one resource change would affect a later resource, that dependency may not be reflected in the dry run.

Health Check Only Mode

Run only health checks without applying resources:

ccm apply manifest.yaml --monitor-only

This is useful for verifying system state without making changes.

Manifests in NATS Object Store

Manifests can be stored in NATS Object Stores, avoiding the need to distribute files locally.

Configure a NATS context for authentication:

nats context add ccm --server nats.example.org --user ccm --password s3cret \
  --description "CCM Configuration Store"

Create an Object Store:

nats obj add CCM --replicas 3 --context ccm

Create a manifest with supporting files:

$ mkdir /tmp/manifest
$ echo 'CCM Managed' > /tmp/manifest/motd
$ cat > /tmp/manifest/manifest.yaml << 'EOF'
ccm:
  resources:
    - file:
        - /etc/motd:
            ensure: present
            source: motd
            owner: root
            group: root
            mode: "0644"
EOF

Package and upload to the Object Store:

$ tar -C /tmp/manifest/ -cvzf /tmp/manifest.tgz .
$ nats obj put CCM manifest.tgz --context ccm

Apply the manifest:

$ ccm apply obj://CCM/manifest.tgz --context ccm
INFO  Using manifest from Object Store in temporary directory bucket=CCM file=manifest.tgz
INFO  file#/etc/motd stable ensure=present runtime=0s provider=posix

Manifests on Web Servers

Store gzipped tar archives on a web server and apply them directly:

$ ccm apply https://example.net/manifest.tar.gz
INFO  Executing manifest manifest=https://example.net/manifest.tar.gz resources=1
INFO  file#/etc/motd stable ensure=present runtime=0s provider=posix

HTTP Basic Auth is supported via URL credentials:

ccm apply https://user:pass@example.net/manifest.tar.gz

Additional Facts

Provide additional facts from the command line or a file:

# Individual facts
$ ccm apply manifest.yaml --fact env=production --fact region=us-east

# Facts from a file
$ ccm apply manifest.yaml --facts custom-facts.yaml

Facts from these sources are merged with system facts, with command-line facts taking precedence.

Agent

For certain use cases, it is useful to run YAML Manifests continuously. For example, you might not want to manage your dotfiles automatically (allowing for local modifications), but you do want to keep Docker up to date.

The CCM Agent runs manifests continuously, loading them from local files, Object Storage, or HTTP(S) URLs, with Key-Value data overlaid.

In time, the Agent will become a key part of a service registry system, allowing for continuous monitoring of managed resources and sharing that state into a registry—enabling other nodes to use file resources to create configurations that combine knowledge of all nodes.

Run Modes

The agent supports two modes of operation that combine to be efficient and fast-reacting:

  • Full manifest apply: Manages the complete state of every resource
  • Health check mode: Runs only Monitoring checks, which can trigger a full manifest apply as remediation

By enabling both modes, you can run health checks very frequently (even at 10- or 20-second intervals) while keeping full Configuration Management runs less frequent (every few hours).

Enabling both modes is optional but recommended. We also recommend adding health checks to your key resources.

Supported Manifest and Data Sources

Manifest Sources

Manifests can be loaded from:

  • Local file: /path/to/manifest.yaml
  • Object Storage: obj://bucket/key.tar.gz
  • HTTP(S): https://example.com/manifest.tar.gz (supports Basic Auth via URL credentials)

Remote sources (Object Storage and HTTP) must be .tar.gz archives containing a manifest.yaml file, templates and file sources.

External Data Sources

For Hiera data resolution, the agent supports:

  • Local file: file:///path/to/data.yaml
  • Key-Value store: kv://bucket/key
  • HTTP(S): https://example.com/data.yaml

Logical Flow

The agent continuously runs and manages manifests as follows:

  1. At startup, the agent fetches data and gathers facts
  2. Starts a worker for each manifest source
    1. Each worker starts watchers to download and manage the manifest (polling every 30 seconds for remote sources)
  3. Triggers workers at the configured interval for a full apply
    1. Each run updates facts (minimum 2-minute interval) and data
    2. Applies each manifest serially
  4. Triggers workers at the configured health check interval
    1. Health check runs do not update facts or data
    2. Runs health checks for each manifest serially
    3. If any health checks are critical (not warning), the agent triggers a full apply for that worker

In the background, object stores and HTTP sources are watched for changes. Updates trigger immediate apply runs with exponential backoff retry on failures.

Prometheus Metrics

When monitor_port is configured, the agent exposes Prometheus metrics on /metrics. These metrics can be used to monitor agent health, track resource states and events, and observe health check statuses.

Agent Metrics

MetricTypeLabelsDescription
choria_ccm_agent_apply_duration_secondsSummarymanifestTime taken to apply manifests
choria_ccm_agent_healthcheck_duration_secondsSummarymanifestsTime taken for health check runs
choria_ccm_agent_healthcheck_remediations_countCountermanifestHealth checks that triggered remediation
choria_ccm_agent_data_resolve_duration_secondsSummary-Time taken to resolve external data
choria_ccm_agent_data_resolve_error_countCounterurlData resolution failures
choria_ccm_agent_facts_resolve_duration_secondsSummary-Time taken to resolve facts
choria_ccm_agent_facts_resolve_error_countCounter-Facts resolution failures
choria_ccm_agent_manifest_fetch_countCountermanifestRemote manifest fetches
choria_ccm_agent_manifest_fetch_error_countCountermanifestRemote manifest fetch failures

Resource Metrics

MetricTypeLabelsDescription
choria_ccm_manifest_apply_duration_secondsSummarysourceTime taken to apply an entire manifest
choria_ccm_resource_apply_duration_secondsSummarytype, provider, nameTime taken to apply a resource
choria_ccm_resource_state_total_countCountertype, nameTotal resources processed
choria_ccm_resource_state_stable_countCountertype, nameResources in stable state
choria_ccm_resource_state_changed_countCountertype, nameResources that changed
choria_ccm_resource_state_refreshed_countCountertype, nameResources that were refreshed
choria_ccm_resource_state_failed_countCountertype, nameResources that failed
choria_ccm_resource_state_error_countCountertype, nameResources with errors
choria_ccm_resource_state_skipped_countCountertype, nameResources that were skipped
choria_ccm_resource_state_noop_countCountertype, nameResources in noop mode

Health Check Metrics

MetricTypeLabelsDescription
choria_ccm_healthcheck_duration_secondsSummarytype, name, checkTime taken for health checks
choria_ccm_healthcheck_status_countCountertype, name, status, checkHealth check results by status

Facts Metrics

MetricTypeLabelsDescription
choria_ccm_facts_gather_duration_secondsSummary-Time taken to gather system facts

Configuration

The agent is included in the ccm binary. To use it, create a configuration file and enable the systemd service.

The configuration file is located at /etc/choria/ccm/config.yaml:

# CCM Agent Configuration Example

# Time between scheduled manifest apply runs.
# Must be at least 30s. Defaults to 5m.
interval: 5m

# Time between health check runs.
# When set, health checks run independently of apply runs and can trigger
# remediation applies when critical issues are detected.
# Omit to disable periodic health checks.
health_check_interval: 1m

# List of manifest sources to apply. Each source creates a separate worker.
# Supported formats:
#   - Local file: /path/to/manifest.yaml
#   - Object store: obj://bucket/key.tar.gz
#   - HTTP(S): https://example.com/manifest.tar.gz
#   - HTTP with Basic Auth: https://user:pass@example.com/manifest.tar.gz
# Remote sources must be .tar.gz archives containing a manifest.yaml file.
manifests:
  - /etc/choria/ccm/manifests/base.yaml
  - obj://ccm-manifests/app.tar.gz
  - https://config.example.com/manifests/web.tar.gz

# Logging level: debug, info, warn, error
log_level: info

# NATS context for authentication. Defaults to 'CCM'.
nats_context: CCM

# Optional URL for external Hiera data resolution.
# Supported formats: file://, kv://, http(s)://
# The resolved data is merged into the manifest data context.
external_data_url: kv://ccm-data/common

# Directory for caching remote manifest sources.
# Defaults to /etc/choria/ccm/source.
cache_dir: /etc/choria/ccm/source

# Port for Prometheus metrics endpoint (/metrics).
# Set to 0 or omit to disable.
monitor_port: 9100

After configuring, start the service:

ccm ensure service ccm-agent running --enable

Monitoring

By default, CCM verifies resource health using the resource’s native state. For example, if a service should be running and systemd reports it as running, CCM considers it healthy.

For deeper validation, all resources support custom health checks. These checks run after a resource is managed and can verify that the resource is functioning correctly, not just present.

Health Check Properties

PropertyDescriptionDefault
commandCommand to execute (Nagios-style check)-
goss_rulesInline Goss validation rules-
nameName for logging and metricsCommand base name
triesNumber of attempts before failing1
try_sleepDuration to wait between retry attempts1s
timeoutMaximum time for command executionNo timeout
formatOutput format interpretationAuto-detected

Each health check must specify either command or goss_rules – they are mutually exclusive. The format is auto-detected based on which field is set (nagios for command, goss for goss_rules), but can be overridden explicitly.

Nagios Format

Health checks using command follow Nagios plugin conventions for exit codes:

Exit CodeStatusDescription
0OKCheck passed
1WARNINGCheck passed with warnings (non-critical)
2CRITICALCheck failed
3+UNKNOWNCheck could not determine status

Example

- service:
    - httpd:
        ensure: running
        enable: true
        health_checks:
          - name: check_http
            command: |
              /usr/lib64/nagios/plugins/check_http -H localhost -p 80 --expect "Acme Inc"
            tries: 5
            try_sleep: 1s
            timeout: 10s
ccm ensure service httpd running \
    --check '/usr/lib64/nagios/plugins/check_http -H localhost -p 80' \
    --check-tries 5 \
    --check-sleep 1s

The CLI supports a single health check per resource. For multiple health checks, use a manifest.

This example verifies that the web server responds with content containing “Acme Inc”. If the check fails, it retries up to 5 times with 1 second between attempts.

Goss Format

Health checks using goss_rules embed Goss validation rules directly in the manifest. This allows you to validate system state – running services, listening ports, file contents, HTTP responses, and more – without writing external check scripts.

The check result is OK when all Goss rules pass, or CRITICAL when any rule fails. See the Goss documentation for the full list of supported resource types and matchers.

Supported Version

Added in version 0.1.0

Example

- service:
    - httpd:
        ensure: running
        enable: true
        health_checks:
          - name: httpd_health
            goss_rules:
              service:
                httpd:
                  running: true
                  enabled: true
              port:
                tcp:80:
                  listening: true
              http:
                http://localhost:
                  status: 200
                  body:
                    - Acme Inc
            tries: 5
            try_sleep: 1s

This validates that the httpd service is running and enabled, port 80 is listening, and the web server responds with a 200 status containing “Acme Inc”.

Templates in Goss Rules

Goss rules are processed through CCM’s own template engine before evaluation. This means you can use the standard {{ }} expression syntax with access to Facts, Data, and Environ, as well as the lookup(), template(), and jet() functions. See the Data section for full details on template syntax.

health_checks:
  - name: app_check
    goss_rules:
      http:
        "http://localhost:{{ Data.app_port }}/health":
          status: 200
          body:
            - "{{ lookup('data.expected_response', 'ok') }}"
      file:
        "{{ Data.config_path }}":
          exists: true

For more complex rules, you can use the template() function with Jet templates:

health_checks:
  - name: app_check
    goss_rules:
      http:
        "http://localhost:{{ template('check_url.jet') }}":
          status: 200
Note

CCM resolves templates before passing rules to Goss. Goss’s own template variables are not used.

Agent Integration

When running the Agent with health_check_interval configured, health checks run independently of full manifest applies. If any health check returns a CRITICAL status, the agent triggers a remediation apply for that manifest.

Design Documents

Design documents provide detailed implementation guidance for CCM’s resource types, providers, and internal components. They are intended for developers contributing to CCM or those seeking to understand specific implementation details.

For end-user documentation on how to use resources, see Resources.

Note

These design documents are largely written with AI assistance and reviewed before publication.

Contents

Each design document covers:

  • Purpose and scope: What the component does and its responsibilities
  • Architecture: How the component fits into CCM’s overall design
  • Implementation details: Key data structures, interfaces, and algorithms
  • Provider contracts: Requirements for implementing new providers
  • Testing considerations: How to test the component

Available Documents

DocumentDescription
ArchiveArchive resource for downloading and extracting archives
ExecExec resource for command execution
FileFile resource for managing files and directories
PackagePackage resource for system package management
ScaffoldScaffold resource for template directory rendering
ServiceService resource for system service management
New TypeHow to add a new resource type to CCM

Subsections of Design Documents

Archive Type

This document describes the design of the archive resource type for downloading and extracting archives.

Overview

The archive resource manages remote archives with three phases:

  • Download: Fetch archive from a URL to local filesystem
  • Extract: Unpack archive contents to a target directory
  • Cleanup: Optionally remove the archive file after extraction

These phases are conditional based on current state and configuration.

Provider Interface

Archive providers must implement the ArchiveProvider interface:

type ArchiveProvider interface {
    model.Provider

    Download(ctx context.Context, properties *model.ArchiveResourceProperties, log model.Logger) error
    Extract(ctx context.Context, properties *model.ArchiveResourceProperties, log model.Logger) error
    Status(ctx context.Context, properties *model.ArchiveResourceProperties) (*model.ArchiveState, error)
}

Method Responsibilities

MethodResponsibility
StatusQuery archive file existence, checksum, attributes, and creates file
DownloadFetch archive from URL, verify checksum, set ownership
ExtractUnpack archive contents to extract parent directory

Status Response

The Status method returns an ArchiveState containing:

type ArchiveState struct {
    CommonResourceState
    Metadata *ArchiveMetadata
}

type ArchiveMetadata struct {
    Name          string    // Archive file path
    Checksum      string    // SHA256 hash of archive
    ArchiveExists bool      // Whether archive file exists
    CreatesExists bool      // Whether creates marker file exists
    Owner         string    // Archive file owner
    Group         string    // Archive file group
    MTime         time.Time // Modification time
    Size          int64     // File size in bytes
    Provider      string    // Provider name (e.g., "http")
}

The Ensure field in CommonResourceState is set to:

  • present if the archive file exists
  • absent if the archive file does not exist

Available Providers

ProviderSourceDocumentation
httpHTTP/HTTPS URLsHTTP

Ensure States

ValueDescription
presentArchive must be downloaded (and optionally extracted)
absentArchive file must not exist

Supported Archive Formats

ExtensionDescription
.tar.gz, .tgzGzip-compressed tar archive
.tarUncompressed tar archive
.zipZIP archive

The URL and local file name must have matching archive type extensions.

Apply Logic

┌─────────────────────────────────────────┐
│ Get current state via Status()          │
└─────────────────┬───────────────────────┘
                  │
                  ▼
┌─────────────────────────────────────────┐
│ Is current state desired state?         │
└─────────────────┬───────────────────────┘
              Yes │         No
                  ▼         │
          ┌───────────┐     │
          │ No change │     │
          └───────────┘     │
                            ▼
              ┌─────────────────────────┐
              │ What is desired ensure? │
              └─────────────┬───────────┘
                            │
            ┌───────────────┴───────────────┐
            │ absent                        │ present
            ▼                               ▼
    ┌───────────────┐             ┌─────────────────────┐
    │ Remove archive│             │ Download needed?    │
    │ file          │             │ (checksum mismatch  │
    └───────────────┘             │  or file missing)   │
                                  └─────────┬───────────┘
                                        Yes │         No
                                            ▼         │
                                    ┌───────────┐     │
                                    │ Download  │     │
                                    └─────┬─────┘     │
                                          │           │
                                          ▼           ▼
                                  ┌─────────────────────────┐
                                  │ Extract needed?         │
                                  │ (extract_parent set AND │
                                  │  (download occurred OR  │
                                  │   creates file missing))│
                                  └─────────┬───────────────┘
                                        Yes │         No
                                            ▼         │
                                    ┌───────────┐     │
                                    │ Extract   │     │
                                    └─────┬─────┘     │
                                          │           │
                                          ▼           ▼
                                  ┌─────────────────────────┐
                                  │ Cleanup enabled?        │
                                  └─────────┬───────────────┘
                                        Yes │         No
                                            ▼         ▼
                                    ┌───────────┐ ┌───────┐
                                    │ Remove    │ │ Done  │
                                    │ archive   │ └───────┘
                                    └───────────┘

Idempotency

The archive resource uses multiple checks for idempotency:

State Checks (in order)

  1. Ensure absent: Archive file must not exist
  2. Creates file: If creates is set, the marker file must exist
  3. Archive existence: If cleanup: false, archive must exist
  4. Owner/Group: Archive file attributes must match
  5. Checksum: If specified, archive checksum must match

Decision Table

ConditionStable?
ensure: absent + archive missingYes
ensure: absent + archive existsNo (remove)
creates file existsYes (skip all)
creates file missingNo (extract needed)
cleanup: false + archive missingNo (download needed)
Archive checksum mismatchNo (re-download needed)
Archive owner/group mismatchNo (re-download needed)

Creates Property

The creates property provides idempotency for extraction:

- archive:
    - /tmp/app.tar.gz:
        url: https://example.com/app.tar.gz
        extract_parent: /opt/app
        creates: /opt/app/bin/app
        owner: root
        group: root

Behavior:

  • If /opt/app/bin/app exists, skip download and extraction
  • Useful when extracted files indicate successful prior extraction
  • Prevents re-extraction on every run

Cleanup Property

The cleanup property removes the archive after extraction:

- archive:
    - /tmp/app.tar.gz:
        url: https://example.com/app.tar.gz
        extract_parent: /opt/app
        creates: /opt/app/bin/app
        cleanup: true
        owner: root
        group: root

Requirements:

  • extract_parent must be set (cleanup only makes sense with extraction)
  • creates must be set to track extraction state

Behavior:

  • After successful extraction, remove the archive file
  • On subsequent runs, creates file prevents re-download

Checksum Verification

When checksum is specified:

- archive:
    - /tmp/app.tar.gz:
        url: https://example.com/app.tar.gz
        checksum: "a1b2c3d4..."
        owner: root
        group: root

Behavior:

  • Downloaded file is verified against SHA256 checksum
  • Existing file checksum is compared to detect changes
  • Checksum mismatch triggers re-download
  • Download fails if fetched content doesn’t match

Authentication

Archives support two authentication methods:

Basic Authentication

- archive:
    - /tmp/app.tar.gz:
        url: https://private.example.com/app.tar.gz
        username: deploy
        password: "{{ lookup('data.password') }}"
        owner: root
        group: root

Custom Headers

- archive:
    - /tmp/app.tar.gz:
        url: https://api.example.com/releases/app.tar.gz
        headers:
          Authorization: "Bearer {{ lookup('data.token') }}"
        owner: root
        group: root

Required Properties

PropertyRequiredDescription
urlYesSource URL for download
ownerYesUsername that owns the archive file
groupYesGroup that owns the archive file

URL Validation

URLs are validated during resource creation:

  • Must be valid URL format
  • Scheme must be http or https
  • Path must end with supported archive extension
  • Extension must match the name property extension

Noop Mode

In noop mode, the archive type:

  1. Queries current state normally
  2. Computes what actions would be taken
  3. Sets appropriate NoopMessage:
    • “Would have downloaded”
    • “Would have extracted”
    • “Would have cleaned up”
    • “Would have removed”
  4. Reports Changed: true if changes would occur
  5. Does not call provider Download/Extract methods
  6. Does not remove files

Multiple actions are joined with “. " (e.g., “Would have downloaded. Would have extracted”).

Desired State Validation

After applying changes (in non-noop mode), the type verifies the archive reached the desired state by calling Status() again and checking all conditions. If validation fails, ErrDesiredStateFailed is returned.

Subsections of Archive Type

HTTP Provider

This document describes the implementation details of the HTTP archive provider for downloading and extracting archives from HTTP/HTTPS URLs.

Provider Selection

The HTTP provider is selected when:

  1. The URL scheme is http or https
  2. The archive file extension is supported (.tar.gz, .tgz, .tar, .zip)
  3. The required extraction tool (tar or unzip) is available in PATH

The IsManageable() function checks these conditions and returns a priority of 1 if all are met.

Operations

Download

Process:

  1. Parse the URL and add Basic Auth credentials if username/password provided
  2. Create HTTP request with custom headers (if specified)
  3. Execute GET request via util.HttpGetResponse()
  4. Verify HTTP 200 status code
  5. Create temporary file in the same directory as the target
  6. Set ownership on temp file before writing content
  7. Copy response body to temp file
  8. Verify checksum if provided
  9. Atomic rename temp file to target path

Atomic Write Pattern:

[parent dir]/archive-name-* (temp file)
    ↓ write content
    ↓ set owner/group
    ↓ verify checksum
    ↓ rename
[parent dir]/archive-name (final file)

The temp file is created in the same directory as the target to ensure os.Rename() is atomic (same filesystem).

Error Handling:

ConditionBehavior
HTTP non-200Return error with status code
Write failureClean up temp file, return error
Checksum mismatchClean up temp file, return error with expected vs actual
Rename failureTemp file cleaned up by defer

Authentication:

MethodImplementation
Basic AuthURL userinfo is passed to HttpGetResponse() which sets Authorization header
Username/Password propertiesEmbedded in URL before request: url.UserPassword(username, password)
Custom HeadersAdded to request via http.Header.Add()

Extract

Process:

  1. Validate ExtractParent is set
  2. Create ExtractParent directory if it doesn’t exist (mode 0755)
  3. Determine archive type from file extension
  4. Execute appropriate extraction command

Extraction Commands:

ExtensionCommand
.tar.gz, .tgztar -xzf <archive> -C <extract_parent>
.tartar -xf <archive> -C <extract_parent>
.zipunzip -d <extract_parent> <archive>

Command Execution:

Commands are executed via model.CommandRunner.ExecuteWithOptions() with:

OptionValue
Commandtar or unzip
ArgsExtraction flags and paths
CwdExtractParent directory
Timeout1 minute

Error Handling:

ConditionBehavior
Unsupported extensionReturn “archive type not supported” error
Command not foundRunner returns error
Non-zero exit codeReturn error with exit code and stderr

Status

Process:

  1. Initialize state with EnsureAbsent default
  2. Check if archive file exists via os.Stat()
  3. If exists: set EnsurePresent, populate metadata (size, mtime, owner, group, checksum)
  4. If Creates property set: check if creates file exists

Metadata Collected:

FieldSource
NameFrom properties
Provider“http”
ArchiveExistsos.Stat() success
SizeFileInfo.Size()
MTimeFileInfo.ModTime()
Ownerutil.GetFileOwner() - resolves UID to username
Grouputil.GetFileOwner() - resolves GID to group name
Checksumutil.Sha256HashFile()
CreatesExistsos.Stat() on Creates path

Idempotency

The provider supports idempotency through the type’s isDesiredState() function:

State Checks (in order)

  1. Ensure Absent: If ensure: absent, archive must not exist
  2. Creates File: If Creates set and file doesn’t exist → not stable
  3. Archive Existence: If cleanup: false, archive must exist
  4. Owner/Group: Must match properties
  5. Checksum: If specified in properties, must match

Note: When cleanup: true, the creates property is required (enforced at validation time).

Decision Matrix

Archive ExistsCreates ExistsChecksum MatchCleanupStable?
NoNoN/AfalseNo (download needed)
YesNoYesfalseNo (extract needed)
YesYesYesfalseYes
NoYesN/AtrueYes
YesYesYestrueNo (cleanup needed)

Checksum Verification

Algorithm: SHA-256

Implementation:

sum, err := util.Sha256HashFile(tempFile)
if sum != properties.Checksum {
    return fmt.Errorf("checksum mismatch, expected %q got %q", properties.Checksum, sum)
}

Timing: Checksum is verified after download completes but before the atomic rename. This ensures:

  • Corrupted downloads are never placed at the target path
  • Temp file is cleaned up on mismatch
  • Clear error message with both expected and actual checksums

Security Considerations

Credential Handling

  • Credentials in URL are redacted in log messages via util.RedactUrlCredentials()
  • Basic Auth header is set by Go’s http.Request.SetBasicAuth(), not manually constructed

Archive Extraction

  • Extraction uses system tar/unzip commands
  • No path traversal protection beyond what the tools provide
  • ExtractParent must be an absolute path (validated in model)

Temporary Files

  • Created with os.CreateTemp() using pattern <archive-name>-*
  • Deferred removal ensures cleanup on all exit paths
  • Ownership set before content written

Platform Support

The provider is Unix-only due to:

  • Dependency on util.GetFileOwner() which uses syscall for UID/GID resolution
  • Dependency on util.ChownFile() for ownership management

Timeouts

OperationTimeoutConfigurable
HTTP Download1 minute (default in HttpGetResponse)No
Archive Extraction1 minuteNo

Large archives may require increased timeouts in future versions.

Exec Type

This document describes the design of the exec resource type for executing commands.

Overview

The exec resource executes commands with idempotency controls:

  • Creates: Skip execution if a file exists
  • Refresh Only: Only execute when triggered by a subscribed resource
  • Exit Codes: Validate success via configurable return codes

Provider Interface

Exec providers must implement the ExecProvider interface:

type ExecProvider interface {
    model.Provider

    Execute(ctx context.Context, properties *model.ExecResourceProperties, log model.Logger) (int, error)
    Status(ctx context.Context, properties *model.ExecResourceProperties) (*model.ExecState, error)
}

Method Responsibilities

MethodResponsibility
StatusCheck if creates file exists, return current state
ExecuteRun the command, return exit code

Status Response

The Status method returns an ExecState containing:

type ExecState struct {
    CommonResourceState

    ExitCode         *int // Exit code from last execution (nil if not run)
    CreatesSatisfied bool // Whether creates file exists
}

The Ensure field in CommonResourceState is set to:

  • present if the creates file exists
  • absent if the creates file does not exist (or not specified)

Available Providers

ProviderExecution MethodDocumentation
posixDirect exec (no shell)Posix
shellVia /bin/sh -cShell

Properties

PropertyTypeDescription
commandstringCommand to run (defaults to name if not set)
cwdstringWorking directory for command execution
environment[]stringAdditional environment variables (KEY=value)
pathstringSearch path for executables (colon-separated)
returns[]intAcceptable exit codes (default: [0])
timeoutstringMaximum execution time (e.g., 30s, 5m)
createsstringFile path; skip execution if exists
refresh_onlyboolOnly execute via subscribe refresh
subscribe[]stringResources to watch for changes (type#name)
logoutputboolLog command output

Apply Logic

┌─────────────────────────────────────────┐
│ Get current state via Status()          │
└─────────────────┬───────────────────────┘
                  │
                  ▼
┌─────────────────────────────────────────┐
│ Check for subscribe refresh             │
└─────────────────┬───────────────────────┘
                  │
    ┌─────────────┴─────────────┐
    │ Subscribed resource       │
    │ changed?                  │
    └─────────────┬─────────────┘
              Yes │         No
                  ▼         │
          ┌───────────┐     │
          │ Execute   │     │
          └───────────┘     │
                            ▼
              ┌─────────────────────────┐
              │ Is desired state met?   │
              │ (creates file exists OR │
              │  refresh_only is true)  │
              └─────────────┬───────────┘
                        Yes │         No
                            ▼         │
                    ┌───────────┐     │
                    │ Skip      │     │
                    └───────────┘     │
                                      ▼
                        ┌─────────────────────────┐
                        │ Is refresh_only = true? │
                        └─────────────┬───────────┘
                                  Yes │         No
                                      ▼         │
                              ┌───────────┐     │
                              │ Skip      │     │
                              └───────────┘     │
                                                ▼
                                        ┌───────────┐
                                        │ Execute   │
                                        └───────────┘

Idempotency

The exec resource provides idempotency through two mechanisms:

Creates Property

The creates property specifies a file that indicates successful prior execution:

- exec:
    - extract-archive:
        command: tar xzf /tmp/app.tar.gz -C /opt
        creates: /opt/app/bin/app

Behavior:

  • If /opt/app/bin/app exists, skip execution
  • Useful for one-time setup commands
  • Provider checks file existence via Status()

Refresh Only Property

The refresh_only property limits execution to subscribe refreshes:

- exec:
    - reload-nginx:
        command: systemctl reload nginx
        refresh_only: true
        subscribe:
          - file#/etc/nginx/nginx.conf

Behavior:

  • Command only runs when subscribed resource changes
  • Without a subscribe trigger, command is skipped
  • Useful for reload/restart commands

Decision Table

ConditionAction
Subscribe triggeredExecute
creates file existsSkip
refresh_only: true + no triggerSkip
refresh_only: false + no createsExecute

Subscribe Behavior

Exec resources can subscribe to other resources and execute when they change:

- file:
    - /etc/app/config.yaml:
        ensure: present
        content: "..."

- exec:
    - reload-app:
        command: systemctl reload app
        refresh_only: true
        subscribe:
          - file#/etc/app/config.yaml

Subscribe takes precedence over other idempotency checks - if a subscribed resource changed, the command executes regardless of creates file existence.

Exit Code Validation

By default, exit code 0 indicates success. The returns property customizes acceptable codes:

- exec:
    - check-status:
        command: /usr/local/bin/check-health
        returns:
          - 0
          - 1
          - 2

Behavior:

  • Command succeeds if exit code is in returns list
  • Command fails if exit code is not in returns list
  • Used for desired state validation after execution

Noop Mode

In noop mode, the exec type:

  1. Queries current state normally (checks creates file)
  2. Evaluates subscribe triggers
  3. Logs what actions would be taken
  4. Sets appropriate NoopMessage:
    • “Would have executed”
    • “Would have executed via subscribe”
  5. Reports Changed: true if execution would occur
  6. Does not call provider Execute method

Desired State Validation

After execution (in non-noop mode), the type verifies success:

func (t *Type) isDesiredState(properties, status) bool {
    // Creates file check takes precedence
    if properties.Creates != "" && status.CreatesSatisfied {
        return true
    }

    // Refresh-only without execution is stable
    if status.ExitCode == nil && properties.RefreshOnly {
        return true
    }

    // Check exit code against acceptable returns
    returns := []int{0}
    if len(properties.Returns) > 0 {
        returns = properties.Returns
    }

    if status.ExitCode != nil {
        return slices.Contains(returns, *status.ExitCode)
    }

    return false
}

If the exit code is not in the acceptable returns list, an ErrDesiredStateFailed error is returned.

Command vs Name

The command property is optional. If not specified, the name is used as the command:

# These are equivalent:
- exec:
    - /usr/bin/myapp --config /etc/myapp.conf:

- exec:
    - run-myapp:
        command: /usr/bin/myapp --config /etc/myapp.conf

Using a descriptive name with explicit command is recommended for clarity.

Environment and Path

Commands can be configured with custom environment:

- exec:
    - build-app:
        command: make build
        cwd: /opt/app
        environment:
          - CC=gcc
          - CFLAGS=-O2
        path: /usr/local/bin:/usr/bin:/bin

Environment:

  • Added to the command’s environment
  • Format: KEY=value
  • Does not replace existing environment

Path:

  • Sets the PATH for executable lookup
  • Must be absolute directories
  • Colon-separated list

Subsections of Exec Type

Posix Provider

This document describes the implementation details of the Posix exec provider for executing commands without a shell.

Provider Selection

The Posix provider is the default exec provider. It is always available and returns priority 1 for all exec resources unless a different provider is explicitly requested via the provider property.

To use the shell provider instead, specify provider: shell in the resource properties.

Comparison with Shell Provider

FeaturePosixShell
Shell invocationNoYes (/bin/sh -c)
Pipes (|)Not supportedSupported
Redirections (>, <)Not supportedSupported
Shell builtins (cd, export)Not supportedSupported
Glob expansionNot supportedSupported
Command substitution ($(...))Not supportedSupported
Argument parsingshellquote.Split()Passed as single string
SecurityLower attack surfaceShell injection possible

When to use Posix (default):

  • Simple commands with arguments
  • When shell features are not needed
  • For better security (no shell injection risk)

When to use Shell:

  • Commands with pipes, redirections, or shell builtins
  • Complex command strings
  • When shell expansion is required

Operations

Execute

Process:

  1. Determine command source (Command property or Name if Command is empty)
  2. Parse command string into words using shellquote.Split()
  3. Extract command (first word) and arguments (remaining words)
  4. Execute via CommandRunner.ExecuteWithOptions()
  5. Optionally log output line-by-line if LogOutput is enabled

Command Parsing:

The command string is parsed using github.com/kballard/go-shellquote, which handles:

SyntaxExampleResult
Simple wordsecho hello world["echo", "hello", "world"]
Single quotesecho 'hello world'["echo", "hello world"]
Double quotesecho "hello world"["echo", "hello world"]
Escaped spacesecho hello\ world["echo", "hello world"]
Mixed quotingecho "it's a test"["echo", "it's a test"]

Execution Options:

OptionSourceDescription
CommandFirst word after parsingExecutable path or name
ArgsRemaining wordsCommand arguments
Cwdproperties.CwdWorking directory
Environmentproperties.EnvironmentAdditional env vars (KEY=VALUE format)
Pathproperties.PathSearch path for executables
Timeoutproperties.ParsedTimeoutMaximum execution time

Output Logging:

When LogOutput: true is set and a user logger is provided:

scanner := bufio.NewScanner(bytes.NewReader(stdout))
for scanner.Scan() {
    log.Info(scanner.Text())
}

Each line of stdout is logged as a separate Info message.

Error Handling:

ConditionBehavior
Empty command stringReturn error: “no command specified”
Invalid shell quotingReturn parsing error (e.g., “Unterminated single quote”)
Runner not configuredReturn error: “no command runner configured”
Command execution failsReturn error from runner
Non-zero exit codeReturn exit code (not an error by itself)

Status

Process:

  1. Create state with EnsurePresent (exec resources are always “present”)
  2. Check if Creates file exists via util.FileExists()
  3. Set CreatesSatisfied accordingly

State Fields:

FieldValue
Protocolio.choria.ccm.v1.resource.exec.state
ResourceTypeexec
NameResource name
Ensurepresent (always)
CreatesSatisfiedtrue if Creates file exists

Idempotency

The exec resource achieves idempotency through multiple mechanisms:

Creates File

If creates is specified and the file exists, the command does not run:

- exec:
    - /usr/bin/tar -xzf app.tar.gz:
        creates: /opt/app/bin/app
        cwd: /opt

RefreshOnly Mode

When refreshonly: true, the command only runs when triggered by a subscribed resource:

- exec:
    - systemctl reload httpd:
        refreshonly: true
        subscribe:
          - file#/etc/httpd/conf/httpd.conf

Exit Code Validation

The returns property specifies acceptable exit codes (default: [0]):

- exec:
    - /opt/app/healthcheck:
        returns: [0, 1, 2]  # 0=healthy, 1=degraded, 2=warning

Decision Flow

┌─────────────────────────────────────────┐
│ Should resource be applied?             │
└─────────────────┬───────────────────────┘
                  │
                  ▼
┌─────────────────────────────────────────┐
│ Subscribe triggered?                     │
│ (subscribed resource changed)            │
└─────────────┬───────────────┬───────────┘
              │ Yes           │ No
              ▼               ▼
┌─────────────────┐   ┌───────────────────┐
│ Execute command │   │ Creates satisfied? │
└─────────────────┘   └─────────┬─────────┘
                                │
                      ┌─────────┴─────────┐
                      │ Yes               │ No
                      ▼                   ▼
              ┌───────────────┐   ┌───────────────────┐
              │ Skip (stable) │   │ RefreshOnly mode? │
              └───────────────┘   └─────────┬─────────┘
                                            │
                                  ┌─────────┴─────────┐
                                  │ Yes               │ No
                                  ▼                   ▼
                          ┌───────────────┐   ┌───────────────┐
                          │ Skip (stable) │   │ Execute       │
                          └───────────────┘   └───────────────┘

Properties Validation

The model validates exec properties before execution:

PropertyValidation
nameMust be parseable by shellquote (balanced quotes)
timeoutMust be valid duration format (e.g., 30s, 5m)
subscribeEach entry must be type#name format
pathEach directory must be absolute (start with /)
environmentEach entry must be KEY=VALUE format with non-empty key and value

Platform Support

The Posix provider works on all platforms supported by Go’s os/exec package. It does not use any platform-specific system calls directly.

The command runner (model.CommandRunner) handles the actual process execution, which may have platform-specific implementations.

Security Considerations

No Shell Injection

Unlike the shell provider, the posix provider does not invoke a shell. Arguments are passed directly to the executable, preventing shell injection attacks:

# Safe with posix provider - $USER is passed literally, not expanded
- exec:
    - /bin/echo $USER:
        provider: posix  # Default

# Potentially dangerous with shell provider - $USER is expanded
- exec:
    - /bin/echo $USER:
        provider: shell

Path Validation

The path property only accepts absolute directory paths, preventing path traversal via relative paths.

Environment Validation

Environment variables must have non-empty keys and values, preventing injection of empty or malformed entries.

Shell Provider

This document describes the implementation details of the Shell exec provider for executing commands via /bin/sh.

Provider Selection

The Shell provider is selected when provider: shell is explicitly specified in the resource properties. It has a lower priority (99) than the Posix provider (1), so it is never automatically selected.

Availability: The provider checks for the existence of /bin/sh via util.FileExists(). If /bin/sh does not exist, the provider is not available.

Comparison with Posix Provider

FeatureShellPosix
Shell invocationYes (/bin/sh -c)No
Pipes (|)SupportedNot supported
Redirections (>, <, >>)SupportedNot supported
Shell builtins (cd, export, source)SupportedNot supported
Glob expansion (*.txt, ?)SupportedNot supported
Command substitution ($(...), `...`)SupportedNot supported
Variable expansion ($VAR, ${VAR})SupportedNot supported
Logical operators (&&, ||)SupportedNot supported
Argument parsingPassed as single stringshellquote.Split()
SecurityShell injection possibleLower attack surface

When to use Shell:

  • Commands with pipes: cat file.txt | grep pattern | sort
  • Commands with redirections: echo "data" > /tmp/file
  • Commands with shell builtins: cd /tmp && pwd
  • Commands with variable expansion: echo $HOME
  • Complex one-liners with logical operators

When to use Posix (default):

  • Simple commands with arguments
  • When shell features are not needed
  • For better security (no shell injection risk)

Operations

Execute

Process:

  1. Determine command source (Command property or Name if Command is empty)
  2. Validate command is not empty
  3. Execute via CommandRunner.ExecuteWithOptions() with /bin/sh -c "<command>"
  4. Optionally log output line-by-line if LogOutput is enabled

Execution Method:

The entire command string is passed to the shell as a single argument:

/bin/sh -c "<entire command string>"

This allows the shell to interpret all shell syntax, including:

  • Pipes and redirections
  • Variable expansion
  • Glob patterns
  • Command substitution
  • Logical operators

Execution Options:

OptionValueDescription
Command/bin/shShell executable path
Args["-c", "<command>"]Shell flag and command string
Cwdproperties.CwdWorking directory
Environmentproperties.EnvironmentAdditional env vars (KEY=VALUE format)
Pathproperties.PathSearch path for executables
Timeoutproperties.ParsedTimeoutMaximum execution time

Output Logging:

When LogOutput: true is set and a user logger is provided:

scanner := bufio.NewScanner(bytes.NewReader(stdout))
for scanner.Scan() {
    log.Info(scanner.Text())
}

Each line of stdout is logged as a separate Info message.

Error Handling:

ConditionBehavior
Empty command stringReturn error: “no command to execute”
Runner not configuredReturn error: “no command runner configured”
Shell not foundProvider not available (checked at selection time)
Command execution failsReturn error from runner
Non-zero exit codeReturn exit code (not an error by itself)

Status

Process:

  1. Create state with EnsurePresent (exec resources are always “present”)
  2. Check if Creates file exists via util.FileExists()
  3. Set CreatesSatisfied accordingly

State Fields:

FieldValue
Protocolio.choria.ccm.v1.resource.exec.state
ResourceTypeexec
NameResource name
Ensurepresent (always)
CreatesSatisfiedtrue if Creates file exists

Use Cases

Pipes and Filters

- exec:
    - filter-logs:
        command: cat /var/log/app.log | grep ERROR | tail -100 > /tmp/errors.txt
        provider: shell
        creates: /tmp/errors.txt

Conditional Execution

- exec:
    - ensure-running:
        command: pgrep myapp || /opt/myapp/bin/start
        provider: shell

Complex Scripts

- exec:
    - deploy-app:
        command: |
          cd /opt/app &&
          git pull origin main &&
          npm install &&
          npm run build &&
          systemctl restart app
        provider: shell
        timeout: 5m

Variable Expansion

- exec:
    - backup-home:
        command: tar -czf /backup/home-$(date +%Y%m%d).tar.gz $HOME
        provider: shell

Idempotency

The shell provider uses the same idempotency mechanisms as the posix provider:

Creates File

If creates is specified and the file exists, the command does not run:

- exec:
    - extract-archive:
        command: cd /opt && tar -xzf /tmp/app.tar.gz
        provider: shell
        creates: /opt/app/bin/app

RefreshOnly Mode

When refreshonly: true, the command only runs when triggered by a subscribed resource:

- exec:
    - reload-nginx:
        command: nginx -t && systemctl reload nginx
        provider: shell
        refreshonly: true
        subscribe:
          - file#/etc/nginx/nginx.conf

Exit Code Validation

The returns property specifies acceptable exit codes (default: [0]):

- exec:
    - check-service:
        command: systemctl is-active myapp || true
        provider: shell
        returns: [0]

Security Considerations

Shell Injection Risk

The shell provider passes the command string directly to /bin/sh, making it vulnerable to shell injection if user input is incorporated:

# DANGEROUS if filename comes from untrusted input
- exec:
    - process-file:
        command: cat {{ user_provided_filename }} | process
        provider: shell

Mitigations:

  • Validate and sanitize any templated values
  • Use the posix provider when shell features aren’t needed
  • Prefer explicit file paths over user-provided values

Environment Variable Exposure

Shell commands can access environment variables, including sensitive ones:

# $SECRET_KEY will be expanded by the shell
- exec:
    - use-secret:
        command: myapp --key=$SECRET_KEY
        provider: shell

Consider using the environment property to explicitly pass required variables rather than relying on inherited environment.

Command Logging

The full command string (including any expanded variables) may appear in logs. Avoid embedding secrets directly in commands:

# BAD - password visible in logs
- exec:
    - bad-example:
        command: mysql -p'secret123' -e 'SELECT 1'
        provider: shell

# BETTER - use environment variable
- exec:
    - better-example:
        command: mysql -p"$MYSQL_PWD" -e 'SELECT 1'
        provider: shell
        environment:
          - MYSQL_PWD={{ lookup('data.mysql_password') }}

Platform Support

The shell provider requires /bin/sh to be available. This is standard on:

  • Linux distributions
  • macOS
  • BSD variants
  • Most Unix-like systems

On Windows, the provider will not be available unless /bin/sh exists (e.g., via WSL or Cygwin).

Shell Compatibility

The provider uses /bin/sh, which is typically:

  • Linux: Often a symlink to bash, dash, or another POSIX-compliant shell
  • macOS: /bin/sh is bash (older) or zsh (newer) in POSIX mode
  • BSD: Usually ash or similar

For maximum portability, use POSIX shell syntax and avoid bash-specific features like:

  • Arrays (arr=(1 2 3))
  • [[ conditionals (use [ instead)
  • source (use . instead)
  • Process substitution (<(command))

File Type

This document describes the design of the file resource type for managing files and directories.

Overview

The file resource manages files and directories with three aspects:

  • Existence: Whether the file/directory exists or is absent
  • Content: The contents of a file (from inline content or source file)
  • Attributes: Owner, group, and permissions

Provider Interface

File providers must implement the FileProvider interface:

type FileProvider interface {
    model.Provider

    CreateDirectory(ctx context.Context, dir string, owner string, group string, mode string) error
    Store(ctx context.Context, file string, contents []byte, source string, owner string, group string, mode string) error
    Status(ctx context.Context, file string) (*model.FileState, error)
}

Method Responsibilities

MethodResponsibility
StatusQuery current file state (existence, type, content hash, attributes)
StoreCreate or update a file with content and attributes
CreateDirectoryCreate a directory with attributes

Status Response

The Status method returns a FileState containing:

type FileState struct {
    CommonResourceState
    Metadata *FileMetadata
}

type FileMetadata struct {
    Name     string         // File path
    Checksum string         // SHA256 hash of contents (files only)
    Owner    string         // Owner username
    Group    string         // Group name
    Mode     string         // Permissions in octal (e.g., "0644")
    Provider string         // Provider name (e.g., "posix")
    MTime    time.Time      // Modification time
    Size     int64          // File size in bytes
    Extended map[string]any // Provider-specific metadata
}

The Ensure field in CommonResourceState is set to:

  • present if a regular file exists
  • directory if a directory exists
  • absent if the path does not exist

Available Providers

ProviderPlatformDocumentation
posixUnix/LinuxPosix

Ensure States

ValueDescription
presentPath must be a regular file with specified content
absentPath must not exist
directoryPath must be a directory

Content Sources

Files can receive content from two mutually exclusive sources:

PropertyDescription
contentsInline string content (template-resolved)
sourcePath to local file to copy from
# Inline content with template
- file:
    - /etc/motd:
        ensure: present
        content: |
          Welcome to {{ lookup('facts.hostname') }}
          Managed by CCM
        owner: root
        group: root
        mode: "0644"

# Copy from source file
- file:
    - /etc/app/config.yaml:
        ensure: present
        source: files/config.yaml
        owner: app
        group: app
        mode: "0640"

When using source, the path is relative to the manifest’s working directory if one is set.

Required Properties

Unlike some resources, file resources require explicit attributes:

PropertyRequiredDescription
ownerYesUsername that owns the file
groupYesGroup that owns the file
modeYesPermissions in octal notation

This prevents accidental creation of files with default or inherited permissions.

Apply Logic

┌─────────────────────────────────────────┐
│ Get current state via Status()          │
└─────────────────┬───────────────────────┘
                  │
                  ▼
┌─────────────────────────────────────────┐
│ Is current state desired state?         │
└─────────────────┬───────────────────────┘
              Yes │         No
                  ▼         │
          ┌───────────┐     │
          │ No change │     │
          └───────────┘     │
                            ▼
              ┌─────────────────────────┐
              │ What is desired ensure? │
              └─────────────┬───────────┘
                            │
    ┌───────────────────────┼───────────────────────┐
    │ absent                │ directory             │ present
    ▼                       ▼                       ▼
┌────────────┐      ┌───────────────┐      ┌───────────────┐
│ Remove     │      │ CreateDir     │      │ Store         │
│ (os.Remove)│      │               │      │               │
└────────────┘      └───────────────┘      └───────────────┘

Idempotency

The file resource checks multiple attributes for idempotency:

State Checks (in order)

  1. Ensure match: Current type matches desired (present/absent/directory)
  2. Content match: SHA256 checksum of contents matches (for ensure: present)
  3. Owner match: Current owner matches desired
  4. Group match: Current group matches desired
  5. Mode match: Current permissions match desired

Decision Table

DesiredCurrent StateAction
absentabsentNone
absentpresent/directoryRemove
directorydirectory + matching attrsNone
directoryabsent/presentCreateDirectory
directorydirectory + wrong attrsCreateDirectory (updates attrs)
presentpresent + matching allNone
presentabsentStore
presentpresent + wrong contentStore
presentpresent + wrong attrsStore

Content Comparison

Content is compared using SHA256 checksums:

SourceChecksum Method
contents propertySha256HashBytes([]byte(contents))
source propertySha256HashFile(adjustedPath)
Existing fileSha256HashFile(filePath)

Mode Validation

File modes are validated during resource creation:

Valid Formats:

  • "0644" - Standard octal
  • "644" - Without leading zero
  • "0o755" - With 0o prefix
  • "0O700" - With 0O prefix

Validation Rules:

  • Must be valid octal number (digits 0-7)
  • Must be ≤ 0777 (no setuid/setgid/sticky via mode)

Path Validation

File paths must be:

  • Absolute (start with /)
  • Clean (no . or .. components, filepath.Clean(path) == path)
if filepath.Clean(p.Name) != p.Name {
    return fmt.Errorf("file path must be absolute")
}

Working Directory

When a manifest has a working directory (e.g., extracted from an archive), the source property is resolved relative to it:

if properties.Source != "" && mgr.WorkingDirectory() != "" {
    source = filepath.Join(mgr.WorkingDirectory(), properties.Source)
}

This allows manifests bundled with their source files to use relative paths.

Noop Mode

In noop mode, the file type:

  1. Queries current state normally
  2. Computes content checksums
  3. Logs what actions would be taken
  4. Sets appropriate NoopMessage:
    • “Would have created the file”
    • “Would have created directory”
    • “Would have removed the file”
  5. Reports Changed: true if changes would occur
  6. Does not call provider Store/CreateDirectory methods
  7. Does not remove files

Desired State Validation

After applying changes (in non-noop mode), the type verifies the file reached the desired state by calling Status() again and checking all attributes match. If validation fails, ErrDesiredStateFailed is returned.

Subsections of File Type

Posix Provider

This document describes the implementation details of the Posix file provider for managing files and directories on Unix-like systems.

Provider Selection

The Posix provider is the default and only file provider. It is always available and returns priority 1 for all file resources.

Operations

Store (Create/Update File)

Process:

  1. Verify parent directory exists
  2. Parse file mode from octal string
  3. Open source file if source property is set
  4. Create temporary file in the same directory as target
  5. Set file permissions on temp file
  6. Write content (from source file or contents property)
  7. Set ownership (chown) on temp file
  8. Close temp file
  9. Atomic rename temp file to target path

Atomic Write Pattern:

[parent dir]/<basename>.* (temp file)
    ↓ chmod (set permissions)
    ↓ write content
    ↓ chown (set owner/group)
    ↓ close
    ↓ rename
[parent dir]/<basename> (final file)

The temp file is created in the same directory as the target to ensure os.Rename() is atomic (same filesystem).

Content Sources:

PropertyBehavior
contentsWrite string directly to file (template-resolved)
sourceCopy from local file path (adjusted for working directory)

If both are empty, an empty file is created.

Error Handling:

ConditionBehavior
Parent directory doesn’t existReturn error: “is not a directory”
Invalid mode formatReturn error from strconv.ParseUint
Source file not foundReturn error from os.Open
Permission deniedReturn error from underlying syscall
Rename failureReturn error: “could not rename temporary file”

CreateDirectory

Process:

  1. Parse file mode from octal string
  2. Create directory and parents via os.MkdirAll()
  3. Lookup numeric UID/GID from owner/group names
  4. Set permissions via os.Chmod() (ensures correct mode even if umask affected MkdirAll)
  5. Set ownership via os.Chown()

Command Sequence:

os.MkdirAll(dir, parsedMode)    // Create directory tree
os.Chmod(dir, parsedMode)        // Ensure correct permissions
os.Chown(dir, uid, gid)          // Set ownership

The explicit Chmod after MkdirAll is necessary because MkdirAll applies the process umask to the mode.

Status

Process:

  1. Initialize state with default metadata
  2. Call os.Stat() on file path
  3. Based on result, populate state accordingly

State Detection:

os.Stat() ResultEnsure ValueMetadata
File existspresentSize, mtime, owner, group, mode, checksum
Directory existsdirectorySize, mtime, owner, group, mode
os.ErrNotExistabsentNone
os.ErrPermissionabsentNone (logged as warning)
Other error(unchanged)None (logged as warning)

Metadata Collection:

FieldSource
NameFrom properties
Provider“posix”
SizeFileInfo.Size()
MTimeFileInfo.ModTime()
Ownerutil.GetFileOwner() - resolves UID to username
Grouputil.GetFileOwner() - resolves GID to group name
Modeutil.GetFileOwner() - octal string (e.g., “0644”)
Checksumutil.Sha256HashFile() - SHA256 hash (files only)

Note: Checksum is only calculated for regular files, not directories.

Idempotency

The file resource achieves idempotency by comparing current state against desired state:

State Checks

The isDesiredState() function checks (in order):

  1. Ensure value matches - present, absent, or directory
  2. Content checksum matches - SHA256 of contents vs existing file (for files only)
  3. Owner matches - Username comparison
  4. Group matches - Group name comparison
  5. Mode matches - Octal permission string comparison

Decision Flow

┌─────────────────────────────────────────┐
│ What is the desired ensure state?       │
└─────────────────┬───────────────────────┘
                  │
    ┌─────────────┼─────────────┬─────────────┐
    │ absent      │ directory   │ present     │
    ▼             ▼             ▼             │
┌───────────┐ ┌───────────┐ ┌───────────────┐ │
│ File      │ │ Directory │ │ File exists?  │ │
│ exists?   │ │ exists?   │ │ Content match?│ │
└─────┬─────┘ └─────┬─────┘ │ Owner match?  │ │
      │             │       │ Group match?  │ │
  Yes │ No      Yes │ No    │ Mode match?   │ │
      ▼             ▼       └───────┬───────┘ │
┌─────────┐ ┌───────────┐           │         │
│ Remove  │ │ Stable    │     All Yes│    No   │
│ file    │ │           │           ▼         ▼
└─────────┘ └───────────┘   ┌───────────┐ ┌───────────┐
                            │ Stable    │ │ Store     │
                            └───────────┘ └───────────┘

Content Comparison

For files with ensure: present:

Content SourceChecksum Calculation
contents propertySha256HashBytes([]byte(contents))
source propertySha256HashFile(adjustedSourcePath)

The source path is adjusted based on the manager’s working directory when set.

Mode Validation

File modes are validated during resource creation:

  1. Strip optional 0o or 0O prefix
  2. Parse as octal number (base 8)
  3. Validate range: must be ≤ 0777

Valid Mode Examples:

InputParsed Value
"0644"0o644
"644"0o644
"0o755"0o755
"0O700"0o700

Invalid Mode Examples:

InputError
"0888"Invalid octal digit
"1777"Exceeds maximum (setuid/setgid not supported via mode)
"rw-r--r--"Not octal format

Ownership Resolution

Owner and group names are resolved to numeric UID/GID via:

uid, gid, err := util.LookupOwnerGroup(owner, group)

This uses the system’s user database (/etc/passwd, /etc/group, or equivalent):

  • user.Lookup(owner) → UID
  • user.LookupGroup(group) → GID

Error Handling:

ConditionBehavior
User not foundReturn error: “could not lookup user”
Group not foundReturn error: “could not lookup group”
Invalid UID/GID formatReturn error from strconv.Atoi

Working Directory Support

When a manager has a working directory set (e.g., from extracted manifest), the source property path is adjusted:

func (t *Type) adjustedSource(properties *model.FileResourceProperties) string {
    source := properties.Source
    if properties.Source != "" && t.mgr.WorkingDirectory() != "" {
        source = filepath.Join(t.mgr.WorkingDirectory(), properties.Source)
    }
    return source
}

This allows manifests to use relative paths for source files bundled with the manifest.

Platform Support

The Posix provider uses Unix-specific system calls:

OperationSystem Call
Get file owner/groupsyscall.Stat_t (UID/GID from stat)
Set ownershipos.Chown()chown(2)
Set permissionsos.Chmod()chmod(2)

The provider has separate implementations for Unix and Windows (file_unix.go, file_windows.go in internal/util), with Windows returning errors for ownership operations.

Security Considerations

Atomic Writes

Files are written atomically via temp file + rename. This prevents:

  • Partial file reads during write
  • Corruption if process is interrupted
  • Race conditions with concurrent readers

Permission Ordering

Permissions and ownership are set on the temp file before rename:

  1. Chmod - Set permissions
  2. Write content
  3. Chown - Set ownership
  4. Rename to target

This ensures the file never exists at the target path with incorrect permissions.

Path Validation

File paths must be absolute and clean (no . or .. components):

if filepath.Clean(p.Name) != p.Name {
    return fmt.Errorf("file path must be absolute")
}

Required Properties

Owner, group, and mode are required properties and cannot be empty, preventing accidental creation of files with default/inherited permissions.

Package Type

This document describes the design of the package resource type for managing software packages.

Overview

The package resource manages software packages with two aspects:

  • Existence: Whether the package is installed or absent
  • Version: The specific version installed (when applicable)

Provider Interface

Package providers must implement the PackageProvider interface:

type PackageProvider interface {
    model.Provider

    Install(ctx context.Context, pkg string, version string) error
    Upgrade(ctx context.Context, pkg string, version string) error
    Downgrade(ctx context.Context, pkg string, version string) error
    Uninstall(ctx context.Context, pkg string) error
    Status(ctx context.Context, pkg string) (*model.PackageState, error)
    VersionCmp(versionA, versionB string, ignoreTrailingZeroes bool) (int, error)
}

Method Responsibilities

MethodResponsibility
StatusQuery current package state (installed version or absent)
InstallInstall package at specified version (or latest if “latest”)
UpgradeUpgrade package to specified version
DowngradeDowngrade package to specified version
UninstallRemove the package
VersionCmpCompare two version strings (-1, 0, 1)

Status Response

The Status method returns a PackageState containing:

type PackageState struct {
    CommonResourceState
    Metadata *PackageMetadata
}

type PackageMetadata struct {
    Name        string         // Package name
    Version     string         // Installed version
    Arch        string         // Architecture (e.g., "x86_64")
    License     string         // Package license
    URL         string         // Package URL
    Summary     string         // Short description
    Description string         // Full description
    Provider    string         // Provider name (e.g., "dnf")
    Extended    map[string]any // Provider-specific metadata
}

The Ensure field in CommonResourceState is set to:

  • The installed version string if the package is installed
  • absent if the package is not installed

Available Providers

ProviderPackage ManagerDocumentation
dnfDNF (Fedora/RHEL)DNF
aptAPT (Debian/Ubuntu)APT

Ensure States

ValueDescription
presentPackage must be installed (any version)
absentPackage must not be installed
latestPackage must be upgraded to latest available
<version>Package must be at specific version
# Install any version
- package:
    - vim:
        ensure: present

# Install latest version
- package:
    - vim:
        ensure: latest

# Install specific version
- package:
    - nginx:
        ensure: "1.24.0-1.el9"

# Remove package
- package:
    - telnet:
        ensure: absent

Apply Logic

Phase 1: Handle Special Cases

┌─────────────────────────────────────────┐
│ Get current state via Status()          │
└─────────────────┬───────────────────────┘
                  │
                  ▼
┌─────────────────────────────────────────┐
│ Is ensure = "latest"?                   │
└─────────────────┬───────────────────────┘
              Yes │         No
                  ▼         │
    ┌─────────────────────┐ │
    │ Is package absent?  │ │
    └─────────────┬───────┘ │
              Yes │     No  │
                  ▼     ▼   │
          ┌────────┐ ┌────────┐
          │Install │ │Upgrade │
          │latest  │ │latest  │
          └────────┘ └────────┘
                            │
                            ▼
              ┌─────────────────────────┐
              │ Is desired state met?   │
              └─────────────┬───────────┘
                        Yes │         No
                            ▼         │
                    ┌───────────┐     │
                    │ No change │     ▼
                    └───────────┘  (Phase 2)

Phase 2: Handle Ensure Values

              ┌─────────────────────────┐
              │ What is desired ensure? │
              └─────────────┬───────────┘
                            │
    ┌───────────────────────┼───────────────────────┐
    │ absent                │ present               │ <version>
    ▼                       ▼                       ▼
┌────────────┐      ┌───────────────┐      ┌───────────────┐
│ Uninstall  │      │ Is absent?    │      │ Is absent?    │
└────────────┘      └───────┬───────┘      └───────┬───────┘
                        Yes │     No           Yes │     No
                            ▼     ▼                ▼     ▼
                    ┌────────┐ ┌────────┐  ┌────────┐ ┌────────────┐
                    │Install │ │No      │  │Install │ │Compare     │
                    │        │ │change  │  │version │ │versions    │
                    └────────┘ └────────┘  └────────┘ └─────┬──────┘
                                                            │
                                           ┌────────────────┼────────────────┐
                                           │ current <      │ current =      │ current >
                                           ▼                ▼                ▼
                                   ┌───────────┐    ┌───────────┐    ┌───────────┐
                                   │ Upgrade   │    │ No change │    │ Downgrade │
                                   └───────────┘    └───────────┘    └───────────┘

Version Comparison

The VersionCmp method compares two version strings:

Return ValueMeaning
-1versionA < versionB (upgrade needed)
0versionA == versionB (no change)
1versionA > versionB (downgrade needed)

Version comparison is delegated to the provider, allowing platform-specific version parsing (e.g., RPM epoch handling, Debian revision suffixes).

Idempotency

The package resource is idempotent through state comparison:

Decision Table

DesiredCurrent StateAction
ensure: presentinstalled (any version)None
ensure: presentabsentInstall
ensure: absentabsentNone
ensure: absentinstalledUninstall
ensure: latestabsentInstall latest
ensure: latestinstalledUpgrade (always runs)
ensure: <version>same versionNone
ensure: <version>older versionUpgrade
ensure: <version>newer versionDowngrade
ensure: <version>absentInstall

Special Case: ensure: latest

When ensure: latest is used:

  • The package manager determines what “latest” means
  • Upgrade is always called when the package exists (package manager is idempotent)
  • The type cannot verify if “latest” was achieved (package managers may report stale data)
  • Desired state validation only checks that the package is not absent

Package Name Validation

Package names are validated to prevent injection attacks:

Allowed Characters:

  • Alphanumeric (a-z, A-Z, 0-9)
  • Period (.), underscore (_), plus (+)
  • Colon (:), tilde (~), hyphen (-)

Rejected:

  • Shell metacharacters (;, |, &, $, etc.)
  • Whitespace
  • Quotes and backticks
  • Path separators

Version strings (when ensure is a version) are also validated for dangerous characters.

Noop Mode

In noop mode, the package type:

  1. Queries current state normally
  2. Computes version comparison
  3. Logs what actions would be taken
  4. Sets appropriate NoopMessage:
    • “Would have installed latest”
    • “Would have upgraded to latest”
    • “Would have installed version X”
    • “Would have upgraded to X”
    • “Would have downgraded to X”
    • “Would have uninstalled”
  5. Reports Changed: true if changes would occur
  6. Does not call provider Install/Upgrade/Downgrade/Uninstall methods

Desired State Validation

After applying changes (in non-noop mode), the type verifies the package reached the desired state:

func (t *Type) isDesiredState(properties, state) bool {
    switch properties.Ensure {
    case "present":
        // Any installed version is acceptable
        return state.Ensure != "absent"

    case "absent":
        return state.Ensure == "absent"

    case "latest":
        // Cannot verify "latest", just check not absent
        return state.Ensure != "absent"

    default:
        // Specific version must match
        return VersionCmp(state.Ensure, properties.Ensure, false) == 0
    }
}

If the desired state is not reached, an ErrDesiredStateFailed error is returned.

Subsections of Package Type

APT Provider

This document describes the implementation details of the APT package provider for Debian-based systems.

Environment

All commands are executed with the following environment variables to ensure non-interactive operation:

VariableValuePurpose
DEBIAN_FRONTENDnoninteractivePrevents dpkg from prompting for user input
APT_LISTBUGS_FRONTENDnoneSuppresses apt-listbugs prompts
APT_LISTCHANGES_FRONTENDnoneSuppresses apt-listchanges prompts

Concurrency

A global package lock (model.PackageGlobalLock) is held during all command executions to prevent concurrent apt/dpkg operations within the same process. This prevents lock contention on /var/lib/dpkg/lock.

Operations

Status Check

Command:

dpkg-query -W -f='${Package} ${Version} ${Architecture} ${db:Status-Status}' <package>

Behavior:

  • Exit code 0 with installed status → Package is present, returns version info
  • Exit code non-zero OR status not installed → Package is absent

Package States: The db:Status-Status field can return various values. Only installed is treated as present:

StatusTreated AsDescription
installedPresentPackage fully installed
config-filesAbsentRemoved but config files remain
half-installedAbsentInstallation started but failed
half-configuredAbsentConfiguration failed
unpackedAbsentUnpacked but not configured
not-installedAbsentNot installed

Treating non-installed states as absent allows apt-get install to repair broken installations.

Install

Ensure Present:

apt-get install -y -q -o DPkg::Options::=--force-confold <package>

Ensure Latest:

apt-cache policy <package>                    # Get candidate version
apt-get install -y -q -o DPkg::Options::=--force-confold <package>=<version>

Specific Version:

apt-get install -y -q -o DPkg::Options::=--force-confold --allow-downgrades <package>=<version>

Flags:

FlagPurpose
-yAssume yes to prompts
-qQuiet output
-o DPkg::Options::=--force-confoldKeep existing config files on upgrade
--allow-downgradesAllow installing older versions (specific version only)

Upgrade

Delegates to Install() with the target version. The --allow-downgrades flag is only added for specific version requests, not for latest.

Downgrade

Delegates to Install() with the target version. The --allow-downgrades flag enables this operation.

Uninstall

Command:

apt-get -q -y remove <package>

Note: Uses remove not purge, so configuration files are preserved. A subsequent install will find existing config files.

Latest Available Version

Command:

apt-cache policy <package>

Parsing: Extracts the Candidate: line from output.

Example output:

zsh:
  Installed: 5.9-8+b18
  Candidate: 5.9-8+b18
  Version table:
 *** 5.9-8+b18 500
        500 http://deb.debian.org/debian trixie/main amd64 Packages
        100 /var/lib/dpkg/status

Version Comparison

Version comparison follows the Debian Policy Manual algorithm.

Version Format

[epoch:]upstream_version[-debian_revision]
ComponentRequiredDescription
epochNoInteger, default 0. Higher epoch always wins.
upstream_versionYesThe main version from upstream
debian_revisionNoDebian-specific revision

Examples:

  • 1.0 → epoch=0, upstream=1.0, revision=""
  • 1:2.0-3 → epoch=1, upstream=2.0, revision=3
  • 2:1.0.0+git-20190109-0ubuntu2 → epoch=2, upstream=1.0.0+git-20190109, revision=0ubuntu2

Comparison Algorithm

  1. Compare epochs numerically - Higher epoch wins regardless of other components

  2. Compare upstream_version and debian_revision using the Debian string comparison:

    The string is processed left-to-right in segments:

    a. Tildes (~) - Compared first. More tildes = earlier version. Tilde sorts before everything, even empty string.

    • 1.0~alpha < 1.0 (tilde before empty)
    • 1.0~~ < 1.0~ (more tildes = earlier)

    b. Letters (A-Za-z) - Compared lexically (ASCII order)

    • Letters sort before non-letters

    c. Non-letters (., +, -) - Compared lexically

    d. Digits - Compared numerically (not lexically)

    • 9 < 13 (numeric comparison)

    These steps repeat until a difference is found or both strings are exhausted.

Comparison Examples

ABResultReason
1.02.0A < BNumeric comparison
1:1.02.0A > BEpoch 1 > epoch 0
1.0~alpha1.0A < BTilde sorts before empty
1.0~alpha1.0~betaA < BLexical: alpha < beta
1.0.11.0.2A < BNumeric: 1 < 2
1.0-11.0-2A < BRevision comparison
1.0a1.0-A < BLetters sort before non-letters

Implementation

The version comparison is implemented in version.go, ported from Puppet’s Puppet::Util::Package::Version::Debian module. It provides:

  • ParseVersion(string) - Parse a version string into components
  • CompareVersionStrings(a, b) - Compare two version strings directly
  • Version.Compare(other) - Compare parsed versions (-1, 0, 1)
  • Helper methods: LessThan, GreaterThan, Equal, etc.

DNF Provider

This document describes the implementation details of the DNF package provider for RHEL/Fedora-based systems.

Concurrency

A global package lock (model.PackageGlobalLock) is held during all command executions to prevent concurrent dnf/rpm operations within the same process. This prevents lock contention on the RPM database.

Operations

Status Check

Command:

rpm -q <package> --queryformat '%{NAME} %|EPOCH?{%{EPOCH}}:{0}| %{VERSION} %{RELEASE} %{ARCH}'

Query Format (NEVRA): The query format extracts the full NEVRA (Name, Epoch, Version, Release, Architecture):

FieldDescription
%{NAME}Package name
%|EPOCH?{%{EPOCH}}:{0}|Epoch (0 if not set)
%{VERSION}Upstream version
%{RELEASE}Release/build number
%{ARCH}Architecture (x86_64, noarch, etc.)

Example output:

zsh 0 5.8 9.el9 x86_64

Behavior:

  • Exit code 0 → Package is present, parses NEVRA components
  • Exit code non-zero → Package is absent

Returned Version Format: The version returned combines VERSION and RELEASE: 5.8-9.el9

The epoch and release are stored separately in the Extended metadata.

Install

Ensure Present or Latest:

dnf install -y <package>

Specific Version:

dnf install -y <package>-<version>

Flags:

FlagPurpose
-yAssume yes to all prompts

Note: DNF uses - (hyphen) to separate package name from version, unlike APT which uses =.

Upgrade

Delegates to Install(). DNF’s install command handles upgrades automatically when a newer version is available or specified.

Downgrade

Command:

dnf downgrade -y <package>-<version>

Note: Unlike upgrade, downgrade uses a dedicated DNF command rather than delegating to install.

Uninstall

Command:

dnf remove -y <package>

Version Format

RPM versions follow the EVR (Epoch:Version-Release) format:

[epoch:]version-release
ComponentRequiredDescription
epochNoInteger, default 0. Higher epoch always wins.
versionYesUpstream version number
releaseYesDistribution-specific release/build number

Examples:

  • 5.8-9.el9 → epoch=0, version=5.8, release=9.el9
  • 1:2.0-3.fc39 → epoch=1, version=2.0, release=3.fc39
  • 0:1.0.0-1.el9 → epoch=0 (explicit), version=1.0.0, release=1.el9

Version Comparison

Version comparison uses a generic algorithm ported from Puppet, implemented in internal/util.VersionCmp().

Algorithm

The version string is tokenized into segments by splitting on -, ., digits, and non-digit sequences. Segments are compared left-to-right:

  1. Hyphens (-) - A hyphen loses to any other character
  2. Dots (.) - A dot loses to any non-hyphen character
  3. Digit sequences - Compared numerically, except:
    • Leading zeros trigger lexical comparison (01 vs 1 compared as strings)
  4. Non-digit sequences - Compared lexically (case-insensitive)

If all segments match, falls back to whole-string comparison.

Trailing Zero Normalization

When ignoreTrailingZeroes is enabled, trailing .0 segments before the first - are removed:

  • 1.0.0-rc11-rc1
  • 2.0.02

This allows 1.0.0 to equal 1.0 when the flag is set.

Comparison Examples

ABResultReason
1.02.0A < BNumeric: 1 < 2
1.101.9A > BNumeric: 10 > 9
1.01.0.1A < BA exhausted first
1.0-11.0-2A < BRelease comparison
1.0a1.0bA < BLexical: a < b
011A < BLeading zero: lexical comparison
1.0.01.0A > BWithout normalization
1.0.01.0A = BWith ignoreTrailingZeroes=true

Implementation

The version comparison is implemented in internal/util/util.go and provides:

  • VersionCmp(a, b, ignoreTrailingZeroes) - Compare two version strings
  • Returns -1 (a < b), 0 (a == b), or 1 (a > b)

Scaffold Type

This document describes the design of the scaffold resource type for rendering template directories into target directories.

Overview

The scaffold resource renders files from a source template directory to a target directory:

  • Rendering: Process templates using Go text/template or Jet engine
  • Synchronization: Detect changed, stable, and purgeable files
  • Cleanup: Optionally remove files in the target not present in the source

Templates have access to facts and data from Hiera, enabling dynamic configuration generation from directory structures.

Provider Interface

Scaffold providers must implement the ScaffoldProvider interface:

type ScaffoldProvider interface {
    model.Provider

    Remove(ctx context.Context, prop *model.ScaffoldResourceProperties, state *model.ScaffoldState) error
    Scaffold(ctx context.Context, env *templates.Env, prop *model.ScaffoldResourceProperties, noop bool) (*model.ScaffoldState, error)
    Status(ctx context.Context, env *templates.Env, prop *model.ScaffoldResourceProperties) (*model.ScaffoldState, error)
}

Method Responsibilities

MethodResponsibility
StatusRender in noop mode to determine current state of managed files
ScaffoldRender templates to target directory (or noop to preview changes)
RemoveDelete managed files (changed and stable) and clean up directories

Status Response

The Status method returns a ScaffoldState containing:

type ScaffoldState struct {
    CommonResourceState
    Metadata *ScaffoldMetadata
}

type ScaffoldMetadata struct {
    Name         string                 // Target directory
    Provider     string                 // Provider name (e.g., "choria")
    TargetExists bool                   // Whether target directory exists
    Changed      []string               // Files created or modified
    Purged       []string               // Files removed (not in source)
    Stable       []string               // Files unchanged
    Engine       ScaffoldResourceEngine // Template engine used
}

The Ensure field in CommonResourceState is set to:

  • present if the target directory exists
  • absent if the target directory does not exist

Available Providers

ProviderEngine SupportDocumentation
choriaGo, JetChoria

Ensure States

ValueDescription
presentTarget directory must exist with rendered template files
absentManaged files must be removed from the target

Template Engines

Two template engines are supported:

EngineLibraryDefault DelimitersDescription
goGo text/template{{ / }}Standard Go templates
jetJet templating[[ / ]]Jet template language

The engine defaults to jet if not specified. Delimiters can be customized via left_delimiter and right_delimiter properties.

Properties

PropertyTypeRequiredDescription
sourcestringYesSource template directory path or URL
enginestringNoTemplate engine: go or jet (default: jet)
skip_emptyboolNoSkip empty files in rendered output
left_delimiterstringNoCustom left template delimiter
right_delimiterstringNoCustom right template delimiter
purgeboolNoRemove files in target not present in source
post[]map[string]stringNoPost-processing: glob pattern to command mapping
# Render configuration templates using Jet engine
- scaffold:
    - /etc/app:
        ensure: present
        source: templates/app
        engine: jet
        purge: true

# Render with Go templates and custom delimiters
- scaffold:
    - /etc/myservice:
        ensure: present
        source: templates/myservice
        engine: go
        left_delimiter: "<<"
        right_delimiter: ">>"

# With post-processing commands
- scaffold:
    - /opt/app:
        ensure: present
        source: templates/app
        post:
          - "*.go": "go fmt {}"

Apply Logic

┌─────────────────────────────────────────┐
│ Get current state via Status()          │
└─────────────────┬───────────────────────┘
                  │
                  ▼
┌─────────────────────────────────────────┐
│ Is current state desired state?         │
└─────────────────┬───────────────────────┘
              Yes │         No
                  ▼         │
          ┌───────────┐     │
          │ No change │     │
          └───────────┘     │
                            ▼
              ┌─────────────────────────┐
              │ What is desired ensure? │
              └─────────────┬───────────┘
                            │
            ┌───────────────┴───────────────┐
            │ absent                        │ present
            ▼                               ▼
      ┌───────────┐                   ┌───────────┐
      │ Noop?     │                   │ Noop?     │
      └─────┬─────┘                   └─────┬─────┘
        Yes │     No                    Yes │     No
            ▼     │                         ▼     │
    ┌────────────┐│                 ┌────────────┐│
    │ Set noop   ││                 │ Set noop   ││
    │ message    ││                 │ message    ││
    └────────────┘│                 └────────────┘│
                  ▼                               ▼
        ┌───────────────┐             ┌─────────────────────┐
        │ Remove all    │             │ Scaffold            │
        │ managed files │             │ (render templates)  │
        │ and empty dirs│             │                     │
        └───────────────┘             └─────────────────────┘

Idempotency

The scaffold resource determines idempotency by rendering templates in noop mode and comparing results against the target directory.

State Checks

  1. Ensure absent: Target must not exist, or no managed files remain on disk (Changed and Stable lists empty). Purged files (files not belonging to the scaffold) do not affect this check.
  2. Ensure present: The Changed list must be empty, and the Purged list must be empty when purge is enabled (all files are stable). When purge is disabled, purged files do not affect stability.

Decision Table

For ensure: absent, purged files never affect stability since they don’t belong to the scaffold. For ensure: present, purged files only affect stability when purge is enabled.

When ensure: absent, the Status method filters Changed and Stable lists to only include files that actually exist on disk, so the state reflects reality after removal rather than what the scaffold would create.

DesiredTarget ExistsChanged FilesPurged FilesPurge EnabledStable?
absentNoN/AN/AN/AYes
absentYesNoneAnyN/AYes (no managed files on disk)
absentYesSomeAnyN/ANo (managed files remain)
presentYesNoneNoneAnyYes
presentYesNoneSomeNoYes (purged files ignored)
presentYesNoneSomeYesNo (purge needed)
presentYesSomeAnyAnyNo (render needed)
presentNoN/AN/AAnyNo (target missing)

Source Resolution

The source property is resolved relative to the manager’s working directory when it is a relative path:

parsed, _ := url.Parse(properties.Source)
if parsed == nil || parsed.Scheme == "" {
    if !filepath.IsAbs(properties.Source) {
        t.prop.Source = filepath.Join(mgr.WorkingDirectory(), properties.Source)
    }
}

This allows manifests bundled with template directories to use relative paths. URL sources (with a scheme) are passed through unchanged.

Path Validation

Target paths (the resource name) must be:

  • Absolute (start with /)
  • Canonical (no . or .. components, filepath.Clean(path) == path)

Post-Processing

The post property defines commands to run on rendered files. Each entry is a map where the key is a glob pattern matched against the file’s basename and the value is a command to execute. Use {} as a placeholder for the file’s full path; if omitted, the path is appended as the last argument.

post:
  - "*.go": "go fmt {}"
  - "*.sh": "chmod +x {}"

Post-processing runs immediately after each file is rendered. Validation ensures neither keys nor values are empty.

Noop Mode

In noop mode, the scaffold type queries the current state via Status() and reports what would change without modifying the filesystem. Neither Scaffold() nor Remove() are called.

For ensure: present, the affected count is the number of changed files plus purged files (when purge is enabled). For ensure: absent, the affected count is the number of changed and stable files plus purged files (when purge is enabled).

DesiredAffected CountMessage
presentChanged + Purged (if purge enabled)Would have changed N scaffold files
absentChanged + Stable + Purged (if purge enabled)Would have removed N scaffold files

Changed is set to true only when the affected count is greater than zero. When the resource is already in the desired state, Changed is false and NoopMessage is empty.

Desired State Validation

After applying changes (in non-noop mode), the type verifies the scaffold reached the desired state by checking the changed and purged file lists. If validation fails, ErrDesiredStateFailed is returned.

Subsections of Scaffold Type

Choria Provider

This document describes the implementation details of the Choria scaffold provider for rendering template directories using the choria-io/scaffold library.

Provider Selection

The Choria provider is the default and only scaffold provider. It is always available and returns priority 1 for all scaffold resources.

Operations

Scaffold (Render Templates)

Process:

  1. Check if target directory exists
  2. Configure scaffold with source, target, engine, delimiters, post-processing, and skip_empty settings
  3. Create scaffold instance using the appropriate engine (scaffold.New() for Go, scaffold.NewJet() for Jet)
  4. Call Render() (real mode) or RenderNoop() (noop mode)
  5. Categorize results into changed, stable, and purged file lists

Scaffold Configuration:

Config FieldSource PropertyDescription
TargetDirectoryNameTarget directory for rendered files
SourceDirectorySourceSource template directory
MergeTargetDirectory(always true)Merge into existing target directory
PostPostPost-processing commands
SkipEmptySkipEmptySkip empty rendered files
CustomLeftDelimiterLeftDelimiterCustom template left delimiter
CustomRightDelimiterRightDelimiterCustom template right delimiter

Engine Selection:

EngineConstructorDefault Delimiters
goscaffold.New(){{ / }}
jetscaffold.NewJet()[[ / ]]

Result Categorization:

Scaffold ActionMetadata ListDescription
FileActionEqualStableFile content unchanged
FileActionAddChangedNew file created
FileActionUpdateChangedExisting file modified
FileActionRemovePurgedFile removed from target

File paths in the metadata lists are absolute paths, constructed by joining the target directory with the relative path from the scaffold result.

Purge Behavior:

When purge is enabled and a file has FileActionRemove, the provider deletes the file from disk during Scaffold(). In noop mode, the removal is logged but not performed. When purge is disabled, purged files are only tracked in metadata and not removed.

Status

Process:

  1. Perform a dry-run render (noop mode) to determine what the scaffold would do
  2. When ensure is absent, filter Changed and Stable lists to only include files that actually exist on disk

The noop render reports what would happen if the scaffold were applied. For ensure: present, this is the desired output — it shows what needs to change. For ensure: absent, the raw render output is misleading after removal (it would show files to be added), so the lists are filtered to reflect what managed files actually remain on disk.

State Detection:

Target DirectoryEnsure ValueMetadata
ExistspresentChanged, stable, and purged file lists from render
ExistsabsentChanged and stable filtered to files on disk, purged from render
Does not existAnyEmpty metadata, TargetExists: false

Remove

Process:

  1. Collect managed files from the state’s Changed and Stable lists (purged files are not removed as they don’t belong to the scaffold)
  2. Remove each file individually
  3. Track parent directories of removed files
  4. Iteratively remove empty directories deepest-first
  5. Stop when no more empty directories can be removed
  6. Best-effort removal of the target directory (only succeeds if empty)

File Removal Order:

Files are collected from two metadata lists:

  1. Changed - Files that were added or modified
  2. Stable - Files that were unchanged

Purged files are not removed because they are unrelated to the scaffold and may belong to other processes.

Directory Cleanup:

For each removed file:
    Track its parent directory

Repeat:
    For each tracked directory:
        Skip if it is the target directory itself
        Skip if not empty
        Remove the directory
        Track its parent directory
Until no more directories removed

Best-effort: remove the target directory (fails silently if not empty)

The target directory is removed if empty after all managed files and subdirectories are cleaned up. If unrelated files remain (purged files), the directory is preserved.

Error Handling:

ConditionBehavior
Non-absolute file pathReturn error immediately
File removal failsLog error, continue with remaining files
Directory removal failsLog error, continue with remaining directories
File does not existSilently skip (os.IsNotExist check)
Target directory removal failsLog at debug level, no error returned

Template Environment

Templates receive the full templates.Env environment, which provides access to:

  • facts - System facts for the managed node
  • data - Hiera-resolved configuration data
  • Template helper functions

This allows templates to generate host-specific configurations based on facts and hierarchical data.

Logging

The provider wraps the CCM logger in a scaffold-compatible interface:

type logger struct {
    log model.Logger
}

func (l *logger) Debugf(format string, v ...any)
func (l *logger) Infof(format string, v ...any)

This adapter translates the scaffold library’s Debugf/Infof calls to CCM’s structured logging.

Platform Support

The Choria provider is platform-independent. It uses the choria-io/scaffold library for template rendering, which operates on standard filesystem operations. No platform-specific system calls are used.

Service Type

This document describes the design of the service resource type for managing system services.

Overview

The service resource manages system services with two independent dimensions:

  • Running state: Whether the service is currently running or stopped
  • Enabled state: Whether the service starts automatically at boot

These are managed independently, allowing combinations like “running but disabled” or “stopped but enabled”.

Provider Interface

Service providers must implement the ServiceProvider interface:

type ServiceProvider interface {
    model.Provider

    Enable(ctx context.Context, service string) error
    Disable(ctx context.Context, service string) error
    Start(ctx context.Context, service string) error
    Stop(ctx context.Context, service string) error
    Restart(ctx context.Context, service string) error
    Status(ctx context.Context, service string) (*model.ServiceState, error)
}

Method Responsibilities

MethodResponsibility
StatusQuery current running and enabled state
StartStart the service if not running
StopStop the service if running
RestartStop and start the service (for refresh)
EnableConfigure service to start at boot
DisableConfigure service to not start at boot

Status Response

The Status method returns a ServiceState containing:

type ServiceState struct {
    CommonResourceState
    Metadata *ServiceMetadata
}

type ServiceMetadata struct {
    Name     string  // Service name
    Provider string  // Provider name (e.g., "systemd")
    Enabled  bool    // Whether service starts at boot
    Running  bool    // Whether service is currently running
}

The Ensure field in CommonResourceState is set to:

  • running if the service is active
  • stopped if the service is inactive

Available Providers

ProviderInit SystemDocumentation
systemdsystemdSystemd

Ensure States

ValueDescription
runningService must be running (default)
stoppedService must be stopped

If ensure is not specified, it defaults to running.

Enable Property

The enable property is a boolean pointer (*bool) with three possible states:

ValueBehavior
trueEnable service to start at boot
falseDisable service from starting at boot
nil (not set)Leave boot configuration unchanged

This allows managing running state without affecting boot configuration:

# Start service but don't change boot config
- service:
    - myapp:
        ensure: running

# Start and enable at boot
- service:
    - myapp:
        ensure: running
        enable: true

# Stop and disable at boot
- service:
    - myapp:
        ensure: stopped
        enable: false

Apply Logic

The service type applies changes in two phases:

Phase 1: Running State

┌─────────────────────────────────────────┐
│ Check for subscribe refresh             │
└─────────────────┬───────────────────────┘
                  │
    ┌─────────────┴─────────────┐
    │ Subscribed resource       │
    │ changed?                  │
    └─────────────┬─────────────┘
              Yes │         │ No
                  ▼         │
    ┌─────────────────────┐ │
    │ ensure=running?     │ │
    │ already running?    │ │
    └─────────────┬───────┘ │
          Yes+Yes │         │
                  ▼         │
          ┌───────────┐     │
          │ Restart   │     │
          └───────────┘     │
                            ▼
              ┌─────────────────────────┐
              │ Compare ensure vs state │
              └─────────────┬───────────┘
                            │
    ┌───────────────────────┼───────────────────────┐
    │ ensure=stopped        │ ensure=running        │
    │ state=running         │ state=stopped         │
    ▼                       ▼                       │
┌────────┐            ┌────────┐                    │
│ Stop   │            │ Start  │                    │
└────────┘            └────────┘                    │
                                                    ▼
                                          ┌───────────────┐
                                          │ No change     │
                                          └───────────────┘

Phase 2: Enabled State

After running state is handled, enabled state is processed:

┌─────────────────────────────────────────┐
│ enable property set?                    │
└─────────────────┬───────────────────────┘
                  │
          nil     │     true/false
          ▼       │
  ┌───────────┐   │
  │ No change │   │
  └───────────┘   │
                  ▼
    ┌─────────────────────────────┐
    │ Compare enable vs enabled   │
    └─────────────┬───────────────┘
                  │
    ┌─────────────┼─────────────┐
    │ enable=true │ enable=false│
    │ !enabled    │ enabled     │
    ▼             ▼             │
┌────────┐  ┌─────────┐         │
│ Enable │  │ Disable │         │
└────────┘  └─────────┘         │
                                ▼
                      ┌───────────────┐
                      │ No change     │
                      └───────────────┘

Subscribe Behavior

Services can subscribe to other resources and restart when they change:

- service:
    - httpd:
        ensure: running
        subscribe:
          - file#/etc/httpd/conf/httpd.conf
          - package#httpd

Special Cases:

ConditionBehavior
ensure: stoppedSubscribe ignored (no restart)
Service not running + ensure: runningStart (not restart)
Service running + ensure: runningRestart

This prevents restarting stopped services and ensures a clean start when the service should be running but isn’t.

Idempotency

The service resource is idempotent through state comparison:

DesiredCurrentAction
ensure: runningrunningNone
ensure: runningstoppedStart
ensure: stoppedstoppedNone
ensure: stoppedrunningStop
enable: trueenabledNone
enable: truedisabledEnable
enable: falseenabledDisable
enable: falsedisabledNone
enable: nilanyNone

Desired State Validation

After applying changes, the type verifies the service reached the desired state:

func (t *Type) isDesiredState(properties, state) bool {
    // Check running state
    if properties.Ensure != state.Ensure {
        return false
    }

    // Check enabled state (only if explicitly set)
    if properties.Enable != nil {
        if *properties.Enable != state.Metadata.Enabled {
            return false
        }
    }

    return true
}

If the desired state is not reached, an ErrDesiredStateFailed error is returned.

Service Name Validation

Service names are validated to prevent injection attacks:

Allowed Characters:

  • Alphanumeric (a-z, A-Z, 0-9)
  • Period (.), underscore (_), plus (+)
  • Colon (:), tilde (~), hyphen (-)

Rejected:

  • Shell metacharacters (;, |, &, etc.)
  • Whitespace
  • Path separators

Noop Mode

In noop mode, the service type:

  1. Queries current state normally
  2. Logs what actions would be taken
  3. Sets appropriate NoopMessage (e.g., “Would have started”, “Would have enabled”)
  4. Reports Changed: true if changes would occur
  5. Does not call provider Start/Stop/Restart/Enable/Disable methods

Subsections of Service Type

Systemd Provider

This document describes the implementation details of the Systemd service provider for managing system services via systemctl.

Provider Selection

The Systemd provider is selected when systemctl is found in the system PATH. The provider checks for the executable using util.ExecutableInPath("systemctl").

Availability Check:

  • Searches PATH for systemctl
  • Returns priority 1 if found
  • Returns unavailable if not found

Concurrency

A global service lock (model.ServiceGlobalLock) is held during all systemctl command executions to prevent concurrent systemd operations within the same process. This prevents race conditions when multiple service resources are managed simultaneously.

func (p *Provider) execute(ctx context.Context, cmd string, args ...string) (...) {
    model.ServiceGlobalLock.Lock()
    defer model.ServiceGlobalLock.Unlock()
    return p.runner.Execute(ctx, cmd, args...)
}

Daemon Reload

The provider performs a systemctl daemon-reload once per provider instance before any service operations. This ensures systemd picks up any unit file changes made by other resources (e.g., file resources managing unit files).

func (p *Provider) maybeReload(ctx context.Context) error {
    p.mu.Lock()
    defer p.mu.Unlock()

    if p.didReload {
        return nil
    }

    _, _, _, err := p.execute(ctx, "systemctl", "daemon-reload")
    return err
}

The reload is performed only once, tracked by the didReload flag.

Operations

Status

Commands:

systemctl is-active --system <service>
systemctl is-enabled --system <service>

Active State Detection:

is-active OutputInterpreted As
activeRunning
inactiveStopped
failedStopped
activatingStopped
OtherError

Enabled State Detection:

is-enabled OutputInterpreted As
enabledEnabled
enabled-runtimeEnabled
aliasEnabled
staticEnabled
indirectEnabled
generatedEnabled
transientEnabled
linkedDisabled
linked-runtimeDisabled
maskedDisabled
masked-runtimeDisabled
disabledDisabled
not-foundError: service not found

Returned State:

FieldValue
Ensurerunning or stopped based on is-active
Metadata.EnabledBoolean from is-enabled
Metadata.RunningBoolean from is-active
Metadata.Provider“systemd”

Start

Command:

systemctl start --system <service>

Called when ensure: running and service is currently stopped.

Stop

Command:

systemctl stop --system <service>

Called when ensure: stopped and service is currently running.

Restart

Command:

systemctl restart --system <service>

Called when a subscribed resource has changed and the service should be refreshed.

Enable

Command:

systemctl enable --system <service>

Called when enable: true and service is currently disabled.

Disable

Command:

systemctl disable --system <service>

Called when enable: false and service is currently enabled.

Command Flags

All commands use the --system flag to explicitly target system-level units (as opposed to user-level units managed with --user).

Decision Flow

Ensure State (Running/Stopped)

┌─────────────────────────────────────────┐
│ Subscribe triggered?                     │
│ (subscribed resource changed)            │
└─────────────┬───────────────┬───────────┘
              │ Yes           │ No
              │               │
              │               ▼
              │       ┌───────────────────┐
              │       │ Ensure = stopped? │
              │       └─────────┬─────────┘
              │           Yes   │   No
              │           ▼     ▼
              │   ┌─────────┐ ┌───────────────────┐
              │   │ Running?│ │ Ensure = running? │
              │   └────┬────┘ └─────────┬─────────┘
              │    Yes │ No         Yes │   No
              │        ▼               ▼
              │   ┌────────┐    ┌─────────┐
              │   │ Stop   │    │ Running?│
              │   └────────┘    └────┬────┘
              │                  No  │ Yes
              │                      ▼
              │                 ┌────────┐
              │                 │ Start  │
              │                 └────────┘
              │
              ▼
      ┌───────────────────────────────┐
      │ Ensure = running?             │
      │ (only restart running services)│
      └───────────────┬───────────────┘
                  Yes │   No
                      ▼
              ┌─────────────┐
              │ Restart     │
              └─────────────┘

Subscribe Behavior Notes:

  • Restart only occurs if ensure: running
  • If service is stopped and ensure: running, it starts instead of restarting
  • Subscribe is ignored if ensure: stopped

Enable State

Enable/disable is processed independently after ensure state:

┌─────────────────────────────────────────┐
│ Enable property set?                     │
└─────────────┬───────────────┬───────────┘
              │ Yes           │ No (nil)
              │               │
              │               ▼
              │       ┌───────────────┐
              │       │ No change     │
              │       └───────────────┘
              ▼
      ┌───────────────────────────────┐
      │ Enable = true?                 │
      └───────────────┬───────────────┘
                  Yes │   No
                      ▼
      ┌───────────────────────────────┐
      │ Currently enabled?             │
      └───────────────┬───────────────┘
                  No  │ Yes
                      ▼
              ┌─────────────┐
              │ Enable      │
              └─────────────┘

      (Similar flow for disable when enable=false)

Idempotency

The service resource checks current state before making changes:

Desired StateCurrent StateAction
runningrunningNone
runningstoppedStart
stoppedstoppedNone
stoppedrunningStop
enable: trueenabledNone
enable: truedisabledEnable
enable: falseenabledDisable
enable: falsedisabledNone
enable: nilanyNone

Service Name Validation

Service names are validated to prevent shell injection:

Dangerous Characters Check:

if dangerousCharsRegex.MatchString(p.Name) {
    return fmt.Errorf("service name contains dangerous characters: %q", p.Name)
}

Allowed Characters:

  • Alphanumeric (a-z, A-Z, 0-9)
  • Period (.)
  • Underscore (_)
  • Plus (+)
  • Colon (:)
  • Tilde (~)
  • Hyphen (-)

Examples:

NameValid
httpdYes
nginx.serviceYes
my-app_v2Yes
app@instanceNo (@ not allowed)
app; rm -rf /No (shell metacharacters)

Subscribe and Refresh

Services can subscribe to other resources and restart when they change:

- file:
    - /etc/myapp/config.yaml:
        ensure: present
        content: "..."
        owner: root
        group: root
        mode: "0644"

- service:
    - myapp:
        ensure: running
        enable: true
        subscribe:
          - file#/etc/myapp/config.yaml

Behavior:

  • When the file resource changes, the service is restarted
  • Restart only occurs if ensure: running
  • If service was stopped and should be running, it starts (not restarts)

Error Handling

ConditionBehavior
systemctl not in PATHProvider unavailable
Service not foundError from is-enabled: “service not found”
Unknown is-active outputError: “invalid systemctl is-active output”
Unknown is-enabled outputError: “invalid systemctl is-enabled output”
Command execution failureError propagated from runner

Platform Support

The Systemd provider requires:

  • Linux with systemd as init system
  • systemctl command available in PATH

It does not support:

  • Non-systemd init systems (SysVinit, Upstart, OpenRC)
  • User-level units (uses --system flag)
  • Windows, macOS, or BSD systems

Adding a Type

This guide documents the process for adding a new resource type to CCM. It uses the archive type as a reference implementation.

Overview

Adding a new resource type requires changes across several packages:

  1. Model definitions - Properties, state, and metadata structs
  2. Resource type implementation - Core logic and provider interface
  3. Provider implementation - Platform-specific operations
  4. Integration points - Factory functions and registry
  5. CLI commands - User-facing command line interface
  6. JSON schemas - Validation for manifests and API requests
  7. Documentation - User and design documentation
  8. CCM Studio - Web-based manifest designer

File Checklist

FileActionPurpose
model/resource_<type>.goCreateProperties, state, metadata structs
model/resource_<type>_test.goCreateProperty validation tests
model/resource.goModifyAdd case to factory function
resources/<type>/<type>.goCreateProvider interface definition
resources/<type>/type.goCreateResource type implementation
resources/<type>/type_test.goCreateResource type tests
resources/<type>/provider_mock_test.goGenerateMock provider for tests
resources/<type>/<provider>/factory.goCreateProvider factory
resources/<type>/<provider>/<provider>.goCreateProvider implementation
resources/<type>/<provider>/<provider>_test.goCreateProvider tests
resources/resources.goModifyAdd case to NewResourceFromProperties
cmd/ensure_<type>.goCreateCLI command handler
cmd/ensure.goModifyRegister CLI command
internal/fs/schemas/manifest.jsonModifyAdd resource schema definitions
internal/fs/schemas/resource_ensure_request.jsonModifyAdd API request schema
docs/content/resources/<type>.mdCreateUser documentation
docs/content/design/<type>/_index.mdCreateDesign documentation
docs/content/design/<type>/<provider>.mdCreateProvider documentation

Step 1: Model Definitions

Create model/resource_<type>.go with the following components.

Constants

const (
    // ResourceStatus<Type>Protocol is the protocol identifier for <type> resource state
    ResourceStatus<Type>Protocol = "io.choria.ccm.v1.resource.<type>.state"

    // <Type>TypeName is the type name for <type> resources
    <Type>TypeName = "<type>"
)

Properties Struct

The properties struct must satisfy model.ResourceProperties:

type ResourceProperties interface {
    CommonProperties() *CommonResourceProperties
    Validate() error
    ResolveTemplates(*templates.Env) error
    ToYamlManifest() (yaml.RawMessage, error)
}

Structure:

type <Type>ResourceProperties struct {
    CommonResourceProperties `yaml:",inline"`

    // Add type-specific fields here
    Url      string `json:"url" yaml:"url"`
    Checksum string `json:"checksum,omitempty" yaml:"checksum,omitempty"`
    // ...
}

Key points:

  • Embed CommonResourceProperties with yaml:",inline" tag
  • Use JSON and YAML struct tags for serialization
  • In Validate(), call p.CommonResourceProperties.Validate() first, then add type-specific validation
  • In ResolveTemplates(), call p.CommonResourceProperties.ResolveTemplates(env) first, then resolve type-specific fields

State Struct

The state struct must satisfy model.ResourceState:

type ResourceState interface {
    CommonState() *CommonResourceState
}

Structure:

type <Type>Metadata struct {
    Name     string `json:"name" yaml:"name"`
    Provider string `json:"provider,omitempty" yaml:"provider,omitempty"`
    // Add fields describing current system state
}

type <Type>State struct {
    CommonResourceState
    Metadata *<Type>Metadata `json:"metadata,omitempty"`
}

Factory Function

Provide a factory function for YAML parsing:

func New<Type>ResourcePropertiesFromYaml(raw yaml.RawMessage) ([]ResourceProperties, error) {
    return parseProperties(raw, <Type>TypeName, func() ResourceProperties {
        return &<Type>ResourceProperties{}
    })
}

Step 2: Resource Type Implementation

Provider Interface (resources/<type>/<type>.go)

Define a type-specific provider interface that embeds model.Provider and adds type-specific methods:

package <type>resource

import (
    "context"

    "github.com/choria-io/ccm/model"
    "github.com/choria-io/ccm/resources/<type>/<provider>"
)

type <Type>Factory interface {
    model.ProviderFactory
}

func init() {
    <provider>.Register()
}

type <Type>Provider interface {
    model.Provider

    Status(ctx context.Context, properties *model.<Type>ResourceProperties) (*model.<Type>State, error)
    // Add provider-specific methods (e.g., Download, Extract for archive)
}

Type Implementation (resources/<type>/type.go)

The Type struct must satisfy both model.Resource and base.EmbeddedResource:

// model.Resource interface
type Resource interface {
    Type() string
    Name() string
    Provider() string
    Properties() ResourceProperties
    Apply(context.Context) (*TransactionEvent, error)
    Info(context.Context) (any, error)
    Healthcheck(ctx context.Context) (*TransactionEvent, error)
}

// base.EmbeddedResource interface
type EmbeddedResource interface {
    NewTransactionEvent() *model.TransactionEvent
    ApplyResource(ctx context.Context) (model.ResourceState, error)
    SelectProvider() (string, error)
    Type() string
}

Embedding *base.Base provides implementations for Apply(), Healthcheck(), Type(), Name(), Properties(), and NewTransactionEvent(). The type must implement:

  • ApplyResource() - core resource application logic
  • SelectProvider() - provider selection
  • Provider() - return current provider name
  • Info() - return resource information

Structure:

type Type struct {
    *base.Base

    prop     *model.<Type>ResourceProperties
    mgr      model.Manager
    log      model.Logger
    provider model.Provider
    facts    map[string]any
    data     map[string]any

    mu sync.Mutex
}

var _ model.Resource = (*Type)(nil)

See resources/archive/type.go for a complete constructor example.

ApplyResource Method

The ApplyResource method (part of base.EmbeddedResource) contains the core logic. It should follow this pattern:

  1. Get initial state via provider.Status()
  2. Check if already in desired state (implement isDesiredState() helper)
  3. If stable, call t.FinalizeState() and return early
  4. Apply changes, respecting t.mgr.NoopMode()
  5. Get final state and verify desired state was achieved
  6. Call t.FinalizeState() with appropriate flags

See resources/archive/type.go:ApplyResource() for a complete example.

Provider Selection Methods

The SelectProvider() method should use registry.FindSuitableProvider() to select an appropriate provider. See resources/archive/type.go for the standard implementation pattern.

Step 3: Provider Implementation

Factory (resources/<type>/<provider>/factory.go)

The factory must satisfy model.ProviderFactory:

type ProviderFactory interface {
    TypeName() string
    Name() string
    New(log Logger, runner CommandRunner) (Provider, error)
    IsManageable(facts map[string]any, properties ResourceProperties) (bool, int, error)
}

The IsManageable method returns:

  • bool - whether this provider can manage the resource
  • int - priority (higher wins when multiple providers match)
  • error - any error encountered

Structure:

package <provider>

import (
    "github.com/choria-io/ccm/internal/registry"
    "github.com/choria-io/ccm/model"
)

const ProviderName = "<provider>"

func Register() {
    registry.MustRegister(&factory{})
}

type factory struct{}

func (p *factory) TypeName() string { return model.<Type>TypeName }
func (p *factory) Name() string     { return ProviderName }
func (p *factory) New(log model.Logger, runner model.CommandRunner) (model.Provider, error) {
    return New<Provider>Provider(log, runner)
}
func (p *factory) IsManageable(facts map[string]any, prop model.ResourceProperties) (bool, int, error) {
    // Type assert and check if this provider can handle the resource
    return true, 1, nil
}

See resources/archive/http/factory.go for a complete example.

Provider Implementation (resources/<type>/<provider>/<provider>.go)

The provider must satisfy the type-specific provider interface defined in Step 2 (which embeds model.Provider):

type Provider interface {
    Name() string
}

Structure:

package <provider>

import (
    "context"

    "github.com/choria-io/ccm/model"
)

type <Provider>Provider struct {
    log    model.Logger
    runner model.CommandRunner
}

func New<Provider>Provider(log model.Logger, runner model.CommandRunner) (*<Provider>Provider, error) {
    return &<Provider>Provider{log: log, runner: runner}, nil
}

func (p *<Provider>Provider) Name() string {
    return ProviderName
}

func (p *<Provider>Provider) Status(ctx context.Context, properties *model.<Type>ResourceProperties) (*model.<Type>State, error) {
    state := &model.<Type>State{
        CommonResourceState: model.NewCommonResourceState(
            model.ResourceStatus<Type>Protocol,
            model.<Type>TypeName,
            properties.Name,
            model.EnsureAbsent,
        ),
        Metadata: &model.<Type>Metadata{
            Name:     properties.Name,
            Provider: ProviderName,
        },
    }

    // Query system state and populate metadata

    return state, nil
}

// Implement other type-specific provider methods...

See resources/archive/http/http.go for a complete example.

Step 4: Integration Points

Update resources/resources.go

Add the import and case statement:

import (
    // ...
    <type>resource "github.com/choria-io/ccm/resources/<type>"
)

func NewResourceFromProperties(ctx context.Context, mgr model.Manager, props model.ResourceProperties) (model.Resource, error) {
    switch rprop := props.(type) {
    // ... existing cases ...
    case *model.<Type>ResourceProperties:
        return <type>resource.New(ctx, mgr, *rprop)
    default:
        return nil, fmt.Errorf("unsupported resource property type %T", rprop)
    }
}

Update model/resource.go

Add the case to NewResourcePropertiesFromYaml:

func NewResourcePropertiesFromYaml(typeName string, rawProperties yaml.RawMessage, env *templates.Env) ([]ResourceProperties, error) {
    switch typeName {
    // ... existing cases ...
    case <Type>TypeName:
        props, err = New<Type>ResourcePropertiesFromYaml(rawProperties)
    default:
        return nil, fmt.Errorf("%w: %s %s", ErrResourceInvalid, ErrUnknownType, typeName)
    }
    // ...
}

Step 5: CLI Command

Create cmd/ensure_<type>.go:

package main

import (
    "github.com/choria-io/ccm/model"
    "github.com/choria-io/fisk"
)

type ensure<Type>Command struct {
    name string
    // Add command-specific fields for flags

    parent *ensureCommand
}

func registerEnsure<Type>Command(ccm *fisk.CmdClause, parent *ensureCommand) {
    cmd := &ensure<Type>Command{parent: parent}

    <type> := ccm.Command("<type>", "<Type> management").Action(cmd.<type>Action)
    <type>.Arg("name", "Resource name").Required().StringVar(&cmd.name)
    // Add type-specific flags

    parent.addCommonFlags(<type>)
}

func (c *ensure<Type>Command) <type>Action(_ *fisk.ParseContext) error {
    properties := model.<Type>ResourceProperties{
        CommonResourceProperties: model.CommonResourceProperties{
            Name:     c.name,
            Ensure:   model.EnsurePresent,
            Provider: c.parent.provider,
        },
        // Set type-specific properties from flags
    }

    return c.parent.commonEnsureResource(&properties)
}

Update cmd/ensure.go:

func registerEnsureCommand(ccm *fisk.Application) {
    // ... existing code ...
    registerEnsure<Type>Command(ens, cmd)
}

Step 6: JSON Schemas

Update internal/fs/schemas/manifest.json

Add to the $defs/resource properties:

"<type>": {
  "oneOf": [
    { "$ref": "#/$defs/<type>ResourceList" },
    { "$ref": "#/$defs/<type>ResourcePropertiesWithName" }
  ]
}

Add resource list definition:

"<type>ResourceList": {
  "type": "array",
  "description": "List of <type> resources to manage (named format)",
  "items": {
    "type": "object",
    "additionalProperties": {
      "$ref": "#/$defs/<type>ResourceProperties"
    },
    "minProperties": 1,
    "maxProperties": 1
  }
}

Add properties definitions:

"<type>ResourcePropertiesWithName": {
  "allOf": [
    { "$ref": "#/$defs/<type>ResourceProperties" },
    {
      "type": "object",
      "properties": {
        "name": {
          "type": "string",
          "description": "Resource name"
        }
      },
      "required": ["name"]
    }
  ]
},
"<type>ResourceProperties": {
  "type": "object",
  "properties": {
    "ensure": {
      "type": "string",
      "enum": ["present", "absent"]
    }
    // Add type-specific properties
  }
}

Update internal/fs/schemas/resource_ensure_request.json

Add to the type enum:

"enum": ["package", "service", "file", "exec", "archive", "<type>"]

Add to properties.oneOf:

{ "$ref": "#/$defs/<type>Properties" }

Add properties definition under $defs:

"<type>Properties": {
  "allOf": [
    { "$ref": "#/$defs/commonProperties" },
    {
      "type": "object",
      "properties": {
        "name": {
          "type": "string",
          "description": "Resource name"
        }
        // Add type-specific properties
      },
      "required": ["name"]
    }
  ]
}

Copy Schemas to Documentation Site

After updating the schema files in internal/fs/schemas/, copy them to docs/static/schemas/v1/ so they are available on the documentation website:

cp internal/fs/schemas/manifest.json docs/static/schemas/v1/manifest.json
cp internal/fs/schemas/resource_ensure_request.json docs/static/schemas/v1/resource_ensure_request.json

Step 7: Generate Mocks

Generate the provider mock for tests:

mockgen -write_generate_directive \
  -source resources/<type>/<type>.go \
  -destination resources/<type>/provider_mock_test.go \
  -package <type>resource

Or use the project command:

abt gen mocks

Step 8: Testing

Model Tests (model/resource_<type>_test.go)

Test property validation:

var _ = Describe("<Type>ResourceProperties", func() {
    Describe("Validate", func() {
        It("should require name", func() {
            p := &model.<Type>ResourceProperties{}
            p.Ensure = model.EnsurePresent
            Expect(p.Validate()).To(MatchError(model.ErrResourceNameRequired))
        })

        It("should validate ensure values", func() {
            p := &model.<Type>ResourceProperties{}
            p.Name = "test"
            p.Ensure = "invalid"
            Expect(p.Validate()).To(HaveOccurred())
        })
    })
})

Type Tests (resources/<type>/type_test.go)

Use the mock manager helper:

var _ = Describe("<Type> Type", func() {
    var mockctl *gomock.Controller

    BeforeEach(func() {
        mockctl = gomock.NewController(GinkgoT())
        registry.Clear()
        // Register mock factory
    })

    AfterEach(func() {
        mockctl.Finish()
    })

    Describe("Apply", func() {
        It("should handle present ensure state", func() {
            mgr, _ := modelmocks.NewManager(facts, data, false, mockctl)
            // Test implementation
        })
    })
})

Key Patterns

State Checking

Always check current state before making changes:

initialStatus, err := p.Status(ctx, t.prop)
if err != nil {
    return nil, err
}

isStable := t.isDesiredState(properties, initialStatus)
if isStable {
    // No changes needed
    t.FinalizeState(initialStatus, noop, "", false, true, false)
    return initialStatus, nil
}

Noop Mode

All resources must respect noop mode:

if !noop {
    // Make actual changes
    t.log.Info("Applying changes")
    err = p.SomeAction(ctx, properties)
} else {
    t.log.Info("Skipping changes as noop")
    noopMessage = "Would have applied changes"
}

Error Handling

Use sentinel errors from model/errors.go:

var (
    ErrResourceInvalid    = errors.New("resource invalid")
    ErrProviderNotFound   = errors.New("provider not found")
    ErrNoSuitableProvider = errors.New("no suitable provider")
    ErrDesiredStateFailed = errors.New("desired state not achieved")
)

Wrap errors with context:

err := os.Remove(path)
if err != nil {
    return fmt.Errorf("could not remove file: %w", err)
}

Template Resolution

The ResolveTemplates method (part of model.ResourceProperties) should resolve all user-facing string fields using templates.ResolveTemplateString(). Always call the embedded CommonResourceProperties.ResolveTemplates(env) first.

Provider Selection

Providers declare manageability via IsManageable on the factory (see model.ProviderFactory in Step 3). Multiple providers can match; the one with highest priority is selected.

Documentation

Create user documentation in docs/content/resources/<type>.md covering:

  • Overview and use cases
  • Ensure states table
  • Properties table with descriptions
  • Usage examples (manifest, CLI, API)

Create design documentation in docs/content/design/<type>/_index.md covering:

  • Provider interface specification
  • State checking logic
  • Apply logic flowchart

Create provider documentation in docs/content/design/<type>/<provider>.md covering:

  • Provider selection criteria
  • Platform requirements
  • Implementation details

CCM Studio

CCM Studio is a web-based manifest designer. After adding a new resource type, update CCM Studio to support it:

Note

CCM Studio is a closed-source project. The maintainers will complete this step.

  • Add the new resource type to the resource palette
  • Create property editors for type-specific fields
  • Add validation matching the JSON schema definitions
  • Update any resource type documentation or help text