Operating Automation – Ansible Playbooks & Roles

Automation for system operations: OS updates/upgrades, Docker cleanup, Mailcow maintenance, Checkmk onboarding, time services, hardening, and more.

Last Update: 2025-11-19

Prerequisites

• Ansible (>= 2.14 recommended)
• Python on target systems, SSH access (key-based authentication preferred)
• Collections (install once):

ansible-galaxy collection install \
	community.docker:3.11.0 \
	community.proxmox:1.4.0 \
	checkmk.general

Notes:

• ansible.cfg sets roles_path = ./roles:/etc/ansible/roles and disables host key checking.
• Sensitive variables are stored in vault.yml (protect with Ansible Vault).

Inventories & Variables

• Examples: inventories/icp-fra-pve1.yml, inventories/icp-frav-packer01.yml
• Group variables: inventories/group_vars/all.yml
• Important OS update variables (defaults in roles/os-updates/defaults/main.yml):
  • os_also_update_mirror (bool, default: true)
  • os_update_mirrors (list of mirror entries)
  • os_update_major_version (bool)
  • os_update_version_codename (e.g., bookworm, trixie)
• Checkmk variables: checkmk_server_url, checkmk_monitoring_site, checkmk_automation_user, checkmk_automation_pass, checkmk_agent_bakery_passphrase, and others.

Vault example (excerpt; store in vault.yml and encrypt with Vault):

checkmk_automation_user: "automation"
checkmk_automation_pass: "<secret>"
checkmk_agent_bakery_passphrase: "<secret>"
# Proxmox API (for major upgrade snapshots)
proxmox_api_host: "<pve.example>"
proxmox_api_user: "<user@pve>"
proxmox_api_token_id: "<token-id>"
proxmox_api_token_secret: "<token>"

Quick Start

1. Install collections (see above)

2. Run playbooks (examples):

# OS update (minor) for all inventory hosts
ansible-playbook -i inventories/icp-fra-pve1.yml playbooks/os-update.yml -K

# OS major upgrade to Debian "trixie" (with Proxmox snapshot and reboot)
ansible-playbook -i inventories/icp-fra-pve1.yml playbooks/os-major-upgrade.yml \
	-e os_update_version_codename=trixie -K

# Change mirrors
ansible-playbook -i inventories/icp-fra-pve1.yml playbooks/os-change-mirror.yml -K

# Configure time service via chronyd
ansible-playbook -i inventories/icp-frav-packer01.yml playbooks/setup-chronyd.yml -K

# Checkmk monitoring (create host, sign/bake agent, register)
ansible-playbook -i inventories/icp-frav-packer01.yml playbooks/setup-checkmk-monitoring.yml --ask-vault-pass

# Deploy ClamAV server (group "clamav-servers")
ansible-playbook -i inventories/icp-fra-pve1.yml playbooks/deploy-clamav-server.yml -K

# Docker: cleanup images only
ansible-playbook -i inventories/icp-frav-packer01.yml playbooks/docker/cleanup-images.yml -K

# Docker: full cleanup (containers/networks/volumes/cache), determine Mailcow first
ansible-playbook -i inventories/icp-frav-packer01.yml playbooks/docker/cleanup-all.yml -K

# Mailcow: update/restart/cleanup in sequence
ansible-playbook -i inventories/icp-frav-packer01.yml playbooks/managed-mailcow/update-mailcow.yaml -K

Playbook Reference

OS & System

• playbooks/os-update.yml

  • Purpose: Standard OS update on Debian. Optionally updates mirrors (os_also_update_mirror).
  • Variables: os_update_major_version (bool), os_update_version_codename (relevant for templates only)
  • Role: os-updates (executes update_mirrors.yaml and upgrade_packages.yaml; reboots on kernel change via handler)

• playbooks/os-major-upgrade.yml

  • Purpose: Debian major upgrade to target codename (e.g., trixie) including Proxmox snapshot before and reboot after.
  • Loads vault.yml (Proxmox API & Checkmk secrets, etc.).
  • Roles/tasks: proxmox-automation:get-vmid, proxmox-automation:create-snapshots, os-updates:update_major_version.
  • Requirement: Collection community.proxmox and valid API tokens.

• playbooks/os-change-mirror.yml

  • Purpose: Change Debian APT mirrors according to os_update_mirrors.
  • Role: os-updates:update_mirrors.

• playbooks/setup-chronyd.yml

  • Purpose: Configure time service with Chrony (systemd-timesyncd is removed).
  • Role: system:setup-timeserver (handler: restart chronyd).

Checkmk Onboarding

• playbooks/setup-checkmk-monitoring.yml
  • Purpose: Create host in Checkmk, sign/bake pending agent jobs, register agent, run discovery.
  • Loads vault.yml (automation user/pass, etc.).
  • Roles/tasks: checkmk-monitoring:create-host, checkmk-monitoring:sign-bake-agents, checkmk.general.agent (TLS/update/registration), checkmk-monitoring:discover-host.
  • Tags: checkmk-deploy (for registration & wait time).
  • Requirement: Collection checkmk.general.

Docker & Mailcow

• playbooks/docker/cleanup-images.yml

  • Purpose: Prune Docker images only; optionally capture Compose stack status (docker_compose_path).
  • Role: docker:cleanup-images.yml (collection community.docker).

• playbooks/docker/cleanup-all.yml

  • Purpose: Full Docker cleanup (containers/images/networks/volumes/builder cache) with running Mailcow stack.
  • Roles/tasks: managed-mailcow:find-mailcow-composedir, docker:get-containerstatus, docker:cleanup-all (only if containers not "false").

• playbooks/managed-mailcow/update-mailcow.yaml

  • Purpose: Update Mailcow via update.sh; optionally restart Docker daemon and cleanup.
  • Variables: github_mailcow_ver (target tag), disk_space_percent_max (threshold), debug.
  • Roles/tasks: roles/managed-mailcow:*, roles/docker:restart-daemon, roles/docker:cleanup-all.

• playbooks/managed-mailcow/start-stop-mailcow.yaml

  • Purpose: Stop and restart Mailcow stack (Compose v2).
  • Roles/tasks: managed-mailcow:find-mailcow-composedir, managed-mailcow:stop-mailcow, managed-mailcow:start-mailcow.

• playbooks/managed-mailcow/check-mailcow-health.yml

  • Purpose: Check HTTP accessibility and ports (25/587/143/993); tolerates errors (ignore_errors).

• playbooks/managed-mailcow/enable-sni-globally.yml

  • Purpose: Set ENABLE_SSL_SNI=y in mailcow.conf; restart stack if changed.

• playbooks/managed-mailcow/change-garbagecleaner.yaml

  • Purpose: Set MAILDIR_GC_TIME to 7 days (10080 minutes) and restart stack if changed.

• playbooks/managed-mailcow/migrate-clamd.yaml

  • Purpose: Switch Rspamd to external/shared ClamAV, disable local ClamAV, restart Rspamd.

• playbooks/managed-mailcow/use-docker-image-proxy.yaml

  • Purpose: Configure Docker daemon proxy & CA, set systemd drop-in, restart Docker.

• playbooks/managed-mailcow/use-syslog-server.yaml

  • Purpose: Switch Docker logging to syslog and restart Mailcow if needed.

• playbooks/managed-mailcow/remove-watchdog-mail.yaml

  • Purpose: Remove WATCHDOG_NOTIFY_EMAIL from mailcow.conf and restart stack.

• playbooks/managed-mailcow/find-roundcube-versions.yaml

  • Purpose: Extract Roundcube version from CHANGELOG.md (under data/web/rc|roundcube|roundcubemail).

• playbooks/managed-mailcow/add-haveged.yaml

  • Purpose: Install haveged package.

Hardening

• playbooks/hardening/manage-ssh-keys.yaml
  • Purpose: Add good keys, remove bad keys; write comment with timestamp.
  • Role: manage-ssh-keys
  • Variables (see roles/manage-ssh-keys/defaults/main.yml):
    • ssh_user (default: root)
    • good_keys (list of allowed keys)
    • bad_keys (list of keys to remove)

ClamAV‑Server

• playbooks/deploy-clamav-server.yml
  • Hosts: clamav-servers
  • Role: deploy-clamd (compiles ClamAV, creates user/group, configures systemd services clamd/freshclam).
  • Variable: clamd_version (default: 1.4.2). IPv6 binding according to template (TCPAddr {{ ansible_default_ipv6.address }}).

Roles & Collections (Overview)

• roles/os-updates – Mirror update, package upgrade, major upgrade including Exim blocking, reboot/apt cleanup handlers.
• roles/docker – Compose v2 status, prune (images/all), Docker daemon restart. Collection: community.docker.
• roles/managed-mailcow – Find Mailcow path, start/stop, update process, helper tasks.
• roles/system – Chrony setup, Docker/MOTD/SSH hardening, disk utility check, service handlers.
• roles/checkmk-monitoring – Create host, discovery, agent bakery/activation. Collection: checkmk.general.
• roles/deploy-clamd – ClamAV build/configuration/templates (systemd units, freshclam/clamd.conf).
• roles/proxmox-automation – Snapshots/VM info (collection: community.proxmox).

Common Commands

# Create/edit vault file
ansible-vault create vault.yml
ansible-vault edit vault.yml

# Syntax check
ansible-playbook -i inventories/icp-fra-pve1.yml playbooks/os-update.yml --syntax-check

# Target only one host group
ansible-playbook -i inventories/icp-fra-pve1.yml playbooks/os-update.yml -l icp-fra-pve1

# Dry run
ansible-playbook -i inventories/icp-fra-pve1.yml playbooks/os-update.yml --check

Notes & Best Practices

• Never commit secrets in plaintext – only provide via vault.yml.
• Always create snapshots/backups before major upgrades (playbook handles Proxmox snapshots automatically if configured).
• community.docker requires a working Docker engine and Compose v2 on the target system.
• Maintain inventory/hosts with IPv6 where possible (repo is prepared for this).

---

Questions or feature requests? Please mention the playbook/use case – we're happy to extend documentation and examples.