Complete firstRunbook
At least three automation-ready runbooks exist so ChatOps commands can trigger real standard procedures.
Without: Without runbooks, ChatOps commands become ad-hoc scripts with no documented effect.
RunsheetDevOps
ChatOps
ComplexityMedium
TimeOngoing
Participants2-20
FormatWorkshop + async
MaturityEstablished
01
Prerequisite
What needs to be finished first
Complete firstIncident Command
Incident roles and escalation paths are defined so it is clear in chat who decides and who executes.
Without: Without clear roles, the incident channel becomes a discussion round with no control.
02
Preparation
What needs to be ready before start
Materials
Team chat (Slack, MS Teams, Mattermost) with bot integration; repository for ChatOps scripts (Hubot, Lita, custom bots); CI/CD hooks; audit log; permissions matrix.
People / roles
A ChatOps maintainer as bot owner; SRE/DevOps with script responsibility; a security reviewer for privileged commands; all team members as users.
Pre-read
List of automation candidates (deploy, restart, status, rollback); tool endpoints and credential strategy; audit requirements; known compliance constraints.
Time needed
Initial setup 1-2 sprints, then continuous expansion
Setup
Connect the bot to chat. Define channel strategy (for example #ops, #incidents, #deploys). Set a permissions matrix per command. Send audit logs to a central logging pipeline. Put confirmation behind all destructive actions.
03
Core question
The one question this method answers
Which recurring operational actions can be triggered safely, traceably, and across the team in chat instead of in scattered tools?
04
Flow
Marker: Phase
| Step | Duration | Action | Hint |
|---|---|---|---|
1Phase 1: Select use cases | 60 min | Collect the top 10 recurring operations (deploy, restart, DB backup, log retrieval, status check). Evaluate frequency, current tooling, and risk for each use case. Select the first 3-5 commands. | High-risk operations (DB drop, prod reset) do not go into the first wave. ChatOps starts with read-only and low-risk actions. |
2Phase 2: Bot setup and permissions | 2-3 days | Choose the bot framework and install it in chat. Define a permission scheme (channel-based or role-based). Store credentials securely (Vault, secret manager). Configure audit logging. | If the bot runs with admin tokens and anyone can address it, that is a backdoor. Least privilege also applies to bots. |
3Phase 3: Implement commands | 1-2 weeks | Write a script per command, check idempotence, add a confirmation prompt for destructive actions. Add a help command with self-documentation. Include a dry-run option where useful. | Without a help command, the bot becomes tribal knowledge. Help text is the first documentation. Idempotence prevents damage from double triggering. |
4Phase 4: Rollout and training | 1 week | Run a dry-run with the team. Channel convention: commands only in the designated channels. Provide a short onboarding guide. Create an audit channel with all executed commands. | If commands run in any channel, audit context gets lost. Strict channel binding makes later forensics easier. |
5Phase 5: Monitoring and expansion | ongoing | Usage metrics: which commands, how often, error rate. Quarterly review for new use cases. Secure high-risk commands with additional approval flows. | A bot without telemetry is a black box. Without usage data, nobody knows whether commands are used or only decorative. |
05
Artifact
What comes out at the end
Form
ChatOps documentation with command list, permissions, audit strategy, onboarding guide, and expansion roadmap. Also includes the bot repository, help text, and audit log pipeline.
Versioning / ownership
Keep bot code in Git with pull request reviews. Command changes require code review and an audit note. Document permission changes with date and rationale.
Tool alternatives
- Hubot (classic), Lita, or Errbot
- Slack-native Workflow Builder with a serverless function
- Microsoft Teams Bot Framework
- Mattermost plugins or Rocket.Chat hooks
- PagerDuty Rundeck or Opsgenie Actions as the interface
chatops-working-template.md
Compact working template for ChatOps with context, input, output artifacts, and next step.
ChatOps Working Template
Goal
Uses chat as a shared interface for operational collaboration.
Context
When and for what do we use this method?
Input
Which data, observations, decisions, or materials are available?
Execution
Short notes along the runsheet.
Output artifacts
- Chat Transcript:
- Action Log:
- Automations:
Assumptions and open questions
- ...
Decision / Next step
Owner, date, and success signal.
06
Example output
Concrete filled scenario, fictional example
chatops-beispiel.md
Concrete filled scenario, fictional example
ChatOps Setup - Platform team workspot, May 2026
Bot: Hubot, deployed in Slack workspace, repo ops/chatbot.
Channels
- #ops-prod: production commands, all engineers + SRE.
- #ops-staging: all engineers, self-service.
- #incidents: only during an active incident.
Commands (wave 1)
| Command | Description | Permission |
|---|---|---|
hubot status <service> | Check a service health state | all |
hubot logs <service> [--lines N] | Retrieve the latest logs | all |
hubot deploy <service> <version> | Start a production deploy | SRE + eng lead |
hubot rollback <service> | Roll back to the previous version | SRE + eng lead |
hubot scale <service> <replicas> | Adjust replica count | SRE |
Audit: every execution is stored in DataDog logs with user, command, arguments, timestamp, and exit code. Retention 1 year.
Wave 2 (planned Q3): database snapshots, feature flag toggles, incident bridge bot.
First metrics (weeks 18-20): 1,842 commands, of which 67% status/logs, 22% deploy, 8% scale, 3% rollback. Error rate 1.4%.
07
Pitfalls
Recognize symptoms and steer against them
Trap
Bot with admin rights and no limits
Symptom
Anyone in the workspace can trigger production actions because the bot can do everything.
What to do
Permissions matrix per command, least privilege. Channel restrictions plus role checks. Destructive commands only with two-person approval.
Trap
No audit log
Symptom
After an incident, nobody can say who triggered which command and when.
What to do
Set up audit from the start. Every command logs user, arguments, and result. Send logs to a central SIEM. Without audit, ChatOps becomes a compliance risk.
Trap
Commands are not idempotent
Symptom
Running a command twice produces a double effect (two deploys, double restart).
What to do
Design for idempotence. On double trigger: lock or throttle. Add confirmation prompts for destructive commands.
Trap
Help texts go stale
Symptom
Help shows commands that no longer exist or arguments that are wrong.
What to do
Generate help text from code annotations instead of maintaining it separately. CI checks that every command has help text.
Trap
Commands in any channel
Symptom
Production deploys run in #random or direct messages, and the audit trail is scattered.
What to do
The bot ignores commands outside the allowed channels with a clear error message. Document allowed channels and explain them during onboarding.
Trap
Growth without review
Symptom
After 6 months there are 80 commands, nobody knows what they do, and security risks pile up.
What to do
Quarterly review of all commands. Remove unused or unclear commands. Re-check permissions. ChatOps needs maintenance like code.
08
Stop criteria
Done signals checkable in under a minute
No audit integration is available, so compliance requirements would be violated.
Permission model cannot be defined (for example the bot cannot query user roles).
Bot owner is not named, so maintenance diffuses.
Use cases are only high-criticality actions (prod drop), so no safe entry point exists.
Chat platform will be replaced within 6 months, so the investment would be wasted.
The team refuses to follow channels and conventions, so audit is impossible.
Finished the runsheet?
Go to the profile for purpose, similar methods, and sources or continue to the next method in the catalog.