Automation
Maintaining a sound cloud infrastructure includes tasks that need to be performed regularly. Resoto can help you automate many of these tasks by defining jobs.
Examples of these tasks include:
- enforcing company policies (e.g., tagging),
- sending notifications about critical infrastructure changes (e.g., to Slack, Discord, or PagerDuty), and
- cleaning up unused resources in your development accounts.
Jobs​
A job is a sequence of commands executed by Resoto at a specific time interval or when a specific event is emitted.
Jobs in Resoto are similar to Unix cron jobs. However, while cron jobs run only on a specific schedule, Resoto jobs can also be triggered based by events.
The typical structure of a job has three parts: a trigger, an optional filter, and an action.
Job Trigger​
The job trigger defines when a job is executed.
There are three types of job triggers: schedule triggers, event triggers, and combined triggers.
Schedule Trigger​
A schedule trigger executes a job at a specific time interval described by a cron expression.
Examples
- Every 5 minutes
--schedule "*/5 * * * *"
- Every day at 3:00am
--schedule "0 3 * * *"
- Every Monday at 4:00am
--schedule "0 4 * * MON"
- Every New Year's Eve at 11:59pm
--schedule "59 23 31 12 *"
See How to Create a Scheduled Job for step-by-step instructions describing how to create a job with a schedule trigger.
Event Trigger​
An event trigger executes a job when a specific event is emitted by Resoto.
Resoto updates the state of resources in the four steps of the collect_and_cleanup
workflow: collect
, cleanup_plan
, cleanup
, and generate_metrics
.
Each of these steps emits events that can be used to trigger jobs.
Examples
- When resource collection is complete and the database reflects the latest state of your resources
--wait-for-event collect_done
- When the cleanup planning process has started to mark resources for cleanup
--wait-for-event cleanup_plan
See Events for a list of events emitted by Resoto.
See How to Create an Event-Based Job for step-by-step instructions describing how to create a job with an event trigger.
Combined Trigger​
A combined trigger combines a schedule trigger with an event trigger, and executes a job when a specific event is emitted after a schedule trigger is fired.
Combined triggers are useful if you want to perform an action on a specific schedule, but only after a specific event is fired.
Example
Let's say you want to clean up development accounts at the end of each week.
To do so, you could define a schedule trigger in addition to specifying the cleanup_plan
event trigger:
--schedule "0 22 * * FRI" --wait-for-event cleanup_plan
The above combined trigger executes the job based on the latest state of resources after 10pm every Friday evening.
See How to Create a Scheduled Event-Based Job for step-by-step instructions describing how to create a job with a combined trigger.
Job Filters & Actions​
The scope and functionality of a job is defined by filters and actions, respectively.
To perform more complex automations, it is possible to define multiple sets of filters and actions delimited by a semicolon. Each set of filters and actions is executed sequentially when the job is triggered.
Job Filters​
A job filter is an optional search that returns the resources of interest. Only resources matching the filter will be processed by the job.
Example
- Instances without an owner tag in the dev or playground accounts
search is(instance) and tag.owner==null and /ancestors.account.reported.name in ["dev", "playground"]
The result of a job filter is a list of resources that is piped as input to job actions.
Job Actions​
A job action is an action command performed on resources matching the optional filter.
Examples
Update resource tags using the
tag update
commandsearch ... | tag update owner=team-cumulus
Delete resource tags using the
tag delete
commandsearch ... | tag delete costcenter
Mark resources for cleanup using the
clean
commandsearch ... | clean "Reason for cleanup"
Create PagerDuty alerts using the
pagerduty
custom commandsearch ... | pagerduty summary="Reason for the alert" dedup_key="xyz"
Send Prometheus Alertmanager alerts using the
alertmanager
custom commandsearch ... | alertmanager name="Description of the alert"
Send Slack notifications the
slack
custom commandsearch ... | slack title="Description of the alert"
Send Discord notifications the
discord
custom commandsearch ... | discord title="Description of the alert"
Stop running AWS EC2 instances using the
aws
commandsearch is(aws_ec2_instance) and instance_status=running and ... | aws ec2 stop-instances --instance-ids {id}
Start stopped AWS EC2 instances using the
aws
commandsearch is(aws_ec2_instance) and instance_status=stopped and ... | aws ec2 start-instances --instance-ids {id}
Send chunked data to a webhook server using the
http
commandsearch ... | chunk 50 | http POST my.node.org/handle
Protect resources from cleanup using the
protect
commandsearch ... | protect
Edit resource metadata using the
set_metadata
commandsearch ... | set_metadata owner=team-cumulus
Job Management​
Jobs are persisted in the Resoto database. The job
command allows you to add, update, delete, and list jobs.
See Related How-To Guides below for step-by-step instructions to create jobs.
Examples
Add a job that sends a Discord notification when compute instances without an
owner
tag are detected> jobs add --id no_owner_tag_in_dev --wait-for-event collect_done 'search is(instance) and tag.owner==null and /ancestors.account.reported.name in ["dev", "playground"] | discord --title "[DEV] Compute instances without owner tag" --message "There are compute instances in the dev accounts without owner tag:"'
Add a job that marks compute instances without an
owner
tag for cleanup every Friday evening> jobs add --id no_owner_mark_cleanup --schedule "0 22 * * FRI" --wait-for-event cleanup_plan 'search is(instance) and tag.owner==null and /ancestors.account.reported.name in ["dev", "playground"] | clean "Compute instances without owner tag"'
List all jobs
> jobs list
​id: no_owner_tag_in_dev
​command: search is(instance) and tag.owner==null and /ancestors.account.reported.name
​ in ["dev", "playground"] | discord --title "[DEV] Compute instances without owner tag"
​ --message "There are compute instances in the dev accounts without owner tag:"
​active: true
​trigger:
​ message_type: collect_done
​
​---
​id: no_owner_mark_cleanup
​command: search is(instance) and tag.owner==null and /ancestors.account.reported.name
​ in ["dev", "playground"] | clean "Compute instances without owner tag"
​active: true
​trigger:
​ cron_expression: 0 22 * * FRI
​wait:
​ message_type: cleanup_plan
Related How-To Guides​
- How to Create a Scheduled Event-Based Job
- How to Create a Scheduled Job
- How to Create an Event-Based Job