Automation

Maintaining a sound cloud infrastructure includes tasks that need to be performed regularly. Resoto can help you automate many of these tasks by defining jobs.

Examples of these tasks include:

• enforcing company policies (e.g., tagging),
• sending notifications about critical infrastructure changes (e.g., to Slack, Discord, or PagerDuty), and
• cleaning up unused resources in your development accounts.

Jobs

A job is a sequence of commands executed by Resoto at a specific time interval or when a specific event is emitted.

Jobs in Resoto are similar to Unix cron jobs. However, while cron jobs run only on a specific schedule, Resoto jobs can also be triggered based by events.

The typical structure of a job has three parts: a trigger, an optional filter, and an action.

Job Trigger

The job trigger defines when a job is executed.

There are three types of job triggers: schedule triggers, event triggers, and combined triggers.

Schedule Trigger

A schedule trigger executes a job at a specific time interval described by a cron expression.

Examples

• Every 5 minutes

  --schedule "*/5 * * * *"
• Every day at 3:00am

  --schedule "0 3 * * *"
• Every Monday at 4:00am

  --schedule "0 4 * * MON"
• Every New Year's Eve at 11:59pm

  --schedule "59 23 31 12 *"

tip

See How to Create a Scheduled Job for step-by-step instructions describing how to create a job with a schedule trigger.

Event Trigger

An event trigger executes a job when a specific event is emitted by Resoto.

Resoto updates the state of resources in the four steps of the collect_and_cleanup workflow: collect, cleanup_plan, cleanup, and generate_metrics.

Each of these steps emits events that can be used to trigger jobs.

Examples

• When resource collection is complete and the database reflects the latest state of your resources

  --wait-for-event collect_done
• When the cleanup planning process has started to mark resources for cleanup

  --wait-for-event cleanup_plan

info

See Events for a list of events emitted by Resoto.

tip

See How to Create an Event-Based Job for step-by-step instructions describing how to create a job with an event trigger.

Combined Trigger

A combined trigger combines a schedule trigger with an event trigger, and executes a job when a specific event is emitted after a schedule trigger is fired.

Combined triggers are useful if you want to perform an action on a specific schedule, but only after a specific event is fired.

Example

Let's say you want to clean up development accounts at the end of each week.

To do so, you could define a schedule trigger in addition to specifying the cleanup_plan event trigger:

--schedule "0 22 * * FRI" --wait-for-event cleanup_plan

The above combined trigger executes the job based on the latest state of resources after 10pm every Friday evening.

tip

See How to Create a Scheduled Event-Based Job for step-by-step instructions describing how to create a job with a combined trigger.

Job Filters & Actions

The scope and functionality of a job is defined by filters and actions, respectively.

To perform more complex automations, it is possible to define multiple sets of filters and actions delimited by a semicolon. Each set of filters and actions is executed sequentially when the job is triggered.

Job Filters

A job filter is an optional search that returns the resources of interest. Only resources matching the filter will be processed by the job.

Example

• Instances without an owner tag in the dev or playground accounts

  search is(instance) and tag.owner==null and /ancestors.account.reported.name in ["dev", "playground"]

The result of a job filter is a list of resources that is piped as input to job actions.

Job Actions

A job action is an action command performed on resources matching the optional filter.

Examples

• Update resource tags using the tag update command

  search ... | tag update owner=team-cumulus

• Delete resource tags using the tag delete command

  search ... | tag delete costcenter

• Mark resources for cleanup using the clean command

  search ... | clean "Reason for cleanup"

• Create PagerDuty alerts using the pagerduty custom command

  search ... | pagerduty summary="Reason for the alert" dedup_key="xyz"

• Send Prometheus Alertmanager alerts using the alertmanager custom command

  search ... | alertmanager name="Description of the alert"

• Send Slack notifications the slack custom command

  search ... | slack title="Description of the alert"

• Send Discord notifications the discord custom command

  search ... | discord title="Description of the alert"

• Stop running AWS EC2 instances using the aws command

  search is(aws_ec2_instance) and instance_status=running and ... | aws ec2 stop-instances --instance-ids {id}

• Start stopped AWS EC2 instances using the aws command

  search is(aws_ec2_instance) and instance_status=stopped and ... | aws ec2 start-instances --instance-ids {id}

• Send chunked data to a webhook server using the http command

  search ... | chunk 50 | http POST my.node.org/handle

• Protect resources from cleanup using the protect command

  search ... | protect

• Edit resource metadata using the set_metadata command

  search ... | set_metadata owner=team-cumulus

Job Management

Jobs are persisted in the Resoto database. The job command allows you to add, update, delete, and list jobs.

tip

See Related How-To Guides below for step-by-step instructions to create jobs.

Examples

• Add a job that sends a Discord notification when compute instances without an owner tag are detected

  > jobs add --id no_owner_tag_in_dev --wait-for-event collect_done 'search is(instance) and tag.owner==null and /ancestors.account.reported.name in ["dev", "playground"] | discord --title "[DEV] Compute instances without owner tag" --message "There are compute instances in the dev accounts without owner tag:"'

• Add a job that marks compute instances without an owner tag for cleanup every Friday evening

  > jobs add --id no_owner_mark_cleanup --schedule "0 22 * * FRI" --wait-for-event cleanup_plan 'search is(instance) and tag.owner==null and /ancestors.account.reported.name in ["dev", "playground"] | clean "Compute instances without owner tag"'

• List all jobs

  > jobs list
  ​id: no_owner_tag_in_dev
  ​command: search is(instance) and tag.owner==null and /ancestors.account.reported.name
  ​  in ["dev", "playground"] | discord --title "[DEV] Compute instances without owner tag"
  ​  --message "There are compute instances in the dev accounts without owner tag:"
  ​active: true
  ​trigger:
  ​  message_type: collect_done
  ​
  ​---
  ​id: no_owner_mark_cleanup
  ​command: search is(instance) and tag.owner==null and /ancestors.account.reported.name
  ​  in ["dev", "playground"] | clean "Compute instances without owner tag"
  ​active: true
  ​trigger:
  ​  cron_expression: 0 22 * * FRI
  ​wait:
  ​  message_type: cleanup_plan

Related How-To Guides

• How to Create a Scheduled Event-Based Job
• How to Create a Scheduled Job
• How to Create an Event-Based Job

Further Reading

• Cloud Data Sync
• job Command
• Command-Line Interface
• Events
• Workflows