Skip to main content

Automation

Maintaining a sound cloud infrastructure includes tasks that need to be performed regularly. Resoto can help you automate many of these tasks by defining jobs.

Examples of these tasks include:

  • enforcing company policies (e.g., tagging),
  • sending notifications about critical infrastructure changes (e.g., to Slack, Discord, or PagerDuty), and
  • cleaning up unused resources in your development accounts.

Jobs​

A job is a sequence of commands executed by Resoto at a specific time interval or when a specific event is emitted.

Jobs in Resoto are similar to Unix cron jobs. However, while cron jobs run only on a specific schedule, Resoto jobs can also be triggered based by events.

Job Overview

The typical structure of a job has three parts: a trigger, an optional filter, and an action.

Job Trigger​

The job trigger defines when a job is executed.

There are three types of job triggers: schedule triggers, event triggers, and combined triggers.

Schedule Trigger​

A schedule trigger executes a job at a specific time interval described by a cron expression.

Examples
  • Every 5 minutes
    --schedule "*/5 * * * *"
  • Every day at 3:00am
    --schedule "0 3 * * *"
  • Every Monday at 4:00am
    --schedule "0 4 * * MON"
  • Every New Year's Eve at 11:59pm
    --schedule "59 23 31 12 *"
tip

See How to Create a Scheduled Job for step-by-step instructions describing how to create a job with a schedule trigger.

Event Trigger​

An event trigger executes a job when a specific event is emitted by Resoto.

Resoto updates the state of resources in the four steps of the collect_and_cleanup workflow: collect, cleanup_plan, cleanup, and generate_metrics.

Each of these steps emits events that can be used to trigger jobs.

Examples
  • When resource collection is complete and the database reflects the latest state of your resources
    --wait-for-event collect_done
  • When the cleanup planning process has started to mark resources for cleanup
    --wait-for-event cleanup_plan
info

See Events for a list of events emitted by Resoto.

tip

See How to Create an Event-Based Job for step-by-step instructions describing how to create a job with an event trigger.

Combined Trigger​

A combined trigger combines a schedule trigger with an event trigger, and executes a job when a specific event is emitted after a schedule trigger is fired.

Combined triggers are useful if you want to perform an action on a specific schedule, but only after a specific event is fired.

Example

Let's say you want to clean up development accounts at the end of each week.

To do so, you could define a schedule trigger in addition to specifying the cleanup_plan event trigger:

--schedule "0 22 * * FRI" --wait-for-event cleanup_plan

The above combined trigger executes the job based on the latest state of resources after 10pm every Friday evening.

tip

See How to Create a Scheduled Event-Based Job for step-by-step instructions describing how to create a job with a combined trigger.

Job Filters & Actions​

The scope and functionality of a job is defined by filters and actions, respectively.

To perform more complex automations, it is possible to define multiple sets of filters and actions delimited by a semicolon. Each set of filters and actions is executed sequentially when the job is triggered.

Job Filters​

A job filter is an optional search that returns the resources of interest. Only resources matching the filter will be processed by the job.

Example
  • Instances without an owner tag in the dev or playground accounts
    search is(instance) and tag.owner==null and /ancestors.account.reported.name in ["dev", "playground"]

The result of a job filter is a list of resources that is piped as input to job actions.

Job Actions​

A job action is an action command performed on resources matching the optional filter.

Examples

  • Update resource tags using the tag update command

    search ... | tag update owner=team-cumulus

  • Delete resource tags using the tag delete command

    search ... | tag delete costcenter

  • Mark resources for cleanup using the clean command

    search ... | clean "Reason for cleanup"

  • Create PagerDuty alerts using the pagerduty custom command

    search ... | pagerduty summary="Reason for the alert" dedup_key="xyz"

  • Send Prometheus Alertmanager alerts using the alertmanager custom command

    search ... | alertmanager name="Description of the alert"

  • Send Slack notifications the slack custom command

    search ... | slack title="Description of the alert"

  • Send Discord notifications the discord custom command

    search ... | discord title="Description of the alert"

  • Stop running AWS EC2 instances using the aws command

    search is(aws_ec2_instance) and instance_status=running and ... | aws ec2 stop-instances --instance-ids {id}

  • Start stopped AWS EC2 instances using the aws command

    search is(aws_ec2_instance) and instance_status=stopped and ... | aws ec2 start-instances --instance-ids {id}

  • Send chunked data to a webhook server using the http command

    search ... | chunk 50 | http POST my.node.org/handle

  • Protect resources from cleanup using the protect command

    search ... | protect

  • Edit resource metadata using the set_metadata command

    search ... | set_metadata owner=team-cumulus

Job Management​

Jobs are persisted in the Resoto database. The jobs command allows you to add, update, delete, and list jobs.

tip

See Related How-To Guides below for step-by-step instructions to create jobs.

Examples

  • Add a job that sends a Discord notification when compute instances without an owner tag are detected

    > jobs add --id no_owner_tag_in_dev --wait-for-event collect_done 'search is(instance) and tag.owner==null and /ancestors.account.reported.name in ["dev", "playground"] | discord --title "[DEV] Compute instances without owner tag" --message "There are compute instances in the dev accounts without owner tag:"'

  • Add a job that marks compute instances without an owner tag for cleanup every Friday evening

    > jobs add --id no_owner_mark_cleanup --schedule "0 22 * * FRI" --wait-for-event cleanup_plan 'search is(instance) and tag.owner==null and /ancestors.account.reported.name in ["dev", "playground"] | clean "Compute instances without owner tag"'

  • List all jobs

    > jobs list
    ​id: no_owner_tag_in_dev
    ​command: search is(instance) and tag.owner==null and /ancestors.account.reported.name
    ​  in ["dev", "playground"] | discord --title "[DEV] Compute instances without owner tag"
    ​  --message "There are compute instances in the dev accounts without owner tag:"
    ​active: true
    ​trigger:
    ​  message_type: collect_done
    ​
    ​---
    ​id: no_owner_mark_cleanup
    ​command: search is(instance) and tag.owner==null and /ancestors.account.reported.name
    ​  in ["dev", "playground"] | clean "Compute instances without owner tag"
    ​active: true
    ​trigger:
    ​  cron_expression: 0 22 * * FRI
    ​wait:
    ​  message_type: cleanup_plan

Further Reading​

Contact Us