"A generation which ignores history has no past—and no future."
— Robert A. Heinlein
While Heinlein's words refer to human history, they also apply to cloud infrastructure. Most of the time, we care about the current state of resources; but sometimes, we want to know the origin of a resource, when a resource was deleted, or when/how a resource was updated.
Such knowledge is necessary in situations where you need to understand the timeline to investigate a specific system behaviour:
- To perform the post-mortem analysis of an outage, we need to know which cloud resources changed and how they changed to yield the behaviour that we saw in our application. Without the ability to review a change log this becomes impossible.
- To understand cost spikes in your cloud billing dashboard, you need to understand what resources were created, when they were created, and by whom they were created. Not only do you need a list of changes, but also the ability to filter, group, sort, and aggregate the data to see the big picture.
- To check for security issues or compliance violations, you may need to reduce the scope to verify only those resources that were created or updated since the previous scan. Even complex checks can be performed on large infrastructures if they are only run against changed resources.
History is a log of events defining your infrastructure. This event log is important, as it will enable you to answer future questions about the state of your infrastructure retrospectively, including tomorrow's questions that have not yet crossed your mind.
Existing Cloud Provider Tools
Without Resoto, you rely on your cloud provider for information. In many cases, cloud providers offer tools yielding at least some information:
"AWS CloudTrail is an AWS service that helps you enable operational and risk auditing, governance, and compliance of your AWS account. Actions taken by a user, role, or an AWS service are recorded as events in CloudTrail. Events include actions taken in the AWS Management Console, AWS Command Line Interface, and AWS SDKs and APIs."
"Google Cloud services write audit logs that record administrative activities and accesses within your Google Cloud resources. Audit logs help you answer 'who did what, where, and when?' within your Google Cloud resources with the same level of transparency as in on-premises environments."
These tools give insights into resource changes; but if your application is spread across different accounts or regions, it is tedious to gather the data. If you have resources in a Kubernetes cluster, cloud provider tools actually won't help at all, further increasing the amount of tedious, manual work. If you want to know how a resource was configured at a specific time, your only hope is that Kubernetes Deployments may have an answer based on the configured resource limit.
Cloud providers do not offer a complete picture of the history of your infrastructure across accounts and regions, they cannot show a resource's configuration at a given point in time, and they completely ignore resources maintained in Kubernetes.
Resoto's history Command
Resoto maintains a record of your resources in its asset inventory graph. Resoto scrapes your infrastructure at regular intervals and tracks the configuration of collected resources. Whenever a change to a resource is detected, Resoto updates its internal representation of the resource and captures the change as an event in its graph database:
Since Resoto does not rely on cloud providers to provide historical data but instead generates it from collected snapshot data, you can use Resoto to get the history of all your resources, no matter where they are running.
Resource changes fall into one of three categories:
| Event | Description | Payload Data |
|---|---|---|
created | A resource was created | Configuration of the created resource |
updated | An existing resource was changed | Configuration of the updated resource |
deleted | A resource was deleted | Configuration of the resource prior to its deletion |
The history command in Resoto supports filtering of events by type and time:
> history --change node_deleted --after 2022-12-02 --before 2022-12-09
change=node_deleted, changed_at=2022-12-03T16:02:17Z, kind=aws_iam_role, name=resoto-eks-stack, cloud=aws, account=prod, region=global
change=node_deleted, changed_at=2022-12-04T16:02:17Z, kind=aws_vpc, name=eks-stack-vpc, cloud=aws, account=dev, region=eu-central-1
change=node_deleted, changed_at=2022-12-07T16:37:12Z, kind=kubernetes_pod, name=resotocore, cloud=k8s, account=resoto, region=default
We defined the absolute times for the before and after filters, but could instead use duration strings (e.g., 2d = 2 days, 4h = 4 hours, 6m = 6 minutes, 2d4h6m = 2 days 4 hours 6 minutes) to specify relative times. Durations are subtracted from the current time:
> history --change node_deleted --after 3d
change=node_deleted, changed_at=2022-12-07T16:37:12Z, kind=kubernetes_pod, name=resotocore, cloud=k8s, account=resoto, region=default
To retrieve the configuration of the deleted resource, we can pipe the result into the dump command. Since the complete configuration is part of any event, we can use this approach for any event, no matter which type of event or which time the resource changed.
> history --change node_deleted --after 3d | dump
id: '4836788'
reported:
id: b9a2ca84-399c-4716-8f1e-3ab47a4f89d4
tags:
kubectl.kubernetes.io/restartedAt: '2022-12-01T23:33:14+01:00'
name: resoto-resotocore-5ff798987-wj7cp
ctime: '2022-12-01T22:33:17Z'
resource_version: '39787763'
namespace: default
labels:
app.kubernetes.io/instance: resoto
app.kubernetes.io/name: resoto
pod-template-hash: 5ff798987
resoto: core
pod_spec:
...
Resoto includes a powerful search DSL that allows filtering resources. The history command understands this search syntax and can perform further filtering. Let's filter to only list Kubernetes deployments with names including core:
> history --after 2022-12-02 is(kubernetes_deployment) and name~core
change=node_updated, changed_at=2022-12-02T16:37:12Z, kind=kubernetes_deployment, name=resoto-resotocore, cloud=k8s, account=resoto, region=default
change=node_updated, changed_at=2022-12-04T00:02:21Z, kind=kubernetes_deployment, name=coredns, cloud=k8s, account=posthog, region=kube-system
You can not only apply filters, but you can also use any other Resoto command (e.g., count and aggregate).
If we want to know which resource kinds have changed the most in the last week, we could use the following command:
> history --after 7d | count kind
aws_iam_access_key: 96
aws_ec2_volume: 141
kubernetes_endpoint: 182
aws_iam_role: 285
kubernetes_node: 450
kubernetes_config_map: 720
total matched: 1874
total unmatched: 0
We can also count changes by type to see how many resources have been created, updated, and deleted in the last week:
> history --after 7d | count /change
node_updated: 412
node_deleted: 480
node_created: 982
total matched: 1874
total unmatched: 0
We can even aggregate by resource owner and event type if you have a tagging policy in place where resources have owner tags:
> history --after 7d | aggregate tags.owner, /change: sum(1) as count
count: 132
group:
owner: team-cumulus
change: node_updated
---
count: 637
group:
owner: team-stratos
change: node_created
---
count: 81
group:
owner: team-cumulus
change: node_created
---
count: 12
group:
owner: team-cirrus
change: node_updated
...
Looking Ahead
We plan to continue improving upon and extending the capabilities of the history command.
For example, we would like to provide a "diff" view that shows the difference between two resource configurations, so that it is possible to not only see changes and their resulting configurations but also the delta of the before and after.
The other feature we would like to introduce is the ability to generate a complete snapshot of the state of your infrastructure at a specific point in time from history data. If this is a feature you are interested in, we would love to get your input! Please join our Discord and let us know what you think.
Conclusion
If you want insights into the who, what, when, and where of infrastructure changes across your clouds, accounts, and regions, why don't you give Resoto a try?
Resoto is free and open-source. Check out the docs for installation instructions, how-to guides, and more.
