Kubernetes is the de-facto standard for orchestrating containerized applications. It is the go-to solution no matter where your infrastructure is running. Resoto can already collect resources in Amazon Web Services, Google Cloud, and DigitalOcean, all of which support Kubernetes.
I'm happy to announce that Resoto now supports collecting Kubernetes resources!
> config edit resoto.worker
In the text editor opened by
config edit, modify the configuration and define the location of your kubeconfig file. By default, Resoto will collect all contexts that are found in the kubeconfig file. It is possible to restrict the contexts to be used by setting the
contexts property and turning
- path: "/home/resoto/.kube/config"
Alternatively, you can also copy & paste the relevant fields directly to the configuration in case the ResotoWorker is running on a machine that does not have access to the file. An example configuration might look like this:
- # The name of the kubernetes cluster.
# The url of the server to connect to.
# The user access token to use to access this cluster.
# The optional CA certificate string.
The Kubernetes collector can collect multiple Kubernetes clusters by defining multiple entries in the
Once this is done, Resoto will start collecting data from all configured clusters on the schedule configured in ResotoCore
resotocore.workflows.collect_and_cleanup.schedule, which defaults to every hour. You can trigger a collection run manually by entering:
> workflows run collect_and_cleanup
Kubernetes Resource Data Model
Every resource found in Kubernetes is represented by a node in the graph. Available resource properties are captured and available to filter and automate.
Resoto has a common data model for resources that abstracts from the underlying provider. Every resource inherits the properties from the base
resource, which defines properties like
Every resource in Kubernetes is also of type
kubernetes_resource, which brings in additional properties like a
labels, and an optional
namespace property, which is defined for all namespaced resources. Annotations of a Kubernetes resource are available as
tags and can be modified via the
Common abstractions apply as well. A PersistentVolume is of base type
volume. So it shares the same kind and properties with other volumes of other cloud providers, e.g. an
EBSVolume in AWS or a
Disk in Google Cloud. The same idea applies to a Kubernetes node which is of base type
Kubernetes has its own way of describing a resource, which is available in Resoto as well. The three main sections
status can be found in almost any resource. The data in the
metadata section is basically covered by the base
resource properties (
tags, etc.). The
spec section usually holds the desired state, while the
status section holds the current state.
Both sections are unique to the resource type. Since Resoto allows searching on any resource property, we made the
status section unique to the resource type. So a
Pod in Resoto has a
pod_status section, while a
Deployment has a
deployment_status section. The relevant spec and status sections have the same specific data model as the Kubernetes resource.
You can find a complete reference at Kubernetes Resource Data Models.
Searching the Graph
Since all Kubernetes resources share the same base kind
kubernetes_resource, we can filter the graph by this resource type. We will count the number of Kubernetes resources by its kind. This will print the kind of the resource and the number of occurrences in ascending order. This is the output for the 2 clusters we have configured:
> search is(kubernetes_resource) | count kind
total matched: 1234
total unmatched: 0
Can we reveal general information about the Kubernetes clusters? This query shows all available clusters including version and server URL:
> search is(kubernetes_cluster) | list cluster_info
major=1, minor=21, platform=linux/amd64, server_url=https://31e9afd3-xxxxx.k8s.ondigitalocean.com
major=1, minor=21, platform=linux/amd64, server_url=https://9a3ac2b5-xxxxx.k8s.ondigitalocean.com
Now we can start asking questions about the state of our Kubernetes infrastructure. For example, how many pods are running on each node in the cluster?
> search is(kubernetes_pod) | count /ancestors.kubernetes_node.reported.name
total matched: 70
total unmatched: 0
You want to see details? Let's retrieve the information that has been collected about a Kubernetes service with name
> search is(kubernetes_service) and name~resotocore | dump
- name: http
- ip: 18.104.22.168
Resoto ships with the ability to search for a value everywhere, no matter where this value is defined. We call this feature full-text search - you can read about all the details here full-text search Let us use this feature, to search an IP address that we found in the resoto core service. We expect it will find exactly the same ResotoCore service:
> search "10.245.133.206"
kind=kubernetes_service, id=df8f6c37-5542-4e17-b2f4-bb6caaa2ba17, name=resoto-resotocore, age=2mo1d, cloud=k8s, account=dev, region=resoto
Let us find all secrets inside Kubernetes, that are shared between more than one pod. Such queries might be useful to find resources, that have a defined relationship in the graph:
> search is(kubernetes_secret) and namespace=resoto with(count>1, <-- is(kubernetes_pod))
kind=kubernetes_secret, id=af1495e4, name=resoto-psk, namespace=resoto, age=2mo1d, cloud=k8s, account=dev, region=resoto
There is more than one secret in this namespace, but only one is shared: the resoto-psk secret holds the private shared key. If we want to see the attached pods, we can use the same query again and list the resources. We will see that ResotoCore, ResotoWorker and ResotoMetrics use the secret resoto-psk. Please note: the secrets that are collected do not sensitive data fields.
> search is(kubernetes_secret) and namespace=resoto with(count>1, <-- is(kubernetes_pod)) <-- is(kubernetes_pod) | list name
Render a graph of all services that are deployed in the
resoto namespace on the
dev cluster. Please note: we are looking at a cluster where we have deployed
resoto as a Helm chart as described in Install Resoto with Kubernetes.
search --with-edges is(kubernetes_service) and namespace=resoto <-[0:]-> | format --dot | write service.dot
This will output a Graphviz file that can be rendered with the
You can see the list of services as successors of the namespace node. Each service has one or more pods attached to it, where the pods reference configmaps and secrets. The data in Resoto is persistent, so you can also find a persistent volume claim that is attached to the database pod. The claim then references the actual persistent volume.
All annotations in Kubernetes are available as tags. Resoto allows to update tags via ResotoShell, as well as delete tags.
Let's tag all pods with the name
resoto in it with the tag
owner. Please note: this tag will be temporary and goes away once the pod is deleted since a pod is only a template instance in Kubernetes. If we wanted a more permanent tag, we would need to adjust the related pod template - the deployment in this case.
> search is(kubernetes_pod) and name~resoto | tag update owner matthias | list name, tags
To make sure the existing resources in the cluster have changed:
$ kubectl get pods resoto-resotocore-67858dbc49-kzh7l -o json | jq .metadata.annotations
Tags can be updated and deleted again:
> search is(kubernetes_pod) and name~resoto | tag delete owner | list name, tags
To delete resources, pipe search results to the
clean command. Let us find volume claims that are not bound to any kubernetes pod and clean them up:
> search is(kubernetes_persistent_volume_claim) with(empty, <-- is(kubernetes_pod)) -[0:1]-> | clean
kind=kubernetes_persistent_volume_claim, id=079affbf, name=single-tvhrrqmm, age=6mo23d, cloud=k8s, account=dev, region=test
kind=kubernetes_persistent_volume, id=c299101c, name=pvc-079affbf, age=6mo23d, cloud=k8s, account=dev, region=test
This query selects all volume claims that have no pod attached and then also select the related volume. The clean command marks the resources for cleanup. The resources will be deleted once the
collect_and_cleanup workflow runs (default: every hour).
We are currently working on the way to connect resources found in Kubernetes to resources found in the underlying cloud provider. Let's assume you have a Kubernetes cluster running on Amazon Web Services, Google Cloud, or DigitalOcean and we want to show the underlying persistent volume provided by the related cloud provider. This would show which compute instance a pod runs on or which underlying volume is provided.