Collector Plugin Development

Plugin source code lives in the plugins directory within the someengineering/resoto GitHub repository. Each plugin is maintained as separate project.

Contributions are made via pull requests to the GitHub repository. You will first need to fork the repository.

Pull requests should target a single plugin. Please refer to Contributing to Components for details on how to clone the repository, set up a virtual environment, start Resoto, and submit your changes for review.

Collector Plugins

Collector plugins allow for importing of arbitrary resources into Resoto in graph form. The most common use case is to gather information about cloud accounts and/or resources. However, any data expressible is graph form can be collected—be it social media accounts, software dependency trees, network topology, steps for cooking your favorite food, etc.

Once the graph data is collected and sent to Resoto Core, the power of Resoto's search is at your fingertips.

📄️

AWS Collector

The AWS collector source code lives in the plugins/aws directory within the someengineering/resoto repository on GitHub.

Plugin Interface

The collector plugin interface is defined as follows:

def collect(self) -> None:
  """Collects all the Cloud Resources"""
  account_graph = collect_account()
  self.graph.merge(account_graph)

The typical flow of the collect method is:

1. An instance of a Graph class is created and filled with cloud resources.

2. The graph is merged with the plugin's internal graph.

Resource Graph Structure

Usually, the graph is structured as follows:

1. The top-level node is the cloud itself. It is added by the collector plugin automatically.

2. The next level is the account (e.g., an AWS account, Google Cloud project, DigitalOcean team, etc.).

3. Where applicable, the following level is the region (e.g., US-East-1, US-West-1, etc.).

4. The remaining levels are cloud-specific resources.

Merging Collected Resources

Resoto uses a thin wrapper on top of NetworkX to operate on graphs. The two most used methods in the wrapper are add_resource and add_edge.

Keep in mind that there are two types of edges, default and delete.

• default edges represent how resources relate to each other (e.g., a resource may be represented by a child node of a region node).

• delete edges define the order in which resources should be cleaned up (e.g., a specific instance needs to be deleted before a particular volume can be deleted).

Please refer to the example collector plugin for a model of how to implement this logic.

Testing

To test a collector plugin, simply launch Resoto and trigger the collect action manually by executing workflow run collect in Resoto Shell.

Once resource collection is complete, you can execute search (<plugin_resource_type>) to see the newly collected resources.

Tips

• Resource properties types must be globally unique.

  For example, if there is a property status: str defined by some other plugin and you add a property status: int, the collection will fail. The reason for such behavior is that all resource properties are indexed for search.

• Try to not introduce unique property names.

  Only do it in event that types are not compatible with existing properties. For example, use status: str instead of plugin_name_resource_name_status: int.

• Don't forget to call sanitize(graph) before checking its properties in tests.

  Please refer to the DigitalOcean plugin as an example.