AWS Collector Development
The AWS collector source code lives in the
plugins/aws directory within the
someengineering/resoto repository on GitHub.
Adding New Resources
Resource definitions are organized by service within
Here's a step-by-step breakdown on how to add a new resource to the AWS collector:
Generate the models.
Resource models are the backbone of the collector. To kickstart support for new resources, there's a nifty
- In the
modelsdictionary, we add the
AwsResotoModelfor the desired resource and run the script. There are plenty of examples in the file already for easy reference.
- The correct
result_shapecan be found in the Botocore data in the local
venv, typically in a file called
service-2.jsonwhere output shapes for all API calls are listed.
- The models are then added to the respective service file under
- Since there is variation across different AWS services, it's good practice to verify the auto-generated model. Put timestamps, id, name, etc. under the respective properties as defined in the base classes
- At the end of each service file, there is a list of
AwsResources. New resources should be added here. If a new service is implemented, this list also needs to be added to collector.py in the parent directory.
- In the
Collect additional information.
AwsResourcemodel comes with a
collect()class method. This method can be overridden if we need to make additional API calls to get all the information.
There are three common scenarios:
A two-step collection process, where first only a list of the existing resources can be retrieved and then follow-up API call get a description for each one. See for example AwsDynamoDbTable.collect().
Separate collection of tags for a resource, e.g. in
Dependent resources, where the second one needs an additional API call parameter. One example case is the collection of
Jobsin AWS Glacier, which are specific to a
Vault. Their collection can therefore only happen within the collection of the vaults. See
Connect to other resources.
Once the new resource(s) are being collected (and new nodes in the graph have been created), edges representing the relationships between connected resources need to be added as well.
There are two ways to add these connections:
Upon collecting dependent resources, edges between the two resource instances can be added immediately.
AwsResourcemodel has a
connect_in_graph()method in which connections can be defined.
AwsApiGatewayResource.connect_in_graph()for defining a single default edge.
AwsBeanstalkEnvironmentfor defining a two-way edge.
dependant_nodeis a shortcut to add both a
defaultedge and a
- Deletion edges are necessary when one of the two resources can't be deleted while the other one is still around.
- In both cases the direction of the edges can be manipulated by using the
- The target node of an edge is identified by keyword arguments. Most commonly used is the class (
clazz) of the target node and
arnas a distinct identifier.
- All edges that were added to a resource need to be reflected in the
reference_kindsattribute of the resource class. This is used to make sure no circularity has been introduced.
reference_kindsis a simple dictionary that contains predecessors and successors along both edges. See
AwsBeanstalkEnvironment.reference_kindsas an example.
Add tagging and deletion methods.
Resoto offers tagging and deletion for resources with a unified command that is provider- and service-agnostic. To make this possible, these methods need to be implemented for each resource.
Please refer to existing resources for examples, e.g.,
Test the new resources.
Resoto has test coverage for the collector itself and for each service. Tests for the AWS collector are located within the
To verify if the collector produces the expected number of nodes and edges, update the counters in
The test suite uses a file-based client for API calls. To verify if the new resources are complete and mapped correctly, provide sample responses in the
These files and the resource tests are clustered by service, similar to how resource definitions are organized in
resoto_plugins_aws/resource. For every "
NewAwsResource" we implemented, we can do a
round_trip_for(NewAwsResource). All existing tests can serve as reference, see
Lint and analyze code.
toxin the terminal from
resoto/plugins/awsas working directory to run
mypy. These checks are required for CI to pass and PRs to be merged.