How to Collect AWS Resource Data
The Amazon Web Services (AWS) collector is configured within the Resoto Worker configuration via the config
command in Resoto Shell.
Prerequisites​
This guide assumes that you have already installed Resoto.
Directions​
1. Enable the AWS Collector​
Open the Resoto Worker configuration via the
config
command in Resoto Shell:> config edit resoto.worker
Add
aws
to the list of collectors by modifying the configuration as follows:resotoworker:
...
# List of collectors to run
collector:
- 'aws'
...
...
2. Authenticate with AWS​
Resoto supports the authentication mechanisms described in the Boto3 SDK documentation. You can authenticate with AWS via an instance profile, an access key, or profiles. These credentials can be defined in the Resoto Worker configuration or as environment variables.
- Resoto Worker Configuration
- Environment Variables
- Instance Profile
- Access Key
- Profiles
Configure an instance profile.
Open the Resoto Worker configuration via the
config
command in Resoto Shell:> config edit resoto.worker
Add the contents of your
credentials
file to theresotoworker
section of the configuration as follows:Resoto Worker configurationresotoworker:
...
write_files_to_home_dir:
- path: ~/.aws/credentials
content: |
[default]
region = us-west-2
role_arn = arn:aws:iam::235059640852:role/Resoto
external_id = a5eMybsyGIowimdZqpZWxxxxxxxxxxxx
credential_source = Ec2InstanceMetadata
...noteIf you do not wish to save the contents of your
credentials
file to Resoto's database, you can alternatively mount the~/.aws
directory to theresotoworker
container.infoSince Resoto is running on your local machine, it can access the
credentials
file directly at~/.aws/credentials
.Modify the
aws
section of the configuration as follows, making sure thataws.access_key_id
andaws.secret_access_key
are set tonull
:Resoto Worker configurationresotoworker:
...
...
aws:
# AWS Access Key ID (null to load from env - recommended)
access_key_id: null
# AWS Secret Access Key (null to load from env - recommended)
secret_access_key: null
...
Access keys in the configuration are visible to anyone with access to your Resoto instance. You can alternatively define the access key via environment variables.
Open the Resoto Worker configuration via the
config
command in Resoto Shell:> config edit resoto.worker
Modify the
aws
section of the configuration as follows:Resoto Worker configurationresotoworker:
...
...
aws:
# AWS Access Key ID (null to load from env - recommended)
access_key_id: 'AKIAZGZKXXXXXXXXXXXX'
# AWS Secret Access Key (null to load from env - recommended)
secret_access_key: 'vO51EW/8ILMGrSBV/Ia9Fov6xZnKxxxxxxxxxxxx'
...
Create a file
~/.aws/credentials
with the desired profiles:~/.aws/credentials[production]
aws_xxx = yyy
[test]
aws_xxx = yyy
[dev]
aws_xxx = yyy
...Open the Resoto Worker configuration via the
config
command in Resoto Shell:> config edit resoto.worker
Add the contents of your
credentials
file to theresotoworker
section of the configuration as follows:Resoto Worker configurationresotoworker:
...
write_files_to_home_dir:
- path: ~/.aws/credentials
content: |
[default]
region = us-west-2
role_arn = arn:aws:iam::235059640852:role/Resoto
external_id = a5eMybsyGIowimdZqpZWxxxxxxxxxxxx
credential_source = Ec2InstanceMetadata
...noteIf you do not wish to save the contents of your
credentials
file to Resoto's database, you can alternatively mount the~/.aws
directory to theresotoworker
container.infoSince Resoto is running on your local machine, it can access the
credentials
file directly at~/.aws/credentials
.Modify the
aws
section of the configuration as follows, adding one or more profile names from your~/.aws/credentials
file:Resoto Worker configurationresotoworker:
...
...
aws:
...
profiles:
- production
- test
- devProfiles can be combined with other AWS options, such as
aws.role
andaws.scrape_org
.noteWhen switching from profiles to another authentication option, be sure to set the value of
aws.profiles
asnull
.
Resoto is meant to run unattended on a server using a service account or instance profile. Resoto supports the same environment variables that the AWS Command-Line Interface does (AWS_ACCESS_KEY_ID
, AWS_SECRET_ACCESS_KEY
, AWS_SESSION_TOKEN
, AWS_ROLE_ARN
, AWS_WEB_IDENTITY_TOKEN_FILE
, AWS_ROLE_SESSION_NAME
, etc.).
When using temporary credentials, however, they should be written to the credentials
or config
file and updated out-of-band instead of using environment variables. This is because the resotoworker
process starts once and then runs forever (i.e., updated environment variables are only reflected upon restart).
You can specify a profile using AWS_PROFILE
and, for local testing, SSO authentication would work as well. However, when Resoto is running unattended in a production environment, SSO credentials that require opening a browser window would not work.
Set the required environment variables (e.g.,
AWS_ACCESS_KEY_ID
andAWS_SECRET_ACCESS_KEY
):- Docker
- Kubernetes
- pip
Add environment variable definitions to the
resotoworker
service indocker-compose.yaml
:docker-compose.yamlservices:
...
resotoworker:
environment:
- AWS_ACCESS_KEY_ID=AKIAZGZKXXXXXXXXXXXX
- AWS_SECRET_ACCESS_KEY=vO51EW/8ILMGrSBV/Ia9Fov6xZnKxxxxxxxxxxxx
...
...Recreate the
resotoworker
container with the updated service definition:$ docker-compose up -d
noteDocker Compose V2 integrated compose functions in to the Docker platform.
In Docker Compose V2, the command is
docker compose
(no hyphen) instead ofdocker-compose
.
Create a secret:
$ kubectl -n resoto create secret generic resoto-auth \
--from-literal=AWS_ACCESS_KEY_ID=AKIAZGZKXXXXXXXXXXXX \
--from-literal=AWS_SECRET_ACCESS_KEY=vO51EW/8ILMGrSBV/Ia9Fov6xZnKxxxxxxxxxxxxUpdate
resoto-values.yaml
as follows:resoto-values.yaml...
resotoworker:
...
extraEnv:
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: resoto-auth
key: AWS_ACCESS_KEY_ID
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: resoto-auth
key: AWS_SECRET_ACCESS_KEY
...Deploy these changes with Helm:
$ helm upgrade resoto resoto/resoto --set image.tag=edge -f resoto-values.yaml
Export the
AWS_ACCESS_KEY_ID
andAWS_SECRET_ACCESS_KEY
environment variables:$ export AWS_ACCESS_KEY_ID=AKIAZGZKXXXXXXXXXXXX
$ export AWS_SECRET_ACCESS_KEY=vO51EW/8ILMGrSBV/Ia9Fov6xZnKxxxxxxxxxxxxRestart the
resotoworker
process.
Open the Resoto Worker configuration via the
config
command in Resoto Shell:> config edit resoto.worker
Modify the
aws
section of the configuration as follows, making sure thataws.access_key_id
andaws.secret_access_key
are set tonull
:Resoto Worker configurationresotoworker:
...
...
aws:
# AWS Access Key ID (null to load from env - recommended)
access_key_id: null
# AWS Secret Access Key (null to load from env - recommended)
secret_access_key: null
...
3. Authorize Resoto Access to AWS​
See How to Roll Out Resoto AWS Permissions with CloudFormation for a step-by-step guide on how to roll out Resoto permissions organization-wide.
If you prefer to deploy the role yourself, the S3 URL of the template to create the ResotoAccess
role is https://resotopublic.s3.amazonaws.com/cf/resoto-role.template.
4. Trigger Resource Collection​
By default, Resoto performs resource collection each hour. To immediately trigger a collect run, use the
workflow run
command in Resoto Shell:> workflow run collect
Once the collect run completes, you can view a summary of collected AWS resources using the following search:
> search is(aws_resource) | count kind