Cloud2SQL is a tool based on Resoto's collector plugins that allows you to collect data from various cloud infrastructure sources and export it to a database (like Snowflake, PostgreSQL, MariaDB, or MySQL) or write it as Parquet, SQLite, or CSV files for ingestion in your data lake.

In this post, I will guide you through the process of installing Cloud2SQL and demonstrate how to export data from AWS to a local SQLite database file.

Whether you are looking to integrate cloud data into your existing SQL workflows or simply want an easy way to access and analyze data from multiple cloud sources, Cloud2SQL is an excellent tool to consider.

One of the key features of Resoto is its ability to collect data from a wide range of cloud providers, including Amazon Web Services (AWS), DigitalOcean, and Google Cloud. This makes it easy to get a comprehensive view of your cloud infrastructure, no matter where it is deployed.

But, what sets Resoto apart from other cloud data collection tools is its ability to enrich the data it collects and make additional connections. This means that Resoto not only gathers raw data about your cloud resources, but also adds additional context and information that can help you better understand your cloud environment.

"A generation which ignores history has no past—and no future."
— Robert A. Heinlein

While Heinlein's words refer to human history, they also apply to cloud infrastructure. Most of the time, we care about the current state of resources; but sometimes, we want to know the origin of a resource, when a resource was deleted, or when/how a resource was updated.

Such knowledge is necessary in situations where you need to understand the timeline to investigate a specific system behaviour:

• To perform the post-mortem analysis of an outage, we need to know which cloud resources changed and how they changed to yield the behaviour that we saw in our application. Without the ability to review a change log this becomes impossible.
• To understand cost spikes in your cloud billing dashboard, you need to understand what resources were created, when they were created, and by whom they were created. Not only do you need a list of changes, but also the ability to filter, group, sort, and aggregate the data to see the big picture.
• To check for security issues or compliance violations, you may need to reduce the scope to verify only those resources that were created or updated since the previous scan. Even complex checks can be performed on large infrastructures if they are only run against changed resources.

History is a log of events defining your infrastructure. This event log is important, as it will enable you to answer future questions about the state of your infrastructure retrospectively, including tomorrow's questions that have not yet crossed your mind.

A Resoto install comes with batteries included; Resoto ships with a command-line interface (CLI) that allows for exploration, insights, and manipulation of your infrastructure. With Resoto's CLI, automating tedious tasks becomes a breeze. Think about enforcing a policy, cleaning up resources, exporting data, or alerting on specific circumstances. See How-To Guides to learn more about possible use cases.

Version 3 of Resoto introduces the ability to extend this capability by defining custom commands programmatically in the language of your choice. If you are familiar with Python, this task becomes super easy, since all the necessary boilerplate code is already provided.

In this blog post, we will implement a new command called hello-world in Python, to show the power and flexibility of this new feature. The simple idea of our new command is adding a greeting to the tags of a selected resource.

Cloud tagging strategies and policies are hailed as one of the most efficient ways to keep your cloud infrastructure controllable. But are they really?

Cloud Resource Tagging​

Generally, the idea is that every piece of cloud service gets tagged (or labeled, in case of Google Cloud) by the developers or maintainers who work with it. This could be accomplished with infrastructure-as-code (IaC) tools (such as Terraform), with a command-line interface (CLI), or in the cloud UI.

Cloud Resource Tagging Policies​

Tagging policies could require that each resource needs tags identifying the owner, cost center, product, project, and/or any other metadata. By being diligent about tagging, resources can be managed via their tags and nothing gets overlooked.

Cloud Resource Tagging Challenges​

In theory, this is the correct way to manage resources; in practice, however, this hardly ever works as intended.

Each tag created is a tag that requires maintenance. Tagging policies may change over time and people can make mistakes (in AWS, for example, tag keys are case sensitive).

And, to properly use tagging on a greenfield cloud account is one thing; to retroactively apply tags to sprawling cloud infrastructure is quite another (especially when utilizing a multi-cloud strategy, where you'd need to repeat any operation over multiple interfaces).