Unified Data Model Blog Posts

As companies grow, their cloud infrastructure quickly becomes fragmented and gets out of control. Data about what resources exist and how resources they relate to each other is tedious to acquire.

In practice, this means that the infrastructure layer often remains a mystery, and engineering teams are unable to see what's happening in their infrastructure. This makes capacity planning impossible, limits organizations' ability to control cloud costs, and leaves teams in the dark about potential security vulnerabilities.

The data to understand cloud growth exists as cloud resource metadata describing the state, configuration, and dependencies of cloud resources. Acquiring and unifying this "infrastructure data" into a single place is the solution for a lot of the problems that infrastructure engineers deal with today—not just cost, but also security and reliability.

But infrastructure is fragmented. Data is locked behind cloud APIs, and the tools that use those APIs to control the deployment of cloud resources. In this post, I'll explain how Resoto acquires infrastructure data, and then uses that data to write code.

Hi there, fellow cloud enthusiast! 🖖

Today, we'll delve into infrastructure fragmentation, exploring what it entails, why it impacts every organization utilizing the cloud, and how building a reactive infrastructure brings order to chaos.

Infrastructure Fragmentation: A Laptop Analogy​

Let's use your laptop as an analogy to better undertand the problem of infrastructure fragmentation.

Over time, performance declines due to a variety of factors, including the accumulation of unnecessary files, software conflicts, and the buildup of outdated or redundant software. And despite your best cleaning efforts, after a year or two its performance is just not what it used to be and everything is a bit more cluttered and disorganized than what it was like when brand new.

Now instead of just you working with that laptop, imagine there were tens, hundreds or even thousands of users. Imagine every cloud user in your organization was working on it as well. That is what your cloud infrastructure looks like.

Infrastructure fragmentation, also known as cloud fragmentation or cloud sprawl, refers to the phenomenon where an organization's cloud infrastructure becomes decentralized and uncoordinated due to the widespread adoption of cloud services by independently operating teams or individuals and a lack of centralized governance.

In other words, infrastructure fragmentation is the uncontrolled proliferation of cloud services and providers within an organization. Infrastructure fragmentation results in a disorganized and difficult-to-manage cloud infrastructure.

Infrastructure fragmentation can create various problems, such as security issues, quota and performance problems as well as increased costs due to redundancies, leaked or abandoned resources and Infrastructure as Code (IaC) drift.

EC2 instances often account for the largest portion of your AWS bill. Yet, it's notoriously difficult to get a simple list of all EC2 instances across all regions and accounts, as threads on StackOverflow and Reddit show.

You also then want to use that list to ask questions about your inventory, such as:

• How many total instances are there?
• Which instances are running?
• Which instances are missing tags?
• Which resources have an expiration date?

In this post, I'll describe how to use Resoto to build an EC2 cloud asset inventory. The baseline inventory is a list with all EC2 instances, which you then can use to create more narrow and detailed views.

Software engineers working with AWS have every cloud service imaginable at their fingertips, and developer velocity could hardly be higher. But, even the most shiny of coins has two sides.

While developers can freely spin up compute instances and databases in addition to less tangible things like Lambda functions or virtual identities—at some point, someone will ask, "What is all of this?"

And as that person hacks away in the CLI trying to get an overview of resources spanning multiple AWS accounts, they will inevitably get frustrated.

While Amazon has been a pioneer in cloud computing and offers the largest array of services, there are some things that just aren't so ideal. Namely, API consistency.

In this post, I describe a few of the challenges and quirks with the AWS API and why we're building Resoto. (Spoiler alert: It is so that you don't have to!)

Today's world of cloud computing is complex. There are many cloud providers, each with their own set of services. Getting insights out of your infrastructure requires specialized understanding of the data from each service.

Cloud Service Diversity​

Properties in different services may have different names but the same meaning, or vice versa. To interpret properties, we need to ensure that values have a defined unit of measurement and one base unit. You can see the challenge if you imagine the many ways you can specify the size of a volume, the number of CPU cores, or even timestamps.