Why Your Cloud Assets Need A Time Machine

Tracking the configuration of cloud resources is essential in the DevOps world. But what happens when you need to look back at past configurations? This is where resource history tracking comes in. By tracking the history of your cloud resources, you can retrieve configurations from any point in time, even if you have since deleted the resource. Furthermore, you can use history tracking to investigate unexpected changes, such as when something goes wrong with your cloud resource.

Using Git For You Cloud’s Asset History

Git is really a great tool to enable you to investigate who has made changes to files and when, and also revert changes, as needed. With the growing complexity of cloud operations, version control is becoming mission critical to business continuity, and is a best practice with DevOps and cloud operations today.

Git Is great when it comes to:

• Retrieving assets configuration from  a particular point in time
• Reverting to a specific version of the asset using Git revert capabilities.
• Looking at the commit’s information about the committer and the commit message to understand the motivation for the change, and hope the person who has done the commit has written good commit messages. (Here is a link for a good blog about commit messages https://cbea.ms/git-commit/)
• Git is generic and all your environments can be managed there in a single place.  This is true even for tools where codification is only in its infancy, like Okta (this is a great post about Okta as Code).
• Ability to compare two points in time to see exactly the change that has happened.

But sometimes Git alone is not enough when it comes to managing resources in your cloud. Even if we don’t want to manually change resources in the cloud - this does still happen from time to time. When this happens Git won’t know about those changes. In this scenario you must constantly be proactive and sample the cloud for changes and track them to not miss such misconfigurations. This “ClickOps” (a manual change) will usually be the changes that interest you when something bad happens.

‍In addition sometimes you will have cloud resources that are not being managed by Infrastructure as Code in a Git repository and the timeline of your repo won’t cover any changes to them at all. There are cases where you want to see the history of unmanaged resources too.

‍Another area that Git breaks down is when the scope is too large. Since Git tracks changes to files, the scope of changes can be too large when dealing with cloud resources. You might have multiple resources in a single file and multiple resources and even files in a single commit, making it harder to track changes to individual resources. Also regular git events won’t be helpful in such a case if you want to revert just a single resource and not all the resources of the commit.

‍So while Git is a great tool, it's not always enough for tracking changes across all of your cloud resources. This is why sometimes you need a timeline for your resources showing what really happened across your infrastructure.

Using Cloud Vendor’s Audit Tools for Cloud’s Asset History

Different Cloud SaaS vendors offer their audit services, such as AWS's CloudTrail or Azure’s Azure Monitor, And Google Cloud’s Cloud Audit Logs. This can be leveraged to see what happened to your cloud resources, and they are great tools with a lot of benefits.

• Audit tools allow you to track changes to all resources, including those not managed by code.
• You won't miss any manual changes, ensuring that you have a complete picture of your resource's history.
• The timeline view provided by audit tools makes it easy to see what happened to a single resource over time.
• Event names help build a full story of what happened to your asset, showing who made changes and what happened to deleted assets.

But sometimes your audit tool also won’t be enough because:

• Audit tools may not provide a full view of a resource's configuration at a certain point in time, preventing you from traveling to the past.
• Each vendor has their own audit tool, which can make it difficult to work with multiple systems if your infrastructure is spread across multiple vendors.
• Knowing past IAC states of your managed resources might be useful when you want to know when it was drifted and why. Vendor audit tools don’t take in count the infrastructure as code and the data there is not helpful enough. Having the IAC state of the resource against its state as part of the resource timeline is powerful.
• Because they audit the change and not full configuration you won’t be able to make a comparison between two points in time in case you need.

Using Cloud’s Change History Feature

Some vendors have implemented their own asset history feature, such as Google Cloud's Change History Tab as part of their Asset Inventory Service.

Its benefits are similar to the benefits of the audit tool:

• Cloud change history tools allow you to monitor and track changes made to your resources, even those that are not managed through code. This ensures that no manual changes go unnoticed, providing a comprehensive overview of your resource's history.
• The timeline view provided by the vendor's change history tool makes it convenient to visualize the sequence of events and alterations that occurred to a specific resource. This chronological representation enhances the understanding of the resource's history.
• The compare feature also enables you to compare the screen between two selected points in time. This comparison assists in comprehending the specific modifications that took place, facilitating analysis and troubleshooting.
• Being in some console with your cloud infrastructure is a great benefit because you don’t need to go to another place and everything is real time because it’s in the same place.

But there are also some disadvantages in these vendors’ asset history features:

• Vendor dependency: Each vendor has its own cloud history tool, and some vendors may not provide this feature at all. This can pose challenges if your infrastructure is spread across multiple vendors, requiring you to work with different systems to access the history of your resources.
• Limited insights into IAC states: Vendor audit tools typically do not consider infrastructure as code (IAC) and provide limited information in this regard. Having the ability to compare the IAC state of a resource with its actual state as part of the resource's timeline would be powerful for understanding drifts and troubleshooting.
• Lack of deleted resource tracking: One notable disadvantage of relying solely on the vendor's asset history feature is the absence of a view for deleted resources. Most vendor consoles do not provide a mechanism to track and retain history for assets that have been deleted. Consequently, once a resource is deleted, its history is lost, making it challenging to trace back and understand the reasons behind its deletion. This limitation emphasizes the need for alternative approaches or tools if preserving the history of deleted resources is essential for your workflow.

Let's Do a Comparison of What's Important

It appears that audit trails, vendor built in asset history and Git are useful tools for reviewing the history of your resources, but each has its own limitations. It would be ideal if there were a tool that could combine the advantages of the three methods we mentioned here.

‍

Using Firefly Asset History

Firefly Asset History provides a convenient solution that combines the benefits of both worlds being multi and cross-cloud while also fusing the added value from both Git and audit tools, to provide a comprehensive view of your inventory. By leveraging the timeline capabilities, you can easily track when a resource was changed and the details of its configuration modifications.

‍
Here is an example of a timeline of a Cloudwatch Log Subscription Filter with a change to its destination ARN to a different environment. We can see when the change has occurred:

In the event that we want to revert the change to the resource,we can use the codification for this purpose by codifying it to the IaC language we use for our infrastructure by clicking “Codify Revision”.

There is no need to manually copy and paste the configuration to the IaC, as you can use Firefly to create a pull request to your Git repository.

Firefly is a one-stop-shop when it comes to asset history, including all your cloud and SaaS vendors. With just one click on the Asset History label, you can access all resources for which history is enabled. For example in the screenshot below we see resources from AWS and from the SaaS provider, Cloudflare with the full Asset History available.

Even if an asset has been deleted, Firefly has you covered. With the Asset History, you can view deleted assets and their previous configurations, just like any other active asset in your inventory so you don’t lose any critical configuration history due to an accidental deletion (or even intentional deletion without backing up the configuration).

Let’s Compare Again

Summary

After exploring the advantages and disadvantages of popular tools such as Git and AWS CloudTrail audit services, and Google Cloud’s Change History it is clear that they are critical tools in our cloud ops toolkit. However, with the growing number of clouds, tools, services and much more––many DevOps teams are opting for consolidation, and prefer to have access to information in a unified system and format. By providing a comprehensive resource history and the ability to restore and revert past configurations, with Firefly this is now possible cross-cloud and even SaaS application configurations.

‍---------

Firefly offers the benefits from all tools mentioned during this article by combining the traceability of audit tools with the reversibility of Git. With Firefly, you can have complete confidence that you won't miss any changes or events affecting your assets and peace of mind that you can revert changes.

So go ahead >> give it a try and see the difference it can make for your team.

Comments

[Sharone Zitzman]

Great post @tamirzheleznyak 💪🔥