🦄 Making great presentations more accessible.
This project aims to enhances multilingual accessibility and discoverability while maintaining the integrity of original content. Detailed transcriptions and keyframes preserve the nuances and technical insights that make each session compelling.
Overview
📖 AWS re:Invent 2025 - Observability & Security unite: Unify your data in Amazon CloudWatch (COP361)
In this video, AWS CloudWatch team presents new features that unify security and observability data to address data fragmentation challenges. Nikhil Kapoor and Avinav Jami demonstrate how organizations can eliminate duplicate log pipelines across security, observability, and compliance teams using CloudWatch as a unified store. Key announcements include support for 30 new AWS services, managed connectors for third-party sources like CrowdStrike and Okta, cross-account centralization with no additional ingestion costs for the first copy, S3 Tables integration for Apache Iceberg compatibility, and Facets for simplified log analysis. Chandra from S&P Global shares their implementation journey, achieving 20-25% cost reduction while meeting requirements across multiple stakeholder teams. The session includes a live demo showing how centralized logs enable faster troubleshooting across accounts and regions, and how different teams can access the same data through their preferred tools without complex ETL pipelines.
; This article is entirely auto-generated while preserving the original presentation content as much as possible. Please note that there may be typos or inaccuracies.
Main Part
The Challenge of Data Fragmentation Across Security and Observability Teams
Welcome everyone, thank you for joining us this afternoon. I hope everyone's having a great re:Invent so far. My name is Nikhil Kapoor. I'm a principal product manager with CloudWatch, and this session is COP 361. We're here to talk about some of the new launches that were announced this morning during the keynote in CloudWatch, specifically how we're helping unify security and observability data within AWS in a simplified manner. With me I have Avinav Jami, director of engineering with CloudWatch, as well as Chandra, who's director of cloud solution engineering with S&P Global, and he'll tell us his story as an early adopter of the solution.
Before we get started, let's recap and level set on what are some of the goals we have for ourselves to cover in this session. First, we'll do a brief recap of what are the common challenges we see that most large companies and enterprises face when it comes to the sheer volume of data they have to deal with, both when it comes to security analytics and observability. When we talk about sheer volume of data, it's mostly logs. We have a love and hate relationship with logs. We'll share some of those challenges that we hear from customers, then we'll talk about what enhancements we announced today to help with those challenges.
Finally, we'll try to leave you with some practical tips you can use, not just about how you can use the solution but also how you can think about your journey as an organization to streamline some of this data management and the challenges we see with data fragmentation. So, just a quick summary of our agenda: we'll go through challenges, we'll talk about the solution, we'll talk about S&P Global's journey with the solution, and finally a recap of takeaways and some helpful resources that you can use.
All right, challenges. A quick show of hands—and I can barely see—but how many of you have something similar in your organization where you have a security team, an observability team, maybe an audit compliance team? Good. I was expecting that, maybe not good once you go through the challenges, but let's walk through them. So typically this is what we see. For valid reasons, you have a dedicated security team whose job is to make sure your company, your customers, and their data is protected, and they're using a dedicated set of tools which are purpose-built for that solution, which makes sense.
Similarly, observability teams have their own set of tools and often more than one tool to get visibility into the operational health and performance of your systems. Finally, in some organizations there's a dedicated audit and compliance framework where all the data needs to land at one place, untouched, and they have their own set of tools. Now that's great. These are serving particular needs. These are specialists who know what they're doing, so they need to be able to do that particular job well. So nothing wrong with that.
But once we go down the stack, we'll see a trend here. Not only are the tools and the people specialized for it, but the underlying data often is tied to that tool, and the end result of that ends up being something like this. When there is a problem and it spans multiple domains, somebody in the security team needs to call up somebody in your DevOps team or SRE and say, "Hey, what happened? Why are you guys not fixing this?" And he or she wakes up at three in the morning and says, "What are you talking about? I don't know anything about it." So we run into this problem of manual communication slowing down insights and slowing down outcomes.
Now we peel a layer further. Why is that happening? Why are those individuals not able to see the same landscape, understand the same problems faster and easier? Often it goes back to the tools. The tools are giving them insights which are siloed to what specific thing that tool was designed to do or set up to do. The humans reacting to the tool can only react to what they see. Then go one step further. Why are the tools not able to get the same insights?
Let's take an example. The security team is suddenly seeing a whole bunch of denials through their firewalls or what have you. At the same time, your application team sees a whole bunch of traffic spike and doesn't know what to do. The security team knows there may be a DDoS attack. The application team doesn't even know—they're trying to scale up their infrastructure thinking they have a whole bunch of customers. Wouldn't it be nice if they could all see that there's a security DDoS attempt going on instead of trying to scale up infrastructure? They need to isolate that particular IP subnet or whatever that is. So that's going back to the people and the tools they are using not working across a common set of information.
The last bit of this layer is actually the most interesting.
When we looked into this, I asked how this data source works, and because the underlying data is largely the same, if you notice, data sources are largely the same. You have your AWS service logs, you have CloudTrail, you have your application logs, you have some third-party tools like CrowdStrike and Okta that you use. All these log sources are largely the same. It's the same data being sent into different places. If anybody here is a developer who manages these data pipelines, I empathize with you. When we looked at several of our customers, they said we have the same logs. Cloud logs need to go here, also need to go there. Often this data is copied from one place to another, and all these ETL pipeline spider webs of information just going from one place to another and back and forth. It is a nightmare to manage, and every time you need to onboard a new workload or a new region, it's the same thing. It just creates complexity.
That's what we are here to discuss. These are the high-level challenges we need to address. How do we get comprehensive insights faster to your teams? How do we increase their ability to detect and react to problems, like the DDoS attack example I mentioned? How do we reduce the operational overhead of this spider web of data pipelines? How quickly can we enable new workloads? And finally, maybe not the least important, is cost. All of this data duplication across three different stores, five different stores, the same data, not to mention the same VPC Flow Logs, the same CloudWatch logs being replicated in several different places. That's the goal we set for ourselves when we started on this journey.
Introducing CloudWatch as a Unified Store: Collection, Curation, Storage, and Analytics
So how can we help? To walk you through this, I'll invite Avinav on stage. We'll walk you through the solution and tell you how we can help. Thank you, Nick. So, especially on the previous slide when we talked about operational overhead, that's been something we have experienced firsthand. Managing your own ETL pipes, loading data, managing versions, and every time the data load gets stuck, figuring out how to do the backfill adds into all the other aspects as well, like introducing more cost. We started by rethinking how we would figure out a solution for our customers that simplifies this.
We thought about how to blur the lines between observability, security, and the audit space. Blurring the lines here doesn't really mean using a single tool for all of your different personas, nor does it mean having the same set of engineers operate on different business problems. But we thought harder about where the problems really lived. We realized the problems actually lived in that underlying data store layer, which is where a lot of the infrastructure heavy lifting happens when you try to create data stores that are comprehensive, that are consistent, that have all the underlying log data and telemetry data until you can get to the point of using the tool. We thought about blurring the lines in that data store space and help you create a unified store and come up with ways in which we can build flexibility into CloudWatch with new features and new capabilities so that you can build that unified store.
Taking one step further, if you want to build that unified store, you need to start thinking about what are all the steps in order to do that. The first step here is to start collecting all the data. The collection phase usually involves bringing your logs from your AWS services, from your applications, from all your third-party sources into a single spot. This is not just about getting the data in the right way, it is also about making sure it consistently arrives in the right way. Next is curation. In your curation phase, you're trying to shape the data to the analytics needs, which means you're thinking about what fields are really valuable, what kind of dashboards will you end up putting, what kind of queries will you end up doing, what kind of information is needed to make that effective?
How can you enrich your logs? How can you make sure the right fields are present in the right form and the transformations that need to happen for it? It also means creating indexes, which help you build fast dashboards or speed up that query that quickly gets you the right information. All of this has to be backed by a flexible storage layer. Storage is where we usually see a lot of the costs happening for our customers. There are different data sets that customers want to retain for longer periods of time, especially for security purposes, and there are some data sets that are more operational in nature. You want to be able to transform the two data sets differently while at the same time keeping essentially a single copy. Lastly, you want to have powerful analytics and insights off of that store. You want to be able to quickly look at your real-time logs and also do deep dives that take multiple query runs.
So here we introduce CloudWatch as a unified store with a whole bunch of features that make CloudWatch a single store for security and observability data. In the first step of collection, CloudWatch today is already available across 65+ services. We are introducing 30 new services so that logs will now be accessible inside CloudWatch. These 30 services include services like Network Load Balancers, CloudFront, and Petrock Agent core logs, among others. We are also introducing third-party logs via a managed connector approach, which means you don't have to write any code to get your logs from your third-party sources. At launch we have 10 connectors, which include CrowdStrike and Okta, and we will continue adding more third-party connectors as the year progresses.
Customers also ask us for security and governance controls at the organizational level, meaning being able to turn on logs for your CloudTrail or your VPC at the organization level. This year we are launching organizational-level enablement support for CloudTrail, VPC, NLB, and so on. In the curation phase, customers often have to shape their data in the way that analytics tools will leverage them. We are introducing out-of-the-box transformers for OCSF and OEL data. Customers can also use our pipelines to shape the data as they need. They can use Grok processors for custom parsing, for field-level operations, and for string manipulation.
We are going to add capabilities for identifying source and type information that we are already aware of and pass that down through our logs. This means source and type information will now be available inside your vendor telemetry, and you can also append that to your custom logs. You can use tags and add source and type information so that you can now operate at a level which is more at the application level instead of thinking about resources, log groups, and so on. At the storage level, the flexibility that our customers most desire is to help control how long the data will stay, the way the data is structured, and whether it will be present in different places.
Many customers, especially in the observability space, want to keep data in their observability accounts, but when it comes to security or for central ops, they want to keep data in a single space. We are introducing a new cross-account, cross-region centralization feature, which brings your log telemetry into a single spot with the help of rules that you can maintain. You can also create separate retention policies. You can transform the logs as they come in. All the enrichment features that I talked about can apply to the centralized store differently from the source stores. So you can create versions of the data that are best suited for your security or your observability needs.
Lastly, any unified data store has to offer a bunch of powerful analytics. CloudWatch today already supports strong features like Insights, live tailing, and top-end analysis. Now we are introducing two more capabilities that are very exciting. The first one is being able to create S3 Tables, essentially connecting you with any analytics engine of your choice that you can now use. The second one is Facets. Facets is the next generation of our indexing feature, where you get essentially an out-of-the-box experience for different fields when you go into the Insights on the query explorer page.
This lets you start that first deep dive when you are not sure where to start a query. You go to your Insights page and you see a field to write a query. But after you enable Facets, you can then go in and see field values. Facets that are at the error level can now help you figure out where exactly your errors are happening. Then maybe you deep dive into it further and find out that you need to look at your service-level facets next. As you click through each of those facets, the page keeps refreshing without you having to write any query. Within our experience, we found that a lot of our initial deep diving is now easier because there is a place to start.
By the time you need to write more complex queries, you can use features like query gen to quickly give it prompts and come up with a new query. So to quickly put together the list of features, we thought about our CloudWatch store. We reimagined it with the opportunity to make it much more flexible and compatible with the needs of the user.
Rather than thinking about it as a data silo, we went through the collection phase and added a bunch of features, including organization-wide enablement and additional support for news sources. We made pipelines a feature available for transforming and enriching your logs. We operated at the storage level, creating a centralized store and giving you open access through S3 tables. Finally, we provided powerful insights that you can now connect to the S3 table. At this point, I'd like to bring in Chandra, who will talk through how his team has been using CloudWatch.
S&P Global's Journey: Simplifying Log Architecture and Reducing Costs by 25%
Thank you. Hi, everyone. Thank you for joining us for this 4 p.m. session. I'm really surprised we have a full house. Let me start with a quick question. How many of you here know about S&P Global and what we do? Just one person? Okay. So you rank all those stocks and all those companies' portfolios, and then you rank them as they come in and go out. Good, very close. How many of you know about the S&P 500 index? Almost everybody, right? That's expected because directly or indirectly, most of us here in this room are getting the benefit of the S&P 500 index.
My name is Chandra, and I lead cloud solution engineering from the corporate division at S&P Global. I'm here to share some of the journeys we took a few months back regarding our logs integration journey. I hope it will help some of you want to take the same path. S&P Global is a global company, formerly known as McGraw Hill Financial. Founded back in 1800, we are a leading provider of benchmarks, data, and solutions for financial markets across the world. We have a strong brand trusted for over 100 years. At this time, we have around 40,000 employees serving our clients globally. We have 50 offices worldwide and 10 data centers. These 10 data centers are mainly used for network gears and connectivity.
Moving on to cloud scale, when you're talking about logging, we need to understand the size of S&P Global quickly across the globe. We have strategic AWS regions to serve our clients within that proximity. This adds up to a large number of accounts, extensive VPCs and cloud resources. This brings substantial volumes of log data, which is the focus of our discussion and the journey we took.
When we were working on revising our existing log pipeline, our stakeholders had a lot of requirements. One requirement is raw logs in a dedicated log archive account, sometimes called a central account. This comes from our legal and compliance team because they want to see unaltered, immutable logs for legal purposes. Next, we need curated logs in a separate, dedicated security tooling account. This comes from our cloud security and information security team. They need visibility on cloud security, what logs we're collecting, and who is accessing what. Another requirement is access logs locally. We frequently hear this from the business units. While we're taking logs to a central location, we also need to have logs locally so we can search and access logs for troubleshooting purposes.
Then we have the Security Operations Center team. They want to have logs especially curated for threat intelligence and incident response. Then, of course, log querying. This is everybody's need, right? They should be able to easily query and get log access when they need it. And then finally SIEM, Security Information and Event Management optimization. This is very crucial because when we have so much of cloud accounts and cloud resources, the data gets massive. At S&P Global, we are actually using a third-party SIEM. We cannot ingest everything into the SIEM because SIEM costs will spike up. So we have to curate and ingest what we need, the logs that we need.
All of these requirements add up to the complexity in the design. This is the design we worked on before we knew the AWS team was actually working on new requirements. At that time, when we were looking at the design, we thought to ask the AWS team if they were working on any new capabilities for CloudWatch. Then we realized they were working on it, and we were so excited that we partnered with the AWS team. They are always available. You can pick up the phone and call them. So they came in and partnered with us, and they actually helped us revise this architecture and reduce the complexity.
When I say complexity, if you look at it here , we have multiple CloudWatch subscriptions. We had to use Firehose to take logs from the divisional account to the central account. We also needed to send logs to the Security account and multiple other accounts. This is the security tooling account, this is the central account, and we had this complexity. Another complexity in the design was that we had many S3 buckets to manage. When we partnered with the AWS team, they said we could actually improve this. This is how it looks now . It is much simpler with fewer components and is CloudWatch native. The interesting part is that the plumbing is taken care of and it comes free of cost.
So what is the result of this journey? We are able to simplify and reduce the cost while simplifying the design. Plus, we are able to achieve a federated model of log storage, curation, and processing. This basically gives us scalability, and we also achieve all our stakeholders' requirements . And then most importantly, cost. As I said, we are expecting at least a 20 to 25 percent cost reduction from the design we have today.
What we learned throughout this journey of our planning and development phase are some valuable lessons. The first one is aggregate with a plan. Basically, do not just collect logs. Have a clear strategy for storage, curation, and access. Aggregate logs thoroughly and keep the focus as simple as possible. Collect and aggregate according to your use cases. That is very important. Do this in dedicated accounts to ensure security and meet your security requirements.
The second lesson is meet your teams where they are. This is where the partnership comes into the picture because there are many requirements. Everybody wants log data their own way. Meet your teams where they are and adapt solutions to existing workflows. Avoid forcing a one-size-fits-all approach. Work with the stakeholders, understand what their needs are, and then build that into your approach. The third lesson is iterative , and I think most projects we run are agile and iterative. What we learned is that we have to start small, learn, and scale. This approach reduces the risk and improves adoption. However, ensure that you have enough logs for your teams.
Deployment Models and Cost Structure: Making Centralization Affordable
So some of the things you'll see in the design, and I'll talk about the deployment models, came from how we saw S&P was trying to adopt and what things we should factor in—not just what customers and the state would be, but what's the incremental journey. So with that, let me talk about some of the deployment models that we have seen so far, other customers that we've been speaking to adopting, and how it may help you get started in your journey.
Before we go to the deployment models, let's talk about costs. I'm sure some of you are wondering about all these managed pipelines, all this replication, centralization—how much is it going to cost? So I broke this down into some of the core features I mentioned. Facets are the out-of-the-box distribution of values available to you without even running a query. Not just expedites your analysis but saves you all this data scan you're doing when you don't even know where you should be looking. They're available free of charge, no additional charge in our standard class.
For context for folks not familiar with CloudWatch, CloudWatch has three high-level pricing dimensions, which we call base pricing dimensions: ingestion, storage, and queries or data scan. There are no changes to those costs. Whatever the price was, we have not increased anything there. It stays as is. In addition, in ingestion we have standard class and infrequent access class, and the difference between them really is what capabilities or processing time capabilities are available in them. Standard class is built more for those real-time operational troubleshooting where you can have indexes, you can do more fast-type real-time query experience, and you can have alarms and all of that.
Infrequent access is more for the logs that the compliance use case, for example. You'd need them to query, but you don't need that real-time analysis at two in the morning. So we try to give flexibility by offering infrequent access at half the cost of standard class. It just gives you the flexibility of using which class is best suited for those logs. So I'll talk about both, but I wanted to set that context first.
Facets are included free of charge for standard class. For infrequent access, because they're not aligned with that use case as I mentioned, they're not supported in infrequent access. S3 Tables integration is an interesting one. As we were thinking about this outcome, we wanted end users to not have to give up on their preferred tool just to use this data store. That was one of our goals—it's not like if you take away that, tell your security teams, "We'll create a unified store, but you have to give up what you're using." It won't work.
So instead, the approach we took was we partnered with the S3 Tables team to have native Apache Iceberg support. So any query tool, whether it's AWS analytics tools like Athena, Redshift, EMR, or any other Iceberg-compatible tool, can query this data in place. S3 Tables integration, including replication, storage, and table maintenance, has no additional charges. It's included in your CloudWatch pricing. You only pay for the queries you run on S3 Tables. The reads get the typical query cost with no additional cost for that as well. It is supported in infrequent access with the same model as standard, no additional cost.
Log centralization was launched about a month or so ago. You may be wondering, "I have all these great things about local teams and divisional teams still having access to their data. Now I centralize in one place, and I centralize again for security in another place. That's all this ingestion and duplication. How is anyone going to pay for that?" So from an ingestion and replication standpoint, there's no additional cost for the first copy. The first aggregated copy is free. The second copy you only pay five cents, which is basically the data transfer cost with no additional ingestion cost.
Keep in mind you have full flexibility of re-ingesting and reprocessing those logs in those central accounts as well, but there's no additional cost for the first copy and five cents for the second copy. Storage charges still apply in addition because at that point you have replicated data and you're managing retention independently. This is an important one to note. So when you centralize, you don't want to have five years of retention across three copies. Pick what your central archival account is, like Chandra mentioned. If that's where you want to have long-term storage, that's where you want to keep the real retention. In your local accounts for developers, testing, and Lambda, they don't need seven years of logs. They just need live tail to work for a couple of days, right? So lower the retention in local accounts.
Pipelines for transformation and enrichment are included at no additional cost. Organization-level log enablement has no additional cost. You just pay for the same log ingestion and storage that ends up, which was already included in CloudWatch pricing. This is also supported in infrequent access. I'm trying to paint a picture of what the comments I was making about free of cost are. This is where all these features we wanted to build in a way that customers can easily try out, see, and iterate without having to think so much about, "Oh, I enabled this one thing, what's going to happen?" We don't want to do that. So that's the cost.
Two Approaches to Implementation: Distributed Centralization vs. Centralize and Distribute
Now that we have that out of the way, how can we get started? I'll talk about a couple of deployment models that we have seen customers use. One is what I'm calling distributed centralization. This is the common example. At the bottom, you see all your log sources that you already have going into CloudWatch, whether it's individual accounts, for example, VPC flow logs across 500 accounts going to those accounts because those teams need them too. The same thing applies with CloudTrail—maybe your local teams also want CloudTrail logs and your central team wants them.
So you enable all those logs. They are available in local accounts. In addition, they get replicated in a fully managed way behind the scenes to an observability account. Whichever logs you want, you can pick and choose to also go to a security account so you can decide that these logs are only relevant for security and only go there. Now at this point, three teams are using those logs in the way they need without compromise. They can transform and enrich independently without impeding each other's workflows and being able to use the tool of their choice, whether it's CloudWatch, Athena, Redshift, some other tool, or Apache Iceberg compatible tools. They can power their workflows without building all of those complex plumbing that they were having to do.
That's one model. The second model, which is similar to what S&P Global has started with, is centralize and distribute. This is the model where you bring everything into one central account, which serves as your compliance single source of truth, but also serves as a single place for you to fork what data needs to go to what other place. Instead of having 200 accounts plumbing this here and plumbing that there, it's one place. And across regions, by the way, not just across accounts. If you operate across 23 different regions for disaster recovery or what have you, you can do that all in one place. Centralization offers a backup region as well.
When you centralize across regions, you have the option to create an active-active backup. So in case something happens to one region, you still have access to your data. In this centralized and distribute model, everything is in one place and then you decide where to send data. If you want to send some logs to the same solution, you can filter what is needed and don't send what is not needed. These are the two different deployment models that we have seen customers use.
Live Demo: From Payment Failures to Root Cause Using Facets and S3 Tables Integration
With that, I think we should do a brief demo at this point. We've covered a lot of slides, so let's switch to the demo. In my setup, I have a demo that's largely trying to represent what I would typically expect most production-grade applications to have. I have a PayStream payments company, if you will, who's processing payments for a large variety of their clients. I am operating across multiple accounts and regions to be closer to my end customers. I ran a payment and I'm seeing some errors, so some payments are failing.
Now, what would I do in this state? If I get a call that payments are failing as a developer, SRE, or support person, the first question I would have at that point is: where do I start? So I go to CloudWatch and I land on this page, the Logs Insights page. I could start by trying to map which region and account to go to and open 15 different tabs to get to that outcome. Or what I'm doing in this case is using the facets that we were showing. Let me zoom in a little so you can see. When I log into this facet, without even running a query, I can do two things. First, I can see if there are any errors happening. I see there are some notification failures, some payment processing issues, and some validation failures. I definitely see some errors are happening here.
So if I now go to which service should I troubleshoot, I'm going to select payment process as the service. I'm going to narrow down before I look at all the different events. As I selected it, without running a query, it updated this other thing that I was interested in live for me. I see there are definitely errors. This is an example of what otherwise I would have done with two or three different queries just to confirm what I just showed you here. Just by collecting this, I can see that this is the service I'm interested in. In real-time, it showed me that in the last hour, we had 690 payments failed. Well, that's good. So I confirmed there were payment failures.
I can now validate where the failures are happening. What I'm going to do now is run a query, but now I have zoomed in my scope. I don't know how to run a query across everything in my logs. I'm just going to try to understand what could be going on. I have a pre-built query here. I'm going to select the facets to run this query.
I'll select the Payment process service and the event type I was interested in. I want to see where the successes are happening too , so let me select both success as well as validation failed. All right, let's run this query and see what kind of results we get. I'm going to make this bigger. All right, so I can see from this that I have different accounts across different regions on the left across my entire real estate. By having logs centralized in one place, not only was I able to analyze them in one place without opening tabs, but I was quickly able to isolate all the errors, this invalid payment failure reason, and I isolated this account in this region. Now I can jump to my next query to understand why this could happen.
What I'll do now is query my CloudTrail logs . I'll select CloudTrail here. The other thing we introduced, which Avina mentioned, is data source. So now instead of thinking about log groups or resources, I know I'm interested in looking at my CloudTrail logs . I'm going to just say CloudTrail and select CloudTrail. I don't need to think about where they are. AWS has already mapped it for me. I'll run this query, and it'll show me what happened to this Lambda function. I'm sure somebody made a change that they didn't intend to. Let me see what happened. Oh, somebody did make a change. They changed some parameter from 128, and look at that, somebody named Cole Pritt. So that's the example of simplified analytics. If you notice, the query capabilities are not that dramatically different outside of facets, but by bringing the data and curating the data the way we were able to in one place, it opens up all these faster and better insights.
All right, so that was the inside part in CloudWatch. What about this other tool thing we mentioned? I'm going to switch to SageMaker Unified Studio. How many of you are familiar with SageMaker Unified Studio? A few people. So I'll walk you through it. SageMaker Unified Studio is AWS's native analytics tool. It gives you the ability to run advanced analytics and data in Jupyter notebooks and Athena queries, and with S3 Tables, they are natively integrated in the catalog. So when I go to my SageMaker Unified Studio, I can see my custom catalog as well as any S3 table catalog.
By having the S3 Tables integration set up in CloudWatch, all the CloudWatch logs that I have associated with this show up under this managed S3 bucket, AWS CloudWatch, and all the log sources I've associated are already here. So going back to the scenario of my demo, which I was showing the payment processing, I have figured out that somebody made a change in Lambda that broke the problem. At the same time, my product team, people like myself, are asking my engineers to say, hey, we need to know which customers are impacted so we can reach out to them and figure out the impact. We need it now. While I'm trying to do all of this fixing the actual problem, I'm also having to deal with all these people who are trying to do their job.
But in the past, what I had to do is export data live and give it to them so they're unblocked and can do their thing. Now I don't have to. I just say, hey, the logs are available in SageMaker. Go find which customers are impacted, and they can analyze and connect with their business data in other S3 tables or S3 buckets in Athena or Redshift and find that customer usage pattern or who's impacted themselves. So different personas, same log, getting the insights.
All right, one last thing I wanted to show in the demo. One other thing we did from a simplification of management at scale. When we're operating in a single account, log groups and resources make sense, but when you think about operating across several hundred accounts over several hundred regions, it can become cumbersome. You need a larger view, a higher level view of your data. So in CloudWatch now we have introduced this new log management experience where in addition to the log groups that folks are already used to and familiar with, you get a high level summary of all of your data so you can see what was ingested, where, and who's taking up space. If you suddenly have a spike in something, you can quickly identify it here. You can see what the most recent run queries were, what configurations are set up, and what data protection policies are enabled or not enabled. So it gives you a top level view of what things are working or not working just from a pure data management standpoint, natively in CloudWatch. Now from analyzing different data sources,
the frequency with which we've mentioned data sources throughout this presentation should tell you how important the connotation of data sources is. We all operate with sources like CrowdStrike data, VPC Flow Logs, or CloudTrail logs, so we wanted to make that sticky and easy for you to identify them. We have added this data source-based experience where most of the AWS sources are automatically mapped for you. You don't have to do anything if you have them enabled. When you land on this page, you'll see them already there. Not only that, they are also schematized by default when schema is available. When you're trying to analyze something, you'll be able to see what fields are available or not available in place.
In this case, Bedrock runtime is by default sending me these four fields. I don't have to think about what they are; they're available. This is important because in the Iceberg experience, we want to make those columns and fields as accessible as possible. We built this whole schema detection experience on top of that. From here, if I go back, you can see that S3 Tables integration is active. Now if I go back, I can enable this S3 Tables integration for one data source if I want to be selective about what I expose to that product manager who I don't know what they want to do. Alternatively, if I want to have a use case where I want to expose all, I can also manage an integration and expose all of those data sources in a multi-select manner.
Key Takeaways: Building Block Approach to Eliminating Data Fragmentation
This is a one-time operation. Once you enable this integration, any future data for those data sources will automatically be mirrored and available for you in the tool of your choice. That covers the demo part. Now let's go back and do a quick recap of everything we talked about. Here are the key takeaways. First, hopefully we all agree that data fragmentation is a problem. Data fragmentation is complex but important for us to solve. We discussed what kinds of challenges and problems data fragmentation can create within your company, within accounts and regions in AWS, and within use cases. There are different levels of data fragmentation, and our goal is to help you eliminate as much as we can.
Second, and this is an important one again, meeting your teams and users where they are is an important part of this journey for our companies and teams to be successful. Uprooting them from what they're used to and their workflows is disruptive, so we wanted to minimize that as much as we possibly can while still trying to limit the duplication and fragmentation of data. Finally, this is why we wanted to talk about both S&P Global's journey as well as some other deployment models. We've built this feature or set of features in a building block manner rather than just one "enable unified store and it works" approach because we realized as we were talking to customers that different customers want to consume different things in different ways.
Rather than giving a very opinionated one-size-fits-all solution, we have given you the building blocks and we encourage you to evaluate which features and capabilities meet your needs at what stage. Maybe today you only need centralization of logs. You don't need pipelines, curation, or third-party sources. That's okay. That's a good start. Six months later you may realize that you can now bring in additional data sources from third parties that fit your need. You can start there. The idea is to identify the right starting point where you are and where you want to get to, and iteratively get there.
That was a lot of talking. There are a couple of QR codes here that may help. One is the news blog which talks about a lot of the features and how you can get started, with various links to other resources available to get started on all these features. On the right we have a broader AWS observability resource guide. There are all these resources about AWS observability, many things we didn't even get to cover here available to you. I encourage you to look into those. With that, thank you so much for coming. We appreciate your time and I hope you found this useful. Please give us feedback; we appreciate the feedback. It helps us learn and improve. Thank you.
; This article is entirely auto-generated using Amazon Bedrock.
Top comments (0)