DEV Community: Kubecost

Leverage Grafana Cloud with Kubecost

Kubecost — Tue, 26 Apr 2022 19:33:11 +0000

We’ve rolled out a Kubecost integration for teams using Grafana Cloud!

Kubecost users can take advantage of the integration to get the best of all worlds—managed observability services, working in concert with cloud cost optimization.

What is Grafana Cloud?

Grafana Cloud is a composable observability platform, integrating metrics, traces, and logs with Grafana. Users get the benefit of the best open source observability software, without the overhead of installing, maintaining, and scaling their observability stack.

As a part of its monitoring services, Grafana Cloud offers a managed backend that can store Prometheus metrics. This hosted service allows you to aggregate and store metrics from multiple Prometheus instances into a single, dedicated space. This centralized location makes it easier to query your data, while also providing long term storage for historical analysis and capacity planning.

Kubecost + Grafana Cloud = 💯

If you wish to reduce the overhead of managing and maintaining your observability stack and keep better track of your cloud spend — Grafana Cloud and Kubecost are here to help!

Kubecost automatically provides cost visibility, savings and optimization recommendations, and ongoing governance for deployments in any Kubernetes environment. Teams can successfully reduce the operational burden of managing multiple cost views and manually tracking spend for each of their services. With the help of Kubecost, teams are even able to track and allocate granular spend on their own.

Customers can now use Kubecost to track cloud spend leveraging their Grafana Cloud stack by using a Custom Prometheus integration. To compliment the integration, we’ve published a Kubecost Grafana Cloud Community Dashboard so that customers can visualize their cost data directly in their hosted Grafana.

Get Started

In this section, we’ll walk through a quick example of how you can use Kubecost and Grafana Cloud in sync. You’ll need a running Kubernetes cluster and a Grafana Cloud account.

Step 0: Install the Grafana Agent in your cluster

Using your Grafana Cloud account credentials, , install the Grafana Agent for Kubernetes on your cluster as a prerequisite for the following steps.

Step 1: Configure Kubecost scraping configuration for the Grafana Agent

Once you’ve set up the Grafana Agent, we’ll need to add some extra configuration to the way Grafana Cloud scrapes metrics, so that Kubecost can offer more accurate cost estimates.

Create a file called extra_scrape_configs.yaml with the following contents, replacing the grafana_prometheus_remoteWrite_url, username and password placeholders to match your Grafana Cloud details, which you’ll find by visiting your organization’s Grafana Cloud Portal -> Prometheus -> Password/API key:

kind: ConfigMap

metadata:

name: grafana-agent

apiVersion: v1

data:

agent.yaml: |

server:

http_listen_port: 12345

metrics:

wal_directory: /tmp/grafana-agent-wal

global:

    scrape_interval: 60s

    external_labels:

    cluster: cloud

configs:

- name: integrations

    remote_write:

    - url: <grafana_prometheus_remoteWrite_url>

    basic_auth:

        username: # Grafana Cloud username

        password: # Grafana Cloud API key password

    scrape_configs:

    - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

    job_name: integrations/kubernetes/cadvisor

    kubernetes_sd_configs:

        - role: node

    metric_relabel_configs:

        - source_labels: [__name__]`

Next, apply the changes in the same namespace as your Grafana Agent deployment:

$ kubectl apply extra_scrape_configs.yaml -n <namespace>

Re-start the Grafana Agent:

$ kubectl rollout restart deployment/grafana-agent -n <namespace>

Step 2: Configure Kubecost to query metrics from Grafana Cloud Prometheus

If you haven’t yet installed Kubecost, use the commands below, or check out our installation guide to get set up. Our standard helm install only takes ~2 minutes:

`$ helm repo add kubecost https://kubecost.github.io/cost-analyzer/

$ helm upgrade -i --create-namespace kubecost kubecost/cost-analyzer --namespace kubecost`

If you already have a Kubecost deployment on your cluster, hooray! Now, we'll set up some basic auth credentials so that Kubecost can query data from Grafana Cloud.

Grab your Grafana Cloud username and API key from Step 1, and create two files in your working directory, called USERNAME and PASSWORD respectively. Then, generate a Kubernetes secret called dbsecret in the same namespace as Kubecost is installed. The namespace is typically kubecost.

$ kubectl create secret generic dbsecret -namespace kubecost --from-file=USERNAME --from-file=PASSWORD

Reload Kubecost, using the secret you’ve just created, and the Prometheus query URL that you can get from your organization’s Grafana Cloud Console -> Prometheus -> Query Endpoint:

$ helm upgrade kubecost kubecost/cost-analyzer --namespace kubecost --set global.prometheus.fqdn=<grafana_prometheus_query_url> --set global.prometheus.enabled=false --set global.prometheus.queryServiceBasicAuthSecretName=dbsecret

That’s it! By now, you should have successfully completed the Kubecost integration with Grafana Cloud. You can view the Kubecost UI in your browser by port-forwarding to http://localhost:9090/ on your machine:

$ kubectl port-forward --namespace kubecost deployment/kubecost-cost-analyzer 9090

From there, you can start by exploring cost allocation trends by clicking the Allocation tab, or by discovering quick savings and cost optimization insights via the Savings tab.

We’ve compiled a list of Getting Started guides to help you take advantage of all of the Kubecost features.

Optionally, you can also add our Kubecost Community Dashboard to your Grafana Cloud organization to visualize your cloud costs in Grafana.

Step 3 (optional): Configure Kubecost recording rules in Grafana Cloud

For even richer Kubecost data, consider adding Prometheus recording rules to Grafana Cloud. While they are optional, they may improve cost accuracy.

We’re here to help!

For more information and troubleshooting, check out the Kubecost documentation and Grafana Cloud documentation.

Join us on Slack for any other help, and general Kubernetes and Cloud Cost optimization banter!

Easily deploy Kubecost on AWS using Amazon EKS Blueprints

Kubecost — Wed, 20 Apr 2022 14:07:53 +0000

If you've ever dabbled with Infrastructure as Code, or it's something you've been interested in doing, we think you're going to like this one... 🥁

Kubecost can now be deployed and configured on Amazon EKS using AWS Cloud Development Kit (CDK) as an EKS Blueprint!

What are Amazon EKS Blueprints?

EKS Blueprints are a way to abstract cloud infrastructure complexity away from software developers. Blueprints allow developers to deploy containerized workloads using tools and languages that they are already familiar with. With the help of EKS Blueprints, teams can consolidate tools and best practices into a single central platform that can then be used across their organization.

A Blueprint is typically composed of multiple AWS or open source products and services, including services for running containers, CI/CD pipelines, capturing logs/metrics, and security enforcement. Blueprints can help package these tools into a cohesive whole and make them available to development teams as a service.

How Kubecost helps

Given the sheer scale and complexity of Kubernetes environments, tracking cloud spend can quickly become problematic. For any organization running services on Kubernetes, having a way to continuously monitor and optimize costs is an important best practice to introduce and automate early on.

Kubecost automatically provides cost visibility, savings optimizations, and ongoing governance for EKS deployments, and presents a unified view of these costs along with those of all other Kubernetes infrastructure. With Kubecost, users reduce the operational burden of managing multiple cost views and manually tracking spend for each of their services.

The Kubecost EKS Blueprint AddOn allows teams to automatically provision and maintain Kubecost across multiple clusters, making it part of the internal development platform for the enterprise. The AddOn can be deployed and configured as part of your EKS Blueprint using AWS CDK (Cloud Development Kit), with just a few lines of TypeScript or JavaScript code!

About the Kubecost EKS Blueprint AddOn

You can use the EKS Blueprint Kubecost AddOn to:

Install Kubecost, along with its dependencies Prometheus, Grafana, and kube-state-metrics in the kubecost namespace from the official Kubecost Helm chart.
Configure a specific Kubecost version.
Set custom values for the values.yaml file.

All using CDK and TypeScript or JavaScript.

How to use the AddOn

Create a CDK project

To get started, you’ll first need to install appropriate versions of AWS CLI, AWS CDK, and then bootstrap a CDK project. Follow the EKS Blueprint Quick Start documentation to go through those steps.

Once you’ve bootstrapped the project, go ahead and install the eks-blueprints library, which will allow you to use all the EKS Blueprint magic.

$ npm install @aws-quickstart/eks-blueprints

Install Kubecost EKS Blueprints AddOn

Install the Kubecost library in your project’s working directory:

$ npm install @kubecost/kubecost-eks-blueprints-addon

Write some TypeScript code 👩‍💻

You'll notice that running the bootstrap command created some files and folders within your working directory. We'll create a blueprint with the Kubecost AddOn, and then apply it to the cluster.

Note that id is the cluster id. You have two options: either use an existing cluster ID in that region, which will install Kubecost on an existing cluster. Alternatively, give it a new id, which will provision a brand new cluster under the associated AWS account in that region.

Replace the contents of bin/.ts (where your-main-file by default is the name of the root project directory) with the following:

import "source-map-support/register";
import * as cdk from "aws-cdk-lib";
import * as blueprints from "@aws-quickstart/eks-blueprints";
import { KubecostAddOn } from "@kubecost/kubecost-eks-blueprints-addon";
const app = new cdk.App();
const addOn = new KubecostAddOn();
const blueprint = blueprints.EksBlueprint.builder()
.addOns(addOn)
.build(app, "my-stack-name");

Deploy and apply the blueprint using the following command:

$ cdk deploy

Give it some time... and there we are! Once you’ve deployed Kubecost this way, you’ll be able to view the dashboards using kubectl port-forward command.

$ kubectl port-forward --namespace kubecost deployment/cost-analyzer 9090

Once the Blueprint is set up, it can be deployed in the same configuration across multiple regions and accounts, making sure that all of your clusters are uniform and consistently running Kubecost. The recommended approach for day-two operations is Git-driven pipelines.

Try it yourself!

Ready to give Kubecost and EKS Blueprints a try? Checkout the full Blueprint documentation and Kubecost AddOn documentation to get started.

Kubecost: bringing open source to cloud cost management

Kubecost — Wed, 23 Feb 2022 16:27:00 +0000

Author: Webb Brown

Last week we announced our $25 million fundraise, so I wanted to take this opportunity to share our origin story, and look ahead to where we are going as a result of this milestone.

Biking across Google’s Mountain View campus, I can still remember how excited I was to hear more about this new project my former teammate was working on: it was called Kubernetes. The project had launched in production the year before, and while I’d read enough to have a sense of its goals, I still hadn’t heard the full story until lunch that day.

Over lunch, I heard all about how the incredible team (a lot of them having worked on Borg) had the grand vision for bringing container orchestration at global scale to any cloud provider or data center. They were planning to do this with an entirely open source framework and a neutral governance model that placed power, control, and flexibility directly in the hands of the community. At the time, I didn’t realize how much lunch that day would shape the next decade of my life—ultimately inspiring me to leave Google, start a company, and raise $25 million with a goal of helping millions of developers one day.

Cost, performance, and health: the vision behind Kubecost
Soon after this experience, my soon-to-be cofounder Ajay Tripathy and I began noticing how early adopters of Kubernetes were struggling with the same challenges we were focused on at Google: balancing cost, performance, and health. The decentralized and dynamic nature of container orchestration was making it harder for Kubernetes users to understand and monitor infrastructure costs alongside performance and reliability metrics—and these same complexities made it significantly harder to optimize across these three areas. Specifically, finding the right amount and type of infrastructure that doesn’t cause major overspending (due to over-provisioning), but also doesn’t create real performance or reliability risks for applications (due to under-provisioning).

Since engineers can’t resist challenging optimization problems, we started brainstorming ways to help teams adopting Kubernetes navigate these complexities. We had used a number of existing third-party solutions, but found those to be focused on managing VM-based spend whereas the challenges Kubernetes users were facing called for new solutions developed specifically for their needs.

Our vision to build something different became clear as we explored different ways to tackle this problem. Our experience at Google taught us that by developing solutions specifically for Kubernetes users (i.e. the engineers working directly with infrastructure and containerized applications), we could empower engineers themselves to directly optimize compute spend—giving them visibility into usage, performance risk, and costs, to ultimately improve their organization’s efficiency.

From the beginning, we knew we wanted to align with the founding principles of Kubernetes: we had to start with an open source project that would empower developers with data and give users ownership over the product’s evolution. This belief led us to build the first open source Kubecost over the course of a weekend. We weren’t sure where it would take us, but we wanted to validate our understanding of the problem and, frankly, needed a way to monitor our own Kubernetes clusters. After what happened next, we quickly realized we needed to start a company to have maximum impact with the project.

Launching Kubecost
Within days of our initial launch in 2019, 100+ teams were using Kubecost to gain visibility into their cost and cost efficiency. Thrilled to have an early community of engineers focused on Kubernetes infrastructure optimization, we started talking to users—and quickly realized that even Kubernetes users that were early in their adoption journey were struggling with these challenges in a meaningful way. Using their insights and feedback, we initially prioritized providing users with great visibility, a choice that was rooted in our belief that you can’t optimize what you can’t measure.

We built our product experience based on 4 strongly held beliefs:

Every team should have free access to our core software, no matter their budget.
Developers should be able to install and experience all Kubecost features in minutes without contacting sales.
Our software should be open, transparent, and emphasize interoperability—users should be able to modify our code and integrate it easily with other open solutions.
Users should have total control over their data—they shouldn’t have to share private company information with us or anyone else to use our software.

After seeing how Kubecost was helping a growing number of teams, we decided to raise a $5 million seed round to pull together our founding team and build a company around the Kubecost open source project. Taking what we viewed as the best parts of Google’s culture (autonomy, open collaboration, and experimentation to name a few) and marrying them with our own values influenced by our open source roots (an emphasis on flexibility, allowing teammates to make decisions about how and where they work best, and optional meetings), we designed our company, Stackwatch, to be as developer-friendly as our product Kubecost was.

Kubernetes at scale, community at the core: the future of Kubecost
Enterprises all over the world are now running containers, with more than 70% of Fortune 100 companies using Kubernetes, as shown in the last Kubernetes Project Report. As companies emerge from the early stages of their adoption journey and start to scale their efforts, many are looking at operating margins with an increasing focus on the impact from infrastructure and cloud bills.

As a result, we’re now monitoring billions of dollars of cloud spend, and as our users move their implementation of Kubernetes to production, we’re partnering closely with them to give them control of spend—helping them build sustainable showback or chargeback models, offering governance solutions, real-time alerting, and more.

Open source is at the core of everything we build, as well as how we work, at Stackwatch. We couldn’t have built Kubecost without our deeply engaged community of users and open source contributors, and our core values of flexibility and neutrality inform every decision we make (starting with our commitment to building tools that are portable and cloud-agnostic).

It’s an exciting time for our company: we’ve just raised $25 million to take our products and team to the next level. Staying true to our mission of empowering engineers through access to meaningful, actionable data, we’re focusing on three key areas to scale our offerings and help way more users:

Expanding our open source contributions (exciting news on this topic soon!) and growing our community
Creating a world-class user experience that doesn’t end at in-product interaction—we aim to truly partner with our customers to add real value, no matter where they are on their Kubernetes adoption journey
Building new tools, more integrations, and different ways to implement Kubecost to reach an even broader audience of users

We’ll be sharing more in the coming weeks about our product roadmap and how you can help us build the future of Kubecost—our story is just beginning!

Announcing Our $25 Million Series A from Coatue to Grow Developer-Centric Community Around Kubernetes Cost Management

Kubecost — Fri, 18 Feb 2022 15:57:28 +0000

As the next major milestone in our journey, we are proud to share that we’ve raised $25 million in Series A funding led by Coatue Managemen t to help developers optimize and run Kubernetes at scale. We’re also excited to announce that our lead investor at Coatue, David Cahn, will join our board of directors. He brings a wealth of expertise from investing in now-public companies like UiPath, Confluent, Snowflake, and Gitlab, as well as leaders in the cloud-native space such as Starburst, Snyk, and Databricks. Our angel investors in this round include Snowflake CFO Michael Scarpelli and Looker co-founder Ben Porterfield, as well as our existing investors First Round and Afore Capital. We're thrilled to be partnering with them, alongside other great investors joining our Series A, as we continue to scale our product and company.

As of a December 2021 CNCF survey, there are now more than 5.5 million developers using Kubernetes and this number is only growing. While solving challenging infrastructure monitoring problems at Google, we saw firsthand how complex it can be to run containers and distributed systems efficiently at scale. That’s why we launched Kubecost in early 2019: to help teams monitor and optimize Kubernetes infrastructure by empowering them with meaningful and insightful data.

We now have thousands of teams—Adobe, Allianz, Capital One, and Under Armour among them—using our product at enterprises across the world, with our solutions now monitoring billions in infrastructure spend and providing cost transparency and optimization to tens of thousands of engineers. Another recent CNCF survey has already identified Kubecost as the most used tool for monitoring Kubernetes spend across all major clouds or in on-prem environments.

Kubecost’s growth speaks to organizations’ increasingly critical need to get an understanding of Kubernetes-related spend. In the rush to application modernization and containerized environments, we’ve seen how inefficient practices, overprovisioning, and a lack of awareness can contribute to unnecessary spending with no guardrails to protect against this. We provide our users and customers with the visibility and optimizations required to rein in Kubernetes spending and save them significant budget—both immediately and with sustainable practices for the long term. We plan to build on this momentum with our Series A funding, by building out more robust product offerings, investing in the open source community, and growing a fantastic team to support our customers along their Kubernetes adoption journey.

As engineers ourselves, we’re committed to building a product experience that serves developers’ needs—giving users the ability to easily install within minutes without ever contacting sales, sharing sensitive data, or giving up control—and ensuring everyone has access to our software, no matter their budget.

We’ve built an active open source community around Kubecost and we ourselves continue to be actively involved in the broader Kubernetes community. Stay tuned for a slew of exciting open source contributions, product launches, and even more resources dedicated to ensuring we bring value to a growing community of users.

Webb & Ajay, cofounders

Want to learn more?

Get started using Kubecost—install the community version: https://www.kubecost.com/install.html
Interested in joining the team? Review our job openings here: jobs.kubecost.com
Join our open source community! Learn how at contribute.kubecost.io
Reach out to us directly: team@kubecost.com or hello.kubecost.io