☁️ Using BigQuery Billing Export to Manage your Invoice ☁️

jsrios — Wed, 15 Jul 2020 20:50:55 +0000

Welcome 😄

I'll be walking you through how to set up Google Cloud Platform Billing Export with BigQuery to help you manage and have more insight into your cloud invoice.

Why do I need this?

It depends.

If you manage Billing at your company or have different side projects you definitely want to be able to manage, analyze, and optimize your cloud costs to fit your company/personal budget. If you're just experimenting with GCP, then it's just something to learn!

Set up a BigQuery Dataset

Navigate to BigQuery

Create a new Dataset

This is the Dataset that the Billing data will be streamed into

Dataset ID: Can be whatever - 'Billing' works
Data Location: Default is fine
Table Expiration: For this walkthrough 2 days is fine, you probably wouldn't want it ever expiring though
Encryption: Google-managed key is fine

Click 'Create Dataset'

You now have a Dataset set up for your Billing data to be streamed into!

Activate the Billing Export Feature

Navigate to Billing

Navigate to Billing Export

Click 'Edit Settings' Button

Note: If you don't see text that says 'Disabled' then you already have the Billing Export feature activated. So you just need to know the Dataset that it's streaming into and check it out.

Select your Project & Billing Dataset

Click Save

And that's it!

Google mentions that it might take a few hours before the data starts streaming into the table, but there's nothing else to do from your end other than wait.

What you can do with this Dataset

The Dataset that Google creates and streams for you is really detailed and they're constantly making updates so that you can gain more insight into your costs and a breakdown of those costs down to the penny. This Dataset can be drilled down and analyzed by project or service and can even show you how much of your costs are in taxes. All you need to do is be creative with your queries.

Extra Resources

Analyzing Google cloud Billing Data with BigQuery

Reconcile your Monthly GCP Invoice with BigQuery Billing Export

Follow Me 😄

Twitter - @jsrioss
Github - @juanri0s

Keeping your Google Cloud Platform Bill to a Minimum with Billing Alerts

jsrios — Tue, 14 Jul 2020 18:50:46 +0000

When experimenting with cloud providers, it's important to keep in mind how to manage costs, because it can be very easy to wake up the next morning with a $2700 bill 😬.

Chris Short

@chrisshort

Good morning, $2700 AWS bill!

Holy shit...

13:26 PM - 04 Jul 2020

66 1323

Even when there might be free tiers or cloud credit, you don't want to accidentally leave an instance of something running and finding out it's all gone and went to your credit card.

I'll be walking through how to manage costs by setting up Billing Alerts in Google Cloud Platform as well as providing further resources for some of the other providers.

Note: This will probably cover most of your cost alerting needs for experimentation, but for larger projects or side projects where you expect decent traffic, you will probably want to take a look at the services being used and optimize the resources needed to run those services alongside setting up alerts.

Setting up Billing Alerts

Navigate to the GCP Billing Console

Navigate to Budgets and Alerts

Create a Budget

50% and 90% threshold for alerts work fine for this walkthrough but Google Cloud Platform allows you to add and remove as many as you want (no duplicates) so choose whatever works for you. For example, you could set up alerts at every 10% threshold like below if you want:

For this walkthrough, there is no need to set up Cloud Pub/Sub or a Monitoring workspace so the two options under Manage notifications can remain unchecked.

And that's it!

You will now be properly alerted for your Google Cloud Platform usage so you won't need to worry about any surprise bills.

As you can see, you can't tell GCP (and other cloud providers) to just stop once you have used $20. Since your service is up and running it can get any amount of traffic at any point in time, so it wouldn't be the easiest thing for cloud providers to manage. So most providers will allow you to set up alerts or alarms so that you can be more aware of how much your project is costing and if it ever reaches a bad point you will be alerted.

The Ideal Solution

Tomasz Łakomy

@tlakomy

AWS should allow you to mark your account as “hey, I’m learning this stuff, this is not a business” and set a *hard limit* to 20 USD per month

Surprise-bill-driven-revenue is a terrible business model, they should get rid of that part

08:18 AM - 07 Jul 2020

208 1526