DEV Community: Giuliano Ribeiro

Programmatically start and stop Google Cloud Compute Engine VM without code

Giuliano Ribeiro — Wed, 24 Feb 2021 09:06:49 +0000

Hi there, long time since my last post, but here I'm!
Today I want to show you a simple but great tip for users/admins of GCE VMs.

Intro

In the past, when we needed to periodically stop or start a VM in GCE was a pain! As we don't have an easy way to do it, we should write a Cloud Functions or include a cron job in a VM (like a central VM with all your cron jobs).
But a few months ago, GCP added a new way to do it using the API directly in the Google Cloud Scheduler (a serverless cron service).

Now, let's do it!

Service Account

First things first: to configure it properly, you should have a service account. As best practice, you need to create a new service account and give the Compute Admin role. But for this tutorial, I'll show using the default Compute Engine service account.
Go to Google Cloud Console > Navigation menu > IAM & Admin > Service Accounts

Copy the email address associated to Compute Engine default service account. This email is always something like 999999999999-compute@developer.gserviceaccount.com

Next!

Cloud Scheduler

The Cloud Scheduler is a serverless cron-like service. It is pretty handy for this kind of job. You can setup jobs to do a variety of calls, like general HTTP calls, App Engine services calls and post to Pubsub topics.

Next, create the jobs. Just click in the Create Job button and let's do it!

New job

The create job form is simple and straight to the point.
The important parts are the HTTP URL and the Auth header ones.

For the HTTP URL you should fill with something like this:

https://compute.googleapis.com/compute/v1/projects/<PROJECT-ID>/zones/<VM-ZONE>/instances/<VM-ID>/<start OR stop>

But first is required to have the Instance ID.

Get Instance ID

Just go to Navigation menu > Compute Engine > Click in your VM to get details

For example, my URL to start would be like this:
https://compute.googleapis.com/compute/v1/projects/titanium-campus-305808/zones/europe-west1-d/instances/514142841644/start

Now we have all information, this is the Create job form:

All Scheduler jobs

As you can see I have 2 jobs, one to start and another to stop my Minecraft server. You can also enable or disable the jobs, so if you don't want the job to run for a specific period. Or, of course, you can setup the frequency of the job for a more complex form, as it respects the cron syntax.

So that's it, my friends, I hope it can help you in any way and as always, send me feedbacks. I'll always try to help you about GCP!

Thanks,

UPDATED 2023: Auto deploy AppEngine app from Gitlab

Giuliano Ribeiro — Wed, 11 Dec 2019 19:50:19 +0000

Nowadays, CI/CD is a common task during Software Development and almost all platforms enable us to do this automatically. It is not different on Gitlab.
Today, I'll show you how to configure your Gitlab project to automatically deploy to Google Cloud App Engine.

Google Cloud part

First things first

First, you need to setup your App Engine project on GCP console. It is necessary because App Engine requires you to set the region of your app.
If you didn't do this, first you will need to setup using this cmd:

Setup your App Engine app.

and follow the instructions.

Updates

The stesp below were made a few years ago, so some permissions were changed.
To simplify the process, I paste here a script. Take a look, and change the appropriated information:

export SA_NAME=gitlab-sa
export GCLOUD_PROJECT=<YOUR_PROJECT_ID_HERE>
export SA_EMAIL=${SA_NAME}@${GCLOUD_PROJECT}.iam.gserviceaccount.com

gcloud services enable appengine.googleapis.com cloudbuild.googleapis.com

gcloud iam service-accounts create ${SA_NAME} --display-name 'Gitlab Service Account to deploy' --project ${GCLOUD_PROJECT}

gcloud iam service-accounts keys create ${SA_NAME}.json --iam-account=SA_EMAIL --project ${GCLOUD_PROJECT}

cat ${SA_NAME}.json

Copy the JSON content and save to include in the Gitlab CICD variables page.

Now continue the process, giving to the service account some permissions:


gcloud projects add-iam-policy-binding ${GCLOUD_PROJECT} \
    --member="serviceAccount:${SA_EMAIL}" \
    --role=roles/appengine.appAdmin

gcloud projects add-iam-policy-binding ${GCLOUD_PROJECT} \
    --member="serviceAccount:${SA_EMAIL}" \
    --role=roles/storage.objectAdmin


gcloud projects add-iam-policy-binding ${GCLOUD_PROJECT} \
    --member="serviceAccount:${SA_EMAIL}" \
    --role=roles/iam.serviceAccountUser

gcloud projects add-iam-policy-binding ${GCLOUD_PROJECT} \
    --member="serviceAccount:${SA_EMAIL}" \
    --role=roles/cloudbuild.builds.builder

Service Account and Permissions

Jump to Gitlab Config part, if you already did the script above.

~~The entire process should be made without human intervention. So, to do this on GCP you'll need a Service Account.~~

~~First, create a Service Account to be used during the process: ~~Service Account Create

Step 1

Step2: ~~Critical step, here you MUST give all those permissions. Without them, the automation will not work properly.~~

Step 2: VERY IMPORTANT!

Step 3: Create a Key

Step 4: choose the JSON option!

In the last step, you should open the file and copy the content. This content will be necessary to configure the Gitlab.

API Enabling

Some APIs must be enabled before trying to run the deploy.

Enable the App Engine Admin API, here:
Enable App Engine Admin API
App Engine Admin API enabling
Enable Cloud Builder API: Cloud Builder API Enable
Cloud Build API enabling

Gitlab Config

Now you have all done on GCP, Gitlab part!
Go to your project and configure the CI/CD options:

CI/CD menu option

Instead of having the content of your Service Account in a file inside your repository, a best practice is to have this as a variable. So, configure a variable, following the image:

Configure the SERVICE_ACCOUNT_KEY variable with the JSON content downloaded in the Step 4.

Gitlab CI file descriptor

To enable all these pieces together is required to "tell" to Gitlab: "Please my friend, run all these now". To do this, you should create a file in your repository called .gitlab-ci.yml, this file describes what you want to do.

Basic gitlab ci descriptor.

File content available here.
https://gitlab.com/giulianobr/simple-go-app/blob/235de13e310d7614f9b8cb2dab0f804c6b82a041/.gitlab-ci.yml

The CI process runs inside the container platform in Gitlab. At the glance, it seems like a Dockerfile, as you should define and base image where your deploy will run.
In this case, I'm using Google's base image that contains the Google Cloud SDK already installed.
To understand more about all configs available on Gitlab CI, take a look here.

Conclusion

This post aims to help you to deploy an app to App Engine Standard. As my example is a Golang app in the 2nd gen runtime, it is a bit easier. To deploy a Java 8 app is a little bit different. If you want any help to setup your Java 8 app, ask me in the comments, I also have an example for this scenario.

I hope it helps you, people!
See you soon.

You can find this project in my Gitlab project.

Cloud Data Fusion, a game-changer for GCP

Giuliano Ribeiro — Thu, 05 Dec 2019 19:41:22 +0000

Continuing the Big Data topic, I want to share with you this post about Google Cloud Data Fusion.

The foundation

Cloud Data Fusion is based on Cask™ Data Application Platform (CDAP).
CDAP was created by a company named Cask and this company was bought by Google last year. CDAP was incorporated into Google Cloud and named as Google Cloud Data Fusion.

CDAP Pipeline page

Data Fusion

Data Fusion is a fully managed CDAP with steroids 🧬.

From the Google Cloud page:

Cloud Data Fusion is a fully managed, cloud-native data integration service that helps users efficiently build and manage ETL/ELT data pipelines. With a graphical interface and a broad open-source library of preconfigured connectors and transformations, Cloud Data Fusion shifts an organization’s focus away from code and integration to insights and action.

On important info about Data Fusion is it rely on Cloud DataProc(Spark), handling the cluster (create and delete) for you.

The game-changer of Data Fusion is the amazing graphic interface providing for the user an easy to use way, to create from a simple transformation pipeline to the complex ones. The best: without a line of code.

The main page of Data Fusion

This post will detail the first 2 options: Wrangler & Integrate
At the end of this post, you can watch a video tutorial where I show a step-by-step using Data Fusion.

Wrangler

Wrangler is the center part to prepare your raw data. You can upload files or connect to a variety of external sources like databases, Kafka, S3, Cloud Storage, BigQuery and Spanner.

Choose you source file or something else

Right after you choose your source, you are redirected to the parsing mode.

Parse the CSV file

As you can see at the top, there's a tab called insights. There you can see some useful graphs about your data:

Insights about your data!

Studio

On the Studio, you have all the great tools to create your data pipeline. The source, the transformation, and the sink. Each one of them with a diversity of choices. Take your tool!

Studio page to design your pipeline

The main page of Studio, on the left you have the Source, Transform, Analytics, Sink, "Conditions and Actions" and "Error Handlers And Alerts". The gray area where you design your pipeline.

A simple but complete pipeline:

Complete pipeline, ingestion from GCS -> Joiner with CloudSQL -> sink to BigQuery

All available tools in the Studio:

All available tools on Studio.

You can also install more sources and sinks, in the Hub:

The HUB

Pipeline

After design your pipeline on the Studio, you need to deploy it. Just click the "Deploy" button, then you can see your pipeline on the next page. On this page, you'll be able to run the job, watch the logs, configure Schedule and also see the Summary of executions.

You pipeline deployed, just waiting to work!

At the top in the center, click on "Summary" to check some facts about your jobs.

Summary about your executions.

Step-by-step

This step-by-step was recorded and edited by me(sorry any issue).
You'll be able to see a complete design and execution of a pipeline. The steps of this pipeline are:

Ingestion a CSV
Parse the CSV
Prepare some columns of the CSV
Use the Join Transformation tool
Connect to a PostgreSQL(CloudSQL) instance
Get information about the States of Brazil
Join it with the correspondent column of the CSV
Output the result in a new table on BigQuery

Conclusion

As you can see, Data Fusion is an amazing tool for data pipelines. Powerful because it can handle as much data you have, taking the advantage of Google Cloud features, and impressively easy to design the workflow. Another great feature is the possibility to create your connector, as it is based on CDAP, you can develop your connector and deploy on HUB.
Data Fusion is GA since last week and has a lot of big customers already using. It is just the beginning of this incredible tool.

As usual, please share this post and consider giving me feedback!

Thank you so much!

App Engine and WebApp with Golang

Giuliano Ribeiro — Mon, 02 Dec 2019 12:29:22 +0000

Now is time to App Engine! My favorite GCP's service.

Intro

App Engine is the oldest service on GCP. It was born before GCP even exists. It's from 2007 and started as an internal Platform created to provide an easy and scalable way to Googlers deploy and serve their projects.

App Engine is a Platform as a Service product where you can deploy your app written in some runtimes, like:

Python
Java
PHP
Golang
Ruby
Node

Before you hear about containers, App Engine already has been using this concept since 2007! It's also true about serverless concept, as you don't need to take care of the underlining infrastructure. You only need to care about your code and the App Engine descriptor file (where you include the information about your necessities).

Versions

App Engine offers to you 2 "flavors", the Standard Environment, and the Flexible environment.

Standard environment

The Standard env is a cloud-native option, you don't need to care about the infra and they have a really fast spin-up engine to create the instances for your traffic in manner of milliseconds.
But in some old runtimes (Python 2.7, Java 8) you may have some issues with some frameworks as the behavior is not the same as you are used. But it's going away (finally!) with the new runtimes (Java 11, Python 3.7 and Go 1.12+), as they are Gvisor-based and the behavior is the same as you are used to.
On Standard env, you pay only for the usage, and its scales to zero. Which means, when there's no traffic, your app "sleep" and you don't pay :) When the traffic come, App Engine spinup your app as fast as possible. In Golang, for example, it takes no more then 1s to "wake up".

Flexible environment

Although the Flexible env is a bit different because is a container-based environment, and relays on Compute Engine VMs. You can use any kind of runtime to run, the important point is: your app should respond in the 8080 port. So Google doesn't care about your runtime but takes care of scalability based on container usage(Memory and CPU).
As this flavor is based on Compute Engine, you should take care about the sizing of theses VMs.

More about App Engine, take a look at the official page: Google Cloud App Engine

IMHO, the Standard is the best option! You have so many great services and features than the Flex. And, of course, the scalability in an easy way (if you need it).

Free tier

A nice "feature" we have on App Engine is the free tier (only for Standard Env). It means you can run your app paying nothing! :)

It is not a special thing on App Engine, you can use some other services on GCP for free too, check this out: GCP Free tier

For App Engine you can use those resources:

Web App

Deploying your app on App Engine you get a URL with https for FREE. This URL is based on App Engine domain, so you are going to have something like this: https://YOUR-GCP-PROJECT-ID.appspot.com

For this post, I choose the Golang runtime, even I'm a Java dev for a long time, in my last years I've been working with Golang instead.

As you can see I'm using the web framework Gin-Gonic.

The unique difference between an ordinary Go web app and this one for App Engine is the file app.yaml. This is the App Engine descriptor, where you need to inform you runtime, and some other options:

Check this page out, for more available options: appm.yaml Configurations

After doing what you want, you only need to deploy your app using the Google Cloud SDK


gcloud app deploy .

It's done, your app is deployed and you can access by the URL provided in the end of the process.

You can have all this Golang code on my Github:

giulianobr / simple-go-app

A simple Go app to serve a static site

simple-go-app

A simple Go app to serve a static site

Run the app

go run main.go

To deploy

Install and setup your Google Cloud SDK.

Run:

gcloud app deploy .

View on GitHub

Conclusion

I hope this post helps you understand, at least, a bit of App Engine and how to use these wonderful services provided by GCP.
As a Platform as a Service, you have the power to focus on your code/app and leave the issues of infrastructure management to Google's team. A really good way to do MVPs, PoCs and to test an idea without the need to create an infrastructure for that.

As usual, please, give feedbacks! :)

Data Lake on Google Cloud Platform

Giuliano Ribeiro — Mon, 25 Nov 2019 09:56:44 +0000

Data Lake on Google Cloud Platform

First of all, thank you for reading my post.
Second, it's a simple overview about Data Lake on GCP.

Intro

Google, as you know, was the last of the big players to enter in the cloud 'war'. But they did it in a very different way, they try to do this focusing on managed services and also focus on Big Data (and Machine Learning) things.
For Big Data, they are good (they ARE really good). With so many years improving their internal workloads for such a huge amount of data, the services provided are high in class.

Words are cheap, show me the "pic" :)

A complete Data Lake solution diagram:

The truth is: the main component on the Data Lake is Google Cloud Storage. The GCS, for short, is the place where you can store all your data. Even the other products in some way also use the GCS to store things. GCS is a powerful service in GCP, with many configs and ways to use it. In a Data Lake, we use it for unstructured data. For structured data, we commonly use CloudSQL(up to 10Tb), Spanner(Global Relational Database), BigTable(Low-latency-NoSQL Database) and BigQuery(Datawarehouse).
For each type of data, we can use one service/product available in GCP.

But to be honest with you, for structured data, BigQuery is the king. You can store and process all sizes of data you have in a manner of seconds.
In BigQuery you can import data in batch or use the Streaming API. Streaming API allows you to send up to 100,000 requests per second.

Of course, Data Lake is a huge topic to discuss, but here I'm trying to show you in a pic and in a few words, how it is possible on GCP. You may notice, but 2 of the most important service for data lake, provided by Google Cloud, is Cloud Storage and BigQuery. But they are not alone in the field, below I wrote down the other "friends" which you can use to create a complete environment for your big data on GCP.

Ingestion

gsutil
Cloud Transfer Service
BigQuery API
Cloud Pub/Sub

Store

Cloud Storage
Bigtable
BigQuery
Spanner

Process

DataPrep
Dataflow
Dataproc

Analyse

BigQuery

Reporting

Data Studio

Conclusion

As I told you before, it is a simple and direct-to-the-point post :)

Data Lake is only one part of the huge topic "Big Data", but it is the starting point. As a fundamental part of this, I'll try to bring more information about it, writing one more post about it. I also will bring more about each one of those services, creating here on dev.to a series of posts to giving me a chance to write the things I know, and bring to all the community information about this topic.

As the first post, I'm absolutely pushing hard (as I'm not a writing person, even more, I'm not an English native speaking/writing). Please, give feedback, in all ways possible. I've been working hard to change myself to create the habit to write. But without feedback, I feel like to write to no one.
I really appreciate your time, thank you so much!