DEV Community: Robert Grootjen

What is Dataplex on Google Cloud? - Explained the simple way

Robert Grootjen — Thu, 27 Jun 2024 14:05:34 +0000

Dataplex is a unified data management service on Google Cloud. It provides a centralized dashboard where you can manage, monitor, secure, and govern all your data across various storage systems like data lakes and data warehouses.

Key features:

1. Unified Data Management
Unifies data lakes, data warehouses, and databases

2. Data Governence and Security
Ensures consistent security.

3. Data Discovery and Cataloging
Automatically discoverys and catalogs data.

4. Data Quality and Monitoring
Checks/monitors data quality and provides data health

5. Collaboration and Data Sharing
Fascilitates colaboration and sharing with team

Simple Metaphor to Understand Dataplex:

Imagine Dataplex as a modern, high-tech library system for a large university:

Traditional Libraries (Traditional Data Management):Each department has its own library with its own cataloging system, rules, and security measures.
Finding a specific book might require visiting multiple libraries and dealing with different sets of rules.

Dataplex (Unified Data Management):A central library system manages all the books across various departmental libraries.

Unified Management: Regardless of where a book (data) is located, you can manage and access it from a single system.

Governance and Security: There is a consistent set of rules and security measures applied to all books, ensuring they are well-protected and accessible only to authorized individuals.
Discovery and Cataloging: The system automatically catalogs new books, making it easy to find them through a unified search interface.

Quality and Monitoring: The library system monitors the condition of books and provides alerts if any book needs maintenance or replacement.

Collaboration and Sharing: Students and faculty can easily share books and resources securely, with the library system tracking who has access to what.

What is GCP Interconnect?

Robert Grootjen — Wed, 26 Jun 2024 14:35:00 +0000

What is GCP Interconnect?

Google Cloud Interconnect is a service that provides a direct connection between your on prem or datacenter and the Google Cloud Network.

Types of Interconnect:

Dedicated Interconnect: Provides a direct physical connection from your data center to Google Cloud.

Partner Interconnect: Uses a supported service provider, like a third-party, to connect your network to Google Cloud.

Methaphor to understand:

Imagine you have a house (your on-premises network) and you frequently need to visit a store (Google Cloud) to get supplies (data and services).

Public Internet Connection:
Public Roads: Using the public roads to get to the store. It can be slow and unpredictable due to traffic, and there is always a risk of encountering hazards (security risks).

GCP Interconnect:
Private Road: Instead of using public roads, you build a private road (Interconnect) directly from your house to the store.

Dedicated Interconnect: You build a private road exclusively for your use, ensuring a fast, secure, and reliable journey every time.

Partner Interconnect: You use a private road provided by a trusted partner, which connects you to the store with similar benefits of speed, security, and reliability.

What is Google Cloud Functions (simplified)?

Robert Grootjen — Tue, 25 Jun 2024 19:52:32 +0000

Google Cloud Functions is a service that runs specific code (a function) in response to events, without needing to keep servers running all the time.

Key features:

Event-driven: Functions are triggered or executed through HTTP requests, changes in database, or uploads to cloud storage

Scalability: Functions scales automatically based on incoming requests

Serverless: No need to manage servers

A Simple Metaphor to Understand Google Cloud Functions:

Imagine you run a bakery.

In this bakery, you have a bell that rings whenever a customer enters, signaling the bakers to start baking a specific type of pastry based on the customer's request.

The Bell: This is like the event trigger in Google Cloud Functions. It could be an HTTP request, a new file uploaded to cloud storage, or a database update.

The Bakers: These are like the cloud functions. They are small, specific tasks (functions) that get executed in response to the bell ringing (event).

The Recipe: This is the code you write for your function, dictating what the bakers should do when the bell rings.

The Bakery Manager: This is Google Cloud, managing the bakers (functions). You don’t worry about hiring more bakers (scaling), ensuring they have the ingredients (resources), or dealing with their schedules (server management). The manager takes care of everything so that the right number of bakers are always ready to respond to the bell.

cloudfunctions #serverless #gcp #googlecloud #cloud #google

GCP for beginners 2024: Build a simple web app with Cloud Run and Cloud Build through terminal

Robert Grootjen — Sun, 23 Jun 2024 16:37:34 +0000

What is Cloud Run?

Cloud run is a fully managed compute platform managed by Google Cloud that automatically scales containers and it allows developers to deploy containerized applications in minutes

What are the key features of Cloud Run?

Auto scaling = Scales up or down depending on traffic
Containerized applications = Run any stateless container
Fully managed = No need to manage servers, google takes care of everything
Pay per use = Pay only for the resources you use

What is Cloud Run used for?

Web applications and API’s = Great for deploying microservices, Restful API’s, and as well web applications
Data processing = Suitable for data transformation and ETL tasks
Background jobs = Can handly simultanious jobs
Event-driven applications = Works perfectly with event sources like cloud storage and pub/sub

It’s a solid combination of Cloud Functions and App Engine, best of both worlds and it can be used as a PaaS or Faas.

Cloud Run in a Metaphor

Imagine you’re running a food truck business. Each containerized application you deploy is like a food truck.

You can design and customize your food truck however you like, with any equipment, recipes (code), and ingredients (dependencies).

Now, think of Cloud Run as a food truck park.

The food truck park is the managed platform where your food trucks operate.

This park offers several features:

First, it provides flexibility—you can bring any type of food truck, serving any cuisine (programming language or framework).

Second, the park has a smart system that monitors customer traffic. When there are more customers, more food trucks are called in to handle the demand. When there are no customers, the food trucks are sent away to save costs.

Lastly, you don’t have to worry about maintaining the park infrastructure (like cleaning, security, or utilities).

The park management (Google Cloud) takes care of all that, so you can focus on your food (application) and customers (users).

How to build a cloud run through cloud shell (Here's the video)

Create a dir

mkdir my-cloud-run-app

cd into the folder we just created

cd my-cloud-run-app

Create a main.py file

nano main.py

from flask import Flask

app = Flask(__name__)

@app.route('/')
def hello():
    return 'Hello, Cloud Build!'

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8080)

Create a requirements.txt with the following

Flask==2.0.1
Werkzeug==2.0.1

Create a dockerfile

nano Dockerfile

# Use the official Python image from the Docker Hub
FROM python:3.9-slim

# Set the working directory
WORKDIR /app

# Copy the requirements file and install dependencies
COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt

# Copy the rest of the application code
COPY . .

# Expose the port the app runs on
EXPOSE 8080

# Run the application
CMD ["python", "main.py"]

Create a cloudbuid.yaml file

nano cloudbuild.yaml

steps:
  - name: 'gcr.io/cloud-builders/docker'
    args: ['build', '-t', 'gcr.io/$PROJECT_ID/my-cloud-run-app', '.']
  - name: 'gcr.io/cloud-builders/docker'
    args: ['push', 'gcr.io/$PROJECT_ID/my-cloud-run-app']
images:
  - 'gcr.io/$PROJECT_ID/my-cloud-run-app'

Now you should have something like this

Run cloud build to containerize the application

gcloud builds submit --config cloudbuild.yaml .

Deploy the cloud run (Make sure you put your Project ID and preferred region for example: us-central1)

gcloud run deploy my-cloud-run-app \
  --image gcr.io/PROJECT_ID/my-cloud-run-app \
  --platform managed \
  --region YOUR_PREFERRED_REGION \
  --allow-unauthenticated

After everything has been succesfully deployed, it will give you an URL to open/access your web app and it should say "Hello, Cloud Builders!"

Now make sure to remove it, after you are done! If not GCP will charge you.

Remove cloud run

gcloud run services delete my-cloud-run-app --platform managed --region YOUR_PREFERRED_REGION

Remove container image

gcloud container images delete gcr.io/YOUR_PROJECT_ID/my-cloud-run-app --force-delete-tags