DEV Community: Ernesto Lopez

List networking information configured during Google Cloud Foundation Setup

Ernesto Lopez — Sun, 17 Nov 2024 15:47:34 +0000

Gogle Cloud Foundation Setup

A foundation setup in Google Cloud is essentially the base layer of your cloud environment. It's the core infrastructure that supports all your applications and workloads. Think of it as the foundation of a house - it needs to be strong, secure, and well-planned to support everything built on top of it.

During the Foundation setup, the relevant activities configured are:
Here's a breakdown of what it entails:

Organizational Structure: How you organize your cloud resources using folders, projects, and the organization itself. This ensures proper resource management and access control.
Identity and Access Management (IAM): Controlling who has access to what resources and what they can do with them. This is crucial for security and compliance.
Networking: Setting up your Virtual Private Cloud (VPC) network, subnets, firewalls, and connectivity to on-premises or other cloud environments.
Security: Implementing security measures like firewalls, organization policies, and data encryption to protect your data and applications.
Automation: Automating tasks like resource provisioning, configuration management, and deployments to increase efficiency and reduce errors.

These steps could be performed using the console throught the Google Cloud setup service

Another option is to use terraform to deploy the environments, for that we can use several options like:

Exporting terraform code from Google Cloud setup
Using Google Cloud Fast Fabric terraform code.
Creating your own code tailored to your needs

Networking in Foundation setup

One of the critical steps during the foundation setup is the networking section, here you will need to decide the network achitecture and other aspects like shared vpcs, connection to onprem, firewall rules, etc.

During foundation creation, the main activities on this phase are:

Shared Virtual Private Cloud (VPC) networks
Configure connectivity between the external provider and Google Cloud.
[OPTIONAL] configure Dyrectory Sync
Set up a path for external egress traffic (Cloud NAT or Public Access)
Implement network security controls (Firewall rules)
Choose an ingress traffic option (example Load Balancer)

Here's the importance of networking in your Google Cloud foundation setup in ten bullet points:

Connectivity: Enables communication between resources, on-premises networks, and the internet.
Security: Allows for firewalls, security tools, and network segmentation to protect resources.
Isolation: Creates boundaries between resources to limit the impact of security breaches.
Performance: Optimizes routing for fast and efficient data flow.
Scalability: Supports future growth and resource expansion.
Flexibility: Adapts to changing application needs and hybrid/multi-cloud environments.
Global Reach: Connects resources across different regions for low-latency access.
Efficiency: Streamlines traffic flow and reduces data transfer costs.
Control: Provides granular control over network traffic and access.
Automation: Enables automated network management for reduced operational overhead.

However there are times where you will not create the network from scratch because this was created by another person in a previous time or maybe the implementation documentation is not in place, for those cases the following list of commands could be very helpful:

List VPCs in a project

gcloud compute networks list --project=<projectName>

This command displays a list of all the Virtual Private Cloud (VPC) networks within a specific Google Cloud project.

You replace with the actual ID of your project to see its networks.

NOTE if you want to see the results in table format use
gcloud config set accessibility/screen_reader false

List VPCs in a project

gcloud compute networks subnets list --network=<vpcName>

This command lists all the subnets that belong to the VPC network. You would replace with the actual name of the VPC network you're interested in. This allows you to explore the subnet structure of different networks in your project.

List Firewall Rules associates with a VPC

Note FIrewall rules listing return too much information, in this case we are going to filter the data and only obtain information that appears in the following table:

gcloud compute firewall-rules list --format="table(
        name,
        network,
        direction,
        priority,
        sourceRanges.list():label=SRC_RANGES,
        destinationRanges.list():label=DEST_RANGES,
        allowed[].map().firewall_rule().list():label=ALLOW,
        denied[].map().firewall_rule().list():label=DENY,
        sourceTags.list():label=SRC_TAGS,
        sourceServiceAccounts.list():label=SRC_SVC_ACCT,
        targetTags.list():label=TARGET_TAGS,
        targetServiceAccounts.list():label=TARGET_SVC_ACCT,
        disabled
    )"

This command lists all firewall rules in your Google Cloud project and displays them in a table format with specific details. Let's break down what each part means:

gcloud compute firewall-rules list: This is the basic command to list all firewall rules.
--format="table(...)": This part specifies that the output should be displayed in a table format. Inside the parentheses, you define the columns you want to see.
name: The name of the firewall rule.
network: The network the rule applies to (e.g., "default" network).
direction: Whether the rule applies to incoming traffic (INGRESS) or outgoing traffic (EGRESS).
priority: The priority of the rule (lower numbers mean higher priority).
sourceRanges.list():label=SRC_RANGES: The source IP address ranges that the rule applies to.
destinationRanges.list():label=DEST_RANGES: The destination IP address ranges that the rule applies to.
allowed[].map().firewall_rule().list():label=ALLOW: The allowed protocols and ports (e.g., tcp:80, udp:53).
denied[].map().firewall_rule().list():label=DENY: The denied protocols and ports.
sourceTags.list():label=SRC_TAGS: Source tags that the rule applies to (tags are labels on VMs).
sourceServiceAccounts.list():label=SRC_SVC_ACCT : Source service accounts that the rule applies to.
targetTags.list():label=TARGET_TAGS: Target tags that the rule applies to.
targetServiceAccounts.list():label=TARGET_SVC_ACCT: Target service accounts that the rule applies to.
disabled : Whether the rule is currently disabled (true or false).

Essentially, this command gives you a comprehensive overview of your firewall rules in a structured table, making it easier to understand your network security configuration.

List VPN Information

gcloud compute vpn-tunnels list --filter="region:( us-central1 europe-west1 )"

This command lists all VPN tunnels in your Google Cloud project that are located in either the us-central1 or europe-west1

gcloud beta compute vpn-tunnels describe <vpnTunnelName> --region <region>

This command provides detailed information about a specific VPN tunnel named in located in the region of your Google Cloud project. It uses the gcloud beta compute vpn-tunnels describe command, which fetches configuration details, status information, and other relevant data about the specified tunnel. This is useful for troubleshooting, monitoring, or simply getting a comprehensive view of a particular VPN tunnel's settings and operational state.

gcloud beta compute vpn-gateways describe <vpnGatewayName> --region <region>

The command retrieves detailed information about a specific VPN gateway in your Google Cloud project, including its configuration, status, and other relevant data. You need to replace with the actual name of your gateway and with its location (e.g., us-central1). This is useful for troubleshooting connection issues, monitoring the gateway's health, and verifying its configuration.

gcloud compute routers describe <vpc-router> --region <region>

This command retrieves detailed information about a Cloud Router in your Google Cloud project. You replace with the name of the router and with its Google Cloud region (e.g., us-central1). This provides insights into the router's configuration, BGP settings, interfaces, and operational status, which is useful for network management, troubleshooting connectivity issues, and verifying that the router is operating as expected.

SUMMARY With these commands you can generate tables with relevant information to know an specific environment or to put it into a delivery summary documentation, however this is way more easier if you use infrastructure as code to generate your infrastructure, that way your code is your own documentation.

3 Aspects to consider when using Google Cloud Serverless VPC Access

Ernesto Lopez — Mon, 26 Dec 2022 22:12:19 +0000

Serverless VPC Access is a service inside Google Cloud that allows to connect serverless services to your Virtual private cloud.By default, services like Cloud Functions, Cloud Run, App Engine uses external endpoints that allow other services to reach to them. In case that you want to keep connection to these services only accessible to other instances inside a VPC and to use private IPs and private DNS you need to use Serverless VPC Access

There are good documentation about how to use it, even for Shared VPC:

Cloud Run: Connecting to a Shared VPC network
Cloud Functions: Connecting to a Shared VPC network
App Engine: Connecting to a Shared VPC network

On the next sections, I will focus on 3 networking aspects that we should consider when using Serverless VPC Access:

IP Addressing

In reality, Serverless VPC Access consist of an access conector that is created using VM instances (On December 2022 there are only 3 types: f1-micro, e2-micro,e2-standard-4). The Instance selection is based on the network Throughput you require and the "cluster" can be minimum 2 instances and maximum 10 instances, in fact this is the default configuration.

NOTE: These instances are not listed inside GCE instance API. You can test it by running gcloud compute instances list

It is recommended to set a custom number for maximum and then increased based on your needs because connectors don't scale down automatically or manually.

When configuring the Serverless VPC connector you need to:

Associate the connector to a VPC network, one connector to one VPC. The VPC can be in the same project or in another project.
Create an IP Range, more on this point Next.This also mean that the Connector is associated with a Region, it is not Global.
Define Escaling configuration (min and max) and instance Type. All instances are the same type, you cannot mix f1-micro with e2-micro` inside the same connector.

Now, let's get to the point, IP Addressing. You must configure /28 IP CIDR Range, unused inside the same VPC (example 192.168.1.0/28). This is important, the iP Addressing range must not overlap with any existing Subnet range inside the VPC.

You cannot use a different mask other than /28, and the reason behind it rest in the maximum configuration, if tyou recall you can have up to 10 instances, a /28 subnet mask will give you 16 IP Address, Google reserve 4 for their use, you have 12 IP Addresses that you can use.

If you try to use a different ip range it will show an error:

As you may have noticed, it evens pre-populate the /28 for you and it is grayed out, an indicator that is not configurable.

Another detail, this IP Range will not appear inside the VPC's Subnet ranges, In the following example you may see 4 subnets

But when you enter inside the VPC Subnets details it appears only 2:

Even if you try to list the subnets using gcloud compute networks subnets list you will not be able to see them. you can use the following CLI command to retrieve the vpc-access subnets:
gcloud compute networks vpc-access connectors list --region=us-central1 --filter='NETWORK=vpc-demo-1'

The best option is to create a Subnet and assign that subnet during connector creation, that way you will be able to track the subnets related to VPC Access:

Note: Remember that you need to use /28 otherwise it will not recognize the subnet during Serverless VPC Connector

Now when we create the connector:

Look the difference between a connector using an Existing Subnet and one using CUSTOM IP.

From the security section, you can use IP Assigned with Firewall Rules, additionally serverless VPC Access Connector is created with two tags that you can use for firewall rules:

Universal network tag: vpc-connector
Unique network tag: vpc-connector-REGION-CONNECTOR_NAME. Example: vpc-connector-us-central1-vpc-access-test-connector

CLOUD NAT

TL;TR
When you use Serverless VPC Access with Cloud NAT, the port allocation for Cloud NAT is for Serverless VPC connector instances and no for functions or container, so if you have 2 Instances in the connector and one public IP Addresses the port allocation is between the instances in the region, incluiding the 2 instances in the connector.

Consider this architecture:

Where we have:

One Cloud Run function that curl an external URL http://ifconfig.me.
One Serverless VPC Connector with Custom IP in us-central1.
Cloud NAT in us-central1 with one public ip address and dynamic port allocation.

If you want to replicate this example, create a cloud run service with a code simialr to this:
`

import os
import requests

from flask import Flask

app = Flask(__name__)


@app.route("/")
def hello_world():
    r = requests.get('http://ifconfig.me')
    return "Hello {}!".format(r)


if __name__ == "__main__":
    app.run(debug=True, host="0.0.0.0", port=int(os.environ.get("PORT", 8080)))

Deploy the cloud run service:

gcloud run deploy curler-private-egress \
  --vpc-connector projects/vpn-site1/locations/us-central1/connectors/run-external-traffic \
  --vpc-egress=all-traffic \
  --platform managed \
  --region us-central1 \
  --no-allow-unauthenticated \
  --max-instances=10
```



Serverless VPC Connector:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4nuizstb5t37jwm7pzcv.png)



And Cloud NAT config similar to this

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/j56l8lg96vt9v42d21rp.png)

After triggering several time the service, by calling the **cloud run service's DNS** from an instance inside the same VPC. The instance's service account require the `cloud run invoker` role.

You can use something like this:



````console
for i in {1..100}
  do
  curl -H "Authorization: Bearer $(gcloud auth print-identity-token)" https://curler-private-egress-61zgbtvvvv-uc.a.run.app
done
```




![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rat2cshwo8dfcs18i4lr.png)

And inside the logs you will see:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jdjz409yyc8jzmeei8gz.png)
Note that the `src_ip` in Cloud NAT's logs (the private IP that make the request to external resources) are the ones from the **Serverless Connector**.

>VMs with an external IP address can have 64,000 TCP, 64,000 UDP, and 64,000 ICMP-query sessions (ping) simultaneously if they have enough compute/memory resources. For Cloud NAT, this limit is reduced to a total of 64,000 connections per VM for all supported protocols combined.

************************************************************
##Deleting the VPC

Let's supose that you try to delete the VPC that have a Serverless VPC Connector:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vcxlhgidgjkcvezu71mx.png)

If there is a subnetwork being used by Serverless VPC Connector, you will NOT be able to delete the VPC, receiving the next error message:
>The deletion of the network failed. Error: The subnetwork resource 'projects/vpc-test1/regions/us-central1/subnetworks/vpc-access-test-serverless-us-central1' is already being used by 'projects/vpc-test1/zones/us-central1-b/instances/aet-uscentral1-vpc--access--test--connector-506w'

First you will need to delete the connector:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gvmd856551eoyluyr6gk.png)

Success!! 
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8c188eymzsu4ytle3xap.png)

But, What happens with a VPC where i create a Connector using Custom IP insteado of an existing subnet?

Spoilert alert!!!! you will receive the same error:
>The deletion of the network failed. Error: Operation type [delete] failed with message "The network resource 'projects/vpc-test1/global/networks/red-test-vpc' is already being used by 'projects/vpc-test1/global/firewalls/aet-uscentral1-rb--vpc--connector--test245-sssss'"

The recommendation is, make sure you review resources using your VPC prior attempting to delete it.

You can use **[Cloud Asset Inventory](https://cloud.google.com/asset-inventory/docs/overview)** service and filter by `compute.Network` and move to Resource TAB, and you will obtain a list of resources that live inside the VPC.

My recommendations to approve Google Associate Cloud Engineer and Professional Cloud Architect certifications in 60 days

Ernesto Lopez — Fri, 13 May 2022 20:16:00 +0000

As a brief introduction, i will let you the links to the certifications so you can start thinking about the first recommendation:

Google Cloud Certified Associate Cloud Engineer
Google Cloud Certified Professional Cloud Architect

Base on the previous list, my first advice is to find two certifications that are closely related, this way you can use the study from the first one as a base for the second one.

NOTE: This is my personal experience, i came from working with AWS and Oracle cloud from several years so i have strong knowledge in cloud computing. Experience is important but it is not all.

ANOTHER NOTE: Google Cloud manage some services in a different way so from time to time you need to forget about waht things are done on AWS or another cloud provider.

Start point > Base knowledge

Here i have two recommendations, if you came from AWS for example of from Azure, on Pluralsight there are two courses that provide a translation to Google Cloud services:

These trainings allow you to map the services that you know with the services on Google Cloud, and to identify the important differences. In my case was very useful for IAM and Storage area.

NEXT, we well keep ourself on Pluralsight, there are Certifications Paths created by Google Cloud, so let me emphasize this, the certifications trainings are created by Google itself.

For Associate Cloud Engineer start with this path:
- Google Cloud Certified Professional Cloud Engineer. The path is around 17 hours of video content and maybe another 12 hours for labs but it will give you a solid ground for knowledge.
For Professional Cloud Architect start with this path:
- Google Cloud Certified Associate Cloud Architect. The path is around *27 hours
of video content and maybe another 20 hours for labs. The advantage with this one is taht it has some trainings from the previous path so you do not need need to view the courses again.

So, if both path share most of the trainings why should a present two different certifications?

Well, the short answer is yo do not need to present both, but it will give you a different set of skills.
For example under ACE beside services and base architecture usage you are tested on hands on activities, specially on CLI, this knowledge is base for cloud administration and for other certifications.
While on PCA you are tested on architecting solutions for specific business needs, and less on CLI commandos or process to configure managed services. Example: you will be tested about when to use certain load balancer but not how to configure it.

OKOK got it, but why Pluralsight?

Well for trainings yu have several options: Pluralsight, Coursera, A Cloud Guru, etc. Why i choose pluralsight? because the trainings are created by google itself and are updated once a year and in some cases twice a year, so you will have updated content for your certifications.

Extra recommenation for base knowledge.

After i passed the ACE exam a beautiful book was released
Visualizing Google Cloud: 101 Illustrated References for Cloud Engineers and Architects from Priyanka Vergadia
The most amazing from this book is that is separated by vertical, example storage, compute, etc and provides you with a graphical way of mapping Google Cloud services and then a brief explanation of each one, ending each chapter with some use cases for those services previously explained.
I use this book from time to time to prepare some presentation for a customer so it is recommended.

Practice, Practice, Practice - Read, Read, Read

You need to get in touch with the Google cloud console, Google provide you with a 300 USD free credits for new customer, so you can create your account and start testing and playing.

Also, you can create an account for CloudSkillboost (Qwiklabs)

You can find hundred of guided labs, with durations from 30 min to 2 hours ,some are free and some will cost you a certain amount of credits.
Labs are grouped on path so you can go specifically to the ACE or PCA paths.
Infrastructure is created for you at the begining each lab so you do not need to worry about spending your Google Cloud credits. And it is destroyed when you end the lab, so do not store anything on it.

Specially for ACE test you need to practice, get confident configuring each services and read the documentation from the Console, sometimes it provides you with informatiion that is asked during the test

This is an example of documentation from the console when creating a cloud run service

Ok practice a lot, got it, but why do i need to read?

Read is one of the activities that i enjoy the most, but is is a personal choice, the fact is that i study from the official learning guides and it helped me a lot to understand certain examples and architectures, so let see what are the guides for each exam:

For Associate Cloud Engineer:
- Official Google Cloud Certified Associate Cloud Engineer Study Guidefrom Dan Sullivan
- This Guide is from 2019, it hasn't been updated BUT as my best understanding it does not need to be updated, you can find example configuration from CLI that are the same right now.
- The book is around 500 pages long, but like 20 to 25% of the book are images from cloud console or diagrams.
- It is for sure that you find variations between images and current Console configuration but you can take advantage of this book from the explanations and practice questions.
- Language is easy to follow and the explanations are concise and direct to the point.
- The books is dividied in chapters by services, and it excels on showing specific use cases for each service and guide you to select the best option for a requirement.
For Professional Cloud Architect:
- Google Cloud Certified Professional Cloud Architect Study Guidefrom Dan Sullivan
- IMPORTANT there are 2 guides on Amazon, the one from 2019 and the new one from April 2022. Buy the new one, Professional Architect exam includes 4 study cases and the 2022 version has the new study cases so this will help you with the current version of the test.
- This book is around 300 pages long, and only like 2% are graphics so this is heavy reading.
- Explanations are simple to understand and include real scenarios use cases.
- The mos useful aspects of this Guide are:
  - There are 2 chapters where Dan explain the use cases and posible analysis to understand requirements, very useful for the test.
  - There is a chapter for Software development process.
  - There is a chapter for Compliance certifications, so if you have experience with compliance you can skip it.
  - Review question has some use case similar to the ones from the test.
  - There is little to nothing about configuration or commands son if you want more on that review the ACE Guide. This guide is base on the current exam so you can use this book as **a blueprint* and look other sources to dive deeper on the concepts presented here.

2 weeks previous the test date

The aspect exposed previously were for learning the basics, getting involved and confident with services and configuration and acquiring deep knowledg for each one.

In my Experience, the week or ween and a half previous to the exam, you need a different type of course. You need a course that has all included and also that provides you with some tips to remember during the test.
Looking for this type of trainings i found an exceptional course for each exam:

For Associate Cloud Engineer:
- GCP Associate Cloud Engineer - Google Cloud Certification from In28Minutes
- This training is about 17 hours, but includes all the subjects needed for the exam.
- There are several lectures with command example and also include code examples so you can test it yourself.
- The training does not just focus on the exam, it also provide examples and configuration for a standard day to day job.
- Under this training i found tips that were not available on the official guide and under the Pluralsight courses.

For Professional Cloud Architect:
- GCP Professional Cloud Architect: Google Cloud Certification from In28Minutes
- The training is about 20 hours of content, it includes some videos from the ACE course but the majority are PCA focused.
- There is a chapter to review the new case studies, this is gold.
- On the Exam there are questions about SREs, Agile, DevOps cuture. And this training has videos for all of that.

During the Test

I am going to talk a little about my experience with each exam.

NOTE i am not going to talk about the process to login remote proctored or in place, just my experience during the test.

For Associate Cloud Engineer:
- 50 Questions in 2 hours, i finished in 1 hour and 15 min.
- Questions were more related to configuration oiptions, selecting a product over another, CLI commands.
- On my test there were heavy focus on storage and database options, so take special care about BigQuery configuration or databases selection.
- This test will not ask anything as specific as : tell me with compute instance type have 16 VCPUs or what is the limit of pods in a cluster, but you need to know tha limitations because it can affect the service selection for the answer.
- The questions are short, the explanation of the cases or requirements are simple. take special attention of the requirements and the things that the client does not want under the configuration.
For Professional Cloud Architect:
- 50 Questions in 2 hours, i completed the test in 1 hour and 51 minutes.
- Know the use cases before presenting, this will help you to prevent spending time reading the use cases. study cases here
- In my test there were 2 cases from the 4 available, and around 14 questions about cases.
- Questions in this exam are long, because they are explaining the business and technical requirements so be prepare to read a lot. By the end of the test i felt tired and needed to re read questions.
- Take special care of business requirements, technical requirements can be met with different products but business requirements are the one that define the correct answer. Remember ARCHITECT test is more focus on Business.
- You can access the study cases at any time during the test.
- try not to loose time on questions, if you spend more than 5 min in a question it is better to move to the next one.

What else?

Well other recommendations are:

You can read some tips and key notes from this repo(WIP)
Visit the Google cloud reference architecture Center : here you can find architecture examples, blueprints, and use cases,
Read the blogs from medium, specially the ones written in Google Cloud community, somo of this provide real world scenarios and configurations --> HERE
Commit to study everyday, this will help to retain concepts and easier the process. the week before the test i spent around 4 to 5 hours a day studying so be prepare for this marathon.

Learning GO (Golang): First steps and the mighty "Hello world"

Ernesto Lopez — Tue, 01 Feb 2022 21:26:28 +0000

These blog post are going to be me showing how i am learning Go, you are going to be part of my travel and i will take advantage of all of you by making sure i am learning the concepts. SO let's start this Journey.

A little background

I always like to start by understanding why something is used for, what are the possibilities with certain technologies or languages and who is using it.

Let's get started by writing some of the main characteristics of the Go language:

Open Source programming language.
It was created at Google in 2007, anounced in 2009 and was released publicly in 2012 (So it has been out there for a while). And was designed thinking in merge the strenghts of C language and the easy syntax of languages like python.
It is a compiled, statically typed language. These two aspects represents a really good learning opportunity as i come from python.
Rich package library. Well, it is less rich than python's library but it is growing.
It was designed thinking on performance, it will use your resources as efficiently as possible, specially CPU.
Provides Goroutines and channels to deal with concurrency.
Kubernetes is written in GO. As a DevOps engineer this is really interesting.
Go generates binaries for your applications with all the dependencies built in, avoiding the need to install runtimes, and also this make GO multiplatform, so code developed on linux can run on MAC or WIndows only by having GO installed on those systems.
It is in the TOP 10 of the most loved languages in stackoverflow

All these features look nice, but then i moved to another question, what is GO used for??

According to GO official website you can use Go for:

Source:GO Use cases

What jumps to my attention was the cloud aspect, so digging a little further i have found reference about using GO for:

Networking automation
CLoud scripts
MAchine learning, and several engineer has been using GO for Data Science.

But related to this aspect, the most interesting part is that GO really excels on infrastructure stuff, this is why Docker, Kubernetes and Prometheus are build on GO.

Really cool features and use cases, but who is using it?

This was my third question, i wanted to know if learning Go will help me on my current job or if it will helpful in the future.

So let's travel again to the oficial GO website and look who is using the language.

Source:Who is using GO

Very interesting, Google was the creator so it should be using it, and the rest of companies using it are important ones. These companies using this language let us know how important it is, and taking a look in pages like LinkedIn you can also notice an increment on GO developers requirements.

Now, i was convinced, this was a good technology to learn, so i started this journey.

First of all, install Go

I have always think that infrastructure and architectura subjects can be learn by studying, but coding requires practicing. THe first step is to install GO, get in touch with the process and lear where are the bins stored and how the language works.

As i saw in several sites, it is recommended to use GO on Linux, it is not mandatory just a recommendation. I will be installing GO 1.17 (*At the date of this post, version 1.18 i think was already release, but this 1.17 is stable).

For the OS, i am using Using Ubuntu 20.04 on a laptop and for coding i am using Visual Studio Code, i have more experience using this software and the GO plugin is really good.

Let's get started, i follow the steps on the official documentation for Linux installation

After the software was downloaded i follow this steps:
GO Installation guide

Note you can download your installer anywhere you want, but you should decompress it on the route /usr/local/bin

** Note ** using this steps yo should log out and them login back so the .profile get the changes on the path.

You can test the installation was correct by running:

(base) ernestol@ubuntu:~/go$ go version
go version go1.17.6 linux/amd64

Now you need to configure your environment for developement

After installing go i created 3 directories under home/go (you should also create the go folder)

(base) ernestol@ubuntu:~$ mkdir go
(base) ernestol@ubuntu:~$ cd go/
(base) ernestol@ubuntu:~/go$ mkdir bin pkg src
(base) ernestol@ubuntu:~/go$ ls
bin  pkg  src

You may ask, why this folders, well i used them for:

bin is where all executables that we create are going to be stored.
pkg is where dependencies packages will be stored
src is where we will store our code

After this steps we can go to the powerful "Hello world!!"

THe Mighty "Hello world!!"

I started by creating a directory inside home/go/src with my hello world project.

(base) ernestol@ubuntu:~$ cd go/
(base) ernestol@ubuntu:~/go$ ls
bin  pkg  src
(base) ernestol@ubuntu:~/go$ cd src/
(base) ernestol@ubuntu:~/go/src$ mkdir basic_golang
(base) ernestol@ubuntu:~/go/src$ cd basic_golang/
(base) ernestol@ubuntu:~/go/src/basic_golang$ pwd
/home/ernestol/go/src/basic_golang

Here we are going to start coding, it is recommended to name the file main.go but i think it is not mandatory. The file it should have an extension .go (This is MANDATORY).

After creating this file we are going to add the following code:

package main

import "fmt"

func main() {
    fmt.Println("HELLO WORLD!!")
}

What looks nice to me that you do not event have to write the import statement, when you save the file, go automatically add you the import because it recognize that you are using the fmt and that you need to import it, this was really nice.

to execute it, you must first compile it (using go build):

(base) ernestol@ubuntu:~/go/src/basic_golang$ ls
src
(base) ernestol@ubuntu:~/go/src/basic_golang$ go build src/main.go

After this step you will notice a new file named main this is the one you will execute

(base) ernestol@ubuntu:~/go/src/basic_golang$ ls -al 
total 1740
drwxrwxr-x 3 ernestol ernestol    4096 feb  1 15:16 .
drwxrwxr-x 3 ernestol ernestol    4096 feb  1 15:10 ..
-rwxrwxr-x 1 ernestol ernestol 1766430 feb  1 15:16 main
drwxrwxr-x 2 ernestol ernestol    4096 feb  1 15:13 src
(base) ernestol@ubuntu:~/go/src/basic_golang$ 


(base) ernestol@ubuntu:~/go/src/basic_golang$ ./main 
HELLO WORLD!!
(base) ernestol@ubuntu:~/go/src/basic_golang$

Another thing that looks very interesting to me was that if you write something that is not along with the syntax best practices, the VS Code plugin will let you know and go will not compile, let see this in action.

Let's say we want the code to appear nice by adding { in a new line:

package main

import "fmt"

func main() 
{
    fmt.Println("HELLO WORLD!!")
}

VS Code will put it as an error:

AN if you try to compile the code you will receive the following errors:

(base) ernestol@ubuntu:~/go/src/basic_golang$ go run src/main.go 
# command-line-arguments
src/main.go:5:6: missing function body
src/main.go:6:1: syntax error: unexpected semicolon or newline before {

SO you better pay attention to those errors,

Also you can notice that even if you add new lines Go will rearrange the code for you, a nice feature if you ask me!

FInally, if you want to run the code as a test without the need to compile it you can run:

(base) ernestol@ubuntu:~/go/src/basic_golang$ go run src/main.go 
HELLO WORLD!!

Note take into account that this option provide less performance but it is a good way to test a change quickly.

THIS WAS MY FIRST BABY STEP
I will be posting more of this to document my journey on this language.

Sources:

Go Official website
GO Blog
A cloud Guru: What is Go? An intro to Google’s Go programming language (aka Golang)
PLatzi training: Curso Básico de Programación en Go
TechWorld with Nana Golang Tutorial for Beginners | Full Go Course
Seven Golang Features you must know about

Automating log downloads from MongoDB Atlas

Ernesto Lopez — Fri, 28 Jan 2022 19:37:35 +0000

MongoDB Atlas can provide 2 types of logs:

Process Logs also known as server logs, these logs are printed in JSON. Process log can include entries such as issues, connections, etc. Messages are similar to this:

{"t":{"$date":"2020-05-01T15:16:17.180+00:00"},"s":"I", "c":"NETWORK", "id":12345, "ctx":"listener", "msg":"Listening on","attr":{"address":"127.0.0.1"}}

Source: MongoDB LOG MESSAGES

Audit Logs these logs must be enabled and allows to audit any users actions inside the mongoDB Atlas cluster, like issued command, source IP, etc.

Information about Enabling auditing logs

These two types of logs has several aspects to consider:

Logs are only available for 30 days, so in case you need more logs you need to have a process in place that can download these logs files. You can manually download these files, but let be honest, you will forget to do it from time to time.
Logs are downloaded in .gz
Logs are generated for each node inside the cluster, if you have a cluster with 3 nodes you will have 3 log files. The bigger log file will be the one corresponding to the master node.
You must required project read access at a minimum to download log files, but in order to enable audit logs you require at least project owner.
Cluster types M0 and M2/M5 does not provide downloadable logs, so you cannot replicate these blog entry on your sandbox or free cluster.

More about manually downloading log files from MongoDB Atlas HERE

As we talked before...

Downloading each log file on daily basis can be time consuming at least, and a really difficult to maintain activity.

THe simplest thing you can do is automate this file download using an script, and it is what we are going to do. We are going to suppose the following:

We have our cluster on MongoDB Atlas.
We are using AWS as a cloud provider. Maybe in the future i will adapt this code for other cloud providers, but this time we are going to use some services from AWS.
THis is a simple script that will run on a daily basis inside a Linux EC2 Instance.
The user will assing an Instance profile so our EC2 Instance will have access to the following AWS Resources:
- AWS S3 to store the log files
- AWS Parameter Store to store the key that allows us to connect to mongo DB Atlas cluster.
MOngoDB Atlas has a connection with your AWS Infrastructure using one of the following:
- VPC Peer connection, more about configure peer connection on MOngo Atlas HERE
- Adding a public IP to your instance and enablig this public IP inside MOngoDB Atlas HERE THIS IS NOT RECOMMENDED

But first, you need some pre requisites

Create the EC2 Instance and install
- Mongo CLI.
- AWS CLI AWS CLI is already installed in AMazon Linux.
Create a S3 Bucket where logs are going to be stored.
Create a PArameter store to keep the MOngo Atlas key.
Define an instance profile with the following permissions:

s3:Read*
s3:Write*
"ssm:GetParameters"

Note Remember to use the specific ARN of your resources, when providing the resource.

Generate an API KEY to connect to your cluster

And assign the permissions:

Store the public key and private key generated inside your parameter stores. REmember to store the values in a key:value fashion.

{
  cluster_id: "your cluster id",
  public_key: "your public api key from atlas",
  private_key: "your private api key from atlas"
}

NOw with all these in place...

Let's move to our code:

#! /bin/bash -e

CLUSTERID=$(aws ssm get-parameters --names "mongodb-atlas-key" --with-decryption --query 'Parameters[*].Value' --output text | grep "cluster_id" | cut -f2 -d ":" | cut -d "\"" -f2)

PUBLICKEY=$(aws ssm get-parameters --names "mongodb-atlas-key" --with-decryption --query 'Parameters[*].Value' --output text | grep "public_key" | cut -f2 -d ":" | cut -d "\"" -f2)

PRIVATEKEY=$(aws ssm get-parameters --names "mongodb-atlas-key" --with-decryption --query 'Parameters[*].Value' --output text | grep "private_key" | cut -f2 -d ":" | cut -d "\"" -f2)

CURRENTDATE=`date +%Y%m%d`
NOW=`date '+%F_%H:%M:%S'`
declare -a StringArray=("node-00-00.snvtr.mongodb.net" "node-00-01.snvtr.mongodb.net" "node-00-02.snvtr.mongodb.net")

for hostname in ${StringArray[@]}; do
  echo "Obtaining logs from ${hostname}"
  curl --user ${PUBLICKEY}:${PRIVATEKEY} --digest \
    --header 'Accept: application/gzip' \
    --request GET "https://cloud.mongodb.com/api/atlas/v1.0/groups/${CLUSTERID}/clusters/${hostname}/logs/mongodb.gz" \
    --output "mongodb-${hostname}-${CURRENTDATE}.gz"

  echo "Uploading logs from ${hostname}"
  aws s3 mv "mongodb-${hostname}-${CURRENTDATE}.gz" s3://mongodb-logs/mongodblogs/${NOW}/mongodb-${hostname}-${CURRENTDATE}.gz
done

for hostname in ${StringArray[@]}; do
  echo "Obtaining Audit logs from ${hostname}"
  curl --user ${PUBLICKEY}:${PRIVATEKEY} --digest \
    --header 'Accept: application/gzip' \
    --request GET "https://cloud.mongodb.com/api/atlas/v1.0/groups/${CLUSTERID}/clusters/${hostname}/logs/mongodb-audit-log.gz" \
    --output "mongodb-audit-log-${hostname}-${CURRENTDATE}.gz"

  echo "Uploading Audit logs from ${hostname}"
  aws s3 mv "mongodb-audit-log-${hostname}-${CURRENTDATE}.gz" s3://mongodb-logs/mongodblogs/${NOW}/mongodb-audit-log-${hostname}-${CURRENTDATE}.gz
done

echo ""
echo "End of script execution..."

Let's see each part

#! /bin/bash -e

CLUSTERID=$(aws ssm get-parameters --names "mongodb-atlas-key" --with-decryption --query 'Parameters[*].Value' --output text | grep "cluster_id" | cut -f2 -d ":" | cut -d "\"" -f2)

PUBLICKEY=$(aws ssm get-parameters --names "mongodb-atlas-key" --with-decryption --query 'Parameters[*].Value' --output text | grep "public_key" | cut -f2 -d ":" | cut -d "\"" -f2)

PRIVATEKEY=$(aws ssm get-parameters --names "mongodb-atlas-key" --with-decryption --query 'Parameters[*].Value' --output text | grep "private_key" | cut -f2 -d ":" | cut -d "\"" -f2)

CURRENTDATE=`date +%Y%m%d`
NOW=`date '+%F_%H:%M:%S'`
declare -a StringArray=("node-00-00.snvtr.mongodb.net" "node-00-01.snvtr.mongodb.net" "node-00-02.snvtr.mongodb.net")

Under this section, we are declaring variables:

CLUSTERID: this value is obtained directly from the parameter store, the value is decrypted and cut.
PUBLICKEY: this value is obtained from the parameter store, decrypted and assigned to the variable.
PRIVATEKEY: this value is obtained from the parameter store using the aws cli command aws ssm get-parameters
CURRENTDATE and NOW, are variables that we set to store the log files inside the s3 bucket.
We also declare the array of cluster nodes names. YOu could also store this values in another parameter store, but as this is not critical you can download write it here.

As you may noticed, these approach allows us to keep critical information out of the code, and in case this information changes you will only need to make the change inside the parameter store. NEw values will be obtain on new executions.

for hostname in ${StringArray[@]}; do
  echo "Obtaining logs from ${hostname}"
  curl --user ${PUBLICKEY}:${PRIVATEKEY} --digest \
    --header 'Accept: application/gzip' \
    --request GET "https://cloud.mongodb.com/api/atlas/v1.0/groups/${CLUSTERID}/clusters/${hostname}/logs/mongodb.gz" \
    --output "mongodb-${hostname}-${CURRENTDATE}.gz"

  echo "Uploading logs from ${hostname}"
  aws s3 mv "mongodb-${hostname}-${CURRENTDATE}.gz" s3://mongodb-logs/mongodblogs/${NOW}/mongodb-${hostname}-${CURRENTDATE}.gz
done

THis for loop will go through the node names and download the process logs
THen it will store them locally in the instance, and after that it will move them to the s3 bucket using the date and the hostname.

BAsically, we are making an API CALL to mongoDB Atlas, and we are authenticating ourself using the key generated.

for hostname in ${StringArray[@]}; do
  echo "Obtaining Audit logs from ${hostname}"
  curl --user ${PUBLICKEY}:${PRIVATEKEY} --digest \
    --header 'Accept: application/gzip' \
    --request GET "https://cloud.mongodb.com/api/atlas/v1.0/groups/${CLUSTERID}/clusters/${hostname}/logs/mongodb-audit-log.gz" \
    --output "mongodb-audit-log-${hostname}-${CURRENTDATE}.gz"

  echo "Uploading Audit logs from ${hostname}"
  aws s3 mv "mongodb-audit-log-${hostname}-${CURRENTDATE}.gz" s3://mongodb-logs/mongodblogs/${NOW}/mongodb-audit-log-${hostname}-${CURRENTDATE}.gz
done

THis for loop will go through the node names and download the audit logs
THen it will store them locally in the instance, and after that it will move them to the s3 bucket using the date and the hostname.

You can grab this script and put it inside a cronjob to be executed every day.
We are not defining the time frame for the logs, by default is 24 hours, by executing this script daily we are donwloading logs from the last 24 hours.

Programmatic API Keys

Do you want to work with env? let's talk about Conda - the basics

Ernesto Lopez — Thu, 20 Jan 2022 20:45:55 +0000

On Python, most of the time, it is a good practice to work with environments, simply because there are different packages or versions to work with between projects, and maintain all that dependencies can be a nightmare on a single environment. There are different options like venv but today we are going to talk a little aboud CONDA

Conda is an open source environment management systems that can be used to install and manage packages too. Ti can be installed on Mac, Linux or Windows, and also provide an Enterprise version

Think of it like having several minicomputers inside your computer.

Basically, you can install any package inside the Anaconda repo

It works with any language including Python, R, Java, JavaScript, etc. But today we are going to work some examples in Python.

Another aspect to consider, you will need to install miniconda or Anaconda to use Conda and the installation will get you also a version of python.

For this example i am going to use:

Anaconda
Ubuntu 20.04

Note make sure you haver installed wget on your ubuntu

Frist of all we are starting by downloading the anaconda script:

wget -O anaconda.sh https://repo.anaconda.com/archive/Anaconda3-2021.11-Linux-x86_64.sh

HERE you will find the list of all the available installers

Then, move to the directory where your anaconda.sh file was stored and execute:

bash anaconda.sh
 #After installantion successfully ends 
conda init

If conda init executed correctly you will see something like this on your prompt:

(base) ernestolopez@mypc:~$

(base) will let you know that conda was successfully installed but if you want to make sure, just run conda info
That will provide you with information such as:

active environment
env location
conda version
python version
user config file

And now we are going to see some useful commands to work with conda.

BASIC COMMANDS TO WORK WITH CONDA

We will start by listing our environments

(base) ernestolopez@mypc:~$ conda env list
# conda environments:
#
base                  *  /home/ernestolopez/anaconda3

As we can see, we only have one environment, so how about if we create another one:

(base) ernestolopez@mypc:~$ conda create --name env2
Collecting package metadata (current_repodata.json): done
Solving environment: done

 ## Package Plan ##

  environment location: /home/ernestolopez/anaconda3/envs/env2



Proceed ([y]/n)? y

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
 #
 # To activate this environment, use
 #
 #     $ conda activate env2
 #
 # To deactivate an active environment, use
 #
 #     $ conda deactivate

If we analize this output, we obser two new commands:

conda activate env2 will activate the new environment and automatically we are going to be inside of it
conda deactivate will deactivate the current environemnt, befor you can delete any environment you will need to deactivate it first.

OK, Ok

So we created an environment, but we didn't tell the environment that we wanted new packages or any different version right?
Let's create a new environment with an specific python version (3.6) and a two new packages.

(base) ernestolopez@mypc:~$ conda create --name=env-demo-36 python=3.6 astor

 #you can add new packages just by adding a space and the name of the package in anaconda repo

When prompt, answer yes

AN now we will get into this new environment and see the packages listed

(base) ernestolopez@mypc:~$ conda activate env-demo-36
(env-demo-36) ernestolopez@mypc:~$ 

(env-demo-36) ernestolopez@mypc:~$ conda list
# packages in environment at /home/ernestolopez/anaconda3/envs/env-demo-36:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
_openmp_mutex             4.5                       1_gnu  
astor                     0.8.1            py36h06a4308_0  
ca-certificates           2021.10.26                         h5101ec6_17  
openssl                   1.1.1m               h7f8727e_0  
pip                       21.2.2           py36h06a4308_0  
python                    3.6.13               h12debd9_1  
readline                  8.1.2                h7f8727e_1  
setuptools                58.0.4           py36h06a4308_0  
sqlite                    3.37.0               hc218d9a_0  
tk                        8.6.11               h1ccaba5_0  
wheel                     0.37.1             pyhd3eb1b0_0  
xz                        5.2.5                h7b6447c_0  
zlib                      1.2.11               h7f8727e_4  


 #THis give you a complete list if we want to see a specific package we can use 

(env-demo-36) ernestolopez@mypc:~$ conda list python
# packages in environment at /home/ernestolopez/anaconda3/envs/env-demo-36:
#
# Name                    Version                   Build  Channel
python                    3.6.13               h12debd9_1

NOTE > We can start developing inside this environment by moving to the folfer of the env /home/ernestolopez/anaconda3/envs/env-demo-36 and writing code .

We can update packages inside our environment:

(env-demo-36) ernestolopez@mypc:~/anaconda3/envs/env-demo-36$ conda update python
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/ernestolopez/anaconda3/envs/env-demo-36

  added / updated specs:
    - python


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    astor-0.8.1                |   py39h06a4308_0          47 KB
    ------------------------------------------------------------
                                           Total:          47 KB

The following NEW packages will be INSTALLED:

  tzdata             pkgs/main/noarch::tzdata-2021e-hda174b7_0

The following packages will be UPDATED


Proceed ([y]/n)? y

Downloading and Extracting Packages
astor-0.8.1          | 47 KB     | ##################################### | 100% 
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
(env-demo-36) ernestolopez@mypc:~/anaconda3/envs/env-demo-36$ conda list python
# packages in environment at /home/ernestolopez/anaconda3/envs/env-demo-36:
#
# Name                    Version                   Build  Channel
python                    3.9.7                h12debd9_1  
(env-demo-36)

WE have just update Python, and now our beautiful name env-demo-36 is not valid anymore because we are not in 3.6 version

In Conda, you cannot rename an environment, so what we want to do es clone our environemnt with a new name, and delete the old one:

(env-demo-36) ernestolopez@mypc:~/anaconda3/envs/env-demo-36$ conda create --name env-demo-39 --copy --clone env-demo-36
Source:      /home/ernestolopez/anaconda3/envs/env-demo-36
Destination: /home/ernestolopez/anaconda3/envs/env-demo-39
Packages: 22
Files: 0
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate env-demo-39
#
# To deactivate an active environment, use
#
#     $ conda deactivate

(env-demo-36)

If you list both environments, you will see that both are similar.

It is time to close this basic tutorial, and now we are going to delete our environment:

(env-demo-36) elopez@winterfell:~/anaconda3/envs/env-demo-36$ conda env list
# conda environments:
#
base                     /home/ernestolopez/anaconda3
env-demo-36           *  /home/ernestolopez/anaconda3/envs/env-demo-36
env-demo-39              /home/ernestolopez/anaconda3/envs/env-demo-39
env2                     /home/ernestolopez/anaconda3/envs/env2

(env-demo-36) ernestolopez@mypc:~/anaconda3/envs/env-demo-36$ conda env remove --name env2

Remove all packages in environment /home/ernestolopez/anaconda3/envs/env2:

(env-demo-36)

If you list your environments again, it will not appear, but what happens if i try to delete my current environment?? *(env-demo-36) *

It will provoke an error, remember i just told you, you need to deactivate your env previous to delete it.

(env-demo-36) ernestolopez@mypc:~/anaconda3/envs/env-demo-36$ conda env remove --name env-demo-36

CondaEnvironmentError: cannot remove current environment. deactivate and run conda remove again

(env-demo-36) ernestolopez@mypc:~/anaconda3/envs/env-demo-36$ conda deactivate
(base) ernestolopez@mypc:~/anaconda3/envs/env-demo-36$ conda env remove --name env-demo-36

Remove all packages in environment /home/ernestolopez/anaconda3/envs/env-demo-36:

AND THAT's IT!!

Hope this information was useful!!!

Working with events in Oracle Cloud Infrastructure Part 1: service basics

Ernesto Lopez — Tue, 11 Jan 2022 22:55:21 +0000

Oracle Cloud Infrastructure Events is a service that allows you to create some automation base on change of state in a service, or maybe based in some sort of input received. Now, let get a step back, and Event is the occurrence of a particular situation. For this situations to be useful, events needs to be with some kind of information, for example, instance name, object name, status code, etc. The information can provide the automation something to work with.

Events are the base of serverless architecture, sometimes also referred to event-driven architecture. Additionally, events are crucial for contemporary software architectures as enables to decouple services, help to make software asynchronous calls and facilitate scaling processes.

An example of an event driven architecture:

You develop an image processing app.
Everytime a new image is uploaded to a bucket it triggers an event.
This event calls a function that grab that image and create a thumbnail from it.
The thumbnail is stored in another bucket.
This triggers another event that sends an email notification to the client.

NOTE OCI function is a serverless solution from OCI, that allows you to run a single purpose software without provisioning hardware or VMs, the software runs base on an event or a schedule job, it is like having functions as a service. More from Functions HERE

Getting back to OCI Events, some of the important characteristics are:

Event service use JSON object to define the event rules. You can think of a rule like the filter that will determine which events are important to consider, to determine what are going to be your inputs. Basically, Rules triggers actions. An example of a rule logic can be:

MATCH event WHERE (
  eventType EQUALS ANY OF (
  com.oraclecloud.computeapi.launchinstance.end
  )
)

This event will trigger when a new compute instance is created

But...

Where is the JSON part??
On the event, itself, let see an example of an event:

{
  "eventType": "com.oraclecloud.computeapi.launchinstance.end",
  "cloudEventsVersion": "0.1",
  "eventTypeVersion": "2.0",
  "source": "ComputeApi",
  "eventTime": "2019-08-16T12:07:42.794Z",
  "contentType": "application/json",
  "data": {
    "compartmentId": "ocid1.compartment.oc1..unique_ID",
    "compartmentName": "example_compartment",
    "resourceName": "my_instance",
    "resourceId": "ocid1.instance.oc1.phx.unique_ID",
    "availabilityDomain": "availability_domain",
    "additionalDetails": {
      "imageId": "ocid1.image.oc1.phx.unique_ID",
      "shape": "VM.Standard2.1",
      "type": "CustomerVmi"
    }
  },
  "eventID": "unique_ID",
  "extensions": {
    "compartmentId": "ocid1.compartment.oc1..unique_ID"
  }
}

This is an example of an instance that have been just created, and will trigger the event.

You can add conditions or filters to further narrow your events.

During an event rule configuration, you can add some attributes to further filter your results

On this example we are filtering results from the event type, for changes only on Sandbox and dev compartments, the whole logic looks like:

MATCH event WHERE (
  eventType EQUALS ANY OF (
  com.oraclecloud.computeapi.launchinstance.end
  )
  AND (
  compartmentName MATCHES ANY OF (
  Sandbox,
  dev
  )
 )
)

Additional to this, we can add Filter conditions based on Tags

MATCH event WHERE (
  eventType EQUALS ANY OF (
  com.oraclecloud.computeapi.launchinstance.end
  )
  AND (
  compartmentName MATCHES ANY OF (
  Sandbox,
  dev
  )
  definedTags INCLUDES ANY OF (
  Oracle-Tags.CreatedBy.elopez
  )
 )
)

Visually something like:

Basically this event will trigger everytime an instance is launch on Sandbox OR dev compartments by the user elopez.

Rules most specify and action.

The main objective for the rules is to provoke something when they are triggered, otherwise will be useless.

Actions are responses defined for event matched

Actions can be created using:
Notifications send the messages to a notification service which can send it to endpoints that are subscribe to the topic

These could be: an email address, email group, slack channel, etc.

Streaming using this service you will ingest your events into a data streams to obtain further analysis and intelligence over this data.

Functions functions can be executed based on events received. Quick example:

you can have an event that triggers every time a new instance is created and this will execute a function that configure monitoring for that new instance.

You need to add permissions so the Events service can call the action services.

Take into consideration any action is deny by default on OCI so you will need to create a policy that permit the Events service to execute the actions.

Basic permission you will need is:

Allow service cloudEvents to use ons-topic in tenancy
Allow service cloudEvents to use functions-family in tenancy

This is for the Tenancy but you can assign it to an specific Compartment

Allow service cloudEvents to use ons-topic in compartment DEV
Allow service cloudEvents to use functions-family in compartment DEV

More about compartments HERE

I want to use OCI Python SDK, where should i begin?

Ernesto Lopez — Fri, 07 Jan 2022 20:00:33 +0000

SDK Refers to a set of software tools used to create software that allow us to manage a specific platform. These tools can include: libraries, processes, documentation, etc.

The OCI python SDK allows us to write code to manage resources in Oracle cloud

You can download the SDK from:

The easiest way is using the pip install, by executing:

pip install oci

Also, remember to add this line to your python code, so you can be able to use the SDK

>>>import oci

If you have follow until here, you may encounter an error, when trying to use any command on the SDK, and why is that? - well, in order to connect to OCI resources, you need:

An OCI Account
A user created in that account, in a group with a policy that grants manage permissions or lower. Examples on policies can be found here
Configure OCI CLI profile on your local computer

I have some good news, you can create a free account in Oracle Cloud and get 300USD credits to test resources. Instructions HERE

Second, configure the CLI is easy you just need to (This is for MAC):

 #Update brew and install cli
brew update && brew install oci-cli

 #verify oci installation
oci --version

Take a look into the Homebrew documentation

Before using the CLI, you need to configure the config file that will contain the required credentials and information for working with Oracle Cloud Infrastructure. By default this file is stored in : ~/.oci/config but you can change it.

So, in order to generate the config file you need to:

 #Move to your home directory and create the .oci folder
mkdir .oci

 #move to the folder and create the config file
cd .oci
touch config

Now, you can vi this file and enter something similar to this:

[ADMIN_USER]
user=ocid1.user.oc1..<unique_ID>
fingerprint=<your_fingerprint>
key_file=keys/admin_key.pem
tenancy = ocid1.tenancy.oc1..<unique_ID>
region = us-phoenix-1

[ADMIN_USER] > you can name this anything youu want, but remember the name as you will use it with Python sdk.
user > here you need to enter the user ocid for the IAM user you created at the beginning. OCID is the unique resource identifier that Oracle cloud infrastructure provide to each resource
fingerprint > refers to the fingerprint of the public key you configure to your user. All the relevant information related to his can be found here
key_file > the .pem file you generated. You should use the complete path if your keys are located in a different directory /Users/elopez/.ssh/admin_key.pem. Detail information for How to Generate an API Signing Key
tenancy > your tenancy OCID. Details on how to obtain my tenancy OCID
region > the region that you are subscribed to (region identifier), Regions and Availability Domains

NOW, WE can start testing some stuff with python SDK.

First you need to establish the connection with OCI, and provide your software with the credentials that will be using.

>>> config = oci.config.from_file(
...     "~/.oci/config",
...     "ADMIN_USER")

Another approach could be to store this information in a .env file (that you should include into your .gitignore)
and reference the file into your code

import oci
import os

config = oci.config.from_file(os.environ.get("CONFIG_PATH"), os.environ.get("OCI_PROFILE"))

and your .env file can look something like this:

CONFIG_PATH = "~/.oci/config"
OCI_PROFILE = "ADMIN_USER"

This is the minimum required to connect with OCI, and will help you establish connection with other services as for example compute:

 # Initialize compute client with default config file
compute_client = oci.core.ComputeClient(config)

Or for the monitoring service

 # Initialize compute client with default config file
monitoring_client = oci.monitoring.MonitoringClient(config)

And now, you can use this to get, for example, a list of compute instances inside a container:

compartment_id_selected = os.environ.get("COMPARTMENT_ID")

list_instances_response = compute_client.list_instances(compartment_id=compartment_id_selected, sort_order="DESC", lifecycle_state="RUNNING")

These are the basic steps to start working with OCI python SDK.

Additional Resources

More about OCI Containers
My personal GITHUB OCI REPO

Oracle has develop a complete API reference and also it helps you by providing code examples:

ORACLE SDK Reference. Look for the API Reference on the left pane menu. Examples can be seen like this one:

# URL > https://docs.oracle.com/en-us/iaas/tools/python-sdk-examples/2.53.1/core/update_instance_configuration.py.html

# This is an automatically generated code sample.
# To make this code sample work in your Oracle Cloud tenancy,
# please replace the values for any parameters whose current values do not fit
# your use case (such as resource IDs, strings containing ‘EXAMPLE’ or ‘unique_id’, and
# boolean, number, and enum parameters with values not fitting your use case).

import oci

# Create a default config using DEFAULT profile in default location
# Refer to
# https://docs.cloud.oracle.com/en-us/iaas/Content/API/Concepts/sdkconfig.htm#SDK_and_CLI_Configuration_File
# for more info
config = oci.config.from_file()


# Initialize service client with default config file
core_client = oci.core.ComputeManagementClient(config)


# Send the request to service, some parameters are not required, see API
# doc for more info
update_instance_configuration_response = core_client.update_instance_configuration(
    instance_configuration_id="ocid1.test.oc1..<unique_ID>EXAMPLE-instanceConfigurationId-Value",
    update_instance_configuration_details=oci.core.models.UpdateInstanceConfigurationDetails(
        defined_tags={
            'EXAMPLE_KEY_4bccp': {
                'EXAMPLE_KEY_l4nah': 'EXAMPLE--Value'}},
        display_name="EXAMPLE-displayName-Value",
        freeform_tags={
            'EXAMPLE_KEY_s14GL': 'EXAMPLE_VALUE_ZZgDFtoA0GvgolAJlyPw'}),
    opc_retry_token="EXAMPLE-opcRetryToken-Value",
    if_match="EXAMPLE-ifMatch-Value")

# Get the data from response
print(update_instance_configuration_response.data)

My personal recommendations to get the OCI Oracle Cloud Infrastructure Architect Associate certification

Ernesto Lopez — Fri, 31 Dec 2021 23:31:40 +0000

On December 30th, I achieved the Oracle Cloud Infrastructure Architect Associate certification after passing the exam OCI Architect 2021 Associate [1Z0-1072-21], just 3 month after even knowing that Oracle had their own Cloud computing offer, and here i plan to write my personal recommendations so you can prepare to this exam.

First of all, you need to know that the exam is not a walk in the meadow, most of the question has two or three possible options but only one is the best, another questions are related so you know that if you have one wrong the other one is wrong to (Or yo can play varying the answers and see what happens).

With certifications, my personal recommendation is to completely avoid exam dumps, Maybe you will find the answers but you are not learning and something even worst, you will lower the appreciation level for that certification and will affect people that studied for their tests.

Now, let start:

Open an OCI free account, instruction here. You will need to practice, practice, practice. Or at least get to know where the services are located and what are the options requested to configure them. Also, The web console is constantly under development, features are added or updated.
Start by studying for the Oracle Foundation certification and take the OCI Foundations 2021 Associate [1Z0-1085-21] test. The training videos under Oracle University are well explained, and contain all the information and examples you will need. You can start HERE
After achieving the foundation certification, take the trainings for the Associate level, again, the video trainings from Oracle are complete and contain all the information you required for the test. Training videos can be found HERE
Follow up the practice videos in your own account, nothing will help you more than doing it yourself.
Read the services FAQs pages, this will provide in depth information about compute instances, storage services, database services, etc. Some questions can be easily resolved knowing the information that appears on those resources.
Exposure is important, besides practicing you can also see the youtube videos from Oracle learning channel, they developed an Oracle Cloud Infrastructure Certified Architect Associate - 1Z0-1072 Exam Preparation video path that contains useful information for day to day job.
Personally, I like to complement my learning with some reading so i bought the following book Oracle Cloud Infrastructure Architect Associate All-In-One Exam Guide (Exam 1z0-1072). This book was developed for a previous version of the exam, but provides some useful commands for day to day job, that could also help you understand some concepts. Example, to upload a multipart file with 5 parallel threads:

oci os object put -ns <object-storage-namespaces> -bn <bucket-name> --file <filepath/filename> --name <object_name_on_bucket> --part-size <size in MB> --parallel-upload-count 5
Take notes, this is important, you can write those notes on paper or use software, i took notes in a txt file using my IDE. Those notes can help you focus on the important subjects.
Last, but not least, go through the Practice Exam: Oracle Cloud Infrastructure Architect Associate Certification it contains useful information, and a couple of question can go to the final test or similar ones.

NOTE also a useful, but not needed resource are the blogs, specially the architecture related posts. This is not mandatory but it can provide you with extra knowledge.

I hope this help you to achieve this certification. Good Luck.

Quickops 3: too many connections to my database and what can i do

Ernesto Lopez — Thu, 23 Dec 2021 23:24:15 +0000

From time to time you may try to connect to your database cluster and receive an error similar to this:

mysqli_connect(): Too many connections

In this case, new services that try to connect to your database may start failing, and even your admin users will not get to the cluster.

So, How are we going to solve this issue?

In my experience you have one of the 3 following options:

Restart the cluster or service: This can provide potential risks of data loss and it doesn't even resolve the issue or provide useful information.
Wait unit a connection is releases and connect to your database. This could be a quick solution or it can take forever.
Check the services that connect to the cluster and release connections. Under this approach you need to review applications configuration's files and look for a service that is not critical and stop it by a moment so you can connect to the cluster with your admin user.

Now, suppose you manage to get a connection and enter the server or cluster.

The following query will provide you with information about users connected to the system and how many connections each user have:

select user, COUNT(*) as c from information_schema.processlist GROUP BY user ORDER BY c DESC LIMIT 10;

Or this one:

select user,count(*) as connections from INFORMATION_SCHEMA.PROCESSLIST group by user order by connections;

You can even see the status of the connections:

SELECT state, COUNT(*) FROM INFORMATION_SCHEMA.PROCESSLIST GROUP BY state;

This will give you the usernames that are consuming the majority of connections.

But we can go further and increase the connection count

mysql> show variables like "max_connections";
--------------
show variables like "max_connections"
--------------
show variables like "max_connections"
--------------
+-----------------+-------+
| Variable_name   | Value |
+-----------------+-------+
| max_connections | 100  |
+-----------------+-------+
1 row in set (0.08 sec)

set global max_connections = 500;

Consider that connections increase resources consumption so special care needs to be taken.

This second part is not a good approach...
Why?

Suppose there is a service that, after a new version release, opens connections to our database but it never closes the connections.
Even if you have more connections slots available eventually you will have this problem again.

So, A better approach is to:

Use the information obtainer to detect the service or services that are consuming most connections.
Review the service config files to detect the amount of connections that the service should use or what is the size of the pool configured.
If it is too high. Talk with the team to see if all that connections are necessary, or if it can be lowered down.
In case the connection pool size is a small number, contact the team it is probably a bug in the software.

If you are in an emergency and needs to release resources you can either stop a service or use this commands:

show full processlist;

 #Find a process that you can kill

kill <PID>;

Care with killing processes this is not a best practice until you have a better diagnose of the issue.

3 ways to use MongoDB without cost (Atlas, mtools, Docker)

Ernesto Lopez — Sat, 18 Dec 2021 01:39:23 +0000

MongoDB is a general-purpose no sql database which provides certain capabilities like replication between nodes, sharding, secondary indexes, range queries, sorting, aggregations, and geospatial indexes. MongoDB is a document-oriented database where rows (from relational database) is replaced by documents which can have any fields inside and you can have more information on one document versus another from the same database. This last point means that there is no predefined schemas.

At the heart of MongoDB there is a key:value so if you see a document representation is similar to JSON or map. And can be as simple as:

{"Salute": "Hello Peter!"}

So Documents are the basic form of data in mongoDB and a collection is similar to have a table.

That's all that we will talk about the basics for MongoDB, because this is a post of testing for free.

Next i am going to present 3 ways you can test your mongoDB database without incurring in any extra cost.

1. MongoDB Atlas Free Cluster

MongoDB Atlas offers a Database as a service with MongoDB Atlas, as part of their offer you can create a free cluster (Always free as long as you keep in this tier) that has similar functionality that a production one.

According to MongoDB Atlas documentation

Free clusters never expire, and provide access to a subset of Atlas features and functionality.

Some of the limitations are:

You can only have one free cluster per project.
You can't upgrade the MongoDB version.
M0 tier is the only one available to deploy free cluster.
Replication factor is 3 nodes, and yes, yo have 3 nodes on free cluster, so replication a similar to prod escenario is easy.
No predefined replica set tag.
you cannot test primary failover, configure encryption at rest or enable auditing.

Oh i forgot to mention, a ReplicaSet is a group of mongo processes (servers, VMs, Containers, etc) that maintain the same data set. At a minimum it is recommended to have a primary node and two replication or secondary node. Also you can have a Primary node, a secondary node and an arbiter (smaller process just to elect a new primary) but this is not recommended.

So your free tier cluster can be seen as this:

Shared, as can be seen in the image refers that the nodes are deployed on shared environment so you cannot configure disk size, memory size, etc.

And, How can you create your free cluster? yo may ask...

First of all go to MongoDB Atlas registration page

Here you have the option to fill al the fields or even better use your google account to sign up, and that's it.

Now, to create the free cluster:

Click on Build a Cluster
Select the option of Starters Cluster or Shared Cluster.
Select your cloud provider. Not all providers or regions support the free cluster. I personally recommends use AWS with us-east-1 region.
Select M0 Sandbox tier as this corresponds to the free tier.
Enter a name for your cluster
Click on Create Cluster and that is all you need to create your cluster, this process will take around 10 min, so you can go and get a cup of coffee.

So, after you cluster is ready, the First Thing you need to do is provide access to your local IP. MongoDB Atlas use a whitelisting approach for this.

On your MongoDB Atlas interface, move to your project and under security click on Network Access

Then, click on Add IP Address
You have the option on enter and IP in CIDR notation, example 188.188.188.21/32
Another option is to provide access to any IP by using 0.0.0.0/0
Or Click Add Current IP Address and automatically will add your public IP.

Note you can even configure a temporary access for 6 hrs, 1 day or 1 week.

Next Step is to create a user to access the cluster , this is different from the user you created at the beginning, this new user is for database access, the previous one was for MongoDB Atlas platform administration.

Enter the important information and provide the permissions through Built-in Role more info HERE

One of the main advantages

of MongoDB Atlas is that allows you to find the exact information you need in an easy way. In this case, to test the access we will connect using mongo shell from our laptop.

you can find the connection URI here

On the next page, select Connect using MongoShell and it will show you something similar to:

Note it also provides the required information to download the mongo shell in your local system.

Note it also provides you with the connection URI and full driver code for several programming languages. Example:

Getting back to our example. Just add the username you created previously and add the ip when it is requested.

☁  .ssh  mongosh "mongodb+srv://cluster0.xxxxx.mongodb.net/myFirstDatabase" --username eltest01
Enter password: **********************
Current Mongosh Log ID: xxxxxxxxxxxxxxxxxxxxxxxx
Connecting to:      mongodb+srv://cluster0.xxxxx.mongodb.net/myFirstDatabase
Using MongoDB:      4.4.10
Using Mongosh:      1.1.7

For mongosh info see: https://docs.mongodb.com/mongodb-shell/


To help improve our products, anonymous usage data is collected and sent to MongoDB periodically (https://www.mongodb.com/legal/privacy-policy).
You can opt-out by running the disableTelemetry() command.

Atlas atlas-xxxxx-shard-0 [primary] myFirstDatabase> show dbs
m201    119 MB
admin   373 kB
local  7.16 GB
Atlas atlas-xxxxx-shard-0 [primary] myFirstDatabase>

2. Creating a docker container with mongodb official Image

Note The image that we use here is a community edition, MongoDB does provides an Enterprise Server that can include some cost.

First, we need to remember that container are self-contained without default access to the host, and also that containers can be considered ephemeral, so as soon as the container is terminated, any data inside of it will be lost.

If we want to have persistent storage we need to create a local directory in our machine and let the mongo container knows about it, in case the container dies a new one can connect to the local storage again.

So lets create a directory:

☁  ~  mkdir tmp
☁  ~  cd tmp
☁  tmp  mkdir datamongodb
☁  tmp

Now we need to obtain or path:

☁  datamongodb  pwd
/Users/ernestol/tmp/datamongodb

After this we can run a container with a bind mount.

According to Docker Documentation

When you use a bind mount, a file or directory on the host machine is mounted into a container. The file or directory is referenced by its absolute path on the host machine

☁  datamongodb  docker run -d --name mongodb01 -v /Users/ernestol/tmp/datamongodb:/data/db mongo
7d375ef665c5adfee4df4217ef9d7xxxxxxxxx41237032879a7ae99ee32d613a
☁  datamongodb  docker ps
CONTAINER ID   IMAGE     COMMAND                  CREATED          STATUS          PORTS       NAMES
7d375ef665c5   mongo     "docker-entrypoint.s…"   43 seconds ago   Up 42 seconds   27017/tcp   mongodb01

Note See that locally and on mongoDB Atlas we use the default port 27017 but you can change this.

Let get into the container to start the mongodb database to see what happens.

☁  datamongodb  docker exec -it mongodb01 bash

root@7d375ef665c5:/# ls /data/db
WiredTiger         WiredTigerHS.wt                       collection-4--3382512612996164992.wt  index-5--3382512612996164992.wt  sizeStorer.wt
WiredTiger.lock    _mdb_catalog.wt                       diagnostic.data                       index-6--3382512612996164992.wt  storage.bson
WiredTiger.turtle  collection-0--3382512612996164992.wt  index-1--3382512612996164992.wt       journal
WiredTiger.wt      collection-2--3382512612996164992.wt  index-3--3382512612996164992.wt       mongod.lock
root@7d375ef665c5:/# mongo
MongoDB shell version v5.0.4
connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("9c275138-c0a6-42d9-af2d-f63cac040318") }
MongoDB server version: 5.0.4
================
Warning: the "mongo" shell has been superseded by "mongosh",
which delivers improved usability and compatibility.The "mongo" shell has been deprecated and will be removed in
an upcoming release.
For installation instructions, see
https://docs.mongodb.com/mongodb-shell/install/
================
Welcome to the MongoDB shell.

        To permanently disable this reminder, run the following command: db.disableFreeMonitoring()
---
> show dbs
admin   0.000GB
config  0.000GB
local   0.000GB
> use test
switched to db test

> db.greets.insertOne({"salute": "Hello Peter!"})
{
    "acknowledged" : true,
    "insertedId" : ObjectId("61bd1e6caecda5e2f0f2932f")
}
>

And notice that those files inside /data/db/ are the sale that are in our local computer and represent the database files form mongodb

☁  datamongodb  ls
WiredTiger                           _mdb_catalog.wt                      diagnostic.data                      index-8--3382512612996164992.wt
WiredTiger.lock                      collection-0--3382512612996164992.wt index-1--3382512612996164992.wt      journal
WiredTiger.turtle                    collection-2--3382512612996164992.wt index-3--3382512612996164992.wt      mongod.lock
WiredTiger.wt                        collection-4--3382512612996164992.wt index-5--3382512612996164992.wt      sizeStorer.wt
WiredTigerHS.wt                      collection-7--3382512612996164992.wt index-6--3382512612996164992.wt      storage.bson
☁  datamongodb

Not you can kill your containers and create others an mount those files.

3. Create a demo replicaset using mtools

mtools is a collection of helper scripts to parse, filter, and visualize MongoDB log files (mongod, mongos). mtools also includes mlaunch, a utility to quickly set up complex MongoDB test environments on a local machine, and mtransfer, a tool for transferring databases between MongoDB instances.

Some of the mtools characteristics:

Written in Python. Requires Python 3.7 >
Not officially supported by MongoDB
Open Source
Only command line interface.
Currently being updated
Tested for mongodb 4.0 or newer.

You can clone the corresponding repo

There is a bunch of good scripts and tools in this repo but the one we want to focus on is the mlaunch

a script to quickly spin up local test environments, including replica sets and sharded systems (requires pymongo)

This required pymongo so make sure you have installed on your local computer.

☁ datamongodb pip3 install pymongo

And also you need to install the mtools

The easiest way to install mtools is via pip3

☁  mtools [develop] pip3 install mtools

Collecting mtools
  Downloading mtools-1.6.4-py3-none-any.whl (2.1 MB)
     |████████████████████████████████| 2.1 MB 1.9 MB/s
Collecting python-dateutil>=2.7
  Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Requirement already satisfied: six in /usr/local/lib/python3.9/site-packages (from mtools) (1.16.0)
Installing collected packages: python-dateutil, mtools

Successfully installed mtools-1.6.4 python-dateutil-2.8.2

You may found that some of the scripts required additional packages

Traceback (most recent call last):
  File "/usr/local/bin/mlaunch", line 5, in <module>
    from mtools.mlaunch.mlaunch import main
  File "/usr/local/lib/python3.9/site-packages/mtools/mlaunch/mlaunch.py", line 20, in <module>
    import psutil
ModuleNotFoundError: No module named 'psutil'

so you may need to install requires packages

IMPORTANT

MONGOD is necessary to run this tools.

mongod is the primary daemon process for the MongoDB system

This is differente from the mongo shell so make sure you install also mongod with

☁  mtools [develop] brew tap mongodb/brew
==> Tapping mongodb/brew
Cloning into '/usr/local/Homebrew/Library/Taps/mongodb/homebrew-brew'...
remote: Enumerating objects: 806, done.
remote: Counting objects: 100% (303/303), done.
remote: Compressing objects: 100% (217/217), done.
remote: Total 806 (delta 153), reused 140 (delta 83), pack-reused 503
Receiving objects: 100% (806/806), 176.54 KiB | 1.26 MiB/s, done.
Resolving deltas: 100% (391/391), done.
Tapped 14 formulae (30 files, 241.7KB).

☁  bin  brew install mongodb-community@5.0
Running `brew update --preinstall`...
==> Downloading https://fastdl.mongodb.org/tools/db/mongodb-database-tools-macos-x86_64-100.5.1.zip
######################################################################## 100.0%
==> Downloading https://fastdl.mongodb.org/osx/mongodb-macos-x86_64-5.0.4.tgz
######################################################################## 100.0%
==> Installing mongodb-community from mongodb/brew
==> Installing dependencies for mongodb/brew/mongodb-community: mongodb-database-tools
==> Installing mongodb/brew/mongodb-community dependency: mongodb-database-tools
🍺  /usr/local/Cellar/mongodb-database-tools/100.5.1: 13 files, 115.7MB, built in 8 seconds
==> Installing mongodb/brew/mongodb-community
==> Caveats
To start mongodb/brew/mongodb-community now and restart at login:
  brew services start mongodb/brew/mongodb-community
Or, if you don't want/need a background service you can just run:
  mongod --config /usr/local/etc/mongod.conf
==> Summary
🍺  /usr/local/Cellar/mongodb-community/5.0.4: 11 files, 181.4MB, built in 8 seconds
==> Running `brew cleanup mongodb-community`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
==> Caveats
==> mongodb-community
To start mongodb/brew/mongodb-community now and restart at login:
  brew services start mongodb/brew/mongodb-community
Or, if you don't want/need a background service you can just run:
  mongod --config /usr/local/etc/mongod.conf

And then run it by using:

☁  data [develop] ⚡  mlaunch init --replicaset nodes 3 --name mymongo --port 3000
launching: "mongod" on port 3000
launching: "mongod" on port 3001
launching: "mongod" on port 3002

Connect using

mongo --host mymongo/$(hostname):3000

Sometimes this could fail because the replicaset is not starting,

You can still launch a single one:

☁  data [develop] ⚡  mlaunch init --single --name mymongo --port 31001 --dir ./data2
A different environment already exists at /Users/ernestolopez/tmp/datamongodb/mtools/mtools/data/data2.
☁  data [develop] ⚡  mlaunch init --single --name mymongo --port 32001 --dir ./data3
launching: "mongod" on port 32001

☁  data [develop] ⚡  ls
data2 data3
☁  data [develop] ⚡  mlaunch list --dir data3

PROCESS    PORT     STATUS     PID

single     32001    running    81184

☁  data [develop] ⚡  mongo --host localhost:32001
\MongoDB shell version v5.0.4

        To enable free monitoring, run the following command: db.enableFreeMonitoring()
        To permanently disable this reminder, run the following command: db.disableFreeMonitoring()
---
>

Quickops 2: Configure Python SDK for OCI

Ernesto Lopez — Fri, 10 Dec 2021 20:24:45 +0000

Before you can start using python SDK for OCI you need to comply with the following requirements, and also certain information you need to have at hand:

Signing public and private key pair.
- OCI request the key to be on PEM format. you can use the following commands

openssl genrsa -out oci-api-key -aes128 2048
chmod go-rwx oci-api-key
openssl rsa -pubout -in oci-api-key -out oci-api-key.pub

Note It is required that the private key has permissions only to the file owner, otherwise you will receive an error when trying to use it on OCI.

Upload the public key to your user on OCI
- Move to Identity > Users > User Details > API Keys

Clic over Add API key - Here you will paste the content from the oci-api-key.pub file you just generated, and click over Add

Take note of the fingerprint that will appear on your console. OCI use fingerprint during authentication phase to review your keys.
Fingerprints looks something like this:

56:73:ff:44:8f:b3:c3:1d:58:12:a8:1f:ff:ff:aa:66

Take note of your user OCI
Take note of the tenancy OCID You can look it on the console (Menu > Administration > Tenancy Details) Remember that: tenant OCID is Root Compartment OCI

You can get more information about Compartments in OCID HERE

Get your Region Identifier This should be the region selected when the account was created, or in case you subscribed to a different region, it should be that region. Example: > region=us-phoenix-1

Now we can proceed to the SDK configuration

Python SDK

In simple words, an SDK is a library that allows a programming language to interact with Oracle cloud infrastructure resources.

For this case we are using Python SDK

1.- So, to install the SDK, first we need to make sure we have python installed.

Note Python SDK works with both version 2 or 3 but use the latest.

☁  .oci  python --version
Python 3.9.7
☁  .oci

On your case you may find that the versions is pointing to python 2. To set python 3 as default on MAC yo can consult one of these:

2.- Update pip as Python SDK for OCI is a PyPi module

Under the documentation:

This is the Python SDK for Oracle Cloud Infrastructure. Python 3.6, 3.7, 3.8 and 3.9 are supported.

And also as a note:

It is highly recommended that a Python virtual environment be used when installing oci.

Let follows the recommendations:

☁  oci-sdk-python  ls
☁  oci-sdk-python  python -m venv ocisdk
☁  oci-sdk-python  ls
ocisdk
☁  oci-sdk-python  source ocisdk/bin/activate


(ocisdk) ☁  oci-sdk-python  python -m pip install --upgrade pip
Requirement already satisfied: pip in ./ocisdk/lib/python3.9/site-packages (21.2.4)
Collecting pip
  Downloading pip-21.3.1-py3-none-any.whl (1.7 MB)
     |████████████████████████████████| 1.7 MB 1.6 MB/s
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 21.2.4
    Uninstalling pip-21.2.4:
      Successfully uninstalled pip-21.2.4
Successfully installed pip-21.3.1
(ocisdk) ☁  oci-sdk-python

We can proceed to install OCI SDK

(ocisdk) ☁  oci-sdk-python  pip install oci
...
Installing collected packages: pycparser, cffi, six, cryptography, pytz, python-dateutil, pyOpenSSL, circuitbreaker, certifi, oci
    Running setup.py install for circuitbreaker ... done
Successfully installed certifi-2021.10.8 cffi-1.15.0 circuitbreaker-1.3.2 cryptography-3.4.7 oci-2.52.1 pyOpenSSL-19.1.0 pycparser-2.21 python-dateutil-2.8.2 pytz-2021.3 six-1.16.0

3.- Create OCI Config file

If you remember, at the beginning i wrote some information you need to have at hand, well now is the moment to use that information.

There are 2 ways to setup your config file,

FIRST, by installing OCI CLI and running the setup dialog

brew update && brew install oci-cli
oci --version

oci setup config

Note More on the official documentation

SECOND, by manually creating the file under ~/.oci

cd
cd .oci
vi config

and inside the config file add the following:

[DEFAULT]
user = <user-ocid>
fingerprint = <fingerprint from your user API key >
tenancy = <tenancy-id or rootCompartment-id>
region = us-phoenix-1
key_file = <the full path to your private key file in pem format>

Note the file should have the following permissions
-rw-------

You can store several profiles on the same file.

4.- Finally!!! Using the SDK

If you deactivate your venv execute the source command again,
Under the venv run

(ocisdk) ☁  oci-sdk-python  python

And inside the Python CLI run

>>> import oci
>>> config = oci.config.from_file("~/.oci/config", "DEFAULT2")
>>> identity = oci.identity.identityClient(config)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'oci.identity' has no attribute 'identityClient'
>>> identity = oci.identity.IdentityClient(config)
>>> availability_domains_list = identity.list_availability_domains(config['tenancy']).data


>>> for availability_domain in availability_domains_list:
...     print('AD: ' + str(availability_domain.name))
...
//look for your results
>>>

You can find information on any known issues with the SDK here and under the “Issues” tab of this project’s Github Repository

DEV Community: Ernesto Lopez

List networking information configured during Google Cloud Foundation Setup

Gogle Cloud Foundation Setup

Networking in Foundation setup

List VPCs in a project

List VPCs in a project

List Firewall Rules associates with a VPC

List VPN Information

3 Aspects to consider when using Google Cloud Serverless VPC Access

IP Addressing

CLOUD NAT

My recommendations to approve Google Associate Cloud Engineer and Professional Cloud Architect certifications in 60 days

Start point > Base knowledge

So, if both path share most of the trainings why should a present two different certifications?

OKOK got it, but why Pluralsight?

Extra recommenation for base knowledge.

Practice, Practice, Practice - Read, Read, Read

Ok practice a lot, got it, but why do i need to read?

2 weeks previous the test date

During the Test

What else?

Learning GO (Golang): First steps and the mighty "Hello world"

A little background

Let's get started by writing some of the main characteristics of the Go language:

All these features look nice, but then i moved to another question, what is GO used for??

According to GO official website you can use Go for:

Really cool features and use cases, but who is using it?

First of all, install Go

Now you need to configure your environment for developement

THe Mighty "Hello world!!"

Sources:

Automating log downloads from MongoDB Atlas

These two types of logs has several aspects to consider:

As we talked before...

THe simplest thing you can do is automate this file download using an script, and it is what we are going to do. We are going to suppose the following:

But first, you need some pre requisites

NOw with all these in place...

Let's see each part

Programmatic API Keys

Do you want to work with env? let's talk about Conda - the basics

BASIC COMMANDS TO WORK WITH CONDA

OK, Ok

AND THAT's IT!!

Hope this information was useful!!!

Working with events in Oracle Cloud Infrastructure Part 1: service basics

An example of an event driven architecture:

Getting back to OCI Events, some of the important characteristics are:

But...

I want to use OCI Python SDK, where should i begin?

NOW, WE can start testing some stuff with python SDK.

Additional Resources

My personal recommendations to get the OCI Oracle Cloud Infrastructure Architect Associate certification

Quickops 3: too many connections to my database and what can i do

So, A better approach is to:

3 ways to use MongoDB without cost (Atlas, mtools, Docker)

1. MongoDB Atlas Free Cluster

One of the main advantages

2. Creating a docker container with mongodb official Image

3. Create a demo replicaset using mtools

IMPORTANT

Quickops 2: Configure Python SDK for OCI

Python SDK

1.- So, to install the SDK, first we need to make sure we have python installed.

2.- Update pip as Python SDK for OCI is a PyPi module

3.- Create OCI Config file

4.- Finally!!! Using the SDK