DEV Community: Srinivasa Vasu

Better Developer Experience: Getting Started with YugabyteDB on Gitpod

Srinivasa Vasu — Thu, 24 Feb 2022 23:34:57 +0000

Developer onboarding and experience gets simplified every day. But while developers use many modern software development techniques—including 12-factor, cloud native, and continuous integration—developer onboarding remains a challenge. Therefore, what developers need is an integrated, self-contained platform that helps them get started with ease.

Gitpod is one such way we can steer into that problem space to create a better developer experience. More specifically, Gitpod provides git-based, fully automated, integrated cloud-native development workflows with the prerequisites configured. This “GitDev” approach offers a preconfigured environment that seamlessly provides a consistent development environment for stream-aligned teams.

In this post, we’ll explore how to get started with YugabyteDB in a Gitpod-driven workspace covering the following workflows:

Creating a single-node instance
Creating a cluster with multiple nodes
Customizing the cluster configuration

First steps

Gitpod is a configurable, ready-to-code cloud development environment accessible via a browser. A Gitpod workspace includes everything we need to develop, such as the Visual Studio Code editor, common languages, tools, and utilities. Instantly, this sets up a cloud-hosted, containerized, and customizable editing environment that is ready to go.

Gitpod doesn’t require anything other than a code editor and Git CLI on your local computer. Therefore, much of the development happens in the cloud through a web browser. But you can refer to the QuickStart section to get started with Gitpod.

We’ll use GitHub as our source code repository. For starters, create a new empty repository yb-git-pod and clone that to the local workstation.

| git clone https://github.com/[user]/yb-git-pod.git

You can find the complete code in this GitHub repository.

Creating the base image

Let’s make a new file— .gitpod.Dockerfile —inside the yb-git-pod directory we cloned to the local workstation. The content of the YugabyteDB Dockerfile is as follows:

Creating a single-node instance

Next, let’s make a new file: .gitpod.yml. The content of this file is as follows:

Next, commit .gitpod.Dockerfile and .gitpod.yml to your GitHub repository. To initialize the Gitpod workspace, launch https://gitpod.io/#[REPO_URL] in a browser window. Replace [REPO_URL] with your repository URL.

A Gitpod workspace gets created based on the definition of the .gitpod.yml file. The “yugabyted start” statement can be part of the “before” or “command” task. These two tasks get executed during the initial workspace creation and re-initialization phases. YugabyteDB starts during the initialization phase, and upon completion, a terminal launches with the ysql shell prompt.

Gitpod manages the exposed ports in the Dockerfile definition. Specifically, it automatically creates the port-forwarding rules. We will get links to access the Web-UI right from this same interface.

Creating a cluster with multiple nodes

We need to update .gitpod.yml with the following multi-node configuration. The complete multi-node configuration spec is available in the .gitpod-cluster.yml file in the repository.

Finally, commit .gitpod.yml with this updated configuration to your GitHub repository and re-initialize the Gitpod workspace https://gitpod.io/#[REPO_URL].

Specifically, this configuration will initialize a three-node cluster based on the loopback interface configuration. The node initialization is sequenced properly using gp sync-done and gp sync-await calls.

Customizing the cluster configuration

The following config builds on top of the previous cluster configuration, providing GFlags and cluster-configuration customization. However, we have configured the cluster with custom placement info in the below snippet. The complete multi-node custom configuration spec is available in the .gitpod-cluster-config.yml file in the repository.

Finally, commit .gitpod.yml with this updated configuration to your GitHub repository and re-initialize the Gitpod workspace https://gitpod.io/#[REPO_URL].

The Web-UI console now reflects the custom configuration changes, as illustrated below:

Conclusion

Additionally, you can check out this article to learn more about integrating microservices with a Gitpod-powered YugabyteDB workspace to improve developer experience. Specifically, we have a pull request submitted for a dedicated YugabyteDB base image for Gitpod.

Finally, if interested, you can follow this pull request to track our progress. Either way, we hope you give this short tutorial a shot.

Any questions? Let us know what you think in the YugabyteDB Community Slack channel.

The post Better Developer Experience: Getting Started with YugabyteDB on Gitpod appeared first on The Distributed SQL Blog.

Cloud Native Java: Integrating YugabyteDB with Spring Boot, Quarkus, and Micronaut

Srinivasa Vasu — Wed, 16 Feb 2022 15:49:00 +0000

Java is the quintessential language runtime for enterprise applications built on monoliths, microservices, and modular architecture patterns. But when it comes to “Enterprise Java,” Spring is the de facto framework of choice.

The Spring ecosystem—with the simplicity of Spring Boot—has grown to provide integration touchpoints to a majority of the Java ecosystem. For starters, it offers a clean abstraction and “glue” code to build cohesive enterprise applications. However, as the ecosystem evolves, two newer frameworks are growing in popularity: Quarkus and Micronaut.

Spring Boot, Quarkus, and Micronaut are seeing massive adoption for cloud native greenfield applications, as well as brownfield modernization efforts. In this blog post, we look at how YugabyteDB’s YSQL smart driver integrates with all three popular microservices frameworks.

Prerequisites

Follow the YB Quickstart instructions to run a local YugabyteDB cluster. Test YugabyteDB’s YSQL API to confirm you have a YSQL service running on “localhost:5433”.
You will need JDK 11 or above. You can use SDKMAN to install the JDK runtime.

Getting started

You can find the complete source code in this GitHub repository. This project has directories for all three frameworks: spring-boot, quarkus, and micronaut. Clone this repository to a local workstation and open the “yb-ms-data” directory in your favorite IDE to easily navigate and explore framework-specific code.

git clone https://github.com/yugabyte/yb-ms-data.git

This source repository consists of a simple JPA-based web application with CRUD functionality. We will focus in this blog only on the database integration point.

Spring Boot

Spring Boot makes it easy to create stand-alone, production-grade Spring-based Applications that you can “just run.”

The following section describes how to build a simple JPA-based web application with the Spring Boot framework for YSQL API using the YugabyteDB JDBC Driver.

For starters, navigate to the springboot framework folder inside the project yb-ms-data directory.

cd yb-ms-data/springboot

Dependencies

This project depends on the following libraries:

implementation("org.springframework.boot:spring-boot-starter-web")
implementation("org.springframework.boot:spring-boot-starter-actuator")
implementation("org.springframework.boot:spring-boot-starter-data-jpa")
implementation("org.flywaydb:flyway-core")
implementation("org.springdoc:springdoc-openapi-ui:1.5.9")
implementation("com.yugabyte:spring-data-yugabytedb-ysql:2.3.0") {
   exclude(module = "jdbc-yugabytedb")
}
implementation("org.springframework.retry:spring-retry")
annotationProcessor("org.springframework.boot:spring-boot-configuration-processor")
annotationProcessor("org.projectlombok:lombok")
developmentOnly("org.springframework.boot:spring-boot-devtools")
compileOnly("org.projectlombok:lombok")
runtimeOnly("io.micrometer:micrometer-registry-prometheus")
implementation("com.yugabyte:jdbc-yugabytedb:42.3.3")
testImplementation("org.springframework.boot:spring-boot-starter-test")
testImplementation("org.flywaydb.flyway-test-extensions:flyway-spring-test:7.0.0")
testImplementation("com.yugabyte:testcontainers-yugabytedb:1.0.0-beta-4")
testImplementation("org.testcontainers:junit-jupiter:1.15.3")

Update the driver dependency library “com.yugabyte:jdbc-yugabytedb:42.3.3” to the latest version. Grab the latest version from YugabyteDB JDBC Driver.

Driver configuration

Refer to the file yb-ms-data/springboot/src/main/resources/application.yaml in the project directory:

spring:
  jpa:
    properties:
      hibernate:
        connection:
          provider_disables_autocommit: true
        default_schema: todo
    open-in-view: false
  datasource:
    url: jdbc:yugabytedb://[hostname:port]/yugabyte?load-balance=true
    username: yugabyte
    password: yugabyte
    driver-class-name: com.yugabyte.Driver
    hikari:
      minimum-idle: 5
      maximum-pool-size: 20
      auto-commit: false

url is the JDBC connection string. You can set YugabyteDB driver-specific properties such as “load-balance” and “topology-keys” as part of this string.
driver-class-name is the JDBC driver class name.

Update the JDBC URL with the appropriate “hostname” and “port” number details “jdbc:yugabytedb://[hostname: port]/yugabyte” in the application.yaml file. Remember to remove the square brackets. It’s a placeholder to indicate the fields that need user inputs.

Build and run the application

Navigate to the springboot folder:
> cd yb-ms-data/springboot
To build the application:
> gradle build
To run and test the application:
> gradle bootRun

Quarkus

Quarkus is a Kubernetes Native Java stack tailored for OpenJDK HotSpot and GraalVM. It’s crafted from best-of-breed Java libraries and standards.

This section describes how to build a simple JPA-based web application with the Quarkus framework for YSQL API using the YugabyteDB JDBC Driver.

For starters, navigate to the quarkus framework folder inside the project yb-ms-data directory.

cd yb-ms-data/quarkus

Dependencies

This project depends on the following libraries:

implementation("io.quarkus:quarkus-hibernate-orm")
implementation("io.quarkus:quarkus-flyway")
implementation("io.quarkus:quarkus-resteasy")
implementation("io.quarkus:quarkus-resteasy-jackson")
implementation("io.quarkus:quarkus-config-yaml")
implementation("io.quarkus:quarkus-agroal")
implementation("io.quarkus:quarkus-smallrye-fault-tolerance")
implementation("com.yugabyte:jdbc-yugabytedb:42.3.3")

Update the driver dependency library “com.yugabyte:jdbc-yugabytedb:42.3.3” to the latest version. Grab the latest version from YugabyteDB JDBC Driver.

Driver configuration

Refer to the file yb-ms-data/quarkus/src/main/resources/application.yaml in the project directory:

quarkus:
  datasource:
    db-kind: pgsql
    jdbc:
      url: jdbc:yugabytedb://[hostname:port]/yugabyte
      driver: com.yugabyte.Driver
      initial-size: 5
      max-size: 20
      additional-jdbc-properties:
        load-balance: true

db-kind indicates the type of db instance. The value can be pqsql or postgresql for PostgreSQL or PostgreSQL API-compatible instances. You can have either of these values as YugabyteDB is PostgreSQL-compliant and reuses the PostgreSQL query layer.
url is the JDBC connection string.
driver is the JDBC driver class name.
additional-jdbc-properties is where YugabyteDB driver-specific properties such as “load-balance” and “topology-keys” can be set.

Next, update the JDBC URL with the appropriate “hostname” and “port” number details “jdbc:yugabytedb://[hostname: port]/yugabyte” in the application.yaml file. Remember to remove the square brackets. It is a placeholder to indicate the fields that need user inputs.

Build and run the application

Navigate to the quarkus folder:
> cd yb-ms-data/quarkus
To build the application:
> gradle quarkusBuild
To run and test the application:
> gradle quarkusDev

Micronaut

Micronaut is a modern, JVM-based, full-stack framework for building modular, easily-testable microservice and serverless applications.

This section describes how to build a simple JPA-based web application with the Micronaut framework for YSQL API using the YugabyteDB JDBC Driver.

Navigate to the micronaut framework folder inside the project yb-ms-data directory.

cd yb-ms-data/micronaut

Dependencies

This project depends on the following libraries:

annotationProcessor("io.micronaut:micronaut-http-validation")
annotationProcessor("io.micronaut.data:micronaut-data-processor")
annotationProcessor("io.micronaut.openapi:micronaut-openapi")
implementation("io.micronaut:micronaut-http-client")
implementation("io.micronaut:micronaut-management")
implementation("io.micronaut:micronaut-runtime")
implementation("io.micronaut.data:micronaut-data-hibernate-jpa")
implementation("io.micronaut.flyway:micronaut-flyway")
implementation("io.micronaut.sql:micronaut-jdbc-hikari")
implementation("io.swagger.core.v3:swagger-annotations")
implementation("javax.annotation:javax.annotation-api")
runtimeOnly("ch.qos.logback:logback-classic")
implementation("io.micronaut:micronaut-validation")
implementation("com.yugabyte:jdbc-yugabytedb:42.3.3")

Update the driver dependency library “com.yugabyte:jdbc-yugabytedb:42.3.3” to the latest version. Grab the latest version from YugabyteDB JDBC driver.

Driver Configuration

Refer to the file yb-ms-data/micronaut/src/main/resources/application.yaml in the project directory:

datasources:
  default:
    url: jdbc:yugabytedb://[hostname:port]/yugabyte
    driverClassName: com.yugabyte.Driver
    data-source-properties:
      load-balance: true
      currentSchema: todo
    username: yugabyte
    password: yugabyte
    minimum-idle: 5
    maximum-pool-size: 20

url is the JDBC connection string.
driverClassName is the JDBC driver class name.
data-source-properties is where YugabyteDB driver-specific properties such as “load-balance” and “topology-keys” can be set.

Update the JDBC URL with the appropriate “hostname” and “port” number details “jdbc:yugabytedb://[hostname: port]/yugabyte” in the application.yaml file. Remember to remove the square brackets. It is a placeholder to indicate the fields that need user inputs.

Build and run the application

Navigate to the micronaut folder:
> cd yb-ms-data/micronaut
To build the application:
> gradle build
To run and test the application:
> gradle run

Conclusion

This post looked at how we can quickly get started with popular cloud native java frameworks using the YugabyteDB JDBC driver without any application-level modifications.

YSQL’s API compatibility with PostgreSQL accelerates developer productivity and onboarding. By integrating with the existing ecosystem, YugabyteDB ensures developers can quickly start using a language they already know and love.

The YugabyteDB JDBC driver is a distributed driver built on the PostgreSQL driver. Although the upstream PostgreSQL JDBC driver works with YugabyteDB, the YugabyteDB driver enhances it by providing additional features such as cluster and topology awareness.

If you haven’t already, check out our Docs site to learn more about YugabyteDB. Any questions? Ask them in the YugabyteDB community Slack channel.

The post Cloud Native Java: Integrating YugabyteDB with Spring Boot, Quarkus, and Micronaut appeared first on The Distributed SQL Blog.

Tutorial: How to Deploy Multi-Region YugabyteDB on GKE Using Multi-Cluster Services

Srinivasa Vasu — Tue, 18 Jan 2022 15:32:16 +0000

The evolution of “build once, run anywhere” containers and Kubernetes—a cloud-agnostic, declarative-driven orchestration API—have made a scalable, self-service platform layer a reality. Even though it is not a one size fits all solution, a majority of business and technical challenges are being addressed. Kubernetes as the common denominator gives scalability, resiliency, and agility to internet-scale applications on various clouds in a predictable, consistent manner. But what good is application layer scalability if the data is still confined to a single vertically scalable server that can’t exceed a predefined limit?

YugabyteDB addresses these challenges. It is an open source, distributed SQL database built for cloud native architecture. YugabyteDB can handle global, internet-scale applications with low query latency and extreme resilience against failures. It also offers the same level of internet-scale similar to Kubernetes but for data on bare metal, virtual machines, and containers deployed on various clouds.

In this blog post, we’ll explore a multi-region deployment of YugabyteDB on Google Kubernetes Engine (GKE) using Google Cloud Platform’s (GCP) native multi-cluster discovery service (MCS). In a Kubernetes cluster, the “Service” object manifest facilitates service discovery and consumption only within the cluster. We need to rely on an off-platform, bespoke solution with Istio-like capabilities to discover services across clusters. But we can build and discover services that span across clusters natively with MCS. Below is an illustration of our multi-region deployment in action.

Assumptions

We will find commands with appropriate placeholders and substitutions throughout this blog based on the following assignments.

Step 1: Ensure Cloud Readiness

We need to enable a couple of GCP service APIs to use this feature. Let’s allow the following APIs.

Enable Kubernetes Engine API.

Enable GKE hub API.

Enable Cloud DNS API.

Enable Traffic Director API.

Enable Cloud Resource Manager API.

In addition to these APIs, enable standard services such as Compute Engine and IAM.

Step 2: Create three GKE Clusters (VPC native)

Create the first cluster in the US region with workload identity enabled.

Create the second cluster in the Europe region with workload identity enabled.

Create the third cluster in the Asia region with workload identity enabled.

Validate the output of the multi-region cluster creation.

Step 3: Enable MCS Discovery

Enable the MCS discovery API and Services:

As workload identity has already been enabled for the clusters, we need to map the Kubernetes service account to impersonate GCP’s service account. This will allow applications running in the cluster to consume GCP services. IAM binding requires the following mapping.

Step 4: Establish Hub Membership

Upon successful registration, the Hub membership service will provision gke-connect and gke-mcs services to the cluster. These are CRDs and controllers to talk to GCP APIs in order to provision the appropriate cloud resources such as network endpoints, mapping rules, and others (as necessary). This service creates a private managed hosted zone and a traffic director mapping rule on successful membership enrollment. The managed zone “clusterset.local” is similar to “cluster.local” but advertises the services across clusters to auto-discover and consume.

To get started, register all three clusters:

After successful membership enrollment, the following objects will be created.

Managed private zone => “clusterset.local”

From there, the traffic director initializes with the following mapping rule.

Once the traffic director initializes, verify the membership status.

This will also set up the appropriate firewall rules. When the service gets provisioned, the right network endpoints with the mapping rules would be automatically created, as illustrated below.

Step 5: Initialize YugabyteDB

As we have been exploring MCS, it’s clear our upstream Helm package won’t work out of the box. Let’s use the upstream chart to generate the template files locally and then make the relevant changes in the local copy. The template variable file for all three regions is available in the gist repo.

Download all three region-specific variable files and an additional service-export.yaml from the remote repo to the local machine and name them as “ap-south1.yaml”, “eu-west1.yaml”, “us-central1.yaml”, and “service-export.yaml”.

As the upstream chart is not updated for cross-cluster service discovery, the broadcast address of the master and tserver service instances would refer to the cluster local “svc.cluster.local” DNS entry. This needs to be updated explicitly with the new managed zone private domain that the hub created during cluster membership enrollment to let the instances communicate between clusters.

To get started, generate Helm templates for all three regions:

Once we generate the template files, we have to update the broadcast address. More specifically, search for this text “–server_broadcast_addresses” in both the master and tserver StatefulSet manifest definition and update both the entries.

The pattern is [INSTANCE_NAME].[MEMBERSHIP].[SERVICE_NAME].[NAMESPACE].svc.clusterset.local. This change is explained in the next section.

Finally, get the container credentials of all three clusters. Once we get the kubeconfig, the local context would be similar to:

Step 6: Deploy YugabyteDB

Connect to all three cluster contexts one by one and execute the generated template file using kubectl CLI.

Now, let’s explore the service-export.yaml file:

“ServiceExport” CRD is added by the MCS service. This will export services outside of the cluster to be discovered and consumed by other services. The controller programmed to react to events from this resource would interact with GCP services to create ‘A’ name records in the internal private managed zone for both yb-tservers and yb-masters headless services. If we verify the “clusterset.local” domain in the console, we would see the following records:

For the exported headless services, there will be two “A” name records:

This is similar to how the in-cluster “svc.cluster.local” DNS works. The DNS creation and propagation would take some time (around 4-5 mins) for the first time. YugabyteDB wouldn’t be able to establish the quorum until the DNS records are made available. Once those entries get propagated, the cluster would be up and running.

When we verify the Kubernetes cluster state for the master and tserver objects, we will find all of them in a healthy state. This is represented below using the Octant dashboard.

We can simulate the region failure by bringing down the StatefulSet replica to zero. Upon failure, the RAFT consensus group reacts and adjusts accordingly to bring the state to normalcy as we still have two surviving instances.

Once we scale up the replica count, the node joins back with the cluster group, and all three regions are again back to a healthy state.

Good usage pattern

Because of the distributed nature of data and the associated secondary indexes in a multi-region deployment, it is beneficial to pin a region as the preferred region to host the tablet leaders. This keeps the network latencies to a minimum and is confined to a region for cross-node RPC calls such as multi-row transactions, secondary index lookups, and other similar operations. As you can see, this is one of the best usage patterns to improve network latencies in a multi-region deployment.

Step 7: Cleanup

Delete YugabyteDB:

Unregister the Hub Membership:

Disable the MCS APIs:

And finally, delete the GKE clusters.

Conclusion

This blog post used GCP’s multi-cluster service discovery to deploy a highly available, fault-tolerant, and geo-distributed YugabyteDB cluster. As shown in the illustration below, a single YugabyteDB cluster distributed across three different regions addresses many use cases, such as region local delivery, geo-partitioning, higher availability, and resiliency.

Give this tutorial a try—and don’t hesitate to let us know what you think in the YugabyteDB Community Slack channel.

The post Tutorial: How to Deploy Multi-Region YugabyteDB on GKE Using Multi-Cluster Services appeared first on The Distributed SQL Blog.