DEV Community: Sergey Pronin

Exposing replica nodes in Percona Operator for PostgreSQL

Sergey Pronin — Thu, 24 Oct 2024 20:53:33 +0000

This is a micro blog post about a feature that I failed to praise in Percona Operator for PostgreSQL - ability to expose replicas separately.

The problem is that when you connect to the PostgreSQL cluster through pgBouncer, it routes all you reads and writes to the primary.

So in that case the role pgBouncer plays here:

Pool connections
Provide single entry point to connect to
In case of primary failure reroute quesries to the new primary

But we heard it a few times, that it is not good enough. Sometimes you want to have your application to scale reads and not hammer your primary.

Starting version 2.4.0 we allow users to expose replica nodes separately. It is happening by default for all clusters already. You can see -replicas service:

NAME                TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)          AGE
cluster1-replicas   ClusterIP      10.118.234.58   <none> 
5432:31812/TCP   11h

To change the service type, user should alter spec.expose section:

  expose:
...
  exposeReplicas:
    type: LoadBalancer

With this exposure your application connection is going to look like this:

Percona offers various software options to run PostgreSQL:

for regular deployments, use battle-tested, performant and reliable Percona Distribution for PostgreSQL;
deploy, manage, scale your databases in Kubernetes with open source Percona Operator for PostgreSQL;
want to get RDS-like experience, but with no vendor-lock and fully open source - get slick UI and API with Percona Everest;
for cloud-based deployments use our SaaS offering - Ivee by Percona (currently in Beta).

And of course if you are looking for help, support, professional support - let our team know: https://www.percona.com/about/contact

Introducing Ivee in Beta or the story of databases UX

Sergey Pronin — Tue, 01 Oct 2024 16:34:52 +0000

TL;DR

We are introducing the early beta version of Ivee by Percona - a fully managed database-as-a-service. It is multi-cloud, supports multiple database technologies and backed by our expertise.

Try it completely for free
Learn more on the website: https://ivee.cloud

The story

I joined Percona 4 years ago and my first assignment was to do Product Management for Kubernetes Operators. Back then we were just starting with the Operators for MySQL and for MongoDB. Now we also have PostgreSQL and one more Operator for MySQL (which provides other replication methods).

The goal was simple - automate the provisioning and management of databases anywhere. 4 years ago databases on Kubernetes was a greenfield. That was the bet for the future, the market was already going there.

Kubernetes Operators' adoption was sky rocketing, mostly because of the two things:

Kubernetes adoption itself
User experience (UX)

To explain a bit on the UX - Operators provided a way not only to deploy databases, but also automate the management (various day-2 operations). With these capabilities, Operators were a good replacement for managed services. On top of it, Percona expertise was playing a huge role in shaping how Operators deploy database clusters. All the knowledge that was accumulated for years are embedded into the logic that Operators execute.

Of course there were some tradeoffs, for example lack of a proper UI and the need to learn the basics of k8s.

Percona Everest

Percona Everest - an open source database platform - is a solution to close the gap between managed services even further. It relies on the Kubernetes Operators, but provides graphical UI and APIs to simplify the experience even more.

It went generally available in June and we've seen a lot of traction and interest for it. Lyrid - a cloud provider - recently announced that they are using Percona Everest to create their own DBaaS. And it is after 3 months of us going GA!

Ivee

As a product person, I'm always thriving to reduce the friction and provide the value to the user faster. It is quite challenging with installable software. That is how we came up with an idea that we should have our own managed DBaaS. You will see our adherence to Percona standards - multi-database, multi-cloud, powered by our expertise and open source technologies.

We launched Ivee in Beta. Our main goal is to capture feedback and iterate fast. I would be extremely excited if you try it out (oh yeah, it is free) and give me your feedback.

Running Gitea with PostgreSQL on Kubernetes

Sergey Pronin — Wed, 04 Sep 2024 08:45:22 +0000

Gitea - an open source tool for hosting version control using Git as well as other collaborative features like bug tracking, code review, continuous integration and more. It is quite easy to deploy it in Kubernetes - it has a helm chart that will also install Redis and PostgreSQL databases through bitnami helm charts.

But this initial deployment with helm charts is hard to manage and scale for production, as it requires proper database management and day-2 operations. To get there, we will see how somebody can deploy Gitea with Percona Operator for PostgreSQL (this is a follow up for this question on the forum)

Deploy PostgreSQL

To simplify, we will use helm chart to deploy the Operator. See instructions in the documentation.

helm install my-operator percona/pg-operator

Once done, we are ready to deploy the database. But there is a caveat with the user management. Starting with PostgreSQL version 15 CREATE permissions are revoked from all users except database owner from the public (or default) schema (see release notes). This change will cause the problem that the user on the forum is facing:

2024/09/03 16:35:38 cmd/migrate.go:40:runMigrate() [F] Failed to initialize ORM engine: migrate: sync: pq: permission denied for schema public
Gitea migrate might fail due to database connection...This init-container will try again in a few seconds

To fix that, we will need to create a user and also add permissions to this user. To automate permission handling, we can use databaseInitSQL section.

Create an init-sql ConfigMap resource:

% cat init-sql-cm.yaml 
apiVersion: v1
kind: ConfigMap
metadata:
  name: init-sql
data:
  init.sql: |
    \c gitea
    GRANT CREATE ON SCHEMA public TO "gitea";

Create the config map:

kubectl apply -f init-sql-cm.yaml

init.sql will connect to the gitea database and grant the user gitea access to SCHEMA public. Feel free to use any other schema.

Now our values.yaml for the operator is going to look like this:

users:
  - name: gitea
    databases:
      - gitea

databaseInitSQL:
  key: init.sql
  name: init-sql

Create the database:

helm install cluster1 percona/pg-db -f values.yaml

Deploy Gitea

Gitea's helm values.yaml is huge. But to connect to the PostgreSQL cluster deployed with the Operator, we just need to alter the following sections:

# define database
gitea:
  config:
    database:
      DB_TYPE: postgres
      HOST: cluster1-pg-db-pgbouncer.default.svc:5432
      USER: gitea
      NAME: gitea
      SSL_MODE: require
      PASSWD: MYPASSWORD
      SCHEMA: public

# disable built-in postgresql
postgresql:
  enabled: false
postgresql-ha:
  enabled: false

Now follow the installation instructions from the official documentation:

helm repo add gitea-charts https://dl.gitea.com/charts/
helm install gitea gitea-charts/gitea -f values.yaml

What is next

It is important to acknowledge that user management in the Percona Operator for PostgreSQL does not really deliver its end of the bargain. Users are created, but you can't start using them right away, as they don't have permissions to use public schema. This breaks a lot of declarative flows.

In the following release of the Operator we are going to fix it, by automatically creating per-user schemas (similar to what CrunchyData PGO is doing).

The issue is going to be fixed in K8SPG-634.

From Zero to Hero: Disaster Recovery for PostgreSQL with Streaming Replication in Kubernetes

Sergey Pronin — Fri, 17 May 2024 11:59:53 +0000

In today’s digital landscape, disaster recovery is essential for any business. As our dependence on data grows, the impact of system outages or data loss becomes more severe, leading to major business interruptions and financial setbacks.

Managing disaster recovery becomes even more complex with multi-cloud or multi-regional PostgreSQL deployments. Percona Operators offer a solution to simplify this process for PostgreSQL clusters running on Kubernetes. The Operator allows businesses to handle multi-cloud or hybrid-cloud PostgreSQL deployments effortlessly, ensuring that crucial data remains accessible and secure, no matter the circumstances.

This article will guide you through setting up disaster recovery using Percona Operator for PostgreSQL and streaming replication.

Design

The design is simple:

Two sites - main and DR (disaster recovery).
- It can be two regions, data centers or even two namespaces
In each site we have an Operator and a PostgreSQL cluster
- In the DR site the cluster is in Standby mode
- We set up streaming replication between these two clusters

Set it up

All examples in this blog post are as usual available in blog-data/pg-k8s-streaming-dr github repository.

Prerequisites:

Kubernetes cluster or clusters (depending on your topology)
Percona Operator for PostgreSQL deployed.
- See quickstart guides.
- Or just use the bundle.yaml that I have in the repository above:

kubectl apply -f https://raw.githubusercontent.com/spron-in/blog-data/master/pg-k8s-streaming-dr/bundle.yaml

Primary

The only specific thing for the Main cluster is that you need to expose it, so that standby can connect to the primary node. To expose the primary node, use the spec.expose section:

spec:
  ...
  expose:
    type: ClusterIP

Use a Service type of your choice. In my case, I have two clusters in different namespaces, so ClusterIP is sufficient. Deploy the cluster as usual:

kubectl apply -f main-cr.yaml -n main-pg

The service that you should use for connecting to standby is called <clustername>-ha (main-ha in my case):

main-ha          ClusterIP   10.118.227.214   <none>        5432/TCP   163m

Standby

TLS certificates

To get the replication working, the Standby cluster would need to authenticate with the Main one. To get there, both clusters must have certificates signed by the same certificate authority (CA). Default replication user _crunchyrepl will be used.

In the simplest case you can copy the certificates from the Main cluster. You need to look out for two files:

main-cluster-cert
main-replication-cert

Copy them to the namespace where DR cluster is going to be running and reference under spec.secrets (I renamed them replacing main with dr):

spec:
  secrets:
    customTLSSecret:
      name: dr-cluster-cert
    customReplicationTLSSecret:
      name: dr-replication-cert

If you are generating your own certificates, just remember the following rules:

Certificates for both Main and Standby clusters must be signed by the same CA
customReplicationTLSSecret must have a Common Name (CN) setting that matches _crunchyrepl, which is a default replication user.

Read more about certificates in the documentation.

Configuration

Apart from setting certificates correctly, you should also set standby configuration.

  standby:
    enabled: true
    host: main-ha.main-pg.svc

standby.enabled controls if it is a standby cluster or not
standby.host must point to the primary node of a Main cluster. In my case it is a main-ha service in another namespace.

Deploy the DR cluster:

kubectl apply -f dr-cr.yaml -n dr-pg

Verify

Once both clusters are up, you can verify that replication is working.

Insert some data into Main cluster
Connect to the DR cluster

To connect to the DR cluster, use the credentials that you used to connect to Main. This also verifies that the connection is working. You should see whatever data you have in the Main cluster in the Disaster Recovery.

Conclusion

Disaster recovery is crucial for maintaining business continuity in today's data-driven environment. Implementing a robust disaster recovery strategy for multi-cloud or multi-regional PostgreSQL deployments can be complex. However, the Percona Operator for PostgreSQL simplifies this process by enabling seamless management of PostgreSQL clusters on Kubernetes. By following the steps outlined in this article, you can set up disaster recovery using Percona Operator and streaming replication, ensuring your critical data remains secure and accessible. This approach not only provides peace of mind but also safeguards against significant business disruptions and financial losses.

At Percona, we are aiming to provide the best open source databases and tooling possible. As the next level of simplification and user experience for databases on Kubernetes, we recently released Percona Everest (currently in Beta). It is a cloud native database platform with a slick UI. It deploys and manages databases on Kubernetes for you without the need to look into YAML manifests.

Try Percona Operator for PostgreSQL | Try Percona Everest

Deploy MongoDB on Kubernetes with ArgoCD and Helm charts

Sergey Pronin — Tue, 23 Apr 2024 09:17:45 +0000

According to CNCF GitOps Microsurvey, adoption of GitOps among Kubernetes practitioners is growing and ArgoCD and Flux are the most widely used CNCF GitOps projects. The desire to simplify application deployment and management in Kubernetes also makes sense, as complexity in Cloud Native space is the most common challenge.

Databases on Kubernetes are complex as well, but Operators take away toil from administrators and developers. In this blog post we will simplify day-1 and day-2 operations for Databases on Kubernetes with a synergetic mix of ArgoCD and Operators, using Percona Operator for MongoDB as an example.

Choosing a strategy

ArgoCD allows the use of both regular YAML manifests and Helm charts. In conversations with our users, we see the tendency to use helm charts when those are available. Helm charts are a universal way to deploy and manage applications on Kubernetes, as a result easier onboarding.

YAMLs though can enable better flexibility, as it gives a direct exposure to manifests.

In this article we provide an example for using Helm charts. Read Deploy PostgreSQL on Kubernetes using Gitops and ArgoCD to get an idea on how to deal with the YAML manifests case, it can be easily adapted for Percona Operator for MongoDB.

Prepare the environment

For starters, you need the following:
Kubernetes cluster
Helm chart. In this article we use the official Percona helm charts.

Deploy and configure ArgoCD

ArgoCD has detailed documentation explaining the installation process. I did the following:

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Action

The flow is the following:

User creates the Argo Application resources
ArgoCD syncs with the helm repository
ArgoCD provisions the resources based on ArgoCD application definitions and syncs with the Helm repository
User connects to the database once it is provisioned

One of the key parts to grasp is that ArgoCD uses helm repository as a source of truth, but it does not deploy helm directly in the Kubernetes cluster. So in the helm list output you will not see anything.

Custom Resource Definitions (CRDs)

CRDs allow users to extend the Kubernetes API. Operators process Custom Resources (CRs) that users create and as a result deploy and manage applications. Custom Resource Definitions must be created before users start creating CRs.

ArgoCD provides a way to sequence the provisioning of the resources, but only within a single application. In the case of a helm chart, we have two different Argo applications.

The best practice here is to ensure that the application that has CRDs is provisioned first. Usual approach is to have role separation. Where Kubernetes cluster administrators provision CRDs and other roles, like developers or DBAs, only focus on database components.

Application to deploy CRDs - argo_psmdb_operator.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: psmdb-operator
  namespace: argocd
spec:
  project: default
  syncPolicy:
    automated: {}
    syncOptions:
    - ServerSideApply=true
  source:
    chart: psmdb-operator
    path: charts/psmdb-operator/
    repoURL: https://percona.github.io/percona-helm-charts/
    targetRevision: 1.15.4
    helm:
      releaseName: psmdb-operator
  destination:
    server: "https://kubernetes.default.svc"
    namespace: default

Some things to note:
- ServerSideApply=true - is a must, otherwise CRD creation will fail with the following error:

time="2024-04-17T14:58:31Z" level=info msg="Updating operation state. phase: Running -> Running, message: 'one or more objects failed to apply, reason: CustomResourceDefinition.apiextensions.k8s.io \"perconaservermongodbs.psmdb.percona.com\" is invalid: metadata.annotations: Too long: must have at most 262144 bytes. Retrying attempt #5 at 2:57PM.' -> 'one or more tasks are running'" application=argocd/psmdb-operator syncId=00006-HyBZr

We enable automated sync in the example, this might not be desired for your use case. Read more about synchronization in the documentation.

Check if application is synchronized:

% kubectl -n argocd get application psmdb-operator
NAME             SYNC STATUS   HEALTH STATUS
psmdb-operator   Synced        Healthy

Once it is synced, you should be able to see the Operator Pod and CRDs deployed.

Deploy the Percona Server for MongoDB cluster

There is no difference between deploying CRDs with Operator and the Custom Resource for database cluster. We use another helm chart and also add extra parameters as an example - argo_psmdb_db.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: psmdb-db
  namespace: argocd
spec:
  project: default
  syncPolicy:
    automated: {}
  source:
    chart: psmdb-db
    path: charts/psmdb-db/
    repoURL: https://percona.github.io/percona-helm-charts/
    targetRevision: 1.15.3
    helm:
      releaseName: psmdb-db
      valuesObject:
      sharding:
          enabled: false
      ...
  destination:
    server: "https://kubernetes.default.svc"
    namespace: default

Attention to valuesObjects section, where we provide some examples of passing values.yaml variables. There are three ways how it can be done:

valuesObjects - the one in our example
values:

source:
  helm:
    values: |
      sharding:
        enabled: false
      ...

values files:

source:
  helm:
    valueFiles:
    - myvalues.yaml

If you do everything correctly, you would see the application synchronized and Custom Resource deployed:

% kubectl -n argocd get application psmdb-db
NAME       SYNC STATUS   HEALTH STATUS
psmdb-db   Synced        Healthy


% kubectl get psmdb
NAME       ENDPOINT   STATUS         AGE
psmdb-db   ...        ready          73s

Conclusion

Deploying the Percona Operator for MongoDB using ArgoCD represents a significant step forward in managing complex database systems within Kubernetes. By integrating Helm charts with GitOps principles, organizations can achieve more consistent, scalable, and error-resistant deployments.

As the cloud-native ecosystem continues to evolve, tools like ArgoCD and Kubernetes Operators will play increasingly critical roles. Their ability to reduce manual overhead and streamline operations is crucial in an environment where agility and reliability are paramount. We encourage practitioners to explore these technologies, experiment with the configurations discussed, and contribute back to the community with any findings or improvements.

Percona is committed to provide open source solutions to deploy and manage databases anywhere. Try out our operators or check the Beta version of an open source cloud native database platform - Percona Everest. It provides a nice GUI and API to deploy databases on Kubernetes.

100% open source Operator for MongoDB | Deploy MongoDB with Percona Everest