Installing YugabyteDB with the Helm Chart creates 4 services. Here is an example where I deployed in AWS as in this previous blog post:
$ kubectl get services -n yb-demo-eu-west-1a
NAME TYPE CLUSTER-IP PORT(S)
yb-master-ui LoadBalancer 10.100.227.70 7000:31261/TCP
yb-masters ClusterIP None 7000/TCP,7100/TCP
yb-tserver-service LoadBalancer 10.100.31.106 6379:31669/TCP,9042:31874/TCP,5433:31376/TCP
yb-tservers ClusterIP None 9000/TCP,12000/TCP,11000/TCP,13000/TCP,9100/TCP,6379/TCP,9042/TCP,5433/TCP
There are two services (LoadBalancer and ClusterIP) for the two StatefulSets (yb-master, the contol plane, and yb-tservers, the data plane)
ClusterIP headless service
The ClusterIP ones are created with no IP which means that they are just DNS entries and not a proxy. This is a headless service that can be used to distribute the connections without an intermediate component, using the Kubernetes DNS.
From the cluster (cluster.local) in a namespace (yb-demo-eu-west-1c), I can connect to any pod in the yb-tserver StatefulSets with the hostname yb-tservers.yb-demo-eu-west-1a.svc.cluster.local
Round robin to one StatefulSets pods
Here is an example, running psql from any pod, and connecting multiple times to this address:
$ kubectl run -it --rm --restart=Never --image postgres psql -- \
psql -h yb-tservers.yb-demo-eu-west-1a.svc.cluster.local \
-p 5433 -U yugabyte -c 'select inet_server_addr()'
inet_server_addr
------------------
192.168.11.13
(1 row)
$ kubectl run -it --rm --restart=Never --image postgres psql -- \
psql -h yb-tservers.yb-demo-eu-west-1a.svc.cluster.local \
-p 5433 -U yugabyte -c 'select inet_server_addr()'
inet_server_addr
------------------
192.168.20.199
(1 row)
The connections went to different nodes. Those are different PostgreSQL backends running in different pods, but they expose the same logical database because YugabyteDB is a Distributed SQL database.
The two IP addresses above are the two pods in the yb-tserver StatefulSets:
$ kubectl get pods -n yb-demo-eu-west-1a -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
yb-master-0 2/2 Running 0 174m 192.168.2.75 ip-192-168-4-117.eu-west-1.compute.internal <none> <none>
yb-tserver-0 2/2 Running 0 174m 192.168.20.199 ip-192-168-4-117.eu-west-1.compute.internal <none> <none>
yb-tserver-1 2/2 Running 0 174m 192.168.11.13 ip-192-168-4-117.eu-west-1.compute.internal <none> <none>
Note that I deployed the StatefulSets in three Availability Zones. The ClusterIP service connects to one AZ and this is probably what you want to do from the application servers, so that they connect in the same AZ. This ensures the minimum latency and maximum availability.
Cluster aware Smart Drivers
However, if you use the YugabyteDB Smart Drivers they will discover all nodes in all AZ:
$ kubectl run -it --rm --restart=Never --image postgres psql -- \
> psql -h yb-tservers.yb-demo-eu-west-1a.svc.cluster.local \
> -p 5433 -U yugabyte -c 'select host,zone from yb_servers()'
host | zone
---------------------------------------------------------------+------------
yb-tserver-0.yb-tservers.yb-demo-eu-west-1c.svc.cluster.local | eu-west-1c
yb-tserver-1.yb-tservers.yb-demo-eu-west-1a.svc.cluster.local | eu-west-1a
yb-tserver-0.yb-tservers.yb-demo-eu-west-1b.svc.cluster.local | eu-west-1b
yb-tserver-1.yb-tservers.yb-demo-eu-west-1b.svc.cluster.local | eu-west-1b
yb-tserver-1.yb-tservers.yb-demo-eu-west-1c.svc.cluster.local | eu-west-1c
yb-tserver-0.yb-tservers.yb-demo-eu-west-1a.svc.cluster.local | eu-west-1a
(6 rows)
This mean that if you want to constrain the connections to one AZ when using the Smart Drivers, you need to use the tolopogy key, like aws.eu-west-1.eu-west-1a for example.
Endpoints
The ClusterIP headless service yb-tservers connects to the yb-tserver pods, 192.168.11.13 and 192.168.20.199 in my case, for all ports exposed by the Table Servers, 5433 being the YSQL one, which is the PostgreSQL-compatible API.
$ kubectl describe service yb-tservers -n yb-demo-eu-west-1a
Name: yb-tservers
Namespace: yb-demo-eu-west-1a
Labels: app=yb-tserver
app.kubernetes.io/managed-by=Helm
chart=yugabyte
component=yugabytedb
heritage=Helm
release=yb-demo
service-type=headless
Annotations: meta.helm.sh/release-name: yb-demo
meta.helm.sh/release-namespace: yb-demo-eu-west-1a
Selector: app=yb-tserver
Type: ClusterIP
IP Family Policy: SingleStack
IP Families: IPv4
IP: None
IPs: None
Port: http-ui 9000/TCP
Endpoints: 192.168.11.13:9000,192.168.20.199:9000
Port: http-ycql-met 12000/TCP
TargetPort: 12000/TCP
Endpoints: 192.168.11.13:12000,192.168.20.199:12000
Port: http-yedis-met 11000/TCP
TargetPort: 11000/TCP
Endpoints: 192.168.11.13:11000,192.168.20.199:11000
Port: http-ysql-met 13000/TCP
TargetPort: 13000/TCP
Endpoints: 192.168.11.13:13000,192.168.20.199:13000
Port: tcp-rpc-port 9100/TCP
TargetPort: 9100/TCP
Endpoints: 192.168.11.13:9100,192.168.20.199:9100
Port: tcp-yedis-port 6379/TCP
TargetPort: 6379/TCP
Endpoints: 192.168.11.13:6379,192.168.20.199:6379
Port: tcp-yql-port 9042/TCP
TargetPort: 9042/TCP
Endpoints: 192.168.11.13:9042,192.168.20.199:9042
Port: tcp-ysql-port 5433/TCP
TargetPort: 5433/TCP
Endpoints: 192.168.11.13:5433,192.168.20.199:5433
Session Affinity: None
Events: <none>
LoadBalancer
In addition to the ClusterIP service, the Helm Chart creates a LoadBalancer. I can see it in the AWS console:
Here is the description from Kubernetes:
$ kubectl describe service yb-tserver-service -n yb-demo-eu-west-1a
Name: yb-tserver-service
Namespace: yb-demo-eu-west-1a
Labels: app=yb-tserver
app.kubernetes.io/managed-by=Helm
chart=yugabyte
component=yugabytedb
heritage=Helm
release=yb-demo
service-type=endpoint
Annotations: meta.helm.sh/release-name: yb-demo
meta.helm.sh/release-namespace: yb-demo-eu-west-1a
Selector: app=yb-tserver
Type: LoadBalancer
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.100.31.106
IPs: 10.100.31.106
LoadBalancer Ingress: a95638155644e470abd19e552bc8ab01-1510239497.eu-west-1.elb.amazonaws.com
Port: tcp-yedis-port 6379/TCP
TargetPort: 6379/TCP
NodePort: tcp-yedis-port 31498/TCP
Endpoints: 192.168.11.13:6379,192.168.20.199:6379
Port: tcp-yql-port 9042/TCP
TargetPort: 9042/TCP
NodePort: tcp-yql-port 30927/TCP
Endpoints: 192.168.11.13:9042,192.168.20.199:9042
Port: tcp-ysql-port 5433/TCP
TargetPort: 5433/TCP
NodePort: tcp-ysql-port 31565/TCP
Endpoints: 192.168.11.13:5433,192.168.20.199:5433
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
The ingress host is the one I can use to connect from outside of the Kubernetes cluster. In this Multi-AZ configuration I have one per AZ. Let's check the round-robin with 100 sucessive connections:
$ for i in {1..100} ; do
psql -h a95638155644e470abd19e552bc8ab01-1510239497.eu-west-1.elb.amazonaws.com -p 5433 \
-t -A -c '
select inet_server_addr()
' ; done | sort | uniq -c
52 192.168.11.13
48 192.168.20.199
48% on yb-tserver-0 and 52% on yb-tserver-1
To create StatefulSets for Multi-AZ (where each has a yb-master and knows about all yb-master in the cluster) I used an override with isMultiAz: True. If you don't want the LoadBalancer, you can disable it with enableLoadbalancer: false
Those values are visible with:
helm show all yugabytedb/yugabyte
and if you don't ind the documentation and comments sufficient, the template is: yugabyte/charts service.yaml
Note that Helm Charts are the maintained and recommended way to Install YugabyteDB on Kubernetes. There's no need for operators as the database itself is self-healing, autonomously repairing itself autonomously without the need for additional components. You scale by scaling the StatefulSets and the connections, SQL processing and data are rebalanced automatically. When a pod is down, the remaining ones do the necessary Raft Leader election to continue to serve consistent reads and writes.
Top comments (0)