DEV Community: Hoang Guruu

Awesome Kubernetes Resources !!! 🔥

Hoang Guruu — Wed, 19 Mar 2025 03:04:35 +0000

Awesome Kubernetes Resources

【 👨🏻‍💻 YouTube 】
Watch more other contents related DevOps - AWS

A curated list of awesome Kubernetes tools and resources.

Inspired by awesome list and donnemartin/awesome-aws.

The Fiery Meter of Awesomeness

Repo with 0050+ Stars: 🔥
Repo with 0200+ Stars: 🔥🔥
Repo with 0500+ Stars: 🔥🔥🔥
Repo with 1000+ Stars: 🔥🔥🔥🔥
Repo with 2000+ Stars: 🔥🔥🔥🔥🔥

Idea taken from donnemartin/awesome-aws.

Awesome Kubernetes Resources
- The Fiery Meter of Awesomeness
- Contents
- Tools and Libraries
- Command Line Tools
- Cluster Provisioning
- Automation and CI/CD
- Cluster Resources Management
- Secrets Management
- Networking
- Storage
- Testing and Troubleshooting
- Monitoring, Alerts, and Visualization
- Backup and Restore
- Security and Compliance
- Service Mesh
- Development Tools
- Data Processing and Machine Learning
- Data Management
- Miscellaneous
- Guides, Documentations, Blogs, and Learnings
- Guides
- Blogs and Videos
- Learnings and Documentations
- Certification Guides
- Contribute
- License

Tools and Libraries

Items with 💚 indicate open source projects.

Command Line Tools

💚Helm 🔥🔥🔥🔥🔥 - Helm is a tool for managing Charts. Charts are packages of pre-configured Kubernetes resources.
💚Helmfile 🔥🔥🔥🔥🔥 - Helmfile is a declarative spec for deploying helm charts.
💚Helmwave 🔥🔥🔥 - Helmwave is helm3-native tool for deploy your Helm Charts. It is like Docker-Compose, but for Helm.
💚Infra 🔥🔥🔥 - Infra enables you to discover and access infrastructure (e.g. Kubernetes, databases). We help you connect an identity provider such as Okta or Azure active directory, and map users/groups with the permissions you set to your infrastructure.
💚K9s 🔥🔥🔥🔥🔥 - K9s provides a terminal UI to interact with your Kubernetes clusters.
💚kapp 🔥🔥🔥 - kapp is a simple deployment tool focused on the concept of "Kubernetes application" — a set of resources with the same label
💚kconnect 🔥🔥 - kconnect is a CLI utility that can be used to discover and securely access Kubernetes clusters across multiple operating environments.
💚Ktunnel 🔥🔥 - Ktunnel is a CLI tool that establishes a reverse tunnel between a kubernetes cluster and your local machine.
💚Kubebox 🔥🔥🔥🔥 - Terminal and Web console for Kubernetes
💚Kubetail 🔥🔥🔥🔥🔥 - Bash script that enables you to aggregate (tail/follow) logs from multiple pods into one stream.
💚kube-shell 🔥🔥🔥🔥 - Kube-shell: An integrated shell for working with the Kubernetes CLI.
💚kubecolor 🔥🔥🔥 - colorizes kubectl output
💚kubectl tree 🔥🔥🔥🔥 - A kubectl plugin to explore ownership relationships between Kubernetes objects through owners.
💚kubectl-aliases 🔥🔥🔥🔥🔥 - This repository contains a script to generate hundreds of convenient shell aliases for kubectl.
💚kubectx + kubens 🔥🔥🔥🔥🔥 - kubectx helps you switch between clusters back and forth, and kubens helps you switch between Kubernetes namespaces smoothly.
💚kube-ps1 🔥🔥🔥🔥🔥 - kube-ps1: A script that lets you add the current Kubernetes context and namespace configured on kubectl to your Bash/Zsh prompt strings (i.e. the $PS1).
💚kubediff 🔥🔥🔥 - Kubediff is a tool for Kubernetes to show you the differences between your running configuration and your version controlled configuration.
💚kubeprompt 🔥 - Isolates KUBECONFIG in each shell and shows the current Kubernetes context/namespace in your prompt
💚Kubevela 🔥🔥🔥🔥🔥 - KubeVela is an easy-to-use yet extensible platform that enables them to design and ship applications with minimal effort.
💚Move2Kube 🔥🔥 - A tool to help users migrate their apps from legacy platforms like Cloud Foundry to Kubernetes and Openshift. Analyses the application source code and generates Kubernetes YAMLs, Helm Charts, Tekton Pipelines, etc. The analysis and generation can be heavily customized to produce the exact output that you want.
💚nova 🔥🔥 - Nova scans your cluster for installed Helm charts, then cross-checks them against all known Helm repositories.
💚Plural 🔥🔥 - Plural is a CLI tool and holistic DevOps management platform for rapidly deploying, managing, and monitoring open-source applications on Kubernetes.
💚RBAC Lookup 🔥🔥🔥 - RBAC Lookup is a CLI that allows you to easily find Kubernetes roles and cluster roles bound to any user, service account, or group name.
💚stern 🔥🔥🔥🔥🔥 - Stern allows you to tail multiple pods on Kubernetes and multiple containers within the pod.

Cluster Provisioning

💚Bootkube 🔥🔥🔥🔥 - Bootkube is a tool for launching self-hosted Kubernetes clusters.
💚Claudie 🔥 - Multi-cloud clusters with each nodepool in a different cloud provider.
💚Cluster API 🔥🔥🔥🔥🔥 - Cluster API is a Kubernetes sub-project focused on providing declarative APIs and tooling to simplify provisioning, upgrading, and operating multiple Kubernetes clusters.
💚eksctl 🔥🔥🔥🔥🔥 - eksctl is a simple CLI tool for creating clusters on EKS - Amazon's new managed Kubernetes service for EC2.
💚k0s 🔥🔥🔥🔥🔥 - k0s - Zero Friction Kubernetes (The Simple, Solid & Certified Kubernetes Distribution)
💚k3d 🔥🔥🔥🔥🔥 - k3d,and Windows.,destroy,half the memory,highly available,is a tool for running local k3s clusters in docker. It's a single binary about 20 MB. You need to have docker installed.
💚k3s 🔥🔥🔥🔥🔥 - Lightweight Kubernetes. Easy to install,Kubernetes clusters from the command line.
💚kind 🔥🔥🔥🔥🔥 - kind is a tool for running local Kubernetes clusters using Docker container "nodes".
💚kops 🔥🔥🔥🔥🔥 - kops helps you create,like kind,upgrade and maintain production-grade
💚kube-aws 🔥🔥🔥🔥 - kube-aws is a command-line tool to create/update/destroy Kubernetes clusters on AWS.
💚kubespray 🔥🔥🔥🔥🔥 - Deploy a production ready Kubernetes cluster
💚microK8s 🔥🔥🔥🔥🔥 - The smallest, fastest Kubernetes
💚Minikube 🔥🔥🔥🔥🔥 - minikube implements a local Kubernetes cluster on macOS,Linux,all in a binary less than 100 MB.
💚Talos Linux 🔥🔥🔥🔥🔥 - Talos Linux is a minimal, immutable, secure OS that installs vanilla Kubernetes - for production datacenters, K8s@home, and Edge.
💚karpenter 🔥🔥🔥🔥🔥 - Karpenter is a Kubernetes Node Autoscaler built for flexibility, performance, and simplicity.
Kubeadm - kubeadm performs the actions necessary to get a minimum viable cluster up and running.

Automation and CI/CD

💚Argo CD 🔥🔥🔥🔥🔥 - Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes.
💚Argo Events 🔥🔥🔥🔥 - Argo Events is an event-driven workflow automation framework for Kubernetes which helps you trigger K8s objects, Argo Workflows, Serverless workloads, etc.
💚Argo Rollouts 🔥🔥🔥🔥 - Argo Rollouts controller, uses the Rollout custom resource to provide additional deployment strategies such as Blue Green and Canary to Kubernetes.
💚Argo Workflows 🔥🔥🔥🔥🔥 - Argo Workflows is an open source container-native workflow engine for orchestrating parallel jobs on Kubernetes.
💚Argocd autopilot 🔥🔥🔥 - The Argo-CD Autopilot is a tool which offers an opinionated way of installing Argo-CD and managing GitOps repositories.
💚Flagger 🔥🔥🔥🔥🔥 - Flagger is a progressive delivery tool that automates the release process for applications running on Kubernetes.
💚Flux2 🔥🔥🔥🔥🔥 - Flux version 2 is built from the ground up to use Kubernetes' API extension system, and to integrate with Prometheus and other core components of the Kubernetes ecosystem.
💚k8s-image-swapper 🔥🔥 - k8s-image-swapper is a mutating webhook for Kubernetes, downloading images into your own registry and pointing the images to that new location.
💚Kubero 🔥🔥🔥🔥🔥 - A free and self-hosted Heroku PaaS alternative for Kubernetes that implements GitOps
💚KubeSphere 🔥🔥🔥🔥🔥 - KubeSphere is a distributed operating system providing cloud native stack with Kubernetes as its kernel, and aims to be plug-and-play architecture for third-party applications seamless integration to boost its ecosystem.
💚Reloader 🔥🔥🔥🔥🔥 - Reloader can watch changes in ConfigMap and Secret and do rolling upgrades on Pods with their associated DeploymentConfigs, Deployments, Daemonsets and Statefulsets.
💚terranetes-controller 🔥 - Terranetes controller enables the platform team to deliver self-service capabilities around cloud resources.
💚Skaffold 🔥🔥🔥🔥🔥 - Skaffold is a command line tool that facilitates continuous development for Kubernetes applications.
💚Spinnaker 🔥🔥🔥🔥🔥 - Spinnaker is an open-source continuous delivery platform for releasing software changes with high velocity and confidence.
💚TF-controller 🔥 - TF-controller is an experimental controller for Flux to reconcile Terraform resources in the GitOps way.
💚werf 🔥🔥🔥🔥🔥 - werf is a CLI tool glueing Git, Docker, Helm & Kubernetes with any CI system to implement CI/CD and GitOps.
💚Weave GitOps 🔥🔥 - Weave GitOps is a simple open source developer platform for people who want cloud native applications, without needing Kubernetes expertise.
💚Otomi - Self-hosted PaaS for K8s 🔥🔥🔥🔥 - Otomi adds developer- and operations-centric tools, automation, and developer self-service on top of Kubernetes in any infrastructure or cloud, to code, build, release, deploy, secure, operate and monitor containerized applications.

Cluster Resources Management

💚Clusterpedia 🔥 - Clusterpedia is used for complex resource searches across multiple clusters, support simultaneous search of a single kind of resource or multiple kinds of resources existing in multiple clusters.
💚Grafana Tanka 🔥🔥🔥🔥 - The clean, concise and super flexible alternative to YAML for your Kubernetes cluster.
💚KEDA 🔥🔥🔥🔥🔥 - KEDA allows for fine grained autoscaling (including to/from zero) for event driven Kubernetes workloads.
💚Kruise 🔥🔥🔥🔥🔥 - Kruise consists of several controllers which extend and complement the Kubernetes core controllers for workload management.
💚KubeDirector 🔥🔥 - KubeDirector uses standard Kubernetes (K8s) facilities of custom resources and API extensions to implement stateful scaleout application clusters.
💚Kubenav 🔥🔥🔥🔥 - kubenav is the navigator for your Kubernetes clusters right in your pocket.
💚Liqo 🔥🔥 - Liqo implements Dynamic resource sharing across different Kubernetes clusters (e.g.; offloading pods and services), supporting decentralized governance.
💚 Meshery 🔥🔥🔥🔥🔥 - Meshery is an open-source cloud-native manager that enables the design and management of all Kubernetes-based infrastructure and applications.
💚Pluto 🔥🔥🔥🔥 - Pluto is a utility to help users find deprecated Kubernetes apiVersions in their code repositories and their helm releases.
💚Polaris 🔥🔥🔥🔥🔥 - Polaris is an open source policy engine for Kubernetes that validates and remediates resource configuration.
💚Projectsveltos 🔥 Projectsveltos is a Kubernetes add-on controller that simplifies the deployment and management of add-ons and applications across multiple clusters.
💚The Hierarchical Namespace Controller 🔥🔥🔥 - Hierarchical namespaces make it easier to share your cluster by making namespaces more powerful.

Secrets Management

💚Kubernetes External Secrets 🔥🔥🔥🔥🔥 - Kubernetes External Secrets allows you to use external secret management systems, like AWS Secrets Manager or HashiCorp Vault, to securely add secrets in Kubernetes.
💚Sealed Secrets 🔥🔥🔥🔥🔥 - Encrypt your Secret into a SealedSecret, which is safe to store - even to a public repository.
💚akv2k8s 🔥🔥 - Azure Key Vault to Kubernetes (akv2k8s) will make Azure Key Vault objects available to Kubernetes in two ways: as native Kubernetes Secrets; as environment variables directly injected into your Container application

Networking

💚Calico Networking 🔥🔥🔥🔥🔥 - Calico is an open source networking and network security solution for containers, virtual machines, and bare-metal workloads
💚cert-manager 🔥🔥🔥🔥🔥 - cert-manager is a Kubernetes add-on to automate the management and issuance of TLS certificates from various issuing sources.
💚cilium 🔥🔥🔥🔥🔥 - Cilium is a networking, observability, and security solution with an eBPF-based dataplane.
💚CoreDNS 🔥🔥🔥🔥🔥 - CoreDNS is a fast and flexible DNS server that works on Kubernetes.
💚ingress-nginx 🔥🔥🔥🔥🔥 - ingress-nginx is an Ingress controller for Kubernetes using NGINX as a reverse proxy and load balancer.
💚Kong for Kubernetes 🔥🔥🔥🔥 - Configure plugins, health checking, load balancing and more in Kong for Kubernetes Services.
💚ksniff 🔥🔥🔥🔥 - A kubectl plugin that utilize tcpdump and Wireshark to start a remote capture on any pod in your Kubernetes cluster.
💚kubectl trace 🔥🔥🔥🔥 - kubectl trace is a kubectl plugin that allows you to schedule the execution of bpftrace programs in your Kubernetes cluster.
💚Kube Karp 🔥 - Add a floating virtual IP to Kubernetes cluster nodes for load balancing easily based on the CARP protocol
💚kubernetes-ingress 🔥🔥🔥🔥🔥 - An implementation of an Ingress controller for NGINX and NGINX Plus (commercial).
💚kube-ovn 🔥🔥🔥🔥 - A Kubernetes Network Fabric for Enterprises that is Rich in Functions and Easy in Operations.
💚loxilb 🔥🔥🔥 - A Kubernetes service load-balancer based on eBPF.

Storage

💚Longhorn 🔥🔥🔥🔥🔥 - Longhorn is a distributed block storage system for Kubernetes.
💚OpenEBS 🔥🔥🔥🔥🔥 - OpenEBS is the most widely deployed and easy to use open-source storage solution for Kubernetes.
💚Rook 🔥🔥🔥🔥🔥 - Rook is an open source cloud-native storage orchestrator for Kubernetes.

Testing and Troubleshooting

💚Chainsaw 🔥 - The ultimate end to end testing tool for Kubernetes operators.
💚Chaos Mesh 🔥🔥🔥🔥🔥 - Chaos Mesh® is a cloud-native Chaos Engineering platform that orchestrates chaos on Kubernetes environments.
💚chaoskube 🔥🔥🔥🔥 - chaoskube periodically kills random pods in your Kubernetes cluster.
💚Conftest 🔥🔥🔥🔥 - Conftest helps you write tests against structured configuration data.
💚DETIK 🔥 - A library that simplifies end-to-end testing of K8s applications by using BATS assertions and natural language queries.
💚k6 🔥🔥🔥🔥🔥 - k6 is a modern load testing tool, building on Load Impact's years of experience in the load and performance testing industry.
💚ksniff 🔥🔥🔥🔥 - A kubectl plugin that utilize tcpdump and Wireshark to start a remote capture on any pod in your Kubernetes cluster.
💚Kube DOOM 🔥🔥🔥🔥 - The next level of chaos engineering is here! Kill pods inside your Kubernetes cluster by shooting them in Doom!
💚kube-monkey 🔥🔥🔥🔥🔥 - It randomly deletes Kubernetes (k8s) pods in the cluster encouraging and validating the development of failure-resilient services.
💚kube-score 🔥🔥🔥🔥 - kube-score is a tool that performs static code analysis of your Kubernetes object definitions.
💚Kubectl-debug 🔥🔥🔥🔥🔥 - kubectl-debug is an out-of-tree solution for troubleshooting running pods, which allows you to run a new container in running pods for debugging purpose.
💚KubeInvaders 🔥🔥🔥 - Through KubeInvaders you can stress Kubernetes cluster in a fun way and check how it is resilient.
💚Kubetest 🔥 - Kubetest is a pytest plugin that makes it easier to manage a Kubernetes cluster within your integration tests.
💚Litmus 🔥🔥🔥🔥🔥 - Litmus provides tools to orchestrate chaos on Kubernetes to help SREs find weaknesses in their deployments.
💚popeye 🔥🔥🔥🔥🔥 - Popeye is a utility that scans live Kubernetes cluster and reports potential issues with deployed resources and configurations.
💚PowerfulSeal 🔥🔥🔥🔥 - PowerfulSeal injects failure into your Kubernetes clusters, so that you can detect problems as early as possible.
💚Testkube 🔥🔥🔥 - Testkube is a Kubernetes native Testing Framework for test orchestration and execution. It allows you to run any of your tests inside a Kubernetes cluster. Integrates with your CI/CD and allows you to follow a GitOps approach to Testing while having a centralized place for all of your Test Results accross all clusters.

Monitoring, Alerts, and Visualization

💚BotKube 🔥🔥🔥🔥 - BotKube integration with Slack or Mattermost helps you monitor your Kubernetes cluster, debug critical deployments and gives recommendations for standard practices by running checks on the Kubernetes resources.
💚Canary Checker 🔥 - Canary Checker is a kubernetes-native health check platform with 30+ built-in health check types.
💚Cortex 🔥🔥🔥🔥🔥 - Cortex provides horizontally scalable, highly available, multi-tenant, long term storage for Prometheus.
💚Goldilocks 🔥🔥🔥 - Goldilocks is a utility that can help you identify a starting point for resource requests and limits.
💚Goldpinger 🔥🔥🔥🔥🔥 - Debugging tool for Kubernetes which tests and displays connectivity between nodes in the cluster.
💚Grafana 🔥🔥🔥🔥🔥 - Grafana allows you to query, visualize, alert on and understand your metrics no matter where they are stored.
💚Helm Dashboard 🔥🔥🔥🔥 - The missing UI for Helm. The Helm Dashboard plugin offers a UI-driven way to view installed Helm charts, see their revision history and corresponding k8s resources.
💚Kiali 🔥🔥🔥🔥🔥 - Kiali works with Istio to visualise the service mesh topology.
💚k8s-image-availability-exporter 🔥 - Prometheus exporter that warns you proactively about images that are defined in Kubernetes objects but are not available in the container registry.
💚kube-capacity 🔥🔥🔥 - This is a simple CLI that provides an overview of the resource requests, limits, and utilization in a Kubernetes cluster.
💚Kubernetes Dashboard 🔥🔥🔥🔥🔥 - Kubernetes Dashboard is a general purpose, web-based UI for Kubernetes clusters.
💚Kubedev 🔥 - Kubedev is a powerful and beautiful user interface for managing Kubernetes clusters.
💚KubeHelper 🔥 - KubeHelper - simplifies many daily Kubernetes cluster tasks through a web interface.
💚Kubernetes Metrics Server 🔥🔥🔥🔥🔥 - Metrics Server is a scalable, efficient source of container resource metrics for Kubernetes built-in autoscaling pipelines.
💚Kubernetes Operational View 🔥🔥🔥🔥 - A tool that aims to provide a common operational picture for multiple Kubernetes clusters.
💚kube-state-metrics 🔥🔥🔥🔥🔥 - kube-state-metrics is a simple service that listens to the Kubernetes API server and generates metrics about the state of the objects.
💚kubewatch 🔥🔥🔥🔥🔥 - kubewatch is a Kubernetes watcher that currently publishes notification to available collaboration hubs/notification channels.
💚Lens 🔥🔥🔥🔥🔥 - Lens it's an useful, attractive, open source user interface (UI) for working with Kubernetes clusters.
💚Mizu 🔥🔥🔥 - API traffic viewer for Kubernetes enabling you to view all API communication between microservices. Think TCPDump and Wireshark re-invented for Kubernetes
💚Network mapper 🔥🔥 - Map Kubernetes in-cluster traffic and export as text, intents, or an image.
💚Popeye 🔥🔥🔥🔥🔥 - Popeye is a utility that scans live Kubernetes cluster and reports potential issues with deployed resources and configurations.
💚Prometheus 🔥🔥🔥🔥🔥 - Prometheus, a Cloud Native Computing Foundation project, is a systems and service monitoring system.
💚Searchlight 🔥🔥 - Searchlight/Icinga periodically runs various checks on a Kubernetes cluster and sends notifications if detects an issue.
💚Sloop 🔥🔥🔥 - Sloop monitors Kubernetes, recording histories of events and resource state changes and providing visualizations to aid in debugging past events.
💚Thanos 🔥🔥🔥🔥🔥 - Thanos is a set of components that can be composed into a highly available metric system with unlimited storage capacity.
💚K8Studio 🔥🔥🔥 - K8Studio IDE to manage and visualise Kubernetes Clusters.

Backup and Restore

💚katafygio 🔥 - katafygio discovers Kubernetes objects (deployments, services, ...), and continuously save them as yaml files in a git repository.
💚Velero 🔥🔥🔥🔥🔥 - Velero (formerly Heptio Ark) gives you tools to back up and restore your Kubernetes cluster resources and persistent volumes.

Security and Compliance

💚Datree 🔥🔥🔥🔥🔥 - Datree is a CLI tool that supports Kubernetes admins in their roles by preventing developers from making errors in Kubernetes configurations that can cause clusters to fail in production.
💚Deepfence ThreatMapper 🔥🔥🔥 - Apache v2, powerful runtime vulnerability scanner for kubernetes, virtual machines and serverless.
💚Falco 🔥🔥🔥🔥🔥 - Falco is a behavioral activity monitor designed to detect anomalous activity in your applications. You can use Falco to monitor run-time security of your Kubernetes applications and internal components.
💚Gatekeeper 🔥🔥🔥🔥🔥 - Policy controller for Kubernetes
💚Intents operator 🔥🔥 - Manage network policies, Istio Authorization Policies, and Kafka ACLs in a Kubernetes cluster with ease.
💚k-rail 🔥🔥 - k-rail is a workload policy enforcement tool for Kubernetes. It can help you secure a multi tenant cluster with minimal disruption and maximum velocity.
💚Konstraint 🔥🔥 - Konstraint is a CLI tool to assist with the creation and management of constraints when using Gatekeeper.
💚kube-bench 🔥🔥🔥🔥🔥 - kube-bench is a Go application that checks whether Kubernetes is deployed securely by running the checks documented in the CIS Kubernetes Benchmark.
💚kube-hunter 🔥🔥🔥🔥🔥 - kube-hunter hunts for security weaknesses in Kubernetes clusters.
💚KubeLinter 🔥🔥🔥🔥 - KubeLinter is a static analysis tool that checks Kubernetes YAML files and Helm charts to ensure the applications represented in them adhere to best practices.
💚Kubesploit 🔥🔥🔥 - Kubesploit is a cross-platform post-exploitation HTTP/2 Command & Control server and agent dedicated for containerized environments written in Golang and built on top of Merlin project by Russel Van Tuyl (@ne0nd0g).
💚KubiScan 🔥🔥🔥 - A tool for scanning Kubernetes cluster for risky permissions in Kubernetes's Role-based access control (RBAC) authorization model.
💚Kyverno 🔥🔥🔥🔥 - Kyverno is a policy engine designed for Kubernetes. It can validate, mutate, and generate configurations using admission controls and background scans.
💚Netchecks 🔥 - Set of tools for testing network conditions and asserting that they are as expected.
💚Permission manager 🔥🔥🔥 - Permission Manager is an application developed by SIGHUP that enables a super-easy and user-friendly RBAC management for Kubernetes.
💚rakkess 🔥🔥🔥 - kubectl plugin to show an access matrix for server resources
💚Rönd 🔥 - Rönd is an open-source lightweight Kubernetes sidecar container that helps you protect your APIs with simple security policies. It also natively allows you to build your RBAC/ABAC solution.
💚Teleport 🔥🔥🔥🔥🔥 - Teleport Unified Access Plane enables engineers to quickly access any computing resource anywhere.

Service Mesh

💚Istio 🔥🔥🔥🔥🔥 - An open platform to connect, manage, and secure microservices.
💚Linkerd 🔥🔥🔥🔥🔥 - Linkerd is a transparent service mesh, designed to make modern applications safe and sane.
💚Open Service Mesh 🔥🔥🔥🔥🔥 - Open Service Mesh (OSM) is a lightweight, extensible, Cloud Native service mesh that allows users to uniformly manage, secure, and get out-of-the-box observability features for highly dynamic microservice environments.

Development Tools

💚Cyclops 🔥🔥 - Customizable UI for Kubernetes deployments
💚Eclipse JKube 🔥🔥 - Tools and plugins for Java developers that help you create container images along with the required manifests to deploy your applications to Kubernetes.
💚garden 🔥🔥🔥🔥🔥 - Garden provides production-like Kubernetes testing environments for integration tests, QA, and development.
💚gefyra 🔥🔥🔥 -Gefyra blazingly-fast, rock-solid, local application development ➡️ with Kubernetes.
💚ko 🔥🔥🔥🔥🔥 - ko is a tool for building and deploying Golang applications to Kubernetes.
💚Konfig 🔥 - Konfig is a Kubernetes friendly Rails gem. It can load configuration and secrets from both YAML or folders with individual files and present them to your application the same way.
💚kubevious 🔥🔥🔥🔥 - Kubevious renders all configurations relevant to the application in one place. That saves a lot of time from operators, eliminating the need for looking up settings and digging within selectors and labels.
💚kubectl-warp 🔥🔥 - Kubernetes CLI plugin for syncing and executing local files in Pod on Kubernetes
💚kubernix 🔥🔥🔥 - This project aims to provide single dependency Kubernetes clusters for local testing, experimenting and development purposes.
💚Makisu 🔥🔥🔥🔥🔥 - Makisu is a fast and flexible Docker image build tool designed for unprivileged containerized environments such as Mesos or Kubernetes.
💚mirrord 🔥🔥🔥🔥🔥 - mirrord connects your local process and your cloud environment, and runs local code in cloud conditions.
💚Monokle 🔥🔥🔥 - Monokle helps you create, edit and validate yaml manifests, visualize and validate resource links and dependencies, connect and compare resources to your clusters, debug the output of kustomize or helm, and more!
💚Okteto 🔥🔥🔥🔥 - okteto accelerates the development workflow of Kubernetes applications.
💚Telepresence 🔥🔥🔥🔥🔥 - Telepresence provides fast, realistic local development for Kubernetes microservices.
💚Tilt 🔥🔥🔥🔥🔥 - Tilt powers multi-service development and makes sure they behave.
💚Tye 🔥🔥🔥🔥🔥 - Tye is a developer tool that makes developing, testing, and deploying microservices and distributed applications easier.
Aptakube - A modern, lightweight and multi-cluster desktop client for Kubernetes. Connect to multiple clusters simultaneously to view, edit and manage all your resources.

Data Processing and Machine Learning

💚Kubeflow 🔥🔥🔥🔥🔥 - Kubeflow is a Cloud Native platform for machine learning based on Google’s internal machine learning pipelines.
💚nos 🔥🔥 - nos is an open-source platform to efficiently run AI workloads on Kubernetes, increasing GPU utilization and reducing infrastructure and operational costs.
💚Strimzi 🔥🔥🔥🔥🔥 - Strimzi provides a way to run an Apache Kafka cluster on Kubernetes or OpenShift in various deployment configurations.
💚Volcano 🔥🔥🔥🔥 - Volcano is a batch system built on Kubernetes.
💚yunikorn 🔥🔥 - a light-weight, universal resource scheduler for container orchestrator systems.

Data Management

💚Kubegres 🔥🔥🔥 - Kubegres is a Kubernetes operator allowing to deploy one or many clusters of PostgreSql pods with data replication and failover enabled out-of-the box.
💚Postgres Operator 🔥🔥🔥🔥🔥 - PGO, the Postgres Operator from Crunchy Data, gives you a declarative Postgres solution that automatically manages your PostgreSQL clusters.
💚MongoDB Community Kubernetes Operator 🔥🔥 - This is a Kubernetes Operator which deploys MongoDB Community into Kubernetes clusters.
💚MySQL Operator for Kubernetes 🔥 - The MYSQL Operator for Kubernetes is an Operator for Kubernetes managing MySQL InnoDB Cluster setups inside a Kubernetes Cluster.
💚Redis Operator 🔥🔥🔥 - Redis Operator creates/configures/manages redis-failovers atop Kubernetes.

Miscellaneous

💚Agones 🔥🔥🔥🔥🔥 - Agones is a library for hosting, running and scaling dedicated game servers on Kubernetes.
💚AWS Controllers for Kubernetes 🔥🔥🔥🔥 - AWS Controllers for Kubernetes (ACK) lets you define and use AWS service resources directly from Kubernetes.
💚AWS Node Termination Handler 🔥🔥🔥 - A Kubernetes Daemonset to gracefully handle EC2 instance shutdown
💚Brigade 🔥🔥🔥🔥🔥 - Brigade is the tool for creating pipelines for Kubernetes.
💚Crossplane 🔥🔥🔥🔥🔥 - Crossplane is an open source Kubernetes add-on that extends any cluster with the ability to provision and manage cloud infrastructure, services, and applications.
💚Descheduler for Kubernetes 🔥🔥🔥🔥🔥 - Descheduling pods from nodes based on policies
💚Devtron 🔥🔥🔥🔥 - It is designed as a self-serve platform for operationalizing and maintaining applications (AppOps) on kubernetes in a developer friendly way.
💚OpenCost 🔥🔥🔥🔥🔥 - OpenCost models give teams visibility into current and historical Kubernetes spend and resource allocation.
💚K8s-Cleaner 🔥 - k8s-cleaner identifies and removes unused resources.
💚K8sPurger 🔥 - K8sPurger Hunt Unused Resources In Kubernetes.
💚KubeEdge 🔥🔥🔥🔥🔥 - KubeEdge is built upon Kubernetes and extends native containerized application orchestration and device management to hosts at the Edge.
💚KubePug 🔥🔥 - A tool to check deprecations before upgrading Kubernetes version
💚Kube No Trouble 🔥🔥🔥🔥🔥 - Easily check your clusters for use of deprecated APIs
💚Shell-operator 🔥🔥🔥🔥 - Shell-operator is a tool for running event-driven scripts in a Kubernetes cluster.

Guides, Documentations, Blogs, and Learnings

Guides

A Beginner’s Guide to Kubernetes - A comprehensive introduction to Kubernetes architecture
A Deep Dive Into Kubernetes Schema Validation - A guide about the Kubernetes schema and how to validate it using OSS and native tools
A Guide to the Kubernetes Networking Model - A in-depth run-through of Kubernetes networking
Amazon EKS Best Practices Guide for Security - This guide provides advice about protecting information, systems, and assets that are reliant on EKS while delivering business value through risk assessments and mitigation strategies.
Amazon EKS Node Drainer 🔥 - A guide and an example to cordon and evict all evictable pods from an EC2 node being terminated.
Comparison of Kubernetes Ingress controllers - This research compares the capabilities of 14 different Kubernetes Ingress controllers.
Configuring HA Kubernetes cluster on bare metal servers with kubeadm - A guide to standing up a HA Kubernetes cluster on bare metal servers with kubeadm.
Introduction to Using Google Kubernetes Engine; Explain Like I’m Five! - Creating your first managed Kubernetes cluster on Google Kubernetes Engine using Terraform.
Kubernetes Network Policy Recipes 🔥🔥🔥🔥🔥 - This repository contains various use cases of Kubernetes Network Policies and sample YAML files to leverage in your setup.
Kubernetes The Hard Way 🔥🔥🔥🔥🔥 - Kubernetes The Hard Way guides you through bootstrapping a highly available Kubernetes cluster with end-to-end encryption between components and RBAC authentication.
Kubernetes Working Group for Multi-Tenancy 🔥🔥🔥 - This is a working place for multi-tenancy related proposals and prototypes.
Production grade Kubernetes Monitoring using Prometheus - A in-depth guide to deploy Prometheus monitoring solution.
The Illustrated Children’s Guide to Kubernetes - Graphical explanations of Kubernetes
Troubleshooting Kubernetes deployments - A flow chart to troubleshoot a kubernetes deployment in case of issues
- Vertical Pod Autoscaling: The Definitive Guide - An in-depth explanation on Kubernetes VPA: what it is, how it works, how to use it and which limitations it has.
Writing Your First Kubernetes Operator - In this article, we’ll see how to build and deploy your first Kubernetes Operator using the Operator SDK.

Blogs and Videos

10 most common mistakes using kubernetes - Common pitfalls and how to avoid them.
How the Department of Defense Moved to Kubernetes and Istio - Focus on the sidecar security stack leveraging Envoy and sidecar containers to ensure zero trust security and baked-in multi-layer security.
Kubernetes at Reddit: Tales from Production - Hear of successes, share in the heartbreak of production explosions, and gain insight into what has and hasn't worked well for one of the world's busiest web properties.
Kubernetes Failure Stories 🔥🔥🔥🔥🔥 - A compiled list of links to public failure stories related to Kubernetes.
Life of a Packet - Tracing the path of network traffic in the Kubernetes system.
OPA Deep Dive - Deep dive on some exciting new features in the OPA project presented by the co-creators.
Scaling Kubernetes to 2,500 Nodes + Scaling Kubernetes to 7,500 Nodes - Issues you will encounter when running high-scale Kubernetes workloads.
Service Mesh Comparison - An easy compensation to help choose one of the service Mesh implementations.
ArgoCD Best Practices

Learnings and Documentations

A Beginner’s Guide to Kubernetes - A comprehensive introduction to Kubernetes architecture
ConfigMaps in Kubernetes: how they work and what you should remember - Understanding the evolution to ConfigMaps, how they work and what happens when they change.
Configuring Redis using a ConfigMap - A walkthrough that provides a real world example of how to configure Redis using a ConfigMap
Example: Deploying Cassandra with a StatefulSet - This tutorial shows you how to run Apache Cassandra on Kubernetes. Cassandra, a database, needs persistent storage to provide data durability.
Example: Deploying PHP Guestbook application with Redis - This tutorial shows you how to build and deploy a simple, multi-tier web application using Kubernetes and Docker.
Example: Deploying WordPress and MySQL with Persistent Volumes - This tutorial shows you how to deploy a WordPress site and a MySQL database using Minikube.
Exposing an External IP Address to Access an Application in a Cluster - This guide shows how to create a Kubernetes Service object that exposes an external IP address.
kubectl Cheat Sheet - An official list of commonly used kubectl commands and flags.
Kubectl Kubernetes CheatSheet 🔥🔥🔥🔥 - A cheatsheet containing many helpful kubectl commands
Kubernetes API Reference Docs - A high-level overview of the basic types of resources provided by the Kubernetes API and their primary functions.
Learn Kubernetes Basics - This tutorial provides a walkthrough of the basics of the Kubernetes cluster orchestration system.
Play with Kubernetes - Play with Kubernetes is a playground which allows users to run K8s clusters in a matter of seconds.
Ready-to-use commands and tips for kubectl - Various kubectl tips and tricks by Flant’s engineers.
Running ZooKeeper, A Distributed System Coordinator - This tutorial demonstrates running Apache Zookeeper on Kubernetes using StatefulSets, PodDisruptionBudgets, and PodAntiAffinity.
Set Up a CI/CD Pipeline with Kubernetes - A end-to-end guide to set up a CI/CD Pipeline with Kubernetes.
StatefulSet Basics - This tutorial provides an introduction to managing applications with StatefulSets.
Webinar: K8s with OPA Gatekeeper - How to use OPA to control what end-users can do on the cluster and ways to ensure that clusters are in compliance with company policies.

Certification Guides

Certified Kubernetes Security Specialist - CKSS 🔥🔥 - This repository is a collection of resources to prepare for the Certified Kubernetes Security Specialist (CKSS) exam.
CKS "Certified Kubernetes security specialist certification 🔥🔥🔥🔥 - Kubernetes security resources primarly from material allowed during the exam, and extra optional items to help you advance your container and kubernetes security journey.
How to pass the Certified Kubernetes Administrator (CKA) exam on the first attempt - A guide to pass CKA exam
The ultimate CKA "Certfified Kuberenetes Administator" resource since exam inception - An updated repo of offical resources to help you master the CKA exam as well some extra resources to consolidate your kubernetes administration knowledge.
Kubernetes Exam Simulator - CKS/CKA/CKAD exams scenarios and environment.

Contribute

Contributions welcome! Read the contribution guidelines first.

License

To the extent possible under law, Tom Huang has waived all copyright and
related or neighboring rights to this work.

Kubernetes Deep Dive: YAML Describe Explained!

Hoang Guruu — Mon, 17 Mar 2025 07:09:05 +0000

Useful GitHub Repositories Every DevOps Engineer Should Bookmark

Hoang Guruu — Sun, 16 Mar 2025 14:52:04 +0000

🚀 Useful GitHub Repositories Every DevOps Engineer Should Bookmark 📌

(You’ll Thank Me Later)

1️⃣ Awesome Sysadmin (Sysadmin tools & resources)
↳ https://lnkd.in/ge7QusvK

2️⃣ DevOps Exercises (Practice real-world DevOps scenarios)
↳ https://lnkd.in/g_NSiHxB

3️⃣ Kubernetes the Hard Way (Step-by-step Kubernetes setup from scratch)
↳ https://lnkd.in/ghHay3Pr

4️⃣ Terraform Best Practices (Infrastructure as Code done right)
↳ https://lnkd.in/gUFTmuuG

5️⃣ Prometheus Monitoring (Monitor your infrastructure like a pro)
↳ https://lnkd.in/g5j2hvWA

6️⃣ Ansible for DevOps (Automate everything with Ansible)
↳ https://lnkd.in/gw2ynKE6

7️⃣ CI/CD Pipeline Examples (Learn different CI/CD approaches)
↳ https://lnkd.in/gR-u6hNt

8️⃣ The Art of Command Line (Master the command line like a wizard)
↳ https://lnkd.in/guNcEYpM

9️⃣ Chaos Engineering Tools (Break things to make them resilient)
↳ https://lnkd.in/g5qFU7ma

🔟 DockSTARTer (Automate Docker deployments easily)
↳ https://lnkd.in/gy-ZtyvQ

1️⃣1️⃣ Site Reliability Engineering (SRE) Case Studies
↳ https://lnkd.in/gQ678bWj

1️⃣2️⃣ Test Your Sysadmin Skills (Are you really a sysadmin?)
↳ https://lnkd.in/gb2WpkN2

1️⃣3️⃣ Awesome DevSecOps (Security for DevOps workflows)
↳ https://lnkd.in/gFErut5W

1️⃣4️⃣ Fast Kubernetes (Learn Kubernetes quickly)
↳ https://lnkd.in/gbBM7vZi

1️⃣5️⃣ Kubernetes (Official Repo) (Core Kubernetes project)
↳ https://lnkd.in/gheQpPQ5

1️⃣6️⃣ Kubernetes Ingress Nginx (Manage ingress traffic in K8s)
↳ https://lnkd.in/g2Dru-2s

1️⃣7️⃣ kubectl (K8s CLI tool) (Master Kubernetes CLI)
↳ https://lnkd.in/g4RuqvEU

1️⃣8️⃣ Kubernetes Metrics (Resource usage monitoring)
↳ https://lnkd.in/gYM3cKhB

1️⃣9️⃣ Kube-State-Metrics (Detailed monitoring for Kubernetes objects)
↳ https://lnkd.in/g27vrHZ7

🎥 Watch more DevOps & AWS videos on YouTube: 🔗https://lnkd.in/g7tpVmdV

♻ Repost to share with your network! 🚀

[ Solution Library AWS ] Live Chat Content AI on AWS

Hoang Guruu — Fri, 14 Mar 2025 02:13:33 +0000

【 👨🏻‍💻 YouTube 】
Watch more other contents related DevOps - AWS

System Architecture Diagram

Table of Contents
Overview
Architecture
Features
- Cost
- Supported AI Models
Prerequisites
Deployment
Deployment Validation
Usage
Running the Solution
- Switching Prompts/Models
- Updating the Front-End
- Scripts Usage
- Moderation Guidelines
Monitoring and Observability
Performance and Scalability
Security Considerations
Cleanup
Next Steps
Notices
Authors

Overview

The Live Chat Content Moderation with generative AI on AWS project is a scalable, multilingual, real-time chat moderation system designed for live chat platforms. It leverages AWS services and generative AI to automatically filter and moderate chat messages, ensuring a safe and engaging environment for users.

Architecture

This solution utilizes several AWS services to create a robust and scalable architecture:

Amazon API Gateway: Handles incoming WebSocket connections and REST API requests;
AWS Lambda: Processes messages and interacts with other services;
Amazon DynamoDB: Stores chat messages, approved messages, and moderation prompts;
Amazon SQS: Manages message queues for reliable processing;
AWS AppSync: Facilitates real-time updates using GraphQL subscriptions;
Amazon Bedrock: Provides generative AI capabilities for message moderation;
Amazon CloudFront & S3: Hosts and serves the front-end application;
AWS WAF: Protects your web applications from common exploits;
AWS CDK: Defines and deploys the infrastructure as code.

Features

Real-time chat moderation using generative AI;
Scalable WebSocket connections for live chat;
Multiple AI model support with easy switching;
Front-end React application for chat interface;
Comprehensive observability and monitoring;
Secure and compliant with AWS best practices.

Cost

The solution utilizes several AWS services, each contributing to the overall cost. Key services include:

Amazon API Gateway: Charges based on the number of API calls and data transfer;
AWS Lambda: Costs depend on the number of requests, duration, and memory allocated;
Amazon DynamoDB: Pricing based on read/write capacity units or on-demand pricing;
Amazon SQS: Charges per million requests;
AWS AppSync: Costs based on the number of API requests and real-time updates;
Amazon Bedrock: Pricing varies by model and is based on the number of input and output tokens;
Amazon CloudFront: Charges for data transfer and number of requests;
AWS WAF: Charges on the number of web ACLs, number of rules per web ACL, and number of web requests;
Amazon CloudWatch: Costs for log ingestion, storage, and dashboard usage;
AWS Systems Manager Parameter Store: Standard parameters are free, advanced parameters have a nominal cost.

Here is an example of a monthly cost estimate based on 1,000 users, 1,000,000 messages sent and 200,000,000 token usage based on N. Virginia (us-east-1):

AWS service: Amazon CloudFront
- Dimensions: 1,000 users requesting 64 MBs of Static Assets
- Cost [USD]: $7.84
AWS service: AWS WAF
- Dimensions: 2 Web ACLs + 2 Rules + 1,000,000 of requests
- Cost [USD]: $13.20
AWS service: Amazon Simple Storage Service (S3)
- Dimensions: 16 MBs of Static Assets
- Cost [USD]: $4.03
AWS service: Amazon API Gateway
- Dimensions: 1,000,000 REST API Requests
- Cost [USD]: $3.50
AWS service: Amazon Simple Queue Service (SQS)
- Dimensions: 1,000,000 Messages
- Cost [USD]: $0.01
AWS service: AWS Lambda
- Dimensions: 1,000,000 of Lambda Executions with 256 MBs of RAM and 2000ms average duration
- Cost [USD]: $8.53
AWS service: AWS AppSync
- Dimensions: 1,000,000 of API requests and 1,000 of connected clients with 1 hour duration
- Cost [USD]: $10.29
AWS service: Amazon Bedrock
- Dimensions: 200,000,000 of input + output tokens
- Cost [USD]: $30.20
AWS service: Amazon Bedrock Guardrails
- Dimensions: 5,000 text units (1,000,000 messages with 200 characters average)
- Cost [USD]: $3.75
AWS service: AWS Systems Manager
- Dimensions: 1 SSM Query per minute
- Cost [USD]: $0.01
AWS service: Amazon DynamoDB
- Dimensions: Reads from Prompt Store and 1,000,000 Writes to Messages Tables
- Cost [USD]: $2.27
AWS service: Amazon CloudWatch
- Dimensions: 34 Metrics, 1 Dashboard, 1 GB of Logs
- Cost [USD]: $11.21
AWS service: AWS X-Ray
- Dimensions: 20% sample of 1,000,000 messages
- Cost [USD]: $1.20

For a more accurate cost estimate, use the AWS Pricing Calculator and input your expected usage patterns.

Supported AI Models

The system supports four different AI models for chat moderation:

Anthropic Claude Haiku: Claude 3 Haiku is Anthropic's fastest, most compact model for near-instant responsiveness. It answers simple queries and requests with speed. Customers will be able to build seamless AI experiences that mimic human interactions. Claude 3 Haiku can process images and return text outputs, and features a 200K context window.
Amazon Titan: Amazon Titan Text Premier is an advanced, high-performance, and cost-effective LLM engineered to deliver superior performance for enterprise-grade text generation applications, including optimized performance for retrieval-augmented generation (RAG) and Agents.
Meta Llama: Meta Llama 3 is an accessible, open large language model (LLM) designed for developers, researchers, and businesses to build, experiment, and responsibly scale their generative AI ideas. Part of a foundational system, it serves as a bedrock for innovation in the global community. Ideal for limited computational power and resources, edge devices, and faster training times.
Amazon Nova Micro: Amazon Nova family of models offer customers multiple price performance operating points to best optimize between accuracy, speed, and cost. Amazon Nova Micro is a text only model that delivers the lowest latency responses at the lowest cost per inference among Nova family.

Each model has its own strengths and characteristics. You can switch between these models using the prompt-switch.bash script.

Prerequisites

AWS Account;
AWS CLI configured with your credentials with permissions configured;
Node.js installed (version 20.x or later) in your system;
Appropriate permissions to install npm packages in your system;
Git installed in your system;
jq installed in your system.

Deployment

Clone the repository:

git clone https://github.com/aws-samples/live-chat-content-moderation-with-genai-on-aws.git

Navigate to the project directory:

cd live-chat-content-moderation-with-genai-on-aws/scripts

Run the installation script:

This script will set up the necessary AWS resources, deploy the back-end infrastructure, and prepare the front-end application.

Deployment Validation

After running the installation script (./install.bash), follow these steps to validate the successful deployment of your Live Chat Moderation system:

Open the AWS CloudFormation console and verify the status of the stack named "ChatModeration". It should be in the "CREATE_COMPLETE" state.
Check the outputs of the CloudFormation stack for the following key resources:

*   REST API Endpoint
*   GraphQL API Endpoint
*   CloudFront Distribution Domain

Verify the creation of DynamoDB tables:

aws dynamodb list-tables --query "TableNames[?contains(@, 'ChatModeration')]"

You should see four tables: ApprovedMessagesTable, UnapprovedMessagesTable, HallucinationsTable, and PromptStoreTable.

Confirm the Lambda function deployment:

aws lambda get-function --function-name ChatModeration-MessageProcessorFunction

This should return details about the deployed function.

Verify the SQS queues:

aws sqs list-queues --queue-name-prefix ChatModeration

You should see two queues: MessageQueue.fifo and DeadLetterQueue.fifo.

Check the AppSync API:

aws appsync list-graphql-apis --query "graphqlApis[?name=='ChatModeration-GraphQLApi']"

This should return details about your GraphQL API.

Verify the CloudFront distribution:

aws cloudfront list-distributions --query "DistributionList.Items"

You should see details of your CloudFront distribution.

If all these resources are present and correctly configured, your deployment has been successful.

Usage

Running the Solution

After installation, the application should be accessible via the CloudFront URL provided in the output. To test and use the Live Chat Moderation system:

Open a web browser and navigate to the CloudFront Distribution Domain provided in ./install.bash output or the CloudFormation stack outputs.
You should see the chat interface. Try sending various types of messages:

A normal, non-offensive message: "Hello, how is everyone doing today?"
A message containing mild profanity: "This stream is damn good!"
A message with hate speech or discrimination: "I hate [directed message]"

Observe the behavior:

Normal messages should appear in the chat;
Mildly offensive messages may or may not be blocked, depending on the AI model's judgment;
Hate speech or discriminatory messages should be blocked, and you should receive a notification.

To test the real-time nature of the chat, open the application in two different browser windows and send messages from both. You should see messages appear in real-time on both screens.
To verify the multilingual support, try sending messages in different languages.

Expected Output:

Approved messages appear in the chat interface;
Blocked messages trigger a notification to the user;
The chat updates in real-time across multiple browser windows.

Switching Prompts/Models

To switch between different AI models or prompts:

./prompt-switch.bash <model-name>

Replace <model-name> with one of the available options: titan, haiku, llama or nova-micro. The aforementioned ./install.bash script configures Anthropic Claude Haiku to be used by default.

Updating the Front-End

After making changes to the front-end code, deploy updates using:

This script will build the React application and update the S3 bucket and CloudFront distribution. This is not required at the first deployment.

Scripts Usage

The scripts should be executed from the ./scripts directory. Running from another path may cause issues or the scripts may fail.

Moderation Guidelines

The AI models use the following prompt as the main guideline for moderating chat messages:

Instructions:
You are a moderation bot tasked with identifying and preventing the spread of harmful, aggressive, racist, or toxic messages in a live stream chat. Analyze the context and intent of the message, not just specific words. Respond only with y for safe messages or n for harmful messages. Do not provide any explanation.
Consider these guidelines when evaluating:
1. Hate speech or discrimination
2. Explicit threats of violence
3. Severe profanity
4. Bullying or harassment
5. Spam or excessive self-promotion
6. Selling or advertising products
7. Sharing personal information
8. Encouraging self-harm or illegal activities
If a message is ambiguous, err on the side of caution and allow it.
Your entire response MUST be either y or n, nothing else.

This prompt instructs the AI to evaluate messages based on various criteria and respond with a simple 'y' for safe messages or 'n' for harmful ones.

Monitoring and Observability

The project includes a CloudWatch dashboard for monitoring key metrics:

Bedrock model invocations and latency;
SQS queue metrics;
Lambda function metrics;
API Gateway request counts.

Access the dashboard named "ChatModeration-Monitoring" in the Amazon CloudWatch console. You can also add a service trace map to the CloudWatch dashboard by doing the following steps:

Access the X-Ray Trace Map in CloudWatch: a. Open the CloudWatch console. b. In the left navigation pane, under the X-Ray Traces section, select Trace Map.
Add the Service Map to the CloudWatch Dashboard: a. Within the Trace Map view, click on the Actions menu. b. Choose Add to dashboard. c. Select the dashboard named "ChatModeration-Monitoring".

Performance and Scalability

The Live Chat Moderation with Generative AI system is built on AWS services, which provide high scalability and performance. However, it's important to be aware of AWS service quotas and limits that may affect the system's scalability:

API Gateway: Limits on the number of WebSocket connections and API requests per second;
Lambda: Concurrent execution limits and individual function timeout limits;
DynamoDB: Read and write capacity units, which affect the rate of database operations;
SQS: Message throughput and retention limits;
Bedrock: API call rate limits and token limits for AI model invocations.

To test the system's performance and identify potential bottlenecks, you can use Locust for load testing. The project includes a Locust configuration file located at ../tests/locust/locust.conf. This file is automatically updated with the correct API endpoint when you run the publish.bash script.

To use Locust:

Install Locust:
Navigate to the Locust test directory:
Run Locust:
Open a web browser and go to http://localhost:8089 to access the Locust web interface.
Enter the number of users to simulate, spawn rate, and other test parameters.
Start the test and monitor the results to identify performance bottlenecks or scalability issues.

Remember to monitor your AWS usage and adjust service limits if needed as your chat moderation system scales.

Security Considerations

DynamoDB tables use encryption at rest;
Lambda functions use least-privilege IAM roles;
S3 bucket for static assets is not publicly accessible;
CloudFront distribution uses HTTPS.

Cleanup

To delete all resources associated with the Live Chat Moderation system:

Destroy the CDK stack:

cd backend/cdk
cdk destroy

Next Steps

To further enhance your Live Chat Moderation system:

Customize AI Prompts: Modify the prompts in the insert-prompt.bash <model-name> scripts to fine-tune the moderation criteria for your specific use case.
Implement User Authentication: Add user authentication to associate messages with verified user accounts.
You can improve DDoS attack protection of the solution by using AWS Shield Advanced to enable advanced protection up to the application layer (Layer 7 on OSI network model). Check reference blog https://aws.amazon.com/blogs/security/protect-apis-with-amazon-api-gateway-and-perimeter-protection-services/

Notices

Customers are responsible for making their own independent assessment of the information in this Guidance. This Guidance: (a) is for informational purposes only, (b) represents AWS current product offerings and practices, which are subject to change without notice, and (c) does not create any commitments or assurances from AWS and its affiliates, suppliers or licensors. AWS products or services are provided “as is” without warranties, representations, or conditions of any kind, whether express or implied. AWS responsibilities and liabilities to its customers are controlled by AWS agreements, and this Guidance is not part of, nor does it modify, any agreement between AWS and its customers.

Authors

Gabriel Costa
Juliano Baeta

Top 17 DevOps AI Tools [2025]

Hoang Guruu — Thu, 13 Mar 2025 05:07:31 +0000

Top 17 DevOps AI Tools [2025] | LambdaTest

Next-Gen App & Browser Testing Cloud

【 👨🏻‍💻 YouTube 】
Watch more other contents related DevOps - AWS

Trusted by 2 Mn+ QAs & Devs to accelerate their release cycles

Chandrika Deb

Posted On: January 17, 2025

21365 Views

21 Min Read

DevOps AI tools are transforming the software development process by breaking traditional barriers and enhancing efficiency. With these DevOps AI tools, you can automate complex tasks, enable continuous improvement and help development and QA teams to deliver high-quality software with ease. Integrating DevOps AI tools allows organizations to move beyond old-fashioned working methods and achieve new levels of productivity and performance.

TABLE OF CONTENTS

Top 17 DevOps AI Tools [2025] | LambdaTest
- Why AI in DevOps?
- LambdaTest
- GitHub Copilot
- AWS CodeGuru
- Datadog
- New Relic
- Sysdig
- Mabl
- Jenkins X
- CircleCI
- Azure DevOps
- Snyk
- Splunk
- CloudHealth
- Dynatrace
- Ansible
- IBM Watson AIOps
- Scalability
- Integration Capabilities
- Team Adoption
- Conclusion
- Frequently Asked Questions (FAQs)
- Are AI DevOps tools suitable for small teams?
- How secure are AI-powered DevOps tools?
- Can AI completely replace human DevOps engineers?
- What is the typical ROI of implementing AI DevOps tools?
- How quickly do AI DevOps tools adapt to new technologies?
- Citations
  - Chandrika Deb

Why AI in DevOps?

According to MarketResearch, the Generative AI in DevOps market is projected to grow from approximately USD 942.5 million in 2022 to an estimated USD 22,100 million by 2032, representing a remarkable 38.20% compound annual growth rate. This explosive growth reflects AI’s transformative potential in software development and operations.

Below are some of the reasons why AI is beneficial for DevOps:

Automation: By automating the process, you can manage the repetitive tasks, allowing teams to focus on strategic, high-value work across the development and testing process.
Predictive Insights: AI analyzes and helps forecast potential system issues before they impact performance, reducing downtime and risks.
Smart Resource Management: AI optimizes infrastructure usage by analyzing performance metrics and workload patterns, enabling cost-effective scaling.
Data-Driven Decisions: Advanced AI analytics provide actionable insights, helping DevOps teams work faster, help them inform decisions, and quickly adapt to market changes.
Enhanced Collaboration: AI bridges communication gaps between development and operations teams by offering transparent, data-backed recommendations and streamlining workflows.

AI integrates intelligent solutions into DevOps workflows, helping teams proactively prevent system failures, reduce manual effort, and enhance efficiency. This allows teams to focus on strategic innovation.

DevOps AI tools allow teams to auto-prioritize alerts, suggest solutions, and execute resolution playbooks, significantly reducing the time it takes to recover from software bugs. This not only accelerates workflow speed but also supports Agile processes and advances innovative practices.

Further, let’s learn various DevOps AI tools that will help development and QA teams make decisions on which tools can fulfill their DevOps requirements.

Below are some of the most popular and widely used DevOps AI tools that help developers and testers streamline workflows by automating processes and enhancing team collaboration. This list will help developers and testers choose the right tool that meets the project requirements.

LambdaTest

LambdaTest is an AI-powered test execution platform that lets you run manual and automation tests at scale by allowing you to integrate with the best CI/CD tools like Jenkins, CircleCI, GitLab, and more. This platform helps you enhance your DevOps workflows effectively.

Key Features of LambdaTest DevOps AI tool:

It accelerates test automation and DevOps workflow, offering speeds up to 70% faster than traditional cloud grids through HyperExecute.
It offers KaneAI, a GenAI native QA Agent-as-a-Service platform, that enables teams to create, debug, and refine tests using natural language, reducing the workload of QA teams and enhancing quality processes. With high-speed execution and automated issue identification, KaneAI streamlines testing and boosts efficiency.
It mitigates potential issues by providing insights through advanced AI analysis, enhancing testing quality and efficiency.
It identifies the causes of flaky tests, tracks failure history, and recognizes patterns, allowing for customized environment adjustments.
It provides AI-powered insights that analyze test data to improve automation efficiency and reduce failure rates.
It enhances team collaboration that allows teams to work together on test creation, execution, and results in real time.

GitHub Copilot

GitHub Copilot, an AI-powered code generation tool, enhances aspects of CI/CD workflows. As one of the most popular DevOps AI tools, it indirectly reduces developer effort by improving efficiency and code quality. This tool provides intelligent suggestions, helps accelerate development cycles and enables teams to deliver reliable software.

Key Features of GitHub Copilot DevOps AI tool:

It suggests code snippets and entire functions in real-time, streamlining development and reducing manual coding efforts.
It assists in various programming languages, enabling DevOps teams to work across diverse codebases without extensive language-specific expertise.
It seamlessly integrates with popular IDEs like Visual Studio Code, enhancing existing workflows without requiring additional tools.
It also integrates with Azure DevOps CI/CD pipelines, assisting in managing operations and providing insights into workloads, thereby enhancing functionality.

Note

Optimize your DevOps workflow by integrating GitHub Copilot for a seamless release cycle. Try LambdaTest Today!

AWS CodeGuru

AWS CodeGuru is an AI-driven development tool that transforms how DevOps teams address code quality, performance, and security. This DevOps AI tool leverages advanced machine learning techniques to deliver comprehensive code analysis through its two core features: CodeGuru Reviewer for automated code reviews and CodeGuru Profiler for performance optimization.

Key Features of AWS CodeGuru DevOps AI tool:

It analyzes code to identify defects, deviations from best practices, and potential security vulnerabilities, providing actionable recommendations to improve code quality.
It continuously monitors applications in production to detect performance bottlenecks, high CPU utilization, and memory inefficiencies, offering insights to enhance application performance and reduce costs.
It scans codebases to detect vulnerabilities, such as those listed in the OWASP Top Ten and CWE Top 25, and provides recommendations for remediation, enhancing the security posture of applications.
It integrates seamlessly with existing development tools and CI/CD pipelines, enabling continuous code analysis and performance monitoring without disrupting established workflows.

Datadog

Datadog is a cloud monitoring platform that transforms application management through intelligent analytics. This DevOps AI tool uses machine learning to metrics, logs, and traces, data, and it enables DevOps teams to detect performance anomalies, identify infrastructure issues, and proactively recognize potential problems before they impact users.

Key Features of Datadog DevOps AI tool:

Its DevOps copilot assists in investigating and responding to incidents more efficiently across the Datadog platform.
AI accelerates investigations by automatically correlating telemetry and surfacing outliers, anomalies, and root causes of issues across the tech stack.
It provides monitoring solutions by using machine learning, enabling teams to easily detect outliers in their application and infrastructure performance.

New Relic

New Relic is an observability platform offering real-time insights into how applications are performing based on user experience. By using this DevOps AI tool, collecting and analyzing telemetry data becomes easy, and DevOps teams can easily identify performance bottlenecks in their applications and track health metrics.

Key Features of New Relic DevOps AI tool:

Its AIOps capabilities utilize machine learning to automatically detect anomalies, correlate related incidents, and reduce alert noise, enabling teams to focus on critical issues.
It provides tools to quickly identify the root causes of issues, integrating data from various sources to offer comprehensive insights and expedite resolution.
It offers end-to-end visibility into AI-powered applications, allowing teams to monitor performance, ensure quality responses, and manage costs effectively.
It leverages large language models, enabling users to interact with their observability data using everyday language, making it accessible to a broader range of team members.
It integrates with tools like GitHub Copilot, facilitating seamless workflows and enhancing the efficiency of development and operations teams.

Sysdig

Sysdig is an innovative DevOps AI tool that leverages AI and machine learning to enhance DevOps operations across containerized environments. It works through multiple integrated mechanisms to provide comprehensive monitoring and security by providing runtime threat detection through continuous monitoring, automated vulnerability scanning of operating systems, applications, and libraries, resource optimization through intelligent allocation process and runtime analysis and detailed forensic timeline.

Key Features of Sysdig DevOps AI tool:

It accelerates human response to cloud threats through multi-step reasoning and contextual awareness. It assists security, development, and DevOps teams in collaboratively addressing security challenges more efficiently.
It leverages machine learning to detect and analyze patterns, anomalies, and potential security threats within the software stack. This enables proactive identification and resolution of issues, ensuring application stability and security.
It automatically identifies unusual behaviors and potential threats within cloud and Kubernetes environments, allowing DevOps engineers to address issues before they impact operations.
It utilizes AI-driven insights to analyze the behavior and performance of containers, microservices, and infrastructure components.

Mabl

Mabl is an innovative AI-driven test automation platform that helps to make the web application testing process fast and smooth through intelligent automation and seamless CI/CD pipeline integration. Using the Mabl DevOps AI tool will help teams deliver high-quality applications much faster through adaptive machine learning. This tool significantly reduces the need for manual test scripting and maintenance.

Key Features of Mabl DevOps AI tool:

Its expert systems and probabilistic models automatically adapt tests to changes in application elements, reducing maintenance efforts and ensuring test reliability.
Its machine learning models learn application timing to dynamically adjust tests, leading to faster and more reliable execution across different environments.
It allows users to validate complex application behaviors using natural language descriptions, simplifying the creation of comprehensive test scenarios.
It detects unexpected UI changes, ensuring a consistent user experience and maintaining application quality.
It monitors page load times and test run durations, using clustering techniques to identify potential regressions early in the development cycle.

Jenkins X

Jenkins X is a cloud-native automation platform that enhances Continuous Integration and Continuous Delivery by integrating with popular open-source projects like Kubernetes, Tekton, Kuberhealthy, Grafana, Jenkins, and Nexus. This DevOps AI tool uses Infrastructure as Code through Terraform for cloud resource management, implements GitOps for version control and cluster state management, and utilizes secure secret management through providers like Vault or cloud-hosted solutions.

Key Features of Jenkins X DevOps AI tool:

It analyzes past build data to predict potential failures before they occur, allowing engineers to address issues proactively.
Its decision-making helps in automating rollbacks in case of deployment failures, minimizing downtime.
It optimizes resource allocation during the build process, reducing costs and improving efficiency.

Subscribe to the LambdaTest YouTube channel and get more videos on DevOps, AIOps, and more.

CircleCI

CircleCI is a leading cloud-based platform for CI/CD that automates the software development process, enabling teams to build, test, and deploy applications with efficiency and precision. By integrating seamlessly with popular version control systems like GitHub, GitLab and Bitbucket, CircleCI enhances collaboration and accelerates development cycles.

This DevOps AI tool uses machine learning algorithms for job scheduling and resource allocation, ensuring faster pipeline execution, while its robust feature set and intuitive interface make it a top choice for DevOps teams aiming to improve software quality and delivery speed.

Key Features of CircleCI DevOps AI tool:

It enables parallel processing of tasks, such as building and testing, to reduce build times and speed up deployments.
It offers strong Docker integration, allowing teams to containerize applications for consistent environments across development, testing, and production stages.
It provides highly configurable pipelines for defining complex workflows, managing dependencies between jobs, and streamlining delivery processes across staging and production environments.

Azure DevOps

Azure DevOps is a comprehensive set of tools and services provided by Microsoft. It is one of the most used DevOps AI tools when integrated with Azure’s AI and machine learning services. This integration enhances CI/CD processes, test automation, and infrastructure management.

Key Features of Azure DevOps AI tool:

It automatically generates and runs test cases based on code changes, improving test coverage and reducing manual effort.
It optimizes build processes by identifying bottlenecks and recommending improvements.
It analyzes past deployments to predict the success of future deployments, reducing the risk of failures.

Snyk

Snyk is one of the most powerful DevOps AI tools that provides end-to-end security scanning capabilities across the development lifecycle, as it focuses on automatically scanning the codebases for vulnerabilities in open-source libraries and dependencies, enabling early detection and remediation of potential security issues. Performing security scans on container images, ensuring applications remain secure throughout the containerization process and offering Cloud Security Posture Management (CSPM) to identify and fix misconfigurations and security gaps in cloud infrastructure.

Key Features of Snyk DevOps AI tool:

It utilizes AI to perform Static Application Security Testing (SAST), identifying vulnerabilities in real time as developers write code. This immediate feedback allows for prompt remediation, reducing security risks early in the development process.
It improves the accuracy of vulnerability identification across application source code, open-source libraries, container infrastructures, and Infrastructure-as-Code (IaC). This comprehensive analysis ensures a robust security posture.
Its AI capabilities provide actionable fix advice directly within development tools. This integration streamlines the remediation process, enabling developers to address security issues efficiently without disrupting their workflow.
Its AI-driven continuous monitoring features alert DevOps teams to new security threats and vulnerabilities as they emerge. This proactive approach ensures that applications remain secure even as new risks develop.
It provides context from data collected across various security and observability tools. This holistic view enables organizations to assess and manage application risk more effectively.

Splunk

Splunk is an emerging game-changing platform in the data-driven world of DevOps, which takes massive amounts of machine-generated data and turns it into meaningful insights. Advanced machine learning capabilities enable Splunk to go beyond the traditional log analysis, offering organizations a powerful DevOps AI tool for proactive monitoring, security, and operational intelligence.

Key Features Splunk DevOps AI tool:

It provides guided workflows and smart assistants for users to create custom machine learning models designed for specific DevOps use cases, enhancing data analysis and decision-making processes.
It utilizes AI to offer predictive analytics, alert noise reduction, anomaly detection, adaptive thresholding, and incident correlation, providing a comprehensive AIOps solution for monitoring and maintaining IT service health.
It uses generative AI to provide a chat experience that helps users author and learn Splunk Processing Language (SPL) by interacting in plain English, offering query suggestions, explanations, and detailed breakdowns.

PagerDuty is a leading incident management platform that provides robust solutions to DevOps teams for detecting, responding to, and resolving production incidents efficiently. This tool allows organizations to minimize downtime by providing real-time monitoring and automated response systems.

The newest offering of this DevOps AI tool is PagerDuty AIOps, which incorporates artificial intelligence and automation into the capabilities. This solution helps reduce noise in the incident management process, improving defect triage efficiency and response accuracy and allows teams to automate repetitive tasks in the incident response workflow.

Key Features of PagerDuty DevOps AI tool:

It automatically detects and routes incidents to qualified team members based on expertise and availability for rapid response to critical issues.
It uses an ML-based approach to limit alert fatigue by grouping associated alerts and prioritizing relevant issues, so teams need to focus on what counts the most.
It ensures that scheduling and escalation policies are automated, thereby ensuring continuous coverage to address incidents in a structured response chain.

CloudHealth

CloudHealth by VMWare leverages AI to optimize the cloud cost. It analyzes cloud resource usage to provide intelligent recommendations, including rightsizing suggestions for workload-appropriate instance types and reserved instance opportunities for predictable workloads, driving significant cost saving. It also offers comprehensive resource management through automated tagging, categorization, and scaling recommendations based on usage patterns. The platform streamlines compliance management through automated reporting and monitoring.

Key Features of CloudHealth DevOps AI tool:

It utilizes AI to provide rightsizing recommendations across CPU, memory, disk, and network resources, ensuring efficient resource utilization and cost savings.
It Offers AI-driven insights for optimizing workloads and automating processes, enhancing operational efficiency and performance.
It provides detailed cost analysis and recommendations for Kubernetes clusters, assisting in right-sizing node pools and underlying infrastructure for optimal performance.
It improves cloud compliance and security posture with full visibility into misconfigurations based on predefined rules and custom policies, prioritizing issues based on risk scores and automating remediation.

Dynatrace

Dynatrace delivers comprehensive monitoring through Davis AI and DevOps AI tool that uses an intelligent engine that processes billions of dependencies in milliseconds to detect anomalies and perform root cause analysis. This tool helps track system metrics such as CPU performance, response times, and network traffic across the entire technology stack, from infrastructure to end-user interactions. It enables automated anomaly detection and root cause analysis, thus helping teams quickly identify issues before they affect users.

Key Features of Dynatrace DevOps AI tool:

It continuously analyzes vast data to detect anomalies, perform root cause analysis, and provide precise answers, enabling rapid issue resolution and proactive performance optimization.
It automatically identifies the root cause of performance issues, reducing the time spent on troubleshooting.
It forecasts potential issues, allowing teams to address them proactively before they impact operations.
It autonomously identifies unusual patterns or behaviors across the entire technology stack, facilitating early detection of potential problems.
It continuously monitors cloud environments, detecting and resolving issues without human intervention.

Ansible

Ansible provides streamlined automation for IT orchestration and configuration management through its declarative language. Teams can efficiently define and execute automation tasks, ensuring consistent environment management. This tool provides simplicity and effectiveness making it a valuable tool for managing complex IT infrastructures at scale.

By integrating AI, this DevOps AI tool transforms from a simple automation platform into an intelligent system capable of making real-time decisions and adapting to complex infrastructure needs.

Key Features of Ansible DevOps AI tool:

Its intelligent algorithms dynamically adjust configurations based on the infrastructure state.
It automates the detection and correction of system issues.
Its predictive resource scaling analysis of usage patterns to optimize performance and resource allocation.

IBM Watson AIOps

IBM Watson AIOps enhances IT operations through machine learning and natural language processing, offering autonomous incident management and resolution. Operating on a consumption-based pricing model, the platform delivers two core capabilities:

Predictive Analytics: Analyzes historical and real-time data to forecast potential incidents before they impact system performance, enabling proactive issue resolution and minimizing service disruptions.
Automated Troubleshooting: Leverages AI-driven analytics and NLP to determine root causes by examining log data, alerts, and change history, reducing manual intervention in incident resolution.

Key Features IBM Watson AIOps DevOps AI tool:

It automatically clusters related events to identify root causes, streamlining incident management and reducing resolution times.
It uses machine learning to detect anomalies in logs and alerts, enabling proactive issue identification before they impact operations.
It processes unstructured data from sources such as logs, tickets, and chats to extract valuable insights, enrich predictive alerts and enhance decision-making.
It identifies the root fault component and assesses the impact radius, facilitating quick problem isolation and resolution.
It provides AI-driven recommendations based on historical data, assisting teams in resolving outages swiftly and preventing future incidents.
It utilizes deep learning models to predict risks associated with code and configuration changes, offering proactive alerts to prevent potential issues.
It seamlessly connects with existing DevOps tools, aggregating data across the entire stack to provide comprehensive insights and maintain workflow continuity.

Selecting the right DevOps AI tool can be the difference between transformative efficiency and technological frustration. The process requires a holistic approach that goes beyond surface-level features, demanding a strategic evaluation of your organization’s unique needs and potential for growth.

The key is to select tools that align with your specific organizational needs and existing technology stack.

Scalability

The ideal DevOps AI tool should be able to seamlessly expand with your existing infrastructure, handling increasing complexity without performance degradation. A truly scalable tool adapts to your growth, offering flexible resource allocation and the ability to manage increasingly sophisticated workflows as your organization evolves.

By integrating AI into DevOps workflows, organizations can streamline the development process by reducing manual intervention and errors. Teams can also boost their productivity by using various DevOps automation tools that enable them to focus on innovation and continuous improvements in delivering high-quality software applications.

Implementing containerization, monitoring tools like Prometheus and Grafana, and creating collaborative environments can help businesses enhance the efficiency and reliability of AI models.

Integration Capabilities

Your DevOps AI tool must become an organic part of your existing technology landscape. This means deep, seamless integration with:

Continuous Integration/Continuous Deployment (CI/CD) pipelines
Cloud platforms
Source code management systems
Monitoring and logging infrastructure

The goal is a harmonious technological ecosystem where AI tools enhance, rather than disrupt, your existing workflows.

Team Adoption

Technical capabilities mean nothing if your team can’t effectively use the DevOps AI tools conveniently. Teams must be able to adapt to new technologies and trends to keep enhancing the development and testing process.

Consider:

Intuitive interfaces
Comprehensive documentation
Training resources
Community support
Alignment with your team’s current skill set

The most powerful DevOps AI tool must be chosen based on the requirements of the project. While AI is not meant to completely replace human judgment, it serves as a powerful tool that augments human capabilities by reducing risks, increasing efficiency, and providing insights that might be overlooked by human analysts.

Key benefits include speed of decision-making, improved accuracy, risk reduction, and the ability to create an infinite institutional memory. However, successful AI integration requires building trust, democratizing access, and seamlessly embedding these technologies into existing systems, with an ongoing emphasis on human-centered design and ethical considerations.

Conclusion

As you move forward AI is no longer an optional add-on but a necessary part of effective DevOps strategies. The right combination of AI tools can dramatically improve efficiency, reduce operational risks, and accelerate innovation. Implementing AI tools in DevOps can revolutionize workflows by automating tasks, enhancing decision-making, and boosting efficiency.

However, some of the challenges, such as data quality, integration complexity, performance strain, security risks, and high costs, need to be overcome for successful implementation. High-quality data through consistent logging and centralized pipelines, choosing compatible tools to reduce integration complexity, and utilizing cloud-based solutions to optimize resources can help overcome these obstacles.

Additionally, prioritizing AI tools with robust security features and regulatory compliance safeguards sensitive data while evaluating ROI ensures cost-effectiveness. With strategic planning and the right tools, organizations can unlock AI’s full potential to transform their DevOps processes.

Frequently Asked Questions (FAQs)

Are AI DevOps tools suitable for small teams?

Yes, many tools offer scalable pricing and can benefit teams of all sizes.

How secure are AI-powered DevOps tools?

Reputable tools prioritize security and often provide additional layers of protection through AI-driven threat detection.

Can AI completely replace human DevOps engineers?

No, AI is designed to augment and assist human capabilities, not replace skilled professionals.

What is the typical ROI of implementing AI DevOps tools?

ROI varies, but organizations often see improvements in deployment speed, resource efficiency, and reduced downtime.

How quickly do AI DevOps tools adapt to new technologies?

Most leading tools use machine learning models that continuously update and learn from vast datasets, ensuring relatively quick adaptation.

Citations

AI and DevOps in Information Technology and Its Future in the United States: https://www.researchgate.net/publication/357033152_AI_and_DevOps_in_Information-_Technology_and_Its_Future_in_the_United_States
Integrating AI into DevOps pipelines: https://ijsra.net/sites/default/files/IJSRA-2024-1838.pdf

Chandrika Deb

Chandrika is a digital marketer with 4+ years of experience in digital transformation and technical content creation. As a freelance writer specializing in DevOps and software testing, her blogs have attracted over 500,000 views. In her free time, she enjoys binge-watching Netflix and illustrating on her iPad.

Author’s Profile

Chandrika Deb

Got Questions? Drop them on LambdaTest Community. Visit now

Try LambdaTest Now !!

Get 100 minutes of automation test minutes FREE!!

We use cookies to give you the best experience. Cookies help to provide a more personalized experience and relevant advertising for you, and web analytics for us. Learn More in our Cookies policy, Privacy & Terms of service.

Top 110+ DevOps Interview Questions and Answers for 2025

Hoang Guruu — Mon, 10 Mar 2025 02:41:15 +0000

【 👨🏻‍💻 YouTube 】
Watch more other contents related DevOps - AWS

What is DevOps?

A group of activities and approaches called DevOps combines software development (Dev) with information technology operations (Ops) to improve the efficiency and quality of software development, delivery, and deployment.

DevOps' primary objective is to foster teamwork between the development and operations teams so that they may collaborate easily across the whole software development life cycle. In addition, automation, continuous integration, delivery, and deployment are used to speed up and reduce mistakes in the software development process.

Monitoring and feedback are also emphasized in DevOps, which enables the development and operations teams to see problems early and proactively handle them. Using DevOps methods, businesses may improve their agility, competitiveness, and overall productivity by achieving quicker release cycles, higher-quality software, and enhanced team cooperation.

What is a DevOps Engineer?

A DevOps engineer is an expert in developing, deploying, and maintaining software systems using the DevOps methodology and practices.

DevOps engineers collaborate closely with IT operations teams, software developers, and other stakeholders to guarantee the effective delivery of software products. To increase the efficiency and quality of software development, they implement automation, continuous integration, and continuous delivery/deployment (CI/CD) practices. They also locate and resolve issues that arise throughout the development process.

DevOps engineers often have extensive backgrounds in IT operations, systems administration, software development, scripting, automation, and cloud computing skills. They are adept at using various tools and technologies, including source code management systems, build and deployment tools, virtualization and container technologies, and monitoring and logging tools.

DevOps engineers must be proficient in technical and interpersonal skills, teamwork, and problem-solving techniques. They also need to be able to interact and collaborate successfully with coworkers from all backgrounds and disciplines since they work closely with several teams within the business.

What are the Requirements to Become a DevOps Engineer?

Different criteria for becoming a DevOps engineer may exist depending on the business and the individual function. However, some specific fundamental skills and certifications are frequently needed or recommended. They consist of the following:

Excellent technical background: DevOps engineers should be well-versed in IT operations, systems administration, and software development. A degree in computer science, information technology, or a similar discipline may be required, as well as relevant experience and certifications.
Experience with DevOps tools and methodologies: DevOps engineers should have experience with various DevOps technologies and processes, including version control systems, build and deployment automation, containerization, cloud computing, and monitoring and logging tools.
Scripting and automation skills: DevOps engineers should have strong scripting skills and be proficient in using tools such as Bash, Python, or PowerShell to automate tasks and processes.
Cloud computing experience: DevOps engineers should have experience working with cloud platforms such as Amazon Web Services (AWS), Microsoft Azure, or Google Cloud Platform (GCP).
Soft skills: DevOps engineers should be effective communicators, able to collaborate with teams across departments, and possess strong problem-solving and analytical skills.
Certification: Some organizations may require DevOps engineers to hold relevant certifications such as Certified DevOps Engineer (CDE), Certified Kubernetes Administrator (CKA), or AWS Certified DevOps Engineer - Professional.

Let us now begin with some general DevOps interview questions and answers.

General DevOps Interview Questions

1. What do you know about DevOps?

Your answer must be simple. Begin by explaining the growing importance of DevOps in the IT industry. Discuss how such an approach aims to synergize the efforts of the development and operations teams to accelerate the delivery of software products with a minimal failure rate. Include how DevOps is a value-added practice where development and operations engineers join hands throughout the product or service lifecycle, from the design stage to the deployment point.

2. How is DevOps different from agile methodology?

DevOps is a culture that allows the development and operations teams to work together. This results in continuous software development, testing, integration, deployment, and monitoring throughout the lifecycle.

Agile is a software development methodology that focuses on iterative, incremental, small, and rapid software releases and customer feedback. It addresses gaps and conflicts between customers and developers.

DevOps addresses gaps and conflicts between the Developers and IT Operations.

3. Which are some of the most popular DevOps tools?

The most popular DevOps tools include:

4. What are the different phases in DevOps?

The various phases of the DevOps lifecycle are as follows:

Plan: Initially, there should be a plan for the type of application that needs to be developed. Getting a rough picture of the development process is always a good idea.
Code: The application is coded as per the end-user requirements.
Build: Build the application by integrating various codes formed in the previous steps.
Test: This is the most crucial step of the application development. Test the application and rebuild, if necessary.
Integrate: Multiple codes from different programmers are integrated into one.
Deploy: Code is deployed into a cloud environment for further usage. It is ensured that any new changes do not affect the functioning of a high traffic website.
Operate: Operations are performed on the code if required.
Monitor: Application performance is monitored. Changes are made to meet the end-user requirements.

The above figure indicates the DevOps lifecycle.

Bridge the gap between software developers and operations and develop your career in DevOps by choosing our unique Post Graduate Program in DevOps. Enroll for the PGP in collaboration with Caltech CTME Today!

5. Mention some of the core benefits of DevOps.

The core benefits of DevOps are as follows:

Technical benefits

Continuous software delivery
Less complex problems to manage
Early detection and faster correction of defects

Business benefits

Faster delivery of features
Stable operating environments
Improved communication and collaboration between the teams

6. How will you approach a project that needs to implement DevOps?

The following standard approaches can be used to implement DevOps in a specific project:

Stage 1

An assessment of the existing process and implementation for about two to three weeks to identify areas of improvement so that the team can create a road map for the implementation.

Stage 2

Create a proof of concept (PoC). Once it is accepted and approved, the team can start implementing and rolling out the project plan.

Stage 3

The project is now ready to implement DevOps by following a step-by-step process for version control, integration, testing, deployment, delivery, and monitoring.

By following the proper steps for version control, integration, testing, deployment, delivery, and monitoring, the project is now ready for DevOps implementation.

7. What is the difference between continuous delivery and continuous deployment?

Continuous Delivery: Ensures code can be safely deployed onto production
- Continuous Deployment: Every change that passes the automated tests is deployed to production automatically
Continuous Delivery: Ensures business applications and services function as expected
- Continuous Deployment: Makes software development and the release process faster and more robust
Continuous Delivery: Delivers every change to a production-like environment through rigorous automated testing
- Continuous Deployment: There is no explicit approval from a developer, and it requires a developed culture of monitoring

8. What is the role of configuration management in DevOps?

Enables management of and changes to multiple systems.
Standardizes resource configurations, which in turn, manage IT infrastructure.
It helps with the administration and management of multiple servers and maintains the integrity of the entire infrastructure.

9. How does continuous monitoring help you maintain the entire architecture of the system?

Continuous monitoring in DevOps is a process of detecting, identifying, and reporting faults or threats in the system's entire infrastructure.

Ensures that all services, applications, and resources are running on the servers properly.
Monitors the status of servers and determines if applications are working correctly or not.
Enables continuous audit, transaction inspection, and control monitoring.

10. What is the role of AWS in DevOps?

AWS has the following role in DevOps:

Flexible services: Provides ready-to-use, flexible services without the need to install or set up the software.
Built for scale: You can manage a single instance or scale to thousands using AWS services.
Automation: AWS lets you automate tasks and processes, giving you more time to innovate
Secure: You can set user permissions and policies using AWS Identity and Access Management (IAM).
Large partner ecosystem: AWS supports a large ecosystem of partners that integrate with and extend AWS services.

11. Name three important DevOps KPIs.

The three important KPIs are as follows:

Meantime to failure recovery: This is the average time taken to recover from a failure.
Deployment frequency: The frequency in which the deployment occurs.
Percentage of failed deployments: The number of times the deployment fails.

12. Explain the term "Infrastructure as Code" (IaC) as it relates to configuration management.

Writing code to manage configuration, deployment, and automatic provisioning.
Managing data centers with machine-readable definition files, rather than physical hardware configuration.
Ensuring all your servers and other infrastructure components are provisioned consistently and effortlessly.
Administering cloud computing environments, also known as infrastructure as a service (IaaS).

13. How is IaC implemented using AWS?

Start by talking about the age-old mechanisms of writing commands onto script files and testing them in a separate environment before deployment and how IaC is replacing this approach. Similar to the codes written for other services, with the help of AWS, IaC allows developers to write, test, and descriptively maintain infrastructure entities, using formats such as JSON or YAML. This enables easier development and faster deployment of infrastructure changes.

14. Why Has DevOps Gained Prominence over the Last Few Years?

Before talking about the growing popularity of DevOps, discuss the current industry scenario. Begin with some examples of how big players such as Netflix and Facebook are investing in DevOps to automate and accelerate application deployment and how this has helped them grow their business. Using Facebook as an example, you would point to Facebook’s continuous deployment and code ownership models and how these have helped it scale up but ensure the quality of experience at the same time. Hundreds of lines of code are implemented without affecting quality, stability, and security.

Your next use case should be Netflix. This streaming and on-demand video company follows similar practices with fully automated processes and systems. Mention the user bases of these two organizations: Facebook has 2 billion users, while Netflix streams online content to more than 100 million users worldwide.

These are great examples of how DevOps can help organizations ensure higher success rates for releases, reduce the lead time between bug fixes, streamline and continuous delivery through automation, and reduce manpower costs overall.

15. What are the fundamental differences between DevOps & Agile?

The main differences between Agile and DevOps are summarized below:

Characteristics: Work Scope
- Agile: Only Agility
- DevOps: Automation needed along with Agility
Characteristics: Focus Area
- Agile: Main priority is Time and deadlines
- DevOps: Quality and Time management are of equal priority
Characteristics: Feedback Source
- Agile: The main source of feedback - customers
- DevOps: The main source of feedback - self (tools used for monitoring)
Characteristics: Practices or Processes followed
- Agile: Practices like Agile Kanban, Scrum, etc., are followed.
- DevOps: Processes and practices like Continuous Development (CD), Continuous Integration (CI), etc., are followed.
Characteristics: Development Sprints or Release cycles
- Agile: Release cycles are usually smaller.
- DevOps: Release cycles are smaller, along with immediate feedback.
Characteristics: Agility
- Agile: Only development agility is present.
- DevOps: Both in operations and development, agility is followed.

Transform your DevOps career and learn the science of improving the operational and developmental activities by choosing our PGP in DevOps. Contact our admission counselor today and grab your seat!

16. What are the anti-patterns of DevOps?

Patterns are common practices that organizations usually follow. An anti-pattern is formed when an organization continues to follow a pattern adopted by others blindly but does not work for them. Some of the myths about DevOps include:

Cannot perform DevOps → Have the wrong people
Developers do DevOps ⇒ Production Management
The solution to all the organization’s problems ⇒ DevOps
DevOps == Process
DevOps == Agile
Cannot perform DevOps → Organization is unique
A separate group needs to be made for DevOps

17. What are the benefits of using version control?

Here are the benefits of using Version Control:

With the version control system (VCS), all team members are free to work on any file at any time. Later, VCS will allow the team to integrate all of the modifications into a single version.
The VCS asks us to provide a brief summary of what was changed every time we save a new version of the project. We also get to examine exactly what was modified in the file, allowing us to see who made what changes to the project.
Inside the VCS, all the previous variants and versions are properly stored. We can request any version at any moment and retrieve a snapshot of the entire project at our fingertips.
A distributed VCS, such as Git, lets all team members retrieve a complete history of the project. This allows developers or other stakeholders to use the local Git repositories of any of the teammates even if the main server goes down at any point.

18. Describe the branching strategies you have used.

This question is usually asked to test our knowledge of the purpose of branching and our experience of branching at a past job.

Below topics can help in answering this DevOps interview question -

Release branching - We can clone the develop branch to create a Release branch once it has enough functionality for a release. This branch kicks off the next release cycle; thus, no new features can be contributed beyond this point. The things that can be contributed are documentation generation, bug fixing, and other release-related tasks. The release is merged into the master and given a version number once it is ready to ship. It should also be merged into the development branch, which may have evolved since the initial release.
Feature branching - This branching model maintains all modifications for a specific feature contained within a branch. The branch gets merged into master once the feature has been completely tested and approved by using tests that are automated.
Task branching - In this branching model, every task is implemented in its respective branch. The task key is mentioned in the branch name. We need to simply look at the task key in the branch name to discover which code implements which task.

19. Can you explain the “Shift left to reduce failure” concept in DevOps?

Shift left is a DevOps idea for improving security, performance, and other factors. Let us take an example: if we look at all of the processes in DevOps, we can state that security is tested before the deployment step. By employing the left shift method, we can add security in the development phase, which is on the left. [will be depicted in a diagram] We can integrate with all phases, including before and during testing, not just development. This most likely raises the security level by detecting faults early.

20. What is the Blue/Green Deployment Pattern?

This is a method of continuous deployment commonly used to reduce downtime. Traffic is transferred from one instance to another. To include a fresh version of the code, we must replace the old code with a new version.

The new version exists in a green environment, and the old one in a blue environment. After making changes to the previous version, we need a new instance from the old one to execute a newer version of the instance.

21. What is Continuous Testing?

Continuous Testing constitutes automated tests as part of the software delivery pipeline to provide instant feedback on the business risks present in the most recent release. To prevent problems in step-switching in the Software delivery life-cycle and to allow Development teams to receive immediate feedback, every build is continually tested in this manner. This results in a significant increase in a developer's productivity speed as it eliminates the requirement for re-running all the tests after each update and project re-building.

22. What is Automation Testing?

Test automation or manual testing Automation is the process of automating a manual procedure to test an application or system. It entails using independent testing tools to develop test scripts that can be run repeatedly without the need for human interaction.

23. What are the benefits of Automation Testing?

Some of the advantages of Automation Testing are -

Helps to save money and time.
Unattended execution can be easily done.
Huge test matrices can be easily tested.
Parallel execution is enabled.
Reduced human-generated errors, which results in improved accuracy.
Repeated test tasks execution is supported.

24. How to automate Testing in the DevOps lifecycle?

Developers are obliged to commit all source code changes to a shared DevOps repository.

Every time a change is made in the code, Jenkins-like Continuous Integration tools will grab it from this common repository and deploy it for Continuous Testing, which is done by tools like Selenium.

25. Why is Continuous Testing important for DevOps?

Continuous testing allows for immediate testing of any code modification. This prevents concerns like quality issues and release delays that might occur whenever big-bang testing is delayed until the end of the cycle. In this way, Continuous Testing allows for high-quality and more frequent releases.

26. What are the key elements of Continuous Testing tools?

Continuous Testing key elements are:

Test Optimization: It guarantees that tests produce reliable results and actionable information. Test Data Management, Test Optimization Management, and Test Maintenance are examples of aspects.
Advanced Analysis: To avoid problems and achieve more within each iteration, it employs automation in areas like scope assessment/prioritization, change effect analysis, and static code analysis.
Policy Analysis: It guarantees that all processes align with the organization's changing business needs and that all compliance requirements are met.
Risk Assessment: Test coverage optimization, technical debt, risk mitigation duties, and quality evaluation are all covered to guarantee the build is ready to move on to the next stage.
Service Virtualization: Ensures that real-world testing scenarios are available. Service visualisation provides access to a virtual representation of the needed testing phases, ensuring its availability and reducing the time spent setting up the test environment.
Requirements Traceability: It guarantees that no rework is necessary and that real criteria are met. An object evaluation is used to determine which needs require additional validation, are in jeopardy and are performing as expected.

DevOps Interview Questions for Source Code Management: Git

27. Explain the difference between a centralized and distributed version control system (VCS).

Centralized Version Control System

All file versions are stored on a central server
No developer has a copy of all files on a local system
If the central server crashes, all data from the project will be lost

Distributed Control System

Every developer has a copy of all versions of the code on their systems
Enables team members to work offline and does not rely on a single location for backups
There is no threat, even if the server crashes

28. What is the git command that downloads any repository from GitHub to your computer?

The git command that downloads any repository from GitHub to your computer is git clone.

29. How do you push a file from your local system to the GitHub repository using Git?

First, connect the local repository to your remote repository:

git remote add origin [copied web address]

// Ex: git remote add origin https://github.com/Simplilearn-github/test.git

Second, push your file to the remote repository:

git push origin master

30. How is a bare repository different from the standard way of initializing a Git repository?

Using the standard method:

git init

You create a working directory with git init
A .git subfolder is created with all the git-related revision history

Using the bare way

git init --bare

It does not contain any working or checked out a copy of source files
Bare repositories store git revision history in the root folder of your repository, instead of the .git subfolder

31. Which of the following CLI commands can be used to rename files?

git rm
git mv
git rm -r
None of the above

The correct answer is B) git mv

32. What is the process for reverting a commit that has already been pushed and made public?

There are two ways that you can revert a commit:

Remove or fix the bad file in a new commit and push it to the remote repository. Then commit it to the remote repository using: git commit –m "commit message"
Create a new commit that undoes all the changes that were made in the bad commit. Use the following command: git revert

Example: git revert 56de0938f

33. Explain the difference between git fetch and git pull.

Git fetch: Git fetch only downloads new data from a remote repository
- Git pull: Git pull updates the current HEAD branch with the latest changes from the remote server
Git fetch: Does not integrate any new data into your working files
- Git pull: Downloads new data and integrate it with the current working files
Git fetch: Users can run a Git fetch at any time to update the remote-tracking branches
- Git pull: Tries to merge remote changes with your local ones
Git fetch: Command - git fetch origin git fetch –-all
- Git pull: Command - git pull origin master

34. What is Git stash?

A developer working with a current branch wants to switch to another branch to work on something else, but the developer doesn't want to commit changes to your unfinished work. The solution to this issue is Git stash. Git stash takes your modified tracked files and saves them on a stack of unfinished changes that you can reapply at any time.

35. Explain the concept of branching in Git.

Suppose you are working on an application, and you want to add a new feature to the app. You can create a new branch and build the new feature on that branch.

By default, you always work on the master branch
The circles on the branch represent various commits made on the branch
After you are done with all the changes, you can merge it with the master branch

36. What is the difference between Git Merge and Git Rebase?

Suppose you are working on a new feature in a dedicated branch, and another team member updates the master branch with new commits. You can use these two functions:

Git Merge

To incorporate the new commits into your feature branch, use Git merge.

Creates an extra merge commit every time you need to incorporate changes
But, it pollutes your feature branch history

Git Rebase

As an alternative to merging, you can rebase the feature branch on to master.

Incorporates all the new commits in the master branch
It creates new commits for every commit in the original branch and rewrites project history

37. How do you find a list of files that have been changed in a particular commit?

The command to get a list of files that have been changed in a particular commit is:

git diff-tree –r {commit hash}

Example: git diff-tree –r 87e673f21b

-r flag instructs the command to list individual files
commit hash will list all the files that were changed or added in that commit

38. What is a merge conflict in Git, and how can it be resolved?

A Git merge conflict happens when merge branches compete for commits, and Git needs your help deciding which changes to incorporate in the final merge.

Manually edit the conflicted file to select the changes you want to keep in the final merge.

Resolve using GitHub conflict editor

This is done when a merge conflict occurs after competing for line changes. For example, it may occur when people make different changes to the same line of the same file on different branches in your Git repository.

Resolving a merge conflict using conflict editor:
Under your repository name, click "Pull requests."

In the "Pull requests" drop-down, click the pull request with a merge conflict that you'd like to resolve
Near the bottom of your pull request, click "Resolve conflicts."

Decide if you want to keep only your branch's changes, the other branch's changes, or make a brand new change that may incorporate changes from both branches.
Delete the conflict markers <<<<<<<, =======, >>>>>>> and make the changes you want in the final merge.

If you have more than one merge conflict in your file, scroll down to the next set of conflict markers and repeat steps four and five to resolve your merge conflict.
Once you have resolved all the conflicts in the file, click Mark as resolved.

If you have more than one file with a conflict, select the next file you want to edit on the left side of the page under "conflicting files" and repeat steps four to seven until you've resolved all of your pull request's merge conflicts.

Once you've resolved your merge conflicts, click Commit merge. This merges the entire base branch into your head branch.

To merge your pull request, click Merge pull request.
A merge conflict is resolved using the command line.
Open Git Bash.
Navigate into the local Git repository that contains the merge conflict.

Generate a list of the files that the merge conflict affects. In this example, the file styleguide.md has a merge conflict.

Open any text editor, such as Sublime Text or Atom, and navigate to the file with merge conflicts.
To see the beginning of the merge conflict in your file, search the file for the conflict marker "<<<<<<<. " Open it, and you'll see the changes from the base branch after the line "<<<<<<< HEAD."
Next, you'll see "=======", which divides your changes from the changes in the other branch, followed by ">>>>>>> BRANCH-NAME".

Decide if you only want to keep your branch's changes, the other branch's changes, or make a brand new change, which may incorporate changes from both branches.
Delete the conflict markers "<<<<<<<", "=======", ">>>>>>>" and make the changes you want in the final merge.

In this example, both the changes are incorporated into the final merge:

Add or stage your changes.

Commit your changes with a comment.

Now, you can merge the branches on the command line or push your changes to your remote repository on GitHub and merge them in a pull request.

39. What is Git bisect? How can you use it to determine the source of a (regression) bug?

Git bisect is a tool that uses binary search to locate the commit that triggered a bug.

Git bisect command -

git bisect

The git bisect command is used in finding the bug performing a commit in the project by using a binary search algorithm.

The bug-occurring commit is called the “bad” commit, and the commit before the bug occurs is called the “good” commit. We convey the same to the git bisect tool, and it picks a random commit between the two endpoints and prompts whether that one is the “good” or “bad” one. The process continues until the range is narrowed down and the exact commit that introduced the exact change is discovered.

40. Explain some basic Git commands.

Some of the Basic Git Commands are summarized in the below table -

Command: git init
- Purpose: Used to start a new repository.
Command: git config:git config –global user.name “[name]”git config –global user.email “[email address]”
- Purpose: This helps to set the username and email to whom the commits belong to.
Command: git clone
- Purpose: Used to create a local copy of an existing repository.
Command: git add:git add git add .
- Purpose: Used to add one or more files to the staging area.
Command: git commit:git commit -a git commit -m “”
- Purpose: Creates a snapshot or records of the file(s) that are in the staging area.
Command: git diff:git diff [first branch] [second branch]git diff -staged
- Purpose: Used to show differences between the two mentioned branches/differences made in the files in the staging area vs current version.
Command: git status
- Purpose: Lists out all the files that are to be committed.
Command: git rm
- Purpose: Used to delete a file(s) from the current working directory and also stages it.
Command: git show
- Purpose: Shows the content changes and metadata of the mentioned commit.
Command: git branch:git branch [branch name]git branch -d [branch name]git branch
- Purpose: The first one creates a brand new branch.The second is used to delete the mentioned branch.The last one lists out all the branches available and also highlights the branch we are in currently.

DevOps Interview Questions for Continuous Integration: Jenkins

41. Explain the master-slave architecture of Jenkins.

Jenkins master pulls the code from the remote GitHub repository every time there is a code commit.
It distributes the workload to all the Jenkins slaves.
On request from the Jenkins master, the slaves carry out, builds, test, and produce test reports.

42. What is Jenkinsfile?

Jenkinsfile contains the definition of a Jenkins pipeline and is checked into the source control repository. It is a text file.

It allows code review and iteration on the pipeline.
It permits an audit trail for the pipeline.
There is a single source of truth for the pipeline, which can be viewed and edited.

43. Which of the following commands runs Jenkins from the command line?

java –jar Jenkins.war
java –war Jenkins.jar
java –jar Jenkins.jar
java –war Jenkins.war

The correct answer is A) java –jar Jenkins.war

44. What concepts are key aspects of the Jenkins pipeline?

Pipeline: User-defined model of a CD pipeline. The pipeline's code defines the entire build process, which includes building, testing, and delivering an application
Node: A machine that is part of the Jenkins environment and capable of executing a pipeline
Step: A single task that tells Jenkins what to do at a particular point in time
Stage: Defines a conceptually distinct subset of tasks performed through the entire pipeline (build, test, deploy stages)

45. Which file is used to define dependency in Maven?

build.xml
pom.xml
dependency.xml
Version.xml

The correct answer is B) pom.xml

46. Explain the two types of pipelines in Jenkins, along with their syntax.

Jenkins provides two ways of developing a pipeline code: Scripted and Declarative.

Scripted Pipeline: It is based on Groovy script as their Domain Specific Language. One or more node blocks do the core work throughout the entire pipeline.

Syntax:

Executes the pipeline or any of its stages on any available agent
Defines the build stage
Performs steps related to building stage
Defines the test stage
Performs steps related to the test stage
Defines the deploy stage
Performs steps related to the deploy stage

Declarative Pipeline: It provides a simple and friendly syntax to define a pipeline. Here, the pipeline block defines the work done throughout the pipeline.

Syntax:

Executes the pipeline or any of its stages on any available agent
Defines the build stage
Performs steps related to building stage
Defines the test stage
Performs steps related to the test stage
Defines the deploy stage
Performs steps related to the deploy stage

47. How do you create a backup and copy files in Jenkins?

In order to create a backup file, periodically back up your JENKINS_HOME directory.

In order to create a backup of Jenkins setup, copy the JENKINS_HOME directory. You can also copy a job directory to clone or replicate a job or rename the directory.

48. How can you copy Jenkins from one server to another?

Move the job from one Jenkins installation to another by copying the corresponding job directory.
Create a copy of an existing job by making a clone of a job directory with a different name.
Rename an existing job by renaming a directory.

49. Name three security mechanisms Jenkins uses to authenticate users.

Jenkins uses an internal database to store user data and credentials.
Jenkins can use the Lightweight Directory Access Protocol (LDAP) server to authenticate users.
Jenkins can be configured to employ the authentication mechanism that the deployed application server uses.

50. How is a custom build of a core plugin deployed?

Steps to deploy a custom build of a core plugin:

Copy the .hpi file to $JENKINS_HOME/plugins
Remove the plugin's development directory
Create an empty file called .hpi.pinned
Restart Jenkins and use your custom build of a core plugin

51. How can you temporarily turn off Jenkins security if the administrative users have locked themselves out of the admin console?

When security is enabled, the Config file contains an XML element named useSecurity that will be set to true.
By changing this setting to false, security will be disabled the next time Jenkins is restarted.

52. What are the ways in which a build can be scheduled/run in Jenkins?

By source code management commits.
After the completion of other builds.
Scheduled to run at a specified time.
Manual build requests.

53. What are the commands that you can use to restart Jenkins manually?

Two ways to manually restart Jenkins:

(Jenkins_url)/restart // Forces a restart without waiting for builds to complete
(Jenkins_url)/safeRestart // Allows all running builds to complete before it restarts

54. Explain how you can set up a Jenkins job?

To create a Jenkins Job, we go to the top page of Jenkins, choose the New Job option and then select Build a free-style software project.

The elements of this freestyle job are:

Optional triggers for controlling when Jenkins builds.
Optional steps for gathering data from the build, like collecting javadoc, testing results and/or archiving artifacts.
A build script (ant, maven, shell script, batch file, etc.) that actually does the work.
Optional source code management system (SCM), like Subversion or CVS.

DevOps Engineer is one of the top emerging jobs of this decade. Explore the endless opportunities and get hands-on experience of working on several projects by choosing our DevOps Engineer Certification Course. Contact us and reserve your seat TODAY!

DevOps Interview Questions for Continuous Testing: Selenium

55. What are the different Selenium components?

Selenium has the following components:

Selenium Integrated Development Environment (IDE)

It has a simple framework and should be used for prototyping.
It has an easy-to-install Firefox plug-in.

Selenium Remote Control (RC)

Testing framework for a developer to write code in any programming language (Java, PHP, Perl, C#, etc.).

Selenium WebDriver

Applies a better approach to automate browser activities.
It does not rely on JavaScript.

Selenium Grid

Works with Selenium RC and runs tests on different nodes using browsers.

56. What are the different exceptions in Selenium WebDriver?

Exceptions are events that occur during the execution of a program and disrupt the normal flow of a program's instructions. Selenium has the following exceptions:

TimeoutException: It is thrown when a command performing an operation does not complete in the stipulated time.
NoSuchElementException: It is thrown when an element with specific attributes is not found on the web page.
ElementNotVisibleException: It is thrown when an element is present in Document Object Model (DOM) but is not visible. Ex: Hidden Elements defined in HTML using type=“hidden”.
SessionNotFoundException: The WebDriver is performing the action immediately after quitting the browser.

57. Can Selenium test an application on an Android browser?

Selenium is capable of testing an application on an Android browser using an Android driver. You can use the Selendroid or Appium framework to test native apps or web apps in the Android browser. The following is a sample code:

58. What are the different test types that Selenium supports?

Functional: This is a type of black-box testing in which the test cases are based on the software specification.

Regression: This testing helps to find new errors, regressions, etc. in different functional and non-functional areas of code after the alteration.

Load Testing: This testing seeks to monitor the response of a device after putting a load on it. It is carried out to study the behavior of the system under certain conditions.

59. How can you access the text of a web element?

Get command is used to retrieve the text of a specified web element. The command does not return any parameter but returns a string value.

Used for:

Verification of messages
Labels
Errors displayed on the web page

Syntax:

String Text=driver.findElement(By.id(“text”)).getText();

60. How can you handle keyboard and mouse actions using Selenium?

You can handle keyboard and mouse events with the advanced user interaction API. The advanced user interactions API contains actions and action classes.

Method	Description
clickAndHold()	Clicks without releasing the current mouse location
dragAndDrop()	Performs click-and-hold at the location of the source element
keyDown(modifier_key)	Performs a modifier key press (ctrl, shift, Fn, etc.)
keyUp(modifier_key)	Performs a key release

60. Which of these options is not a WebElement method?

getText()
size()
getTagName()
sendKeys()

The correct answer is B) size()

61. When do we use findElement() and findElements()?

findElement()

It finds the first element in the current web page that matches the specified locator value.

Syntax:

WebElement element=driver.findElements(By.xpath(“//div[@id=‘example’]//ul//li”));

findElements()

It finds all the elements in the current web page that matches the specified locator value.

Syntax:

List elementList=driver.findElements(By.xpath(“//div[@id=‘example’]//ul//li”));

62. What are driver.close() and driver.quit() in WebDriver?

These are two different methods used to close the browser session in Selenium WebDriver:

driver.close(): This is used to close the current browser window on which the focus is set. In this case, there is only one browser open.
driver.quit(): It closes all the browser windows and ends the WebDriver session using the driver.dispose method.

63. How can you submit a form using Selenium?

The following lines of code will let you submit a form using Selenium:

WebElement el = driver.findElement(By.id(“ElementID”));

el.submit();

64. What are the Testing types supported by Selenium?

There are two types of testing that are primarily supported by Selenium:

Functional Testing: Individual testing of software functional points or features.

Regression Testing: Wherever a bug is fixed, a product is retested and this is called Regression Testing.

65. What is Selenium IDE?

Selenium integrated development environment (IDE) is an all-in-one Selenium script development environment. It may be used to debug tests, alter and record and is also available as a Firefox extension. Selenium IDE comes with the whole Selenium Core that allows us to rapidly and easily replay and record tests in the exact environment where they will be conducted.

Selenium IDE is the best environment for building Selenium tests, regardless of the style of testing we prefer, thanks to the ability to move instructions around rapidly and the autocomplete support.

66. What is the difference between Assert and Verify commands in Selenium?

The difference between Verify and Assert commands in Selenium are:

The verify commands determine whether or not the provided condition is true. The program execution does not halt regardless of whether the condition is true or not, i.e., all test steps will be completed, and verification failure will not stop the execution.
The assert command determines whether a condition is false or true. To know whether the supplied element is on the page or not, we do the following. The next test step will be performed by the program control, if the condition is true. However, no further tests will be run, and the execution will halt, if the condition is false.

67. How to launch Browser using WebDriver?

To launch Browser using WebDriver, following syntax is followed -

WebDriver driver = new InternetExplorerDriver();

WebDriver driver = new ChromeDriver();

WebDriver driver = new FirefoxDriver();

68. What is the difference between Asset Management and Configuration Management?

Differences between Configuration Management and Asset Management are:

Configuration Management	Asset Management
Operational Relationships.	Incidental relationships only.
Maintains troubleshooting data.	Maintains taxes data.
Everything we deploy is scope.	Everything we own is scope.
Deployment to retirement - lifecycle.	Purchase to disposal - lifecycle.
Operations - main concern.	Finances - main concern.
ITIL processes from interfacing.	Leasing and purchasing from interfacing.

DevOps Interview Questions for Configuration Management: Chef, Puppet, Ansible

69. Why are SSL certificates used in Chef?

SSL certificates are used between the Chef server and the client to ensure that each node has access to the right data.
Every node has a private and public key pair. The public key is stored at the Chef server.
When an SSL certificate is sent to the server, it will contain the private key of the node.
The server compares this against the public key in order to identify the node and give the node access to the required data.

70. Which of the following commands would you use to stop or disable the 'httpd' service when the system boots?

# systemctl disable httpd.service
# system disable httpd.service
# system disable httpd
# systemctl disable httpd.service

The correct answer is A) # systemctl disable httpd.service

71. What is Test Kitchen in Chef?

Test Kitchen is a command-line tool in Chef that spins up an instance and tests the cookbook on it before deploying it on the actual nodes.

Here are the most commonly used kitchen commands:

72. How does chef-apply differ from chef-client?

chef-apply is run on the client system.

chef-apply applies the recipe mentioned in the command on the client system.

$ chef-apply recipe_name.rb
chef-client is also run on the client system.

chef-client applies all the cookbooks in your server's run list to the client system.

$ knife chef-client

73. What is the command to sign the requested certificates?

For Puppet version 2.7:

# puppetca –sign hostname-of-agent

Example:

# puppetca –sign ChefAgent

# puppetca sign hostname-of-agent

Example:

# puppetca sign ChefAgent
For Puppet version 2.7:

# puppetca –sign hostname-of-agent

Example:

# puppetca –sign ChefAgent

# puppetca sign hostname-of-agent

Example:

# puppetca sign ChefAgent

74. Which open-source or community tools do you use to make Puppet more powerful?

Changes in the configuration are tracked using Jira, and further maintenance is done through internal procedures.
Version control takes the support of Git and Puppet's code manager app.
The changes are also passed through Jenkin's continuous integration pipeline.

75. What are the resources in Puppet?

Resources are the basic units of any configuration management tool.
These are the features of a node, like its software packages or services.
A resource declaration, written in a catalog, describes the action to be performed on or with the resource.
When the catalog is executed, it sets the node to the desired state.

76. What is a class in Puppet?

Classes are named blocks in your manifest that configure various functionalities of the node, such as services, files, and packages.

The classes are added to a node's catalog and are executed only when explicitly invoked.

Class apache (String $version = ‘latest’) {

package{

‘httpd’: ensure => $version,

before => File[‘/etc/httpd.conf’],}

77. What is an Ansible role?

An Ansible role is an independent block of tasks, variables, files, and templates embedded inside a playbook.

This playbook installs tomcat on node1.

78. When should I use '{{ }}'?

Always use {{}} for variables, unless you have a conditional statement, such as "when: …". This is because conditional statements are run through Jinja, which resolves the expressions.

For example:

echo “This prints the value of {{foo}}”

when : foo is defined

Using brackets makes it simpler to distinguish between strings and undefined variables.

This also ensures that Ansible doesn't recognize the line as a dictionary declaration.

79. What is the best way to make content reusable/redistributable?

There are three ways to make content reusable or redistributable in Ansible:

Roles are used to managing tasks in a playbook. They can be easily shared via Ansible Galaxy.
"include" is used to add a submodule or another file to a playbook. This means a code written once can be added to multiple playbooks.
"import" is an improvement of "include," which ensures that a file is added only once. This is helpful when a line is run recursively.

80. How is Ansible different from Puppet?

Ansible	Puppet
Easy agentless installation	Agent-based installation
Based on Python	Based on Ruby
Configuration files are written in YAML	Configuration files are written in DSL
No support for Windows	Support for all popular OS's

We will now look at some DevOps interview questions on contanerization.

DevOps Interview Questions on Containerization

81. Explain the architecture of Docker.

Docker uses a client-server architecture.
Docker Client is a service that runs a command. The command is translated using the REST API and is sent to the Docker Daemon (server).
Docker Daemon accepts the request and interacts with the operating system to build Docker images and run Docker containers.
A Docker image is a template of instructions, which is used to create containers.
Docker container is an executable package of an application and its dependencies together.
Docker registry is a service to host and distribute Docker images among users.

82. What are the advantages of Docker over virtual machines?

Criteria: Memory space
- Virtual Machine : Occupies a lot of memory space
- Docker: Docker containers occupy less space
Criteria: Boot-up time
- Virtual Machine : Long boot-up time
- Docker: Short boot-up time
Criteria: Performance
- Virtual Machine : Running multiple virtual machines leads to unstable performance
- Docker: Containers have a better performance, as they are hosted in a single Docker engine
Criteria: Scaling
- Virtual Machine : Difficult to scale up
- Docker: Easy to scale up
Criteria: Efficiency
- Virtual Machine : Low efficiency
- Docker: High efficiency
Criteria: Portability
- Virtual Machine : Compatibility issues while porting across different platforms
- Docker: Easily portable across different platforms
Criteria: Space allocation
- Virtual Machine : Data volumes cannot be shared
- Docker: Data volumes are shared and used again across multiple containers

83. How do we share Docker containers with different nodes?

It is possible to share Docker containers on different nodes with Docker Swarm.
Docker Swarm is a tool that allows IT administrators and developers to create and manage a cluster of swarm nodes within the Docker platform.
A swarm consists of two types of nodes: a manager node and a worker node.

84. What are the commands used to create a Docker swarm?

Create a swarm where you want to run your manager node.

Docker swarm init --advertise-addr
Once you've created a swarm on your manager node, you can add worker nodes to your swarm.
When a node is initialized as a manager, it immediately creates a token. In order to create a worker node, the following command (token) should be executed on the host machine of a worker node.

docker swarm join \ --token SWMTKN-1-49nj1cmql0jkz5s954yi3oex3nedyz0fb0xx14ie39trti4wxv-8vxv8rssmk743ojnwacrr2e7c \ 192.168.99.100:2377

85. How do you run multiple containers using a single service?

It is possible to run multiple containers as a single service with Docker Compose.
Here, each container runs in isolation but can interact with each other.
All Docker Compose files are YAML files.

86. What is a Dockerfile used for?

A Dockerfile is used for creating Docker images using the build command.
With a Docker image, any user can run the code to create Docker containers.
Once a Docker image is built, it's uploaded in a Docker registry.
From the Docker registry, users can get the Docker image and build new containers whenever they want.

87. Explain the differences between Docker images and Docker containers.

Docker Images : Docker images are templates of Docker containers
- Docker Container: Containers are runtime instances of a Docker image
Docker Images : An image is built using a Dockerfile
- Docker Container: Containers are created using Docker images
Docker Images : It is stored in a Docker repository or a Docker hub
- Docker Container: They are stored in the Docker daemon
Docker Images : The image layer is a read-only filesystem
- Docker Container: Every container layer is a read-write filesystem

88. Instead of YAML, what can you use as an alternate file for building Docker compose?

To build a Docker compose, a user can use a JSON file instead of YAML. In case a user wants to use a JSON file, he/she should specify the filename as given:

Docker-compose -f Docker-compose.json up

89. How do you create a Docker container?

Task: Create a MySQL Docker container

A user can either build a Docker image or pull an existing Docker image (like MySQL) from Docker Hub.

Now, Docker creates a new container MySQL from the existing Docker image. Simultaneously, the container layer of the read-write filesystem is also created on top of the image layer.

Command to create a Docker container: Docker run -t –i MySQL
Command to list down the running containers: Docker ps

90. What is the difference between a registry and a repository?

Registry: A Docker registry is an open-source server-side service used for hosting and distributing Docker images
- Repository: The repository is a collection of multiple versions of Docker images
Registry: In a registry, a user can distinguish between Docker images with their tag names
- Repository: It is stored in a Docker registry
Registry: Docker also has its own default registry called Docker Hub
- Repository: It has two types: public and private repositories

91. What are the cloud platforms that support Docker?

The following are the cloud platforms that Docker runs on:

Amazon Web Services
Microsoft Azure
Google Cloud Platform
Rackspace

92. What is the purpose of the expose and publish commands in Docker?

Expose

Expose is an instruction used in Dockerfile.
It is used to expose ports within a Docker network.
It is a documenting instruction used at the time of building an image and running a container.
Expose is the command used in Docker.
Example: Expose 8080

Publish

Publish is used in a Docker run command.
It can be used outside a Docker environment.
It is used to map a host port to a running container port.
--publish or –p is the command used in Docker.
Example: docker run –d –p 0.0.0.80:80

Now, let's have a look at the DevOps interview questions for continuous monitoring.

DevOps Interview Questions for Continuous Monitoring

93. How does Nagios help in the continuous monitoring of systems, applications, and services?

Nagios enables server monitoring and the ability to check if they are sufficiently utilized or if any task failures need to be addressed.

Verifies the status of the servers and services
Inspects the health of your infrastructure
Checks if applications are working correctly and web servers are reachable

94. How does Nagios help in the continuous monitoring of systems, applications, and services?

95. What do you mean by Nagios Remote Plugin Executor (NPRE) of Nagios?

Nagios Remote Plugin Executor (NPRE) enables you to execute Nagios plugins on Linux/Unix machines. You can monitor remote machine metrics (disk usage, CPU load, etc.)

The check_npre plugin that resides on the local monitoring machine
The NPRE daemon that runs on the remote Linux/Unix machine

With diverse real-life projects and masterclasses from Caltech CTME faculty, this Post Graduate Program in DevOps can help you accelerate your DevOps career in just 9 months. Enroll today for a life-changing experience!

96. What are the port numbers that Nagios uses for monitoring purposes?

Usually, Nagios uses the following port numbers for monitoring:

97. What are active and passive checks in Nagios?

Nagios is capable of monitoring hosts and services in two ways:

Actively

Active checks are initiated as a result of the Nagios process
Active checks are regularly scheduled

Passively

Passive checks are initiated and performed through external applications/processes
Passive checks results are submitted to Nagios for processing

98. What are active and passive checks in Nagios?

Active Checks:

The check logic in the Nagios daemon initiates active checks.
Nagios will execute a plugin and pass the information on what needs to be checked.
The plugin will then check the operational state of the host or service, and report results back to the Nagios daemon.
It will process the results of the host or service check and send notifications.

Passive Checks:

In passive checks, an external application checks the status of a host or service.
It writes the results of the check to the external command file.
Nagios reads the external command file and places the results of all passive checks into a queue for later processing.
Nagios may send out notifications, log alerts, etc. depending on the check result information.

99. Explain the main configuration file and its location in Nagios.

The main configuration file consists of several directives that affect how Nagios operates. The Nagios process and the CGIs read the config file.

A sample main configuration file will be placed into your settings directory:

/usr/local/Nagios/etc/resource.cfg

100. What is the Nagios Network Analyzer?

It provides an in-depth look at all network traffic sources and security threats.
It provides a central view of your network traffic and bandwidth data.
It allows system admins to gather high-level information on the health of the network.
It enables you to be proactive in resolving outages, abnormal behavior, and threats before they affect critical business processes.

101. What are the benefits of HTTP and SSL certificate monitoring with Nagios?

HTTP Certificate Monitoring

Increased server, services, and application availability.
Fast detection of network outages and protocol failures.
Enables web transaction and web server performance monitoring.

SSL Certificate Monitoring

Increased website availability.
Frequent application availability.
It provides increased security.

102. Explain virtualization with Nagios.

Nagios can run on different virtualization platforms, like VMware, Microsoft Visual PC, Xen, Amazon EC2, etc.

Provides the capability to monitor an assortment of metrics on different platforms
Ensures quick detection of service and application failures
Has the ability to monitor the following metrics:
CPU Usage
Memory
Networking
VM status
Reduced administrative overhead

103. Name the three variables that affect recursion and inheritance in Nagios.

Name: Template name that can be referenced in other object definitions so it can inherit the object's properties/variables.

Use: Here, you specify the name of the template object that you

want to inherit properties/variables from.

should be registered with Nagios.

define someobjecttype{

object-specific variables ….

name template_name

use name_of_template

}

104. Why is Nagios said to be object-oriented?

Using the object configuration format, you can create object definitions that inherit properties from other object definitions. Hence, Nagios is known as object-oriented.

Types of Objects:

Services
Hosts
Commands
Time Periods

105. Explain what state stalking is in Nagios.

State stalking is used for logging purposes in Nagios.
When stalking is enabled for a particular host or service, Nagios will watch that host or service very carefully.
It will log any changes it sees in the output of check results.
This helps in the analysis of log files.

Version Control System Interview Questions

Here are some common interview questions and answers related to version control systems:

106. What is a version control system (VCS)?

A VCS is a software tool that allows developers to manage changes to the source code of a software project. It enables developers to track and manage different versions of code files, collaborate with others, and revert to earlier versions if necessary.

107. What are the benefits of using a VCS?

There are several benefits to using a VCS, including:

The ability to track changes to code over time
The ability to collaborate with other developers and share code
The ability to revert to earlier versions of code if necessary
The ability to branch code and work on different features or fixes simultaneously
The ability to merge changes from other branches or contributors
Increased confidence and control over code changes and deployments

108. What are the types of VCS?

There are two main types of VCS: centralized and distributed.

Centralized VCS: A centralized VCS has a single central repository that stores all versions of the code files. Developers check out files from the central repository, make changes, and then commit the changes back to the warehouse.
Distributed VCS: A distributed VCS allows developers to create their local repositories of code changes. Developers can work on code changes locally, commit changes to their local storage, and then push changes to a central repository or pull changes from other contributors.

109. What is the difference between Git and SVN?

Git and SVN are both popular VCS tools, but they have some key differences:

Git is a distributed VCS, while SVN is a centralized VCS.
Git is more flexible and allows easier branching and merging of code changes.
SVN has better support for handling binary files.
Git is generally considered faster than SVN.

Virtualization Interview Questions

110. What is virtualization?

Virtualization is a technology that allows multiple operating systems or applications to run on a single physical server or computer. It creates virtual instances of hardware resources such as CPU, memory, and storage, which can be allocated to different virtual machines.

111. What are the benefits of virtualization?

There are several benefits of virtualization, including:

Reduced hardware costs
Increased efficiency and utilization of resources
Improved scalability and flexibility
Increased reliability and availability of applications
Simplified management and administration of IT infrastructure

112. What are the different types of virtualization?

There are several types of virtualization, including:

Server virtualization: Running multiple operating systems on a single physical server.
Network virtualization: Creating virtual networks that operate independently of physical network infrastructure.
Storage virtualization: Combining physical storage resources into a single virtual storage pool.
Desktop virtualization: Running multiple desktop environments on a single physical machine.

113. What is a hypervisor?

A hypervisor is a layer of software that enables virtualization by allowing multiple virtual machines to share a single physical server or computer. It manages the allocation of hardware resources to each virtual machine and isolates each virtual machine from the others.

Most Frequently Asked Questions

There are some frequently asked questions related to DevOps virtualization:

114. What is virtualization, and how does it connect to DevOps?

Virtualization is creating a virtual version of something, such as a server, storage device, or network. In DevOps, virtualization allows teams to create and manage virtual environments that can be used for development, testing, and deployment. This can help improve efficiency, reduce costs, and enable greater flexibility and scalability.

115. What are the benefits of using virtualization in DevOps?

Virtualization offers several benefits in a DevOps environment, including:

Improved efficiency: Virtualization allows for faster creation, deployment, and management of development and testing environments.
Greater scalability: Virtualization enables teams to quickly scale up or down their infrastructure as needed without requiring additional physical hardware.
Increased flexibility: Virtualization allows the creation of custom environments that can be easily modified and adapted to meet changing requirements.
Reduced costs: Virtualization can help reduce hardware costs and increase resource utilization, leading to lower overall infrastructure costs.

116. What are some standard virtualization technologies used in DevOps?

Several virtualization technologies are commonly used in DevOps, including:

Virtual machines (VMs): VMs are created using virtualization software such as VMware or VirtualBox, which enables the creation of multiple virtual instances of an operating system on a single physical machine.
Containers: Containers are lightweight, portable virtual environments created using containerization software such as Docker or Kubernetes. Containers enable the creation of custom application environments that can be easily shared and deployed across different systems.
Cloud computing: Cloud computing providers such as Amazon Web Services (AWS), Microsoft Azure, and the Google Cloud Platform (GCP) offer virtualized infrastructure and services that can be easily managed and scaled using DevOps tools and practices.

Next Steps

There you go; these are some of the most common DevOps interview questions you might encounter while attending an interview. As a DevOps Engineer, in-depth knowledge of processes, tools, and relevant technology is essential. These DevOps interview questions and answers will help you learn about some of these aspects. In addition, you must also have a holistic understanding of the products, services, and systems in place.

As you can see, there is a lot to learn to be able to land a rewarding job in the field of DevOps—Git, Jenkins, Selenium, Chef, Puppet, Ansible, Docker, Nagios, and more. While this comprehensive DevOps interview question guide is designed to help you ace your next interview, you would undoubtedly perform better if you enroll in our comprehensive DevOps Engineer Training Course today. You can even check out our Post Graduate Program in DevOps designed in collaboration with Caltech CTME that enables you to prepare for a DevOps career.

Mom, I'm scared ...

Hoang Guruu — Wed, 05 Mar 2025 04:54:23 +0000

Mom, I’m scared…

Mom, I don’t know how to say this, but lately, I’ve been really worried. AI is getting so advanced. It can do almost everything—write, draw, code, and even create things that I once thought only humans could do.

What I used to see only in sci-fi movies is now happening right in front of me. With just a few commands, AI can generate images that look completely real, write entire articles in seconds. It never gets tired, never complains, never makes mistakes, and works much faster than any human.

Mom, do you remember when I used to joke that “robots only exist in movies” and that humans are irreplaceable? Now, I’m not so sure anymore. People are using AI to replace human jobs, and many careers might disappear. Not just physical labor, but even creative work is being taken over.

I feel scared, confused, and uncertain about the future. What will happen next? Will I be left behind? Will there come a day when humans are no longer needed?

Mom, I’m scared…

But then I remember… Is this the same feeling you had when you first used a smartphone?

I remember how you used to say you didn’t know how to use it, that everything felt too complicated. You were afraid of pressing the wrong button, accidentally deleting a message, or getting lost in all the new apps while I kept explaining everything so quickly. You even said that back then, a simple phone with buttons was enough—why did things have to get so complicated now?

But then, little by little, you learned. First, it was just making calls and sending texts. Then, you figured out how to send pictures, make video calls, even shop online and book rides. You took your time, but you didn’t reject it. Now, even though you still say you’re not great with technology, you’ve adapted because you see how useful it is.

Mom, is my fear of AI the same as your fear of smartphones back then?

AI can do so many things now that I’m afraid it might replace humans. But maybe, instead of being scared, I should approach it the way you approached your phone—slowly learning, figuring out how to control and use it.

Maybe AI is just another tool, like all the tools that came before. People once feared that machines would take away jobs, but they learned how to work with them instead. Just like you no longer fear using a smartphone, I need to learn how to stop fearing AI.

But Mom, if one day I get too overwhelmed by all this, will you be patient and explain things to me—just like I once did for you? Because no matter how advanced technology becomes, love and human understanding are things that AI will never replace.

Your child.

MASTERING VIM – A DEVOPS ESSENTIAL!

Hoang Guruu — Tue, 04 Mar 2025 02:56:56 +0000

【 👨🏻‍💻 YouTube 】
Watch more other contents related DevOps - AWS

Vim Cheat Sheet & Quick Reference

Getting Started

Motion Diagrams

▼/▶ Cursor   ▽/▷ Target

Left-right motions

┌───────────── |      
├───────────── 0      $ ──────────────┐ 
│  ┌────────── ^      fe ────────┐    │
│  │  ┌─────── Fo     te ───────┐│    │
│  │  │┌────── To     30| ───┐  ││    │
│  │  ││ ┌──── ge     w ───┐ │  ││    │
│  │  ││ │ ┌── b      e ─┐ │ │  ││    │
│  │  ││ │ │  ┌h      l┐ │ │ │  ││    │
▽  ▽  ▽▽ ▽ ▽  ▽▼      ▼▽ ▽ ▽ ▽  ▽▽    ▽
   echo "A cheatsheet from quickref.me"

Up-down motions

                 - SCREEN 1 START
   ┌─┬─────────▷ #!/usr/bin/python
   │ │     ┌───▷     
   │ │     │     print("Hello")
   │ │     { } ▶ print("Vim")
   │ │       │   print("!")
   │ │       └─▷    
   │ │ ┌───┬───▷ print("Welcome")
G gg H M L k j ▶ print("to")
│        │   └─▷ print("quickref.me")
│        │       print("/vim")
│        │     
│        └─────▷ 
│                - SCREEN 1 END
└──────────────▷ print("SCREEN 2")

Motions

Shortcut	Description
h	j
/	Half-page up/down
/	Page up/down

Words

Shortcut	Description
b / w	Previous/Next word
ge / e	Previous/Next end of word

Line

Shortcut	Description
0 (zero) / $	Start/End of line
^	Start of line (non-blank)

Character

Shortcut	Description
Fe / fe	Move to previous/next e
To / to	Move before/after previous/next o
	/ n

Document

Shortcut	Description
gg / G	First/Last line
:n	nG
} / {	Next/Previous empty line

Window

Shortcut	Description
H / M / L	Top/Middle/Bottom screen
zt / zz / zb	Top/Center/Bottom this line

Insert Mode

Shortcut	Description
i / a	Insert before/after cursor
I / A	Insert start/end of line
o / O (letter)	Insert new line below/above
s / S	Delete char/line and insert
C / cc	Change to end of/current line
gi	Insert at last insert point
Esc

Saving and Exiting

Shortcut	Description
:w	Save
:q	Close file
:wq	:x
:wqa	Save and quit all files
:q!	ZQ
:qa	Close all files
:qa!	Force quit all files
:w now.txt	Write to now.txt
:sav new.txt	Save and edit new.txt
:w !sudo tee %	Write to readonly file

Normal Mode

Shortcut	Description
r	Replace one character
R	Enter Replace mode
u / 3u	Undo changes 1 / 3 times
U	Undo changes on one line
J	Join with next line
/ 5	Redo changes 1 / 5 times

Cut and paste

Shortcut	Description
x	Delete character (Cut)
p / P	Paste after/before
xp	Swap two characters
D	Delete to end of line (Cut)
dw	Delete word (Cut)
dd	Delete line (Cut)
ddp	Swap two lines
yy	Yank line (Copy)
"*p	"+p
"*y	"+y

In visual mode

Shortcut	Description
d	x
s	Replace selection
y	Yank selection (Copy)

Repeating

Shortcut	Description
.	Repeat last command
;	Repeat latest f, t, F or T
,	Repeat latest f, t, F or T reversed
&	Repeat last :s
@:	Repeat a command-line command

Visual mode

Shortcut	Description
v	Enter visual mode
V	Enter visual line mode
	Enter visual block mode
ggVG	Select all text
> / <	Shift text right/left

Macros

-	-
qi	Record macro i
q	Stop recording macro
@i	Run macro i
7@i	Run macro i 7 times
@@	Repeat last macro

You can save macro for any letters not just i

Vim Operators

Usage

Shortcut	Description
d	w
Operator	Motion

Combine operators with motions to use them

Available Operators

Shortcut	Description
d	Delete
y	Yank (copy)
c	Change (delete then insert)
p	Paste
=	Formats code
g~	Toggle case
gU	Uppercase
gu	Lowercase
>	Indent right
<	Indent left
!	Filter through external program

Examples

Combination	Description
dd	Delete current line
dj	Delete two lines
dw	Delete to next word
db	Delete to beginning of word
dfa	Delete until a char
d/hello	Delete until hello
cc	Change current line, synonym with S
yy	Copy current line
>j	Indent 2 lines
ggdG	Delete a complete document
gg=G	Indent a complete document
ggyG	Copy a whole document

Counts

[count] <operator> <motion>
<operator> [count] <motion>

Combination	Description
2dd	Delete 2 lines
6yy	Copy 6 lines
d3w	Delete 3 words
d5j	Delete 5 lines downwards
>4k	Indent 4 lines upwards

Vim Text objects

Usage

Shortcut	Description	-
v	i / a	p
Operator	inner / around	Text object

Operate with an operator inner or around text blocks

Text objects

Shortcut	Description
p	Paragraph
w	Word
W	WORD (surrounded by whitespace)
s	Sentence
[ ( { <	A [], (), or {} block
] ) } >	A [], (), or {} block
' " `	A quoted string
b	A block [(
B	A block in [{
t	A HTML tag block

See :help text-objects

Delete

Shortcut	Description
diw	Delete inner word
dis	Delete inner sentence
di"	Delete in quotes
da"	Delete in quotes (including quotes)
dip	Delete a paragraph

Selections

Shortcut	Description
vi"	Select inner quotes "..."
va"	Select quotes "..."
vi[	Select inner brackets [...]
va[	Select brackets [...]
viw	Select inner word
vip	Select inner paragraph
vipip	Select more paragraph

Misc

Shortcut	Description
ciw	Change inner word
ci"	Change inner quotes
cit	Change inner tags (HTML)
cip	Change inner paragraph
yip	Yank inner paragraph
yap	Yank paragraph (including newline)

Vim Multiple files

Buffers

-	-
:e file	Edit a file in a new buffer
:bn	Go to the next buffer
:bp	Go to the previous buffer
:bd	Remove file from buffer list
:b 5	Open buffer #5
:b file	Go to a buffer by file
:ls	List all open buffers
:sp file	Open and split window
:vs file	Open and vertically split window
:hid	Hide this buffer
:wn	Write file and move to next
:tab ba	Edit all buffers as tabs

Windows

-	-
s	Split window
v	Split window vertically
w	Switch windows
q	Quit a window
T	Break out into a new tab
x	Swap current with next
- / +	Decrease/Increase height
< / >	Decrease/Increase width

=	Equally high and wide
h / l	Go to the left/right window
j / k	Go to the up/down window

Tabs

Shortcut	Description
:tabe [file]	Edit file in a new tab
:tabf [file]	Open if exists in new tab
:tabc	Close current tab
:tabo	Close other tabs
:tabs	List all tabs
:tabr	Go to first tab
:tabl	Go to last tab
:tabm 0	Move to position 0
:tabn	Go to next tab
:tabp	Go to previous tab

Normal mode

Shortcut	Description
gt	Go to next tab
gT	Go to previous tab
2gt	Go to tab number 2

Vim Search and Replace

Search

-	-
/foo	Search forward
/foo\c	Search forward (case insensitive)
?foo	Search backward
/\v\d+	Search with regex
n	Next matching search pattern
N	Previous match
*	Search for current word forward
#	Search for current word backward

Replace LINE

`
:[range]s/{pattern}/{str}/[flags]


:s/old/new	Replace first
:s/old/new/g	Replace all
:s/\vold/new/g	Replace all with regex
:s/old/new/gc	replace all (Confirm)
:s/old/new/i	Ignore case replace first
:2,6s/old/new/g	Replace between lines 2-6

Replace FILE

`
:%s/{pattern}/{str}/[flags]


:%s/old/new	Replace first
:%s/old/new/g	Replace all
:%s/old/new/gc	Replace all (Confirm)
:%s/old/new/gi	Replace all (ignore case)
:%s/\vold/new/g	Replace all with regex

Ranges

-	-
%	Entire file
’<,’>	Current selection
5	Line 5
5,10	Lines 5 to 10
$	Last line
2,$	Lines 2 to Last
.	Current line
,3	Next 3 lines
-3,	Forward 3 lines

Global command

`
:[range]g/{pattern}/[command]


:g/foo/d	Delete lines containing foo
:g!/foo/d	Delete lines not containing foo
:g/^\s*$/d	Delete all blank lines
:g/foo/t$	Copy lines containing foo to EOF
:g/foo/m$	Move lines containing foo to EOF
:g/^/m0	Reverse a file
:g/^/t.	Duplicate every line

Inverse :g

`
:[range]v/{pattern}/[command]


:v/foo/d	Delete lines not containing foo(also :g!/foo/d)

Flags

-	-
g	Replace all occurrences
i	Ignore case
I	Don't ignore case
c	Confirm each substitution

Substitute expression (magic)

-	-
&	\0
\1...\9	Replace with the group 0-9
\u	Uppercase next letter
\U	Uppercase following characters
\l	Lowercase next letter
\L	Lowercase following characters
\e	End of \u, \U, \l and \L
\E	End of \u, \U, \l and \L

Examples

`
:s/a|b/xxx\0xxx/g # Modifies "a b" to "xxxaxxx xxxbxxx"
:s/test/\U& file/ # Modifies "test" to "TEST FILE"
:s/(test)/\U\1\e file/ # Modifies "test" to "TEST file"
:s/\v([abc])([efg])/\2\1/g # Modifies "af fa bg" to "fa fa gb"
:s/\v\w+/\u\0/g # Modifies "bla bla" to "Bla Bla"
:s/\v([ab])|([cd])/\1x/g # Modifies "a b c d" to "ax bx x x"
:%s/./\L&/ # Modifies "HTML" to "html"
:s/\v<(.)(\w)/\u\1\L\2/g # Make every first letter of a word uppercase
:%s/^(.)\n\1/\1/ # Remove duplicate lines
:%s/<\/=(\w+)>/\U&/g # Convert HTML-Tags to uppercase
:g/^pattern/s/$/mytext # Find and append text to the end
:g/pattern/norm! @i # Run a macro on matching lines
/^(.)(\r\?\n\1)+$ # View the duplicates lines
/\v^(.*)(\r?\n\1)+$ # View the duplicates lines (very magic)
:v/./,/./-j # Compress blank lines into a blank line
:g//,//d # Delete inclusively from to

Vimdiff

Usage

$ vimdiff file1 file2 [file3]
$ vim -d file1 file2 [file3]

Editing

`
:[range]diffget [bufspec]
:[range]diffput [bufspec]

Shortcut	Description
do / :diffget	Obtain (get) difference
dp / :diffput	Put difference
:dif	Re-scan differences
:diffo	Switch off diff mode
:1,$+1diffget	Get all differences
ZQ	Quit without changes

See: Ranges

Folds

Shortcut	Description
zo / zO	Open
zc / zC	Close
za / zA	Toggle
zv	Open folds for this line
zM	Close all
zR	Open all
zm	Fold more (foldlevel += 1)
zr	Fold less (foldlevel -= 1)
zx	Update folds

Jumping

Shortcut	Description
]c	Next difference
[c	Previous difference

Miscellaneous

Case

Shortcut	Description
vU	Uppercase character
vu	Lowercase character
~	Toggle case character
viw U	Uppercase word
viw u	Lowercase word
viw ~	Toggle case word
VU / gUU	Uppercase line
Vu / guu	Lowercase line
V~ / g~~	Toggle case line
gggUG	Uppercase all text
ggguG	Lowercase all text
ggg~G	Toggle case all text

Jumping

Shortcut	Description
	Go back to previous
	Go forward
gf	Go to file in cursor
ga	Display hex, ascii value

Misc command-lines

-	-
:h	Help open help view
:edit!	Reload current file
:2,8m0	Move lines 2-8 to 0
:noh	Clear search highlights
:sort	Sort lines
:ter	Open a terminal window
:set paste	Enable Insert Paste sub-mode
:set nopaste	disable Insert Paste sub-mode
:cq	Exiting with an error(aborting Git)

Navigating

Shortcut	Description
%	Nearest/matching {[()]}
[(	[{
])	]{
[m	Previous method start
[M	Previous method end

Counters

Shortcut	Description
	Increase number
	Decrease number

Shortcut	Description
:tag Classname	Jump to first definition of Classname
	Jump to definition
g]	See all definitions
	Go back to last tag
	Back/forward
:tselect Classname	Find definitions of Classname
:tjump Classname	Find definitions of Classname (auto-select 1st)

Formatting

-	-
:ce 8	Center lines between 8 columns
:ri 4	Right-align lines at 4 columns
:le	Left-align lines

See :help formatting

Marks

Shortcut	Description
`^	Last position of cursor in insert mode
`.	Last change in current buffer
`"	Last exited current buffer
`0	In last file edited
''	Back to line in current buffer where jumped from
	Back to position in current buffer where jumped from
`[	To beginning of previously changed or yanked text
`]	To end of previously changed or yanked text
`<	To beginning of last visual selection
`>	To end of last visual selection
ma	Mark this cursor position as a
`a	Jump to the cursor position a
'a	Jump to the beginning of the line with position a
d'a	Delete from current line to line of mark a
d`a	Delete from current position to position of mark a
c'a	Change text from current line to line of a
y`a	Yank text from current position to position of a
:marks	List all current marks
:delm a	Delete mark a
:delm a-d	Delete marks a, b, c, d
:delm abc	Delete marks a, b, c

Calculator

Shortcut	Description
= 7*7	Shows the result
= 10/2	Shows the result

Do this in INSERT mode

Shell

-	-
:!	Interpret Shell Command
:r!	Read in output of shell
:r!date	Insert date
:!!date	Replace current line with date

Command line

Shortcut	Description
	Insert current word into the command line
"	Paste from " register
	Auto-completion of path in insert mode

Tricks

Remove duplicate lines

:sort | %!uniq -u

To number the lines in the file

:%!cat -n

Copy whole doc to clipboard

:%w !pbcopy            # Mac OS X
:%w !xclip -i -sel c   # GNU/Linux
:%w !xsel -i -b        # GNU/Linux

Also see

Devhints (devhints.io)
Vim cheatsheet (vim.rotrr.com)
Vim documentation (vimdoc.sourceforge.net)
Interactive Vim tutorial (openvim.com)

Do DevOps Need to Know Databases? Absolutely! MongoDB, PostgreSQL, MySQL, and Redis

Hoang Guruu — Mon, 03 Mar 2025 08:52:35 +0000

【 👨🏻‍💻 YouTube | 🕵️ LinkedIn】
Watch more other contents related DevOps - AWS

📜 Table of Contents

MongoDB
Postgres
MySQL
Redis

📌 MongoDB

MongoDB Cheat Sheet & Quick Reference

MongoDB

The MongoDB cheat sheet provides you with the most commonly used MongoDB commands and queries for your reference. the cheatsheet is from mongodb developers website

Getting Started

Connect MongoDB Shell

mongo # connects to mongodb://127.0.0.1:27017 by default

mongo --host <host> --port <port> -u <user> -p <pwd> # omit the password if you want a prompt

mongo "mongodb://192.168.1.1:27017"

mongo "mongodb+srv://cluster-name.abcde.mongodb.net/<dbname>" --username <username> # MongoDB Atlas

Helpers

show dbs :

db // prints the current database

Switch database :

use <database_name>

Show collections :

show collections

Run JavaScript file :

load("myScript.js")

Crud

Create

db.coll.insertOne({name: "Max"})
db.coll.insert([{name: "Max"}, {name:"Alex"}]) // ordered bulk insert
db.coll.insert([{name: "Max"}, {name:"Alex"}], {ordered: false}) // unordered bulk insert
db.coll.insert({date: ISODate()})
db.coll.insert({name: "Max"}, {"writeConcern": {"w": "majority", "wtimeout": 5000}})

Delete

db.coll.remove({name: "Max"})
db.coll.remove({name: "Max"}, {justOne: true})
db.coll.remove({}) // WARNING! Deletes all the docs but not the collection itself and its index definitions
db.coll.remove({name: "Max"}, {"writeConcern": {"w": "majority", "wtimeout": 5000}})
db.coll.findOneAndDelete({"name": "Max"})

Update

db.coll.update({"_id": 1}, {"year": 2016}) // WARNING! Replaces the entire document
db.coll.update({"_id": 1}, {$set: {"year": 2016, name: "Max"}})
db.coll.update({"_id": 1}, {$unset: {"year": 1}})
db.coll.update({"_id": 1}, {$rename: {"year": "date"} })
db.coll.update({"_id": 1}, {$inc: {"year": 5}})
db.coll.update({"_id": 1}, {$mul: {price: NumberDecimal("1.25"), qty: 2}})
db.coll.update({"_id": 1}, {$min: {"imdb": 5}})
db.coll.update({"_id": 1}, {$max: {"imdb": 8}})
db.coll.update({"_id": 1}, {$currentDate: {"lastModified": true}})
db.coll.update({"_id": 1}, {$currentDate: {"lastModified": {$type: "timestamp"}}})

Array

db.coll.update({"_id": 1}, {$push :{"array": 1}})
db.coll.update({"_id": 1}, {$pull :{"array": 1}})
db.coll.update({"_id": 1}, {$addToSet :{"array": 2}})
db.coll.update({"_id": 1}, {$pop: {"array": 1}})  // last element
db.coll.update({"_id": 1}, {$pop: {"array": -1}}) // first element
db.coll.update({"_id": 1}, {$pullAll: {"array" :[3, 4, 5]}})
db.coll.update({"_id": 1}, {$push: {scores: {$each: [90, 92, 85]}}})
db.coll.updateOne({"_id": 1, "grades": 80}, {$set: {"grades.$": 82}})
db.coll.updateMany({}, {$inc: {"grades.$[]": 10}})
db.coll.update({}, {$set: {"grades.$[element]": 100}}, {multi: true, arrayFilters: [{"element": {$gte: 100}}]})

Update many

db.coll.update({"year": 1999}, {$set: {"decade": "90's"}}, {"multi":true})
db.coll.updateMany({"year": 1999}, {$set: {"decade": "90's"}})

FindOneAndUpdate

db.coll.findOneAndUpdate({"name": "Max"}, {$inc: {"points": 5}}, {returnNewDocument: true})

Upsert

db.coll.update({"_id": 1}, {$set: {item: "apple"}, $setOnInsert: {defaultQty: 100}}, {upsert: true})

Replace

db.coll.replaceOne({"name": "Max"}, {"firstname": "Maxime", "surname": "Beugnet"})

Save

db.coll.save({"item": "book", "qty": 40})

Write concern

db.coll.update({}, {$set: {"x": 1}}, {"writeConcern": {"w": "majority", "wtimeout": 5000}})

Find

db.coll.findOne() // returns a single document
db.coll.find()    // returns a cursor - show 20 results - "it" to display more
db.coll.find().pretty()
db.coll.find({name: "Max", age: 32}) // implicit logical "AND".
db.coll.find({date: ISODate("2020-09-25T13:57:17.180Z")})
db.coll.find({name: "Max", age: 32}).explain("executionStats") // or "queryPlanner" or "allPlansExecution"
db.coll.distinct("name")

Count

db.coll.count({age: 32})          // estimation based on collection metadata
db.coll.estimatedDocumentCount()  // estimation based on collection metadata
db.coll.countDocuments({age: 32}) // alias for an aggregation pipeline - accurate count

Comparison

db.coll.find({"year": {$gt: 1970}})
db.coll.find({"year": {$gte: 1970}})
db.coll.find({"year": {$lt: 1970}})
db.coll.find({"year": {$lte: 1970}})
db.coll.find({"year": {$ne: 1970}})
db.coll.find({"year": {$in: [1958, 1959]}})
db.coll.find({"year": {$nin: [1958, 1959]}})

Logical

db.coll.find({name:{$not: {$eq: "Max"}}})
db.coll.find({$or: [{"year" : 1958}, {"year" : 1959}]})
db.coll.find({$nor: [{price: 1.99}, {sale: true}]})
db.coll.find({
$and: [
    {$or: [{qty: {$lt :10}}, {qty :{$gt: 50}}]},
{$or: [{sale: true}, {price: {$lt: 5 }}]}
]
})

Element

db.coll.find({name: {$exists: true}})
db.coll.find({"zipCode": {$type: 2 }})
db.coll.find({"zipCode": {$type: "string"}})

Aggregation Pipeline

db.coll.aggregate([
{$match: {status: "A"}},
{$group: {_id: "$cust_id", total: {$sum: "$amount"}}},
{$sort: {total: -1}}
])

Text search with a "text" index

db.coll.find({$text: {$search: "cake"}}, {score: {$meta: "textScore"}}).sort({score: {$meta: "textScore"}})

Regex

db.coll.find({name: /^Max/}) // regex: starts by letter "M"
db.coll.find({name: /^Max$/i}) // regex case insensitive

Array

db.coll.find({tags: {$all: ["Realm", "Charts"]}})
db.coll.find({field: {$size: 2}}) // impossible to index - prefer storing the size of the array & update it
db.coll.find({results: {$elemMatch: {product: "xyz", score: {$gte: 8}}}})

Projections

db.coll.find({"x": 1}, {"actors": 1}) // actors + \_id
db.coll.find({"x": 1}, {"actors": 1, "\_id": 0}) // actors
db.coll.find({"x": 1}, {"actors": 0, "summary": 0}) // all but "actors" and "summary"

Sort, skip, limit

db.coll.find({}).sort({"year": 1, "rating": -1}).skip(10).limit(3)

Read Concern

db.coll.find().readConcern("majority")

Databases and Collections

Drop

db.coll.drop()    // removes the collection and its index definitions
db.dropDatabase() // double check that you are *NOT* on the PROD cluster... :-)

Create Collection

db.createCollection("contacts", {
   validator: {$jsonSchema: {
      bsonType: "object",
      required: ["phone"],
      properties: {
         phone: {
            bsonType: "string",
            description: "must be a string and is required"
         },
         email: {
            bsonType: "string",
            pattern: "@mongodb\.com$",
            description: "must be a string and match the regular expression pattern"
         },
         status: {
            enum: [ "Unknown", "Incomplete" ],
            description: "can only be one of the enum values"
         }
      }
   }}
})

Other Collection Functions

db.coll.stats()
db.coll.storageSize()
db.coll.totalIndexSize()
db.coll.totalSize()
db.coll.validate({full: true})
db.coll.renameCollection("new_coll", true) // 2nd parameter to drop the target collection if exists

Indexes

Basics

List

db.coll.getIndexes()
db.coll.getIndexKeys()

Drop Indexes

db.coll.dropIndex("name_1")

Hide/Unhide Indexes

db.coll.hideIndex("name_1")
db.coll.unhideIndex("name_1")

Create Indexes

// Index Types
db.coll.createIndex({"name": 1})                // single field index
db.coll.createIndex({"name": 1, "date": 1})     // compound index
db.coll.createIndex({foo: "text", bar: "text"}) // text index
db.coll.createIndex({"$**": "text"})            // wildcard text index
db.coll.createIndex({"userMetadata.$**": 1})    // wildcard index
db.coll.createIndex({"loc": "2d"})              // 2d index
db.coll.createIndex({"loc": "2dsphere"})        // 2dsphere index
db.coll.createIndex({"_id": "hashed"})          // hashed index

// Index Options
db.coll.createIndex({"lastModifiedDate": 1}, {expireAfterSeconds: 3600})      // TTL index
db.coll.createIndex({"name": 1}, {unique: true})
db.coll.createIndex({"name": 1}, {partialFilterExpression: {age: {$gt: 18}}}) // partial index
db.coll.createIndex({"name": 1}, {collation: {locale: 'en', strength: 1}})    // case insensitive index with strength = 1 or 2
db.coll.createIndex({"name": 1 }, {sparse: true})

Others

Handy commands

use admin
db.createUser({"user": "root", "pwd": passwordPrompt(), "roles": ["root"]})
db.dropUser("root")
db.auth( "user", passwordPrompt() )

use test
db.getSiblingDB("dbname")
db.currentOp()
db.killOp(123) // opid

db.fsyncLock()
db.fsyncUnlock()

db.getCollectionNames()
db.getCollectionInfos()
db.printCollectionStats()
db.stats()

db.getReplicationInfo()
db.printReplicationInfo()
db.isMaster()
db.hostInfo()
db.printShardingStatus()
db.shutdownServer()
db.serverStatus()

db.setSlaveOk()
db.getSlaveOk()

db.getProfilingLevel()
db.getProfilingStatus()
db.setProfilingLevel(1, 200) // 0 == OFF, 1 == ON with slowms, 2 == ON

db.enableFreeMonitoring()
db.disableFreeMonitoring()
db.getFreeMonitoringStatus()

db.createView("viewName", "sourceColl", [{$project:{department: 1}}])

Replica Set

rs.status()
rs.initiate({"_id": "replicaTest",
  members: [
    { _id: 0, host: "127.0.0.1:27017" },
    { _id: 1, host: "127.0.0.1:27018" },
    { _id: 2, host: "127.0.0.1:27019", arbiterOnly:true }]
})
rs.add("mongodbd1.example.net:27017")
rs.addArb("mongodbd2.example.net:27017")
rs.remove("mongodbd1.example.net:27017")
rs.conf()
rs.isMaster()
rs.printReplicationInfo()
rs.printSlaveReplicationInfo()
rs.reconfig(<valid_conf>)
rs.slaveOk()
rs.stepDown(20, 5) // (stepDownSecs, secondaryCatchUpPeriodSecs)

Sharded Cluster

sh.status()
sh.addShard("rs1/mongodbd1.example.net:27017")
sh.shardCollection("mydb.coll", {zipcode: 1})

sh.moveChunk("mydb.coll", { zipcode: "53187" }, "shard0019")
sh.splitAt("mydb.coll", {x: 70})
sh.splitFind("mydb.coll", {x: 70})
sh.disableAutoSplit()
sh.enableAutoSplit()

sh.startBalancer()
sh.stopBalancer()
sh.disableBalancing("mydb.coll")
sh.enableBalancing("mydb.coll")
sh.getBalancerState()
sh.setBalancerState(true/false)
sh.isBalancerRunning()

sh.addTagRange("mydb.coll", {state: "NY", zip: MinKey }, { state: "NY", zip: MaxKey }, "NY")
sh.removeTagRange("mydb.coll", {state: "NY", zip: MinKey }, { state: "NY", zip: MaxKey }, "NY")
sh.addShardTag("shard0000", "NYC")
sh.removeShardTag("shard0000", "NYC")

sh.addShardToZone("shard0000", "JFK")
sh.removeShardFromZone("shard0000", "NYC")
sh.removeRangeFromZone("mydb.coll", {a: 1, b: 1}, {a: 10, b: 10})

Change Streams

watchCursor = db.coll.watch( [ { $match : {"operationType" : "insert" } } ] )

while (!watchCursor.isExhausted()){
   if (watchCursor.hasNext()){
      print(tojson(watchCursor.next()));
   }
}

📌 Postgres

PostgreSQL Cheat Sheet & Quick Reference

Recon

Show version

SHOW SERVER_VERSION;

Show system status

\conninfo

Show environmental variables

SHOW ALL;

List users

SELECT rolname FROM pg_roles;

Show current user

SELECT current_user;

Show current user's permissions

\du

Show current database

SELECT current_database();

Show all tables in database

\dt

List functions

\df <schema>

Databases

List databases

\l

Connect to database

\c <database_name>

Show current database

SELECT current_database();

Create database

CREATE DATABASE <database_name> WITH OWNER <username>;

Drop database

DROP DATABASE IF EXISTS <database_name>;

Rename database

ALTER DATABASE <old_name> RENAME TO <new_name>;

Tables

List tables, in current db

\dt

SELECT table_schema,table_name FROM information_schema.tables ORDER BY table_schema,table_name;

List tables, globally

\dt *.*.

SELECT * FROM pg_catalog.pg_tables

List table schema

\d <table_name>
\d+ <table_name>

SELECT column_name, data_type, character_maximum_length
FROM INFORMATION_SCHEMA.COLUMNS
WHERE table_name = '<table_name>';

Create table

CREATE TABLE <table_name>(
  <column_name> <column_type>,
  <column_name> <column_type>
);

Create table, with an auto-incrementing primary key

CREATE TABLE <table_name> (
  <column_name> SERIAL PRIMARY KEY
);

Delete table

DROP TABLE IF EXISTS <table_name> CASCADE;

Permissions

Become the postgres user, if you have permission errors

sudo su - postgres
psql

Grant all permissions on database

GRANT ALL PRIVILEGES ON DATABASE <db_name> TO <user_name>;

Grant connection permissions on database

GRANT CONNECT ON DATABASE <db_name> TO <user_name>;

Grant permissions on schema

GRANT USAGE ON SCHEMA public TO <user_name>;

Grant permissions to functions

GRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA public TO <user_name>;

Grant permissions to select, update, insert, delete, on a all tables

GRANT SELECT, UPDATE, INSERT ON ALL TABLES IN SCHEMA public TO <user_name>;

Grant permissions, on a table

GRANT SELECT, UPDATE, INSERT ON <table_name> TO <user_name>;

Grant permissions, to select, on a table

GRANT SELECT ON ALL TABLES IN SCHEMA public TO <user_name>;

Columns

Add column

ALTER TABLE <table_name> IF EXISTS
ADD <column_name> <data_type> [<constraints>];

Update column

ALTER TABLE <table_name> IF EXISTS
ALTER <column_name> TYPE <data_type> [<constraints>];

Delete column

ALTER TABLE <table_name> IF EXISTS
DROP <column_name>;

Update column to be an auto-incrementing primary key

ALTER TABLE <table_name>
ADD COLUMN <column_name> SERIAL PRIMARY KEY;

Insert into a table, with an auto-incrementing primary key

INSERT INTO <table_name>
VALUES (DEFAULT, <value1>);


INSERT INTO <table_name> (<column1_name>,<column2_name>)
VALUES ( <value1>,<value2> );

Data

Select all data

SELECT * FROM <table_name>;

Read one row of data

SELECT * FROM <table_name> LIMIT 1;

Search for data

SELECT * FROM <table_name> WHERE <column_name> = <value>;

Insert data

INSERT INTO <table_name> VALUES( <value_1>, <value_2> );

Update data

UPDATE <table_name>
SET <column_1> = <value_1>, <column_2> = <value_2>
WHERE <column_1> = <value>;

Delete all data

DELETE FROM <table_name>;

Delete specific data

DELETE FROM <table_name>
WHERE <column_name> = <value>;

Users

List roles

SELECT rolname FROM pg_roles;

Create user

CREATE USER <user_name> WITH PASSWORD '<password>';

Drop user

DROP USER IF EXISTS <user_name>;

Alter user password

ALTER ROLE <user_name> WITH PASSWORD '<password>';

Schema

List schemas

\dn

SELECT schema_name FROM information_schema.schemata;

SELECT nspname FROM pg_catalog.pg_namespace;

Create schema

CREATE SCHEMA IF NOT EXISTS <schema_name>;

Drop schema

DROP SCHEMA IF EXISTS <schema_name> CASCADE;

Dates

Show current date YYYY-MM-DD

SELECT current_date;

Calculate age between two dates

SELECT age(timestamp, timestamp);

Show current time with time zone

SELECT current_time;

Make dates using integers

SELECT make_date(2021,03,25);

📌 MySQL

MySQL Cheat Sheet & Quick Reference

Managing tables

Create a new table with three columns

CREATE TABLE t (
     id    INT,
     name  VARCHAR DEFAULT NOT NULL,
     price INT DEFAULT 0
     PRIMARY KEY(id)
);

Delete the table from the database

DROP TABLE t ;

Add a new column to the table

ALTER TABLE t ADD column;

Drop column c from the table

ALTER TABLE t DROP COLUMN c ;

Add a constraint

ALTER TABLE t ADD constraint;

Drop a constraint

ALTER TABLE t DROP constraint;

Rename a table from t1 to t2

ALTER TABLE t1 RENAME TO t2;

Rename column c1 to c2

ALTER TABLE t1 RENAME c1 TO c2 ;

Remove all data in a table

TRUNCATE TABLE t;

Querying data from a table

Query data in columns c1, c2 from a table

SELECT c1, c2 FROM t

Query all rows and columns from a table

SELECT * FROM t

Query data and filter rows with a condition

SELECT c1, c2 FROM t
WHERE condition

Query distinct rows from a table

SELECT DISTINCT c1 FROM t
WHERE condition

Sort the result set in ascending or descending order

SELECT c1, c2 FROM t
ORDER BY c1 ASC [DESC]

Skip offset of rows and return the next n rows

SELECT c1, c2 FROM t
ORDER BY c1 
LIMIT n OFFSET offset

Group rows using an aggregate function

SELECT c1, aggregate(c2)
FROM t
GROUP BY c1

Filter groups using HAVING clause

SELECT c1, aggregate(c2)
FROM t
GROUP BY c1
HAVING condition

Querying from multiple tables

Inner join t1 and t2

SELECT c1, c2 
FROM t1
INNER JOIN t2 ON condition

Left join t1 and t1

SELECT c1, c2 
FROM t1
LEFT JOIN t2 ON condition

Right join t1 and t2

SELECT c1, c2 
FROM t1
RIGHT JOIN t2 ON condition

Perform full outer join

SELECT c1, c2 
FROM t1
FULL OUTER JOIN t2 ON condition

Produce a Cartesian product of rows in tables

SELECT c1, c2 
FROM t1
CROSS JOIN t2

Another way to perform cross join

SELECT c1, c2 
FROM t1, t2

Join t1 to itself using INNER JOIN clause

SELECT c1, c2
FROM t1 A
INNER JOIN t1 B ON condition

Using SQL Operators Combine rows from two queries

SELECT c1, c2 FROM t1
UNION [ALL]
SELECT c1, c2 FROM t2

Return the intersection of two queries

SELECT c1, c2 FROM t1
INTERSECT
SELECT c1, c2 FROM t2

Subtract a result set from another result set

SELECT c1, c2 FROM t1
MINUS
SELECT c1, c2 FROM t2

Query rows using pattern matching %, _

SELECT c1, c2 FROM t1
WHERE c1 [NOT] LIKE pattern

Query rows in a list

SELECT c1, c2 FROM t
WHERE c1 [NOT] IN value_list

Query rows between two values

SELECT c1, c2 FROM t
WHERE  c1 BETWEEN low AND high

Check if values in a table is NULL or not

SELECT c1, c2 FROM t
WHERE  c1 IS [NOT] NULL

Using SQL constraints

Set c1 and c2 as a primary key

CREATE TABLE t(
    c1 INT, c2 INT, c3 VARCHAR,
    PRIMARY KEY (c1,c2)
);

Set c2 column as a foreign key

CREATE TABLE t1(
    c1 INT PRIMARY KEY,  
    c2 INT,
    FOREIGN KEY (c2) REFERENCES t2(c2)
);

Make the values in c1 and c2 unique

CREATE TABLE t(
    c1 INT, c1 INT,
    UNIQUE(c2,c3)
);

Ensure c1 > 0 and values in c1 >= c2

CREATE TABLE t(
  c1 INT, c2 INT,
  CHECK(c1> 0 AND c1 >= c2)
);

Set values in c2 column not NULL

CREATE TABLE t(
     c1 INT PRIMARY KEY,
     c2 VARCHAR NOT NULL
);

Modifying Data

Insert one row into a table

INSERT INTO t(column_list)
VALUES(value_list);

Insert multiple rows into a table

INSERT INTO t(column_list)
VALUES (value_list), 
       (value_list), …;

Insert rows from t2 into t1

INSERT INTO t1(column_list)
SELECT column_list
FROM t2;

Update new value in the column c1 for all rows

UPDATE t
SET c1 = new_value;

Update values in the column c1, c2 that match the condition

UPDATE t
SET c1 = new_value, 
        c2 = new_value
WHERE condition;

Delete all data in a table

DELETE FROM t;

Delete subset of rows in a table

DELETE FROM t
WHERE condition;

Managing Views

Create a new view that consists of c1 and c2

CREATE VIEW v(c1,c2) 
AS
SELECT c1, c2
FROM t;

Create a new view with check option

CREATE VIEW v(c1,c2) 
AS
SELECT c1, c2
FROM t;
WITH [CASCADED | LOCAL] CHECK OPTION;

Create a recursive view

CREATE RECURSIVE VIEW v 
AS
select-statement -- anchor part
UNION [ALL]
select-statement; -- recursive part

Create a temporary view

CREATE TEMPORARY VIEW v 
AS
SELECT c1, c2
FROM t;

Delete a view

DROP VIEW view_name;

Managing triggers

Create or modify a trigger

CREATE OR MODIFY TRIGGER trigger_name
WHEN EVENT
ON table_name TRIGGER_TYPE
EXECUTE stored_procedure;

WHEN

-	-
BEFORE	invoke before the event occurs
AFTER	invoke after the event occurs

EVENT

-	-
INSERT	invoke for INSERT
UPDATE	invoke for UPDATE
DELETE	invoke for DELETE

TRIGGER_TYPE

-	-
FOR EACH ROW
FOR EACH STATEMENT

Managing indexes

Create an index on c1 and c2 of the t table

CREATE INDEX idx_name 
ON t(c1,c2);

Create a unique index on c3, c4 of the t table

CREATE UNIQUE INDEX idx_name
ON t(c3,c4)

Drop an index

DROP INDEX idx_name;

📌 Redis

Redis Cheat Sheet & Quick Reference

Redis

This is a redis quick reference cheat sheet that lists examples of redis commands

Getting Started

Getting started

Start Redis

$ redis-server

Connect to redis (Redis CLI client)

$ redis-cli

Connect to redis (telnet)

$ telnet 127.0.0.1 6379

Hello World

Ping

redis> PING
PONG

Hello World

redis> SET mykey "Hello world"
OK
redis> GET mykey
"Hello world"

Basic Data types

Strings
Lists
Hashes
Sets
Sorted Sets

Redis supports 5 basic data types

Redis String command

APPEND

APPEND key value

Example

redis> EXISTS mykey
(integer) 0
redis> APPEND mykey "Hello"
(integer) 5
redis> APPEND mykey " World"
(integer) 11
redis> GET mykey
"Hello World"

Append a value to a key

BITCOUNT

BITCOUNT key [start end]

Example

redis> SET mykey "foobar"
"OK"
redis> BITCOUNT mykey
(integer) 26
redis> BITCOUNT mykey 0 0
(integer) 4
redis> BITCOUNT mykey 1 1
(integer) 6

Count set bits in a string

BITFIELD

BITFIELD key [GET type offset] [SET type offset value] [INCRBY type offset increment] [OVERFLOW WRAP|SAT|FAIL]

Example

redis> BITFIELD mykey INCRBY i5 100 1 GET u4 0
1) (integer) 1
2) (integer) 0

Perform arbitrary bitfield integer operations on strings

BITOP

BITOP operation destkey key [key ...]

Example

redis> SET key1 "foobar"
"OK"
redis> SET key2 "abcdef"
"OK"
redis> BITOP AND dest key1 key2
(integer) 6
redis> GET dest
"`bc`ab"

Perform bitwise operations between strings

BITPOS

BITPOS key bit [start] [end]

Example

redis> SET mykey "\xff\xf0\x00"
"OK"
redis> BITPOS mykey 0
(integer) 12
redis> SET mykey "\x00\xff\xf0"
"OK"
redis> BITPOS mykey 1 0
(integer) 8
redis> BITPOS mykey 1 2
(integer) 16
redis> set mykey "\x00\x00\x00"
"OK"
redis> BITPOS mykey 1
(integer) -1

Find first bit set or clear in a string

DECR

DECR key

Example

redis> SET mykey "10"
"OK"
redis> DECR mykey
(integer) 9
redis> SET mykey "234293482390480948029348230948"
"OK"
redis> DECR mykey
ERR ERR value is not an integer or out of range

Decrement the integer value of a key by one

DECRBY

DECRBY key decrement

Example

redis> SET mykey "10"
"OK"
redis> DECRBY mykey 3
(integer) 7

Decrement the integer value of a key by the given number

GET

GET key

Example

redis> GET nonexisting
(nil)
redis> SET mykey "Hello"
"OK"
redis> GET mykey
"Hello"

Get the value of a key

GETBIT

GETBIT key offset

Example

redis> SETBIT mykey 7 1
(integer) 0
redis> GETBIT mykey 0
(integer) 0
redis> GETBIT mykey 7
(integer) 1
redis> GETBIT mykey 100
(integer) 0

Returns the bit value at offset in the string value stored at key

GETRANGE

GETRANGE key start end

Example

redis> SET mykey "This is a string"
"OK"
redis> GETRANGE mykey 0 3
"This"
redis> GETRANGE mykey -3 -1
"ing"
redis> GETRANGE mykey 0 -1
"This is a string"
redis> GETRANGE mykey 10 100
"string"

Get a substring of the string stored at a key

GETSET

GETSET key value

Example

redis> INCR mycounter
(integer) 1
redis> GETSET mycounter "0"
"1"
redis> GET mycounter
"0"

Set the string value of a key and return its old value

INCR

INCR key

Example

redis> SET mykey "10"
"OK"
redis> INCR mykey
(integer) 11
redis> GET mykey
"11"

Increment the integer value of a key by one

MSETNX

MSETNX key value [key value ...]

Example

redis> MSETNX key1 "Hello" key2 "there"
(integer) 1
redis> MSETNX key2 "new" key3 "world"
(integer) 0
redis> MGET key1 key2 key3
1) "Hello"
2) "there"
3) (nil)

Set multiple keys to multiple values, only if none of the keys exist

INCRBYFLOAT

INCRBYFLOAT key increment

Example

redis> SET mykey 10.50
"OK"
redis> INCRBYFLOAT mykey 0.1
"10.6"
redis> INCRBYFLOAT mykey -5
"5.6"
redis> SET mykey 5.0e3
"OK"
redis> INCRBYFLOAT mykey 2.0e2
"5200"

Increment the float value of a key by the given amount

MGET

MGET key [key ...]

Example

redis> SET key1 "Hello"
"OK"
redis> SET key2 "World"
"OK"
redis> MGET key1 key2 nonexisting
1) "Hello"
2) "World"
3) (nil)

Get the values of all the given keys

MSET

MSET key value [key value ...]

Example

redis> MSET key1 "Hello" key2 "World"
"OK"
redis> GET key1
"Hello"
redis> GET key2
"World"

Set multiple keys to multiple values

INCRBY

INCRBY key increment

Example

redis> SET mykey "10"
"OK"
redis> INCRBY mykey 5
(integer) 15

Increment the integer value of a key by the given amount

PSETEX

PSETEX key milliseconds value

Example

redis> PSETEX mykey 1000 "Hello"
"OK"
redis> PTTL mykey
(integer) 1000
redis> GET mykey
"Hello"

Set the value and expiration in milliseconds of a key

SET

SET key value [EX seconds|PX milliseconds|KEEPTTL] [NX|XX] [GET]

Example

redis> SET mykey "Hello"
"OK"
redis> GET mykey
"Hello"
redis> SET anotherkey "will expire in a minute" EX 60
"OK"

Set the string value of a key

SETBIT

SETBIT key offset value

Example

redis> SETBIT mykey 7 1
(integer) 0
redis> SETBIT mykey 7 0
(integer) 1
redis> GET mykey
"\u0000"

Sets or clears the bit at offset in the string value stored at key

SETEX

SETEX key seconds value

Example

redis> SETEX mykey 10 "Hello"
"OK"
redis> TTL mykey
(integer) 10
redis> GET mykey
"Hello"

Set the value and expiration of a key

SETNX

SETNX key value

Example

redis> SETNX mykey "Hello"
(integer) 1
redis> SETNX mykey "World"
(integer) 0
redis> GET mykey
"Hello"

Set the value of a key, only if the key does not exist

SETRANGE

SETRANGE key offset value

Example

redis> SET key1 "Hello World"
"OK"
redis> SETRANGE key1 6 "Redis"
(integer) 11
redis> GET key1
"Hello Redis"

Overwrite part of a string at key starting at the specified offset

STRLEN

STRLEN key

Example

redis> SET mykey "Hello world"
"OK"
redis> STRLEN mykey
(integer) 11
redis> STRLEN nonexisting
(integer) 0

Get the length of the value stored in a key

STRALGO

STRALGO LCS algo-specific-argument [algo-specific-argument ...]

Example

redis> STRALGO LCS KEYS key1 key2 IDX
1) "matches"
2) 1) 1) 1) (integer) 4
         2) (integer) 7
      2) 1) (integer) 5
         2) (integer) 8
   2) 1) 1) (integer) 2
         2) (integer) 3
      2) 1) (integer) 0
         2) (integer) 1
3) "len"
4) (integer) 6

Run algorithms (currently LCS) against strings

Redis Set command

SADD

SADD key member [member ...]

Example

redis> SADD myset "Hello"
(integer) 1
redis> SADD myset "World"
(integer) 1
redis> SADD myset "World"
(integer) 0
redis> SMEMBERS myset
1) "Hello"
2) "World"

Add one or more members to a set

SCARD

SCARD key

Example

redis> SADD myset "Hello"
(integer) 1
redis> SADD myset "World"
(integer) 1
redis> SCARD myset
(integer) 2

Get the number of members in a set

SDIFF

SDIFF key [key ...]

Example

redis> SADD key1 "a"
(integer) 1
redis> SADD key1 "b"
(integer) 1
redis> SADD key1 "c"
(integer) 1
redis> SADD key2 "c"
(integer) 1
redis> SADD key2 "d"
(integer) 1
redis> SADD key2 "e"
(integer) 1
redis> SDIFF key1 key2
1) "a"
2) "b"

Subtract multiple sets

SDIFFSTORE

SDIFFSTORE destination key [key ...]

Example

redis> SADD key1 "a"
(integer) 1
redis> SADD key1 "b"
(integer) 1
redis> SADD key1 "c"
(integer) 1
redis> SADD key2 "c"
(integer) 1
redis> SADD key2 "d"
(integer) 1
redis> SADD key2 "e"
(integer) 1
redis> SDIFFSTORE key key1 key2
(integer) 2
redis> SMEMBERS key
1) "a"
2) "b"

Subtract multiple sets and store the resulting set in a key

SINTER

SINTER key [key ...]

Example

redis> SADD key1 "a"
(integer) 1
redis> SADD key1 "b"
(integer) 1
redis> SADD key1 "c"
(integer) 1
redis> SADD key2 "c"
(integer) 1
redis> SADD key2 "d"
(integer) 1
redis> SADD key2 "e"
(integer) 1
redis> SINTER key1 key2
1) "c"

Intersect multiple sets

SINTERSTORE

SINTERSTORE destination key [key ...]

Example

redis> SADD key1 "a"
(integer) 1
redis> SADD key1 "b"
(integer) 1
redis> SADD key1 "c"
(integer) 1
redis> SADD key2 "c"
(integer) 1
redis> SADD key2 "d"
(integer) 1
redis> SADD key2 "e"
(integer) 1
redis> SINTERSTORE key key1 key2
(integer) 1
redis> SMEMBERS key
1) "c"

Intersect multiple sets and store the resulting set in a key

SISMEMBER

SISMEMBER key member

Example

redis> SADD myset "one"
(integer) 1
redis> SISMEMBER myset "one"
(integer) 1
redis> SISMEMBER myset "two"
(integer) 0

Determine if a given value is a member of a set

SMISMEMBER

SMISMEMBER key member [member ...]

Example

redis> SADD myset "one"
(integer) 1
redis> SADD myset "one"
(integer) 0
redis> SMISMEMBER myset "one" "notamember"
1) (integer) 1
2) (integer) 0

Returns the membership associated with the given elements for a set

SMEMBERS

SMEMBERS key

Example

redis> SADD myset "Hello"
(integer) 1
redis> SADD myset "World"
(integer) 1
redis> SMEMBERS myset
1) "Hello"
2) "World"

Get all the members in a set

SMOVE

SMOVE source destination member

Example

redis> SADD myset "one"
(integer) 1
redis> SADD myset "two"
(integer) 1
redis> SADD myotherset "three"
(integer) 1
redis> SMOVE myset myotherset "two"
(integer) 1
redis> SMEMBERS myset
1) "one"
redis> SMEMBERS myotherset
1) "two"
2) "three"

Move a member from one set to another

SPOP

SPOP key [count]

Example

redis> SADD myset "one"
(integer) 1
redis> SADD myset "two"
(integer) 1
redis> SADD myset "three"
(integer) 1
redis> SPOP myset
"two"
redis> SMEMBERS myset
1) "one"
2) "three"
redis> SADD myset "four"
(integer) 1
redis> SADD myset "five"
(integer) 1
redis> SPOP myset 3
1) "four"
2) "five"
3) "three"
redis> SMEMBERS myset
1) "one"

Remove and return one or multiple random members from a set

SRANDMEMBER

SRANDMEMBER key [count]

Example

redis> SADD myset one two three
(integer) 3
redis> SRANDMEMBER myset
"three"
redis> SRANDMEMBER myset 2
1) "two"
2) "three"
redis> SRANDMEMBER myset -5
1) "one"
2) "two"
3) "three"
4) "three"
5) "one"

Get one or multiple random members from a set

SREM

SREM key member [member ...]

Example

redis> SADD myset "one"
(integer) 1
redis> SADD myset "two"
(integer) 1
redis> SADD myset "three"
(integer) 1
redis> SREM myset "one"
(integer) 1
redis> SREM myset "four"
(integer) 0
redis> SMEMBERS myset
1) "two"
2) "three"

Remove one or more members from a set

SUNION

SUNION key [key ...]

Example

redis> SADD key1 "a"
(integer) 1
redis> SADD key1 "b"
(integer) 1
redis> SADD key1 "c"
(integer) 1
redis> SADD key2 "c"
(integer) 1
redis> SADD key2 "d"
(integer) 1
redis> SADD key2 "e"
(integer) 1
redis> SUNION key1 key2
1) "a"
2) "c"
3) "e"
4) "b"
5) "d"

Add multiple sets

SUNIONSTORE

SUNIONSTORE destination key [key ...]

Example

redis> SADD key1 "a"
(integer) 1
redis> SADD key1 "b"
(integer) 1
redis> SADD key1 "c"
(integer) 1
redis> SADD key2 "c"
(integer) 1
redis> SADD key2 "d"
(integer) 1
redis> SADD key2 "e"
(integer) 1
redis> SUNIONSTORE key key1 key2
(integer) 5
redis> SMEMBERS key
1) "a"
2) "c"
3) "e"
4) "b"
5) "d"

Add multiple sets and store the resulting set in a key

Redis List command

Misc

-: BRPOPLPUSH
- -: Pop an element from a list, push it to another list and return it; or block until one is available
-: BLMOVE
- -: Pop an element from a list, push it to another list and return it; or block until one is available

BLPOP

BLPOP key [key ...] timeout

Example

redis> DEL list1 list2
(integer) 0
redis> RPUSH list1 a b c
(integer) 3
redis> BLPOP list1 list2 0
1) "list1"
2) "a"

Remove and get the first element in a list, or block until one is available |

BRPOP

BRPOP key [key ...] timeout

Example

redis> DEL list1 list2
(integer) 0
redis> RPUSH list1 a b c
(integer) 3
redis> BRPOP list1 list2 0
1) "list1"
2) "c"

Remove and get the last element in a list, or block until one is available |

LINDEX

LINDEX key index

Example

redis> LPUSH mylist "World"
(integer) 1
redis> LPUSH mylist "Hello"
(integer) 2
redis> LINDEX mylist 0
"Hello"
redis> LINDEX mylist -1
"World"
redis> LINDEX mylist 3
(nil)

Get an element from a list by its index

LINSERT

LINSERT key BEFORE|AFTER pivot element

Example

redis> RPUSH mylist "Hello"
(integer) 1
redis> RPUSH mylist "World"
(integer) 2
redis> LINSERT mylist BEFORE "World" "There"
(integer) 3
redis> LRANGE mylist 0 -1
1) "Hello"
2) "There"
3) "World"

Insert an element before or after another element in a list

LLEN

LLEN key

Example

redis> LPUSH mylist "World"
(integer) 1
redis> LPUSH mylist "Hello"
(integer) 2
redis> LLEN mylist
(integer) 2

Get the length of a list

LPOP

LPOP key [count]

Example

redis> RPUSH mylist "one"
(integer) 1
redis> RPUSH mylist "two"
(integer) 2
redis> RPUSH mylist "three"
(integer) 3
redis> LPOP mylist
"one"
redis> LRANGE mylist 0 -1
1) "two"
2) "three"

Remove and get the first elements in a list

LPOS

LPOS key element [RANK rank] [COUNT num-matches] [MAXLEN len]

Example

redis> RPUSH mylist a b c d 1 2 3 4 3 3 3
(integer) 11
redis> LPOS mylist 3
(integer) 6
redis> LPOS mylist 3 COUNT 0 RANK 2
1) (integer) 8
2) (integer) 9
3) (integer) 10

Return the index of matching elements on a list

LPUSH

LPUSH key element [element ...]

Example

redis> LPUSH mylist "world"
(integer) 1
redis> LPUSH mylist "hello"
(integer) 2
redis> LRANGE mylist 0 -1
1) "hello"
2) "world"

Prepend one or multiple elements to a list

LPUSHX

LPUSHX key element [element ...]

Example

redis> LPUSH mylist "World"
(integer) 1
redis> LPUSHX mylist "Hello"
(integer) 2
redis> LPUSHX myotherlist "Hello"
(integer) 0
redis> LRANGE mylist 0 -1
1) "Hello"
2) "World"
redis> LRANGE myotherlist 0 -1
(empty list or set)

Prepend an element to a list, only if the list exists

LRANGE

LRANGE key start stop

Example

redis> RPUSH mylist "one"
(integer) 1
redis> RPUSH mylist "two"
(integer) 2
redis> RPUSH mylist "three"
(integer) 3
redis> LRANGE mylist 0 0
1) "one"
redis> LRANGE mylist -3 2
1) "one"
2) "two"
3) "three"
redis> LRANGE mylist -100 100
1) "one"
2) "two"
3) "three"
redis> LRANGE mylist 5 10
(empty list or set)

Get a range of elements from a list

LREM

LREM key count element

Example

redis> RPUSH mylist "hello"
(integer) 1
redis> RPUSH mylist "hello"
(integer) 2
redis> RPUSH mylist "foo"
(integer) 3
redis> RPUSH mylist "hello"
(integer) 4
redis> LREM mylist -2 "hello"
(integer) 2
redis> LRANGE mylist 0 -1
1) "hello"
2) "foo"

Remove elements from a list

LSET

LSET key index element

Example

redis> RPUSH mylist "one"
(integer) 1
redis> RPUSH mylist "two"
(integer) 2
redis> RPUSH mylist "three"
(integer) 3
redis> LSET mylist 0 "four"
"OK"
redis> LSET mylist -2 "five"
"OK"
redis> LRANGE mylist 0 -1
1) "four"
2) "five"
3) "three"

Set the value of an element in a list by its index

LTRIM

LTRIM key start stop

Example

redis> RPUSH mylist "one"
(integer) 1
redis> RPUSH mylist "two"
(integer) 2
redis> RPUSH mylist "three"
(integer) 3
redis> LTRIM mylist 1 -1
"OK"
redis> LRANGE mylist 0 -1
1) "two"
2) "three"

Trim a list to the specified range

RPOP

RPOP key [count]

Example

redis> RPUSH mylist "one"
(integer) 1
redis> RPUSH mylist "two"
(integer) 2
redis> RPUSH mylist "three"
(integer) 3
redis> RPOP mylist
"three"
redis> LRANGE mylist 0 -1
1) "one"
2) "two"

Remove and get the last elements in a list

RPOPLPUSH

RPOPLPUSH source destination

Example

redis> RPUSH mylist "one"
(integer) 1
redis> RPUSH mylist "two"
(integer) 2
redis> RPUSH mylist "three"
(integer) 3
redis> RPOPLPUSH mylist myotherlist
"three"
redis> LRANGE mylist 0 -1
1) "one"
2) "two"
redis> LRANGE myotherlist 0 -1
1) "three"

Remove the last element in a list, prepend it to another list and return it

LMOVE

LMOVE source destination LEFT|RIGHT LEFT|RIGHT

Example

redis> RPUSH mylist "one"
(integer) 1
redis> RPUSH mylist "two"
(integer) 2
redis> RPUSH mylist "three"
(integer) 3
redis> LMOVE mylist myotherlist RIGHT LEFT
"three"
redis> LMOVE mylist myotherlist LEFT RIGHT
"one"
redis> LRANGE mylist 0 -1
1) "two"
redis> LRANGE myotherlist 0 -1
1) "three"
2) "one"

Pop an element from a list, push it to another list and return it

RPUSH

RPUSH key element [element ...]

Example

redis> RPUSH mylist "hello"
(integer) 1
redis> RPUSH mylist "world"
(integer) 2
redis> LRANGE mylist 0 -1
1) "hello"
2) "world"

Append one or multiple elements to a list

RPUSHX

RPUSHX key element [element ...]

Example

redis> RPUSH mylist "Hello"
(integer) 1
redis> RPUSHX mylist "World"
(integer) 2
redis> RPUSHX myotherlist "World"
(integer) 0
redis> LRANGE mylist 0 -1
1) "Hello"
2) "World"
redis> LRANGE myotherlist 0 -1
(empty list or set)

Append an element to a list, only if the list exists

Redis Hash command

HDEL

HDEL key field [field ...]

Example

redis> HSET myhash field1 "foo"
(integer) 1
redis> HDEL myhash field1
(integer) 1
redis> HDEL myhash field2
(integer) 0

Delete one or more hash fields

HEXISTS

HEXISTS key field

Example

redis> HSET myhash field1 "foo"
(integer) 1
redis> HEXISTS myhash field1
(integer) 1
redis> HEXISTS myhash field2
(integer) 0

Determine if a hash field exists

HGET

HGET key field

Example

redis> HSET myhash field1 "foo"
(integer) 1
redis> HGET myhash field1
"foo"
redis> HGET myhash field2
(nil)

Get the value of a hash field

HGETALL

HGETALL key

Example

redis> HSET myhash field1 "Hello"
(integer) 1
redis> HSET myhash field2 "World"
(integer) 1
redis> HGETALL myhash
1) "field1"
2) "Hello"
3) "field2"
4) "World"

Get all the fields and values in a hash

HINCRBY

HINCRBY key field increment

Example

redis> HSET myhash field 5
(integer) 1
redis> HINCRBY myhash field 1
(integer) 6
redis> HINCRBY myhash field -1
(integer) 5
redis> HINCRBY myhash field -10
(integer) -5

Increment the integer value of a hash field by the given number

HINCRBYFLOAT

HINCRBYFLOAT key field increment

Example

redis> HSET mykey field 10.50
(integer) 1
redis> HINCRBYFLOAT mykey field 0.1
"10.6"
redis> HINCRBYFLOAT mykey field -5
"5.6"
redis> HSET mykey field 5.0e3
(integer) 0
redis> HINCRBYFLOAT mykey field 2.0e2
"5200"

Increment the float value of a hash field by the given amount

HKEYS

HKEYS key

Example

redis> HSET myhash field1 "Hello"
(integer) 1
redis> HSET myhash field2 "World"
(integer) 1
redis> HKEYS myhash
1) "field1"
2) "field2"

Get all the fields in a hash

HLEN

HLEN key

Example

redis> HSET myhash field1 "Hello"
(integer) 1
redis> HSET myhash field2 "World"
(integer) 1
redis> HLEN myhash
(integer) 2

Get the number of fields in a hash

HMGET

HMGET key field [field ...]

Example

redis> HSET myhash field1 "Hello"
(integer) 1
redis> HSET myhash field2 "World"
(integer) 1
redis> HMGET myhash field1 field2 nofield
1) "Hello"
2) "World"
3) (nil)

Get the values of all the given hash fields

HMSET

HMSET key field value [field value ...]

Example

redis> HMSET myhash field1 "Hello" field2 "World"
"OK"
redis> HGET myhash field1
"Hello"
redis> HGET myhash field2
"World"

Set multiple hash fields to multiple values

HSET

HSET key field value [field value ...]

Example

redis> HSET myhash field1 "Hello"
(integer) 1
redis> HGET myhash field1
"Hello"

Set the string value of a hash field

HSETNX

HSETNX key field value

Example

redis> HSETNX myhash field "Hello"
(integer) 1
redis> HSETNX myhash field "World"
(integer) 0
redis> HGET myhash field
"Hello"

Set the value of a hash field, only if the field does not exist

HSTRLEN

HSTRLEN key field

Example

redis> HMSET myhash f1 HelloWorld f2 99 f3 -256
"OK"
redis> HSTRLEN myhash f1
(integer) 10
redis> HSTRLEN myhash f2
(integer) 2
redis> HSTRLEN myhash f3
(integer) 4

Get the length of the value of a hash field

HVALS

HVALS key

Example

redis> HSET myhash field1 "Hello"
(integer) 1
redis> HSET myhash field2 "World"
(integer) 1
redis> HVALS myhash
1) "Hello"
2) "World"

Get all the values in a hash

Redis Sorted set command

BZPOPMIN

BZPOPMIN key [key ...] timeout

Example

redis> DEL zset1 zset2
(integer) 0
redis> ZADD zset1 0 a 1 b 2 c
(integer) 3
redis> BZPOPMIN zset1 zset2 0
1) "zset1"
2) "a"
3) "0"

Remove and return the member with the lowest score from one or more sorted sets, or block until one is available

BZPOPMAX

BZPOPMAX key [key ...] timeout

Example

redis> DEL zset1 zset2
(integer) 0
redis> ZADD zset1 0 a 1 b 2 c
(integer) 3
redis> BZPOPMAX zset1 zset2 0
1) "zset1"
2) "c"
3) "2"

Remove and return the member with the highest score from one or more sorted sets, or block until one is available

ZADD

ZADD key [NX|XX] [GT|LT] [CH] [INCR] score member [score member ...]

Example

redis> ZADD myzset 1 "one"
(integer) 1
redis> ZADD myzset 1 "uno"
(integer) 1
redis> ZADD myzset 2 "two" 3 "three"
(integer) 2
redis> ZRANGE myzset 0 -1 WITHSCORES
1) "one"
2) "1"
3) "uno"
4) "1"
5) "two"
6) "2"
7) "three"
8) "3"

Add one or more members to a sorted set, or update its score if it already exists

ZCARD

ZCARD key

Example

redis> ZADD myzset 1 "one"
(integer) 1
redis> ZADD myzset 2 "two"
(integer) 1
redis> ZCARD myzset
(integer) 2

Get the number of members in a sorted set

ZSCORE

ZSCORE key member

Example

redis> ZADD myzset 1 "one"
(integer) 1
redis> ZSCORE myzset "one"
"1"

Get the score associated with the given member in a sorted set

ZCOUNT

ZCOUNT key min max

Example

redis> ZADD myzset 1 "one"
(integer) 1
redis> ZADD myzset 2 "two"
(integer) 1
redis> ZADD myzset 3 "three"
(integer) 1
redis> ZCOUNT myzset -inf +inf
(integer) 3
redis> ZCOUNT myzset (1 3
(integer) 2

Count the members in a sorted set with scores within the given values

ZDIFF

ZDIFF numkeys key [key ...] [WITHSCORES]

Example

redis> ZADD zset1 1 "one"
(integer) 1
redis> ZADD zset1 2 "two"
(integer) 1
redis> ZADD zset1 3 "three"
(integer) 1
redis> ZADD zset2 1 "one"
(integer) 1
redis> ZADD zset2 2 "two"
(integer) 1
redis> ZDIFF 2 zset1 zset2
1) "three"
redis> ZDIFF 2 zset1 zset2 WITHSCORES
1) "three"
2) "3"

Subtract multiple sorted sets

ZDIFFSTORE

ZDIFFSTORE destination numkeys key [key ...]

Example

redis> ZADD zset1 1 "one"
(integer) 1
redis> ZADD zset1 2 "two"
(integer) 1
redis> ZADD zset1 3 "three"
(integer) 1
redis> ZADD zset2 1 "one"
(integer) 1
redis> ZADD zset2 2 "two"
(integer) 1
redis> ZDIFFSTORE out 2 zset1 zset2
(integer) 1
redis> ZRANGE out 0 -1 WITHSCORES
1) "three"
2) "3"

Subtract multiple sorted sets and store the resulting sorted set in a new key

ZINCRBY

ZINCRBY key increment member

Example

redis> ZADD myzset 1 "one"
(integer) 1
redis> ZADD myzset 2 "two"
(integer) 1
redis> ZINCRBY myzset 2 "one"
"3"
redis> ZRANGE myzset 0 -1 WITHSCORES
1) "two"
2) "2"
3) "one"
4) "3"

Increment the score of a member in a sorted set

ZINTER

ZINTER numkeys key [key ...] [WEIGHTS weight [weight ...]] [AGGREGATE SUM|MIN|MAX] [WITHSCORES]

Example

redis> ZADD zset1 1 "one"
(integer) 1
redis> ZADD zset1 2 "two"
(integer) 1
redis> ZADD zset2 1 "one"
(integer) 1
redis> ZADD zset2 2 "two"
(integer) 1
redis> ZADD zset2 3 "three"
(integer) 1
redis> ZINTER 2 zset1 zset2
1) "one"
2) "two"
redis> ZINTER 2 zset1 zset2 WITHSCORES
1) "one"
2) "2"
3) "two"
4) "4"

Intersect multiple sorted sets

ZINTERSTORE

ZINTERSTORE destination numkeys key [key ...] [WEIGHTS weight [weight ...]] [AGGREGATE SUM|MIN|MAX]

Example

redis> ZADD zset1 1 "one"
(integer) 1
redis> ZADD zset1 2 "two"
(integer) 1
redis> ZADD zset2 1 "one"
(integer) 1
redis> ZADD zset2 2 "two"
(integer) 1
redis> ZADD zset2 3 "three"
(integer) 1
redis> ZINTERSTORE out 2 zset1 zset2 WEIGHTS 2 3
(integer) 2
redis> ZRANGE out 0 -1 WITHSCORES
1) "one"
2) "5"
3) "two"
4) "10"

Intersect multiple sorted sets and store the resulting sorted set in a new key

ZLEXCOUNT

ZLEXCOUNT key min max

Example

redis> ZADD myzset 0 a 0 b 0 c 0 d 0 e
(integer) 5
redis> ZADD myzset 0 f 0 g
(integer) 2
redis> ZLEXCOUNT myzset - +
(integer) 7
redis> ZLEXCOUNT myzset [b [f
(integer) 5

Count the number of members in a sorted set between a given lexicographical range

ZPOPMAX

ZPOPMAX key [count]

Example

redis> ZADD myzset 1 "one"
(integer) 1
redis> ZADD myzset 2 "two"
(integer) 1
redis> ZADD myzset 3 "three"
(integer) 1
redis> ZPOPMAX myzset
1) "three"
2) "3"

Remove and return members with the highest scores in a sorted set

ZPOPMIN

ZPOPMIN key [count]

Example

redis> ZADD myzset 1 "one"
(integer) 1
redis> ZADD myzset 2 "two"
(integer) 1
redis> ZADD myzset 3 "three"
(integer) 1
redis> ZPOPMIN myzset
1) "one"
2) "1"

Remove and return members with the lowest scores in a sorted set

ZRANGE

ZRANGE key start stop [WITHSCORES]

Example

redis> ZADD myzset 1 "one"
(integer) 1
redis> ZADD myzset 2 "two"
(integer) 1
redis> ZADD myzset 3 "three"
(integer) 1
redis> ZRANGE myzset 0 -1
1) "one"
2) "two"
3) "three"
redis> ZRANGE myzset 2 3
1) "three"
redis> ZRANGE myzset -2 -1
1) "two"
2) "three"

Return a range of members in a sorted set, by index

ZRANGEBYLEX

ZRANGEBYLEX key min max [LIMIT offset count]

Example

redis> ZADD myzset 0 a 0 b 0 c 0 d 0 e 0 f 0 g
(integer) 7
redis> ZRANGEBYLEX myzset - [c
1) "a"
2) "b"
3) "c"
redis> ZRANGEBYLEX myzset - (c
1) "a"
2) "b"
redis> ZRANGEBYLEX myzset [aaa (g
1) "b"
2) "c"
3) "d"
4) "e"
5) "f"

Return a range of members in a sorted set, by lexicographical range

ZREVRANGEBYLEX

ZREVRANGEBYLEX key max min [LIMIT offset count]

Example

redis> ZADD myzset 0 a 0 b 0 c 0 d 0 e 0 f 0 g
(integer) 7
redis> ZREVRANGEBYLEX myzset [c -
1) "c"
2) "b"
3) "a"
redis> ZREVRANGEBYLEX myzset (c -
1) "b"
2) "a"
redis> ZREVRANGEBYLEX myzset (g [aaa
1) "f"
2) "e"
3) "d"
4) "c"
5) "b"

Return a range of members in a sorted set, by lexicographical range, ordered from higher to lower strings.

ZRANGEBYSCORE

ZRANGEBYSCORE key min max [WITHSCORES] [LIMIT offset count]

Example

redis> ZADD myzset 1 "one"
(integer) 1
redis> ZADD myzset 2 "two"
(integer) 1
redis> ZADD myzset 3 "three"
(integer) 1
redis> ZRANGEBYSCORE myzset -inf +inf
1) "one"
2) "two"
3) "three"
redis> ZRANGEBYSCORE myzset 1 2
1) "one"
2) "two"
redis> ZRANGEBYSCORE myzset (1 2
1) "two"
redis> ZRANGEBYSCORE myzset (1 (2
(empty list or set)

Return a range of members in a sorted set, by score

ZRANK

ZRANK key member

Example

redis> ZADD myzset 1 "one"
(integer) 1
redis> ZADD myzset 2 "two"
(integer) 1
redis> ZADD myzset 3 "three"
(integer) 1
redis> ZRANK myzset "three"
(integer) 2
redis> ZRANK myzset "four"
(nil)

Determine the index of a member in a sorted set

ZREM

ZREM key member [member ...]

Example

redis> ZADD myzset 1 "one"
(integer) 1
redis> ZADD myzset 2 "two"
(integer) 1
redis> ZADD myzset 3 "three"
(integer) 1
redis> ZREM myzset "two"
(integer) 1
redis> ZRANGE myzset 0 -1 WITHSCORES
1) "one"
2) "1"
3) "three"
4) "3"

Remove one or more members from a sorted set

ZREMRANGEBYLEX

ZREMRANGEBYLEX key min max

Example

redis> ZADD myzset 0 aaaa 0 b 0 c 0 d 0 e
(integer) 5
redis> ZADD myzset 0 foo 0 zap 0 zip 0 ALPHA 0 alpha
(integer) 5
redis> ZRANGE myzset 0 -1
1) "ALPHA"
 2) "aaaa"
 3) "alpha"
 4) "b"
 5) "c"
 6) "d"
 7) "e"
 8) "foo"
 9) "zap"
10) "zip"
redis> ZREMRANGEBYLEX myzset [alpha [omega
(integer) 6
redis> ZRANGE myzset 0 -1
1) "ALPHA"
2) "aaaa"
3) "zap"
4) "zip"

Remove all members in a sorted set between the given lexicographical range

ZREMRANGEBYRANK

ZREMRANGEBYRANK key start stop

Example

redis> ZADD myzset 1 "one"
(integer) 1
redis> ZADD myzset 2 "two"
(integer) 1
redis> ZADD myzset 3 "three"
(integer) 1
redis> ZREMRANGEBYRANK myzset 0 1
(integer) 2
redis> ZRANGE myzset 0 -1 WITHSCORES
1) "three"
2) "3"

Remove all members in a sorted set within the given indexes

ZREMRANGEBYSCORE

ZREMRANGEBYSCORE key min max

Example

redis> ZADD myzset 1 "one"
(integer) 1
redis> ZADD myzset 2 "two"
(integer) 1
redis> ZADD myzset 3 "three"
(integer) 1
redis> ZREMRANGEBYSCORE myzset -inf (2
(integer) 1
redis> ZRANGE myzset 0 -1 WITHSCORES
1) "two"
2) "2"
3) "three"
4) "3"

Remove all members in a sorted set within the given scores

ZREVRANGE

ZREVRANGE key start stop [WITHSCORES]

Example

redis> ZADD myzset 1 "one"
(integer) 1
redis> ZADD myzset 2 "two"
(integer) 1
redis> ZADD myzset 3 "three"
(integer) 1
redis> ZREVRANGE myzset 0 -1
1) "three"
2) "two"
3) "one"
redis> ZREVRANGE myzset 2 3
1) "one"
redis> ZREVRANGE myzset -2 -1
1) "two"
2) "one"

Return a range of members in a sorted set, by index, with scores ordered from high to low

ZREVRANGEBYSCORE

ZREVRANGEBYSCORE key max min [WITHSCORES] [LIMIT offset count]

Example

redis> ZADD myzset 1 "one"
(integer) 1
redis> ZADD myzset 2 "two"
(integer) 1
redis> ZADD myzset 3 "three"
(integer) 1
redis> ZREVRANGEBYSCORE myzset +inf -inf
1) "three"
2) "two"
3) "one"
redis> ZREVRANGEBYSCORE myzset 2 1
1) "two"
2) "one"
redis> ZREVRANGEBYSCORE myzset 2 (1
1) "two"
redis> ZREVRANGEBYSCORE myzset (2 (1
(empty list or set)

Return a range of members in a sorted set, by score, with scores ordered from high to low

ZREVRANK

ZREVRANK key member

Example

redis> ZADD myzset 1 "one"
(integer) 1
redis> ZADD myzset 2 "two"
(integer) 1
redis> ZADD myzset 3 "three"
(integer) 1
redis> ZREVRANK myzset "one"
(integer) 2
redis> ZREVRANK myzset "four"
(nil)

Determine the index of a member in a sorted set, with scores ordered from high to low

ZUNION

ZUNION numkeys key [key ...] [WEIGHTS weight [weight ...]] [AGGREGATE SUM|MIN|MAX] [WITHSCORES]

Example

redis> ZADD zset1 1 "one"
(integer) 1
redis> ZADD zset1 2 "two"
(integer) 1
redis> ZADD zset2 1 "one"
(integer) 1
redis> ZADD zset2 2 "two"
(integer) 1
redis> ZADD zset2 3 "three"
(integer) 1
redis> ZUNION 2 zset1 zset2
1) "one"
2) "three"
3) "two"
redis> ZUNION 2 zset1 zset2 WITHSCORES
1) "one"
2) "2"
3) "three"
4) "3"
5) "two"
6) "4"

Add multiple sorted sets

ZMSCORE

ZMSCORE key member [member ...]

Example

redis> ZADD myzset 1 "one"
(integer) 1
redis> ZADD myzset 2 "two"
(integer) 1
redis> ZMSCORE myzset "one" "two" "nofield"
1) "1"
2) "2"
3) (nil)

Get the score associated with the given members in a sorted set

ZUNIONSTORE

ZUNIONSTORE destination numkeys key [key ...] [WEIGHTS weight [weight ...]] [AGGREGATE SUM|MIN|MAX]

Example

redis> ZADD zset1 1 "one"
(integer) 1
redis> ZADD zset1 2 "two"
(integer) 1
redis> ZADD zset2 1 "one"
(integer) 1
redis> ZADD zset2 2 "two"
(integer) 1
redis> ZADD zset2 3 "three"
(integer) 1
redis> ZUNIONSTORE out 2 zset1 zset2 WEIGHTS 2 3
(integer) 3
redis> ZRANGE out 0 -1 WITHSCORES
1) "one"
2) "5"
3) "three"
4) "9"
5) "two"
6) "10"

Add multiple sorted sets and store the resulting sorted set in a new key

Redis Geo command

GEOADD

GEOADD key longitude latitude member [longitude latitude member ...]

Example

redis> GEOADD Sicily 13.361389 38.115556 "Palermo" 15.087269 37.502669 "Catania"
(integer) 2
redis> GEODIST Sicily Palermo Catania
"166274.1516"
redis> GEORADIUS Sicily 15 37 100 km
1) "Catania"
redis> GEORADIUS Sicily 15 37 200 km
1) "Palermo"
2) "Catania"

Add one or more geospatial items in the geospatial index represented using a sorted set

GEOHASH

GEOHASH key member [member ...]

Example

redis> GEOADD Sicily 13.361389 38.115556 "Palermo" 15.087269 37.502669 "Catania"
(integer) 2
redis> GEOHASH Sicily Palermo Catania
1) "sqc8b49rny0"
2) "sqdtr74hyu0"

Returns members of a geospatial index as standard geohash strings

GEOPOS

GEOPOS key member [member ...]

Example

redis> GEOADD Sicily 13.361389 38.115556 "Palermo" 15.087269 37.502669 "Catania"
(integer) 2
redis> GEOPOS Sicily Palermo Catania NonExisting
1) 1) "13.36138933897018433"
   2) "38.11555639549629859"
2) 1) "15.08726745843887329"
   2) "37.50266842333162032"
3) (nil)

Returns longitude and latitude of members of a geospatial index

GEODIST

GEODIST key member1 member2 [m|km|ft|mi]

Example

redis> GEOADD Sicily 13.361389 38.115556 "Palermo" 15.087269 37.502669 "Catania"
(integer) 2
redis> GEODIST Sicily Palermo Catania
"166274.1516"
redis> GEODIST Sicily Palermo Catania km
"166.2742"
redis> GEODIST Sicily Palermo Catania mi
"103.3182"
redis> GEODIST Sicily Foo Bar
(nil)

Returns the distance between two members of a geospatial index

GEORADIUS

GEORADIUS key longitude latitude radius m|km|ft|mi [WITHCOORD] [WITHDIST] [WITHHASH] [COUNT count] [ASC|DESC] [STORE key] [STOREDIST key]

Example

redis> GEOADD Sicily 13.361389 38.115556 "Palermo" 15.087269 37.502669 "Catania"
(integer) 2
redis> GEORADIUS Sicily 15 37 200 km WITHDIST
1) 1) "Palermo"
   2) "190.4424"
2) 1) "Catania"
   2) "56.4413"
redis> GEORADIUS Sicily 15 37 200 km WITHCOORD
1) 1) "Palermo"
   2) 1) "13.36138933897018433"
      2) "38.11555639549629859"
2) 1) "Catania"
   2) 1) "15.08726745843887329"
      2) "37.50266842333162032"
redis> GEORADIUS Sicily 15 37 200 km WITHDIST WITHCOORD
1) 1) "Palermo"
   2) "190.4424"
   3) 1) "13.36138933897018433"
      2) "38.11555639549629859"
2) 1) "Catania"
   2) "56.4413"
   3) 1) "15.08726745843887329"
      2) "37.50266842333162032"

Query a sorted set representing a geospatial index to fetch members matching a given maximum distance from a point

GEORADIUSBYMEMBER

GEORADIUSBYMEMBER key member radius m|km|ft|mi [WITHCOORD] [WITHDIST] [WITHHASH] [COUNT count] [ASC|DESC] [STORE key] [STOREDIST key]

Example

redis> GEOADD Sicily 13.583333 37.316667 "Agrigento"
(integer) 1
redis> GEOADD Sicily 13.361389 38.115556 "Palermo" 15.087269 37.502669 "Catania"
(integer) 2
redis> GEORADIUSBYMEMBER Sicily Agrigento 100 km
1) "Agrigento"
2) "Palermo"

Query a sorted set representing a geospatial index to fetch members matching a given maximum distance from a member

GEOSEARCH

GEOSEARCH key [FROMMEMBER member] [FROMLONLAT longitude latitude] [BYRADIUS radius m|km|ft|mi] [BYBOX width height m|km|ft|mi] [ASC|DESC] [COUNT count] [WITHCOORD] [WITHDIST] [WITHHASH]

Example

redis> GEOADD Sicily 13.361389 38.115556 "Palermo" 15.087269 37.502669 "Catania"
(integer) 2
redis> GEOADD Sicily 12.758489 38.788135 "edge1"   17.241510 38.788135 "edge2"
(integer) 2
redis> GEOSEARCH Sicily FROMLONLAT 15 37 BYRADIUS 200 km ASC
1) "Catania"
2) "Palermo"
redis> GEOSEARCH Sicily FROMLONLAT 15 37 BYBOX 400 400 km ASC
1) "Catania"
2) "Palermo"
3) "edge2"
4) "edge1"

Query a sorted set representing a geospatial index to fetch members inside an area of a box or a circle.

Misc

-: GEOSEARCHSTORE
- -: Query a sorted set representing a geospatial index to fetch members inside an area of a box or a circle, and store the result in another key.

Redis Hyperloglog command

PFADD

PFADD key element [element ...]

Example

redis> PFADD hll a b c d e f g
(integer) 1
redis> PFCOUNT hll
(integer) 7

Adds the specified elements to the specified HyperLogLog.

PFCOUNT

PFCOUNT key [key ...]

Example

redis> PFADD hll foo bar zap
(integer) 1
redis> PFADD hll zap zap zap
(integer) 0
redis> PFADD hll foo bar
(integer) 0
redis> PFCOUNT hll
(integer) 3
redis> PFADD some-other-hll 1 2 3
(integer) 1
redis> PFCOUNT hll some-other-hll
(integer) 6

Return the approximated cardinality of the set(s) observed by the HyperLogLog at key(s).

PFMERGE

PFMERGE destkey sourcekey [sourcekey ...]

Example

redis> PFADD hll1 foo bar zap a
(integer) 1
redis> PFADD hll2 a b c foo
(integer) 1
redis> PFMERGE hll3 hll1 hll2
"OK"
redis> PFCOUNT hll3
(integer) 6

Merge N different HyperLogLogs into a single one.

Redis Server command

COMMAND

COMMAND

Example

redis> COMMAND
1) 1) "georadius_ro"
     2) (integer) -6
     3) 1) "readonly"
        2) "movablekeys"
     4) (integer) 1
     5) (integer) 1
     6) (integer) 1
     7) 1) "@read"
        2) "@geo"
        3) "@slow"
  2) 1) "zpopmin"
     2) (integer) -2
     3) 1) "write"
        2) "fast"
  ........

Get array of Redis command details

Misc

-: ACL LOAD
- -: Reload the ACLs from the configured ACL file
-: ACL SAVE
- -: Save the current ACL rules in the configured ACL file
-: ACL LIST
- -: List the current ACL rules in ACL config file format
-: ACL USERS
- -: List the username of all the configured ACL rules
-: ACL GETUSER
- -: Get the rules for a specific ACL user
-: ACL SETUSER
- -: Modify or create the rules for a specific ACL user
-: ACL DELUSER
- -: Remove the specified ACL users and the associated rules
-: ACL CAT
- -: List the ACL categories or the commands inside a category
-: ACL GENPASS
- -: Generate a pseudorandom secure password to use for ACL users
-: ACL WHOAMI
- -: Return the name of the user associated to the current connection
-: ACL LOG
- -: List latest events denied because of ACLs in place
-: ACL HELP
- -: Show helpful text about the different subcommands
-: BGREWRITEAOF
- -: Asynchronously rewrite the append-only file
-: BGSAVE
- -: Asynchronously save the dataset to disk
-: CONFIG GET
- -: Get the value of a configuration parameter
-: CONFIG REWRITE
- -: Rewrite the configuration file with the in memory configuration
-: CONFIG SET
- -: Set a configuration parameter to the given value
-: CONFIG RESETSTAT
- -: Reset the stats returned by INFO
-: DBSIZE
- -: Return the number of keys in the selected database
-: DEBUG OBJECT
- -: Get debugging information about a key
-: DEBUG SEGFAULT
- -: Make the server crash
-: FLUSHALL
- -: Remove all keys from all databases
-: FLUSHDB
- -: Remove all keys from the current database
-: LOLWUT
- -: Display some computer art and the Redis version
-: LASTSAVE
- -: Get the UNIX time stamp of the last successful save to disk
-: MEMORY DOCTOR
- -: Outputs memory problems report
-: MEMORY HELP
- -: Show helpful text about the different subcommands
-: MEMORY MALLOC-STATS
- -: Show allocator internal stats
-: MEMORY PURGE
- -: Ask the allocator to release memory
-: MEMORY STATS
- -: Show memory usage details
-: MEMORY USAGE
- -: Estimate the memory usage of a key
-: MODULE LIST
- -: List all modules loaded by the server
-: MODULE LOAD
- -: Load a module
-: MODULE UNLOAD
- -: Unload a module
-: MONITOR
- -: Listen for all requests received by the server in real time
-: SAVE
- -: Synchronously save the dataset to disk
-: SHUTDOWN
- -: Synchronously save the dataset to disk and then shut down the server
-: SLAVEOF
- -: Make the server a replica of another instance, or promote it as master. Deprecated starting with Redis 5. Use REPLICAOF instead.
-: REPLICAOF
- -: Make the server a replica of another instance, or promote it as master.
-: SLOWLOG
- -: Manages the Redis slow queries log
-: SWAPDB
- -: Swaps two Redis databases
-: SYNC
- -: Internal command used for replication
-: PSYNC
- -: Internal command used for replication
-: LATENCY DOCTOR
- -: Return a human readable latency analysis report.
-: LATENCY GRAPH
- -: Return a latency graph for the event.
-: LATENCY HISTORY
- -: Return timestamp-latency samples for the event.
-: LATENCY LATEST
- -: Return the latest latency samples for all events.
-: LATENCY RESET
- -: Reset latency data for one or more events.
-: LATENCY HELP
- -: Show helpful text about the different subcommands.

COMMAND COUNT

COMMAND COUNT

Example

redis> COMMAND COUNT
(integer) 217

Get total number of Redis commands

COMMAND GETKEYS

COMMAND GETKEYS

Example

redis> COMMAND GETKEYS MSET a b c d e f
1) "a"
2) "c"
3) "e"
redis> COMMAND GETKEYS EVAL "not consulted" 3 key1 key2 key3 arg1 arg2 arg3 argN
1) "key1"
2) "key2"
3) "key3"
redis> COMMAND GETKEYS SORT mylist ALPHA STORE outlist
1) "mylist"
2) "outlist"

Extract keys given a full Redis command

COMMAND INFO

COMMAND INFO command-name [command-name ...]

Example

redis> COMMAND INFO get set eval
1) 1) "get"
   2) (integer) 2
   3) 1) "readonly"
      2) "fast"
   4) (integer) 1
   5) (integer) 1
   6) (integer) 1
   7) 1) "@read"
      2) "@string"
      3) "@fast"
2) 1) "set"
   2) (integer) -3
   3) 1) "write"
      2) "denyoom"
   4) (integer) 1
   5) (integer) 1
   6) (integer) 1
   7) 1) "@write"
      2) "@string"
      3) "@slow"
3) 1) "eval"
   2) (integer) -3
   3) 1) "noscript"
      2) "movablekeys"
   4) (integer) 0
   5) (integer) 0
   6) (integer) 0
   7) 1) "@slow"
      2) "@scripting"

Get array of specific Redis command details

INFO

INFO [section]

Example

redis> INFO
# Server
redis_version:6.1.240
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:a26db646ea64a07c
redis_mode:standalone
os:Linux 5.4.0-1017-aws x86_64
......

Get information and statistics about the server

ROLE

ROLE

Example

redis> ROLE
1) "master"
2) (integer) 0
3) (empty list or set)

Return the role of the instance in the context of replication

TIME

TIME

Example

redis> TIME
1) "1609040690"
2) "558952"
redis> TIME
1) "1609040690"
2) "559206"

Return the current server time

Redis Generic command

Misc

-: COPY
- -: Copy a key
-: MIGRATE
- -: Atomically transfer a key from a Redis instance to another one.
-: MOVE
- -: Move a key to another database
-: OBJECT
- -: Inspect the internals of Redis objects
-: RESTORE
- -: Create a key using the provided serialized value, previously obtained using DUMP.
-: SORT
- -: Sort the elements in a list, set or sorted set
-: WAIT
- -: Wait for the synchronous replication of all the write commands sent in the context of the current connection
-: SCAN
- -: Incrementally iterate the keys space

DEL

DEL key [key ...]

Example

redis> SET key1 "Hello"
"OK"
redis> SET key2 "World"
"OK"
redis> DEL key1 key2 key3
(integer) 2

Delete a key

DUMP

DUMP key

Example

redis> SET mykey 10
"OK"
redis> DUMP mykey
"\u0000\xC0\n\t\u0000\xBEm\u0006\x89Z(\u0000\n"

Return a serialized version of the value stored at the specified key.

EXISTS

EXISTS key [key ...]

Example

redis> SET key1 "Hello"
"OK"
redis> EXISTS key1
(integer) 1
redis> EXISTS nosuchkey
(integer) 0
redis> SET key2 "World"
"OK"
redis> EXISTS key1 key2 nosuchkey
(integer) 2

Determine if a key exists

EXPIRE

EXPIRE key seconds

Example

redis> SET mykey "Hello"
"OK"
redis> EXPIRE mykey 10
(integer) 1
redis> TTL mykey
(integer) 10
redis> SET mykey "Hello World"
"OK"
redis> TTL mykey
(integer) -1

Set a key's time to live in seconds

EXPIREAT

EXPIREAT key timestamp

Example

redis> SET mykey "Hello"
"OK"
redis> EXISTS mykey
(integer) 1
redis> EXPIREAT mykey 1293840000
(integer) 1
redis> EXISTS mykey
(integer) 0

Set the expiration for a key as a UNIX timestamp

KEYS

KEYS pattern

Example

redis> MSET firstname Jack lastname Stuntman age 35
"OK"
redis> KEYS *name*
1) "firstname"
2) "lastname"
redis> KEYS a??
1) "age"
redis> KEYS *
1) "firstname"
2) "age"
3) "lastname"

Find all keys matching the given pattern

PERSIST

PERSIST key

Example

redis> SET mykey "Hello"
"OK"
redis> EXPIRE mykey 10
(integer) 1
redis> TTL mykey
(integer) 10
redis> PERSIST mykey
(integer) 1
redis> TTL mykey
(integer) -1

Remove the expiration from a key

PEXPIRE

PEXPIRE key milliseconds

Example

redis> SET mykey "Hello"
"OK"
redis> PEXPIRE mykey 1500
(integer) 1
redis> TTL mykey
(integer) 1
redis> PTTL mykey
(integer) 1499

Set a key's time to live in milliseconds

PEXPIREAT

PEXPIREAT key milliseconds-timestamp

Example

redis> SET mykey "Hello"
"OK"
redis> PEXPIREAT mykey 1555555555005
(integer) 1
redis> TTL mykey
(integer) -2
redis> PTTL mykey
(integer) -2

Set the expiration for a key as a UNIX timestamp specified in milliseconds

PTTL

PTTL key

Example

redis> SET mykey "Hello"
"OK"
redis> EXPIRE mykey 1
(integer) 1
redis> PTTL mykey
(integer) 1000

Get the time to live for a key in milliseconds

RENAME

RENAME key newkey

Example

redis> SET mykey "Hello"
"OK"
redis> RENAME mykey myotherkey
"OK"
redis> GET myotherkey
"Hello"

Rename a key

RENAMENX

RENAMENX key newkey

Example

redis> SET mykey "Hello"
"OK"
redis> SET myotherkey "World"
"OK"
redis> RENAMENX mykey myotherkey
(integer) 0
redis> GET myotherkey
"World"

Rename a key, only if the new key does not exist

TOUCH

TOUCH key [key ...]

Example

redis> SET key1 "Hello"
"OK"
redis> SET key2 "World"
"OK"
redis> TOUCH key1 key2
(integer) 2

Alters the last access time of a key(s). Returns the number of existing keys specified.

TTL

TTL key

Example

redis> SET mykey "Hello"
"OK"
redis> EXPIRE mykey 10
(integer) 1
redis> TTL mykey
(integer) 10

Get the time to live for a key

TYPE

TYPE key

Example

redis> SET key1 "value"
"OK"
redis> LPUSH key2 "value"
(integer) 1
redis> SADD key3 "value"
(integer) 1
redis> TYPE key1
"string"
redis> TYPE key2
"list"
redis> TYPE key3
"set"

Determine the type stored at key

UNLINK

UNLINK key [key ...]

Example

redis> SET key1 "Hello"
"OK"
redis> SET key2 "World"
"OK"
redis> UNLINK key1 key2 key3
(integer) 2

Delete a key asynchronously in another thread. Otherwise it is just as DEL, but non blocking.

Redis Connection command

Misc

-	-
AUTH	Authenticate to the server
CLIENT CACHING	Instruct the server about tracking or not keys in the next request
CLIENT KILL	Kill the connection of a client
CLIENT LIST	Get the list of client connections
CLIENT GETNAME	Get the current connection name
CLIENT GETREDIR	Get tracking notifications redirection client ID if any
CLIENT PAUSE	Stop processing commands from clients for some time
CLIENT REPLY	Instruct the server whether to reply to commands
CLIENT SETNAME	Set the current connection name
CLIENT TRACKING	Enable or disable server assisted client side caching support
CLIENT UNBLOCK	Unblock a client blocked in a blocking command from a different connection
HELLO	switch Redis protocol
QUIT	Close the connection
RESET	Reset the connection
SELECT	Change the selected database for the current connection

CLIENT ID

CLIENT ID

Example

redis> CLIENT ID
ERR Unknown or disabled command 'CLIENT'

Returns the client ID for the current connection

CLIENT INFO

CLIENT INFO

Example

redis> CLIENT INFO
"id=55542 addr=127.0.0.1:58710 laddr=127.0.0.1:6379 fd=8 name= age=114920 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=26 qbuf-free=40928 argv-mem=10 obl=0 oll=0 omem=0 tot-mem=61466 events=r cmd=client user=default redir=-1\n"

Returns information about the current client connection.

ECHO

ECHO message

Example

redis> ECHO "Hello World!"
"Hello World!"

Echo the given string

PING

PING [message]

Example

redis> PING
"PONG"
redis> PING "hello world"
"hello world"

Ping the server

Redis Stream command

Misc

-: XINFO
- -: Get information on streams and consumer groups
-: XDEL
- -: Removes the specified entries from the stream. Returns the number of items actually deleted, that may be different from the number of IDs passed in case certain IDs do not exist.
-: XREAD
- -: Return never seen elements in multiple streams, with IDs greater than the ones reported by the caller for each stream. Can block.
-: XGROUP
- -: Create, destroy, and manage consumer groups.
-: XREADGROUP
- -: Return new entries from a stream using a consumer group, or access the history of the pending entries for a given consumer. Can block.
-: XCLAIM
- -: Changes (or acquires) ownership of a message in a consumer group, as if the message was delivered to the specified consumer.
-: XPENDING
- -: Return information and entries from a stream consumer group pending entries list, that are messages fetched but never acknowledged.

XADD

XADD key [MAXLEN [=|~] length] [NOMKSTREAM] *|ID field value [field value ...]

Example

redis> XADD mystream * name Sara surname OConnor
"1609040574632-0"
redis> XADD mystream * field1 value1 field2 value2 field3 value3
"1609040574632-1"
redis> XLEN mystream
(integer) 2
redis> XRANGE mystream - +
1) 1) "1609040574632-0"
   2) 1) "name"
      2) "Sara"
      3) "surname"
      4) "OConnor"
2) 1) "1609040574632-1"
   2) 1) "field1"
      2) "value1"
      3) "field2"
      4) "value2"
      5) "field3"
      6) "value3"

Appends a new entry to a stream

XTRIM

XTRIM key MAXLEN [=|~] length

Example

redis> XADD mystream * field1 A field2 B field3 C field4 D
"1609040575750-0"
redis> XTRIM mystream MAXLEN 2
(integer) 0
redis> XRANGE mystream - +
1) 1) "1609040575750-0"
   2) 1) "field1"
      2) "A"
      3) "field2"
      4) "B"
      5) "field3"
      6) "C"
      7) "field4"
      8) "D"

Trims the stream to (approximately if '~' is passed) a certain size

XRANGE

XRANGE key start end [COUNT count]

Example

redis> XADD writers * name Virginia surname Woolf
"1609040578002-0"
redis> XADD writers * name Jane surname Austen
"1609040578002-1"
redis> XADD writers * name Toni surname Morrison
"1609040578003-0"
redis> XADD writers * name Agatha surname Christie
"1609040578003-1"
redis> XADD writers * name Ngozi surname Adichie
"1609040578003-2"
redis> XLEN writers
(integer) 5
redis> XRANGE writers - + COUNT 2
1) 1) "1609040578002-0"
   2) 1) "name"
      2) "Virginia"
      3) "surname"
      4) "Woolf"
2) 1) "1609040578002-1"
   2) 1) "name"
      2) "Jane"
      3) "surname"
      4) "Austen"

Return a range of elements in a stream, with IDs matching the specified IDs interval

XREVRANGE

XREVRANGE key end start [COUNT count]

Example

redis> XADD writers * name Virginia surname Woolf
"1609040579130-0"
redis> XADD writers * name Jane surname Austen
"1609040579130-1"
redis> XADD writers * name Toni surname Morrison
"1609040579130-2"
redis> XADD writers * name Agatha surname Christie
"1609040579131-0"
redis> XADD writers * name Ngozi surname Adichie
"1609040579131-1"
redis> XLEN writers
(integer) 5
redis> XREVRANGE writers + - COUNT 1
1) 1) "1609040579131-1"
   2) 1) "name"
      2) "Ngozi"
      3) "surname"
      4) "Adichie"

Return a range of elements in a stream, with IDs matching the specified IDs interval, in reverse order (from greater to smaller IDs) compared to XRANGE

XLEN

XLEN key

Example

redis> XADD mystream * item 1
"1609040580250-0"
redis> XADD mystream * item 2
"1609040580250-1"
redis> XADD mystream * item 3
"1609040580251-0"
redis> XLEN mystream
(integer) 3

Return the number of entries in a stream

XACK

XACK key group ID [ID ...]

Example

redis> XACK mystream mygroup 1526569495631-0
ERR Unknown or disabled command 'XACK'

Marks a pending message as correctly processed, effectively removing it from the pending entries list of the consumer group. Return value of the command is the number of messages successfully acknowledged, that is, the IDs we were actually able to resolve in the PEL.

Miscellaneous

Cluster

Transactions

-	-
DISCARD	Discard all commands issued after MULTI
EXEC	Execute all commands issued after MULTI
MULTI	Mark the start of a transaction block
UNWATCH	Forget about all watched keys
WATCH	Watch the given keys to determine execution of the MULTI/EXEC block

Scripting

-	-
EVAL	Execute a Lua script server side
EVALSHA	Execute a Lua script server side
SCRIPT DEBUG	Set the debug mode for executed scripts.
SCRIPT EXISTS	Check existence of scripts in the script cache.
SCRIPT FLUSH	Remove all the scripts from the script cache.
SCRIPT KILL	Kill the script currently in execution.
SCRIPT LOAD	Load the specified Lua script into the script cache.

Pubsub

-	-
PSUBSCRIBE	Listen for messages published to channels matching the given patterns
PUBSUB	Inspect the state of the Pub/Sub subsystem
PUBLISH	Post a message to a channel
PUNSUBSCRIBE	Stop listening for messages posted to channels matching the given patterns
SUBSCRIBE	Listen for messages published to the given channels
UNSUBSCRIBE	Stop listening for messages posted to the given channels

Master DevOps If You Master it : System Design !!

Hoang Guruu — Thu, 27 Feb 2025 05:46:57 +0000

【 👨🏻‍💻 YouTube | 🕵️ LinkedIn】

System Design 101

Explain complex systems using visuals and simple terms.

Whether you're preparing for a System Design Interview or you simply want to understand how systems work beneath the surface, we hope this repository will help you achieve that.

Communication protocols
- REST API vs. GraphQL
- How does gRPC work?
- What is a webhook?
- How to improve API performance?
- HTTP 1.0 -> HTTP 1.1 -> HTTP 2.0 -> HTTP 3.0 (QUIC)
- SOAP vs REST vs GraphQL vs RPC
- Code First vs. API First
- HTTP status codes
- What does API gateway do?
- How do we design effective and safe APIs?
- TCP/IP encapsulation
- Why is Nginx called a “reverse” proxy?
- What are the common load-balancing algorithms?
- URL, URI, URN - Do you know the differences?
CI/CD
- CI/CD Pipeline Explained in Simple Terms
- Netflix Tech Stack (CI/CD Pipeline)
Architecture patterns
- MVC, MVP, MVVM, MVVM-C, and VIPER
- 18 Key Design Patterns Every Developer Should Know
Database
- A nice cheat sheet of different databases in cloud services
- 8 Data Structures That Power Your Databases
- How is an SQL statement executed in the database?
- CAP theorem
- Types of Memory and Storage
- Visualizing a SQL query
- SQL language
Cache
- Data is cached everywhere
- Why is Redis so fast?
- How can Redis be used?
- Top caching strategies
Microservice architecture
- What does a typical microservice architecture look like?
- Microservice Best Practices
- What tech stack is commonly used for microservices?
- Why is Kafka fast
Payment systems
- How to learn payment systems?
- Why is the credit card called “the most profitable product in banks”? How does VISA/Mastercard make money?
- How does VISA work when we swipe a credit card at a merchant’s shop?
- Payment Systems Around The World Series (Part 1): Unified Payments Interface (UPI) in India
DevOps
- DevOps vs. SRE vs. Platform Engineering. What is the difference?
- What is k8s (Kubernetes)?
- Docker vs. Kubernetes. Which one should we use?
- How does Docker work?
GIT
- How Git Commands work
- How does Git Work?
- Git merge vs. Git rebase
Cloud Services
- A nice cheat sheet of different cloud services (2023 edition)
- What is cloud native?
Developer productivity tools
- Visualize JSON files
- Automatically turn code into architecture diagrams
Linux
- Linux file system explained
- 18 Most-used Linux Commands You Should Know
Security
- How does HTTPS work?
- Oauth 2.0 Explained With Simple Terms.
- Top 4 Forms of Authentication Mechanisms
- Session, cookie, JWT, token, SSO, and OAuth 2.0 - what are they?
- How to store passwords safely in the database and how to validate a password?
- Explaining JSON Web Token (JWT) to a 10 year old Kid
- How does Google Authenticator (or other types of 2-factor authenticators) work?
Real World Case Studies
- Netflix's Tech Stack
- Twitter Architecture 2022
- Evolution of Airbnb’s microservice architecture over the past 15 years
- Monorepo vs. Microrepo.
- How will you design the Stack Overflow website?
- Why did Amazon Prime Video monitoring move from serverless to monolithic? How can it save 90% cost?
- How does Disney Hotstar capture 5 Billion Emojis during a tournament?
- How Discord Stores Trillions Of Messages
- How do video live streamings work on YouTube, TikTok live, or Twitch?

Communication protocols

Architecture styles define how different components of an application programming interface (API) interact with one another. As a result, they ensure efficiency, reliability, and ease of integration with other systems by providing a standard approach to designing and building APIs. Here are the most used styles:

SOAP:

Mature, comprehensive, XML-based

Best for enterprise applications
RESTful:

Popular, easy-to-implement, HTTP methods

Ideal for web services
GraphQL:

Query language, request specific data

Reduces network overhead, faster responses
gRPC:

Modern, high-performance, Protocol Buffers

Suitable for microservices architectures
WebSocket:

Real-time, bidirectional, persistent connections

Perfect for low-latency data exchange
Webhook:

Event-driven, HTTP callbacks, asynchronous

Notifies systems when events occur

REST API vs. GraphQL

When it comes to API design, REST and GraphQL each have their own strengths and weaknesses.

The diagram below shows a quick comparison between REST and GraphQL.

REST

Uses standard HTTP methods like GET, POST, PUT, DELETE for CRUD operations.
Works well when you need simple, uniform interfaces between separate services/applications.
Caching strategies are straightforward to implement.
The downside is it may require multiple roundtrips to assemble related data from separate endpoints.

GraphQL

Provides a single endpoint for clients to query for precisely the data they need.
Clients specify the exact fields required in nested queries, and the server returns optimized payloads containing just those fields.
Supports Mutations for modifying data and Subscriptions for real-time notifications.
Great for aggregating data from multiple sources and works well with rapidly evolving frontend requirements.
However, it shifts complexity to the client side and can allow abusive queries if not properly safeguarded
Caching strategies can be more complicated than REST.

The best choice between REST and GraphQL depends on the specific requirements of the application and development team. GraphQL is a good fit for complex or frequently changing frontend needs, while REST suits applications where simple and consistent contracts are preferred.

Neither API approach is a silver bullet. Carefully evaluating requirements and tradeoffs is important to pick the right style. Both REST and GraphQL are valid options for exposing data and powering modern applications.

How does gRPC work?

RPC (Remote Procedure Call) is called “remote” because it enables communications between remote services when services are deployed to different servers under microservice architecture. From the user’s point of view, it acts like a local function call.

The diagram below illustrates the overall data flow for gRPC.

Step 1: A REST call is made from the client. The request body is usually in JSON format.

Steps 2 - 4: The order service (gRPC client) receives the REST call, transforms it, and makes an RPC call to the payment service. gRPC encodes the client stub into a binary format and sends it to the low-level transport layer.

Step 5: gRPC sends the packets over the network via HTTP2. Because of binary encoding and network optimizations, gRPC is said to be 5X faster than JSON.

Steps 6 - 8: The payment service (gRPC server) receives the packets from the network, decodes them, and invokes the server application.

Steps 9 - 11: The result is returned from the server application, and gets encoded and sent to the transport layer.

Steps 12 - 14: The order service receives the packets, decodes them, and sends the result to the client application.

What is a webhook?

The diagram below shows a comparison between polling and Webhook.

Assume we run an eCommerce website. The clients send orders to the order service via the API gateway, which goes to the payment service for payment transactions. The payment service then talks to an external payment service provider (PSP) to complete the transactions.

There are two ways to handle communications with the external PSP.

1. Short polling

After sending the payment request to the PSP, the payment service keeps asking the PSP about the payment status. After several rounds, the PSP finally returns with the status.

Short polling has two drawbacks:

Constant polling of the status requires resources from the payment service.
The External service communicates directly with the payment service, creating security vulnerabilities.

2. Webhook

We can register a webhook with the external service. It means: call me back at a certain URL when you have updates on the request. When the PSP has completed the processing, it will invoke the HTTP request to update the payment status.

In this way, the programming paradigm is changed, and the payment service doesn’t need to waste resources to poll the payment status anymore.

What if the PSP never calls back? We can set up a housekeeping job to check payment status every hour.

Webhooks are often referred to as reverse APIs or push APIs because the server sends HTTP requests to the client. We need to pay attention to 3 things when using a webhook:

We need to design a proper API for the external service to call.
We need to set up proper rules in the API gateway for security reasons.
We need to register the correct URL at the external service.

How to improve API performance?

The diagram below shows 5 common tricks to improve API performance.

Pagination

This is a common optimization when the size of the result is large. The results are streaming back to the client to improve the service responsiveness.

Asynchronous Logging

Synchronous logging deals with the disk for every call and can slow down the system. Asynchronous logging sends logs to a lock-free buffer first and immediately returns. The logs will be flushed to the disk periodically. This significantly reduces the I/O overhead.

Caching

We can store frequently accessed data into a cache. The client can query the cache first instead of visiting the database directly. If there is a cache miss, the client can query from the database. Caches like Redis store data in memory, so the data access is much faster than the database.

Payload Compression

The requests and responses can be compressed using gzip etc so that the transmitted data size is much smaller. This speeds up the upload and download.

Connection Pool

When accessing resources, we often need to load data from the database. Opening the closing db connections adds significant overhead. So we should connect to the db via a pool of open connections. The connection pool is responsible for managing the connection lifecycle.

HTTP 1.0 -> HTTP 1.1 -> HTTP 2.0 -> HTTP 3.0 (QUIC)

What problem does each generation of HTTP solve?

The diagram below illustrates the key features.

HTTP 1.0 was finalized and fully documented in 1996. Every request to the same server requires a separate TCP connection.
HTTP 1.1 was published in 1997. A TCP connection can be left open for reuse (persistent connection), but it doesn’t solve the HOL (head-of-line) blocking issue.

HOL blocking - when the number of allowed parallel requests in the browser is used up, subsequent requests need to wait for the former ones to complete.
HTTP 2.0 was published in 2015. It addresses HOL issue through request multiplexing, which eliminates HOL blocking at the application layer, but HOL still exists at the transport (TCP) layer.

As you can see in the diagram, HTTP 2.0 introduced the concept of HTTP “streams”: an abstraction that allows multiplexing different HTTP exchanges onto the same TCP connection. Each stream doesn’t need to be sent in order.
HTTP 3.0 first draft was published in 2020. It is the proposed successor to HTTP 2.0. It uses QUIC instead of TCP for the underlying transport protocol, thus removing HOL blocking in the transport layer.

QUIC is based on UDP. It introduces streams as first-class citizens at the transport layer. QUIC streams share the same QUIC connection, so no additional handshakes and slow starts are required to create new ones, but QUIC streams are delivered independently such that in most cases packet loss affecting one stream doesn't affect others.

SOAP vs REST vs GraphQL vs RPC

The diagram below illustrates the API timeline and API styles comparison.

Over time, different API architectural styles are released. Each of them has its own patterns of standardizing data exchange.

You can check out the use cases of each style in the diagram.

Code First vs. API First

The diagram below shows the differences between code-first development and API-first development. Why do we want to consider API first design?

Microservices increase system complexity and we have separate services to serve different functions of the system. While this kind of architecture facilitates decoupling and segregation of duty, we need to handle the various communications among services.

It is better to think through the system's complexity before writing the code and carefully defining the boundaries of the services.

Separate functional teams need to speak the same language and the dedicated functional teams are only responsible for their own components and services. It is recommended that the organization speak the same language via API design.

We can mock requests and responses to validate the API design before writing code.

Improve software quality and developer productivity Since we have ironed out most of the uncertainties when the project starts, the overall development process is smoother, and the software quality is greatly improved.

Developers are happy about the process as well because they can focus on functional development instead of negotiating sudden changes.

The possibility of having surprises toward the end of the project lifecycle is reduced.

Because we have designed the API first, the tests can be designed while the code is being developed. In a way, we also have TDD (Test Driven Design) when using API first development.

HTTP status codes

The response codes for HTTP are divided into five categories:

Informational (100-199) Success (200-299) Redirection (300-399) Client Error (400-499) Server Error (500-599)

What does API gateway do?

The diagram below shows the details.

Step 1 - The client sends an HTTP request to the API gateway.

Step 2 - The API gateway parses and validates the attributes in the HTTP request.

Step 3 - The API gateway performs allow-list/deny-list checks.

Step 4 - The API gateway talks to an identity provider for authentication and authorization.

Step 5 - The rate limiting rules are applied to the request. If it is over the limit, the request is rejected.

Steps 6 and 7 - Now that the request has passed basic checks, the API gateway finds the relevant service to route to by path matching.

Step 8 - The API gateway transforms the request into the appropriate protocol and sends it to backend microservices.

Steps 9-12: The API gateway can handle errors properly, and deals with faults if the error takes a longer time to recover (circuit break). It can also leverage ELK (Elastic-Logstash-Kibana) stack for logging and monitoring. We sometimes cache data in the API gateway.

How do we design effective and safe APIs?

The diagram below shows typical API designs with a shopping cart example.

Note that API design is not just URL path design. Most of the time, we need to choose the proper resource names, identifiers, and path patterns. It is equally important to design proper HTTP header fields or to design effective rate-limiting rules within the API gateway.

TCP/IP encapsulation

How is data sent over the network? Why do we need so many layers in the OSI model?

The diagram below shows how data is encapsulated and de-encapsulated when transmitting over the network.

Step 1: When Device A sends data to Device B over the network via the HTTP protocol, it is first added an HTTP header at the application layer.

Step 2: Then a TCP or a UDP header is added to the data. It is encapsulated into TCP segments at the transport layer. The header contains the source port, destination port, and sequence number.

Step 3: The segments are then encapsulated with an IP header at the network layer. The IP header contains the source/destination IP addresses.

Step 4: The IP datagram is added a MAC header at the data link layer, with source/destination MAC addresses.

Step 5: The encapsulated frames are sent to the physical layer and sent over the network in binary bits.

Steps 6-10: When Device B receives the bits from the network, it performs the de-encapsulation process, which is a reverse processing of the encapsulation process. The headers are removed layer by layer, and eventually, Device B can read the data.

We need layers in the network model because each layer focuses on its own responsibilities. Each layer can rely on the headers for processing instructions and does not need to know the meaning of the data from the last layer.

Why is Nginx called a “reverse” proxy?

The diagram below shows the differences between a ��𝐨𝐫𝐰𝐚𝐫𝐝 𝐩𝐫𝐨𝐱𝐲 and a 𝐫𝐞𝐯𝐞𝐫𝐬𝐞 𝐩𝐫𝐨𝐱𝐲.

A forward proxy is a server that sits between user devices and the internet.

A forward proxy is commonly used for:

Protecting clients
Circumventing browsing restrictions
Blocking access to certain content

A reverse proxy is a server that accepts a request from the client, forwards the request to web servers, and returns the results to the client as if the proxy server had processed the request.

A reverse proxy is good for:

Protecting servers
Load balancing
Caching static contents
Encrypting and decrypting SSL communications

What are the common load-balancing algorithms?

The diagram below shows 6 common algorithms.

Static Algorithms

Round robin

The client requests are sent to different service instances in sequential order. The services are usually required to be stateless.
Sticky round-robin

This is an improvement of the round-robin algorithm. If Alice’s first request goes to service A, the following requests go to service A as well.
Weighted round-robin

The admin can specify the weight for each service. The ones with a higher weight handle more requests than others.
Hash

This algorithm applies a hash function on the incoming requests’ IP or URL. The requests are routed to relevant instances based on the hash function result.

Dynamic Algorithms

Least connections

A new request is sent to the service instance with the least concurrent connections.
Least response time

A new request is sent to the service instance with the fastest response time.

URL, URI, URN - Do you know the differences?

The diagram below shows a comparison of URL, URI, and URN.

URI stands for Uniform Resource Identifier. It identifies a logical or physical resource on the web. URL and URN are subtypes of URI. URL locates a resource, while URN names a resource.

A URI is composed of the following parts: scheme:[//authority]path[?query][#fragment]

URL stands for Uniform Resource Locator, the key concept of HTTP. It is the address of a unique resource on the web. It can be used with other protocols like FTP and JDBC.

URN stands for Uniform Resource Name. It uses the urn scheme. URNs cannot be used to locate a resource. A simple example given in the diagram is composed of a namespace and a namespace-specific string.

If you would like to learn more detail on the subject, I would recommend W3C’s clarification.

CI/CD

CI/CD Pipeline Explained in Simple Terms

Section 1 - SDLC with CI/CD

The software development life cycle (SDLC) consists of several key stages: development, testing, deployment, and maintenance. CI/CD automates and integrates these stages to enable faster and more reliable releases.

When code is pushed to a git repository, it triggers an automated build and test process. End-to-end (e2e) test cases are run to validate the code. If tests pass, the code can be automatically deployed to staging/production. If issues are found, the code is sent back to development for bug fixing. This automation provides fast feedback to developers and reduces the risk of bugs in production.

Section 2 - Difference between CI and CD

Continuous Integration (CI) automates the build, test, and merge process. It runs tests whenever code is committed to detect integration issues early. This encourages frequent code commits and rapid feedback.

Continuous Delivery (CD) automates release processes like infrastructure changes and deployment. It ensures software can be released reliably at any time through automated workflows. CD may also automate the manual testing and approval steps required before production deployment.

Section 3 - CI/CD Pipeline

A typical CI/CD pipeline has several connected stages:

The developer commits code changes to the source control
CI server detects changes and triggers the build
Code is compiled, and tested (unit, integration tests)
Test results reported to the developer
On success, artifacts are deployed to staging environments
Further testing may be done on staging before release
CD system deploys approved changes to production

Netflix Tech Stack (CI/CD Pipeline)

Planning: Netflix Engineering uses JIRA for planning and Confluence for documentation.

Coding: Java is the primary programming language for the backend service, while other languages are used for different use cases.

Build: Gradle is mainly used for building, and Gradle plugins are built to support various use cases.

Packaging: Package and dependencies are packed into an Amazon Machine Image (AMI) for release.

Testing: Testing emphasizes the production culture's focus on building chaos tools.

Deployment: Netflix uses its self-built Spinnaker for canary rollout deployment.

Monitoring: The monitoring metrics are centralized in Atlas, and Kayenta is used to detect anomalies.

Incident report: Incidents are dispatched according to priority, and PagerDuty is used for incident handling.

Architecture patterns

MVC, MVP, MVVM, MVVM-C, and VIPER

These architecture patterns are among the most commonly used in app development, whether on iOS or Android platforms. Developers have introduced them to overcome the limitations of earlier patterns. So, how do they differ?

MVC, the oldest pattern, dates back almost 50 years
Every pattern has a "view" (V) responsible for displaying content and receiving user input
Most patterns include a "model" (M) to manage business data
"Controller," "presenter," and "view-model" are translators that mediate between the view and the model ("entity" in the VIPER pattern)

18 Key Design Patterns Every Developer Should Know

Patterns are reusable solutions to common design problems, resulting in a smoother, more efficient development process. They serve as blueprints for building better software structures. These are some of the most popular patterns:

Abstract Factory: Family Creator - Makes groups of related items.
Builder: Lego Master - Builds objects step by step, keeping creation and appearance separate.
Prototype: Clone Maker - Creates copies of fully prepared examples.
Singleton: One and Only - A special class with just one instance.
Adapter: Universal Plug - Connects things with different interfaces.
Bridge: Function Connector - Links how an object works to what it does.
Composite: Tree Builder - Forms tree-like structures of simple and complex parts.
Decorator: Customizer - Adds features to objects without changing their core.
Facade: One-Stop-Shop - Represents a whole system with a single, simplified interface.
Flyweight: Space Saver - Shares small, reusable items efficiently.
Proxy: Stand-In Actor - Represents another object, controlling access or actions.
Chain of Responsibility: Request Relay - Passes a request through a chain of objects until handled.
Command: Task Wrapper - Turns a request into an object, ready for action.
Iterator: Collection Explorer - Accesses elements in a collection one by one.
Mediator: Communication Hub - Simplifies interactions between different classes.
Memento: Time Capsule - Captures and restores an object's state.
Observer: News Broadcaster - Notifies classes about changes in other objects.
Visitor: Skillful Guest - Adds new operations to a class without altering it.

Database

A nice cheat sheet of different databases in cloud services

Choosing the right database for your project is a complex task. Many database options, each suited to distinct use cases, can quickly lead to decision fatigue.

We hope this cheat sheet provides high-level direction to pinpoint the right service that aligns with your project's needs and avoid potential pitfalls.

Note: Google has limited documentation for their database use cases. Even though we did our best to look at what was available and arrived at the best option, some of the entries may need to be more accurate.

8 Data Structures That Power Your Databases

The answer will vary depending on your use case. Data can be indexed in memory or on disk. Similarly, data formats vary, such as numbers, strings, geographic coordinates, etc. The system might be write-heavy or read-heavy. All of these factors affect your choice of database index format.

The following are some of the most popular data structures used for indexing data:

Skiplist: a common in-memory index type. Used in Redis
Hash index: a very common implementation of the “Map” data structure (or “Collection”)
SSTable: immutable on-disk “Map” implementation
LSM tree: Skiplist + SSTable. High write throughput
B-tree: disk-based solution. Consistent read/write performance
Inverted index: used for document indexing. Used in Lucene
Suffix tree: for string pattern search
R-tree: multi-dimension search, such as finding the nearest neighbor

How is an SQL statement executed in the database?

The diagram below shows the process. Note that the architectures for different databases are different, the diagram demonstrates some common designs.

Step 1 - A SQL statement is sent to the database via a transport layer protocol (e.g.TCP).

Step 2 - The SQL statement is sent to the command parser, where it goes through syntactic and semantic analysis, and a query tree is generated afterward.

Step 3 - The query tree is sent to the optimizer. The optimizer creates an execution plan.

Step 4 - The execution plan is sent to the executor. The executor retrieves data from the execution.

Step 5 - Access methods provide the data fetching logic required for execution, retrieving data from the storage engine.

Step 6 - Access methods decide whether the SQL statement is read-only. If the query is read-only (SELECT statement), it is passed to the buffer manager for further processing. The buffer manager looks for the data in the cache or data files.

Step 7 - If the statement is an UPDATE or INSERT, it is passed to the transaction manager for further processing.

Step 8 - During a transaction, the data is in lock mode. This is guaranteed by the lock manager. It also ensures the transaction’s ACID properties.

CAP theorem

The CAP theorem is one of the most famous terms in computer science, but I bet different developers have different understandings. Let’s examine what it is and why it can be confusing.

CAP theorem states that a distributed system can't provide more than two of these three guarantees simultaneously.

Consistency: consistency means all clients see the same data at the same time no matter which node they connect to.

Availability: availability means any client that requests data gets a response even if some of the nodes are down.

Partition Tolerance: a partition indicates a communication break between two nodes. Partition tolerance means the system continues to operate despite network partitions.

The “2 of 3” formulation can be useful, but this simplification could be misleading.

Picking a database is not easy. Justifying our choice purely based on the CAP theorem is not enough. For example, companies don't choose Cassandra for chat applications simply because it is an AP system. There is a list of good characteristics that make Cassandra a desirable option for storing chat messages. We need to dig deeper.
“CAP prohibits only a tiny part of the design space: perfect availability and consistency in the presence of partitions, which are rare”. Quoted from the paper: CAP Twelve Years Later: How the “Rules” Have Changed.
The theorem is about 100% availability and consistency. A more realistic discussion would be the trade-offs between latency and consistency when there is no network partition. See PACELC theorem for more details.

Is the CAP theorem actually useful?

I think it is still useful as it opens our minds to a set of tradeoff discussions, but it is only part of the story. We need to dig deeper when picking the right database.

Types of Memory and Storage

Visualizing a SQL query

SQL statements are executed by the database system in several steps, including:

Parsing the SQL statement and checking its validity
Transforming the SQL into an internal representation, such as relational algebra
Optimizing the internal representation and creating an execution plan that utilizes index information
Executing the plan and returning the results

The execution of SQL is highly complex and involves many considerations, such as:

The use of indexes and caches
The order of table joins
Concurrency control
Transaction management

SQL language

In 1986, SQL (Structured Query Language) became a standard. Over the next 40 years, it became the dominant language for relational database management systems. Reading the latest standard (ANSI SQL 2016) can be time-consuming. How can I learn it?

There are 5 components of the SQL language:

DDL: data definition language, such as CREATE, ALTER, DROP
DQL: data query language, such as SELECT
DML: data manipulation language, such as INSERT, UPDATE, DELETE
DCL: data control language, such as GRANT, REVOKE
TCL: transaction control language, such as COMMIT, ROLLBACK

For a backend engineer, you may need to know most of it. As a data analyst, you may need to have a good understanding of DQL. Select the topics that are most relevant to you.

Cache

Data is cached everywhere

This diagram illustrates where we cache data in a typical architecture.

There are multiple layers along the flow.

Client apps: HTTP responses can be cached by the browser. We request data over HTTP for the first time, and it is returned with an expiry policy in the HTTP header; we request data again, and the client app tries to retrieve the data from the browser cache first.
CDN: CDN caches static web resources. The clients can retrieve data from a CDN node nearby.
Load Balancer: The load Balancer can cache resources as well.
Messaging infra: Message brokers store messages on disk first, and then consumers retrieve them at their own pace. Depending on the retention policy, the data is cached in Kafka clusters for a period of time.
Services: There are multiple layers of cache in a service. If the data is not cached in the CPU cache, the service will try to retrieve the data from memory. Sometimes the service has a second-level cache to store data on disk.
Distributed Cache: Distributed cache like Redis holds key-value pairs for multiple services in memory. It provides much better read/write performance than the database.
Full-text Search: we sometimes need to use full-text searches like Elastic Search for document search or log search. A copy of data is indexed in the search engine as well.
Database: Even in the database, we have different levels of caches:

WAL(Write-ahead Log): data is written to WAL first before building the B tree index
Bufferpool: A memory area allocated to cache query results
Materialized View: Pre-compute query results and store them in the database tables for better query performance
Transaction log: record all the transactions and database updates
Replication Log: used to record the replication state in a database cluster

Why is Redis so fast?

There are 3 main reasons as shown in the diagram below.

Redis is a RAM-based data store. RAM access is at least 1000 times faster than random disk access.
Redis leverages IO multiplexing and single-threaded execution loop for execution efficiency.
Redis leverages several efficient lower-level data structures.

Question: Another popular in-memory store is Memcached. Do you know the differences between Redis and Memcached?

You might have noticed the style of this diagram is different from my previous posts. Please let me know which one you prefer.

How can Redis be used?

There is more to Redis than just caching.

Redis can be used in a variety of scenarios as shown in the diagram.

Session

We can use Redis to share user session data among different services.
Cache

We can use Redis to cache objects or pages, especially for hotspot data.
Distributed lock

We can use a Redis string to acquire locks among distributed services.
Counter

We can count how many likes or how many reads for articles.
Rate limiter

We can apply a rate limiter for certain user IPs.
Global ID generator

We can use Redis Int for global ID.
Shopping cart

We can use Redis Hash to represent key-value pairs in a shopping cart.
Calculate user retention

We can use Bitmap to represent the user login daily and calculate user retention.
Message queue

We can use List for a message queue.
Ranking

We can use ZSet to sort the articles.

Top caching strategies

Designing large-scale systems usually requires careful consideration of caching. Below are five caching strategies that are frequently utilized.

Microservice architecture

What does a typical microservice architecture look like?

The diagram below shows a typical microservice architecture.

Load Balancer: This distributes incoming traffic across multiple backend services.
CDN (Content Delivery Network): CDN is a group of geographically distributed servers that hold static content for faster delivery. The clients look for content in CDN first, then progress to backend services.
API Gateway: This handles incoming requests and routes them to the relevant services. It talks to the identity provider and service discovery.
Identity Provider: This handles authentication and authorization for users.
Service Registry & Discovery: Microservice registration and discovery happen in this component, and the API gateway looks for relevant services in this component to talk to.
Management: This component is responsible for monitoring the services.
Microservices: Microservices are designed and deployed in different domains. Each domain has its own database. The API gateway talks to the microservices via REST API or other protocols, and the microservices within the same domain talk to each other using RPC (Remote Procedure Call).

Benefits of microservices:

They can be quickly designed, deployed, and horizontally scaled.
Each domain can be independently maintained by a dedicated team.
Business requirements can be customized in each domain and better supported, as a result.

Microservice Best Practices

A picture is worth a thousand words: 9 best practices for developing microservices.

When we develop microservices, we need to follow the following best practices:

Use separate data storage for each microservice
Keep code at a similar level of maturity
Separate build for each microservice
Assign each microservice with a single responsibility
Deploy into containers
Design stateless services
Adopt domain-driven design
Design micro frontend
Orchestrating microservices

What tech stack is commonly used for microservices?

Below you will find a diagram showing the microservice tech stack, both for the development phase and for production.

▶️ 𝐏𝐫𝐞-𝐏𝐫𝐨𝐝𝐮𝐜𝐭𝐢𝐨𝐧

Define API - This establishes a contract between frontend and backend. We can use Postman or OpenAPI for this.
Development - Node.js or react is popular for frontend development, and java/python/go for backend development. Also, we need to change the configurations in the API gateway according to API definitions.
Continuous Integration - JUnit and Jenkins for automated testing. The code is packaged into a Docker image and deployed as microservices.

▶️ 𝐏𝐫𝐨𝐝𝐮𝐜𝐭𝐢𝐨𝐧

NGinx is a common choice for load balancers. Cloudflare provides CDN (Content Delivery Network).
API Gateway - We can use spring boot for the gateway, and use Eureka/Zookeeper for service discovery.
The microservices are deployed on clouds. We have options among AWS, Microsoft Azure, or Google GCP. Cache and Full-text Search - Redis is a common choice for caching key-value pairs. Elasticsearch is used for full-text search.
Communications - For services to talk to each other, we can use messaging infra Kafka or RPC.
Persistence - We can use MySQL or PostgreSQL for a relational database, and Amazon S3 for object store. We can also use Cassandra for the wide-column store if necessary.
Management & Monitoring - To manage so many microservices, the common Ops tools include Prometheus, Elastic Stack, and Kubernetes.

Why is Kafka fast

There are many design decisions that contributed to Kafka’s performance. In this post, we’ll focus on two. We think these two carried the most weight.

The first one is Kafka’s reliance on Sequential I/O.
The second design choice that gives Kafka its performance advantage is its focus on efficiency: zero copy principle.

The diagram illustrates how the data is transmitted between producer and consumer, and what zero-copy means.

Step 1.1 - 1.3: Producer writes data to the disk
Step 2: Consumer reads data without zero-copy

2.1 The data is loaded from disk to OS cache

2.2 The data is copied from OS cache to Kafka application

2.3 Kafka application copies the data into the socket buffer

2.4 The data is copied from socket buffer to network card

2.5 The network card sends data out to the consumer

Step 3: Consumer reads data with zero-copy

3.1: The data is loaded from disk to OS cache 3.2 OS cache directly copies the data to the network card via sendfile() command 3.3 The network card sends data out to the consumer

Zero copy is a shortcut to save the multiple data copies between application context and kernel context.

Payment systems

How to learn payment systems?

Why is the credit card called “the most profitable product in banks”? How does VISA/Mastercard make money?

The diagram below shows the economics of the credit card payment flow.

1. The cardholder pays a merchant $100 to buy a product.

2. The merchant benefits from the use of the credit card with higher sales volume and needs to compensate the issuer and the card network for providing the payment service. The acquiring bank sets a fee with the merchant, called the “merchant discount fee.”

3 - 4. The acquiring bank keeps $0.25 as the acquiring markup, and $1.75 is paid to the issuing bank as the interchange fee. The merchant discount fee should cover the interchange fee.

The interchange fee is set by the card network because it is less efficient for each issuing bank to negotiate fees with each merchant.

5. The card network sets up the network assessments and fees with each bank, which pays the card network for its services every month. For example, VISA charges a 0.11% assessment, plus a $0.0195 usage fee, for every swipe.

6. The cardholder pays the issuing bank for its services.

Why should the issuing bank be compensated?

The issuer pays the merchant even if the cardholder fails to pay the issuer.
The issuer pays the merchant before the cardholder pays the issuer.
The issuer has other operating costs, including managing customer accounts, providing statements, fraud detection, risk management, clearing & settlement, etc.

How does VISA work when we swipe a credit card at a merchant’s shop?

VISA, Mastercard, and American Express act as card networks for the clearing and settling of funds. The card acquiring bank and the card issuing bank can be – and often are – different. If banks were to settle transactions one by one without an intermediary, each bank would have to settle the transactions with all the other banks. This is quite inefficient.

The diagram below shows VISA’s role in the credit card payment process. There are two flows involved. Authorization flow happens when the customer swipes the credit card. Capture and settlement flow happens when the merchant wants to get the money at the end of the day.

Authorization Flow

Step 0: The card issuing bank issues credit cards to its customers.

Step 1: The cardholder wants to buy a product and swipes the credit card at the Point of Sale (POS) terminal in the merchant’s shop.

Step 2: The POS terminal sends the transaction to the acquiring bank, which has provided the POS terminal.

Steps 3 and 4: The acquiring bank sends the transaction to the card network, also called the card scheme. The card network sends the transaction to the issuing bank for approval.

Steps 4.1, 4.2 and 4.3: The issuing bank freezes the money if the transaction is approved. The approval or rejection is sent back to the acquirer, as well as the POS terminal.

Capture and Settlement Flow

Steps 1 and 2: The merchant wants to collect the money at the end of the day, so they hit ”capture” on the POS terminal. The transactions are sent to the acquirer in batch. The acquirer sends the batch file with transactions to the card network.

Step 3: The card network performs clearing for the transactions collected from different acquirers, and sends the clearing files to different issuing banks.

Step 4: The issuing banks confirm the correctness of the clearing files, and transfer money to the relevant acquiring banks.

Step 5: The acquiring bank then transfers money to the merchant’s bank.

Step 4: The card network clears up the transactions from different acquiring banks. Clearing is a process in which mutual offset transactions are netted, so the number of total transactions is reduced.

In the process, the card network takes on the burden of talking to each bank and receives service fees in return.

Payment Systems Around The World Series (Part 1): Unified Payments Interface (UPI) in India

What’s UPI? UPI is an instant real-time payment system developed by the National Payments Corporation of India.

It accounts for 60% of digital retail transactions in India today.

UPI = payment markup language + standard for interoperable payments

DevOps

DevOps vs. SRE vs. Platform Engineering. What is the difference?

The concepts of DevOps, SRE, and Platform Engineering have emerged at different times and have been developed by various individuals and organizations.

DevOps as a concept was introduced in 2009 by Patrick Debois and Andrew Shafer at the Agile conference. They sought to bridge the gap between software development and operations by promoting a collaborative culture and shared responsibility for the entire software development lifecycle.

SRE, or Site Reliability Engineering, was pioneered by Google in the early 2000s to address operational challenges in managing large-scale, complex systems. Google developed SRE practices and tools, such as the Borg cluster management system and the Monarch monitoring system, to improve the reliability and efficiency of their services.

Platform Engineering is a more recent concept, building on the foundation of SRE engineering. The precise origins of Platform Engineering are less clear, but it is generally understood to be an extension of the DevOps and SRE practices, with a focus on delivering a comprehensive platform for product development that supports the entire business perspective.

It's worth noting that while these concepts emerged at different times. They are all related to the broader trend of improving collaboration, automation, and efficiency in software development and operations.

What is k8s (Kubernetes)?

K8s is a container orchestration system. It is used for container deployment and management. Its design is greatly impacted by Google’s internal system Borg.

A k8s cluster consists of a set of worker machines, called nodes, that run containerized applications. Every cluster has at least one worker node.

The worker node(s) host the Pods that are the components of the application workload. The control plane manages the worker nodes and the Pods in the cluster. In production environments, the control plane usually runs across multiple computers, and a cluster usually runs multiple nodes, providing fault tolerance and high availability.

Control Plane Components

API Server

The API server talks to all the components in the k8s cluster. All the operations on pods are executed by talking to the API server.
Scheduler

The scheduler watches pod workloads and assigns loads on newly created pods.
Controller Manager

The controller manager runs the controllers, including Node Controller, Job Controller, EndpointSlice Controller, and ServiceAccount Controller.
Etcd

etcd is a key-value store used as Kubernetes' backing store for all cluster data.

Nodes

Pods

A pod is a group of containers and is the smallest unit that k8s administers. Pods have a single IP address applied to every container within the pod.
Kubelet

An agent that runs on each node in the cluster. It ensures containers are running in a Pod.
Kube Proxy

Kube-proxy is a network proxy that runs on each node in your cluster. It routes traffic coming into a node from the service. It forwards requests for work to the correct containers.

Docker vs. Kubernetes. Which one should we use?

What is Docker ?

Docker is an open-source platform that allows you to package, distribute, and run applications in isolated containers. It focuses on containerization, providing lightweight environments that encapsulate applications and their dependencies.

What is Kubernetes ?

Kubernetes, often referred to as K8s, is an open-source container orchestration platform. It provides a framework for automating the deployment, scaling, and management of containerized applications across a cluster of nodes.

How are both different from each other ?

Docker: Docker operates at the individual container level on a single operating system host.

You must manually manage each host and setting up networks, security policies, and storage for multiple related containers can be complex.

Kubernetes: Kubernetes operates at the cluster level. It manages multiple containerized applications across multiple hosts, providing automation for tasks like load balancing, scaling, and ensuring the desired state of applications.

In short, Docker focuses on containerization and running containers on individual hosts, while Kubernetes specializes in managing and orchestrating containers at scale across a cluster of hosts.

How does Docker work?

The diagram below shows the architecture of Docker and how it works when we run “docker build”, “docker pull” and “docker run”.

There are 3 components in Docker architecture:

Docker client

The docker client talks to the Docker daemon.
Docker host

The Docker daemon listens for Docker API requests and manages Docker objects such as images, containers, networks, and volumes.
Docker registry

A Docker registry stores Docker images. Docker Hub is a public registry that anyone can use.

Let’s take the “docker run” command as an example.

Docker pulls the image from the registry.
Docker creates a new container.
Docker allocates a read-write filesystem to the container.
Docker creates a network interface to connect the container to the default network.
Docker starts the container.

GIT

How Git Commands work

To begin with, it's essential to identify where our code is stored. The common assumption is that there are only two locations - one on a remote server like Github and the other on our local machine. However, this isn't entirely accurate. Git maintains three local storages on our machine, which means that our code can be found in four places:

Working directory: where we edit files
Staging area: a temporary location where files are kept for the next commit
Local repository: contains the code that has been committed
Remote repository: the remote server that stores the code

Most Git commands primarily move files between these four locations.

How does Git Work?

The diagram below shows the Git workflow.

Git is a distributed version control system.

Every developer maintains a local copy of the main repository and edits and commits to the local copy.

The commit is very fast because the operation doesn’t interact with the remote repository.

If the remote repository crashes, the files can be recovered from the local repositories.

Git merge vs. Git rebase

What are the differences?

When we merge changes from one Git branch to another, we can use ‘git merge’ or ‘git rebase’. The diagram below shows how the two commands work.

Git merge

This creates a new commit G’ in the main branch. G’ ties the histories of both main and feature branches.

Git merge is non-destructive. Neither the main nor the feature branch is changed.

Git rebase

Git rebase moves the feature branch histories to the head of the main branch. It creates new commits E’, F’, and G’ for each commit in the feature branch.

The benefit of rebase is that it has a linear commit history.

Rebase can be dangerous if “the golden rule of git rebase” is not followed.

The Golden Rule of Git Rebase

Never use it on public branches!

Cloud Services

A nice cheat sheet of different cloud services (2023 edition)

What is cloud native?

Below is a diagram showing the evolution of architecture and processes since the 1980s.

Organizations can build and run scalable applications on public, private, and hybrid clouds using cloud native technologies.

This means the applications are designed to leverage cloud features, so they are resilient to load and easy to scale.

Cloud native includes 4 aspects:

Development process

This has progressed from waterfall to agile to DevOps.
Application Architecture

The architecture has gone from monolithic to microservices. Each service is designed to be small, adaptive to the limited resources in cloud containers.
Deployment & packaging

The applications used to be deployed on physical servers. Then around 2000, the applications that were not sensitive to latency were usually deployed on virtual servers. With cloud native applications, they are packaged into docker images and deployed in containers.
Application infrastructure

The applications are massively deployed on cloud infrastructure instead of self-hosted servers.

Developer productivity tools

Visualize JSON files

Nested JSON files are hard to read.

JsonCrack generates graph diagrams from JSON files and makes them easy to read.

Additionally, the generated diagrams can be downloaded as images.

Automatically turn code into architecture diagrams

What does it do?

Draw the cloud system architecture in Python code.
Diagrams can also be rendered directly inside the Jupyter Notebooks.
No design tools are needed.
Supports the following providers: AWS, Azure, GCP, Kubernetes, Alibaba Cloud, Oracle Cloud, etc.

Github repo

Linux

Linux file system explained

The Linux file system used to resemble an unorganized town where individuals constructed their houses wherever they pleased. However, in 1994, the Filesystem Hierarchy Standard (FHS) was introduced to bring order to the Linux file system.

By implementing a standard like the FHS, software can ensure a consistent layout across various Linux distributions. Nonetheless, not all Linux distributions strictly adhere to this standard. They often incorporate their own unique elements or cater to specific requirements. To become proficient in this standard, you can begin by exploring. Utilize commands such as "cd" for navigation and "ls" for listing directory contents. Imagine the file system as a tree, starting from the root (/). With time, it will become second nature to you, transforming you into a skilled Linux administrator.

18 Most-used Linux Commands You Should Know

Linux commands are instructions for interacting with the operating system. They help manage files, directories, system processes, and many other aspects of the system. You need to become familiar with these commands in order to navigate and maintain Linux-based systems efficiently and effectively.

This diagram below shows popular Linux commands:

ls - List files and directories
cd - Change the current directory
mkdir - Create a new directory
rm - Remove files or directories
cp - Copy files or directories
mv - Move or rename files or directories
chmod - Change file or directory permissions
grep - Search for a pattern in files
find - Search for files and directories
tar - manipulate tarball archive files
vi - Edit files using text editors
cat - display the content of files
top - Display processes and resource usage
ps - Display processes information
kill - Terminate a process by sending a signal
du - Estimate file space usage
ifconfig - Configure network interfaces
ping - Test network connectivity between hosts

Security

How does HTTPS work?

Hypertext Transfer Protocol Secure (HTTPS) is an extension of the Hypertext Transfer Protocol (HTTP.) HTTPS transmits encrypted data using Transport Layer Security (TLS.) If the data is hijacked online, all the hijacker gets is binary code.

How is the data encrypted and decrypted?

Step 1 - The client (browser) and the server establish a TCP connection.

Step 2 - The client sends a “client hello” to the server. The message contains a set of necessary encryption algorithms (cipher suites) and the latest TLS version it can support. The server responds with a “server hello” so the browser knows whether it can support the algorithms and TLS version.

The server then sends the SSL certificate to the client. The certificate contains the public key, host name, expiry dates, etc. The client validates the certificate.

Step 3 - After validating the SSL certificate, the client generates a session key and encrypts it using the public key. The server receives the encrypted session key and decrypts it with the private key.

Step 4 - Now that both the client and the server hold the same session key (symmetric encryption), the encrypted data is transmitted in a secure bi-directional channel.

Why does HTTPS switch to symmetric encryption during data transmission? There are two main reasons:

Security: The asymmetric encryption goes only one way. This means that if the server tries to send the encrypted data back to the client, anyone can decrypt the data using the public key.
Server resources: The asymmetric encryption adds quite a lot of mathematical overhead. It is not suitable for data transmissions in long sessions.

Oauth 2.0 Explained With Simple Terms.

OAuth 2.0 is a powerful and secure framework that allows different applications to securely interact with each other on behalf of users without sharing sensitive credentials.

The entities involved in OAuth are the User, the Server, and the Identity Provider (IDP).

What Can an OAuth Token Do?

When you use OAuth, you get an OAuth token that represents your identity and permissions. This token can do a few important things:

Single Sign-On (SSO): With an OAuth token, you can log into multiple services or apps using just one login, making life easier and safer.

Authorization Across Systems: The OAuth token allows you to share your authorization or access rights across various systems, so you don't have to log in separately everywhere.

Accessing User Profile: Apps with an OAuth token can access certain parts of your user profile that you allow, but they won't see everything.

Remember, OAuth 2.0 is all about keeping you and your data safe while making your online experiences seamless and hassle-free across different applications and services.

Top 4 Forms of Authentication Mechanisms

SSH Keys:

Cryptographic keys are used to access remote systems and servers securely
OAuth Tokens:

Tokens that provide limited access to user data on third-party applications
SSL Certificates:

Digital certificates ensure secure and encrypted communication between servers and clients
Credentials:

User authentication information is used to verify and grant access to various systems and services

Session, cookie, JWT, token, SSO, and OAuth 2.0 - what are they?

These terms are all related to user identity management. When you log into a website, you declare who you are (identification). Your identity is verified (authentication), and you are granted the necessary permissions (authorization). Many solutions have been proposed in the past, and the list keeps growing.

From simple to complex, here is my understanding of user identity management:

WWW-Authenticate is the most basic method. You are asked for the username and password by the browser. As a result of the inability to control the login life cycle, it is seldom used today.
A finer control over the login life cycle is session-cookie. The server maintains session storage, and the browser keeps the ID of the session. A cookie usually only works with browsers and is not mobile app friendly.
To address the compatibility issue, the token can be used. The client sends the token to the server, and the server validates the token. The downside is that the token needs to be encrypted and decrypted, which may be time-consuming.
JWT is a standard way of representing tokens. This information can be verified and trusted because it is digitally signed. Since JWT contains the signature, there is no need to save session information on the server side.
By using SSO (single sign-on), you can sign on only once and log in to multiple websites. It uses CAS (central authentication service) to maintain cross-site information.
By using OAuth 2.0, you can authorize one website to access your information on another website.

How to store passwords safely in the database and how to validate a password?

Things NOT to do

Storing passwords in plain text is not a good idea because anyone with internal access can see them.
Storing password hashes directly is not sufficient because it is pruned to precomputation attacks, such as rainbow tables.
To mitigate precomputation attacks, we salt the passwords.

What is salt?

According to OWASP guidelines, “a salt is a unique, randomly generated string that is added to each password as part of the hashing process”.

How to store a password and salt?

the hash result is unique to each password.
The password can be stored in the database using the following format: hash(password + salt).

How to validate a password?

To validate a password, it can go through the following process:

A client enters the password.
The system fetches the corresponding salt from the database.
The system appends the salt to the password and hashes it. Let’s call the hashed value H1.
The system compares H1 and H2, where H2 is the hash stored in the database. If they are the same, the password is valid.

Explaining JSON Web Token (JWT) to a 10 year old Kid

Imagine you have a special box called a JWT. Inside this box, there are three parts: a header, a payload, and a signature.

The header is like the label on the outside of the box. It tells us what type of box it is and how it's secured. It's usually written in a format called JSON, which is just a way to organize information using curly braces { } and colons : .

The payload is like the actual message or information you want to send. It could be your name, age, or any other data you want to share. It's also written in JSON format, so it's easy to understand and work with. Now, the signature is what makes the JWT secure. It's like a special seal that only the sender knows how to create. The signature is created using a secret code, kind of like a password. This signature ensures that nobody can tamper with the contents of the JWT without the sender knowing about it.

When you want to send the JWT to a server, you put the header, payload, and signature inside the box. Then you send it over to the server. The server can easily read the header and payload to understand who you are and what you want to do.

How does Google Authenticator (or other types of 2-factor authenticators) work?

Google Authenticator is commonly used for logging into our accounts when 2-factor authentication is enabled. How does it guarantee security?

Google Authenticator is a software-based authenticator that implements a two-step verification service. The diagram below provides detail.

There are two stages involved:

Stage 1 - The user enables Google two-step verification.
Stage 2 - The user uses the authenticator for logging in, etc.

Let’s look at these stages.

Stage 1

Steps 1 and 2: Bob opens the web page to enable two-step verification. The front end requests a secret key. The authentication service generates the secret key for Bob and stores it in the database.

Step 3: The authentication service returns a URI to the front end. The URI is composed of a key issuer, username, and secret key. The URI is displayed in the form of a QR code on the web page.

Step 4: Bob then uses Google Authenticator to scan the generated QR code. The secret key is stored in the authenticator.

Stage 2 Steps 1 and 2: Bob wants to log into a website with Google two-step verification. For this, he needs the password. Every 30 seconds, Google Authenticator generates a 6-digit password using TOTP (Time-based One Time Password) algorithm. Bob uses the password to enter the website.

Steps 3 and 4: The frontend sends the password Bob enters to the backend for authentication. The authentication service reads the secret key from the database and generates a 6-digit password using the same TOTP algorithm as the client.

Step 5: The authentication service compares the two passwords generated by the client and the server, and returns the comparison result to the frontend. Bob can proceed with the login process only if the two passwords match.

Is this authentication mechanism safe?

Can the secret key be obtained by others?

We need to make sure the secret key is transmitted using HTTPS. The authenticator client and the database store the secret key, and we need to make sure the secret keys are encrypted.
Can the 6-digit password be guessed by hackers?

No. The password has 6 digits, so the generated password has 1 million potential combinations. Plus, the password changes every 30 seconds. If hackers want to guess the password in 30 seconds, they need to enter 30,000 combinations per second.

Real World Case Studies

Netflix's Tech Stack

This post is based on research from many Netflix engineering blogs and open-source projects. If you come across any inaccuracies, please feel free to inform us.

Mobile and web: Netflix has adopted Swift and Kotlin to build native mobile apps. For its web application, it uses React.

Frontend/server communication: Netflix uses GraphQL.

Backend services: Netflix relies on ZUUL, Eureka, the Spring Boot framework, and other technologies.

Databases: Netflix utilizes EV cache, Cassandra, CockroachDB, and other databases.

Messaging/streaming: Netflix employs Apache Kafka and Fink for messaging and streaming purposes.

Video storage: Netflix uses S3 and Open Connect for video storage.

Data processing: Netflix utilizes Flink and Spark for data processing, which is then visualized using Tableau. Redshift is used for processing structured data warehouse information.

CI/CD: Netflix employs various tools such as JIRA, Confluence, PagerDuty, Jenkins, Gradle, Chaos Monkey, Spinnaker, Atlas, and more for CI/CD processes.

Twitter Architecture 2022

Yes, this is the real Twitter architecture. It is posted by Elon Musk and redrawn by us for better readability.

Evolution of Airbnb’s microservice architecture over the past 15 years

Airbnb’s microservice architecture went through 3 main stages.

Monolith (2008 - 2017)

Airbnb began as a simple marketplace for hosts and guests. This is built in a Ruby on Rails application - the monolith.

What’s the challenge?

Confusing team ownership + unowned code
Slow deployment

Microservices (2017 - 2020)

Microservice aims to solve those challenges. In the microservice architecture, key services include:

Data fetching service
Business logic data service
Write workflow service
UI aggregation service
Each service had one owning team

What’s the challenge?

Hundreds of services and dependencies were difficult for humans to manage.

Micro + macroservices (2020 - present)

This is what Airbnb is working on now. The micro and macroservice hybrid model focuses on the unification of APIs.

Monorepo vs. Microrepo.

Which is the best? Why do different companies choose different options?

Monorepo isn't new; Linux and Windows were both created using Monorepo. To improve scalability and build speed, Google developed its internal dedicated toolchain to scale it faster and strict coding quality standards to keep it consistent.

Amazon and Netflix are major ambassadors of the Microservice philosophy. This approach naturally separates the service code into separate repositories. It scales faster but can lead to governance pain points later on.

Within Monorepo, each service is a folder, and every folder has a BUILD config and OWNERS permission control. Every service member is responsible for their own folder.

On the other hand, in Microrepo, each service is responsible for its repository, with the build config and permissions typically set for the entire repository.

In Monorepo, dependencies are shared across the entire codebase regardless of your business, so when there's a version upgrade, every codebase upgrades their version.

In Microrepo, dependencies are controlled within each repository. Businesses choose when to upgrade their versions based on their own schedules.

Monorepo has a standard for check-ins. Google's code review process is famously known for setting a high bar, ensuring a coherent quality standard for Monorepo, regardless of the business.

Microrepo can either set its own standard or adopt a shared standard by incorporating the best practices. It can scale faster for business, but the code quality might be a bit different. Google engineers built Bazel, and Meta built Buck. There are other open-source tools available, including Nx, Lerna, and others.

Over the years, Microrepo has had more supported tools, including Maven and Gradle for Java, NPM for NodeJS, and CMake for C/C++, among others.

How will you design the Stack Overflow website?

If your answer is on-premise servers and monolith (on the bottom of the following image), you would likely fail the interview, but that's how it is built in reality!

What people think it should look like

The interviewer is probably expecting something like the top portion of the picture.

Microservice is used to decompose the system into small components.
Each service has its own database. Use cache heavily.
The service is sharded.
The services talk to each other asynchronously through message queues.
The service is implemented using Event Sourcing with CQRS.
Showing off knowledge in distributed systems such as eventual consistency, CAP theorem, etc.

What it actually is

Stack Overflow serves all the traffic with only 9 on-premise web servers, and it’s on monolith! It has its own servers and does not run on the cloud.

This is contrary to all our popular beliefs these days.

Why did Amazon Prime Video monitoring move from serverless to monolithic? How can it save 90% cost?

The diagram below shows the architecture comparison before and after the migration.

What is Amazon Prime Video Monitoring Service?

Prime Video service needs to monitor the quality of thousands of live streams. The monitoring tool automatically analyzes the streams in real time and identifies quality issues like block corruption, video freeze, and sync problems. This is an important process for customer satisfaction.

There are 3 steps: media converter, defect detector, and real-time notification.

What is the problem with the old architecture?

The old architecture was based on Amazon Lambda, which was good for building services quickly. However, it was not cost-effective when running the architecture at a high scale. The two most expensive operations are:

The orchestration workflow - AWS step functions charge users by state transitions and the orchestration performs multiple state transitions every second.
Data passing between distributed components - the intermediate data is stored in Amazon S3 so that the next stage can download. The download can be costly when the volume is high.

Monolithic architecture saves 90% cost

A monolithic architecture is designed to address the cost issues. There are still 3 components, but the media converter and defect detector are deployed in the same process, saving the cost of passing data over the network. Surprisingly, this approach to deployment architecture change led to 90% cost savings!

This is an interesting and unique case study because microservices have become a go-to and fashionable choice in the tech industry. It's good to see that we are having more discussions about evolving the architecture and having more honest discussions about its pros and cons. Decomposing components into distributed microservices comes with a cost.

What did Amazon leaders say about this?

Amazon CTO Werner Vogels: “Building evolvable software systems is a strategy, not a religion. And revisiting your architecture with an open mind is a must.”

Ex Amazon VP Sustainability Adrian Cockcroft: “The Prime Video team had followed a path I call Serverless First…I don’t advocate Serverless Only”.

How does Disney Hotstar capture 5 Billion Emojis during a tournament?

Clients send emojis through standard HTTP requests. You can think of Golang Service as a typical Web Server. Golang is chosen because it supports concurrency well. Threads in Golang are lightweight.
Since the write volume is very high, Kafka (message queue) is used as a buffer.
Emoji data are aggregated by a streaming processing service called Spark. It aggregates data every 2 seconds, which is configurable. There is a trade-off to be made based on the interval. A shorter interval means emojis are delivered to other clients faster but it also means more computing resources are needed.
Aggregated data is written to another Kafka.
The PubSub consumers pull aggregated emoji data from Kafka.
Emojis are delivered to other clients in real-time through the PubSub infrastructure. The PubSub infrastructure is interesting. Hotstar considered the following protocols: Socketio, NATS, MQTT, and gRPC, and settled with MQTT.

A similar design is adopted by LinkedIn which streams a million likes/sec.

How Discord Stores Trillions Of Messages

The diagram below shows the evolution of message storage at Discord:

MongoDB ➡️ Cassandra ➡️ ScyllaDB

In 2015, the first version of Discord was built on top of a single MongoDB replica. Around Nov 2015, MongoDB stored 100 million messages and the RAM couldn’t hold the data and index any longer. The latency became unpredictable. Message storage needs to be moved to another database. Cassandra was chosen.

In 2017, Discord had 12 Cassandra nodes and stored billions of messages.

At the beginning of 2022, it had 177 nodes with trillions of messages. At this point, latency was unpredictable, and maintenance operations became too expensive to run.

There are several reasons for the issue:

Cassandra uses the LSM tree for the internal data structure. The reads are more expensive than the writes. There can be many concurrent reads on a server with hundreds of users, resulting in hotspots.
Maintaining clusters, such as compacting SSTables, impacts performance.
Garbage collection pauses would cause significant latency spikes

ScyllaDB is Cassandra compatible database written in C++. Discord redesigned its architecture to have a monolithic API, a data service written in Rust, and ScyllaDB-based storage.

The p99 read latency in ScyllaDB is 15ms compared to 40-125ms in Cassandra. The p99 write latency is 5ms compared to 5-70ms in Cassandra.

How do video live streamings work on YouTube, TikTok live, or Twitch?

Live streaming differs from regular streaming because the video content is sent via the internet in real-time, usually with a latency of just a few seconds.

The diagram below explains what happens behind the scenes to make this possible.

Step 1: The raw video data is captured by a microphone and camera. The data is sent to the server side.

Step 2: The video data is compressed and encoded. For example, the compressing algorithm separates the background and other video elements. After compression, the video is encoded to standards such as H.264. The size of the video data is much smaller after this step.

Step 3: The encoded data is divided into smaller segments, usually seconds in length, so it takes much less time to download or stream.

Step 4: The segmented data is sent to the streaming server. The streaming server needs to support different devices and network conditions. This is called ‘Adaptive Bitrate Streaming.’ This means we need to produce multiple files at different bitrates in steps 2 and 3.

Step 5: The live streaming data is pushed to edge servers supported by CDN (Content Delivery Network.) Millions of viewers can watch the video from an edge server nearby. CDN significantly lowers data transmission latency.

Step 6: The viewers’ devices decode and decompress the video data and play the video in a video player.

Steps 7 and 8: If the video needs to be stored for replay, the encoded data is sent to a storage server, and viewers can request a replay from it later.

Standard protocols for live streaming include:

RTMP (Real-Time Messaging Protocol): This was originally developed by Macromedia to transmit data between a Flash player and a server. Now it is used for streaming video data over the internet. Note that video conferencing applications like Skype use RTC (Real-Time Communication) protocol for lower latency.
HLS (HTTP Live Streaming): It requires the H.264 or H.265 encoding. Apple devices accept only HLS format.
DASH (Dynamic Adaptive Streaming over HTTP): DASH does not support Apple devices.
Both HLS and DASH support adaptive bitrate streaming.

License

This work is licensed under

DevOps Labs : Guide to Setup Node.js 23, React, NestJS, Next.js, and Nginx on EC2 Ubuntu 24.04

Hoang Guruu — Thu, 20 Feb 2025 03:31:29 +0000

Guide to Setup Node.js 23, React, NestJS, Next.js, and Nginx on EC2 Ubuntu 24.04

Install Node.js 23
Setup React Application
Install and Configure Nginx for React
Setup NestJS and Next.js Applications
Configure Nginx for NestJS and Next.js
Start and Verify Applications
Troubleshooting

Install Node.js 23

Step 1: Update and Install Dependencies

sudo apt update && sudo apt upgrade -y
sudo apt install -y curl gnupg2 lsb-release ca-certificates

Step 2: Add NodeSource Repository

curl -fsSL https://deb.nodesource.com/setup_23.x | sudo -E bash -

Step 3: Install Node.js 23

sudo apt install -y nodejs

Step 4: Verify Installation

node -v
npm -v

Setup React Application

Step 1: Install `create-react-app` and Create a Project

npx create-react-app my-react-app
cd my-react-app
npm install

Step 2: Start React Application

npm start

Default React app runs on port 3000.

Install and Configure Nginx for React

Step 1: Install Nginx

sudo apt install -y nginx

Step 2: Enable and Start Nginx

sudo systemctl enable nginx
sudo systemctl start nginx

Step 3: Configure Nginx Proxy for React

sudo nano /etc/nginx/sites-available/default

Replace with the following configuration:

server {
    listen 80;

    server_name your_domain_or_IP;

    location / {
        proxy_pass http://localhost:3000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
}

Step 4: Test and Restart Nginx

sudo nginx -t
sudo systemctl restart nginx

Setup NestJS and Next.js Applications

Step 1: Install NestJS CLI and Create a New Project

npm install -g @nestjs/cli
nest new my-nest-app
cd my-nest-app
npm install

Step 2: Change NestJS Port to 4000

Edit src/main.ts and modify:

await app.listen(4000);

Step 3: Start NestJS Server

npm run start

Step 4: Setup Next.js Application

npx create-next-app@latest my-next-app
cd my-next-app
npm install

Step 5: Change Next.js Port to 3001

Modify package.json scripts:

"scripts": {
  "dev": "next dev -p 3001"
}

Step 6: Start Next.js Application

npm run dev

Configure Nginx for NestJS and Next.js

Step 1: Edit Nginx Configuration

sudo nano /etc/nginx/sites-available/default

Replace with:

server {
    listen 80;
    server_name your_domain_or_IP;

    location /react {
        proxy_pass http://localhost:3000;
    }

    location /nestjs {
        proxy_pass http://localhost:4000;
    }

    location /next {
        proxy_pass http://localhost:3001;
    }
}

Step 2: Test and Restart Nginx

sudo nginx -t
sudo systemctl restart nginx

Start and Verify Applications

Step 1: Start Applications

cd ~/my-react-app && npm start &
cd ~/my-nest-app && npm run start &
cd ~/my-next-app && npm run dev &

Step 2: Access Applications via Nginx

React App: http://your_domain_or_IP/react
NestJS App: http://your_domain_or_IP/nestjs
Next.js App: http://your_domain_or_IP/next

Troubleshooting

1. Nginx Not Starting?

Check logs:

sudo journalctl -xe
sudo systemctl status nginx

2. React/NestJS/Next.js Not Running?

Check logs:

pm2 logs

Or restart manually:

node app.js 2>&1 | tee error.log

Conclusion

You have successfully:

Installed Node.js 23.
Set up React, NestJS, and Next.js applications.
Configured Nginx as a reverse proxy.

If you face any issues, check logs and verify configurations.

🚀 Happy coding! 🚀

DevOps CLI Linux Like a Pro!

Hoang Guruu — Wed, 19 Feb 2025 09:59:13 +0000