DEV Community: suresh devops

From DevSecOps to Operational Resilience: 26 Engineering Improvements Across Cloud, Kubernetes and ML Infrastructure

suresh devops — Wed, 21 Oct 2026 03:48:54 +0000

*From Security Scanning to Operational Resilience: Engineering Improvements Across DevSecOps, Cloud and ML Infrastructure *

Lessons from integrating security into CI/CD, strengthening observability, automating recovery, modernising API gateways and improving production operations.

Modern platform engineering is not just about deploying applications faster. It is about building secure, reliable, recoverable and maintainable systems while reducing manual operational effort.

In a multi-cloud environment supporting pharmaceutical SaaS applications, API platforms, Kubernetes clusters and machine learning services, operational excellence requires a combination of DevSecOps, observability, automation, capacity management and disciplined change control.

Over time, I have worked on a series of engineering improvements across CI/CD pipelines, on-premises infrastructure, Azure Kubernetes Service (AKS), Oracle Cloud Infrastructure (OCI), API gateways, databases, monitoring platforms and ML workloads.

These improvements share a common objective: shift operational practices from reactive troubleshooting towards proactive controls, repeatable automation and evidence-based engineering decisions.

This article explores the key improvements, the engineering considerations behind them and the operational value they bring.

1. Integrating Gitleaks with CI/CD: Bringing Secret Detection into the Development Workflow

Security vulnerabilities can enter a production environment long before deployment. Hard-coded credentials, API tokens, passwords and private keys committed to source control are examples of risks that should be detected as early as possible.

To address this, I integrated Gitleaks into the CI/CD workflow to identify potential secrets in source code and related repository content.

Why this matters

When credentials are committed to a repository, the risk extends beyond the immediate application. Credentials may be copied into build logs, cached artifacts, cloned repositories or downstream environments. Removing the secret from the latest commit does not necessarily remove it from the repository's history or eliminate the need to rotate it.

Engineering approach

The integration introduces secret scanning into the software delivery process, allowing findings to be identified closer to the point where code changes are introduced.

A mature implementation should include:

Scanning code changes and, where appropriate, repository history.
Reviewing findings to distinguish genuine secrets from false positives.
Assigning ownership for remediation.
Rotating exposed credentials rather than relying solely on deleting them from source code.
Defining severity-based handling and exception procedures.
Retaining scan results and remediation evidence according to policy.

The next level of maturity is to enforce pipeline failure for confirmed policy violations, subject to an agreed exception process. Detection and build-blocking enforcement are separate controls and should be implemented and verified independently.

Operational value: Earlier detection of credential exposure, improved security awareness in development workflows and a foundation for policy-driven DevSecOps.

2. Strengthening Security Visibility with Wazuh Across On-Premises Servers

Security controls are more effective when infrastructure events can be observed centrally.

I worked on installing Wazuh across on-premises servers to improve security monitoring and centralise visibility into relevant host events.

Key considerations

A successful deployment involves more than installing an agent. It requires attention to:

Agent connectivity and health.
Collection of relevant security events.
Alert rules and severity classification.
Investigation and escalation procedures.
Ownership of findings and remediation.
Monitoring of agent coverage and reporting gaps.

The operational objective is to make security events visible to the appropriate teams and provide a consistent basis for investigation.

A useful follow-up measure is coverage: the number of in-scope servers reporting successfully compared with the total number of servers that should be monitored. Additional measures can include alert investigation time and the age of unresolved high-priority findings.

Operational value: Improved security visibility across the on-premises server estate and a stronger basis for operational risk monitoring.

3. Evaluating Root-Access Changes Through a Proof of Concept

Privileged access is one of the most sensitive areas of infrastructure administration. Excessive or poorly controlled root access can increase the impact of accidental changes, compromised credentials and unauthorised activity.

I worked on a proof of concept for changing root-access practices, with broader rollout identified as a subsequent step.

A controlled approach to privileged access should consider:

Whether administrative access can be performed through named accounts and approved privilege elevation.
Authentication and authorisation requirements.
Audit logging for privileged commands and sessions.
Emergency access procedures.
Service dependencies that may require special treatment.
A tested rollback and recovery approach.

A proof of concept should validate compatibility with existing administration procedures before wider deployment. Production rollout should include a staged migration, clear ownership, exception handling and a method to recover access if the new controls fail.

Operational value: A structured path towards stronger privileged-access governance without introducing avoidable operational disruption.

4. Upgrading Trivy to Improve Vulnerability-Scanning Currency

Container and infrastructure security scanning must evolve alongside the tools, libraries and vulnerabilities they are intended to detect.

I upgraded Trivy from version 0.58.1 to 0.64.1 as part of maintaining the security tooling used in the engineering environment.

Tool upgrades are an important operational activity, but the upgrade alone does not establish that vulnerabilities have been eliminated.

A sound upgrade process should include:

Reviewing release notes and compatibility considerations.
Testing the new version against representative images and repositories.
Validating vulnerability database access and scan output.
Comparing findings and investigating material differences.
Confirming that CI/CD integrations continue to operate correctly.
Documenting the deployed version and the upgrade outcome.

The broader objective is to ensure that scanning remains a dependable part of the delivery process and that findings can be prioritised and remediated.

Operational value: Updated vulnerability-scanning tooling and a stronger foundation for maintaining consistent container security checks.

5. Resolving Certificate Trust Issues Between On-Premises Containers and OCI OpenSearch

Secure communication between distributed systems depends on correct certificate trust configuration.

An SSL connectivity issue was encountered when on-premises containers communicated with OCI OpenSearch. The issue was resolved by adding the required CA certificate to the operating system on the OCI server.

This illustrates an important troubleshooting principle: TLS failures are not always caused by application code or network connectivity. They can arise from certificate chains, trust stores, hostname verification, certificate expiry or configuration differences between environments.

A systematic investigation should validate:

The certificate presented by the remote endpoint.
The certificate chain and relevant CA certificates.
The trust store used by the client process.
Hostname and certificate validity checks.
Certificate expiry and renewal procedures.
Connectivity and TLS negotiation using approved diagnostic tools.

The durable fix should also be documented and incorporated into configuration management where appropriate, rather than relying on manual changes that may be lost during rebuilds.

Operational value: Restored secure connectivity and a clearer understanding of certificate trust dependencies between environments.

6. Improving Cloud Hygiene by Removing Unused Route 53 Records

Cloud resources and DNS records can remain after services are migrated, retired or replaced. Unused records create unnecessary configuration complexity and may introduce security, ownership or troubleshooting risks.

I removed unused Route 53 URLs after identifying records that were no longer required.

DNS cleanup should be performed carefully because a record that appears unused may still support a legacy integration, scheduled job, external client or certificate-validation workflow.

A controlled cleanup process should include:

Identifying the record owner and associated service.
Reviewing traffic and dependency evidence where available.
Confirming that the target service has been decommissioned or migrated.
Checking certificate, redirect and integration dependencies.
Recording the change and retaining a recovery path where appropriate.
Validating service behaviour after removal.

Operational value: Reduced DNS clutter and improved configuration maintainability, provided that dependency checks and post-change validation are completed.

7. API Gateway Modernisation: WSO2, Kong, Azure API Management and OCI API Gateway

API gateways are critical infrastructure components because they sit on the path between clients and backend services. Their configuration, availability and migration strategy can directly affect application connectivity.

My API gateway improvement work included WSO2-to-Kong migration activities, Kong failover-related implementation, API migration from Azure API Management to OCI API Gateway and work involving migration of services to Standard API Manager from Basic SKU.

Addressing OCI API Gateway service limits

During the migration to OCI API Gateway, a service configuration constraint involving the limit of 50 APIs per service required attention. The engineering team reduced the number of APIs configured per service to remain below that limit.

This is an example of adapting API architecture to the constraints of the target platform instead of treating migration as a direct copy of the source configuration.

Resolving Jasper API compatibility issues

Jasper APIs that were invoked from Azure did not function as expected after the move to OCI API Gateway. OCI Functions were used to address the integration issue.

The broader engineering lesson is that API migration requires functional validation, not just successful deployment. Differences in request handling, authentication, integration behaviour, routing and platform capabilities can affect API compatibility.

Reliability and failover

Kong failover implementation and related proof-of-concept activities were also part of the improvement work.

A robust failover implementation should verify:

Health detection and failure thresholds.
Routing to the correct healthy endpoint.
Connection and session behaviour.
Authentication, rate limiting and TLS configuration.
Recovery when the original node becomes healthy again.
Logging, alerting and operational ownership.

A failover design should be tested under controlled failure scenarios before it is treated as production-ready.

Operational value: Improved migration readiness, more deliberate gateway architecture and a structured approach to API availability and failover.

8. Automating Kong API Manager Administration

Manual API management can become difficult to maintain as the number of services, configurations and environments grows.

I developed an automated script for managing Kong API Manager activities.

Automation is especially useful for repeatable administrative tasks when it reduces configuration inconsistency and improves execution speed. However, API-management automation must also protect against accidental changes to routes, credentials, upstream services or access policies.

A reliable automation workflow should include:

Validated inputs and environment selection.
Authentication using appropriately protected credentials.
Pre-change checks and configuration validation.
Logging of actions and results.
Error handling and safe failure behaviour.
Change traceability and controlled permissions.
Post-change verification.

Operational value: A foundation for more repeatable API administration and reduced reliance on manual configuration steps.

9. Building a More Reliable Backup and Recovery Practice

Backups are only useful when they are available, consistent and recoverable when required.

I implemented automated backup processes across several infrastructure and data platforms, including:

Zabbix
Kong
Grafana
Prometheus
Weaviate
Dgraph
Legal-Elastic
Redis
PostgreSQL
MLflow

Additional work included automated PostgreSQL database backups and the use of an NFS server on separate box to receive backups from the five-node Elasticsearch cluster on Master Node.

The objective was to make backup execution more repeatable and reduce dependence on manual intervention.

Backup automation is only the first step

A production-ready backup process should also monitor completion, validate backup integrity, control access, manage retention and verify that the recovery process works.

Key questions include:

Did the backup complete?
Does the backup contain the expected data?
Is it protected from accidental or unauthorised deletion?
Can it be restored independently of the production failure?
Can the service be recovered within its agreed RTO?
Is the amount of data lost within the agreed RPO?

These checks are essential because a successful backup job is not proof of recoverability.

Operational value: More repeatable backup execution across a diverse technology stack and a foundation for better recovery readiness.

10. Restoring Legal-Elastic and Weaviate Data

Backup processes become more meaningful when recovery is demonstrated.

As part of the operational work, I restored Legal-Elastic on Backup Server and restored Weaviate data on Master node.

These activities provided practical experience with data restoration and the operational dependencies involved in bringing data services back into service.

A restoration procedure should validate more than whether a process starts. Depending on the platform, verification may include:

Database or index health.
Expected collections, indices or records.
Data consistency and application connectivity.
Authentication and access controls.
Application-level queries.
Evidence that the restored service is fit for its intended use.

The next maturity step is to formalise recurring restore tests, record the duration and results, and compare them against recovery objectives.

Operational value: Demonstrated restoration capability for the specific systems involved and a basis for improving recovery procedures.

11. Recovering a 2 TB Disk Snapshot for a UAT Environment

A disk recovery activity involved using a 2 TB disk snapshot to create and mount a disk on the UAT virtual machine on Azure. The restored data was checked with Ayan, after which the original 2 TB disk was depleted and decommissioned.

This activity highlights the importance of managing the complete lifecycle of recovered storage—not just creating the disk.

A controlled process should confirm:

The correct snapshot and recovery point were selected.
The restored disk was attached and mounted safely.
The expected data was available and validated by the relevant owner.
The UAT environment could access the required data.
The original disk was no longer required before decommissioning.
The storage change and decommissioning were documented.

The principle is simple: validate the recovery target and data before retiring the original resource.

Operational value: Recovered storage and validated data availability for the UAT environment, followed by the removal of the original disk after confirmation.

12. Expanding Elasticsearch Cluster Capacity

I added Node 4 and Node 5 as Elasticsearch nodes to an existing cluster having 3 nodes.

Adding nodes can increase available capacity and provide more flexibility in distributing data and workload. However, node addition must be accompanied by cluster-health and capacity validation.

Important checks include:

Cluster health before and after the change.
Node discovery and membership.
Shard allocation and rebalancing.
Disk utilisation and storage watermarks.
Heap utilisation, CPU and query latency.
Replication and recovery activity.
Any application impact during rebalancing.

Operational value: Expanded the cluster's node footprint, creating additional capacity potential. The actual performance or capacity improvement should be established from before-and-after metrics.

13. Resource Optimisation Through Pod and Docker Profiling

Infrastructure capacity decisions should be based on observed workloads rather than assumptions.

I carried out proof-of-concept work for pod profiling and then profiled pods across Development, QA, Sales and UAT Kubernetes clusters. I also performed Docker profiling for ML services.

Profiling helps identify:

CPU and memory utilisation patterns.
Workloads with excessive or insufficient resource requests.
Potential memory leaks or unexpected growth.
Containers that frequently restart or encounter OOM termination.
Opportunities for resource right-sizing.
Workloads that may benefit from isolation or separate scaling policies.

This is particularly important for ML workloads, where resource demand can vary significantly between model types and execution patterns.

Profiling should cover normal usage as well as peak periods. Reducing resource allocations based only on average usage can introduce instability during traffic spikes or concurrent workloads.

Operational value: Better evidence for capacity planning and resource optimisation across Kubernetes and containerised ML services.

14. ML Service Rationalisation and Ollama Configuration

I worked on ML service deprecation activities, Ollama configuration changes for ML services and Ollama model availability on GPU hosts.

These activities relate to managing an evolving ML infrastructure footprint and aligning deployed services and models with operational requirements.

For service deprecation, a controlled process should verify service ownership, consumer dependencies, traffic usage, replacement availability and rollback requirements before a service is retired.

For model and GPU-host configuration, important operational considerations include:

GPU and memory utilisation.
Model compatibility and loading behaviour.
Concurrent request capacity.
Model-serving latency and error rates.
Process health and recovery.
Version consistency and configuration management.

The objective is to maintain a service footprint that is operationally supportable while ensuring that active workloads continue to meet their functional requirements.

Operational value: Improved management of ML service and model configurations. Quantified resource savings or inference-performance improvements should be reported only after measurement.

15. Automating ML Service Deployment

I completed proof-of-concept work for ML service deployment automation, with broader automation across all ML services identified as a next step.

ML deployment automation can reduce manual configuration, improve repeatability and help maintain consistency across multiple environments.

A mature workflow should cover:

Versioned deployment configuration.
Validation of required secrets and environment variables.
Image and dependency checks.
Resource configuration and health probes.
Deployment execution with appropriate permissions.
Post-deployment smoke tests and service validation.
Failure handling and rollback.
Audit-friendly records of what was deployed and when.

The important distinction is between a working proof of concept and a fully operationalised deployment framework. Broader rollout should include testing, documentation, ownership and support procedures.

Operational value: A foundation for consistent ML deployment and a path towards reducing manual deployment effort.

16. Operational Resilience Through Swap Configuration

Swap configuration was standardised based on available host memory:

Hosts with 8 GB RAM: 4 GB swap.
Hosts with 16 GB RAM: 8 GB swap.
Hosts with 32 GB RAM: 8 GB swap.

Swap can provide a buffer during memory pressure, but it is not a replacement for adequate physical memory. Excessive swapping can also degrade application performance, especially for latency-sensitive services.

A production implementation should monitor swap usage, memory pressure, OOM events and application response times. It should also consider workload characteristics and the performance implications of swapping.

Operational value: More consistent swap configuration across the specified host sizes, with the actual reliability benefit dependent on workload behaviour and monitoring evidence.

17. Safer Service Isolation and Environment Segregation

I worked on separating the vigilance data service by deploying vigilance-qa in the QA cluster and vigilance-prod in the Prod cluster. I also worked on service-ner environment segregation.

Separating services by environment can improve operational isolation, reduce the risk of accidental cross-environment interaction and make deployment ownership clearer.

However, segregation requires more than separate deployment names. It should also consider:

Namespace and cluster boundaries.
Service accounts and permissions.
Network access.
Environment-specific secrets and configuration.
Database and external-service connectivity.
Monitoring, logging and alert routing.
Deployment and rollback procedures.

Operational value: Clearer environment separation and a more controlled foundation for managing service configurations across QA and Sales.

18. Improving Kubernetes Deployment Safety with Health Probes and Strategies

I updated deployment.yaml configurations with liveness probes, readiness probes and deployment strategies for backend services across Development, QA and Prod environments.

These controls address different aspects of application availability.

Readiness probes help prevent traffic from reaching pods that are not ready to serve requests.
Liveness probes help detect applications that need to be restarted.
Deployment strategies control how application versions are rolled out and how old and new instances coexist during a change.

Correct configuration is critical. A liveness probe that is too aggressive can cause restart loops, while an inadequate readiness probe can expose users to an application that has started but is not ready.

Deployment validation should cover normal startup, slow startup, dependency failures, rolling updates and rollback scenarios.

Operational value: Stronger deployment-level health controls and a more structured approach to managing backend service updates.

19. Liquibase CI/CD for Database Changes

Database changes require a controlled approach because schema modifications can affect application compatibility, data integrity and recovery.

I worked on Liquibase CI/CD for database delivery.

A well-designed database deployment pipeline should include:

Version-controlled migration scripts.
Review and approval of changes.
Validation against the target database.
Controlled execution in the appropriate environment.
Compatibility checks between application and schema versions.
Verification after migration.
A documented recovery or forward-fix strategy.

Database rollback requires special consideration. Not every schema or data transformation can be safely reversed, so the recovery strategy must be designed for the specific change.

Operational value: A foundation for repeatable, traceable database changes and closer integration between application and database delivery processes.

20. Jenkins Upgrade and Deployment Hygiene

I upgraded Jenkins from version 2.479.2 to 2.504.3.

CI/CD infrastructure is itself a critical production dependency. An upgrade should therefore include plugin compatibility checks, configuration and job validation, backup verification, access-control review and post-upgrade testing.

I also updated deployment scripts to automatically delete TAR files from 52 hosts after deployment, and backed up deployment scripts from those hosts to Bitbucket.

These activities address two operational concerns: managing deployment artefacts after use and retaining version-controlled copies of operational scripts.

A strong implementation should verify that:

The artefact is no longer needed by the running application or rollback process.
Cleanup only targets the intended files and directories.
Script failures are logged and surfaced.
The deployment outcome is not reported as successful when required cleanup or validation fails.
The version-controlled script is accessible to authorised maintainers.

Operational value: More consistent deployment cleanup and improved version-control coverage for deployment scripts.

21. Standardising the Linux Operating-System Baseline

I worked on upgrading Ubuntu from version 20.04 to 22.04.

Operating-system upgrades require a balance between security maintenance, application compatibility and service availability. A safe upgrade process should include dependency assessment, testing, backup validation, change approval, rollback planning and post-upgrade health checks.

For production systems, I would also verify monitoring-agent compatibility, service startup behaviour, network configuration, scheduled jobs and application dependencies.

Operational value: Progress towards a more current operating-system baseline for the systems included in the upgrade.

22. Managing ML Automation with Locking Controls

I worked on a lock script for ML automation.

Concurrent automation can create conflicts when multiple jobs attempt to modify the same resource, deploy the same service or run an operation that should only have one active execution.

Locking controls can help prevent duplicate execution, but they must handle stale locks, abnormal termination, ownership and safe release.

A production-ready implementation should define:

What resource is protected.
How lock ownership is established.
How stale locks are detected and recovered.
How concurrent execution attempts are handled.
What happens when a job fails midway.
How execution is logged and monitored.

Operational value: A mechanism for controlling concurrent ML automation and reducing the risk of conflicting operations, subject to the lock implementation and its failure handling.

23. Infrastructure Connectivity and Network Resilience

Network availability is a fundamental dependency for monitoring, application connectivity and operational support.

I changed the primary ISP from ACT to TATA after the ACT Fair Usage Policy threshold was exceeded and firewall bandwidth dropped to 3 Mbps.

The change addressed an operational connectivity constraint. It also illustrates why network-provider performance should be evaluated against actual business requirements rather than assumed service capacity.

Useful follow-up measurements include:

Available bandwidth and utilisation.
Packet loss and latency.
Connection stability.
VPN and remote-access performance.
Incident frequency and duration.
Failover capability and recovery time.

Where possible, network changes should be validated against representative workloads and documented alongside the associated firewall and routing configuration.

Operational value: A change in primary ISP to address a bandwidth limitation affecting operational connectivity.

24. Observability with OpenTelemetry

I worked on introducing an OpenTelemetry pod in AKS.

OpenTelemetry provides a common approach for collecting telemetry such as traces, metrics and logs through supported instrumentation and collectors. Its value comes from making it easier to correlate application behaviour across distributed components.

For a production rollout, I would validate:

Which telemetry signals are collected.
Whether the instrumentation covers the required services.
Exporter and collector configuration.
Access controls and protection of sensitive telemetry.
Sampling, retention and data-volume implications.
Collector health and telemetry delivery.
Integration with the organisation's observability platform.

Operational value: A foundation for more consistent telemetry collection and cross-service troubleshooting, with the actual observability improvement depending on instrumentation coverage and integration.

25. Resolving an Infrastructure Capacity Constraint in the Network Path

The network-provider change from ACT to TATA also illustrates a broader operational principle: infrastructure capacity issues can originate outside the application or server layer.

When firewall throughput falls, application symptoms can resemble a backend problem—slow requests, timeouts, unstable remote sessions or delayed monitoring data.

A layered investigation should therefore examine the full service path, including the local network, firewall, ISP, VPN, routing and remote endpoint.

Operational value: A practical reminder to correlate application symptoms with network performance before making unnecessary changes to application infrastructure.

26. Bringing the Improvements Together

The individual changes become more valuable when managed as part of a coherent operational improvement programme.

I group this work into six engineering themes:

Security and control

Gitleaks integration with CI/CD.
Wazuh deployment across on-premises servers.
Trivy upgrade.
Root-access change proof of concept.
Certificate trust remediation.
DNS hygiene.

Reliability and recovery

Automated backups across multiple platforms.
Legal-Elastic and Weaviate restoration.
Disk snapshot recovery for UAT.
Elasticsearch cluster expansion.
Kong failover-related implementation.
Kubernetes health probes and deployment strategies.

Automation and engineering consistency

Kong API Manager automation.
Liquibase CI/CD.
Jenkins upgrade.
ML deployment automation proof of concept.
ML automation locking.
Deployment-script cleanup and version control.

Cloud and API modernisation

WSO2-to-Kong migration work.
Azure API Management to OCI API Gateway migration.
OCI API configuration-limit remediation.
Jasper API integration remediation using OCI Functions.
API migration work involving Standard API Manager.

Capacity and platform optimisation

Pod and Docker profiling.
ML service rationalisation.
Ollama configuration and GPU model management.
Swap configuration.
Elasticsearch node expansion.

Operational hygiene and environment management

Linux OS upgrades.
Service segregation across environments.
Removal of unused DNS records.
Network-provider change to address bandwidth constraints.
OpenTelemetry deployment work.

27. Measuring Operational Improvement

Engineering changes should ultimately be evaluated using evidence. For this programme, I would track a focused set of measures.

Improvement area	Useful measure
Secret scanning	Repositories covered, findings triaged and remediation time
Security monitoring	Server coverage, agent health and high-priority finding age
Vulnerability scanning	Scan coverage, finding severity and remediation status
Backup and recovery	Backup success, restore-test success, RPO and RTO
API gateway reliability	Availability, request errors, latency and failover test results
Kubernetes operations	Pod restarts, OOM events, deployment failures and readiness failures
Capacity optimisation	Resource utilisation, headroom and cost per workload
Automation	Manual effort avoided, automation failures and repeat-ticket reduction
Database delivery	Migration success, rollback or forward-fix events and post-deployment validation
Network performance	Latency, packet loss, throughput and connectivity incidents

I would establish a baseline before claiming a percentage improvement. Where measurements are unavailable, the correct approach is to describe the engineering change and its intended benefit without inventing an outcome.

Conclusion: Engineering Operations as a Continuous Improvement Discipline

These improvements demonstrate how DevSecOps, observability, automation, infrastructure management and recovery planning work together.

Security scanning helps identify risks earlier. Monitoring makes operational behaviour visible. Backup and restoration processes support recovery. Kubernetes health controls improve deployment management. API gateway and infrastructure changes address platform constraints. Automation reduces repetitive work, while capacity profiling provides evidence for optimisation.

The next level of maturity is to connect these activities to measurable operational outcomes: fewer recurring incidents, faster and safer recovery, more reliable deployments, stronger security-control coverage and better use of infrastructure resources.

For me, the central principle is straightforward:

Operational excellence is not the absence of incidents. It is the ability to detect problems early, recover safely, learn systematically and engineer the conditions that make the next incident less likely.

That is the direction I continue to pursue across DevSecOps, cloud platforms, API infrastructure and ML operations.

No-Code Platform Deployment : Surprising Lessons from Appsmith on Azure

suresh devops — Wed, 30 Sep 2026 05:00:19 +0000

*1. Introduction: The Complexity Behind the "No-Code" Curtain *

No-code platforms are built on a compelling architectural paradox: they deliver an abstract, frictionless experience for the end-user while demanding a highly sophisticated, high-performance infrastructure under the hood. While spinning up a local development environment is a matter of a few Docker commands, re-architecting for cloud-native resilience on Microsoft Azure is a different beast entirely.

Taking a tool like Appsmith from a local machine to a commercial-grade SaaS requires moving beyond simple containerization. It demands a transition into a world of managed services, multi-tenant security models, and global traffic orchestration. This guide reveals the architectural blueprints necessary to bridge that gap, transforming a local build into an enterprise-ready Azure deployment.

2. The Invisible Engine: Why the RTS Server is Non-Negotiable

The Real-Time Server (RTS) is the "heartbeat" of the Appsmith stack, facilitating the reactive nature of the platform. In a local testing environment, this typically involves pinning the environment to Node version 20.11.1 and executing ./start-server.sh within the packages directory. When moving to Azure, the RTS becomes the primary networking pivot point.

"Appsmith’s architecture relies heavily on specific networking constraints (like the RTS server communicating with the main backend)..."

In a production Azure environment, the RTS cannot be an afterthought. Whether deployed in Azure Container Apps (ACA) or Azure Kubernetes Service (AKS), you must ensure low-latency, persistent connectivity between the RTS and the main Java/Maven backend. If this networking link is brittle, the real-time synchronization that users expect from a modern no-code IDE will degrade, leading to state inconsistencies.

3. The MongoDB "Replica Set" Requirement: More Than Just a Database

A common pitfall in Appsmith deployments is treating the database as a standard standalone instance. Appsmith requires MongoDB to be initialized with a replica set to handle its internal event-driven architecture. Locally, this is achieved by starting the container with the --replSet rs0 flag and executing the mandatory initialization:

rs.initiate({"_id": "rs0", "members": [{"_id": 0, "host": "localhost:27017"}]})

In a professional Azure deployment, we shift this operational burden to Azure Cosmos DB for MongoDB. The transformative value of this managed service is that it handles replica set requirements and high availability "under the hood," removing the manual overhead of managing mongosh commands. To complement this, Azure Cache for Redis should be integrated as the session and caching layer. For cost-efficiency in multi-tenant builds, architects can utilize a single Redis tier and segment data using unique prefix keys, providing isolation without the price tag of multiple dedicated instances.

4. The "Scale-to-Zero" Secret: Why Azure Container Apps (ACA) Wins for SaaS

For platforms targeting product-led growth (PLG), the infrastructure must be as elastic as the user base. Azure Container Apps (ACA) is our recommended path because it enables a "Scale-to-Zero" model that traditional VMs or always-on clusters cannot match.

Cost-Efficiency for Inactive Tenants: Drastically reduces hosting overhead by only charging for compute when a user is actively building or executing queries.
Isolated Environments: ACA provides a secure, serverless container boundary for each tenant, ensuring that one customer’s heavy workload does not impact the compute availability of another.

5. Choosing Your Fortress: The High-Stakes Choice Between Siloed and Shared Tenancy

Architects must decide between "Siloed (Multi-Instance)" and "Shared (Logical) Backend" models. This choice dictates your security posture and your Azure bill. For Shared models, Microsoft Entra External ID (formerly Azure AD B2C) becomes the essential pillar for managing cross-tenant identity and routing users to their specific workspace.

Feature The Winner / Architectural Trade-off

Data Isolation Siloed:
Offers physical separation; zero risk of cross-tenant leaks.
Cost Shared:
Higher efficiency; tenants share unused cluster headroom.
No-Code Complexity Shared:
Simple infra, but requires complex "tenant_id" app-level logic.

Scalability Siloed: Predictable; avoids the "noisy neighbor" resource exhaustion.

6. The "Noisy Neighbor" Problem: The Logical vs. Physical Reality

When opting for a Shared Backend model, you face the "Noisy Neighbor" challenge. Here, logical separation relies on the integrity of your application code—specifically, "bug-free code filters" where every query must be scoped by a tenant_id.

To mitigate this at the database level, Azure Cosmos DB for MongoDB utilizes hierarchical partition keys (e.g., /tenant_id). This strategy co-locates a single tenant's data within the same physical partition, optimizing performance and providing a structural layer of separation even within a shared cluster.

"Requires aggressive resource throttling to avoid 'noisy neighbor' issues."

Without these throttles, a single tenant’s runaway script could consume the RU/s (Request Units) of the entire cluster, starving other users.

7. Conclusion: The Future of Democratized Infrastructure

The transition from a local Dockerized setup to a global Azure deployment underscores the evolution of democratized infrastructure. By integrating managed services like Azure Cosmos DB for data resilience and Azure Front Door for global routing, SSL termination, and WAF (Web Application Firewall) protection, enterprise-grade multi-tenancy is now accessible to every developer.

As we move toward a "scale-to-zero" world, we must ask ourselves: Is the biggest challenge of the next decade building the app itself, or orchestrating the infrastructure that lets it live?

Disaster Recovery Strategies for AKS Web Applications Backed by On-Prem ML Services

suresh devops — Fri, 11 Sep 2026 04:05:19 +0000

Popular DR Strategies in Cloud:

Backup and Restore (Lowest Cost, Slowest Speed)
Pilot Light (Low Cost, Moderate Speed)
Warm Standby (Higher Cost, Fast Speed)
Multi-Site (Highest Cost, Fastest Speed)

We planned below Disaster Recovery strategies for our Web applications hosted on AKS , backed up by 50 + ML Services running on On-Prem:

Our Infra setup with running hybrid architectures: Many web application layers deployed on Azure Kubernetes Service (AKS) in the cloud, calling back into machine learning inference services still hosted on-premises — often because of data residency rules, GPU capex already sunk into a data center, or ML models trained on sensitive data that can't leave the corporate network.

This hybrid pattern is powerful, but it introduces a DR problem that pure cloud-native architectures don't have: your failure domain now spans two estates you don't fully control together — a cloud region that can fail, and a data center that can fail independently of it, connected by a network link that can also fail on its own.

This post adapts the classic four-tier DR framework (Backup & Restore → Pilot Light → Warm Standby → Multi-Site) to this AKS + on-prem ML topology, and then walks through a concrete highly-available, multi-zone reference architecture.

The Two Foundational Metrics:

Every DR conversation starts with the same two numbers, and in a hybrid AKS/on-prem design you should define them separately for each tier of the stack (web/API layer, data layer, and the ML inference layer), because they will not be the same:

• Recovery Time Objective (RTO) — how long the business can tolerate the web app or the ML-backed feature being down.

• Recovery Point Objective (RPO) — how much data (user data,
feature-store data, model state) you can afford to lose.

A typical mismatch: the web tier might have an RTO of 15 minutes, while the on-prem ML training pipeline has an RTO of 24 hours because it depends on a single data center's GPU cluster. Your DR strategy has to account for that gap explicitly, usually through graceful degradation rather than pretending both tiers fail over at the same speed.

The Four DR Tiers, Adapted for AKS + On-Prem ML
As you move from lowest cost to highest cost, recovery speed increases (lower RTO/RPO). Here's how each tier translates when the backend dependency is an on-prem ML service rather than another cloud region.

1. Backup and Restore (Lowest Cost, Slowest Speed)

How it works in this topology:

• AKS/cluster state: Back up Kubernetes manifests, Helm releases, and persistent volumes with Velero (targeting an Azure Storage Account), plus Azure Backup for any managed disks.
• Container images: Enable geo-replication on Azure Container Registry (ACR) so images are already durable in a second region without a manual restore step.
• Application/config data: Regular exports of Azure SQL/Cosmos DB via native backup, or Azure Backup Vault for PaaS data stores.
• On-prem ML artifacts: Nightly (or more frequent) replication of trained model binaries, feature-store snapshots, and the model registry (e.g., MLflow) to Azure Blob Storage. This is the critical, often-forgotten piece — if the on-prem site is unreachable, you need the last known good model sitting in the cloud, not just the app tier.
Cost vs. Speed: Minimal ongoing spend — nothing is running idle. But recovery means provisioning a fresh AKS cluster, redeploying workloads from Velero, and standing up an emergency inference service (see the "ML fallback" pattern below) from the last replicated model. Expect RTO in hours, RPO measured from the last backup cycle.

2. Pilot Light (Low Cost, Moderate Speed)

How it works:

• A minimal secondary AKS cluster exists in a second Azure region, defined entirely as code (Bicep, Terraform, or ARM templates), scaled to near-zero (e.g., a single system node pool, no user workloads running).
• Core namespaces, secrets (via Azure Key Vault + CSI driver), and networking (VNet, private endpoints) are pre-provisioned so the "light" is already on — you're not starting from a blank slate.
• On-prem ML fallback: A lightweight, always-warm inference stub — a smaller/quantized model or a cached-prediction service — is kept deployed in the secondary region's AKS cluster. It isn't meant to replace the full on-prem model; it's meant to keep the application functional but degraded while the primary ML path is restored.
Cost vs. Speed: Low ongoing cost since only skeleton infrastructure runs continuously. Failover means scaling the node pools up via automation and pointing traffic to the secondary cluster — RTO typically drops to tens of minutes.

3. Warm Standby (Higher Cost, Fast Speed)

How it works:

• A partially scaled AKS cluster runs continuously in the secondary region with real (not stub) application pods and the Horizontal Pod Autoscaler and Cluster Autoscaler configured to burst on failover.

• Data layer is continuously replicated (e.g., Azure SQL active geo-replication, Cosmos DB multi-region writes).

• For the ML dependency, this tier is where a secondary on-prem site or a colocated edge inference cluster becomes worth the investment — either a second data center with replicated models, or GPU-backed nodes in Azure (via Azure Machine Learning endpoints or GPU node pools in AKS) that mirror the on-prem model's behavior closely enough to serve as a live hot backup, fed by continuous model-artifact sync.

• Azure Traffic Manager or Azure Front Door health-probes the primary path (app + on-prem ML) and can redirect traffic before a human intervenes.

Cost vs. Speed: Meaningfully higher cost — you're running real capacity around the clock. In return, RTO drops to single-digit minutes because the standby path is already serving low-volume traffic and simply needs to absorb the full load.

4. Multi-Site / Active-Active (Highest Cost, Fastest Speed)

How it works:

• Fully active AKS clusters in two or more Azure regions, each capable of independently serving 100% of traffic, fronted by Azure Front Door or Traffic Manager with active-active routing.
• The ML dependency is the hard part here: true multi-site requires either (a) two independent on-prem data centers with synchronized model registries and load-balanced inference traffic over redundant ExpressRoute circuits, or (b) accepting that ML inference is migrated into the cloud (Azure ML / GPU AKS node pools in each region) so the cloud becomes multi-site while on-prem becomes a batch training source rather than a real-time dependency.
• This is also where you decouple the app from synchronous calls to on-prem ML entirely, using an event/queue-based pattern (Azure Service Bus, Event Hubs, or Kafka) so a slow or unreachable on-prem inference path never blocks the web tier — the app degrades to cached/default predictions instead of failing outright.

Cost vs. Speed: Highest cost by far — full redundant capacity, redundant network circuits, and (usually) a redesign to remove hard synchronous coupling to a single on-prem site. In exchange, RTO/RPO approach near-zero, and the on-prem site's failure stops being a single point of failure for the whole product.

Comparison at a Glance :

Tier Ongoing Cost RTO RPO On-Prem ML Handling

Backup & Restore $ Hours Hours (last backup) Model artifacts backed up to cloud storage; emergency stub built on demand

Pilot Light $$ Tens of minutes Tens of minutes Warm stub/quantized model kept ready in secondary region

Warm Standby $$$ Single-digit minutes Minutes Hot secondary inference path (2nd DC or cloud GPU nodes), continuously synced

Multi-Site $$$$ Near-zero Near-zero Dual on-prem sites or ML migrated fully into cloud; async decoupling via queues

Designing a Highly Available, Multi-Zone AKS Web Application
and extending it to cover the on-prem hop:

• Azure Front Door: Acts as the global entry point and CDN — caches static content at the edge, applies WAF policies, and performs global health-probing/failover across regions, similar in role to CloudFront + Route 53 combined.

• Azure Load Balancer / Application Gateway (with AGIC): Distributes traffic across AKS ingress controllers, providing L4/L7 load balancing and SSL termination in front of the cluster.

• AKS with Availability Zones: Deploy node pools across all three Availability Zones in the region, and set topologySpreadConstraints / pod anti-affinity so replicas of each deployment are spread across zones — this is the direct equivalent of an EC2 Auto Scaling Group spanning multiple AZs. Combine with the Cluster Autoscaler and HPA so pod and node counts flex with demand automatically.

• Azure Cache for Redis: Sits in front of both the application data layer and, importantly, in front of the on-prem ML calls — caching recent inference results so that a transient blip in on-prem connectivity or latency doesn't degrade every user request, only new/uncached ones.

• Hybrid connectivity resilience: This is the piece with no AWS-native equivalent, since it's the link back to the data center. Provision redundant ExpressRoute circuits (from different peering locations) with a Site-to-Site VPN as a tertiary fallback, and front the on-prem ML endpoint with Azure Private Link/Private Endpoint so the AKS workloads always resolve to the healthiest available path. Wrap every call to the on-prem service in a circuit breaker (e.g., Polly in .NET, resilience4j-style patterns in other stacks) so a struggling on-prem dependency fails fast and the app falls back to cached or default responses instead of cascading timeouts.

Putting it together

A request flows: Azure Front Door → Application Gateway → AKS ingress (zone-redundant pods) → Redis cache check → ExpressRoute (primary) / VPN (fallback) → on-prem ML inference → response cached in Redis for subsequent requests.

Every hop in that chain has its own failure mode, and the DR tier you choose should be applied consciously at each hop rather than uniformly across the whole system — it's common, and often the right cost/speed trade-off, to run the web/app tier at Warm Standby while the on-prem ML tier only reaches Pilot Light, compensated for with aggressive caching and graceful degradation in the application logic.

Key Takeaways

Define RTO/RPO per tier, not just per application — the web layer and the on-prem ML layer will almost never share the same recovery targets.
Backup on-prem ML artifacts (models, feature stores, registries) to cloud storage regardless of which DR tier you choose elsewhere — it's cheap insurance and enables an emergency fallback at any tier.
Decouple the web tier from synchronous, blocking calls to on-prem ML wherever possible using caching and queue-based patterns — this alone can turn an on-prem outage from a full outage into a graceful degradation.
Treat network connectivity (ExpressRoute/VPN) as its own failure domain with its own redundancy plan — it's the one link in this architecture that doesn't exist in a pure single-cloud design.
Multi-zone AKS deployment (Front Door, App Gateway, zone-spread node pools, Redis) gives you regional high availability essentially "for free" at moderate cost — treat it as the baseline, then layer the four DR tiers on top for cross-region and cross-site resilience.

Our Ollama /LLM Production Setup

suresh devops — Sat, 08 Aug 2026 05:16:35 +0000

Response to Ollama Production Queries to the previous post

Excellent questions — you've clearly been in the trenches with Ollama at scale! Let me address your specific concerns about our production setup, now updated with our actual deployment architecture.

On GPU Residency & VRAM Thrashing

You're absolutely right — this is the single biggest operational challenge with Ollama in production. We've implemented a hybrid strategy:

Our Approach:

Hot Models Pinned in Memory:
- qwen3:8b (primary inference)
- bge-m3 (embedding service)
- qwen3:4b-instruct (NER-specific)

These stay resident with OLLAMA_KEEP_ALIVE=-1 in the container environment.

Cold Models on Separate Instance:
- gemma3:4b, llama3.2, custom RatnaDveLinga
- These run on a secondary Ollama instance with a lower concurrency limit
- The load balancer routes to this instance only for specific workloads
VRAM Monitoring:

   # We run this in a sidecar container
   watch -n 1 nvidia-smi --query-gpu=memory.used,memory.free --format=csv

The Keep-Alive Tuning We Use:

# docker-compose environment section
environment:
  - OLLAMA_KEEP_ALIVE=5m  # Not -1 to avoid holding all models
  - OLLAMA_NUM_PARALLEL=4
  - OLLAMA_MAX_LOADED_MODELS=3  # Critical! Prevents thrashing

Key insight: Setting OLLAMA_MAX_LOADED_MODELS to 3 forces eviction behavior to be predictable rather than reactive. We learned this after a production incident where p99 latency jumped from 200ms to 8 seconds during a batch processing job!

On Concurrent Load & Service Architecture

Great observation about OLLAMA_NUM_PARALLEL — default is 1 in many versions! Here's our production design at scale:

Our Actual Deployment Architecture:

Service	Instance Count	RAM (GB)	CPU (Cores)	Purpose
Ollama	21	-	-	LLM inference engine
Ollama-WebUI	21	-	-	Web interface for model management
forms-service-ner-v4-api	2	15	7.5	Production NER API endpoints
forms-service-ner-v4-celery	3	15	7.5	Async NER task workers (prod)
forms-service-ner-qa-api	2	15	3.9	QA environment NER APIs
forms-service-ner-qa-celery-server	6	15	3.9	QA async task workers
forms-service-ner-api(sales)	4	15	3.9	Production -specific NER APIs
forms-service-ner-celery(sales)	4	15	3.9	Production async task workers
signal-summarizer-llm	2	-	-	Main summarizer service
signal-summarizer-llm workflow-aggregator	9	-	-	Workflow orchestration
signal-summarizer-llm worker-model1	9	-	-	Model execution workers
signal-summarizer-llm celery-flower	1	-	-	Celery monitoring dashboard

Load Distribution Strategy:

                    ┌──────────────────────────────────────┐
                    │       KONG API Gateway               │
                    │  (Least Connection Algorithm)        │
                    └─────────────┬────────────────────────┘
                                  │
            ┌─────────────────────┼─────────────────────┐
            │                     │                     │
    ┌───────▼──────┐    ┌────────▼────────┐    ┌───────▼──────┐
    │  NER Services │    │  Summarizer     │    │  Ad-hoc      │
    │  (17 pods)    │    │  Services       │    │  Queries     │
    └───────┬──────┘    │  (20 pods)      │    └───────┬──────┘
            │           └────────┬────────┘            │
            └─────────────────────┼─────────────────────┘
                                  │
                    ┌─────────────▼─────────────┐
                    │   21 Ollama Instances      │
                    │   (Distributed Pool)       │
                    └────────────────────────────┘

How KONG's Least Connection Load Balancing Helps:

1. NER Service Distribution (17 instances total):

Production NER: 2 API + 3 Celery = 5 instances (high memory/CPU)
QA NER: 2 API + 6 Celery = 8 instances (lower CPU, more workers)
Sales NER: 4 API + 4 Celery = 8 instances (balanced)

KONG routes traffic to the least loaded instance, which is crucial for:

Preventing any single NER instance from being overwhelmed by large document batches
Allowing Celery workers to scale independently based on queue depth
Isolating QA traffic from production traffic

2. Summarizer Service Distribution (20 instances total):

Main service: 2 instances
Workflow aggregators: 9 instances (coordinating multi-step summaries)
Model workers: 9 instances (doing the heavy lifting)
Celery flower: 1 instance (monitoring)

Request Flow with Backpressure:

# Simplified version of our queue management with distributed instances
class OllamaLoadBalancer:
    def __init__(self):
        # Track load across all 21 Ollama instances
        self.instance_pool = self.discover_ollama_instances()

    def route_request(self, model, prompt, service_type):
        # Find least loaded instance via KONG
        if service_type == 'ner':
            # Route to NER-specific Celery queue
            # Celery distributes across 13 NER workers (2+3+4+4)
            instance = self.get_least_loaded(['forms-ner-v4', 'forms-ner-qa', 'forms-ner-sales'])
        elif service_type == 'summarizer':
            # Route to summarizer Celery queue
            # Celery distributes across 9 model workers
            instance = self.get_least_loaded(['summarizer-workflow', 'summarizer-model1'])

        # Set appropriate parallelism per service type
        if service_type == 'ner':
            ollama_timeout = 30  # seconds
            ollama_concurrency = 4  # OLLAMA_NUM_PARALLEL
        else:  # summarizer
            ollama_timeout = 120  # seconds (larger context)
            ollama_concurrency = 2  # careful with VRAM

    def get_least_loaded(self, services):
        # Query KONG's least-connection algorithm
        return kong_api.get_least_connected_instance(services)

What We've Learned at This Scale

The Good:

21 Ollama instances across servers provide excellent fault tolerance
Least connection algorithm naturally handles hot spots
Separate QA vs Production instances (8 NER QA pods) allow safe testing
Sales-specific NER (8 pods) isolates business-critical workloads

The Challenges:

Model Consistency: Keeping all 21 Ollama instances synchronized with same models and versions
- Solution: We use a shared NFS volume for models, or have a model-sync cron job
Celery Worker Configuration: With 13 NER Celery workers and 9 summarizer workers, we had to tune:

   # Celery config
   CELERYD_PREFETCH_MULTIPLIER = 1  # Prevent worker hogging
   CELERY_ACKS_LATE = True  # Re-queue if worker crashes
   CELERY_TASK_TIME_LIMIT = 300  # 5 min max
   CELERY_TASK_SOFT_TIME_LIMIT = 240

KONG Routing Complexity: With least-connection across heterogeneous instance types (some 7.5 CPU cores, some 3.9), we had to implement:

   -- Kong plugin to weigh instances by capacity
   -- Instance with 7.5 cores should get ~2x traffic of 3.9 core instance

Specific Q & A

Q: "Given your NER and summarizer run on the large-document path, how are you handling concurrent load — one instance per service, or a shared pool with request queuing in front?"

A: Distributed pool with service-specific queues:

NER: 17 total instances (5 prod + 8 QA + 4 sales) each with OLLAMA_NUM_PARALLEL=4 → 68 concurrent NER requests possible
Summarizer: 20 total instances (2 main + 9 workflow + 9 workers) with OLLAMA_NUM_PARALLEL=2 → 40 concurrent summarizer requests possible
All fronted by KONG with least-connection algorithm → no single point of overload

The key insight: With 21 Ollama instances, even if one gets busy, KONG routes to the next least-loaded. Celery handles the queue depth and retries.

Critical Monitoring in Production

We track these metrics per service:

Service	Critical Metrics	Alert Threshold
NER (All)	Queue length > 50 per celery worker	Scale up workers
NER (Prod)	99th percentile latency > 3s	Investigate model/instance
Summarizer	99th percentile latency > 8s	Check GPU usage
Ollama Instances	Memory usage > 80%	Evict cold models
KONG	504/502 errors rate > 1%	Check instance health

One More Thing: Metrics Are Your Friend

With 21 Ollama instances and 37 service instances (NER + Summarizer), observability is critical:

# Distributed tracing across services
# We use OpenTelemetry with Jaeger
# This helps identify if latency is in:
# 1. KONG routing
# 2. Celery queuing
# 3. Ollama inference
# 4. Network between instances

Bottom line: Your assessment is spot-on. The production challenges with Ollama aren't getting it running — they're keeping it running predictably under mixed workloads across 21 instances. We've found that explicit model management with OLLAMA_MAX_LOADED_MODELS, proper Celery tuning, and KONG's least-connection balancing are non-negotiable at this scale.

The 21-instance deployment gives us:

High availability (can lose 2-3 instances without impact)
Scalability (can route QA vs Prod traffic separately)
Performance (least-connection ensures no single instance is overwhelmed)

Happy to dive deeper into any of these aspects further 🚀

Posted our LLM / Ollama process here: https://dev.to/suresh_devops_ffa0728a190/powering-ml-services-with-ollama-a-complete-setup-for-self-hosted-llms-3o7f

suresh devops — Fri, 07 Aug 2026 09:08:36 +0000

suresh devops

Aug 7

Powering ML Services with Ollama: A Complete Setup for Self-Hosted LLMs

#llm #ollama #mlops #machinelearning

3 min read

Powering ML Services with Ollama: A Complete Setup for Self-Hosted LLMs

suresh devops — Fri, 07 Aug 2026 09:05:23 +0000

Our ML Services need LLMs to process the large documents and data. We started using Ollama since 2024 and the learnings below:

Ollama, an open-source tool that packages LLMs into a simple CLI and REST server, has emerged as a game-changer for developers and organizations seeking privacy, cost-efficiency, and offline capabilities

*What is Ollama and Why Use It? *

Ollama is often described as "Docker for LLMs" – you pull a model by name, and Ollama handles quantization, memory mapping, and inference runtime behind the scenes. The core reasons developers choose Ollama over cloud APIs include:

Privacy: Your prompts and data never leave your machine
Cost: Zero per-token charges after the one-time model download
Latency: No network round-trip, especially fast on modern GPUs
Offline use: Works without any internet connection once models are downloaded

Our ML Services Environment

Our setup runs a variety of models optimized for different tasks, including:

Qwen3 series (8B, 4B-instruct, 0.6B) for general-purpose inference
Llama3.2 and custom variants for specialized tasks
Gemma3:4b and Gemma2:2b for lightweight processing
BGE-M3 for embedding generation
llm – a custom model created from GGUF format

We also maintain two dedicated services:

ner: For Named Entity Recognition in forms
summarizer-llm: For summarizing signals and communications

Complete Setup Walkthrough

Install Dependencies

Start by updating your system and installing essential packages:

apt update && apt upgrade -y
apt install git docker-compose curl

Install NVIDIA Drivers and Container Toolkit

For GPU acceleration (which provides 10-100x speedup over CPU inference) , install NVIDIA drivers and the NVIDIA Container Toolkit:

distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list

sudo apt-get install -y nvidia-docker2
sudo systemctl restart docker
sudo ubuntu-drivers autoinstall

Reboot the system after driver installation.

Clone and Configure the Ollama Repository

git clone https://github.com/sujithrpillai/ollama.git
cd ollama

Create a docker-compose.yml file with GPU support:

version: '3.8'

services:
  ollama:
    image: ollama/ollama:latest
    hostname: ollama
    ports:
      - "11434:11434"
    volumes:
      - ./models:/root/.ollama/models
    networks:
      - genai-network
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    runtime: nvidia
    restart: always

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    volumes:
      - ./backend/data:/app/backend/data
    networks:
      - genai-network
    restart: always

networks:
  genai-network:
    driver: bridge
    name: genai-network

The GPU configuration in the deploy section enables hardware acceleration for supported NVIDIA GPUs .

Start the Services

docker-compose up -d

Pull Models

Access the Ollama container and download models:

docker exec -it ollama ollama pull qwen3:8b
docker exec -it ollama ollama pull gemma2:2b
docker exec -it ollama ollama pull llama3.2:latest

Create Custom Models from GGUF Files

Ollama allows creating custom models from GGUF format files :

sudo apt-get install git-lfs
git clone https://huggingface.co/org/llm-v3-270k-GGUF
curl -fsSL https://ollama.com/install.sh | sh

Create a Modelfile:

FROM /home/RatnaDveLinga-v3-270k-GGUF/RatnaDveLinga-v3-270k.gguf

Then create the model:

ollama create llmname -f Modelfile

Verify Setup

Check models available:

root@ollama:/ ollama list

NAME                                ID              SIZE      MODIFIED
qwen3:8b                            500a1f067a9f    5.2 GB    8 months ago
qwen3:4b-instruct                   0edcdef34593    2.5 GB    8 months ago
qwen2.5:1.5b                        65ec06548149    986 MB    9 months ago
gemma3:4b                           a2af6cc3eb7f    3.3 GB    13 months ago
llama3.2:latest                     a80c4f17acd5    2.0 GB    13 months ago
bge-m3:latest                       790764642607    1.2 GB    13 months ago
RatnaDveLinga:latest                decc819a53b1    1.7 GB    21 months ago

Model Management Commands

Essential CLI commands for managing your Ollama setup :

Command	Description
`ollama list`	List all downloaded models
`ollama pull <model>`	Download a model without running it
`ollama run <model>`	Run a model interactively
`ollama stop <model>`	Stop a running model
`ollama rm <model>`	Remove a model from disk
`ollama ps`	Show currently running models
`ollama show <model>`	Display model metadata

Performance Considerations

GPU acceleration can provide 10-100x faster inference compared to CPU-only operation . To verify GPU usage:

docker exec -it ollama nvidia-smi

The Open WebUI interface available at http://localhost:3000 provides a user-friendly way to interact with models, pull new ones, and manage your setup without using the command line .

Conclusion

This setup provides a robust, self-hosted LLM environment with GPU acceleration, a comprehensive model library, and a web interface for easy interaction. Whether you're running specialized NER services, summarization tasks, or general-purpose inference, Ollama combined with Open WebUI delivers a powerful and private AI platform.

Happy prompting!

Why Your Elasticsearch Backups Might Fail When You Need Them Most

suresh devops — Tue, 04 Aug 2026 04:01:43 +0000

Why Your Elasticsearch Backups Might Fail When You Need Them Most: Lessons from the July 2026 Recovery

In the world of Site Reliability Engineering, there is a painful, often career-defining difference between having a backup and having a recovery. This reality hit our team with visceral clarity during a high-stakes production recovery event in July 2026.

While we believed our standard operating procedures were robust, the technical nuances of Elasticsearch 7.17.x proved otherwise. Our previous SOP didn't just have "gaps"—it allowed for a "Partial" success state that nearly resulted in total data loss.

The following insights are derived from the hard-won lessons of that recovery. We have since scrapped the old methods and codified these findings into our formal SOP (Version 1.0). If you are responsible for Elasticsearch clusters, these changes are the difference between a clean restoration and a catastrophic failure.

The "Partial" Snapshot is a Silent Failure:

The most dangerous trap in Elasticsearch administration is accepting a snapshot that isn't perfect. In July 2026, we learned that a snapshot can complete with a state of PARTIAL, a "silent failure" that often occurs due to node shutdowns, shard unavailability, or filesystem issues on the NFS server. A partial snapshot means some shards were simply skipped, leaving invisible data holes that you won't discover until your application is back online and throwing errors.

Our updated SOP now strictly forbids proceeding with a restoration if the state is anything other than SUCCESS. If you encounter a PARTIAL state, do not attempt to overwrite or restart the full process immediately. Instead, you must resolve the underlying infrastructure issue and take a NEW, dedicated snapshot specifically for the missing indices.

"A backup is considered successful when: Snapshot state is SUCCESS."

The Global State Bloat:
When executing a backup, the default behaviour is often to capture the cluster’s global state. While this seems comprehensive, it is an architectural poison when restoring to a different environment. During our recovery, we moved data from our source cluster (Primary Node) to a target cluster (Restore Cluster). Including the global state risked "poisoning" the new cluster with shard allocation awareness attributes and persistent tasks that were specific to the old hardware.

By explicitly setting "include_global_state": false, we ensure the backup is truly portable.

PUT /snapshot/es_backup/all_indices
{
"indices":"*",
"include_global_state":false
}

This prevents the target cluster from trying to mimic the source's runtime configuration, which is essential for a clean start on the Restore Cluster cluster.

Don't Restore the "Ghost" Indices:

A common reflex during a disaster is to restore everything inside the snapshot. However, our recovery post-mortem identified several "ghost" indices—system indices like .tasks and .geoip_databases—that should be ignored.

These are runtime indices that Elasticsearch 7.17.x recreates automatically. Attempting to force a restore of .tasks, for example, can cause the target cluster to hang or crash as it tries to resume background tasks that were specific to the source cluster's state.

Always exclude these system indices to allow the target cluster’s native operations to initialize correctly.

The "Green" Health Mandate

A successful recovery begins long before the PUT command is run; it starts with the health of the source. One of our key lessons from July 2026 was that snapshots taken during node restarts or maintenance windows are inherently unreliable.

We now mandate a "Status Green" precondition. If your cluster is "Yellow," you have unassigned or initializing shards. Taking a snapshot of a degraded cluster is effectively building on shifting sand—it is the primary recipe for the PARTIAL failures mentioned earlier. Before you touch the backup API, verify that there are no relocating shards and the cluster is stable.

Metadata is the True Backup

A snapshot file residing on the NFS server is useless if the cluster that created it is gone and you have no record of what was inside. During the heat of a recovery, you cannot rely on the Elasticsearch API to tell you what you have if the cluster itself is down.

We now require engineers to archive snapshot metadata—JSON output detailing index counts, shard status, and failure logs—off-cluster. This metadata should be stored in a version-controlled repository or a secondary secure bucket, not just on the cluster being backed up.

GET /_snapshot/es_backup/?pretty

"Archive snapshot metadata (indices, failures, timestamps, shard status) along with operational records for future audits and disaster recovery."
This allows you to verify that the document counts on your new Restore Cluster exactly match what was captured from the Primary Node source, providing the empirical proof required for a successful sign-off.
The Quarterly Integrity Test

The "set it and forget it" mentality is the enemy of Site Reliability. You do not have a backup until you have proven you can restore it. Our SOP now mandates a test restore on a non-production cluster under three specific triggers:

• Quarterly: A routine scheduled validation.
• Post-Upgrade: Immediately following any Elasticsearch version change.
• Infrastructure Change: Whenever there are modifications to the NFS backup server or network pathing.

These tests verify more than just data; they validate your Recovery Time Objective (RTO) and ensure that your application remains compatible with the restored mappings and settings.

Conclusion: The Future of Resilience

The transition to Version 1.0 of our Elasticsearch SOP marks our shift from "checking a box" to building true operational resilience. By enforcing "Green" state preconditions, excluding dangerous global state, and rigorously archiving metadata off-cluster, we have replaced "Partial" hope with technical certainty.

The July 2026 recovery was a wake-up call that forced us to evolve. If your primary cluster vanished today, would your metadata stored off the Backup server be enough to rebuild, or are you currently relying on a "Partial" hope?

Can we survive as DevOps Engineers in AI Era ?

suresh devops — Sat, 21 Feb 2026 12:11:06 +0000

_(Haunted by AI and trying to find answers - Notes -1)
_

As DevOps professionals, the most dangerous thing we can do right now is mistake our ability to write a CI/CD pipeline for job security. We are witnessing a violent divergence in the DevOps career path.

AI is not coming for the architects; it is coming for the executors—the engineers who have spent their careers living at "YAML-level depth."

The central thesis of this new era is : The manual pipeline builder is a dying breed. The AI-Orchestrating Architect is the future.

*The Great Split: Tool Operators vs. Platform Architects
*
The industry is bifurcating into two distinct paths with vastly different lifespans. Understanding which path you are on is the difference between professional obsolescence and becoming a linchpin of the enterprise.

The Divergent Paths

• Path 1: The Tool Operator (High Risk) This role is defined by routine execution: writing pipelines, drafting Helm charts, deploying containers, and patching minor configuration drift. Because AI can already automate 60–70% of these commodity tasks, this layer of the profession is rapidly becoming a low-value commodity.

• Path 2: The Platform Architect / Reliability Strategist (High Value) This is the domain of the AI-Orchestrating Architect. It involves high-level systemic design: failure domains, multi-region architecture, SLO/SLA strategy, and governing how AI is integrated into the SDLC.
Analysis: The automation of commodity tasks makes the move to Path 2 an urgent strategic necessity. To survive this shift, you must stop being the person who moves the bricks and start being the person who designs the structural integrity of the building.

"AI will replace DevOps engineers who stay at YAML-level depth."

1: Distributed Systems Depth Over CLI Commands

Mastering Kubernetes syntax or specific CLI flags is no longer a competitive advantage—it’s a baseline. AI can generate infrastructure code and deploy clusters with high proficiency. However, AI lacks the systemic intuition required to prevent or debug a total system collapse. To reach the architect level, you must master the mechanics of distributed systems:

• Consensus & Leader Election: Mastering the Raft protocol and how state is maintained.
• The CAP Theorem: Navigating the brutal trade-offs between Consistency, Availability, and Partition Tolerance.
• Failure Management: Managing Eventual Consistency, Backpressure, Circuit Breaking, and the logic required to survive Retry Storms.
• Idempotency: Ensuring that system operations can be repeated without unintended side effects.

Analysis: While AI can build the infrastructure, human architectural thinking is required to manage the complexity of distributed failure. AI might build the bridge, but the Architect understands why the resonance of a thousand footsteps will make it collapse.

2: The SRE Mindset and the "Error Budget"

The definition of "success" is shifting from binary uptime to Site Reliability Engineering (SRE). A Strategic Lead doesn't just ask if a service is "up"; they manage the reliability of the system as a product feature.

Key SRE pillars from the source include:

• SLO Design and Golden Signals: Measuring the metrics that actually impact the business.
• Error Budgets: Using data to negotiate the tension between feature velocity and system stability.
• Incident Analysis and Chaos Engineering: Proactively testing resilience and performing deep-dive postmortems to find the "why" behind the "what."

Analysis: Designing for failure before it happens is a uniquely human skill. It requires a proactive, strategic mindset that views every incident not as a chore, but as an architectural data point.

3: FinOps – Cost is Now an Architectural Pillar

In the age of massive cloud scale, infrastructure cost is no longer an accounting problem—it is a primary architectural constraint. As an architect, you must treat "Cost Architecture" with the same rigor as security.

This requires mastering:

• Bin Packing Math: Optimizing resource density to ensure maximum ROI on compute.
• Spot vs. On-Demand Strategy: Designing workloads that can survive the volatility of spot instances to slash OpEx.
• Observability Cost Control: Ensuring the cost of monitoring the system doesn't exceed the value of the data gathered.
• Workload Rightsizing: Constant, automated tuning of resource requests to match actual utilization.

Analysis: AI can generate a thousand instances in seconds, but it lacks the organizational context to know if those instances are a waste of capital. FinOps isn't just about saving money; it’s about reclaiming the innovation budget.

4: The Rise of the Internal Developer Platform (IDP)

The future of DevOps is Platform Engineering. This marks a fundamental shift from "deploying applications" to "designing the systems that deploy applications."

The goal of the Architect is to move from being a manual bottleneck to becoming a platform provider. This is achieved through:

• Golden Path Templates: "Secure-by-default" workflows that make the right way the easy way.
• Guardrails and Policy as Code: Automated governance that prevents disasters before they are committed to main.
• Self-Service Environments: Empowering developers to move at speed within safe, pre-defined boundaries.

Analysis: When you stop being the person who manually runs the deployment and start being the architect of the IDP, you become a force multiplier for the entire engineering organization.

5: Judgment Under Uncertainty – The Irreplaceable Skill

What makes a professional truly irreplaceable is not a collection of certifications, but judgment under uncertainty. You must become the AI-Orchestrating Architect who uses AI for root cause exploration, log pattern detection, threat modeling, and IaC reviews, but retains the final word.

Critical "When to..." decisions include:

• When to re-architect a legacy system vs. when to maintain the "technical debt."
• When to deprecate a service that no longer serves the business.
• When to accept calculated risk for the sake of market speed.
"AI suggests. Architect decides."

Analysis: This level of judgment is your career "moat." It is the protective layer that automation cannot cross because it requires context, experience, and the accountability to own the outcome.

A Strategic Roadmap:

Transitioning from an executor to a Reliability Strategist requires a structured evolution:

Foundations & Distributed Systems Deep dive into Raft/Consensus, master observability, design High Availability (HA) across zones/regions, and take the lead on high-stakes incident analysis.

Platform Ownership & Governance Own the platform architecture, design organization-wide cost governance models, and build the reusable DevOps frameworks that others consume.

Strategy & Enterprise Architecture Define the enterprise-level DevOps strategy, lead multi-cluster/multi-region global designs, and move into the role of Principal Engineer or Enterprise Architect.

The Hard Truth:

The era of the "pipeline builder" is ending. The demand for those who can navigate platform engineering, reliability, cost optimization, and AI governance is exploding.

Ignoring AI is the only certain way to be replaced. Those who integrate AI to handle the "grunt work" of log analysis and threat modeling will always outperform those who try to compete with the machine.

DevOps as “platform + reliability + cost + AI governance” will expand massively

How we fixed Real Kubernetes Production Incidents

suresh devops — Wed, 18 Feb 2026 15:30:28 +0000

We Used 'Kubernetes Advanced Scheduling' to Fix Real Production Incidents as below:

Kubernetes scheduling looks simple — until production traffic hits.
In lower environments, the default scheduler works beautifully.

In production, under load, across multiple AZs, with service mesh, sidecars, GPUs, and mixed workloads?

That’s when we discovered:
The scheduler is not “basic.”

It’s powerful — if we know how to use it.

Based on real issues we resolved in a live Kubernetes environment running critical workloads, these were 2 AM incidents.

Incident 1: All Pods in One AZ – The Near-Outage

The Situation

We had a 3-zone cluster.
A critical API deployment had 9 replicas.
One evening, during a traffic spike, one Availability Zone had network degradation.
Suddenly, 6 of our 9 pods were unreachable.
Why?
Because the default scheduler had “helpfully” packed pods into fewer zones for efficiency.
ReplicaSet guarantees count.
It does NOT guarantee distribution.

That night we learned:
High Availability is not automatic.

The Fix: Pod Topology Spread Constraints

We implemented:
• topologyKey: topology.kubernetes.io/zone
• maxSkew: 1
• whenUnsatisfiable: DoNotSchedule

This forced the scheduler to maintain balanced pod distribution across zones.

What Changed Immediately

• Pods spread 3-3-3 across AZs
• Rolling updates maintained balance
• Zone failure no longer meant partial collapse

**Why This Matters

Unlike anti-affinity (which is binary), Topology Spread Constraints are quantitative.

They enforce distribution math.
That’s the difference.

Real Production Lesson

During a rolling update before the fix:

New pods filled one AZ first.
Traffic imbalance followed.
Latency spikes happened.

After implementing spread constraints:

Updates remained balanced from the first pod.
No traffic concentration.

Incident 2: Node Memory Pressure – “But Requests Look Fine”
The Situation

We were using:
• Service mesh (Istio sidecars)
• Kata Containers for isolated workloads

Pods were requesting 512MB.
Nodes were showing allocatable capacity sufficient for 20 pods.

But after deploying 15 pods:

• Memory pressure started
• Evictions happened
• Latency increased

Metrics didn’t make sense.
Until we realized:
The scheduler was blind to runtime overhead.

The Fix: Pod Overhead via RuntimeClass

We defined a RuntimeClass with:

overhead:
podFixed:
memory: "50Mi"
cpu: "50m"
Now every pod using that RuntimeClass automatically added overhead to scheduling math.

What Changed?

Before:
Scheduler thought 512MB per pod.

After:
Scheduler calculated ~562MB per pod.
Node fitting became accurate.
Evictions stopped.

Critical Insight

Pod Overhead creates an “invisible container” in scheduling math.

Without it:
You overcommit silently.

With it:
Scheduling becomes financially and operationally accurate.

Incident 3: GPU Pods Landing on Spot Nodes

The Situation

We had:

• GPU nodes (on-demand)
• General compute nodes (Spot instances)
A training workload got scheduled on a Spot node without GPU.

Result:

• CrashLoopBackOff
• Wasted compute
• Delayed ML training pipeline

Node labels existed.
But scheduling preference wasn’t strict enough.

The Fix: Scheduler Profiles

Instead of rewriting the scheduler, we:

• Created a separate scheduler profile
• Configured specific scoring behavior for ML workloads
• Used taints + tolerations + stricter filtering

Now:

• General workloads used default scheduling
• ML workloads used a GPU-aware scheduling profile

Production Impact

• Zero misplacements
• Predictable GPU utilization
• Better cost control

Incident 4: Bin Packing Wasn’t Actually Packing

The Situation

We were optimizing for cost.
Using bin packing strategy (MostRequested scoring).
But nodes were underutilized by 5–7%.

Finance asked:

“Why are we scaling nodes when CPU is still free?”
The missing variable?
Pod Overhead wasn’t being considered in our mental math.

The Real Scheduler Math

When NodeResourceFit runs:

Total Pod Compute =
(Sum of container requests) + (Pod Overhead)

During scoring:
Scheduler evaluates:
(Current Node Usage + Total Pod Compute)

With overhead included,
Bin packing becomes more accurate.
Without understanding this,
You miscalculate cluster density.

After adjusting runtime overhead correctly:
• Node utilization improved
• Fewer nodes required
• Monthly cloud cost dropped noticeably

What Most Teams Don’t Realize

The scheduler is not a black box.

It has phases:

Filtering
Scoring
Binding

You can influence each phase.
Advanced scheduling is not “nice to have.”

It is production engineering.

Lessons learnt

Replica count ≠ High Availability : Use Topology Spread Constraints.
Container request ≠ Total consumption : Use Pod Overhead when sidecars or microVMs are involved.
One scheduler profile ≠ All workloads : Use Scheduler Profiles for workload classes.
Bin packing requires correct math Understand NodeResourceFit + Overhead interaction.

Running Kubernetes vs Engineering Kubernetes

Anyone can deploy pods.
Engineering Kubernetes means:
• Designing failure domains intentionally
• Accounting for invisible compute
• Aligning scheduling with business rules
• Optimizing cost through math, not guesswork

The scheduler is not just a placement tool.

It is:
A control plane decision engine.
And this separates:
Clusters that survive production
from
Clusters that collapse under it.

(You no need to be a Kubestronaut to learn and apply not so known features of Kubernetes. All you need is a 'crisis' at 2 AM)

DevOps Books - A DevOps enthusiast's bucket list

suresh devops — Wed, 18 Feb 2026 09:57:28 +0000

Today’s default learning path for most techies looks like this:

Problem → Panic → Open GPT → Copy → Deploy → Pray.

We’ve optimized for speed of answers, not depth of understanding.

Here’s the uncomfortable truth:

GPT can give you answers.
Books rewire how you think.

And DevOps at the Architect level?
It’s not about YAML. It’s about perception.

If you want to think like a DevOps Architect — beyond what you do — you need something deeper than prompts.

You need frameworks.
You need mental models.
You need immersion.

The DevOps Architect Bucket List

If you truly want to think beyond tools and certifications, add these to your professional bucket list:

☐ The Phoenix Project – Understand organizational chaos
☐ Accelerate – Learn what high performance actually looks like
☐ Continuous Delivery – Master deployment flow
☐ The DevOps Handbook – Structure transformation
☐ The Docker Book – Build container foundations
☐ Lean IT: Enabling and Sustaining Your Lean Transformation – Eliminate systemic waste
☐ The Journey to DevOps – A Testing Perspective – Strengthen quality culture

This is about intentionally studying 7 foundational works that shape how DevOps Architects think — beyond how tool operates, configure pipelines.

Book: The Phoenix Project

Author: Gene Kim

Starts with story.

This book humanizes DevOps. It exposes silos, burnout, firefighting culture, and broken flow.

Before you design scalable systems, you must understand why organizations collapse under their own complexity.

Architects first learn to see the system.

Book: Accelerate

Authors:
Nicole Forsgren, Jez Humble, Gene Kim

(Based on research from the State of DevOps Reports)

This book moves DevOps from belief to evidence.

Deployment frequency.
Lead time.
Change failure rate.
MTTR.

Now you’re not “suggesting improvements.”
You’re defending architecture with science.
Architects use data as leverage.

Book: Continuous Delivery

Authors:
Jez Humble, David Farley

This is your technical deep dive.

Understanding how code moves safely from commit to production is more important than learning any specific CI/CD tool.

Tools evolve.
Delivery principles do not.

Spend time mapping these concepts into your real infrastructure.

Architects understand flow before scale.

Book: The DevOps Handbook

Authors:
Gene Kim, Patrick Debois, John Willis, Jez Humble

If DevOps had a constitution, this would be it.

Culture.
Automation.
Measurement.
Sharing.

This book connects technical practices with leadership and organizational design.

It transforms DevOps from “engineering initiative” into “business strategy.”

Architects think beyond pipelines.

Book: The Docker Book

Author: James Turnbull

Before Kubernetes.
Before large-scale orchestration.

You must understand containers deeply.

Isolation.
Portability.
Consistency.

Modern cloud architecture stands on this foundation.

Architects master primitives before abstractions.

Book: Lean IT: Enabling and Sustaining Your Lean Transformation

Authors:
Mike Orzen, Steven Bell

DevOps without Lean becomes automation theater.

Lean IT teaches systems thinking.

Every delay.
Every handoff.
Every unnecessary approval.

That’s waste.

Architects remove friction before automating it.

Book: The Journey to DevOps – A Testing Perspective

Author: Chris Riley

This phase sharpens your quality mindset.

DevOps is incomplete without testing strategy.

Automation without validation is chaos at scale.

This book reinforces that DevOps maturity includes strong testing culture and continuous validation.

Architects design reliability into the system — not after it breaks.

The Real Difference comes from Long-form reading:

It Builds pattern recognition, Strengthens systems thinking, Improves strategic communication , Develops architectural intuition

Most engineers consume snippets.
Architects build frameworks.

These Seven books.
(May be in Twelve months?)

That’s the blueprint.

By committing to this roadmap, you move from:

Tool operator
to
Systems thinker
to
Strategic DevOps Architect

Most engineers collect certifications

Architects collect mental models.

Choose wisely.

(I completed some. The rest are scheduled — long before the final production shutdown)

My Accidental 20-Year DevOps Journey : from Cruise Control to Cloud Native

suresh devops — Mon, 16 Feb 2026 04:49:48 +0000

**Or: How I Learned to Stop Worrying and start living with Pipelines

2007.

I was there, typing away in a dimly lit office, surrounded by the gentle hum of servers that were actually in the same building (kids, ask your parents about "server rooms").

My tools of the trade? CruiseControl (bless its XML-filled heart). Ant scripts that looked like ancient scrolls. ClearCase—the version control system that required a PhD in theoretical physics and the patience of a saint. And good old SVC, because Git was still just a twinkle in Linus Torvalds' eye.

Back then, we called it "Build Release and Management." BRM. It sounded like a mild skin condition or perhaps a small-town accounting firm. Nobody outside our immediate team knew what we did. When people asked my job title, I'd say "I handle builds" and watch their eyes glaze over faster than a Windows 98 boot screen.

I had absolutely zero idea that I was standing at the foot of a mountain that would consume the next two decades of my professional life—and fundamentally reshape how the entire world builds software.

The Accidental Pioneer:

Here's the thing about being "early" to something: you don't know it's early. You just know you're tired of manually deploying code at 2 AM while the developer who broke the build sleeps peacefully, dreaming of clean compilations.
We weren't trying to start a revolution. We were just trying to get CruiseControl to send emails that weren't caught by the corporate spam filter.
But somewhere between wrestling with ClearCase branching strategies and writing Ant targets that would make a masochist wince, something was happening. The seeds were being planted. The industry was slowly realizing: "Wait, maybe the people who write code and the people who run it should probably... talk to each other?"
Revolutionary stuff, I know.

The Unstoppable Tidal Wave

Then DevOps happened. Or rather, DevOps happened to us.
Suddenly, everyone cared about what we did. Developers wanted to know about deployment pipelines. Ops wanted to understand the build process. Managers started using words like "synergy" and "continuous delivery" in meetings, usually incorrectly, but the intent was there!

The tools exploded:
• Jenkins rose from the ashes of Hudson
• Git arrived like a glorious messiah of branching
• Puppet, Chef, Ansible—we could finally script servers like code
• Docker containers (containers! that weren't Solaris Zones!)
• Kubernetes—because one container is lonely, but thousands need a therapist
• Terraform—infrastructure as code, because why should apps have all the fun?
Every few years, just when I thought, "Okay, surely we've automated everything now," the universe would respond: "Hold my beer."

From BRM Newbie to DevSecOps Eng. Lead

Fast forward to 2026. Two decades later:

I'm now DevSecOps strategist, FinOps Focal, ML Ops Starter, AI OPS enthusiast, Forced Innovationist across multiple Platforms.

The naive BRM learner who once celebrated getting CruiseControl to trigger a build without crashing now spends days thinking about:

• Shifting security left—so far left that security is practically in the parking lot before development even starts

• Platform engineering—building internal developer platforms that feel like magic, but with better documentation

• Compliance as code—because auditors shouldn't need spreadsheets in 2026

• MLOps—applying DevOps principles to models that have opinions about cats vs. dogs

• FinOps—because the cloud isn't actually "infinite," it's just "very large and surprisingly expensive"

• Cognitive load reduction—fancy term for "making sure my engineers don't quit to become goat farmers"

The tools have changed. The complexity has multiplied exponentially. But the mission remains surprisingly similar: get good software to users reliably, securely, and without waking anyone up at 2 AM.

Because We're All Still Here…

After twenty years, here's what I've learned:

Lesson 1. Nothing ages faster than "cutting edge."

I once gave a talk about why Maven was the future. I stand by what I said, but I also stand by my cargo shorts from 2007. Some things should stay in the past.

Lesson 2. The more things change, the more they break.

We have GitOps, policy-as-code, canary deployments, and chaos engineering. And yet, somewhere today, a developer is pushing directly to main and wondering why prod is on fire. That developer was me in 2008. It might still be me occasionally in 2026.

Lesson 3. DevOps is still growing because software is still eating the world.
Every industry, every company, every idea eventually becomes software-driven. And software needs to be built, deployed, and run. We're not running out of work anytime soon.

Lesson 4. The best tool is still the one your team will actually use.
Kubernetes is amazing. So is Bash. So is that weird shell script Frank wrote in 2019 that everyone pretends to understand. The technology matters less than the culture.

Lesson 5. We're all still learning.

I've been doing this for twenty years. I still Google basic YAML syntax. I still panic when a pipeline fails. I still celebrate small victories like a deployment that works on the first try.

I have had infinite lessons with finite scars in my DevOps journey.

To start with my 'epiphany' during my DevOps dark days, I would refer below books for 'Serious only' DevOps enthusiasts.

Patrick Debois - The DevOps Handbook
Gene Kim - The Phoenix Project, The DevOps Handbook
Jez Humble - Continuous Delivery, The DevOps Handbook
John Willis - The DevOps Handbook, Docker tutorial series
James Turnbull- The Docker Book, The Art Of Monitoring
Nicole Forsgren- State of DevOps Report, The Data on DevOps
Alan Shimel- Founder DevOps.com & DevOpsInstitute.com
Mike Orzen - Lean IT: Enabling and Sustaining Your Lean Transformation, Field Guide
Chris Riley - DevOps.com, The Journey to DevOps – A Testing Perspective
Sean Hull - Authors AWS, databases, scalability, and the cloud

What's Next?

If someone told me in 2007 that I'd still be doing this in 2026, writing blogs about DevSecOps and platform engineering, I'd have laughed. Then asked if ClearCase finally added Git integration. (It didn't. It really didn't.)

But here we are. DevOps isn't a job title anymore—it's how software gets made. It's not a team—it's a culture. It's not a trend—it's the baseline.

And the best part? It's still growing. Still evolving. Still full of problems to solve and puzzles to untangle.

To the BRM newbies, the DevOps engineers, the platform builders, the SREs, the release managers, and everyone who's ever spent four hours debugging a YAML indentation issue:

Welcome. The journey is weird, wonderful, and absolutely nowhere near finished.
Come for the automation. Stay for the people. And never, ever trust a deployment on a Friday afternoon.

A Former BRM Enthusiast, Current YAML Survivor

P.S. - If anyone needs me, I'll be writing Ant scripts for old times' sake. Just kidding. I'm not a masochist.

Kubernetes Advanced Scheduling ( Hidden gems of Kubernetes )

suresh devops — Sun, 15 Feb 2026 12:40:49 +0000

Kubernetes is often described as a scheduler’s operating system. The default scheduler (kube-scheduler) does an excellent job of placing pods on nodes based on basic resource requests and limits. However, in complex, production-grade environments, the default "fit and spread" logic is often not enough.

While Custom Resource Definitions (CRDs) and Operators get most of the spotlight in the ecosystem, the control plane itself hides a treasure trove of powerful scheduling levers. Here is a deep dive into the advanced scheduling features that separate a functional cluster from a finely-tuned, production-ready one.

Problem 1: Pod Topology Spread Constraints: Orchestrating Disaster Recovery

The Problem:
By default, a ReplicaSet ensures a certain number of replicas are running, but it doesn't care where they run. In a cloud environment, if all pods land on the same Availability Zone (AZ) and that zone fails, you experience a full outage. Similarly, if pods are packed onto the same node, a node failure takes out the entire service.

The Solution:
Pod Topology Spread Constraints allow you to enforce strict or best-effort distribution rules across arbitrary failure domains (e.g., zones, nodes, or even custom host labels).

How it works:
You define a topologySpreadConstraints field in your Pod spec. You specify:
• topologyKey : The label key on nodes that defines the domain (e.g., topology.kubernetes.io/zone).
• maxSkew : The maximum allowable difference in the number of pods across domains.
• whenUnsatisfiable : What to do if the skew can't be met (DoNotSchedule vs ScheduleAnyway).

Technical Deep Dive:
Unlike PodAffinity/AntiAffinity, which are binary (must be/not be), Spread Constraints are quantitative. They look at the distribution delta.
• Use Case 1: Strict HA: You can ensure that if you have 3 zones, a deployment of 6 pods is scheduled exactly 2 per zone. If a zone is unhealthy and a pod reschedules, the scheduler will wait until the zone recovers or another pod moves to maintain balance.
• Use Case 2: Rolling Updates: During a rolling update, new pods are created. Without spread constraints, the scheduler might fill up one zone first to bin-pack. With spread constraints, the new pods are spread evenly from the start, maintaining balance during the transition.

Problem 2. Pod Overhead: Accounting for the Invisible Resources

The Problem:
When using "microVMs" (Kata Containers, gVisor) or sidecar-heavy meshes (Istio), the user container is not the only thing consuming resources. The runtime shim or the sidecar proxy consumes CPU and memory before the main process starts. If the scheduler ignores this, it will overcommit the node, leading to throttling or node pressure.

The Solution:
Pod Overhead is a feature used primarily with RuntimeClasses. It allows you to define the resources consumed by the infrastructure (the runtime) per pod.

How it works:
You define a RuntimeClass that includes an overhead field. Any pod using that RuntimeClass automatically adds this overhead to its scheduling calculations.

Technical Deep Dive:
Scheduling is a math problem. The scheduler sums pod.spec.containers[*].resources.requests to determine node fit.
Pod Overhead injects an additional "invisible container" into this calculation.

• Example: A Kata container might need 50MB of memory and 5% CPU for the VM kernel and agent. The user requests 512MB for their app. The scheduler will see a total demand of 562MB. Without this, the node would be overprovisioned, and the VM might crash.
• Monitoring Impact: This also affects kubectl top node. The "allocatable" resources now account for this overhead, giving a more accurate picture of node utilization.

Problem 3. Scheduler Profiles & Extenders: Custom Logic Without Custom Code

The Problem:
You need a specific scheduling rule (e.g., "Don't schedule GPU pods on nodes with Spot instances," or "Prefer nodes with SSDs for databases"). Rewriting the entire Kubernetes scheduler from scratch is a daunting and fragile task.

The Solution:
Scheduler Profiles (introduced in v1.18) and Scheduler Extenders (legacy but powerful) allow you to inject custom logic into the scheduling pipeline.
• Scheduler Profiles (Multi-point): Allow you to run multiple scheduling configurations in parallel. You can configure which set of plugins (default or custom) run for specific pods.
• Scheduler Extenders: A process external to the scheduler that acts as a "webhook" for scheduling. The scheduler sends it a list of filtered nodes, and the extender filters or prioritizes them further.

Technical Deep Dive:
The scheduling cycle is split into phases: Filtering (Predicates), Scoring (Priorities), and Binding.
• Multi-scheduling: With Profiles, you could have one profile for general workloads that bin-packs tightly, and another profile for critical workloads that spreads thinly across nodes.
• Extenders: Imagine you have a hardware accelerator connected via USB to some nodes. The scheduler doesn't know about USB devices. An extender can look at the pod annotation, check an inventory database, and filter out nodes that don't have the USB device plugged in, allowing the pod to land only on physically capable hardware.

Problem 4. NodeResourceFit with Pod Overhead: The Hidden Math

The Problem:
While we covered Pod Overhead, it exists in isolation. The real magic happens when the scheduler's NodeResourceFit plugin interacts with it. Many engineers assume that if they set Pod Overhead, the scheduler just "adds it." But understanding how it adds it reveals potential pitfalls.

The Solution:
The NodeResourceFit plugin is the component that checks if a node has enough resources. Its integration with Pod Overhead ensures that overhead is treated as a first-class citizen during the Filtering and Scoring phases.

Technical Deep Dive:
There is a nuance here regarding Scoring. The scoring algorithm usually calculates a score based on the ratio of pod requests to node allocatable resources. With Pod Overhead:

Total Pod Compute: (Sum of Container Requests) + (Pod Overhead)
Node Consumption Calculation: When scoring, the scheduler looks at the current node usage + Total Pod Compute.
MostRequested vs LeastRequested: If you are using MostRequested (bin packing) strategy, the inclusion of overhead means the scheduler will actually pack nodes tighter because it accounts for the "wasted" overhead resource, ensuring the user payload is dense relative to the total claimed resources. If you forget this, your bin-packing algorithm will be inaccurate by the margin of your overhead, potentially leaving CPU on the table.

Conclusion :

These scheduling features represent the difference between "running Kubernetes" and "engineering Kubernetes." By leveraging Topology Spread Constraints, you ensure business continuity. By using Pod Overhead, you maintain financial and operational accuracy in mixed runtime environments. And by utilizing Scheduler Profiles, you unlock the ability to tailor the control plane to your specific hardware and business logic without fighting the upstream project.

The scheduler is not a black box; it is a configurable engine. These "hidden gems" are the keys to unlocking its full potential.