DEV Community: Chetan Menge

Mastering DevOps Branching: Your Ultimate Guide to Git Flow, Trunk, Tag-Based, and Hybrid Strategies

Chetan Menge — Tue, 13 Aug 2024 10:10:41 +0000

This guide explores popular Git branching strategies for DevOps environments, focusing on Git Flow, Trunk-Based Development (TDD), Tag-Based, and Hybrid approaches. We'll compare their strengths and weaknesses, highlight challenges they address, and discuss their suitability for web applications and ETL processes / applications.

Branching Strategies:

Git Flow:

Workflow: Separate branches for development (develop), feature development (feature/), hotfixes (hotfix/), and releases (release/*). Features are merged into develop, releases are branched from develop, and hotfixes are applied to both develop and the release branch.
Pros: Clear separation of concerns, promotes stable releases, good for large teams with complex projects.
Cons: Overhead of managing multiple branches, potential merge conflicts.
Challenges Addressed: Feature isolation, release stability, collaboration in large teams.
Suitability: Complex web applications, large-scale ETL projects.

Reference Diagram

Trunk-Based Development (TDD):

Workflow: All development happens directly on the main branch (master). Frequent integrations and deployments ensure a stable codebase.
Pros: Encourages collaboration and rapid feedback, simplifies branch management.
Cons: Requires disciplined development practices and strong CI/CD. Not ideal for large or complex projects.
Challenges Addressed: Streamlined development workflow, continuous integration testing.
Suitability: Small to medium-sized web applications, ETL processes requiring frequent updates.

Tag-Based:

Workflow: Short-lived feature branches are merged directly into the main branch (master). Releases are tagged on the main branch.
Pros: Simple, fast releases, good for small teams or projects with frequent deployments.
Cons: Main branch can become unstable, difficult to rollback changes.
Challenges Addressed: Rapid development cycles, continuous integration and delivery (CI/CD).
Suitability: Simple web applications, small ETL pipelines.

Hybrid:

Workflow: Combines elements of Git Flow and Tag-Based. May involve short-lived feature branches merged into a develop branch before integration with the main branch, or feature branches directly merged to the main branch with frequent tagging for releases.
Pros: Flexible, adaptable to project needs, balances stability with development speed.
Cons: Requires careful planning and discipline to avoid complexity.
Challenges Addressed: Adaptability to project size and complexity, maintaining release stability.
Suitability: Both web applications and ETL projects, depending on specific needs.

Choosing the Right Strategy:

The best strategy hinges on your project's specific needs. Consider these factors:

Project Size and Complexity: Larger projects with multiple developers might benefit from the structure of Git Flow.
Release Frequency: Frequent releases might favor a Tag-Based or Hybrid approach.
Team Experience: A Hybrid approach might require more discipline but offers greater flexibility.
Deployment Frequency: TDD or a Hybrid with frequent main branch merges works well for rapid deployments.

Additional Notes:

CI/CD Integration: All these strategies can benefit from a strong CI/CD pipeline to automate testing, integration, and deployment.

Web Applications vs. ETL Apps: Both strategies can be applied to both web applications and ETL processes. ETL might benefit from more frequent deployments and rollbacks, which could influence the choice.

By understanding these strategies and their implications, developers can choose the approach that best promotes efficient development, stable releases, and collaboration within their DevOps environment.

GCP Cloud Armor - How to Leverage and add extra layer of security

Chetan Menge — Fri, 21 Jun 2024 11:59:31 +0000

In today's digital world, securing your internet-facing applications is paramount. Distributing Denial-of-Service (DDoS) attacks, web application vulnerabilities, and malicious bots can significantly disrupt your services and damage your reputation. Google Cloud Armor offers a robust solution to fortify your application's defences. This blog post, aimed at developers, explores how Cloud Armor bolsters your application security on Google Cloud Platform (GCP).

What is Cloud Armor?

Cloud Armor is a globally-distributed Web Application Firewall (WAF) and DDoS mitigation service offered by GCP. It acts as a security shield, positioned in front of your internet-facing applications, filtering malicious traffic before it reaches your backend servers. Cloud Armor offers a multi-layered defence against various threats, including:

DDoS Attacks: Cloud Armor safeguards your applications from volumetric (L3/L4) and Layer 7 DDoS attacks, ensuring service availability during traffic surges.
Web Application Attacks: Pre-configured WAF rules based on OWASP Top 10 risks help mitigate common web vulnerabilities like SQL injection and cross-site scripting (XSS).

Benefits of Cloud Armor

Enhanced Security: Cloud Armor provides a comprehensive security solution, safeguarding your applications from a broad spectrum of threats.
Improved Performance: By filtering malicious traffic at the edge, Cloud Armor reduces the load on your backend servers, enhancing application performance.
Simplified Management: Cloud Armor offers a user-friendly interface for managing security policies and monitoring traffic patterns.
Global Scale: Cloud Armor's globally distributed network ensures consistent protection across all your GCP regions.

Implementation with a Reference Diagram

Users access your application through the internet.
Traffic is routed through Cloud Load Balancing, which can be integrated with Cloud Armor.
Cloud Armor's WAF engine inspects incoming traffic, filtering out malicious requests based on pre-configured rules or custom policies.
Legitimate traffic is forwarded to your application servers.

Sample Policy

Pros and Cons of using Cloud Armor

Pros:

Robust security against DDoS attacks and web application vulnerabilities.
Improved application performance and availability.
Simplified security management with a user-friendly interface.
Scalable protection that adapts to your application's traffic patterns.

Cons:

Additional cost associated with Cloud Armor usage.
May require configuration adjustments for existing applications.
Might introduce slight latency due to additional processing at the edge.

Cost Considerations

Cloud Armor charges are based on incoming and outgoing request counts. You can leverage GCP's free tier for limited usage. Pay-as-you-go pricing applies for exceeding the free tier limits. Refer to GCP's pricing documentation for detailed cost information https://cloud.google.com/armor/pricing

101- Databricks Model Serving- Saving Cost

Chetan Menge — Wed, 08 May 2024 10:12:21 +0000

Started exploring and trying Databricks instruct LLM. Was going over the Databricks Marketplace and installed and served the model by following steps listed in the Sample provided Notebook.

Was able to serve model successfully and interacted with it very well by proving few prompts. Its was after couple of days, got realised when saw budget alert notification that, planned budget got exceeded way beyond.

Lesson Learned,

LLM Download from Market place is free
Serving LLM - Similar to cloud hosted resources cost saving, there is way to scale down served LLM endpoint when not in use
Model which is accessed from Marketplace can be serve using "Databricks Model Serving" approach which server Model as REST endpoint using serverless compute.

Please find below details with screenshot for reference, for downloading and serving DBRX Model.

Model Download

On Databricks Workspace portal, we can go to Marketplace and search for LLM. E.g search for DBRX models.

Model and its details will be shown as below,

You can select / Click on Get instant access, to download model into your environment.

Validation of Model in Unity Catalog

Once downloaded, model will be available in unity catalog as shown below

If its listed in unity catalog that means model got downloaded and available for use.

Serving Model thru Endpoint

You can go to unity catalog and select specific model e.g. dbrx_instruct. You can create the endpoint and server model by clicking the “Serve this model” button above in the model UI.

Below page will be prompted to select the configuration before serving the model

Saving Cost of Serving Model Endpoint

While serving the model , make sure to expand the Advance Configuration section, which has option of "Scale to Zero" Please refer below screenshot for the details.

If the "scale to zero" is not selected, the minimum charge will depend on the minimum provisioned concurrency specified by the chosen concurrency range.

If ‘scale to zero’ is selected, scale to zero happens automatically after 30 minutes of no requests, at which time the endpoint enters the fully scaled-to-zero (idle) state. You are not charged during this time period. When a new request is made, the endpoint exits this idle state and begins scaling up at which point you begin getting charged.

Reference :-

Model Serving Pricing | Databricks

Databricks Model Serving simplifies the deployment of machine learning models as APIs, enabling real-time predictions within seconds or milliseconds.

databricks.com