DEV Community: Srinivas Ettedi

Google cloud storage and its life cycle Management

Srinivas Ettedi — Mon, 21 Jul 2025 09:48:19 +0000

Google Cloud Storage Lifecycle Management: A Comprehensive Guide

This document provides a comprehensive overview of Google Cloud Storage (GCS) lifecycle management, detailing its functionalities, configuration options, and practical use cases. Lifecycle management is a powerful feature within GCS that automates the process of transitioning objects between storage classes or deleting them based on predefined rules. This automation helps optimize storage costs, improve data governance, and streamline data management workflows. We will explore the various aspects of lifecycle management, including its benefits, configuration methods, and real-world scenarios where it proves invaluable.

What is Google Cloud Storage Lifecycle Management?

Google Cloud Storage lifecycle management is a feature that automatically manages the lifecycle of your objects stored in GCS buckets. It allows you to define rules that specify actions to be taken on objects based on their age, storage class, creation date, or other criteria. These actions can include:

Transitioning to a different storage class: Moving objects to a cheaper storage class (e.g., from Standard to Nearline, Coldline, or Archive) as they become less frequently accessed.

Deleting objects: Permanently removing objects that are no longer needed, such as old logs, temporary files, or outdated backups.

By automating these tasks, lifecycle management helps you:

Reduce storage costs: By moving infrequently accessed data to cheaper storage classes, you can significantly lower your storage bills.

Improve data governance: By automatically deleting old data, you can ensure compliance with data retention policies and reduce the risk of storing unnecessary information.

Simplify data management: Automating lifecycle management tasks frees up your time and resources to focus on other important aspects of your data management strategy.

How Lifecycle Management Works

Lifecycle management rules are defined at the bucket level and apply to all objects within that bucket (or a subset of objects based on object name prefixes). Each rule consists of a condition and an action.

Conditions: Specify when the action should be taken. Common conditions include:

Age: The number of days since the object was created.

CreatedBefore: A specific date before which the object was created.

NumberOfNewerVersions: The number of newer versions of the object that exist.

IsLive: Whether the object is the live version (relevant for versioned buckets).

MatchesStorageClass: The current storage class of the object.

Prefix: A prefix that the object name must match.

DaysSinceCustomTime: The number of days since a custom time was set on the object.

CustomTimeBefore: A specific date before which the custom time was set on the object.

Actions: Specify what should happen when the condition is met. Common actions include:

Delete: Permanently deletes the object.

SetStorageClass: Transitions the object to a different storage class.

AbortIncompleteMultipartUpload: Aborts incomplete multipart uploads.

SetCustomTime: Sets a custom time on the object.

When an object meets the conditions of a lifecycle rule, the specified action is automatically executed. GCS periodically evaluates objects against the defined rules and applies the appropriate actions.

Configuring Lifecycle Management

You can configure lifecycle management rules using several methods:

Google Cloud Console: The web-based interface provides a user-friendly way to create and manage lifecycle rules.

gsutil command-line tool: A powerful command-line tool for interacting with GCS, allowing you to define rules in a YAML file and apply them to buckets.

Cloud Storage API: Programmatically manage lifecycle rules using the Cloud Storage API in various programming languages.

Terraform: Infrastructure-as-code tool to define and manage your lifecycle rules alongside your other cloud resources.

Here's an example of a lifecycle rule defined in a YAML file for use with gsutil:

rules:

action:
type: Delete
condition:
age: 365

action:
type: SetStorageClass
storageClass: NEARLINE
condition:
age: 30

action:
type: SetStorageClass
storageClass: COLDLINE
condition:
age: 90

This rule set does the following:

Deletes objects older than 365 days.

Transitions objects older than 30 days to the Nearline storage class.

Transitions objects older than 90 days to the Coldline storage class.

To apply this rule to a bucket named my-bucket, you would use the following gsutil command:

gsutil lifecycle set lifecycle.yaml gs://my-bucket

Use Cases for Lifecycle Management

Lifecycle management is a versatile tool that can be used in a variety of scenarios. Here are some common use cases:

Archiving Logs: Automatically move old log files to cheaper storage classes (Coldline or Archive) after a certain period. This is useful for retaining logs for compliance or auditing purposes without incurring high storage costs.

Managing Backups: Delete old backups after a specified retention period. This helps to reduce storage costs and ensure that you are only storing the backups that you need.

Temporary Data Storage: Automatically delete temporary files or data that is no longer needed. This is useful for cleaning up temporary storage areas and preventing them from filling up with unnecessary data.

Compliance and Data Retention: Enforce data retention policies by automatically deleting data after a certain period. This helps to ensure compliance with regulatory requirements.

Media Asset Management: Transition infrequently accessed media assets (images, videos, audio files) to cheaper storage classes. This is useful for managing large media libraries where some assets are rarely accessed.

Big Data Analytics: Move older datasets to cheaper storage classes after they have been analyzed. This helps to reduce storage costs for large datasets that are only accessed periodically.

Software Development: Delete old build artifacts or temporary files after a certain period. This helps to keep your development environment clean and organized.

Best Practices for Lifecycle Management

Start with a plan: Before implementing lifecycle management, carefully consider your data retention policies and storage requirements.

Test your rules: Before applying lifecycle rules to production data, test them in a non-production environment to ensure that they are working as expected.

Monitor your rules: Regularly monitor your lifecycle rules to ensure that they are still meeting your needs and that they are not causing any unexpected issues.

Use object prefixes: Use object prefixes to apply lifecycle rules to specific subsets of objects within a bucket.

Consider versioning: If you are using object versioning, be aware that lifecycle rules can affect both live and non-current versions of objects.

Understand storage class transitions: Be aware of the retrieval costs associated with different storage classes. While cheaper storage classes can save you money on storage costs, they may incur higher retrieval costs if you need to access the data frequently.

Use custom time: Leverage the custom time feature to control the lifecycle of objects based on a specific event or date, rather than just the creation date.

Conclusion

Google Cloud Storage lifecycle management is a powerful tool for automating the management of your data in GCS. By defining rules that specify actions to be taken on objects based on their age, storage class, or other criteria, you can significantly reduce storage costs, improve data governance, and simplify data management workflows. By understanding the various features and best practices of lifecycle management, you can effectively leverage it to optimize your storage strategy and improve your overall data management efficiency.

Google cloud Run Cloud Build and Functions

Srinivas Ettedi — Mon, 21 Jul 2025 09:44:41 +0000

Cloud Run, Cloud Build, and Cloud Functions

This blog post explores the power of combining Google Cloud Run, Cloud Build, and Cloud Functions to create a robust and scalable serverless architecture. We'll delve into each service individually, highlighting their strengths and use cases, and then demonstrate how they can be integrated to build a complete CI/CD pipeline for deploying and managing serverless applications. This trifecta offers developers a streamlined and efficient way to build, deploy, and scale applications without the complexities of managing underlying infrastructure.

Cloud Run: Containerized Serverless Execution

Cloud Run is a fully managed compute platform that enables you to run stateless containers invocable via HTTP requests. It abstracts away the complexities of infrastructure management, allowing you to focus solely on writing code. Key features of Cloud Run include:

Container-based: Cloud Run leverages containers, providing flexibility in choosing programming languages, libraries, and system dependencies. You can package your application into a Docker image and deploy it to Cloud Run.

Fully Managed: Google Cloud handles all the underlying infrastructure, including scaling, patching, and security. This reduces operational overhead and allows you to focus on development.

Scalability: Cloud Run automatically scales your application based on incoming traffic, ensuring optimal performance and cost efficiency. It scales down to zero when there are no requests, minimizing costs during periods of inactivity.

HTTP-Driven: Cloud Run services are invoked via HTTP requests, making them ideal for building APIs, web applications, and event-driven systems.

Integration: Cloud Run integrates seamlessly with other Google Cloud services, such as Cloud Build, Cloud Functions, and Cloud Logging.

Use Cases:

Web Applications: Hosting static websites or dynamic web applications.

APIs: Building RESTful APIs for mobile apps, web applications, or other services.

Event Processing: Handling events from Cloud Pub/Sub or other event sources.

Microservices: Deploying individual microservices as independent Cloud Run services.

Cloud Build: Automated CI/CD Pipelines

Cloud Build is a fully managed CI/CD (Continuous Integration/Continuous Delivery) platform that automates the process of building, testing, and deploying software. It allows you to define build pipelines using YAML configuration files, specifying the steps required to transform source code into deployable artifacts. Key features of Cloud Build include:

Automated Builds: Cloud Build automatically triggers builds based on code changes in repositories like Cloud Source Repositories, GitHub, or Bitbucket.

Container-based Builds: Build steps are executed within Docker containers, providing a consistent and reproducible build environment.

Customizable Pipelines: You can define custom build steps using Docker images, allowing you to integrate any tool or process into your pipeline.

Integration with Google Cloud: Cloud Build integrates seamlessly with other Google Cloud services, such as Cloud Run, Cloud Functions, and Google Kubernetes Engine (GKE).

Security: Cloud Build provides security features such as access control and vulnerability scanning.

Use Cases:

Continuous Integration: Automatically building and testing code changes whenever they are committed to a repository.

Continuous Delivery: Automatically deploying applications to various environments, such as development, staging, and production.

Infrastructure as Code: Building and deploying infrastructure changes using tools like Terraform or Ansible.

Automated Testing: Running unit tests, integration tests, and end-to-end tests as part of the build process.

Cloud Functions: Event-Driven Serverless Functions

Cloud Functions is a serverless execution environment for building and connecting cloud services. It allows you to write single-purpose, event-driven functions that are executed in response to events from various sources, such as Cloud Storage, Cloud Pub/Sub, or HTTP requests. Key features of Cloud Functions include:

Event-Driven: Cloud Functions are triggered by events, making them ideal for building event-driven architectures.

Serverless: Google Cloud manages the underlying infrastructure, allowing you to focus on writing code.

Scalability: Cloud Functions automatically scale based on the number of incoming events, ensuring optimal performance and cost efficiency.

Pay-per-Use: You only pay for the compute time consumed by your functions, making it a cost-effective solution for event processing.

Integration: Cloud Functions integrates seamlessly with other Google Cloud services, such as Cloud Storage, Cloud Pub/Sub, and Cloud Firestore.

Use Cases:

Data Processing: Processing data uploaded to Cloud Storage.

Event Handling: Responding to events from Cloud Pub/Sub.

API Endpoints: Creating simple API endpoints for mobile apps or web applications.

Background Tasks: Performing background tasks, such as sending emails or generating reports.

Integrating Cloud Run, Cloud Build, and Cloud Functions

The true power of these services lies in their ability to be integrated into a cohesive serverless architecture. Here's a common scenario:

Code Change: A developer commits code changes to a repository (e.g., GitHub).

Cloud Build Trigger: Cloud Build is triggered by the code change.

Build Process: Cloud Build executes a build pipeline defined in a YAML configuration file. This pipeline might include steps such as:

Building a Docker image of the application.

Running unit tests and integration tests.

Pushing the Docker image to Container Registry.

Cloud Run Deployment: Cloud Build deploys the new Docker image to Cloud Run, updating the service with the latest version of the application.

Cloud Functions Trigger (Optional): A Cloud Function can be triggered by the Cloud Run deployment to perform post-deployment tasks, such as updating a database or sending a notification.

Example Scenario: Image Resizing Service

Let's imagine we want to build a simple image resizing service.

Cloud Function (Trigger): A Cloud Function is triggered when a new image is uploaded to a Cloud Storage bucket.

Cloud Function (Logic): The Cloud Function resizes the image and stores the resized image in another Cloud Storage bucket.

Cloud Run (API): A Cloud Run service provides an API endpoint to retrieve the resized images.

Cloud Build (CI/CD): Cloud Build automates the process of building, testing, and deploying the Cloud Function and Cloud Run service.

Benefits of this Integration:

Automated Deployment: Cloud Build automates the entire deployment process, reducing manual effort and the risk of errors.

Scalability: Cloud Run and Cloud Functions automatically scale based on demand, ensuring optimal performance.

Cost Efficiency: You only pay for the resources you use, minimizing costs during periods of inactivity.

Simplified Management: Google Cloud manages the underlying infrastructure, allowing you to focus on building and deploying your application.

Faster Development Cycles: Automated CI/CD pipelines enable faster development cycles and quicker time to market.

Conclusion

Cloud Run, Cloud Build, and Cloud Functions provide a powerful and versatile platform for building serverless applications. By combining these services, you can create a robust and scalable architecture that simplifies development, reduces operational overhead, and optimizes costs. Embracing this serverless trifecta allows developers to focus on building innovative solutions without being bogged down by infrastructure management. As serverless computing continues to evolve, mastering these tools will be crucial for building modern, cloud-native applications.

Ingress and ingress controller

Srinivas Ettedi — Mon, 21 Jul 2025 09:41:18 +0000

What is Ingress?
In simple terms, Ingress is an API object in Kubernetes that manages external access to your services, typically over HTTP/HTTPS. Instead of exposing each service with a separate Load Balancer or Node Port, Ingress lets you define rules to route traffic based on hostnames, paths, etc.

Example:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: example-ingress
spec:
rules:
- host: app.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: my-app-service
port:
number: 80

⚙️ What is an Ingress Controller?
Ingress is just a set of rules. To enforce those rules, you need an Ingress Controller — a specialized LoadBalancer running inside your cluster. It listens to changes in Ingress resources and updates its config accordingly.

Popular Ingress Controllers:

NGINX Ingress Controller (most widely used)

Traefik

HAProxy

Istio Gateway (if you're using service mesh)

🚦Why Use Ingress?
✅ Centralized traffic management

✅ SSL/TLS termination (HTTPS)

✅ Path-based or host-based routing

✅ Easy integration with Let's Encrypt (via cert-manager)

✅ Clean URLs and security policies

🔐 Pro Tip: Secure Your Ingress
Use TLS with certificates (automated via cert-manager)

Limit access with annotations or network policies

Use external authentication (OAuth2 proxy, SSO)

Enable rate-limiting and Web Application Firewall (WAF)

🛠 Common Issues
❌ Ingress resource created but not working? Check if an Ingress Controller is deployed!

🔄 Changed Ingress config not updating? Look at the controller logs (kubectl logs ).

📶 404 errors? Check your paths, service names, and port definitions.