DEV Community: rowjay007

Go and PocketBase: Shortified!

rowjay007 — Wed, 08 Oct 2025 16:09:45 +0000

Whether you're building your first web service or your hundredth microservice, the URL shortener remains a compelling case study. It's conceptually simple, quick to prototype, yet deceptively complex at scale.

A single handler that accepts a URL, generates a random string, stores both in a database, and returns the short code. This works for demos, but what happens when that prototype becomes production-critical? When you need robust error handling, input validation, and graceful failure modes?

That gap between "works on my machine" and "scales in production" is where our journey begins.

Together, we'll build a URL shortening service using Go and PocketBase, transforming a monolithic handler into a clean, layered architecture. I'll show you how to structure Go services using clean architecture principles, implement structured error handling, design stable API contracts, and build resilient external integrations. By the end, you'll have both a complete service and a blueprint for your next maintainable Go application.

The Monolithic Handler: A Lesson in Technical Debt

Let's start with the honest reality of version one. Our goal is to get a functional endpoint up and running as quickly as possible. Using Gin's simplicity, we naturally gravitate toward putting everything in a single handler. The result: a function doing far too much work.

Consider this initial createShortURL handler:

func createShortURL(c *gin.Context) {
    var request struct {
        URL string `json:"url"`
    }
    if err := c.BindJSON(&request); err != nil {
        c.JSON(400, gin.H{"error": "invalid request"})
        return
    }

    if !strings.HasPrefix(request.URL, "http") {
        c.JSON(400, gin.H{"error": "invalid URL"})
        return
    }

    shortCode := utils.GenerateRandomString(6)

    _, err := db.Exec("INSERT INTO urls (url, short_code) VALUES (?, ?)", request.URL, shortCode)
    if err != nil {
        log.Printf("db error: %v", err)
        c.JSON(500, gin.H{"error": "internal server error"})
        return
    }

    c.JSON(200, gin.H{"short_code": shortCode})
}

This code works, and we ship it. But we've just created insurmountable technical debt. Let's examine the specific engineering failures we've introduced:

Untestable: Unit testing requires mocking the entire Gin context and database layer—or maintaining a live database. Testing business logic in isolation becomes impossible.
Violates Single Responsibility: This function handles HTTP decoding, input validation, ID generation, database interaction, and HTTP encoding. Five distinct responsibilities in one place.
Framework Coupling: Business logic is inseparable from Gin. Exposing the same functionality via gRPC or CLI requires complete rewrites.
Opaque Failures: Generic 500 errors provide no diagnostic information. Database failures, constraint violations, and application bugs all look identical.

This is the technical debt that will cripple our project. The first step to paying it down is establishing clear boundaries. This isn't about adding unnecessary complexity—it's about creating a structure that enables future growth and maintainability.

The Power of Delegation

Now, let's look at how we can build that same handler using a clean architecture approach. Notice what it doesn't do:

func (h *URLHandler) CreateShortURL(c *gin.Context) {
    var req dto.CreateURLRequest
    if err := c.ShouldBindJSON(&req); err != nil {
        h.handleServiceError(c, serviceErrors.NewBadRequestError("handler.CreateShortURL", "invalid request body", err))
        return
    }

    response, err := h.service.CreateShortURL(c.Request.Context(), &req)
    if err != nil {
        h.handleServiceError(c, err)
        return
    }

    c.JSON(http.StatusCreated, response)
}

This is a night-and-day difference. Our new handler is lean, focused, and incredibly simple. Its only responsibilities are to decode the request, call the service, and encode the response. It contains no business logic, no validation, and no database calls.

The benefits are immediate:

Testability: We can easily test this handler by providing a mock URLService. We don't need a database or complex setup. We can verify its behavior in complete isolation.
Maintainability: If we need to change how short codes are generated, we don't touch the handler. If we need to add a new validation rule, we don't touch the handler. Its purpose is stable and unchanging.
Clarity: A new engineer can look at this and immediately grasp the flow of control without getting bogged down in implementation details.

By adopting this clean, delegated approach, we've already made our system more robust and maintainable. Now, let's explore the architecture that makes this clean separation possible.

Layered Architecture: Enforcing Separation of Concerns

To solve the problems of our prototype, we need to enforce a strict separation of concerns. We'll adopt a layered architecture inspired by the principles of Clean Architecture. This isn't an academic exercise—it's a pragmatic approach to building resilient software.

The core principle is the Dependency Rule: all dependencies must point inwards, toward the core business logic. The outer layers know about the inner layers, but the inner layers know nothing about the outer ones.

Our project structure reflects these layers:

internal/
├── handlers/        # Layer 4: The Web Framework (Gin)
├── services/        # Layer 3: The Business Logic
├── repository/      # Layer 2: The Data Access Abstraction
└── models/          # Layer 1: The Core Domain Entities

// Supporting Packages
├── dto/             # API Contracts (Request/Response Structs)
├── errors/          # Custom Error Types
└── validator/       # Input Validation Logic

This structure is enforced using Go's interfaces, which allow us to implement the Dependency Inversion Principle. Instead of high-level modules depending on low-level modules, both depend on abstractions (interfaces).

The Repository Interface (The Data Contract):
This interface defines what we can do with our data, but not how. It's the contract between our business logic and the persistence layer.

type URLRepository interface {
    Create(ctx context.Context, shortURL *models.ShortURL) error
    GetByShortCode(ctx context.Context, shortCode string) (*models.ShortURL, error)
    ExistsByShortCode(ctx context.Context, shortCode string) (bool, error)
    UpdateAccessCount(ctx context.Context, shortCode string) error
}

The Service Interface (The Business Logic Contract):
This defines the core capabilities of our application, completely independent of any web framework.

type URLService interface {
    CreateShortURL(ctx context.Context, req *dto.CreateURLRequest) (*dto.CreateURLResponse, error)
    GetOriginalURL(ctx context.Context, shortCode string) (*dto.GetURLResponse, error)
    GetStats(ctx context.Context, shortCode string) (*dto.GetStatsResponse, error)
}

Wiring It All Together (Dependency Injection):
Our main.go function becomes the composition root. It's the only place in the application that knows about the concrete implementations of these interfaces. It builds the dependency graph and injects the concrete types.

func main() {
    config := loadConfig()
    pocketBaseClient := database.NewPBClient(config.PocketBaseURL)

    urlRepo := repository.NewURLRepository(pocketBaseClient)
    urlService := services.NewURLService(urlRepo, validator.NewURLValidator())
    urlHandler := handlers.NewURLHandler(urlService)

    router := gin.Default()
    router.POST("/api/v1/shorten", urlHandler.CreateShortURL)
    router.GET("/api/v1/:code", urlHandler.GetOriginalURL)
    router.GET("/api/v1/:code/stats", urlHandler.GetStats)

    router.Run(":8080")
}

This is a paradigm shift. Each layer can now be tested in isolation by mocking its dependencies—the service layer mocks the repository, the handler layer mocks the service. We have broken the tight coupling that made our prototype so fragile.

This architecture allows us to swap out components with minimal impact. If we decide to move from PocketBase's REST API to a direct SQLite driver or even to a different database like PostgreSQL, we only need to write a new implementation of the URLRepository interface. The service and handler layers remain completely untouched. This is the definition of a maintainable and scalable system.

The Flow of a Request: A Practical Walkthrough

Let's trace the lifecycle of a single API call to see how our layers work together. A user wants to create a new short URL by sending a POST request to /api/v1/shorten.

The Handler (handlers/url_handler.go):
- The Gin router receives the incoming HTTP request and routes it to our CreateShortURL handler.
- The handler's first and only job is to manage the HTTP interaction. It uses c.ShouldBindJSON() to decode the JSON payload into a dto.CreateURLRequest struct.
- If binding fails, the handler immediately returns a 400 Bad Request response. It knows nothing about why the binding failed, only that the request was malformed.
- If binding is successful, the handler calls the service layer: h.service.CreateShortURL(ctx, &req).
- The handler then waits for the service to return either a dto.CreateURLResponse or an error.
- If an error is returned, it passes the error to the handleServiceError function to be translated into the correct HTTP status code and response body.
- If a response is returned, it serializes the DTO to JSON and sends it back to the client with a 201 Created status code.
The Service (services/url_service.go):
- The CreateShortURL method in the service receives the dto.CreateURLRequest.
- Its first action is to perform validation by calling the validator.ValidateURL() and validator.ValidateShortCode() methods. If validation fails, it returns a ServiceError with the code ErrorCodeValidation.
- Next, it orchestrates the business logic. If the user provided a custom code, it calls repo.ExistsByShortCode(). If the code exists, it returns a ServiceError with the code ErrorCodeDuplicate.
- If no custom code was provided, it enters a loop to generate a unique short code, calling utils.GenerateShortCode() and repo.ExistsByShortCode() until a unique code is found.
- Once it has a valid URL and a unique short code, it constructs a models.ShortURL domain object.
- It then calls repo.Create(ctx, &shortURL) to persist the new record.
- Finally, it maps the resulting models.ShortURL (now populated with an ID and timestamps from the database) to a dto.CreateURLResponse and returns it to the handler.
The Repository (repository/url_repository.go):
- The Create method receives the models.ShortURL object.
- It maps this domain model to a pocketBaseCreateRequest struct, which matches the schema expected by the PocketBase API.
- It constructs an http.Request with the appropriate method, URL, and body, ensuring the context is passed along for cancellation and timeouts.
- It executes the HTTP request against the PocketBase server.
- It then interprets the HTTP response. A 201 Created is a success. A 409 Conflict is translated into our serviceErrors.NewDuplicateError. Any other non-2xx status is translated into a serviceErrors.NewInternalError.
- On success, it decodes the response body from PocketBase to get the database-generated ID and timestamps, and updates the original models.ShortURL object with this information before returning.

This clear, unidirectional flow is the hallmark of a well-architected system. Each layer has a specific job, and it communicates with the layers adjacent to it through well-defined contracts (interfaces and DTOs). This makes the system easy to reason about, debug, and extend.

Defining the Lines: Stable API Contracts with DTOs

A critical discipline in building robust APIs is the strict separation of your internal data structures (domain models) from your external data structures (the API contract). Let's look at the common anti-pattern first.

The Anti-Pattern: Exposing Your Database Models Directly

In a rush to get things working, it's incredibly tempting to use your internal models.ShortURL struct directly in your handler for both request binding and response serialization.

func (h *URLHandler) CreateShortURL(c *gin.Context) {
    var url models.ShortURL
    if err := c.ShouldBindJSON(&url); err != nil {
        c.JSON(http.StatusBadRequest, gin.H{"error": "invalid request"})
        return
    }

    c.JSON(http.StatusCreated, url)
}

This shortcut seems efficient, but it creates a tight, brittle coupling between your public API and your private database schema. This leads to several serious problems:

The Problem with Direct Exposure:

Breaking Changes: If you rename a database column, you've just introduced a breaking change to your public API. Your API becomes a fragile mirror of your database schema.
Information Leakage: Your internal models may contain fields that are irrelevant or sensitive and should not be exposed to the client, such as internal flags, metadata, or hashed values.
Use-Case Mismatch: The data structure that's optimal for your database is rarely the optimal structure for a specific API response. A GET request might need a subset of fields, while a stats endpoint might need aggregated data.

The Solution: Defining Explicit API Contracts with DTOs

To solve these problems, you must treat your API as a public, versioned contract. The best way to do this is with Data Transfer Objects (DTOs). These are simple, plain structs whose sole purpose is to define the exact shape of your API's requests and responses. They live in their own dto/ package and are the public face of your service.

Domain Model (internal source of truth):

package models

import "time"

type ShortURL struct {
    ID          string    `db:"id"`
    URL         string    `db:"url"`
    ShortCode   string    `db:"short_code"`
    AccessCount int64     `db:"access_count"`
    Created     time.Time `db:"created"`
    Updated     time.Time `db:"updated"`
}

API Contract (external interface):

package dto

import "time"

type CreateURLRequest struct {
    URL        string  `json:"url" validate:"required,url,max=2048"`
    CustomCode *string `json:"customCode,omitempty" validate:"omitempty,min=4,max=20,alphanum"`
}

type CreateURLResponse struct {
    ID        string    `json:"id"`
    URL       string    `json:"url"`
    ShortCode string    `json:"shortCode"`
    CreatedAt time.Time `json:"createdAt"`
}

type GetStatsResponse struct {
    ShortCode   string    `json:"shortCode"`
    AccessCount int64     `json:"accessCount"`
    CreatedAt   time.Time `json:"createdAt"`
}

The service layer acts as the mediator, responsible for mapping between these two worlds. It takes an incoming CreateURLRequest DTO, validates it, and then maps its data into a models.ShortURL domain object to be sent to the repository. Conversely, when fetching data, it takes a models.ShortURL from the repository and maps it to a CreateURLResponse or GetStatsResponse DTO before returning it to the handler.

This mapping is a small price to pay for immense flexibility. It decouples our API contract from our database schema, allowing each to evolve independently. It's a crucial step in building an API that is stable, secure, and designed for its consumers.

The Mapper Pattern: A Clean Implementation

To keep our service layer clean, we can even introduce explicit mapper functions. While not strictly necessary for a project of this size, it's a pattern that scales well. These mappers can live in the service layer or even their own package.

func toCreateURLResponse(url *models.ShortURL) *dto.CreateURLResponse {
    return &dto.CreateURLResponse{
        ID:        url.ID,
        URL:       url.URL,
        ShortCode: url.ShortCode,
        CreatedAt: url.Created,
    }
}

func fromCreateURLRequest(req *dto.CreateURLRequest) (*models.ShortURL, error) {
    return &models.ShortURL{
        URL:       req.URL,
        ShortCode: "",
    }, nil
}

By using mappers, the service layer's intent becomes crystal clear:

func (s *urlServiceImpl) CreateShortURL(ctx context.Context, req *dto.CreateURLRequest) (*dto.CreateURLResponse, error) {
    if err := s.validator.ValidateURL(req.URL); err != nil {
        return nil, err
    }

    shortURL, err := fromCreateURLRequest(req)
    if err != nil {
        return nil, serviceErrors.NewInternalError("service.Create", "failed to map request", err)
    }

    shortURL.ShortCode = s.generateUniqueShortCode(ctx)

    if err := s.repo.Create(ctx, shortURL); err != nil {
        return nil, err
    }

    return toCreateURLResponse(shortURL), nil
}

This level of discipline ensures that the boundaries between layers remain pristine. The DTOs define the public-facing language of our API, while the domain models define the internal language of our business logic. The service layer acts as the professional translator between the two.

Beyond 500: A Strategy for Meaningful Error Handling

One of the most telling signs of a fragile system is its error handling. Let's examine the common anti-pattern that leads to opaque, unhelpful APIs.

The Anti-Pattern: Generic Errors and String Comparisons

In a simple implementation, it's common to see error handling like this:

func (s *urlServiceImpl) GetOriginalURL(ctx context.Context, shortCode string) (string, error) {
    url, err := s.repo.GetByShortCode(ctx, shortCode)
    if err != nil {
        if err == sql.ErrNoRows {
            return "", fmt.Errorf("not found")
        }
        return "", fmt.Errorf("internal error")
    }
    return url.URL, nil
}

_, err := h.service.GetOriginalURL(c.Request.Context(), shortCode)
if err != nil {
    if err.Error() == "not found" {
        c.JSON(http.StatusNotFound, gin.H{"error": "URL not found"})
    } else {
        c.JSON(http.StatusInternalServerError, gin.H{"error": "An internal error occurred"})
    }
    return
}

This approach is a debugging nightmare for several reasons:

Loss of Context: The original error from the database (sql.ErrNoRows) is discarded. We lose the valuable context of what exactly went wrong.
Fragile String Comparisons: Relying on err.Error() == "not found" is incredibly brittle. If a developer changes the error message in the service, the handler's logic breaks. This is a common source of bugs.
Ambiguity: A generic "internal error" tells the client and our operations team nothing. Is the database down? Is there a bug in our query? We have no way to know without digging through logs.

The Solution: Errors as First-Class, Structured Citizens

To fix this, you must elevate errors to be a core, designed part of your system. Instead of passing around simple strings, we'll create a dedicated errors/ package to define a structured, custom error type that will be used throughout the application.

The ServiceError Struct:

package errors

type ErrorCode int

const (
    ErrorCodeNotFound ErrorCode = iota + 1
    ErrorCodeDuplicate
    ErrorCodeValidation
    ErrorCodeInternal
)

type ServiceError struct {
    Op      string    // Operation that failed
    Code    ErrorCode // Machine-readable type
    Message string    // Human-readable message
    Err     error     // Wrapped original error
}

func (e *ServiceError) Error() string {
    return fmt.Sprintf("%s: %s", e.Op, e.Message)
}

func NewNotFoundError(op, message string) *ServiceError {
    return &ServiceError{Op: op, Code: ErrorCodeNotFound, Message: message}
}

With this structure in place, our repository and service layers can now return rich, meaningful errors. Instead of fmt.Errorf("not found"), our repository now returns serviceErrors.NewNotFoundError("repository.GetByShortCode", "short URL not found").

This provides several key advantages:

Context: We know the exact operation (Op) that failed.
Classification: We have a machine-readable Code that allows us to programmatically handle different error types.
Clarity: We have a clear Message intended for the end-user.
Traceability: We can wrap the original low-level error (Err) for detailed logging without exposing it to the client.

Centralized Error Handling in the Handler:
The true power of this pattern is realized in the handler layer, where we can create a single, centralized function to translate any ServiceError into the correct HTTP response.

func (h *URLHandler) handleServiceError(c *gin.Context, err error) {
    var serviceErr *serviceErrors.ServiceError
    if errors.As(err, &serviceErr) {
        statusCode := map[serviceErrors.ErrorCode]int{
            serviceErrors.ErrorCodeNotFound:   http.StatusNotFound,
            serviceErrors.ErrorCodeDuplicate:  http.StatusConflict,
            serviceErrors.ErrorCodeValidation: http.StatusBadRequest,
        }[serviceErr.Code]
        if statusCode == 0 {
            statusCode = http.StatusInternalServerError
        }

        log.Error().Err(serviceErr.Err).Str("op", serviceErr.Op).Msg(serviceErr.Message)
        c.JSON(statusCode, dto.ErrorResponse{Error: serviceErr.Message})
        return
    }

    log.Error().Err(err).Msg("Unexpected error")
    c.JSON(http.StatusInternalServerError, dto.ErrorResponse{Error: "Internal server error"})
}

This function is a game-changer. It ensures that all API error responses are consistent and predictable. It guarantees that we log rich, contextual information for debugging while never leaking implementation details to the client. This systematic approach to error handling transformed our service from brittle to resilient.

The Power of Structured Logging

Notice the logging line within our handleServiceError function:

log.Error().Err(serviceErr.Err).Str("op", serviceErr.Op).Msg(serviceErr.Message)

This is not an accident. We are using the zerolog library to produce structured, JSON-formatted logs. When this error is logged, it won't be a simple, unparseable string. It will be a rich, machine-readable JSON object:

{
  "level": "error",
  "op": "repository.GetByShortCode",
  "error": "record not found",
  "message": "short URL not found",
  "time": "2023-10-27T10:00:00Z"
}

This is a massive operational advantage. We can now ship these logs to a centralized logging platform (like Elasticsearch, Datadog, or Logz.io) and perform powerful queries. We can easily filter for all errors originating from a specific operation (op), count the occurrences of different error codes, and set up automated alerts for spikes in internal server errors. This is the difference between reactive debugging (grepping through raw text files) and proactive observability.

By treating errors as structured data, both in our API responses and in our internal logs, we build a system that is not only more reliable but also far easier to monitor and maintain at scale.

First Line of Defense: Proactive and Centralized Validation

Defensive programming is key to building secure and reliable systems. The best way to handle invalid data is to reject it at the earliest possible moment. Let's examine the common anti-pattern that leads to security holes and inconsistent behavior.

The Anti-Pattern: Scattered and Inconsistent Validation

When you're moving quickly, it's tempting to sprinkle validation checks directly within your service or handler logic wherever they seem to be needed.

func (s *urlServiceImpl) CreateShortURL(ctx context.Context, req *dto.CreateURLRequest) (*dto.CreateURLResponse, error) {
    if len(req.URL) > 2048 {
        return nil, serviceErrors.NewValidationError("service.Create", "URL exceeds maximum length", nil)
    }
    if !strings.HasPrefix(req.URL, "http") {
        return nil, serviceErrors.NewValidationError("service.Create", "Invalid URL format", nil)
    }

    // Business logic mixed with validation
    shortCode, err := s.generateUniqueShortCode(ctx)
    // ...
}

This approach quickly becomes a maintenance nightmare:

Violation of DRY (Don't Repeat Yourself): If multiple service methods need to validate a URL, you'll end up copying and pasting the same validation logic, leading to inconsistencies when one is updated and the other is forgotten.
Mixing Concerns: The service layer's job is to orchestrate business logic, not to be an expert in the minutiae of what constitutes a valid URL string. This mixing of responsibilities makes the code harder to read and reason about.
Inconsistent Rules: Without a central authority for validation, it's easy for different parts of the application to end up with slightly different rules, leading to unpredictable behavior for your users.

The Solution: The Gatekeeper Pattern

To solve this, we'll create a dedicated validation layer. The validator/ package is responsible for one thing and one thing only: validating raw, untrusted input from the outside world. It acts as a gatekeeper, ensuring that no invalid data ever reaches your business logic.

This validator is a pure function of its inputs, with no side effects and no knowledge of business logic or databases.

package validator

import (
    "net/url"
    "strings"
    "github.com/rowjay/url-shortening-service/internal/constants"
    "github.com/rowjay/url-shortening-service/internal/errors"
)

type URLValidator struct {
    blockedDomains []string
}

func NewURLValidator() *URLValidator {
    return &URLValidator{blockedDomains: constants.BlockedDomains}
}

func (v *URLValidator) ValidateURL(rawURL string) error {
    if len(rawURL) > constants.MaxURLLength {
        return errors.NewValidationError("validator.ValidateURL", "URL exceeds maximum length", nil)
    }

    parsed, err := url.ParseRequestURI(rawURL)
    if err != nil || (parsed.Scheme != "http" && parsed.Scheme != "https") {
        return errors.NewValidationError("validator.ValidateURL", "Invalid URL format", err)
    }

    for _, domain := range v.blockedDomains {
        if strings.Contains(parsed.Host, domain) {
            return errors.NewValidationError("validator.ValidateURL", "URL domain is blocked", nil)
        }
    }

    return nil
}

The URLService is the consumer of this validator. Before performing any action, it first ensures the input is sane. This is the Gatekeeper Pattern.

func (s *urlServiceImpl) CreateShortURL(ctx context.Context, req *dto.CreateURLRequest) (*dto.CreateURLResponse, error) {
    if err := s.validator.ValidateURL(req.URL); err != nil {
        return nil, err
    }
    if req.CustomCode != nil {
        if err := s.validator.ValidateShortCode(*req.CustomCode); err != nil {
            return nil, err
        }
    }

    shortCode := req.CustomCode
    if shortCode == nil {
        code, err := s.generateUniqueShortCode(ctx)
        if err != nil {
            return nil, err
        }
        shortCode = &code
    }

    shortURL := &models.ShortURL{
        URL:       req.URL,
        ShortCode: *shortCode,
    }

    if err := s.repo.Create(ctx, shortURL); err != nil {
        return nil, err
    }

    return &dto.CreateURLResponse{
        ID:        shortURL.ID,
        URL:       shortURL.URL,
        ShortCode: shortURL.ShortCode,
        CreatedAt: shortURL.Created,
    }, nil
}

By centralizing validation, we ensure that our rules are applied consistently across all use cases. If we need to update our URL validation logic (e.g., to block more domains), we only need to change it in one place. This makes the system more secure and easier to maintain. The service layer can trust that any data it receives has already been vetted, allowing it to focus purely on business logic.

Types of Validation Implemented

Our URLValidator is responsible for several distinct checks, each serving a critical purpose:

Syntactical Validation: We use url.ParseRequestURI to ensure the URL is structurally valid according to RFC 3986. This is the first line of defense against malformed input.
Protocol Validation: We explicitly check that the URL scheme is either http or https. This prevents abuse with other schemes like ftp://, file://, or potentially malicious custom schemes.
Semantic Validation (Domain Blocking): This is a crucial security feature. Our validator checks the URL's host against a configurable list of blocked domains. This allows us to prevent our service from being used to shorten links to known malware sites, phishing pages, or internal resources. This list is managed in our constants package, making it easy to update without changing the validation logic itself.
Length Constraints: We enforce a maximum length for both the original URL and any custom short codes. This is a simple but effective way to prevent denial-of-service attacks that attempt to exhaust our database storage with absurdly long strings.
Character Set Validation: For custom short codes, we enforce an alphanumeric character set. This prevents users from injecting special characters, control characters, or potentially harmful script tags into our URLs.

By performing these checks in a dedicated, centralized validator, we create a single source of truth for our application's input rules. This makes the system more secure, more predictable, and easier to audit.

Talking to the Outside World: A Resilient PocketBase Repository

With our internal structure sorted, it's time to connect to the outside world. The repository layer is the bridge between our application's domain and the persistence layer. It's also a place where it's easy to create leaky abstractions.

The Anti-Pattern: The Leaky Repository

A common mistake is to write a repository that doesn't fully abstract away the details of the external service it's communicating with. The logic of HTTP requests, status codes, and external data formats can bleed into the service layer.

func (s *urlServiceImpl) CreateShortURL(ctx context.Context, req *dto.CreateURLRequest) (*dto.CreateURLResponse, error) {
    shortURL := &models.ShortURL{
        URL:       req.URL,
        ShortCode: req.CustomCode,
    }

    pbReq := map[string]interface{}{
        "URL": shortURL.URL,
        "ShortCode": shortURL.ShortCode,
    }

    resp, err := s.repo.Create(ctx, pbReq)
    if err != nil {
        return nil, err
    }

    if resp.StatusCode == http.StatusConflict {
        return nil, serviceErrors.NewDuplicateError("service.Create", "short code exists")
    }

    return toCreateURLResponse(shortURL), nil
}

This approach breaks our clean architecture:

Leaky Abstraction: The service layer is now coupled to the implementation details of PocketBase. It has to know what the request body should look like and how to interpret HTTP status codes. This violates the Dependency Rule.
Reduced Testability: Testing the service now requires mocking the http.Response, which is cumbersome. The service should only need to know about our domain models and errors.
Difficult to Swap: If we wanted to replace PocketBase with a direct SQL database, we would have to rewrite significant portions of our service layer, not just the repository.

The Solution: The Adapter Pattern

Our repository should be a true Adapter. Its job is to adapt the interface of our application (the URLRepository interface, which speaks in our domain models) to the interface of the external service (the PocketBase REST API, which speaks in HTTP and JSON). The service layer should be completely shielded from these external details.

This choice informs a critical architectural decision: we will treat PocketBase not as an embedded database, but as a remote, external service. Our application will communicate with it exclusively over HTTP. This approach has profound benefits for maintainability and scalability:

True Decoupling: Our Go service is now completely decoupled from its storage backend. The repository layer's job is to be an expert HTTP client for the PocketBase API. We could swap PocketBase for a different database service (like Supabase or a custom REST API) simply by writing a new repository that conforms to the URLRepository interface. The core business logic would remain untouched.
Leveraging External Features: PocketBase offers more than just storage. It has its own validation rules, authentication, and a real-time event system. By communicating via its API, we can leverage these features without having to implement them ourselves in our Go service.
Forced Cleanliness: The network boundary is a powerful forcing function. It prevents us from writing leaky abstractions where database-specific logic (like SQL queries or transaction management) bleeds into our business layer. Our repository must translate our internal domain models into HTTP requests and translate HTTP responses back into our domain models and errors.

Let's examine the concrete implementation of our urlRepositoryImpl to see this in practice.

type urlRepositoryImpl struct {
    pb *database.PBClient
}

func (r *urlRepositoryImpl) Create(ctx context.Context, shortURL *models.ShortURL) error {
    reqBody := pocketBaseCreateRequest{
        URL:         shortURL.URL,
        ShortCode:   shortURL.ShortCode,
        AccessCount: 0,
    }

    jsonBody, err := json.Marshal(reqBody)
    if err != nil {
        return serviceErrors.NewInternalError("repository.Create", "failed to marshal request", err)
    }

    endpoint := fmt.Sprintf("%s/api/collections/%s/records", r.pb.BaseURL, constants.ShortURLsCollection)
    req, err := http.NewRequestWithContext(ctx, "POST", endpoint, bytes.NewBuffer(jsonBody))
    if err != nil {
        return serviceErrors.NewInternalError("repository.Create", "failed to create request", err)
    }
    req.Header.Set("Content-Type", "application/json")

    resp, err := r.pb.HTTPClient.Do(req)
    if err != nil {
        return serviceErrors.NewInternalError("repository.Create", "failed to execute request", err)
    }
    defer resp.Body.Close()

    if resp.StatusCode != http.StatusCreated {
        if resp.StatusCode == http.StatusConflict {
            return serviceErrors.NewDuplicateError("repository.Create", "short code already exists")
        }
        return serviceErrors.NewInternalError("repository.Create", "PocketBase returned an error", fmt.Errorf("status: %d", resp.StatusCode))
    }

    var pbResp pocketBaseResponse
    if err := json.NewDecoder(resp.Body).Decode(&pbResp); err != nil {
        return serviceErrors.NewInternalError("repository.Create", "failed to decode response", err)
    }

    shortURL.ID = pbResp.ID
    shortURL.Created = pbResp.Created
    shortURL.Updated = pbResp.Updated

    return nil
}

This repository is a perfect example of an Adapter. It adapts our application's internal interfaces to the external interface of the PocketBase API. It is the sole gatekeeper of this external communication, creating a clean and testable boundary between our service and the outside world.

The Lifeline of a Request: Mastering `context.Context`

If there's one pattern that separates a professional Go service from an amateur one, it's the disciplined use of context.Context. Neglecting it is one of the most common sources of bugs, performance degradation, and cascading failures in distributed systems.

Imagine a user sends a request to your service, which then makes a call to a downstream API (like PocketBase). What happens if the user closes their browser? Or if the downstream API hangs and never responds? Without context, your server is left in the dark. It will continue to process the request, hold the connection open, and consume memory and CPU cycles, all for a result that no one is waiting for. Now, multiply this by thousands of requests per second. This is how you build a service that will reliably fall over under pressure.

context is Go's elegant solution to this problem. It provides a request-scoped "lifeline" that carries cancellation signals, deadlines, and other values across API boundaries. It must be the first argument to any function that is part of a request's call chain, especially those involving I/O.

Our Context Propagation Strategy

We implemented a simple but powerful context strategy:

Origination in the Handler: The lifecycle of our context begins in the HTTP handler. The Gin framework automatically provides a context.Context for each incoming request, which we can access via c.Request.Context(). This context is automatically cancelled if the client disconnects.
Propagation Through the Service: Every method in our URLService interface accepts ctx as its first argument. It does not inspect or modify the context; it simply acts as a carrier, passing it down to the repository layer.
Termination in the Repository: The repository is where the context is finally consumed. It uses the context to create deadline-aware and cancellable outbound HTTP requests.

Let's look at the critical piece of code in our repository:

func (r *urlRepositoryImpl) GetByShortCode(ctx context.Context, shortCode string) (*models.ShortURL, error) {
    ctx, cancel := context.WithTimeout(ctx, constants.RequestTimeout)
    defer cancel()

    filter := fmt.Sprintf("short_code='%s'", shortCode)
    reqURL := fmt.Sprintf("%s/api/collections/%s/records?filter=%s", r.pb.BaseURL, constants.ShortURLsCollection, url.QueryEscape(filter))

    req, err := http.NewRequestWithContext(ctx, "GET", reqURL, nil)
    if err != nil {
        return nil, serviceErrors.NewInternalError("repository.GetByShortCode", "failed to create request", err)
    }

    resp, err := r.pb.HTTPClient.Do(req)
    if err != nil {
        return nil, serviceErrors.NewInternalError("repository.GetByShortCode", "failed to execute request", err)
    }
    defer resp.Body.Close()

    if resp.StatusCode == http.StatusNotFound {
        return nil, serviceErrors.NewNotFoundError("repository.GetByShortCode", "short URL not found")
    }
    if resp.StatusCode != http.StatusOK {
        return nil, serviceErrors.NewInternalError("repository.GetByShortCode", "PocketBase returned an error", fmt.Errorf("status: %d", resp.StatusCode))
    }

    var pbResp struct {
        Items []pocketBaseShortURL `json:"items"`
    }
    if err := json.NewDecoder(resp.Body).Decode(&pbResp); err != nil {
        return nil, serviceErrors.NewInternalError("repository.GetByShortCode", "failed to decode response", err)
    }

    if len(pbResp.Items) == 0 {
        return nil, serviceErrors.NewNotFoundError("repository.GetByShortCode", "short URL not found")
    }

    item := pbResp.Items[0]
    return &models.ShortURL{
        ID:          item.ID,
        URL:         item.URL,
        ShortCode:   item.ShortCode,
        AccessCount: item.AccessCount,
        Created:     item.Created,
        Updated:     item.Updated,
    }, nil
}

This implementation gives us two crucial guarantees:

Fail-Fast on Timeouts: We will never wait more than 30 seconds for PocketBase to respond. This prevents slow downstream services from causing a cascading failure in our application.
Work Cancellation: If the initial HTTP request is cancelled by the user, that cancellation signal will propagate down and cause our http.Client to immediately abort the outbound request to PocketBase. We stop wasting resources the instant the work is no longer needed.

This disciplined use of context is the bedrock of a resilient Go service.

From Prototype to Better Code: A Blueprint for Your Next Service

We started with a common scenario: a simple script that solved a problem but was riddled with technical debt. Through a series of deliberate engineering decisions, we transformed it into a robust service that is a pleasure to work on. This wasn't about premature optimization or adding layers for the sake of it; it was about making pragmatic choices that lead to a more maintainable, testable, and resilient system.

Let's recap the actionable principles you can apply to your next Go project:

Isolate Your Business Logic: Your core logic should have no knowledge of your web framework, database, or any other external concern. Encapsulate it in a service layer that depends only on interfaces.
Define Contracts with Interfaces: Use interfaces to define the boundaries between your layers (URLService, URLRepository). This is the key to testability and modularity.
Separate Public and Private Models: Use DTOs to define your public API contract. This decouples your API from your internal data structures, preventing breaking changes and information leakage.
Treat Errors as Structured Data: Don't just return error strings. Return custom error types that contain rich, structured context. This allows for consistent error handling and powerful, structured logging.
Validate at the Edge: Create a dedicated validation layer to act as a gatekeeper for all incoming requests. Your business logic should only ever operate on data that has been proven to be safe and valid.
Propagate Context Everywhere: Make context.Context the first argument to every function in your request path. This is your lifeline for handling timeouts, cancellations, and building resilient systems.

By embracing these principles, you move from simply writing code to engineering a system. You build a well-organized workshop where any engineer can confidently and safely contribute, rather than a chaotic garage where only the original creator knows where the tools are. The initial investment in this structure pays dividends in reduced bugs, faster feature development, and a more maintainable codebase.

Conclusion

We began with a familiar scenario: a monolithic handler that worked but couldn't scale. Through our systematic application of clean architecture principles, we've transformed it into a resilient, maintainable service. This transformation wasn't about adding complexity—it was about organizing complexity in a way that actually makes our lives easier.

Our URL shortener now demonstrates six critical patterns that you can apply to any Go service: domain isolation through service layers, interface-based contracts for testability, explicit API boundaries via DTOs, structured error handling for observability, centralized input validation for security, and context-aware operations for resilience.

Apply these patterns incrementally to your existing services. Start with error handling and input validation—they'll give you immediate value with minimal refactoring. Then introduce service interfaces and DTOs to create those testable boundaries we've been talking about. Save the repository pattern for last, as it requires the most architectural change but provides the greatest long-term flexibility.

Remember, the goal isn't perfection from day one. It's building systems that can evolve safely, scale predictably, and welcome new contributors without requiring archaeological expeditions through legacy code. You've got this!

Project Resources

GitHub Repository: github.com/rowjay007/url-shortening-service
Original Project Brief: roadmap.sh/projects/url-shortening-service
Author: @rowjay

Production-Ready Caching Proxy in Go

rowjay007 — Wed, 17 Sep 2025 11:52:00 +0000

I've seen the same pattern play out time and again: a successful application's biggest enemy is its own growth. As traffic scales, the backend inevitably becomes a bottleneck, leading to slower response times and higher costs. To solve this, I often turn to one of the most powerful tools we have: a caching proxy.

In this post, I'm going to walk you through how I designed and built a production-ready caching proxy in Go. We're going to move beyond a simple 'hello world' example and dive deep into the architectural decisions, trade-offs, and patterns that are required to build a service you can truly rely on in a demanding enterprise environment.

My Core Principles:

For me, it's a promise that the service delivers on five key principles. These were the non-negotiable requirements I set for this project:

Resilience: The proxy can't just fail; it has to handle upstream failures gracefully and never cause a cascading outage.
Observability: I need to be able to see everything. The service can't be a black box; it must provide deep insights through structured logs, metrics, and traces.
Scalability: It has to be ready for 10x the traffic from day one, designed for clean, horizontal scaling.
Security: Security must be built-in, not bolted on. I designed it to operate on a principle of least privilege.
Maintainability: I wanted to build a codebase that other engineers would enjoy working on—clean, modular, and easy to test.

Everything we're about to explore was built on this foundation.

The Architecture: A Tour of the Components

I'm a firm believer in the Separation of Concerns. I designed this proxy not as a monolith, but as a collection of specialized Go packages that work together. Each one has a single, clear job.

cache-proxy/
├── cmd/caching-proxy/main.go     # The Conductor: Starts and stops the service
├── internal/
│   ├── cache/cache.go            # The Heart: The high-performance cache itself
│   ├── config/config.go          # The Brain: Manages all configuration
│   ├── errors/errors.go          # The Safety Net: Handles errors predictably
│   ├── health/health.go          # The Pulse: Reports the service's health
│   ├── logger/logger.go          # The Scribe: Writes structured logs
│   ├── middleware/middleware.go  # The Guardian: Protects and enriches requests
│   └── proxy/proxy.go            # The Engine Room: The core HTTP server
└── Taskfile.yml                  # The Toolkit: For building and testing

I stick to the standard Go project layout because it makes the codebase immediately familiar to any Go developer. Now, let's dive into the components I'm most proud of.

The Heart: A High-Concurrency Caching Engine

At the core of the proxy is the caching engine. I knew from the start that a simple map[string][]byte wouldn't cut it; in a concurrent system, that's a guaranteed race condition. So, I built the engine in internal/cache/cache.go specifically for thread safety and high performance.

I started by defining a clean Cache interface. This is one of my favourite patterns in Go because it makes the code incredibly modular. It means that later on, we can easily swap in a Redis-backed cache without having to rewrite the entire proxy.


type Cache interface {
    Get(key string) (*Entry, bool)
    Set(key, value string, ttl time.Duration)
    Delete(key string) bool
    Stats() Stats
}

My InMemoryCache implementation showcases a few pragmatic Go concurrency patterns:


type InMemoryCache struct {
    mu         sync.RWMutex
    items      map[string]*Entry
    maxSize    int
    stats      Stats
    defaultTTL time.Duration
    cleanup    *time.Ticker
}

I chose a sync.RWMutex for locking. A standard Mutex would have created a bottleneck, since cache reads are far more common than writes. The RWMutex allows for unlimited concurrent readers, which is a huge performance win.
I use a background goroutine for cache cleanup. A simple time.Ticker wakes up a goroutine periodically to purge expired items. This is an elegant, low-overhead way to handle TTLs and prevent stale data.
I enforced a maxSize to prevent memory leaks. An unbounded cache is a dangerous thing. My implementation guarantees a predictable memory footprint by evicting the oldest item when the cache is full.
The cache tracks its own metrics. It's not a black box. I made sure it tracks hits, misses, and evictions—vital signs that we can expose through an API for monitoring.

The Brain: Smart, 12-Factor Configuration

I'm a zealot when it comes to configuration. Hardcoding settings is a non-starter for me, so I strictly followed the 12-Factor App methodology. All configuration is externalized and managed in a single place: internal/config/config.go.


type Config struct {
    Port         int           `json:"port"`
    Origin       string        `json:"origin"`
    CacheTTL     time.Duration `json:"cache_ttl"`
    CacheSize    int           `json:"cache_size"`
    Timeout      time.Duration `json:"timeout"`
    LogLevel     string        `json:"log_level"`
    // ... and more
}

func (c *Config) Validate() error {
    if c.Port <= 0 || c.Port > 65535 {
        return errors.New("invalid port")
    }
    // ... other validations
    return nil
}

Environment is King: I designed the service so that a single, immutable Docker image can be promoted from dev to staging to prod. The behaviour is controlled entirely by environment variables.
Fail Fast: The very first thing the application does on startup is validate its configuration. If a setting is invalid, it fails immediately with a clear error message. This has saved me countless hours of debugging mysterious runtime failures.

The Engine Room: A Resilient Proxy Server

This is where all the components come together. In internal/proxy/proxy.go, I use the Gin framework to orchestrate the flow of requests. It's fast, reliable, and has great support for middleware.

I'm a big proponent of Dependency Injection, so I made sure the server doesn't create its own dependencies (like the cache or logger). Instead, we inject them when the server is created. This is a crucial pattern that makes our components loosely coupled and a breeze to unit test, because you can just pass in mocks.


func New(cfg *config.Config, cache cache.Cache, logger logger.Logger) (*Server, error) {
    // ... initialization ...

    engine.Use(
        middleware.RequestID(),       
        middleware.Logger(logger),    
        middleware.CORS(cfg.EnableCORS), 
        middleware.Security(),       
        middleware.Metrics(),        
    )

    return server, nil
}

The Middleware Pipeline: I think of middleware as an assembly line for our requests. Every request that comes in passes through a standard set of steps: it gets a unique ID for tracing, it's logged, security headers are added, and more. This keeps my core proxy logic clean and focused on its main job: caching.
Zero-Downtime Deployments: In production, you can't just pull the plug on a server. I built the proxy to listen for shutdown signals (SIGINT, SIGTERM) and perform a graceful shutdown. It stops accepting new requests but gives in-flight requests a chance to finish. For me, this is a non-negotiable feature for any serious service.

The Watchful Eye: A Trilogy of Observability

I have a simple rule: if I can't see what a service is doing, I don't trust it in production. That's why I built the proxy to be transparent, giving us a clear view into its health and behavior through three key mechanisms.

Structured Logging: I chose Zerolog to write logs as structured JSON. This is a game-changer. Every log line is a machine-readable event, enriched with context like the request_id and duration. When you're debugging an issue at 3 AM, being able to filter and search these logs in a platform like Splunk or Datadog is a lifesaver.
```
log.Info().
    Str("method", c.Request.Method).
    Str("path", c.Request.URL.Path).
    Int("status", c.Writer.Status()).
    Str("request_id", requestID).
    Dur("duration", duration).
    Msg("Request processed")
```
Health Probes: I included Kubernetes-native health endpoints: /health/live (is the process running?) and /health/ready (can it serve traffic?). This is the language we use to let orchestrators like Kubernetes automatically manage the service, enabling a self-healing system.
Metrics: I added a hook to export key metrics. This is where we'd track things like cache hit ratios, request latencies, and error rates. This is the data we feed into tools like Prometheus and Grafana to build dashboards and set up alerts, so we know about a problem before our users do.

The Safety Net: Predictable Error Handling

I can't stand APIs that return vague or inconsistent errors. To solve this, I implemented a custom AppError type for all our error responses.


type AppError struct {
    Code       string `json:"code"`
    Message    string `json:"message"`
    StatusCode int    `json:"-"`
    Err        error  `json:"-"`
}

This guarantees that every error is a clean, structured JSON object with a machine-readable code, a human-readable message, and the correct HTTP StatusCode. This makes life better for the developers consuming our API and makes my debugging process far more straightforward.

The Conductor: Orchestrating the Application Lifecycle

I think of the main.go file as the conductor of our orchestra. Its job isn't to do the work itself, but to make sure all the other components are initialized in the right order and that the service starts and stops cleanly.


func main() {
    cfg, err := config.ParseFlags()

    log := logger.NewWithLevel(logLevel)
    cacheInstance := cache.New(cacheConfig)

    server, err := proxy.New(cfg, cacheInstance, log)

    ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
    defer stop()

    go func() { server.Start() }()

    <-ctx.Done()

    server.Shutdown(shutdownCtx)
}

Thanks to the modular design I chose, the main function is incredibly simple and readable. It tells a clear story about the service's lifecycle without getting bogged down in the implementation details of each component.

From Code to Cloud: My Blueprint for Operations

I believe that great code is only half the story. A service is only as good as its operational story—how we build, deploy, and manage it. I designed this proxy from day one with a smooth operational lifecycle in mind.

My Toolkit: Taskfile for Automation

I've moved on from Makefiles. For this project, I used Taskfile.yml for a modern, declarative approach to build automation. For me, this is all about consistency.

# Taskfile.yml

version: "3"

tasks:
  build:
    desc: "Build the application binary."
    cmds:
      - go build -o caching-proxy ./cmd/caching-proxy

  test:
    desc: "Run all unit tests."
    cmds:
      - go test -v ./...

  lint:
    desc: "Run the golangci-lint linter."
    cmds:
      - golangci-lint run

  docker:
    desc: "Build the Docker container image."
    cmds:
      - docker build -t cache-proxy:latest .

The Taskfile gives us a single, self-documenting entry point for every common task. It guarantees that the command I run on my laptop is the exact same command our CI server runs, which eliminates that whole class of "works on my machine" problems.

The Vessel: A Production-Ready Dockerfile

I designed this service to be deployed as a container, and the Dockerfile I wrote uses a multi-stage build. In my opinion, this is a non-negotiable best practice for creating lean and secure production images.


FROM golang:1.19-alpine AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o /caching-proxy ./cmd/caching-proxy

FROM alpine:latest
RUN apk --no-cache add ca-certificates

WORKDIR /root/
COPY --from=builder /caching-proxy .

CMD ["./caching-proxy"]

Lean and Mean: The final image I produce starts from a minimal alpine base and copies only the compiled Go binary. No source code, no build tools. This drastically reduces the image size and its attack surface.
Completely Static: I use CGO_ENABLED=0 to build a statically-linked binary. This means it has zero dependencies on the host OS, making the container incredibly portable.

Thriving in the Kubernetes Ecosystem

I engineered this service to be a first-class citizen in a Kubernetes environment. The features we've built map directly to the core concepts I rely on for cloud-native operations:

Our 12-Factor design allows for Declarative Configuration via ConfigMaps and Secrets.
The /health/live and /health/ready probes enable Automated Healing, which Kubernetes uses to manage rolling updates and restarts.
My graceful shutdown logic supports High Availability, ensuring pods can be terminated without dropping user connections.
The bounded, in-memory cache allows for Stable Performance by setting predictable resource limits.

Fortifying the Gates: My Approach to Security

For me, security isn't a feature you bolt on at the end; it's a foundational requirement. I used a defense-in-depth strategy for this proxy.

The First Line of Defense: Security Middleware

I wrote a simple but powerful gatekeeper in middleware.Security() that applies critical HTTP security headers to every response.


func Security() gin.HandlerFunc {
    return func(c *gin.Context) {
        c.Header("X-Content-Type-Options", "nosniff")
        c.Header("X-Frame-Options", "DENY")
        c.Header("X-XSS-Protection", "1; mode=block")
        c.Next()
    }
}

These headers instruct browsers to enable built-in protections against common attacks like Cross-Site Scripting (XSS) and clickjacking.

The Principle of Least Privilege

I configured the Dockerfile to run the service as a non-root user, but in a real production environment, I'd take it even further in the Kubernetes manifest:


securityContext:
  runAsNonRoot: true
  runAsUser: 1001
  readOnlyRootFilesystem: true
  allowPrivilegeEscalation: false

This is a powerful security control. If an attacker finds a vulnerability, their blast radius is tiny. They're trapped as a low-privilege user with a read-only filesystem.

Architecting for Tomorrow: The Path to Hyperscale

A great design solves today's problems while keeping an eye on tomorrow's. The modular, interface-driven design I chose provides a clear roadmap for future evolution.

Beyond a Single Node: Distributed Caching

The InMemoryCache is fast, but its state is local to a single pod. At scale, this leads to cache fragmentation. The beauty of the cache.Cache interface I defined is that we can swap in a RedisCache implementation without changing a single line of the proxy server.


type RedisCache struct {
    client *redis.Client
}

func (r *RedisCache) Get(key string) (*Entry, bool) {
}

func (r *RedisCache) Set(key, value string, ttl time.Duration) {
}

server, err := proxy.New(cfg, cacheInstance, log)

By simply changing the component we inject, we can transform the service from a node-local cache to a globally consistent, distributed caching tier.

Embracing the Service Mesh

As a microservices ecosystem grows, managing concerns like mTLS or retries in every service becomes a nightmare. A service mesh like Istio or Linkerd can offload this complexity to the infrastructure layer.

I designed our proxy to be a perfect fit for a service mesh. Its commitment to observability provides the rich data that meshes need to operate. We could even simplify our application by removing logic for things like mTLS, delegating them entirely to the mesh.

This forward-looking design—built for composition and integration—is a hallmark of modern enterprise architecture.

The Crucible of Quality: My Testing Strategy

Code without tests is just a bug waiting to happen. My philosophy isn't about chasing 100% code coverage, but about achieving maximum confidence with a smart, layered testing strategy.

The Foundation: Unit Tests

For me, unit tests are the bedrock. They're fast, isolated, and verify that each individual component does its job correctly. For example, I have tests that hammer the cache key generation logic to ensure it's deterministic.


func TestGenerateKey(t *testing.T) {
    c := New(Config{DefaultTTL: 1 * time.Minute})

    key1 := c.GenerateKey("GET", "/users/123", "sort=asc")
    key2 := c.GenerateKey("GET", "/users/123", "sort=asc")
    key3 := c.GenerateKey("GET", "/users/123", "sort=desc")

    assert.Equal(t, key1, key2, "Keys with identical inputs should be identical")
    assert.NotEqual(t, key1, key3, "Keys with different inputs should not be identical")
}

This gives me high confidence that the building blocks of our system are solid.

The Seams: Integration Tests

Next, I use integration tests to make sure our components play nicely together. They focus on the contracts between our packages, like verifying that the proxy server calls the cache module correctly.


func TestProxyHandler_CacheInteraction(t *testing.T) {
    mockCache := new(mocks.Cache)
    logger := logger.New()
    cfg := &config.Config{Origin: "http://example.com"}
    server, _ := New(cfg, mockCache, logger)

    mockCache.On("Get", mock.Anything).Return(nil, false).Once()

    w := httptest.NewRecorder()
    req, _ := http.NewRequest("GET", "/test", nil)
    server.engine.ServeHTTP(w, req)

    mockCache.AssertCalled(t, "Get", mock.Anything)
}

By using a real HTTP server but a mocked Cache interface, I can validate this interaction without the brittleness of a full end-to-end test. It's a powerful way I test the seams of the architecture.

The Full Picture: End-to-End (E2E) Tests

Finally, I use E2E tests for the ultimate reality check. They treat the entire application as a black box and verify a real user journey. For us, the most critical journey is the cache HIT/MISS cycle.


./caching-proxy --origin="http://localhost:9090" &
PROXY_PID=$!

RESPONSE_ONE=$(curl -s -I http://localhost:8080/data)
echo "$RESPONSE_ONE" | grep "X-Cache: MISS"

RESPONSE_TWO=$(curl -s -I http://localhost:8080/data)
echo "$RESPONSE_TWO" | grep "X-Cache: HIT"

kill $PROXY_PID

This layered strategy gives me the confidence to deploy quickly and safely.

The Art of the Trade-off: My Engineering Decisions

Great engineering isn't about finding perfect solutions; it's about making smart, deliberate compromises. Here are a few key trade-offs I made during this project.

In-Memory Cache vs. Distributed Cache: I chose a simple, in-memory cache. The benefit is blistering speed and zero operational overhead. The compromise is that the cache isn't shared between instances. My rationale? It's the simplest thing that works, and our interface-driven design means we can easily swap to Redis later if we need to.
Simple Eviction vs. True LRU: I decided to evict the oldest item, not the least recently used. Why? Because a truly performant, thread-safe LRU is a complex beast. My simple policy is good enough for most workloads and keeps the code radically simpler and easier to maintain. I chose pragmatism over algorithmic perfection.

A Final Thought: Your Turn to Build

We've journeyed from a high-level vision to the real-world trade-offs that define professional engineering. I've shown you how I built this proxy, but this is more than just a service; it's a collection of battle-tested patterns for building software that lasts.

But this isn't the end of the story. I see it as a challenge. How would you improve it? Could you build a smarter eviction policy? Integrate it with Prometheus and Grafana? The path to mastery is paved with relentless curiosity. Take these ideas, challenge them, and go build something great.

References

The initial project idea was inspired by the "Caching Server in Go" project on roadmap.sh.
The complete source code for this project is available on GitHub.

Deploying NGINX with Vagrant & Making It Publicly Accessible

rowjay007 — Fri, 31 Jan 2025 15:44:07 +0000

📌 Introduction

As part of the HNG DevOps Stage 0 challenge, I successfully deployed an NGINX web server on a fresh Ubuntu machine using an Azure Virtual Machine (VM). Unlike my initial Vagrant-based deployment, this approach ensures full-time availability and public accessibility over the internet.

This blog post documents the full process, challenges faced, and how I overcame them.

1️⃣ Setting Up the Environment

🖥️ Prerequisites

Before getting started, ensure you have the following:

✔️ An Azure Account (for deploying the VM)

✔️ SSH Client (for connecting to the VM)

✔️ A basic understanding of NGINX and Ubuntu

2️⃣ Creating the Azure Virtual Machine (VM)

1️⃣ Go to Azure Portal and create a new Ubuntu 20.04 LTS VM.

2️⃣ Choose the right instance type (Standard B1s is a good starting point).

3️⃣ Enable SSH login and allow HTTP (port 80) traffic.

4️⃣ Assign a Public IP Address for remote access.

5️⃣ Deploy the VM and connect via SSH:

ssh <username>@102.37.154.154

3️⃣ Installing and Configuring NGINX

After logging into the Azure VM, I installed NGINX with:

sudo apt update -y
sudo apt install nginx -y
sudo systemctl start nginx
sudo systemctl enable nginx

To confirm NGINX was running, I checked its status:

sudo systemctl status nginx

✔️ Result: "Active (running)" ✅

4️⃣ Creating the Custom Welcome Page

Next, I customized the default NGINX page by modifying /var/www/html/index.html:

echo '<html>
<head>
  <title>Welcome</title>
  <style>
    body { display: flex; justify-content: center; align-items: center; height: 100vh; font-family: Arial, sans-serif; }
    h1 { text-align: center; }
  </style>
</head>
<body>
  <h1>Welcome to DevOps Stage 0 - Rowland Adimoha/Rowjay</h1>
</body>
</html>' | sudo tee /var/www/html/index.html > /dev/null

Then, I restarted NGINX to apply the changes:

sudo systemctl restart nginx

Now, my website was ready to be accessed via the public IP!

5️⃣ Configuring Azure Firewall for Public Access

By default, Azure blocks port 80. To allow external access, I had to update the Network Security Group (NSG):

1️⃣ Go to Azure Portal → Virtual Machine → Networking.

2️⃣ Add an inbound rule:

Source: Any
Source Port Ranges: *
Destination: Any
Service: Custom
Destination Port Ranges: 80
Protocol: TCP
Action: Allow
Priority: 310 3️⃣ Save the rule, and my site became accessible from anywhere on the internet at:

👉 http://102.37.154.154/

6️⃣ Testing the Deployment

🔹 Browser Test: Visiting http://102.37.154.154 displayed:

"Welcome to DevOps Stage 0 - Rowland Adimoha/Rowjay"

🔹 NGINX Header Check:

I confirmed my server was serving requests properly:

curl -I http://102.37.154.154

✔️ The response contained "Server: nginx", confirming that NGINX was active.

🔹 Verifying Uptime:

I ran:

sudo systemctl status nginx

✔️ NGINX was running without errors. ✅

🚀 Challenges Faced & Solutions

❌ Issue 1: Website Not Loading Externally

Problem: After configuring NGINX, I could access the site locally but not from other devices.

Solution: The issue was port 80 being blocked in Azure NSG. After adding an allow rule for port 80, the site became publicly accessible.

❌ Issue 2: NGINX Not Restarting After Reboot

Problem: NGINX didn’t automatically restart after VM reboots.

Solution: I ran:

sudo systemctl enable nginx

✔️ This ensured NGINX starts automatically on boot.

🎯 Lessons Learned & How This Contributes to My DevOps Growth

💡 1. Cloud Infrastructure Management

This task helped me understand Azure VM deployment and network configurations for web applications.

💡 2. Debugging & Troubleshooting Network Issues

I learned how to troubleshoot cloud firewall rules to enable public access.

💡 3. Web Server Automation

I automated NGINX installation and web page deployment using shell scripts.

🔗 References (Cited Properly)

As I continue on this DevOps journey, I look forward to exploring career opportunities in the field. If you're looking for skilled DevOps Engineers, check out:

Building a REST API with Golang, Gin, and Pocketbase

rowjay007 — Tue, 21 Jan 2025 16:06:30 +0000

Introduction

Building a REST API with modern technologies can be an engaging yet complex process, especially when you are working with unfamiliar stacks. As a Go developer, I decided to explore the combination of Golang with the Gin framework and Pocketbase for building a REST API. This choice was driven by a desire to experiment with a different tech stack while also considering the positive feedback I had heard from the developer community regarding Pocketbase's user-friendly features.

We’ll walk you through setting up the environment, configuring the API routes, and creating a service layer to interact with Pocketbase for user authentication and data management. By the end of this tutorial, you will have a fully functional REST API capable of user registration and login.

Why Choose Gin and Pocketbase?

Gin Framework:

Performance: Known for its speed and low latency, making it ideal for high-performance applications.
Simplicity and Flexibility: Easy to set up, scale, and maintain.
Middleware Support: Built-in middleware enables seamless integration of common features like logging and authentication.

Pocketbase:

User Management: Out-of-the-box support for user registration, login, and password reset functionalities.
Real-Time Database: Simplifies real-time data management with minimal configuration.
Ease of Use: Pocketbase abstracts away many backend complexities, allowing developers to focus on building the core application.

⚠️ Important Note About Pocketbase:

Pocketbase is under active development and may undergo breaking changes before version 1.0.0. It is not recommended for production use unless you are comfortable with frequent updates and manual migrations.

Prerequisites

Golang: Install Go from Go Downloads.
GitHub Account: Required for cloning repositories and managing code.
Pocketbase: Download the latest release from Pocketbase GitHub.

Setting Up the Project Structure

The directory structure of the project is as follows:

walkit/
├── .github/
├── cmd/
│   └── server/
├── config/
├── internal/
│   ├── middleware/
│   ├── handler/
│   ├── model/
│   ├── repository/
│   ├── service/
│   ├── routes/
├── pb_data/
├── pkg/
│   ├── util/
│   └── logger/
├── air.toml
├── .env
├── Dockerfile
├── pocketbase
├── Makefile
├── go.mod
├── go.sum
└── README.md

Setting Up Pocketbase

Download and Install Pocketbase:

Download the latest release of Pocketbase from the GitHub repository.
Extract the files and place them in your project folder.
To start Pocketbase, run the following command:

   ./pocketbase serve

This will start Pocketbase at http://127.0.0.1:8090.

Navigate to the Pocketbase dashboard at http://127.0.0.1:8090/_/ and set up a root password for the admin interface.

Once the server is running, Pocketbase provides two essential endpoints:

REST API: http://127.0.0.1:8090/api/
Dashboard: http://127.0.0.1:8090/_/

Configuring API URLs

In the config.go file, we'll use Viper to manage and load configuration settings, such as the Pocketbase API URL and JWT secret. This ensures that the Golang application can securely interact with Pocketbase for user authentication and management.

package config

import (
    "log"
    "github.com/spf13/viper"
)

type Config struct {
    BaseURL           string
    JWTSecret         string
    Environment       string
    CORSAllowedOrigins []string
    Port              string
}

func LoadConfig() *Config {
    viper.SetConfigName("config")
    viper.SetConfigType("yaml")
    viper.AddConfigPath("./config")
    viper.AddConfigPath(".")
    viper.AutomaticEnv()
    viper.SetDefault("cors_allowed_origins", []string{"*"})

    if err := viper.ReadInConfig(); err != nil {
        log.Printf("Error reading config file, using defaults: %v", err)
    }

    corsAllowedOrigins := viper.GetStringSlice("cors_allowed_origins")
    if len(corsAllowedOrigins) == 0 {
        corsAllowedOrigins = []string{"*"}
    }

    return &Config{
        BaseURL:           viper.GetString("pocket_base_url"),
        JWTSecret:         viper.GetString("jwt_secret"),
        Environment:       viper.GetString("app_env"),
        CORSAllowedOrigins: corsAllowedOrigins,
        Port:              viper.GetString("port"),
    }
}

The config.yml file should be structured as follows:

pocket_base_url: "http://127.0.0.1:8090/api"
jwt_secret: "your_jwt_secret_key"
app_env: "development"
cors_allowed_origins:
  - "*"
port: "8080"

Setting Up Golang with Gin

Install Gin:

To install the Gin framework in your Go project, run the following command:

go get -u github.com/gin-gonic/gin

This will install Gin and its dependencies into your Go workspace.

API Endpoints and `main.go` Implementation

Here are the two primary API endpoints for registering and logging in users:

POST /api/v1/auth/register – Allows a new user to register.
POST /api/v1/auth/login – Allows a registered user to log in and receive a JWT token.

package main

import (
    "context"
    "net/http"
    "os"
    "os/signal"
    "syscall"
    "time"

    "github.com/gin-contrib/cors"
    "github.com/gin-gonic/gin"
    "github.com/rowjay007/walkit/config"
    "github.com/rowjay007/walkit/internal/routes"
    "github.com/rowjay007/walkit/pkg/logger"
)

func main() {
    logger := logger.New()
    cfg := config.LoadConfig()

    if cfg.Environment == "production" {
        gin.SetMode(gin.ReleaseMode)
    }

    router := gin.New()
    router.Use(gin.Recovery())

    corsConfig := cors.DefaultConfig()
    corsConfig.AllowOrigins = cfg.CORSAllowedOrigins
    router.Use(cors.New(corsConfig))

    routes.LoadRoutes(router)

    srv := &http.Server{
        Addr:         ":" + cfg.Port,
        Handler:      router,
        ReadTimeout:  15 * time.Second,
        WriteTimeout: 15 * time.Second,
        IdleTimeout:  60 * time.Second,
    }

    go func() {
        logger.Info("Starting server on port " + cfg.Port)
        if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
            logger.Fatal("Failed to start server: " + err.Error())
        }
    }()

    quit := make(chan os.Signal, 1)
    signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
    <-quit

    logger.Info("Shutting down server...")
    ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
    defer cancel()

    if err := srv.Shutdown(ctx); err != nil {
        logger.Fatal("Server forced to shutdown: " + err.Error())
    }

    logger.Info("Server exiting")
}

Integrating Repository, Service, and Handler

Repository Layer (`repository.go`):

This layer interacts directly with Pocketbase for data management, such as registering users or logging them in.

package repository

import (
    "bytes"
    "encoding/json"
    "fmt"
    "net/http"
    "github.com/rowjay007/walkit/config"
    "github.com/rowjay007/walkit/internal/model"
)

func AuthAPI() string {
    return config.LoadConfig().BaseURL + "/collections/users"
}

func RegisterUser(user model.User) error {
    userJSON, err := json.Marshal(user)
    if err != nil {
        return fmt.Errorf("error marshaling user data: %w", err)
    }

    resp, err := http.Post(AuthAPI(), "application/json", bytes.NewBuffer(userJSON))
    if err != nil {
        return fmt.Errorf("error making request: %w", err)
    }
    defer resp.Body.Close()

    if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusCreated {
        return fmt.Errorf("registration failed: %v", resp.Status)
    }

    return nil
}

func LoginUser(login model.LoginRequest) (*model.LoginResponse, error) {
    loginJSON, err := json.Marshal(login)
    if err != nil {
        return nil, fmt.Errorf("error marshaling login data: %w", err)
    }

    req, err := http.NewRequest("POST", AuthAPI()+"/auth-with-password", bytes.NewBuffer(loginJSON))
    if err != nil {
        return

 nil, fmt.Errorf("error creating request: %w", err)
    }
    req.Header.Set("Content-Type", "application/json")

    client := &http.Client{}
    resp, err := client.Do(req)
    if err != nil {
        return nil, fmt.Errorf("error making request: %w", err)
    }
    defer resp.Body.Close()

    var response model.LoginResponse
    // Handle response decoding here
    return &response, nil
}

Service Layer (`service.go`):

This layer handles business logic and interacts with the repository.

package service

import (
    "github.com/rowjay007/walkit/internal/model"
    "github.com/rowjay007/walkit/internal/repository"
)

func LoginUser(login model.LoginRequest) (*model.LoginResponse, error) {
    return repository.LoginUser(login)
}

func RegisterUser(user model.User) error {
    return repository.RegisterUser(user)
}

Handler Layer (`handler.go`):

This layer handles HTTP requests and responses.

package handler

import (
    "fmt"
    "net/http"
    "strings"
    "github.com/gin-gonic/gin"
    "github.com/rowjay007/walkit/internal/model"
    "github.com/rowjay007/walkit/internal/service"
    "github.com/rowjay007/walkit/pkg/util"
)

func LoginUser(c *gin.Context) {
    var login model.LoginRequest
    if err := c.ShouldBindJSON(&login); err != nil {
        util.RespondWithError(c, http.StatusBadRequest, fmt.Sprintf("Invalid request payload: %v", err))
        return
    }

    if err := validateLoginInput(login); err != nil {
        util.RespondWithError(c, http.StatusBadRequest, err.Error())
        return
    }

    response, err := service.LoginUser(login)
    if err != nil {
        statusCode := determineStatusCode(err)
        util.RespondWithError(c, statusCode, "Login failed. Please check your credentials.")
        return
    }

    util.RespondWithJSON(c, http.StatusOK, response)
}

func validateLoginInput(login model.LoginRequest) error {
    if strings.TrimSpace(login.Identity) == "" {
        return fmt.Errorf("identity cannot be empty")
    }
    if len(login.Password) < 8 {
        return fmt.Errorf("password must be at least 8 characters")
    }
    return nil
}

func determineStatusCode(err error) int {
    if strings.Contains(strings.ToLower(err.Error()), "invalid credentials") {
        return http.StatusUnauthorized
    }
    if strings.Contains(strings.ToLower(err.Error()), "not found") {
        return http.StatusNotFound
    }
    return http.StatusInternalServerError
}

Response Utilities (`util.go`):

This file provides functions to send consistent responses.

package util

import (
    "github.com/gin-gonic/gin"
)

func RespondWithError(c *gin.Context, code int, message string) {
    c.JSON(code, gin.H{"error": message})
}

func RespondWithJSON(c *gin.Context, code int, payload interface{}) {
    c.JSON(code, payload)
}

JWT Middleware (`middleware.go`):

This middleware ensures that a valid JWT token is included in the request header.

package middleware

import (
    "fmt"
    "net/http"
    "strings"
    "github.com/dgrijalva/jwt-go"
    "github.com/gin-gonic/gin"
    "github.com/rowjay007/walkit/config"
)

func JWTAuthMiddleware(c *gin.Context) {
    tokenString := c.GetHeader("Authorization")
    if tokenString == "" {
        c.JSON(http.StatusUnauthorized, gin.H{"error": "Authorization token is required"})
        c.Abort()
        return
    }

    tokenString = strings.TrimPrefix(tokenString, "Bearer ")

    token, err := jwt.Parse(tokenString, func(token *jwt.Token) (interface{}, error) {
        if _, ok := token.Method.(*jwt.SigningMethodHMAC); !ok {
            return nil, fmt.Errorf("unexpected signing method: %v", token.Header["alg"])
        }
        return []byte(config.LoadConfig().JWTSecret), nil
    })

    if err != nil || !token.Valid {
        c.JSON(http.StatusUnauthorized, gin.H{"error": "Invalid or expired token"})
        c.Abort()
        return
    }

    c.Next()
}

Here’s the updated conclusion with a link to the complete code:

Conclusion

In this tutorial, we’ve built a REST API using Golang, Gin, and Pocketbase. The guide covered everything from project structure to implementing user authentication with JWT tokens. We also demonstrated how to integrate multiple layers (repository, service, handler) for better modularity and scalability.

With this setup, you can start extending your API with more features, such as user profile management, JWT token refresh, and other advanced functionalities.

Next Steps

Add Password Reset: Implement a password reset flow using JWT for authentication.
Use More Pocketbase Features: Pocketbase offers additional real-time features that can be useful for interactive applications.
Test the API: Write tests for API endpoints to ensure robustness.
Deploy: Deploy the application on platforms like AWS, Google Cloud, or DigitalOcean.

By following this tutorial, you've established a solid foundation for building secure and scalable APIs using Golang, Gin, and Pocketbase.

You can find the complete code for this project on GitHub here.

Securing Express APIs using OAuth2 and JSON Web Tokens

rowjay007 — Mon, 02 Sep 2024 10:48:16 +0000

Introduction

How secure are your users? In today's digital landscape, where data breaches and cyber attacks are increasingly common, implementing robust authentication in your web applications is no longer optional—it's essential. As developers, we are responsible for protecting our users' data and maintaining our applications' integrity.

In a world where cybersecurity is more critical than ever, implementing robust authentication in your Express.js application isn't just a technical necessity; it's a safeguard for your entire platform. This guide will walk you through building a secure authentication system using TypeScript, ensuring your application is resilient and scalable.

We'll explore everything from basic username/password authentication to advanced strategies like JSON Web Tokens (JWT) and OAuth 2.0, equipping you with the tools to protect your users and your application from unauthorized access.

Setting Up Your Development Environment

Before we proceed with the implementation, it's crucial to set up your development environment properly. This ensures a smooth development process and helps maintain consistency across your team. Here's what you'll need:

Prerequisites:

Node.js (version 14 or later)
npm (usually comes with Node.js)
A code editor (I recommend Visual Studio Code for its excellent TypeScript support)

Start by installing TypeScript globally on your system. This allows you to use TypeScript's compiler anywhere on your machine. Additionally, install ts-node, which enables you to run TypeScript files directly without a separate compilation step.

npm install -g typescript
npm install -g ts-node

With these tools in place, you're ready to start building secure Express.js applications with TypeScript. The combination of Express.js and TypeScript provides a robust foundation for creating scalable and maintainable web applications.

Understanding Authentication in Express.js

Authentication is the process of verifying the identity of a user, device, or system. In the context of web applications, it typically involves validating user credentials and creating a session or token to maintain the user's authenticated state.

Why is authentication crucial for your Express.js applications?

Data Protection: It safeguards sensitive user information from unauthorized access.
Access Control: It allows you to restrict certain parts of your application to authenticated users only.
Personalization: With authenticated users, you can provide personalized experiences and content.
Accountability: Authentication helps in tracking user actions, which is crucial for auditing and compliance.

In Express.js, authentication is usually implemented as middleware. This middleware intercepts requests, verifies the user's identity, and either allows the request to proceed or denies access if the authentication fails.

There are several authentication strategies you can implement in Express.js:

Session-based Authentication: This traditional method involves storing user session information on the server.
Token-based Authentication: Popular for stateless authentication, especially in Single Page Applications (SPAs) and mobile apps.
OAuth: Allows users to grant limited access to their resources on one site to another site, without sharing their credentials.

Each strategy has its pros and cons, and the choice depends on your specific application requirements. In this guide, we'll focus on implementing token-based authentication using JSON Web Tokens (JWT), as it's widely used in modern web applications.

Setting Up Your Express.js Project with TypeScript

Setting up an Express.js project with TypeScript involves a few more steps than a standard JavaScript project, but the benefits in terms of code quality and maintainability are significant.

First, initialize your project and install the necessary dependencies:

mkdir express-auth-demo
cd express-auth-demo
npm init -y
npm install express body-parser
npm install --save-dev typescript @types/node @types/express @types/body-parser

Next, create a tsconfig.json file in your project root. This file configures the TypeScript compiler options for your project:

{
  "compilerOptions": {
    "target": "es6",
    "module": "commonjs",
    "outDir": "./dist",
    "strict": true,
    "esModuleInterop": true
  },
  "include": ["src/**/*"]
}

With the configuration in place, create your main application file (e.g., src/app.ts) using TypeScript syntax:

import express, { Request, Response } from 'express';
import bodyParser from 'body-parser';

const app = express();
const port = 3000;

app.use(bodyParser.json());

app.get('/', (req: Request, res: Response) => {
  res.send('Welcome to our secure Express.js API with TypeScript!');
});

app.listen(port, () => {
  console.log(`Server running on http://localhost:${port}`);
});

export default app;

To run your TypeScript application during development, you can use ts-node:

ts-node src/app.ts

Implementing Basic Username/Password Authentication

Let's start with a simple username/password authentication system. While this method isn't the most secure for production environments, it serves as a good starting point to understand the basics of authentication in Express.js.

In this implementation, we'll create two routes:

/register: Allows users to create a new account.
/login: Authenticates users based on their credentials.

For simplicity, we'll store user information in memory. In a real-world application, you'd use a database to persist this data.

import express, { Request, Response } from 'express';
import bodyParser from 'body-parser';

interface User {
  username: string;
  password: string;
}

const app = express();
const port = 3000;

app.use(bodyParser.json());

const users: User[] = [];

app.post('/register', (req: Request, res: Response) => {
  const { username, password } = req.body;

  if (users.find(user => user.username === username)) {
    return res.status(400).json({ message: 'Username already exists' });
  }

  users.push({ username, password });
  res.status(201).json({ message: 'User registered successfully' });
});

app.post('/login', (req: Request, res: Response) => {
  const { username, password } = req.body;

  const user = users.find(user => user.username === username && user.password === password);

  if (user) {
    res.json({ message: 'Login successful' });
  } else {
    res.status(401).json({ message: 'Invalid credentials' });
  }
});

app.listen(port, () => {
  console.log(`Server running on http://localhost:${port}`);
});

export default app;

Key points to consider in this implementation:

Use TypeScript interfaces to define the structure of user objects.
Implement proper error handling and status codes.
Validate input data before processing.

While this basic implementation works, it's not secure enough for production use. In the next section, we'll enhance it with password hashing.

Enhancing Security with Password Hashing

Storing passwords in plain text is a major security risk. If an attacker gains access to your user database, they would have immediate access to all user accounts. To mitigate this risk, we'll use the bcrypt library to hash passwords before storing them.

Bcrypt is a password hashing function designed by Niels Provos and David Mazières. It's based on the Blowfish cipher and incorporates a salt to protect against rainbow table attacks. Here's why bcrypt is a good choice:

It's slow by design, making brute-force attacks impractical.
It automatically handles salt generation and incorporation.
It's future-proof, allowing you to increase the work factor as computers get faster.

Let's update our code to use bcrypt:

import express, { Request, Response } from 'express';
import bodyParser from 'body-parser';
import bcrypt from 'bcrypt';

interface User {
  username: string;
  password: string;
}

const app = express();
const port = 3000;

app.use(bodyParser.json());

const users: User[] = [];

app.post('/register', async (req: Request, res: Response) => {
  const { username, password } = req.body;

  if (users.find(user => user.username === username)) {
    return res.status(400).json({ message: 'Username already exists' });
  }

  const hashedPassword = await bcrypt.hash(password, 10);
  users.push({ username, password: hashedPassword });
  res.status(201).json({ message: 'User registered successfully' });
});

app.post('/login', async (req: Request, res: Response) => {
  const { username, password } = req.body;

  const user = users.find(user => user.username === username);

  if (user && await bcrypt.compare(password, user.password)) {
    res.json({ message: 'Login successful' });
  } else {
    res.status(401).json({ message: 'Invalid credentials' });
  }
});

app.listen(port, () => {
  console.log(`Server running on http://localhost:${port}`);
});

export default app;

When implementing password hashing:

Hash passwords during user registration before storing them.
During login, compare the provided password with the stored hash.
Never store or transmit passwords in plain text.

Token-Based Authentication with JSON Web Tokens (JWT)

Token-based authentication is a stateless authentication method that works particularly well with modern web architectures. JSON Web Tokens (JWTs) have become a popular choice for implementing token-based authentication.

A JWT consists of three parts:

Header: Contains metadata about the token (like the hashing algorithm used).
Payload: Contains claims (statements about the user and additional data).
Signature: Ensures the token hasn't been altered.

Here's how JWT authentication

typically works:

User logs in with their credentials.
Server verifies the credentials and generates a JWT.
Server sends the JWT back to the client.
Client stores the JWT (usually in local storage) and sends it with subsequent requests.
Server verifies the JWT for each request to protected routes.

Let's implement JWT authentication in our Express.js application:

import express, { Request, Response, NextFunction } from 'express';
import bodyParser from 'body-parser';
import bcrypt from 'bcrypt';
import jwt from 'jsonwebtoken';

interface User {
  username: string;
  password: string;
}

const app = express();
const port = 3000;
const SECRET_KEY = 'your-secret-key'; // In production, use an environment variable

app.use(bodyParser.json());

const users: User[] = [];

app.post('/register', async (req: Request, res: Response) => {
  const { username, password } = req.body;

  if (users.find(user => user.username === username)) {
    return res.status(400).json({ message: 'Username already exists' });
  }

  const hashedPassword = await bcrypt.hash(password, 10);
  users.push({ username, password: hashedPassword });
  res.status(201).json({ message: 'User registered successfully' });
});

app.post('/login', async (req: Request, res: Response) => {
  const { username, password } = req.body;

  const user = users.find(user => user.username === username);

  if (user && await bcrypt.compare(password, user.password)) {
    const token = jwt.sign({ username: user.username }, SECRET_KEY, { expiresIn: '1h' });
    res.json({ message: 'Login successful', token });
  } else {
    res.status(401).json({ message: 'Invalid credentials' });
  }
});

function authenticateToken(req: Request, res: Response, next: NextFunction) {
  const authHeader = req.headers['authorization'];
  const token = authHeader && authHeader.split(' ')[1];

  if (token == null) return res.sendStatus(401);

  jwt.verify(token, SECRET_KEY, (err: any, user: any) => {
    if (err) return res.sendStatus(403);
    req.user = user;
    next();
  });
}

app.get('/protected', authenticateToken, (req: Request, res: Response) => {
  res.json({ message: 'This is a protected route', user: req.user });
});

app.listen(port, () => {
  console.log(`Server running on http://localhost:${port}`);
});

export default app;

When implementing JWT authentication:

Use environment variables to store your secret key.
Set an expiration time for your tokens.
Implement token refresh mechanisms for long-lived sessions.
Store tokens securely on the client-side (avoid storing in local storage if possible).

Implementing OAuth 2.0 for Third-Party Authentication

OAuth 2.0 is an authorization framework that enables applications to obtain limited access to user accounts on an HTTP service. It's widely used for implementing "Sign in with Google/Facebook/GitHub" functionality.

Implementing OAuth 2.0 in your Express.js application involves several steps:

Register your application with the OAuth provider (e.g., Google, Facebook).
Install necessary packages (e.g., passport, passport-google-oauth20).
Configure Passport strategies for your chosen providers.
Set up routes for initiating the OAuth flow and handling callbacks.
Implement logic to create or update user accounts based on OAuth data.

Here's an example of implementing Google OAuth:

import express from 'express';
import passport from 'passport';
import { Strategy as GoogleStrategy } from 'passport-google-oauth20';

const app = express();

passport.use(new GoogleStrategy({
    clientID: GOOGLE_CLIENT_ID,
    clientSecret: GOOGLE_CLIENT_SECRET,
    callbackURL: "http://localhost:3000/auth/google/callback"
  },
  function(accessToken, refreshToken, profile, cb) {
    // Here you would find or create a user in your database
    return cb(null, profile);
  }
));

app.get('/auth/google',
  passport.authenticate('google', { scope: ['profile', 'email'] }));

app.get('/auth/google/callback', 
  passport.authenticate('google', { failureRedirect: '/login' }),
  function(req, res) {
    // Successful authentication, redirect home.
    res.redirect('/');
  });

When implementing OAuth:

Keep client IDs and secrets secure (use environment variables).
Implement proper error handling for the OAuth flow.
Consider offering multiple OAuth providers to give users choices.
Always validate and sanitize data received from OAuth providers before using it in your application.

Best Practices for Secure Authentication

Implementing authentication is just the first step. To ensure your system remains secure, consider these best practices:

Use HTTPS: Encrypt all communication between client and server.
Implement Rate Limiting: Prevent brute-force attacks by limiting login attempts.
Use Secure Password Reset Mechanisms: Avoid security questions; use time-limited tokens sent via email.
Implement Multi-Factor Authentication: Add an extra layer of security for sensitive operations.
Keep Dependencies Updated: Regularly update your packages to patch known vulnerabilities.
Use Secure Session Management: Set proper cookie flags (HttpOnly, Secure, SameSite).
Implement Proper Error Handling: Avoid leaking sensitive information through error messages.
Use Password Strength Meters: Encourage users to create strong passwords.
Implement Account Lockout Mechanisms: Temporarily lock accounts after multiple failed login attempts.
Use Security Headers: Implement headers like Content-Security-Policy to protect against certain types of attacks.

Testing Your Authentication System

Testing is crucial to ensure your authentication system is working as expected. Write unit tests and integration tests for the following:

Registration and Login: Ensure the processes work correctly and handle edge cases.
Token Generation and Validation: Verify that tokens are issued and validated properly.
OAuth Flows: Test the integration with third-party providers.
Security Vulnerabilities: Test for common vulnerabilities like SQL injection, cross-site scripting (XSS), and cross-site request forgery (CSRF).

Consider using tools like Postman for manual testing and frameworks like Jest for automated tests.

Conclusion and Next Steps

Securing your Express.js API with robust authentication is a critical step in safeguarding your application and user data. By implementing the strategies outlined in this guide—ranging from basic username/password authentication to advanced token-based and OAuth 2.0 authentication—you can create a secure and scalable authentication system for your Express.js applications.

As you continue to develop and maintain your application, stay informed about the latest security practices and continuously evaluate and improve your authentication mechanisms. Your commitment to security will protect your users and enhance your platform's overall integrity and trustworthiness.

Software Releases: A Developer's Guide to Smooth Deployments

rowjay007 — Mon, 12 Aug 2024 20:01:14 +0000

Imagine pushing a critical update that brings down your entire production system. Now imagine having the confidence to roll back that change in seconds. This is the power of effective software release management. In today's fast-paced tech world, where 84% of organizations have adopted DevOps practices (State of DevOps Report 2023), mastering release management is no longer optional—it's essential.

This guide will walk you through proven techniques for managing software releases, using a Go application with the Gin framework as our case study. Whether you're a seasoned DevOps professional or a developer looking to level up your skills, you'll find actionable insights to enhance your release strategy.

The Foundations: Versioning Your Application
Environment Management: From Development to Production
Deployment Strategies: Blue-Green and Beyond
The Safety Net: Mastering Rollbacks
DevOps and Developer Collaboration: Bridging the Gap
Monitoring and Metrics: The Key to Continuous Improvement
Quick Start Guide: 5 Steps to Better Releases

1. The Foundations: Versioning Your Application

Effective versioning is the bedrock of successful release management. It's not just about keeping track of changes; it's about communicating clearly with your team and users about what to expect.

Semantic Versioning: Your New Best Friend

Semantic Versioning (SemVer) provides a clear structure for version numbers: MAJOR.MINOR.PATCH. Here's how it works:

MAJOR: Incremented for incompatible API changes.
MINOR: Added for new features in a backwards-compatible manner.
PATCH: Incremented for backwards-compatible bug fixes.

Let's implement this in our Go application:

// version.go
package main

const (
    Version   = "1.2.3"
    BuildDate = "2024-08-12"
)

Now, let's use this in our Gin application:

// main.go
package main

import (
    "github.com/gin-gonic/gin"
    "net/http"
)

func main() {
    r := gin.Default()

    r.GET("/version", func(c *gin.Context) {
        c.JSON(http.StatusOK, gin.H{
            "version":   Version,
            "buildDate": BuildDate,
        })
    })

    r.Run()
}

Pro Tip: Include versioning details in your configuration files for easy reference:

# config.yaml
version: "1.2.3"
buildDate: "2024-08-12"

By implementing clear versioning, you're not just organizing your code—you're setting the stage for smoother releases and easier troubleshooting down the line.

2. Environment Management: From Development to Production

Managing different environments is crucial for ensuring stability and smooth transitions. Let's explore how to set up and manage feature, development, staging, and production environments effectively.

Feature Environment: Where Innovation Thrives

Purpose: Develop and test new features separately.
Best Practice: Use feature flags to enable or disable features dynamically.

# feature-flags.yaml
features:
  newPaymentGateway:
    enabled: true
  betaUserInterface:
    enabled: false

Development Environment: The Integration Playground

Purpose: Integrate and test code changes before staging.
Best Practice: Mirror production as closely as possible.

# dev-config.yaml
database:
  host: dev-db.example.com
  port: 5432
logging:
  level: debug

Staging Environment: The Dress Rehearsal

Purpose: Test releases in a production-like setting.
Best Practice: Perform extensive testing, including load and performance tests.

# staging-config.yaml
database:
  host: staging-db.example.com
  port: 5432
logging:
  level: info

Production Environment: Where Performance Meets Reliability

Purpose: Serve live traffic with the highest stability requirements.
Best Practice: Implement robust monitoring and quick rollback capabilities.

# prod-config.yaml
database:
  host: prod-db.example.com
  port: 5432
logging:
  level: warn

By clearly defining and managing these environments, you create a pipeline that nurtures innovation while safeguarding stability.

3. Deployment Strategies: Blue-Green and Beyond

Now that we have our environments set up, let's explore deployment strategies that minimize downtime and risk.

Blue-Green Deployment: The Art of Seamless Transitions

Blue-green deployment uses two identical production environments, allowing you to switch between them with zero downtime.

Here's how it works:

Deploy the new version to the "Green" environment.
Run tests on the Green environment.
Switch traffic from Blue to Green.
Keep Blue as a quick rollback option.

Let's see this in action with a Kubernetes configuration:

# blue-green-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app-green
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
      environment: green
  template:
    metadata:
      labels:
        app: my-app
        environment: green
    spec:
      containers:
      - name: my-app
        image: my-app:1.2.3-green
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: my-app-service
spec:
  selector:
    app: my-app
    environment: green
  ports:
  - port: 8080

By implementing blue-green deployment, you can reduce downtime to near-zero and increase confidence in your releases.

4. The Safety Net: Mastering Rollbacks

Even with the best preparation, things can go wrong. That's where rollbacks come in. Let's explore how to implement effective rollback strategies.

Preparation: The Key to Quick Recovery

Maintain detailed version history in your version control system.
Implement comprehensive automated tests to catch issues early.

Instant Rollback: The Quick Switch

In a blue-green setup, rolling back can be as simple as redirecting traffic back to the "Blue" environment:

# rollback-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: my-app-service
spec:
  selector:
    app: my-app
    environment: blue  # Switch back to blue
  ports:
  - port: 8080

Advanced Rollback Techniques

Canary Releases: Gradually release changes to a subset of users.
Feature Toggles: Switch features on/off without redeploying.

# canary-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app-canary
spec:
  replicas: 1  # Start with a small subset
  selector:
    matchLabels:
      app: my-app
      environment: canary
  template:
    metadata:
      labels:
        app: my-app
        environment: canary
    spec:
      containers:
      - name: my-app
        image: my-app:1.2.3-canary
        ports:
        - containerPort: 8080

By implementing these rollback strategies, you create a safety net that allows you to move fast without breaking things.

5. DevOps and Developer Collaboration: Bridging the Gap

Effective release management isn't just about tools and techniques—it's about people working together seamlessly. Here's how to foster collaboration between DevOps and development teams.

Integrated Workflows: Communication is Key

Use tools like Slack or Microsoft Teams for real-time discussions.
Align goals and metrics. Both teams should work towards common objectives like stability, performance, and user satisfaction.

Automated CI/CD Pipelines: The Backbone of Collaboration

Implement robust Continuous Integration (CI) and Continuous Deployment (CD) pipelines:

# .github/workflows/ci-cd.yml
name: CI/CD Pipeline

on:
  push:
    branches:
      - main

jobs:
  build-and-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Set up Go
        uses: actions/setup-go@v2
        with:
          go-version: '1.20'
      - name: Build
        run: go build -v ./...
      - name: Test
        run: go test -v ./...

  deploy:
    needs: build-and-test
    runs-on: ubuntu-latest
    steps:
      - name: Deploy to Kubernetes
        run: |
          kubectl apply -f blue-green-deployment.yaml
          kubectl apply -f prod-config.yaml

Feedback Loops: Continuous Improvement

Conduct post-deployment reviews to gather insights and identify areas for improvement.
Collaborate on incident management and post-mortems to refine processes and prevent future issues.

By fostering this collaboration, you create a culture of shared responsibility and continuous improvement.

6. Monitoring and Metrics: The Key to Continuous Improvement

Effective monitoring ensures you understand the impact of releases and catch issues early. Let's explore how to set up robust monitoring.

Monitoring Tools: Your Eyes and Ears in Production

Prometheus: For metrics collection and alerting.
Grafana: For visualizing metrics and creating dashboards.

Prometheus Configuration:

# prometheus.yml
scrape_configs:
  - job_name: 'my-app'
    static_configs:
      - targets: ['localhost:9090']

Grafana Dashboard:

{
  "dashboard": {
    "id": null,
    "uid":

 "my-app-dashboard",
    "title": "My App Dashboard",
    "panels": [
      {
        "type": "graph",
        "title": "Request Rate",
        "targets": [
          {
            "expr": "rate(http_requests_total[5m])",
            "interval": "",
            "format": "time_series"
          }
        ]
      }
    ]
  }
}

Real-Time Monitoring: Staying Ahead of Issues

Set up alerts for critical metrics like error rates and response times.
Continuously monitor application performance and user experience.

By implementing robust monitoring, you gain the insights needed to make data-driven decisions about your releases.

7. Quick Start Guide: 5 Steps to Better Releases

Ready to level up your release management? Here are five steps you can take today:

Implement Semantic Versioning in your next commit.
Set up a basic blue-green deployment for one of your services.
Create a simple feature flag system to toggle new features.
Establish a regular "release retrospective" meeting with your team.
Set up basic monitoring with Prometheus and Grafana.

Conclusion: Empowering Your Release Strategy

Mastering software releases is a journey, not a destination. By implementing these techniques—from versioning to monitoring—you're not just improving your deployment process. You're building confidence, reducing stress, and creating space for innovation.

Remember, the goal isn't perfection but continuous improvement. Start small, learn from each release, and gradually refine your process. With these tools and strategies at your disposal, you're well-equipped to navigate the complexities of modern software development.

This technique, while widely applicable across various platforms like Azure, AWS, GCP, and other cloud environments, is valuable for any setting where minimizing downtime and maintaining application stability is critical. With a collaborative approach between DevOps and developers, and by integrating robust monitoring and metrics, you can navigate the complexities of software releases with confidence and precision.

References

Managing Technical Debt: Strategies for Long-Term Success

rowjay007 — Mon, 05 Aug 2024 16:41:24 +0000

Earlier this week, I was discussing technical debt with a friend. We talked about how code can be shipped with issues because some developers do not refactor, check security coverage, or apply any form of test coverage once the code works. In our discussion, we compared refactoring to driving to a popular mall. Imagine there's a better and faster shortcut, but due to fear of getting lost, we keep taking the old route countless times without searching for new paths. This fear of the unknown, or simply sticking to what’s familiar, mirrors how teams often accumulate and neglect technical debt in software development.

Much like an overused route, technical debt represents shortcuts taken during development to meet deadlines or achieve immediate results. These shortcuts can lead to inefficiencies, making future modifications more challenging and costly. If we don't address this debt, it can accumulate, just as our reluctance to explore new routes can lead to a longer and more frustrating journey. In this article, we'll explore strategies for managing technical debt effectively, ensuring that our software projects are successful in the short term and sustainable and efficient in the long run.

Understanding Technical Debt

Before diving into management strategies, it's essential to grasp what technical debt entails and its different forms:

Code Debt: This occurs when code is written in a way that is complex, redundant, or difficult to maintain. It includes poorly named variables, inefficient algorithms, and a lack of adherence to coding standards.
Design Debt: Arises from suboptimal design choices that hinder scalability and flexibility. Examples include tightly coupled components and lack of modularity.
Documentation Debt: This involves insufficient or outdated documentation that makes it harder for new developers to understand the codebase and for the team to maintain the system.
Testing Debt: Results from inadequate test coverage or outdated tests, leading to a fragile codebase that is prone to bugs and regressions.
Infrastructure Debt: Refers to outdated or improperly configured infrastructure, such as obsolete servers or inefficient database designs.

Understanding these categories helps us pinpoint where technical debt exists and how to address it.

The Consequences of Accumulating Technical Debt

Accumulating technical debt without a repayment plan can have several adverse effects:

Decreased Productivity: Developers may spend more time understanding and fixing issues rather than building new features. This reduces overall productivity and slows down progress.
Increased Costs: Addressing technical debt later in the development cycle or after deployment is often more costly and time-consuming than dealing with it upfront.
Reduced Quality: Compromises made to expedite delivery can lead to bugs, security vulnerabilities, and performance issues, ultimately affecting the product's quality.
Lower Morale: Working with a debt-ridden codebase can be frustrating and demoralizing, leading to decreased team morale and higher turnover rates.

Recognizing these impacts emphasizes the importance of managing technical debt proactively.

Strategies for Managing Technical Debt

1. Prioritize and Categorize Debt

Effective management of technical debt starts with identifying and prioritizing it. Use tools and techniques such as static code analysis, code reviews, and automated testing to detect debt areas. Classify debt based on its impact and urgency:

High Priority: Debt that affects critical functionality, security, or performance.
Medium Priority: Debt impacting maintainability and scalability.
Low Priority: Minor issues that do not significantly affect the current development process.

Prioritizing debt ensures that the most critical issues are addressed first, minimizing their negative impact.

2. Establish a Debt Log

Maintaining a debt log helps document and track all identified technical debt. This log should include:

Description: A clear explanation of the debt.
Category: Classification (code, design, documentation, etc.).
Impact: Potential consequences of not addressing the debt.
Priority: Urgency of addressing the debt.
Owner: Team or individual responsible for resolving the debt.
Status: Current progress on addressing the debt.

A debt log provides visibility and accountability, ensuring that technical debt is systematically managed.

3. Integrate Debt Management into the Development Process

Managing technical debt should be part of the development workflow. Implement these practices:

Incorporate Debt Reviews: Discuss technical debt in regular code reviews and sprint planning meetings.
Allocate Time for Debt Repayment: Dedicate a portion of each sprint to addressing high-priority debt.
Automate Detection and Remediation: Use tools like linters and static code analyzers to detect and fix common issues proactively.

Incorporating debt management into daily activities promotes ongoing progress and helps prevent accumulation.

4. Refactor Regularly

Refactoring involves restructuring current code without altering its external behaviour to enhance readability, maintainability, and performance. Let’s illustrate with a simple example.

Original Code

Here’s a Go function that calculates discounts based on customer type. It has redundancy and complexity:

package main

import "fmt"

func calculateDiscount(price float64, customerType string) float64 {
    if customerType == "regular" {
        return price * 0.9
    } else if customerType == "vip" {
        return price * 0.8
    } else if customerType == "employee" {
        return price * 0.7
    }
    return price
}

func main() {
    fmt.Println(calculateDiscount(100, "vip")) // Output: 80
}

Refactored Code

Here’s a refactored version that eliminates redundancy:

package main

import "fmt"

// Refactored function to calculate discount
func calculateDiscount(price float64, discountRate float64) float64 {
    return price * discountRate
}

// Function to get discount rate based on customer type
func getDiscountRate(customerType string) float64 {
    switch customerType {
    case "regular":
        return 0.9
    case "vip":
        return 0.8
    case "employee":
        return 0.7
    default:
        return 1.0
    }
}

func main() {
    discountRate := getDiscountRate("vip")
    fmt.Println(calculateDiscount(100, discountRate)) // Output: 80
}

This refactoring improves readability and maintainability.

5. Invest in Training and Knowledge Sharing

Technical debt often arises from a lack of knowledge or understanding of best practices. Address this by:

Providing Training: Offer workshops, courses, and resources on coding standards, design patterns, and testing methodologies.
Fostering Collaboration: Encourage pair programming, code reviews, and knowledge-sharing sessions.
Documenting Standards: Create comprehensive documentation on coding standards and architectural guidelines.

Training and knowledge sharing empower your team to make informed decisions and minimize technical debt.

6. Balance Short-Term Gains with Long-Term Goals

Balancing immediate needs with long-term goals is crucial. Here’s how:

Adopt a Sustainable Pace: Avoid shortcuts that compromise quality. Plan realistic timelines that accommodate both quality and sustainability.
Evaluate Trade-offs: Weigh the benefits of shortcuts against their long-term costs. Make informed decisions based on this evaluation.
Set Clear Guidelines: Establish guidelines for when and how technical debt can be incurred, ensuring it is a deliberate choice.

Balancing short-term and long-term considerations helps maintain a healthy codebase.

7. Monitor and Measure Progress

To ensure effective debt management, regularly monitor and measure progress using metrics such as:

Code Quality: Track metrics like code complexity, duplication, and adherence to coding standards.
Test Coverage: Measure the extent of automated test coverage and identify areas needing improvement.
Issue Resolution Time: Monitor the time taken to resolve technical debt issues, aiming for continuous improvement.

Monitoring provides insights into the effectiveness of your strategies and highlights areas for further improvement.

8. Foster a Culture of Quality

Creating a culture of quality within your development team is essential for managing technical debt. Here’s how:

Promote Ownership: Encourage developers to take ownership of their code and its quality.
Recognize Efforts: Acknowledge and reward efforts to reduce technical debt and improve code quality.
Encourage Feedback: Foster an environment where constructive feedback is welcomed and acted upon.

A culture of quality ensures that managing technical debt becomes a shared responsibility and a core aspect of your development process.

Tools and Techniques for Managing Technical Debt

1. Static Code Analysis Tools

Static code analysis tools help detect potential issues early. Examples include:

SonarQube: Provides detailed insights into code quality, highlighting bugs, code smells, and security vulnerabilities.
ESLint: Focuses on identifying and fixing problems in JavaScript code.
PMD: Analyzes Java code for potential issues and offers recommendations for improvement.

Integrating these tools into your CI/CD pipeline allows for early detection and remediation of technical debt.

2. Automated Testing Frameworks

Automated testing frameworks help to ensure that code changes do not introduce new issues. Examples include:

JUnit: A popular framework for writing and running tests in Java.
pytest: A strong Python framework that supports various types of tests and fixtures.
Selenium: An open-source tool for automating web browser interactions.

Automated testing provides a safety net, allowing for refactoring and optimization with confidence.

3. Continuous Integration and Continuous Deployment (CI/CD) Tools

CI/CD tools automate the integration and deployment

process. Examples include:

Jenkins: An open-source automation server that supports building, testing, and deploying code.
GitHub Actions: Provides automation workflows directly within GitHub repositories.
CircleCI: Offers fast and scalable CI/CD pipelines for building and deploying code.

CI/CD tools streamline the development process, ensuring that technical debt is managed continuously.

4. Technical Debt Dashboard

Creating a technical debt dashboard provides real-time visibility into your codebase’s state. It can include:

Code Quality Metrics: Visualize the overall quality of your codebase, highlighting areas with high technical debt.
Debt Trends: Track changes in technical debt over time, identifying patterns and trends.
Issue Distribution: Show the distribution of technical debt across different categories.

A technical debt dashboard helps monitor progress and make data-driven decisions.

5. Gamification

Gamifying technical debt management can motivate developers. Here’s how:

Point System: Assign points for identifying and resolving technical debt issues.
Leaderboards: Display a leaderboard showing top contributors to debt reduction.
Rewards: Offer rewards for significant contributions, such as gift cards or public recognition.

Gamification makes managing technical debt engaging and rewarding.

6. Technical Debt Retirement Plan

Develop a technical debt retirement plan to systematically reduce debt. The plan should include:

Debt Assessment: Conduct a thorough assessment to identify and prioritize all technical debt.
Repayment Schedule: Create a schedule for addressing high-priority debt, allocating time and resources.
Progress Reviews: Regularly review progress and adjust the plan as needed.

A technical debt retirement plan ensures a structured approach to debt reduction.

Conclusion

As we’ve explored, managing technical debt is much like finding the optimal route to a destination. Just as we might avoid exploring new shortcuts due to fear or unfamiliarity, teams can accumulate technical debt by avoiding necessary refactoring or updates. However, just as finding and using that better route can lead to a more pleasant journey, addressing technical debt can lead to a more maintainable, efficient, and high-quality codebase.

By understanding the various forms of technical debt and implementing effective strategies—such as prioritizing debt, integrating debt management into the development process, and fostering a quality culture—we can navigate our development journey more effectively.

Just as taking the time to explore and map out better routes can save time and frustration in the long run, managing technical debt proactively ensures that our projects remain robust and adaptable. Embrace these strategies and keep a focus on both immediate and long-term goals to transform technical debt from a daunting challenge into a manageable and surmountable aspect of software development.

Remember, the path to success in managing technical debt is not about avoiding new routes but about understanding and navigating them effectively. With a proactive approach and commitment to quality, we can ensure that our software remains reliable and efficient, just like finding the perfect route to our destination.

For further reading and a comprehensive guide on managing technical debt, consider these resources:

CI/CD Pipelines Guide with GitHub Actions, SonarCloud, Codecov, and Sentry

rowjay007 — Mon, 29 Jul 2024 22:58:44 +0000

I've been developing an open-source REST API for a book management system. I've set up all the components for an enterprise application, including Docker, Prometheus, and Grafana. Additionally, I've established a thorough CI/CD pipeline. Here's a summary of how I configured the CI/CD pipeline using GitHub Actions to check test coverage, conduct code analysis, create build tags, and monitor performance errors.

In this article, I will guide you through setting up a CI/CD pipeline using GitHub Actions. We will configure the pipeline to include test coverage reports, code quality analysis, automatic tag generation, release creation, and performance monitoring.

Prerequisites

Before we proceed with the CI/CD configuration, please ensure you have the following:

GitHub Account: Please ensure you have a GitHub account with an existing public repository, as you will need a premium account for private repositories.
SonarCloud Account: Create a SonarCloud account and obtain the Project key and Organization keys, which will be added to GitHub secrets.
Codecov Account: Set up a Codecov account and add the CODECOV_TOKEN to the secret keys of your GitHub repository project.
Sentry Project: Ensure you have an existing Sentry project. Add your CHANGELOG_RELEASE from your profile settings to the project repository for error monitoring, so that the tag will correlate with Sentry production.

Configuring the CI/CD Pipeline

Let's break down the configuration of our CI/CD pipeline step by step.

Setting Up the Build Job

First, we need a build job needs to be created to run on the ubuntu-latest environment. This job should check out the code, set up Node.js, and install the necessary dependencies.

name: CI/CD Pipeline
on:
  push:
    branches:
      - main
jobs:
  build:
    name: Build
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v3
      - name: Set up Node.js
        uses: actions/setup-node@v2
        with:
          node-version: '20'
      - name: Install dependencies
        run: npm ci

In this job, we first check out the code from the repository. Next, we set up Node.js using the actions/setup-node action, specifying version 20. Finally, we install the project dependencies using npm ci.

Running Tests and Uploading Coverage Reports

Next, we will create a test job to execute our tests and then upload the coverage reports to Codecov.

  test:
    name: Run Tests
    runs-on: ubuntu-latest
    needs: build
    steps:
      - name: Checkout code
        uses: actions/checkout@v3
      - name: Set up Node.js
        uses: actions/setup-node@v2
        with:
          node-version: '20'
      - name: Install dependencies
        run: npm ci
      - name: Run tests with coverage
        run: npm run test:coverage
      - name: Upload coverage reports to Codecov
        uses: codecov/codecov-action@v3
        with:
          files: ./coverage/lcov.info
          flags: unittests
          name: codecov-umbrella
          fail_ci_if_error: true
        env:
          CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}

This job is triggered only after the build job is completed successfully. It checks out the code, sets up Node.js, and installs the dependencies. Then, it runs the tests with coverage and uploads the coverage report to Codecov.

Performing Code Analysis with SonarCloud

Let's set up a job to perform static code analysis using SonarCloud.

  sonarcloud:
    name: SonarCloud Analysis
    runs-on: ubuntu-latest
    needs: test
    steps:
      - name: Checkout code
        uses: actions/checkout@v3
        with:
          fetch-depth: 0
      - name: Set up Node.js
        uses: actions/setup-node@v2
        with:
          node-version: '20'
      - name: Install dependencies
        run: npm ci
      - name: SonarCloud Scan
        uses: SonarSource/sonarcloud-github-action@master
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
        with:
          projectBaseDir: .
          args: >
            -Dsonar.organization=rowjay007
            -Dsonar.projectKey=nardy-books
            -Dsonar.sources=src
            -Dsonar.tests=src/test
            -Dsonar.javascript.lcov.reportPaths=coverage/lcov.info
            -Dsonar.typescript.tsconfigPath=tsconfig.json
            -Dsonar.exclusions=**/node_modules/**,**/dist/**,**/coverage/**,**/public/**
            -Dsonar.test.inclusions=**/*.test.ts

In this job, we perform a comprehensive code analysis using SonarCloud. We clone the code repository with the full commit history, set up Node.js, and install dependencies. The SonarCloud action then executes the scan and uploads the results.

Generating Git Tags

We then set up a job to automatically generate Git tags.

  generate_git_tags:
    name: Generate Git Tags
    needs: [test, sonarcloud]
    runs-on: ubuntu-latest
    outputs:
      output_new_tag: ${{ steps.taggerFinal.outputs.new_tag }}
    steps:
      - name: Checkout code
        uses: actions/checkout@v3
        with:
          fetch-depth: 0
      - name: Generate Final Version
        id: taggerFinal
        uses: anothrNick/github-tag-action@1.67.0
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          WITH_V: true
      - name: Echo New Tag
        run: |
          echo "The next new tag will be: ${{ steps.taggerFinal.outputs.new_tag }}"

This job uses the github-tag-action to automatically create a new version tag based on the repository's state, and then outputs the new tag's value.

Creating a GitHub Release

Let's create our GitHub Release, which will publish a new release based on the generated tag.

  generate_git_release:
    needs: [test, sonarcloud, generate_git_tags]
    name: GitHub Release
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Release Action
        uses: ncipollo/release-action@v1.14.0
        with:
          tag: ${{ needs.generate_git_tags.outputs.output_new_tag }}
          token: ${{ secrets.CHANGELOG_RELEASE }}

This job creates a GitHub release using the generated tag. It checks out the code and uses the release-action to publish the release.

Monitoring with Sentry

Finally, we integrate Sentry to manage our releases and monitor performance errors.

  generate_sentry_release:
    needs: [test, sonarcloud, generate_git_tags, generate_git_release]
    name: Sentry Release
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
        with:
          fetch-depth: 0
      - name: Split Repo Name
        uses: jungwinter/split@v2.0.0
        id: split_repo_name
        with:
          separator: '/'
          msg: ${{ github.repository }}
      - name: Echo Repo name
        run: echo "${{ steps.split_repo_name.outputs._1 }}"
      - name: Sentry Release
        uses: getsentry/action-release@v1.7.0
        env:
          SENTRY_AUTH_TOKEN: ${{ secrets.SENTRY_AUTH_TOKEN }}
          SENTRY_ORG: ${{ secrets.SENTRY_ORG }}
          SENTRY_PROJECT: ${{ secrets.SENTRY_PROJECT }}
        with:
          environment: 'production'
          version: '${{ steps.split_repo_name.outputs._1 }}@${{ needs.generate_git_tags.outputs.output_new_tag }}'
          sourcemaps: './build'
          url_prefix: '~'

In this job, our task involves reviewing the code and extracting the repository name. The Sentry action entails creating the release in Sentry, uploading sourcemaps, and linking the release with the new version tag. This process is crucial for effectively tracking errors and managing releases.

Complete YAML Configuration

To provide a comprehensive view of the entire pipeline, here’s the complete YAML configuration file:

name: CI/CD Pipeline
on:
  push:
    branches:
      - main
jobs:
  build:
    name: Build
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v3
      - name: Set up Node.js
        uses: actions/setup-node@v2
        with:
          node-version: '20'


 - name: Install dependencies
        run: npm ci

  test:
    name: Run Tests
    runs-on: ubuntu-latest
    needs: build
    steps:
      - name: Checkout code
        uses: actions/checkout@v3
      - name: Set up Node.js
        uses: actions/setup-node@v2
        with:
          node-version: '20'
      - name: Install dependencies
        run: npm ci
      - name: Run tests with coverage
        run: npm run test:coverage
      - name: Upload coverage reports to Codecov
        uses: codecov/codecov-action@v3
        with:
          files: ./coverage/lcov.info
          flags: unittests
          name: codecov-umbrella
          fail_ci_if_error: true
        env:
          CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}

  sonarcloud:
    name: SonarCloud Analysis
    runs-on: ubuntu-latest
    needs: test
    steps:
      - name: Checkout code
        uses: actions/checkout@v3
        with:
          fetch-depth: 0
      - name: Set up Node.js
        uses: actions/setup-node@v2
        with:
          node-version: '20'
      - name: Install dependencies
        run: npm ci
      - name: SonarCloud Scan
        uses: SonarSource/sonarcloud-github-action@master
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
        with:
          projectBaseDir: .
          args: >
            -Dsonar.organization=rowjay007
            -Dsonar.projectKey=nardy-books
            -Dsonar.sources=src
            -Dsonar.tests=src/test
            -Dsonar.javascript.lcov.reportPaths=coverage/lcov.info
            -Dsonar.typescript.tsconfigPath=tsconfig.json
            -Dsonar.exclusions=**/node_modules/**,**/dist/**,**/coverage/**,**/public/**
            -Dsonar.test.inclusions=**/*.test.ts

  generate_git_tags:
    name: Generate Git Tags
    needs: [test, sonarcloud]
    runs-on: ubuntu-latest
    outputs:
      output_new_tag: ${{ steps.taggerFinal.outputs.new_tag }}
    steps:
      - name: Checkout code
        uses: actions/checkout@v3
        with:
          fetch-depth: 0
      - name: Generate Final Version
        id: taggerFinal
        uses: anothrNick/github-tag-action@1.67.0
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          WITH_V: true
      - name: Echo New Tag
        run: |
          echo "The next new tag will be: ${{ steps.taggerFinal.outputs.new_tag }}"

  generate_git_release:
    needs: [test, sonarcloud, generate_git_tags]
    name: GitHub Release
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Release Action
        uses: ncipollo/release-action@v1.14.0
        with:
          tag: ${{ needs.generate_git_tags.outputs.output_new_tag }}
          token: ${{ secrets.CHANGELOG_RELEASE }}

  generate_sentry_release:
    needs: [test, sonarcloud, generate_git_tags, generate_git_release]
    name: Sentry Release
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
        with:
          fetch-depth: 0
      - name: Split Repo Name
        uses: jungwinter/split@v2.0.0
        id: split_repo_name
        with:
          separator: '/'
          msg: ${{ github.repository }}
      - name: Echo Repo name
        run: echo "${{ steps.split_repo_name.outputs._1 }}"
      - name: Sentry Release
        uses: getsentry/action-release@v1.7.0
        env:
          SENTRY_AUTH_TOKEN: ${{ secrets.SENTRY_AUTH_TOKEN }}
          SENTRY_ORG: ${{ secrets.SENTRY_ORG }}
          SENTRY_PROJECT: ${{ secrets.SENTRY_PROJECT }}
        with:
          environment: 'production'
          version: '${{ steps.split_repo_name.outputs._1 }}@${{ needs.generate_git_tags.outputs.output_new_tag }}'
          sourcemaps: './build'
          url_prefix: '~'

Conclusion

In this article, we have configured a comprehensive CI/CD pipeline using GitHub Actions, SonarCloud, Codecov, and Sentry. This setup ensures our code is built, tested, analyzed for code quality, and monitored for performance issues.

To enhance this pipeline further, you can integrate platform-as-a-service (PaaS) providers like Azure or AWS for deployment. Additionally, incorporating Docker and Docker Hub can streamline your deployment process, making it easier to manage and deploy your applications.

By setting up a robust CI/CD pipeline, you ensure a more reliable, efficient, and maintainable development process, ultimately leading to better software quality and faster delivery.

Feel free to modify this pipeline to better fit your project’s needs and enjoy the advantages of automated and efficient development.

References:
GitHub Actions Documentation
SonarCloud GitHub Action
Codecov GitHub Action
Sentry GitHub Action
Nardy Book REST API

Go Mastery: Advanced Structs and Interfaces

rowjay007 — Mon, 22 Jul 2024 21:27:17 +0000

When building a side project or embarking on a new software development endeavour, it's crucial to consider the architectural and organizational aspects of your code. Whether you're a seasoned developer or new to the Go programming language, understanding how to effectively use structs and interfaces can make a significant difference in how you design, manage, and scale your projects.

Imagine you're tasked with creating a robust application. You need a way to model data efficiently and ensure your code is flexible and maintainable. This is where Go's structs and interfaces come into play. They are not just foundational elements; they are powerful tools that can transform how you handle data and interact with different components of your application.

In this article, we'll dive into the world of structs and interfaces in Go. We'll explore what they are, why they matter, and how you can use them to build better Go applications. Along the way, we'll cover practical examples, advanced use cases, and real-world scenarios to help you master these concepts.

What Are Structs?

Structs in Go are composite data types that group variables under a single name. They are similar to classes in other programming languages but without inheritance. Structs are the primary way to define and organize data in Go, allowing you to model complex entities with multiple attributes.

For instance, in a library management system, if you need to represent books with various attributes such as title, author, and publication year, you can use a struct to model a book like this:

package main

import "fmt"

// Define the Book struct
type Book struct {
    Title  string
    Author string
    Year   int
}

func main() {
    // Create an instance of Book
    myBook := Book{
        Title:  "Go Programming",
        Author: "John Doe",
        Year:   2024,
    }

    // Print the book details
    fmt.Println("Title:", myBook.Title)
    fmt.Println("Author:", myBook.Author)
    fmt.Println("Year:", myBook.Year)
}

In this example:

Book Struct: Defines a Book with fields for title, author, and year.
Instance Creation: We create a Book instance and initialize it with values.
Printing Details: We access and print the book’s details.

Structs help you encapsulate related data and provide a clear, organized way to work with complex information.

Struct Tags: Adding Metadata

Struct tags in Go provide a way to add metadata to your struct fields. They are used for various purposes, such as serialization, validation, and documentation. For example, if you want to serialize a Book struct into JSON, you can use tags to specify the JSON keys :

package main

import (
    "encoding/json"
    "fmt"
)

// Define the Book struct with JSON tags
type Book struct {
    Title  string `json:"title"`
    Author string `json:"author"`
    Year   int    `json:"year"`
}

func main() {
    // Create an instance of Book
    myBook := Book{
        Title:  "Go Programming",
        Author: "John Doe",
        Year:   2024,
    }

    // Serialize the Book instance to JSON
    bookJSON, _ := json.Marshal(myBook)
    fmt.Println(string(bookJSON))
}

In this example:

JSON Tags: Define how struct fields are named in the JSON output.
Serialization: Convert the Book instance to JSON format, which can be useful for APIs or data storage.

Methods on Structs: Adding Behaviour

Methods in Go allow structs to have behaviour in addition to data. By defining methods on structs, you can perform operations related to the data they contain. Here’s how you can add a Details method to the Book struct :

package main

import "fmt"

// Define the Book struct
type Book struct {
    Title  string
    Author string
    Year   int
}

// Method to get book details
func (b Book) Details() string {
    return fmt.Sprintf("Title: %s, Author: %s, Year: %d", b.Title, b.Author, b.Year)
}

func main() {
    // Create an instance of Book
    myBook := Book{
        Title:  "Go Programming",
        Author: "John Doe",
        Year:   2024,
    }

    // Call the Details method
    fmt.Println(myBook.Details())
}

In this example:

Details Method: Provides a formatted string with the book’s information.
Usage: We call the Details method to get a summary of the book.

Methods enhance the functionality of structs, enabling you to encapsulate both data and behaviour.

What Are Interfaces?

Interfaces in Go are a powerful feature that enables you to define methods without specifying exact types. They allow different types to be used interchangeably as long as they implement the required methods. This flexibility is key to writing modular and reusable code.

Let's define a Speaker interface with a single method Speak:

package main

import "fmt"

// Define the Speaker interface
type Speaker interface {
    Speak() string
}

// Define a struct that implements the Speaker interface
type Person struct {
    Name string
}

func (p Person) Speak() string {
    return "Hello, my name is " + p.Name
}

func main() {
    // Create an instance of Person
    p := Person{Name: "Alice"}

    // Declare a variable of type Speaker
    var s Speaker

    // Assign the Person instance to the Speaker variable
    s = p

    // Call the Speak method
    fmt.Println(s.Speak())
}

In this example:

Speaker Interface: Defines a contract with the Speak method.
Person Struct: Implements the Speak method, thus satisfying the interface.
Usage: We use a Speaker variable to hold a Person instance and call the Speak method.

Interfaces provide a way to achieve polymorphism, where different types can be treated the same if they implement the same interface.

The Empty Interface: Flexibility at Its Best

The empty interface (interface{}) is a catch-all type that can hold any value. It’s incredibly versatile and useful when you need a container for values of unknown or varied types.

Here’s an example of using the empty interface:

package main

import "fmt"

// Function that accepts an empty interface
func printValue(value interface{}) {
    fmt.Println(value)
}

func main() {
    // Call the function with different types
    printValue(42)
    printValue("Hello, World!")
    printValue(3.14)
}

In this example:

Empty Interface: interface{} can hold values of any type.
Function Usage: We pass different types of values to the printValue function .

Structs Implementing Interfaces

One of Go's strengths is how structs and interfaces can work together. By implementing interfaces, structs can be used interchangeably, making your code more flexible and modular.

Consider a Vehicle interface with a Drive method and two different structs, Car and Bike:

package main

import "fmt"

// Define the Vehicle interface
type Vehicle interface {
    Drive() string
}

// Define the Car struct
type Car struct {
    Make  string
    Model string
}

func (c Car) Drive() string {
    return "Driving a " + c.Make + " " + c.Model
}

// Define the Bike struct
type Bike struct {
    Brand string
}

func (b Bike) Drive() string {
    return "Riding a " + b.Brand + " bike"
}

func main() {
    // Create instances of Car and Bike
    myCar := Car{Make: "Toyota", Model: "Corolla"}
    myBike := Bike{Brand: "Yamaha"}

    // Declare a slice of Vehicle
    vehicles := []Vehicle{myCar, myBike}

    // Iterate through the slice and call Drive method
    for _, v := range vehicles {
        fmt.Println(v.Drive())
    }
}

In this example:

Vehicle Interface: Defines a common method Drive.
Car and Bike Structs: Implement the Drive method, thus satisfying the Vehicle interface.
Usage: A slice of Vehicle holds both Car and Bike instances and iterates over them.

Interface Composition: Building Complex Behaviours

Go allows you to compose interfaces, creating more complex and specialized interfaces by combining simpler ones. This is useful for creating more granular and reusable components.

Here’s an example of interface composition:

package main

import "fmt"

// Define the Speaker interface
type Speaker interface {
    Speak() string
}

// Define the Writer interface
type Writer interface {
    Write() string
}

// Define a new interface that combines Speaker and Writer
type Communicator interface {
    Speaker
    Writer
}

// Define a struct that implements Communicator
type Person struct {
    Name

 string
}

func (p Person) Speak() string {
    return "Hello, my name is " + p.Name
}

func (p Person) Write() string {
    return "Writing a letter."
}

func main() {
    // Create an instance of Person
    p := Person{Name: "Alice"}

    // Declare a variable of type Communicator
    var c Communicator

    // Assign the Person instance to the Communicator variable
    c = p

    // Call methods from both Speaker and Writer interfaces
    fmt.Println(c.Speak())
    fmt.Println(c.Write())
}

In this example:

Communicator Interface: Combines Speaker and Writer interfaces.
Person Struct: Implements both interfaces, satisfying the Communicator interface.

Applying Structs and Interfaces in App

Building a Simple Inventory System

To illustrate how structs and interfaces work together, let’s build a simple inventory system that manages both products and services. This example will demonstrate how you can use structs and interfaces to handle different types of items.

package main

import "fmt"

// Define the Item interface
type Item interface {
    Name() string
    Price() float64
}

// Define the Product struct
type Product struct {
    name  string
    price float64
}

func (p Product) Name() string {
    return p.name
}

func (p Product) Price() float64 {
    return p.price
}

// Define the Service struct
type Service struct {
    description string
    fee         float64
}

func (s Service) Name() string {
    return s.description
}

func (s Service) Price() float64 {
    return s.fee
}

func main() {
    // Create instances of Product and Service
    prod := Product{name: "Laptop", price: 1200.00}
    serv := Service{description: "Repair", fee: 150.00}

    // Declare a slice of Item
    inventory := []Item{prod, serv}

    // Print details of each item in inventory
    for _, item := range inventory {
        fmt.Printf("Item: %s, Price: %.2f\n", item.Name(), item.Price())
    }
}

In this example:

Item Interface: Defines common methods for items.
Product and Service Structs: Implement the Item interface.
Inventory: Stores and manages different item types using a slice of Item.

Leveraging Dependency Injection

Dependency injection is a design pattern that helps manage dependencies by injecting them into a component rather than hardcoding them. This pattern enhances flexibility and makes testing easier.

Here’s a basic example of dependency injection using interfaces:

package main

import "fmt"

// Define the Database interface
type Database interface {
    Query(query string) string
}

// Define the MySQL struct
type MySQL struct{}

func (m MySQL) Query(query string) string {
    return "MySQL query result for: " + query
}

// Define the MongoDB struct
type MongoDB struct{}

func (m MongoDB) Query(query string) string {
    return "MongoDB query result for: " + query
}

// Define a function that accepts a Database interface
func performQuery(db Database, query string) {
    result := db.Query(query)
    fmt.Println(result)
}

func main() {
    // Create instances of MySQL and MongoDB
    mysql := MySQL{}
    mongodb := MongoDB{}

    // Perform queries using different database implementations
    performQuery(mysql, "SELECT * FROM users")
    performQuery(mongodb, "SELECT * FROM products")
}

In this example:

Database Interface: Defines a Query method.
MySQL and MongoDB Structs: Implement the Database interface.
Dependency Injection: The performQuery function can work with any type that implements the Database interface.

Advanced Topics: Diving Deeper

Type Assertions: Uncovering the Underlying Type

Type assertions are used to retrieve the concrete type of an interface. This can be helpful when you need to work with the actual type stored in an interface.

Here’s an example of type assertions:

package main

import "fmt"

// Define the Item interface
type Item interface {
    Name() string
}

// Define the Product struct
type Product struct {
    name string
}

func (p Product) Name() string {
    return p.name
}

// Define a function that asserts the type of an interface
func assertType(i interface{}) {
    if v, ok := i.(Product); ok {
        fmt.Println("Type is Product, Name:", v.Name())
    } else {
        fmt.Println("Type is not Product")
    }
}

func main() {
    p := Product{name: "Laptop"}

    assertType(p)
}

In this example:

Type Assertion: Checks if the interface holds a Product type and retrieves it if true.

Type Switches: Handling Multiple Types

Type switches allow you to handle different types stored in an interface using a switch-like syntax. This provides a clean way to deal with multiple possible types.

Here’s an example of using a type switch:

package main

import "fmt"

// Define a function that uses a type switch
func handleType(i interface{}) {
    switch v := i.(type) {
    case string:
        fmt.Println("Type is string, Value:", v)
    case int:
        fmt.Println("Type is int, Value:", v)
    default:
        fmt.Println("Unknown type")
    }
}

func main() {
    handleType("Hello")
    handleType(42)
    handleType(3.14)
}

In this example:

Type Switch: Handles different types and performs actions based on the actual type.

Conclusion

Structs and interfaces are powerful features in Go that, when mastered, can greatly enhance your programming skills. They provide the tools you need to build flexible, modular, and maintainable applications.

As you continue to work with Go, keep experimenting with structs and interfaces. Try implementing new interfaces, composing them, and using them in various scenarios. The more you practice, the more proficient you'll become at leveraging these concepts to create high-quality software.

References:

"Effective Go." The Go Programming Language. Effective Go.
Donovan, Alan A.A., and Brian W. Kernighan. The Go Programming Language. Addison-Wesley, 2015.
Cox-Buday, Katrina. Concurrency in Go: Tools and Techniques for Developers. O'Reilly Media, 2017.
"Go by Example: Structs." Go by Example. Go by Example.
"Go by Example: Interfaces." Go by Example. Go by Example.

Unlocking the Potential of Pointers in Go

rowjay007 — Mon, 15 Jul 2024 11:04:29 +0000

Pointers in Go programming are pivotal for efficient memory management and performance optimization. Mastering pointers allows developers precise control over memory addresses, significantly enhancing data manipulation capabilities. This article offers a comprehensive guide to effectively utilizing pointers in Go.

Introduction to Pointers

Pointers in Go are variables that store memory addresses rather than actual values. By referencing these addresses, developers can directly interact with data stored in memory, leading to more efficient data management and improved performance.

Basics of Pointers

Fundamentally, in a Go program, every variable resides at a specific memory location identified by its address. A pointer variable holds these addresses, enabling indirect access to the stored data.

Declaring Pointers

To declare a pointer in Go, specify the type of variable it will reference using the * symbol:

var ptr *int // Declare a pointer to an integer

Here, ptr is a pointer to an integer (*int). It can store the memory address of an integer variable.

Initializing Pointers

Pointers can be initialized using the address-of operator (&) to obtain the memory address of a variable:

num := 42
ptr = &num

This assigns the memory address of num to ptr, allowing ptr to reference and modify num.

Dereferencing Pointers

Accessing the value stored at a memory address referenced by a pointer is done using the dereferencing operator *:

fmt.Println(*ptr)

In this example, *ptr retrieves the value stored at the memory address pointed to by ptr, which is 42.

Pointers to Different Types

Pointers can reference any data type, including structs. For example, when managing a user profile: Pointers can reference any data type, including structs. For example, managing a user profile:

type UserProfile struct {
    ID    string
    Name  string
    Email string
}

profile := UserProfile{ID: "12345", Name: "John Doe", Email: "john.doe@example.com"}
ptr := &profile

fmt.Println(ptr.Name) 
fmt.Println(ptr.Email)

Here, ptr allows indirect access to fields of the profile struct, enabling efficient data manipulation.

Benefits of Using Pointers

Efficient Memory Management

Pointers enable functions to modify data directly, optimizing memory usage.

func updateEmail(profile *UserProfile, newEmail string) {
    profile.Email = newEmail
}

func main() {
    profile := UserProfile{ID: "12345", Name: "John Doe", Email: "john.doe@example.com"}
    updateEmail(&profile, "updated.email@example.com")
    fmt.Println(profile.Email) 
}

By passing a pointer to updateEmail, the function can directly modify the Email field of the UserProfile struct, avoiding the overhead of copying the entire struct.

Passing Large Structures

Remember, passing large structures by pointer can improve performance by avoiding excessive memory usage.

type DetailedUserProfile struct {
    ID        string
    Name      string
    Email     string
    Address   string
    Phone     string
    Biography string
}

func updateBiography(profile *DetailedUserProfile, biography string) {
    profile.Biography = biography
}

func main() {
    profile := DetailedUserProfile{ID: "12345", Name: "John Doe", Email: "john.doe@example.com"}
    updateBiography(&profile, "John Doe is a software engineer with over 10 years of experience...")
    fmt.Println(profile.Biography) 
}

Here, updateBiography updates the Biography field of DetailedUserProfile efficiently by passing a pointer to the struct.

Common Use Cases for Pointers

Managing User Sessions

Pointers are useful for managing user sessions in web applications:

type Session struct {
    UserID string
    Token  string
}

func createSession(userID string) *Session {
    return &Session{UserID: userID, Token: "some-generated-token"}
}

func main() {
    session := createSession("user123")
    fmt.Println(session.UserID) 
    fmt.Println(session.Token) 
}

By returning a pointer to Session from createSession, the function efficiently manages session data.

Modifying Function Arguments

Using pointers allows functions to modify arguments directly:

type Settings struct {
    Theme string
    NotificationsEnabled bool
}

func updateSettings(settings *Settings, theme string, notificationsEnabled bool) {
    settings.Theme = theme
    settings.NotificationsEnabled = notificationsEnabled
}

func main() {
    userSettings := Settings{Theme: "light", NotificationsEnabled: true}
    updateSettings(&userSettings, "dark", false)
    fmt.Println(userSettings.Theme)
    fmt.Println(userSettings.NotificationsEnabled) 
}

Here, updateSettings modifies Settings directly by passing a pointer to the struct, ensuring efficient handling of configuration updates.

Best Practices for Using Pointers

Avoiding Null Pointers

Always remember to check if a pointer is nil before dereferencing to prevent runtime errors.

func safeDereference(ptr *int) {
    if ptr != nil {
        fmt.Println(*ptr)
    } else {
        fmt.Println("Pointer is nil")
    }
}

func main() {
    var ptr *int
    safeDereference(ptr) 

    num := 42
    ptr = &num
    safeDereference(ptr) 
}

By checking for nil, you avoid potential crashes when dereferencing pointers.

Minimizing Pointer Usage

While powerful, excessive use of pointers can overly complicate code. Pointers should be used selectively to enhance performance and functionality.While powerful, excessive pointer usage can complicate code. Use pointers judiciously for performance and functionality:

type CachedData struct {
    Data []byte
}

func processCachedData(cachedData *CachedData) {
    // Process cached data
}

func main() {
    data := CachedData{Data: make([]byte, 1024)}
    processCachedData(&data)
}

It's important to minimize the use of pointers to keep the code clear and maintain its robustness.

Conclusion

Pointers are fundamental in Go programming as they enable efficient memory management and enhance performance. Mastering pointers allows developers to optimize data handling, effectively manage complex structures, and improve application performance. By following best practices and using pointers judiciously, Go programmers can unleash the full potential of this powerful language feature.

This article has explored pointers in Go in-depth, covering their declaration, usage, benefits, common use cases, best practices, and more. With this foundational understanding, developers can confidently integrate pointers into their Go applications, ensuring robust and efficient code.

Navigating Go: Mastering Slices for Flexible Data Handling

rowjay007 — Mon, 08 Jul 2024 08:48:49 +0000

Slices in Go are fundamental for managing datasets with flexibility and efficiency. They offer more flexibility than arrays, allowing for dynamic resizing and more sophisticated operations. This article will walk you through the key aspects of using slices in Go, helping you understand their capabilities and how to use them effectively.

Introduction to Slices

Slices are dynamic and flexible views of the elements of an array. They offer powerful features for working with datasets that are more adaptable than arrays.

Declaration and Initialization

Declaring and initializing slices in Go is simple. Here are some ways to create slices:

// Declare an empty slice
var numbers []int

// Initialize a slice with values
numbers := []int{1, 2, 3, 4, 5}

You can also create a slice from an existing array:

arr := [5]int{10, 20, 30, 40, 50}
slice := arr[1:4] // slice contains {20, 30, 40}

Characteristics of Slices

Dynamic Size: Slices can dynamically grow or shrink, unlike arrays, which have a fixed size.
Reference Type: Slices are references to arrays, meaning that changes made to the slice will affect the underlying array.
Capacity and Length: A slice has two properties: length and capacity. The size indicates the number of elements in the slice, while the capacity is the maximum number of elements it can hold before reallocation is required.

Working with Slices

Accessing and Modifying Elements

Accessing and modifying elements in a slice is straightforward.

numbers := []int{10, 20, 30}
fmt.Println(numbers[0]) // Output: 10
numbers[1] = 25
fmt.Println(numbers[1]) // Output: 25

Appending Elements

Slices in Go can be dynamically resized using the append function:

numbers := []int{10, 20}
numbers = append(numbers, 30)
fmt.Println(numbers) // Output: [10 20 30]

You can also append multiple elements at once:

numbers = append(numbers, 40, 50)
fmt.Println(numbers) // Output: [10 20 30 40 50]

Slicing Slices

Slices can be further sliced to create sub-slices:

numbers := []int{10, 20, 30, 40, 50}
subslice := numbers[1:4] // subslice contains {20, 30, 40}
fmt.Println(subslice)

Capacity and Length

Understanding the length and capacity of slices is crucial for efficient data handling:

numbers := make([]int, 5, 10) // A slice with length 5 and capacity 10
fmt.Println(len(numbers)) // Output: 5
fmt.Println(cap(numbers)) // Output: 10

Iterating Over Slices

Iterating over a slice can be done using a for loop:

numbers := []int{10, 20, 30}
for i, num := range numbers {
    fmt.Printf("Index %d: %d\n", i, num)
}

Passing Slices to Functions

Passing slices to functions allows for flexible and dynamic data management. Here's an example:

func addElement(slice []int, element int) []int {
    return append(slice, element)
}

func main() {
    numbers := []int{10, 20}
    numbers = addElement(numbers, 30)
    fmt.Println(numbers) // Output: [10 20 30]
}

Performance Considerations

Slices offer efficient data handling, but there are performance considerations to keep in mind:

Memory Allocation: When a slice exceeds its capacity, Go allocates a new array and copies the elements, which can be costly for large slices.
Copying Slices: Assigning one slice to another creates a new reference, not a copy of the data. This means modifications through one reference affect the other.

Use Cases for Slices

Example 1: Dynamic Arrays

Slices are perfect for situations where the number of elements is unknown beforehand.

func collectData(data []int) []int {
    var results []int
    for _, value := range data {
        if value > 0 {
            results = append(results, value)
        }
    }
    return results
}

func main() {
    data := []int{1, -2, 3, 4, -5}
    positiveData := collectData(data)
    fmt.Println(positiveData) // Output: [1 3 4]
}

Example 2: Processing Input Data

Slices are perfect for handling variable-sized input data:

func processData(inputs []string) {
    for _, input := range inputs {
        fmt.Println("Processing:", input)
    }
}

func main() {
    inputs := []string{"input1", "input2", "input3"}
    processData(inputs)
}

Example 3: Filtering Data

Slices can be used to filter and create subsets of data:

func filterData(data []int, threshold int) []int {
    var filtered []int
    for _, value := range data {
        if value > threshold {
            filtered = append(filtered, value)
        }
    }
    return filtered
}

func main() {
    data := []int{10, 20, 5, 15, 25}
    filteredData := filterData(data, 15)
    fmt.Println(filteredData) // Output: [20 25]
}

Example 4: Efficient Stacks

Slices can be used to implement stack-like data structures efficiently:

type Stack struct {
    elements []int
}

func (s *Stack) Push(value int) {
    s.elements = append(s.elements, value)
}

func (s *Stack) Pop() int {
    if len(s.elements) == 0 {
        return -1 // Stack is empty
    }
    val := s.elements[len(s.elements)-1]
    s.elements = s.elements[:len(s.elements)-1]
    return val
}

func main() {
    stack := &Stack{}
    stack.Push(10)
    stack.Push(20)
    fmt.Println(stack.Pop()) // Output: 20
    fmt.Println(stack.Pop()) // Output: 10
}

Conclusion

Slices in Go are a powerful and flexible way to handle datasets. Their dynamic nature allows for efficient resizing, appending, and slicing, making them ideal for various applications. By mastering slices, you can take full advantage of Go’s capabilities for dynamic data management. Stay tuned for the next part of this series, where we will explore advanced features and usage patterns of slices.

This article offers a complete guide to understanding and using slices in Go. Feel free to reach out if you have any questions or need further assistance!

Navigating Go: Mastering Arrays for Efficient Data Handling

rowjay007 — Mon, 01 Jul 2024 15:12:02 +0000

Imagine you are the librarian of a well-organized library. Each section of your library contains books of the same genre, and each section has a fixed number of shelves. You know exactly where each book is located because each shelf has a specific spot for each book. This precise organization and fixed structure make it easy to find any book quickly.

In this article, we'll delve into arrays in Go, employing the library analogy to elucidate the concepts clearly and intuitively. Additionally, we'll draw upon explanations from W3Schools, GeeksforGeeks, and Tutorialspoint to bolster our understanding.

Introduction to Arrays

In Go, an array is like a section in a library where each shelf (or array) has a fixed number of slots for books (or elements). Arrays are fixed-size collections that store elements of the same type, providing fast and reliable access to data.

Declaration and Initialization

Just as you might decide to allocate a specific number of shelves for a genre, you declare an array with a specific size in Go:

var shelf [5]int
shelf[0] = 10
shelf[1] = 20
shelf[2] = 30
shelf[3] = 40
shelf[4] = 50

// Or shorthand
books := [5]int{10, 20, 30, 40, 50}

You can also initialize your array with default values or use ellipsis (...) to let Go determine the size based on the number of initial values:

emptyShelf := [5]int{}              // 5 shelves initialized to zero
filledShelf := [...]int{1, 2, 3, 4} // 4 shelves automatically sized

Characteristics of Arrays

Fixed Size: Just as your library has a fixed number of shelves, an array has a fixed size.
Value Type: Like a physical copy of a book, arrays are value types. Assigning an array to another variable copies all its elements.
Performance: Accessing a specific book on a shelf is fast because you know exactly where it is. Similarly, accessing elements in an array is very fast.
Compile-time Check: The number of shelves (array size) is known at compile-time, reducing the chance of runtime errors related to out-of-bounds access.

Working with Arrays

Accessing Elements

Finding a book on a specific shelf is straightforward, just as accessing elements in an array is:

shelf := [5]int{10, 20, 30, 40, 50}
fmt.Println(shelf[0]) // Output: 10
fmt.Println(shelf[4]) // Output: 50

Iterating Over Arrays

To check every book on a shelf, you might go through each one sequentially. In Go, you can iterate over array elements using a for loop and the range keyword:

shelf := [5]int{10, 20, 30, 40, 50}
for i, book := range shelf {
    fmt.Printf("Shelf %d, Book: %d\n", i, book)
}

Or use a traditional for loop:

for i := 0; i < len(shelf); i++ {
    fmt.Printf("Shelf %d, Book: %d\n", i, shelf[i])
}

Passing Arrays to Functions

When you lend an array (or shelf) to a friend, Go makes a copy. If you want them to modify the original, you should lend a pointer to the array.

func updateShelf(s [5]int) {
    s[0] = 100
}

func updateShelfPointer(s *[5]int) {
    s[0] = 100
}

func main() {
    shelf := [5]int{10, 20, 30, 40, 50}

    updateShelf(shelf)
    fmt.Println(shelf) // Output: [10 20 30 40 50]

    updateShelfPointer(&shelf)
    fmt.Println(shelf) // Output: [100 20 30 40 50]
}

Multidimensional Arrays

Sometimes you need multiple sections in your library. A multidimensional array in Go represents this concept:

var library [3][3]int
library[0][0] = 1
library[1][1] = 2
library[2][2] = 3

// Or shorthand
library := [3][3]int{
    {1, 0, 0},
    {0, 2, 0},
    {0, 0, 3},
}

Performance Considerations

Arrays in Go, like well-organized shelves, offer fast access times. This efficiency is due to their contiguous memory layout.

Cache Efficiency: Accessing array elements is cache-friendly due to contiguous memory allocation, resulting in fewer cache misses.
Memory Overhead: Arrays do not have the overhead associated with dynamic memory allocation, making them lightweight and efficient.
Fixed Size: The fixed size means no resizing, leading to predictable performance.

Use Cases for Arrays

Example 1: Fixed-size Buffers

When your library needs a fixed number of shelves for new arrivals, arrays are ideal:

func readBooks(buffer [512]byte) {
    for i := range buffer {
        buffer[i] = byte(i)
    }
    fmt.Println(buffer)
}

func main() {
    var newBooks [512]byte
    readBooks(newBooks)
}

Example 2: Static Lookup Tables

For quick access to a specific genre or section, arrays are perfect:

var genres = [16]string{"Fiction", "Non-Fiction", "Mystery", "Sci-Fi", "Fantasy", "Biography", "History", "Science"}

func getGenre(index int) string {
    if index < 0 || index >= len(genres) {
        return "Unknown"
    }
    return genres[index]
}

func main() {
    fmt.Println(getGenre(2)) // Output: Mystery
    fmt.Println(getGenre(6)) // Output: History
}

Example 3: Matrix and Grid Representations

For arranging books in a matrix format:

func printLibrary(matrix [3][3]int) {
    for _, row := range matrix {
        for _, val := range row {
            fmt.Printf("%d ", val)
        }
        fmt.Println()
    }
}

func main() {
    library := [3][3]int{
        {1, 2, 3},
        {4, 5, 6},
        {7, 8, 9},
    }
    printLibrary(library)
}

Example 4: Image Processing

For representing sections of images:

type Pixel struct {
    R, G, B byte
}

func applyFilter(image [100][100]Pixel) [100][100]Pixel {
    var result [100][100]Pixel
    for i := 0; i < 100; i++ {
        for j := 0; j < 100; j++ {
            gray := (image[i][j].R + image[i][j].G + image[i][j].B) / 3
            result[i][j] = Pixel{gray, gray, gray}
        }
    }
    return result
}

func main() {
    var img [100][100]Pixel
    // Initialize img with some data...
    filteredImg := applyFilter(img)
    fmt.Println(filteredImg)
}

Limitations of Arrays

Just like having a fixed number of shelves limits the number of books you can store, arrays have their limitations:

Fixed Size: Once declared, the size cannot be changed.
Value Semantics: Assigning an array to another variable copies all its elements, which can be inefficient for large arrays.
Syntax: Working with complex, multidimensional arrays can be verbose and cumbersome.

Conclusion

Arrays in Go are similar to fixed shelves as earlier stated in a well-organized library. They provide fast and predictable access to data, making them ideal for situations where the number of elements is known in advance. Although they have limitations in terms of flexibility and value semantics, their efficiency makes them extremely valuable for performance-critical applications.

By mastering arrays, you can efficiently handle fixed-size collections of elements in Go. Stay tuned for the next part of this series, where we will explore slices—Go’s flexible and powerful data structures for handling dynamic collections.

DEV Community: rowjay007

Go and PocketBase: Shortified!

The Monolithic Handler: A Lesson in Technical Debt

The Power of Delegation

Layered Architecture: Enforcing Separation of Concerns

The Flow of a Request: A Practical Walkthrough

Defining the Lines: Stable API Contracts with DTOs

The Anti-Pattern: Exposing Your Database Models Directly

The Solution: Defining Explicit API Contracts with DTOs

The Mapper Pattern: A Clean Implementation

Beyond 500: A Strategy for Meaningful Error Handling

The Anti-Pattern: Generic Errors and String Comparisons

The Solution: Errors as First-Class, Structured Citizens

The Power of Structured Logging

First Line of Defense: Proactive and Centralized Validation

The Anti-Pattern: Scattered and Inconsistent Validation

The Solution: The Gatekeeper Pattern

Types of Validation Implemented

Talking to the Outside World: A Resilient PocketBase Repository

The Anti-Pattern: The Leaky Repository

The Solution: The Adapter Pattern

The Lifeline of a Request: Mastering context.Context

Our Context Propagation Strategy

From Prototype to Better Code: A Blueprint for Your Next Service

Conclusion

Project Resources

Production-Ready Caching Proxy in Go

My Core Principles:

The Architecture: A Tour of the Components

The Heart: A High-Concurrency Caching Engine

The Brain: Smart, 12-Factor Configuration

The Engine Room: A Resilient Proxy Server

The Watchful Eye: A Trilogy of Observability

The Safety Net: Predictable Error Handling

The Conductor: Orchestrating the Application Lifecycle

From Code to Cloud: My Blueprint for Operations

My Toolkit: Taskfile for Automation

The Vessel: A Production-Ready Dockerfile

Thriving in the Kubernetes Ecosystem

Fortifying the Gates: My Approach to Security

The First Line of Defense: Security Middleware

The Principle of Least Privilege

Architecting for Tomorrow: The Path to Hyperscale

Beyond a Single Node: Distributed Caching

Embracing the Service Mesh

The Crucible of Quality: My Testing Strategy

The Foundation: Unit Tests

The Seams: Integration Tests

The Full Picture: End-to-End (E2E) Tests

The Art of the Trade-off: My Engineering Decisions

A Final Thought: Your Turn to Build

References

Deploying NGINX with Vagrant & Making It Publicly Accessible

📌 Introduction

1️⃣ Setting Up the Environment

🖥️ Prerequisites

2️⃣ Creating the Azure Virtual Machine (VM)

3️⃣ Installing and Configuring NGINX

4️⃣ Creating the Custom Welcome Page

5️⃣ Configuring Azure Firewall for Public Access

6️⃣ Testing the Deployment

🚀 Challenges Faced & Solutions

❌ Issue 1: Website Not Loading Externally

❌ Issue 2: NGINX Not Restarting After Reboot

🎯 Lessons Learned & How This Contributes to My DevOps Growth

🔗 References (Cited Properly)

Building a REST API with Golang, Gin, and Pocketbase

Introduction

Why Choose Gin and Pocketbase?

Gin Framework:

Pocketbase:

⚠️ Important Note About Pocketbase:

Prerequisites

Setting Up the Project Structure

Setting Up Pocketbase

Download and Install Pocketbase:

Configuring API URLs

Setting Up Golang with Gin

Install Gin:

API Endpoints and main.go Implementation

The Lifeline of a Request: Mastering `context.Context`

API Endpoints and `main.go` Implementation

Repository Layer (`repository.go`):

Service Layer (`service.go`):

Handler Layer (`handler.go`):

Response Utilities (`util.go`):

JWT Middleware (`middleware.go`):