DEV Community: Huynh Thanh Phuc

Why Do You Push Code During Work Hours?" - How an Interview Question Led Me to Build a Delayed Commit Feature

Huynh Thanh Phuc — Tue, 21 Oct 2025 11:13:09 +0000

The Interview That Changed My Perspective

Picture this: I'm sitting in a technical interview, feeling pretty confident about my answers so far. Then the interviewer glances at my GitHub profile and asks a seemingly innocent question:

"I notice you push code mostly during work hours. Can you tell me about your development workflow?"

I froze.

Not because I was doing anything wrong, but because I suddenly realized how much our commit timestamps reveal about our work habits. In that moment, I understood that my GitHub contribution graph wasn't just showing my coding activity—it was inadvertently broadcasting my work schedule to the world.

This interview question got me thinking: What if developers want more control over when their commits appear to be made?

The Problem We Don't Talk About

As developers, we live in a transparent world. Our GitHub profiles are our resumes, our contribution graphs are our performance metrics, and our commit histories are open books. But here's the uncomfortable truth:

Work-life boundaries are blurring: That green square at 2 AM? It might just be because you finally fixed that bug, but it signals "always available" to some employers.
Timezone confusion: Working from Asia but applying to European companies? Your commit times might look odd to recruiters.
Privacy concerns: Not everyone wants their daily routine visible to the world.
Unconscious bias: Some people judge productivity by when commits happen, not what they contain.

After that interview, I decided to do something about it. I built the Delayed Commit feature for GoCommit, my AI-powered commit message generator.

What Is Delayed Commit?

The concept is simple but powerful: prevent commits during your defined work hours and let you schedule them for later.

Here's how it works:

You configure your "restricted hours" (e.g., 9 AM - 5 PM)
When you try to commit during those hours, GoCommit intercepts it
A friendly UI appears, suggesting alternative times
You pick a time (or enter a custom one)
Your commit is created with that timestamp

No scripts, no manual git commands, no fumbling with --date flags. Just a smooth, integrated experience.

The Architecture Behind the Magic

Building this feature was fascinating. Here's what happens under the hood:

1. Time Validation

func isTimeInRestrictedRange(currentTime time.Time, config DelayedCommitConfig) (bool, time.Time, error) {
    hour := currentTime.Hour()

    if hour >= config.RestrictedStartHour && hour < config.RestrictedEndHour {
        // Calculate when restrictions end
        endTime := time.Date(
            currentTime.Year(), currentTime.Month(), currentTime.Day(),
            config.RestrictedEndHour, 0, 0, 0, currentTime.Location(),
        )
        return true, endTime, nil
    }

    return false, time.Time{}, nil
}

2. Smart Time Suggestions

The system generates suggested commit times at configurable intervals (default: 20 minutes) starting from when your restrictions end:

┌──────────────────────────────────────────────────────────┐
│ Commit during work hours (09:00-17:00) detected         │
│ Current time: 14:30                                      │
│                                                          │
│ Select commit time:                                      │
│  → 17:20 (5:20 PM) - Today                              │
│  • 17:40 (5:40 PM) - Today                              │
│  • 18:00 (6:00 PM) - Today                              │
│  • 18:20 (6:20 PM) - Today                              │
│  • Enter custom time...                                  │
│                                                          │
│ ↑↓: Move  Enter: Select  Esc: Use current time anyway   │
└──────────────────────────────────────────────────────────┘

3. Git Integration

Git actually supports custom timestamps natively! The trick is using both the --date flag and the GIT_COMMITTER_DATE environment variable:

func executeDelayedCommit(message string, timestamp time.Time) *exec.Cmd {
    dateStr := timestamp.Format(time.RFC3339)

    cmd := exec.Command("git", "commit", "-m", message, "--date", dateStr)
    cmd.Env = append(os.Environ(),
        fmt.Sprintf("GIT_COMMITTER_DATE=%s", dateStr))

    return cmd
}

This ensures both the author date (when the code was written) and committer date (when the commit was recorded) match your chosen time.

Real-World Use Cases

Since implementing this feature, I've discovered several legitimate use cases:

1. Maintaining Work-Life Boundaries

Developers who want to separate personal projects from work hours without actually changing when they code.

2. Timezone Management

Remote workers who want their contributions to appear during "normal" hours in their employer's timezone.

3. Privacy Protection

Freelancers who don't want clients tracking their exact working hours.

4. Contribution Graph Aesthetics

Let's be honest—some people just want a nicer-looking contribution graph. And that's okay!

5. Retroactive Commits

Sometimes you work on code offline or forget to commit. Being able to set accurate timestamps helps maintain an honest history.

The Ethics Question

Now, I know what you're thinking: "Isn't this... dishonest?"

Let's talk about it.

Here's my take: Git timestamps have always been flexible. You can manually set them, rebase changes, or use any number of tools to modify history. This feature doesn't enable anything new—it just makes an existing capability more accessible and user-friendly.

The real question is: Should your coding schedule be public information?

I argue it shouldn't. Your value as a developer comes from:

The quality of your code
Your problem-solving skills
Your collaboration and communication
The impact of your work

Not from whether you pushed at 2 PM or 2 AM.

This tool helps you control your narrative without lying. You're still writing the code, still making the commits—you're just choosing when they appear in your history.

Getting Started

Want to try it? Here's how to set up GoCommit with delayed commits:

1. Install GoCommit

# Linux/macOS
curl -sSL https://raw.githubusercontent.com/thanhphuchuynh/gocommit/main/install.sh | bash

# Or download from releases
# https://github.com/thanhphuchuynh/gocommit/releases

2. Configure Your API Key

GoCommit uses AI to generate commit messages, so you'll need an API key:

gocommit --config

Choose between Google Gemini or OpenRouter (which gives you access to Claude, GPT-4, Llama, and more).

3. Enable Delayed Commits

gocommit --config-delayed

You'll be prompted to set:

Restricted start hour (e.g., 9 for 9 AM)
Restricted end hour (e.g., 17 for 5 PM)
Suggestion interval (e.g., 20 minutes)

4. Use It Naturally

git add .
gocommit

That's it! If you're committing during restricted hours, you'll see the time selection UI. Otherwise, it works exactly like before.

Configuration File

Under the hood, it's just a simple JSON config at ~/.gocommit.json:

{
  "api_key": "your-api-key",
  "logging_enabled": true,
  "icon_mode": false,
  "delayed_commit": {
    "enabled": true,
    "restricted_start_hour": 9,
    "restricted_end_hour": 17,
    "suggestion_interval_min": 20
  }
}

P.S. If you found this useful, give GoCommit a star on GitHub! It helps more developers discover the tool. Also, GoCommit generates your commit messages with AI, so you get both better messages AND schedule control in one tool.

About the Project

GoCommit is an AI-powered commit message generator that:

Analyzes your staged changes automatically
Generates meaningful, conventional commit messages
Supports multiple AI providers (Gemini, Claude, GPT-4, Llama, etc.)
Includes built-in logging for prompt improvement
Now features delayed commits for schedule control

It's free, open source, and designed to make your Git workflow smoother. Check it out!

GoCommit: Never Write Another Git Commit Message Again 🤖

Huynh Thanh Phuc — Sun, 01 Jun 2025 12:27:29 +0000

We've all been there. You've just spent hours implementing a complex feature or fixing a tricky bug, and then... you stare at the terminal, trying to come up with a meaningful commit message. Should it be "fix stuff" or "update code"? 😅

What if I told you there's a better way? Meet GoCommit - an AI-powered CLI tool that generates meaningful, conventional commit messages automatically by analyzing your staged changes.

The Problem 🤔

Writing good commit messages is hard. It requires:

Understanding what changed and why
Following consistent formatting conventions
Being descriptive but concise
Categorizing changes appropriately (feat, fix, docs, etc.)

Most developers either spend too much time crafting the perfect message or resort to generic ones like "update" or "fix". Both approaches hurt code maintainability and team collaboration.

The Solution ✨

GoCommit leverages Google's Gemini AI to analyze your git diff and generate contextually appropriate commit messages that follow the conventional commits specification.

Here's what makes it special:

🎯 Smart Analysis

The tool examines your staged changes and understands the context - whether you're adding a feature, fixing a bug, updating documentation, or refactoring code.

📝 Conventional Commits

All generated messages follow the conventional commits format:

feat: A new feature
fix: A bug fix
docs: Documentation changes
refactor: Code refactoring
perf: Performance improvements
And more...

🔧 Zero Configuration Hassle

Built-in configuration wizard makes setup a breeze - just run gocommit --config and you're ready to go.

📊 Comprehensive Logging

Every interaction is logged for analysis and prompt improvement, helping the tool get better over time.

Quick Start 🚀

Installation

Linux/macOS:

curl -sSL https://raw.githubusercontent.com/thanhphuchuynh/gocommit/main/install.sh | bash

Windows (PowerShell):

Invoke-WebRequest -Uri "https://raw.githubusercontent.com/thanhphuchuynh/gocommit/main/install.ps1" -OutFile "install.ps1"; .\install.ps1

Setup

# Configure your Gemini API key
gocommit --config

Usage

# Stage your changes
git add .

# Generate and commit with AI
gocommit

That's it! GoCommit will analyze your changes and create a meaningful commit message automatically.

Real-World Example 💡

Let's say you've just added user authentication to your web app. Instead of writing:

git commit -m "auth stuff"

GoCommit might generate:

feat(auth): implement user authentication with JWT tokens

- Add login and registration endpoints
- Implement JWT token generation and validation
- Add password hashing with bcrypt
- Create middleware for protected routes

Much better, right? 🎉

Technical Deep Dive 🔧

Architecture

GoCommit is built in Go and follows a clean architecture:

├── main.go              # CLI entry point and orchestration
├── config/              # Configuration management
│   └── config.go       # API key validation and storage
├── logger/              # Request/response logging
│   └── logger.go       # JSON structured logging
└── tools/               # Analysis utilities
    └── analyze_logs.go  # Usage pattern analysis

Key Features

🔐 Secure Configuration

API keys are validated (must start with "AIza" and be 39 characters)
Stored securely in user's home directory
Easy reconfiguration with --config flag

📈 Smart Logging
All interactions are logged in JSON format for analysis:

{
  "timestamp": "2024-01-15T10:30:00Z",
  "git_diff": "...",
  "ai_prompt": "...",
  "ai_response": "...",
  "user_selection": "...",
  "success": true
}

🔍 Built-in Analytics
Run analysis on your usage patterns:

go run tools/analyze_logs.go

Get insights into:

Success rates and error patterns
Most common commit types
Daily activity patterns
User choice preferences

Cross-Platform Support

GoCommit supports all major platforms:

Linux (x64, ARM64)
macOS (Intel, Apple Silicon)
Windows (x64)
FreeBSD (x64)

Pre-built binaries are available for all platforms, or you can build from source.

Why Choose GoCommit? 🌟

vs. Manual Commit Messages

Consistency: Never worry about format variations again
Speed: Generate messages in seconds, not minutes
Quality: AI understands context better than quick human thoughts

vs. Other AI Tools

Specialized: Built specifically for git workflows
Offline-First: Only sends diffs to AI, not your entire codebase
Logging: Built-in analytics to improve over time
Easy Installation: One-command setup across all platforms

vs. IDE Extensions

Universal: Works with any editor or terminal
Lightweight: No IDE dependencies
Scriptable: Easy to integrate into workflows and scripts

Future Roadmap 🛣️

Some exciting features being considered:

Support for additional AI providers (OpenAI, Claude, etc.)
Template customization for team-specific formats
Integration with popular git workflows
Commit message suggestions based on issue numbers
Team analytics and best practices sharing

Get Started Today! 🎯

Ready to level up your git workflow? Here's how to get started:

Install GoCommit:

   curl -sSL https://raw.githubusercontent.com/thanhphuchuynh/gocommit/main/install.sh | bash

Get a Gemini API key from Google AI Studio
Configure and start using:

   gocommit --config
   # Stage some changes
   git add .
   gocommit

Contributing 🤝

GoCommit is open source and welcomes contributions! Whether it's:

Bug reports and feature requests
Code contributions
Documentation improvements
Usage feedback and suggestions

Check out the GitHub repository to get involved.

Conclusion 🎉

Writing good commit messages doesn't have to be a chore. With GoCommit, you can focus on writing great code while AI handles the documentation part.

The tool is designed by developers, for developers, with real-world workflows in mind. It's fast, reliable, and gets better over time through usage analytics.

Give it a try and let me know what you think! Your future self (and your teammates) will thank you for the clear, consistent commit history.

Have you tried GoCommit? What's your experience with AI-powered development tools? Share your thoughts in the comments below! 👇

P.S. If you found this useful, consider starring the GitHub repo to help others discover it! ⭐

YouTube's transcript feature with proxy

Huynh Thanh Phuc — Tue, 01 Apr 2025 10:52:42 +0000

Unlocking YouTube's Hidden Transcript API

YouTube's transcript feature is a powerful but often overlooked resource. While there's no official API endpoint for accessing transcripts, we can programmatically fetch them. In this article, I'll show you how to work with YouTube's hidden transcript API and share insights from building a TypeScript library for this purpose.

Understanding YouTube Transcripts

YouTube provides transcripts for many videos, either auto-generated or manually added. These transcripts include:

Text content
Timestamps for each segment
Duration information
Language information

The Technical Implementation

Here's how to fetch transcripts using TypeScript:

import { YoutubeTranscript } from 't-youtube-transcript-fetcher';

// Basic usage with video URL
const transcript = await YoutubeTranscript.fetchTranscript(
  'https://www.youtube.com/watch?v=dQw4w9WgXcQ'
);

// Using video ID with specific language
const spanishTranscript = await YoutubeTranscript.fetchTranscript(
  'dQw4w9WgXcQ',
  { lang: 'es' }
);

The response is an array of transcript segments:

interface TranscriptSegment {
  text: string;     // The text content
  duration: number; // Duration in seconds
  offset: number;   // Start time in seconds
  lang: string;     // Language code
}

Handling Edge Cases

When working with YouTube's transcript API, you'll need to handle various scenarios:

Rate Limiting: YouTube may rate-limit requests if too many are made in quick succession
Private Videos: Some videos may be private or unavailable
Disabled Transcripts: Content creators can disable transcripts
Language Availability: Not all languages may be available for every video

Here's how to implement robust error handling:

try {
  const transcript = await YoutubeTranscript.fetchTranscript(videoId);
  // Process transcript
} catch (error) {
  if (error instanceof RateLimitError) {
    console.error('Too many requests, try again later');
  } else if (error instanceof VideoUnavailableError) {
    console.error('Video is private or doesn\'t exist');
  } else if (error instanceof TranscriptDisabledError) {
    console.error('Transcripts are disabled for this video');
  } else if (error instanceof NoTranscriptError) {
    console.error('No transcripts available');
  } else if (error instanceof LanguageNotFoundError) {
    console.error('Requested language not available');
  }
}

Advanced Usage: Proxy Support

For scenarios requiring proxy support (e.g., handling geographic restrictions or load balancing), you can configure proxy settings:

import { HttpsProxyAgent } from 'https-proxy-agent';

// Using a pre-configured proxy agent
const proxyAgent = new HttpsProxyAgent('https://username:password@proxy.example.com:8080');
const transcript = await YoutubeTranscript.fetchTranscript('VIDEO_ID', {
  proxyAgent: proxyAgent
});

// Alternative: Using proxy configuration object
const transcriptWithProxyConfig = await YoutubeTranscript.fetchTranscript('VIDEO_ID', {
  proxy: {
    host: 'http://proxy.example.com:8080',
    auth: {
      username: 'your-username',
      password: 'your-password'
    }
  }
});

Practical Applications

This API access enables various applications:

Content Analysis: Analyze video content programmatically
Subtitle Generation: Create custom subtitles or closed captions
Search Indexing: Make video content searchable
Language Learning Tools: Build tools for studying content in different languages
Accessibility Features: Improve video accessibility with text-based content

Best Practices

When working with YouTube's transcript API:

Implement Rate Limiting: Add delays between requests to avoid hitting limits
Cache Results: Store transcripts to minimize API calls
Handle Languages Gracefully: Check language availability before requesting specific translations
Validate Input: Ensure video IDs/URLs are valid before making requests
Consider Proxy Rotation: For large-scale applications, rotate through multiple proxies

Conclusion

YouTube's hidden transcript API offers powerful capabilities for developers. While not officially documented, it provides reliable access to video transcripts when handled properly. The TypeScript implementation shown here demonstrates how to access this functionality while handling edge cases and providing proxy support for more robust applications.

The complete implementation is available as an npm package: t-youtube-transcript-fetcher. Check out the source code on GitHub for more details and implementation examples.

Remember to respect YouTube's terms of service and implement proper rate limiting in your applications. Happy coding! 🚀

Building a Simple High-Performance PubSub System with Functional Go (PushInterval)

Huynh Thanh Phuc — Sat, 29 Mar 2025 04:02:17 +0000

Building a High-Performance PubSub System with Functional Go

As a software engineer passionate about building robust distributed systems, I'd like to share my experience creating t-pubsub, a high-performance publish-subscribe system built with Go using functional programming principles. This project combines modern Go practices with functional programming to create a reliable, metrics-enabled message delivery system.

Why Another PubSub System?

Most PubSub implementations use object-oriented patterns with mutable state, leading to potential race conditions and hard-to-track bugs in concurrent scenarios. t-pubsub takes a different approach by embracing functional programming principles:

Immutability: Operations return new instances instead of modifying existing state
Pure Functions: Minimize side effects for predictable behavior
Higher-Order Functions: Functions that operate on other functions for flexibility

Core Architecture

The system is built around several key components:

// ConsumerFunc defines the signature for consumer functions
type ConsumerFunc func(msg model.Message)

// MessageStore is a functional interface for message operations
type MessageStore interface {
    AddMessage(msg model.Message) MessageStore
    GetMessages() []model.Message
    GetMessageByID(id string) (model.Message, bool)
    ReplaceMessage(id string, msg model.Message) MessageStore
    ClearMessages() MessageStore
}

The core PubSub type encapsulates the message store and consumer functions while maintaining thread safety:

type PubSub struct {
    store        MessageStore
    consumers    []ConsumerFunc
    mutex        sync.RWMutex
    pushInterval time.Duration
    stopChan     chan struct{}
    metrics      *MetricsRegistry
}

Functional Design Patterns

1. Immutable State Management

Instead of modifying state directly, operations return new instances:

// Subscribe adds a new consumer (functional approach)
func Subscribe(ps *PubSub, consumer ConsumerFunc) *PubSub {
    // Create a copy of consumers slice (immutability)
    newConsumers := make([]ConsumerFunc, len(ps.consumers), len(ps.consumers)+1)
    copy(newConsumers, ps.consumers)

    // Add the new consumer
    newConsumers = append(newConsumers, consumer)

    // Create a new PubSub with updated consumers
    newPS := &PubSub{
        store:        ps.store,
        consumers:    newConsumers,
        mutex:        sync.RWMutex{},
        pushInterval: ps.pushInterval,
        stopChan:     ps.stopChan,
        metrics:      ps.metrics,
    }

    return newPS
}

2. Functional Options Pattern

Configuration is handled through functional options, making it flexible and extensible:

// Create PubSub with custom options
ps := pubsub.NewPubSub(
    pubsub.WithPushInterval(100 * time.Millisecond),
    pubsub.WithMessageStore(customStore),
    pubsub.WithMetrics(customMetrics),
)

Real-Time Performance Monitoring

One of the standout features is comprehensive metrics collection integrated with Prometheus and Grafana. The system tracks:

Message rates (published/delivered)
Queue size and utilization
Consumer latency percentiles
Processing time distributions
Error rates

// Metrics are updated automatically during operations
ps.metrics.MessagesPublished.Inc()
ps.metrics.QueueSize.Inc()
ps.metrics.MessagesInQueue.Inc()
ps.metrics.QueueUtilization.Set(float64(len(queueSize)) / float64(1000) * 100)

Batch Processing Capabilities

For high-throughput scenarios, the system supports batch processing:

batchConfig := pubsub.NewBatchConsumerConfig().
    WithBatchSize(10).
    WithTimeout(5 * time.Second)

batchProcessor := func(batch []model.Message) {
    fmt.Printf("Processing batch of %d messages\n", len(batch))
}

ps = pubsub.SubscribeBatchConsumer(ps, batchConfig, batchProcessor)

Performance Tuning

The system includes built-in benchmarking tools for performance optimization:

curl -X POST http://localhost:2112/benchmark \
  -H "Content-Type: application/json" \
  -d '{
    "messageCount": 1000,
    "concurrentProducers": 4,
    "messageSize": 512
  }'

You can also run continuous simulations to test different scenarios:

curl -X POST http://localhost:2112/simulate/start \
  -H "Content-Type: application/json" \
  -d '{
    "messageRate": 50,
    "numberProducers": 2,
    "numberConsumers": 3,
    "duration": "10m"
  }'

Containerized Deployment

The system is Docker-ready with a complete monitoring stack:

PubSub Service: The main application
Prometheus: Metrics collection
Grafana: Real-time visualization

The included Grafana dashboards provide instant visibility into:

Message throughput
Queue metrics
Consumer performance
System health indicators

Conclusion

Building t-pubsub with functional programming principles in Go has resulted in a system that is:

Thread-safe by design
Easy to reason about
Highly testable
Performance-optimized
Ready for production use

The combination of functional patterns with Go's strong concurrency support creates a robust foundation for building distributed messaging systems. The integrated metrics and monitoring capabilities make it suitable for production environments where observability is crucial.

You can find the complete source code on GitHub.

How to automatically scale down a GKE node pool in the staging cluster to save money, with notifications via Slack.

Huynh Thanh Phuc — Tue, 05 Nov 2024 18:00:23 +0000

Overview

This guide demonstrates how to implement an automated/manual solution for scaling down Kubernetes nodepools during off-hours while keeping your team informed through Slack notifications. This approach can significantly reduce cloud costs by ensuring compute resource aren't idle during non-business hours.

Prerequisites

Kubernetes cluster (in post, I will use GKE)
Slack workspace with permissions to create Webhooks
KubeCTL access to your cluster
... ## Solution Components
1. Golang CronJob to push notify yes/no scale up/down
2. Slack notify and recieve message confirm from user
3. Node Scale exc

Implement:

1. Create Slack bot and workspace

https://slack.com/help/articles/206845317-Create-a-Slack-workspace
https://programmingpercy.tech/blog/develop-a-slack-bot-using-golang/

2. Create GKE cluster and create node pool on it

https://cloud.google.com/kubernetes-engine/docs/deploy-app-cluster

3. Prepare k8s conf and bot

Prepare conf

connect to cluster and get kube/congfig Check config

cat ~/.kube/config

Code push message to Slack

type TypeQuestionType string

const (
    QuestionTypeYesNoScaleDown TypeQuestionType = "yes_no_scale_down"
    QuestionTypeYesNoScaleUp TypeQuestionType = "yes_no_scale_up"
)

func sendYesNoQuestion(client *slack.Client, channelID string, question string, typeQuestion TypeQuestionType) (string, error) {
    attachment := slack.Attachment{
        Text: question,
        CallbackID: string(typeQuestion),
        Actions: []slack.AttachmentAction{
            {
                Name: "yes",
                Text: "Yes",
                Type: "button",
                Value: "yes",
            },
            {
                Name: "no", 
                Text: "No",
                Type: "button",
                Value: "no",
            },
        },
    }
    _, timestamp, err := client.PostMessage(
        channelID,
        slack.MsgOptionAttachments(attachment),
    )
    if err != nil {
        return "", err
    }
    lastQuestionTimestamp = timestamp
    return "Response pending", nil
}

result test:

Flow chart

DEMO

https://www.youtube.com/watch?v=EybkQ9KJJbo
github: https://github.com/thanhphuchuynh/bot-slack-autoscale-gke

Building a Simple TCP Port Scanner in Go

Huynh Thanh Phuc — Fri, 01 Dec 2023 09:24:34 +0000

Hey! 🚀

I'm excited to share a simple yet powerful tool I've built using Go - a TCP port scanner. This tool allows you to quickly identify open ports on a target server, making it a handy utility for network administrators and security enthusiasts. Let's dive into the code and explore how it works!

https://github.com/ThanhPhucHuynh/tcp-scanner-go-basic/tree/main



func worker(ports, results chan int) {
    for p := range ports {
        address := fmt.Sprintf("scanme.nmap.org:%d", p)
        conn, err := net.Dial("tcp", address)
        if err != nil {
            results <- 0
            continue
        }
        conn.Close()
        results <- p
    }
}

And call this func in main



ports := make(chan int, MAX_PORTS)

results := make(chan int)

var open_ports []int

for i := 0; i < cap(ports); i++ {

    go worker(ports, results)

}

go func() {

    for i := 1; i <= MAX_PORTS; i++ {

        ports <- i

    }

}()

for i := 0; i < MAX_PORTS; i++ {

    port := <-results

    if port != 0 {

        open_ports = append(open_ports, port)

    }

}

Understanding the Code

First, let's break down the key components of the code:

Goroutines and Channels: The use of goroutines and channels in Go is central to the concurrent execution of tasks. Goroutines are lightweight threads, and channels facilitate communication between them.

worker() Function: This function is responsible for attempting to establish a TCP connection to a given port on the target server. If the connection is successful, the port is considered open, and its number is sent through the results channel.

main() Function: The main function sets up the necessary channels and goroutines, distributing the task of scanning ports among the workers. It then collects the results, filters out closed ports, and finally, prints the open ports in ascending order.

Running the Scanner
To run the port scanner, simply copy the code into a Go file (e.g., main.go) and execute it. The scanner defaults to scanning the well-known ports (1-65535) on the target server scanme.nmap.org. You can customize the target address by modifying the address variable in the worker() function.



go run main.go

Result

How It Works

The scanner utilizes the net.Dial function to attempt a TCP connection to each port. If the connection is successful, the port is considered open, and its number is added to the list of open_ports. The final list is then sorted and printed.

Conclusion

This simple TCP port scanner in Go demonstrates the language's concurrency capabilities. It's a great starting point for those looking to explore network programming or enhance their Go skills. Feel free to experiment, modify, and extend the code to suit your specific needs!

I hope you find this tool useful in your network exploration endeavors. Happy coding!

Building a TypeScript Simple Channel (Like Golang)

Huynh Thanh Phuc — Fri, 17 Nov 2023 10:02:47 +0000

Introduction:

Modern web development often involves managing asynchronous communication between different parts of an application. One powerful tool in this realm is the implementation of channels, providing a clean and efficient way for components or processes to exchange data. In this blog post, we'll explore the creation of a TypeScript Channel, a versatile and elegant solution for asynchronous communication.

Understanding the Need for Channels

In complex applications, different components or modules often need to communicate asynchronously. Traditional callback patterns or event listeners can become cumbersome and lead to callback hell. Channels provide a structured and organized way for these components to share data without tightly coupling them.

TypeScript Channel

The TypeScript Channel class presented here offers a simple yet powerful interface for asynchronous communication. Let's break down its key components and functionalities.

import { Channel } from "typescript-channel";
const chan = new Channel<string>();
...
chan.send(`hello from`); 
...
for await(const data of chan) { // use async iterator to receive data
        console.log(`Received: ${data}`);
 }

Link NPM: https://www.npmjs.com/package/typescript-channel

Conclusion

The TypeScript Simple Channel class provides a powerful and flexible solution for managing asynchronous communication in your applications. By abstracting away the complexities of asynchronous patterns, this implementation fosters cleaner and more maintainable code. Whether you're building a frontend application with React or a backend service with Node.js, incorporating channels can greatly enhance the efficiency and readability of your codebase.

Buy Me a Coffee:

Efficient Retries: Implementing Fibonacci Backoff in JavaScript Fetch Requests

Huynh Thanh Phuc — Fri, 17 Nov 2023 09:53:39 +0000

Introduction:

In the world of web development, handling network requests is a common task. However, network hiccups, server issues, or other transient problems can lead to failed requests. To enhance the robustness of your application, implementing a retry mechanism is crucial. In this blog post, we'll explore an interesting approach called Fibonacci Backoff and learn how to implement it in JavaScript fetch requests.

Understanding the Need for Retry Mechanisms

Network requests are inherently unpredictable. Various factors such as server load, network congestion, or temporary outages can cause requests to fail. To ensure a smooth user experience, it's essential to handle these failures gracefully. This is where retry mechanisms come into play.

The Concept of Fibonacci Backoff

Fibonacci Backoff is a retry strategy that introduces increasing wait times between successive retries. The wait times are determined by the Fibonacci sequence, where each subsequent value is the sum of the two preceding ones. This approach prevents overwhelming the server with rapid, repeated requests while providing a gradually increasing backoff period.

The TypeScript Implementation

Let's delve into a practical implementation of Fibonacci Backoff in TS, specifically in the context of fetch requests. The code snippet provided demonstrates a reusable function, fetchDataWithFibonacciBackoff, designed to fetch data from a given URL with retry capabilities.

export default async function fetchDataWithFibonacciBackoff(
    url: string,
    init?: RequestInit,
    { tries } = { tries: 6 },
): Promise<Response> {
    let controller: AbortController;
    let attempt = 0;
    let fib = [1, 2];

    while (attempt <= tries) {
        try {
            controller = new AbortController();
            const signal = controller.signal;

            const response = await fetch(url, { ...init, signal: signal });
            if (response.ok) {
                return response;
            } else {
                throw new Error(`Request failed with status: ${response.status}`);
            }
        } catch (e) {
            if (attempt === tries) {
                controller.abort();
                throw e; // If it's the last attempt, re-throw the error
            } else {
                const waitTime = fib[attempt];
                console.log(`Attempt ${attempt + 1}/${tries + 1} failed. Retrying in ${waitTime} seconds.`);
                console.error(e);
                await new Promise(resolve => setTimeout(resolve, waitTime * 1000)); // Wait for the specified time
                attempt++;

                // Generate the next Fibonacci number
                fib.push(fib[fib.length - 1] + fib[fib.length - 2]);
            }
        }
    }

    controller.abort();
    throw new Error(`All ${tries + 1} attempts failed.`);
}

Conclusion

Implementing retry mechanisms is a crucial aspect of building robust and resilient applications. The Fibonacci Backoff strategy provides an elegant solution, introducing a balance between responsiveness and avoiding overwhelming server resources. By incorporating this approach into your fetch requests, you can enhance the reliability of your web applications in the face of transient network issues.

Buy Me a Coffee:

Managing Worker Pool State with Prometheus and Grafana in Go - not detail

Huynh Thanh Phuc — Tue, 12 Sep 2023 15:42:22 +0000

In this article, we'll explore how to manage the state of jobs in a Worker Pool in Go while leveraging the power of Prometheus and Grafana for monitoring and visualization.

Introduction

Worker Pools are a common concurrency pattern used in software development to efficiently process tasks or jobs concurrently. Monitoring the state of these worker pools is crucial for ensuring the health and performance of your application. Prometheus and Grafana are excellent tools for this purpose.

Prerequisites

Before we dive into the code, make sure you have Prometheus and Grafana set up and running. If you haven't already, follow the official documentation for installation and configuration.

Setting Up Prometheus Metrics

To begin, let's instrument our Go application to expose Prometheus metrics. We'll create a simple worker pool and monitor the number of jobs in progress.



// Import necessary packages

import (

    "net/http"

    "github.com/prometheus/client_golang/prometheus"

    "github.com/prometheus/client_golang/prometheus/promhttp"

)

// Define Prometheus metrics

var (

    jobQueue              = promauto.NewGauge(prometheus.GaugeOpts{Name: "workerpool_jobs_queued", Help: "Number of jobs currently in the queue"})

    activeWorkers         = promauto.NewGauge(prometheus.GaugeOpts{Name: "workerpool_active_workers", Help: "Number of currently active workers"})

    jobState              = promauto.NewGaugeVec(prometheus.GaugeOpts{Name: "job_state", Help: "The state of the job"}, []string{"jobID", "state"})

    jobProcessingDuration = promauto.NewHistogram(prometheus.HistogramOpts{Name: "job_processing_duration_seconds", Help: "The duration it takes to process a job", Buckets: prometheus.LinearBuckets(1, 1, 10)})

)

// Job represents a unit of work

type Job struct {

    ID    string

    Data  string

    State string // states can be "Queued", "Processing", "Completed"

}

func main() {

    // Register Prometheus metrics

    prometheus.MustRegister(jobsInProgress)

<span class="c">// Start a web server to expose metrics</span>
<span class="n">http</span><span class="o">.</span><span class="n">Handle</span><span class="p">(</span><span class="s">"/metrics"</span><span class="p">,</span> <span class="n">promhttp</span><span class="o">.</span><span class="n">Handler</span><span class="p">())</span>
<span class="k">go</span> <span class="k">func</span><span class="p">()</span> <span class="p">{</span>
    <span class="n">http</span><span class="o">.</span><span class="n">ListenAndServe</span><span class="p">(</span><span class="s">":8080"</span><span class="p">,</span> <span class="no">nil</span><span class="p">)</span>
<span class="p">}()</span>

<span class="c">// Your worker pool logic goes here</span>
<span class="c">// Don't forget to update jobsInProgress as jobs are processed</span>

<span class="c">// Example:</span>
<span class="c">// Assign jobs using goroutines</span>
<span class="k">for</span> <span class="n">i</span> <span class="o">:=</span> <span class="m">1</span><span class="p">;</span> <span class="n">i</span> <span class="o">&lt;=</span> <span class="n">numJobs</span><span class="p">;</span> <span class="n">i</span><span class="o">++</span> <span class="p">{</span>
    <span class="k">go</span> <span class="n">assignJob</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">jobs</span><span class="p">,</span> <span class="n">done</span><span class="p">,</span> <span class="o">&amp;</span><span class="n">wg</span><span class="p">)</span>
<span class="p">}</span>



}

// assignJob creates a new job, sends it for processing, and waits for it to complete

func assignJob(i int, jobs chan<- Job, done <-chan Job, wg *sync.WaitGroup) {

    defer wg.Done()

<span class="n">jobID</span> <span class="o">:=</span> <span class="n">fmt</span><span class="o">.</span><span class="n">Sprintf</span><span class="p">(</span><span class="s">"%s-%d"</span><span class="p">,</span> <span class="n">uuid</span><span class="o">.</span><span class="n">New</span><span class="p">()</span><span class="o">.</span><span class="n">String</span><span class="p">(),</span> <span class="n">i</span><span class="p">)</span>
<span class="n">job</span> <span class="o">:=</span> <span class="o">&amp;</span><span class="n">Job</span><span class="p">{</span><span class="n">ID</span><span class="o">:</span> <span class="n">jobID</span><span class="p">,</span> <span class="n">Data</span><span class="o">:</span> <span class="n">fmt</span><span class="o">.</span><span class="n">Sprintf</span><span class="p">(</span><span class="s">"Data_%d"</span><span class="p">,</span> <span class="n">i</span><span class="p">),</span> <span class="n">State</span><span class="o">:</span> <span class="s">"Queued"</span><span class="p">}</span>
<span class="n">updateJobState</span><span class="p">(</span><span class="n">job</span><span class="p">,</span> <span class="s">"Queued"</span><span class="p">)</span>

<span class="n">jobQueue</span><span class="o">.</span><span class="n">Inc</span><span class="p">()</span>
<span class="n">jobs</span> <span class="o">&lt;-</span> <span class="n">job</span>
<span class="o">&lt;-</span><span class="n">done</span>
<span class="n">jobQueue</span><span class="o">.</span><span class="n">Dec</span><span class="p">()</span>



}

Configuring Grafana Dashboards

With Prometheus metrics exposed, we can now create Grafana dashboards to visualize the state of our worker pool. Configure Grafana to use Prometheus as a data source and create panels that display the job_state metric.



{

    "annotations": {

      "list": [

        {

          "builtIn": 1,

          "datasource": {

            "type": "grafana",

            "uid": "-- Grafana --"

          },

          "enable": true,

          "hide": true,

          "iconColor": "rgba(0, 211, 255, 1)",

          "name": "Annotations & Alerts",

          "type": "dashboard"

        }

      ]

    },

    "editable": true,

    "fiscalYearStartMonth": 0,

    "graphTooltip": 0,

    "id": 2,

    "links": [],

    "liveNow": false,

    "panels": [

      {

        "datasource": {},

        "fieldConfig": {

          "defaults": {

            "color": {

              "fixedColor": "transparent",

              "mode": "fixed"

            },

            "mappings": [],

            "thresholds": {

              "mode": "absolute",

              "steps": [

                {

                  "color": "green",

                  "value": null

                },

                {

                  "color": "red",

                  "value": 80

                }

              ]

            },

            "unit": "none"

          },

          "overrides": []

        },

        "gridPos": {

          "h": 8,

          "w": 6,

          "x": 0,

          "y": 0

        },

        "id": 5,

        "options": {

          "colorMode": "background",

          "graphMode": "area",

          "justifyMode": "auto",

          "orientation": "auto",

          "reduceOptions": {

            "calcs": [

              "lastNotNull"

            ],

            "fields": "",

            "values": false

          },

          "textMode": "auto"

        },

        "pluginVersion": "10.1.0",

        "targets": [

          {

            "datasource": {

              "type": "prometheus",

              "uid": "e294f615-a82c-43ef-a960-86ba0e5b5b11"

            },

            "editorMode": "code",

            "expr": "count(job_state{state=\"Queued\"})",

            "instant": false,

            "legendFormat": "auto",

            "range": true,

            "refId": "A"

          }

        ],

        "title": "Total Jobs",

        "type": "stat"

      },

      {

        "datasource": {},

        "fieldConfig": {

          "defaults": {

            "color": {

              "mode": "thresholds"

            },

            "mappings": [],

            "thresholds": {

              "mode": "absolute",

              "steps": [

                {

                  "color": "green",

                  "value": null

                },

                {

                  "color": "red",

                  "value": 80

                }

              ]

            },

            "unit": "s"

          },

          "overrides": []

        },

        "gridPos": {

          "h": 8,

          "w": 18,

          "x": 6,

          "y": 0

        },

        "id": 3,

        "options": {

          "colorMode": "value",

          "graphMode": "area",

          "justifyMode": "auto",

          "orientation": "auto",

          "reduceOptions": {

            "calcs": [

              "lastNotNull"

            ],

            "fields": "",

            "values": false

          },

          "textMode": "auto"

        },

        "pluginVersion": "10.1.0",

        "targets": [

          {

            "datasource": {

              "type": "prometheus",

              "uid": "eba8c540-ab46-4987-9347-6112f4e0a081"

            },

            "disableTextWrap": false,

            "editorMode": "code",

            "expr": "avg(rate(job_processing_duration_seconds_sum[5m]) / rate(job_processing_duration_seconds_count[5m]))",

            "fullMetaSearch": false,

            "includeNullMetadata": true,

            "instant": false,

            "legendFormat": "auto",

            "range": true,

            "refId": "A",

            "useBackend": false

          }

        ],

        "title": "Time average",

        "type": "stat"

      },

      {

        "datasource": {

          "type": "prometheus",

          "uid": "e6f319b5-9c5c-48df-96f7-40fc9b1c189c"

        },

        "fieldConfig": {

          "defaults": {

            "color": {

              "mode": "thresholds"

            },

            "mappings": [],

            "thresholds": {

              "mode": "absolute",

              "steps": [

                {

                  "color": "green",

                  "value": null

                },

                {

                  "color": "red",

                  "value": 80

                }

              ]

            }

          },

          "overrides": [

            {

              "matcher": {

                "id": "byName",

                "options": "count(job_state{state=\"Completed\"})"

              },

              "properties": [

                {

                  "id": "displayName",

                  "value": "Complete"

                }

              ]

            },

            {

              "matcher": {

                "id": "byName",

                "options": "count(job_state{state=\"Processing\"} unless on(jobID) (job_state{state=\"Completed\"})) or\nvector(0)"

              },

              "properties": [

                {

                  "id": "displayName",

                  "value": "Processing"

                },

                {

                  "id": "color",

                  "value": {

                    "fixedColor": "blue",

                    "mode": "fixed"

                  }

                }

              ]

            },

            {

              "matcher": {

                "id": "byName",

                "options": "count(job_state{state=\"Queued\"} unless on(jobID) (job_state{state=\"Processing\"} or job_state{state=\"Completed\"})) or\nvector(0)"

              },

              "properties": [

                {

                  "id": "displayName",

                  "value": "Queue"

                },

                {

                  "id": "color",

                  "value": {

                    "fixedColor": "yellow",

                    "mode": "fixed"

                  }

                }

              ]

            }

          ]

        },

        "gridPos": {

          "h": 8,

          "w": 24,

          "x": 0,

          "y": 8

        },

        "id": 4,

        "options": {

          "colorMode": "value",

          "graphMode": "area",

          "justifyMode": "auto",

          "orientation": "auto",

          "reduceOptions": {

            "calcs": [

              "lastNotNull"

            ],

            "fields": "",

            "values": false

          },

          "textMode": "auto"

        },

        "pluginVersion": "10.1.0",

        "targets": [

          {

            "datasource": {

              "type": "prometheus",

              "uid": "a93b3831-a11d-4ef3-b5e7-54bc557e33c0"

            },

            "editorMode": "code",

            "expr": "count(job_state{state=\"Queued\"} unless on(jobID) (job_state{state=\"Processing\"} or job_state{state=\"Completed\"})) or\nvector(0)",

            "hide": false,

            "instant": false,

            "legendFormat": "auto",

            "range": true,

            "refId": "Queue"

          },

          {

            "datasource": {

              "type": "prometheus",

              "uid": "eba8c540-ab46-4987-9347-6112f4e0a081"

            },

            "editorMode": "code",

            "exemplar": false,

            "expr": "count(job_state{state=\"Processing\"} unless on(jobID) (job_state{state=\"Completed\"})) or\nvector(0)",

            "hide": false,

            "instant": false,

            "legendFormat": "auto",

            "range": true,

            "refId": "C"

          },

          {

            "datasource": {

              "type": "prometheus",

              "uid": "eba8c540-ab46-4987-9347-6112f4e0a081"

            },

            "editorMode": "code",

            "expr": "count(job_state{state=\"Completed\"})",

            "instant": false,

            "legendFormat": "__auto",

            "range": true,

            "refId": "A"

          }

        ],

        "title": "State Jobs",

        "type": "stat"

      }

    ],

    "refresh": "5s",

    "schemaVersion": 38,

    "style": "dark",

    "tags": [],

    "templating": {

      "list": []

    },

    "time": {

      "from": "now-5m",

      "to": "now"

    },

    "timepicker": {},

    "timezone": "",

    "title": "New dashboard Copy",

    "uid": "ee9cb0d1-2e66-4f16-8dae-166000f285cd",

    "version": 1,

    "weekStart": ""

  }

Logging Worker Pool State

While Prometheus and Grafana are excellent for monitoring, it's often helpful to log the state of your worker pool for debugging and record-keeping purposes. We can use Go's standard logging package for this.



// processJob simulates processing a job by a worker

func worker(id int, jobs <-chan Job, done chan<- Job) {

    for job := range jobs {

        startTime := time.Now()

    <span class="n">activeWorkers</span><span class="o">.</span><span class="n">Inc</span><span class="p">()</span>
    <span class="n">updateJobState</span><span class="p">(</span><span class="n">job</span><span class="p">,</span> <span class="s">"Processing"</span><span class="p">)</span>

    <span class="c">// Simulating job processing</span>
    <span class="n">time</span><span class="o">.</span><span class="n">Sleep</span><span class="p">(</span><span class="n">time</span><span class="o">.</span><span class="n">Duration</span><span class="p">(</span><span class="n">rand</span><span class="o">.</span><span class="n">Intn</span><span class="p">(</span><span class="m">10000</span><span class="p">)</span><span class="o">+</span><span class="m">100</span><span class="p">)</span> <span class="o">*</span> <span class="n">time</span><span class="o">.</span><span class="n">Millisecond</span><span class="p">)</span>

    <span class="n">activeWorkers</span><span class="o">.</span><span class="n">Dec</span><span class="p">()</span>
    <span class="n">jobProcessingDuration</span><span class="o">.</span><span class="n">Observe</span><span class="p">(</span><span class="n">time</span><span class="o">.</span><span class="n">Since</span><span class="p">(</span><span class="n">startTime</span><span class="p">)</span><span class="o">.</span><span class="n">Seconds</span><span class="p">())</span>
    <span class="n">updateJobState</span><span class="p">(</span><span class="n">job</span><span class="p">,</span> <span class="s">"Completed"</span><span class="p">)</span>

    <span class="n">done</span> <span class="o">&lt;-</span> <span class="n">job</span>
<span class="p">}</span>



}

Result

Conclusion

In this article, we've learned how to manage the state of jobs in a Worker Pool in Go while integrating Prometheus and Grafana for monitoring. We've also seen how to log the state of the worker pool for debugging purposes. Monitoring and logging are crucial for maintaining the reliability and performance of your applications, and these tools make the process much more accessible.

Now, go ahead and apply these techniques to your Go applications to ensure they run smoothly and efficiently, even under heavy workloads.

Happy coding!

repo: https://github.com/ThanhPhucHuynh/go-grafana-prometheus-worker-pool

Buy Me a Coffee:

Understanding "Connection Reset by Peer" in GoLang: A Troubleshooting Guide

Huynh Thanh Phuc — Fri, 02 Jun 2023 03:16:32 +0000

Introduction

When working with network connections in application, you may encounter the error message "Connection Reset by Peer". This error message can be confusing and frustrating, especially if you are not familiar with its meaning. In this troubleshooting guide, we will explore what "Connection Reset by Peer" means, what causes it, and how to fix it in your applications.

What is "Connection Reset by Peer"?

"Connection Reset by Peer" is an error message that can occur when working with network connections. It indicates that the remote server or client closed the connection unexpectedly, resulting in the termination of the connection.

This error message can be caused by a variety of factors, such as network connectivity issues, misconfigured firewalls, or improper handling of network errors in your code. Some common causes of "Connection Reset by Peer" error messages include:

Network Issues: The client and server may be on different networks, and network-related problems such as firewalls, proxies, or routers could be interfering with the connection.
Server Overload: The server may be overwhelmed with requests, resulting in it terminating connections to maintain stability.
Incorrect Protocol Handling: If the client and server are using different protocols or versions, it can lead to unexpected behavior and connection resets.
Firewall or Security Configuration: A misconfigured firewall or security settings may reject or terminate connections, triggering the error.

Troubleshooting "Connection Reset by Peer":

Verify Connectivity: Ensure that both the client and server have a stable and reliable network connection. Check for any network-related issues or restrictions that might be causing the error.
Review Server Logs: Examine the server logs to identify any potential issues, such as server overload or resource exhaustion. These logs can provide valuable insights into the root cause of the connection reset.
Check Protocol Compatibility: Confirm that both the client and server are using the same protocol and compatible versions. Incompatible protocols can result in unexpected behavior and connection resets.
Modify Firewall or Security Settings: Temporarily disable firewalls or security measures to determine if they are causing the connection reset. If so, adjust the settings accordingly to allow the connection.
Implement Connection Retry Mechanism: To handle transient network issues or server overload situations, consider implementing a connection retry mechanism in your code. This allows the client to automatically reconnect and recover from connection resets.
Optimize Code and Resource Usage: Review your code to ensure that it is efficient and not unnecessarily consuming excessive resources. Optimize the code and resource usage to reduce the likelihood of encountering connection reset errors.

Example

I was getting a code simple HTTP server and the client



func client() {
    for i := 0; i < 100; i++ {
        res, err := http.Post("http://localhost:9000/test", "text/json", bytes.NewReader(nil))
        if err != nil {
            log.Printf("error sending: %s", err)
            continue
        }

        body, err := ioutil.ReadAll(res.Body)
        defer res.Body.Close()
        if err != nil {
            log.Printf("error reading: %s", err)
            continue
        }
        fmt.Printf("iteration #%0d read: %s", i, string(body))
        time.Sleep(time.Millisecond * 200)
    }
}

func srv() {
    mux := http.NewServeMux()
    mux.HandleFunc("/test", func(w http.ResponseWriter, req *http.Request) {
        fmt.Println(w, "test\n")
    })

    srv := &http.Server{
        Addr:        ":9000",
        Handler:     mux,
        IdleTimeout: time.Millisecond * 300,
    }
    srv.ListenAndServe()
}

func main() {
    go srv()
    time.Sleep(3 * time.Second)
    client()
}

When excusing this code, I get some errors:



2023/06/02 09:44:20 error sending: Post "http://localhost:9000/test": http: server closed idle connection
iteration #16 read: test
2023/06/02 09:44:20 error sending: Post "http://localhost:9000/test": EOF
iteration #18 read: test
2023/06/02 09:44:20 error sending: Post "http://localhost:9000/test": EOF
iteration #20 read: test
2023/06/02 09:44:20 error sending: Post "http://localhost:9000/test": read tcp 127.0.0.1:43258->127.0.0.1:9000: read: connection reset by peer

Cause*:*

HTTP 1.1 introduced the concept of persistent connections, enabling a client to reuse the same connection for multiple requests to a server. Instead of closing the connection immediately after receiving a response, the connection remains open, allowing subsequent requests to benefit from reduced latency and improved performance. This persistence is achieved by keeping the TCP connection alive.

Connection Closure and Non-Idempotent Requests:
In certain scenarios, a server may close the persistent connection unexpectedly, resulting in errors for the client. It is important to understand that non-idempotent requests, such as POST and PATCH, should not be retried when the connection is closed.

Idempotent requests are those that produce the same outcome regardless of how many times they are repeated. On the other hand, non-idempotent requests have the potential to cause side effects on the server with each execution. POST, for example, is commonly used to create new records, and multiple executions would result in the creation of multiple records.

Handling Connection Closure:
When a connection is closed, the client cannot be certain whether the request was successfully executed by the server. Hence, according to RFC 7230, which defines the HTTP 1.1 protocol, it is considered unsafe for the client to automatically retry non-idempotent requests such as POST. The client should instead treat the connection closure as a definitive response from the server and proceed accordingly.

A request method is considered "idempotent" if the intended effect on the server of multiple identical requests with that method is the same as the effect for a single request RFC 7230 4.2.2

Automatic retrying of non-idempotent requests could lead to unintended consequences, such as duplicate records being created on the server. Since the client cannot be certain whether the original request was processed by the server before the connection was closed, retrying the request would violate the idempotent nature of the operation.

My solution:

Disable persistent connection on the client side
Reduce connection timeout on the client side to be less than server’s



  // Set Transport
    t := http.DefaultTransport.(*http.Transport).Clone()
    t.DisableKeepAlives = true
  // option 2
    // t.IdleConnTimeout = time.Millisecond * 100
    c := &http.Client{
        Transport: t,
    }



res, err := c.Post("http://localhost:9000/test", "text/json", bytes.NewReader(nil))

Result:



iteration #0 read: test
iteration #1 read: test
....
iteration #98 read: test
iteration #99 read: test

https://go.dev/play/p/Aw1PXsmm9lm

Conclusion

The "connection reset by peer" error is a common challenge faced by developers working with network programming. By understanding the causes, implementing proper troubleshooting techniques, and following preventive measures, you can effectively overcome this error and build robust and reliable network applications. Remember to leverage the power of GoLang's error-handling mechanisms and regularly monitor and optimize your server's performance to minimize the occurrence of connection reset errors.

Buy Me a Coffee:

Simplifying Message Queueing with Golang and Amazon SQS

Huynh Thanh Phuc — Sun, 28 May 2023 02:20:23 +0000

Introduction

Message queueing is a critical aspect of modern application development, enabling asynchronous communication between different components or services. One of the popular cloud-base message queue services is Amazon Simple Queue Service (SQS). In this blog post, we will explore how to leverage the power of Golang to interact with SQS ad build a simple, efficient message queue system.

What is Amazon SQS?

Amazon Simple Queue Service (SQS) is a fully managed message queue service provided by Amazon Web Services (AWS). It offers a reliable, scalable, and highly available infrastructure for decoupling the components of distributed systems. SQS enables you to send, store, and receive messages between software components, providing flexibility and durability to your application architecture.

Getting Started with Golang and SQS

To begin, you'll need to have a basic understanding of the Go programming language and an AWS account to access Amazon SQS. Let's walk through the steps to set up your development environment and start using Golang with SQS.

Installing the AWS SDK for Go
The AWS SDK for Go provides a convenient way to interact with various AWS services, including SQS. Install the SDK by running the following command in your terminal
```
go get -u github.com/aws/aws-sdk-go
```
Configuring AWS Credentials

To authenticate your application with AWS, you need to provide your AWS
access key ID and secret access key. You can either set these
credentials as environment variables or use AWS CLI's aws configure command to store them in a configuration file.
```
aws configure
```

Importing the Required Packages

Import the necessary packages to interact with SQS and handle AWS SDK operations in your Go code

import (
    "github.com/aws/aws-sdk-go/aws"
    "github.com/aws/aws-sdk-go/aws/session"
    "github.com/aws/aws-sdk-go/service/sqs"
)

Creating an SQS Client

Create an SQS client using the AWS SDK for Go and the configured session

sess := session.Must(session.NewSessionWithOptions(session.Options{
    SharedConfigState: session.SharedConfigEnable,
}))

svc := sqs.New(sess)

Sending Messages to SQS

To send a message to an SQS queue, use the SendMessage function and provide the QueueUrl and MessageBody parameters

queueURL := "YOUR_QUEUE_URL"
messageBody := "Hello, SQS!"

_, err := svc.SendMessage(&sqs.SendMessageInput{
    QueueUrl:    aws.String(queueURL),
    MessageBody: aws.String(messageBody),
})

Receiving Messages from SQS

To receive messages from an SQS queue, use the ReceiveMessage function and specify the QueueUrl

queueURL := "YOUR_QUEUE_URL"

result, err := svc.ReceiveMessage(&sqs.ReceiveMessageInput{
    QueueUrl: aws.String(queueURL),
})
if err != nil {
    // Handle error
}

for _, msg := range result.Messages {
    // Process the received message
    fmt.Println(*msg.Body)
}

Conclusion

By combining the power of Golang and Amazon SQS, you can easily implement a robust message queueing system that facilitates asynchronous communication between components of your distributed applications. Golang's simplicity and concurrency features make it an ideal choice for building high-performance systems, while SQS provides the scalability, durability, and reliability required for modern cloud-based architectures.

Buy Me a Coffee:

Customizing Error Messages in Struct Validation Using Tags in Go

Huynh Thanh Phuc — Sun, 21 May 2023 01:41:47 +0000

Introduction

Struct validation is an essential aspect of building robust applications in Go. It ensures that the data we receive from the process adheres to the expected format and constraints. While the standard validation package in Go provides useful features, customize error messages in struct validation using tags, along with a practical example.

Overview of Struct Validation in Go

Package: https://pkg.go.dev/github.com/go-playground/validator/v10

Struct validation is a crucial process in software development. It ensures that the data received during the process adheres to the expected format and constraints, preventing errors and ensuring data integrity. In Go, this process is made easy by the standard validation package, which provides a variety of useful features.

The validation package in Go allows developers to specify constraints for struct fields using tags. These constraints can include maximum and minimum values, regular expressions, and custom validation functions. By adding these tags to the struct fields, developers can ensure that the data being received meets the expected format and constraints.

Another benefit of using the validation package in Go is the ability to customize error messages. Developers can add tags to their struct fields that specify custom error messages for each constraint. This can help to make error messages more informative and user-friendly, improving the overall user experience.

To demonstrate the power of struct validation in Go, let's consider a practical example. Suppose we are building an e-commerce platform that requires users to enter their payment information. Using the validation package, we can ensure that the credit card number entered by the user is valid and meets the expected format. We can also ensure that the expiration date is valid and that the CVV code is three digits long.

By using the validation package in Go, we can ensure that our e-commerce platform is secure and error-free. This will improve the user experience and help to build trust with our customers.

Section 2: Standard Struct Validation in Go

The validation package in Go provides a variety of useful features for struct validation. To use the validation package, we first need to create a validator instance. We can create a new validator instance using the New() function provided by the validation package.



import "github.com/go-playground/validator/v10"

// Create a new validator instance
validate := validator.New()

Once we have a validator instance, we can use it to validate our struct fields. To do this, we add tags to our struct fields that specify the desired constraints.



type PaymentInfo struct {
    CreditCardNumber string `validate:"required"`
    CVV              string `validate:"required,len=3"`
}

In this example, we have added tags to the fields of our PaymentInfo struct. These tags specify that the CreditCardNumber field must required, the CVV field must contain exactly three digits.

We can then use the validator instance to validate our struct fields.



// Create a new PaymentInfo instance
paymentInfo := PaymentInfo{
    CreditCardNumber: "4111111111111111",
    CVV:              "123",
}

// Validate the PaymentInfo instance
err := validate.Struct(paymentInfo)
if err != nil {
    // Handle validation error
}

If any of the struct fields fail validation, the validate.Struct() function will return a validation error. We can then handle this error in our application code.

Key: 'PaymentInfo.CreditCardNumber' Error:Field validation for 'CreditCardNumber' failed on the 'required' tag
Key: 'PaymentInfo.CVV' Error:Field validation for 'CVV' failed on the 'len' tag

By using the validation package in Go, we can ensure that our struct fields meet the expected format and constraints, improving the overall quality and security of our applications.

Section 3: Customizing Error Messages with Tags

In addition to specifying constraints for struct fields, we can also customize error messages using tags. To do this, we add a message tag to our struct field tags.



type PaymentInfo struct {
    CreditCardNumber string `validate:"required" errormgs:"Invalid credit card is required xxx"`
    CVV              string `validate:"required,len=3" errormgs:"CVV code must be three digits long"`
}

func (m *PaymentInfo) Validate(validate *validator.Validate) error {
    return ValidateFunc[PaymentInfo](*m, validate)
}

My ValidateFunc:



const tagCustom = "errormgs"

func errorTagFunc[T interface{}](obj interface{}, snp string, fieldname, actualTag string) error {
    o := obj.(T)

    if !strings.Contains(snp, fieldname) {
        return nil
    }

    fieldArr := strings.Split(snp, ".")
    rsf := reflect.TypeOf(o)

    for i := 1; i < len(fieldArr); i++ {
        field, found := rsf.FieldByName(fieldArr[i])
        if found {
            if fieldArr[i] == fieldname {
                customMessage := field.Tag.Get(tagCustom)
                if customMessage != "" {
                    return fmt.Errorf("%s: %s (%s)", fieldname, customMessage, actualTag)
                }
                return nil
            } else {
                if field.Type.Kind() == reflect.Ptr {
                    // If the field type is a pointer, dereference it
                    rsf = field.Type.Elem()
                } else {
                    rsf = field.Type
                }
            }
        }
    }
    return nil
}

func ValidateFunc[T interface{}](obj interface{}, validate *validator.Validate) (errs error) {
    o := obj.(T)

    defer func() {
        if r := recover(); r != nil {
            fmt.Println("Recovered in Validate:", r)
            errs = fmt.Errorf("can't validate %+v", r)
        }
    }()

    if err := validate.Struct(o); err != nil {
        errorValid := err.(validator.ValidationErrors)
        for _, e := range errorValid {
            // snp  X.Y.Z
            snp := e.StructNamespace()
            errmgs := errorTagFunc[T](obj, snp, e.Field(), e.ActualTag())
            if errmgs != nil {
                errs = errors.Join(errs, fmt.Errorf("%w", errmgs))
            } else {
                errs = errors.Join(errs, fmt.Errorf("%w", e))
            }
        }
    }

    if errs != nil {
        return errs
    }

    return nil
}

Section 4: Practical Example



// Create a new PaymentInfo instance
paymentInfo := PaymentInfo{
    CreditCardNumber: "",
    CVV:              "123",
}

// Validate the PaymentInfo instance
err := paymentInfo.Validate(validate)

The error messages we specified in the errormgs tag will be returned if the field fails validation. For example, if we set the CreditCardNumber to an empty string and validate the PaymentInfo instance, the error message "Invalid credit card is required xxx" will be returned. By customizing error messages in this way, we can make them more informative and user-friendly, improving the overall user experience of our applications.



CreditCardNumber: Invalid credit card is required xxx (required)
CVV: CVV code must be three digits long (len)

https://go.dev/play/p/Tigv2u0o_51

Section 6: Conclusion

In conclusion, struct validation is an essential aspect of building robust applications in Go. While the standard validation package provides a variety of useful features, customizing error messages with tags can make them more informative and user-friendly, enhancing the overall user experience. By following the practical example presented in this article, developers can ensure that their struct fields meet the expected format and constraints, building trust with their users and improving the quality of their applications.

Buy Me a Coffee: