DEV Community: Muhammetberdi Jepbarov

Merging Redis Serialized HyperLogLog Sets in Golang (Without Redis Commands)

Muhammetberdi Jepbarov — Tue, 25 Feb 2025 08:29:09 +0000

I was looking through questions on Stack Overflow and noticed that one friend needed help with his problem. He wanted to merge two HyperLogLog sets stored in Redis, but instead of using Redis commands like PFMERGE, he wanted to do it inside his Golang application. The challenge was retrieving the serialized HyperLogLog data, deserializing it, merging the sets, and then using the merged result.

At first, this seemed straightforward. Redis provides PFADD for adding elements and PFCOUNT for getting an estimated count. But the real issue? Redis does not store HyperLogLog sets as raw serialized data. That means you can't simply use GET to retrieve a HyperLogLog and merge it manually in Go.

Understanding the Problem

When you use HyperLogLog in Redis, the data is stored in a special format optimized for cardinality estimation. Running GET hll1 does not return a serialized HyperLogLog object but instead results in an error. This happens because Redis does not expose the internal structure of a HyperLogLog directly.

A naive approach would be to assume that Redis stores the raw HyperLogLog object, retrieve it, and attempt to merge it. But that will not work. Instead, a more effective solution is to manage the HyperLogLog instances inside the Go application and store them in Redis as serialized byte arrays.

The Correct Approach: Managing HyperLogLog in Go

Instead of relying on Redis to handle HyperLogLog merging, we should:

Create HyperLogLog instances in Go.
Serialize them and store the serialized data in Redis.
Retrieve them later, deserialize them, and merge multiple instances.
Estimate the final count in Go.

Step 1: Serializing a HyperLogLog and Storing it in Redis

To properly store a HyperLogLog instance, we need to serialize it before saving it in Redis. We use encoding/gob to convert the object into a byte slice:

func serializeHLL(hll *hyperloglog.Sketch) ([]byte, error) {
    var buf bytes.Buffer
    enc := gob.NewEncoder(&buf)
    err := enc.Encode(hll)
    return buf.Bytes(), err
}

In the main() function, we create a HyperLogLog instance, insert some values, serialize it, and store it in Redis:

func main() {
    ctx := context.Background()
    rdb := redis.NewClient(&redis.Options{
        Addr: "localhost:6379",
    })

    hll1 := hyperloglog.New()
    hll1.Insert([]byte("foo"))
    hll1.Insert([]byte("bar"))

    data1, err := serializeHLL(hll1)
    if err != nil {
        log.Fatal("Error serializing hll1:", err)
    }
    rdb.Set(ctx, "hll1", data1, 0)
}

This approach ensures that we have full control over our HyperLogLog data structure.

Step 2: Retrieving and Merging HyperLogLog Sets

To merge two HyperLogLog sets, we first need to deserialize them from the stored byte data in Redis.

func deserializeHLL(data []byte) (*hyperloglog.Sketch, error) {
    var hll hyperloglog.Sketch
    buf := bytes.NewBuffer(data)
    dec := gob.NewDecoder(buf)
    err := dec.Decode(&hll)
    return &hll, err
}

Now, we retrieve the stored HyperLogLog sets, deserialize them, and merge them:

func main() {
    ctx := context.Background()
    rdb := redis.NewClient(&redis.Options{
        Addr: "localhost:6379",
    })

    raw1, err := rdb.Get(ctx, "hll1").Bytes()
    if err != nil {
        log.Fatal("Error fetching hll1:", err)
    }

    raw2, err := rdb.Get(ctx, "hll2").Bytes()
    if err != nil {
        log.Fatal("Error fetching hll2:", err)
    }

    hll1Deserialized, err := deserializeHLL(raw1)
    if err != nil {
        log.Fatal("Error deserializing hll1:", err)
    }

    hll2Deserialized, err := deserializeHLL(raw2)
    if err != nil {
        log.Fatal("Error deserializing hll2:", err)
    }

    err = hll1Deserialized.Merge(hll2Deserialized)
    if err != nil {
        log.Fatal("Error merging HyperLogLog sets:", err)
    }

    fmt.Println("Estimated count after merge:", hll1Deserialized.Estimate())
}

This method ensures we can fully control the HyperLogLog lifecycle, from creation to storage and retrieval.

Why This Works and Other Approaches Fail

Attempting to retrieve HyperLogLog directly from Redis does not work because Redis does not store it as a serialized object.
Using PFMERGE in Redis works but does not allow merging outside Redis, making the logic less flexible.
Manually managing HyperLogLog instances in Go ensures better control, allowing serialization, merging, and estimation without depending on Redis-specific operations.

Conclusion

Instead of relying on Redis commands, we can store serialized HyperLogLog sets manually in Redis, retrieve them in Go, merge them, and get accurate estimates. This gives us more control and flexibility when working with approximate counting in a distributed system.

This approach is beneficial when you need HyperLogLog merging logic outside of Redis, such as in microservices, offline processing, or custom caching layers.

Next time you work with HyperLogLog in Go, try managing your own serialized instances—it might save you a lot of trouble! 🚀

Building Mobile Apps Without a Backend: The Power of Database Gateway API

Muhammetberdi Jepbarov — Thu, 20 Feb 2025 15:21:03 +0000

The Pain of Backend Development

If you've ever built a mobile or web app that interacts with a database, you know the struggle—designing API endpoints, handling authentication, writing business logic, and ensuring scalability. Sometimes, all you need is a simple way to query the database and retrieve structured JSON responses without going through the entire process of backend development.

That’s exactly why many years ago I built the Database Gateway API—a solution that allows developers to interact directly with PostgreSQL and MSSQL databases via a secure API, without writing a full-fledged backend.

The Idea Behind Database Gateway API

The goal was simple: Why should you have to build an entire backend when all you need is an API for your database?

I needed a lightweight yet powerful solution that could:

Eliminate the need for a dedicated backend in simple applications.
Enable mobile and web apps to access database data seamlessly.
Provide an easy way to deploy APIs for database interactions with minimal setup.
Integrate with any existing system, whether it’s an enterprise accounting platform or a retail point-of-sale (POS) system.

This led to the creation of the Database Gateway API—a framework-agnostic API layer that acts as a bridge between your database and applications.

How It Works

The Database Gateway API is a standalone service that connects to your PostgreSQL or MSSQL database and exposes SQL queries as API endpoints. It takes care of request parsing, security, and response formatting, allowing you to focus on building your application instead of writing backend logic.

Key Features:

✅ Direct SQL Query Execution: Supports SELECT, INSERT, UPDATE, and DELETE operations via API requests.

✅ Automatic JSON Responses: Converts database query results into well-structured JSON.

✅ Easy Integration with Mobile & Web Apps: No need for a complex backend—just plug it into your frontend.

✅ Secure & Configurable: Set up role-based access, API keys, and rate limiting.

✅ Supports Complex Queries & Joins: Fetch relational data easily, just like using SQL directly.

✅ Minimal Deployment Overhead: Run it as a standalone service or containerized in Docker.

Real-World Impact: 29 Deployments & 150+ Devices

This API has been successfully deployed across 29 businesses, powering over 150 devices, primarily in shops and malls. It integrates seamlessly with mobile apps and existing accounting systems, allowing businesses to:

Sync sales data in real-time, eliminating manual record-keeping.
Improve customer experience by linking mobile apps to inventory and POS systems.
Enable effortless data exchange between different applications without writing additional backend logic.

Why You Should Use It

If you’re a developer building a mobile app, web dashboard, or prototype, the Database Gateway API can save you weeks of development time by handling database queries and responses automatically.

Instead of spinning up a full backend, setting up ORM models, and writing CRUD endpoints, you simply install the gateway, configure it with your database, and start making API requests.

✅ Instant API – Set up in minutes and start making SQL queries right away.
✅ Effortless Integration – Works with mobile, web, and desktop apps.
✅ JSON-Formatted Responses – Your queries return well-structured JSON, ready to use.
✅ Perfect for Prototyping – Quickly test database interactions without a full backend.
✅ Optimized for Performance – Execute fast queries with minimal setup.

🚀 How It Works
🔍 Querying Your Database

Need to fetch data? Just send a POST request with your SQL query:

POST 127.0.0.1:8000/api/v1/make-db-request
{
"query_string": "SELECT * FROM tbl_mg_materials",
"base64_columns": ["group_code", "image_pict", "firm_id_guid"]
}

Pro tip: Use base64_columns to encode image BLOBs or sensitive data!
📊 The Response

The API returns a structured JSON response:

{
"data": [...],
"status": 1,
"total": 2,
"message": "db query result"
}

⚠️ Important Considerations

🔴 Security First! – This API executes raw SQL, so make sure to restrict access and validate inputs.
🔴 Database Changes? – Schema updates might require adjustments to your API queries.
🔴 Use with Caution – Best for internal tools, rapid prototyping, and trusted environments.

Want to try it out? The Database Gateway API is open-source and available for anyone to use and contribute to. You can integrate it into your next project and cut down on backend development time significantly.

Check out the repository: [Db-Gateway-Go]

Let me know if you have questions or ideas for improvements—happy coding! 🚀

Как я разработал библиотеку Viewscount: Решение проблемы органического подсчета просмотров в Golang

Muhammetberdi Jepbarov — Thu, 20 Feb 2025 14:17:58 +0000

Как разработчики, мы часто сталкиваемся с необходимостью отслеживания просмотров контента на различных платформах. Будь то отслеживание просмотров для статей, видео или продуктов, большинство приложений полагаются на простую колонку view_count в своих таблицах базы данных. Проблема возникает, когда вы не хотите писать отдельное API для каждой таблицы, которая нуждается в подсчете просмотров, и при этом важно обеспечить безопасность от таких атак, как DoS (Denial of Service). Это та проблема, с которой я столкнулся, и причина, по которой я создал библиотеку Viewscount.

В большинстве приложений просмотры отслеживаются в той или иной форме — будь то для блога, страницы продукта или профиля пользователя. Однако создание решения, которое было бы масштабируемым и безопасным, может быть сложной задачей. Вы можете выбрать путь написания индивидуальных API для каждой из этих таблиц, но это отнимает много времени и не масштабируется. А что если можно отслеживать просмотры, не пиша API для каждой новой таблицы? Что если существует более эффективный способ интеграции этой функциональности в любое приложение? Именно в этот момент я начал работать над созданием Viewscount.

Необходимость единого решения

Когда вы строите систему с отслеживанием просмотров, одним из важных факторов является сохранение достоверности данных. В некоторых случаях злоумышленники могут попытаться искусственно увеличить счетчик просмотров, делая повторные запросы — здесь важно предотвратить такие атаки, как DoS (Denial of Service). Без надежного решения ваше приложение становится уязвимым для подобных атак, что подрывает достоверность данных.

Так появилась идея создания библиотеки Viewscount. Я хотел создать решение, которое позволило бы легко отслеживать органические просмотры для любых таблиц, независимо от фреймворка, и при этом предоставить возможность простого интегрирования этого решения в любое приложение, защищая от ненадежных инкрементов. Эта библиотека разработана для того, чтобы интегрироваться с любым Golang фреймворком, таким как Gin, Chi или даже базовым net/http, и быть независимой от базы данных, поддерживая PostgreSQL.

Как работает Viewscount

Когда вы добавляете Viewscount в свое приложение, вы можете начать отслеживать просмотры сразу — без необходимости писать отдельные API для каждой таблицы. Допустим, у вас есть таблицы, такие как tbl_driver или tbl_vehicle, каждая из которых имеет колонку view_count. Вместо того чтобы создавать API специально для инкрементации счетчика просмотров каждый раз, когда пользователь просматривает страницу, вы можете использовать Viewscount для автоматического отслеживания этого процесса.

Вот как это работает на практике:

В вашей базе данных просто добавьте колонку view_count в ваши таблицы:

CREATE TABLE tbl_driver
(
    id              SERIAL PRIMARY KEY,
    first_name      VARCHAR(100) NOT NULL DEFAULT '',
    view_count      INT          NOT NULL DEFAULT 0,
    deleted         INT          NOT NULL DEFAULT 0
);

Аналогично можно сделать и для других таблиц. Не нужно писать сложные запросы или API — просто добавьте эту колонку для отслеживания просмотров.

Интеграция Viewscount в ваше приложение

Самая лучшая часть библиотеки Viewscount заключается в том, что она является независимой от фреймворков. Независимо от того, используете ли вы Gin, Chi или даже net/http, вы можете легко интегрировать ее в ваше приложение. Вот как вы можете добавить ее как middleware в Gin:

Сначала нужно инициализировать view tracker в вашем приложении:

package middlewares

import (
    "fmt"
    "github.com/gin-gonic/gin"
    "github.com/mikebionic/viewscount"
    "time"
)

var viewTracker *viewscount.ViewTracker

func InitializeViewTracker(db interface{}, minutesGap time.Duration) {
    viewTracker = viewscount.NewViewTracker(db, minutesGap)
}

func ViewCounterMiddleware(tableName string) gin.HandlerFunc {
    return func(c *gin.Context) {
        id := c.Param("id")
        if id == "" {
            c.Next()
            return
        }

        idInt := 0
        fmt.Sscanf(id, "%d", &idInt)

        if err := viewTracker.HandleView(c.Request, tableName, idInt); err != nil {
            fmt.Printf("Ошибка отслеживания просмотра: %v\n", err)
        }
        c.Next()
    }
}

Как только middleware настроено, вы можете добавить его к вашим маршрутам. Вот как это делается:

func main() {
    // Инициализация базы данных и middleware
    middlewares.InitializeViewTracker(database.DB, 10)  

    // Определение маршрутов
    router.GET("/:id", middlewares.ViewCounterMiddleware("tbl_driver"), services.GetDriver)
    //...
}

Эта простая настройка интегрирует библиотеку Viewscount в ваше приложение. Middleware будет автоматически отслеживать просмотры на указанной таблице и увеличивать значение в колонке view_count по мере необходимости, при этом обеспечивая защиту от атак.

Преимущества Viewscount

Легкая интеграция: Всего несколько строк кода — и вы уже отслеживаете просмотры на любой из ваших таблиц.
Независимость от фреймворков: Библиотека разработана для работы с любым Golang фреймворком, что позволяет использовать ее в проекте независимо от стека.
Защита от атак DoS: Ограничивая быстрые запросы, которые могут искусственно увеличить счетчик просмотров, Viewscount гарантирует, что данные остаются достоверными и точными.
Эффективность и масштабируемость: Не нужно создавать кастомные API или сложные запросы — используйте библиотеку и отслеживайте просмотры в реальном времени.

Заключение

Viewscount решает общую проблему разработки: как отслеживать просмотры на различных таблицах, обеспечивая безопасность и эффективность, при этом не писать отдельные API для каждой новой таблицы. Легкость интеграции, защита от атак и независимость от фреймворков делают ее мощным инструментом для разработчиков, создающих современные Golang-приложения.

Не стесняйтесь внести вклад в проект и сообщать о проблемах или предложениях через GitHub Viewscount. Я буду рад увидеть, как эта библиотека поможет другим в их разработке!

Счастливого кодинга!

How I Developed the Viewscount Library: Solving the Problem of Organic View Counting in Golang

Muhammetberdi Jepbarov — Thu, 20 Feb 2025 14:16:56 +0000

As developers, we often encounter the need to track views on content across various platforms. Whether it's tracking views for articles, videos, or products, most applications rely on a simple view_count column in their database tables. The challenge arises when you don’t want to go through the hassle of writing a separate API for each table that needs view tracking, while also ensuring that your system is secure from abuse like DoS (Denial of Service) attacks. This is the problem I encountered and the reason I created the Viewscount library.

In most applications, views are tracked in some form—whether it’s for a blog post, a product listing, or even user profile pages. However, implementing a solution that is both scalable and secure can become cumbersome. You could go the route of writing custom APIs for each of these tables, but that’s time-consuming and doesn’t scale well. What if there was a way to track these views without rewriting APIs for every new table? What if there was a more efficient way to integrate this functionality into any app? That’s when I started working on Viewscount.

The Need for a Unified Solution

When building any system with views tracking, one of the critical concerns is ensuring that the view count remains authentic. In some cases, malicious users may attempt to artificially inflate the view count through repeated requests—this is where preventing DOS (Denial of Service) type increments comes into play. Without a solid solution, your application becomes vulnerable to these types of attacks, undermining the integrity of your view count.

Thus, the idea for Viewscount was born. I wanted a way to seamlessly count organic views across any tables, regardless of the framework, while providing an easy-to-integrate middleware solution that prevents abusive increments. This library is designed to integrate with any Golang web framework, like Gin, Chi, or even basic net/http, and is database-agnostic, supporting PostgreSQL seamlessly.

How Viewscount Works

When you add Viewscount to your application, you can begin tracking views instantly—without having to write separate APIs for each table. Let’s say you have tables like tbl_driver or tbl_vehicle, each with a view_count column. Instead of building an API specifically to increment the view_count each time a user views a page, you can use Viewscount to track this automatically.

Here’s how it works in practice:

In your database, you simply add the view_count column to your tables like so:

CREATE TABLE tbl_driver
(
    id              SERIAL PRIMARY KEY,
    first_name      VARCHAR(100) NOT NULL DEFAULT '',
    view_count      INT          NOT NULL DEFAULT 0,
    deleted         INT          NOT NULL DEFAULT 0
);

The same goes for your other tables. No need for complicated queries or APIs—just a simple column to track views.

Integrating Viewscount into Your Application

The best part of Viewscount is that it’s designed to be framework-agnostic. Whether you're using Gin, Chi, or even net/http, you can easily integrate it into your application. Here’s how you can add it as middleware in Gin:

First, you’ll need to initialize the view tracker in your app:

package middlewares

import (
    "fmt"
    "github.com/gin-gonic/gin"
    "github.com/mikebionic/viewscount"
    "time"
)

var viewTracker *viewscount.ViewTracker

func InitializeViewTracker(db interface{}, minutesGap time.Duration) {
    viewTracker = viewscount.NewViewTracker(db, minutesGap)
}

func ViewCounterMiddleware(tableName string) gin.HandlerFunc {
    return func(c *gin.Context) {
        id := c.Param("id")
        if id == "" {
            c.Next()
            return
        }

        idInt := 0
        fmt.Sscanf(id, "%d", &idInt)

        if err := viewTracker.HandleView(c.Request, tableName, idInt); err != nil {
            fmt.Printf("Error tracking view: %v\n", err)
        }
        c.Next()
    }
}

Once you’ve set up the middleware, you can add it to your routes. Here’s how you do it:

func main() {
    // Initialize the database and middleware
    middlewares.InitializeViewTracker(database.DB, 10)  

    // Define routes
    router.GET("/:id", middlewares.ViewCounterMiddleware("tbl_driver"), services.GetDriver)
    //...
}

This simple setup integrates the Viewscount library into your existing app. The middleware will automatically track views on the specified table and increment the view_count column as needed, while also providing protection against abuse.

The Benefits of Viewscount

Seamless Integration: With just a few lines of code, you can integrate view counting into any of your existing tables.
Framework-Agnostic: The library is designed to work with any Golang web framework, making it easy to use in your project regardless of the stack.
Prevention of DOS Attacks: By limiting rapid requests that could artificially inflate view counts, Viewscount ensures that the data remains reliable and accurate.
Efficient and Scalable: No need for custom APIs or complex queries—just use the library and track views automatically in real time.

Conclusion

Viewscount solves a common problem in application development: how to track views across tables securely and efficiently without having to build a separate API for each table. Its easy integration, ability to prevent DOS-style attacks, and framework-agnostic design make it a powerful tool for developers building modern Golang applications.

Feel free to contribute to the project and raise issues or suggestions via Viewscount GitHub. I’d love to see how this library can help others in their development journey!

Happy coding!

Hacking Go-TFE and Fetching All Workspaces in Terraform Enterprise: A Journey Through Pagination

Muhammetberdi Jepbarov — Sun, 16 Feb 2025 11:59:36 +0000

Terraform Enterprise (TFE) is a powerful platform for organizations using Terraform at scale. It provides collaboration, governance, and self-service workflows for infrastructure automation. As organizations grow, managing infrastructure becomes increasingly complex, often requiring the ability to programmatically interact with TFE using its robust API.

Recently, I found myself in a situation where I needed to fetch all workspaces for a given organization in TFE. Workspaces, as you may know, are the fundamental units in Terraform where runs occur. Each workspace holds the Terraform state file, which tracks infrastructure resources. For automation and reporting, I needed to pull in the full list of workspaces, but I quickly ran into a seemingly simple yet frustrating problem: pagination.

The Goal: Get All Workspaces

Our goal was clear — we wanted to retrieve every single workspace associated with a particular organization, without worrying about pagination limits. Whether for generating reports, validating configurations, or orchestrating CI/CD workflows, having a full view of all workspaces was crucial.

The go-tfe library by HashiCorp provides an elegant way to interact with TFE’s API. The most straightforward way to list workspaces looks something like this:

workSpaces, err := tfeClient.Workspaces.List(ctx, orgName, nil)

However, there was a catch.

The Pain Point: Pagination Woes

When we ran this code, we noticed something odd — it only returned 20 workspaces. After digging into the documentation and the source code of the go-tfe library, it became clear that the API defaults to a page size of 20. Even when we explicitly set the page size to the maximum limit of 100, it still only gave us one page of results:

workSpaces, err := tfeClient.Workspaces.List(ctx, orgName, &tfe.WorkspaceListOptions{
    ListOptions: tfe.ListOptions{
        PageNumber: 1,
        PageSize:   100,
    },
})

The issue was evident: the API uses pagination. This means that if your organization has more than 100 workspaces (which is not uncommon for larger teams), you have to make multiple requests to get all of them.

But we didn’t want to be bound by pagination. We wanted all the workspaces in one go — a clean, consolidated list.

The Solution: Handling Pagination Manually

To work around the pagination limit, the only solution was to manually paginate through the results until all workspaces were fetched. We crafted the following solution:

func fetchAllWorkspaces(ctx context.Context, client *tfe.Client, orgName string) ([]*tfe.Workspace, error) {
    var allWorkspaces []*tfe.Workspace
    page := 1
    pageSize := 100 // Max allowed page size

    for {
        opts := &tfe.WorkspaceListOptions{
            ListOptions: tfe.ListOptions{
                PageNumber: page,
                PageSize:   pageSize,
            },
        }

        workspaces, err := client.Workspaces.List(ctx, orgName, opts)
        if err != nil {
            return nil, err
        }

        allWorkspaces = append(allWorkspaces, workspaces.Items...)

        if len(workspaces.Items) < pageSize {
            break // No more pages left
        }

        page++
    }

    return allWorkspaces, nil
}

Breaking Down the Solution

Let’s walk through the logic step by step:

Initialization: We start with an empty slice to hold all workspaces and set the page number to 1.
Pagination Loop: Inside the for loop, we make requests using the ListOptions, setting both the page number and the maximum page size.
Appending Results: After each request, we append the returned workspaces to our allWorkspaces slice.
Exit Condition: If the number of workspaces in the current response is less than the page size (meaning we’ve reached the last page), we break out of the loop.
Result: Finally, we return the full list of workspaces.

Why This Matters

Fetching all workspaces might seem like a trivial task at first glance, but it highlights a crucial aspect of working with APIs — understanding and handling pagination correctly.

When dealing with APIs that support pagination:

Always check default page sizes.
Understand the max page size allowed.
Implement proper loop termination to avoid infinite loops.
Consider API rate limits when making multiple requests.

In our case, the solution was not only about fetching data but about ensuring the stability and reliability of our automation pipelines.

Conclusion

In the end, handling pagination in TFE’s API required us to step back, understand the underlying mechanics, and build a robust way to gather all the data we needed. The final solution is now part of our infrastructure tooling, allowing us to work seamlessly with Terraform Enterprise’s workspaces.

So next time you hit a wall with an API and pagination, remember — it’s not a bug; it’s a feature. Master it, and your automation game will only grow stronger.

Happy coding!

terraform #golang #go-tfe #workspaces #fetching #coding #solution #server #cicd

7 Beneficial News in Go 1.24 That Improves A Programming Experience!

Muhammetberdi Jepbarov — Thu, 13 Feb 2025 18:55:04 +0000

Go 1.24 brings an exciting set of improvements, making the language more efficient, secure, and developer-friendly. Whether you're optimizing performance, improving maintainability, or simplifying debugging, these updates can make a big difference in your projects.

Let's dive into the highlights and see how they can benefit you!

1. Performance Enhancements in Garbage Collection

Go 1.24 introduces optimizations to the garbage collector (GC), reducing pause times and improving memory efficiency.

Why It Matters

If you're working with high-throughput applications like:

Real-time streaming services
Large-scale web applications
High-performance microservices

You’ll see improved responsiveness and reduced latency.

Example Use Case

Before Go 1.24, GC pauses could cause brief performance hiccups in applications processing millions of requests per second. With these improvements, apps handle memory more smoothly, reducing disruptions.

2. Improved Typing and Map Enhancements

Go 1.24 refines type inference and simplifies working with maps.

Why It Matters

More convenient type handling in generics
Easier manipulation of maps with better performance

Example Use Case

data := map[string]int{"apple": 5, "banana": 10}

// New optimizations make working with maps smoother and faster
if qty, ok := data["banana"]; ok {
    fmt.Println("Banana count:", qty)
}

This makes handling maps more predictable and performant in real-world applications.

3. Better Error Handling with `errors.Join`

Error handling has always been a key part of Go, and Go 1.24 improves it by enhancing errors.Join for wrapping multiple errors in a more readable and structured way.

Why It Matters

If your application depends on multiple subsystems (e.g., databases, external APIs), error handling can become complex. This update makes it easier to aggregate errors and present them more clearly.

Example Use Case

package main

import (
    "errors"
    "fmt"
)

func main() {
    err1 := errors.New("database connection failed")
    err2 := errors.New("cache unavailable")

    err := errors.Join(err1, err2)
    fmt.Println(err) // Output: database connection failed; cache unavailable
}

With this change, debugging multi-source failures becomes more intuitive and less time-consuming.

4. `os.OpenRoot()`: Simplified Directory Access

A new function, os.OpenRoot(), has been added to make directory-based operations more convenient and secure.

Why It Matters

Previously, handling root directory paths safely required workarounds. With os.OpenRoot(), directory manipulation becomes easier and more secure.

Example Use Case

package main

import (
    "os"
    "log"
)

func main() {
    dir, err := os.OpenRoot()
    if err != nil {
        log.Fatal(err)
    }
    defer dir.Close()
    log.Println("Root directory opened successfully")
}

This small helper simplifies working with directories in cross-platform applications.

5. JSON Serialization with `omitzero`

A long-requested feature, omitzero, has been introduced for JSON serialization. It allows you to automatically omit zero-value fields when marshaling structs.

Why It Matters

Previously, developers had to manually filter out zero values or use workarounds. Now, you can use omitzero for cleaner JSON outputs without unnecessary fields.

Example Use Case

package main

import (
    "encoding/json"
    "fmt"
)

type User struct {
    Name  string `json:"name"`
    Age   int    `json:"age,omitempty"`
    Email string `json:"email,omitzero"`
}

func main() {
    user := User{Name: "John"}
    data, _ := json.Marshal(user)
    fmt.Println(string(data)) // Output: {"name":"John"}
}

With omitzero, you no longer need to pre-process structs to remove empty fields manually.

6. Improvements to `cmd/go` for Module Proxy Handling

Go 1.24 refines the go command, making module proxy interactions more efficient. This results in faster builds and better dependency management.

Why It Matters

If you're using large Go projects with multiple dependencies, you’ll notice:

Faster go get execution
Improved caching of modules
Better resilience in network-constrained environments

Example Use Case

Before Go 1.24, resolving modules over slow or unreliable networks could be frustrating. Now, with improved proxy handling, you experience fewer build failures and faster package retrieval.

7. Language Refinements and Internal Improvements

Go 1.24 also brings refinements to language internals, including better compiler optimizations and standard library improvements. These updates lead to:

More optimized runtime performance
Smaller binary sizes
Enhanced debugging tools

Why It Matters

For developers building cloud-native applications, CLI tools, or serverless functions, every byte and CPU cycle counts. Go 1.24 helps ensure that your applications run efficiently without unnecessary overhead.

Example Use Case

If you maintain a CLI tool that users download frequently, a smaller binary reduces bandwidth costs and improves execution speed on resource-constrained environments.

Final Thoughts: Why You Should Upgrade to Go 1.24

Go 1.24 is more than just an incremental update—it's a step forward in efficiency, reliability, and developer experience. With key improvements in garbage collection, error handling, typing, module proxy interactions, JSON handling, and performance optimizations, upgrading ensures you stay ahead with a more robust Go environment.

Upgrade today and experience the difference! 🚀

NASA Space Apps Challenge

Muhammetberdi Jepbarov — Sun, 09 Feb 2025 07:47:07 +0000

NASA Space Apps Challenge - dünýäde iň uly we elýeterli hakatonlardan biri bolup, NASA tarapyndan 2012-nji ýylda döredilen global innowasiýa bäsleşigidir. Maksady - jemgyýetiň dürli künjeginden gelen täzelikçi zehinleri bir ýere jemläp, NASA-nyň we beýleki hyzmatdaş guramalaryň kosmos bilen bagly maglumatlaryny ulanmak arkaly häzirki zaman meselelerine çözgüt tapmakdyr. Bu çärä programmistler, dizaýnerler, inženerler, alymlar, mugallymlar we dünýäni gowulandyrmak isleýän ähli adamlar gatnaşyp biler.
Registrasiýa

NASA Space Apps Challenge-na gatnaşmak üçin şu ädimleri ýerine ýetirip bilersiňiz:

Ofisial saýta girip, şahsy maglumatlaryňyzy doldurmak arkaly hasap açyň. Bu saýta girip bilersiňiz: spaceappschallenge.org
Bäsleşigiň ýerleşýän ýerini saýlaň (ýa-da onlaýn formatda hem gatnaşyp bilersiňiz).
Topara goşulyň ýa-da öz toparyňyzy dörediň.

Bäsleşikdäki Challenge-ler

Her ýyl çärä birnäçe kategoriýada täze we döwrebap meseleler (challenge-ler) hödürlenýär. Meseleleriň käbir görnüşleri şulardan ybarat:

Ýer we howa üýtgemesi: Adamzat üçin ýeri we howa üýtgemegini çözmek.
Käin öwrenmek: Planetalaryň we ýyldyzlaryň hereketini analiz etmek.
Innowasiýa we tehnologiýa: Täze tehnologiýalar we işleşýän çözgütler döretmek.
Data analitikasy: NASA-nyň açyk maglumatlaryny ulanmak bilen, maglumatlaryň üstünde işlemek.
Häsiýet we dizaýn: Dizaýnlaryň we infografikalaryň kömegi bilen kosmos maglumatlaryny köpçülige düşündirmek.

Bu meseleler hem-de olaryň düşündirişi bäsleşik başlamazdan öň maglumatlarda berilýär. Her kim öz gyzyklanýan meselelerini saýlap, çözgüt işläp taýýarlap biler.
Şoňa meňzeş çäreler

Google Code-in: Okuwa gatnaşýan ýaş programmistler üçin bäsleşik.
Hack The Planet: Ekologiýa we klimata degişli meseleleri çözmek maksady bilen geçirilen global hakaton.
European Space Agency (ESA) Challenge: ESA tarapyndan gurnalan kosmos temasyndaky bäsleşikler.

NASA Space Apps Challenge, pikir alşyp, täze çözgütler döretmek we tehnologik mümkinçilikleri ösdürmek isleýänler üçin ajaýyp mümkinçilikdir.

https://www.spaceappschallenge.org/nasa-space-apps-2024/2024-local-events/ashgabat/

The 2024 NASA Space Apps Challenge took place on October 5-6, 2024. It is a global hackathon that welcomes participants from all over the world, both virtually and in-person, to work on challenges related to space and Earth science using NASA's open data. This year's theme, "The Sun Touches Everything," encourages participants to explore the influence of the Sun in various fields such as data visualization, AI, machine learning, and more.

You can register for the event and find all necessary details on the official website spaceappschallenge.org. For questions about local events or organizing participation, it's best to contact local leads or check the platform's contact information.

Winners of the hackathon can receive global recognition. In previous years, there have been categories such as "Most Disruptive," "Best Use of Data," and "People’s Choice." These awards are evaluated by experts from NASA and other partner space agencies

Deploying Secure Go Apps on GCP: A Journey from Local to Cloud

Muhammetberdi Jepbarov — Sat, 08 Feb 2025 13:19:30 +0000

It was 3 AM, and I was staring at my computer screen, trying to figure out why our Go application kept crashing in production. Our small e-commerce startup had just experienced its first major traffic spike, and our hosting setup wasn’t ready for it. That night changed how I thought about deploying Go applications forever.

The Problem We Faced

Our team had built a simple but powerful inventory management system using Go. It worked perfectly on our local machines, but when we deployed it to production, we faced several challenges:

Random crashes during peak hours
Security vulnerabilities we hadn’t considered
Scaling issues that cost us customers
Credentials being accidentally exposed in our code

Enter Google Cloud Platform (GCP)

After that difficult night, we decided to rebuild our deployment process using GCP. Here’s the story of how we transformed our application deployment from a source of stress into a smooth, secure operation.

Setting Up a Secure Foundation

First, let’s look at how we structured our Go application for secure deployment. Here’s a basic example of our initial setup:

`package main
import (
"log"
"os"
"net/http"
"github.com/joho/godotenv"
)
func main() {
// Load environment variables
if err := godotenv.Load(); err != nil {
log.Printf("No .env file found")
}

// Get port from environment variable
port := os.Getenv("PORT")
if port == "" {
port = "8080"
}

http.HandleFunc("/", handleHome)
log.Printf("Starting server on port %s", port)
log.Fatal(http.ListenAndServe(":" + port, nil))
}
func handleHome(w http.ResponseWriter, r *http.Request) {
// Implement secure headers
w.Header().Set("X-Frame-Options", "DENY")
w.Header().Set("X-Content-Type-Options", "nosniff")
w.Header().Set("Content-Security-Policy", "default-src 'self'")

w.Write([]byte("Welcome to our secure Go application!"))
}`

Securing Secrets with GCP Secret Manager

One of our biggest concerns was managing secrets. Instead of storing sensitive data in environment variables, we moved to GCP Secret Manager:
`
import (
secretmanager "cloud.google.com/go/secretmanager/apiv1"
secretmanagerpb "google.golang.org/genproto/googleapis/cloud/secretmanager/v1"
)
func getSecret(projectID, secretID string) (string, error) {
ctx := context.Background()
client, err := secretmanager.NewClient(ctx)
if err != nil {
return "", err
}
defer client.Close()

name := fmt.Sprintf("projects/%s/secrets/%s/versions/latest", projectID, secretID)
req := &secretmanagerpb.AccessSecretVersionRequest{
Name: name,
}

result, err := client.AccessSecretVersion(ctx, req)
if err != nil {
return "", err
}

return string(result.Payload.Data), nil
}`

Deploying to Cloud Run

We chose Cloud Run for deployment because it offered:

Automatic scaling
Pay-per-use pricing
Built-in security features
Simple deployment process

Here’s our Dockerfile that we use for deployment:
`
FROM golang:1.21-alpine
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN go build -o main .

Use a minimal alpine image for the final stage

FROM alpine:latest
WORKDIR /app
COPY - from=0 /app/main .

Run as non-root user

RUN adduser -D appuser
USER appuser
CMD ["./main"]`

Implementing Security Best Practices

We learned to follow these key security practices:

Always use HTTPS
Implement rate limiting
Use proper logging
Regular security scanning
Implement proper authentication

Here’s an example of how we implemented rate limiting:

import ( "golang.org/x/time/rate" "sync" ) type IPRateLimiter struct { ips map[string]*rate.Limiter mu *sync.RWMutex r rate.Limit b int } func NewIPRateLimiter(r rate.Limit, b int) *IPRateLimiter { return &IPRateLimiter{ ips: make(map[string]*rate.Limiter), mu: &sync.RWMutex{}, r: r, b: b, } } func (i *IPRateLimiter) AddIP(ip string) *rate.Limiter { i.mu.Lock() defer i.mu.Unlock() limiter := rate.NewLimiter(i.r, i.b) i.ips[ip] = limiter return limiter } func (i *IPRateLimiter) GetLimiter(ip string) *rate.Limiter { i.mu.Lock() limiter, exists := i.ips[ip] if !exists { i.mu.Unlock() return i.AddIP(ip) } i.mu.Unlock() return limiter }
The Results

After implementing these changes:

Our application handled traffic spikes smoothly
Security vulnerabilities were significantly reduced
Deployment became a straightforward process
Our team could sleep better at night!

Where Can You Use This?

This setup is perfect for:

E-commerce platforms
API services
Web applications
Microservices
Data processing systems
Business automation tools

Conclusion

That 3 AM incident taught us valuable lessons about deploying Go applications securely. By leveraging GCP’s features and following security best practices, we transformed our deployment process from a source of stress into a reliable system.

Remember: security isn’t a one-time setup but a continuous journey. Keep learning, keep improving, and most importantly, keep your applications secure.

Want to learn more? Check out the official Go documentation and GCP’s security best practices guide for more detailed information.

How to build an EASIEST Smart Traffic App using OpenCV with RaspberryPi and Arduino | IoT Application

Muhammetberdi Jepbarov — Sat, 08 Feb 2025 13:17:20 +0000

Automation system for traffic control, vehicle and pedestrian detection.

Project Source code is here

Basic goals

This project is intended to improve road traffic processes and ensure pedestrian safety.

The program should:

Detect vehicles and read traffic information
Make calculations and set favorable traffic light times
Keep pedestrians and vehicles safe

How does it work?

Using the technology of computer image and video processing, the program reads traffic conditions from cameras. This is necessary in order to obtain information about traffic congestion, the number of pedestrians.

The data is transferred to the micro controller, which in turn controls the state and time of the traffic light.

Activate the system

To start the system, you must use a computer or microcomputer with preinstalled software. Also connected to one of the USB connectors of the Arduino microprocessor. To get information about traffic, the program needs to indicate from which sources

you need to read the video stream. It can be two webcams connected to USB ports, or a link to online cameras.

After that, two windows with a video stream will appear on the graphical user interface of the computer. It is necessary for calibrating cameras. After calibrating, you must press the “C” key on the keyboard, after which the program will display two video streams with information about the proportions of objects on the road and their absence. If at this stage you start to arrange cars, the number of proportions will begin to increase and the data on occupancy will be sent to the microcontroller that controls the time of the traffic light. As a result, if there is more vehicle on one road than on the other, the microprocessor will set the time of the green light of the traffic light, determined from the proportion value.

Also, the software part of the image recognition system has another version, which provides machine learning and the creation of car models, according to which the recognition system and proportion calculation will work.

Further development

The next steps would be:

improved recognition
determination of the required recognition zone for a clearer analysis
timely automatic calibrations
identifying traffic information on a web page for road safety officials

Programming languages

Python version above 3.6

C++ with Arduino library

Needed materials

Material + quantity

RaspberryPi 4 x1

Web camera / Network camera stream x2

Arduino x1

Led strip x12 diodes

DEV Community: Muhammetberdi Jepbarov

Merging Redis Serialized HyperLogLog Sets in Golang (Without Redis Commands)

Understanding the Problem

The Correct Approach: Managing HyperLogLog in Go

Step 1: Serializing a HyperLogLog and Storing it in Redis

Step 2: Retrieving and Merging HyperLogLog Sets

Why This Works and Other Approaches Fail

Conclusion

Building Mobile Apps Without a Backend: The Power of Database Gateway API

The Pain of Backend Development

The Idea Behind Database Gateway API

How It Works

Key Features:

Real-World Impact: 29 Deployments & 150+ Devices

Why You Should Use It

Как я разработал библиотеку Viewscount: Решение проблемы органического подсчета просмотров в Golang

Необходимость единого решения

Как работает Viewscount

Интеграция Viewscount в ваше приложение

Преимущества Viewscount

Заключение

How I Developed the Viewscount Library: Solving the Problem of Organic View Counting in Golang

The Need for a Unified Solution

How Viewscount Works

Integrating Viewscount into Your Application

The Benefits of Viewscount

Conclusion

Hacking Go-TFE and Fetching All Workspaces in Terraform Enterprise: A Journey Through Pagination

The Goal: Get All Workspaces

The Pain Point: Pagination Woes

The Solution: Handling Pagination Manually

Breaking Down the Solution

Why This Matters

Conclusion

terraform #golang #go-tfe #workspaces #fetching #coding #solution #server #cicd

7 Beneficial News in Go 1.24 That Improves A Programming Experience!

1. Performance Enhancements in Garbage Collection

Why It Matters

Example Use Case

2. Improved Typing and Map Enhancements

Why It Matters

Example Use Case

3. Better Error Handling with errors.Join

Why It Matters

Example Use Case

4. os.OpenRoot(): Simplified Directory Access

Why It Matters

Example Use Case

5. JSON Serialization with omitzero

Why It Matters

Example Use Case

6. Improvements to cmd/go for Module Proxy Handling

Why It Matters

Example Use Case

7. Language Refinements and Internal Improvements

Why It Matters

Example Use Case

Final Thoughts: Why You Should Upgrade to Go 1.24

NASA Space Apps Challenge

Deploying Secure Go Apps on GCP: A Journey from Local to Cloud

Securing Secrets with GCP Secret Manager

Deploying to Cloud Run

Use a minimal alpine image for the final stage

Run as non-root user

Implementing Security Best Practices

How to build an EASIEST Smart Traffic App using OpenCV with RaspberryPi and Arduino | IoT Application

3. Better Error Handling with `errors.Join`

4. `os.OpenRoot()`: Simplified Directory Access

5. JSON Serialization with `omitzero`

6. Improvements to `cmd/go` for Module Proxy Handling