DEV Community: Tentang Anak Tech Team

Basic Concept & Benefit of API Gateway

Rizky Darmawan — Thu, 27 Mar 2025 04:52:06 +0000

Photo by Javier Allegue Barros on Unsplash

Api gateways are an important component in modern software architecture, especially in systems that implement microservices. Api Gateway acts as the main gateway to receive all incoming API requests. With this role, API Gateway simplifies API management and improves overall system performance and security.

Using API Gateway helps build systems that are scalable and easy to maintain. As system complexity increases, API Gateway plays a critical role in managing effective integration and communication between various backend services and clients.

In this article, we will learn basic concepts and benefits from using API Gateway. The hope is that this article can provide new understanding or strengthen the insights you already have.

What is API Gateway ?

An API Gateway is an API management tool that sits between a client and a collection of backend services. It acts as an intermediary, routing client requests to the appropriate backend service required to fulfill them and then returning the corresponding response Additionally, an API Gateway provides several important features such as load balancing, circuit breaking, logging, authentication, and caching, making it a crucial component in modern API architecture.

In today's fast-paced world of digital development, efficiency is everything, which is why API Gateways have become essential in modern software projects. Acting as a central point of control for API across microservices architecture handling thousands of concurrent API calls efficiently.

How does API Gateway work ?

Api Gateway work with receive request from internal and external, called "API Calls". API Gateway as software layer that manage traffic for routes them ti appropriate API and then delver the responses to the particular user or devic thath made the request.

This the explain API Gateway Workflow from image above:

A client (browser, mobile app, or another service) sends an API request.
The API Gateway processes the request with routes the request to the correct backend service.
The backend service processes the request and sends a response.
The API Gateway modifies or enhances the response.
Finally, the response is returned to the client.

Benefits of API Gateway

Using an API Gateway can bring several benefits such as enhance performance, improve security, easier maintenance of multiple backend services. Here are some key benefits of using an API Gateway:

1. Enhanced security

An API Gateway enforce authentication and authorization policies, ensuring that only authorized users can access backend services. Additionally, it can help mitigate security threats through:

Rate limiting to prevent API abuse and DDoS attacks.
IP whitelisting and blacklisting to restrict access.
TLS encryption to secure communication between clients and services.

2. Better monitoring and visibility

An API gateway can collect metrics and other data about the requests and responses, providing valuable insights into the performance and behavior of the system. This can help to identify and diagnose problems, and improve the overall reliability and resilience of the system.

3. Data Format Transformation

An API Gateway can convert request and response or data data format (e.g., JSON to XML). Making it easier to manage data formats when needed.

4. API Versioning and Backward Compatibility

An API Gateway can manage multiple versions of an API, allowing developers to introduce new features or make changes without breaking existing clients. This enables a smoother transition for clients and reduces the risk of service disruptions.

5. Enhanced Error Handling

An API Gateway can provide a consistent way to handle errors and generate error responses, improving the user experience and making it easier to diagnose and fix issues.

Drawbacks of Impelementation API Gateway

In addition to having many benefits that we get when we implement API Gateway, on the other hand we also have disadvantages in using it, here are some points:

1. Single Point of Failure

Since the API Gateway acts as a central access point for all API requests, it can become a single point of failure if not properly managed. If the gateway experiences downtime, it could disrupt the entire API ecosystem.

2. Additional Complexity

Integrating an API Gateway introduces an additional layer of complexity to your architecture. It requires careful configuration, monitoring, and maintenance. If not managed efficiently, it can slow down development and increase operational overhead.

3. Increased Latency

An API Gateway processes incoming requests before forwarding them to the appropriate services, which can introduce latency. This delay may impact overall system performance, especially if multiple processing steps (such as authentication, logging, and rate limiting) are involved.

4. Cost

Operating an API Gateway, particularly in high-traffic environments, can add significant costs to your infrastructure. Expenses may include hosting, licensing fees, or managed API Gateway services from cloud providers.

Simple Example of an API Gateway

To implement an API Gateway, you can use a popular API Gateway such as Nginx, Kong, KrakenD, and many more popular choices. However, our goal for the topic is to understand how API Gateway works. So, we can understand by simply building a simple API Gateway using golang. Let's begin.

For the complete code you can see following repository: Click me

The structure Folder

We can create a simply the API Gateway following the structure folder:

├── Makefile
├── api_gateway
│   ├── go.mod
│   ├── go.sum
│   └── main.go
├── service_cart
│   ├── go.mod
│   ├── go.sum
│   └── main.go
├── service_order
│   ├── go.mod
│   ├── go.sum
│   └── main.go
└── service_product
    ├── go.mod
    ├── go.sum
    └── main.go

The rate limitter function code

First, we can create a file with the name main.go in the api_gateway folder which contains:

package main

import (
    "time"

    "github.com/gofiber/fiber/v2"
    "github.com/gofiber/fiber/v2/middleware/limiter"
)

func rateLimiter() fiber.Handler {
    return limiter.New(limiter.Config{
        Expiration: time.Second * 60,
        Max:        5,
    })

}

The rateLimiter() function is used to limit the number of requests per client. Where this rate limiter is useful for Preventing abuse & DDoS, Controlling API traffic, Protecting backend services from overload.

With the configuration Expiration: time.Second, Max: 5, each client can only make 5 requests per second. If this limit is reached, the client will receive error HTTP 429 Too Many Requests. After 1 second, the limit will be reset and the client can send requests again.

The proxy function code

Second, we also create a function with the name reverseProxy:


package main

import (
    // ... other import libraries
        "log"
        "github.com/gofiber/fiber/v2/middleware/proxy"
)

// ... rate limitter function

func reverseProxy(target string) fiber.Handler {
    return func(c *fiber.Ctx) error {
        url := target + c.OriginalURL()
        return proxy.Do(c, url)
    }
}

The proxy(target string) function is used to forward requests (proxy requests) from clients to other backend servers specified in the target. This function returns a middleware handler (fiber.Handler) that can be used on Fiber.

We also see c.OriginalURL() returns the path and query parameters of the original request sent by the client. For example, if the client accesses http://localhost:3000/api/orders?id=10, then:

/api/orders?id=10

And last we return proxy.Do(c, url) is used to forward requests to the destination url. This url is a combination of target and c.OriginalURL(), so all incoming requests will be directed to another server with the same path. In this way, requests that enter the API Gateway will be automatically directed to the appropriate backend service without the client knowing.

Then main function code

And latest file we create main function, is a entrypoint of this application. In this case, it acts as an API Gateway. The code is like this:

package main

import (
    // ... other import libraries
        "github.com/gofiber/fiber/v2/middleware/logger"
)


func main() {
    app := fiber.New()

    app.Use(logger.New())

    // Implement rate limitter
    app.Use(rateLimiter())

    // Implement proxy
    // Redirect request to service product
    app.Use("/product", reverseProxy("http://localhost:5001"))

    // Redirect request to service order
    app.Use("/order", reverseProxy("http://localhost:5002"))

    // Redirect request to service cart
    app.Use("/cart", reverseProxy("http://localhost:5003"))

    port := ":3000"
    log.Printf("Starting API Gateway in port %s", port)
    log.Fatal(app.Listen(port))
}

We can focus on the code part:

    // Implement rate limitter
    app.Use(rateLimiter())

This code applies rate limiting to the entire application with Fiber's limiter middleware. This middleware is attached with app.Use(rateLimiter()), which means it will apply to all routes. The function rateLimiter() returns the rate limiter middleware we created earlier with limiter.New().

And also for some of these code:

    // Implement proxy
    // Redirect request to service product
    app.Use("/product", reverseProxy("http://localhost:5001"))

    // Redirect request to service order
    app.Use("/order", reverseProxy("http://localhost:5002"))

    // Redirect request to service cart
    app.Use("/cart", reverseProxy("http://localhost:5003"))

This code uses the reverseProxy function we created earlier, which will forward the request to the appropriate service.

As we know in the The structure Folder section, we also have several service folders that will be directed according to incoming requests from clients.

For the complete code you can see following repository: Click me

Let's run it

To run it there are several ways, if Makefile is installed on your computer you only need to run the command below in the terminal on root folder project:

make run

If not you can run it one by one by going into all folders and starting to run with the command:

cd service_product && go run main.go

cd service_order && go run main.go

cd service_cart && go run main.go

cd api_gateway && go run main.go

If successful, you can see display like this:

From the image above you can see several services running on their respective ports.

Service API Gateway running on port 3000

Service product running on port 5001

Service order running on port 5002

Service cart running on port 5003

Here is the advantage of using API Gateway, imagine you have more than 3 services and on the client we have several features that must be run simultaneously. Do we have to define the url with each specific port? of course it is very troublesome. And later one of the services must change its url address, of course it is even more troublesome to have to refactor here and there on the client side.

But with API Gateway you only need to know the url on the API Gateway and define the appropriate url, let's try to access several services via the API Gateway url only. Run this command:

curl http://localhost:3000/product/1
curl http://localhost:3000/order
curl http://localhost:3000/cart

And the result is you can access multiple services with just one URL.

And in the log you can see that the api gateway actually forwards the request to the service that corresponds to the address being called.

And I give a challenging task, try to make more requests than the maximum rate limiter that we have set in the rateLimiter function until you get the error HTTP 429 Too Many Requests.

Conclusion

API Gateway acts as the primary gateway to manage request traffic in a microservice system. With rate limiting, APIs are more secure from abuse such as DDoS. Reverse proxies allow requests to be directed to the appropriate backend service without changing the client side, increasing flexibility and ease of management. In addition, API Gateway also supports caching, logging, monitoring, and other security features such as authentication & authorization. With proper implementation, API Gateway can improve the security, performance, and scalability of the overall system.

However, implementing API Gateway also has some drawbacks that we must be aware of such as a single point of failure, additional complexity, increased latency, and higher operational costs. Despite these drawbacks, a well-implemented API Gateway remains essential for scalable and secure microservice architectures.

Hopefully this article can help to improve understanding or recall what is already known. If you have additions or corrections to the discussion above, let's discuss it in the comments column. Hopefully it helps 👋.

Reading References

Great Code Starts with Great Planning

Irfan Adisukma — Mon, 24 Feb 2025 07:47:58 +0000

Image Source: GrowMy.tech

Every developer dreams of writing clean, efficient, and impactful code. But what many overlook is the critical step that comes before the first line of code is even written: planning. Without a clear roadmap, projects can quickly spiral into chaos, leading to wasted time, frustration, and subpar results. Taking the time to carefully plan your approach not only sets a solid foundation but also makes the coding process smoother and more productive. In this article, we'll explore why planning is the cornerstone of successful development and how it can elevate your coding projects to new heights.

The Journey of a Beginner Developer

The internet is full of technical tutorials and advice on how to code effectively. But let’s take a step back and look at the bigger picture. No matter how many mistakes you make or how often things don’t go as planned, persistence is key. The journey of a developer comes with challenges, but it’s also incredibly rewarding. Stay committed, keep learning, and trust that your efforts will pay off over time. The code you write today has the potential to make a real impact.

It’s normal to feel discouraged sometimes, even the most experienced ones, has faced similar struggles. If you're feeling stuck, remember that it's just part of the learning process. Even I come across difficult topics that feel overwhelming at times, but with consistent effort, improvement always follows. Keep going, you’re making progress every day.

The Beginner Stage

As a beginner, it often feels like you're on an emotional rollercoaster. Some days, you’re unstoppable, completely focused on coding, excited about your progress, and confident that you’ve found your passion. On other days, the challenges feel too difficult.
A tricky concept or a bug that’s hard to fix might make you question your path, thinking, “Is this really what I want to do for the rest of my life?” It’s frustrating, and it’s easy to feel like the struggle is holding you back.

Navigating the Highs and Lows

This stage is tough, but it’s also crucial. It’s a phase where you’ll inevitably make mistakes and lots of them. There are many reasons for this. Perhaps you’re rushing to land your first job in tech, or maybe you’ve taken on low budget projects just to make ends meet. These pressures can lead to poor decisions, overly optimistic timelines, and code that isn’t as clean or efficient as you’d like. But here’s the truth, this is normal. Every seasoned developer you admire has been through this phase. It’s a rite of passage that helps you grow. You may feel like you’re just stumbling along, but every misstep teaches you something valuable. If you keep pushing forward, you’ll find your rhythm and reach the next level of your journey.

I can confidently tell you this, it’s all worth it. The struggles, the late nights, and even the failed projects they all contribute to your development as a programmer. So, keep coding. Keep learning. The only way to improve is to persevere.

Finding Strength in the Beginner Stage

While this stage might feel overwhelming, it’s also one of the most transformative phases of your journey. It’s here that you develop qualities that will serve you for the rest of your career. Among the most important are:

Willpower: The determination to keep going, even when the odds feel stacked against you.
Curiosity: An insatiable hunger to learn more and explore uncharted territories in programming.
Ambition: The drive to push past your limits and achieve your goals.
Resilience: The courage to embrace failures and use them as stepping stones toward success. Humility: The willingness to accept feedback, no matter how critical, and use it to improve.

These traits are your superpowers during this stage. Use them to fuel your growth. Let your ambition and curiosity guide you to new knowledge. Push yourself to take on challenges that seem just beyond your reach. And when you stumble, remember that mistakes are simply opportunities to learn.

At this stage, it’s also important to focus on the bigger picture. Yes, you and I have goals whether that’s building better apps, landing a dream job, or contributing to impactful projects. But our purpose as developers goes beyond personal achievements. With every line of code, we have the opportunity to add value to the world. By solving problems, simplifying processes, and sharing our knowledge, we can create meaningful change.

Plan First, Code Later

A great real world example of the "Plan First, Code Later" approach is how Instagram was initially developed.

In 2010, Instagram (then called "Burbn") was just an idea in the minds of its founders, Kevin Systrom and Mike Krieger. Instead of jumping straight into coding, they followed a structured planning approach before writing a single line of code.

Case Study: Instagram's Early Development

Background
In 2010, Instagram (then called "Burbn") was just an idea in the minds of its founders, Kevin Systrom and Mike Krieger. Instead of jumping straight into coding, they followed a structured planning approach before writing a single line of code.

Step 1: Analyzing Product Requirements
Initially, Burbn was a location based check-in app with a photo sharing feature. After conducting market research and analyzing user feedback, the team realized that users were mainly interested in the photo-sharing feature rather than the check-ins.

Lesson: Instead of coding an unnecessary check-in system, they refined their product requirements and pivoted towards a dedicated photo sharing app.

Step 2: System Architecture & Scalability Planning
Before coding, the team planned how the backend should handle millions of photos and user interactions. They chose AWS for hosting and used Django as their backend framework for rapid development while ensuring scalability.

Lesson: Without proper planning, they could have picked a technology stack that wouldn't scale effectively.

Step 3: UX/UI Planning Before Development
They designed a simple and intuitive user interface before writing any code. They focused on a minimalist approach, reducing clutter and unnecessary features, leading to an easy to use app.

Lesson: By planning the UI/UX beforehand, they avoided the need for major redesigns after coding.

Outcome
Because of their careful planning and iterative approach, Instagram launched with a lightweight, focused, and scalable system. Within two months, it gained 1 million users, proving the power of planning before coding.

Source: strategizeyourcareer

In programming, clarity in requirements, design, and implementation plays a crucial role in minimizing the number of fixes or bugs. Let's break this down:

Clarify Early → Fewer Fixes Later
When requirements are well defined, developers know exactly what to build. This prevents misunderstandings and unnecessary revisions.
A clear design ensures that architectural decisions are sound from the start, reducing structural issues later.
Writing clean and well documented code makes debugging and future maintenance easier, reducing the chances of introducing new bugs when fixing existing ones.
Lack of Clarity → More Fixes Needed
If requirements are vague, developers might make incorrect assumptions, leading to features that don’t match expectations.
A poorly designed system can cause scalability and maintainability problems, requiring constant refactoring.
Ambiguous or unstructured code can lead to hard to find bugs, increasing debugging time and causing more regressions.

As a beginner, you’ve likely experienced the highs and lows of learning to code. The excitement of progress mixed with the frustration of setbacks is part of the process. But here’s something that can make this stage more manageable: developing a strong foundation in planning before you dive into coding.

Let's take an example of "Plan First, Code Later" in a real coding scenario using Flutter

Before Planning (Jumping Straight into Code)

class UserListScreen extends StatefulWidget {
  @override
  _UserListScreenState createState() => _UserListScreenState();
}

class _UserListScreenState extends State<UserListScreen> {
  List<dynamic> users = [];

  @override
  void initState() {
    super.initState();
    fetchUsers();
  }

  Future<void> fetchUsers() async {
    final response = await http.get(Uri.parse('https://jsonplaceholder.typicode.com/users'));
    if (response.statusCode == 200) {
      setState(() {
        users = json.decode(response.body);
      });
    }
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(title: Text('Users')),
      body: users.isEmpty
          ? Center(child: CircularProgressIndicator())
          : ListView.builder(
              itemCount: users.length,
              itemBuilder: (context, index) {
                return ListTile(
                  title: Text(users[index]['name']),
                  subtitle: Text(users[index]['email']),
                );
              },
            ),
    );
  }
}

Problems with This Approach

Mixes UI and Business Logic: The fetchUsers() function is inside the UI, making it hard to test and maintain.
Tight Coupling: If the API changes, we must modify the UI code as well.
No Error Handling: What if the API fails? There’s no proper error message.
Not Scalable: If we add caching or other data sources (like a database), we will need major rewrites.

After Planning (More Effective & Maintainable Code)

Step 1: Define a Model (User.dart)

class User {
  final int id;
  final String name;
  final String email;

  User({required this.id, required this.name, required this.email});

  factory User.fromJson(Map<String, dynamic> json) {
    return User(
      id: json['id'],
      name: json['name'],
      email: json['email'],
    );
  }
}

Step 2: Separate Data Layer (UserService.dart)

import 'dart:convert';
import 'package:http/http.dart' as http;
import 'user.dart';

class UserService {
  Future<List<User>> fetchUsers() async {
    final response = await http.get(Uri.parse('https://jsonplaceholder.typicode.com/users'));

    if (response.statusCode == 200) {
      List<dynamic> data = json.decode(response.body);
      return data.map((json) => User.fromJson(json)).toList();
    } else {
      throw Exception('Failed to load users');
    }
  }
}

Step 3: Use State Management (UserProvider.dart)

import 'package:flutter/material.dart';
import 'user_service.dart';
import 'user.dart';

class UserProvider extends ChangeNotifier {
  final UserService _userService = UserService();
  List<User> _users = [];
  bool _isLoading = true;
  String? _error;

  List<User> get users => _users;
  bool get isLoading => _isLoading;
  String? get error => _error;

  UserProvider() {
    fetchUsers();
  }

  Future<void> fetchUsers() async {
    try {
      _isLoading = true;
      notifyListeners();

      _users = await _userService.fetchUsers();
      _error = null;
    } catch (e) {
      _error = e.toString();
    } finally {
      _isLoading = false;
      notifyListeners();
    }
  }
}

Step 4: UI with Separation of Concerns (UserListScreen.dart)

import 'package:flutter/material.dart';
import 'package:provider/provider.dart';
import 'user_provider.dart';

class UserListScreen extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(title: Text('Users')),
      body: Consumer<UserProvider>(
        builder: (context, userProvider, child) {
          if (userProvider.isLoading) {
            return Center(child: CircularProgressIndicator());
          }

          if (userProvider.error != null) {
            return Center(child: Text('Error: ${userProvider.error}'));
          }

          return ListView.builder(
            itemCount: userProvider.users.length,
            itemBuilder: (context, index) {
              return ListTile(
                title: Text(userProvider.users[index].name),
                subtitle: Text(userProvider.users[index].email),
              );
            },
          );
        },
      ),
    );
  }
}

Step 5: Setup in main.dart

import 'package:flutter/material.dart';
import 'package:provider/provider.dart';
import 'user_provider.dart';
import 'user_list_screen.dart';

void main() {
  runApp(
    MultiProvider(
      providers: [
        ChangeNotifierProvider(create: (_) => UserProvider()),
      ],
      child: MyApp(),
    ),
  );
}

class MyApp extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      home: UserListScreen(),
    );
  }
}

Benefits of This Planned Approach

Separation of Concerns – Business logic, API calls, and UI are separate.
Scalability – Easy to add caching, local storage, or database.
Better Error Handling – Provides clear error messages.
Easier Testing – Can unit test the UserService without involving UI.
Less Boilerplate in UI – The UI is now focused only on presentation.

When starting a new project or feature, it’s tempting to jump straight into coding because it feels productive. However, skipping the planning phase often leads to inefficiencies, unnecessary mistakes, and wasted effort. Taking the time to plan ensures a smoother development process and helps create cleaner, more maintainable code.

Planning doesn’t always mean lengthy documentation. For small tasks, it might be as simple as confirming requirements: “You want the button text changed to blue, right?” For larger projects, it involves defining specifications, gathering feedback, and securing approval. Without this step, you risk solving the wrong problem or overlooking key considerations.

Writing code too soon can also make it harder to communicate ideas, especially with non technical colleagues. Conversations, diagrams, and brainstorming sessions are often faster ways to clarify concepts than jumping into code. While coding can sometimes help explore a problem, any early code should be treated as a rough draft rather than a final solution.

Even informal planning breaking down a task into steps or considering potential challenges can help avoid issues later. By thinking ahead, you can refine your approach before writing code, ultimately saving time and reducing frustration.

Good planning doesn’t slow you down; it helps you move faster by preventing costly mistakes and ensuring you build the right solution from the start.

Thank you for reading this article! I hope it helps you understand the importance of planning before coding and how it can lead to better, more efficient development. Happy coding! 🚀

References:
letterstoanewdeveloper
Michael Rothrock
Instagram Was First Called 'Burbn'

Connection Pool in Backend Development: Basic Concept, Benefits, and Implementation

Rizky Darmawan — Thu, 06 Feb 2025 02:43:38 +0000

Photo by Christine Tutunjian on Unsplash

Connection Pooling is a mechanism that creates and manages a pool of database connections that can be used by applications. This concept is important in managing connections to the database with the aim of optimizing resource use and improving the performance of applications that frequently interact with the database.

Instead of creating a new connection every time it is needed (which is expensive in terms of time and resources), connection pools allow applications to borrow/use existing connections and return them to the pool when they are finished using them. That's why it's called connection pool.

Why connection pool is important ?

The important question is why is connection polling important? why not just make 1 connection to be used alternately. Let's discuss 1 by 1 why connection pooling is important.

Better performance

Opening a new connection to the database can take time because it goes through several processes such as authentication, network settings and so on. With a connection pool, applications can avoid this process because connections are already available if needed.

Resource savings

Without a pool, if the application gets 100 requests, the application will create 100 new connections. This can burden the database server which is always busy carrying out the initialization process. The connection pool can limit the number of connections used simultaneously, this keeps the database server load stable.

Scalability

Connection pooling helps applications manage load well when the application scales up. With good connection pooling settings, we can handle a large number of requests coming at the same time and ensure the application remains responsive.

Load configuration

The connection pool can be configured with a maximum number of connections according to needs. So it can prevent applications from flooding connections to the database.

How to Works ?

Now that we understand why connection pooling is important to use, we will now discuss how connection polling works.

Pool Initialization

The first time an application is run, it makes a certain number of connections to the database (for example, 10 connections). This number is called Pool size. This connection will be idling waiting to be used.

Connection usage

When the application needs a connection to the database it no longer creates a new connection, instead it requests a connection from the pool. If a connection is available in the pool, it will be assigned to the application. If no connections are available, the application may have to wait until a connection is released and returns to the pool after being used by another process.

Restore connection

Once an application has finished using a connection, it will not close the connection. Instead, the connection is returned to the pool to be used again by the next request.

The common main parameter in connection pool

The following are some of the main parameters that are commonly used when creating a connection pool. I will try to explain in detail followed by analogies that can help us understand each context better.

Pool Size

We can determine the number of connections provided in Pool Size. If all connections are used then when there is a request to use the connection it will queue up waiting for an available connection or it will fail if the queue is full.

For example, if we set the max pool size to 10, only 10 active connections can be used by the application. If an 11th request comes, this request will wait for a connection from the pool that is already in use. The analogy is like this:

Imagine there is a parking lot, if all the slots are full, a new car that wants to enter (connection request) has to wait until another car comes out (idle connection).
If we want to accommodate more cars, we have to build more parking slots (increase MaxPoolSize) so that we can accommodate more cars. But there are increased maintenance costs (server resources).

Minimum Connections

Minimum connection is the minimum number of connections that will be created the first time the application is run. This is a connection that will always be on standby waiting to be used, even when there are no requests. For example, if we set minimum pool size 5, the pool will ensure there are always 5 idle connections. Even if there are no active requests, the pool will still make sure these 5 connections are open to reduce the connection opening time when a request comes in.

This is useful for anticipating erratic loads, for example suddenly our application gets a lot of requests and connections are available ready to use without having to open a new connection. The analogy is like this:

Imagine that a restaurant has 10 chairs that are deliberately not arranged in the dining room, but are stored in the warehouse. But when there are a lot of visitors, we need extra chairs, we don't need long to buy them at the shop. But just need to go to the warehouse and pick up the 10 chairs needed for use.

Idle Connection

Idle connections are the number of connections that are not being used but remain open in the pool, which can also be called "idle connections". This parameter is usually set by:

Maximum Idle Time: How long an idle connection is maintained before being closed.
Maximum Idle Connections: The number of idle connections allowed to remain in the pool.

For example, if we set MaxIdleConnections = 5, the pool will maintain a maximum of 5 idle connections. If there are more idle connections it will be closed. If there are no new requests within a certain time (for example, 10 minutes), these idle connections may be removed to save resources. The analogy is like this:

In the supermarket there are 5 cashiers, 3 cashiers are serving buyers (active connection) and 2 cashiers are on standby in their positions without a queue of buyers (idle connection) and are ready whenever a buyer comes.
Supermarkets have a rule, "If the cashier is on standby for 30 minutes without a customer, then they can go home." These 30 minutes are idle time.

Timeout

We can set the maximum time the application waits for a connection from the pool before it fails (error). This is done to prevent applications from waiting too long for available connections from the pool, if the pool is busy or full. This is useful to prevent the application from waiting too long for a connection which will cause the application to "freeze" where it is better to give an error to the user, so that it can provide information to the user to try again rather than resulting in a freeze request.

For example, if we set a timeout of 5 seconds, if a request does not get a connection, after waiting for 5 seconds it will receive an error. The point to remember is that a timeout that is too short can cause the request to fail too quickly and a timeout that is too long means the request will hang for too long. The analogy is like this:

We are queuing to buy at a restaurant (request), but we can't wait for the queue (timeout limit). And we decided to leave the restaurant (return error).

Max lifetime

Lifetime determines how long a connection can live before being reset or closed by the pool, even if the connection is active. Usually used to avoid old connection problems or update configurations to ensure connections remain fresh and reliable.

For example, if we set max lifetime 20 minutes, any connection that has been used for 20 minutes will be closed and removed from the pool and will be replaced by a new connection that will be created.

This is important because sometimes some database servers have a time limit for old connections which can prevent stale connections** problems. The analogy is like this:

An office has 3 security officers per shift. Each officer works a maximum of 8 hours (Max Lifetime)
After 8 hours, the officers must be replaced with new officers, even though the previous 5 officers are still in good condition, there are no problems and work is running smoothly. But it prevents officers from getting tired and losing concentration when checking incoming visitors (requests).

Implementation

Now we will try to implement a connection pool using Golang. Note the following code:

package main

import (
    "database/sql"
    "fmt"
    "log"
    "time"

    _ "github.com/go-sql-driver/mysql"
)

func main() {
    dsn := "username:password@tcp(127.0.0.1:3306)/dbname?parseTime=true"

    db, err := sql.Open("mysql", dsn)
    if err != nil {
        log.Fatalf("Failed to open connection to database: %v", err)
    }
    defer db.Close()

    db.SetMaxOpenConns(10)
    db.SetMaxIdleConns(5)
    db.SetConnMaxLifetime(time.Minute * 10)
    db.SetConnMaxIdleTime(time.Minute * 5)

    err = db.Ping()
    if err != nil {
        log.Fatalf("Failed to ping database: %v", err)
    }
    fmt.Println("Successfully connected to database!")

    rows, err := db.Query("SELECT id, name FROM users")
    if err != nil {
        log.Fatalf("Failed to run query: %v", err)
    }
    defer rows.Close()

    for rows.Next() {
        var id int
        var name string
        if err := rows.Scan(&id, &name); err != nil {
            log.Fatalf("Failed to read query results: %v", err)
        }
        fmt.Printf("ID: %d, Name: %s\n", id, name)
    }

    if err := rows.Err(); err != nil {
        log.Fatalf("Error after iteration: %v", err)
    }
}

We can focus on the following code snippet:

db.SetMaxOpenConns(10)
db.SetMaxIdleConns(5)
db.SetConnMaxLifetime(time.Minute * 10)
db.SetConnMaxIdleTime(time.Minute * 5)

Here's the explanation:

SetMaxOpenConns(10): Determines the maximum number of connections that can be open simultaneously.
SetMaxIdleConns(5): Determines the maximum number of idle connections maintained in the pool.
SetConnMaxLifetime(time.Minute * 10): Determines how long a connection can live before being reset or closed.
SetConnMaxIdleTime(time.Minute * 5): Determines how long an idle connection can be maintained before being closed.

Conclusion

The use of connection pooling is very important, as this can anticipate the application running smoothly when handling an erratic load of incoming requests. But the right settings must also be implemented, so that the use of connection pooling can be more optimal.

If you have additions or corrections to the discussion above, let's discuss it in the comments column. Hope it helps 👋.

Mastering ENUMs in Go

Rizky Darmawan — Tue, 24 Dec 2024 02:41:21 +0000

Image by Hans-Peter Gauster from Unsplash

Let's say we are building an E-commerce API that will receive several orders, each order process has several statuses such as Pending, Processed, Shipped, Delivered, Cancelled. And our application receives input strings which will be stored in the database, for example the status is Processed, received Process, Processing or something else that causes data inconsistencies. Here Enum has an important role.

In Golang enums unlike other languages such as Java or C# which offer built-in support for enums, Go takes a different approach. In Go, enums are not a native language feature, but developers have several techniques that can be used to achieve similar functionality.

Understanding Enum

In Golang, enums (short for enumerations) provide a way to represent a set of named constants. Although Go does not have built-in enum types like some other languages, developers can emulate enum-like behavior using constants or custom types. Let's discuss the purpose and syntax of enums in Go:

Objective

Readability and Maintainability: Enums make code more readable and easy to understand by giving meaningful names to specific values. This increases the maintainability of the code because the purpose of each constant becomes easier to understand.
Type Safety: Enums help enforce type safety by restricting variables to a predefined set of values. This reduces the possibility of runtime errors caused by using incorrect values.

Create Enums

Here we will discuss step by step how to create an enum in Golang, so that it is easy to understand at each stage we will explain the meaning of the code we write.

Create a New Type

The first thing we will do is create a new type for the enum we need. The method is quite easy, we only need to use the keyword type and followed by the name of the type here with the name StatusOrder and the type here we define the type unsigned integer like this:

type StatusOrder uint

Well, just make it easy.

Define constant ENUM

With the new type that we have created, now is the time for us to define some of the order statuses that we have with constants. Where we define the type StatusOrder which we create as the type, like this:

const (
    Pending StatusOrder = iota
    Processed
    Shipped
    Delivered
    Cancelled
)

Maybe you ask, what is the keyword iota? This keyword makes GO assign a value of 0 to the first constant and then increase the value by 1 sequentially for each subsequent constant. This makes it easier for us rather than defining the values manually 1 by 1. About iota you can read here.

Function Strings

The next step we will take is to create a String function that is used to represent each string value from the StatusOrder enum.

func (s StatusOrder) String() string {
    switch s {
    case Pending:
        return "Pending"
    case Processed:
        return "Processed"
    case Shipped:
        return "Shipped"
    case Delivered:
        return "Delivered"
    case Cancelled:
        return "Cancelled"
    default:
        return "Unknown"
    }
}

Does the function name have to be String? We'll discuss it at the end.

Testing

Now we will carry out testing, the last code we will write is, function main and in it we print the results using the help of the fmt package. yes.

func main() {

    processed := Processed
    fmt.Printf("Order Status: %s (%d)\n", processed, processed)

    pending := Pending
    fmt.Printf("Order Status: %s (%d)\n", pending, pending)

}

Now the complete code can be seen here.

Of course, we carry out the last step and we will see the results like this:

Fmt Stringer

The question may arise, why is the String function also called when we call the enum constant?. The answer is because we use the fmt package. The fmt package explicitly uses the fmt.Stringer interface to process types that implement the String() method. So, if you don't use fmt, the String() method will not be called automatically. To explain more, you can explore in more detail here.

Conclusion

Although Golang does not offer native enum types, the techniques we learn here are often used in building applications with Golang. And for the type itself, we can freely use other types, not just integer. By utilizing this technique, readability, ease of maintenance and security can be significantly improved.

Maybe there are some points explained above that you feel are lacking, we can discuss them in the comments column below. Hope it helps 👋.

Reading References

Avoid Boilerplate with Code Generator in Flutter

Firman Maulana — Thu, 19 Dec 2024 03:08:49 +0000

As a software developer, you may encounter situations like writing code repeatedly for standard tasks. Imagine creating an application that needs integration with various APIs, managing JSON data, or performing tests. These tasks usually involve writing a lot of boilerplate code, which is time-consuming and increases the chance of mistakes.

This can be very frustrating, especially when deadlines are approaching or when you are working on a complex project with many dependencies. Instead of focusing on innovation and solving more creative problems, you get stuck in a seemingly endless routine of writing code. This wastes valuable time and reduces your enthusiasm as a developer.

This is where the importance of using code generators comes in. By utilizing code generation, you can reduce your workload in application development. You can automate repetitive processes, allowing you to focus on more strategic aspects and solve project-related problems. How can you use this in Flutter projects? Let's find out!

The Dart ecosystem provides developers with a variety of powerful tools, one of which is code generation techniques. Yes, that's right! As the name suggests, this tool helps programmers generate code so that we don't need to write the whole code, but only the parts we need.

Code Generation with build_runner

Creating applications with Flutter often involves many common tasks, such as:

Deserializing JSON
Using backend APIs
Managing navigation
Writing tests

Typically, completing these tasks involves writing repetitive code, which can be time-consuming and error-prone. However, we can reduce the amount of code we need to write manually by using build_runner to generate the required code.

In simple terms, build_runner is a tool for generating output files from input files. We can create a code generator that works with this mechanism, which allows us to read input files, usually written in Dart code, and generate the corresponding output files.

Using this generator, we only need to provide a minimal configuration for the desired code. After the build process is complete, we will receive a valid Dart code.

Here are the steps to use build_runner:

Add it as a development dependency in the pubspec.yaml file:

   dev_dependencies:
     build_runner: x.y.z

Run the code generation command:

   flutter pub run build_runner build

After that, build_runner will execute all the code generators added as dependencies and generate output based on the input.

Watch Mode and Handling Existing Files

build_runner also has a watch mode that monitors the file system and automatically updates the output files when the input files change. If files are considered existing and not generated by build_runner, a reminder will appear to delete those files.

The -d option (short for --delete-conflicting-outputs) allows build_runner to delete conflicting files before starting code generation, making the process easier.

For a more detailed explanation, you can check here: build_runner documentation.

Implementation

There are many packages we can use, but this time we will look at some of the most popular ones. Some of these packages work with the build_runner tool. They make app development easier, so developers can focus more on what they want to create instead of how to create it. Here are some commonly used packages:

json_serializable
This package makes it easy to turn JSON data into Dart objects and vice versa.
By using these packages together, we can work faster and more efficiently when building Flutter apps.

For example, we can create an app that shows data from a JSON file:

 "id": 1,
     "user": {
       "name": "Asep Saepul",
       "avatar": "https://images.pexels.com/photos/27659838/pexels-photo-27659838/free-photo-of-portrait-of-a-man-on-a-white-background.jpeg?auto=compress&cs=tinysrgb&w=800",
       "place": "Bandung, Indonesia"
     },
     "content": {
       "isLike": false,
       "image": "https://images.pexels.com/photos/3386600/pexels-photo-3386600.jpeg?auto=compress&cs=tinysrgb&w=600",
       "likes": "1.000 likes",
       "description": "Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor."
     }
   },

To be used further in the application and take advantage of the typed nature of Dart, this data must be transformed into an instance of a Dart class:

class Feed {
 final int id;
 final User user;
 Content content;


 Feed({
   required this.id,
   required this.user,
   required this.content,
 });
}


class Content {
 final String image;
 late final String likes;
 final String description;
 bool isLike;
  Content({
   required this.image,
   required this.likes,
   required this.description,
   required this.isLike,
 });
}


class User {
 final String name;
 final String avatar;
 final String place;


 User({
   required this.name,
   required this.avatar,
   required this.place,
 });
}

To implement JSON deserialization in the fromJson() and toJson() methods within a class to allow the following usage:

class Feed {
  final int id;
  final User user;
  Content content;

  Feed({
    required this.id,
    required this.user,
    required this.content,
  });

  // Method to convert Feed object to JSON
  Map<String, dynamic> toJson() {
    return {
      'id': id,
      'user': user.toJson(),
      'content': content.toJson(),
    };
  }

  // Method to create Feed object from JSON
  factory Feed.fromJson(Map<String, dynamic> json) {
    return Feed(
      id: json['id'],
      user: User.fromJson(json['user']),
      content: Content.fromJson(json['content']),
    );
  }
}

class Content {
  final String image;
  late final String likes;
  final String description;
  bool isLike;

  Content({
    required this.image,
    required this.likes,
    required this.description,
    required this.isLike,
  });

  // Method to convert Content object to JSON
  Map<String, dynamic> toJson() {
    return {
      'image': image,
      'likes': likes,
      'description': description,
      'isLike': isLike,
    };
  }

  // Method to create Content object from JSON
  factory Content.fromJson(Map<String, dynamic> json) {
    return Content(
      image: json['image'],
      likes: json['likes'],
      description: json['description'],
      isLike: json['isLike'],
    );
  }
}

class User {
  final String name;
  final String avatar;
  final String place;

  User({
    required this.name,
    required this.avatar,
    required this.place,
  });

  // Method to convert User object to JSON
  Map<String, dynamic> toJson() {
    return {
      'name': name,
      'avatar': avatar,
      'place': place,
    };
  }

  // Method to create User object from JSON
  factory User.fromJson(Map<String, dynamic> json) {
    return User(
      name: json['name'],
      avatar: json['avatar'],
      place: json['place'],
    );
  }
}

Here, the static method Feed.fromJson() handles the parsing logic for the various field types in Feed, such as int and the custom classes User and Content, which also have User.fromJson() and Content.fromJson() factories. Additionally, the example above also includes the Feed.toJson() method that performs the inverse transformation.

The json_serializable package allows for the same result with much less code. It can generate methods like Feed.fromJson() and Feed.toJson() and provides a mechanism to perform all the formatting adjustments as in the manual implementation above.

To take advantage of code generation for JSON deserialization, add the following dependency in your pubspec.yaml file:

dependencies:
  json_annotation: x.y.z

dev_dependencies:
  build_runner: x.y.z
  json_serializable: x.y.z

Then an example of implementation by creating a Feed class as follows:

part feed.g.dart';

@JsonSerializable(explicitToJson: true, includeIfNull: false)
class Feed with _$Feed {
  const factory Feed({
    required int id,
    required User user,
    required Content content,
  }) = _Feed;

  factory Feed.fromJson(Map<String, dynamic> json) => _$FeedFromJson(json);

Map<String, dynamic> toJson() => _$FeedToJson(this);
}

The @JsonSerializable() annotation above the Feed class indicates that this class is subject to code generation, and the declaration of the feed.g.dart section file ensures that the generated code is in the same feed.dart. Once the code is generated, a new feed.g.dart file is added with the private methods _$FeedFromJson() and _$FeedToJson(), allowing the same results as the previous manual implementation.

Using code generation significantly reduces the amount of code that needs to be written compared to a manual JSON deserialization implementation. Additionally, every time the JSON format or Dart class structure is updated, a manual implementation requires changes in several places, which are easy to miss until a serialization exception is raised at runtime. With code generation, we can ensure that all necessary changes are applied automatically.

This guide only shows the basic usage of the json_serializable package. Check the official documentation for more information.

Enhance Classes with Freezed

There are several operations that developers perform on Dart class instances explicitly or implicitly:

value-based comparisons
hash code calculations
using string representations
making copies with modifications

The implementations of some of these operations, such as the ==() operator, hashCode, and toString(), are already built into the language but do not provide the best development experience and therefore require improvement, such as the copyWith() method, must be written from scratch.

Let’s consider the Feed class from the example above as an example. Two instances with the same field value are not equal, and their hash values are also different:

final feed1 = Feed(
        id: 1,
        user: User(
          name: 'John Dhoe',
          avatar:
              'https://images.png’,
          place: 'Bandung, Indonesia',
        ),
        content: Content(
          isLike: false,
          image:
                 'https://images.png’,
          likes: '1.000 likes',
          description:
              'Lorem ipsum dolor sit amet',
        ),
      ),
final feed2 = Feed(
        id: 1,
        user: User(
          name: 'John Dhoe',
          avatar:
              'https://images.png’,
          place: 'Bandung, Indonesia',
        ),
        content: Content(
          isLike: false,
          image:
                 'https://images.png’,
          likes: '1.000 likes',
          description:
              'Lorem ipsum dolor sit amet',
        ),
      ),
assert(feed1 == feed2);                    // false
assert(feed1.hashCode == feed2.hashCode);  // false

This can cause problems when Feed instances are added to a Set, used as Map keys, or verified in tests.

At the same time, the string representations of two instances with different field values are the same and return an Instance of Feed:

assert(feed1.toString() == feed2.toString());  // true

To enhance the Feed class, we need to provide implementations of the following methods:

   // Override == operator
  @override
  bool operator ==(Object other) =>
      identical(this, other) ||
      other is Feed &&
          runtimeType == other.runtimeType &&
          id == other.id &&
          user == other.user &&
          content == other.content;

  // Override hashCode
  @override
  int get hashCode =>
      id.hashCode ^ user.hashCode ^ content.hashCode;

But the problem doesn't stop there, because the user == other.user check will likely fail most of the time even if both collections are empty or contain the same objects. To compare collections based on their contents, it should be replaced with:

const DeepCollectionEquality().equals(user, other.user)

The Freezed package allows us to not worry about this situation. To take advantage of Freezed in code generation to enhance Dart classes, add this dependency in the pubspec.yaml file:

dependencies:
  freezed_annotation: x.y.z

dev_dependencies:
  build_runner: x.y.z
  freezed: x.y.z

And here is an example of its implementation:

@freezed
class Feed with _$Feed {
 const factory Feed({
   required int id,
   required User user,
   required Content content,
 }) = _Feed;


 factory Feed.fromJson(Map<String, dynamic> json) => _$FeedFromJson(json);
}


@freezed
class Content with _$Content {
 const factory Content({
   required String image,
   required String likes,
   required String description,
   required bool isLike,
 }) = _Content;


 factory Content.fromJson(Map<String, dynamic> json) => _$ContentFromJson(json);
}


@freezed
class User with _$User {
 const factory User({
   required String name,
   required String avatar,
   required String place,
 }) = _User;


 factory User.fromJson(Map<String, dynamic> json) => _$UserFromJson(json);
}

The @freezed annotation above the Feed class indicates that it is subject to code generation, and the declaration of the 'feed.freezed.dart' file section ensures that the generated code is part of the same as feed.dart.

Check out the Feed, Content, and User classes. You can check from the examples repository for the complete implementations. Once the code is generated, a new file feed.freezed.dart is added with implementations of the ==(), hashCode, toString(), and copyWith() operator methods, allowing for the same results as the manual implementation above.

As with JSON deserialization, using code generation to enhance Dart classes reduces the effort of creating and maintaining them, ensuring that any necessary changes are automatically applied when the class structure changes.

This guide only shows the basic usage of the freezed package. It has many useful features and can also be easily combined with json_serializable. Check the official documentation for more information.

There are still many packages that you can use for code generation, such as for RESTful API Consumption needs you can use Retrofit, Flutter Dependency Injection with Injectable, Documentation and Tests with mockito, and many more. Hopefully, in the next article, we can try to learn each of these packages to compare and learn how to implement them in our projects.

But for now, I think it's enough to know examples of using json_serializable and freezed as a first step to finding out the benefits of codegen and how we use it in a flutter project.

Drawback

Okay, we already know how many benefits we can get from code generators. But keep in mind, we often have to look at things from both sides. Yes, codegen also has disadvantages that we need to consider before using it. In my Flutter project, I found some things when using codegen:

Time-consuming: Basic code generation can be time-consuming for complex data structures or requirements.
Initial setup: Code generation requires some initial setup.
Slows down the build process: The generated code can slow the build process.
Clutters the project: The generated code can clutter your project with extra files.
Requires team members to run the codegen step: If you don't commit the generated files to git, every team member must remember to run the codegen step.
Requires a custom CI build step: If you don't commit the generated files to git, a custom CI build step is required to build the app.

Source Code

A simple Flutter demo project using Freezed package is available on GitHub.🚀

Conclusion

Code generation is a powerful tool that can make us more productive by facilitating many repetitive or typical tasks in Flutter application development.

In addition, those packages not only speed up the development process but also maintain the consistency and accuracy of the generated code. By utilizing these tools, developers can streamline their workflows, improve project maintenance, and ensure that structural changes in the code are automatically updated, which is an important step in maintaining the quality and sustainability of the application in the future.

However, besides the many benefits of using code generation, there are also things to consider. For example, parts of the code that have changed and impacted other parts must be re-generated, which may take more time than changing it manually. So, it is adjusted again to the needs of the project you are working on. If you have tried it, don't hesitate to share your experience in the comments; hopefully, it's useful!

References

https://docs.flutter.dev/
https://pub.dev/packages/build_runner
https://pub.dev/packages/freezed

Profiling Memory In Go

dikac — Tue, 17 Dec 2024 02:00:47 +0000

Efficient memory management is critical in Golang applications, particularly in high-concurrency environments, long-running services, or data-intensive tasks. Profiling memory usage helps diagnose issues, optimize performance, and prevent out-of-memory (OOM) errors. This guide provides a comprehensive approach to profiling memory usage from a Go endpoint.

Why Memory Profiling Matters

Memory profiling identifies inefficient memory usage, memory leaks, and excessive allocations in your application. Without proper profiling, memory issues can lead to performance degradation, higher costs, and service downtime.

Common Causes of High Memory Usage

Memory leaks: Unintended memory retention due to data structures not being cleared up.
Excessive allocations: Large slices, maps, or other data structures consuming significant memory.

Setting Up Memory Profiling in Go

To profile memory usage in a Go application, you can use tools like pprof for runtime profiling and parca for continuous profiling. Here’s how to set up and use these tools effectively.

Profiling Tools

pprof

A built-in Go tool that provides profiling for memory, CPU, goroutines, and more.
- Documentation
Parca

A continuous profiling tool that provides real-time insights by collecting data from pprof.
- Documentation
Stress Testing

Generate load to simulate real-world usage and observe memory behavior under stress. For our case we use SoapUI.

Using `pprof`

Since pprof is built-in tool, installation is not required, include the following snippet to enable pprof in your application:

import (
    _ "net/http/pprof"
)

func main() {
    go func () {
    log.Print(http.ListenAndServe(":1234", nil))
    }()
}

This exposes pprof on port 1234. Access profiling data by visiting http://localhost:1234/debug/pprof/ or using tools like go tool pprof.

Using `parca` for Continuous Profiling

To install parca see https://github.com/parca-dev/parca, after successfully installing parca, configure parca.yaml job_name.static_configs.targets set the same port number as pprof (in this example 1234)

then you can run command

parca --config-path="parca.yaml"

if successful you will see message similar to

level=info name=parca ts=2024-10-30T06:19:44.5149184Z caller=factory.go:53 msg="loading bucket configuration"
level=info name=parca ts=2024-10-30T06:19:44.5159183Z caller=badger.go:54 msg="Set nextTxnTs to 0"
level=info name=parca ts=2024-10-30T06:19:44.517917Z caller=server.go:90 msg="starting server" addr=:7070

addr=:7070 is where you can access parca web interface, port number might be different depend on configuration

if all setup successful, you can access parca on web browser

There is a multiple profiling type, for Memory usage you can use

if you encounter any issue, you should consult documentation since different environment might require different solution

pprof https://pkg.go.dev/net/http/pprof
parca https://github.com/parca-dev/parca

Identifying Memory Usage

Stress Testing

Before profiling, simulate high traffic using stress-testing tools in our case we use SoapUI. Stress tests help replicate conditions leading to memory issues.

Analyzing Memory Usage

After completing a stress test, monitor memory usage over time using the parca dashboard.

Click on the graphs to access detailed profiles.

Using the icicle graph, examine the stack and corresponding memory usage. Wider lines indicate higher memory consumption. This visualization helps pinpoint processes consuming significant memory.

In our application, a process with substantial memory usage was identified:

Memory Optimization

Memory optimization is a complex topic that varies depending on the application and its environment. Here are some practical techniques:

Selective Data Loading: Load only the necessary data to significantly reduce memory allocation.
Avoiding Pointers: Use value types instead of pointers to minimize heap allocations.
Predefining Data Lengths: Specify lengths for known-size data structures to improve memory efficiency.

Upon further investigation, we discovered that the data retrieved from the cache was excessively large. We needed to validate whether such a large dataset was truly necessary for our logic flow.

In our case, it turned out that this large dataset was not required. Therefore, we optimized the process by selectively removing unnecessary data. After re-running the tests, memory usage was reduced by approximately 50%.

Previous Implementation

After selectively removing unneeded data

With help of this method we can easily narrow down and correct memory usage, in our case Selective Data Loading is the correct method for reducing memory usage.

Conclusion

Memory profiling is a critical practice for maintaining the performance and stability of Go applications. By leveraging tools like pprof and parca, you can identify memory issues, optimize resource usage, and ensure your application performs reliably under various loads. Regular profiling and proactive optimization help address memory-related challenges effectively.

Modularization in Flutter App

Edo Lubis — Mon, 16 Dec 2024 09:07:57 +0000

As the complexity of a project increases, so does the amount of code we write. Therefore, maintaining code quality, scalability, and readability becomes even more important. One approach often used by engineers to address these challenges is modularization.

What is modularization? Simply put, modularization is the process of breaking down a problem into smaller, more specific tasks or components, so that each part has a clear and focused purpose.

However keep in mind, modularization doesn’t necessarily reduce the complexity of your project. The overall complexity may remain the same, but modularization makes it easier to manage and maintain.

We can separate modules based on features, by layers (such as UI, model, etc.), or a combination of both, depending on our needs and preferences.

When modularizing, it's important to remember that good modularization is characterized by high cohesion, low coupling and bounded context.

High cohesion means that each module has a specific function and focuses on a single purpose. For example, a module responsible for user authentication should only handle tasks like verifying user credentials or managing login sessions. It should not include unrelated functions, such as processing payments or managing user preferences.

Low coupling means that modules do not rely excessively on other modules, so changes in one module do not have a significant impact on other modules. For example, the user authentication module should be designed in such a way that it operates independently of other modules.

Bounded context means that each module should operate within its own domain, with its own set of rules and responsibilities. It ensures that the boundaries of a module are clearly defined, preventing its domain from spilling over into others. as an analogy, imagine you are designing an amusement park with various rides. Each ride in the park has its own theme and different rules, such as the roller coaster, haunted house, or giant Ferris wheel. Each ride has clear boundaries, with features and rules that apply only within that ride.

In the roller coaster, the main rule is speed and passenger safety. All design and interactions in the roller coaster are focused on providing a thrilling and safe high-speed experience.

In the haunted house, the main rule is to create suspense and surprise. All design elements and interactions in the haunted house focus on creating a spooky and mysterious atmosphere.

In the giant Ferris wheel, the main rule is to enjoy the view. This ride is designed to offer a calm and scenic experience, without speed or tension.

Even though all these rides are part of the same amusement park, each operates within its own bounded context, with its own rules, definition and goals.

learn more about cohession and coupling

Lets take the example of an ecommerce application where a user can Login, Browse a list of products, Add products to the cart, Make a payment and chose the shipping method.

This application can be modularized as follows:

User Module: Handles user authentication (login, signup, password reset).
Product Module: Handles product list, product details, and product filtering/searching.
Cart Module: Manages the shopping cart, including adding/removing items, displaying total price, and updating quantities.
Payment Module: Processes payments and manages transactions.
Shipping Module: Manages shipping options and delivery methods.

And make sure, each module has boundaries defined based on the principle of bounded context, which means that each domain should have its own rules and responsibilities, ensuring that it operates independently within its own scope. This approach guarantees that each module has high cohesion (meaning all related functionalities are grouped together) and low coupling (meaning minimal dependencies on other modules).

Modular Approach in flutter using flutter package

In Flutter, we can adopt a modular approach using Flutter packages. For example, if we create a new application named "TradingApp," we can organize the modules based on features such as Wallet, Login, and Market.

Create a new Flutter project.
After creating the TradingApp project, create the login module by going to File → New → New Flutter Project, and select Package.

Then, create the other modules. The project structure will look like this

To enable TradingApp to use the other modules, we need to link them with import dependencies, similar to Flutter libraries.

This way, we can use different modules within our application.

We can also group modules into a single folder in our project. This will make the project structure and pubspec.yaml file look like this:

Or If we prefer not to keep the modules in the same repository, we can use a different approach by placing each module in separate Git repositories.

Conclussion

Modularization enhances software development by breaking complex projects into manageable, focused components, which improves code quality, scalability, and maintainability. This approach allows for better organization, easier updates, and reusability of code, and facilitates collaboration among team members. And, for modularization to be effective, it is crucial to ensure high cohesion within modules and maintain low coupling between them.

Flutter Image Compression: Ensuring High Quality

Irfan Adisukma — Thu, 14 Nov 2024 06:05:58 +0000

In today's digital age, we often find ourselves taking countless photos and capturing memories on our devices. As developers, we want to make sure these images are not just beautiful but also efficient when it comes to uploading them to a server. This is where image compression comes into play.

In this article, we’ll explore the ins and outs of image compression for Flutter, focusing specifically on how to balance quality and performance for seamless uploads. Whether you're building a social media app or a photo-sharing platform, understanding the art of compression can greatly enhance user experience while keeping your application running smoothly. Let's dive in!

Image compression is the process of reducing the file size of an image without significantly compromising its quality. It involves removing unnecessary data or utilizing algorithms to make the file smaller. This is essential for several reasons, especially when uploading images to a server.

Firstly, larger image sizes can lead to longer upload times. For users with slower internet connections or limited bandwidth, this delay can result in frustration and a poor overall experience. By compressing images before uploading, developers can ensure faster transfer speeds, making the process more seamless for users.

Secondly, uploading large images can strain server resources. This can lead to increased storage costs and slower response times for other users accessing the server. By utilizing image compression, developers not only enhance the user experience but also optimize server performance and efficiency.

So, understanding and implementing image compression is vital for any Flutter developer looking to improve their app's performance and user satisfaction, particularly in scenarios where image uploads are a common requirement.

Types of Image Conversion

When it comes to image compression, there are two main types: lossy compression and lossless compression. Each has its pros and cons, making them suitable for different situations.

Lossy Compression

Image Source

What it is: Lossy compression reduces image size by permanently removing some of the image’s data. It focuses on discarding details that are less noticeable to the human eye, making the file significantly smaller.

Impact on Quality: This approach often leads to a small reduction in image quality, which may or may not be visible depending on the level of compression used. Higher compression ratios (smaller file sizes) often result in more noticeable quality loss, like blurring or pixelation.

Best Use Cases: Ideal for photos or images with lots of colors and gradients, like user-uploaded images or social media photos. JPEG is a common format that uses lossy compression.

Lossless Compression

Image Source

What it is: Lossless compression reduces file size without removing any image data. It achieves smaller files by finding and compressing patterns within the image, but no data is lost, so you can fully restore the original image.

Impact on Quality: Since no data is discarded, image quality remains intact. However, lossless compression doesn’t reduce file size as dramatically as lossy compression, especially for complex images.

Best Use Cases: Suitable for graphics, icons, or images where high fidelity is crucial, like logos or text-heavy images. PNG and GIF formats often use lossless compression.

Choosing Between Lossy and Lossless

When Quality is Crucial: Use lossless compression to retain full detail, especially if users may zoom in or if the image contains text or important details.

When Reducing File Size is Key: Use lossy compression when you need a smaller file size and can afford slight quality loss, especially for thumbnails or images in feed views where high resolution isn't necessary.

In summary, the choice between lossy and lossless compression hinges on your specific needs: prioritize speed and smaller files with lossy compression, or opt for higher quality with lossless compression. Understanding these options will help you optimize image uploads in your Flutter app.

In this article, we'll focus more on lossy compression to achieve slight quality loss and smaller file sizes.

Original (974Kb)

Image Source
I have this original size file with 974Kb file size.

And then i do compression with 90% Quality.

90% Quality (318Kb)

With 90% quality we can achieve the compression rate at approximately 67.35% from 974Kb to 318Kb.

50% Quality (109Kb)

With 50% quality we can achieve the compression rate at approximately 88.81% from 974Kb to 109Kb.

10% Quality (55Kb)

With 10% quality we can achieve the compression rate at approximately 94.35% from 974Kb to 55Kb.

Based on the quality comparisons above, you can decide which compression level works best for your needs. If image quality is a priority (for product photos or detailed images), higher quality settings (e.g., 90 or above) should be used to maintain clarity, though they result in larger file sizes.

If file size is more important, such as for thumbnails or gallery images, lower quality settings (e.g., 50 or even 10) can be used to significantly reduce file size, though some visual degradation will occur. The choice depends on whether you prioritize visual fidelity or performance and size optimization.

Code Implementation

Here’s an example of a function to compress an image in Flutter using the flutter_image_compress package:

You only need these two packages:
Flutter Image Compress Package
Flutter Image Picker

Pick Image

import 'package:image_picker/image_picker.dart';

final ImagePicker picker = ImagePicker();
File? selectedImage;

Future<void> pickImage() async {
    final pickedFile = await picker.pickImage(source: ImageSource.gallery);
    if (pickedFile != null) {
      setState(() {
        selectedImage = File(pickedFile.path);
      });
    }
  }

Compress Image

import 'package:path/path.dart' as path;
import 'package:flutter_image_compress/flutter_image_compress.dart';

Future<XFile> compressImageFile(
    {
      required File imageFile, 
      int quality = 80, 
      CompressFormat format = CompressFormat.jpeg
    }) async {

    DateTime time = DateTime.now();
    final String targetPath = path.join(
      Directory.systemTemp.path, 'imagetemp-${format.name}-$quality-${time.second}.${format.name}'
    );

    final XFile? compressedImageFile = await FlutterImageCompress.compressAndGetFile(
      imageFile.path,
      targetPath,
      quality: quality,
      format: format
    );

    if (compressedImageFile == null){
      throw ("Image compression failed! Please try again.");
    }
    debugPrint("Compressed image saved to: ${compressedImageFile.path}");
    return compressedImageFile;
  }

UI Example

Column(
  mainAxisAlignment: MainAxisAlignment.center,
  children: [
    if (selectedImage != null) 
      Image.file(
        selectedImage!
      ),
    TextButton(
      onPressed: () async {
        await pickImage();
      }, 
      child: const Text("Pick Image")
    ),

    ElevatedButton(
      onPressed: () {
        if (selectedImage != null) {
          compressImageFile(
            imageFile: selectedImage!,
            quality: 10
          );
        }
      }, 
      child: const Text("Compress")
    )
  ],
)

If you'd like to view the code for the sample usage app above, please visit:
Github

References:
www.dhiwise.com
www.adobe.com
cyberhoot.com

Defensive Programming as a Backend Developer: Building Robust and Secure Systems

Rizky Darmawan — Mon, 11 Nov 2024 03:10:17 +0000

In backend application development, ensuring application security and stability is an absolute must. The backend is the backbone of the application, which is responsible for handling business logic, storing data, and interacting with external systems. Writing strong and reliable code is essential for all software developers. However, no matter how careful we are, bugs and unexpected situations can still occur. This is where Defensive programming comes into play.

Defensive programming is a coding practice aimed at ensuring that software functions correctly even when unexpected events or invalid input occur. For a backend developer defensive programming is an important approach, which allows us to design applications that can survive bad input, system errors, and external attacks.

In this article, we will discuss several points on how defensive programming can be applied to increase the resilience of backend applications and provide examples in the Golang language to illustrate how to implement it, although its implementation is not tied to just one language.

Validate Input Data from External Sources

External sources such as third-party API users, or databases can be entry points for various attacks, especially if the input provided is invalid or malicious. Attacks such as SQL injection, cross-site scripting (XSS), and command injection often occur due to input that is not properly filtered. We can apply the following points to overcome this:

1. Input Sanitation

Any input from outside sources must be sanitized. Input sanitization is the process of ensuring that data received from users or external sources is cleaned of malicious characters before it is used in an application, especially when the data is used for SQL queries, system commands, or output to the browser. For example, if you receive string input from a form, avoid entering that input directly into a SQL query without sanitization. What you can do is use prepared statements to prevent SQL Injection. With prepared statements, user input is not allowed to be part of the SQL command, but rather as a safe parameter. For example, we have input to check the user's username and password for login and our backend system, rather than directly running the following query:

// direct query with parameters
query := fmt.Sprintf("SELECT * FROM users WHERE username = '%s' AND password = '%s'", "john_doe", "12345")
rows, err := db.Query(query)

It is better to use prepared statements, where we ensure that user input is never executed as part of a SQL query. Input is considered a value, not part of a SQL command as follows:

// prepared statement
stmt, err := db.Prepare("SELECT * FROM users WHERE username = ? AND password = ?")
if err != nil {
    log.Printf("Error while prepared statement: %v", err)
    return
}
defer stmt.Close()

// excecution prepared statement
username := "john_doe"
password := "12345"
rows, err := stmt.Query(username, password)
if err != nil {
    log.Printf("Error while run query: %v", err)
    return
}
defer rows.Close()

2. Data Type Validation

Data type validation is the process of checking and ensuring that the type of data received is as expected before further processing. This is important to prevent errors in data processing and also reduce security risks, such as SQL injection and XSS. For example, if an API expects an email format, validate that the input is indeed an email format. The following is an example of carrying out the data type validation process in Golang:

func main() {
    input := "hello@world.com"

    // Use regex to validate email format
    re := regexp.MustCompile(`^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,} $`)
    if !re.MatchString(input) {
        fmt.Println("Invalid email format.")
        return
    }

    fmt.Println("Valid email:", input)
}

Here, we use a regular expression to validate whether the input corresponds to the correct email format.

3. Whitelist & Blacklist

Whitelist and blacklist are two approaches used to validate and manage user input in applications. Both have the goal of improving security and preventing exploits, but they work and are applied differently.

- whitelist

Whitelisting is an approach where only certain data or characters are allowed to be processed by the application. In the context of user input, a whitelist determines what is considered "safe" and allows only input that meets those criteria. It is often used to validate very specific data. Example of use:


func isValidUsername(username string) bool {
    // using regex to check allowed characters 
    re := regexp.MustCompile(`^[a-zA-Z0-9_-]+$`)
    return re.MatchString(username)
}

func main() {
    input := "user_name-123"
    if isValidUsername(input) {
        fmt.Println("Valid username:", input)
    } else {
        fmt.Println("Invalid username.")
    }
}

In the example above, the use of regex ^[a-zA-Z0-9_-]+$ is used to limit the characters allowed in the username. Only letters, numbers, _ (underscore), and - (dash) are accepted.

- Blacklist

Blacklisting is an approach where you specify certain data or characters that are not allowed. In this case, all other inputs are considered safe, except those on the blacklist. This approach is often used when it is difficult to predict all possible safe characters. Example of use:

func isValidInput(input string) bool {
    // List not allowed characters
    blacklist := []string{"<", ">", "&"}

    for _, char := range blacklist {
        if strings.Contains(input, char) {
            return false
        }
    }
    return true
}

func main() {
    input := "hello & welcome"

    if isValidInput(input) {
        fmt.Println("Valid input:", input)
    } else {
        fmt.Println("Invalid input.")
    }
}

Here, we use a blacklist to check whether the input contains unwanted characters. If present, the input is considered invalid.

4. Input Limit

Input Limit is the practice of limiting the amount of data or size of input that can be accepted from users in an application. The goal is to prevent excessive use of resources, improve application performance, and protect against potential attacks, such as Denial of Service (DoS) or SQL injection. The following is an example of its implementation:

func validateUsername(username string) bool {
    if len(username) > 20 {
        return false // Username is too long username is too long
    }
    return true // username is valid
}

func main() {
    input := "ThisIsAReallyLongUsername"

    if validateUsername(input) {
        fmt.Println("Valid username:", input)
    } else {
        fmt.Println("Invalid username: username cannot exceed 20 characters.")
    }
}

Of course, some of the points above can be applied to any programming language and can be used using libraries that are widely available so that the data validation process is easier than carrying out a manual validation process like some of the examples above.

Data Security

Sensitive data such as personal information or credentials must always be protected, both when stored and when transferred. We can apply the following points to overcome this:

1. Data encryption

Sensitive data such as passwords or credit card information should always be encrypted. Encryption involves the process of converting readable data (plaintext) into an unintelligible form (ciphertext) using an encryption algorithm and encryption key. The main purpose of encryption is to ensure that even if data falls into the wrong hands, it cannot be read without the right key to decrypt it.

Use modern encryption algorithms such as AES and avoid outdated algorithms such as MD5 or SHA1.

2. Use HTTPS

Use HTTPS (Hypertext Transfer Protocol Secure) to ensure that all data transferred between the client and server is properly encrypted. HTTPS (Hypertext Transfer Protocol Secure) is a secure version of HTTP, which is a protocol used for data transfer between a browser (client) and a server.

HTTPS works by adding a layer of security by using TLS (Transport Layer Security) or SSL (Secure Sockets Layer) to encrypt data sent over the network. This is especially important in backend development because it prevents third parties (such as hackers) from reading or modifying data that is being sent between the client and the server.

3. Tokenization & Masking

In certain cases, tokenization or masking can be used to reduce the risk of a data breach. Tokenization and Masking are two data security techniques frequently used in backend applications to protect sensitive data.

Tokenization is the process of replacing sensitive data (such as credit card numbers) with meaningless values called “tokens.” The original data is stored separately and can only be linked back to the token through a system that has access to the system that manages the token. This is important because if a data leak occurs, but the leaked data is only a token and not real data, the risk of attack is greatly reduced because the token has no value outside the system context. For example, in online payments, credit card numbers can be replaced with tokens so that card data does not actually need to be stored on the server.

Data masking is the process of hiding part of sensitive data so that only some parts are visible. This is often used in user interfaces or reports to prevent complete information from being exposed. How does this work? data masking protects sensitive information when displayed to users who do not need to know the full data. Even if the data displayed in the UI or logs is stolen, the masked data only shows partial information, so the risk is lower. An example that we may often see is hiding part of a credit card number in a display like this: **** **** **** 1234.

Error Management & Logging

In building a system errors will always occur, but how we handle them determines whether our application can survive without causing damage or loss of data. Good logging helps track problems in production systems and detect suspicious patterns. Here are some points that we do:

1. Handle Errors Appropriately

Don't just catch errors without taking action or providing information. It's best to log errors and provide a good fallback to avoid a total crash. For example, if the connection to the database fails, try retrying or switching to read-only mode temporarily.

2. Granular Logging

Log critical events such as API requests, authentication failures, or database connection problems with a sufficient level of detail. However, remember to avoid recording sensitive information such as passwords in logs.

3. Log Splitting

Use different log levels such as INFO, WARNING, and ERROR to separate different types of events. More granular logs make it easier for developers to diagnose problems.

For a more complete discussion and examples of Error Management & Logging I have written an article about this here: Observability - Why logging is important

Avoid mistakes elegantly

Crashing is a response that occurs when the system encounters an unexpected error. However, errors can be avoided or minimized through better handling. The strategies below can help us handle errors elegantly to make the system more resilient.

1. Timeouts

When backend applications access external services or wait for resources such as databases, it is important to use timeouts. Timeouts prevent code from being "stuck" in an indefinite waiting state, thereby maintaining application performance and avoiding deadlocks. The following is an example of implementing a timeout when accessing an external service:

client := http.Client{
    Timeout: 5 * time.Second,  // Set timeout 5 detik
}

resp, err := client.Get("https://example.com/api")
if err != nil {
    fmt.Println("Request failed:", err)
    return
}
defer resp.Body.Close()

fmt.Println("Response received:", resp.Status)

In this example, we set a 5 second timeout for HTTP requests. If the service doesn't respond within 5 seconds, the app won't hang, and we can handle the error by announcing the failure.

2. Resource Existence Checks

Before accessing other resources such as databases, files or APIs, always verify their existence and availability. This prevents applications from making requests to resources that may not exist, which saves time and avoids unnecessary errors. Here's an example of how we check:

func main() {
    filePath := "./data.txt"
    _, err := os.Stat(filePath) // check status file

    if os.IsNotExist(err) {
        fmt.Printf("File %s not found.\n", filePath)
    } else {
        fmt.Printf("File %s is available.\n", filePath)
    }
}

3. Fallback

When the system carries out certain processes, we might get an error. For example, our system sends an OTP during the authentication process. When the OTP sending process fails or gets an error, we can use a fallback mechanism or backup method, such as diverting OTP sending to our backup provider.

By implementing the strategies above, we can reduce the frequency of errors that cause crashes, maintain application performance, and improve the overall user experience.

Secure and Solid API Design

We can see that APIs function as the backbone for applications or web that require dynamic data resources. However, because they are considered such a critical backbone, APIs are often the main target for attacks, so it is important to ensure APIs that are secure and difficult to exploit.

1. Authentication & Authorization

Use strong authentication (such as OAuth or JWT) to ensure that only authorized users or systems can access our APIs. Implement granular access control to limit the access rights of clients who access the API that we have. This can prevent clients with malicious intent from accessing the resources we have.

2. Rate Limiting and Throttling

Rate Limiting and Throttling are critical to controlling API usage and preventing misuse or overloading. The application of rate limiting is also to prevent attacks such as DDoS or excessive exploitation of the API by users or bots, and also to maintain the stability and availability of the API for all users. Rate Limiting works by regulating the volume of requests from API clients within set intervals, ensuring fair and equal access to the resources we have.

Efficient Use of Resources

Backend systems often have to manage various resources such as database connections, files, or memory. Efficient resource management is critical to maintaining application stability and performance. By handling resources wisely, we can avoid problems such as "resource leaks" and system failures. The following are several strategies that can be used for efficient use of resources:

1. Connection Pooling

Connection Pooling is a technique where a number of connections to a service such as a database are recycled rather than creating a new connection every time an application needs access. With connection pooling, applications do not need to create expensive new connections every time there is a request, but instead use existing connections in the "pool".

Why is connection pooling important?, Every connection to a database or other service has overhead in terms of time and memory. Continuously opening and closing connections can drain system resources. By recycling existing connections, applications can respond more quickly and reduce the load on external services such as databases.

2. File & Memory Management

When dealing with files or memory allocation, it is important to ensure that resources are released after they have been used. If not, it could cause resource leaks which result in the application running out of memory or file handles.

For example, in Golang we can use the built-in function defer to ensure files, connections, or other resources are always released after they are finished using them. Then also check for errors (error handling) when working with resources, so that if a problem occurs, the resource can still be released correctly.

Here's a little code snippet of how we use defer to turn off an operation and close it after a function has finished carrying out a process:

func main() {
    // open file process
    file, err := os.Open("example.txt")
    if err != nil {
        fmt.Println("Error opening file:", err)
        return
    }

    // Make sure the file is closed when finishedinished
    defer file.Close()

    // Process file...
}

By using defer, we ensure that the file will be closed as soon as the function exits, whether it is successful or an error occurs. This helps prevent file handle leaks that can occur if you forget to close a file.

3. Asynchronous Operations

In situations where the application performs heavy tasks or slow I/O operations (such as accessing large files, or calling external APIs), running it Asynchronously is an efficient way to avoid blocking the application. Where applications that perform synchronous operations can be hampered while waiting for heavy tasks to complete, which will affect overall performance. So the use of asynchronous processes is sometimes important because it allows applications to continue other tasks while waiting for I/O or network tasks to complete.

Let's say we have a case where we clear the Redis cache after data input. The cache deletion process can be run asynchronously so as not to slow down the main process. This means the application does not need to wait for Redis to respond before continuing to execute the next code. As a result, the application remains responsive and efficient.

Conclusion

As a backend developer, defensive programming is an essential approach to ensuring that the applications we build can handle errors, invalid input, and external threats properly. By practicing the techniques above, we can ensure that the backend applications we build are more secure, stable and resilient in facing various unexpected situations. Defensive programming helps us create systems that not only function well, but are also able to survive the worst conditions.

Of course, there may be many other important points that are not discussed above, you can add or provide suggestions in the comments column above.

Hopefully it's useful 👋.

Reference

Master Multiple Density Image Assets in Flutter: Boost Your App’s Visuals!

Vincentius Westley — Tue, 08 Oct 2024 03:19:39 +0000

As the mobile landscape continues to evolve, the range of screen sizes and resolutions across devices has expanded significantly. From compact smartphones to expansive tablets, ensuring that your application’s visuals remain sharp and consistent across all these devices is crucial. This is where the concept of multiple density image assets comes into play, especially in the context of Flutter—a popular framework for cross-platform mobile development.

📄 Understanding Screen Density and Image Assets
Before getting into the details of Flutter it’s important to understand the basics of screen density and image assets:

Screen Density (DPI): Density refers to the number of pixels within a given physical area of the screen, typically measured in dots per inch (DPI). Devices with higher screen densities have more pixels per inch, which makes their displays sharper and clearer.
Image Assets: These are the visual elements (like icons, buttons, and background images) used in an app. To appear sharp on screens with different densities, these assets need to be provided in multiple resolutions.

Why Multiple Density Image Assets Matter
1. Enhanced Visual Quality: High-resolution devices demand high-quality images to avoid appearing pixelated. By providing multiple density image assets, you ensure that your images look sharp and clear on any device, regardless of its screen density.

2. Optimized Performance: Using the appropriate image for each screen density can significantly improve performance. Loading images that are too large for a given screen’s resolution wastes memory and processing power, leading to slower performance and higher power consumption. Multiple density assets allow Flutter to select the most suitable image, optimizing both memory usage and performance.

3. Consistent User Experience: Ensuring that images look consistent across various devices is crucial for a seamless user experience. Whether your app is being used on a low-resolution screen or a high-resolution one, users should have a consistent visual experience. Multiple density image assets help achieve this by providing the right image for each screen type.

4. Scalability Across Devices: As new devices with varying screen resolutions are released, having multiple density assets makes your app scalable and future-proof. It ensures that your app can easily adapt to new hardware without requiring significant changes or additions to your image assets.

Implementing Multiple Density Image Assets in Flutter
Flutter simplifies the process of handling multiple density image assets through its asset management system. Here’s a step-by-step guide on how to implement them effectively:

1. Preparing Image Assets

For each image asset, prepare multiple versions at different resolutions. Commonly, these are:

1x for baseline resolution (mdpi)
1.5x for low-density screens (hdpi)
2x for medium-density screens (xhdpi)
3x for high-density screens (xxhdpi)
4x for extra-high-density screens (xxxhdpi)

For example, if you have an icon named logo.webp for a baseline resolution, you should also provide logo@1.5x.webp, logo@2x.webp, logo@3x.webp, and logo@4x.webp.

📄 - In this example, I'm using WebP for all the images because WebP offers better image compression. It's a good idea to use WebP for your images too. To learn more, visit this link.

2. Organizing Assets in the Project

Organize these assets in your Flutter project’s assets directory. A typical structure might look like this:

assets/
  - images/
      - logo.webp
      - 1.0x/
        - logo.webp
      - 1.5x/
        - logo.webp
      - 2.0x/
        - logo.webp
      - 3.0x/
        - logo.webp
      - 4.0x/
        - logo.webp

Defining Assets in pubspec.yaml

In your pubspec.yaml file, list the image assets. Flutter will automatically detect and use the appropriate resolution based on the device’s screen density:

flutter:
  assets:
    - assets/images/logo.webp

Using Assets in Your Flutter Code

When you need to use these assets in your Flutter widgets, simply refer to the base asset name. Flutter’s asset resolution system will automatically select the most suitable image:

Image.asset('assets/images/logo.webp');

Best Practices for Managing Multiple Density Assets
1. Asset Compression: Optimize your images for size without sacrificing quality. Tools like Respresso can help compress images while maintaining their visual fidelity and also splitting your image to all density.

2. Regular Updates: As new devices with higher resolutions are released, ensure that your assets are updated accordingly to support these new screens.

3. Testing on Multiple Devices: Always test your app on a variety of devices and screen densities to ensure that your assets render correctly and the user experience remains consistent.

⛔ Trade-offs of Implementing Multiple Density Image Assets

While multiple density image assets offer some benefits, they also come with certain trade-offs that developers should consider:

Increased App Size
In Flutter providing multiple versions of each image for different screen densities will inevitably increase the overall size of your app. This can be a concern for users with limited storage or slow download speeds. Large app sizes can also affect your app’s ranking and reviews, as users often prefer smaller, quicker-to-download apps.

📄 - The way multiple image densities are handled differs between native apps and Flutter. In native apps, using different densities can reduce the app's download size because only the appropriate density images are downloaded for each device. However, in Flutter, all image densities are included in the app package, which means the app download size isn't reduced by splitting images based on density.

Development and Maintenance Overhead
Managing multiple versions of each image can increase the complexity of your development process. You need to ensure that all images are updated consistently across all densities, which can be time-consuming. This also requires designers and developers to work closely together to maintain image quality and consistency across different resolutions.

Balancing Trade-offs
When deciding whether to implement multiple density image assets, consider the nature of your app and your target audience:

- High-End Visuals: If your app relies heavily on visual quality (e.g., a graphic-rich game or a photo editing app), investing in multiple density assets is crucial despite the increased app size.

- Performance Optimization: For apps that prioritize performance and efficient resource use, multiple density assets can reduce CPU load and power consumption, making the app more responsive.
**

App Size Constraints**: If your app is targeted at regions with limited internet connectivity or users who are sensitive to download sizes, you might choose to use fewer resolutions or use methods that deliver assets as needed.

✨ Conclusion
Implementing multiple density image assets in Flutter is a strategic choice that balances visual quality, performance, and app size. By providing the right assets for each screen density, you ensure your app looks sharp and performs well across a variety of devices. However, this comes at the cost of increased app size and development complexity.

As a Flutter developer, it’s essential to evaluate your app’s specific needs and user demographics when deciding how to manage image assets. Whether you choose to implement multiple density image assets for optimal performance or streamline assets to minimize app size, the decision should align with your app’s goals and user expectations. By carefully considering these factors, you can deliver a high-quality user experience while maintaining efficient app performance.

How to Handle Time Zones and Sync Your Software on the server side using Go

Rizky Darmawan — Tue, 01 Oct 2024 02:08:22 +0000

When your application starts on a large scale, the increase in users will increase. What is very likely to happen is that the user's location is not only in the same area, it could be in another area that has a different time zone. So as a Backend developer, things related to handling time zone differences are very important to think about.

I recently came across an issue involving time zones. Let's be honest, dealing with dates and times is one of the most complicated areas a human being has to deal with. And this was an opportunity for me to learn how to properly handle dates and times on the server side.

In this article, I will share my experience on how I handle time zone differences on the server side as a Backend developer. Maybe if someone is willing to correct and provide additional input that would be valuable for me.

About Time Zone

Time zones are a standard time division system used throughout the world to organize and standardize the measurement of time. This concept emerged as a response to the need for global time coordination, especially along with developments in communications and transportation technology.

The basic principle of time zones is the division of the earth into 24 zones. Each time zone is generally one hour different from the zone next to it. The main reference for time zones is Greenwich Mean Time (GMT) or Coordinated Universal Time (UTC), which is at the zero degrees longitude line that passes through Greenwich, England.

Illustration by Hellerick from Wikimedia Commons

A small example is when the clock shows 12:00 noon in Jakarta, in New York the time shows 00:00 or midnight. This means that while Jakartans are enjoying lunch, New Yorkers are just starting their new day. From here you can certainly imagine the importance of handling time zones correctly in building an application.

Timezone handling on the server side

After we have seen the explanation above, now we will go into the points that can be done when our server application receives a request from a client that accesses our API to handle its time zone.

In this article, I will discuss several approaches to handling time zones on the server side. Here I will use code examples in the Golang (Go) language. Golang has a time package for working with time-related data which is considered quite complete. Here are some points that we will discuss:

How to save date to DB
Conversion of user's local time
Testing and Validation

How to save date to DB

The first thing we will discuss is which time we will save in the database, for example we have an ecommerce application that does flash sales, where our application is already on an international scale.

When a user processes a purchase transaction in America or if the user is in Indonesia, the user will send their different local time to the server. The question is, will our database store time data according to the user's local time? If the answer is yes, it is likely a complicated problem when we want to retrieve the data or for example the admin wants to do data processing, which user makes the earliest transaction.

To overcome this, the best practice is to store transaction times in the UTC (Coordinated Universal Time) time zone which is the primary time standard for clocks and time settings. Here is the application of the time to UTC.

package main

import (
    "fmt"
    "time"
)

func main() {

    now := time.Now()
    fmt.Printf("Local time: %s\n", now)

    nowInUTC := now.UTC()
    fmt.Printf("UTC time: %s\n", nowInUTC)

}

Let's see the meaning of the code above.

First, in the now := time.Now() code line, this line uses the Now() function from the time package to get the current time based on the system's local time zone. The result is stored in the current variable.

Then, in the nowInUTC := now.UTC() line, here we convert the local time (now) to UTC time using the UTC() method. The results are stored in a new variable nowInUTC and this time can be stored on the server, where it is hoped that developers can avoid ambiguity and errors that may arise due to time zone differences between servers and users in various regions with different time zones.

Here are the results if we run the code above:

But this is not always the best solution that you should use. There may be some points you can keep in mind in certain use cases, one of which is is it true that our users come from different time zones? If it's not possible, perhaps storing the time in UTC will add complexity to your code.

Change time to user locale

At the discussion point above, we have agreed to store user time data in one location, namely UTC. Now how users can see accurate time according to their location. An example of the discussion above is a flash sale on an e-commerce application that we have, where users also want to know information about which user made the first transaction. So at this point, converting the time we store in the database to the user's local time is another important thing that we should not ignore.

The approach I take is that the server side always asks the client to send the timezone on the user side. This can be done on the request side where the client sends a header with the key timezone and has the user's timezone value. For example, Indonesia has 3 time zone divisions, namely WIT(+9), WITA(+8), WIB(+7). Where each zone has a difference of 1 hour. If previously on our server we stored UTC time at 00.00, then on the WIT side it was at 09.00, then on the WITA side it was at 08.00 and WIB at 07.00.

Here's an example code to handle the above case:

package main

import (
    "fmt"
    "time"
)

func ParseTimezoneFromString(tz string) *time.Location {

    if len(tz) > 0 {

        t, err := time.Parse("2006 -07:00", fmt.Sprintf("2007 %s", tz))

        if err != nil {

            panic(err)

        } else {

            return t.Location()
        }
    }

    return time.Now().Location()
}

func main() {

    timeServerInUTC := "2024-08-04 00:00:00"
    nowInUTC, err := time.Parse("2006-01-02 15:04:05", timeServerInUTC)
    if err != nil {
        panic(err)
    }

    fmt.Printf("UTC time: %s\n", nowInUTC)

    witLocation := ParseTimezoneFromString("+09:00")

    nowInWIT := nowInUTC.In(witLocation)
    fmt.Printf("WIT time: %s\n", nowInWIT)

    witaLocation := ParseTimezoneFromString("+08:00")

    nowInWITA := nowInUTC.In(witaLocation)
    fmt.Printf("WITA time: %s\n", nowInWITA)

    wibLocation := ParseTimezoneFromString("+07:00")

    nowInWIB := nowInUTC.In(wibLocation)
    fmt.Printf("WIB time: %s\n", nowInWIB)

}

credit to dikac for create this function ParseTimezoneFromString

Let's understand the meaning of the code above:

First, we create a function ParseTimezoneFromString, where this function is used to find the time location based on the argument tz or timezone of the given user location. If the string value tz is valid, we will convert the string's timezone using the time.Parse function to convert the string to a time object, then extract the location (timezone) from that object. And we also handle if the string is empty or parsing fails, the function returns the local time zone of the system.

func ParseTimezoneFromString(tz string) *time.Location {

    if len(tz) > 0 {

        t, err := time.Parse("2006 -07:00", fmt.Sprintf("2007 %s", tz))

        if err != nil {

            panic(err)

        } else {

            return t.Location()
        }
    }

    return time.Now().Location()
}

Next we also define time data in the following string format:

timeServerInUTC := "2024-08-04 00:00:00"

nowInUTC, err := time.Parse("2006-01-02 15:04:05", timeServerInUTC)
if err != nil {
    panic(err)
}

You can think of this as the timing data that we get from the server. And parse it into a time object.

Next, we try to find the user's accurate location based on the ParseTimezoneFromString function that we previously created based on the string argument that we defined. What needs to be paid attention to is that this string argument is what is meant by the value of the timezone header sent by the client via the incoming request.

We can use the location we get from the ParseTimezoneFromString function to convert or shift the time we get from the server to the user's local time. We can do this using the In function which is also included in the time package which we can see in the following code snippet:

nowInWIT := nowInUTC.In(witLocation)
nowInWITA := nowInUTC.In(witaLocation)
nowInWIB := nowInUTC.In(wibLocation)

If we run it, we will get the time that corresponds to the timezone location that we defined.

Testing and Validation

The final point is no less important, namely testing and validation. When the development process often causes developers to make unexpected mistakes, testing and validation are always important.

In the discussion of point 2 above, the ParseTimezoneFromString function has been important in handling our time zones. Repeated testing and validation are important to make our applications get results that meet our expectations.

For testing, it is recommended to use unit tests, where testing will be carried out on the smallest unit with several scenarios that can be added. The more scenarios there are, the less likely it is to handle these time differences.

Conclusion

Handling time zones can indeed be tricky for backend developers. However, it's important to remember that every challenging task we overcome contributes to our growth and skill improvement. Properly managing time zones is not just a technical necessity, it ensures accuracy in scheduling and provides a smooth experience for our application users across different regions.

The points shared in this article about storing times in UTC, converting to user local times, and implementing robust conversion functions are starting points in tackling this complex issue. However, I acknowledge that there may be shortcomings or areas for improvement in the approaches discussed. This is why additional input and suggestions from the developer community are invaluable.

I sincerely hope that the insights and code examples provided in this article will be helpful to you in the future when you encounter time zone-related challenges in your projects. Remember, the goal is to create applications that work seamlessly for users, regardless of their geographical location.

Let's continue this discussion in the comments section below. I'd love to hear about your experiences with handling time zones, any challenges you've faced, or alternative approaches you've found effective. Your insights could be extremely valuable to me and other readers facing similar issues.

Thank you for reading, and I hope this article proves useful in your development journey. Let's keep learning and improving together! 👋

Reading References

Flutter Scrolling Techniques: Using GlobalKey and Scrollable.ensureVisible() for Precise Navigation

Irfan Adisukma — Fri, 27 Sep 2024 09:27:02 +0000

In Flutter development, creating smooth and intuitive user experiences often involves guiding users to specific parts of the interface. One powerful technique to achieve this is programmatically scrolling to a particular widget. Whether it’s navigating to a section in a long form, focusing on a specific item in a list, or highlighting a new message in a chat, the ability to scroll to a specific widget enhances the overall usability and interactivity of an app.

Flutter provides several tools, such as GlobalKey, and Scrollable.ensureVisible(), that allow developers to precisely control scrolling behavior. By leveraging these tools, developers can ensure that important content is brought into view, reduce the need for excessive manual scrolling, and create more seamless navigation experiences. This not only helps in improving user engagement but also makes the application more accessible and efficient, especially for apps with complex layouts or dynamic content.

In this article, we’ll explore how to effectively scroll to a specific widget in Flutter using these techniques, ensuring that your app's navigation is as smooth and user-friendly as possible.

Understanding the Components

When building a Flutter app, having control over how users navigate through content is crucial for creating a smooth and engaging experience. To achieve precise scrolling behavior, three key components can be used: GlobalKey, and Scrollable.ensureVisible(). Let's dive deeper into each of these components and understand their roles.

GlobalKey: In Flutter, GlobalKey is a type of key that is used to uniquely identify a widget within the entire widget tree. It enables developers to access the widget's context, state, and position, which is essential when you need to perform actions such as scrolling to a particular widget or manipulating its state. By assigning a GlobalKey to a widget, you can easily reference it later in your code to trigger actions like scrolling, focusing, or even retrieving information about its dimensions and position. And it's compatible with all Flutter versions.
Scrollable.ensureVisible(): The Scrollable.ensureVisible() method is a utility function provided by Flutter that ensures a given widget is visible within its scrollable ancestor. When called, it automatically calculates the distance and direction to scroll in order to bring the target widget into view. This is particularly useful for creating smooth and automated scrolling effects, such as jumping to a specific section of a form, highlighting a newly added item in a list, or scrolling to a widget based on user actions (e.g., clicking a button). By combining Scrollable.ensureVisible() with GlobalKey, developers can programmatically control the visibility of widgets, enhancing navigation and user experience. This feature available and supported from Flutter 1.5.4 onwards. So make sure you are using latest version.

By understanding these two components— GlobalKey, and Scrollable.ensureVisible()—you can effectively implement advanced scrolling behaviors in your Flutter app, ensuring that users can easily navigate and interact with content in a seamless and intuitive way.

Creating the UI

To demonstrate, let's create a simple UI containing a list of items.

Step-by-Step Example
Here is a basic example of setting up:

ListView.builder(
 shrinkWrap: true,
 itemCount: 50,
 physics: const NeverScrollableScrollPhysics(),
 itemBuilder: (context, index) {
   return ListTile(title: Text("Item $index"));
 }
)

Now let's add GlobalKey to identify widgets

final GlobalKey _scrollKey = GlobalKey();

Assign this key to the widget within your ListView:

ListTile(
  key: _scrollKey,
  title: Text('Scroll to Me!'),
)

And the code should be like this with additional handling to override generated list item with the GlobalKey:

ListView.builder(
 shrinkWrap: true,
 itemCount: 50,
 physics: const NeverScrollableScrollPhysics(),
 itemBuilder: (context, index) {
   if (index == 25) {
     return ListTile(
      key: _scrollKey,
      title: Text("Scroll to Me!")
     );
   }
   return ListTile(title: Text("Item $index"));
 }
)

Now let's add ScrollC

Let's add one final touch by adding Button to trigger the scroll using Scrollable.ensureVisible():

void _scrollToTarget() {
  // Ensure the widget is built
  WidgetsBinding.instance.addPostFrameCallback((_) {
    final context = _scrollKey.currentContext;
    if (context != null) {
      Scrollable.ensureVisible(
        context,
        duration: const Duration(seconds: 1), 
        curve: Curves.easeInOut);
      }
   });
 }

ListView(
 children: [
  ElevatedButton(
    onPressed: () async {
      _scrollToTarget();
    },
    child: const Text('Scroll to Widget'),
  ),
  ListView.builder(
    shrinkWrap: true,
    itemCount: 50,
    physics: const NeverScrollableScrollPhysics(),
    itemBuilder: (context, index) {
      if (index == 25) {
       return ListTile(
        key: _scrollKey,
        title: Text("Scroll to Me!")
       );
      }
      return ListTile(title: Text("Item $index"));
     }
   )
  ],
)

Here's the output!

How Scrollable.ensureVisible works?

The Scrollable.ensureVisible() method takes in the BuildContext of a widget and scrolls the nearest scrollable parent to make that widget fully visible.

When Scrollable.ensureVisible() is called, Flutter calculates the position of the widget represented by the context in its scrollable ancestor.
If the widget is already fully visible, no scrolling occurs. If not, the method scrolls the closest scrollable widget (like CustomScrollView, ListView, etc.) to bring the target widget into view.
The scrolling animation takes place over the specified duration and follows the provided curve, resulting in a smooth scroll to the target widget.

This is real example of how we implement this technique on Tentang Anak App, we have a feature section that must be highlighted and we help user to find this section by tapping the button, and it will automatically scroll to the specific section.

Additional Tips

When implementing scrolling to a specific widget in Flutter using GlobalKey, and Scrollable.ensureVisible(), it's essential to consider performance and handle potential edge cases to ensure a smooth and responsive user experience. Here are some additional tips to help you optimize your implementation:

Performance Considerations

When working with large lists or complex scrollable content, performance optimization is crucial. Here are some strategies to enhance scrolling and rendering performance:

Avoid Unnecessary Rebuilds: Ensure that widgets within your ListView do not rebuild unnecessarily. Use const constructors where possible, and consider using state management solutions like Provider or Riverpod to manage the state outside of the widget tree, avoiding frequent rebuilds.
Virtualization with ListView.builder and GridView.builder: If your use case involves scrolling a long list of items (e.g., 100+ items), consider using ListView.builder or GridView.builder instead of SliverList to optimize memory usage. These builders only render visible items and reuse widgets that move out of the viewport.
Throttle or Debounce User Actions: If you allow users to trigger scrolling actions frequently (e.g., tapping buttons to scroll to different items), consider throttling or debouncing these actions to prevent excessive animations and rendering.
Use the automaticKeepAlive property: For complex widgets that need to maintain their state while scrolling, consider using AutomaticKeepAlive widgets to manage their lifecycles better without unnecessarily keeping off-screen widgets alive.

Handling Edge Cases

Proper handling of edge cases ensures that your scroll functionality works reliably in all scenarios:

Ensure Widget Exists in the Viewport: Before calling Scrollable.ensureVisible(), ensure that the target widget is part of the scrollable content. If the widget is dynamically added or removed (e.g., in response to user actions), you need to verify that it is indeed within the scrollable ancestor when the method is called. Failing to do so can result in exceptions or unexpected behavior.
Handle Already Visible Widgets: If the widget is already fully visible in the viewport, Scrollable.ensureVisible() won’t trigger any scrolling. Make sure to handle such cases gracefully to avoid redundant animations or confusing the user.
Handle Partially Visible Widgets: Consider scenarios where the widget is only partially visible (e.g., cut off at the top or bottom of the viewport). Scrollable.ensureVisible() will scroll to bring it fully into view, but you might want to adjust the duration or curve for smoother transitions, especially if only a small amount of scrolling is needed.
Check for Nested Scroll Views: If you have nested scroll views, calling Scrollable.ensureVisible() may not work as expected if the target widget is inside a nested scroll view. Ensure that the method is called on the correct scrollable parent and that the widget’s context is properly obtained from the right GlobalKey.
Manage Focus and Accessibility: After scrolling to a widget, you may want to give it focus (e.g., for an input field) or announce it to assistive technologies. Use FocusScope.of(context).requestFocus() or Semantics to enhance accessibility and provide visual feedback.
Consider Scroll Offset Adjustments: Sometimes, you may want to scroll a bit past the widget or stop before it reaches the top or bottom of the viewport to provide some padding or context. You can use alignment or calculate offsets manually to achieve more precise positioning.

By considering these performance tips and handling edge cases, you can implement scrolling to specific widgets in a way that is both efficient and robust, ensuring a seamless and intuitive experience for your Flutter app users. This approach not only improves usability but also contributes to a polished, professional app that handles a variety of user interactions gracefully.

Conclusion

Scrolling to a specific widget in Flutter can significantly enhance the user experience by ensuring that important content is always visible and easily accessible.

In summary, using GlobalKey, and Scrollable.ensureVisible() together provides a powerful toolkit for creating responsive and user-friendly scrolling experiences in Flutter apps. This approach not only guides users through different sections of an app smoothly but also enhances the overall usability and accessibility, leading to a more polished and professional application.

See the full code
GitHub

DEV Community: Tentang Anak Tech Team

Basic Concept & Benefit of API Gateway

Photo by Javier Allegue Barros on Unsplash

What is API Gateway ?

How does API Gateway work ?

Benefits of API Gateway

1. Enhanced security

2. Better monitoring and visibility

3. Data Format Transformation

4. API Versioning and Backward Compatibility

5. Enhanced Error Handling

Drawbacks of Impelementation API Gateway

1. Single Point of Failure

2. Additional Complexity

3. Increased Latency

4. Cost

Simple Example of an API Gateway

The structure Folder

The rate limitter function code

The proxy function code

Then main function code

Let's run it

Conclusion

Reading References

Great Code Starts with Great Planning

The Journey of a Beginner Developer

The Beginner Stage

Navigating the Highs and Lows

Finding Strength in the Beginner Stage

Plan First, Code Later

Case Study: Instagram's Early Development

Before Planning (Jumping Straight into Code)

After Planning (More Effective & Maintainable Code)

Step 3: Use State Management (UserProvider.dart)

Step 4: UI with Separation of Concerns (UserListScreen.dart)

Step 5: Setup in main.dart

Connection Pool in Backend Development: Basic Concept, Benefits, and Implementation

Photo by Christine Tutunjian on Unsplash

Why connection pool is important ?

Better performance

Resource savings

Scalability

Load configuration

How to Works ?

Pool Initialization

Connection usage

Restore connection

The common main parameter in connection pool

Pool Size

Minimum Connections

Idle Connection

Timeout

Max lifetime

Implementation

Conclusion

Mastering ENUMs in Go

Image by Hans-Peter Gauster from Unsplash

Understanding Enum

Objective

Create Enums

Create a New Type

Define constant ENUM

Function Strings

Testing

Fmt Stringer

Conclusion

Reading References

Avoid Boilerplate with Code Generator in Flutter

Code Generation with build_runner

Implementation

Drawback

Source Code

Conclusion

References

Profiling Memory In Go

Why Memory Profiling Matters

Common Causes of High Memory Usage

Setting Up Memory Profiling in Go

Profiling Tools

Using pprof

Using `pprof`

Using `parca` for Continuous Profiling