In modern software development, performance is often a critical factor, especially when dealing with applications that require fetching and processing large amounts of data. Recently, I worked on optimizing a Go method that fetches user data and statistics from a database. By introducing concurrency, I was able to significantly reduce the execution time of the method. In this post, I’ll walk you through the changes I made and the results I achieved.
The Problem
The method in question, GetAllUsersData, is responsible for fetching a list of users along with various statistics such as the highest and lowest age, average age, highest and lowest salary, and average salary. Initially, the method was implemented sequentially, meaning each database query was executed one after the other. Here’s a simplified version of the original implementation:
// GetAllUsersData fetches user data and statistics
func (c *Controller) GetAllUsersData(limit, offset int, orderBy, sort string) (map[string]interface{}, error) {
methodName := "GetAllUsersData"
c.Logger.Debug("method start", zap.String("method", methodName))
start := time.Now()
var (
userList []*models.User
count string
intUserHighAge int
intUserLowAge int
fltUserAvgAge float64
fltUserAvgSalary float64
fltUserLowSalary float64
fltUserHighSalary float64
err error
)
userList, count, err = c.Repo.GetAllUserDB(limit, offset, orderBy, sort)
if err != nil {
return nil, err
}
intUserHighAge, err = c.Repo.GetUserHighAge()
if err != nil {
return nil, err
}
intUserLowAge, err = c.Repo.GetUserLowAge()
if err != nil {
return nil, err
}
fltUserAvgAge, err = c.Repo.GetUserAvgAge()
if err != nil {
return nil, err
}
fltUserLowSalary, err = c.Repo.GetUserLowSalary()
if err != nil {
return nil, err
}
fltUserHighSalary, err = c.Repo.GetUserHighSalary()
if err != nil {
return nil, err
}
fltUserAvgSalary, err = c.Repo.GetUserAvgSalary()
if err != nil {
return nil, err
}
responseUserObj := models.ResponseUser{
HighAge: strconv.Itoa(intUserHighAge),
LowAge: strconv.Itoa(intUserLowAge),
AvgAge: fmt.Sprintf("%.2f", fltUserAvgAge),
HighSalary: fmt.Sprintf("%.2f", fltUserHighSalary),
LowSalary: fmt.Sprintf("%.2f", fltUserLowSalary),
AvgSalary: fmt.Sprintf("%.2f", fltUserAvgSalary),
Count: count,
List: userList,
}
var responseObj map[string]interface{}
err = mapstructure.Decode(responseUserObj, &responseObj)
if err != nil {
return nil, err
}
duration := time.Since(start)
seconds := int(duration.Seconds()) % 60
milliseconds := duration.Milliseconds() % 1000
strDuration := fmt.Sprintf("%02d.%03d", seconds, milliseconds)
c.Logger.Debug("method end", zap.String("method", methodName), zap.String("Time elapsed for this method:", strDuration))
return responseObj, nil
}
While this approach worked, it was not optimal. Each database call had to wait for the previous one to complete, leading to longer execution times. For example, the method took 64 milliseconds to complete in its sequential form.
The Solution: Introducing Concurrency
To improve performance, I decided to leverage Go’s concurrency model using Goroutines and the errgroup package. The idea was to execute all database queries concurrently, allowing them to run in parallel and reduce the total execution time.
Here’s the updated version of the method:
// GetAllUsersData fetches user data and statistics concurrently.
func (c *Controller) GetAllUsersData(limit, offset int, orderBy, sort string) (map[string]interface{}, error) {
methodName := "GetAllUsersData"
c.Logger.Debug("method start", zap.String("method", methodName))
start := time.Now()
//ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
//defer cancel()
g, _ := errgroup.WithContext(context.Background())
var (
userList []*models.User
count string
intUserHighAge int
intUserLowAge int
fltUserAvgAge float64
fltUserAvgSalary float64
fltUserLowSalary float64
fltUserHighSalary float64
)
g.Go(func() error {
var err error
userList, count, err = c.Repo.GetAllUserDB(limit, offset, orderBy, sort)
if err != nil {
return err
}
return nil
})
g.Go(func() error {
var err error
intUserHighAge, err = c.Repo.GetUserHighAge()
if err != nil {
return err
}
return nil
})
g.Go(func() error {
var err error
intUserLowAge, err = c.Repo.GetUserLowAge()
if err != nil {
return err
}
return nil
})
g.Go(func() error {
var err error
fltUserAvgAge, err = c.Repo.GetUserAvgAge()
if err != nil {
return err
}
return nil
})
g.Go(func() error {
var err error
fltUserLowSalary, err = c.Repo.GetUserLowSalary()
if err != nil {
return err
}
return nil
})
g.Go(func() error {
var err error
fltUserHighSalary, err = c.Repo.GetUserHighSalary()
if err != nil {
return err
}
return nil
})
g.Go(func() error {
var err error
fltUserAvgSalary, err = c.Repo.GetUserAvgSalary()
if err != nil {
return err
}
return nil
})
if err := g.Wait(); err != nil {
return nil, err
}
responseUserObj := models.ResponseUser{
HighAge: strconv.Itoa(intUserHighAge),
LowAge: strconv.Itoa(intUserLowAge),
AvgAge: fmt.Sprintf("%.2f", fltUserAvgAge),
HighSalary: fmt.Sprintf("%.2f", fltUserHighSalary),
LowSalary: fmt.Sprintf("%.2f", fltUserLowSalary),
AvgSalary: fmt.Sprintf("%.2f", fltUserAvgSalary),
Count: count,
List: userList,
}
var responseObj map[string]interface{}
err := mapstructure.Decode(responseUserObj, &responseObj)
if err != nil {
return nil, err
}
duration := time.Since(start)
seconds := int(duration.Seconds()) % 60
milliseconds := duration.Milliseconds() % 1000
strDuration := fmt.Sprintf("%02d.%03d", seconds, milliseconds)
c.Logger.Debug("method end", zap.String("method", methodName), zap.String("Time elapsed for this method:", strDuration))
return responseObj, nil
}
In this version, each database query is executed in a separate Goroutine. The errgroup package ensures that all Goroutines complete successfully and handles any errors that may occur.
The Results
After implementing concurrency, I ran the method and compared its performance with the original sequential version. The results were impressive:
Without Concurrency: 64 milliseconds
With Concurrency: 24 milliseconds
By running the database queries concurrently, the method’s execution time was reduced by 62.5%. This is a significant improvement, especially in scenarios where the method is called frequently or where low latency is critical.
Key Takeaways
- Concurrency is Powerful: Go’s Goroutines and channels make it easy to implement concurrency, allowing you to execute multiple tasks in parallel and improve performance.
- Error Handling Matters: Using errgroup simplifies error handling in concurrent operations, ensuring that any errors are properly propagated and handled.
- Measure and Optimize: Always measure the performance of your code before and after optimization to quantify the impact of your changes.
Conclusion
Introducing concurrency in the GetAllUsersData method was a straightforward yet highly effective way to improve its performance. By running database queries concurrently, I was able to reduce the execution time from 64 milliseconds to 24 milliseconds—a significant improvement. If you’re working on similar performance-critical applications, I highly recommend exploring Go’s concurrency features to unlock faster and more efficient code.
Top comments (0)