I am trying to merge all json data from paginated api responses with Go, this is how it looks like

Background

Once upon a time, I need to consume list of users data from an api which uses json response. As I predicted earlier, the api provider creates a pagination responses when the consumer interacts with list. Unfortunately I need to consume all users which means I have to hit every possible pagination query parameter to fetch all users.

here's the api endpoint

.../api/v2/users?page=1&per_page=4

and here's the api response

{
  "meta": {
    "page": 1,
    "per_page": 5,
    "total_pages": 100
  },
  "data": [
    {
      "id": "ef5b4299-b8d0-40c2-8442-2fdc90e342ae",
      "firstname": "Cale",
      "lastname": "Schmidt",
      "email": "[e-mail]"
    },
    {
      "id": "c0328632-1f18-4cf6-ad53-17278ffcc7bb",
      "firstname": "Norval",
      "lastname": "Feest",
      "email": "[e-mail]"
    },
    {
      "id": "0c467516-444f-4d01-b40d-82f7a646820a",
      "firstname": "Muriel",
      "lastname": "Berge",
      "email": "[e-mail]"
    },
    {
      "id": "2942a4b0-eac4-463c-87e3-6c7d0406cc81",
      "firstname": "Cletus",
      "lastname": "Considine",
      "email": "[e-mail]"
    },
    {
      "id": "2942a4b0-eac4-463c-87e3-6c7d0406cc81",
      "firstname": "Cletus",
      "lastname": "Considine",
      "email": "[e-mail]"
    }
  ]
}

First thing that comes to my mind when i face this problem is i go to first page and get last page information and then i will iterate to reach every possible page and merge every user from api response. To improve performance then I need to do it asynchronously for each page. As a result the user data I manage to merge is not sequential and I'm ok with that.

Code

I developed a module which provide the functionality as i mention earlier. You can get it in this repository.

Usage

Create a struct that representing json response which bind api response to a static data type.

type Meta struct {
    Page    int `json:"page"`
    PerPage int `json:"per_page"`
    TotalPages int `json:"total_pages"`
}

type User struct {
    Uuid  string `json:"id" gorm:"column:id;primaryKey"`
    FirstName string `json:"firstname" gorm:"column:firstname"`
    LastName  string `json:"lastname" gorm:"column:lastname"`
    Email     string `json:"email" gorm:"column:email"`
}

type Response struct {
    Meta Meta   `json:"meta"`
    User []User `json:"data"`
}

func (resp *Response) GetBoundary() int {
    return resp.Meta.TotalPages
}

In case you're asking about tag "gorm" , yes i will use it.
Then i create module instance with some configurations like this

type Meta struct {
    Page    int `json:"page"`
    PerPage int `json:"per_page"`
    TotalPages int `json:"total_pages"`
}

type User struct {
    Uuid  string `json:"id" gorm:"column:id;primaryKey"`
    FirstName string `json:"firstname" gorm:"column:firstname"`
    LastName  string `json:"lastname" gorm:"column:lastname"`
    Email     string `json:"email" gorm:"column:email"`
}

type Response struct {
    Meta Meta   `json:"meta"`
    User []User `json:"data"`
}

func (resp *Response) GetBoundary() int {
    return resp.Meta.TotalPages
}

func main() {

   var jsonPages Response

  pag, err := paginationaggregator.NewPaginationAggregator( 
    &paginationaggregator.PaginationAggregatorConfig{
      URL:"http://127.0.0.1:3000/api/v2/users?per_page=5&page=%d",
      Client: &http.Client{},
      JsonPage: &jsonPages,
      Concurrent: 5,
    },
  )

}

As you can see, the library provides Concurrent configuration. This configuration means if you need to make 100 requests for each page then there will be 20 batch asynchronous requests or each batch will make 5 asynchronous requests until all page fetched. I think it will prevent api server considers all these requests as DDOS attacks and prevents requests from hitting rate limiters.

Insert user to database

After successfully retrieving all user then I need to insert it into database only if user email does not exist. Rather than waiting for all user successfully retrieved, the library provides a function so we can process it in each batch.

type Meta struct {
   Page       int `json:"page"`
   PerPage    int `json:"per_page"`
   TotalPages int `json:"total_pages"`
}

type User struct {
   Uuid      string `json:"id" gorm:"column:id;primaryKey"`
   FirstName string `json:"firstname" gorm:"column:firstname"`
   LastName  string `json:"lastname" gorm:"column:lastname"`
   Email     string `json:"email" gorm:"column:email"`
}

type Response struct {
   Meta Meta   `json:"meta"`
   User []User `json:"data"`
}

func (resp *Response) GetBoundary() int {
   return resp.Meta.TotalPages
}

var Mysql *gorm.DB

func main() {

  var jsonPages Response

  Mysql = connectMysql()
  defer func() {
     sql, _ := Mysql.DB()
     sql.Close()
  }()

  pag, err := paginationaggregator.NewPaginationAggregator(
   &paginationaggregator.PaginationAggregatorConfig{
      URL:"http://127.0.0.1:3000/api/v2/users?per_page=5&page=%d",
      Client: &http.Client{},
      JsonPage: &jsonPages,
      Concurrent: 5,
      ConcurrentBatch: processingBatch,
     },
  )

  if err != nil {
     fmt.Println(err)
     return
  }

  res, err := pag.Get()

  if err != nil {
     fmt.Println(err)
     return
  }

  for _, val := range res {
     fmt.Println(val.Response.Data)
  }
}

// select and insert user for every batch
func processingBatch(
    batchResult []paginationaggregator.HttpInteraction
) error {

   var newUsers []User

   for _, val := range batchResult {

      resp := Response{}

      if err := json.Unmarshal([]byte(val.Response.Data), 
                   &resp); err != nil {
         continue
      }

      for _, user := range resp.User {

       userTmp := User{}

           // check existing user by email
           if err := Mysql.Where("email = ?", 
                user.Email).First(&userTmp).Error; err != nil {

              if errors.Is(err, gorm.ErrRecordNotFound) {
                       // users not exists so append it
                  newUsers = append(newUsers, user)
              }

             continue

          }
       }
    }

    if err := Mysql.Create(&newUsers).Error; err != nil {
         return err
    }

    return nil
}

func connectMysql() *gorm.DB {

    dsn := "root:yourpassword@tcp(127.0.0.1:3306)/peoples"

    DB, err := gorm.Open(mysql.Open(dsn), &gorm.Config{
        Logger: logger.Default.LogMode(logger.Silent),
    })

    if err != nil {
    panic(err)
    }

    return DB
}

Reference

• https://github.com/Mhakimamransyah/go-pagination-aggregate