DEV Community: Luis Dourado

The serverless challenge

Luis Dourado — Mon, 02 Sep 2024 23:37:40 +0000

Serverless computing, particularly platforms like AWS Lambda, has revolutionized the way we build and deploy applications. It promises scalability, cost-efficiency, and reduced operational overhead. However, as I discovered during a recent project migrating a product to a serverless architecture, the reality can be quite different from the promise. In this post, I'll share my experiences and the challenges I faced, hoping to provide insights for others considering or already working with serverless technologies.

The Allure of Serverless

Before diving into the challenges, it's worth remembering why serverless is so appealing:

Scalability: Automatic scaling based on demand.
Cost-efficiency: Pay only for what you use.
Reduced operational overhead: No server management required.
Rapid deployment: Quick updates and rollbacks.

These benefits are real and can be transformative for many applications. However, they come with their own set of challenges that aren't always apparent at first glance.

The Database Dilemma: Connections and Scaling

One of the first major hurdles I encountered was related to database connections. In a traditional server environment, you have a fixed number of servers with a predictable number of connections. In a serverless world, this predictability goes out the window.

Lambda functions can scale rapidly, creating many concurrent executions. Each of these executions potentially needs its own database connection. This can quickly lead to exhausting your database's connection limit, resulting in errors and degraded performance.

Moreover, the issue of cold starts compounds this problem. When a Lambda function hasn't been invoked for a while, AWS will spin up a new instance to handle the request. If you suddenly get a spike in traffic, you might have dozens or even hundreds of new Lambda instances all trying to establish database connections simultaneously. This can overwhelm your database, leading to timeouts and failures.

In our case, we initially saw frequent connection errors during traffic spikes. Our solution was to migrate to Aurora Serverless and use the RDS Data API. This allowed us to handle the connection pooling at the database level, rather than in our application code. While effective, it was a significant and unexpected change to our architecture.

We also implemented a connection pooling solution at the application level, using a library that maintains a pool of database connections that can be shared across Lambda invocations. This helped reduce the number of new connections being created, but it required careful tuning to balance performance and resource usage.

The Cost Conundrum: When Serverless Isn't Cheap

While serverless can be cost-effective, it's not always the case. The pay-per-use model is attractive, but it can also lead to surprising bills if you're not careful.

Long-running functions can become expensive quickly. In our project, we had a few functions that were processing large amounts of data, sometimes running for several minutes. These functions, while infrequent, were contributing significantly to our overall costs.

Cold starts not only affect performance but also impact cost. Each cold start means your function takes longer to execute, and you're paying for that execution time. If you have frequent cold starts across many functions, these costs can add up quickly.

Data transfer is another often-overlooked cost factor. Moving large amounts of data in and out of Lambda can be costly, especially if you're transferring data between regions or to external services.

Perhaps the most surprising cost factor was related to the overzealous scaling of our functions. During traffic spikes, our functions would scale up rapidly to handle the load. While this ensured we could handle the traffic, it also meant we were running (and paying for) many more function instances than we typically needed.

After a careful analysis of our usage patterns and costs, we found that for our specific use case and expected traffic patterns, a traditional ECS deployment might actually be more cost-effective than our serverless setup. This realization led us to adopt a hybrid approach, using serverless for some components and traditional deployments for others.

Performance Pitfalls: The Cold Start Terror

Performance in a serverless environment can be unpredictable, primarily due to cold starts. A cold start occurs when a Lambda function is invoked for the first time or after a period of inactivity. During a cold start, AWS needs to provision your function's container and bootstrap your runtime, which can introduce significant latency.

In our project, we experienced this firsthand with our homepage. The function responsible for rendering the homepage needed to process a large amount of data, sometimes taking up to 10 seconds to render on a cold start. This was a far cry from the snappy performance we were aiming for.

The impact of cold starts varies depending on the language and runtime you're using. We found that our Node.js functions generally had faster cold start times compared to our Python functions. However, even with Node.js, functions with many dependencies still suffered from noticeable cold start latency.

To mitigate this, we implemented several strategies:

Provisioned Concurrency: For our most critical functions, we used AWS Lambda Provisioned Concurrency to keep a certain number of function instances warm and ready to respond quickly.
Function Optimization: We worked on optimizing our functions, reducing dependencies where possible and lazy-loading modules that weren't always needed.
Intelligent Routing: We implemented a routing layer that would direct traffic to warm function instances when available, reducing the likelihood of users hitting a cold start.
Prewarming: For predictable traffic patterns, we implemented a prewarming strategy, invoking our functions periodically to keep them warm.

Despite these efforts, cold starts remained a challenge, particularly for less frequently accessed parts of our application, which of course, spiked our costs.

The Debugging Dilemma: Observability in a Serverless World

Debugging serverless applications introduced a whole new set of challenges. In a traditional application, you can often reproduce issues locally or SSH into a server to investigate. With serverless, each function execution is isolated, making it harder to trace issues across invocations.

We found that our existing logging and monitoring solutions weren't adequate for this new environment. While AWS provides CloudWatch for logs and metrics, setting up comprehensive logging and monitoring for a complex serverless application proved to be a significant undertaking.

We ended up implementing a distributed tracing solution using AWS X-Ray, which allowed us to trace requests as they moved through our various Lambda functions and other AWS services. This was crucial for understanding performance bottlenecks and errors that spanned multiple function invocations.

Another challenge was reproducing issues locally. The ephemeral nature of Lambda environments can make it difficult to replicate the exact conditions under which an error occurred. We invested time in developing a robust local development environment using tools like LocalStack and the AWS SAM CLI, which allowed us to run and debug our functions locally.

We also implemented extensive error handling and logging within our functions. Each function was designed to catch and log errors comprehensively, including relevant context information. This was crucial for post-mortem analysis of issues that only occurred in the production environment.

Architectural Avalanche: Complexity in Simplicity

While serverless architectures can simplify some aspects of application design, they often introduce complexity in other areas. Breaking down an application into small, single-purpose functions sounds great in theory, but it can lead to a sprawling, hard-to-manage architecture if not done carefully.

In our project, we initially went too far in breaking down our application into tiny functions. While this gave us fine-grained control and scaling, it also made it harder to understand the overall flow of our application. We found ourselves dealing with complex chains of function invocations, where an error in one function could have cascading effects that were hard to trace.

State management became a significant challenge. With stateless functions, we needed to rethink how we handled application state. We ended up relying heavily on external services like DynamoDB for state management, which added another layer of complexity to our application.

The reliance on other AWS services also increased the overall complexity of our system. Our application ended up being a complex choreography of Lambda functions, API Gateway, DynamoDB, S3, and several other AWS services. While each service was powerful and well-documented, the interactions between them created a complex system that was challenging to reason about and troubleshoot.

To manage this complexity, we implemented several strategies:

Service Aggregation: We started grouping related functions into larger services, deploying them together and managing them as a unit.
API Gateway Service Proxies: Instead of having API Gateway invoke Lambda functions directly, we used service proxies to route requests to internal APIs hosted on ECS. This gave us more flexibility in our internal architecture while maintaining the serverless facade.
Event-Driven Architecture: We made extensive use of AWS EventBridge to implement an event-driven architecture, which helped decouple our services and made the overall system more resilient and scalable.
Infrastructure as Code: We used AWS CDK to define our entire infrastructure as code, which helped us manage the complexity and ensure consistency across environments.

Limitations and Constraints: Working Within Boundaries

As we dove deeper into serverless development, we encountered various limitations and constraints that required creative solutions:

Execution Time Limits: AWS Lambda has a maximum execution time of 15 minutes. While this is sufficient for most use cases, we had a few long-running processes that exceeded this limit. We had to redesign these processes, breaking them into smaller steps that could be chained together using Step Functions.

Payload Size Limits: Lambda functions have limits on the size of incoming and outgoing payloads. We ran into this limit when trying to process large files. Our solution was to use S3 for file storage and pass S3 references between functions instead of the actual file content.

VPC Cold Starts: When we needed to access resources in a VPC, we discovered that the cold start times were significantly longer. We mitigated this by using Provisioned Concurrency for these functions and optimizing our VPC configuration.

Limited Local Storage: The ephemeral nature of Lambda's file system meant we couldn't rely on local file storage. For scenarios where we needed to process large amounts of data, we used EFS (Elastic File System) integrated with Lambda, which provided a persistent file system accessible across function invocations.

Dependency Management: Managing dependencies in a serverless environment proved challenging, especially for larger projects. We invested time in optimizing our deployment packages, using techniques like layer sharing and careful dependency management to reduce package sizes and improve cold start times.

Strategies for Success

Despite these challenges, serverless can still be a powerful tool when used appropriately. Here are some strategies we've found helpful:

Careful Planning: Thoroughly analyze your application's needs and traffic patterns before committing to serverless. Consider factors like execution time, memory usage, and I/O patterns.
Hybrid Approaches: Don't be afraid to mix serverless with traditional deployments. We found that using serverless for API layers and event-driven processes, while keeping data-intensive operations on EC2 or ECS, gave us the best of both worlds.
Optimization Focus: Continuously work on optimizing your functions. This includes code optimization, reducing dependencies, and efficient use of memory.
Caching Strategies: Implement robust caching at various levels - in your functions, at the API Gateway level, and in your data layer - to reduce latency and database load.
Comprehensive Monitoring: Invest in a robust observability solution. This includes logging, metrics, tracing, and alerting. Tools like AWS X-Ray, CloudWatch Logs Insights, and third-party observability platforms can be invaluable.
Continuous Learning: The serverless ecosystem is rapidly evolving. Stay updated with best practices, new features, and community insights. Attend conferences, participate in online forums, and experiment with new services and features.
Cost Monitoring and Optimization: Implement detailed cost allocation tags and regularly review your AWS bill. Use tools like AWS Cost Explorer and set up billing alarms to avoid surprise costs.
Developer Experience: Invest in tools and processes that improve the local development experience. This includes local Lambda runners, test frameworks, and CI/CD pipelines optimized for serverless deployments.

Conclusion

Serverless computing offers exciting possibilities, but it's not a silver bullet. It comes with its own set of challenges that require careful consideration and planning. In our project, we ultimately decided on a hybrid approach, using serverless for lighter workloads and traditional deployments for more demanding components.

The key takeaway is to approach serverless with a clear understanding of its strengths and limitations. It's a powerful tool, but like any technology, it needs to be applied judiciously and with a thorough understanding of your specific use case.

Remember, the best architecture is the one that solves your problem effectively, whether that's serverless, traditional, or a mix of both. Don't be afraid to experiment, but also be ready to adapt your approach based on real-world performance and costs.

Serverless has fundamentally changed how we think about building and scaling applications. While it presents challenges, it also offers unprecedented flexibility and scalability. By understanding and preparing for these challenges, we can harness the full power of serverless architectures to build robust, scalable, and cost-effective applications.

Development Principles

Luis Dourado — Mon, 02 Sep 2024 23:35:58 +0000

Programming is a journey of continuous improvement. As we gain experience, we encounter practices and principles that help us refine our craft, leading to higher-quality code. This article will guide you through key principles and practices that can help you become a better programmer, including SOLID principles, design patterns, and coding standards. We'll explore how these concepts can improve your development process.

A note on the examples: I've chosen Go for the code examples due to its simplicity and readability – it's often described as "executable pseudocode." If you're not familiar with Go, don't worry! This is an excellent opportunity to learn.

As you encounter unfamiliar syntax or concepts, take a moment to look them up. This process of discovery can be a fun bonus to your learning experience. Remember, the principles we're discussing apply across programming languages, so focus on understanding the concepts rather than the specifics of Go syntax.

1. Embrace Programming Standards

One of the first steps to becoming a better programmer is to adhere to programming standards. Standards provide guidelines for aspects like naming conventions, code organization, and file structure. By following these conventions, you ensure that your code is consistent and easy to read, which is crucial for collaboration and long-term maintenance.

Each programming language typically has its own set of conventions. For Go, these are outlined in the official Go style guide. It's important to learn and embrace the standards relevant to your context, whether it's your team's conventions or language-specific guidelines.

Let's look at an example of how following standards can improve code readability:

// Before: Inconsistent styling
func dosomething(x int,y string)string{
if x>10{
return y+"is big"
}else{
return y+"is small"
}}

This code has several issues:

Inconsistent spacing and indentation make it difficult to understand the code's structure.
The function name doesn't follow Go's camelCase convention.
The unnecessary else clause adds cognitive complexity.

Now, let's refactor this code to follow Go standards:

// After: Following Go standards
func doSomething(x int, y string) string {
    if x > 10 {
        return y + " is big"
    }
    return y + " is small"
}

In this improved version:

Proper indentation clearly shows the code's structure.
The function name follows Go's naming convention.
The else clause is removed, simplifying the logic.

These changes aren't just about aesthetics; they significantly improve code readability and maintainability. When working in a team, consistently applying these standards makes it easier for everyone to understand and work with the codebase.

Even if your team doesn't have established standards, you can take the initiative to follow widely-accepted conventions in the programming community. Over time, this practice will lead to more readable, maintainable code across your projects.

2. Follow Design Principles

Programming design principles are guidelines that help you write better code. These principles can be applied not only to code architecture but also to system design and even some aspects of development processes.
There are many design principles, and some are more relevant in specific contexts. Others are more general like KISS (Keep It Simple, Stupid) or YAGNI (You Ain't Gonna Need It).
Among these general principles, the SOLID principles are some of the most impactful. Let's explore each principle with a focus on how they can improve your code.

Single Responsibility Principle (SRP)

This principle encourages you to design components (functions, classes, or modules) with a single, well-defined purpose. When a component has multiple responsibilities, it becomes harder to understand, test, and maintain.

Let's look at an example of refactoring a function to adhere to SRP:

// Before: A function doing too much
func processOrder(order Order) error {
    // Validate order
    if order.Total <= 0 {
        return errors.New("invalid order total")
    }
    // Save to database
    db.Save(order)
    // Send confirmation email
    sendEmail(order.CustomerEmail, "Order Confirmation", orderDetails(order))
    // Update inventory
    for _, item := range order.Items {
        updateInventory(item.ID, item.Quantity)
    }
    return nil
}

This function is doing multiple unrelated tasks: validating the order, saving it to the database, sending an email, and updating inventory. Let's break it down into separate functions, each with a single responsibility:

// After: Breaking it down into single responsibilities
func processOrder(order Order) error {
    if err := validateOrder(order); err != nil {
        return err
    }
    if err := saveOrder(order); err != nil {
        return err
    }
    if err := sendOrderConfirmation(order); err != nil {
        return err
    }
    return updateInventoryForOrder(order)
}

Now, let's implement each of these functions:

func validateOrder(order Order) error {
    if order.Total <= 0 {
        return errors.New("invalid order total")
    }
    return nil
}

func saveOrder(order Order) error {
    return db.Save(order)
}

func sendOrderConfirmation(order Order) error {
    return sendEmail(order.CustomerEmail, "Order Confirmation", orderDetails(order))
}

func updateInventoryForOrder(order Order) error {
    for _, item := range order.Items {
        if err := updateInventory(item.ID, item.Quantity); err != nil {
            return err
        }
    }
    return nil
}

In this refactored version:

Each function has a single, clear responsibility.
The code is more modular and easier to test.
Individual components can be reused or modified without affecting others.

Trust me, your future self will thank you for this level of organization.

Open-Closed Principle (OCP)

The Open-Closed Principle advises that software entities should be open for extension but closed for modification. This means you should be able to add new functionality without changing existing code.

Liskov Substitution Principle (LSP)

LSP states that objects of a superclass should be replaceable with objects of its subclasses without affecting the correctness of the program. This ensures that inheritance hierarchies are well-designed and maintainable.

Interface Segregation Principle (ISP)

ISP suggests that no code should be forced to depend on methods it does not use. In practice, this often means creating smaller, more focused interfaces.This modularity makes your code easier to manage and test.

Here's an example demonstrating ISP:

// Before: A large interface that many structs only partially implement
type Worker interface {
    DoWork()
    TakeBreak()
    GetPaid()
    FileTicket()
}

This large interface forces implementers to define methods they might not need. Let's break it down into smaller, more focused interfaces:

// After: Smaller, more focused interfaces
type Worker interface {
    DoWork()
}

type BreakTaker interface {
    TakeBreak()
}

type Payable interface {
    GetPaid()
}

type TicketFiler interface {
    FileTicket()
}

Now, structs can implement only the interfaces they need:

type Developer struct{}

func (d Developer) DoWork() {
    fmt.Println("Writing code")
}

func (d Developer) TakeBreak() {
    fmt.Println("Browsing Reddit")
}

func (d Developer) FileTicket() {
    fmt.Println("Creating a Jira ticket")
}

type Contractor struct{}

func (c Contractor) DoWork() {
    fmt.Println("Completing assigned task")
}

func (c Contractor) GetPaid() {
    fmt.Println("Invoicing for work done")
}

This modularity makes your code easier to manage and test. In Go, interfaces are implicitly implemented, which makes this principle particularly easy to apply.

Dependency Inversion Principle (DIP)

DIP promotes the use of abstractions rather than concrete implementations. By depending on interfaces or abstract classes, you decouple your code, making it more flexible and easier to maintain. This also facilitates easier testing by allowing mock implementations.

Applying the SOLID (see what I did there?) principles leads to decoupled, modular code that is easier to maintain, scale, reuse, and test. I’ve found that these principles, while sometimes challenging to apply at first, have consistently led to more robust and flexible codebases.

While these principles are valuable, remember that they are guidelines, not strict rules.There will always be exceptions to the rules, but it’s important to remember that following these principles is a continuous process and not a one-time event. It will take time and effort to develop good habits, but the rewards are well worth it.

3. Utilize Design Patterns

Design patterns provide reusable solutions to common programming problems. They are not rigid implementations but rather templates that can be adapted to fit specific needs. Many design patterns are related to SOLID principles, often aiming to uphold one or more of these principles in their design.

Design patterns are typically categorized into three types:

Creational Patterns

These patterns deal with object creation mechanisms. An example is the Factory Method pattern, which creates objects based on a set of criteria while abstracting the instantiation logic.

Let's look at a simple Factory Method example in Go:

type PaymentMethod interface {
    Pay(amount float64) string
}

type CashPayment struct{}

func (c CashPayment) Pay(amount float64) string {
    return fmt.Sprintf("Paid %.2f using cash", amount)
}

type CreditCardPayment struct{}

func (cc CreditCardPayment) Pay(amount float64) string {
    return fmt.Sprintf("Paid %.2f using credit card", amount)
}

Here, we define a PaymentMethod interface and two concrete implementations. Now, let's create a factory function:

func GetPaymentMethod(method string) (PaymentMethod, error) {
    switch method {
    case "cash":
        return CashPayment{}, nil
    case "credit":
        return CreditCardPayment{}, nil
    default:
        return nil, fmt.Errorf("Payment method %s not supported", method)
    }
}

This factory function creates the appropriate payment method based on the input string. Here's how you might use it:

method, err := GetPaymentMethod("cash")
if err != nil {
    fmt.Println(err)
    return
}
fmt.Println(method.Pay(42.42))

This pattern allows for easy extension of payment methods without modifying existing code.

Structural Patterns

Structural patterns deal with object composition, promoting better interaction between classes. The Adapter pattern, for example, allows incompatible interfaces to work together.

Behavioral Patterns

Behavioral patterns focus on communication between objects. The Observer pattern is a common behavioral pattern that facilitates a publish-subscribe model, enabling objects to react to events.

It's important to note that there are many more design patterns, and some are more relevant in specific contexts. For example, game development might heavily use the Object Pool pattern, while it's less common in web development.

Design patterns help solve recurring problems and create a universal vocabulary among developers. However, don't feel pressured to learn all patterns at once. Instead, familiarize yourself with the concepts, and when facing a new problem, consider reviewing relevant patterns that might offer a solution. Over time, you'll naturally incorporate these patterns into your design process.

4. Practice Good Naming Conventions

Clear naming conventions are crucial for writing readable and maintainable code. This practice is closely related to programming standards and deserves special attention.

Use Descriptive Names

Choose names that clearly describe the purpose of the variable, function, or class. Avoid unnecessary encodings or cryptic abbreviations.

Consider this poorly named function:

// Bad
func calc(a, b int) int {
    return a + b
}

Now, let's improve it with more descriptive names:

// Good
func calculateSum(firstNumber, secondNumber int) int {
    return firstNumber + secondNumber
}

However, be cautious not to go to the extreme with overly long names:

// Too verbose
func calculateSumOfTwoIntegersAndReturnTheResult(firstInteger, secondInteger int) int {
    return firstInteger + secondInteger
}

Balance Clarity and Conciseness

Aim for names that are clear but not overly verbose. Good naming practices make your code self-documenting, reducing the need for excessive comments.

Prefer Good Names to Comments

Often, the need for comments arises from poorly named elements in your code. If you find yourself writing a comment to explain what a piece of code does, consider whether you could rename variables or functions to make the code self-explanatory.

Replace Magic Values with Named Constants

Using named constants instead of hard-coded values clarifies their meaning and helps keep your code consistent.

// Bad
if user.Age >= 18 {
    // Allow access
}

// Good
const LegalAge = 18
if user.Age >= LegalAge {
    // Allow access
}

By following these naming conventions, you'll create code that's easier to read, understand, and maintain.

5. Prioritize Testing

Testing is an essential practice for ensuring that your code behaves as expected. While there are many established opinions on testing methodologies, it's okay to develop an approach that works best for you and your team.

Unit Testing

Unit tests focus on individual modules in isolation. They provide quick feedback on whether specific parts of your code are functioning correctly.

Here's a simple example of a unit test in Go:

func TestCalculateSum(t *testing.T) {
    result := calculateSum(3, 4)
    expected := 7
    if result != expected {
        t.Errorf("calculateSum(3, 4) = %d; want %d", result, expected)
    }
}

Integration Testing

Integration tests examine how different modules work together. They help identify issues that might arise from interactions between various parts of the code.

End-to-End Testing

End-to-end tests simulate user interactions with the entire application. They validate that the system works as a whole, providing a user-centric view of functionality.

Remember, well-tested code is not only more reliable but also easier to refactor and maintain over time. The SOLID principles we discussed earlier can make testing easier by encouraging modular, decoupled code that is simpler to isolate and validate.

6. Take Your Time to Plan and Execute

While it might be tempting to rush through projects, especially when deadlines are tight, thoughtful planning is crucial for long-term success. Rushing can lead to technical debt and future maintenance challenges.

Take the time to carefully consider architectural decisions and plan your approach. Building a solid foundation early on will save time and effort in the long run. However, be cautious not to over-plan—find a balance that works for your project's needs.

Different projects may require varying levels of planning. A small prototype might need minimal upfront design, while a large, complex system could benefit from more extensive planning. The key is to be flexible and adjust your approach based on the project's requirements and constraints.

Conclusion

By following these principles and practices, you can become a better programmer. Focus on writing code that is clear, maintainable, and thoughtfully structured. Remember the words of Martin Fowler: "Any fool can write code that a computer can understand. Good programmers write code that humans can understand."

Becoming a better programmer is a continuous process. Don't be discouraged if you don't get everything right immediately. Keep learning, keep practicing, and most importantly, keep coding. With time and dedication, you'll see significant improvements in your skills and the quality of your work.

Here are some resources you might find helpful for further learning:

For general programming principles: Programming Principles
For understanding the importance of basics: The Basics
For a deep dive into software design: The Philosophy of Software Design and its discussion
For practical programming advice: The Pragmatic Programmer
For improving existing code: Refactoring

Now, armed with these principles, start applying them to your projects. Consider creating a small web API that goes beyond simple CRUD operations, or develop a tool that solves a problem you face in your daily work. Remember, the best way to learn is by doing. Happy coding!

Combining APIs Responses in Node

Luis Dourado — Wed, 29 Mar 2023 21:50:41 +0000

Introduction

Have you ever encountered an API endpoint that doesn't provide all the information you need in a single response? This can be frustrating, especially when you're developing a project that requires data from multiple sources. In such cases, you might find yourself having to make multiple queries to the same endpoint and combine the responses in your frontend code.

In this tutorial, we'll explore how you can create a simple Node.js server to process API responses and send the combined data back to your client in a single query. This technique can also be applied in the client-side code.

Prerequisites

Before we begin, make sure that you have the following tools installed on your system:

Node.js
npm (Node.js Package Manager)

Getting Started

To get started, let's create a new Node.js project. Open up your terminal and navigate to the directory where you want to create your project. Once there, run the following command:

npm init -y

This will initialize a new Node.js project with default settings. Next, let's install some dependencies that we'll need for our project:

npm install express axios

Express: A web framework for Node.js.
Axios: A promise-based HTTP client for making API requests.

With these dependencies installed, let's create a new file called index.js in the root directory of our project.

In this file, we'll first import the necessary modules and set up our server:

const express = require('express');
const app = express();
const port = 8080;

This sets up a basic server that listens for requests on port 8080. You can choose any other available port if you prefer.

Next, let's create some variables to store the API endpoints we'll be querying:

const postUrl = 'https://jsonplaceholder.typicode.com/posts';
const userUrl = 'https://jsonplaceholder.typicode.com/users';

We'll be using the JSONPlaceholder API for this tutorial, which provides a set of mock endpoints for testing and prototyping.

Now, let's use the axios module to make requests to these endpoints and store the responses in files:

const fs = require('fs/promises');
const axios = require('axios');

const fetchPostsAndUsers = async () => {
  try {
    const [postsResponse, usersResponse] = await Promise.all([
      axios.get(postUrl),
      axios.get(userUrl),
    ]);

    await fs.writeFile('posts.json', JSON.stringify(postsResponse.data));
    await fs.writeFile('users.json', JSON.stringify(usersResponse.data));
  } catch (error) {
    console.error(error);
  }
};

This code fetches data from the two API endpoints, stores the responses in files called posts.json and users.json, and converts the JSON data into string format using JSON.stringify().

Now, let's read the data from these files and combine it into a single response:

const getUsersPosts = async () => {
  try {
    const [postsData, usersData] = await Promise.all([
      fs.readFile('posts.json', 'utf8'),
      fs.readFile('users.json', 'utf8'),
    ]);

    const posts = JSON.parse(postsData);
    const users = JSON.parse(usersData);

    const usersPosts = posts.map((post) => {
      const user = users.find((user) => user.id === post.userId).name;
      return {
        ...post,
        user,
      };
    });

    return usersPosts;
  } catch (error) {
    console.error(error);
    return [];
  }
};

This code reads the data from the two files using fs.readFileSync(), parses the JSON data using JSON.parse(), and combines the data into a single response using the map() function.

This is where you can choose what data will be returned in the new combined response, in this example we are adding only the name property into the posts response.

Finally, let's create a new endpoint on our server that returns the combined data and call the functions:

 fetchPostsAndUsers()
 app.get('/posts', (req, res) => {
  const usersPosts = await getUsersPosts();
    res.send(usersPosts);
  });

This is the final code:

const express = require('express');
const app = express();
const port = 8080;
const axios = require('axios');
const fs = require('fs/promises');

const postUrl = 'https://jsonplaceholder.typicode.com/posts/';
const userUrl = 'https://jsonplaceholder.typicode.com/users/';

const fetchPostsAndUsers = async () => {
  try {
    const [postsResponse, usersResponse] = await Promise.all([
      axios.get(postUrl),
      axios.get(userUrl),
    ]);

    await fs.writeFile('posts.json', JSON.stringify(postsResponse.data));
    await fs.writeFile('users.json', JSON.stringify(usersResponse.data));
  } catch (error) {
    console.error(error);
  }
};



const getUsersPosts = async () => {
  try {
    const [postsData, usersData] = await Promise.all([
      fs.readFile('posts.json', 'utf8'),
      fs.readFile('users.json', 'utf8'),
    ]);

    const posts = JSON.parse(postsData);
    const users = JSON.parse(usersData);

    const usersPosts = posts.map((post) => {
      const user = users.find((user) => user.id === post.userId).name;
      return {
        ...post,
        user,
      };
    });

    return usersPosts;
  } catch (error) {
    console.error(error);
    return [];
  }
};

fetchPostsAndUsers()

app.get('/posts', async (req, res) => {
  const usersPosts = await getUsersPosts();
  res.send(usersPosts);
});

app.listen(port, () => {
  console.log(`Example app listening at http://localhost:${port}`);
});

In the above code, we have created a new endpoint /posts that returns an array of posts with the user field added. This is achieved by reading the data from the posts.json and users.json files and then processing the data to combine the necessary fields.

To test our endpoint, we can use a tool like Postman or just open the URL http://localhost:8080/posts in a browser. You should see a list of posts with the user field added, like this:

[
  {
    "userId": 1,
    "id": 1,
    "title": "sunt aut facere repellat provident occaecati excepturi optio reprehenderit",
    "body": "quia et suscipit\nsuscipit... ",
    "user": "Leanne Graham"
  },
  {
    "userId": 1,
    "id": 2,
    "title": "qui est esse",
    "body": "est rerum tempore vitae\nsequi sint... ",
    "user": "Leanne Graham"
  },
]

And that's it! We have successfully created a simple server that processes data from two API endpoints and returns the combined data in a single query.

This approach can be used in a variety of scenarios where you need to combine data from multiple API endpoints, and it can also be extended to handle more complex logic and data processing.

I hope you found this tutorial useful, and feel free to leave any questions or comments in the comments section below!