DEV Community: farizmamad

How Software Engineering Enables National-Scale Product Distribution

farizmamad — Mon, 03 Feb 2025 03:01:02 +0000

Are you looking to expand your physical product distribution beyond local markets? As a technical leader who's helped a company scale from local to national distribution, I've learned that success lies in the intersection of technology and business operations.

Let me share how the right software engineering approach can transform your distribution capabilities:

The Business Challenge

Local success often creates an urgent need to expand. But scaling distribution isn't just about hiring more people or opening new warehouses. It requires systems that can:

Handle exponentially growing transaction volumes
Maintain speed and reliability across regions
Support real-time decision making
Ensure accurate tracking and auditing

Key Success Metrics

Before diving into solutions, let's focus on what matters:

Sales Team Efficiency

Application response time under 2 seconds for field sales
99.9% system availability
Automated visit scheduling and routing

Operation Reliability

Real-time inventory tracking
trackable order processing
Full audit capability

Customer Satisfaction

Instant order confirmation
Accurate delivery estimates
Transparent tracking

The Strategic Approach

success comes from aligning technology with business operations:

Infrastructure Strategy

Regional server deployment for optimal performance
Automated scaling to handle the high traffic during peak hours
Multi-platform accessibility like web or mobile app

Team Alignment

Share a clear and common objectives
Structured team according to job description
Iterative product delivery
Empowered and accountable teams

Technical Excellence

Optimized response times
Efficient database management
lightweight customer-facing application
Scalable architecture

The company see improvements in business process:

40% reduction in order processing time
2x increase in sales team productivity
Significant sales leader planning time reduction through scheduling automation
increased revenue

Here are a common framework for getting started

Assess your current systems
Identify bottlenecks
Prioritize improvements
Start with quick wins
Build for scale

What distribution challenges is your business facing? Let's discuss in the comments.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
If you're looking for technical leader who brings dedication to your business journey, let's connect. I'm open to discussing how my experience in building scalable systems and leading high-performing teams could help drive your business growth.

MongoDB Model.bulkWrite method solved my issue importing documents from hours to 5 minutes

farizmamad — Fri, 01 Mar 2024 01:07:03 +0000

Situation

I have a Node.js backend service using the NestJS framework that depends on an object modeling tool called Mongoose to connect to the MongoDB database. It serves to import data from CSV to the database via API endpoint. The CSV consists of >10k rows, each has multiple columns. The service has tasks to:

Convert the CSV data to an array of JavaScript objects.
Calculate the value of new fields for each object using some business logic.
Insert each object into the database OR update the existing document if the new field value matches.

Task

It was relatively easy to do task 1 by utilizing the csv-parse library to convert rows of columns of data into an array of JavaScript objects. The problem happened when doing calculations on each object and when inserting the object into the database. Calculations on each object mean it might query the database, do mathematic calculations, do string manipulation, etc. Hence, it resulted in different updates for each object. Furthermore, the database operation could be updateOne or create depending on the value of new field. Below is the pseudocode on the 1st version of the code.

Guess how is the performance? To process 1k of data it took 1 hour or more! If an error occurred, then it took longer. Imagine how long I need to wait to process 10k of data. It was such a wasted time.

Action and Result

That's why I explored the root cause of this problem, and I found out that updateOne or create commands on every looping consumes most of the time. Why? Because in updateOne or create, it did a round-trip communication to the database. Moreover, create needs time to index. Thus, n data = n round-trip where each round-trip may take 100 ms to 1000 ms.

It was where I found the solution: Model.bulkWrite. Instead of doing n round-trip communication to the database in every loop, we do it in batch. The idea was to only do calculations on each loop. After that, it only sends the insert or update operations at one call to do write operations in batch. Now the service can process 8k data only in under 5 minutes. Below is the code after being fixed.

Short about Model.bulkWrite

By definition from MongoDB, Model.bulkWrite or db.collection.bulkWrite is a method that

"Performs multiple write operations with controls for order of execution".

It receives write operations as input and returns counts for each write operation, and an array containing an _id for each inserted or upserted document.

In Mongoose, the details look like this.

# write operations:
- insertOne
- replaceOne
- updateOne
- updateMany
- deleteOne
- deleteMany
# options: some bulk write options.
# counts:
- insertedCount
- matchedCount
- modifiedCount
- deletedCount
- upsertedCount
# upsertedIds: array of ObjectIds

So I recommend using Model.bulkWrite in your application if you have a similar use case, where your application does write operations on lots of data to MongoDB with different operations on each document.

Tool for Integration Testing in NestJS-based Backend using Mongodb Replica Set

farizmamad — Mon, 15 Jan 2024 02:47:17 +0000

Backend services built on top of NestJS framework is able to utilize the test framework Jest. NestJS and Jest come up with solutions to develop unit tests and end-to-end tests. While both are necessary, they are out of scope of this document. Here we present a documentation for integration tests, which is where the tests are run against a service integrated with its dependencies, e.g. database, cloud features, etc.

Situation

Previously tests in my project were done manually by QA or Product Ops. For simple cases, this practice is enough. However, the project is growing and now has advance features for business process. Advance features are hard to be tested by detail in manual test, because it is exhausting and prone to human error. Furthermore, QA or Product Ops are handed with other daily tasks that takes time.

Another problem is coming from service dependencies which have advance features. For example, Mongo DB with its Node.js library called mongoose has a "Transaction" feature that is really important for data consistency. Me and my team have experienced data inconsistency in the mid of 2022. QA and Product Ops found the tests are passed in manual test. However, in production inconsistency is still exist.

Task

Manual test can't be used in advance engineering problem. So, test automation is a way to brake this hard issue. Problems coming from dependencies can't be tested in unit test, yet it is overkill to be tested in end-to-end test. That's why engineer offer a solution to create integration test.

Action

Engineer prepare requirements and steps to do integration test in backend services.

Requirements

Shell scripts containing mongodb replica set initiation placed in root with directory scripts/setup.sh. Here pre-defined script for up to 3 replica set with 1 primary.
Docker-compose file placed in root, docker-compose.yml. This file will create mongodb replica set containers and one db setup container.
Npm scripts for pretest, test, and posttest in package.json.
Create a file inside test directory, test/jest-integration.json, to enable jest identify test cases for integration test.

5. .env file placed at the root of the project containing the DB_URI_TEST

DB_URI_TEST=mongodb://localhost:27017/db-test?readPreference=primary&directConnection=true

Create test cases inside file name with pattern <any case>.integration.spec.ts in test directory. For example:

test/order.integration.spec.ts
test/product.integration.spec.ts

Make sure to connect to the primary host before all test cases.

Steps to run the tests

Open a terminal at root project directory.
Run test command, npm run test:integration. Note that we don't need to call pretest and posttest commands. NestJS will call pretest before the test, and call posttest after all test cases are passed.
Npm will first run pretest script.
Wait the tests until it stops.
The test is done.

Note:

if tests are passed, npm will run posttest script.
else, test will be stopped without running the posttest. It allows us to fix the bug and then rerun the test command.