Optimizing Data Pipelines for Fiix Dating App

Working for Fiix, formerly known as Jfiix, a mobile dating application, I took on the task of refining a crucial data pipeline. This pipeline served as the backbone of the app’s user engagement strategy. The pipeline analyzed user interactions—likes and messages—to generate match suggestions that would be delivered via push notifications. These suggestions aimed to reduce churn, boost active users, improve retention rates, increase conversion rates, and ultimately elevate the lifetime value (LTV) of the platform's user base.

Challenges Faced

The project presented two significant challenges:

1. Technical Challenge:
   The process required over 15 hours to complete daily and exhibited a concerning trend of increased processing time as the user base grew. Preparing for user base growth to support further scaling became crucial.

2. Business Challenge:
   Despite the intensive computations, the number of actual matches produced was limited, resulting in a poor return on investment (ROI). This inefficiency jeopardized the business goal of driving user re-engagement and retention.

Root Cause Analysis

At the core of the problem was the method used to identify potential matches. The logic involved a multi-step relationship analysis:

For example, if User A likes or messages User B, and User B then interacts with User C, this suggests User C might have interests similar to User A. This chain of interactions can help identify potential matches based on shared connections and interests. Building on this, if User C interacts with User D, there is a possibility that User D could also be a good match for User A.

The computational burden stemmed from a series of Cartesian product operations, which combine every row of one dataset with every row of another. This process exponentially increased the data volume being processed, leading to significant challenges in memory and computation. This led to excessive memory usage, data spills, high I/O operations, and intensive CPU demands.

The Solution: A Data-First Mindset

To address these challenges, I devised and implemented a new pipeline with a data-first approach, emphasizing efficiency at each stage. Here are the steps I took:

1. Database Optimization:

Configured the MySQL database for write optimization by fine-tuning database internals and optimizing the host server settings.

This ensured smoother data ingestion and retrieval processes.

1. Dedicated Interaction Tables:

Set up dedicated tables to store daily user interactions, isolating this data for streamlined processing.

1. Layered Processing Workflow:

Broke the pipeline into seven distinct processing steps. For example, the first step involved identifying unique user interactions and storing them in a dedicated temporary table. Subsequent steps layered additional insights, such as filtering for active users, mapping interaction chains, and prioritizing based on engagement metrics.

Each step added a new layer of processed data and wrote the results to temporary tables.

This layered approach reduced the need to hold large datasets in memory and allowed the database to perform efficient, incremental operations.

1. Indexing and Partitioning:

Leveraged indexing and partitioning to accelerate query performance and reduce I/O operations.

1. Incremental Data Processing:

Designed the pipeline to process only new data each day, minimizing redundant computations.

Results Achieved

The revamped pipeline delivered transformative results:

• Performance Improvement:

  • Reduced processing time from over 15 hours to under 2 hours.
  • Enabled the system to handle significantly larger datasets without resource bottlenecks.

• Increased Matches:

  • Boosted the number of matches produced by approximately 600%, as measured by the total count of successful user connections each day compared to the previous pipeline. This increase led to a noticeable improvement in user engagement, with more users returning to the app after receiving match suggestions.
  • Enhanced the relevance of match suggestions, leading to higher user satisfaction.

• Business Impact:

  • Achieved the primary goal of reducing churn and increasing user engagement.
  • Contributed to improved retention rates, higher conversion rates, and greater LTV.

Reflections

This project provided several key insights that were instrumental in its success:

1. Incremental and Modular Design: Breaking down complex problems into smaller, manageable steps was critical for achieving both efficiency and scalability.
2. Effective Database Optimization: Leveraging features like indexing, partitioning, and write optimization resulted in substantial performance improvements.
3. Understanding User Interaction Patterns: A deep analysis of user relationships and interactions was central to building an effective match suggestion system.

These insights highlight the value of adopting a data-first mindset and engineering solutions that align technical efficiency with business objectives. By embracing a structured and incremental approach, we were able to overcome significant challenges and deliver measurable value to the Fiix platform and its users.