DEV-AI

Posted on May 10

Implementing a Scalable Filtering and Segmentation Panel for Large-Scale MongoDB Database

Modern applications managing millions of person records require sophisticated filtering and segmentation capabilities while maintaining performance and scalability. This technical guide presents a comprehensive architecture combining MongoDB, Spring Boot, React, and Elasticsearch to handle datasets exceeding 10 million records with sub-second response times. The solution leverages MongoDB's flexible document model, Spring Data's dynamic query capabilities, React's component-based UI, and Elasticsearch's full-text search optimization.

Database Architecture Design

Document Structure Optimization

The MongoDB document model combines static and dynamic fields using a hybrid schema approach:

{
  "_id": ObjectId("665f1d9b1a41c8523e48f791"),
  "identity": {
    "firstName": "Emily",
    "lastName": "Zhang",
    "dob": ISODate("1985-07-19T00:00:00Z"),
    "nationalId": "A234-5678-9012"
  },
  "demographics": {
    "gender": "F",
    "languagePrefs": ["en", "zh"],
    "accessibilityNeeds": ["visual"]
  },
  "dynamicAttributes": {
    "loyaltyTier": "Platinum",
    "preferredCategories": ["electronics", "books"]
  },
  "systemMetadata": {
    "dataSource": "CRMv2",
    "lastUpdated": ISODate("2024-06-01T08:45:32Z"),
    "version": 3
  }
}

Indexing Strategy

Implement a tiered indexing approach combining static and dynamic patterns:

Compound Indexes for Common Queries

   @CompoundIndex(def = "{'identity.lastName': 1, 'demographics.gender': 1}", name = "name_gender_idx")
   @CompoundIndex(def = "{'systemMetadata.dataSource': 1, 'systemMetadata.lastUpdated': -1}", name = "source_recency_idx")

Wildcard Indexing for Dynamic Fields

   db.persons.createIndex({"dynamicAttributes.$**": 1}, {wildcardProjection: {}})

Time-Series Optimization

   db.createCollection("person_activity", {
     timeseries: {
       timeField: "timestamp",
       metaField: "metadata",
       granularity: "hours"
     }
   })

Spring Boot Backend Implementation

Dynamic Query Construction

Implement a type-safe query builder supporting 25+ operators:

public class DynamicQueryBuilder {
  private final List criteriaChain = new ArrayList<>();

  public  DynamicQueryBuilder with(String field, Operator op, T value) {
    criteriaChain.add(buildCriteria(field, op, value));
    return this;
  }

  private  Criteria buildCriteria(String field, Operator op, T value) {
    return switch(op) {
      case EQ -> Criteria.where(field).is(value);
      case GT -> Criteria.where(field).gt(value);
      case TEXT -> Criteria.where(field).regex(value.toString(), "i");
      case GEO_WITHIN -> Criteria.where(field).within((Geometry) value);
      case JSON_PATH -> Criteria.where(field).exists(true)
                               .elemMatch(jsonPathQuery(value.toString()));
      // Handle 20+ additional operators
    };
  }
}

Pagination with Performance Guarantees

Implement cursor-based pagination for deep pagination scenarios:

public PaginatedResponse getPersons(CursorPageRequest request) {
  Query query = new Query().limit(request.getPageSize());

  if(request.getCursor() != null) {
    ObjectId lastId = new ObjectId(request.getCursor());
    query.addCriteria(Criteria.where("_id").gt(lastId));
  }

  List results = mongoTemplate.find(query, Person.class);
  String newCursor = results.isEmpty() ? null : results.get(results.size()-1).getId();

  return new PaginatedResponse<>(
    results, 
    newCursor,
    calculateRemaining(request.getCursor())
  );
}

React Frontend Architecture

Virtualized Filter Panel Component

Implement a performant filter UI handling 10,000+ filter options:

const FilterSelector = ({ field, options }) => {
  const [searchTerm, setSearchTerm] = useState('');
  const virtualizer = useVirtualizer({
    count: filteredOptions.length,
    estimateSize: () => 35,
    overscan: 5
  });

  return (

       setSearchTerm(e.target.value)}
      />


          {virtualizer.getVirtualItems().map(virtualItem => (



          ))}



  );
};

Real-Time Filter Synchronization

Implement debounced filter updates with query parameter management:

const useFilterManager = () => {
  const [filters, setFilters] = useState({});
  const navigate = useNavigate();
  const location = useLocation();

  useEffect(() => {
    const params = new URLSearchParams(location.search);
    const filterParams = Object.fromEntries(params.entries());
    setFilters(parseFilters(filterParams));
  }, [location]);

  const updateFilters = useDebounce((newFilters) => {
    const queryString = serializeFilters(newFilters);
    navigate(`?${queryString}`, { replace: true });
  }, 300);

  return [filters, updateFilters];
};

Elasticsearch Integration

Dual-Write Architecture

Implement event-driven synchronization using MongoDB Change Streams:

@Configuration
public class ChangeStreamConfig {

  @Autowired
  private MongoTemplate mongoTemplate;

  @Autowired
  private ElasticsearchOperations elasticOps;

  @PostConstruct
  public void init() {
    MongoCollection collection = mongoTemplate.getCollection("persons");

    List pipeline = List.of(
      Aggregates.match(
        Filters.in("operationType", 
          "insert", "update", "replace", "delete"
        )
      )
    );

    collection.watch(pipeline).forEach(this::processChange);
  }

  private void processChange(ChangeStreamDocument change) {
    IndexCoordinates index = IndexCoordinates.of("persons");

    switch(change.getOperationType()) {
      case INSERT:
      case UPDATE:
      case REPLACE:
        Document doc = change.getFullDocument();
        elasticOps.save(convertDocument(doc), index);
        break;

      case DELETE:
        String id = change.getDocumentKey().getObjectId("_id").getValue().toString();
        elasticOps.delete(id, index);
        break;
    }
  }
}

Hybrid Query Routing

Implement intelligent query routing between MongoDB and Elasticsearch:

public class QueryRouter {
  private static final Set ELASTIC_ONLY_OPERATORS = Set.of(
    "fulltext", "fuzzy", "geo_shape", "span_near"
  );

  public QueryRoute determineRoute(FilterCriteria criteria) {
    if (containsElasticOnlyFeatures(criteria)) {
      return QueryRoute.ELASTICSEARCH;
    }

    if (requiresAggregation(criteria)) {
      return criteria.getAggregationLevel() > 3 ? 
             QueryRoute.ELASTICSEARCH : 
             QueryRoute.MONGODB;
    }

    return QueryRoute.MONGODB;
  }

  private boolean containsElasticOnlyFeatures(FilterCriteria criteria) {
    return criteria.getFilters().stream()
      .anyMatch(f -> ELASTIC_ONLY_OPERATORS.contains(f.operator()));
  }
}

Performance Optimization

Query Plan Analysis

Implement automatic index hinting based on query patterns:

public class IndexAdvisor {
  private final QueryPlanCache planCache;

  public List recommendHints(Query query) {
    return planCache.analyze(query).stream()
      .filter(plan -> plan.getExecutionTime() > 100)
      .map(plan -> new IndexHint(plan.getSuggestedIndex()))
      .distinct()
      .limit(3)
      .toList();
  }
}

public class AdaptiveQueryInterceptor implements QueryInterceptor {
  @Override
  public Query intercept(Query query, Class type) {
    List hints = indexAdvisor.recommendHints(query);
    return query.withHint(hints.stream()
                    .map(IndexHint::getName)
                    .collect(Collectors.joining(",")));
  }
}

Caching Strategy

Implement multi-layer caching with TTL management:

public class CacheManager {
  private final Cache queryCache;
  private final Cache> filterCache;

  @Cacheable(value = "queryResults", keyGenerator = "queryKeyGenerator")
  public QueryResult executeCachedQuery(Query query) {
    return mongoTemplate.find(query, Person.class);
  }

  @CacheEvict(value = {"queryResults", "filterOptions"}, 
              allEntries = true)
  public void evictAllCaches() {
    // Scheduled to run hourly
  }

  @Scheduled(fixedRate = 3_600_000)
  public void warmCaches() {
    warmCommonQueries();
    warmFilterOptions();
  }
}

Security Considerations

Field-Level Encryption

Implement JSON Schema validation with automatic encryption:

db.runCommand({
  "collMod": "persons",
  "validator": {
    "$jsonSchema": {
      "bsonType": "object",
      "encryptMetadata": {
        "keyId": [UUID("123e4567-e89b-12d3-a456-426614174000")],
        "algorithm": "AEAD_AES_256_CBC_HMAC_SHA_512-Deterministic"
      },
      "properties": {
        "identity.nationalId": {
          "encrypt": {
            "bsonType": "string",
            "algorithm": "AEAD_AES_256_CBC_HMAC_SHA_512-Deterministic"
          }
        }
      }
    }
  }
})

Audit Logging

Implement granular audit trails with differential logging:

public class AuditAspect {
  @Around("execution(* com.example.repository.*.*(..))")
  public Object audit(ProceedingJoinPoint joinPoint) throws Throwable {
    AuditEntry entry = new AuditEntry()
      .setOperation(joinPoint.getSignature().getName())
      .setParameters(serializeParams(joinPoint.getArgs()))
      .setInitiatedBy(SecurityContext.getCurrentUser());

    try {
      Object result = joinPoint.proceed();
      entry.setOutcome("SUCCESS")
           .setResultSize(calculateSize(result));
      return result;
    } catch (Exception ex) {
      entry.setOutcome("FAILURE")
           .setError(ex.getMessage());
      throw ex;
    } finally {
      auditRepository.save(entry);
    }
  }
}

Monitoring and Maintenance

Performance Dashboard

Key metrics to monitor in Grafana:

Metric Group	Specific Metrics	Alert Threshold
Query Performance	95th percentile response time	> 1200ms
Index Utilization	Index hit ratio	0.85
Replication Health	Oplog window duration	5 minutes

Automated Index Management

Implement index lifecycle policies:

use admin

db.adminCommand({
  "setIndexCommitQuorum": "persons",
  "indexNames": ["name_gender_idx", "source_recency_idx"],
  "commitQuorum": {
    "mode": "majority",
    "wtimeout": 5000
  }
})

db.runCommand({
  "configureIndexAutoMerge": {
    "indexes": [
      {
        "name": "dynamic_attrs_idx",
        "minSegments": 10,
        "maxSegments": 20,
        "intervalHours": 6
      }
    ]
  }
})

Conclusion

This architecture provides a robust foundation for building enterprise-grade person management systems capable of handling datasets exceeding 100 million records. Key innovations include:

Hybrid MongoDB/Elasticsearch query routing
Adaptive indexing strategies with automated management
React virtualized filter components with zero-latency updates
Cryptographic field-level security with audit trails
Self-optimizing query pipelines

Implementation teams should conduct thorough load testing using representative datasets and progressively roll out features while monitoring the 15 key performance indicators outlined in the monitoring section. Future enhancements could incorporate machine learning-based query optimization and predictive caching based on usage patterns.