Partition once versus filter twice for bulk email validation

The decision

The partition function separates the array into the elements that satisfy the condition from those that do not docs. The filter function iterates over an array and applies an expression that returns matching values docs. Both scripts compute the task to split payload.records into accepted ids and rejected records with a reason, plus a retry count, by an email match.

First approach

The partition once approach uses the import line import * from dw::core::Arrays.

%dw 2.0
import * from dw::core::Arrays
output application/json
var split = payload.records partition (r) ->
    (r.email default "") matches /.+@.+\..+/
---
{
  accepted: split.success map (r) -> r.id,
  rejected: split.failure map (r) -> {
    id: r.id,
    reason: "missing or invalid email"
  },
  retryCount: sizeOf(split.failure)
}

Second approach

The filter twice approach requires no import.

%dw 2.0
output application/json
var valid = payload.records filter ((r) -> (r.email default "") matches /.+@.+\..+/)
var invalid = payload.records filter ((r) -> not ((r.email default "") matches /.+@.+\..+/))
---
{
  accepted: valid map (r) -> r.id,
  rejected: invalid map (r) -> {
    id: r.id,
    reason: "missing or invalid email"
  },
  retryCount: sizeOf(invalid)
}

Same input, same output

{
  "records": [
    { "id": "ORD-1001", "email": "[e-mail]" },
    { "id": "ORD-1002", "email": "bad-address" },
    { "id": "ORD-1003", "email": "[e-mail]" },
    { "id": "ORD-1004" },
    { "id": "ORD-1005", "email": "[e-mail]" }
  ]
}

Both scripts print the same output for this input.

{
  "accepted": [
    "ORD-1001",
    "ORD-1003",
    "ORD-1005"
  ],
  "rejected": [
    {
      "id": "ORD-1002",
      "reason": "missing or invalid email"
    },
    {
      "id": "ORD-1004",
      "reason": "missing or invalid email"
    }
  ],
  "retryCount": 2
}

Measured

Input	Records	Script	Runs	Min ms	Median ms	Max ms	Source
small	5	partition once	10	77	84	94	verified in sandbox
large	50000	partition once	10	378	383	388	verified in sandbox
small	5	filter twice	10	52	55.5	62	verified in sandbox
large	50000	filter twice	10	319	326.5	335	verified in sandbox
small	5	baseline	10	50	51.5	60	verified in sandbox
large	50000	baseline	10	143	147	154	verified in sandbox

Script	Input traversals	Imports	Lines	Source
partition once	1	import * from dw::core::Arrays	14	verified in sandbox
filter twice	2	none	13	verified in sandbox
baseline	1	none	4	verified in sandbox

At 5 records the timing ranges did not overlap and partition once measured 77 to 94 ms while filter twice measured 52 to 62 ms.
At 50000 records the timing ranges did not overlap and partition once measured 378 to 388 ms while filter twice measured 319 to 335 ms.

When I choose which

At 5 records the timing ranges do not overlap. At 50000 records the timing ranges do not overlap. I choose filter twice because it carried the lower median at both record counts and its two traversals avoid the import line. See how to partition bulk records into accepted and rejected lists Partition bulk records into accepted and rejected lists.

Runs shown: DataWeave CLI 2.12.2

104 DataWeave patterns + 8 Exchange modules with 208 MUnit tests: github.com/shakarbisetty/mulesoft-cookbook | 60-second walkthroughs: youtube.com/@SanThaParv

Comments

[Aman]

Great side-by-side comparison. Beyond just cleaner syntax, partition is a massive win in high-throughput Mule runtimes because it cuts iteration to $O(N)$ and avoids running the regex engine twice over the entire record set.

On enterprise payloads (say 500k+ records), have you benchmarked the heap memory delta between holding split.success and split.failure in memory versus streaming/chunking? Also, did you notice any regex compilation overhead differences in DataWeave when dealing with null/empty email defaults at scale?

Definitely recommending the partition pattern to our integration team.