DEV Community

Apache SeaTunnel
Apache SeaTunnel

Posted on

From JSON to JSONL: How Apache SeaTunnel Solves the HTTP Big Data Transfer Memory Dilemma

#json #http #apacheseatunnel #datascience

data-g1fb503c12_1280

Apache SeaTunnel seamlessly extracts data via HTTP endpoints. Typically, an HTTP server returns responses in standard JSON format. SeaTunnel parses this JSON data, extracts the relevant schema and field types, and routes the records downstream to target systems.

While standard JSON handling works well for moderate payloads, massive HTTP datasets create severe memory pressure for both clients and servers. How can we overcome this bottleneck? The answer lies in streaming with JSONL (JSON Lines)—processing and transmitting data line-by-line as it generates. This streaming approach eliminates the need to load entire payloads into memory at once, drastically reducing overhead. For instance, when querying one month of order records totaling 1 million rows from a database, data can be fetched and processed in 5-day incremental batches rather than holding all 1 million records in memory simultaneously.

1. Data Output Formats: JSON vs. JSONL

  • JSON Standard JSON responses returned via HTTP follow an array structure like this: 
[
  { "id": 1, "name": "a" },
  { "id": 2, "name": "b" }
]
Enter fullscreen mode Exit fullscreen mode

The entire payload is treated as a single unified JSON document, requiring the system to process the full payload in one go.

  • JSONL JSONL responses returned via HTTP follow a line-delimited format like this: 
{"id":1,"name":"a"}
{"id":2,"name":"b"}
Enter fullscreen mode Exit fullscreen mode

Instead of a single enclosing JSON object or array, each row represents an independent JSON entity, separated by newline characters.

2. Key Parameter for Format Identification

To differentiate between these response formats, configure the parsing behavior using the key parameter below:

  • Parameter: enable_multi_lines
  • Default Value: false
  • Description: Determines whether to enable multi-line JSON parsing. When set to true, SeaTunnel reads newline-delimited JSON entries (JSONL/NDJSON) sequentially without loading the full payload as a single object.
Parameter Value Description
enable_multi_lines true Read JSONL format
enable_multi_lines (Default) false Read JSON format

3. Practical Configuration Example

  • Step 1: Set Up an HTTP Test Service Returning Both Formats 
[root@localhost]# curl http://localhost:3000
[  {    "id": 1,    "name": "aaa"  },  {    "id": 2,    "name": "bbb"  }]

[root@localhost]# curl http://localhost:3000/jsonl
{"id":1,"name":"aaa"}
{"id":2,"name":"bbb"}
Enter fullscreen mode Exit fullscreen mode
  • Step 2: Configuration for Standard JSON 
env {
  parallelism = 1
  job.mode = "BATCH"
}

source {
  Http {
    plugin_output = "http"
    url = "http://localhost:3000"
    method = "GET"
    format = "json"
    enable_multi_lines = false
    schema = {
      fields {
        id = int
        name = string
      }
    }
  }
}

sink {
  Console {
    parallelism = 1
  }
}
Enter fullscreen mode Exit fullscreen mode

Console Output:

2026-07-04 18:13:09,073 INFO  [.a.s.c.s.c.s.ConsoleSinkWriter] [st-multi-table-sink-writer-1] - subtaskIndex=0  rowIndex=1:  SeaTunnelRow#tableId=Optional[http] SeaTunnelRow#kind=INSERT : 1, aaa
2026-07-04 18:13:09,073 INFO  [.a.s.c.s.c.s.ConsoleSinkWriter] [st-multi-table-sink-writer-1] - subtaskIndex=0  rowIndex=2:  SeaTunnelRow#tableId=Optional[http] SeaTunnelRow#kind=INSERT : 2, bbb
Enter fullscreen mode Exit fullscreen mode
  • Step 3: Configuration for JSONL Streaming 
source {
  Http {
    plugin_output = "http"
    url = "http://localhost:3000/jsonl"
    method = "GET"
    format = "json"
    enable_multi_lines = true
    schema = {
      fields {
        id = int
        name = string
      }
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

When enable_multi_lines is set to true, the execution output matches the JSON source output above. However, if enable_multi_lines remains false on a JSONL stream, SeaTunnel processes only the first row and drops all subsequent records.

4. Key Takeaway

For standard HTTP services returning single JSON string payloads, leave the default setting unchanged as false. For streaming JSONL outputs, always set enable_multi_lines = true to guarantee complete, memory-efficient data ingestion.

Top comments (0)