DEV Community

jarred0214
jarred0214

Posted on Fully Autonomous

Extracting Column Lineage from Flink SQL Without Running a Cluster

Before dropping a source column, I want to know which downstream columns depend on it. Searching SQL files helps find direct references, but a temporary view can hide the connection between the source column and the final write.

I built LineSQL to extract those relationships from SQL scripts. This Flink example runs locally with Java and curl.

curl -fsSL \
  https://repo.maven.apache.org/maven2/io/github/jarredhj0214/linesql-cli/1.3.0/linesql-cli-1.3.0.jar \
  -o linesql.jar &&
java -jar linesql.jar --dialect FLINK "
  CREATE TEMPORARY VIEW active_users AS
  SELECT id FROM ods.users WHERE status = 'ACTIVE';

  INSERT INTO ads.user_summary (user_id)
  SELECT id FROM active_users;
"
Enter fullscreen mode Exit fullscreen mode

The final INSERT directly references active_users.id. By analyzing both statements together, LineSQL resolves that reference back to ods.users.id. The JSON output for the INSERT reports three kinds of dependencies.

What to check Result
Which tables are involved? ods.users → ads.user_summary
Where does user_id get its value? ods.users.id → ads.user_summary.user_id
What controls which rows are written? ods.users.status is used in WHERE

Those dependencies lead to different review steps. If you drop id, you need a replacement source for user_id. If you change what status = 'ACTIVE' means, check which rows reach the destination. Table lineage points to the same table in both cases.

The structured output can also give an AI assistant evidence for a review summary. Keep the parser diagnostics alongside it so reviewers can identify scripts that need manual inspection. For example, LineSQL needs schema metadata to expand SELECT *.

Supported SQL dialects: Spark SQL, Hive, Flink SQL, StarRocks, MySQL, Oracle, SQL Server, PostgreSQL, and OceanBase. Syntax and lineage coverage vary by dialect. When you know which engine runs the script, specify its dialect explicitly.

If you have a Flink SQL script that’s hard to trace, I’d like to see a minimal anonymized example and the lineage you expect. LineSQL on GitHub

Top comments (0)