SeaTunnel Community Monthly Report For September

Hi community, the SeaTunnel Community's September Monthly Report is here! This report highlights the major developments in the SeaTunnel community over the past month. Stay tuned for updates!

SeaTunnel Monthly Merge Stars

A big thank you to the following contributors for their outstanding contributions to Apache SeaTunnel last month (in no particular order):

@hawk9821,@zuhdan,@liugddx,@Hisoka-X,@chl-wxp,@dailai,@arshadmohammad,@hailin0,@loustler,@Carl-Zhou-CN,@jiamin13579,@corgy-w,@wuchunfu,@sunxiaojian,@EricJoy2048,@liunaijie,@BilwaST,@CosmosNi,@Cheun99,@ZhangWeike2000,@PeppaPage,@zhangshenghang,@happyboy1024,@FuYouJ

apache/seatunnel

fix

• [hotfix]fix docker image build script @liunaijie
• [Fix]fix docker image publish setting @liunaijie
• [Fix][Zeta]If Zeta not a TCP discovery, it cannot find other members @loustler
• [Fix][CI]Fix CI loss document check when raise PR @Hisoka-X
• [Hotfix][ci]Fix github ci License header error @hawk9821
• [Fix][Connector-V2]Fix iceberg throw java: package sun.security.krb5 does not exist when use jdk 11 @sunxiaojian
• [Hotfix][Connector-V2]Release resources when task is closed for iceberg sinkwriter @dailai
• [Fix][Connecotr-V2]Fix paimon dynamic bucket tale in primary key is not first @hawk9821
• [Hotfix][Connector-V2]Release resources even the task is crashed for paimon sink @dailai
• [Fix][Connector-V2]Fix paimon e2e error @hawk9821
• [Fix][CI]Add doc sync to fix ci error @zhangshenghang
• [Fix][Connector-V2]Fix http source can not read streaming @CosmosNi
• [Fix][Connector-V2]Fix known directory create and delete ignore issues @corgy-w
• [Fix][Connector-V2]Fix jdbc test case failed @Hisoka-X
• [Hotfix][Core]Fix concurrency exceptions when MultiTableSink#PrepareCommit @hailin0
• [Fix][Connector-V2]Fixed iceberg sink can not handle uppercase fields @sunxiaojian
• [Fix][Connector-V2]Fix some throwable error not be caught @Hisoka-X
• [Fix][Connector-v2]Throw Exception in sql query for JdbcCatalog in table or db exists query @dailai
• [Hotfix][CDC]Fix ddl duplicate execution error when config multi_table_sink_replica @hailin0
• [Fix]Fix oracle query table 'char' 'varchar' length @chl-wxp
• [Fix][Transforms-V2]Fix LLM transform can not parse boolean value type @hawk9821
• [Hotfix]Fix iceberg missing column comment when savemode create table @hailin0
• [Fix][Document]Fix typos in PostgreSQL-CDC doc @zuhdan
• [Fix][Common]Fix the CommonError msg for paimon sink @dailai
• [Hotfix][Connector-v2]Fix the ClassCastException for connector-mongodb @dailai
• [Fix][JDBC]Fix starrocks jdbc dialect catalog conflict with starrocks connector @liunaijie
• [Fix][Test]Fix FixSlotResourceTest unstable issue @liunaijie
• [Fix][E2E]remote loading driver ignores the certificate @hawk9821
• [Fix][Document]Fix document build error @Hisoka-X
• [Hotfix][Connector-v2]Fix the NullPointerException for jdbc oracle which used the table_list @dailai

improve

• [Improve][E2E]update doris image to official version @hawk9821
• [Improve][Spark]Convert array type to exact type @corgy-w
• [Improve][Connector-v2]Remove useless code and add changelog doc for paimon sink @dailai
• [Improve][doc]Improve startrocks sink doc @jiamin13579
• [Improve][Doc]Add logging into sidebars.js @Hisoka-X
• [Improve][Docs]Add tuning guide for doris sink in streaming mode @dailai
• [Improve][Doc]Improve engine deployment doc for skip deploy client @Hisoka-X
• [Improve][Doc]Spark use doc update @Carl-Zhou-CN
• [Improve][CI]Move paimon into single task @Hisoka-X
• [Improve][Transform]Improve inner sql query for map field type @liunaijie
• [Improve][Zeta]Add log for tryTriggerPendingCheckpoint because the wrong time of server @dailai
• [Improve][Connector-V2]Optimize sqlserver package structure @corgy-w
• [Feature][Flink]Support multiple tables read and write @PeppaPage
• [Feature][Zeta][Core]Support output log file of job @hailin0
• [Improve][Doc]Add some connector icons @zhangshenghang
• [Improve][Core]Config variables update doc and add test case @zhangshenghang
• [Feature][Connector-V2]SqlServer support user-defined type @Hisoka-X
• [Improve][doc]Fixed hive sink doc @jiamin13579
• [Improve][Connector-V2]Optimize milvus code @corgy-w
• [Feature][Connector-V2]Assert support multi-table check @Hisoka-X
• [Feature][Connector-v2]Support streaming read for paimon @dailai
• [Feature][Connector-V2]jdbc saphana source tablepath support view and synonym @chl-wxp
• [Feature][Connector-V2]sftp file sink suport multiple table and save mode @chl-wxp
• [improve][Redis]Redis scan command supports versions 5, 6, 7 @FuYouJ
• [Feature][Connector-V2]Ftp file sink suport multiple table and save mode @chl-wxp
• [Feature][Connector-V2]Optimize hudi sink @happyboy1024
• [Improve] [Connector-V2]Optimize milvus-connector config code @corgy-w
• [Improve][Jdbc]Jdbc truncate table should check table not database @Hisoka-X
• [Feature][Rest Api]Additional rest api documentation. @liugddx
• [Improve][Iceberg]Add savemode create table primaryKey testcase @hailin0
• [Feature][Transforms-V2]LLM transforms Support custom field name @hawk9821
• [Improve][Connector-V2]Time supports default value @corgy-w
• [Improve][Connector-V2]Support read archive compress file @corgy-w
• [Improve][Document]Refactor job env config document @Hisoka-X
• [Feature][Transforms-V2]LLM transforms Support KimiAI @hawk9821
• [Improve][Doc]Add doc for mysql-cdc schema evolution @dailai
• [Improve][Document]Add remind for job config file suffix @Hisoka-X
• [Improve][doc]Fixed telemetry doc @corgy-w
• [Feature][Transforms-V2]llm trans support field projection @hawk9821
• [Feature][REST-API]Add threaddump rest api @liugddx
• [Feature][Core]shell batch cancel task @zhangshenghang
• [Improve][Docs]resource isolation add update node tags link @Cheun99
• [Feature][kafka]Add arg poll.timeout for interval poll messages @CosmosNi
• [Feature][rest-api]Add whether master node identifier @liugddx
• [Improve][Transform-v2]Refactor a transformRow from FilterFieldTransform @loustler
• [Improve][Docs]Add a usage method for a keyword in sql-config @dailai
• [Improve][Connector-V2]Flink support embedding transform @corgy-w
• [Improve][Transform]LLM transforms Support Doubao @corgy-w
• [Improve][Zeta]Split the classloader of task group @Hisoka-X
• [Improve][Test][Connector-V2][MongoDB]Add few test cases for BsonToRowDataConverters @loustler
• [Improve][Document]Update jar download link and formats sidebar @Hisoka-X
• [Improve][Connector-V2][MongoDB]A BsonInt32 will be convert to a long type @loustler
• [Feature][zeta]Support exposing monitoring metrics by prometheus exporter protocol @corgy-w
• [Feature]Support config variable substitution with default value @zhangshenghang
• [Feature][Connector-V2]Paimon Sink supported truncate table @hawk9821
• [Improve][Connector-v2]Improve the exception msg in case-sensitive case for paimon sink @dailai
• [Improve][Document]Improve quick start and build from source code @Hisoka-X
• [Feature][Zeta]add rest api to update node tags @Cheun99
• [Feature][Transform]Add embedding transform @corgy-w
• [Improve][Transform]Support errorHandleWay on jsonpath transform @hailin0
• [Feature][Connector-V2][Hbase]implement hbase catalog @zhangshenghang
• [Feature][Elastic search]Support multi-table source feature @FuYouJ
• [Feature][Zeta]Support slf4j mdc tracing job id output into logs @hailin0
• [Feature][Connector-V2]Support tidb cdc connector source @sunxiaojian

others

• [Docs]update Dynamic-compile document @liunaijie
• [Doc]Fix document symbol escape error @zhangshenghang
• [Chore]Code specification adjustments @ZhangWeike2000
• [Chore]Output job-id in zeta example log @hailin0
• [Docs][kafka]Add kafka protobuf doc @zhangshenghang
• [Docs]add docker cluster guide @liunaijie # apache/seatunnel

improve

• [Feature][Connector-Paimon]Support dynamic bucket splitting improves Paimon writing efficiency @hawk9821
• [Improve][EventService]improve event code and extract event code to EventService @EricJoy2048
• [Improve][Kafka]kafka source refactored some reader read logic @Carl-Zhou-CN

apache/seatunnel-website

fix

• [Fix]Fix can not import icons @Hisoka-X
• [Fix]Fix team title hard to notice when use dark mode @Hisoka-X
• [Fix]Fix team title hard to notice when use dark mode @Hisoka-X

improve

• [Improve]Update favicon.ico @Hisoka-X
• [Improve]Support dark mode @Hisoka-X
• [Improve]Remove mentor tab cause SeaTunnel is TLP @Hisoka-X

others

• Add doc card @zhangshenghang # apache/seatunnel-web

fix

• [Bug][Seatunnel-web][DB2]DB2 Datasource fails due to improper use of database and schema names. @arshadmohammad
• [Bug][Seatunnel-web][Hive]An error occurred during the attempt to save a job using the Hive connector. @arshadmohammad
• [Bug] [Seatunnel-web][Hive]Hive data source is not appearing in UI during the configuration of the Hive source. @arshadmohammad
• [Bug] [Seatunnel-web] [Metrics]NPE while fetching metrics from seatunnel-engine. @arshadmohammad
• [Bug][Seatunnel-web]Task instance filter is not working as expected @arshadmohammad
• [Bug] [Seatunnel-web]Job mode is hardcoded, job create API is always creates BATCH job @arshadmohammad
• [Bug] [Seatunnel-web]GET /seatunnel/api/v1/job/definition fails when job mode is passed @arshadmohammad
• [Bug] [Seatunnel-web]Job instance delete is not working @arshadmohammad

improve

• [Improvement][Seatunnel-web]Support H2 database in integration tests @arshadmohammad
• [Improvement][Seatunnel-web][Hive-JDBC]Add Hive-JDBC into seatunnel-web @arshadmohammad
• [Improvement][Seatunnel-web]Rename package com.whaleops.datasource to org.apache.seatunnel @arshadmohammad
• [Improve][SeaTunnel-Web]Unified use of JSONUtils @wuchunfu
• [Improve][SeaTunnel-Web]Unified version @wuchunfu
• [Improve][SeaTunnel-Web]Change JobMode and EngineType to enum type to avoid hard coding @wuchunfu
• [Improvement][Seatunnel-web]dom4j-1.6.1.jar has multiple CVEs, updat… @BilwaST
• [Feature][Seatunnel-web]Add support to configure placeholder with default value in the job config. @arshadmohammad
• [Improve]Update README.md for version 2.3.8 @Hisoka-X
• [Improve][SeaTunnel-Web]Change JobStatus to enum type to avoid hard coding @wuchunfu
• [Improvement] [Seatunnel-web]velocity-1.7.jar have multiple CVEs, upgrade all maven plugins which depend on this. @arshadmohammad