Building Metadata Capabilities in Apache SeaTunnel: A Committer’s Journey

#programming #ai #beginners #javascript

Recently, Apache SeaTunnel welcomed several talented and highly motivated new Committers, and Wang Xuepeng is one of them.

As a long-time contributor, Wang Xuepeng’s promotion to Committer was no coincidence. Over the years, he has quietly contributed a tremendous amount to the community, and everyone has witnessed his dedication. From first stepping into the open-source world to becoming a Committer of an Apache top-level project, he has accumulated plenty of stories and valuable insights along the way.

What inspired his journey? What experiences and lessons does he want to share with the community? Let’s take a closer look at this exclusive interview with him!

Personal Introduction

Interview Transcript

How long have you been involved in open source? What attracts you to open source?

I started getting involved in open source in 2023. What attracts me most is the sense of achievement when the code I write can actually be used within the industry.

When did you start contributing to SeaTunnel? What was the trigger?

I joined WhaleOps in 2023, which was also when I first started engaging with open source.

Now that you’ve been elected as a SeaTunnel Committer, could you summarize your contributions to the community, including both code and non-code contributions?

Most of my major feature PRs have focused on building SeaTunnel’s metadata capabilities.

When running SeaTunnel jobs and writing job configurations, users often need to manually enter datasource connection information. For file-based tasks, users also need to manually define field mappings. To address these issues, I designed an SPI interface called MetadataProvider.

The interface mainly exposes two methods:

Map<String, Object> datasourceMap(String connectorIdentifier, String metaDataDatasourceId);
Optional<TableSchema> tableSchema(String metaDataTableId);

Previously, some users in the community mentioned that datasource usernames and passwords were stored in Nacos with read-only access permissions. In scenarios like this, users can implement a custom metadata center to better protect sensitive connection information.

Community Contribution Summary

PR Link	Description
https://github.com/apache/seatunnel/pull/5663	Added `save_mode` functionality to SeaTunnel
https://github.com/apache/seatunnel/pull/10402	Integrated Gravitino with SeaTunnel
https://github.com/apache/seatunnel/pull/10586	Designed the metadata SPI interface for SeaTunnel
https://github.com/apache/seatunnel/pull/10657	Enhanced the metadata SPI interface for SeaTunnel
https://github.com/apache/seatunnel/pull/10838	Added dynamic metadata functionality based on the metadata SPI interface

After contributing to SeaTunnel for so long, you must have developed a deep understanding of both the project and the community. Compared with competing products, what do you think are SeaTunnel’s strengths and weaknesses? What keeps you actively involved in the community?

One major advantage of SeaTunnel is the flexibility of its engine choices. Teams already familiar with Flink or Spark can adopt it with a very low learning curve. For lightweight data synchronization scenarios, the Zeta engine is an even better choice.

As for weaknesses, I think the web platform still has a lot of room for improvement.

What attracts me most to the SeaTunnel community is the opportunity to discuss implementation solutions with talented contributors from different technical fields. It helps me improve my own skills while broadening my perspective.

Have you ever done any secondary development based on SeaTunnel’s shortcomings? Have you contributed those improvements back to the community? Could you briefly introduce your solution?

Yes, I’ve done secondary development for SeaTunnel. Most of the time, when I encounter bugs during usage, I first fix them in our company repository and then submit the same fixes back to the open-source community.

What kind of support do you hope the SeaTunnel community can provide for your personal growth in the future?

As long as people actively participate in community discussions — whether it’s creating issues, submitting PRs, or reviewing PRs — they will definitely improve their technical abilities.

What does the Committer role mean to you? What responsibilities should a Committer take within the community?

I believe a Committer should first ensure code quality. Secondly, Committers should help guide the community in a positive direction, such as mentoring newcomers on how to submit PRs.

Now that you’ve become a Committer, what would you like to say to the community? Do you have any suggestions for the project’s future development?

First of all, I’m very happy to become a Committer. It means becoming part of the Apache Foundation, which is truly a valuable identity and experience. I also want to thank all the community members who guided and helped me along the way.