The Apache SeaTunnel Community Welcomes A New Committer From India!

Great news! The community has recently welcomed a new Indian Committer, Mohammad Arshad!

Mohammad, a contributor from Bangalore, is a senior data engineer and consultant at Visa with extensive experience in big data. As one of the world’s largest payment processing companies, Visa holds a significant position in India’s digital payments and fintech sectors.

With his team, Mohammad Arshad optimized and integrated SeaTunnel Web, successfully resolving key issues that had previously existed, greatly enhancing the user experience and solving a pressing challenge for the community.

In this edition of our community event, we had the pleasure of conducting an in-depth interview with Mohammad to learn how he became involved with SeaTunnel. Let’s dive into the inspiring story of this talented engineer from India!

Disclaimer: The views expressed in this talk are my own and do not necessarily reflect any official views.

Personal Profile

• Name: Mohammad Arshad
• Nationality: Bangalore, India
• Profession： Data Engineer — — Senior Consultant at Visa
• GitHub ID: arshadmohammad.
• Expertise: Working in the Big Data domain.
• Hobbies: In my free time, I enjoy watching movies and listening to Urdu poetry.

You can find more details about Arshad on his linked-in profile:
https://www.linkedin.com/in/arshad31415/

Contribution to SeaTunnel web

Arshad and his team worked together to optimize the features of SeaTunnel Web, addressed several key issues, and enhanced the overall functionality, including:

• System Integration: SeaTunnel Web was integrated with other existing systems by adding new APIs to manage jobs. These APIs enable creating, retrieving, updating, deleting, and executing jobs, thereby improving interoperability and streamlining workflows.
• Parameterized Query Execution: Previously, there was no option to parameterize query execution. This feature was introduced to allow users to pass parameters dynamically, making queries more flexible and reusable.
• Enhanced Testing Capabilities: To make testing more efficient, features such as FakeSource and Console data sources were added. Additionally, integration tests were written to verify functionality on real instances of SeaTunnel and SeaTunnel Web. These tests ensure that the system performs reliably under various conditions.
• Bug Fixes: Numerous bugs in SeaTunnel Web were addressed to ensure it could handle current use cases effectively. While there are still unresolved issues, these fixes have significantly improved stability and usability. ## Validating Various Use Cases Though have not used SeaTunnel in his work or personal research yet, Arshad is actively exploring its features and adding enhancements, while also validating various use cases to ensure they are fully supported. He aims to transfer data from one data storage to another in a secure, reliable, and fast manner with minimal manual intervention. Once all use cases are met, he plans to start using SeaTunnel extensively.

A Passionate Open-Source Participant

Arshad has been involved in open source since around 2017. In addition to SeaTunnel, he is a committer for Apache Ambari and Apache ZooKeeper. He has also contributed to various other projects, such as Hadoop and HBase.

He is passionate about open source because it provides opportunities to learn and contribute to the community. The rapid pace at which community projects grow would be nearly impossible for a single company to achieve without incurring significant costs.
Given that Hadoop and its ecosystem projects are open source, it makes perfect sense to be part of this collaborative environment.

When he first knew about Apache SeaTunnel, he was working on other Apache projects such as Ambari and ZooKeeper, and was seeking a replacement for NiFi and has discovered Apache SeaTunnel.

He found the SeaTunnel community to be incredibly helpful and responsive. The support he has received from the community has been outstanding.

For instance, he would often check to ensure his PR’s CI was successful before going to bed, only to find it merged the next morning.

This kind of prompt response has been both encouraging and motivating. Mentioning this, he expressed his gratitude to Jia Fan and David for their exceptional support.

Contributing to SeaTunnel has been a smooth and comfortable experience

As a new contributor to the SeaTunnel community, Arshad describes the process of contributing to SeaTunnel to be a smooth and comfortable experience. The process is straightforward and does not present any difficulties.

SeaTunel Or NiFi?

Before using SeaTunnel, Arshad compared SeaTunnel with other similar tools like NiFi.

The result shows that SeaTunnel appears to be a better fit for their use case because it supports distributed computation, has a wider range of connectors, utilizes a more updated technology stack, and has a more active community.

Ultimately, among many options, they chose SeaTunnel.

Expected New Features of SeaTunnel

For SeaTunnel, Arshad expects the following new features and optimizations:

1. Operability, Maintainability, Metrics, and Monitoring: Enhance the operational and maintenance capabilities of SeaTunnel. Maybe integration with Ambari for these features would be beneficial. Add comprehensive metrics and monitoring solutions
2. SeaTunnel Web Enhancements: Improve SeaTunnel Web by adding more features and all connector and transformation support.
3. Security Improvements: Use upgraded versions of dependencies to address security vulnerabilities and upgrade to Java 17 for both SeaTunnel and SeaTunnel Web.
4. Role-Based Access Control (RBAC): Introduce RBAC in SeaTunnel Web for better access management.
5. LDAP Support: Add LDAP support in SeaTunnel Web for centralized authentication.
6. Deployment Architecture: Improve the deployment architecture so end users do not need direct access to SeaTunnel machines. Users should interact only with SeaTunnel Web for better security management.
7. Engine Support in SeaTunnel Web: Ensure SeaTunnel Web supports job execution for all engines supported by SeaTunnel, such as Zeta, Spark, and Flink.

As Arshad mentioned, SeaTunnel still has plenty of room for improvement, and many ambitious goals are yet to be achieved. This will require the strength of the community to realize these goals sooner. We welcome passionate individuals to join this group and contribute their wisdom to the project!