As this year’s Summer of Open Source (OSPP) program comes to an end, developers from the Apache DolphinScheduler community have completed their long development journey and achieved impressive results — including a brand-new gRPC task plugin for DolphinScheduler.
This contribution comes from Haibo Zhao, an outstanding student majoring in Electrical Engineering at Beijing Jiaotong University. Let’s take a look at how he successfully completed this development task during the program!
Personal Introduction
Name: Haibo Zhao
University: Beijing Jiaotong University, Electrical Engineering
Mentor: Tan Hengliang
GitHub ID: npofsi
Research Interests: Time-series data analysis, model optimization and control
Hobbies: Digital painting, gourmet food
Project Title
- Add a gRPC Task Plugin for Apache DolphinScheduler
Project Overview
In OSPP 2025, I was responsible for developing a new gRPC task plugin for Apache DolphinScheduler, enabling workflow tasks to interact directly with gRPC services.
Similar to DolphinScheduler’s HTTP task, this plugin allows workflows to access external gRPC service interfaces. It was designed using the same configuration logic as the HTTP plugin and implemented based on DolphinScheduler’s SPI plugin mechanism.
As a distributed workflow scheduler listed in the CNCF Landscape, DolphinScheduler has seen broad adoption in cloud-native environments. However, it previously lacked support for the gRPC protocol, which is widely used in such scenarios — thus motivating this project.
Development Path
Based on this requirement, I analyzed how users typically invoke gRPC services within workflows. With guidance from my mentor, I designed an interaction model similar to API debugging tools, allowing separate configuration of schema and request messages, which are then merged during execution.
This led to a plugin architecture that preprocesses protobuf and data on the frontend and merges them on the backend.
When configuring the gRPC task, the frontend accepts parameters such as the gRPC service endpoint, protobuf service definition, request message, and SSL configuration.
Upon saving, protobuf.js compiles the protobuf definition into a specific descriptor and passes it to the backend. During execution, the backend reconstructs the protobuf descriptor (which defines the message’s binary format), merges it with the user-provided JSON message, and sends a gRPC request to the configured service endpoint.
Project Outcomes
- Completed the design and implementation of the DolphinScheduler gRPC task plugin, which enables dynamic gRPC calls at Java runtime.
- Introduced a configuration model similar to gRPC debugging tools, allowing reuse of existing configuration files and test messages.
- Finished comprehensive unit and integration tests, covering major features and edge cases.
- Provided detailed documentation, parameter descriptions, and usage examples.
Key Technical Highlights
The most challenging technical aspect was implementing dynamic invocation in gRPC.
Unlike RESTful APIs that use a schema-on-read JSON format, gRPC uses schema-on-write through predefined protobuf definitions. These must be compiled ahead of time and are not transmitted with data. This design reduces payload size, improves efficiency and consistency, and allows more compile-time checks—but it also limits runtime flexibility.
To address this, I used protobuf.js as the compiler to pre-compile protobuf files on the frontend. Task data includes both the source protobuf and its intermediate descriptor, enabling the backend to build binary formats directly without depending on protoc. This means users can make gRPC requests without installing protoc.
After completing the development, the DolphinScheduler community conducted a short interview to learn more about participants’ experiences and reflections. Below is the interview transcript:
Q1: Among all projects, why did you choose Apache DolphinScheduler?
A: As a distributed workflow scheduling system, DolphinScheduler’s architecture itself is fascinating. It covers key topics such as task scheduling, distributed systems, and cloud-native design — areas I’ve always wanted to explore.
gRPC, as the preferred RPC framework for modern microservices, has also been a technology I wanted to study deeply.
Additionally, the DolphinScheduler community is very mature. During the application phase, I communicated with mentors who clearly explained the project goals and expectations. The documentation was extensive, and the community environment was friendly — all of which gave me confidence to complete the project successfully.
Q2: Does this project relate to your academic studies?
A: My academic background is in electrical engineering, focusing on control and optimization of renewable energy systems. The power grid itself is a complex distributed system requiring high coordination — concepts closely related to cloud-native technologies.
For example, China’s State Grid uses Kubernetes to manage a large number of monitoring nodes. Therefore, I wanted to learn more about technologies that could eventually be applied in my field.
The open-source community provides a platform for deep technical discussions and collaboration. I really enjoy exchanging ideas and working with people from different backgrounds to build better solutions. It’s a great way to meet like-minded peers and discover new opportunities.
Q3: How has participating in this project influenced your studies and future plans?
A: I learned a lot about gRPC and plan to use it in other systems in the future. I also gained a deeper understanding of workflow systems like DolphinScheduler, as well as valuable testing experience.
Most importantly, I felt a strong sense of participation. With my mentor’s encouragement, I was able to complete the entire process — from submitting the RFC and issue to having my PR merged. That sense of accomplishment was deeply rewarding.
Q4: What was the biggest challenge during development, and how did you overcome it?
A: The biggest challenge wasn’t a specific technical problem, but time management.
Most of my time is devoted to my lab research, so I had to contribute to the open-source project in the evenings and on weekends. However, the community’s constant positive feedback kept me motivated. The sense of achievement, belonging, and recognition from contributing value was my biggest driving force.
Q5: How long have you been involved in open source? Do you enjoy it?
A: My first exposure to open source came from a book about JavaScript optimization, which introduced me to open-source projects. Although I didn’t participate directly at the time, it sparked my curiosity.
Later, after using open-source software and learning its philosophy, I decided to join in. The experience has been very enjoyable. Unlike working in isolation, open source amplifies individual contributions through sharing and feedback, helping me grow faster.
I’ve also learned to pay more attention to upstream projects and gained a lot of practical experience.
Q6: What was your first impression of the DolphinScheduler community? Any feedback or suggestions?
A: My first impression came during the application phase. The mentors were friendly and patient, answering all my questions in detail.
The community’s comprehensive documentation also helped me quickly set up the environment and run the software successfully — showing the community’s professionalism and maturity.
Through OSPP, I learned a lot about software development and hope to further understand how open-source communities operate.
One improvement I’d suggest is updating the SPI-related documentation, which is currently somewhat outdated. Enhancing this could make plugin development easier and faster, boosting DolphinScheduler’s ecosystem.
Q7: Will you continue contributing to the Apache DolphinScheduler community?
A: Yes, definitely. Especially with ongoing maintenance of the gRPC plugin — I plan to further improve its maintainability and replace some low-maintainability components.
Top comments (0)