New Release: Enhanced Apache NiFi Connector for Pulsar v2.1.0
I'm excited to announce the availability of an updated version of the Apache NiFi connector for Pulsar! This week, we dedicated time to implementing much-needed improvements that will enhance your data streaming experience.
Community-Driven Improvements
First and foremost, I want to extend our heartfelt gratitude to the community members who took the time to report issues and provide valuable feedback. Your contributions are essential to making this connector more robust and reliable for everyone.
Key Changes in This Release
- Added support for OAuth2 credentials - Enhanced authentication flexibility by supporting clientId/clientSecret instead of requiring private key files (see issue #85 for details)
- Added Pulsar MessageID and message properties to the outbound FlowFiles - Pulsar MessageID and message properties are properly captured and forwarded to outbound FlowFiles (addresses issue #67)
- Optimized publisher resource management - Improved performance by reusing PublisherLease objects instead of creating new publishers for each FlowFile
- Updated to Apache NiFi 2.1.0 - Latest NiFi version support with newest features and security updates
- Updated to Apache Pulsar 3.3.7 - Latest stable Pulsar release integration
What's New
This release includes several key improvements and updates that significantly enhance the connector's functionality and performance:
New Features
- Added Pulsar MessageID and message properties to the outbound FlowFiles
- Added support for OAuth2 authentication using clientId, clientSecret
Enhanced OAuth2 Authentication Support
We've expanded authentication options by adding support for OAuth2 authentication using clientId and clientSecret credentials. This provides a more flexible alternative to private key file-based authentication, making it easier to integrate with modern cloud-based Pulsar deployments and enterprise authentication systems.
Added Pulsar MessageID and Message Properties to Outbound FlowFiles
The connector now properly captures and forwards Pulsar MessageID and message properties to outbound FlowFiles. This enhancement ensures that important message metadata is preserved throughout your data processing pipeline, enabling better message tracking, debugging, and downstream processing decisions based on message properties.
Optimized Publisher Resource Management
We've implemented a smarter approach to managing Pulsar publishers by reusing existing PublisherLease objects when possible, rather than storing publishers in a cache. This architectural improvement simplifies the design while preventing the unnecessary creation of new Pulsar Publisher instances for every FlowFile, resulting in better resource utilization and improved performance under high-throughput scenarios.
Platform Updates
- Updated to Apache NiFi 2.1.0: Ensuring compatibility with the newest features and security updates
- Updated to Apache Pulsar 3.3.7: Latest stable release providing improved performance and reliability
Getting Started
The updated connector maintains backward compatibility while providing enhanced functionality. You can download the latest version from Maven Central and find installation instructions in the project repository.
Looking Forward
I remain committed to maintaining and improving this connector based on community feedback. If you encounter any issues or have suggestions for future enhancements, please don't hesitate to open an issue in our GitHub repository.
Thank you again to our community for your continued support and contributions. Together, we're building better tools for real-time data processing and streaming.
For technical support or questions about the Apache NiFi connector for Pulsar, visit my GitHub repository or reach out to the community.*
As an Apache Pulsar committer, I'm always interested in hearing about your experiences with streaming data technologies. Feel free to reach out with questions or share your own insights!
Top comments (0)