Apache DolphinScheduler is a distributed, decentralized, and scalable visual DAG workflow task scheduling system. It is committed to solving the complex dependencies in data processing workflows, making the scheduling system ready to use.
Ambari is a tool for managing Hadoop clusters. Although Ambari does not directly support the installation of DolphinScheduler, we can integrate it into Ambari through a custom service. This article will describe how to use Ambari to install DolphinScheduler.
1. Ambari Service Installation Package
Place the service installation package that needs to be integrated into Ambari in the following directory:
/var/lib/ambari-server/resources/stacks/HDP/3.1/services
2. Restart the Ambari-server Service
sudo ambari-server restart
After the integration, you can proceed with the service installation on the Ambari web page.
- Add Service and Select DolphinScheduler
- DolphinScheduler Service Configuration Choose to use MySQL as the metadata database for the DolphinScheduler service.
- Service Selection for Installation Select two DS-master nodes, at least two DS-ALERTS, DS-APIs, and deploy the remaining nodes as DS-worker nodes.
- Error Handling for hue-desktop-ini If you have configured namenode-ha before installing DolphinScheduler, you may encounter errors related to 'hue-desktop-ini'. You need to modify the params.py script.
Script location: /var/lib/ambari-server/resources/stacks/HDP/3.3/services/DOLPHINSCHEDULER/package/scripts/params.py
Modify the content:
# hdfs_httpfs_host = config['configurations']['hue-desktop-ini']['hdfs_httpfs_host']
hdfs_httpfs_host = "dfs_ha_enabled" # Comment out the original line and assign any value to hdfs_httpfs_host
After making the changes, save and restart the ambari-server service to reinstall DolphinScheduler.
- Copy MySQL Driver to All Nodes' libs Directory
cp /tmp/mysql-connector-j-8.0.33.jar /usr/hdp/current/dolphinscheduler/alert-server/libs/
cp /tmp/mysql-connector-j-8.0.33.jar /usr/hdp/current/dolphinscheduler/api-server/libs/
cp /tmp/mysql-connector-j-8.0.33.jar /usr/hdp/current/dolphinscheduler/master-server/libs/
cp /tmp/mysql-connector-j-8.0.33.jar /usr/hdp/current/dolphinscheduler/worker-server/libs/
cp /tmp/mysql-connector-j-8.0.33.jar /usr/hdp/3.3.1.0-004/dolphinscheduler/tools/libs/
- Change Ownership of All Services to hdfs User
chown -R hdfs:hdfs /var/log/dolphinscheduler
chown -R hdfs:hdfs /usr/hdp/3.3.1.0-004/dolphinscheduler
chown -R hdfs:hdfs /etc/dolphinscheduler/
ln -s /usr/hdp/3.1.4.0-315/dolphinscheduler dolphinscheduler
After making these changes, start DolphinScheduler from the Ambari management platform. The metadata will be automatically initialized and tables will be created during the startup process.
Top comments (0)