🚀 Deploying qData Open Source: A Docker Guide for a Full Data Stack

#opensource #devops #api #dataengineering

Deploying a complete data platform can be a complex task, but the open-source version of qData simplifies this with a Docker Compose setup. With a single command, you can spin up an entire suite of services, including the core platform, DolphinScheduler, Hadoop, and Spark.

This guide walks you through the deployment process, from pre-flight checks to verifying your first data task, ensuring a successful launch.

🛠️ Pre-Deployment Checklist

Before running any commands, let's ensure your environment is ready to avoid common pitfalls.

Confirm Server Architecture
Run uname -m to check your system's architecture.
- x86_64 corresponds to amd64 images.
- aarch64 corresponds to ARM64 images. A mismatch here will result in an exec format error.
Check Docker Version
Ensure you have Docker >= 19.03 and Docker Compose >= 2.20.2. You can verify this with:
```
docker -v
docker compose version
```

Check System Resources
Use free -h and df -h to confirm you have sufficient memory and disk space. If a container exits with code 137, it's a classic sign of an Out-Of-Memory (OOM) kill.
Check Port Availability
The deployment requires ports 80 (for qData) and 12345 (for DolphinScheduler). Check for conflicts with:
```
ss -lntp | grep -E ':(80|12345)\b'
```

📦 Extract and Backup

First, extract the qData deployment package and the docker.zip file inside it. Before making any changes, it's a best practice to back up the original configuration files.

cd ~/qData/docker
cp .env ".env.bak.$(date +%Y%m%d%H%M%S)"
mkdir -p backup
cp docker-compose*.yml backup/

Pro Tip: If you edited any scripts in a Windows environment, you might encounter a bad interpreter error due to line ending issues. Fix this by running sed -i 's/\r//' <script_path> and ensure the script has execute permissions with chmod 755.

🗄️ Select Your Primary Database

qData supports both MySQL and Dameng DM8. You need to choose one by setting the DB_TYPE variable in your .env file.

Using MySQL:
1. Set DB_TYPE=mysql in the .env file.
2. All subsequent docker compose commands must include the -f docker-compose-mysql.yml parameter.
Using Dameng DM8:
1. Set DB_TYPE=dm8 in the .env file.
2. Use the default Compose file without any extra parameters.

⚙️ Initialize the Database

Crucial Step: You must initialize the database before starting all the services.

For Dameng DM8:

sudo docker compose --profile schema up -d

For MySQL:

sudo docker compose -f docker-compose-mysql.yml --profile schema up -d

After running the command, use docker compose logs to check the output. Ensure there are no errors and that the database container is running correctly before proceeding.

▶️ Start All Services

Once the database is initialized successfully, you can bring up the entire platform.

For Dameng DM8:

sudo docker compose --profile all up -d

For MySQL:

sudo docker compose -f docker-compose-mysql.yml --profile all up -d

Use docker ps -a to verify that the core containers are in an Up state and that ports 80 and 12345 are correctly mapped.

✅ Verify the Complete Service Chain

Just because the web pages load doesn't mean the deployment is fully functional. Let's verify the entire data pipeline from task submission to result writing.

Check Service Access:

curl -I http://127.0.0.1:80
curl -I http://127.0.0.1:12345/dolphinscheduler/ui/home

Log in to the Systems:
- qData: http://<Server_IP>:80 (User: qData, Pass: qData123)
- DolphinScheduler: http://<Server_IP>:12345/dolphinscheduler/ui/home (User: admin, Pass: dolphinscheduler123)

Create and Execute a Test Task: Create a simple data integration task in qData (e.g., Table Input → Field Mapping → Table Output). After submission, trace the execution:
- Does qData generate a running instance?
- Does DolphinScheduler generate a corresponding workflow instance?
- Does the Worker pick up the task?
- Does Spark generate the application?
- Does the target table contain the expected data?

The deployment is only considered a success when this entire chain runs without issues.

🚨 Troubleshooting Common Issues

Incorrect Startup Command: When using MySQL, forgetting to add -f docker-compose-mysql.yml to your commands is a common mistake.
Port Conflict: An Error port is already allocated message means another process is using the port. Stop the conflicting program or modify the port mapping in the Compose file.
Container Restarting: Use docker inspect <container_name> to check the exit code. Code 137 usually means insufficient memory, while 126/127 points to script permission or path issues.
Cannot Access Web Page: If curl fails locally, check the container logs. If it works locally but not from an external machine, check your server's firewall and security groups.
No Instance After Task Submission: Focus on the connection between qData and DolphinScheduler. Verify the API address, Token validity, and the status of the Master/Worker nodes.

🛠️ Common Maintenance Commands

The following commands use the default Compose file as an example. Remember to add -f docker-compose-mysql.yml when using MySQL.

Action	Command
View Status	`sudo docker compose --profile all ps --all`
View Logs	`sudo docker compose --profile all logs -f`
Stop/Start	`sudo docker compose --profile all stop` / `start`
Restart Service	`sudo docker compose --profile all restart`
Reset Environment	`sudo docker compose --profile all down`