DEV Community

Apache SeaTunnel
Apache SeaTunnel

Posted on

Modernizing Infrastructure: Seamless Data Migration to HighGo DB with Apache SeaTunnel

#seatunnel #database

Wondering how to interface Apache SeaTunnel with HighGo Database? This article shares hands-on experience. HighGo Database is built on the PostgreSQL kernel, allowing it to be connected directly using standard JDBC drivers. Below are configuration examples for HighGo MySQL-mode to PG-mode migration and Doris-to-HighGo data transfers.

1. Introduction to HighGo Database

HighGo is a leading Chinese database vendor specializing in enterprise-grade applications. Built on the PostgreSQL kernel, it is a prominent player in China's domestic IT modernization ecosystem (Xinchuang), similar to KingBase.

Key Features:

  • Fully compatible with the PostgreSQL protocol.
  • Certified for government and critical infrastructure IT standards.
  • Utilizes standard PostgreSQL drivers (no proprietary drivers required).
  • Supports multiple deployment modes (Standalone, Primary-Standby, Distributed).

HighGo offers both PG and MySQL compatibility modes. You can treat it as native PG or MySQL; standard JDBC and tools like Navicat connect seamlessly. One minor tip: when using older versions of Navicat with HighGo's MySQL mode, you may need to select the "Legacy" client driver in settings to avoid metadata errors when opening tables.

2. Practical Read/Write Scenarios

2.1 Reading HighGo MySQL Mode to HighGo PG Mode

You can paste this configuration directly into a SeaTunnel node within DolphinScheduler. Unlike some competitors that require PG drivers to access MySQL-compatible schemas, HighGo acts as a native MySQL instance (using the MySQL JDBC driver).

env {
  parallelism = 2
  job.mode = "BATCH"
}
source {
  Jdbc {
    driver = "com.mysql.cj.jdbc.Driver"
    url = "jdbc:mysql://192.168.0.110:3306/public"
    user = "root"
    password = "root"
    query = "SELECT * FROM public.tb_dict;"
  }
}
sink {
  jdbc {
    url = "jdbc:postgresql://192.168.0.119:5866/datadb"
    driver = "org.postgresql.Driver"
    user = "highgo"
    password = "highgo"
    generate_sink_sql = true
    database = datacenter
    table = data_schema.dim_public_dict_info
    schema_save_mode = "CREATE_SCHEMA_WHEN_NOT_EXIST"
    field_ide="LOWERCASE"
    data_save_mode="DROP_DATA"
  }
}
Enter fullscreen mode Exit fullscreen mode

The execution is as smooth as silk.

2.2 Meeting Compliance and Migration Requirements

If your existing system uses non-domestic databases (e.g., Apache Doris) but your production environment mandates a transition to certified domestic platforms, SeaTunnel serves as the perfect migration bridge. You can treat Doris as a high-performance engine to process data before writing it back to the compliant HighGo DB.

env {
  parallelism = 2
  job.mode = "BATCH"
}
source {
  Jdbc {
    url = "jdbc:mysql://192.168.0.120:9030/data_statistics"
    driver = "com.mysql.cj.jdbc.Driver"
    connection_check_timeout_sec = 100
    user = "root"
    password = "root"
    "table_list" = [
      { "table_path" = "data_statistics.data_develop_data_source_yw" },
      { "table_path" = "data_statistics.data_develop_data_source_type" },
      { "table_path" = "data_statistics.data_develop_data_source_ip" }
    ]
  }
}
sink {
  jdbc {
    url = "jdbc:postgresql://192.168.0.119:5866/datadb"
    driver = "org.postgresql.Driver"
    user = "highgo"
    password = "highgo"
    generate_sink_sql = true
    database = datadb
    table = "data_schema.${table_name}"
    data_save_mode = "DROP_DATA"
  }
}
Enter fullscreen mode Exit fullscreen mode

3. Summary

From my experience, the combination of Doris + DolphinScheduler + SeaTunnel has become the "New Trinity" of data engineering. While DolphinScheduler and Doris handle most ETL tasks via catalogs, SeaTunnel acts as the ultimate fail-safe for complex migrations or specialized domestic database integrations.

Top comments (0)