How to Unify Your Analytics Ecosystem Without Rebuilding Your Data Pipeline From Scratch
1. Introduction
When it comes to real-time analytics, people tend to focus first on how fast the engine runs. But in an enterprise setting, raw speed is only one piece of the puzzle. For an analytical database to truly land in production, it must integrate smoothly with the data ecosystem the enterprise has already built. Data is typically scattered across file systems, message queues, and existing operational databases, while analytical results need to flow back into reports, dashboards, and downstream applications. If every integration point requires custom adapter code to develop and maintain, the adoption cost quickly becomes prohibitive.
The traditional approach is to stitch together multiple specialized components into an analytics pipeline. Data might first go into Kafka for buffering, then through Flink for stream processing, land in Hive for historical storage, and finally be loaded into ClickHouse for querying. Each layer means another round of synchronization and transformation. The longer the pipeline, the harder it is to control end-to-end latency, and the greater the consistency risk between systems. More practically, development teams spend enormous effort on data movement and pipeline maintenance, leaving less time for actual business logic.
Therefore, real-time analytics capabilities require a complete set of supporting features, centered on three core concerns: how data comes in, how results get out after processing, and how the entire process is managed. These three concerns span the full pipeline from data ingestion through processing to consumption. OceanBase’s approach is to use a unified engine to interface with the existing ecosystem, minimizing intermediate layers rather than asking enterprises to add even more components. At the same time, OceanBase supports in-database data warehouse layering, allowing data to be progressively refined from ODS (Operational Data Store) to DWS (Data Warehouse Summary) to ADS (Application Data Store) within a single database, with Flink handling inter-layer data flow to further reduce dependence on external systems. The following sections walk through these capabilities in order of data flow.
2. Data Integration: Ingestion and Flow of Multi-Source Data
Getting data into the database is the first step of analytics. Data from different sources, at different scales, and with different freshness requirements calls for different ingestion methods. OceanBase provides capabilities across three dimensions: batch import, external tables, and real-time synchronization.
2.1 Batch Import
For scenarios that require loading large volumes of data in one shot, OceanBase provides Direct Load. Data bypasses the SQL parsing and transaction processing layers and directly generates the underlying storage files (SSTables), skipping the overhead of per-row transaction commits and delivering significantly higher efficiency for bulk writes.
This approach is well suited for initial database seeding, historical data migration, monthly archiving, and ETL batch loading. These scenarios share common characteristics: large data volumes and low requirements for per-row transaction semantics, which play directly to Direct Load’s strengths. Compared to row-by-row inserts, Direct Load processes a batch of data as a whole, reducing repeated transaction commits and log writes. The larger the data volume, the more pronounced the time savings.
Complementing Direct Load, the OBLOADER tool handles batch import and export of structured data, managing the mapping between data file formats and database table structures to make bulk loading operations simpler and more controllable.
2.2 External Tables and Data Lake Integration
Much analytical data does not reside inside the database but is stored as files in object storage or distributed file systems. The core idea behind OceanBase’s external table capability is to avoid moving data altogether — instead, it defines mappings to external files directly within the database.
In terms of file formats, external tables support the universal CSV text format as well as Parquet and ORC columnar storage formats. Columnar formats organize data by column, achieving high compression ratios and reading only the columns needed during scans, making them ideal for analytical scenarios. For storage systems, external tables are compatible with S3, OSS, and other object storage services, as well as HDFS and ODPS. For metadata management, external tables can connect to Hive Metastore (HMS) and ODPS Catalog to form a unified metadata view, and support table formats such as Hive and Iceberg.
Building on this, External Catalog can register database and table information from multiple external data sources into OceanBase in a unified manner, allowing external data to be queried directly with standard SQL without the need to manually create external tables one by one. Data already sitting in a data lake does not need to be copied into the database again, reducing both storage redundancy and the latency introduced by data movement.
2.3 Real-Time Synchronization and Data Collection
Analytics scenarios often demand data that is as fresh as possible, requiring continuous synchronization of changes from operational databases and timely ingestion of various log and event data. OceanBase supports integration with a wide range of synchronization and collection tools, covering everything from log collection to database synchronization.
For log collection, Flume and Logstash collect log data from application servers such as Nginx and Apache Tomcat, performing simple parsing and formatting before pushing data into OceanBase. This type of log data is high-volume and high-frequency, typically used for behavioral analysis, operations monitoring, and similar scenarios.
For database synchronization, OMS (OceanBase Migration Service) is OceanBase’s built-in migration and synchronization tool, supporting both full migration and incremental sync across migrations from MySQL, Oracle, PostgreSQL, PolarDB, and other databases into OceanBase. DataX is Alibaba’s open-source heterogeneous data source synchronization tool, supporting batch data synchronization across multiple database types, suitable for periodic full or incremental extraction tasks.
On the data processing side, dbt handles SQL-level ETL transformation logic, while DataWorks provides data integration and development platform capabilities on Alibaba Cloud. Kafka, as a message queue, handles buffering and distribution of streaming data and is often used in conjunction with the synchronization tools mentioned above.
The fact that these tools can connect directly means that most existing enterprise data pipelines can be reused — there is no need to build an entirely new synchronization pipeline just to integrate with OceanBase.
2.4 Flink Connector Ecosystem
In real-time data integration scenarios, Apache Flink is one of the most widely used stream processing engines today. OceanBase provides multiple official Flink Connectors covering the full pipeline from data writing to change data capture, adapting to different performance requirements and use cases.
The JDBC Connector reads and writes OceanBase through the standard JDBC protocol, supporting both MySQL and Oracle compatibility modes. It includes built-in Druid connection pool management and batch write optimization (buffering plus batch commit), along with retry mechanisms. It is suited for common scenarios such as real-time data synchronization, CDC incremental writes, and writing stream processing results to the database, making it the most versatile choice.
The DirectLoad Connector is based on the Direct Load API. Data bypasses the SQL parsing layer and writes directly to the underlying storage files. Write throughput is far higher than the JDBC approach, making it ideal for high-throughput scenarios such as large-scale data import, ETL batch processing, and historical data migration.
The OBKV HBase Connector writes to OceanBase via the OBKV HBase API, suited for HBase-compatible scenarios and wide-table storage requirements.
Flink CDC (Source) reads change data from OceanBase, supporting both MySQL and Oracle compatibility modes, and synchronizes incremental data to downstream systems. It is suited for scenarios such as syncing data out of OceanBase and data lake ingestion. At the same time, Flink CDC can also listen to change logs from external databases (MySQL, Oracle, PostgreSQL, etc.) and write incremental data into OceanBase in real time, making it one of the core channels for data ingestion.
In addition, the Flink Connector JDBC (community edition) and Apache SeaTunnel can also be used to process OceanBase data through Flink, providing more options for existing technology stacks.
In an in-database data warehouse layering architecture, Flink also takes on the role of inter-layer processing. Data flows from ODS through cleansing into DWS, then aggregates into ADS, with real-time flow between each layer handled by Flink. This way, OceanBase provides unified storage while Flink handles inter-layer computation, and together they form a lightweight real-time data warehouse solution that reduces dependence on multiple external systems.
A quick selection guide: use the JDBC Connector for regular writes; choose DirectLoad Connector for bulk data loading; use OBKV HBase Connector for HBase compatibility needs; use Flink CDC when you need to read changes from OceanBase.
2.5 Spark Connector Ecosystem
If Flink’s strength lies in real-time stream processing, Spark excels at large-scale batch processing and interactive analytics. The two are complementary: Flink handles continuous incremental computation, while Spark handles periodic full scans, complex ETL, and ad-hoc queries. OceanBase provides connectivity for both, allowing users to flexibly choose the compute engine based on the scenario.
Starting from v1.1, the OceanBase Spark Connector is deeply integrated with Spark Catalog, which is the currently recommended usage approach. Its key features include:
Zero-code access with full SQL interaction. Spark-SQL can directly read, write, and manage databases and tables in OceanBase, supporting common operations such as SHOW DATABASES, SHOW TABLES, and CTAS without writing additional code.
Adaptive partitioning and parallel reads. The Connector automatically recognizes OceanBase partition table structures and splits read tasks based on partition information, enabling multi-partition parallel scanning that fully leverages the distributed computing power of the Spark cluster.
Predicate pushdown. Filter conditions are pushed down to OceanBase for execution, reducing the amount of data transferred over the network and improving query efficiency.
Automatic schema inference. No need to manually define table structures — the Connector automatically discovers column information and type mappings for OceanBase tables.
Flexible write modes. Supports both JDBC writes and Direct Load writes. JDBC is suited for regular write scenarios, while Direct Load is designed for bulk data loading, bypassing the SQL layer and writing directly to storage for higher throughput.
Upsert support for primary key tables. In MySQL mode, this is based on the INSERT … ON DUPLICATE KEY UPDATE syntax; in Oracle mode, it is based on the MERGE syntax.
Supports both MySQL and Oracle compatibility modes simultaneously.
Typical use cases: leveraging Spark’s distributed computing power for batch analytics and ETL processing on massive data in OceanBase; syncing data from Hive or data lakes into OceanBase via Spark; completing heavy computation tasks in the Spark cluster and efficiently writing results back to OceanBase via Direct Load.
In addition to the Catalog Connector, OceanBase also provides the OBKV HBase Connector for Spark for data access in HBase-compatible scenarios. The legacy Spark Connector (for Spark 2.x) is being gradually retired; new projects are recommended to use the Catalog approach.
3. SQL Compatibility: Lowering the Barrier to Entry
Once data is in the database, how it gets queried is another critical concern. If the query syntax differs significantly from the enterprise’s existing systems, application modification and staff training both incur costs. OceanBase has invested heavily in SQL compatibility, with the goal of enabling existing applications and tools to connect with minimal changes.
3.1 MySQL Compatibility
OceanBase’s MySQL compatibility operates on two levels. At the protocol level, OceanBase communicates using the MySQL protocol, so existing MySQL clients and drivers — such as JDBC and Python connectors — can connect directly without replacement. At the syntax level, DDL, DML, and common functions are largely consistent.
The implications for upstream and downstream systems are straightforward: BI tools, ORM frameworks, and applications originally built against MySQL typically do not need code changes when switching to OceanBase. For teams already built on the MySQL ecosystem, this significantly reduces migration costs.
3.2 Oracle Compatibility
For systems that have traditionally used Oracle, OceanBase provides Oracle compatibility. Perhaps the most practically significant aspect is compatibility with PL/SQL stored procedures. Many traditional enterprises have encapsulated core business logic in Oracle stored procedures. If the database is not compatible, it means rewriting all that business logic with a new technology stack — high cost, long cycle, and significant risk.
OceanBase’s Oracle compatibility allows these stored procedures to be migrated almost as-is. SQL compatibility reaches 99.9%, application modification costs are very low, and the majority of existing stored procedures and business logic can be migrated smoothly.
3.3 Standard SQL Analytical Capabilities
Analytical queries are often more complex than online business queries, commonly involving large table joins, multi-level nested subqueries, window functions (such as ranking and sliding averages), and recursive CTE queries. OceanBase covers these commonly used standard SQL syntax features for analytics, enabling analysts to express complex statistical logic in familiar ways.
Building on this, OceanBase also provides materialized view capabilities. Materialized views pre-compute and store the results of complex queries, so subsequent queries can read directly from the materialized view instead of recomputing every time. OceanBase supports near-real-time and real-time materialized view refresh, as well as automatic SQL rewriting — when a query can match a materialized view, it is automatically routed to it. For high-frequency, repeated analytical queries, this can dramatically reduce response times.
By contrast, some big data platforms have limited support for standard SQL. When encountering complex joins or window functions, developers are often forced to use low-level APIs or hand-written UDFs, which raises the barrier and increases ongoing maintenance costs.
4. Database Management and Operations Ecosystem
Once data can flow in and be queried, the next requirement is ensuring the entire system can be managed stably. Management and operations capabilities determine the system’s long-term maintainability. OceanBase integrates with the common operations ecosystem across three dimensions: scheduling and orchestration, observability, and resource management.
4.1 Scheduling and Orchestration
A complete analytics task is rarely accomplished in a single step. It typically involves multiple stages — data extraction, cleansing and transformation, aggregation computation, and result writing — with dependencies between steps that need to be triggered in order or based on conditions. The role of a scheduling platform is to define these steps as a DAG (Directed Acyclic Graph) and execute them automatically according to a schedule.
OceanBase supports integration with mainstream scheduling and data flow management tools such as Airflow, DolphinScheduler, and NiFi. Airflow and DolphinScheduler excel at defining complex task dependencies and timed scheduling policies. Apache NiFi is a data flow management tool that specializes in visually defining data collection, routing, and transformation processes, suited for scenarios that require flexible orchestration of data flows. Once integrated with these platforms, analytics task orchestration can directly reuse the enterprise’s existing scheduling infrastructure without developing a separate scheduling logic layer.
4.2 Observability
Operations teams need to understand the system’s running state at all times, especially in multi-tenant environments where different business lines share a cluster and need to independently monitor their resource usage and query performance. OceanBase integrates with the Prometheus and Grafana monitoring stack, with Prometheus collecting metrics and Grafana handling visualization and alert configuration.
In addition to the open-source approach, OceanBase also supports integration with commercial APM platforms such as Datadog, meeting the needs of enterprises already using Datadog for unified monitoring and avoiding the need to maintain a separate monitoring system.
Monitoring coverage includes node CPU, memory, and disk I/O; query latency distributions; slow query statistics; and tenant-level resource consumption. With these metrics, issues can be quickly pinpointed when they arise, and the data also supports capacity planning and performance tuning on an ongoing basis.
4.3 Multi-Tenant Resource Management
When online transactions and analytics tasks run on the same cluster, resource contention is a common problem. If analytics queries consume too much CPU or I/O, online business may be affected. OceanBase provides multiple levels of resource isolation to handle this.
Tenants are logically independent database instances with independent resource quotas that do not interfere with each other. Foreground and background tasks are also isolated — background operations such as compaction and merging do not disrupt foreground user queries. At a finer granularity, OceanBase supports user-level and SQL-level resource groups, allowing specific users or specific SQL statements to be assigned dedicated resource groups for fine-grained control. For mixed TP/AP workloads, OceanBase supports strong isolation between TP/AP replicas — analytics queries can be routed to read-only replicas, physically separated from the primary replicas handling online transactions.
This way, multiple workload types can safely coexist on the same physical cluster without interfering with each other.
5. BI and Data Visualization
Analytical results ultimately need to be seen and used by business users, so the ability to connect seamlessly with BI tools directly determines the practical value of analytics capabilities. OceanBase is compatible with mainstream platforms including Superset, Grafana, Tableau, Power BI, QuickBI, and GuanData, allowing analytical results to be displayed directly in these tools. Superset and Grafana are open-source options suited for flexible customization; Tableau and Power BI are enterprise-grade business intelligence tools with rich capabilities in data modeling and interactive exploration.
The significance of self-service analytics is that business users can drag and drop fields, set filter conditions, and build reports on their own, without submitting every request to the data development team and waiting for scheduling. This shortens the cycle from asking a question to seeing the data, enabling people who understand the business to validate ideas directly, and extending analytics capabilities from the development team to the business team.
In terms of real-world results, one international retail brand previously used a combination of ADB, Deltalake, and ClickHouse, with different technology stacks for ETL and serving, resulting in lengthy integration cycles and difficult maintenance. After migrating to OceanBase, a single cluster provides both ETL and serving capabilities with MySQL compatibility, reducing overall database costs by more than 40%. Another example is a provincial mobile carrier that used external tables to read CSV, ORC, and other files directly from HDFS, then used Direct Load for real-time import, achieving a 69% overall performance improvement over the existing system.
6. AI/ML Integration
With the proliferation of large language models and AI applications, databases serve not only as the foundation for traditional analytics but are increasingly taking on data storage and retrieval needs for AI applications. Through standard interfaces and vector search capabilities, OceanBase can integrate with mainstream AI/ML ecosystem tools.
Python is the most widely used programming language in the AI/ML space. Because OceanBase is compatible with the MySQL protocol, developers can connect directly to OceanBase using common drivers such as pymysql and SQLAlchemy, performing data reads, writes, and analysis in Python environments without an additional adaptation layer.
In LLM application development, LangChain and LlamaIndex are the currently dominant frameworks. They support using OceanBase as a vector storage backend for storing text embeddings and providing similarity search capabilities. Combined with model services such as OpenAI, enterprises can build RAG (Retrieval-Augmented Generation) applications on OceanBase, enabling large models to retrieve relevant context from the database when answering questions, improving both accuracy and timeliness. OceanBase’s vector search capabilities allow structured data queries and semantic retrieval to be completed within the same database, eliminating the need to deploy a separate vector database for AI applications.
7. Data Governance
Data quality is the prerequisite for trustworthy analytical results. If source data has missing values, duplicates, or format anomalies, downstream analysis and decision-making will be affected. OceanBase is compatible with mainstream data quality governance tools, enabling enterprises to perform quality control at every stage of data ingestion, processing, and output.
Soda is a data quality monitoring tool that supports defining data quality rules (such as field non-null rates, numerical ranges, uniqueness constraints, etc.) and periodically checking data against these rules, alerting when anomalies are detected. Great Expectations is a data validation framework in the Python ecosystem, used to define and verify expected data characteristics, well suited for integration into ETL pipelines for automated validation. Data Profiler focuses on data profiling, helping understand data distributions, outliers, and statistical features, which is especially useful during initial data onboarding. These tools connect to OceanBase through standard database connections, allowing enterprises to embed data quality checks into existing data pipelines without building a separate governance system.
8. Summary
The core of ecosystem support is making data flow more smoothly, rather than forcing enterprises to adapt to the database. OceanBase’s approach is to use a unified engine to interface with the existing data ecosystem — from data ingestion, layered processing, query analytics, and visualization through to AI applications and data governance — minimizing the glue code and synchronization pipelines between systems, so that data can come in, be processed quickly, be seen clearly, and be managed effectively.
Compared to the traditional approach of stitching together multiple components, this reduces pipeline complexity and lowers the latency and consistency risks that come with overly long pipelines, making it easier for real-time analytics capabilities to truly land in enterprise production environments.
Top comments (0)