Overview
In modern data integration scenarios, seamless connectivity between cloud platforms is critical. One common requirement is integrating Oracle Integration Cloud (OIC) with Databricks for performing table-based operations.
However, a key limitation arises:
❗ Databricks does not provide native RDS-style connectivity for direct table operations from OIC.
To overcome this limitation, we implement a scalable and efficient architecture:
Proposed Solution Architecture
OIC → AWS S3 → Databricks (External Tables)
This approach leverages AWS S3 as an intermediate storage layer, allowing OIC to push data, which Databricks can then consume via external tables.
Architecture Flow
1. OIC (Oracle Integration Cloud)
- Acts as the data integration layer
- Extracts and prepares data from upstream systems
- Pushes data files (CSV/JSON/Parquet) to AWS S3
2. AWS S3 (Storage Layer)
- Serves as a staging area for data
- Stores files uploaded by OIC
- Organizes data into structured folders (e.g., by date/source)
3. Databricks (Processing Layer)
- Uses External Tables to read data directly from S3
- Processes data using Spark SQL / PySpark
- Performs transformations and analytics
Why External Tables?
Since Databricks cannot directly expose tables like traditional RDS systems:
✅ External tables allow querying data stored in S3
✅ No need to move data into Databricks-managed storage
✅ Enables scalable and cost-effective data access
Example:
CREATE TABLE customer_data
USING CSV
LOCATION 's3://your-bucket/path/'
OPTIONS (header = "true");
Required Connection Details for OIC
To enable the integration (OIC → S3), the following details are required:
📦 AWS S3 Access Details
- Bucket Name
- Region
- Folder Path (if applicable)
- Access Key ID
- Secret Access Key
🔗 Connectivity Configuration in OIC
Configure REST Adapter or FTP Adapter / Stage File Action
Use pre-signed URLs or API-based upload (if secured)
Ensure proper authentication mechanism (AWS Signature v4)
⚙️ OIC Design Reference
Step 1: Data Extraction
Fetch data from source application (DB / SaaS)
Step 2: Data Transformation
Convert to required format (CSV/JSON)
Step 3: File Creation
Use Stage File action in OIC
Step 4: Upload to S3
Use REST API or direct adapter
Push file into S3 bucket
✅ Advantages of This Approach
🔹 Decouples OIC and Databricks
🔹 Highly scalable using S3 storage
🔹 Cost-efficient (pay-as-you-go)
🔹 Supports batch and near real-time processing
🔹 Simplifies integration architecture
⚠️ Key Considerations
Ensure data format compatibility (CSV, Parquet preferred)
Maintain naming conventions in S3
Handle data partitioning for performance
Secure credentials using OCI Vault / AWS IAM policies
Monitor file upload success and retries in OIC
🎯 Conclusion
Due to the absence of native RDS-style table connectivity in Databricks, using an intermediary layer like AWS S3 is the most effective integration pattern.
The combination of:
OIC for orchestration
S3 for storage
Databricks external tables for processing
provides a robust, scalable, and cloud-native solution.
Top comments (0)