Automation of project migration with Bob (yes Bob again)!
TLDR; What is watsonx.ai?
In the rapidly evolving world of Generative AI, organizations are moving past simple experimentation and toward full-scale production. IBM watsonx.ai is the enterprise-grade studio designed specifically for this transition. As a core component of the broader watsonx platform, it provides a unified workspace where data scientists and developers can build, train, tune, and deploy both traditional machine learning and cutting-edge generative AI models.
Unlike consumer-facing AI tools, watsonx.ai is built with the “AI builder” in mind, offering a robust environment that balances open-source flexibility with enterprise-level security.
The Core Pillars of watsonx.ai
- The Foundation Model Library: Access a curated selection of IBM’s own Granite models, alongside popular open-source models from the Hugging Face community (like Llama and Mistral).
- The Prompt Lab: A specialized sandbox for prompt engineering, allowing users to experiment with zero-shot and few-shot learning to refine model outputs without deep coding.
- The Tuning Studio: A powerful environment for fine-tuning foundation models on proprietary enterprise data, ensuring the AI understands your specific business context and terminology.
- Full AI Lifecycle Management: From initial data preparation to model monitoring and deployment, it provides the tools needed to manage the entire “ModelOps” pipeline in one place.
What is a project in watsonx.ai?
In watsonx.ai, projects serve as central, collaborative workspaces where teams organize resources and manage the end-to-end lifecycle of AI and machine learning solutions. These projects house a variety of data assets, ranging from direct file uploads to remote connections with databases, as well as operational assets like Jupyter Notebooks, Python scripts, and SPSS Modeler flows. Beyond traditional machine learning, projects are the primary hub for generative AI work, containing Prompt Lab experiments, Tuning Studio sessions, and AI Agent configurations. By integrating these components with shared collaborators, compute resources, and versioning, watsonx.ai projects ensure that data scientists and engineers can transition seamlessly from data preparation and model tuning to full-scale deployment within a unified, governed environment.
Key Components of a watsonx.ai Project
To give your readers a clear breakdown, here are the essential elements found within a project:
- Data Assets: Local files (CSV, JSON), connected data from external clouds (S3, Azure, Snowflake), and refined data products.
- Generative AI Tools: Saved prompts from the Prompt Lab and custom model tuning experiments from the Tuning Studio.
- Operational Assets: Integrated Jupyter Notebooks (Python/R), automated machine learning pipelines (AutoAI), and model metadata.
- Environments & Runtimes: The underlying compute power (CPU/GPU) and software specifications used to run your code and train models.
- Collaborators: Access control settings that allow you to manage permissions for data scientists, engineers, and viewers.
In addition to the tools and notebooks, every watsonx.ai project is underpinned by two critical infrastructure components: IBM Cloud Object Storage (COS) and the Machine Learning (ML) Service.
The Backbone: Cloud Object Storage (COS)
Think of Cloud Object Storage as the project’s “hard drive.” Every time you create a project, it is automatically linked to a dedicated COS bucket.
- Role: It stores all your physical files, including uploaded CSVs, saved model files, and Jupyter Notebook .ipynb files.
- Importance for Automation: When you automate an export, the system essentially packages these files from the bucket. To automate “importing” into a new project, the assets must be programmatically transferred and registered into the new project’s specific COS bucket.
The Brain: watsonx.ai Runtime (formerly Machine Learning Service)
The ML Service (now part of the watsonx.ai Runtime) acts as the project’s “engine.”
- Role: It provides the compute power and the API framework needed to train traditional models, host deployment spaces, and run your automation scripts.
- Project Element: Within a project, this manifests as “Software Specifications” and “Runtimes” which define the Python version and libraries (like
scikit-learnorpytorch) to execute your work.
Project Isolation
In watsonx.ai, projects are strictly isolated environments by design, ensuring that work within one project remains invisible and inaccessible to any other. This logical separation is a cornerstone of enterprise-grade security, primarily achieved through multi-tenant architecture where each project is anchored to its own dedicated IBM Cloud Object Storage (COS) bucket. This ensures that data assets, model weights, and prompt configurations are physically stored in discrete locations. Furthermore, isolation is enforced through Identity and Access Management (IAM) and Role-Based Access Control (RBAC), which require users to be explicitly added as collaborators to each specific project; even users within the same organization cannot browse or access a project unless granted permission. This boundary also extends to compute resources, where runtimes and hardware allocations are scoped to the individual project level, preventing “noisy neighbor” effects and ensuring that a heavy training job in one project does not compromise the performance or security of another.
Key Reasons for Project Isolation
| Reason | Explanation |
| --------------------- | ------------------------------------------------------------ |
| Data Privacy | Sensitive training data and proprietary prompts are stored in a project-specific bucket, ensuring no data leakage between different teams or departments. |
| Security Compliance | Isolation allows organizations to meet regulatory standards (like SOC2 or HIPAA) by maintaining clear boundaries and audit trails for who accessed what data. |
| Environment Integrity | Each project can have its own software specifications and library versions, preventing conflicts where an update in one project breaks the code in another. |
| Governance | By isolating projects, you can apply different lifecycle policies—such as stricter monitoring or approvals for a "Production" project versus a "Sandbox" project. |
Back to Essentials…🎒
Having established the architecture and the importance of project isolation, let’s return to the core mission of this guide. While watsonx.ai is designed for enterprise scale, the “human element” often necessitates a more programmatic approach.
In my experience, the drive to automate the import and export process usually stems from two primary challenges:
- Bridging the Technical Gap: Often, project stakeholders or interlocutors have the necessary platform access but lack the “hands-on” familiarity with the UI. Automation allows us to deliver results or move assets without requiring every team member to navigate complex menu structures.
- Scaling Project Migrations: While moving one project manually is a minor task, moving ten, twenty, or fifty projects across different regions or accounts becomes a significant bottleneck. Automation transforms a repetitive, multi-day chore into a single, repeatable script.
The “Easy Way”: Manual Migration
It is worth noting that for a one-off task, the manual process in watsonx.ai is remarkably straightforward. By using the “Export” feature within the project UI, the platform packages all your selected assets into a single ZIP file, which can then be uploaded into a new project environment.
The “Automated way”: and God bless Bob 😁
As readers of my previous posts will remember, we have a reliable partner in “Bob,” our resident automation expert here at IBM. I tasked him with architecting a fully automated, end-to-end procedure for importing and exporting watsonx.ai projects to eliminate the manual overhead we discussed.
Bob certainly delivered. What follows is a professional, streamlined workflow that handles the heavy lifting programmatically, ensuring that your project migrations are consistent, secure, and — most importantly — fast. Hereafter is the process of automation ⬇️
- Create an “.env” file with all the necessary credentials/token and information.
# ============================================
# WATSONX.AI PROJECT MIGRATION
# ============================================
# IBM Cloud API Key
WXAI_API_KEY=your-api-key-here
# Project ID
WXAI_PROJECT_ID=your-project-id-here
# watsonx.ai instance URL (optional, defaults to us-south)
# Options:
# - https://us-south.ml.cloud.ibm.com (US South)
# - https://eu-de.ml.cloud.ibm.com (EU Germany)
# - https://eu-gb.ml.cloud.ibm.com (EU United Kingdom)
# - https://jp-tok.ml.cloud.ibm.com (Japan Tokyo)
WXAI_URL=https://us-south.ml.cloud.ibm.com
# ============================================
# OPTIONAL SETTINGS
# ============================================
# Poll interval in seconds (default: 5)
# WXAI_POLL_INTERVAL=5
# Maximum wait time in seconds (default: 3600)
# WXAI_MAX_WAIT_TIME=3600
# ============================================
# MIGRATION WORKFLOWS
# ============================================
# watsonx.ai Projects:
# 1. For EXPORT: Set WXAI_* credentials for source instance
# 2. Run: python wxai_export.py
# 3. For IMPORT: Update WXAI_* credentials for target instance
# 4. Run: python wxai_import.py --file <export-file>.zip
- Install the requirements for the Python code;
# Watson AI Migration Tools Requirements
# Install with: pip install -r requirements.txt
#
# Note: If you encounter compilation errors with Python 3.14+,
# please use Python 3.11 or 3.12 for better compatibility.
# IBM Watson AI SDK (for watsonx.ai project migration)
ibm-watsonx-ai>=1.0.0,<2.0.0
# HTTP requests library (for Watson Orchestrate agent migration)
requests>=2.31.0,<3.0.0
# Environment variable management
python-dotenv>=1.0.0,<2.0.0
# Optional: For enhanced logging and formatting
colorlog>=6.7.0,<7.0.0
Works with Python < v14!
brew install python@3.12
python3.12 -m venv venv
pip install --upgrade pip
source venv/bin/activate
pip install -r requirements.txt
- The “Export” code 📤
#!/usr/bin/env python3
"""
Watson AI Project Export Tool
Exports projects from watsonx.ai instance using IBM API key and project ID.
"""
import os
import sys
import time
import argparse
import logging
from datetime import datetime
from pathlib import Path
from dotenv import load_dotenv
from ibm_watsonx_ai import APIClient
# Load environment variables from .env file
load_dotenv()
# Configure logging
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(levelname)s - %(message)s',
handlers=[
logging.FileHandler('wxai_export.log'),
logging.StreamHandler(sys.stdout)
]
)
logger = logging.getLogger(__name__)
class WatsonXExporter:
"""Handles exporting projects from watsonx.ai"""
def __init__(self, api_key: str, project_id: str, url: str = "https://us-south.ml.cloud.ibm.com"):
"""
Initialize the exporter.
Args:
api_key: IBM Cloud API key
project_id: Source project ID to export
url: watsonx.ai instance URL (default: us-south)
"""
self.api_key = api_key
self.project_id = project_id
self.url = url
self.client = None
def connect(self):
"""Establish connection to watsonx.ai"""
try:
credentials = {
"url": self.url,
"apikey": self.api_key
}
self.client = APIClient(credentials)
logger.info(f"Successfully connected to watsonx.ai at {self.url}")
return True
except Exception as e:
logger.error(f"Failed to connect to watsonx.ai: {str(e)}")
return False
def export_project(self, export_name: str = None, output_path: str = None,
all_assets: bool = True, poll_interval: int = 5,
max_wait_time: int = 3600):
"""
Export a project from watsonx.ai.
Args:
export_name: Name for the export (default: auto-generated with timestamp)
output_path: Path where to save the export ZIP file
all_assets: Export all assets (True) or specific assets (False)
poll_interval: Seconds between status checks (default: 5)
max_wait_time: Maximum time to wait for export completion in seconds (default: 3600)
Returns:
str: Path to the exported ZIP file, or None if failed
"""
if not self.client:
logger.error("Not connected. Call connect() first.")
return None
try:
# Generate export name with timestamp if not provided
if not export_name:
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
export_name = f"wxai_export_{timestamp}"
# Generate output path if not provided
if not output_path:
output_path = f"{export_name}.zip"
logger.info(f"Starting export of project {self.project_id}")
logger.info(f"Export name: {export_name}")
# Create export payload - use the client's ConfigurationMetaNames
export_payload = {
self.client.export_assets.ConfigurationMetaNames.NAME: export_name,
self.client.export_assets.ConfigurationMetaNames.ALL_ASSETS: all_assets
}
# Start the export
export_details = self.client.export_assets.start(
project_id=self.project_id,
meta_props=export_payload
)
export_id = export_details['metadata']['id']
logger.info(f"Export initiated with ID: {export_id}")
# Poll for completion
start_time = time.time()
while True:
elapsed_time = time.time() - start_time
if elapsed_time > max_wait_time:
logger.error(f"Export timed out after {max_wait_time} seconds")
return None
status_details = self.client.export_assets.get_details(
self.project_id,
export_id
)
status = status_details['entity']['status']['state']
logger.info(f"Export status: {status} (elapsed: {int(elapsed_time)}s)")
if status == 'completed':
logger.info("Export completed successfully")
break
elif status == 'failed':
error_msg = status_details.get('entity', {}).get('status', {}).get('message', 'Unknown error')
logger.error(f"Export failed: {error_msg}")
return None
time.sleep(poll_interval)
# Download the exported content
logger.info(f"Downloading export to {output_path}")
self.client.export_assets.get_exported_content(
self.project_id,
export_id,
file_path=output_path
)
# Verify file was created
if os.path.exists(output_path):
file_size = os.path.getsize(output_path)
logger.info(f"Export successful! File saved: {output_path} ({file_size} bytes)")
return output_path
else:
logger.error("Export file was not created")
return None
except Exception as e:
logger.error(f"Export failed with error: {str(e)}")
return None
def main():
"""Main entry point for the export tool"""
parser = argparse.ArgumentParser(
description='Export projects from watsonx.ai',
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog="""
Examples:
# Export using environment variables
export WXAI_API_KEY="your-api-key"
export WXAI_PROJECT_ID="your-project-id"
python wxai_export.py
# Export with command line arguments
python wxai_export.py --api-key YOUR_KEY --project-id YOUR_PROJECT_ID
# Export to specific file with custom URL
python wxai_export.py --api-key YOUR_KEY --project-id YOUR_PROJECT_ID \\
--output my_export.zip --url https://eu-de.ml.cloud.ibm.com
"""
)
parser.add_argument(
'--api-key',
help='IBM Cloud API key (or set WXAI_API_KEY env var)',
default=os.getenv('WXAI_API_KEY')
)
parser.add_argument(
'--project-id',
help='Source project ID to export (or set WXAI_PROJECT_ID env var)',
default=os.getenv('WXAI_PROJECT_ID')
)
parser.add_argument(
'--url',
help='watsonx.ai instance URL (default: us-south)',
default=os.getenv('WXAI_URL', 'https://us-south.ml.cloud.ibm.com')
)
parser.add_argument(
'--output',
help='Output ZIP file path (default: auto-generated)',
default=None
)
parser.add_argument(
'--export-name',
help='Name for the export (default: auto-generated with timestamp)',
default=None
)
parser.add_argument(
'--poll-interval',
type=int,
help='Seconds between status checks (default: 5)',
default=5
)
parser.add_argument(
'--max-wait-time',
type=int,
help='Maximum time to wait for export in seconds (default: 3600)',
default=3600
)
args = parser.parse_args()
# Validate required arguments
if not args.api_key:
logger.error("API key is required. Provide via --api-key or WXAI_API_KEY environment variable")
sys.exit(1)
if not args.project_id:
logger.error("Project ID is required. Provide via --project-id or WXAI_PROJECT_ID environment variable")
sys.exit(1)
# Create exporter and run
exporter = WatsonXExporter(
api_key=args.api_key,
project_id=args.project_id,
url=args.url
)
if not exporter.connect():
logger.error("Failed to connect to watsonx.ai")
sys.exit(1)
result = exporter.export_project(
export_name=args.export_name,
output_path=args.output,
poll_interval=args.poll_interval,
max_wait_time=args.max_wait_time
)
if result:
logger.info(f"Export completed successfully: {result}")
sys.exit(0)
else:
logger.error("Export failed")
sys.exit(1)
if __name__ == "__main__":
main()
# Made with Bob
- And the import 📩
#!/usr/bin/env python3
"""
Watson AI Project Import Tool
Imports projects to watsonx.ai instance using IBM API key and project ID.
"""
import os
import sys
import time
import argparse
import logging
from datetime import datetime
from pathlib import Path
from dotenv import load_dotenv
from ibm_watsonx_ai import APIClient
# Load environment variables from .env file
load_dotenv()
# Configure logging
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(levelname)s - %(message)s',
handlers=[
logging.FileHandler('wxai_import.log'),
logging.StreamHandler(sys.stdout)
]
)
logger = logging.getLogger(__name__)
class WatsonXImporter:
"""Handles importing projects to watsonx.ai"""
def __init__(self, api_key: str, project_id: str, url: str = "https://us-south.ml.cloud.ibm.com"):
"""
Initialize the importer.
Args:
api_key: IBM Cloud API key
project_id: Target project ID to import into
url: watsonx.ai instance URL (default: us-south)
"""
self.api_key = api_key
self.project_id = project_id
self.url = url
self.client = None
def connect(self):
"""Establish connection to watsonx.ai"""
try:
credentials = {
"url": self.url,
"apikey": self.api_key
}
self.client = APIClient(credentials)
logger.info(f"Successfully connected to watsonx.ai at {self.url}")
return True
except Exception as e:
logger.error(f"Failed to connect to watsonx.ai: {str(e)}")
return False
def import_project(self, import_file: str, poll_interval: int = 5,
max_wait_time: int = 3600):
"""
Import a project to watsonx.ai.
Args:
import_file: Path to the ZIP file to import
poll_interval: Seconds between status checks (default: 5)
max_wait_time: Maximum time to wait for import completion in seconds (default: 3600)
Returns:
dict: Import details if successful, None if failed
"""
if not self.client:
logger.error("Not connected. Call connect() first.")
return None
# Verify import file exists
if not os.path.exists(import_file):
logger.error(f"Import file not found: {import_file}")
return None
file_size = os.path.getsize(import_file)
logger.info(f"Import file: {import_file} ({file_size} bytes)")
try:
logger.info(f"Starting import to project {self.project_id}")
# Start the import
import_details = self.client.import_assets.start(
project_id=self.project_id,
file_path=import_file
)
import_id = import_details['metadata']['id']
logger.info(f"Import initiated with ID: {import_id}")
# Poll for completion
start_time = time.time()
while True:
elapsed_time = time.time() - start_time
if elapsed_time > max_wait_time:
logger.error(f"Import timed out after {max_wait_time} seconds")
return None
try:
status_details = self.client.import_assets.get_details(
self.project_id,
import_id
)
status = status_details['entity']['status']['state']
logger.info(f"Import status: {status} (elapsed: {int(elapsed_time)}s)")
if status == 'completed':
logger.info("Import completed successfully")
# Log summary of imported assets
if 'entity' in status_details and 'import_summary' in status_details['entity']:
summary = status_details['entity']['import_summary']
logger.info("Import Summary:")
for key, value in summary.items():
logger.info(f" {key}: {value}")
return status_details
elif status == 'failed':
error_msg = status_details.get('entity', {}).get('status', {}).get('message', 'Unknown error')
logger.error(f"Import failed: {error_msg}")
# Log any error details
if 'entity' in status_details and 'errors' in status_details['entity']:
errors = status_details['entity']['errors']
logger.error(f"Error details: {errors}")
return None
except Exception as e:
logger.warning(f"Error checking status: {str(e)}")
time.sleep(poll_interval)
except Exception as e:
logger.error(f"Import failed with error: {str(e)}")
return None
def validate_import_file(self, import_file: str):
"""
Validate the import file before attempting import.
Args:
import_file: Path to the ZIP file to validate
Returns:
bool: True if valid, False otherwise
"""
if not os.path.exists(import_file):
logger.error(f"File does not exist: {import_file}")
return False
if not import_file.lower().endswith('.zip'):
logger.error(f"File must be a ZIP file: {import_file}")
return False
file_size = os.path.getsize(import_file)
if file_size == 0:
logger.error(f"File is empty: {import_file}")
return False
logger.info(f"Import file validation passed: {import_file} ({file_size} bytes)")
return True
def main():
"""Main entry point for the import tool"""
parser = argparse.ArgumentParser(
description='Import projects to watsonx.ai',
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog="""
Examples:
# Import using environment variables
export WXAI_API_KEY="your-api-key"
export WXAI_PROJECT_ID="your-project-id"
python wxai_import.py --file migration_package.zip
# Import with command line arguments
python wxai_import.py --api-key YOUR_KEY --project-id YOUR_PROJECT_ID \\
--file migration_package.zip
# Import to different region
python wxai_import.py --api-key YOUR_KEY --project-id YOUR_PROJECT_ID \\
--file my_export.zip --url https://eu-de.ml.cloud.ibm.com
"""
)
parser.add_argument(
'--api-key',
help='IBM Cloud API key (or set WXAI_API_KEY env var)',
default=os.getenv('WXAI_API_KEY')
)
parser.add_argument(
'--project-id',
help='Target project ID to import into (or set WXAI_PROJECT_ID env var)',
default=os.getenv('WXAI_PROJECT_ID')
)
parser.add_argument(
'--url',
help='watsonx.ai instance URL (default: us-south)',
default=os.getenv('WXAI_URL', 'https://us-south.ml.cloud.ibm.com')
)
parser.add_argument(
'--file',
required=True,
help='Path to the ZIP file to import'
)
parser.add_argument(
'--poll-interval',
type=int,
help='Seconds between status checks (default: 5)',
default=5
)
parser.add_argument(
'--max-wait-time',
type=int,
help='Maximum time to wait for import in seconds (default: 3600)',
default=3600
)
parser.add_argument(
'--skip-validation',
action='store_true',
help='Skip import file validation'
)
args = parser.parse_args()
# Validate required arguments
if not args.api_key:
logger.error("API key is required. Provide via --api-key or WXAI_API_KEY environment variable")
sys.exit(1)
if not args.project_id:
logger.error("Project ID is required. Provide via --project-id or WXAI_PROJECT_ID environment variable")
sys.exit(1)
# Create importer
importer = WatsonXImporter(
api_key=args.api_key,
project_id=args.project_id,
url=args.url
)
# Validate import file
if not args.skip_validation:
if not importer.validate_import_file(args.file):
logger.error("Import file validation failed")
sys.exit(1)
# Connect and import
if not importer.connect():
logger.error("Failed to connect to watsonx.ai")
sys.exit(1)
result = importer.import_project(
import_file=args.file,
poll_interval=args.poll_interval,
max_wait_time=args.max_wait_time
)
if result:
logger.info("Import completed successfully")
sys.exit(0)
else:
logger.error("Import failed")
sys.exit(1)
if __name__ == "__main__":
main()
# Made with Bob
Et voilà 🎯
Conclusion
Automating the movement of watsonx.ai projects is more than just a technical convenience; it is a fundamental step toward achieving true ModelOps at scale. By moving away from manual UI interactions and leveraging the power of “Bob’s” automated procedures, you eliminate the risk of human error, save valuable engineering time, and ensure that even non-technical stakeholders can benefit from seamless project migrations. As your AI initiatives grow from single experiments into enterprise-wide deployments, these automation scripts will serve as the essential bridge across your isolated environments. Now that you have the tools and the code to streamline your workflow, you can spend less time managing files and more time building the next generation of AI solutions.
Thanks for reading, if you want more 'Bob-approved' automation tips for subscribe to the blog to stay updated on the latest posts.
Links
- watsonx.ai: https://www.ibm.com/products/watsonx-ai
- watsonx.ai Samples: https://github.com/IBM/watsonx-ai-samples
- Project Bob: https://www.ibm.com/products/bob
Top comments (0)