Fine-Tuning and Inference in LLMs: From Custom Models to Production Deployment

As organizations move beyond using pre-trained models, two critical concepts become essential: fine-tuning and inference. Fine-tuning allows you to customize models for specific domains, while inference enables you to deploy these models to serve predictions in production. This guide explores both processes, with a focus on enterprise platforms like Oracle Cloud Infrastructure (OCI).

Understanding Fine-Tuning

Fine-tuning is the process of taking a pretrained foundational model and providing additional training using custom data to optimize the model for specific tasks, domains, or organizational needs.

Why Fine-Tune?

While pre-trained models possess broad knowledge, they often need domain-specific adaptation for optimal performance. Fine-tuning addresses several key challenges:

1. Domain Specialization: Adapting models to industry-specific terminology and patterns (medical, legal, financial)
2. Instruction Following: Teaching models to follow specific prompt formats or instruction styles
3. Task Optimization: Improving performance on particular tasks like classification, summarization, or question-answering
4. Brand Voice: Aligning outputs with organizational tone and style
5. Accuracy Improvement: Enhancing correctness for specialized knowledge domains

The Fine-Tuning Process

The typical workflow for creating a custom model involves:

Step 1: Create a Dedicated AI Cluster
On cloud platforms like Oracle Cloud, you provision GPU-based compute resources exclusively for training workloads. These dedicated clusters provide:

• Isolated infrastructure for proprietary data
• Appropriate GPU resources sized for model complexity
• Full control over training environment

Step 2: Gather and Prepare Training Data
Training data must be formatted correctly, typically as JSONL (JSON Lines) files where each line contains a separate JSON object with prompt-completion pairs:

{"prompt": "<first prompt>", "completion": "<expected completion>"}
{"prompt": "<second prompt>", "completion": "<expected completion>"}

Data Requirements:

• Quality over quantity—clean, relevant examples matter more than volume
• Consistent formatting across all examples
• Representative coverage of use cases

Step 3: Configure and Start Fine-Tuning
Select hyperparameters including:

• Number of training epochs
• Learning rate
• Batch size
• Early stopping criteria
• Fine-tuning method (T-Few, Vanilla, LoRA)

Step 4: Monitor and Validate
During training, monitor metrics like:

• Loss (should decrease over epochs)
• Accuracy (proportion of correct completions)
• Validation performance
• Overfitting indicators

Step 5: Deploy the Fine-Tuned Model
Once training completes, the custom model is ready for deployment to an inference endpoint.

Fine-Tuning Methods: Understanding the Approaches

Different fine-tuning methods offer trade-offs between efficiency, performance, and resource requirements.

T-Few: Parameter-Efficient for Small Datasets

T-Few (Training Few) is a parameter-efficient fine-tuning method that selectively updates only a fraction of the model weights, making it computationally less expensive than full fine-tuning.

Key Characteristics:

• Recommended for small datasets (<100,000 samples)
• Typical use case: fine-tuning base model to follow different prompt formats or instructions
• Reduces training costs significantly
• Lower inference costs since fewer parameters change
• Faster training time

When to Use T-Few:

• Limited training data available
• Need to adjust instruction-following behavior
• Want to experiment with multiple fine-tuned versions
• Budget constraints on compute resources

Important Note: T-Few reduces inference costs because it's computationally less expensive—only a subset of parameters are modified, requiring less processing power during predictions.

Vanilla Fine-Tuning: Deep Semantic Understanding

Vanilla fine-tuning updates more model parameters for deeper adaptation.

Key Characteristics:

• Recommended for large datasets (>100,000-1M+ samples)
• Applied for complicated semantic understanding improvement (e.g., enhancing model understanding about a topic)
• More comprehensive model adaptation
• Higher computational requirements

When to Use Vanilla:

• Large, high-quality training datasets available
• Need deep semantic understanding improvements
• Performance is priority over efficiency
• Domain requires substantial knowledge injection

Warning: Using small datasets for Vanilla method might cause overfitting—the model gives great results for training data but can't generalize to unseen data.

For Vanilla fine-tuning on Cohere models, you can specify the number of layers to optimize, providing granular control over the adaptation process.

LoRA: Low-Rank Adaptation

LoRA (Low-Rank Adaptation) has become the most popular parameter-efficient fine-tuning method, striking an optimal balance between quality and efficiency.

How LoRA Works:

LoRA freezes pre-trained model weights and introduces trainable low-rank decomposition matrices into transformer layers. Instead of updating billions of parameters, LoRA trains small adapter matrices representing only 1-5% of original parameters.

The method makes fine-tuning large models more efficient by adding smaller matrices that transform inputs and outputs rather than updating all original parameters.

Key Advantages:

• Dramatically reduces memory requirements while preserving base model capabilities
• Can even outperform full fine-tuning by avoiding catastrophic forgetting
• Works on single GPUs for most models
• Ideal for experimentation and multi-adapter workflows
• LoRA adapter is typically just a few megabytes while pretrained base model is several gigabytes

LoRA Hyperparameters:

Key LoRA parameters include: lora_r (rank), lora_alpha (scaling), lora_dropout, lora_target_linear, and lora_target_modules.

Deployment Considerations:

During inference, both the adapter and pretrained LLM need to be loaded, so memory requirement remains similar. If weights aren't merged, there's slight increase in inference latency.

The PEFT library allows merging adapter weights with base model in a single line of code, eliminating latency overhead.

QLoRA: Maximum Memory Efficiency

QLoRA takes LoRA further by adding 4-bit quantization of the base model.

QLoRA combines LoRA adapters with 4-bit quantization—frozen weights stored in 4-bit precision while LoRA adapters train in higher precision, then gradients backpropagate through quantized model.

Advantages:

• Enables large model fine-tuning on consumer hardware
• Maximum memory efficiency
• Achieves efficiency to tune very large models using 4-bit quantization

Trade-offs:

• Slight accuracy reduction compared to LoRA
• More complex setup
• Best for memory-constrained environments

Choosing Your Fine-Tuning Method

The LoRA vs full fine-tuning vs QLoRA decision depends on GPU budget, accuracy requirements, and deployment strategy:

Full Fine-Tuning:

• Updates every parameter
• Highest possible task-specific accuracy
• Requires multi-GPU clusters (A100/H100)
• Significantly more training time

LoRA:

• Balance of quality and efficiency
• Works on single GPUs
• Ideal for experimentation and multi-adapter workflows

QLoRA:

• Maximum memory efficiency
• Enables large model fine-tuning on limited hardware
• Slight accuracy trade-off

Production Strategy: Many production teams use LoRA for experimentation, then full fine-tune the winning configuration for maximum production accuracy.

Understanding Inference

Inference refers to the process of using a trained machine learning model to make predictions or decisions based on new input data. Once a model is trained (or fine-tuned), inference is how it generates outputs in production.

The Inference Pipeline

1. Input Processing: User request is received and preprocessed
2. Model Loading: Trained model (and adapter if using LoRA) loaded into memory
3. Forward Pass: Input processed through model layers
4. Output Generation: Model produces predictions/text
5. Post-Processing: Output formatted and returned to user

Real-Time vs Batch Inference

Real-Time Inference:

• Designed for synchronous low-latency requests—when you invoke endpoint, results returned in endpoint's response
• Suitable for chatbots, interactive applications
• Requires consistent availability
• Lower latency requirements (milliseconds to seconds)

Batch Inference:

• Designed for long-running batch processing—when you invoke batch endpoint, you generate batch job that performs actual work
• Processes large volumes of data asynchronously
• Optimized for throughput over latency
• Cost-effective for non-time-sensitive workloads

Model Endpoints: Serving Your Models

A model endpoint is a designated point on a dedicated AI cluster where an LLM can accept user requests and send back responses, such as model-generated text.

Endpoint Fundamentals

When you deploy an LLM, you make it available for use in a website, application, or other production environment. Deployment involves hosting the model on a server or in cloud and creating an API or interface for users to interact with the model.

Think of an endpoint as a container that can house multiple model deployments. Endpoint names need to be unique in a region.

Creating Endpoints: The Process

Step 1: Create a Dedicated AI Cluster for Hosting
Unlike fine-tuning clusters, hosting clusters are optimized for inference workloads:

• Stable, high-throughput performance
• Private GPUs ensuring data never leaves your environment
• Zero-downtime scaling capabilities
• Multiple cluster sizes for different model requirements

Step 2: Create the Endpoint
Configure endpoint settings:

• Name and region
• Authentication method (typically API keys)
• Rate limiting and quotas
• Content moderation filters (optional)
• Monitoring and logging preferences

Step 3: Deploy Model to Endpoint
Attach your fine-tuned model (or pretrained model) to the endpoint:

• Select model version
• Configure instance count
• Set autoscaling rules
• Define health check parameters

Step 4: Test and Monitor

• Validate endpoint functionality
• Monitor latency and throughput
• Track error rates
• Optimize based on usage patterns

Endpoint Capabilities and Limits

OCI Generative AI Endpoint Rules:

A hosting dedicated AI cluster can have up to 50 endpoints, enabling diverse use cases:

• Host multiple model versions for A/B testing
• Host several versions of custom model on one cluster (applies to Cohere Command models fine-tuned with T-Few method)
• Deploy different models for different applications
• Separate development/staging/production endpoints

Capacity Management:

Monitor remaining capacity on hosting endpoints to ensure adequate resources. To increase call volume supported by hosting cluster, you can increase its instance count.

Cluster Types in OCI Generative AI

OCI provides two distinct cluster types optimized for different workloads:

Fine-Tuning Clusters

Purpose: Training pretrained foundational models with custom data

Characteristics:

• GPU resources sized for training workloads
• Optimized for compute-intensive operations
• Temporary usage during training jobs
• Support for T-Few, Vanilla, and LoRA methods

Cluster Unit Types:
Different sizes available based on model and training method requirements.

Hosting Clusters

Purpose: Hosting custom model endpoints for inference

Characteristics:

• Optimized for low-latency serving
• Stable, predictable performance
• Persistent availability for production workloads
• Support for multiple concurrent endpoints

Cluster Unit Types:

• Small Cohere/Generic units: For smaller models and lower throughput
• Large Generic units: For 70B parameter models
• Large Generic 2/4 units: For massive 405B parameter models

Best Practices for Production Deployments

Fine-Tuning Best Practices

1. Start with Quality Data: Clean, representative training data matters more than volume
2. Choose Appropriate Method: Match fine-tuning strategy to dataset size and goals
3. Monitor Training Metrics: Watch for overfitting and convergence issues
4. Validate Thoroughly: Test on held-out validation sets before deployment
5. Version Control: Track model versions, datasets, and hyperparameters
6. Budget Wisely: LoRA for experiments, full fine-tune for production if needed

Inference Best Practices

1. Right-Size Clusters: Match compute resources to expected load
2. Implement Monitoring: Track latency, throughput, error rates, costs
3. Enable Autoscaling: Handle traffic spikes without over-provisioning
4. Use Load Balancing: Distribute requests across multiple instances
5. Cache When Possible: Store responses for common queries
6. Implement Retries: Handle transient failures gracefully
7. Set Rate Limits: Protect endpoints from abuse and control costs

Security Considerations

1. Authentication: Use API keys or OAuth for endpoint access
2. Encryption: Ensure data encrypted in transit and at rest
3. Access Control: Implement role-based access control (RBAC)
4. Audit Logging: Track all inference requests for compliance
5. Content Moderation: Filter inappropriate inputs/outputs
6. Data Isolation: Keep training data and inference separate

Cost Optimization Strategies

Fine-Tuning Costs

• Use Parameter-Efficient Methods: LoRA and T-Few reduce compute requirements
• Right-Size Training Data: More data isn't always better
• Optimize Hyperparameters: Fewer epochs if model converges quickly
• Share Clusters: Multiple fine-tuning jobs on same cluster infrastructure

Inference Costs

• Serverless for Variable Load: Pay only for actual usage
• Dedicated for Consistent Load: More cost-effective at scale
• Batch Processing: Use batch inference for non-urgent workloads
• Model Optimization: Quantization and pruning reduce compute needs
• Request Batching: Process multiple requests together for efficiency

The Future of Fine-Tuning and Inference

Emerging Trends

Multi-Adapter Systems:
One advantage of adapter pattern is ability to deploy single large pretrained model with task-specific adapters—enabling efficient multi-tenancy and personalization.

Distributed Inference:
Prefill/Decode Disaggregation reduces time to first token (TTFT) and gets more predictable time per output token (TPOT) by splitting inference into prefill servers handling prompts and decode servers handling responses.

Expert Parallelism:
Deploy very large Mixture-of-Experts (MoE) models like DeepSeek-R1 and significantly reduce end-to-end latency by scaling with Data Parallelism and Expert Parallelism.

Hardware Diversity:
Support expanding beyond NVIDIA GPUs to AMD GPUs, Google TPUs, Intel XPUs, and other accelerators.

Conclusion

Fine-tuning and inference represent the critical bridge from research to production for LLMs. Understanding these processes enables organizations to:

• Customize models for specific domains and use cases
• Deploy efficiently using parameter-efficient methods like LoRA and T-Few
• Serve at scale through dedicated endpoints and clusters
• Optimize costs by matching methods to requirements
• Maintain security through proper isolation and access controls

Key takeaways:

• T-Few: Best for small datasets (<100K samples) and instruction adaptation
• LoRA: Optimal balance of efficiency and performance for most use cases
• Vanilla: For large datasets requiring deep semantic understanding
• Endpoints: Container for deployments, supporting up to 50 per cluster on OCI
• Cluster types: Separate infrastructure for fine-tuning (training) and hosting (inference)
• Cost optimization: Match deployment strategy to workload characteristics

Whether building chatbots, implementing domain-specific assistants, or creating production AI applications, mastering fine-tuning and inference is essential for success with LLMs.