DEV Community: Paulo Torres

Building Scalable E-commerce Systems: A Journey Through Microservices with ASP.NET Core and Kubernetes

Paulo Torres — Mon, 10 Feb 2025 18:58:22 +0000

The Challenge

Our monolithic application served us well initially, handling about 10,000 daily users. However, as our user base grew to 100,000+ daily active users, we started experiencing several issues:

Deployment bottlenecks where updating a single feature required deploying the entire application
Scaling difficulties when different components needed different resources
Development team conflicts as multiple teams worked on the same codebase
Difficulty implementing new technologies due to tight coupling

Our Microservices Solution

Architecture Overview

We broke down our monolith into these core microservices:

Product Catalog Service
- Handles product information, categories, and search
- Built with ASP.NET Core 8.0
- Uses MongoDB for flexible product attributes
- Implements Elasticsearch for full-text search
Order Processing Service
- Manages order lifecycle and payment processing
- Uses ASP.NET Core minimal APIs
- SQL Server with Entity Framework Core
- Implements Saga pattern for distributed transactions
Inventory Service
- Real-time inventory management
- Event-driven architecture using Azure Service Bus
- Redis caching for high-performance stock checks
User Management Service
- Handles authentication and user profiles
- IdentityServer4 integration
- PostgreSQL database

Infrastructure Implementation

# Example Kubernetes deployment for Product Catalog Service
apiVersion: apps/v1
kind: Deployment
metadata:
  name: product-catalog
spec:
  replicas: 3
  selector:
    matchLabels:
      app: product-catalog
  template:
    metadata:
      labels:
        app: product-catalog
    spec:
      containers:
      - name: product-catalog
        image: techmart/product-catalog:1.0
        resources:
          requests:
            memory: "256Mi"
            cpu: "200m"
          limits:
            memory: "512Mi"
            cpu: "500m"
        ports:
        - containerPort: 80
        env:
        - name: ASPNETCORE_ENVIRONMENT
          value: "Production"
        - name: ConnectionStrings__MongoDB
          valueFrom:
            secretKeyRef:
              name: mongodb-secrets
              key: connection-string

Inter-Service Communication

We implemented both synchronous and asynchronous communication patterns:

Synchronous Communication
- REST APIs for direct service-to-service communication
- gRPC for performance-critical operations
- API Gateway pattern using Ocelot
Asynchronous Communication
- Event-driven architecture using Azure Service Bus
- Dead letter queues for failed messages
- Event sourcing for order processing

// Example of handling events in Order Processing Service
public class OrderCreatedEventHandler : IEventHandler<OrderCreatedEvent>
{
    private readonly IInventoryService _inventoryService;
    private readonly IMessageBus _messageBus;

    public async Task HandleAsync(OrderCreatedEvent @event)
    {
        try
        {
            // Reserve inventory
            var inventoryReserved = await _inventoryService.ReserveInventoryAsync(
                @event.OrderId,
                @event.Products
            );

            if (inventoryReserved)
            {
                // Publish inventory reserved event
                await _messageBus.PublishAsync(new InventoryReservedEvent(@event.OrderId));
            }
            else
            {
                // Handle insufficient inventory
                await _messageBus.PublishAsync(new OrderFailedEvent(
                    @event.OrderId,
                    "Insufficient inventory"
                ));
            }
        }
        catch (Exception ex)
        {
            // Handle errors and compensating transactions
            await _messageBus.PublishAsync(new OrderFailedEvent(
                @event.OrderId,
                ex.Message
            ));
        }
    }
}

Monitoring and Observability

We implemented comprehensive monitoring using:

Prometheus for metrics collection
Grafana for visualization
Elasticsearch, Fluentd, and Kibana (EFK stack) for log aggregation
Jaeger for distributed tracing

Service Mesh Implementation

We adopted Istio as our service mesh to handle:

Traffic management
Security
Observability
Load balancing
Circuit breaking

Lessons Learned

Start Small
Rather than migrating everything at once, we started with non-critical services. This allowed us to learn and adjust our approach without risking core business functions.
Data Management is Critical
We spent significant time planning our data architecture. Each service owns its data, and we implemented careful boundaries to maintain data consistency.
Automation is Essential
We invested heavily in CI/CD pipelines using Azure DevOps, automating everything from build to deployment:

# Example Azure DevOps pipeline
trigger:
  branches:
    include:
    - main
    - feature/*

pool:
  vmImage: 'ubuntu-latest'

variables:
  dockerRegistry: 'container-registry'
  imageRepository: 'product-catalog'
  tag: '$(Build.BuildId)'

stages:
- stage: Build
  jobs:
  - job: BuildAndTest
    steps:
    - script: dotnet build --configuration Release
    - script: dotnet test
    - task: Docker@2
      inputs:
        containerRegistry: $(dockerRegistry)
        repository: $(imageRepository)
        command: 'buildAndPush'
        Dockerfile: '**/Dockerfile'
        tags: |
          $(tag)
          latest

Performance Optimization We implemented various strategies to maintain high performance:

Caching with Redis
Message queuing for asynchronous operations
Horizontal scaling based on metrics
CDN integration for static content

Results

After six months of gradual migration:

Deployment frequency increased from bi-weekly to daily
System availability improved from 99.9% to 99.99%
Average response time decreased by 40%
Development team productivity increased by 35%

Future Improvements

We're currently working on:

Implementing chaos engineering practices
Expanding our API gateway capabilities
Enhancing our monitoring with AI-powered anomaly detection
Implementing blue-green deployments for zero-downtime updates

Conclusion

Migrating to microservices was a challenging but rewarding journey. The key to our success was taking an incremental approach, maintaining strong team communication, and focusing on automation and observability from the start.

Remember that microservices aren't a silver bullet - they solve specific problems but also introduce complexity. Ensure your organization has the technical maturity and real need for microservices before embarking on this journey.

Building Resilient and Fault-Tolerant Applications with Azure Resiliency Patterns

Paulo Torres — Mon, 20 Jan 2025 20:12:23 +0000

In today’s cloud-driven world, building resilient and fault-tolerant applications has become a cornerstone of successful software engineering. System outages, latency spikes, and service failures are inevitable in any distributed architecture. However, by designing applications with resilience in mind, we can ensure they recover gracefully, continue functioning, and maintain a seamless user experience.

This article explores how to build resilient and fault-tolerant applications using Azure Resiliency Patterns and .NET Core, providing real-world examples and actionable strategies for enterprise-level solutions.

Why Resilience Matters

Modern cloud applications often span multiple services and regions, relying on APIs, databases, and third-party integrations. Without proper resilience mechanisms, a single point of failure can cascade, impacting system reliability and user trust. The goal of resilience is to:

Ensure high availability even in the face of failures.
Minimize downtime and user impact during outages.
Enable systems to recover automatically without manual intervention.

Azure Resiliency Patterns Overview

Microsoft Azure provides several tools and design patterns to enable fault-tolerant systems. Below are the key resiliency patterns and how they can be applied in real-world applications:

1. Retry Pattern

The Retry Pattern ensures that transient failures, such as network glitches or temporary service unavailability, don’t result in permanent errors.

Implementation in .NET Core with Polly

Polly is a .NET library for implementing resilience and transient-fault-handling strategies like retries.

Example: Retry for an Azure SQL Database Call

var retryPolicy = Policy
    .Handle<SqlException>()
    .WaitAndRetryAsync(3, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)), 
        (exception, timeSpan, retryCount, context) =>
        {
            Console.WriteLine($"Retry {retryCount} for operation. Error: {exception.Message}");
        });

await retryPolicy.ExecuteAsync(async () =>
{
    // Simulated database call
    await using var connection = new SqlConnection("<Azure-SQL-ConnectionString>");
    await connection.OpenAsync();
    // Execute query or command
});

In this example:

The app retries failed database operations three times.
Retries occur with an exponential backoff (e.g., 1s, 2s, 4s).

2. Circuit Breaker Pattern

The Circuit Breaker Pattern prevents your application from repeatedly trying to access a failing service. Instead, it opens the circuit after a specified number of failures, blocking further requests until the service recovers.

Implementation with Polly

var circuitBreakerPolicy = Policy
    .Handle<Exception>()
    .CircuitBreakerAsync(2, TimeSpan.FromSeconds(30), 
        onBreak: (exception, duration) =>
        {
            Console.WriteLine("Circuit opened!");
        },
        onReset: () =>
        {
            Console.WriteLine("Circuit closed, operations resume.");
        });

await circuitBreakerPolicy.ExecuteAsync(async () =>
{
    // Simulate a failing service call
    throw new HttpRequestException("Service unavailable");
});

In this example:

After two consecutive failures, the circuit opens for 30 seconds.
This prevents overwhelming the failing service with repeated calls.

3. Bulkhead Isolation

The Bulkhead Pattern isolates different parts of the system to limit the impact of a failure in one component. For example, if a single API endpoint becomes unresponsive, other endpoints remain operational.

Implementation in Azure App Service

Use Azure App Service Deployment Slots or scale out specific microservices to isolate workloads.

Example: Isolating API Calls Using Task Parallelism in .NET

var tasks = new List<Task>
{
    Task.Run(() => CallServiceAAsync()),
    Task.Run(() => CallServiceBAsync())
};

await Task.WhenAll(tasks);

async Task CallServiceAAsync()
{
    // Call Service A
}

async Task CallServiceBAsync()
{
    // Call Service B
}

Each service call is executed in its own isolated thread, ensuring that a failure in one service doesn’t block others.

4. Timeout Pattern

The Timeout Pattern ensures that operations don’t hang indefinitely. If a service call doesn’t respond within a specified time frame, it fails gracefully.

Implementation in .NET Core with Polly

var timeoutPolicy = Policy
    .TimeoutAsync(TimeSpan.FromSeconds(5), 
        (context, timeSpan, task) =>
        {
            Console.WriteLine($"Operation timed out after {timeSpan.Seconds} seconds.");
            return Task.CompletedTask;
        });

await timeoutPolicy.ExecuteAsync(async () =>
{
    // Simulated long-running operation
    await Task.Delay(10000); // Simulates a delayed service response
});

5. Failover Pattern

The Failover Pattern ensures that if one service or region fails, traffic is redirected to another available instance.

Implementation with Azure Traffic Manager

Azure Traffic Manager enables automatic failover for global applications. Configure multiple endpoints in different Azure regions and set the failover priority.

Create a Traffic Manager Profile:
- Configure multiple Azure App Services or VMs in different regions.
- Set routing methods: Priority or Performance.
Modify Your Application’s DNS Settings:
- Route traffic through the Traffic Manager’s DNS name.

6. Queue-Based Load Leveling

The Queue-Based Load Leveling Pattern decouples producers (e.g., APIs) from consumers (e.g., background jobs) using message queues like Azure Service Bus or Azure Storage Queues.

Example: Azure Service Bus with .NET Core

Producer: Enqueue messages to the queue.

var client = new ServiceBusClient("<Connection-String>");
var sender = client.CreateSender("<Queue-Name>");

await sender.SendMessageAsync(new ServiceBusMessage("Order Processed"));

Consumer: Dequeue messages and process them.

var processor = client.CreateProcessor("<Queue-Name>");

processor.ProcessMessageAsync += async args =>
{
    Console.WriteLine($"Received: {args.Message.Body.ToString()}");
    await args.CompleteMessageAsync(args.Message);
};
await processor.StartProcessingAsync();

This pattern ensures that spikes in traffic don’t overwhelm the backend system.

Real-World Use Case: E-Commerce Application

Consider an e-commerce application built with .NET Core and hosted on Azure. Here's how resiliency patterns ensure reliability:

Order Processing:
- Use Azure Service Bus to queue orders and ensure the system handles traffic spikes smoothly.
- Implement retries for database writes to Azure SQL.
Inventory Management:
- Use Circuit Breaker to prevent repeated calls to a failing inventory service.
- Use Bulkhead Isolation to separate inventory checks from other API operations.
Global Customer Base:
- Use Azure Traffic Manager for failover between regions.
- Cache frequently accessed product details using Azure Cache for Redis.
Payment Processing:
- Implement Timeout Pattern to ensure payment gateways don’t hang indefinitely.
- Use Retry Pattern for transient failures during payment gateway API calls.

Best Practices for Building Resilient Applications

Test for Failures: Use tools like Chaos Studio to simulate failures and validate your system’s resiliency.
Monitor Continuously: Leverage Azure Monitor and Application Insights to detect issues early.
Automate Scaling: Enable auto-scaling for Azure App Services, Azure Functions, and databases to handle sudden traffic surges.
Document SLAs and Recovery Plans: Clearly define service-level agreements and recovery strategies for critical services.

Conclusion

Building resilient and fault-tolerant applications is not just about handling failures but embracing them as an inevitable part of distributed systems. By implementing Azure resiliency patterns like Retry, Circuit Breaker, and Queue-Based Load Leveling, you can create robust applications that recover gracefully and maintain a seamless user experience.

With tools like Azure Traffic Manager, Azure Service Bus, and libraries like Polly in .NET Core, developers have the resources to design systems that are not only resilient but also scalable and cost-efficient. By incorporating these patterns into your architecture, you ensure that your applications are prepared to handle the unpredictable challenges of the cloud-native world.

Let resilience be the backbone of your application’s success! 🚀

Optimizing Performance in Cloud-Native Applications with .NET Core 8 and Azure

Paulo Torres — Thu, 16 Jan 2025 17:29:24 +0000

In today’s fast-paced digital landscape, optimizing performance in cloud-native applications is not just a luxury—it’s a necessity. As businesses scale, so do the demands on their systems, and any inefficiency in code, database queries, or cloud resource utilization can lead to higher costs and reduced user satisfaction. This article focuses on real-world strategies for optimizing performance in cloud-native applications built using .NET Core 8 and Azure, offering practical insights and examples to help you deliver fast, scalable, and cost-effective solutions.

The Importance of Performance Optimization

Cloud-native applications leverage distributed architectures, making performance optimization a multi-faceted challenge. Poorly performing applications can lead to:

Higher operational costs due to inefficient resource utilization.
User dissatisfaction caused by high latency or downtime.
Scalability bottlenecks when handling increased traffic.

By adopting best practices in .NET Core 8 and Azure, engineers can address these challenges effectively.

1. Optimize API Performance with .NET Core 8

Efficient Asynchronous Programming

In cloud-native applications, API endpoints often handle multiple requests concurrently. Using async/await ensures that threads are not blocked while waiting for I/O operations like database queries or HTTP requests.

Example: Optimized API Controller

[ApiController]
[Route("api/[controller]")]
public class ProductsController : ControllerBase
{
    private readonly IProductService _productService;

    public ProductsController(IProductService productService)
    {
        _productService = productService;
    }

    [HttpGet("{id}")]
    public async Task<IActionResult> GetProductByIdAsync(int id)
    {
        var product = await _productService.GetProductByIdAsync(id);
        if (product == null) return NotFound();
        return Ok(product);
    }
}

Caching Frequently Accessed Data

Use Azure Cache for Redis to cache frequently accessed data and reduce database load.

Example: Implementing Redis Caching

public class ProductService : IProductService
{
    private readonly IDistributedCache _cache;
    private readonly IProductRepository _repository;

    public ProductService(IDistributedCache cache, IProductRepository repository)
    {
        _cache = cache;
        _repository = repository;
    }

    public async Task<Product> GetProductByIdAsync(int id)
    {
        var cacheKey = $"Product_{id}";
        var cachedProduct = await _cache.GetStringAsync(cacheKey);

        if (!string.IsNullOrEmpty(cachedProduct))
            return JsonConvert.DeserializeObject<Product>(cachedProduct);

        var product = await _repository.GetProductByIdAsync(id);
        if (product != null)
        {
            await _cache.SetStringAsync(
                cacheKey,
                JsonConvert.SerializeObject(product),
                new DistributedCacheEntryOptions { AbsoluteExpirationRelativeToNow = TimeSpan.FromMinutes(10) }
            );
        }
        return product;
    }
}

2. Optimize Database Interactions

Use Efficient Querying Techniques

Inefficient database queries can become a significant bottleneck. Use Entity Framework Core with proper query optimizations, like avoiding unnecessary joins and loading only the required data.

Example: Using `.AsNoTracking()` for Read-Only Queries

public async Task<List<Product>> GetProductsAsync()
{
    return await _dbContext.Products
        .AsNoTracking() // Improves performance for read-only operations
        .Where(p => p.IsActive)
        .ToListAsync();
}

Database Indexing

Ensure your database has proper indexing for frequently queried fields. For example, indexing a ProductId or IsActive field can significantly improve performance for queries that filter by these columns.

Scaling Databases with Azure SQL

Use Read Replicas for handling read-heavy workloads.
Scale Azure SQL elastically based on demand using Auto-Scale.

3. Leverage Azure-Specific Optimizations

Azure Functions for Background Processing

Offload heavy, non-blocking tasks to Azure Functions to reduce the load on your API.

Real-World Scenario: Sending Order Confirmation Emails

Instead of processing emails synchronously in your API, send them via an Azure Function triggered by Azure Event Grid.

Step 1: Publish Event to Event Grid

public async Task NotifyOrderCreatedAsync(Order order)
{
    var eventGridClient = new EventGridPublisherClient(new Uri("<your-event-grid-url>"), new AzureKeyCredential("<your-key>"));
    var orderEvent = new EventGridEvent(
        "OrderCreated",
        "Order.Events.Created",
        "1.0",
        order
    );
    await eventGridClient.SendEventAsync(orderEvent);
}

Step 2: Azure Function to Handle the Event

[FunctionName("ProcessOrderCreated")]
public async Task Run([EventGridTrigger] EventGridEvent eventGridEvent)
{
    var order = JsonConvert.DeserializeObject<Order>(eventGridEvent.Data.ToString());
    await _emailService.SendOrderConfirmationEmailAsync(order);
}

Static Content Delivery with Azure Front Door

Serve static content like images, JavaScript, and CSS through Azure Front Door for low latency and high availability. Pair this with Azure Blob Storage to store your static assets.

4. Monitor and Diagnose Performance Issues

Application Insights

Use Azure Application Insights to monitor the performance and detect bottlenecks in real time. Key metrics include:

Request Latency: Measure the time taken by API endpoints.
Dependency Calls: Track latency and failure rates for external services like databases and APIs.

Example: Adding Application Insights to .NET Core

dotnet add package Microsoft.ApplicationInsights.AspNetCore

Program.cs

builder.Services.AddApplicationInsightsTelemetry();

5. Implement Load Testing and Auto-Scaling

Load Testing

Use Azure Load Testing to simulate real-world traffic and identify bottlenecks in your application.

Key Metrics to Monitor:

Request/Response Time
Error Rates
Resource Utilization (CPU, Memory)

Auto-Scaling with Azure App Service

Configure auto-scaling rules to adjust resources based on metrics like CPU usage or request count.

Example: Auto-Scale Rules

Scale out to more instances when CPU > 70% for 5 minutes.
Scale in when CPU < 30% for 10 minutes.

Conclusion

Optimizing performance in cloud-native applications requires a holistic approach—starting from efficient code practices in .NET Core 8 to leveraging Azure's cloud-native capabilities like Redis caching, Event Grid, and Application Insights. By combining these best practices, you can ensure your applications are fast, scalable, and cost-effective.

Remember, performance optimization is not a one-time activity. It’s an iterative process that evolves as your application scales and user demands grow. With .NET Core 8 and Azure, you have all the tools you need to build cloud-native applications that deliver exceptional performance.

Fine-Tuning Large Language Models (LLMs) with .NET Core, Python, and Azure

Paulo Torres — Mon, 13 Jan 2025 15:59:27 +0000

Introduction
Why Fine-Tune Large Language Models?
Solution Overview
Setting Up the Environment
Training and Fine-Tuning with Python
Integrating the Fine-Tuned Model in .NET Core
Deploying to Azure
Best Practices
Conclusion

1. Introduction

Large Language Models (LLMs) have gained significant traction for their ability to understand and generate human-like text. However, many organizations have unique datasets and domain-specific vocabularies that general-purpose models may not fully capture. Fine-tuning allows developers to tailor these large models to a specific context or industry, improving accuracy and relevance.

In this article, we’ll explore how to fine-tune an LLM using Python, then integrate and deploy the resulting model into a .NET Core C# application, all on Microsoft Azure for scalability and convenience.

2. Why Fine-Tune Large Language Models?

Domain Specificity: LLMs can be fine-tuned to use industry-specific jargon, product names, or specialized terminology.
Performance Boost: Fine-tuning often reduces errors and increases relevance in use cases like customer service, research, and analytics.
Reduced Costs: Rather than building a model from scratch, you customize an existing powerful LLM.
Time Efficiency: You leverage pre-trained weights and only adjust final layers or parameters, expediting the process.

3. Solution Overview

Components and Technologies

Python for Fine-Tuning
- Popular libraries (e.g., Hugging Face Transformers, PyTorch)
- Streamlined process for loading and adapting pre-trained models
.NET Core C# for Integration
- Back-end service or API exposing the fine-tuned model
- Strongly typed language, familiar to many enterprise developers
Azure Services
- Azure Machine Learning for training and model management
- Azure Storage for data and model artifacts
- Azure App Service or Azure Functions for hosting the .NET Core application
- Azure Key Vault (optional) to secure credentials

4. Setting Up the Environment

Prerequisites

Azure Subscription: Required to create resources such as Machine Learning workspaces and App Services.
Python 3.8+: Installed locally for model fine-tuning.
.NET 6/7/8 SDK: To create and run the .NET Core C# application.
Visual Studio 2022 or Visual Studio Code: Recommended IDEs.
Azure CLI: For provisioning and managing Azure services via the terminal.
Docker (optional): Useful for containerizing your application, if desired.

5. Training and Fine-Tuning with Python

This example uses Hugging Face Transformers—one of the most widely adopted libraries for LLM fine-tuning.

5.1 Set Up a Virtual Environment

python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

5.2 Install Dependencies

pip install torch transformers azureml-sdk

5.3 Create an Azure Machine Learning Workspace

Resource Group and Workspace:

   az group create --name LLMFinetuneRG --location eastus
   az ml workspace create --name LLMFinetuneWS --resource-group LLMFinetuneRG

Configure local environment to connect to the workspace (using a config.json file or environment variables).

5.4 Fine-Tuning Script (train.py)

import os
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments, Trainer
from azureml.core import Workspace, Run

# Connect to Azure ML
ws = Workspace.from_config()
run = Run.get_context()

model_name = "gpt2"  # Example model
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

# Load your custom dataset (local or from Azure Storage)
# Example: a text file or a dataset in Azure ML
train_texts = ["Your domain-specific text here..."]  # Simplified
train_encodings = tokenizer(train_texts, truncation=True, padding=True)

class CustomDataset(torch.utils.data.Dataset):
    def __init__(self, encodings):
        self.encodings = encodings
    def __len__(self):
        return len(self.encodings["input_ids"])
    def __getitem__(self, idx):
        return {k: torch.tensor(v[idx]) for k, v in self.encodings.items()}

train_dataset = CustomDataset(train_encodings)

training_args = TrainingArguments(
    output_dir="./results",
    num_train_epochs=3,
    per_device_train_batch_size=2,
    save_steps=100,
    logging_steps=100
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
)

trainer.train()

# Save the fine-tuned model
trainer.save_model("./fine_tuned_model")
tokenizer.save_pretrained("./fine_tuned_model")

5.5 Register the Model in Azure

from azureml.core.model import Model

model = Model.register(
    workspace=ws,
    model_path="./fine_tuned_model",
    model_name="myFineTunedLLM"
)

At this point, your fine-tuned model is stored in Azure Machine Learning for easy access and version control.

6. Integrating the Fine-Tuned Model in .NET Core

6.1 Create a .NET Core Web API Project

dotnet new webapi -n FineTunedLLMApi
cd FineTunedLLMApi

6.2 Add Dependencies

HttpClient for calling Azure endpoints or local inference API
Newtonsoft.Json (if you prefer JSON.NET for serialization)
Azure.Storage.Blobs or Azure.Identity for secure access to Azure resources

dotnet add package Microsoft.Extensions.Http
dotnet add package Microsoft.Azure.Storage.Blob
dotnet add package Newtonsoft.Json

6.3 ModelConsumerService.cs

Assume you deployed the fine-tuned model as a web service (e.g., using Azure Container Instances or a custom endpoint in Azure ML). The following snippet calls that service to get completions.

using Newtonsoft.Json;
using System.Net.Http;
using System.Text;
using System.Threading.Tasks;

public class ModelConsumerService
{
    private readonly HttpClient _httpClient;

    public ModelConsumerService(IHttpClientFactory httpClientFactory)
    {
        _httpClient = httpClientFactory.CreateClient("FineTunedModel");
    }

    public async Task<string> GetCompletionAsync(string prompt)
    {
        var requestBody = new { prompt = prompt };
        var content = new StringContent(
            JsonConvert.SerializeObject(requestBody),
            Encoding.UTF8, 
            "application/json");

        var response = await _httpClient.PostAsync("/predict", content);
        response.EnsureSuccessStatusCode();
        return await response.Content.ReadAsStringAsync();
    }
}

6.4 LLMController.cs

using Microsoft.AspNetCore.Mvc;
using System.Threading.Tasks;

[ApiController]
[Route("[controller]")]
public class LLMController : ControllerBase
{
    private readonly ModelConsumerService _modelService;

    public LLMController(ModelConsumerService modelService)
    {
        _modelService = modelService;
    }

    [HttpPost("complete")]
    public async Task<IActionResult> CompletePrompt([FromBody] PromptRequest request)
    {
        var result = await _modelService.GetCompletionAsync(request.Prompt);
        return Ok(new { Completion = result });
    }
}

public class PromptRequest
{
    public string Prompt { get; set; }
}

6.5 Configure Your .NET Core App

In Program.cs or Startup.cs:

var builder = WebApplication.CreateBuilder(args);

// Register HttpClient
builder.Services.AddHttpClient("FineTunedModel", client =>
{
    client.BaseAddress = new Uri("https://your-model-endpoint/");
});

// Register the ModelConsumerService
builder.Services.AddTransient<ModelConsumerService>();

builder.Services.AddControllers();
var app = builder.Build();

app.MapControllers();
app.Run();

7. Deploying to Azure

Azure App Service:
- Easiest route for many .NET Core apps.
- Create a new Web App from the Azure Portal or via the CLI.

   az webapp create --resource-group LLMFinetuneRG --plan MyPlan --name FineTunedLLMWebApp

Azure Functions (Optional):
- Ideal for running serverless, event-driven logic if your usage is intermittent or scheduled.
Azure Kubernetes Service (AKS) (Advanced):
- Ideal for large-scale deployments.
- Containerize your app using Docker and push it to Azure Container Registry (ACR).

8. Best Practices

Data Privacy: Ensure you handle sensitive or proprietary data responsibly, especially during model training.
Monitoring & Logging: Integrate Azure Application Insights to monitor performance, track usage, and detect anomalies.
Security: Use Azure Key Vault to store secrets (API keys, connection strings).
Model Versioning: Keep track of different fine-tuned versions in Azure ML; roll back to older versions if needed.
Prompt Engineering: Refine your prompts to get the best results from your fine-tuned model.

9. Conclusion

Fine-tuning LLMs with Python and Azure Machine Learning, then integrating them into a .NET Core application, allows you to build powerful domain-specific AI solutions. This combination is an excellent choice for organizations seeking to leverage the best of Python’s AI ecosystem and .NET’s enterprise capabilities, all backed by the scalability of Azure.

With careful planning around security, data governance, and DevOps, you can roll out a production-ready solution that meets real-world needs—delivering accurate, domain-specific language capabilities in a robust, maintainable framework.

Building AI Agents for Automated Trading Systems Using .NET C# Semantic Kernel, Azure AI Services, and Azure Functions

Paulo Torres — Tue, 07 Jan 2025 18:31:35 +0000

In the dynamic world of financial trading, leveraging Artificial Intelligence (AI) to automate trading strategies has become a game-changer. Integrating AI agents into trading systems allows for real-time data analysis, predictive modeling, and autonomous decision-making. This comprehensive guide explores how to build a real-world AI agent for automated trading using .NET C# Semantic Kernel, .NET Core C# 8, ASP.NET Core Web API, Azure AI Services, Azure Functions, Azure Key Vault, Azure Cosmos DB (MongoDB API), Azure Kubernetes Service (AKS), Python, and other relevant technologies.

Introduction
Prerequisites
Architecture Overview
Setting Up the Development Environment
Developing the AI Model with Python and Azure Machine Learning
Utilizing .NET C# Semantic Kernel for AI Agents
Creating the .NET Core Web API
Integrating the AI Model into the .NET Core Application
Implementing Data Storage with Azure Cosmos DB
Securing Secrets with Azure Key Vault
Using Azure Functions for Event-Driven Processing
Containerizing the Application with Docker
Deploying to Azure Kubernetes Service (AKS)
Monitoring and Logging
Conclusion

Introduction

Automated trading systems utilize algorithms and AI models to execute trades based on predefined strategies without human intervention. By incorporating AI agents, these systems can learn from historical data, predict market trends, and make autonomous decisions. This guide demonstrates how to build such an AI-powered trading agent using cutting-edge technologies and cloud services, including the .NET C# Semantic Kernel, which facilitates the integration of AI models into .NET applications.

Prerequisites

Azure Subscription: Access to Microsoft Azure services.
Development Tools:
- Visual Studio 2022 or Visual Studio Code with .NET Core SDK installed.
- Python 3.8+ with necessary libraries.
Azure CLI: For managing Azure resources from the command line.
Docker: Installed on your development machine.
Azure Kubernetes Service (AKS): Basic understanding.
.NET C# Semantic Kernel: Installed and configured.

Architecture Overview

The architecture consists of the following components:

AI Model: Developed in Python using Azure Machine Learning.
.NET C# Semantic Kernel: Integrates AI capabilities into the .NET Core application.
ASP.NET Core Web API: Serves as the backend, interacting with the AI agent and handling trade executions.
Azure Cosmos DB: Stores trading data and model predictions.
Azure Key Vault: Secures sensitive information like API keys and connection strings.
Azure Functions: Handles event-driven processes such as real-time data ingestion.
Azure Kubernetes Service (AKS): Hosts the containerized application for scalability and high availability.
Azure AI Services: Provides additional AI capabilities if needed.

Setting Up the Development Environment

Install .NET Core SDK and Tools

Download and install the latest .NET Core SDK.
Install Visual Studio 2022 or Visual Studio Code.

Install .NET C# Semantic Kernel

Clone the Semantic Kernel repository:

  git clone https://github.com/microsoft/semantic-kernel.git

Build and install the Semantic Kernel library.

Install Python and Libraries

Download and install Python 3.8+.
Install necessary Python libraries:

  pip install pandas numpy scikit-learn joblib azureml-sdk

Install Azure CLI

Download and install the Azure CLI.

Install Docker

Download and install Docker Desktop.

Developing the AI Model with Python and Azure Machine Learning

Step 1: Define the Trading Strategy

Determine the AI model's purpose, such as predicting stock price movements or classifying market trends.

Step 2: Set Up Azure Machine Learning Workspace

Create a Resource Group:

   az group create --name TradingAI_RG --location eastus

Create an Azure Machine Learning Workspace:

   az ml workspace create -w TradingAI_WS -g TradingAI_RG

Step 3: Develop the AI Model

Create a Python script train_model.py for training the model.

import pandas as pd
import numpy as np
from sklearn.ensemble import RandomForestClassifier
import joblib
from azureml.core import Workspace, Dataset

# Connect to Azure ML Workspace
ws = Workspace.from_config()

# Load Dataset
dataset = Dataset.get_by_name(ws, name='historical_stock_data')
df = dataset.to_pandas_dataframe()

# Data Preprocessing
# ... (cleaning, feature engineering)

# Split Data
X = df.drop('target', axis=1)
y = df['target']

# Train Model
model = RandomForestClassifier(n_estimators=100)
model.fit(X, y)

# Save Model
joblib.dump(model, 'model.joblib')

Step 4: Register the Model in Azure ML

from azureml.core.model import Model

model = Model.register(workspace=ws,
                       model_path='model.joblib',
                       model_name='TradingAIModel')

Utilizing .NET C# Semantic Kernel for AI Agents

The Semantic Kernel is an open-source SDK that allows .NET developers to integrate AI models, including Large Language Models (LLMs), into their applications.

Step 1: Install the Semantic Kernel NuGet Package

dotnet add package Microsoft.SemanticKernel

Step 2: Integrate the AI Model into the Semantic Kernel

Create a new class TradingAgentKernel.cs to define the AI agent's capabilities.

using Microsoft.SemanticKernel;
using Microsoft.SemanticKernel.Orchestration;
using Microsoft.SemanticKernel.SkillDefinition;

public class TradingAgentKernel
{
    private readonly IKernel _kernel;

    public TradingAgentKernel()
    {
        _kernel = Kernel.Builder.Build();

        // Register AI skills or functions
        _kernel.RegisterSemanticFunction("PredictMarketTrend", PredictMarketTrend);
    }

    [SKFunction]
    public async Task<SKContext> PredictMarketTrend(SKContext context)
    {
        // Implement logic to call the AI model
        var features = context.Variables["features"];
        var prediction = await CallAIModelAsync(features);
        context.Variables["prediction"] = prediction.ToString();
        return context;
    }

    private async Task<int> CallAIModelAsync(string features)
    {
        // Logic to call the AI model (e.g., via REST API)
        // ...
        return prediction;
    }

    public async Task<string> ExecuteAsync(string input)
    {
        var context = await _kernel.RunAsync(input, "PredictMarketTrend");
        return context.Variables["prediction"];
    }
}

Step 3: Leverage Azure OpenAI Service (Optional)

If you want to integrate LLMs like GPT-3 into your agent:

_kernel.Config.AddAzureOpenAICompletionBackend(
    "azureOpenAI",
    "deploymentName",
    "https://your-openai-endpoint.openai.azure.com/",
    "your-api-key"
);

Creating the .NET Core Web API

Step 1: Initialize the Project

dotnet new webapi -n TradingAIAPI
cd TradingAIAPI

Step 2: Install Necessary NuGet Packages

dotnet add package Microsoft.Azure.Cosmos
dotnet add package Azure.Security.KeyVault.Secrets
dotnet add package Microsoft.Extensions.Configuration.AzureKeyVault
dotnet add package Microsoft.Azure.Services.AppAuthentication
dotnet add package Microsoft.SemanticKernel

Step 3: Set Up Project Structure

Controllers: For API endpoints.
Services: For business logic.
Models: For data structures.

Step 4: Configure App Settings

Create appsettings.json with placeholders.

{
  "Azure": {
    "KeyVault": "https://your-keyvault-name.vault.azure.net/",
    "CosmosDb": {
      "Endpoint": "your-cosmosdb-endpoint",
      "Database": "TradingDB",
      "Container": "Trades"
    }
  }
}

Integrating the AI Model into the .NET Core Application

Step 1: Use Semantic Kernel in the Application

TradingAgentKernel.cs (as defined earlier) will be used in the application.

Step 2: Implement the Controller

TradesController.cs

[ApiController]
[Route("[controller]")]
public class TradesController : ControllerBase
{
    private readonly TradingAgentKernel _tradingAgentKernel;
    private readonly TradeRepository _tradeRepository;

    public TradesController(TradingAgentKernel tradingAgentKernel, TradeRepository tradeRepository)
    {
        _tradingAgentKernel = tradingAgentKernel;
        _tradeRepository = tradeRepository;
    }

    [HttpPost("execute")]
    public async Task<IActionResult> ExecuteTrade([FromBody] TradeRequest request)
    {
        // Use Semantic Kernel to get prediction
        var prediction = await _tradingAgentKernel.ExecuteAsync(request.FeaturesJson);

        if (int.Parse(prediction) == 1)
        {
            // Logic to execute buy order
            await _tradeRepository.AddTradeAsync(new Trade
            {
                Symbol = request.Symbol,
                Price = request.Price,
                Timestamp = DateTime.UtcNow,
                Action = "Buy"
            });
        }
        else
        {
            // Logic to execute sell order
            await _tradeRepository.AddTradeAsync(new Trade
            {
                Symbol = request.Symbol,
                Price = request.Price,
                Timestamp = DateTime.UtcNow,
                Action = "Sell"
            });
        }

        return Ok(new { Success = true });
    }
}

TradeRequest Model

public class TradeRequest
{
    public string Symbol { get; set; }
    public double Price { get; set; }
    public string FeaturesJson { get; set; } // JSON string of features
}

Implementing Data Storage with Azure Cosmos DB

Follow the same steps as before, using the MongoDB Driver for .NET Core.

TradeRepository.cs remains unchanged.

Securing Secrets with Azure Key Vault

Step 1: Create an Azure Key Vault

az keyvault create --name TradingAIKeyVault --resource-group TradingAI_RG --location eastus

Step 2: Store Secrets

az keyvault secret set --vault-name TradingAIKeyVault --name "CosmosDbConnectionString" --value "your-connection-string"

Step 3: Configure the Application to Use Key Vault

Install Packages

dotnet add package Azure.Extensions.AspNetCore.Configuration.Secrets
dotnet add package Azure.Identity

Modify Program.cs

using Azure.Identity;

public class Program
{
    public static void Main(string[] args)
    {
        var host = CreateHostBuilder(args).Build();
        host.Run();
    }

    public static IHostBuilder CreateHostBuilder(string[] args) =>
        Host.CreateDefaultBuilder(args)
            .ConfigureAppConfiguration((context, config) =>
            {
                var builtConfig = config.Build();
                var keyVaultName = builtConfig["Azure:KeyVault"];
                if (!string.IsNullOrEmpty(keyVaultName))
                {
                    var keyVaultUri = new Uri(keyVaultName);
                    config.AddAzureKeyVault(keyVaultUri, new DefaultAzureCredential());
                }
            })
            .ConfigureWebHostDefaults(webBuilder =>
            {
                webBuilder.UseStartup<Startup>();
            });
}

Using Azure Functions for Event-Driven Processing

Azure Functions can be used to handle real-time data ingestion or to trigger trading actions based on specific events.

Step 1: Create an Azure Function Project

func init TradingAIFunctions --dotnet
cd TradingAIFunctions
func new --name MarketDataIngestion --template "HttpTrigger" --authlevel "Function"

Step 2: Implement the Function

MarketDataIngestion.cs

using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.AspNetCore.Http;
using Microsoft.Extensions.Logging;

public static class MarketDataIngestion
{
    [FunctionName("MarketDataIngestion")]
    public static async Task<IActionResult> Run(
        [HttpTrigger(AuthorizationLevel.Function, "post", Route = null)] HttpRequest req,
        ILogger log)
    {
        // Parse incoming market data
        var requestBody = await new StreamReader(req.Body).ReadToEndAsync();
        var marketData = JsonConvert.DeserializeObject<MarketData>(requestBody);

        // Logic to process data or trigger trading actions
        // ...

        return new OkResult();
    }
}

public class MarketData
{
    public string Symbol { get; set; }
    public double Price { get; set; }
    // Other relevant fields
}

Step 3: Deploy the Function to Azure

func azure functionapp publish TradingAIFunctionsApp

Step 4: Integrate with the Main Application

Use Azure Event Grid or Azure Service Bus to communicate between Azure Functions and the main application.

Containerizing the Application with Docker

Follow the same steps as before, ensuring that the Dockerfile includes the necessary configurations for the Semantic Kernel and any dependencies.

Dockerfile

# Base image
FROM mcr.microsoft.com/dotnet/aspnet:8.0 AS base
WORKDIR /app

# Copy files and build
FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build
WORKDIR /src
COPY ["TradingAIAPI.csproj", "./"]
RUN dotnet restore "./TradingAIAPI.csproj"
COPY . .
RUN dotnet build "TradingAIAPI.csproj" -c Release -o /app/build

# Publish
FROM build AS publish
RUN dotnet publish "TradingAIAPI.csproj" -c Release -o /app/publish

# Final stage
FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "TradingAIAPI.dll"]

Deploying to Azure Kubernetes Service (AKS)

Follow the same steps as before to create an AKS cluster, push Docker images to Azure Container Registry, and deploy the application.

Monitoring and Logging

Step 1: Enable Azure Monitor for Containers

az aks enable-addons -a monitoring -n TradingAICluster -g TradingAI_RG

Step 2: Use Application Insights

Install Package

dotnet add package Microsoft.ApplicationInsights.AspNetCore

Configure in Startup.cs

public void ConfigureServices(IServiceCollection services)
{
    services.AddApplicationInsightsTelemetry("Your_Instrumentation_Key");
    services.AddControllers();
}

Step 3: View Logs and Metrics

Access Azure Portal to view logs in Azure Monitor.
Use Application Insights for application-level telemetry.

Conclusion

By integrating .NET C# Semantic Kernel and Azure Functions into your automated trading system, you enhance the AI capabilities and enable event-driven processing, which is crucial for real-time trading applications. The combination of these technologies with Azure AI Services, Azure Key Vault, Azure Cosmos DB, AKS, and Python allows you to build a sophisticated, scalable, and secure AI agent for automated trading.

About the Author

As a Senior Software Engineer and Senior AI Engineer specializing in .NET C#, Azure, and AI, I bring extensive experience in developing enterprise-grade applications and AI solutions. This article combines practical knowledge and best practices to help professionals build advanced automated trading systems.

References

Developing an Equity Business Application with .NET 8, Azure, and AI

Paulo Torres — Thu, 07 Nov 2024 19:01:05 +0000

In the competitive world of finance, equity investors require sophisticated tools to make informed decisions. By leveraging .NET 8, Azure, and Artificial Intelligence (AI), developers can build robust applications that provide real-time insights, predictive analytics, and a seamless user experience. This article presents a real-world example of creating an equity business application for investors, complete with code snippets, configurations, and detailed explanations.

Scenario: Building an Equity Investment Platform

Our goal is to develop an equity investment platform that offers:

Real-time stock data visualization
Predictive analytics for stock performance
Personalized investment recommendations
Secure user authentication and data protection

Setting Up the Development Environment

Prerequisites

Here is an outline of the key steps involved:

Now, let's dive into the development process.

Let's walk through the process step by step.

Step 1: Creating the .NET 8 Web Application

Initialize a new ASP.NET Core Web Application.

dotnet new webapp -n EquityInvestorPlatform
cd EquityInvestorPlatform

Project Structure

Frontend: ASP.NET Core Razor Pages or Blazor for interactive UI
Backend: ASP.NET Core Web API for data processing
Data Layer: Entity Framework Core with SQL Server
AI Integration: Azure Machine Learning for predictive analytics
Authentication: Azure Active Directory (Azure AD) for secure login

Step 2: Implementing Secure User Authentication with Azure AD

Registering the Application in Azure AD

Navigate to the Azure Portal and go to Azure Active Directory.
Under App registrations, click New registration.
Enter a Name for your app (e.g., "EquityInvestorPlatform").
Set the Redirect URI to https://localhost:5001/signin-oidc.
Click Register.

Configuring the Application

Install the necessary NuGet packages.

dotnet add package Microsoft.AspNetCore.Authentication.AzureAD.UI
dotnet add package Microsoft.Identity.Web

Modify Program.cs to configure authentication.

using Microsoft.Identity.Web;
using Microsoft.AspNetCore.Authentication;

var builder = WebApplication.CreateBuilder(args);

// Add services to the container.
builder.Services.AddRazorPages();

// Configure Azure AD authentication
builder.Services.AddAuthentication(OpenIdConnectDefaults.AuthenticationScheme)
    .AddMicrosoftIdentityWebApp(builder.Configuration.GetSection("AzureAd"));

var app = builder.Build();

// Enable authentication middleware
app.UseAuthentication();
app.UseAuthorization();

app.MapRazorPages();

app.Run();

Add the Azure AD configuration to appsettings.json.

"AzureAd": {
  "Instance": "https://login.microsoftonline.com/",
  "Domain": "yourdomain.onmicrosoft.com",
  "TenantId": "your-tenant-id",
  "ClientId": "your-client-id",
  "CallbackPath": "/signin-oidc"
}

Protecting Routes

Apply the [Authorize] attribute to secure pages.

using Microsoft.AspNetCore.Authorization;
using Microsoft.AspNetCore.Mvc.RazorPages;

[Authorize]
public class DashboardModel : PageModel
{
    public void OnGet()
    {
    }
}

Step 3: Integrating Real-Time Stock Data

Using a Third-Party API

We'll use a financial data API like Alpha Vantage or Yahoo Finance.

Installing HttpClientFactory

dotnet add package Microsoft.Extensions.Http

Configuring Services

In Program.cs:

builder.Services.AddHttpClient("StockDataClient", client =>
{
    client.BaseAddress = new Uri("https://www.alphavantage.co/");
});

Creating the Stock Service

public class StockService
{
    private readonly HttpClient _httpClient;
    private readonly string _apiKey = "YOUR_ALPHA_VANTAGE_API_KEY";

    public StockService(IHttpClientFactory httpClientFactory)
    {
        _httpClient = httpClientFactory.CreateClient("StockDataClient");
    }

    public async Task<StockQuote> GetStockQuoteAsync(string symbol)
    {
        var response = await _httpClient.GetAsync($"/query?function=GLOBAL_QUOTE&symbol={symbol}&apikey={_apiKey}");
        response.EnsureSuccessStatusCode();

        var content = await response.Content.ReadAsStringAsync();
        // Deserialize JSON to StockQuote object
        var stockQuote = JsonConvert.DeserializeObject<StockQuote>(content);
        return stockQuote;
    }
}

Creating Models

public class StockQuote
{
    [JsonProperty("Global Quote")]
    public GlobalQuote GlobalQuote { get; set; }
}

public class GlobalQuote
{
    [JsonProperty("01. symbol")]
    public string Symbol { get; set; }

    [JsonProperty("05. price")]
    public string Price { get; set; }

    // Add other properties as needed
}

Displaying Data in the UI

In your Razor Page:

@page
@model IndexModel
@inject StockService StockService

<h2>Stock Quote</h2>
<form method="post">
    <input type="text" name="Symbol" placeholder="Enter Stock Symbol" />
    <button type="submit">Get Quote</button>
</form>

@if (Model.StockQuote != null)
{
    <div>
        <p>Symbol: @Model.StockQuote.GlobalQuote.Symbol</p>
        <p>Price: @Model.StockQuote.GlobalQuote.Price</p>
    </div>
}

@functions {
    public StockQuote StockQuote { get; set; }

    public async Task OnPostAsync(string symbol)
    {
        StockQuote = await StockService.GetStockQuoteAsync(symbol);
    }
}

Step 4: Implementing Predictive Analytics with Azure Machine Learning

Creating a Predictive Model

Use Azure Machine Learning Studio to:

Prepare Data: Historical stock data.
Train Model: Use regression algorithms to predict future stock prices.
Deploy Model: Expose the model as a REST endpoint.

Consuming the Model in the Application

Installing Necessary Packages

dotnet add package Newtonsoft.Json

Creating the Prediction Service

public class PredictionService
{
    private readonly HttpClient _httpClient;
    private readonly string _endpoint = "YOUR_AZURE_ML_ENDPOINT";
    private readonly string _apiKey = "YOUR_AZURE_ML_API_KEY";

    public PredictionService(IHttpClientFactory httpClientFactory)
    {
        _httpClient = httpClientFactory.CreateClient();
    }

    public async Task<PredictionResult> PredictStockPriceAsync(PredictionInput input)
    {
        var request = new HttpRequestMessage(HttpMethod.Post, _endpoint);
        request.Headers.Add("Authorization", $"Bearer {_apiKey}");
        request.Content = new StringContent(JsonConvert.SerializeObject(input), Encoding.UTF8, "application/json");

        var response = await _httpClient.SendAsync(request);
        response.EnsureSuccessStatusCode();

        var content = await response.Content.ReadAsStringAsync();
        var predictionResult = JsonConvert.DeserializeObject<PredictionResult>(content);
        return predictionResult;
    }
}

public class PredictionInput
{
    public string Symbol { get; set; }
    public DateTime Date { get; set; }
    // Other features as required
}

public class PredictionResult
{
    public float PredictedPrice { get; set; }
}

Integrating with the UI

In your Razor Page:

@page
@model PredictionModel
@inject PredictionService PredictionService

<h2>Predict Stock Price</h2>
<form method="post">
    <input type="text" name="Symbol" placeholder="Enter Stock Symbol" />
    <input type="date" name="Date" />
    <button type="submit">Predict Price</button>
</form>

@if (Model.PredictedPrice != null)
{
    <div>
        <p>Predicted Price: @Model.PredictedPrice</p>
    </div>
}

@functions {
    public float? PredictedPrice { get; set; }

    public async Task OnPostAsync(string symbol, DateTime date)
    {
        var input = new PredictionInput { Symbol = symbol, Date = date };
        var result = await PredictionService.PredictStockPriceAsync(input);
        PredictedPrice = result.PredictedPrice;
    }
}

Step 5: Providing Personalized Investment Recommendations

Using AI for Recommendations

Developing the Recommendation Model

Data: User portfolio, investment history, risk tolerance.
Algorithm: Collaborative filtering or content-based filtering.
Deployment: Use Azure Machine Learning to deploy the model as a service.

Consuming the Recommendation Service

public class RecommendationService
{
    private readonly HttpClient _httpClient;
    private readonly string _endpoint = "YOUR_RECOMMENDATION_API_ENDPOINT";
    private readonly string _apiKey = "YOUR_API_KEY";

    public RecommendationService(IHttpClientFactory httpClientFactory)
    {
        _httpClient = httpClientFactory.CreateClient();
    }

    public async Task<List<Recommendation>> GetRecommendationsAsync(string userId)
    {
        var request = new HttpRequestMessage(HttpMethod.Get, $"{_endpoint}?userId={userId}");
        request.Headers.Add("Authorization", $"Bearer {_apiKey}");

        var response = await _httpClient.SendAsync(request);
        response.EnsureSuccessStatusCode();

        var content = await response.Content.ReadAsStringAsync();
        var recommendations = JsonConvert.DeserializeObject<List<Recommendation>>(content);
        return recommendations;
    }
}

public class Recommendation
{
    public string Symbol { get; set; }
    public float ExpectedReturn { get; set; }
}

Displaying Recommendations in the Dashboard

@page
@model DashboardModel
@inject RecommendationService RecommendationService

<h2>Your Personalized Recommendations</h2>

@if (Model.Recommendations != null)
{
    <ul>
        @foreach (var rec in Model.Recommendations)
        {
            <li>@rec.Symbol - Expected Return: @rec.ExpectedReturn%</li>
        }
    </ul>
}

@functions {
    public List<Recommendation> Recommendations { get; set; }

    public async Task OnGetAsync()
    {
        var userId = User.Identity.Name;
        Recommendations = await RecommendationService.GetRecommendationsAsync(userId);
    }
}

Step 6: Data Storage with Entity Framework Core and SQL Server

Installing Entity Framework Core

dotnet add package Microsoft.EntityFrameworkCore.SqlServer
dotnet add package Microsoft.EntityFrameworkCore.Tools

Creating the Data Context

public class EquityDbContext : DbContext
{
    public EquityDbContext(DbContextOptions<EquityDbContext> options)
        : base(options)
    {
    }

    public DbSet<UserPortfolio> Portfolios { get; set; }
    public DbSet<StockTransaction> Transactions { get; set; }
}

public class UserPortfolio
{
    public int Id { get; set; }
    public string UserId { get; set; }
    public List<StockTransaction> Transactions { get; set; }
}

public class StockTransaction
{
    public int Id { get; set; }
    public string Symbol { get; set; }
    public int Shares { get; set; }
    public decimal PriceAtPurchase { get; set; }
    public DateTime PurchaseDate { get; set; }
}

Configuring the Database Connection

In appsettings.json:

"ConnectionStrings": {
  "EquityDb": "Server=(localdb)\\mssqllocaldb;Database=EquityDb;Trusted_Connection=True;"
}

In Program.cs:

builder.Services.AddDbContext<EquityDbContext>(options =>
    options.UseSqlServer(builder.Configuration.GetConnectionString("EquityDb")));

Applying Migrations

dotnet ef migrations add InitialCreate
dotnet ef database update

Step 7: Deploying to Azure

Creating Azure Resources

App Service: For hosting the web application.
Azure SQL Database: For data storage.
Azure Machine Learning Services: For AI models.
Azure Key Vault: For managing secrets.

Deployment Steps

Publish the Application

Use Visual Studio or Azure CLI to publish the application to Azure App Service.

# Create a resource group
az group create --name EquityInvestorRG --location "East US"

# Create an App Service plan
az appservice plan create --name EquityInvestorPlan --resource-group EquityInvestorRG --sku B1

# Create a Web App
az webapp create --resource-group EquityInvestorRG --plan EquityInvestorPlan --name EquityInvestorApp

# Deploy the app
dotnet publish -c Release
az webapp deploy --resource-group EquityInvestorRG --name EquityInvestorApp --src-path ./bin/Release/net8.0/publish

Configure Connection Strings and App Settings

Set environment variables for connection strings and API keys.

az webapp config connection-string set --resource-group EquityInvestorRG --name EquityInvestorApp \
    --settings EquityDb="Server=tcp:yourserver.database.windows.net;Database=EquityDb;User ID=...;Password=...;" --connection-string-type SQLAzure

az webapp config appsettings set --resource-group EquityInvestorRG --name EquityInvestorApp --settings \
    AzureMLApiKey="YOUR_AZURE_ML_API_KEY" \
    AlphaVantageApiKey="YOUR_ALPHA_VANTAGE_API_KEY"

Step 8: Ensuring Security and Compliance

Using Azure Key Vault

Store sensitive information like API keys and connection strings in Azure Key Vault.

Create a Key Vault

az keyvault create --name EquityInvestorVault --resource-group EquityInvestorRG --location "East US"

Add Secrets to Key Vault

az keyvault secret set --vault-name EquityInvestorVault --name "AlphaVantageApiKey" --value "YOUR_API_KEY"

Configure the App to Access Key Vault

In Program.cs, use Managed Identity to access Key Vault.

builder.Configuration.AddAzureKeyVault(new Uri("https://EquityInvestorVault.vault.azure.net/"), new DefaultAzureCredential());

Enabling HTTPS and SSL

Force HTTPS in the Azure App Service settings and configure SSL certificates if using custom domains.

Conclusion

By integrating .NET 8, Azure services, and AI technologies, we've developed a comprehensive equity investment platform that meets the needs of modern investors. This application demonstrates how cutting-edge technologies can be combined to deliver real-time data, predictive analytics, and personalized experiences while maintaining high security and compliance standards.

Securing APIs with OAuth2 and JWT in .NET Core

Paulo Torres — Mon, 16 Sep 2024 19:15:22 +0000

In today's digital ecosystem, securing APIs is more critical than ever. As applications become increasingly interconnected, ensuring that your APIs are protected against unauthorized access is paramount. In this article, we'll explore how to secure APIs using OAuth2 and JSON Web Tokens (JWT) in .NET Core, providing a step-by-step guide along with real-world examples.

Understanding OAuth2 and JWT

OAuth2 is an industry-standard protocol for authorization that enables third-party applications to access user data without exposing credentials. It provides a secure way to delegate access, allowing users to grant limited permissions to apps.

JSON Web Tokens (JWT) are a compact and self-contained way for securely transmitting information between parties as a JSON object. They're widely used for authentication and authorization in web applications due to their efficiency and security.

Why Use OAuth2 and JWT in .NET Core?

Scalability: Stateless authentication with JWT reduces server overhead.
Security: Robust encryption and signature validation protect against tampering.
Interoperability: JWTs are language-agnostic and can be used across different platforms.
Performance: Reduced need for database lookups enhances API responsiveness.

Setting Up Your .NET Core API

Step 1: Create a New Web API Project

dotnet new webapi -n SecureApiExample
cd SecureApiExample

Step 2: Install Necessary NuGet Packages

dotnet add package Microsoft.AspNetCore.Authentication.JwtBearer

Implementing OAuth2 and JWT Authentication

Step 1: Configure Authentication Services

In your Program.cs or Startup.cs (depending on your .NET Core version), configure the authentication services.

using Microsoft.AspNetCore.Authentication.JwtBearer;
using Microsoft.IdentityModel.Tokens;
using System.Text;

var builder = WebApplication.CreateBuilder(args);

// Add services to the container.
builder.Services.AddControllers();

// Configure JWT Authentication
var key = Encoding.ASCII.GetBytes("YourSuperSecretKey");
builder.Services.AddAuthentication(options =>
{
    options.DefaultAuthenticateScheme = JwtBearerDefaults.AuthenticationScheme;
    options.DefaultChallengeScheme = JwtBearerDefaults.AuthenticationScheme;
})
.AddJwtBearer(options =>
{
    options.RequireHttpsMetadata = false; // Should be true in production
    options.SaveToken = true;
    options.TokenValidationParameters = new TokenValidationParameters
    {
        ValidateIssuerSigningKey = true,
        IssuerSigningKey = new SymmetricSecurityKey(key),
        ValidateIssuer = false, // Set to true in production
        ValidateAudience = false, // Set to true in production
        ClockSkew = TimeSpan.Zero
    };
});

var app = builder.Build();

// Enable authentication middleware
app.UseAuthentication();
app.UseAuthorization();

app.MapControllers();

app.Run();

Step 2: Create a Token Generator

Create a controller AuthController to handle user login and token generation.

using Microsoft.AspNetCore.Mvc;
using Microsoft.IdentityModel.Tokens;
using System.IdentityModel.Tokens.Jwt;
using System.Security.Claims;
using System.Text;

[ApiController]
[Route("[controller]")]
public class AuthController : ControllerBase
{
    [HttpPost("login")]
    public IActionResult Login([FromBody] UserCredentials credentials)
    {
        // Validate the user credentials (this is just a demo, use a user store in production)
        if (credentials.Username != "user" || credentials.Password != "password")
            return Unauthorized();

        var tokenHandler = new JwtSecurityTokenHandler();
        var key = Encoding.ASCII.GetBytes("YourSuperSecretKey");
        var tokenDescriptor = new SecurityTokenDescriptor
        {
            Subject = new ClaimsIdentity(new Claim[]
            {
                new Claim(ClaimTypes.Name, credentials.Username),
                new Claim(ClaimTypes.Role, "Admin")
            }),
            Expires = DateTime.UtcNow.AddHours(1),
            SigningCredentials = new SigningCredentials(
                new SymmetricSecurityKey(key), SecurityAlgorithms.HmacSha256Signature)
        };
        var token = tokenHandler.CreateToken(tokenDescriptor);
        var jwtToken = tokenHandler.WriteToken(token);

        return Ok(new { token = jwtToken });
    }
}

public class UserCredentials
{
    public string Username { get; set; }
    public string Password { get; set; }
}

Step 3: Secure Your API Endpoints

Apply the [Authorize] attribute to secure your controllers or actions.

using Microsoft.AspNetCore.Authorization;
using Microsoft.AspNetCore.Mvc;

[Authorize]
[ApiController]
[Route("[controller]")]
public class ValuesController : ControllerBase
{
    [HttpGet]
    public IActionResult GetValues()
    {
        return Ok(new string[] { "Value1", "Value2" });
    }
}

Step 4: Test Your Secured API

Obtain a JWT Token

Use Postman or a similar tool to send a POST request to /auth/login with the following JSON body:

{
    "username": "user",
    "password": "password"
}

Access Protected Endpoint

With the token received, set the Authorization header in your GET request to /values:

Authorization: Bearer your_jwt_token_here

Real-World Example: Securing a Multi-Tier Application

Imagine a financial application where sensitive customer data is accessed via APIs. By implementing OAuth2 and JWT:

Clients: Mobile apps and web clients authenticate once and receive a token.
API Gateway: Validates JWTs before routing requests to microservices.
Microservices: Each service verifies the token and checks user roles and permissions embedded within.

Best Practices

Use HTTPS: Always encrypt communication to prevent token interception.
Keep Tokens Short-Lived: Reduce risk by limiting token lifespan.
Store Secrets Securely: Use environment variables or Azure Key Vault.
Validate Token Claims: Always check issuer, audience, and expiration.
Implement Refresh Tokens: Allow users to renew tokens without re-authenticating.

Conclusion

Securing your APIs with OAuth2 and JWT in .NET Core is a robust solution for modern applications. It not only enhances security but also streamlines authentication and authorization processes. By following the steps and best practices outlined above, you can protect your APIs from unauthorized access while providing a seamless experience for legitimate users.

Performance Optimization in .NET Core: A Comprehensive Guide

Paulo Torres — Thu, 12 Sep 2024 18:41:10 +0000

Performance optimization is a critical aspect of modern software development. With .NET Core, you can build highly scalable and efficient applications, but as complexity grows, it’s essential to adopt best practices and techniques to ensure peak performance. In this article, we’ll explore advanced performance optimization techniques in .NET Core, using real-world examples to demonstrate their practical applications.

Why Performance Optimization Matters

Performance directly impacts user experience, resource usage, scalability, and operating costs. A well-optimized application will:

Reduce response times: Deliver faster results to users.
Minimize resource consumption: Optimize CPU, memory, and bandwidth usage.
Handle high traffic loads: Scale efficiently to accommodate more users.
Improve application stability: Reduce downtime and errors due to performance bottlenecks.

Key Areas for Optimization

Performance optimization in .NET Core can be divided into several key areas:

Optimizing I/O Operations
Improving Memory Management
Asynchronous Programming
Efficient Use of Dependency Injection
Caching Strategies
Database Optimization

1. Optimizing I/O Operations

I/O-bound operations such as file reads/writes, network calls, and database interactions can be major performance bottlenecks. To optimize I/O in .NET Core:

Real-World Example: Asynchronous File Handling

Synchronous I/O operations block threads, reducing the number of concurrent requests your application can handle. Always prefer asynchronous methods for I/O operations:

public async Task<string> ReadFileAsync(string filePath)
{
    using (var reader = new StreamReader(filePath))
    {
        return await reader.ReadToEndAsync();
    }
}

By making I/O operations asynchronous, the application can process other tasks while waiting for the I/O operation to complete, improving overall throughput.

Buffering I/O Operations

Using buffer techniques can also optimize I/O operations. For example, reading data in chunks rather than loading an entire file into memory reduces memory pressure.

byte[] buffer = new byte[4096]; // 4KB buffer
using (var fileStream = new FileStream(filePath, FileMode.Open, FileAccess.Read))
{
    int bytesRead;
    while ((bytesRead = await fileStream.ReadAsync(buffer, 0, buffer.Length)) > 0)
    {
        // Process each chunk
    }
}

2. Improving Memory Management

Memory management is crucial for reducing overhead and improving performance. Inefficient memory usage can lead to increased garbage collection (GC) activity, which can slow down your application.

Use `Span<T>` and `Memory<T>`

Instead of allocating large memory blocks, .NET Core provides Span<T> and Memory<T> to work with contiguous memory efficiently. These types avoid heap allocations and reduce the need for garbage collection.

public void ProcessBuffer(Span<byte> buffer)
{
    // Process buffer data without allocations
}

By using Span<T>, you can pass slices of arrays or memory buffers without copying data, reducing memory pressure.

Avoid Large Object Heap (LOH) Allocations

.NET allocates objects larger than 85,000 bytes in the Large Object Heap (LOH). Frequent LOH allocations can cause memory fragmentation. To avoid this:

Reuse large objects: Instead of frequently creating new large objects, reuse them wherever possible.
Use arrays with caution: Be mindful of allocating large arrays, as they quickly end up in the LOH.

3. Asynchronous Programming

Using asynchronous programming techniques allows your application to handle multiple tasks concurrently without blocking threads, significantly improving throughput for high-traffic applications.

Example: Async HTTP Requests

When making HTTP calls, prefer HttpClient with asynchronous methods to avoid blocking threads:

public async Task<string> FetchDataAsync(string url)
{
    using var client = new HttpClient();
    return await client.GetStringAsync(url);
}

Asynchronous programming is particularly useful in web applications where multiple I/O-bound operations (like database calls or API requests) are made.

4. Efficient Use of Dependency Injection

Dependency Injection (DI) is integral to .NET Core, but it can become a performance bottleneck if not used efficiently.

Scoped vs Transient vs Singleton Services

Understanding the lifecycle of services is critical:

Transient services are created each time they are requested. Use them for stateless and lightweight services.
Scoped services are created once per request. Ideal for database contexts.
Singleton services are created once and shared across the application’s lifetime. Use them for heavy or costly services.

Avoid Overusing Transient Services

If a service is expensive to create (e.g., a service that performs complex calculations), avoid registering it as Transient. Use Scoped or Singleton where appropriate to minimize performance overhead.

services.AddScoped<IProductService, ProductService>();

5. Caching Strategies

Implementing caching strategies can significantly reduce the load on your backend systems, improving response times and reducing resource consumption.

In-Memory Caching

In-memory caching is useful for caching data that is frequently requested but changes infrequently.

services.AddMemoryCache();

Use the IMemoryCache interface to store and retrieve cached data:

public class ProductService
{
    private readonly IMemoryCache _cache;

    public ProductService(IMemoryCache cache)
    {
        _cache = cache;
    }

    public Product GetProduct(int id)
    {
        return _cache.GetOrCreate($"product_{id}", entry =>
        {
            entry.AbsoluteExpirationRelativeToNow = TimeSpan.FromMinutes(5);
            return FetchProductFromDatabase(id);
        });
    }
}

Distributed Caching

For larger, distributed applications, use distributed caches like Redis. This helps ensure that multiple instances of your application can share cached data across servers.

6. Database Optimization

Database performance is crucial for .NET Core applications, especially when dealing with large datasets or frequent queries.

Use Asynchronous Database Queries

As with I/O operations, database queries should be performed asynchronously to avoid blocking threads.

public async Task<Product> GetProductAsync(int id)
{
    return await _dbContext.Products.FirstOrDefaultAsync(p => p.Id == id);
}

Indexing and Query Optimization

Ensure that your database queries are optimized with proper indexing. Indexes significantly improve the performance of SELECT queries but can slow down INSERT and UPDATE operations. Analyze query performance using database tools like SQL Server Profiler or Entity Framework’s DbContext logging.

Real-World Example: Optimizing a .NET Core Web Application

Imagine a high-traffic e-commerce application built using .NET Core. The application’s performance degrades as traffic increases, with users experiencing slow response times and occasional timeouts. Here's how you could optimize the application:

Optimize I/O: Replace synchronous database and file operations with asynchronous ones, allowing the application to handle more concurrent requests.
Implement Caching: Cache frequently accessed product information using Redis to reduce the number of database hits.
Reduce Memory Pressure: Refactor large object allocations to use Span<T> and Memory<T>, minimizing garbage collection overhead.
Optimize SQL Queries: Analyze slow database queries and add indexes to improve performance, reducing query times from seconds to milliseconds.
Use Dependency Injection Efficiently: Change transient services that perform costly operations to singleton or scoped services, reducing the time spent on service instantiation.

Conclusion

Optimizing the performance of .NET Core applications is essential for ensuring that they scale efficiently, provide a seamless user experience, and remain cost-effective. By adopting best practices in asynchronous programming, memory management, caching, and database optimization, you can significantly improve the performance of your applications. Performance tuning is an ongoing process, and regular profiling, monitoring, and analysis are critical to maintaining a high-performance .NET Core application.

Building Scalable APIs with .NET Core and GraphQL

Paulo Torres — Mon, 26 Aug 2024 20:28:22 +0000

In the modern era of web development, building scalable and efficient APIs is crucial for handling the demands of dynamic applications. Traditional REST APIs have served developers well, but GraphQL offers a powerful alternative that can enhance flexibility, reduce over-fetching, and improve performance. This article delves into how you can build scalable APIs using .NET Core and GraphQL, providing advanced techniques and real-world examples to demonstrate the advantages of this approach.

Understanding GraphQL

What is GraphQL?

GraphQL is a query language for your API, and it offers a more efficient, powerful, and flexible alternative to REST. Developed by Facebook, GraphQL allows clients to request exactly the data they need, avoiding the over-fetching or under-fetching common in REST APIs. It also enables clients to aggregate data from multiple sources in a single request.

Key Features of GraphQL:

Single Endpoint: Unlike REST, which typically uses multiple endpoints, GraphQL consolidates all requests through a single endpoint.
Client-Driven Queries: Clients specify exactly what data they need, resulting in more efficient data retrieval.
Strongly Typed Schema: GraphQL uses a strongly typed schema that defines the types of data that can be queried.

Setting Up a .NET Core Project with GraphQL

Step 1: Create a New .NET Core Web API Project

To start, create a new .NET Core Web API project:

dotnet new webapi -n GraphQLApi
cd GraphQLApi

Step 2: Add GraphQL NuGet Packages

Add the necessary GraphQL packages to your project:

dotnet add package HotChocolate.AspNetCore
dotnet add package HotChocolate.Data
dotnet add package HotChocolate.Types

Step 3: Define Your GraphQL Schema

In GraphQL, the schema defines the structure of the API, including the types of data and the queries available.

Example: Defining a Product Schema

Create a Product model:

public class Product
{
    public int Id { get; set; }
    public string Name { get; set; }
    public decimal Price { get; set; }
}

Define a GraphQL type for the Product model:

public class ProductType : ObjectType<Product>
{
    protected override void Configure(IObjectTypeDescriptor<Product> descriptor)
    {
        descriptor.Field(p => p.Id).Type<NonNullType<IdType>>();
        descriptor.Field(p => p.Name).Type<NonNullType<StringType>>();
        descriptor.Field(p => p.Price).Type<NonNullType<DecimalType>>();
    }
}

Step 4: Implement Queries and Mutations

In GraphQL, queries are used to fetch data, while mutations are used to modify data.

Example: Implementing a Query

Create a query class to retrieve products:

public class Query
{
    public IQueryable<Product> GetProducts([Service] ProductDbContext context) =>
        context.Products;
}

public void ConfigureServices(IServiceCollection services)
{
    services.AddPooledDbContextFactory<ProductDbContext>(options =>
        options.UseInMemoryDatabase("ProductsDb"));

    services
        .AddGraphQLServer()
        .AddQueryType<Query>()
        .AddType<ProductType>();
}

Step 5: Set Up the Database Context

Set up an in-memory database for demonstration purposes:

public class ProductDbContext : DbContext
{
    public ProductDbContext(DbContextOptions<ProductDbContext> options)
        : base(options) { }

    public DbSet<Product> Products { get; set; }
}

Seed the database with some initial data:

public static class DataSeeder
{
    public static void Seed(ProductDbContext context)
    {
        context.Products.AddRange(
            new Product { Name = "Laptop", Price = 999.99M },
            new Product { Name = "Smartphone", Price = 499.99M }
        );
        context.SaveChanges();
    }
}

Step 6: Build and Run the API

Build and run your API:

dotnet run

Navigate to https://localhost:5001/graphql to explore the GraphQL Playground, where you can test your queries and mutations.

Advanced Techniques for Scalability

1. Pagination with GraphQL

To manage large datasets, implement pagination in your GraphQL API. GraphQL supports several pagination strategies, including offset-based and cursor-based pagination.

Example: Cursor-Based Pagination

public class ProductConnection
{
    public Connection<Product> GetProducts(
        int first,
        string after,
        [Service] ProductDbContext context)
    {
        return context.Products
            .OrderBy(p => p.Id)
            .ToConnection(first, after);
    }
}

2. Caching with DataLoader

DataLoader is a utility that helps minimize the number of database queries by batching and caching requests. This is particularly useful in GraphQL when you have to resolve fields across multiple types that might require separate database queries.

Example: Using DataLoader

public class ProductByIdDataLoader : BatchDataLoader<int, Product>
{
    private readonly IDbContextFactory<ProductDbContext> _dbContextFactory;

    public ProductByIdDataLoader(
        IBatchScheduler batchScheduler,
        IDbContextFactory<ProductDbContext> dbContextFactory)
        : base(batchScheduler)
    {
        _dbContextFactory = dbContextFactory;
    }

    protected override async Task<IReadOnlyDictionary<int, Product>> LoadBatchAsync(
        IReadOnlyList<int> keys,
        CancellationToken cancellationToken)
    {
        await using var dbContext = _dbContextFactory.CreateDbContext();

        return await dbContext.Products
            .Where(p => keys.Contains(p.Id))
            .ToDictionaryAsync(p => p.Id, cancellationToken);
    }
}

3. Optimizing Performance with Query Complexity Analysis

GraphQL allows clients to specify exactly what data they need, but this can lead to complex queries that impact performance. Implement query complexity analysis to prevent abuse by limiting the depth and complexity of queries.

Example: Implementing Complexity Analysis

services.AddGraphQLServer()
    .AddQueryType<Query>()
    .AddType<ProductType>()
    .AddMaxComplexityRule(100)
    .AddMaxDepthRule(10);

Conclusion

Building scalable APIs with .NET Core and GraphQL offers numerous advantages, from improved efficiency in data retrieval to the flexibility of client-driven queries. By leveraging GraphQL’s powerful features, such as single endpoint access, query complexity management, and caching, you can create APIs that are not only scalable but also highly responsive to the needs of modern applications. Whether you're building an API for internal use or public consumption, integrating GraphQL with .NET Core positions your applications for success in the evolving landscape of web development.

Building Advanced Solutions with Power Apps and .NET Core

Paulo Torres — Mon, 19 Aug 2024 13:54:35 +0000

Power Apps is a powerful platform that enables businesses to create custom applications without extensive coding knowledge. However, for more complex scenarios, integrating Power Apps with .NET Core allows developers to build advanced, scalable solutions that leverage the full power of the .NET ecosystem. This article explores how to integrate Power Apps with .NET Core, using real-world examples to illustrate advanced techniques for building robust, enterprise-level applications.

Understanding Power Apps and .NET Core Integration

What is Power Apps?

Power Apps is a low-code platform developed by Microsoft that allows users to create custom business applications. It provides a range of templates and pre-built components, enabling users to quickly build apps that can be used across various devices. Power Apps integrates seamlessly with other Microsoft services like SharePoint, Dynamics 365, and Office 365, making it a versatile tool for business process automation.

Why Integrate with .NET Core?

While Power Apps is powerful, there are scenarios where its capabilities might not be sufficient for complex business logic or performance-critical operations. Integrating Power Apps with .NET Core allows you to:

Leverage the full capabilities of the .NET ecosystem.
Build custom APIs that can be consumed by Power Apps.
Implement advanced business logic that goes beyond the built-in capabilities of Power Apps.
Securely access on-premises and cloud-based resources.

Setting Up the Integration

Step 1: Create a .NET Core API

The first step in integrating Power Apps with .NET Core is to create a .NET Core API that will serve as the backend for your Power App.

Create a new .NET Core Web API project:

   dotnet new webapi -n PowerAppsIntegration
   cd PowerAppsIntegration

Define your API models: For this example, let's create a simple model representing a product.

   public class Product
   {
       public int Id { get; set; }
       public string Name { get; set; }
       public decimal Price { get; set; }
   }

Create the API controller: Implement the controller to handle HTTP requests.

   [ApiController]
   [Route("api/[controller]")]
   public class ProductsController : ControllerBase
   {
       private static readonly List<Product> Products = new List<Product>
       {
           new Product { Id = 1, Name = "Laptop", Price = 999.99M },
           new Product { Id = 2, Name = "Smartphone", Price = 499.99M }
       };

       [HttpGet]
       public ActionResult<IEnumerable<Product>> Get()
       {
           return Products;
       }

       [HttpGet("{id}")]
       public ActionResult<Product> Get(int id)
       {
           var product = Products.FirstOrDefault(p => p.Id == id);
           if (product == null) return NotFound();
           return product;
       }

       [HttpPost]
       public ActionResult<Product> Post(Product product)
       {
           product.Id = Products.Max(p => p.Id) + 1;
           Products.Add(product);
           return CreatedAtAction(nameof(Get), new { id = product.Id }, product);
       }
   }

Deploy the API: Deploy the .NET Core API to Azure or any other cloud service. Ensure that the API is accessible from the internet, as Power Apps will need to consume it.

Step 2: Secure the API with Azure AD

Security is a critical aspect when integrating Power Apps with .NET Core. One effective way to secure your API is by using Azure Active Directory (Azure AD) for authentication and authorization.

Register the API in Azure AD: Go to the Azure portal and register your API as an application in Azure AD. Configure the API permissions and define scopes that will be required by clients (e.g., Power Apps).
Update the .NET Core API to use Azure AD: Install the necessary NuGet packages and update the Startup.cs to configure authentication.

   public void ConfigureServices(IServiceCollection services)
   {
       services.AddAuthentication(AzureADDefaults.BearerAuthenticationScheme)
           .AddAzureADBearer(options => Configuration.Bind("AzureAd", options));

       services.AddControllers();
   }

   public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
   {
       app.UseRouting();

       app.UseAuthentication();
       app.UseAuthorization();

       app.UseEndpoints(endpoints =>
       {
           endpoints.MapControllers();
       });
   }

Test the secured API: Use tools like Postman to test the API with Azure AD authentication, ensuring that only authorized requests can access the API.

Step 3: Connect Power Apps to the .NET Core API

Now that your API is deployed and secured, the next step is to connect Power Apps to this API.

Create a new Power App: Open Power Apps and create a new app (Canvas App or Model-Driven App) depending on your requirements.
Add a custom connector: In Power Apps, navigate to "Data" -> "Custom connectors" and create a new connector. Provide the API URL, define the operations (GET, POST, etc.), and configure the authentication to use OAuth2 with Azure AD.
Use the connector in Power Apps: Once the custom connector is set up, you can use it to connect to your .NET Core API. For example, you can create a form in Power Apps that allows users to submit data, which is then sent to your API.

   // Example formula in Power Apps to create a new product
   SubmitForm(CreateProductForm);

Display data from the API: You can also retrieve data from the API and display it in Power Apps. For example, use a gallery to list products:

   // Example formula to retrieve products
   Products = MyApi.GetProducts();

Advanced Scenarios

1. Implementing Complex Business Logic

.NET Core allows you to implement complex business logic that goes beyond the capabilities of Power Apps. For instance, you can perform complex calculations, integrate with third-party services, or apply advanced validation rules in the backend.

Example: Calculating Discounts Based on User Role

In your .NET Core API, you can implement logic to apply discounts based on the user's role:

[Authorize(Roles = "Admin")]
[HttpPost("CalculateDiscount")]
public ActionResult<decimal> CalculateDiscount(Product product)
{
    var discount = product.Price * 0.1M;
    return Ok(discount);
}

2. Integrating with On-Premises Systems

In some scenarios, your Power Apps solution may need to interact with on-premises systems. You can achieve this by using a hybrid architecture where your .NET Core API connects to on-premises resources using technologies like Azure Hybrid Connections or a VPN.

Example: Accessing On-Premises SQL Server

Your .NET Core API can connect to an on-premises SQL Server instance to retrieve or update data, and then expose this functionality to Power Apps:

public class OnPremisesDataService
{
    private readonly string _connectionString;

    public OnPremisesDataService(IConfiguration configuration)
    {
        _connectionString = configuration.GetConnectionString("OnPremisesSqlServer");
    }

    public IEnumerable<Product> GetProducts()
    {
        using var connection = new SqlConnection(_connectionString);
        return connection.Query<Product>("SELECT * FROM Products");
    }
}

3. Using Power Automate for Workflow Automation

Power Automate (formerly Microsoft Flow) can be used alongside Power Apps and .NET Core to automate workflows. For example, when a new record is created in Power Apps, Power Automate can trigger a workflow that calls your .NET Core API, sends an email notification, or updates another system.

Example: Triggering a Workflow on Record Creation

In Power Automate, create a flow that triggers when a new item is created in Power Apps:

Trigger: When a new item is created in Power Apps
Action: HTTP request to your .NET Core API to process the data
Action: Send an email notification to the user

Conclusion

Integrating Power Apps with .NET Core opens up a world of possibilities for building advanced, enterprise-grade applications. By leveraging the full power of the .NET ecosystem, you can implement complex business logic, integrate with on-premises systems, and securely expose APIs to Power Apps. Whether you're building custom connectors, implementing advanced authentication, or automating workflows, the combination of Power Apps and .NET Core provides a flexible, scalable, and powerful platform for modern application development.

AI and Machine Learning Integration in .NET Core with ML.NET

Paulo Torres — Thu, 15 Aug 2024 13:33:22 +0000

Artificial intelligence (AI) and machine learning (ML) are transforming industries by enabling data-driven decision-making, automating complex tasks, and providing deep insights through predictive analytics. With .NET Core and ML.NET, Microsoft’s open-source machine learning framework, developers can integrate AI and ML capabilities directly into their .NET applications. This article delves into advanced techniques for integrating AI and ML in .NET Core using ML.NET, with real-world examples to illustrate these concepts.

Introduction to ML.NET

ML.NET is a cross-platform, open-source machine learning framework developed by Microsoft. It allows .NET developers to build, train, and deploy custom machine learning models directly within .NET applications. ML.NET provides a variety of pre-built algorithms for tasks such as classification, regression, clustering, and more. The framework also supports integration with popular ML tools like TensorFlow, ONNX, and AutoML for more advanced scenarios.

Key Features of ML.NET:

Custom ML Models: Build and train custom models using C# or F#.
Integration with .NET Core: Seamlessly integrate ML models into .NET Core applications.
Support for Popular ML Algorithms: Includes support for a wide range of machine learning algorithms.
Cross-Platform: Run on Windows, Linux, and macOS.

Setting Up ML.NET in a .NET Core Project

Step 1: Create a New .NET Core Project

First, create a new .NET Core console application:

dotnet new console -n AIWithMLNET
cd AIWithMLNET

Step 2: Add ML.NET NuGet Packages

Add the ML.NET package to your project:

dotnet add package Microsoft.ML

Step 3: Import Necessary Namespaces

In your Program.cs file, import the necessary ML.NET namespaces:

using Microsoft.ML;
using Microsoft.ML.Data;

Real-World Example: Predicting Housing Prices

To demonstrate the integration of ML.NET in a .NET Core application, we’ll build a model to predict housing prices based on various features such as the number of bedrooms, square footage, and location.

Step 4: Define the Data Model

Create a data model class to represent the input features and the prediction output:

public class HousingData
{
    [LoadColumn(0)]
    public float Size { get; set; }

    [LoadColumn(1)]
    public float Bedrooms { get; set; }

    [LoadColumn(2)]
    public string Location { get; set; }

    [LoadColumn(3)]
    public float Price { get; set; }
}

public class HousingPricePrediction
{
    [ColumnName("Score")]
    public float Price { get; set; }
}

Step 5: Load and Prepare the Data

Load the data from a CSV file and prepare it for training:

var context = new MLContext();
var dataPath = "housing_data.csv";
IDataView data = context.Data.LoadFromTextFile<HousingData>(dataPath, hasHeader: true, separatorChar: ',');

var split = context.Data.TrainTestSplit(data, testFraction: 0.2);
var trainData = split.TrainSet;
var testData = split.TestSet;

Step 6: Define the Data Processing Pipeline

ML.NET allows you to define a pipeline for data preprocessing and model training. In this example, we’ll encode categorical data, normalize the numeric data, and define a regression algorithm for training:

var pipeline = context.Transforms.Categorical.OneHotEncoding("Location")
    .Append(context.Transforms.Concatenate("Features", "Size", "Bedrooms", "Location"))
    .Append(context.Transforms.NormalizeMinMax("Features"))
    .Append(context.Regression.Trainers.FastTree());

Step 7: Train the Model

Train the model using the training dataset:

var model = pipeline.Fit(trainData);

Step 8: Evaluate the Model

Evaluate the model’s performance using the test dataset:

var predictions = model.Transform(testData);
var metrics = context.Regression.Evaluate(predictions, labelColumnName: "Price");

Console.WriteLine($"R^2: {metrics.RSquared}");
Console.WriteLine($"MAE: {metrics.MeanAbsoluteError}");

Step 9: Make Predictions

Once the model is trained and evaluated, you can use it to make predictions on new data:

var predictionEngine = context.Model.CreatePredictionEngine<HousingData, HousingPricePrediction>(model);
var newHouse = new HousingData { Size = 2000, Bedrooms = 3, Location = "Downtown" };
var prediction = predictionEngine.Predict(newHouse);

Console.WriteLine($"Predicted Price: {prediction.Price}");

Advanced Integration: Using Pre-Trained Models with TensorFlow

ML.NET also allows integration with pre-trained models built using TensorFlow. This is particularly useful for tasks like image classification, where training a model from scratch can be time-consuming and require a large dataset.

Step 10: Integrate a Pre-Trained TensorFlow Model

To integrate a TensorFlow model, first, ensure that the TensorFlow .NET binding is installed:

dotnet add package SciSharp.TensorFlow.Redist

Next, load the TensorFlow model and use it for predictions:

var tensorFlowModel = context.Model.LoadTensorFlowModel("model.pb");
var pipeline = tensorFlowModel.ScoreTensorName("output_score")
    .Append(tensorFlowModel.Output("input", "output_score"))
    .Append(context.Transforms.Conversion.MapKeyToValue("PredictedLabel", "PredictedLabel"));

Deploying ML.NET Models in ASP.NET Core

Step 11: Deploying the Model

Integrating ML.NET models into ASP.NET Core allows you to expose the model as a RESTful API. First, create an ASP.NET Core Web API project:

dotnet new webapi -n MLWebAPI
cd MLWebAPI

In the controller, load the trained model and create an endpoint to return predictions:

[Route("api/[controller]")]
[ApiController]
public class PredictionController : ControllerBase
{
    private readonly PredictionEngine<HousingData, HousingPricePrediction> _predictionEngine;

    public PredictionController(PredictionEngine<HousingData, HousingPricePrediction> predictionEngine)
    {
        _predictionEngine = predictionEngine;
    }

    [HttpPost]
    public ActionResult<HousingPricePrediction> Predict([FromBody] HousingData data)
    {
        var prediction = _predictionEngine.Predict(data);
        return Ok(prediction);
    }
}

Conclusion

Integrating AI and machine learning into .NET Core applications using ML.NET opens up a world of possibilities for developers. Whether you’re building predictive models for business applications or leveraging pre-trained models for more complex tasks like image recognition, ML.NET provides a robust and flexible framework for integrating machine learning into your software solutions. With its seamless integration into the .NET ecosystem, ML.NET allows you to build, train, and deploy machine learning models with ease, all within the comfort of the .NET environment.

Blazor WebAssembly for Full-Stack Development: A Comprehensive Guide

Paulo Torres — Mon, 12 Aug 2024 13:51:25 +0000

Blazor WebAssembly has revolutionized web development by allowing developers to build full-stack web applications using .NET and C#. Unlike traditional frameworks that require JavaScript for client-side logic, Blazor enables developers to write client-side code in C# and run it directly in the browser. This article delves into advanced techniques for leveraging Blazor WebAssembly in full-stack development, complete with real-world examples.

What is Blazor WebAssembly?

Blazor WebAssembly is a framework in the .NET ecosystem that allows developers to build interactive web applications using C# instead of JavaScript. It compiles C# code into WebAssembly, a low-level assembly-like language that runs in the browser. This enables .NET developers to build full-stack applications using a single programming language, simplifying the development process and fostering a more cohesive codebase.

Key Features of Blazor WebAssembly:

Single-Language Development: Use C# for both client and server-side code.
WebAssembly: Run compiled .NET code directly in the browser for near-native performance.
Reusable Components: Build UI components that can be reused across different parts of the application.
Seamless Integration: Integrate with existing .NET libraries and APIs, including ASP.NET Core, Entity Framework Core, and more.

Setting Up a Blazor WebAssembly Project

Step 1: Create a New Blazor WebAssembly Project

Using the .NET CLI or Visual Studio, create a new Blazor WebAssembly project:

dotnet new blazorwasm -o MyBlazorApp
cd MyBlazorApp

Step 2: Understand the Project Structure

Blazor WebAssembly projects have a distinct structure:

wwwroot: Contains static assets such as CSS, JavaScript, and images.
Pages: Holds Razor components, which are .razor files defining the UI and logic for individual pages.
Shared: Contains shared components that can be used across multiple pages.
Program.cs: The entry point for the application, where services are configured and the app is bootstrapped.

Step 3: Build a Real-World Example: Task Management Application

Creating the Task Model

First, create a model class to represent tasks:

public class TaskItem
{
    public int Id { get; set; }
    public string Title { get; set; }
    public bool IsCompleted { get; set; }
}

Creating the Task Service

Next, create a service to manage tasks. This service will handle CRUD operations.

public class TaskService
{
    private readonly List<TaskItem> _tasks = new List<TaskItem>();

    public TaskService()
    {
        // Sample data
        _tasks.Add(new TaskItem { Id = 1, Title = "Learn Blazor", IsCompleted = false });
        _tasks.Add(new TaskItem { Id = 2, Title = "Build a Blazor App", IsCompleted = false });
    }

    public IEnumerable<TaskItem> GetTasks() => _tasks;

    public TaskItem GetTaskById(int id) => _tasks.FirstOrDefault(t => t.Id == id);

    public void AddTask(TaskItem task) => _tasks.Add(task);

    public void UpdateTask(TaskItem task)
    {
        var existingTask = GetTaskById(task.Id);
        if (existingTask != null)
        {
            existingTask.Title = task.Title;
            existingTask.IsCompleted = task.IsCompleted;
        }
    }

    public void DeleteTask(int id) => _tasks.RemoveAll(t => t.Id == id);
}

Registering the Service in `Program.cs`

Add the TaskService to the dependency injection container:

builder.Services.AddSingleton<TaskService>();

Creating the Task List Component

Create a new Razor component to display the list of tasks:

@page "/tasks"
@inject TaskService TaskService

<h3>Task List</h3>

<ul>
    @foreach (var task in TaskService.GetTasks())
    {
        <li>
            <input type="checkbox" @bind="task.IsCompleted" />
            @task.Title
        </li>
    }
</ul>

Adding Task Management Features

Add features to create, update, and delete tasks:

@code {
    private string newTaskTitle;

    private void AddTask()
    {
        var newTask = new TaskItem
        {
            Id = TaskService.GetTasks().Max(t => t.Id) + 1,
            Title = newTaskTitle,
            IsCompleted = false
        };
        TaskService.AddTask(newTask);
        newTaskTitle = string.Empty;
    }

    private void DeleteTask(TaskItem task)
    {
        TaskService.DeleteTask(task.Id);
    }
}

Advanced Blazor WebAssembly Techniques

1. Implementing Authentication and Authorization

Blazor WebAssembly supports various authentication methods, including ASP.NET Core Identity and third-party providers like Google or Microsoft. Implementing authentication involves configuring services in Program.cs, creating custom authentication states, and protecting routes with [Authorize].

Example: Configuring Authentication with Azure AD B2C

builder.Services.AddMsalAuthentication(options =>
{
    builder.Configuration.Bind("AzureAdB2C", options.ProviderOptions.Authentication);
});

2. Server-Side Blazor with SignalR

While Blazor WebAssembly runs entirely in the browser, Blazor Server relies on SignalR to manage client-server communication. This hybrid approach allows for real-time updates and state management on the server.

Example: Real-Time Notifications

public class NotificationService
{
    public event Action OnNotificationReceived;

    public void ReceiveNotification(string message)
    {
        OnNotificationReceived?.Invoke(message);
    }
}

3. Integrating with APIs

Blazor WebAssembly applications often need to consume external APIs. Use HttpClient to interact with RESTful services.

Example: Consuming a REST API

@inject HttpClient Http

@code {
    private IEnumerable<TaskItem> tasks;

    protected override async Task OnInitializedAsync()
    {
        tasks = await Http.GetFromJsonAsync<IEnumerable<TaskItem>>("https://api.example.com/tasks");
    }
}

4. Optimizing Performance

Blazor WebAssembly's performance can be optimized by leveraging techniques like lazy loading, JavaScript interop, and pre-rendering.

Example: Lazy Loading Components

@page "/lazy"
@using System.Threading.Tasks

<h3>Lazy Loaded Component</h3>

@if (isLoading)
{
    <p>Loading...</p>
}
else
{
    <p>Data Loaded!</p>
}

@code {
    private bool isLoading = true;

    protected override async Task OnInitializedAsync()
    {
        await Task.Delay(2000); // Simulate a delay
        isLoading = false;
    }
}

5. Deploying Blazor WebAssembly Applications

Blazor WebAssembly applications can be deployed to various platforms, including Azure Static Web Apps, GitHub Pages, or as a Progressive Web App (PWA).

Example: Deploying to Azure Static Web Apps

Create a new Static Web App in the Azure portal.
Configure GitHub Actions to automate the deployment process.
Push your code to GitHub, and Azure will automatically build and deploy the application.

Conclusion

Blazor WebAssembly represents a significant shift in web development, allowing .NET developers to leverage their skills across the full stack. By enabling C# code to run directly in the browser, Blazor eliminates the need for JavaScript, providing a unified development experience. Through real-world examples, this article has demonstrated how to build a full-stack application with Blazor WebAssembly, covering everything from basic setup to advanced features like authentication, real-time updates, and performance optimization. As Blazor continues to evolve, it offers exciting possibilities for building modern, interactive web applications using .NET.

DEV Community: Paulo Torres

Building Scalable E-commerce Systems: A Journey Through Microservices with ASP.NET Core and Kubernetes

The Challenge

Our Microservices Solution

Architecture Overview

Infrastructure Implementation

Inter-Service Communication

Monitoring and Observability

Service Mesh Implementation

Lessons Learned

Results

Future Improvements

Conclusion

Building Resilient and Fault-Tolerant Applications with Azure Resiliency Patterns

Why Resilience Matters

Azure Resiliency Patterns Overview

1. Retry Pattern

Implementation in .NET Core with Polly

2. Circuit Breaker Pattern

Implementation with Polly

3. Bulkhead Isolation

Implementation in Azure App Service

4. Timeout Pattern

Implementation in .NET Core with Polly

5. Failover Pattern

Implementation with Azure Traffic Manager

6. Queue-Based Load Leveling

Example: Azure Service Bus with .NET Core

Real-World Use Case: E-Commerce Application

Best Practices for Building Resilient Applications

Conclusion

Optimizing Performance in Cloud-Native Applications with .NET Core 8 and Azure

The Importance of Performance Optimization

1. Optimize API Performance with .NET Core 8

Efficient Asynchronous Programming

Example: Optimized API Controller

Caching Frequently Accessed Data

Example: Implementing Redis Caching

2. Optimize Database Interactions

Use Efficient Querying Techniques

Example: Using .AsNoTracking() for Read-Only Queries

Database Indexing

Scaling Databases with Azure SQL

3. Leverage Azure-Specific Optimizations

Azure Functions for Background Processing

Real-World Scenario: Sending Order Confirmation Emails

Static Content Delivery with Azure Front Door

4. Monitor and Diagnose Performance Issues

Application Insights

Example: Adding Application Insights to .NET Core

5. Implement Load Testing and Auto-Scaling

Load Testing

Key Metrics to Monitor:

Auto-Scaling with Azure App Service

Conclusion

Fine-Tuning Large Language Models (LLMs) with .NET Core, Python, and Azure

Table of Contents

1. Introduction

2. Why Fine-Tune Large Language Models?

3. Solution Overview

Components and Technologies

4. Setting Up the Environment

Prerequisites

5. Training and Fine-Tuning with Python

5.1 Set Up a Virtual Environment

5.2 Install Dependencies

5.3 Create an Azure Machine Learning Workspace

5.4 Fine-Tuning Script (train.py)

5.5 Register the Model in Azure

6. Integrating the Fine-Tuned Model in .NET Core

6.1 Create a .NET Core Web API Project

6.2 Add Dependencies

6.3 ModelConsumerService.cs

6.4 LLMController.cs

6.5 Configure Your .NET Core App

7. Deploying to Azure

8. Best Practices

9. Conclusion

Building AI Agents for Automated Trading Systems Using .NET C# Semantic Kernel, Azure AI Services, and Azure Functions

Table of Contents

Example: Using `.AsNoTracking()` for Read-Only Queries