DEV Community: Yaroslav Yarmoshyk

The cost of self-hosted LLM model in AWS

Yaroslav Yarmoshyk — Fri, 28 Feb 2025 10:27:38 +0000

Introduction

There are numerous reasons why you’d like to run an LLM model locally and isolated from the internet instead of using the public OpenAI, Meta or Deepseek apis.

For me the most important are the following:

Data privacy
- Some industries (healthcare, finance, legal) require sensitive or proprietary data to remain on-premises or within specific geographic regions
- Stringent regulations (e.g., HIPAA, GDPR) by avoiding data transfers to external third-party services.
Security
- The content, generated or processed, must remain confidential. A local solution prevents sending queries to an external API
- You have end-to-end control (network, physical access, encryption at rest/in transit) when models are self-hosted.

Everything that includes PII of your clients or the business confidential information should never be uploaded to the public services.

In order to meet these requirements, while still being able to use LLM to boost performance of your organization, the local setup in your cloud account can be the golden bullet.

What is the financial side of this setup?

I prepared an approximate forecast to run the LLama models v3.2 based on requirements in AWS Cloud.

In my calculations I covered the following 2 cases:

The LLM should be online only during the working hours (40h/week)
LLM should be available 24/7
No saving plans, reserved instances, upfront payment included.

Llama Name	Possible EC2 Instance	Instance Details	Monthly Price (40 hrs/week)	Monthly Price (168 hrs/week)
Llama 3.2 1B Instruct	g4dn.xlarge	16GB RAM 4 vCPUs 1 GPU (NVIDIA T4)	$91.42	$383.98
Llama 3.2 3B Instruct	g4dn.2xlarge	32GB RAM 8 vCPUs 1 GPU (NVIDIA T4)	$130.70	$548.96
Llama 3.2 11B Vision	g5.8xlarge	128GB RAM 32 vCPUs 1 GPU (24GB Memory)	$429.33	$1,803.04
Llama 3.2 90B Vision	g5.48xlarge	768GB RAM 192 vCPUs 1 GPU (192GB Memory)	$2,834.85	$11,906.24

Notes on the Table

Possible EC2 Instance was selected based on the LLama models v3.2 requirements
- For smaller Instruct models (1B, 3B), a single g4dn or g5 instance with an NVIDIA T4 should be enough.
- For 11B Vision, the g5.8xlarge meets the minimum 22 GB VRAM requirement (A10G has 24 GB VRAM).
- For 90B Vision, you typically need multiple high-end GPUs. The g5.48xlarge offers 8× A100 GPUs (40 GB each = 320 GB total VRAM) plus sufficient CPU and RAM.
Monthly Price Calculated from approximate On-Demand hourly rates in us-west-2 (Oregon). Prices shown are for:
- 160 hours/month (40 hrs/week)
- 720 hours/month (24×7 usage). Actual AWS rates may vary slightly by region and can change over time.
Storage The selected instances typically come with local NVMe SSD volumes. In production, you’ll often attach an EBS volume to meet or exceed the required disk space. EBS costs are not included in the prices above.

Here is a link to the pricing calculator. You can use it as a baseline in your cost forecasts.

Optimization Options

Reserved Instances or Savings Plans can drastically reduce hourly rates.
Spot Instances offer lower prices but can be interrupted.
For large models, you might also explore distributed training/inference techniques to scale across multiple smaller GPUs.

Always confirm instance pricing using the official AWS Pricing Calculator or up-to-date AWS documentation for G4 instaces and G5 instances.

Detect Inappropriate Content with AWS Rekognition

Yaroslav Yarmoshyk — Wed, 15 Jan 2025 09:39:13 +0000

Introduction

In this article I want to describe how to use AWS Rekognition service to detect and block the images that doesn't comply with your content policy and can affect the reputation of your website, leading to user dissatisfaction, loss of trust, and potential legal issues.

This case is applicable to any website that allows users to upload images with subesequent location of these images in the S3 bucket.

Let’s say you have a website with the feature of file uploads. Users can publish inappropriate images or videos that can affect the reputation of your website, leading to user dissatisfaction, loss of trust, and potential legal issues.

Another possible case is related to Educational Platforms where one of your teachers accidentally uploads his home videos instead of learning material.

Problem: User-uploaded images may contain explicit or harmful content such as nudity, violence, hate symbols or other inappropriate material. If such images remain publicly accessible, they can violate platform policies, offend users and potentially lead to legal issues.
Solution: use AWS Rekognition to automatically detect such content and move it into a secure, non-public location. The system ensures that inappropriate content is promptly removed from public access.

This is more a solution design rather then How-To implementation guide.

Disclaimer

AWS Rekognition is a comprehensive image and video analysis service offered by Amazon Web Services (AWS). It is powered by a deep learning technology and requires no machine learning expertise to use.

It provides object and scene detection, allowing for the identification of various elements within images and videos. Facial analysis and recognition features allows it to detect faces, emotions and even celebrity recognition, making it useful for security and personalization applications.

Additionally, content moderation tools help automatically identify and filter inappropriate or explicit content, ensuring compliance and safety. The following article will be primarily focused on using the recognition for content moderation.

For those of you who is looking for more detailed information, you can visit the AWS Rekognition Overview and check its Key Features.

Automated Content Moderation With AWS Recognition

Data Flow Summary

Upload Path: User → CloudFront → Public S3 Bucket
Processing Path: CloudWatch Event → Lambda Function → AWS Rekognition
Response Path: If flagged → Move to Secure S3 Bucket + SES Email Notification

User Interaction:
- Action: A user uploads an erotic image.
- Storage: The image is stored in a public Amazon S3 bucket, which is accessible via Amazon CloudFront for content delivery.
Event Trigger:
- Service: Amazon CloudWatch Event Rule
- Function: Scheduled to trigger a Lambda function every hour, ensuring periodic checks of newly uploaded content.
Lambda Function Execution:
- Language: Python
- Tasks:
  - File Retrieval: The function scans the S3 bucket to identify and list files that have been uploaded in the last 62 minutes.
  - Content Analysis: For each identified image, the Lambda function invokes AWS Rekognition to analyze the content against predefined labels. The the DetectLabels operation is used
AWS Rekognition Analysis:
- Labels Checked:
  - Detected Nudity
  - Violence
  - Gambling
  - Rude Gestures
  - Hate Symbols
  - Drugs & Tobacco
  - Alcohol Use
  - Exposed Buttocks or Anus
  - Explicit Nudity
  - Explicit Sexual Activity
  - Obstructed Intimate Parts
- Outcome: Determines whether any of the specified labels are present in the image.
Conditional Handling Based on Analysis:
- If Labels are Detected:
  - File Management: The image is moved from the public S3 bucket to a secure, non-public S3 bucket to prevent further public access. You can not set ACLs on an object level. This will not block access to the object. You can either move it into a non-public folder of the existing s3 bucket or into another S3 Bucket without public access.
  - Notification: An email notification is sent to recognized@example.com using Amazon Simple Email Service (SES), alerting the relevant parties about the detection.

Speaking about the possible list of labels that can be detected

Customers can download the list of supported labels and object bounding boxes from our documentation page or from the 'Label detection' tab of the Amazon Rekognition Console. In addition, on the Rekognition console, customers can use a search bar to easily check whether their label is already supported or not. Using the same interface, customers can request new labels that they would like Amazon Rekognition to support, or provide any other product feedback.

External links:

Protect nginx ingress with AWS WAF and AWS Shield

Yaroslav Yarmoshyk — Mon, 09 Dec 2024 09:19:16 +0000

I have

The application running in AWS EKS. The application frontend is exposed to the world over nginx ingress controller.

I need to

Enable AWS WAF and AWS Shield protection from Application and Network attacks with AWS WAF and AWS Shield.

Why do I need this?

AWS WAF helps to prevent malicious attacks such as SQL injection attacks and cross-site scripting, aligning with OWASP top 10 list. Complete list of features is available here.

AWS Shield is primarily used to protect from distributed denial of service (DDoS) attacks. It automatically detects threats to the environment

I know that

Nginx ingress controller creates Network Load Balancer for it's service.

Also I know the OSI model and the Application is level 7 while NLB is level 4. Such integrations are incompatible.
AWS WAF doesn't work for Network Load Balancers.

AWS WAF

is applicable only to the following resources

Amazon CloudFront distribution
Amazon API Gateway REST API
Application Load Balancer
AWS AppSync GraphQL API
Amazon Cognito user pool
AWS App Runner service
AWS Verified Access instance

AWS Shield

.. comes with 2 subscription models Standard and Advanced.

AWS Shield Advanced provides expanded DDoS attack protection for

Amazon EC2 instances
Elastic Load Balancing load balancers
CloudFront distributions
Route 53 hosted zones
AWS Global Accelerator standard accelerators.

AWS Shield Advanced incurs additional charges. The pricing breakdown can be found here. To be short: it costs ~$3k/month with a yearly subscription so you need to pay $36k to enable it for your AWS organization.

AWS Shield Standard is automatically included at no extra cost beyond AWS managed services. This means that you have no control over the configuration of Shield standard. It is used by AWS to protect the managed services like AWS Cloudfront distributions, route53 resolvers and global accelerator.

AWS CloudFront is the answer

Taking a look at the statements above we can use AWS Cloudfront to take advantage of built-in AWS Shield basic and enable filtering with AWS WAF Rules. The only trick is to disable caching where it is not needed. Really often caching can affect your workload experience.

Additionally you will need to ensure your NGINX Ingress Controller on EKS handles only traffic routed through CloudFront.
To achieve this you can Configure CloudFront to add a custom header (e.g., X-CloudFront-Secret) to all requests and next update your NGINX Ingress configuration to validate this custom header.

Use the following annotation in the Ingress resource to set up a whitelist of allowed headers.
Example:

nginx.ingress.kubernetes.io/server-snippet: |
  if ($http_x_cloudfront_secret != "YourSecretValue") {
    return 403;
  }

It has to be applied to every Ingress resource in your workloads.

Alternatively you can apply the custom header validation globally at the controller level by modifying the NGINX configuration through a ConfigMap or custom template.

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
data:
  http-snippet: |
    map $http_x_cloudfront_secret $valid_secret {
      default 0;
      "YourSecretValue" 1;
    }

    server {
      if ($valid_secret = 0) {
        return 403;
      }
    }

Bonus part

You can not be 100% confident that managed services will save you from all the problems. This is why I recommend to create the custom WAF ACL with the list of IPs that should be blocked from accessing your website.

In this case you will have a quick path to block IP address or IP CIDR if you see the suspicious activity coming form those to remain safe while you are sorting out what is missing in your security configuration.

How to enable AWS S3 replication between Global AWS Region and AWS China

Yaroslav Yarmoshyk — Mon, 04 Nov 2024 10:29:49 +0000

Recently I was struggling to enable S3 replication from China to EU region and I made it work using the Data Transfer Hub.

Actually it is not an AWS managed solution. It consists of the number of AWS resource such as EC2, SQS, Lambda functions and DynamoDB table.

I didn't need a fancy UI so I used the simplified s3-plugin approach

It is well documented so there is not much to add, however I'd like to outline couple of points:

You need to deploy the stack into global region.
The IAM user has to be created in China.
The entire solution is automated in Cloudformation.
The DataTransferS3Stack.template didn't work for me. There is something wrong with IAM policy on line #1866. Everything worked out without this resource.

The actual resource composition diagram is the following:

Proper setup of IAM federation in Multi-account AWS Organization for Terragrunt

Yaroslav Yarmoshyk — Tue, 20 Aug 2024 13:44:12 +0000

Introduction

In this article I want to describe how to configure the IAM relationships in a multi-account AWS organization with AWS SSO to allow managing infrastructure as a code with terragrunt/terraform from both CI/CD runner and local PCs.

This is NOT a beginner guide. This is the solution design with very brief code examples.

The picture above is the most common setup when we have an IAM spoke/assumer role in the shared account that assumes the spoken/doers roles in the application accounts.

Every spoken/doer role in the application accounts has the IAM policy attached that allows them to manage resources.

The following schema has more details:

According to this diagram we have EC2 instances that are registered as self-hosted runners in the CI/CD system (Github Actions, GitlabCI, Jenkins, etc.)

The spoke/assumer role is attached to the instance. When we run terraform/terragrunt on this instance it can assume the spoken/doers roles in other accounts and apply the infrastructure as a code changes.

This is achieved by the following in aws provider configuration

provider "aws" {
  assume_role {
    role_arn     = "arn:aws:iam::123456789012:role/doer-role"
    session_name = "doer-session-123456789012"
  }
}

If you are using terragrunt (which is highly recommended) then it will automatically create the following resources using this spoked/doer role in every target account:

S3 bucket for statefiles
Dynamodb for state file locks

Shared resources in the Shared aws account

Everything is OK until you don’t need to create shared resources in the Shared aws account. For example global secrets or resource for monitoring (Managed Prometheus) or log aggregation (Opensearch)

In this case one more spoken/doer role is needed in the shared account and terragrunt will create the resources for statefiles.

Reference terragrunt outputs between accounts

The problem appears when you need to reference the outputs of terragrunt resource from shared as inputs for resources in other account(s).

Here is the example of terragrunt folder structure

In this case

The eks_controllers in dev/stage/prod depend on the opensearch and prometheus in a shared account.
The transit_gateway_attachments depend on the transit_gateway in shared account

And the problem is the spoken/doer roles of these accounts can not access the terraform statefile of the shared account

You apply infrastructure updates to a shared account first, get outputs and hardcode them as inputs for workload accounts.
But what if there are more dependencies?

Use single s3 bucket for terraform state

Terraform allows the use of a single state bucket to locate state files of multiple accounts.

This is being achieved with Assume Role Configuration:

terraform {
  backend "s3" {
    bucket = "terraform-state-prod"
    key    = "network/terraform.tfstate"
    region = "us-east-1"
    assume_role = {
      role_arn           = "arn:aws:iam::SHARED-ACCOUNT-ID:role/state-mgmt-role"
    }
  }
}

In terragrunt, the remote state with single role looks like the following:

remote_state {
  backend = "s3"
  config = {
    bucket         = "terraform-state-SHARED-ACCOUNT-ID"
    dynamodb_table = "terraform-state-lock-SHARED-ACCOUNT-ID"
    key            = "${path_relative_to_include()}/terraform.tfstate"
    role_arn       = "arn:aws:iam::SHARED-ACCOUNT-ID:role/state-mgmt-role”
    region         = "us-east-1"
  }
}

In this case we will need to create one more role for state management in the shared account and the infrastructure will look like the following:

At this stage we can reference the outputs of the resources from 0-shared account as inputs for resources in 1-dev, 2-stage and 3-prod accounts as shown below:

Happy days if all we need is to run a terragrunt plan or terragrunt apply from the CI/CD pipeline.

Run terragrunt plan locally

What if we need some kind of breaking glass access in case of CI/CD failure so that we could update infrastructure from our workstation or run terragrunt plan locally?

Let’s add the AWS SSO into our setup.

How to make it work with SSO roles and consolidated terraform state management role?
There are two ways how to make it happen

Option #1: Allow all IAM roles to assume state management role

If you have AWS SSO then you have the following AWSReserverdSSO_* IAM roles in all accounts with random suffix in the the name

If you copy environment variables from AWS SSO page to authorize in AWS and run your terragrunt locally then it will not work until you update the trust relationships of the state-managemt role with the arns of IAM Roles

Basically you’ll need to add every AWSReserverdSSO_* role from each AWS account in your organization into the list of allowed principles of the state-management role in shared account.

This seems to be complicated from operations and automation perspective

Option #2: Allow IAM roles from Management account to assume spoke/assumer role

This is the best approach.

Configure the AWS CLI with IAM Identity Center authentication
Update the list of IAM principals in trust policy of spoke/assumer role with the arn of the SSO roles you’d like to be able to assume it.
Add the profiles into your ~/.aws/config file.

[profile shared]
sso_session = sso
sso_account_id = 123456789011
sso_role_name = AWSAdministratorAccess
region = us-west-2
output = json

[profile assumer]
source_profile = shared
role_arn = arn:aws:iam::123456789011:role/infrastructure-assumer-role

Make sure the update the following:

sso_account_id
sso_role_name
role_arn
region

This will configure your workstation to perform the following actions:

sts:assume the AWSAdministratorAccess or AWSPoweruserAccess SSO role from a shared account.
STS:Assume infrastructure assumer role from shared account
All other roles will be assumed by terragrunt automatically.

Usage:

aws sso login --sso-session sso
export AWS_PROFILE=assumer

Follow the steps in your browser.
After this you can run

terragrunt plan

How to migrate DNS records from CloudFlare to AWS Route53 with Terraform&Terragrunt

Yaroslav Yarmoshyk — Sun, 26 May 2024 11:26:54 +0000

Possible reasons

There are multiple reasons for such migration. The most common are the following:

You'd like to use external-dns controller in your EKS cluster to manage DNS records automatically for you, however the CloudFlare support is still in beta and you don't want to use it for production workloads.
You want to take the advantages of AWS WebApplication Firewall instead of CloudFlare WAF.

There might be other reasons but I faced the 2 in the most resent project.

You'll need to put either Cloudfront distribution or ApplicationLoadBalancer (ALB) in front of your web application to use AWS WAF because it provides the application level protection so it can not be enabled for NetworkLoadBalancer (NLB)

Migration flow

Read all the records from the existing CloudFlare DNS zone. You can re-use the python script I've prepared. The automation is available in github.com/yyarmoshyk/read-cloudflare-dns-records The readme file describes how to use it.
Create DNS zone in AWS You don't need to invest much efforts into this. Feel free to re-use the existing terraform-aws-route53 community module
Create DNS records in AWS The script above produces the json output that can be used as an input for the terraform-aws-route53/records terraform module

  {
    "name": "example.com",
    "type": "A",
    "ttl": 300,
    "records": [
      "10.10.10.10"
    ]
  }

The output should be saved into the file. Next the contents can be read with terrafrom/terragrunt and specified as inputs to the terraform-aws-route53/records terrafrom module

    records_jsonencoded = jsondecode(file("dns_records.json"))

Update NameServer configuration in your current DNS registrar. For this you'll need to refer to the documentation of the DNS provider where your domain is registered.

I will not cover running terragrunt apply procedure here. There are many documents about this over the internet.

Closing words

Most of the time you'll spend on creating the API token in CloudFlare and injecting the route53 provisioning into your existing IaaC structure.
Basically we extract the data from cloudflare, convert it into proper format, next create all records with terragrunt or terraform.

AWS Cloud Platform for highly loaded WordPress website

Yaroslav Yarmoshyk — Mon, 29 Apr 2024 09:46:01 +0000

Today I'd like to share the simple (or not), highly available and secure design of the AWS Cloud Solution to host the highly loaded WordPress website. I think it is also applicable to other popular CMS such as Joomla, Drupal, etc.

The diagram doesn't cover the multi-OU setup for simplicity purposes.

Explanations

Networking

Public - the primary purpose for this network is to locate load balancer and NAT gateways (1 per availability zone).
Private - all EC2 instances will be hosted here. I skipped routing tables to simplify the schema. Ofcourse you need to define routing rules for Private subnets.
Isolated - is a home for multi-AZ Aurora RDS and EFS mount endpoints.

Internet Gateway is used to communicate with the world.

File Storage

Elastic File System (EFS) is planned to be used to locate the code of the website. It has to be mounted to every webserver root (ex. /var/www/html).

AWS Backup is to be used to create scheduled backups.

S3 is intended to be used for static files. To be honest I didn't test this approach but I found the following WordPress plugin that allows to locate uploads to s3. So I'd appreciate if whoever decides to implement this setup could drop off a comment whether it works or not.

If it doesn't work then we'll need to add S3 File Gateway to the schema and mount s3 into $ROOT/wp-content/uploads folder

I think S3 Intelligent tiering should be enabled for it to enable automatic storage cost savings by automatically moving data to the most cost-effective access tier when access patterns change.

Data Storage

The RDS Aurora with MySQL engine is to be used. It doesn't require much configuration except of the right sizing. It is recommended to store the password and read/write endpoint address in SSM Parameter Store or AWS Secret Manager and next read it on EC2 boot with userdata and expose as environment variables to be read by WordPress.

The automatic daily snapshotting should be enabled. Sometimes you might need to recover from it instead of point in time recovery.

Public Access and Content Delivery

In order to expose the website to the internet the combination of the following resources will be used:

Application Load Balancer. The load balancer will distribute the load to the EC2 instances in the autoscaling group. In case of really high traffic this can be updated to Network Loadbalancer because.
CloudFront is a CDN that will deliver traffic to the users from the closes edge location and will provide front-end caching capabilities to reduce the load on the compute. Also it will be configured to deliver static files directly from the S3 bucket (URI: /wp-content/uploads/*)

Security

This area includes multiple items:

Cloudfront comes with AWS Shield standard enabled. No configuration is needed at this point. However we need to create proper S3 bucket policy to allow reading objects only with a certain CloudFront origin access identity. Here is the manual.
AWS Web Application Firewall (WAF) with a set of rules is to be used to protect ALB from L7 (Application) attacks. A good practice is to create the ip ruleset and custom ACL linked to it to have an ability to blacklist IPs on manually (ex. during DDoS attacks). Here is the manual
KMS was skipped to simplify the schema but it is recommended to create customer-managed key for every resource and use those to encrypt data. Resources to be covered are the following:
1. S3 buckets
2. EBS volumes of EC2 instances
3. RDS volumes and snapshots
Macie is a great ML-powered solution to audit the files in S3 buckets and detect any sensitive data published (ex. PII, passwords, etc.)
AWS Certificate Manager (ACM) is probably one of the oldest services in AWS that allows to create SSl certificates to enable traffic encryption. The certificates are to be wired to CloudFront distribution.
Security groups were skipped in the diagram to avoid visual complexity. 3 security groups are to be created:
1. ALB - allow access to loadbalancer. Port 80 is to be open to 0.0.0.0/0. Since we will do ssl termination at CloudFront level, there is no need to listen to port 443. Transit encryption is optional inside AWS networks.
2. Compute - allow access from loadbalancer to EC2. Port 80 open to alb-security-group-id.
3. EFS - ports 111 and 2049 (TCP + UDP) are to be opened to compute-security-group-id
4. RDS - port 3306 (TCP) is to be opened to compute-security-group-id
IAM role and instance profile to be attached to the EC2 instances has to have sufficient permissions to find and mount EFS volumes. Also it needs to have permissions to read/write in the S3 bucket for static files. Use IAM policy generator to create the IAM policy

Compute

I use autoscaling group to start instances in multiple availability zones with scaling policy based on built-in metric of CPU utilization that is being sent to CloudWatch. Standard approach. If the load goes up then we need more servers to handle it.

The missing piece of puzzle is the AMI "golden image" that will be used to start the instances in autoscaling group. The AMI has to have NGINX and PHP installed with the list of required modules enabled. The great tool to brew one is hashicorp packer.

Additionally, the userdata script should find the EFS mount point in the current Availability Zone and mount it to the NGINX WebServer root as I mentioned earlier.

Automation possibilities

I am not only a big fan of hashicorp terraform. I'm also one of the early adopters of it. So this is my main go-to Infrastructure as a Code tool. However all the resources I use are supported by other IaaC solutions such as AWS CloudFormation and AWS CDK. You definitely got to use one to avoid loosing the track of resources you create.

Summary

This solution satisfies the following requirements to safely run your website in AWS Cloud:

High availability (multi AZ provisioning)
On-demand capacity (autoscaling)
Encryption at rest and in transit (KMS + SSL)
Least access privilege (IAM, Security Groups with explicit rules)
Data safety (AWS Backup + Automatic snapshotting of RDS)
Security audit (Macie)
Secure communications (AWS Shield + WAF + 3-tier networking)

It is relatively easy to deploy code updates. All you need to do is just to copy over the updated files to NFS. No restarts are required.

The bottleneck here can be network limitations of NFS. In case of really high traffic you might need to adjust throughput settings but be aware of additional costs.