The latest articles on DEV Community by Aviram Galim (@avigalim). https://dev.to/avigalim https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3937347%2F449a3967-4b66-4c82-bd1c-5e578df01678.jpg https://dev.to/avigalim en Aviram Galim Mon, 18 May 2026 06:37:43 +0000 https://dev.to/avigalim/how-i-deployed-llama-31-on-aws-ec2-g4dnxlarge-with-llamacpp-real-numbers-5020 https://dev.to/avigalim/how-i-deployed-llama-31-on-aws-ec2-g4dnxlarge-with-llamacpp-real-numbers-5020 <p>Tired of paying per token? I set up a self-hosted Llama 3.1 inference endpoint on an AWS GPU instance using llama.cpp. Here's what it actually looks like end to end.</p> <h2> The Setup </h2> <ul> <li>Instance: g4dn.xlarge (NVIDIA Tesla T4, 15 GB VRAM) - $0.53/hour on-demand</li> <li>Model: Llama 3.1 8B Instruct, Q4_K_M quantized (4.58 GiB)</li> <li>Backend: llama.cpp compiled with '-DGGML_CUDA=ON'</li> <li>API: OpenAI-compatible REST endpoint on port 8080</li> </ul> <h2> Real benchmark numbers </h2> <div class="table-wrapper-paragraph"><table> <thead> <tr> <th>Test</th> <th>Result</th> </tr> </thead> <tbody> <tr> <td>Prompt processing (pp512)</td> <td>1,093 tokens/sec</td> </tr> <tr> <td>Text generation (tg128)</td> <td>34.36 tokens/sec</td> </tr> <tr> <td>VRAM usage</td> <td>5,292 MiB</td> </tr> </tbody> </table></div> <h2> A few things I learned </h2> <p><strong>The Deep Learning Base GPU AMI is worth it.</strong> CUDA drivers, build tools, cmake, git - all pre-installed. Saves you an hour of setup that nobody wants to document.</p> <p><strong>The CUDA build takes ~90 minutes on 4 vCPUs.</strong> <code>-DGGML_CUDA=ON</code> is the flag that matters. Without it you're running on CPU and inference is ~10x slower. Snapshot your EBS volume after the build so you never wait again.</p> <p><strong>GPU instance quotas start at 0 on new AWS accounts.</strong> Request your quota increase before you start - it takes up to 2 hours and will block you mid-exercise otherwise.</p> <p><strong>Q4_K_M is the sweet spot for the T4.</strong> Full fp16 needs ~16 GB VRAM (too tight for the T4's 15 GB). Q4_K_M fits in 5.2 GB with minimal quality loss.</p> <p><strong>llama-server exposes an OpenAI-compatible API.</strong> Point your existing code at the new endpoint URL - no other changes needed.</p> <h2> The full guide </h2> <p>I wrote up every step with real terminal output and screenshots:</p> <p>👉 <a href="https://gizmojack.com/how-to-deploy-llama-3-1-on-aws-ec2-g4dn-xlarge-for-under-1-hour-a-complete-guide/" rel="noopener noreferrer">https://gizmojack.com/how-to-deploy-llama-3-1-on-aws-ec2-g4dn-xlarge-for-under-1-hour-a-complete-guide/</a></p> <p>Covers AMI selection, security group setup, CUDA build, model download, server flags, benchmarking, and cost optimization tips.</p> <p>If you have questions or need help setting this up for your company, reach me via the contact form at <a href="https://gizmojack.com/contact-me/" rel="noopener noreferrer">https://gizmojack.com/contact-me/</a></p> aws llm devops opensource