The latest articles on DEV Community by Mati (@mattick27). https://dev.to/mattick27 https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F1207801%2F9c3c3cdd-c5a0-4d98-b8a7-6386a05650b8.jpg https://dev.to/mattick27 en Mati Sun, 26 Nov 2023 15:51:07 +0000 https://dev.to/mattick27/run-llama7b-100-tps-with-a10-444e https://dev.to/mattick27/run-llama7b-100-tps-with-a10-444e <p>Hi Folk.</p> <p>I've been working on setting up and managing TensorRT-LLM and Triton backend scripts to build the Llama2-7b model in FP16, int8, and int4 formats.</p> <p>I ran a benchmark with int4 and achieved an inference speed of approximately 100 tokens per second.</p> <p><a href="https://github.com/vultureprime/deploy-ai-model/blob/main/aws-example/Llama7b-TensorRT-LLM/README_eng.md">Github Llama7b-TensorRT-LLM</a></p> machinelearning tutorial ai bash