Faster Rust development on AWS EC2 with VSCode

I dread running cargo build --release.

It's not that I dread build failures or compiler warnings - I dread the wait.

My current Rust project takes about a minute to build on my laptop. It's not long enough to do something meaningful, like read HackerNews, but is long enough to make it feel agonizingly boring. If I don't have the money for a faster laptop, maybe I could build a really fast Rust development server on AWS?

Cost of Intel NUC vs EC2 Instance

A friend of mine is using an Intel NUC bare-bones computer as his local development server. It is quite powerful for its tiny size and will outperform my ageing Surface by a wide margin. We'll use the cost of a new NUC and the performance of my Surface as the baseline to compare different options.

• New NUC cost: $1,000 incl shipping, credit card fees, etc
• Useful lifetime: 2 years
• Residual cost: $200 in today's money (discounted for inflation and the cost of sale)
• Total cost: $900 in today's money ($1000 outlay + $100 interest - $200 residual cost)

Somehow I find it painful to part with $1,000 for a gadget I may not need in a few months, even if it's as cute as a NUC. Let's see what an EC2 instance may cost.

• t3.2xlarge AWS instance: 33c per hour or $247 per month or $5,942 over 2 years or 6 NUCs
• Working hours usage only: 5 days * 10 hrs * 4 weeks = 200hrs per month or $1,584 over 2 years
• Realistic usage: 100 hrs per month or $792 over 2 years
• Total cost: $33 per month

So, it comes down to $33 per month vs. $1,000 upfront. That's an easy choice, but how much faster will the builds be?

Rust Compilation Benchmarks on EC2

I compared Rust build time on different instance types using a very simple Rust project for logging AWS Lambda input into CloudWatch.

• Source code: https://github.com/rimutaka/rusty_lambdas/tree/master/json_logger
• AMI: ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20200408
• Build command: cargo build --release --target x86_64-unknown-linux-musl

Type	vCPU	Mem	Cost/hr	Avg build	Builds/hr	100 hrs	200 hrs	1,200 hrs
t2.micro/u	1	1	$0.0116	40.25s	89	too slow
t2.small/u	1	2	$0.0230	38.81s	93	too slow
t2.med/u	2	4	$0.0464	21.00s	171	too slow
t2.large	2	8	$0.0928	21.50s	167	too slow
t2.large/u	2	8	$0.0928	20.89s	172	too slow
t2.xlarge	4	16	$0.1856	11.59s	311	$19	$37	$223
t3.xlarge	4	16	$0.1664	13.83s	260	$17	$33	$200
c5.xlarge	4	8	$0.1700	11.22s	321	$17	$34	$204
a1.xlarge	4	8	$0.1020	25.93s	139	too slow
c5a.xlarge	4	8	$0.1540	11.18s	322	$15	$31	$185
c6g.xlarge	4	8	$0.1360	11.69s	308	$14	$27	$163
t2.2xlarge	8	32	$0.3712	7.86s	458	$37	$74	$445
t3.2xlarge	8	32	$0.3328	7.79s	462	$33	$67	$399
c5.2xlarge	8	16	$0.3400	7.04s	512	$34	$68	$408
c5a.2xlarge	8	16	$0.3080	6.93s	519	$31	$62	$370

The average compilation time proved that the defining factor is the number of vCPUs (cores) - the more the better. Factors like extra memory, EBS optimization or network throughput didn't make any difference. I did use Unlimited option (u in t2.micro/u) for some tests.

Surprisingly, the new ARM-based instances (a1.xlarge, c6g.xlarge) were slower than x86, but provided more value. I also ran into several toolchain issues with them. It may be a bit too early to consider them for this particular application.

Another unexpected result was that t2.micro has the lowest cost per compilation. It is too slow for real-time builds, but it gets there eventually. I would use it for a build server in a CI/CD pipeline where the build duration is not critical.

The time saved per build is relative to the size of the build. The next graph shows the ratio of improvement with EC2 time compared to my current Surface/VSCode/WSL2/Rust setup. I may be saving 15s per build on this test project, but some builds may run into minutes, so the higher the ratio the more time it will save.

And finally, consider some measure of value vs the monthly cost. It looks like c5-types provide the best value for money for real-time Rust builds. I arbitrary defined value = time_saving_ratio/cost_per_build, which can be simplified to value = 1/(instance_cost * avg_compilation_time^2).

I would go with c5a.2xlarge for larger builds to get 2.89x improvement or c5a.xlarge for smaller ones with 1.79x. They seem to be the optimal choice between speed, value and what I am prepared to pay for it.

Total cost: $31 per month (c5a.2xlarge @ 100 hrs per month, 7s, 2.58x faster)

---

The rest of the post is a detailed step by step guide of setting up VSCode with a Rust development server on EC2.

AWS set up

The primary purpose of this development environment is building AWS Lambda functions. That adds a few extra steps with x86_64-unknown-linux-musl target and other specifics.

This section assumes that you already know how to configure a security group, launch an instance and connect to it. Let's get started ...

1. Launch a t3.micro instance with Ubuntu Server 18.04 LTS (HVM), SSD Volume Type AMI
2. Connect to the instance via SSH
3. Run the following series of commands to install Rust and AWS CLI

sudo apt-get update
sudo apt-get install musl
sudo apt-get install build-essential -y
sudo apt-get install musl-tools
sudo apt-get install zip -y
curl https://sh.rustup.rs -sSf | sh -s -- -y
source $HOME/.cargo/env
rustup --version
rustup target add x86_64-unknown-linux-musl

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
aws --version
rm awscliv2.zip

You may need to stop the process to troubleshoot if you get any errors. I included some links that helped me along the way.

• https://forge.rust-lang.org/infra/other-installation-methods.html
• https://github.com/awslabs/aws-lambda-rust-runtime/issues/17#issuecomment-577837255
• https://stackoverflow.com/questions/52445961/how-do-i-fix-the-rust-error-linker-cc-not-found-for-debian-on-windows-10
• https://aws.amazon.com/blogs/opensource/rust-runtime-for-aws-lambda/

AWS config

We need to prepare the environment to easily start/stop the instance and deploy test Lambdas.

1. Create 2 IAM policies (StartStopDevEC2 and UpdateOrInvokeLambda)
2. Create an EC2 role with UpdateOrInvokeLambda policy and attach it to the Rust instance
3. Allocate a new Elastic IP and associate it with the Rust instance
4. Create a new AWS CLI user in IAM with StartStopDevEC2 policy
5. Copy the user credentials into ~/.aws/credentials on your computer running VSCode. Add them as a named profile, e.g. I called mine [vscode-syd-ld-start-stop]
6. Create an alarm inside EC2 instance monitoring tab for auto-stopping the Rust instance when not in use:
   • Stop this instance when Sum of NetPacketsIn < 100 for 1 Hr
   • Change the default name to something nice, like Stop Rust Dev EC2

IAM policy templates

Replace the a/c (028534811986) region (ap-southeast-2, us-east-1) and instance IDs (i-026a4507e766ebe50) in the templates with your values.

This is StartStopDevEC2 policy for starting your Rust EC2 instance from inside VSCode:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "ec2:StartInstances",
                "ec2:StopInstances"
            ],
            "Resource": "arn:aws:ec2:ap-southeast-2:028534811986:instance/i-026a4507e766ebe50"
        }
    ]
}

This is UpdateOrInvokeLambda policy for updating Lambda function code from your EC2:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "lambda:UpdateFunctionCode",
                "lambda:InvokeFunction",
                "lambda:InvokeAsync"
            ],
            "Resource": "arn:aws:lambda:us-east-1:028534811986:function:*"
        }
    ]
}

Launch the EC2 instance in the AWS location closest to you to minimize the network latency. Lambdas can be deployed in a different location with the rest of your infra - the latency from the Lambda to you is not critical. For me, the nearest AWS data centre is in Sydney (ap-southeast-2) and my Lambdas live in us-east-1.

VS Code configuration

Starting / stopping Rust EC2 instance

• Install vscode-command-runner extension
• Add start-instances custom command to VSCode settings.json with your instance, region and profile IDs:

"command-runner.commands": {
  "Start vscode-syd VM": "aws ec2 start-instances --instance-ids i-026a4507e766ebe50 --profile vscode-syd-ld-start-stop --region ap-southeast-2"
}

Starting the instance: press Ctrl+Shift+R and select Start vscode-syd VM. This shortcut works only if your VSCode is running locally. Give it a minute or so for the instance to initialize before connecting to it.

Stopping the instance: disconnect all SSH sessions and the instance will be stopped 1 hr later by the EC2 alarm described in AWS Config section. You can add stop-instances custom command if you want to stop it immediately.

Connecting to the EC2 instance from VSCode

• Configure SSH details: type "ssh open" in the command pallet, select Remote SSH: Open configuration file and add the following entry. Note, IdentityFile should be located on the computer running your VSCode.

Host rust-aws-syd-ld
  HostName 13.54.209.171
  User ubuntu
  IdentityFile ~/.ssh/dev-ld-syd.pem

• Connect to the instance:
  • Open a remote window (click on >< at the bottom left corner of your VSCode screen or search for "SSH Connect" in the command pallet)
  • Watch for prompts at the top of the screen: select Linux and Continue
  • Let the VS install its components on the remote system

Learn more about remote SSH connections in VSCode from https://code.visualstudio.com/docs/remote/ssh.

Setting up Rust for VSCode

• Install rust-analyzer VSCode extension on the remote machine
• Consider other useful extensions: Better TOML, Code Spell Checker, CodeLLDB
• Open the terminal and create a folder for Rust projects, e.g. mkdir rust + cd rust
• Clone a demo lambda project git clone https://github.com/rimutaka/rusty_lambdas.git
• Open json_logger folder in your VSCode, then open main.rs file. Wait for a prompt from rust-analyzer to install its server (watch the bottom-right corner of the screen).

Build and deploy a demo Lambda function

If you are still running on that t2.micro instance it is time to upgrade to c5.xlarge and build the demo project for deployment to AWS Lambda:

cargo build --release --target x86_64-unknown-linux-musl

Package the Lambda with Rust custom runtime:

cp ./target/x86_64-unknown-linux-musl/release/lambda ./bootstrap && zip lambda.zip bootstrap && rm bootstrap

Push the code to AWS Lambda. The Lambda must already exist for update-function-code command to work. Replace the region and function name with your IDs.

aws lambda update-function-code --region us-east-1 --function-name JsonLogger --zip-file fileb://lambda.zip

I noticed that IntelliSense within VSCode stops working from time to time. If that happens to you, search the command pallet for Rust Analyzer:Restart Server. It should start picking up errors and expanding the syntax as soon as your Rust Analyzer restarted.

---

Comments with questions, corrections or suggestions are very welcome.

Comments

[David Laban]

This is quite similar to the approach I settled on for rustc development in the middle of summer. I ended up using visual studio online ("codespaces") on azure. I had a lot of problems with connection stability though, and often wished that I just had an ssh tunnel under my control. I may try something more like your setup next time.

[Max]

This comment arrived in an email from someone who is not a Dev.To member ...

I just read your blog post and I was honestly wondering why one would buy a 1000$ NUC as a build machine. Since you probably meant Intel NUC 9 Extreme Kit NUC9i5QNX I have to say that would probably not be my first choice for building rust. One problem is that these small form factor PCs usually don't have adequate cooling for the CPU to run at full throttle. The more obvious problem is that with Intel, and especially with their NUCs you don't really get the most bang for the buck (especially for Rust compilation).

If you'd give me 1000$ to select a Rust build server, I would probably choose an AMD Ryzen 3900X processor. Not only does it have significantly better single core performance (according to Geekbench) but it has 12 Cores (24 threads) and it "only" costs about 500$, leaving 500$ spare to build a computer around it (which should be more than enough).

Also I'm quite certain you would get much more than a 2 year useful lifespan out of that computer.

[Richard Pringle]

RemoteSSH is brilliant! I was trying to work on a rust project on small laptop (only 2 cores) and it was impossible. Even Rust-Analyzer was burning through CPU cycles.

I set it up with my desktop everything is working super smoothly (and was so easy to set up).

As with the comment that you posted, the simple fact that it's so easy to work with a remote server is great.

Thanks for sharing!

[frank martinez]

Max, nice recipe. Been looking for an easy deploy, host solution for Rust . Cloudflare sort of gets close albeit very opinionated.

Assume most of these steps can be run from Terraform.