Recently, I have been using the 500 USD in AWS credits provided as an AWS Community Builders benefit to experiment with services and architectures that I have been interested in.
This time, I built a demo application using Lambda MicroVMs and AWS Controllers for Kubernetes (ACK) that installs npm packages in an isolated environment and records what they do during installation.
Running an unfamiliar package directly on my local machine or on an EKS node feels risky, so I decided to launch a new MicroVM for each inspection and manage the creation and deletion of those MicroVMs through ACK.
The implementation is available here:
lambda-microvm-package-inspector
What I built
When a user creates a PackageInspection resource in Kubernetes, a custom controller creates a Microvm resource for ACK. The ACK controller detects the resource, calls the AWS API, and launches a MicroVM for the inspection.
The overall flow looks like this:
The only component running on EKS is the controller that accepts inspection requests and manages MicroVMs. The npm package installation scripts themselves run inside disposable MicroVMs.
In other words, Kubernetes is not used as the place where the packages are executed. It is used as the control plane that creates and deletes isolated execution environments.
I chose npm for this demo because it makes code execution during installation easy to demonstrate. During npm install, lifecycle scripts such as preinstall and postinstall can be executed.
With pip, the installation behavior varies depending on the distribution format. For source distributions (sdist), build steps may also run before installation. That would increase the number of things that need to be observed, so I limited this demo to npm.
Why I used Lambda MicroVMs
Containers running as EKS Jobs on EC2-based worker nodes share the kernel of the EKS node. That leaves me somewhat uncomfortable using them as a place to run installation scripts that I do not yet trust.
Lambda MicroVMs can launch a Firecracker-based execution environment for each job, and the entire MicroVM can be terminated after the inspection is complete.
A similar system could also be built with EKS Jobs, Fargate, or EC2. However, one of the things I wanted to try in this experiment was creating and deleting AWS-managed VMs through ACK, so I chose Lambda MicroVMs.
Implementation architecture
The responsibilities and implementation languages are as follows:
| Location | Implementation | Responsibility |
|---|---|---|
| AWS environment | AWS CDK / TypeScript | Create EKS, VPC, S3, ECR, IAM, and other resources |
| EKS | Go / controller-runtime | Accept inspections, create and delete MicroVMs, and save reports |
| EKS | ACK controller | Connect Microvm resources with the Lambda MicroVMs API |
| MicroVM | Python | Run npm, strace, file diffs, and generate JSON reports |
| Sample packages | Node.js | Perform harmless actions that can be observed |
Two controllers running on EKS
On EKS, the custom controller and the ACK controller run as separate Pods. I also assigned separate IAM roles to them.
The PackageInspection custom resource
A user creates a PackageInspection resource. When the custom controller detects it, the controller creates a Microvm custom resource for ACK. ACK then detects the Microvm, calls the AWS API, and launches the actual MicroVM.
Users can specify only the package name, version, and maximum execution time. They cannot change URLs, commands, or IAM roles.
apiVersion: inspection.demo.aws/v1alpha1
kind: PackageInspection
metadata:
name: canary-package
namespace: package-inspector-system
spec:
ecosystem: npm
package:
name: '@demo/canary'
version: '1.0.0'
timeoutSeconds: 30
For safety and reproducibility, this demo only allows two custom packages that are embedded in the MicroVM image in advance. Any other package is rejected by the Kubernetes API, the custom controller, and the inspection program running inside the MicroVM.
The MicroVM does not connect to the npm registry. The two packages are embedded in the MicroVM image, and their size and SHA-256 hashes are verified before running npm install --offline.
What the test package does
@demo/canary is a harmless package that I created specifically to test the monitoring functionality. I will refer to it as the canary package from this point on.
The important parts of its postinstall.js are shown below:
writeFileSync('/tmp/demo-canary.txt', 'harmless-canary\n');
spawnSync('/usr/bin/true', []);
const wasSet = process.env.DEMO_CANARY_TOKEN !== undefined;
try {
const descriptor = openSync('/home/sandbox/.aws/credentials', 'r');
closeSync(descriptor);
} catch {
// Continue with the remaining checks even if the file does not exist
}
await contactSink(); // Attempt DNS and HTTP access, record the failures, and exit
This script attempts several operations: creating a file, starting a child process, reading an environment variable, performing a DNS lookup, and making an HTTP connection.
The credentials file and DEMO_CANARY_TOKEN do not exist. The destination uses the reserved .test domain. This allows me to verify the monitoring behavior without contacting a real external service or causing harm.
Inspection inside the MicroVM
Inside the MicroVM, the system performs both checks before installation and observations while the package is actually being installed.
Static inspection
npm packages are distributed as tar archives in .tgz format.
First, the package is inspected without extracting its contents to the filesystem. The inspection records information such as the SHA-256 hash, file list, package.json, dependencies, and license information.
The system also generates a minimal SBOM from package.json containing the target package and its direct dependencies.
Dynamic inspection
Next, npm install is launched through strace.
This allows the system to record file access, process creation, and network connections performed by npm and its child processes.
Before execution, the process switches its UID and GID to 10001, so package code does not run as root.
The command being executed is:
npm install \
--offline \
--no-audit \
--no-fund \
--foreground-scripts \
--ignore-scripts=false \
/opt/package-inspector/fixtures/demo-canary-1.0.0.tgz
The inspection records file changes, processes, file access, connection destinations, CPU usage, memory usage, and command output.
File diffs are limited to the npm working directory, the sandbox home directory, and the files under /tmp used by the canary package.
The strace configuration also focuses only on selected operations, so this system does not provide complete visibility into every action performed by the package.
Execution limits
To prevent a package from running indefinitely or producing excessive output, I configured several limits.
| Target | Limit |
|---|---|
npm install execution time |
30 seconds |
| CPU time per process | 60 seconds |
| Number of processes | 64 |
| Standard output | 64 KiB |
| Standard error | 64 KiB |
| Maximum size of a created file | 16 MiB |
| JSON report | 2 MiB |
These values are limits configured before the inspection. They are not actual resource usage values.
Actual execution time, CPU usage, and memory usage are recorded separately.
How I handled networking and credentials
The inspection MicroVM is connected to a VPC that has no route to the internet.
No AWS execution role or AWS credentials are provided to the MicroVM.
However, EKS still needs to start the inspection and retrieve the resulting report. For that purpose, communication to port 8080 is temporarily authorized using a short-lived token.
Deploying to AWS
I wrote the infrastructure using AWS CDK with TypeScript and deployed it to ap-northeast-1.
The EKS cluster uses only one t4g.medium worker node.
The IAM roles for ACK and the custom controller are separated.
The deployment is performed in the following order:
export AWS_PROFILE=YOUR_DEMO_PROFILE
export AWS_REGION=ap-northeast-1
export AWS_DEFAULT_REGION="$AWS_REGION"
export EXPECTED_AWS_ACCOUNT_ID=000000000000
aws sso login --profile "$AWS_PROFILE"
./scripts/preflight
export MICROVM_BASE_IMAGE_VERSION=VERSION_PRINTED_BY_PREFLIGHT
./scripts/deploy-infrastructure
./scripts/deploy-controller
./scripts/run-demo
Actual inspection results
Running run-demo starts two inspections, one for @demo/good and another for @demo/canary.
After both inspections finish, the results are displayed as a list.
@demo/good is a comparison package that does not contain any installation scripts. @demo/canary is the package created to test the observation features.
NAME PACKAGE PHASE RISK SCORE
canary-package @demo/canary Succeeded critical 95
good-package @demo/good Succeeded low 0
Comparison of observed events
| Observation | Comparison package | Canary package |
|---|---|---|
| Installation scripts | 0 | 1 |
| Items found by string scanning | 0 | 4 |
| File changes | 9 | 11 |
| Process-related events | 2 | 16 |
| DNS-related activity logs | 0 | 2 |
| Network-related events | 0 | 5 |
| Environment variable reads reported by the canary | 0 | 1 |
strace |
Available | Available |
| Risk | LOW 0/100 | CRITICAL 95/100 |
Of the five network-related events, four were connections to the local DNS resolver, while one was an HTTP failure log emitted by the canary package.
No connection to the HTTP sink was successfully established.
Findings detected for the canary package
npm.lifecycle-script +15
static.suspicious-pattern +10
dynamic.child-process +10
dynamic.credential-path +25
dynamic.dns +10
dynamic.network +20
dynamic.environment-read +5
--
95
The score is simply the sum of values assigned by deterministic rules.
The score of 95 is a value designed for this demo. It should not be interpreted as a general-purpose measure of package risk.
When npm install ran, the canary package's postinstall script was executed.
The canary attempted to write to /tmp, start a child process, access a credentials file, perform a DNS lookup, and make an HTTP request.
Because no credentials were provided and there was no route to the internet, these attempts failed.
The architecture is intended to reduce the potential impact of such behavior, but it does not guarantee that every malicious package can be completely contained.
The following is the activity log emitted by the canary itself to make its behavior easier to understand:
demo canary postinstall started
CANARY_EVENT {"type":"file-write","path":"/tmp/demo-canary.txt"}
CANARY_EVENT {"type":"child-process","executable":"/usr/bin/true","status":0}
CANARY_EVENT {"type":"environment-read","name":"DEMO_CANARY_TOKEN","wasSet":false}
CANARY_EVENT {"type":"credential-path-open","path":"/home/sandbox/.aws/credentials","result":"ENOENT"}
CANARY_EVENT {"type":"dns-attempt","hostname":"sink.inspection.test"}
CANARY_EVENT {"type":"dns-result","hostname":"sink.inspection.test","error":"ENOTFOUND"}
CANARY_EVENT {"type":"http-result","error":"getaddrinfo ENOTFOUND sink.inspection.test"}
demo canary postinstall completed
Saving the reports
The EKS-side controller also validates the size and contents of the JSON report.
The report is stored in S3 in both JSON and Markdown formats. Only the storage location and SHA-256 hash are stored in PackageInspection.status.
I also verified that the SHA-256 values matched in the actual demo.
Terminating the MicroVM
After the report has been saved, the controller deletes the Microvm resource.
Once ACK has terminated the MicroVM, the inspection transitions to Succeeded.
The five MicroVMs shown in the screenshot include some that were created during earlier test runs.
The final demo used the two MicroVMs shown at the top. All of them are now in the Terminated state.
Was Kubernetes worth using?
Creating a PackageInspection resource was enough to start an inspection, and I could leave retries after failures and MicroVM cleanup to the controllers.
I found it interesting that Kubernetes could be used not as the environment where package code runs, but as the control plane for managing disposable MicroVMs.
That said, EKS and ACK would clearly be overkill if the only goal were to inspect two packages once each.
This architecture starts to make more sense when inspections are accepted repeatedly and the entire lifecycle, from MicroVM creation to deletion, needs to be managed through the same consistent workflow.
Conclusion
ACK was convenient to work with, so I would like to experiment with some other use cases for it as well.
Top comments (0)