Recent supply chain attacks make me feel that the phase has clearly changed.
This article is not a complete solution. It is a summary of what I have been thinking about development environments and CI/CD after seeing recent supply chain attacks.
In the past, many attacks targeted people with low computer literacy. For example, attackers asked users to open suspicious emails, run suspicious attachments, or enter credentials into fake websites.
Of course, these attacks are still serious. But in many cases, there was at least some room for individual caution.
Recent attacks are different.
They clearly target developers.
Developers often have stronger privileges than normal business users. They may have GitHub tokens, npm tokens, SSH keys, cloud credentials, and access to repositories that can affect CI/CD or production systems.
From an attacker's point of view, compromising a developer is very attractive. If they can compromise a developer, they may be able to affect the software supply chain itself.
AI coding also changes the situation. More people can now write code with AI. This is a good thing. But it also means that more people may have strong privileges without enough knowledge about development environments, dependencies, CI/CD, and secret management.
So protecting developers is becoming more important than ever.
This does not mean that EDR, DNS filtering, device management, browser protection, authentication systems, and secret management on devices are useless. They are still necessary.
But they are not enough.
We need to protect not only developer devices, but also the act of development itself.
How do we handle dependencies? What is running in CI/CD? Where are secrets stored? How do we review AI-generated code? How do we handle code from outside?
The development process itself must become safer.
CI/CD also needs protection
CI/CD is as important as developer devices.
CI/CD systems such as GitHub Actions build, test, deploy, and operate infrastructure. This means they may touch very powerful secrets, such as cloud credentials, signing keys, and registry tokens.
If CI/CD is compromised, the problem is not only source code leakage. Released artifacts may be modified. Cloud credentials may be sent outside. Infrastructure may be directly operated by attackers.
The GITHUB_TOKEN in GitHub Actions also has permissions to call the GitHub API from workflows. These permissions can be controlled, but if workflows or events are configured incorrectly, attackers may be able to use unexpected permissions.
https://docs.github.com/actions/reference/authentication-in-a-workflow
https://docs.github.com/en/actions/concepts/security/github_token
GitHub also provides documentation about secure use of GitHub Actions.
https://docs.github.com/en/actions/reference/security/secure-use
For CI/CD security, one important point is observability.
We need to know what actually happened in CI/CD.
Recently, I tried cicd-sensor and found it interesting.
https://github.com/cicd-sensor/cicd-sensor
It is like EDR for CI/CD. It uses eBPF to observe what happens during CI/CD jobs. It can record processes, file access, network access, and suspicious behavior.
Just being able to see which process ran, which files were touched, and where the job communicated is valuable.
CI/CD usually only shows what is written in logs. But that is no longer enough.
This is not as simple as saying "this IP address is suspicious."
For example, access to 169.254.169.254 may be access to the AWS metadata service. It may be legitimate. So we need to know the calling process.
Did a process started by npm install read credentials?
Did it access many different credentials?
Did it make network connections that do not usually happen in normal builds?
Without this context, detection is not practical.
Of course, observability tools do not solve everything. Rules need tuning. We also need to decide what to do after detection.
But at least we can move away from a state where we do not know what happened in CI/CD.
I think a realistic first step is detection and evidence collection. If we try to block everything from the beginning, we may break the developer experience.
First, make things visible. Then block only truly dangerous behavior.
Protecting dependency entry points and CI/CD execution
In this context, tools such as Takumi Guard and Takumi Runner are interesting.
Takumi Guard is a registry proxy that improves npm supply chain security.
https://flatt.tech/takumi/features/guard
https://shisho.dev/docs/ja/r/202603-takumi-guard/
By setting the registry URL in .npmrc, it can block malicious packages and track package installation.
This direction is important.
It is not realistic for humans to read all dependency code. It may be possible to ask AI to review dependencies, but continuously reviewing all dependencies deeply is expensive.
So one realistic approach is to stop dangerous packages at install time.
Another important point is traceability. If a package is later found to be malicious, we need to know where it may have been installed or used.
Takumi Runner is also interesting in the same context.
https://flatt.tech/takumi/features/runner
https://shisho.dev/docs/ja/t/runner/
It works as a self-hosted runner for GitHub Actions. It records processes, network access, and file operations during workflow execution by using eBPF. It also visualizes traces and helps triage suspicious behavior.
The design is practical because users can start using it by changing runs-on in existing workflows.
If improving CI/CD security requires large workflow changes, adoption becomes hard. Moving this responsibility to the runner side is a good approach.
Protecting the dependency entry point and observing CI/CD execution are both ways to make development itself safer.
AI changes vulnerability management
AI is improving the ability to find vulnerabilities.
Advanced AI agents that find vulnerabilities get a lot of attention. But even current AI review tools can already find some issues.
First, we should continuously fix issues that current AI tools can find.
I do not know whether the current increase in AI-found vulnerabilities is temporary or will continue for a long time. But at least for now, we should assume this situation will continue.
AI review is becoming better. It can help reduce new vulnerabilities, find missing authorization checks, and detect dangerous implementations earlier.
So I think AI review should become part of the development process.
But AI review is not perfect.
In particular, it is not realistic to deeply review all dependency code every time. We need to separate what AI review can protect and what should be protected by dependency management or runtime monitoring.
OSS and the GitHub ecosystem are changing
The OSS and GitHub ecosystem is also changing quickly.
Accepting pull requests from outside has been very important for OSS. People send code, receive reviews, communicate with maintainers, and build trust through that process.
In traditional OSS, sending code was also a way to build trust.
But AI has made it much cheaper to create pull requests.
If many AI-generated pull requests are sent to projects, pull requests may stop being a signal of trust. They may become a source of maintainer fatigue.
This does not mean every OSS project should close pull requests. Many projects depend on external contributions, and that will continue to be important.
But for some projects, keeping "anyone can send a pull request" may become too expensive.
AI-generated PRs, malicious PRs, and low-quality issues can all increase maintainer burden. I think OSS operation models may change.
GitHub has also added repository settings for configuring pull request access.
If a project does not accept pull requests, the reason to use GitHub may become weaker. Some projects may move to GitLab, Forgejo, or other platforms.
At the same time, many OSS projects depend on GitHub Actions compute resources. That free infrastructure supports many OSS projects.
So GitHub Actions and other CI/CD platforms are convenient, but they are also becoming important attack targets.
We need to evaluate vulnerabilities calmly
Vulnerability reports are also becoming difficult to evaluate.
Finding vulnerabilities is important. Security researchers and security companies do necessary work.
But the severity of a vulnerability is not decided only by the type of bug.
We need to look at attack conditions.
Can it be attacked in a realistic environment?
Is it exploitable with the default settings?
Does it require another vulnerability?
Does it require a special environment?
Is it privilege escalation, information leakage, or remote code execution?
Sensational titles get attention. But if vulnerabilities with limited attack conditions are treated too loudly, users may not know what to prioritize.
As a result, truly important issues may be hidden.
This is also a burden for OSS maintainers.
We should not underestimate vulnerabilities. But we need to look at attack conditions calmly and prioritize correctly.
How much should we depend on OSS libraries?
Can we develop web applications without OSS libraries?
In modern frontend development, avoiding React or similar major libraries is usually not realistic. We need standard libraries and widely used libraries.
But we should become more careful about small convenience libraries.
AI usually does not review the inside of dependencies. Technically it may be possible, but it is not easy in terms of cost and responsibility.
This means that when we use a library, the inside of that library often becomes a blind spot for AI review.
Using a library itself now has some risk.
Of course, I am not saying we should stop using all libraries.
For areas such as authentication, cryptography, sanitization, OAuth, and OIDC, we should not easily write our own implementations. These areas require expertise, and bugs can be critical. We should use trusted implementations.
But for small libraries that only make things a little more convenient, it may sometimes be safer to let AI generate the code, review it with AI, and then review it by humans.
The important thing is to choose dependencies carefully.
We should reduce unnecessary convenience dependencies.
npm is convenient, but difficult for security
npm is convenient, but it has difficult security aspects.
In package.json, we can specify version ranges for dependencies. This means newer versions may be installed unintentionally. Lockfiles fix the actual installed versions, but if operation is wrong, unexpected updates can still happen.
https://docs.npmjs.com/cli/configuring-npm/package-json
npm also has lifecycle scripts. Scripts can run during install and other phases. This has legitimate uses, but it is also attractive for attackers.
https://docs.npmjs.com/cli/using-npm/scripts/
npx is also convenient. But it makes it easy to run commands from external packages. From a security point of view, we should be careful with it.
https://docs.npmjs.com/cli/commands/npx/
Ruby and PHP also have dependency problems. But today they are often used inside Docker.
npm is different in many cases. npm often runs directly on developer machines.
npm touches developer machines directly. Secrets may exist there. That is why attackers target it.
Go takes a different approach. Module versions are written explicitly in go.mod, and module graph resolution has its own design.
At first, this may feel inconvenient. But now I feel it was a safer design.
Package managers should move not only toward convenience, but also toward safety.
Moving development environments to the cloud or sandbox
Another possible direction is to avoid building development environments directly on local machines.
Instead, development can happen in cloud VMs or containers.
If all network communication can be monitored, it becomes easier to detect dangerous package installs and suspicious communication. It is also useful because secrets do not need to be placed on personal developer devices.
If the environment is compromised, it can be discarded and recreated.
Cloud development environments such as GitHub Codespaces are one example of this direction.
https://github.com/features/codespaces
But we do not need to move everything to the cloud from the beginning.
A more realistic approach is to run only dangerous operations in a sandbox.
For example:
npm installnpx- building code from outside
- running AI-generated code
These operations should not always run directly on developer machines. It may be safer to run them in isolated environments.
In this sandbox, it is not enough to simply block network access.
We should record process execution, file access, and network communication by using technologies such as eBPF. When necessary, the sandbox should also block dangerous behavior.
For example:
- A package install tries to read an SSH key.
- A process tries to read cloud credentials.
- A command makes unusual external network connections.
- A process accesses the metadata service.
We need to record not only the destination IP address, but also which command caused it, which dependency process caused it, which files were read, and where it communicated.
Without this level of visibility, investigation and blocking are difficult.
So making development environments safer is not only about moving them to the cloud.
We need to isolate dangerous operations, observe what happens inside them, and block dangerous behavior when needed.
However, introducing this to old projects all at once is difficult.
Migration cost is high. Developer productivity may fall at first. IDE and debugging experiences may also become worse.
So gradual adoption is more realistic.
Start with new projects.
Start with high-risk repositories.
Run only npm install and npx in a sandbox.
Move only secret-handling work to cloud environments.
That kind of step-by-step approach seems practical.
Always keep systems rebuildable
We need to continuously update not only applications, but also infrastructure.
Linux kernels, container runtimes, base images, and language runtimes continuously get vulnerabilities.
We should not panic every time a vulnerability is announced. Instead, we should build systems that can be updated regularly.
In practice, moving toward containers is important.
Deploy at least once a week.
Update base images and runtimes regularly.
Do not keep using old container images.
Replace Kubernetes nodes and VMs when needed.
The important thing is not to work hard only after a vulnerability appears.
The important thing is to always keep systems rebuildable.
This applies to development environments, CI/CD, and production environments.
We should not assume that systems will never be compromised. We should make systems disposable, rebuildable, and observable.
Conclusion
Developers are no longer outside the attack surface.
For attackers, developers are now high-value targets.
AI coding also increases the number of people who can participate in development. This is a good thing. But it may also increase the number of people with strong privileges before knowledge about development environments, dependencies, CI/CD, and secret management is shared enough.
Developer devices, CI/CD, package managers, OSS libraries, GitHub Actions, and cloud credentials are all connected.
If one part is compromised, the whole service may be affected.
Basic endpoint protection and authentication systems are necessary. But they are not enough to make development safe.
What matters now is making development itself safer.
We should choose dependencies carefully.
We should stop dangerous packages at the entry point.
We should add AI review to the development process.
We should observe what happens in CI/CD.
We should not place secrets carelessly on developer devices or CI/CD.
We should run dangerous operations such as npm install, npx, and external code execution in sandboxes that can record and block behavior.
We should keep infrastructure and development environments rebuildable.
Tools such as cicd-sensor, Takumi Guard, and Takumi Runner are interesting examples of this direction.
cicd-sensor:
https://github.com/cicd-sensor/cicd-sensor
Takumi Guard:
https://flatt.tech/takumi/features/guard
https://shisho.dev/docs/r/202603-takumi-guard/
Takumi Runner:
https://flatt.tech/takumi/features/runner
https://shisho.dev/docs/r/202603-takumi-guard/
Now that developers are attack targets, development environments and CI/CD must be protected like production systems.
This is not only about stronger internal IT management.
It is about changing the way we develop software toward a safer model.
Top comments (0)