We all know that crypto mining is negatively impacting many things in the world. And now it's ruining something else in a way no one has seen comin...
Some comments have been hidden by the post's author - find out more
For further actions, you may consider blocking this person and/or reporting abuse
This is insane.
I won't deny it was a genius discovery and clever tactic but horribly detrimental to those of us who rely on these free and excellent services daily for genuine use cases.
I like the free minutes approach rather than paywall or credits.
However I wouldn't mind GitHub requiring credit or debit card information to prevent abuse. Similar to how Firebase prevents abuse by requiring your card details. It works.
Not everyone has easy access to a credit card. Nothing Genius about it.
I agree with you. It’s a real problem, I hope as I’ve mention that they will find a solution that doesn’t require limiting others (aka us) for something abusers do, but rather changing the approach of the whole system
What about privacy issues?? I don't want to give my card info to Microsoft or any company for that matter.
I just pay for my CI. One of the few things that can be done.
This isn't always enough. If you have an open-source project that allows pull requests, and automatically runs tests against these, someone can weasel in there and inject a miner in that process.
They get the rewards, you get stuck with the tab.
The most insulting part of this is the amount gained by the attacker is usually a fraction of the cost to you. For each $1 you spend on CI services they might make a tenth of a cent, or in many cases even less. To make any amount of money at all they need to operate at a huge scale, which is why this problem is so bad.
There are other problems with letting the CI run on PRs without supervision. For example, someone could try to steal credentials being used in the build. I don't recommend doing this.
Letting strangers run arbitrary code in PRs with responsibility falling under the repository owner was always gunna turn out bad surely?
Seems like self-hosted might be the way to go in the really soon. 🤔
I don't like the idea of running and managing my own Jenkins server because a few bad actors are exploiting CI pipelines.
Its a good opportunity to learn more on devops for me. If you want something hands off you can use the AWS / DO / azure cloud plugin to create and destroy VMs on demand. If this isn't your thing you can always pay for the CI services themselves.
Personally, I'd recommend terraform or Ansible specifically for the creation of cloud resources. The Jenkins agent would be able to launch those using plug-ins.
Yes I create the master using terraform, but the agents are spun up on demand. Keeps the costs reasonably low without loosing out on performance.
Sure, but there are other scenarios in which users may not be able to do so… like for example for OSS projects
I pay for CI on my OSS projects.
You’re one of a kind 😉 jokes aside, there are many factors. It is always good have more choices and not being limited, especially if the limitations are in place because of abusers
Great article/video! I fully support the newer Proof-of-Stake mining method rather than the Proof-of-Work(PoW) mining like they do to current Bitcoin/Ethereum/Monero.
But as long as Bitcoin still living, the PoW will never ceased. At this rate only hope those CI/CD services can have a mechanism to ban all mining activities as soon as they detected it.
This would be just like dealing with hackers and cyber criminals. You create a game of cat and mouse between the exploiter and the security apparatus.
Thanks. The problem is that it’s almost impossible detecting it until it’s too late 😟
This is Bad.
I started doing open source projects before CI/CD and I know how painful it was to release software without CI/CD. It's just so painful. CI/CD is like a dream come true. These crypto monkeys are just shit. Why would they damage the very industry that brought these crypto currencies in the first place.
I think there is a way to prevent this by identifying which programs are invoked in CI and just block that user. Like is the program invoked is a mining software, that's pretty doable.
It’s not that easy. You can write your own mining software… how would a platform recognize that?
Well for most cases, miners only use popular software so it's easy to block most cases.
Thanks for the article. I learned something new and horrifying. Before, I was hoping that the Chinese govt and others were likely to wake up and charge these miners for their externalities. But now I see that they are just thieves stealing computing (and therefore also environmental) resources. I’m even more disgusted by the business.
Just now (June 14:th and 15:th), AWS CodeBuild had extended delays in execution for a number of accounts.
Also, a number of accounts in at least us-east-1 and eu-west-1 could (can not) not run CodeBuild due to AccountServiceLimitExceededException errors.
My guess is that AWS CodeBuild may have been hit by this kind of activity and that AWS blocked some account/organisation+region combinations, where they may have found suspicious execution patterns and perhaps somewhat new accounts.
Yes I’m pretty sure that’s the case… unfortunately 😟
To be fair with Travis CI, they were also bought, which is probably part of the reason to abandon the free tier.
But it is very inconvenient. Yesterday changed the CI from Travis to Github Actions, for a legitimate Open Source project, which on average maybe runs once a month. And before Travis became so slow the integration tests started to break.
Super interesting article! I didn't realise cryptomining was affecting so many services.
Thanks! Unfortunately it is 😕
Very interesting article!
Out of curiosity, do you know how cloud providers realized their platforms were being used for mining?
I’m not sure. If I have to guess, checking the network traffic to wallets/crypto accounts. But that would be “too late” because all the computation is already done 😩
THX YOU !!!!!
It’s so bad to devs that use this services in real projects and need to CI services. The future is pay or not use that.
Yes, it has a big impact, especially for small projects or devs that can't / won't afford to pay for CI.
What did we do to deserve this?
I know… it really sucks…
Hi Davide,
Excellent discussion.
I did not realize that the scope of this problem impacted CI/CD so strongly.
Keep up the good work!
Interesting, I have always wondered what's stopping them from abusing these free workflows, or some services that charges based on task executed rather than duration or compute power required.
As it turns out... not much :)