DEV Community: Maksim Bober

Algorithmic Trading Showdown: Self-Hosting vs. Cloud Hosting - Uncover the Ultimate Winner!

Maksim Bober — Thu, 27 Jul 2023 15:31:17 +0000

P.S. Pardon my viral title, I'm trying to test out if viral titles on DEV have an effect on stats (-/+), it might be a separate post later 😄

Let's face it; you are not exceptional. In all likelihood, you will perform worse than long-term investors. You will trade more often than them, and you will be learning. The bottom line, doing algo-trading, you will lose most of the time. After a couple of years (in the best-case scenario), you might see positive yearly returns in areas where the markets are not efficient. Those inefficiencies will be temporary and will have limited capacity or high risks. If not, Big Corp would already be sucking all alpha out of it.

Now if you accept the things above and you still want to go into it, the name of your game will be preserving the capital. Cloud costs money and time. Money is pretty obvious. Time is something less so.

Money in cloud

The first step, you would need to have OHCL data pulled daily. A higher resolution is better since you can always resample to a lower resolution and not the other way around. At this point, you would probably get a data provider for 1min OHCL, which would cost you anything from 20$-90$ USD per month. For that much money, you would be able to pull daily or the last 30 days of OHCL with 1min resolution.

You would still have a rate limit. With a rate limit of 300 requests per minute will take you 23 min to pull 7000 stocks or 10 min to pull 3000. Now, how much would it cost you to run an EC2 machine for 23 min a day? (30*23) / 60 * 0.0255 = 0.29325 so it will be 30 cents per month to host this on your EC2 machine given that the EC2 machine runs daily for 23 minutes. This is the cheapest Vcpu: 1 and 2 GiB machine. Now, you would need to figure out how to provision EC2 machine with your pulling logic and figure out how to trigger it on schedule. So that's extra couple of cents per year. Now, AWS prices are not set in stone and could increase so will your budget. All of this data would need to be written to somewhere cheapest and easiest in EC2 is S3. You would need to read this data onto you machine for analysis which is an extra $+ for you. What if you make a mistake and you need to know that the job has failed? That is another expense +.

So what costs money for us so far: data vendor + EC2 + S3 + monitoring + management time. I added management time since it's the time that you are not spending working on your trading strategy. Now let's say you have found a strategy that requires data to be scraped from a browser or vendor has to be pulled 8h per day. Now your AWS costs will start growing more than your data vendor costs. There will be a point when the money you spend for your infra will outgrow the possible edge you can find.

Now let's talk about self-hosting:

This is also not rosy; it will require a higher starting cost but will be kept flat for far longer than your cloud solution. To get started, you need hardware if you have hardware lying around, this is all fine for you. You can set up a Linux machine with ssh access and deploy your fetcher/strategy. You could set up inexpensive monitoring with Cronitor or Discord bot so that you know when the bot fails. The downsides to this approach are that you need to operate your own server and keep it up and running. In my experience, if you keep your server simple, do not run k8s or docker and just deploy bare metal. It will only fail if there is an electricity outage. It should not be a problem initially since the costs you save will make up for it. If you know Rust or C++ you can compile your fetcher to your hardware. Keep it simple, no abstractions, just fetching and writing to storage. All of your data operations will be done on your main machine using Python during the exploration phase.

So what costs money for us so far: data vendor + initial hardware. Note, initial hardware and scaling will be the same cost and the more you will run it cheaper it will become.

When to move to the cloud?

When you start making more money than you would pay for cloud hosting. Moving to the cloud will not make you money automatically. Unless you rely on the speed of access and you want to position your server as close as possible to the exchange for some HFT stuff.

TLDR

In the early stages of algo-trading, you'll face losses; thus, maintaining a cost-effective setup for data handling and processing becomes vital to survival - you should consider transitioning to the cloud only when your trading earnings consistently exceed the costs associated with cloud services.

How to build openssl when you broke it.

Maksim Bober — Wed, 26 Jul 2023 22:55:54 +0000

# Step 1: Update your system
sudo apt update

# Step 2: Install required dependencies
sudo apt install build-essential

# Step 3: Download and extract OpenSSL source
wget https://www.openssl.org/source/openssl-1.1.1l.tar.gz
tar -xf openssl-1.1.1l.tar.gz
cd openssl-1.1.1l

# Step 4: Configure and build OpenSSL
./config
make
# Step 5: Test that everything works
make test

# Step 6: Install OpenSSL
sudo make install

# Step 7: Verify the installation
openssl version

Look into Telegram username auction

Maksim Bober — Tue, 14 Feb 2023 16:17:50 +0000

Telegram has started an auction on short usernames on the TON blockchain. Here’s a good explanation of the historical context of the TON blockchain. For those who don’t know, initially, TON was developed by Telegram, and there were plans to do ICO, but SEC intervened, and Telegram was forced to “distance” itself from the project, but here we are, and TON is part of the in-app experience.

You can see the auction here.

What’s going on with the top 200

There are some outliers that I excluded to be able to plot the majority of auctions.

Most of the usernames are being sold between 755$ and 1000$.

If we plot all of the bids on a single line:

We can see that some usernames go for up to a million.

It’s interesting to see how the most popular usernames are related to existing brands. It’s not surprising, though, if you know the economics of domain name prices. A more accessible and catchier domain name makes it easy to discover/remember your website. Website visits translate into $ pretty quickly.

Not sure if official brands are buying Telegram usernames or if those are speculators trying to flip a famous brand’s name, or if someone with a lot of TON does it for some other purpose (for fun?).

The auction has been enough to generate a price movement on TON, although it was much less than in November 2021.

What’s the bigger picture?

Telegram has 700 million users, and Twitter has 230 million.

Telegram distills the social network model of FB to the bare minimum, a communication channel between people. Now that it has a vast reach, it monetizes using standard internet practices. It has a payment system supported by TON, sells addresses to companies or individuals, adds facilities to manage large channels, allows the publication of different types of content, and has voice chat … You get the point.

Predicting outcomes of Kentucky Derby 2022 race

Maksim Bober — Sat, 07 May 2022 00:05:13 +0000

Context

Kentucky Derby is a horse race that takes place each year in Louisville, Kentucky, United States. I learned about this race from one of my coworkers. He has been doing a side project analyzing past races to predict the outcome of the upcoming horse race. I didn't know anything about horse racing and thought it would be fun to do the same project. I would make predictions and see if they are accurate or not. I would also learn a lesson about how to profit in the real world using DS.

Predictions

As a result of my analysis I found that two candidates are likely to finish in the top 3.

Horse	Trainer	Jockey
Charge It	Todd A. Pletcher	Luis Saez
Simplification	Antonio Sano	Jose Ortiz

Why?

Mostly because of this.
After running a community detection algorithm on top of the social network of jockey and trainer I found this.

There are clear community clusters in the dataset. After suggestion of my coworker to find out what percentage of the means falls into the cluster it turned out that 50% of all wins that happened from 2015-2021 fall into the red cluster.

After finding out if there are trainers from this winning cluster participate in the current race I found out that there are 2 trainers participating but they have 4 horses so there are 4 choices to choose from. Then I norrowed down my scope to two by finding out which jockeys have won before (in my notebook you can see why it's reasonable to make such an assumption).

Why not to use ML?

I'm not sure that this problem is truly "random". I have an intuition that it runs simillar to an old boy network so by doing a community analysis I might get a better shot at getting an edge.

Why do I think that it's an old boy network? Well, primarily because of Bob Baffert, who has a history of drugging his horses and being disqualified from participating in horse competition until 2023. Coincidently he falls into this red cluster that wins 50% of the time. It might be a coincidence, but knowing about Tour De France scandal and Doping in Olympic games by Russia, I would not be surprised that something like this might exist.

Outcomes

So the results of the horse race came in, none of my predictions made into the top 3. Simplification came in close but Charge It has came in 8. So the predictions weren't as good as I thought.

Finished	Horse	Trainer	Jockey
8	Charge It	Todd A. Pletcher	Luis Saez
4	Simplification	Antonio Sano	Jose Ortiz

The winners were:

Finished	Horse	Trainer	Jockey
1	Rich Strike	Eric Reed	Sonny Leon
2	Epicenter	Steve Asmussen	Joel Rosario
3	Zandon	Chad C. Brown	Flavien Prat

Interesting to see that at least one of the trainers that appeared in our winning cluster has made it into top 3, so we weren't too far out with our analysis after all.

Resources

Temperature, BTC price and crypto farm earnings

Maksim Bober — Mon, 21 Feb 2022 03:09:14 +0000

I'm following work of Stephen Wolfram and I got excited by his idea of a computational essay.

I'm also into crypto trading, so I'm always looking for ways of creating some alpha over other crypto-traders (speculators 😋). One of the ideas that made intuitive sense to me was to use temperature at locations of crypto-farms to generate a trading signal somehow. Thus, I decided to explore this idea through a computational essay that you can view here. Sorry, dev.to does not support embeds from Wolfram 😢.

TLDR:

There appears to be some relationship, at least sometimes. Note, the influence from news (Elon twitting) seems to be influencing crypto much stronger than any of the temperature highs or lows at crypto farm locations.

How to profit from it (likely loose money fast)? 🤑

Global Energy Forecast offers a free API that allows to see Heating Degree Days and Cooling Degree Days for any location in the world. Essentially, it's theoretically possible to use it to forecast energy usage for any location in the world. For example, high HDD might force crypto farmers to go offline since electricity companies prioritize residents. So this might negatively affect BTC price, so you might use Kraken Short to try to make money on it (likely loose).

LeetCode vs Projects (Hedgehog or Fox)

Maksim Bober — Mon, 14 Feb 2022 00:06:05 +0000

What would you do to get hired, Litecode or work on projects?

To me, it seems to be the exploitation vs exploration tradeoff. You can specialize and become very good at solving algorithmic challenges. You can even get hired for this in FAANG.

However, building projects and solving your problems with the software will be more rewarding in the long game. It will teach you the experience needed to build and maintain projects as well as you are going to have an opportunity to make money out of it. 💰 It'll be easier for you to answer technical questions at the interviews since you will talk about challenges you encountered when working on your project.

As you will be working on your projects, you are likely to run into problems that will require some algorithmic solution. No worries, you will learn how to solve those once you start Googling and researching how to do what you want to do.

TLDR: Litecode will teach you one thing, working on your own projects will teach you many.

Lessons learned: Migrating Spark App to AWS EMR

Maksim Bober — Fri, 04 Feb 2022 13:53:11 +0000

Some background, the startup that I was working at got acquired, and suddenly all of my programming effort was focused on migrating ML infrastructure to a parent company's infrastructure. I didn't know anything about AWS and EMR at the time, so I had to learn on the job 💪

So here are my newbie thoughts about AWS EMR good, ugly and bad sides.

Good

Management of Spark cluster is done by AWS, so you can spend time on delivering the value instead of mantaining the cluster. ❤️
Read/Writes from S3 out of the box. Data input and output can live in S3 which makes it very easy to share data with teamates. ❤️ ❤️ ❤️

Ugly

Logs...
- To get the logs of you application, you would need to jump through bunch of hoops. Live logs of a running application on the cluster? (Very, very tricky todo). 😭😭
- You can get the logs after the application has finished running, by ssh into EMR master and runing yarn logs --applicationId.
- You can also wait ~5min and get them from the S3 bucket where EMR saves these logs 😭

Bad

Finetuning hole...
- Something that you could run with 4 nodes in your own Spark cluster could take 30 nodes with autoscaling in EMR.
No Spark standalone mode, no nice debugging...
- EMR does not support Spark standalone mode. So you would write your PySpark app submit it to EMR cluster as a job. Once it fails, you would need to go through ENORMOUS stack trace of Yarn, Spark and your application to see the error. No more pleasant debugging experience from the comfort of your IDE 😭😭

Conclusion

Overall, AWS EMR still requires maintenance effort. It's much more involved compared to a simple web app. You would need to understand Spark's inner workings to fine-tune it to get the logs quickly and not bankrupt you in the process with its compute autoscale. (You could disable autoscaling, but then you would need to spend more time tuning your resources)

Getting humble bundle books

Maksim Bober — Sat, 29 Jan 2022 20:38:05 +0000

I have been using Humble Bundle Books for at least three years. However, they still haven't implemented a user-friendly way to download all the books from the order page. They have an option to download all the books, but when you click on it, you will have one popup for each book to download, and you would need to do it multiple times if you want your books in different formats.

So I decided to use my ⌨️🐒skillz to come up with a better solution.

Here I present to you humble_get a cli tool that would allow you to download all of your Humble Bundle books from the comfort of your terminal.

pip install humble-get

# Get YOUR_KEY from humble_bundle page
# https://www.humblebundle.com/downloads?key=YOUR_KEY

# And get all of your books in a nice directory:
humble_get -key YOUR_KEY -format epub -p ./my/books/directory

Resources

Bobrinik / humble_bundle_books

CLI for getting your Humble Bundle Books

Humble book bundles are hard to download from the order page.

If you are going to click on the download button you are going to end up with one popup window per book. Which is a painfull UX to say the least.

So, I decided to implement a command line app in Python that would make it easier to download books from the order page into a directory of my choosing.

Installation

pip install humble-get

How to use it?

In order to use this script, you would need to have a humble bundle key.

You can get it by checking the URL right after buying your bundle.

https://www.humblebundle.com/downloads?key=YOUR_KEY

humble_get -key YOUR_KEY -format epub -p ./my/books/directory
# OR
humble_get -key YOUR_KEY -format pdf -p ./my/books/directory
# OR
humble_get -key YOUR_KEY -format all -p ./my/books/directory

View on GitHub

Kazakhstan and BTC: What would happen to BTC if its hash rate suddenly drops?

Maksim Bober — Thu, 06 Jan 2022 03:48:44 +0000

On the 5th of January 2022, Kazahstan answered this question. On this day, the Kazakhstan government has increased prices on gas 2x, leading to protests which caused the government to shutdown Kazakh internet for 7 hours.

18.10% of the world hash rate is in Kazakhstan, according to https://ccaf.io/cbeci/mining_map. In other words, 18.10% of the hash rate has vanished from the BTC network.

VP of Research at The Block also found that the drop in hash rate was 12%.

Larry Cermak

@lawmaster

Internet has been down in Kazahstan for a few hours where roughly 18% if Bitcoin's hashrate is.

Looks like the impact on the hashrate is so far 12% drop just in the last few hours since internet went down.

16:23 PM - 05 Jan 2022

So we predict that the same number of BTC transactions will go through a smaller number of nodes; thus, BTC transaction fees will go up.

Let's double-check it with Glassnode data and indeed the fees have went up.

How does it affect the BTC price?

Unfortunately, I don't know. BTC prices went down today, but the stock market also fell. It seems unlikely to be related to Kazahstan events and BTC fees. Some say that the fall in BTC and the stock market is related to the US government's announcement of interest rate hikes.

TLDR: If hashrate disappears, fees go up 🤷

Getting an edge in a board game (part 0)

Maksim Bober — Fri, 31 Dec 2021 23:35:03 +0000

I'm on vacation, and because of Covid-19 and Omicron variant, I end up spending a lot of time at home. One of my favourite pass times is to play Gremlins Inc with my friends.

Gremlins Inc is a board game available on Windows, macOS and Linux.

This is how the game looks.

There are three main components at a high level: players, board, and your card deck. An elementary explanation is that you play cards on your turn and move your game piece according to a number displayed on the card. If you play smart, you can play multiple cards during your turn.

Since I'm a developer and want to over-analyze this game even if it does not affect my bottom line, I decided to go and check if I could get some alpha (edge) over my friends.

👺 Evil planning

There are different ways to get an advantage, but I'm interested in the one related to stats and ML (since ML is very 🔥 right now 😎 💸).

Since the game has three main components: players, board and card deck, my edge can come with problem framing for one or more of these components 🔎.

I know that there have been done a lot DS done for HeathStone Game which is another board game that also uses cards although in a slightly different way. I believe in standing on the shoulders of giants so I would shamlessly climb on the research done for HearthStone and try to 🛠️ it into Gremlins Inc.

The first low hanging problem is to learn what's the card distribution in Gremlins Inc and do some DS viz to see if I can spot some actionable insights.

Resources for inspiring reasearch
- Dive in HearthStone from a data perspective
- Analysis of gameplay strategies in hearthstone: a data science approachapproach

Tech stuff that I ❤️ reading or 👀

Maksim Bober — Sat, 11 Dec 2021 17:34:01 +0000

Books

Elements of Clojure by Zachary Tellman ⭐⭐⭐⭐
Hackers & Painters by Paul Graham ⭐⭐⭐
Datascience at the command line 2ed

Papers

The magical Number Seven, Plus or Minus Two found when reading Elements of Clojure by Zachary Tellman

Interviews

Interview with a creator of R language

Conference talks

Computer Science Metanotation bu Guy Steele 🔥🔥
Growing a Language, by Guy Steele🔥🔥🔥

Fundamental ML papers

No Free Lunch Theorem
- It proves that you need to make assumptions about the data to choose the best model possible.
Tidy Data

Great Research in DS

Anton Antonov blog 🔥🔥🔥🔥🔥
- This blog is a rare gem
Ilya Kostrikov

Is it possible to do triangular arbitrage on Coinbase?

Maksim Bober — Mon, 04 Oct 2021 02:49:11 +0000

Triangular arbitrage is the result of a discrepancy between three foreign currencies that occurs when the currency's exchange rates do not exactly match up. These opportunities are rare and traders who take advantage of them usually have advanced computer equipment and/or programs to automate the process. - Investopedia

Illustration of how it works?

Let's use a fictitious ideal example.

You can see here that if you have 1 unit of USDC you can sell it for 3 units of BTC. Then you can sell 3 units of BTC for 1 unit of ETH each so you get 3 units of ETH. Now, you call sell 3 units of ETH for 2 units of USDC each, so you get 6 units of USDC. Thus by going in such a cycle you can make initial investment of 1 unit into 6 units. Pretty sweet deal 🚀🚀🚀

Let's try seeing if we can do the same thing on Coinbase Pro and make some real 💸.

How to get data?

After reading Coinbase API docs

You will get trading pairs that are available on Coinbase Pro.

You can save them to a file and then load them into your Python script to do the analysis.

Let's explore it 🔬

I decided to analyze this data with Wolfram Language since it's much easier and faster.

Interactive example in Wolfram Cloud

Here is how the directed graph of trading pairs looks like.

Now, we need to find out if there are any cycles that we can exploit to do Triangular Arbitrage.

Interestingly, there are no cycles within the trading pair graph. So it seems that you cannot do triangular arbitrage within the broker without withdrawing your money and then buying crypto again 😞

View live triangular arbitrage opportunities on Binance
- https://eugenioclrc.github.io/binance-crypto-triangular-arbitrage/