DEV Community: Tom Hughes

Building My First Trading Bot (And How It Lost All My Money)

Tom Hughes — Mon, 25 Jan 2021 21:43:12 +0000

It was 2017 and I’d just moved to Thailand. I was loving every minute of it; Working from coffee shops, going to events at night and mingling with other people who were also working online, or at least attempting to.

Life was good.

The only issue was that I was hemorrhaging money, it was a small hemorrhage (cost of living is low in Thailand), but a hemorrhage nonetheless.

I had no income source. My justification for this was that I was learning to write code, and soon enough this new knowledge would be able to provide me with all the money I’d ever need.

I loved the idea of being able to make anything with JUST a laptop and an internet connection, I was determined to make this lifestyle work.

The Dream

I had a great idea. I was going to build a program that would automatically place bets for me.

Furthermore, this bot would be intelligent, it would know exactly how much money I had and would increase / decrease the amount bet accordingly.

If this worked, I could sit back and drink yai leos (large beers) while my bot diligently printed money.

The idea was simple, I’d subscribe to lots of different betting ‘tipsters’, and any selections that more than one tipster recommended, I’d bet on it.

These tipsters were all claiming to be profitable gamblers, so my thinking was that by only betting on the ones that more than one tipster recommended, I’d be increasing the profitability of the bets.

Building the Dream

I discovered a pretty clever way to input all the data. Whenever a tipster would send me an email with the days recommended tips, I would forward it to my bot. The bot would then parse out the information and put it into a big database.

Once I had all the day's selections in the database, I needed a way to be able to tell if there were any duplicate tips.

This is actually harder than it sounds. There was a range of inconsistencies between the way each tipster sent out their tips, such as;

Kempsey 18:30 #5 Little Miss Sunshine
Kempsey 6:30pm #5 Little Ms Sunshine

Notice the inconsistent use of apostrophes, 24hr time, abbreviations etc.

All this made it very hard to actually match the data together. This led me into the topic of ‘fuzzy matching’.

Fuzzy Matching attempts to match datasets that aren’t 100% the same.

The first thing I did was attempt to normalize all the data, by stripping out all the apostrophes and special characters, and converting everything to lowercase.

Then I found a library that could detect and convert 12 hour time into 24 hour time, so I used that.

The beauty about programming is that you don’t need to do everything yourself, there’s public online libraries where you can pull and use other peoples (often much more sophisticated) code. Perfect for a newbie like me!

Now the above example looked like this;

kempsey 18:30 #5 little miss sunshine
kempsey 18:30 #5 little ms sunshine

Finally I found an algorithm known as the Levenshtein Algorithm. This is a form of fuzzy matching that compares how many substitutions are required to make two strings the same.

For example ‘dog’ compared to ‘dag’ has a Levenshtein distance of 1, because all we need to do to make the words match is replace ‘a’ with ‘o’.

I decided that any two tips that had a Levenshtein distance lower than 5 would be assumed to be the same.

Input 1	Input 2	Levenshtein Distance	The Same?
kempsey 18:30 #5 little miss sunshine	kempsey 18:30 #5 little ms sunshine	2	Yes
kempsey 18:30 #5 little miss sunshine	kempsey 15:00 #3 dance monkey	21	No

Now that all the hard stuff was out of the way, all I had to do was implement the actual placing of the bets.

There’s a lot more technical stuff involved in how this was made, but parsing and normalizing the tips was by far the most challenging part of the build.

All my bets were going to be placed on Betfair, which is a betting exchange where people can place bets against other punters. Betfair is great because they have a really easy to use API.

Using the Betfair API to place the bets was pretty straight forward. I ran a query to get a list of all of the horses that were running today, then found the ID of the horse that corresponded to the horse I wanted to bet on (also using the Levenshtein algorithm) and I was off to the races (literally).

Now, it’s safe to say that this whole codebase was a sloppy mess. I’m no zealot about writing clean efficient code, but this was sloppy even by my standards (to be fair, I was still learning).

Sloppiness aside, I was done.

I managed to build a bot that would receive emails, parse data, normalize that data and then query Betfair’s API to place the bets using real money.

100% automated, no input required by me, brilliant!

It would even send me an email with the bets it was placing (as shown below).

The Dream Gone Wrong

Now all there was to do was fund my Betfair account and let the bot work it’s magic!

As I was still a complete beginner, I decided to only bet a TINY amount of money.

The amount I was going to bet would be determined by the odds that the horse was suggested at. I would always bet to profit a certain amount of money, around $100.

Odds	Stake	Return	Profit
10	$11	$110	$100
5	$25	$125	$100
2.5	$66.6	$166.6	$100

Notice how the lower the odds, the more I bet?

This is good because I’d be betting more money on the more horses that were more likely to win, and less on the outsiders.

After a few days of the bot running I received this email.

Oh no.

I knew something was horribly wrong because I would never bet on a horse at odds of 1.08.

Because of the way I was staking, I knew that this was likely going to result in my bot betting my ENTIRE BALANCE on this one horse.

Suddenly the nice message I programmed the bot to send “have a great night” was like a kick in the nuts.

“Hey Tom, I’ve bet all your money. Have a great night”

Sure enough, I log into my Betfair account and see the following.

Ouch, instead of betting a tiny amount of money like I wanted, it bet over $2000.

Considering I didn’t have an income source, this was a devastating blow.

What the hell went wrong?

The Aftermath

I went back into my emails and found the issue.

One of the tipsters emails that I was parsing had added in a new column unexpectedly.

The column that my bot normally would find the suggested odds, was actually a new column called ‘value for money’, a number between 1 and 1.2.

The ACTUAL suggested odds for this horse was 6.5.

But not all hope was lost. Betfair is a market where you can buy AND sell bets.

Maybe I could just sell the bet back to the market? Not quite it turns out.

Betfair operates like an exchange. People put in their orders that they are willing to buy and sell tips at. Because my bot put $2304 on this horse at odds of 1.08, Betfair matched my bet with ANY order that was higher than 1.08.

Considering the market odds for this horse was actually 6.5, I accepted pretty much every order that was available. No price was too low for me! I chowed through that order book like Pacman.

This can sometimes happen in the stock market, and is known as a ‘fat thumb’ (lol).

Literally because someone with a ‘fat thumb’ accidentally hits the wrong key on the keyboard (because of their enormous thumb) and buys/sells WAY more than they want to.

This has the result of eating through the order book, much like my bet did. Below is an example of what this looks like;

Notice the big long red line? That’s a textbook fat thumb. Someone with a lot of money accidentally sold $10 million of something instead of $10,000.

After a ‘fat thumb’ event, the market quickly re-stabilizes as seen above. It’s nothing more than a blip.

Because of this phenomenon, the odds for the horse that I bet $2304 on quickly rose back up to the market average of 6.5.

Given that I got average odds of 2.56, there was no way I could sell the bet without guaranteeing a massive loss. I decided to ride the bet.

I thought about writing a big paragraph about how nervous and excited I was to watch the race, imagining the glory if it won, but it’s not really the point of this post. Long story short, the horse lost and with it went my $2304.

I deactivated the bot pretty quickly after this.

The Postmortem

Looking back now as an experienced developer, there’s literally infinite things I could have done to prevent this.

The most obvious (and simple) one is implementing what’s known as a guard clause.

A guard clause is a boolean expression that HAS to evaluate to true in order for a function to continue executing.

As an example, a function that fails if any of its inputs are negative might have a guard clause like;

return if input < 0

It’s a simple check that everything is as it should be before running.

Below is the function I wrote that is responsible for placing the bet (written in ruby);

def place_bet(market, horse, stake, odds)
    @order = @client.place_orders({
        marketId: market,
        instructions: [{
            orderType: "LIMIT",
            selectionId: horse,
            side: "BACK",
            limitOrder: {
                size: stake,
                price: odds,
                persistenceType: "MARKET_ON_CLOSE"
            }
        }]
    })
    @order
end

Notice there’s no checks or anything. This could place a bet for $1,000,000 and my bot wouldn’t bat an eyelid.

I could add one line of code to fix that;

return if stake > 200

That’s it. Now any bet > $200 would be considered invalid and wouldn't be placed.

That simple, obvious line of code could have saved me $2304.

That is, until the next bug comes along.

Thanks for reading, if you enjoyed this post please subscribe to my blog (https://tomhughes.cc). I write about my experiences as a solo developer, what it’s like living overseas, gambling/investing as an income and more :)

How To Turn Your VPN Into A Proxy Using Python

Tom Hughes — Thu, 02 Jul 2020 23:32:27 +0000

I love scraping data.

I can write a script that in a few seconds can pull data from a site, filter out all the html tags and javascript mumbo jumbo, and spit out the exact data that I want in a beautiful, useable format (preferably JSON).

Without web scraping that would take me HOURS of copy and pasting.

One frustrating part about web scraping though is that generally site owners don't want you scraping their site. Which is totally fair enough.

However, if you're still hell bent on web scraping, you can use what's known as a 'proxy' to hide your IP address.

This makes it much harder for websites to stop you scraping them.

A proxy works by tunnelling all your requests through a seperate server.

For the site owner, it looks like it's the seperate server that's making the request, and they are. But then they are relaying that request right back to you, sneaky!

Today I'm going to show you how to use any commercial VPN (NordVPN, ExpressVPN etc) with the requests library in Python to level up your web scraping game.

First off, we're going to import the libraries we want to use. In this tutorial we're just going to use the requests library.



import requests

Using a proxy with the requests library is done with the following structure;



requests.get(url, proxies=proxy)

That's it. How damn easy is that!

So what is that 'proxy' object we passed into the get function?

The proxy object is a dictionary that maps each protocol (http, https, ftp etc) to a specific proxy in the following format;



proxy = {
    'http': "username:password@host",
    'https': "username:password@host"
}

Now we just need to fill in the blanks here. I'm using NordVPN but any popular VPN service will work (ExpressVPN, SurfShark etc).

Your username and password will be the same as the one you use to login to your VPN.

Notice in the proxy string, the characters : and @ are used to seperate the username, password and host. If you have these characters in your username or password, the interpreter will get confused and the proxy won't work.

For this reason we need to encode our username and password, more info can be found on that here. For reference, @ becomes %40 and : becomes %3A.

Now we just need to fill in the 'host' part of the string.

Navigating to your VPN providers website, there should be a section that lists all their servers, with NordVPN there's a 'servers' link on the homepage that gives you all the information you need;

Using the above information, the host we're going to use is au473.nordvpn.com.

So our full proxy object becomes;



proxy = {
    'http': "tom%40gmail.com:password123@au473.nordvpn.com",
    'https': "tom%40gmail.com:password123@au473.nordvpn.com"
}

These aren't my real login details, but you knew that.

Putting it all together we get;



import requests
proxy = {
    'http': "tom%40gmail.com:password123@au473.nordvpn.com",
    'https': "tom%40gmail.com:password123@au473.nordvpn.com"
}
requests.get('https://google.com',proxies=proxy)

And that's it! Now all the requests you make will LOOK like they're coming from NordVPN, cool huh!

We've managed to turn any VPN service into a proxy with a few short lines of code.

Hopefully you've learnt something new today :)

If you want to be EVEN more stealthy when web scraping, I'll be writing more articles here on the topic, so be sure to follow me to stay updated!

3 Painful Lessons I’ve Learnt As A Solo Developer

Tom Hughes — Wed, 01 Jul 2020 02:31:06 +0000

I’ve flirted with the idea of calling myself a developer for years now. Yet, whenever the words come out of my mouth, it just doesn’t feel right.

I don’t have a computer science background, I’m largely self taught and I don’t work in a team.

I learnt to code almost exclusively from youtube tutorials, blog posts and stack overflow.

While that’s good, there’s definitely some ‘gotchas’ that I wish I’d known before starting out. This article is written for anyone hacking it on their own, I hope I can save you some time.

Avoid Tutorial Hell at all costs.

Tutorial hell is the syndrome where you're perpetually completing tutorials, without ever making any meaningful progress.

"Once I learn React I’ll have more job prospects, and Angular couldn’t hurt!”

In my early days, I felt this insidious loop all too personally. CSS tutorials, Ruby on Rails tutorials, DevOps tutorials, you name it.

The dangerous thing about this is that you’re never actually implementing anything you're learning.

The likelihood of ANY of this information sticking is next to none. It’s the equivalent of an overweight person reading weight loss book after weight loss book, without ever exercising.

The only time things really started to stick was when I was banging my head against the wall building my own project. This will likely be the case for you, too.

Prioritise development speed over everything.

This is the single biggest source of wasted time for me, and I imagine many others.

I’m a perfectionist at heart. When I launch a website, I want users to be able to reset their passwords, chat with other users in real-time and so much more. 100% test coverage would be nice, too.

As a solo developer, I don’t have the time to implement all these features. Especially when 99% of my projects fail. I’ve now learnt to write remarkably ugly code that ships at least 5 times faster than my prior self.

Does it incur a lot of technical debt? Sure. I’d rather that then spend time building unnecessary features that no one uses.

If a project ever sees traction, then and only then will I go back and clean it all up.

Side note, there should be a word to describe the hours of wasted time building features that are never used (technical waste?)

Learn one or two languages, and learn them well.

Have you ever found yourself googling the most basic of things? I have, many times. The amount of google searches I’ve done for “How to find the length of an array in [insert language here]” is embarrassing. These searches are not only a waste of time, but an unnecessary context switch.

By deciding to focus on only one or two languages, you can gain a better grasp of the fundamentals and syntax of said languages, avoiding these costly searches. For example, I only code in Python and Javascript (and some SQL). This decision has improved my overall developer experience and speed greatly.

So there’s the 3 main things I’ve learnt as a solo developer. The hardest point for me to implement was sticking to two languages. It can feel like you’re falling behind when you see all these articles popping up about a language or framework you have never even heard of.

Let me know in the comments the things that you’ve done to improve your experience as a developer, I’d love to hear them.