Crawling Thousands of URL's? Import sys.

Ben — Mon, 30 May 2022 21:20:26 +0000

This post is mainly here to act as a mental note for me in the future! However, you might find it helpful.

I'm coding a bot that loops through a CSV file of about 15,000 URL's, these are added to a Set when successfully scraped.

But, when Firefox or Chrome driver couldn't load the website, my bot would require a restart. The scraper function would need to restart again and check to see if the URL was in the Set.

This would throw an exception saying something to the effect of 'maximum recursion depth exceeded'

If you get this error when running your Python code try this:

import sys sys.setrecursionlimit(40000)

Courtesy of coderjack this will increase the capacity of the stack and allow the code to run.

Be careful with the number you set this to, especially if you are on an old machine. The spinning circle may pay a visit.

HOW TO SET UP A FREE CRYPTOCURRENCY TRADING BOT - BEGINNER'S GUIDE

Ben — Sun, 10 Oct 2021 14:54:03 +0000

Here are the links to the resources you will need to get started:

Video: https://www.youtube.com/watch?v=vtDKiXgHzxA

Bot code: https://github.com/whittlem/pycryptobot

Trading Exchanges that work with this Bot:
Coinbase: https://www.coinbase.com/join/olney_0c
Binance: https://www.binance.com/en/register?ref=73328165

The Raspberry Pi Shopping List:

Raspberry Pi (Model 4b shown in this video with 4GB memory)
Official UK Raspberry Pi 4 Power Supply (5.1V 3A)
Micro SD - Pre-Loaded with NOOBS (Official Card & Adapter) Store: https://uk.pi-supply.com/?ref=r9urn2si3b

(WORLDWIDE SHIPPING)

FIRSTLY BEFORE THE VIDEO:

Open your Pi terminal window and type:

sudo apt update

then:

sudo apt install git

Learn the Basics of Python, enough to edit and make changes to the bot. Python is one of the easiest coding languages to learn and the language of the future with Machine Learning and AI: https://amzn.to/3v4iZfz

Michael, the developer of the bot code, writes regularly about updates and has more information about how to install and configure the bot here: https://whittle.medium.com

How to Install the Bot on a Raspberry Pi:
https://github.com/yanone/raspberrypycryptobot

TradingView (Great app for price/candlestick charts):
https://www.tradingview.com/gopro/?share_your_love=Ben23O

Investopedia:
These charts are recommended:
https://www.investopedia.com/terms/e/ema.asp
https://www.investopedia.com/terms/g/goldencross.asp

Please note that some of these links above are affiliated. By clicking on them and making purchases/signing up it will help support further articles/videos on this project.

DISCLAIMER: The information provided in this post does not constitute investment advice, financial advice, trading advice, or any other sort of advice and you should not treat any of the website's content as such. I do not recommend that any cryptocurrency should be bought, sold, or held by you. Do conduct your own due diligence and consult a qualified financial advisor before making any investment decisions.

DEV Community: Ben

Crawling Thousands of URL's? Import sys.

HOW TO SET UP A FREE CRYPTOCURRENCY TRADING BOT - BEGINNER'S GUIDE