DEV Community: joseph

Monitor your internet with python

joseph — Fri, 23 Oct 2020 11:32:42 +0000

I live in Germany and Germany is known for its diversity. The people are diverse, the culture is diverse and the quality of internet connections are equally as diverse. Having spoken to my internet provider who insisted I had no problems with my internet connection I decided to write a program to monitor my upload and download speed.

Getting started

We will be using two cool packages in python namely speedtest and matplotlib. speedtest will provide us with the information about our internet connection and matplotlib will allow us to generate graphs based on this information.

Whether you're in a virtual environment or not using one you will have to install speedtest and matplotlib.

pip install matplotlib
pip install speedtest-cli

We will also be using a few built in python packages datetime and csv. CSV (or comma separated values) are a quick way of storing data. We will be using this to store the information about our internet speeds and then using matplotlib to make this information visual.

Gathering data

Create a python file called monitor.py

touch monitor.py

and let's put the following code on the inside

import speedtest
s = speedtest.Speedtest()
while True:
    print(s.download(), s.upload())

and we should be greeted by the following output before hitting ctrl + C to get out of this infinite loop.

75020309.22943133 24381170.373616524
105192450.00822952 40653433.153288215

So by creating a new instance of speedtest as s and testing the upload and download speed we are given the upload and download speed in bits per second. To convert this to megabits per second (Mb/s) we can do the following to include the time of the test too:

import speedtest
import datetime
s = speedtest.Speedtest()
while True:
  time = datetime.datetime.now().strftime("%H:%M:%S")
  downspeed = round((round(s.download()) / 1048576), 2)
  upspeed = round((round(s.upload()) / 1048576), 2)
  print(f"time: {time}, downspeed: {downspeed} Mb/s, upspeed: {upspeed} Mb/s")

which yields:

time: 12:44:15, downspeed: 95.04 Mb/s, upspeed: 32.85 Mb/s
time: 12:44:35, downspeed: 99.46 Mb/s, upspeed: 38.76 Mb/s
time: 12:44:56, downspeed: 100.59 Mb/s, upspeed: 38.94 Mb/s

Now we will move on to recording this in a CSV file. CSVs are large text files which values separated by commas. Here is an example of one of mine:

time,downspeed,upspeed
12:17:01,100.05,38.28
12:17:21,100.53,37.85
12:17:42,74.87,25.92

We start with a header row then have the information split up. We can think about this like an excel sheet:

In order to record into a csv file in python we need to import the CSV package and 'open' a CSV file (if one doesn't exist it will create one).

import speedtest
import datetime
import csv

s = speedtest.Speedtest()

with open('test.csv', mode='w') as speedcsv:
    csv_writer = csv.DictWriter(speedcsv,
                                fieldnames=['time', 'downspeed', 'upspeed'])
    csv_writer.writeheader()
    while True:
        time = datetime.datetime.now().strftime("%H:%M:%S")
        downspeed = round((round(s.download()) / 1048576), 2)
        upspeed = round((round(s.upload()) / 1048576), 2)
        csv_writer.writerow({
            'time': time,
            'downspeed': downspeed,
            "upspeed": upspeed
        })

So while you let this code run for 4-5 minutes we can discuss what is going on. Line 7 with open essentiallly creates a csv file with the name test.csv with the headers name, downspeed and upspeed and writers them into the csv. Then the loop begins and every time a test is performed by speedtest it writes a new row into the csv with the time, download speed and upload speed we specified before. So let's go and look at that now.

time,downspeed,upspeed
12:51:16,99.29,38.66
12:51:37,100.67,38.79
12:51:57,99.7,38.79
12:52:17,92.89,31.99
12:52:38,99.4,38.96

Cool now we are gathering information and storing it in a csv. You could do all sorts of clever stuff with the filename like name it today's date using datetime but for now I will keep it as test.

You could also set up a check to see if a certain amount of time has passed and to stop the application after that but I will leave that up to your own creativity.

Visualising the data

Let's make another python file to generate the graph of our internet connection. This is where we will use matplotlib.

touch make_graph.py

import matplotlib.pyplot as plt
import csv
import matplotlib.ticker as ticker
times = []
download = []
upload = []

The easiest way to generate a graph is by using an array. In order to populate our arrays we will have to iterate through our new csv file

with open('test.csv', 'r') as csvfile:
    plots = csv.reader(csvfile, delimiter=',')
    next(csvfile)
    for row in plots:
        times.append(str(row[0]))
        download.append(float(row[1]))
        upload.append(float(row[2]))

print(times, "\n", download, "\n", upload)

['12:51:16', '12:51:37', '12:51:57', '12:52:17', '12:52:38']
 [99.29, 100.67, 99.7, 92.89, 99.4]
 [38.66, 38.79, 38.79, 31.99, 38.96]

So now we are parsing our data! The next(csvfile) essentially skipss the row of headers (that were for our benefit only, not python's). Now we come on to using matplotlib which I am by no standards and expert on. Their documentation is extensive.

plt.figure(30, 30)
plt.plot(times, download, label='download', color='r')
plt.plot(times, upload, label='upload', color='b')
plt.xlabel('time')
plt.ylabel('speed in Mb/s')
plt.title("internet speed")
plt.legend()
plt.savefig('test_graph.jpg', bbox_inches='tight')

This code tells matplotlib or plt to create a figure with the lines labelled 'download' and 'upload' to be plotted against their respective arrays. So for every item in the download and and time array a mark will be made on the graph against it. Finally the x and y axis get labels and the graph gets saved.

Potential improvements

This is a short form of a project I recently finished which contains automated uploads to a cloud platform, schedules itself daily does some annotations of the lowest download speed. I initially made it to complain to my internet provider however I know that it will be like showing a graph to a brick wall.

If you have any questions about this tutorial please don't hesitate to reach out to me.

@jsphwllng

What is your go-to Emoji on tech twitter?

joseph — Fri, 18 Sep 2020 23:09:02 +0000

Creating a webscraping twitter bot in Python.

joseph — Thu, 17 Sep 2020 14:45:15 +0000

Let's look at building a twitter bot that scrapes information from one part of the web and then spits it out on twitter. There are a few good examples if you want to generate ideas for a twitter bot such as @bbcbweaking, @poemexe and my own @rnghaiku. In this example we will write something that simply finds the first dev.to post and tweets the title.

Requirements:

Python
Text editor
Browser
Patience

Before we write even a line of code we need to download two incredibly powerful modules namely: beautifulsoup and tweepy. Beautifulsoup will handle the webscraping and tweepy will handle the twitter API requests.

Webscraping

Webscraping is one of the most powerful tools that a coder has and it's the first thing I look to do in each language that I touch. we will be using beautifulsoup for this and you can install it from your terminal with the following:

pip install beautifulsoup4

Create a new .py document and add the following lines to the top in order to import all the functions that beautifulsoup (which I will refer to now as bs4) has for us:

import requests
from bs4 import BeautifulSoup

You now need to define the page you want bs4 to open which you can do like this:

page = reqeusts.get("http://dev.to")
soup = BeautifulSoup(page.content, "html.parser")

In order to test that bs4 is doing it's job you can simply print the content of this page to see what we have

print(soup)

Now we are loading ALL the html from the dev.to which is neither readable nor tweetable. We now need to refine our search using the built in "find" function from bs4. In order to find the titles of each post we need to dive into the HTML of dev.to. We can do this by either reading all the HTML (boring, long, painful) or simply use your favourite web browser to do the same.

We can see in the DOM above that each title is buried under several HTML classes and ids. In order to get to these we're going to have to dig through the elements in order to return an array of titles. We can do it like this and I will explain later:

from bs4 import BeautifulSoup
import requests

page = requests.get("http://dev.to")
soup = BeautifulSoup(page.content, "html.parser")
home = soup.find(class_="articles-list crayons-layout__content")
posts = home.find_all(class_="crayons-story__indentation")

for post in posts:
    title = post.find("h2", class_="crayons-story__title")
    print(title.text.strip())

Now lets go through each line of code in order.

page = requests.get("http://dev.to")
soup = BeautifulSoup(page.content, "html.parser")

These open the HTML page and returns the result as a bs4.BeautifulSoup object. We can then find objects within this soup of HTML with the following:

home = soup.find(class_="articles-list crayons-layout__content")
posts = home.find_all(class_="crayons-story__indentation")

If you look at the DOM above the red coloured section refers to everything inside the "home" variable and the posts is everything inside the green. The posts is now an array (or a bs4.element.ResultSet) that we can iterate through as follows

for post in posts:
    title = post.find("h2", class_="crayons-story__title")
    print(title.text.strip())

For every post in the list of posts we search for the h2 with the class of crayons-story__title and print the stripped title for it. This is the purple section of the highlighted DOM. If everything goes correctly you should see a list of posts like this:

But obviously we only want the first one of these so we can just do the following:

top_post = posts[0].find("h2", class_="crayons-story__title").text.strip()

Which in my case returns the Stop Using React post. For selecting other elements such as <a href="foo"> tags please check out this tutorial from compjour.

Anyway now we have something that we want to tweet (which can be any string or even images) we can now look at tweeting.

Tweeting

For this we will use tweepy whose documentation is equally as strong and useful as bs4's and I recommend that you check out. Install in the usual way:

pip install tweepy

and on the top of your python document don't forget to include tweepy

import tweepy

and now we need to set up a twitter app for your bot to operate in. You can do this by going here and creating an app and filling in the simple form, confirm your email then you will have to wait for your developer application to be confirmed. In the meantime, before we get our API keys, we can do some more work on the tweet we want to send.

from bs4 import BeautifulSoup
import tweepy
import requests

def scrape():
    page = requests.get("http://dev.to")
    soup = BeautifulSoup(page.content, "html.parser")
    home = soup.find(class_="articles-list crayons-layout__content")
    posts = home.find_all(class_="crayons-story__indention")
    top_post = posts[0].find("h2", class_="crayons-story__title").text.strip()


    tweet(top_post)

def tweet(top_post):
    consumer_key = "#"
    consumer_secret = "#"
    access_token = "#"
    access_token_secret = "#"
    auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
    auth.set_access_token(access_token, access_token_secret)
    api = tweepy.API(auth)
    api.update_status(top_post)

scrape()

There's a lot going on here so let's slowly go through it. The tweet function that we wrote will take one argument of 'top post' which is what we figured out in the scrape section. The consumer_key, consumer_secret, access_token and access_token_secret are all API keys provided to us by twitter and should be some long, unreadable string. To learn what these tokens actually do I would recommend that you check out the tweepy documentation.

Once you receive the email confirming your API keys be sure to copy them into your function so they function as expected.

Once all of this has been completed we can host our bot on AWS or my personal recommendation heroku. For a good guide on hosting a bot on heroku please check out this great article from data_bae. Now just run your app and let's see what you get!

joseph 🍇

@jsphwllng

Stop Using React

14:39 PM - 17 Sep 2020

0 0

I, of course, advocate using React as I have the hopes of getting a job in Berlin where React is dominating the job market so I will have some explaining to do in my twitter later.

If you have any questions about this article or my bad syntax please message me at @jsphwllng

Reflections on a bootcamp.

joseph — Wed, 16 Sep 2020 16:36:24 +0000

I recently completed a 9 week Full Stack Development bootcamp in Berlin and I have some thoughts on the whole process and hopefully I can help some people decide whether or not it is for them.

The before

I self-studied coding, before the bootcamp, in my spare time. Coding was and continues to be one of my hobbies. It's satisfying, results can come quick especially with languages like Python and Ruby. I decided to elevate my hobby into a career after a few boring career ventures into Project Management and Legal Consultancy. My coding level before the bootcamp was not strong in any sense of the word. I was comfortable with datatypes in Python, I could iterate over an array and I was happy enough to copy code. I applied to this bootcamp after a friend of a friend had completed it.

For me attending a bootcamp was a big gamble. I knew I had the capacity to become a strong coder if I could dedicate a year or more to teaching myself however the prospect of being accelerated by a school convinced me to leave my Project Management position and try and enter the Tech World.

I had a month and a half between my job ending and my enrollment in the bootcamp and during this time I fixated on completing as much of the pre-reading and working on a few small projects of my own.

Applying

The entry bar for coding camps is simply time and money. I applied to a number of bootcamps in Berlin before settling on one and each of the bootcamps had similar entry requirements. Generally consisting of 50 hours of work on one of the various online studying platforms (freecodecamp, codecademy, etc.) with a focus on the language of choice. My bootcamp focused on Ruby which is an incredibly simple language to use and then some closing thoughts on HTML, CSS and Javascript. Aside from being able to complete the pre-course work there is obviously the cost. This bootcamp in Berlin cost ~€6000.

Luckily, for me, the German Government was able to fund my tuition however a large portion of the people on the bootcamp were self-funded and, surprisingly to me, there was very little difference in the attitude between the two groups of people.

It was obvious, even from the start, that some people were attending the bootcamp to transition careers and others were there simply for "something to do" during the COVID situation where numerous companies in Berlin had let their staff go on Kurzarbeit.

The first week

The atmosphere at the coding bootcamp was very interesting. A large mix of ages and genders (but very ethnically homogeneous) getting together to learn. It had been over 5 years since I had left any form of educational establishment whereas some people were joining directly from university and others had full careers that they had been able to take a leave of absence from to code.

The first day, in which we were setting up our laptops (to UNIX systems) was a foreshadow of some things to come. We were shown basic git commands that the students would repeat with little understanding if they had not seen them before. Problems often arose with git among the students, most of which had to be solved by teachers or those with experience in version management.

From the second day onward we used Ruby. In the short-term Ruby seemed like the most sensible "native" language for new students. I had used Python prior to the bootcamp and I would rank Python as slightly more complex than Ruby in its syntax. For new coders Ruby is, I think, a good gateway into coding. However when more complicated syntax was introduced with JavaScript some of the students struggled as they had been used to the simplicity of Ruby.

Each day during the first 7 weeks of the bootcamp we were given between 2 and 6 challenges to solve. If you have used Hackerrank or Codewars then you are familiar with the style of the challenges: you are told to create, more times than not, a function/method that takes an input and then outputs something as a result. Below you can see a typical challenge/answer combo

Specs

Write a method array_to_hash which takes an Array as an argument
and returns a Hash.
If no block is given, then the hash keys should just be integer indexes of elements in the array, converted as Strings.
If a block is given, call it passing the array index and use what’s returned as the hash key.

def array_to_hash(array)
  hash = {}                                      
  array.each_with_index do |item, index|           
    key = block_given? ? yield(index) : index.to_s 
    hash[key] = item                               
  end
  return hash                                     
end

During our time completing these challenges we had access to the Teaching Assistants, contactable by raising a ticket, who would assist in the problem. The code was exponential in the fact that the previous days code would build the foundations of the next's.

The challenge system was a very good way of building upon the knowledge we were given in our 2 hour morning lecture. The morning lecture would introduce the concepts of any given day (for example the first day would be a focus on strings, the second arrays) before letting us loose on the challenges.

As we had completed the prep work prior to the bootcamp the first two weeks of the bootcamp felt, to me, slightly redundant. We were repeating code that we had studied not too long ago. In this regard I would not consider bootcamps to be any better than Hackerrank or FreeCodeCamp where you could simply read a tutorial and then continue coding on your own. Where my bootcamp, and I presume all coding bootcamps, excel is the environment that you are in.

Coding by committee

Each morning the inbuilt system would assign you a buddy. Your role with your buddy that day would be to discuss the code, help troubleshoot each other's problems and to generally build up a friendly environment within the bootcamp which would become crucial in the later weeks.

It was obvious from the first couple of weeks who the stand-out coders were in our batch. More often than not these coders were people who had prior experience coding. The buddy system, for these more experienced people, was a hindrance in terms of the actual coding but in terms of the social skills it proved invaluable. I know, personally, I could have completed the challenges each day in a fraction of the time if I was left on my own however, having now spoken to recruiters and engineers alike, it is equally as important to have strong teaching skills if you want to enter the tech world. By having to explain how or why code works helped my own understanding sometimes but more importantly it let me learn how to teach.

The times that this group-think coding really had an impact was a lot more casual. Sitting with friends and complaining about the challenges, discussing workarounds to get the output to be correct or simply deleting the test spec file was where we built out network and where the real learning happened. Showing code to my contemporaries and having them laugh at my incorrect variable names or debugging together helped me learn how write readable code and how to write shorter code. This is not something you can teach in a lecture.

Copying and pasting

Day 0 copying and pasting:

Open a terminal and type this, replacing the email with yours (the same one you used to create your GitHub account). It will prompt for information. Just press enter until it asks for a passphrase.

mkdir -p ~/.ssh && ssh-keygen -t ed25519 -o -a 100 -f ~/.ssh/id_ed25519 -C "TYPE_YOUR_EMAIL@HERE.com"

Day -1 copying and pasting:

Setup

To setup a favicon, you can use Rails' favicon_link_tag helper.

Best practice:

    Resize your logo image in 32x32
    Upload it in a favicon generator, this will generate a favicon.ico file
    Add it to your app/assets/images folder
    Place your favicon_link_tag in your layout's <head>:

<!-- app/views/layouts/application.html.erb -->
<%= favicon_link_tag %>

One of my main criticism with the bootcamp format, and maybe this is actually a criticism of coding in general, is that we learned just enough to code. From the first day we were almost blindly pasting things into the console without much real understanding of what we were doing. Even at the last day as we were deploying our final product that we were just copying and pasting things into the terminal to satisfy some tests.

Obviously this is how you have to do it in a bootcamp environment because I can't imagine explaining how a compiler works to a group of students who only learned how an if/else statements works less than 8 weeks prior.

My main criticism, however, was the lack of education on how to continue learning. For 9 weeks the students at the bootcamp followed a very structured set of lectures, challenges and tests however now that I am two weeks outside the bootcamp system it has been an interesting transition into learning how to make my own projects, learn new concepts and overall maintain the pace that the bootcamp had set up.

I understand that this is how the majority of education goes, for example when I was studying my bachelors I wasn't exactly prepared to self study in the legal world however I think with tech there is some exception. The tech world is largely self-taught and reading things like documentation is a legitimate skill that you need to develop. The bootcamp felt like its end goal was to get students through the bootcamp and not necessarily produce self-reliant coders.

The stack

I realise that I should maybe give some insight into what we were actually taught as this will allow me to accelerate this diary entry into the interesting final weeks of the bootcamp. The bootcamp's modules are as follows:

Ruby (2 and a half weeks ending with basic OOP)
Databases (1 week starting with SQL before ending on PostgreSQL)
**Front end (1 and a half weeks covering HTML, JS and CSS)
Rails (2 weeks of building webapps ranging from simple websites to an AIRBNB clone)
Project weeks (2 weeks of building our own project)

Before we move onto the project week it is worth discussing how each of these modules went as I believe the pacing and order could be slightly improved.

Ruby

The pacing that the Ruby module was taught I largely can't argue with. I personally thought that the challenges were a bit simple however that is more of a reflection on myself than it is the course.

Databases

Databases were the area I, personally, struggled with the most. This is where I got the most insight into how the TAs operate and the insight that they give. The teaching staff at the bootcamp, including the TAs, are stellar. TAs are assigned each day based on their strengths and very rarely did I have a call with someone where I felt that they hadn't contributed somehow.

Front end

I would like to see Javascript be the first language that we are taught in the bootcamp. The beauty of Ruby relies on its simplicity and it's easy to read nature but this isn't fully appreciated until you touch on other languages. What surprised me about the order of teaching is we started off with the simpler language.

I think that having a week of Javascript learning how to iterate over arrays and all the unusual complexities that Javascript presents would be a shock to the system that the tonic of Ruby would be welcome.

Rails

I personally think for a topic as complicated and wide-reaching as Rails, or any framework for that matter, this bootcamp did a surprisingly good job at teaching. My criticism in this area, which I think might be a critique of the whole construct of bootcamps is the reliance on copying and pasting that I spoke about before.

Project week

This is the real meat and potatoes of the bootcamp. Project week is the last 2 weeks of the bootcamp and is the first real insight into the developer life. There is no real guidelines to follow and you just have to produce something with the knowledge that you gain during the previous 7 weeks. It is a stressful, chaotic time where a group of four students, with one student acting as the lead developer have to work together. In the previous 7 weeks we were simply satisfying tests and then you are let loose.

You are forced to learn how to work as a team, to use code that you hadn't ever seen before and I should really write another piece on this. Generally it was a fantastic experience.

Wrap up

Was a coding a good experience? Yes, it was incredible. I grew as a coder, I grew as a person and I have come away from it with an insight into a world that I wouldn't otherwise have. Is it for everyone? I'm not so sure. I think the people who got the most out of the bootcamp are the ones who have a clear mindset about what they want to get out of it. Some people were hoping to change industry, others wanted to be able to project manage in a group of coders. I feel like those who got the least out of the experience were the people who just signed up to learn something.

Will a coding bootcamp hold up in allowing me to get a job in the real world? I don't know yet.

If you have any questions about coding bootcamps feel free to reach out to me @jsphWllng