DEV Community: Min

Machine Learning Certifications

Min — Thu, 26 Jan 2023 00:08:19 +0000

I had some time over the end of year / new year to go over some machine learning concepts and cloud technologies. I have used cloud computing here and there - whether it was to spin up a Minecraft server to play with friends, develop a webpage to help with tutoring, or to develop AI applications for work - but I wanted to get an overview of what the three cloud providers had to offer. I also got to fill in some of my many gaps in my knowledge in this vast field of data science and machine learning. Overall, it was a good learning experience. As a side effect, I now have some new badges.

While there were some differences in how the three major cloud provides approach machine learning, their certifications focused on similar ideas: (a) taking models into production with scalability and reliability, (b) how to tackle issues such as privacy, bias, and explainability, and (c) how and when to leverage existing AI/ML solutions (cloud specific or through AutoML). In short, MLOps seemed to be a common theme amongst the three.

MLOps is said to be a combination of machine learning, devops, and data engineering. So it can be very challenging to learn. So here are some recommendations.

Prerequisite Knowledge

These certifications are aimed at professionals who already have some knowledge and experience in machine learning. Maybe you are a software engineer who is increasingly having to incorporate ML solutions. Perhaps you are a statistician dipping your toes in the cloud. Or you are a manager / consultant working in the AI field.

In any case, if you are new to programming and/or machine learning - you will need to start there.

🎓 Computer Science for Web Programming Professional Certificate by Harvard through edX is a great introduction to computer science. It touches on Python, JavaScript, application development, databases and SQL, how the internet works, APIs, and a bit about security.

🎓 Machine Learning Specialization by Stanford through Coursera is a great introduction to machine learning.

Books

📖 An Introduction to Statistical Learning by James, Witten, Hastie, and Tibshirani. It is never a bad idea to re-read this book for the "classical" machine learning models.

📖 Deep Learning with Python by Francois Chollet is another great introductory book, which covers deep learning models as well as working with images and text.

📖 Machine Learning Engineering by Andriy Burkov is a good book that touches on many of the MLOps topics that all machine learning engineers should know.

📖 Fundamentals of Data Engineering by Reis and Housley provides a very good overview of all things data engineering in a vendor/product-independent way.

Courses

🎓 Machine Learning DevOps Engineer Nanodegree by Udacity covered a lot of ground (including tools such as MLflow and FastAPI, and concepts such as writing clean code and automated testing). Despite its cost, Udacity is also a great platform. Their nanodegrees take a project-based learning approach. Very hands-on and a very industry-informed curation of projects. You won't pass unless your project meets all criteria - so the nanodegrees can be quite time consuming, but their feedback system is amazing, and you can resubmit your projects as many times as you need to keep improving.

🎓 Cloud Academy was a great learning platform - especially with the hands-on labs where they provide you temporary access to AWS, Azure, and Google Cloud, so that you can play around with the services you are learning about, rather than just reading about them. They have learning paths (which include video lessons, hands-on labs, quizzes, and practice exams) for the data engineering and machine learning certifications (and plenty more) on all three major cloud platforms.

Practice Exams

Avoid using "brain dumps" (where people try to remember exam questions and share them). Not only is it cheating, but there is no quality control - the questions would be worded incorrectly, or the "answers" will be wrong almost all of the time. The "best" or "official" practice exams are:

Microsoft Azure: Measure Up. Amazing quality, basically the same coverage, style, and difficulty as the real thing.
AWS: Tutorials Dojo. Good explanations, and similar level of difficulty as the real thing.
Google Cloud: This one doesn't have an "official" partner. But the closest thing I could find was Whizlabs. Varying quality from question to question, but reasonable quality. I can confirm that these were not brain dumps. And I can confirm that the questions were mostly useful to study.

JavaScript for Good: Unfinished

Min — Sun, 26 Apr 2020 23:50:13 +0000

2020 is the year that I finally decided to learn web development and I started the Full Stack Web Developer Nanodegree with Udacity. But this wasn't the first time I had tinkered with the web stuff. Doing this nanodegree and the pandemic situation made me think about my previous adventures that involved JavaScript. In a way, I think the takeaway from this post is that programming can be useful even for people who are not developers, but also for developers to think about any past unfinished projects they'd like to revisit.

The background

I was deep in my quarter-life crisis and I wanted to do something that mattered. So I quit my PhD degree in Physics and became a school teacher in a disadvantaged school. I am no longer a teacher, nor have I returned to my PhD (perhaps a story for another time), but I had observed the vast gap that exists between regular schools and disadvantaged schools. Online, or computer-based, learning that I thought to be the future, did not seem like a viable option for many of these schools that struggled to afford textbooks let alone functioning computers. The KhanAcademy that I had loved and used for tutoring, was not suitable for classrooms like this. Managing students through limited resources, on top of having to keep the students from being distracted by the entire internet, as well as dealing with lost passwords and other unnecessary troubleshooting, was challenging to say the least.

The project

Fixing education is not something that I could have ever tackled alone. But a very small part of the problem that I had faced was a solvable one. What I needed was a free, no login, printable, educational resource online. I was a mathematics and science teacher, so (once I had left teaching) I started working on a mathematics worksheet generator (among other unfinished projects).

The solution was static webpages that would generate problems using JavaScript, HTML, and CSS, every time the page was reloaded. I wanted to make it entirely browser-based so that nothing had to be installed, internet connection wasn't required, and that there were no logins required. I had to design it so that when I hit print, the page prints nicely.

This was back in 2014, and I had not returned to it since, other than to use it to print off some worksheets from time to time as I tutored. But I decided to host it on Firebase so that I could simply give the link to my tutoring students to access. Here is the webpage as it was developed back in 2014, with the navigation page added using Bulma.

Mathematics Test Generator

Some of the programming was a little challenging, as I developed everything from scratch, including the drawing of the graphs and the shapes - in a way that was both randomly generated and randomly oriented. The questions also had to be random, but in such a way that the answers to the questions would be calculatable without calculators. The answers needed to be simplified (with surds) and factorised. It was fun to work on.

The issues

So why did I discontinue my work on it? I was unemployed. This, and a few other projects, were all in the desire to give something to the community. Close to three years of unemployment was hard. A PhD dropout without anything to show for work experience for the recent 3 years at the time meant that I could not get a job, even after over 100 applications. Once I had my first very minimum-pay job, I was overworking to compensate for my late start.

Lack of money, lack of time, and ultimately a lack in my belief that this project was worthy of pushing further, were my excuses for not finishing this project.

And I can't realistically see myself returning to it. Not only because I am a much busier person, but also because of my lack of development skills back in 2014. My code was spaghetti - it was crude, unorganised, messy... disgusting to look at. I'd have to start from scratch.

So what

With a huge number of students studying from home and online, given the COVID-19 situation, I wondered how things may have turned out had I continued to work on some sort of educational website for the past 5 years. Maybe, just maybe, I could have done something useful in my 20's.

Do you have an unfinished project? Why did you abandon it? Would you ever return to it?

If you are interested in getting started in web development (which is also a good introduction to programming in general), then this other post might be useful to you.

Learn Web Development for Free

Min ・ Apr 9 '20 ・ 8 min read

#javascript #python #webdev #sql

Learn Web Development for Free

Min — Thu, 09 Apr 2020 07:26:30 +0000

Motivation and Background

My brother has been developing an interest in programming recently. Data science, artificial intelligence, web development, ..., you know, all the cool stuff. Eventually, he built up enough motivation to start learning web development and asked me how he might get started. I'm no expert in web development (I'm more of a data science guy), but as a former teacher, I was keen on putting together a beginner-friendly "curriculum" of sorts.

He had touched HTML and maybe a programming language before. But that was close to two decades ago, and my brother hadn't really mastered it then nor has he done any programming since. So I can treat him as a complete beginner. There were a few things I had to keep in mind:

Time: My brother is working full time, and has to take care of his baby. I have to take a minimalistic approach. Harvard's CS50 on edX.org is amazing - they take complete beginners through the basics of computer science and include a web development course using Python. But it does go through a lot using the C programming language before they get to the web stuff. Besides, unless you sign up, you often cannot follow along what they code in their lectures because they use CS50 specific tools sometimes. I had to mix and match different resources that fit well together.
Convenience: My brother is keen to study in between his busy work/life. I want it to be browser-based as long as possible. Installing tools and setting up environments can be tricky for beginners, and his work computer runs Windows while his home computer is a Mac. It is better for beginners to get started on the code straight away, as trying to mess with tools can be daunting and off-putting.
Pedagogy: I don't want to sound too much like an ex-teacher, but there are progressions that make more sense in terms of learning. Less complex ideas should be introduced first before more complex ideas are introduced. Sounds obvious, but some courses out there are focussed on the job-readiness, and therefore try to get to the most popular tools as soon as possible. For example, out of the front-end SPA technologies, Svelte appears to be the easiest choice to get started, but most courses offer React.
Documentation: There is a lot to be said about learning from video and learning from reading. I won't go into it here, but suffice it to say, that at some point, all developers need to learn how to read the documentation. But not all documentation is created equally. Following the docs for FastAPI is far easier than following those of Flask, although there are far more tutorials made about Flask. The choices I present needed to consider how clear something was explained through a mixture of video and documents.
Language: Obvious necessities are HTML, CSS, JavaScript, and SQL. That, in theory, should be enough, with Node.js and Express.js pretty much covering the backend part of web development. But my brother initially was and still is, interested in data science as well. So I thought Python would be a good thing to start to get used to now.

Oh, and I wanted to collect completely FREE resources.

The result of all these considerations was a curriculum that I have named "Web Dev for Bro". If you are in a similar situation, getting started in Web Development without any programming experience, and don't have much time, you may find this useful too. Here it is:

1 Introduction to Programming

1.1 HTML/CSS

Just do the basic tutorials. No need to do all of them.

HTML Tutorial

CSS Tutorial

1.2 JavaScript

Just do the basic tutorials. No need to do all of them.

JavaScript Tutorial

1.3 Python

Just do the basic tutorials. No need to do all of them.

Python Tutorial

1.4 SQL

Just to the basic tutorials and the SQL Database tutorials.

SQL Tutorial

2 Introduction to Web Development

2.1 Concepts

Only need to watch. No need to follow along. Just try to get used to the concepts at this point.

CS50 2019 - Web Track - YouTube

2.2 Tools and Setup

These tutorials are more about having the necessary tools to be a JavaScript and Python developer. Follow along with the installation and familiarise yourself with the tools. You don't need to be fully comfortable with these tools yet. Simply installing these tools is enough at this point. All of the subsequent courses will provide some guidance on how to use these tools.

VS Code

Visual Studio Code Intro & Setup

JavaScript and npm

NPM Crash Course

Python and pipenv

There are many ways of managing the environment for python projects. Pipenv should be the "best", but some of the tutorials, later on, will use virtualenv. Both are fine, and those tutorials will show you how to use virtualenv. On your own projects, try to use pipenv.

Pipenv Crash Course

Git/GitHub

Git & GitHub Crash Course For Beginners

SQLite and PostgreSQL

There are many databases. Some of the most popular are PostgreSQL, MySQL, and MongoDB. For most cases, PostgreSQL is the best option. All of these databases require a database "server". This can be cumbersome, so developers often use SQLite during development and then switch over to PostgreSQL during production. So it is good to know a bit of both. The SQL language is mostly the same so there is nothing "new" to learn from what was covered in W3School, but here are some tutorials on how to work with SQLite and PostgreSQL.

Sqlite 3 Python Tutorial

PostgreSQL - Installation & Overview

2.3 Practice

Follow along with this introductory tutorial. There will be a concept that hasn't been introduced before called ORM, which will use a tool called SQLAlchemy. Don't freak out, just follow along as an introduction for now. It will be covered in the future.

Learn Flask for Python - Full Tutorial

🍎 By this point, you should have a basic idea of how to make small and simple web apps.

3 Web Development Fundamentals

3.1 Concepts

Things about to get more advanced from here on. Watch the following video lectures. There is no need to follow along, but you may need to re-watch some lectures a couple of times to really absorb the concepts.

CS50's Web Programming with Python and JavaScript - YouTube

3.2 Practice

Follow along the following tutorials to get some practice of backend and frontend web development. Some of these tutorials may be similar, but it is worth doing all of them for practice. Each tutorial has a slightly different focus, so it is well worth going through all of them. Note that you'll be moving towards separating the frontend from the backend. The backend provides the API and the frontend consumes the API.

Backend

Follow along with this tutorial to built a full web app using Flask.

Python Flask Tutorial for Beginners - Full Course in 3 hours (2020)

While it is possible to build full apps using Flask and its templates, modern apps often separate the backend from the frontend. This means that Flask will only be used to provide an API that the frontend part of the app can use.

One tool that can help you test these APIs as you build the backend before building the frontend is called Postman. As with other tool videos, you don't need to remember everything yet, but install postman and familiarise yourself.

Postman API Crash Course for Beginners [2020] - Learn Postman in 1 hour

Now follow along with this tutorial to build an API.

REST API With Flask & SQL Alchemy

Frontend

There are many frontend frameworks, one of the easiest to get started is Svelte. Follow along with this tutorial to get a feel for it.

Svelte 3 - Quickstart Tutorial (Getting Started With Svelte.js)

You will feel like you need to go back and revise JavaScript. This is perfectly normal and there is no harm in going back to basics as you need. But just to show how amazingly simple Svelte will be once you get it, here is a 3 minute tutorial that shows you how to build a todo app in 15 lines of code.

Intro to Svelte - Todo app in 3 minutes and 15 lines of code

And here is a longer one that you can follow along with to build a to do app with a bit more complexity. Also, towards the end of this tutorial there is an introduction to how to connect such frontend apps to the backend API.

Svelte v3 - Basics - Todo App

🔥 By this point, you should have a pretty good idea of how to make simple web apps.

4 Master Web Development

You are already ready to start building hobby web apps. To become better, you need to become pretty good with at least one frontend framework (you've met Svelte) and at least one backend framework (you've met Flask and Django). In addition, you will need to get comfortable with deploying web apps online (Heroku, for example). Underlying all this is being good at Python and JavaScript.

4.1 Learn a Frontend "Framework"

Many choices here:

Svelte
Vue
Angular
React

Pick one and master it. Svelte is the most elegant and easiest to learn. It is also the best introduction to frontend technologies. The official documentation has a great tutorial.

Svelte

To get a job in frontend development, you need to know the popular tools. The next thing I would learn would be Vue. This is optional, and I would try and make a few web apps using Svelte first.

4.2 Learn a Backend "Framework"

Many choices here too:

Flask
Django
FastAPI
Express

As with the frontend case, pick one and master it. FastAPI is the most modern python framework. The official documentation has a great tutorial. In fact, FastAPI has the best official tutorial that covers a lot of concepts including things like security. However, it does not have many video tutorials on youtube or anywhere else yet. The framework design is very similar to Flask, so the knowledge should be transferrable.

FastAPI

In the Python world, the most popular frameworks are Flask and Django. But the backend development market is fragmented into Python, JavaScript, Ruby, PHP, Java Scala, Go, Rust, and many more languages. I would stick with Python and JavaScript. For JavaScript, Express is the most popular backend framework. The next thing I would learn is Express. Again, this is optional, and I would try and make a few web apps using FastAPI first.

4.3 Figure out how to deploy web apps

There are many options here and things keep changing all the time. Some starting points are:

Docker
Heroku
Firebase

Really, these things are great. Seriously great. There should be plenty of tutorials out there on how to use these tools. Some Youtube channels below will have a lot of tutorials on these.

😎 By this point, you're good to go bro. You've got this.

5 Continue Learning

Youtube

Traversy Media
Academind
The Net Ninja
Fireship
Pretty Printed
Code Drip
Web Dev Simplified
Tech with Tim
Corey Schafer
Svelte Master

Udemy

They do massive discounts fairly often. So when there is a discount, purchase some courses that you are interested in. Svelte, Vue, Flask, Express, Heroku, Docker, Firebase, Authentication, Payment, etc.

Location Data for Cities and Towns

Min — Wed, 19 Feb 2020 20:10:21 +0000

Why might you want location data for cities and towns?

Have you ever wanted to geocode some address data, but only to the city/town level - say, for privacy reasons or entity resolution purposes?

Or maybe you just need to plot out all the major cities around the world on a map?

There is no easy way of obtaining such data.

You could try:

Google (or some other commercial) geocoding APIs on your lists of city names and towns, but that will probably cost you money.
Open Street Maps (and APIs that support using OSM), but this isn't easy. Especially if you have a bulk data you are trying to geocode.

WikiData has a very good source.

OSM is not the only open data available. WikiData also has quite a bit of such data.

This is how you would obtain the locations of cities and towns around the world.

Go to https://query.wikidata.org

and use the following sparql query:

SELECT DISTINCT ?citytownLabel ?countryLabel ?loc
WHERE {
   ?citytown wdt:P31/wdt:P279* wd:Q7930989 .
   ?citytown wdt:P625 ?loc .
   OPTIONAL { ?citytown wdt:P17 ?country } .
   SERVICE wikibase:label {
      bd:serviceParam wikibase:language "en" .
   }
}

After about a minute, you should get something like:

You can simply download the data as csv or json.

You can also take a look by displaying the data on a map (there should be a dropdown on the left.

Interesting to see which communities are more active in open data contribution.

And if you are interested in contributing the location data, or are a school teacher who can make such "data contribution" a geography class project, then the following query will give you a list of cities and towns without a location data, and the link to their wikidata pages.

SELECT DISTINCT ?citytown ?citytownLabel ?countryLabel ?loc
WHERE {
   ?citytown wdt:P31/wdt:P279* wd:Q7930989 .
   ?citytown wdt:P17 ?country .
   FILTER NOT EXISTS { ?citytown wdt:P625 ?loc }.
   SERVICE wikibase:label {
      bd:serviceParam wikibase:language "en" .
   }
}

You will also realise that the towns that do not have location data tends to be missing a lot of other data too, like their English labels.

WikiData and the community of contributors are amazing.

Best Free Data Courses for Beginners

Min — Wed, 29 Jan 2020 19:24:17 +0000

You might think this is yet another post about free resources for Python. In one sense, yes it is. But really, it is not.

Make your decision to learn SQL and Python (maybe even some R)

It is the year 2020, and there are still post on blogs and Linkedin encouraging people to learn Python. Gone are the days of Excel Data Analysts. Watching youtube videos and reading blog posts, this overall trend may very well be true, but we are far from living in a time where all Data Analysts (and BI Analysts) know how to code. Many have progressed to using PowerBI or Tableau on top of Excel, but still, have much reluctance to learn to code.

Many others have written about why Data Analysts should learn to program/code, and I do not have much to add there. But I will quickly mention three barriers that people face:

I'm not a mathematics/programming/technical person.
I don't have the time.
The degrees/courses seem expensive.

As an ex-teacher, I hate the first statement. These skills are not beyond the reach of anybody. It is merely up to your motivation and mindset. And if you are not up for getting "technical" then perhaps get a different job.

The second excuse is more realistic, and I don't think I can help much, other than to say that 5 hours per week for a couple of years can take you a long way. As professionals, we should always strive to learn and improve.

Cost. Now, this is where this post can really help you. Whether you are a student wanting to learn data science, a BI Analyst wanting to learn Python, or a corporate person (finance, HR, marketing, etc.) wanting to use analytics in your job, these courses will help you get started for free.

Free does not have to be 💩

I have taken many (paid) courses from edX, Coursera, Udacity, DataCamp, Udemy, and a degree from a University. They all have their strengths and weaknesses, styles and emphases. I'll make a separate post about my thoughts on those perhaps, but for now, let me introduce their (surprisingly less known) free counterparts.

Firstly, while edX and Coursera are great, their "free" version did not turn out to be completely free. Last time I checked, the free auditing came with a time limit, usually a few weeks before you lose the free access. Secondly, university degrees are (to my knowledge) not free, nor is Udemy. Lastly, while Udemy can be cheap when their discounts are on, even at the low cost, I wouldn't recommend Udemy for data-science-related courses - despite most other posts listing Udemy and Coursera as their top two choices.

That leaves DataCamp and Udacity. What surprised me is that many people did not seem to know that they offer free courses and tutorials. I am also adding the Kaggle courses (again, surprisingly lesser-known) to the list.

The free courses

Udacity

If you go to https://www.udacity.com/courses/all, you will see the entire list of Udacity's courses. You can then filter to see only the Free Courses. My top picks for beginners are:

Kaggle

Kaggle is not just the data science competition grounds it once was. Kaggle not only provides free online compute environment, it also has several courses offered for free. The Udacity free courses are more video based, while the Kaggle courses will be entirely notebook based. The Kaggle courses will complement the Udacity courses nicely.

DataCamp

DataCamp is very affordable and very comprehensive in its library of courses, covering spreadsheets, SQL, Tableau, R, Scala, Python, and probably growing. But, if you want to access their free content beyond just the first parts of their paid courses, they do have a blog-like community-created tutorial content as well. These are not created by DataCamp, and their qualities may vary, but some of the great ones are kindly tagged as "must read". These tutorials can be a great supplement to other courses. Here are my picks:

Improving your data wrangling game

SQL Tutorial: How To Write Better Querie | DataCamp
Using PostgreSQL in Python | DataCamp
Introduction to MongoDB and Python | DataCamp
Importing Data into Pandas | DataCamp

Improving your pandas game

Pandas Tutorial: DataFrames in Python | DataCamp
5 Tips To Write Idiomatic Pandas Code | DataCamp
Python Exploratory Data Analysis Tutorial | DataCamp
Python Time Series Analysis Tutorial | DataCamp

Practicing machine learning

Essentials of Linear Regression in Python | DataCamp
KNN Classification using Scikit-learn | DataCamp
K-Means Clustering with scikit-learn | DataCamp
(Tutorial) Principal Component Analysis (PCA) in Python | DataCamp

Conclusion

As you can see, there are plenty of free resources available from very reputable training providers. You can go a very long way with these free courses alone, but I would recommend that once you are done testing the waters out, then go for the paid courses from a combination of education providers:

Bonus: Youtube

Depending on your interests, there are a plethora of resources available on Youtube as well (although, you would need to be able to sift through what is good and what is not). Some very popular Youtubers turned out to be plaigerisers, and some others, whilst pretty good, write code in a style that I would not want to collaborate with. Don't get me wrong, there are some that are great, it is just that you would already have to have a sense of what is a "better" code to decide which videos are high quality. Having said that, there are really great "theory-explaning" videos on Youtube that cannot be found elsewhere. So my advice would be: don't try to learn SQL, R, and Python using Youtube, but use Youtube to look up some theories you want to get a quick understanding of. I've prepared a playlist for machine learning if you are interested.

Happy learning!

Documenting SQL with Markdown and Diagrams

Min — Sun, 26 Jan 2020 04:15:34 +0000

Nobody seems to document SQL codes

I find that SQL code is often undocumented. A lot of SQL code don't get the sweet documentation love that python and javascript codes so often do.

Some commented out lines of code, which supposedly apply a particular business rule over another, are left hanging without what they were supposed to do. Some comments allude to warnings about the consequences of changing particular join conditions, without an explanation as to why.

Many SQL codes can go over a dozen layers of join paths deep, with multiple subqueries, badly written split-apply-combine methods, and inexplicably named aliases.

Someday that might bite us on our backsides

Executives of the business may one day get an automatically generated report one day and question "how the figures were generated" (Think "why is this number so low?"). With people moving from one job to another, undocumented SQL code jibberish is all that the analyst has to go by (if the analyst is given access to code at least).

A lot of data pipelines rely on SQL, and they really should be documented better - explain to the user the code that they are looking at. Throw in a diagram perhaps.

Let's use markdown in a comment block

But we shouldn't have to write a separate document that does these things. Separating the documentation from the code increases the chances that the documentation will not be updated. Documentation should really live in the code as comments. I like the simplicity of markdown, and think it would be nice if I could use markdown within the comment block of the sql to document the code.

There's a tool for that.

A little while ago, I posted about documenting Python code for data science. I had built a package called mindoc that allowed me to document my pipeline code using markdown within my code.

Documenting Python Data Science Code with mindoc

Min ・ Jan 18 '20

#python #documentation #datascience #productivity

Now I have expanded the tool to help document SQL code and added support for diagrams (using mermaid js)!

See this github page for how it works: https://minchulkim87.github.io/mindoc/

Documenting Python Data Science Code with mindoc

Min — Sat, 18 Jan 2020 12:00:59 +0000

You can write amazingly readable code in python.
Think of the hierarchies of abstraction the same you would reports.
mindoc can help you document your code like markdown.

https://minchulkim87.github.io/mindoc/

Write amazingly readable code in python.

This talk was a game-changer to me:

I shared it with my colleague, and it changed how we used the already-amazing Jupyter notebook.

I was already trying to write code as readable to other humans as I though possible, but the talk gave me a better template to work with.

Basically, the idea is to write code that even a non-dev could understand if they tried.

In pandas, for example, the equivalent way of writing code would be:

(df.pipe(clean_data)
   .pipe(group_data_by_customer_type)
   .pipe(compute_statistics)
   .pipe(generate_visualisation))

Even a non-data-person or a non-coder would recognise that the data was being cleaned, then grouped by customer type, then some statistics are being calculated before the visualisation is created.

With this newfound excitement over just how clean our code started to look, we quickly started re-factoring our data pipeline codes.

And then came the awkward moment

The great thing is that we were able to abstract away what steps like the "clean_data" were! If the reader was curious about what was meant by "cleaning" the data in this code, they simply need to look at the clean_data code, which may look like the following:

def clean_data(df: pd.DataFrame) -> pd.DataFrame:
    df = (df.pipe(rename_columns)
            .pipe(remove_special_characters)
            .pipe(correct_date_format)
            .pipe(replace_values))
    return df

Again, the steps taken in the clean_data "phase" were obvious. Again, you could abstract away further details. Great!

Then we hit our first moment of 'um...'. We constantly had decisions to make about exactly what we would abstract away. And to what end? Were we to get down to the lowest level of "atomic" manipulations of the data?

Once you get to over a hundred such functions, how do you order them? In the order they are used? What if they are used multiple times? Alphabetically? But then the code stops "explaining itself".

Our project seemed way too large for this way of coding to scale to our needs. Not to mention that even the most readable code becomes too "technical" to the manager or other business people who are not likely programmers or "data people". We needed a way to write "documentation" for both the other wizards and the muggles.

Then came the Aha! moment

I remembered watching another amazing talk (for a javascript audience, but the message applied to any developer):

Then I suggested that we think of organising the code as if we were writing a Word document. The levels of abstraction were equivalent to levels of headings. If we were to write the code in English, what would the title, headings, and subheadings be?

We decided it was a good idea to arrange our code this way, and started to use the comments in python to indicate headings.

The commenting syntax resembles a markdown header.

# This is a comment. Or... is it a header?

Surely this made sense to a lot of other people. Surely, there was some standard practice around documenting data analysis, data engineering, and data science projects with code.

We took to the python documentation tools.

We envied using Sphinx. But while it seemed great for documenting packages, it didn't meet the needs of a data project.

First of all, "documentation" for python modules uses docstrings heavily. These are useful for creating a manual of sorts to the user of the module. What methods are available, what arguments I can give this function, etc.

But the point of this new way of coding was that the code was "self-describing". Did I really need to start writing like this?:

def clean_data(df: pd.DataFrame) -> pd.DataFrame:
    """
    This function cleans the dataframe.

    First the columns are renamed.
    Then the special characters are removed.
    Then the date format is corrected.
    Then some values are replaced with others.

    Args:
        df (pd.DataFrame): Pandas DataFrame to be cleaned

    Returns:
        pd.DataFrame: The cleaned Pandas DataFrame
    """
    df = (df.pipe(rename_columns)
            .pipe(remove_special_characters)
            .pipe(correct_date_format)
            .pipe(replace_values))
    return df

Surely, the names of the functions and the python type hinting already fully details the function. The "documentation" we needed was to document the process rather than the tool. Like I wrote above, the documentation of the tool is absolutely required. The processes used in a data project that needs to be used by others? Not so much. The process should be documented though.

There was Pycco, which we drew some ideas from, but the presentation made it feel like it was annotating the code rather than documenting the code. Again, there is a place for annotation, but it didn't quite meet our needs.

There was an added limitation of being a public servant. We did not have the luxury of using "pip install" - especially when it came to upgrading JupyterLab or extending them. Even if we discovered a solution, if it required packages we did not have access to, we would not be able to use it that easily.

So. I decided to write a simple documentation tool.

Introducing mindoc

It is a simple tool that converts a .py file into an HTML document.

It is basically a flipped version of markdown.

In markdown, you write a document and put code between fenced triplets of backticks.
With mindoc, you write code and put the documentation between fenced triplets of quotes.

The use of the headers allows you to organise your code in hierarchical levels, and "hide" away whichever level of granularity you wish to make the logic of the code more readable.

It automatically generates the table of contents for you too.

Go check it out on GitHub!

The page that you see from the link is the .py file (converted into HTML).

Company data using Wikidata

Min — Sat, 11 Jan 2020 04:56:13 +0000

Company data is useful.
Company information is hard to find in bulk.
Wikidata can be a useful starting point.
Wikidata can be great for collecting other information too.
Let's contribute to open data, not just open source codes.

Company data is really useful

Whether you are doing competitor analysis, econometric analysis, or policy analysis, it would be great to have information about companies. Working in the government, we are often asking how a new public policy might affect small businesses. Necessarily, we need to have data about the list of businesses, and whether they are large or small.

Collecting information about companies is hard

You may think that there are government bodies that collect this information. After all, corporate taxes are a big part of how governments source their funds. However, just because one part of the government has a certain set of data, it does not mean that other parts of the government can access that data. Much of these restrictions are necessary for privacy - and I am not here to debate data sharing and governance policies - but we can agree that having access to such data can benefit researchers.

Some options for company data

Buy the data

For the very reason that the company data is hard to get, there are organisations that sell such information. Organisations such as universities and government agencies purchase these data, without many alternatives.

Get data provided by some governments

Some governments are pretty good with organising and releasing data. Australia, for example, has data.gov.au where they host information such as a list of businesses and their ABNs (Australian Business Numbers) so that you can identify them. Also on data.gov.au is a dataset released for tax transparency purposes for "large" companies.

A quick side note: definitions of large companies can vary. You could use the annual turnover (revenue) and/or the number of employees as measures of size. It would depend on what question you are trying to answer.

Try and get publically available data

You could try some form of web scraping. Some very nice people at www.peopledatalabs.com have released what they have collected (I suspect from Linkedin, by scraping or by using the API). This dataset would be limited to those with a Linkedin presence and has no information about revenue.

Companies publish annual financial reports and searching for these online could be one way of gathering information. I guess that is what some of the data vendors do. This, as you can imagine, is a tedious and time-consuming task. Perhaps there should be an open-source community whose mission is the make this data available and accessible. Such an initiative does in fact exist: OpenCorporates. But to me, this data was exposed more as a search tool (as opposed to a downloadable bulk data), and there seems to be a lot of inconsistent duplication of data. At least for what I was trying to do, I could not work out how to even select the "closest" match for the search results. I'm sure that there are many use cases for this tool, but it did not suit my use case. Also, this data isn't exactly "free".

Wikidata

This is where Wikidata comes in. "Wikidata?" you may ask. Here is what the website says:

Wikidata is a free and open knowledge base that can be read and edited by both humans and machines.
Wikidata acts as central storage for the structured data of its Wikimedia sister projects including Wikipedia, Wikivoyage, Wiktionary, Wikisource, and others.

There is an enthusiastic talk about it in youtube as well:

It is really amazing. We should really consider supporting the Wikimedia Foundation

So I have been playing around with the SPARQL (feels like SQL but with its own dialect) (tutorial here).

Specifically for finding "large companies" I have developed these following codes that you can run yourself at Wikidata Query Service. Note that I have tried to gather ISNI and GRID ids as the identifiers that can be used to join with other sources of information if required.

For largest companies by (latest available information about) number of employees:

SELECT DISTINCT ?business ?isni ?grid_id ?businessLabel ?officialname ?shortname ?countryLabel ?employees
WHERE {
   ?business wdt:P31/wdt:P279* wd:Q4830453 .
   SERVICE wikibase:label { bd:serviceParam wikibase:language "en" } .
   ?business wdt:P17 ?country .
   OPTIONAL { ?business wdt:P213 ?isni } .
   OPTIONAL { ?business wdt:P2427 ?grid_id } .
   OPTIONAL { ?business wdt:P1448 ?officialname FILTER( LANG(?officialname) = "en" ) } .
   OPTIONAL { ?business wdt:P1813 ?shortname FILTER( LANG(?shortname) = "en" ) } .
   ?business wdt:P1128 ?employees .
   FILTER( ?employees >= 200 )
}
ORDER BY DESC (?employees)

For largest companies by (latest available information about) revenue (converted to USD):

SELECT DISTINCT ?business ?isni ?grid_id ?businessLabel ?officialname ?shortname ?countryLabel ?revenue_usd
WHERE
{
   {
      SELECT DISTINCT ?business ?isni ?grid_id ?businessLabel ?officialname ?shortname ?countryLabel (MAX(?date) AS ?max_date
      WHERE {
         ?business wdt:P31/wdt:P279* wd:Q4830453 .
         SERVICE wikibase:label { bd:serviceParam wikibase:language "en" } .
         OPTIONAL { ?business wdt:P213 ?isni } .
         OPTIONAL { ?business wdt:P2427 ?grid_id } .
         OPTIONAL { ?business wdt:P1448 ?officialname FILTER( LANG(?officialname) = "en" ) } .
         OPTIONAL { ?business wdt:P1813 ?shortname FILTER( LANG(?shortname) = "en" ) } .
         ?business wdt:P17 ?country .
         ?business p:P2139 ?statement .
         OPTIONAL { ?statement pq:P585 ?date } .
      }
      GROUP BY ?business ?isni ?grid_id ?businessLabel ?officialname ?shortname ?countryLabel
   } OPTIONAL {
      SELECT DISTINCT ?business ?date (MAX(?revenue_usd_recorded) AS ?revenue_usd)
      WHERE {
         ?business wdt:P31/wdt:P279* wd:Q4830453 .
         ?business p:P2139 ?statement .
         OPTIONAL { ?statement pq:P585 ?date } .
         {
            ?statement psv:P2139 [
               wikibase:quantityAmount ?revenue; wikibase:quantityUnit wd:Q4917
            ] .
            BIND( wd:Q4917 AS ?unit ) .
            BIND( ?revenue AS ?revenue_usd_recorded ) .
            FILTER( ?revenue_usd_recorded > 100000000 )
         } UNION {
            ?statement psv:P2139 [
               wikibase:quantityAmount ?revenue; wikibase:quantityUnit ?unit
            ] .
            FILTER( ?unit != wd:Q4917 ) .
            ?unit p:P2284 ?unit_statement .
            ?unit_statement
               psv:P2284 [ wikibase:quantityUnit wd:Q4917; wikibase:quantityAmount ?usd ] .
            BIND( ?revenue * ?usd AS ?revenue_usd_recorded ) .
            FILTER( ?revenue_usd_recorded > 100000000 )
         }
      }
      GROUP BY ?business ?date
   } .
   FILTER( ?date = ?max_date )
}
ORDER BY DESC(?revenue_usd)

Not all companies will have entries for employee numbers or revenue. I have also tried getting publicly traded (securities) companies with the assumptions that such companies will be large. For these companies, I have also extracted their ISINs.

SELECT DISTINCT ?business ?isni ?grid_id ?businessLabel ?officialname ?shortname ?countryLabel ?stockexchangeLabel ?isin
WHERE {
   ?business wdt:P31/wdt:P279* wd:Q4830453 .
   ?business wdt:P17 ?country .
   ?business wdt:P414 ?stockexchange .
   SERVICE wikibase:label { bd:serviceParam wikibase:language "en" } .
   OPTIONAL { ?business wdt:P213 ?isni } .
   OPTIONAL { ?business wdt:P2427 ?grid_id } .
   OPTIONAL { ?business wdt:P1448 ?officialname FILTER( LANG(?officialname) = "en" ) } .
   OPTIONAL { ?business wdt:P1813 ?shortname FILTER( LANG(?shortname) = "en" ) } .
   OPTIONAL { ?business wdt:P946 ?isin } .
}
ORDER BY ?business

I hope that someone may find this information useful. But I also hope that others can improve on this, because I would really like to access reliable and maintained data about companies - one with (official) identifiers, revenue data, employee numbers, and ultimate global owners. I must admit, I am not great with SPARQL.

Also, I can't thank all the people contributing to Wikipedia enough. Again: We should really consider supporting the Wikimedia Foundation - through contributing data or donations.

My Data Science Tech Stack 2020

Min — Tue, 31 Dec 2019 05:32:15 +0000

There is a lot to learn in data science.
We can group the technologies by the subfields of data science.
There are a few key technologies for each subfield to focus on.
Creating this personal tech stack list was a fun and useful exercise.

There is a lot of things to know in Data Science.

If we tried to survey the technologies used by Data Scientists, we might get a picture like this:

This list is in no way comprehensive (this is already filtered based on my personal interests). On top of that, this list will change year-to-year. Keeping up with everything will be impossible. Thankfully, no Data Scientists will need to know or use all of these tools.

But since we should strive to be T-shaped people, we should at least learn a good chunk of these technologies, right? But where do we begin? How much should we learn, and what technologies? First, we should discuss briefly the nature of the profession itself.

Data science is a broad and loosely defined field.

Data science is still relatively young and continues to evolve. Once popularised by the tagline "sexiest job of the 21st century", many people were attracted to the interesting profession.

What may have begun as an application of statistics to solve business problems, is now a name that encompasses areas of big data engineering, visualisation, machine learning, deep learning and artificial intelligence. The rapid evolution was in part due to the breadth of areas in which data science can be applied to, but also because the technologies have also developed rapidly. The number of skills that a Data Scientist must possess has grown with the nebulous definition of the job.

I imagine a one-person Data Scientist in a small organisation would have a different set of tasks to do compared to a Data Scientist in a team within a large organisation. I also imagine that the exact job will depend very much on the industry and the nature of the organisation. Compounded with the rapid decrease in job tenure (or an increase in job mobility), this variance in the job description requires the practising Data Scientist to keep up with a large number of skills and technologies.

If we had to group some of the subfields of data science, they would look something like this:

Data Analysis: This part of the job is about understanding the data. It involves data wrangling, exploratory data analysis, and "explanatory" data analysis. In a larger team, dedicated Data Analysts will perform these tasks.
Data Visualisation: This part of the job is about communicating the data, usually to a non-technical audience. In a larger team, dedicated Business Intelligence Analysts will perform these tasks, although this can be a part of the Data Analyst's duties.
Machine Learning: This part of the profession is probably where the "sexy" comes from. Using regression, classification, and clustering to solve a wide range of problems including computer vision and natural language processing. Sometimes, the people who develop new and better ways of solving problems through machine learning are called Machine Learning Scientists and the people who implement the solutions are called Machine Learning Engineers.
Data Engineering: This part of the field has become so important that Data Engineers are more in demand than Data Scientists. To do data science, we need data and tools. Making these available is what data engineering is about.
Cloud DevOps: More and more, both the data and the tools required to do data science are being made available on the cloud. Navigating a large number of cloud products, managing the scalable infrastructure, and managing the access and security are the duties of the Cloud DevOps Engineers.
Web Development: This part might seem out of place, but if we consider the end-to-end data science projects, then the web is most likely the prototyping or deployment solution. In larger teams, there may be a team of Front-End and Full-Stack Developers.

Sure, these groupings are not clear cut and there are overlaps. At least, these groups give me some way of organisation. I have noted in the brief descriptions, these roles can be carried out by dedicated specialists in the team. But in a small organisation, it could be up to the one-person generalist data scientist to carry out all of these functions.

Whether we need to perform all of these roles or not, it would be helpful to understand a little bit about what other people in the team do. Or perhaps you are looking to switch your career track, say from a data analyst to a data engineer or from a web developer to a machine learning engineer, in which case, you will benefit by knowing something about everything.

Coming back to the tech stack, we can (loosely) group the technologies according to these roles.

EDIT: (Notes on what these are added towards the end of the post)

We can't be experts in everything.

Scanning through job descriptions and MOOCs, we can probably narrow down the very employable stack to something like the following:

Even this is too much to truly master. Even if I have touch points with all of these technologies, I wouldn't need to learn all of them. I would either be working with specialists or only use them to the extent that can be handled by reading through the documentation.

But I think I can remove some "duplicates" or remove some from my personal "core" stack from a learning standpoint.

So, my tech stack is...

After much consideration, my data science "core" stack for the coming years will look something like this:

Even though this is just a silly exercise, I still struggled to come up with the final, and personal, tech stack. I am not sure that this is the ideal minimalist stack. And even if it was, I won't be able to use many of these at my current job. I would probably use this "data science core stack" if I were to embark on a personal project or start my own tech company. There are other technologies that I have learned that I will continue to keep up with (Tidyverse, Spark, and Airflow), additional technologies I will learn a little bit about this year (Vue perhaps) knowing that I probably won't use them. I also recognise that one must choose the right tool for the job and that this list may look different in a couple of years' time.

Nevertheless, I think doing this kind of exercise once in a while is helpful in reassessing the pros and cons of each technology and getting the feel for the overall landscape. I have probably read over a hundred blog posts and a hundred videos about the latest developments and commentaries about the technologies listed in the first diagram and many more. This in itself was a valuable exercise. It also helped me narrow my focus on what I wanted to learn and why I should prioritise them, because realistically, I won't be able to (nor do I need to) learn all of them.

So, what's your tech stack?

Some thoughts...

I've jotted down some thoughts that went through my head as I was reducing my tech stack. These are all personal opinions, but the final tech stack is a personal one. So I think that it's okay.

There is a lot of knowledge that is a co-requisite for some packages like statsmodels, scikit-learn, and Tensorflow. We shouldn't forget that there are whole fields of mathematics such as probability, statistics, linear algebra, vector calculus, econometrics, and machine learning algorithms that form the foundations of data analysis and machine learning. These take up a very big part of the skillset of a data scientist and therefore provides motivation to reduce what would be included in the core tech stack.
Computation is also a big part of data science, and good programming skills are essential: best practices (think the Zen of Python, PEP8, and PEP484 for example), programming paradigms (imperative vs declarative, procedural vs object-oriented vs functional, etc.), data structures and algorithms. It is not enough to write code that works, it is also important to write code that other humans can read. Getting deep into the best practises and latest inclusions in Python, JavaScript, R, and SQL is itself quite a demanding. Venturing into other languages such as Scala, Go, Rust or C may not be a luxury that all of us can afford. Another reason to minimise the tech stack.
Contrary to some popular narrative 80% of a data scientist's job is not data wrangling and cleaning. Half of the job would be meeting various people to understand requirements, communicate insights, and educate the benefits; as well as administrative duties, data governance duties, and professional development. Technical parts are perhaps half the story, and professional Data Scientists should also develop their management, communication, and design thinking skills. Yet another reason to reduce the technology fatigue by minimising the tech stack.
I think that some technologies are similar enough that learning one would provide the transferrable skills required to learn others. For example, knowing Tableau will probably make learning PowerBI much easier. So learn one, and we can pick up the different nuances if the job requires.
Similarly, where two different technologies essentially do the same job but are just different implementations, we can consider "duplicates". For example, PyTorch and Tensorflow are both very good deep learning packages, picking any one of the two would be a good choice.
Some competing technologies are all worth keeping for different reasons. ggplot2 is the defacto visualisation tool for R. I have read some blog posts written in 2019 that still claim that R has better visualisation capabilities compared to Python. I think this is one of the reasons why we should take some time once in a while to update our knowledge about the data science tech landscape. Altair is arguably a better implementation of the grammar of graphics than ggplot2. But Altair uses Vega-lite (built on top of D3) which is very suitable for the web. For "print", I think that seaborn is the best. Within notebook environments, I think that Plotly Express is a very good candidate. At this point, I don't think I could choose between Altair, seaborn, and Plotly Express. All three are declarative and really easy to learn and use. I would probably continue to use all three. I would consider Altair and seaborn to be a part of the pandas ecosystem, and Plotly Express to be a part of the Plotly ecosystem (together with Plotly Dash).
Some technologies are easier to learn than others. For example, React and Angular are powerful front-end frameworks (or library), but may not be the easiest to master. Some say that Vue, another front-end framework, takes the best of both styles and is easier to learn. Given that I am not looking to specialise in web development, I think Vue or even Svelte will suffice.
In fact, some technologies are so easy that they are almost not worth "learning" or need keeping up-to-date that much. For example, HTML, CSS, Excel and Tableau. I think I can put these under the "assumed skills" category.
One could say the same about SQL, but I think there are enough dialects within SQL and No-SQL "languages" built similar to SQL that it is worth keep reading up on. In my diagram, I am including all these peripheral and related things within "SQL".
Speaking of including all related things, much like Tidyverse contains a lot of packages within it, I am including a lot of related packages within the pandas logo in the diagram above: NumPy, SciPy, matplotlib, seaborn, Altair, pandas-profiling, and pyjanitor for example. But I have separated out statsmodels because of the magnitude of co-requisite knowledge required to wield this package.
Some technologies are "closer together". For example, while R may be better for statistics and econometrics, Python's statsmodels have caught up significantly. Since Python is useful for web development as well as machine learning, an argument can be made for using statsmodels over R. This is a hard balance to make. On the one hand, there are economic gains to be made by minimising the number of languages to strive for mastery in. But R still appears to be more sophisticated. And in my experience, learning Tidyverse (in R) helped me become a better pandas (in Python) user. In this post, I am somewhat trying to be more economical, so if I had to pick, I would drop R and focus on Python. I would still happily use R if required for a specific job.
Some technologies are "more native" than others. Spark and Airflow are perhaps more mature and widely used compared with the Dask and Prefect duo. But people on Youtube and tech blogs tell me that Dask and Prefect are more "pythonic" than Spark and Airflow. Similar parallels can be drawn in Javascript land with Svelte versus React. I am not certain that these comparisons can be meaningful nor whether the differences are significant. This is another difficult balance to strike. On the one hand, we want to be more employable by keeping up with the technologies in vogue. On the other hand, concepts such as good data engineering and programming best practices are transferrable, and we prefer to learn what is easier to implement and learn. Both Spark and Airflow are easy enough to use if they have been made available, but I wouldn't particularly enjoy configuring Spark clusters and Airflow on my laptop.
The "cloud" technologies are developing too rapidly, and I am not sure about learning a core stack just in case my future job will require me knowing how to use them. With the release of products like Google Cloud Run, I am not sure whether Kubernetes would be worth learning for the generalist data scientist.

List of tools in image

Languages

Python
SQL
R
Scala
HTML
CSS
JavaScript
TypeScript
C++
Java
Go
Rust

Python + SQL + HTML + CSS + JavaScript will probably take you a very long way.

Data Analysis

Tidyverse ecosystem

dplyr
tidyr
readr
ggplot2
shiny

Plus a whole bunch of packages not included in image. Just use "Tidyverse". R tends to have individual libraries for doing just about any stats. While I cannot list them all, Tidyverse should be central.

Pandas ecosystem

pandas
numpy
scipy
statsmodels
matplotlib
seaborn
altair*
plotly**

The pandas ecosystem is the Python equivalent to Tidyverse of R.

*Altair really belongs in the Vega family (JavaScript-based, and D3.js based), which the altair provides the Python API for.

**Unlike R (ggplot), viz tools in Python does not have a king (yet). I included plotly in the pandas ecosystem in the list, but it is an ecosystem on its own and extends beyond just python. Sorry.

Data Engineering

Spark
Dask
Airflow
Prefect
Kafka
PostgreSQL
MySQL
MongoDB
Cassandra
Elastic
Presto
Redis

Web Development

Svelte
Vue
Angular
React
D3.js
GraphQL (Graphene, Apollo etc.)
ExpressJS
NodeJS
Flask
SQLAlchemy
Heroku
Firebase

Cloud DevOps

AWS
Azure
GCP
Docker
Kubernetes
Jenkins
Ansible

Machine Learning

mlr
Scikit-Learn
Keras (absorbed into TensorFlow)
TensorFlow
PyTorch
OpenCV
spaCy
H2O.ai
OpenAI

Business Intelligence

(Or.. all-in-ones?)

Excel
SAS
PowerBI
Tableau
Knime
Alteryx

DEV Community: Min

Machine Learning Certifications

Prerequisite Knowledge

Books

Courses

Practice Exams

JavaScript for Good: Unfinished

The background

The project

The issues

So what

Learn Web Development for Free

Min ・ Apr 9 '20 ・ 8 min read

Learn Web Development for Free

Motivation and Background

1 Introduction to Programming

1.1 HTML/CSS

1.2 JavaScript

1.3 Python

1.4 SQL

2 Introduction to Web Development

2.1 Concepts

2.2 Tools and Setup

VS Code

JavaScript and npm

Python and pipenv

Git/GitHub

SQLite and PostgreSQL

2.3 Practice

3 Web Development Fundamentals

3.1 Concepts

3.2 Practice

Backend

Frontend

4 Master Web Development

4.1 Learn a Frontend "Framework"

4.2 Learn a Backend "Framework"

4.3 Figure out how to deploy web apps

5 Continue Learning

Youtube

Udemy

Location Data for Cities and Towns

Why might you want location data for cities and towns?

There is no easy way of obtaining such data.

WikiData has a very good source.

Best Free Data Courses for Beginners

Make your decision to learn SQL and Python (maybe even some R)

Free does not have to be 💩

The free courses

Udacity

Programming

Statistics

Data Analysis

Data Wrangling and Machine Learning

Kaggle

DataCamp

Improving your data wrangling game

Improving your pandas game

Practicing machine learning

Conclusion

Bonus: Youtube

Documenting SQL with Markdown and Diagrams

Nobody seems to document SQL codes

Someday that might bite us on our backsides

Let's use markdown in a comment block

There's a tool for that.

Documenting Python Data Science Code with mindoc

Min ・ Jan 18 '20

Documenting Python Data Science Code with mindoc

Write amazingly readable code in python.

And then came the awkward moment

Then came the Aha! moment

We took to the python documentation tools.

Introducing mindoc

Company data using Wikidata

Company data is really useful

Collecting information about companies is hard

Some options for company data

Buy the data

Get data provided by some governments