DEV Community: Sam Gould

Web Development for Data Scientists: Core functionality and a DevSecOps foundation

Sam Gould — Wed, 23 Nov 2022 15:42:10 +0000

In my last post I outlined a number of different approaches for setting up a simple website architecture. I mentioned that websites are typically hosted on Cloud VM/VPS or managed hosting solutions, and recommended some follow-up actions which take a Hello World site to a more robust production level. In this post I'm going to expand on some of these follow-up actions with a focus on bolstering our DevSecOps so that we have a solid foundation from which to build. Some points will be specific to WordPress sites, but most of the DevSecOps applies in general (to Linux servers). Before I get to DevSecOps however, I'm going to quickly cover my overall approach to task management and implementation of some core site elements.

How do I manage my web development tasks?

Personally, I find it important to maintain structured notes and TODO lists for my projects (not just web dev). I am using the following structure which I created to manage my web development tasks:

Deployment
Plugins
Design
Content
Integrations
DevOps
Server management and security

Deployment, Plugins and Design

What is quite nice about this task management structure, besides being essentially MECE and easy to understand, is that at the beginning it can roughly be followed in order. In the first post, we covered Deployment of the Hello World site (in particular I used DigitalOcean's 1-click WordPress launcher to set up a Droplet VM which hosts samgould.net). The next step for me was to install a WordPress theme (which is very easy), essentially a Plugin, and then to configure the Design options. While not the focus of this article, the basic actions I would cover here are (all found under Appearance > Customise in WordPress):

Define a tagline
Upload a 512x512 favicon
Upload a logo (150x150 suggested but you can also use a banner)
Pick a nice colour and font scheme
Define a sitewide header and footer (NB: it's possible to remove the theme copyright text by going into your site html root directory, which is /var/www/your-site/ where $your-site may depend on your Apache virtual host config, and then going to the relevant theme file, which for me is wp-content/themes/theme-name/includes/template-tags.php, and editing the php code; however I'm not 100% sure if this is permitted by copyright and/or the theme license (GPL v3 for me) so I decided to be a good citizen and leave it in place. It is interesting to know that this is where the processing happens under-the-hood, though.)
Create a public nickname for the admin user, to be used in the default WordPress template ("content posted by X"); or it can be removed completely.
Set a site timezone (under Settings > General).

Content

You would probably then want to add some basic content to your site! I added stuff like:

Basic homepage
About Me / Bio page with links to socials
Core functionality - a blog and list of freelance services for me; you might want an e-commerce store or something else

Another important piece of functionality is enabling people to contact you. Of course it is easy to write your email address or link to other socials, but it looks quite professional to be able to receive email to you@yourdomain. I discovered that, while it is easy to receive emails like this (very simple to set up via your domain registrar), sending emails requires an SMTP server. This is commonly achieved via paid or free tier plugins but could also be self-hosted.

Integrations

The only work I have done here is beginning to automate the way I share my content across different socials. I have started posting to DEV.to and am using a script I wrote which reformats WordPress markup so it is ready to post. This is somewhat tangential and should not be considered a necessary step to setting up your website.

DevOps

In the context of this article, by DevOps what I really mean is: "how do I manage my code and other site assets in a way which is flexible for development and resilient to failure?". In particular - version control and backups. I would also echo DigitalOcean's best practice advice and first set up a non-root user for any work you do on the server. In the code below, we will call this user _non_root_user_.

Version control

At this stage, all you really need is a Git repo to hold relevant files from your site codebase. For my WordPress site, this is simply custom theme and plugin code (and at this stage I don't even have any custom plugins). As mentioned in the Design section above, the site root on your server may depend on your HTML server config; mine depends on Apache virtual hosts and can be double checked by checking which sites are within /etc/apache2/sites-enabled. As a data scientist you (should) probably know how to set up a Git repo, but there were a few gotchas which I discovered along the way:

cd /var/www/your-site/
git init
gotcha #1: sudo chown -R non_root_user /var/www/your-site/ (explanation here)
gotcha #2: if you try to interact with the filesystem from outside the server (e.g. upload media files via web UI) then you will now get errors because the default www-data worker no longer owns the files. After you finish working with the git repo you can reset it with chown -R www-data /var/www/your-site/. I'm not sure if there is a better way to handle this…
gotcha #3: nano .gitignore (this is a gotcha in the sense that it is crucially important you define (and save!) the correct gitignore. Do not commit passwords/secrets to your repo! In particular wp-config.php must be excluded. I used Sal Ferrarello's "surgical" .gitignore as my starting point. Make sure you populate and save the file.)
git add .
git commit -m "First commit"
Create a repo, e.g. on github, and do git remote add origin https://github.com/your-github-username/your-repo.git
git branch -M main
git push origin main
gotcha #4: although the command output says it requires a password, it actually needs a Personal Access Token.

Congrats, you have now synced your mutable (WordPress) site files to a Git repo. But this is not every component of the site - we also need to create a backup of our content and config. For that we will use phpMyAdmin (although there are multiple ways to do this).

Database backup

What is phpMyAdmin?

As WordPress puts it:

"An administrator’s tool of sorts, phpMyAdmin is a PHP script meant for giving users the ability to interact with their MySQL databases. WordPress stores all of its information in the MySQL database and interacts with the database to generate information within your WordPress site. A “raw” view of the data, tables and fields stored in the MySQL database is accessible through phpMyAdmin."

How to use phpMyAdmin to backup a WordPress site

Step 1: install phpMyAdmin: sudo apt install phpmyadmin

With an Apache server, you may need to run these additional commands: sudo ln -s /etc/phpmyadmin/apache.conf /etc/apache2/conf-available/phpmyadmin.conf && sudo a2enconf phpmyadmin.conf && sudo service apache2 reload. I only needed to run the apache2 reload for some reason.
With an Nginx server, you may need to run these additional commands: sudo ln -s /user/share/phpmyadmin /var/www/my-site/phpmyadmin && nginx -s reload

phpMyAdmin can then be accessed via https://yourdomain/phpmyadmin (NB: see the 'Server management and security' section below for instructions on enabling https, or get it automatically from the DigitalOcean 1-click installer). You can log in with the MySQL root user created during the DigitalOcean 1-click deployment when our LAMP stack was configured (see the first post), or create a new database user with appropriate permissions.

Step 2: run a simple backup using phpMyAdmin. This is very simple to do - instructions can be found here.

Congrats, you have now backed up pretty much everything you need to be able to restore your (WordPress) site in case something breaks in production!

What about CI/CD?

A pattern for deploying different codebase versions is blue/green deployment. This is possible to do, for example using Apache virtual hosts, but in my opinion is overkill at this stage.

Server management and security

One of the first, most important steps to securing your server is implementing a firewall to block unwanted traffic. The simplest configuration tool is the Uncomplicated Firewall (UFW) and the suggested configuration is to allow only SSH (port 22, rate limited), HTTP (port 80), and HTTPS (port 443) access.

With a firewall (i.e. with appropriately exposed ports), you can further secure your website by enabling encrypted connections over HTTPS/SSL. This protects user privacy, data and has additional benefits like preferential treatment by search engines. The simplest way to implement this protection is by using the Certbot tool, which handles the certification and renewal/reminder processes.

We can block further unwanted traffic by implementing DDoS prevention. The DigitalOcean 1-click installer for WordPress uses two layers, namely fail2ban (should be implemented on all architectures) and disabling XML-RPC (WordPress-specific).

Finally, we want to make sure that our server stays updated to mitigate against any potential exploits. When using a Cloud VM/VPS, your Cloud provider will most likely push messages into the SSH console so that upon login you can see if you need to run updates (udo apt update && sudo apt upgrade). Since vulnerabilities should be patched in a timely manner, it is sometimes recommended to run updates in an automated and unattended manner (i.e. as soon as they are available).

Conclusion

We have completed our first batch of activities from each section of my task management framework, taking us from a simple Hello World to a site which is version controlled, backed-up and secured from attackers. Congratulations!

Disclaimer: I am not a security professional and you should DYOR. You are responsible for the security of your server/website.

This content was originally posted on samgould.net

Introduction to Web Development for Data Scientists

Sam Gould — Thu, 17 Nov 2022 14:20:44 +0000

You are a data scientist used to helping organisations answer their strategic questions using data and technology. When you are in engineering mode, you spend your time:

building data pipelines in Python and SQL
modelling and analysing data in Python or R
using REST APIs and Linux containers for lightweight app deployments

You now want to build a website. What are the right tools and approaches for getting started with web development?

This is the situation in which I recently found myself while building samgould.net. There are lots of reasons to build a website:

Startups/SaaS businesses need websites and web apps
It gives you a self-owned platform for creativity and expression
Web dev is a new topic to learn

If you are working as a data scientist then you already have a lot of the foundational technical skillset required for web development. But the modern proliferation of languages, frameworks and platforms can make getting started seem daunting. Here's how I did it.

Let's tackle our Hello World foe

How is a website structured?

My experience in deploying machine learning MVPs gave me a rough mental model of a website's internals as a starting point. We will need a backend handling application/business logic, a frontend providing pages for a user to view, and some kind of API connecting the two. Incoming internet requests are pointed at our server host via a DNS lookup (mapping URL to IP). We will probably also need some layer handling incoming traffic (API/routing and traffic load balancing).

A rough mental model of a website architecture

What are the main types of website?

We can refine this picture by considering the use case. In a simple blog, the frontend user experience is the same for everyone: I request some content, like an article, and the website serves it to me; but in a more complex application, the experience might depend on the user and some data exchange with the server. The key distinction here is that of static vs. dynamic websites. A dynamic site typically uses static templates which it dynamically populates based on client requests. An idea of our desired website functionality in these terms will be important for technology selection and architecture.

What are the main web dev technologies?

There are various ways to research this question. I looked at posts on popular developer sites StackOverflow, IndieHackers, DEV.to, Reddit:

A minimal static site is, at its core, HTML and CSS, with bits of JavaScript sprinkled in for dynamic functionality. However, as a Python-using data scientist, I am used to high-level frameworks doing the heavy lifting for me, and I don't fancy the sound of learning three new languages. But which one to pick? There is clearly a massive array of web dev frameworks out there. In any form of programming, there are many ways to skin a cat. The most important thing is to pick something which works well enough for us to build out our use case. With a Python background, the frameworks which jump out to me are Django and Flask.

Python web frameworks

Django

Django is "a high-level Python web framework that encourages rapid development and clean, pragmatic design". It is appealing as a "batteries-included" framework which handles a lot of boilerplate and functionality which is important but unfamiliar to a developer from a non-web background. It is considered "somewhat opinionated" and utilises a 'model-view-template' design pattern. A model is a piece of backend logic which is invoked by a view, which is an HTTP request handler. Views use templates to format the data for display to the client.

The official docs provide the best place to begin with Django development (project setup etc.) but there are alternative tutorials too. The docs explain the concept of WSGI (the Web Server Gateway Interface - the standard which lets your Python code communicate with web requests) - Django's startproject command sets up a minimal default WSGI configuration.

Looking at our diagram, the other missing puzzle piece is hosting. Django applications are typically wrapped in lightweight web server frameworks (Nginx or Apache) to expose ports and allow connections in (i.e. HTTP requests to the frontend), and run on Cloud VMs, for example, in order to have a public IP address. It is possible, but not recommended, to run on your own hardware as a server.

With this new knowledge of Django, we can update our mental model:

High level architecture of a website using Django

If you do choose to develop with Django, then consider exploring popular frameworks and toolkits. To stay on the pulse of the Django ecosystem, see Filip Němeček's DjangoFeeds.

Flask

Flask is an alternative Python library, often compared with Django: "a lightweight WSGI web application framework. It is designed to make getting started quick and easy, with the ability to scale up to complex applications". As can be seen from the source code, it can spin up a Hello World API routing example in just a few lines of extremely simple, Pythonic code. I personally wanted to explore heavier features such as front-end admin panels without spending too much time on a learning curve, so did not delve deeper into Flask - one to revisit in future.

Alternative microframeworks include FastAPI and Starlite.

Static Site Generators

As we have seen, developing a website in Python is an extremely viable option. But suppose that you have a laser focus on a simple use case: a static blog site. In this scenario, a Static Site Generator (SSG) might be the more efficient option.

Let's slightly redefine our mental model. A common pattern for developing a site which serves static content involves less two-way communication than in Django's MVT: content is created when the site is developed, not when the user requests it, so why not generate all of our site's HTML at build time too? We can push our content (typically a collection of Markdown files) into a CI/CD platform (e.g. GitHub), run an SSG to inject it into templates and spit out a bunch of static HTML pages, and then serve these to the user. This typically makes for an extremely fast website with minimal development overhead.

Hugo

A nice SSG is Hugo, which can be used to quickly set up a blog. It is trivial to then host these static files on a CDN hosting service like Netlify or GitHub Pages, although deploying through an Nginx server is also very doable.

High level architecture of a website using Hugo

Hugo is not the only SSG, there are many others.

Content Management Systems

With the previous two methods, we can successfully host our content on the internet. So what happens when our scope expands and we need to add new site functionalities, like collaborative editing of front-end content?

A Content Management System (CMS) can be used to create and edit a site's content. Like with an SSG, this content is then injected into HTML templates to be served on the front-end. The difference, architecturally speaking, is that the CMS can be dynamically coupled with the front-end. From a development perspective, popular CMSes are highly-featured with highly mature plugin ecosystems, meaning it is quicker to implement advanced site functionality such as e-commerce integration and user access roles.

Note: Jamstack is the name of a web development architecture based on core principles around decoupling the front end web experience from the data and business layers, with a focus on delivery as static sites. Both SSGs and CMSes can be used in this way: a CMS can be used in 'headless' mode - i.e. backend only, requiring a separate presentation layer to handle design, site structure and templates. This CMS usage mode is technically more aligned to Jamstack, but the distinction appears to make little practical difference during early stage web dev.

WordPress (the open source WordPress.org, not the managed service WordPress.com) is the most popular CMS in use today. It is straightforward to deploy a WordPress instance into a VM running Nginx or Apache.

High level architecture of a website running WordPress

From my perspective, this is a great option. It gives me control over my server - I have more fine-grained control because I am paying for IaaS (a VM server) rather than PaaS (website hosting). I am (still) using open source tools which are, on the whole, very approachable from a data science background. Personally, WordPress itself is still a bit of a black box of PHP code, which is why I stopped representing the distinction between back- and front-ends in the diagram, but the trade-off is that I have immediate access to plugins and integrations.

This architecture is one implementation of what is known as the LAMP/LEMP stack: Linux (my Ubuntu VM), Apache/Nginx (pronounced Engine-X, hence "E"), MySQL, PHP. For reference, a typical JS web app stack would be something like MEAN or MERN (but there are lots of other possibilities).

Conclusion

There are multiple ways to build a website, and the choice of one should balance use case requirements against ease of development. Each approach conforms to the basic server hosting mental model coming from a Python/ML/DS background, with key distinctions depending on website archetype (static/dynamic web page/app) - but essentially we are putting a bunch of HTML files on a server and exposing them to the internet via a webserver service. Django lets Python developers focus on implementing business logic, Hugo can easily spin up static blogs, and WordPress provides instant access to mature plugins. There is no single right way to do web development - but it is important to remember the end goal and not fixate on the technology choice.

I have skimmed over some details in this article, notably DNS setup (which is really quite straightforward) and best practices for server maintenance, which include the following recommendations. You can also refer to DigitalOcean's 1-click WordPress installer to see what additional configurations it performs on its Apache server:

For a live production site, you should go on to explore:

Security - fail2ban and DDoS prevention (DigitalOcean disables XML-RPC)
Backups - for example via git or using the Cloud hosting provider's services
Staging and deployment - commonly using the blue/green pattern via Apache's virtual hosts file
SEO
Sustainable content development - e.g. using content frameworks

but these are topics for future posts! I hope this was helpful for anyone looking to develop their site.

This content was originally posted on samgould.net