DEV Community: Vasco Abelha

How to still use Crawlers in Client-Side Websites

Vasco Abelha — Wed, 12 Aug 2020 11:57:32 +0000

Originally this was published on my blog. You can find the publication here!

If you wanna discuss anything feel free to reach me on Twitter.

Introduction

In this post, I will describe a solution that I built for an already existing React Client-Side platform, in which people wanted to be able to share specific content on their feeds.

This publication is useful for developers:

have an already built a Client-Side Website (doesn't need to be solely React)
want to understand how we can interact with different crawlers.

Technologies used:

VPS where the project was hosted
Nginx;
ExpressJS (It doesn't matter what you are using).
ReactJS
Facebook SDK - OpenGraph

Contextualization

Whenever you share a link to a website in Facebook, Twitter, or any other social platform, they spawn a crawler that will scrape your website in order to look for meta tags that can help them understand what they are looking at and how they can share it - App, Card, Summary, Large Card, etcetera.

One of the biggest problems in a React Client-Side website is that everything is rendered through JavaScript. If you use a Browser or a Crawler that doesn't process JS, you will just be presented with a blank page - You need to enable JavaScript to run this app. This applies to Facebook or Twitter Crawlers.

In the end, if you share an URL from your website on one of these social platforms, you won't get any type of card or information from your website.

Note: You can use https://cards-dev.twitter.com/validator to verify and test yourself.

To the Left we Have a React Client Side Website. To the Right we have a static website.

In both Websites, I have React-Helmet (which allows modifications to your document head), yet the left side still shows no meta-tags fetched by the crawlers due to requiring JavaScript to render.

Show what the Crawlers want to see

If we are hosting the website on a typical Virtual Private Server, then there is a good chance that we are using a web server like apache, nginx or lighttpd to process the incoming HTTP requests.
Thus a web server like Nginx is the perfect place to "trick" him and proxy him into a renderer HTML with the information that we want the crawler to see.
For this we need:

To know which requests come from the crawlers;
a service that renders Dynamic HTML Content;
Update NGINX to link crawlers to the new service.

Crawlers Identification

After researching Facebook and Twitter Documentation we can identify the crawlers by the following user-agent strings:

facebookexternalhit/1.1 (Facebook)
Twitterbot(Twitter)

Service to render Dynamic HTML

You have other types of solutions. You can pretty much use anything that renders an HTML webpage.

In this case, I had an already established set of services available through expressjs, so I stuck with it and created one endpoint that would take params (in this case a news publication identifier) and return an HTML page with every kind of head and meta tags that I wanted to be scraped by the crawlers.

Note: The URL must be equal to the one where I view the news publication.

Example of the service:

//(routes/social.js -> socialRoutes)
...
router.get("/news/:id", async (req, res) => {

    const { id } = req.params;
    const {news} = await getNews(id);
    res.set("Content-Type", "text/html");
  res.send(`<!DOCTYPE html>
  <html>
        <head>
            <link rel="canonical" href="${news.url}" />
            <meta property="og:title" content="${news.title}>" />
            <meta property="og:description" content="${news.description}" />
            <meta property="og:image" content="${news.cover_image}" />
            <meta property="og:url" content="${news.url}" />
            <meta name="twitter:title" content="${news.title}>" />
            <meta name="twitter:description" content="${news.description}" />
            <meta name="twitter:image" content="${news.cover_image}" />
            <meta name="twitter:url" content="${news.url}" />
            <meta name="twitter:card" content="summary" />
        </head>
  </html>
`);
});

//server.js
...
app.use("/social", socialRoutes);
...
app.listen(3500, () => {
  console.log('started at localhost:3500');
});

Update NGINX and Send Crawlers to our Service

With knowing the user-agent strings of the crawlers and having already defined our service to generate HTML pages free of javascript.
We can now "trick" the crawlers with the help of NGINX and send them to our services instead of the real webpage.
Usually, if you are using a react app under Nginx, your default.conf file will be generally similar to this:

server{
    root /var/www/html;

    # Add index.php to the list if you are using PHP
    index index.html index.htm index.nginx-debian.html;

    server_name www.example.com example.com;

    location / {
        try_files $uri /index.html; 
    }
}

Nevertheless, this isn't enough, because the crawlers will still go to our files located in root and only see blank pages due to javascript rendering.

Therefore we need to add a prior condition to verify the user-agent before sending them to our project folder.

server{
    root /var/www/html;

    # Add index.php to the list if you are using PHP
    index index.html index.htm index.nginx-debian.html;

    server_name www.example.com example.com;

    location / {
        # Here we proxy the request to our api if user-agent matches any of these regular expressions
        if ($http_user_agent ~ facebookexternalhit|Twittterbot) {
            proxy_pass http://localhost:3500/social$uri$args;
        }
        try_files $uri /index.html; 
    }
}

Conclusion

Every time we have a new request that matches the user-agents of Facebook and Twitter, we will proxy it to our service for HTML rendering. Thus, in turn, allowing the crawlers to process our "not-so-real" webpage as the real one and fetch the meta-tags needed to share our website.

As long as you have some kind of middleware that can act as a reverse proxy, then you can still allow client-side web applications to be scraped by crawlers that don't execute javascript.

Nevertheless, if possible you should take a look at Static Side Generators or Server-Side Rendering Frameworks.

This publication is only useful to shed some light on how you can interact with crawlers and to possibly guide or help someone in anything similar that they are working on.

How to easily add HTTPS to your Server for free!

Vasco Abelha — Tue, 04 Aug 2020 15:53:09 +0000

Originally this was published on my blog. You can find the publication here! If you wanna discuss anything feel free to reach me on my Twitter

Long gone are the days where if you wanted to have https, you had to pay out at least more 10$ for a certificate.

Nowadays, if for any reason you are hosting a website or an API in your Virtual Private Server on AWS, DigitalOcean, and any other VPS Hosting; most likely you got a Linux distribution with a basic configuration and no SSL. Luckily, now we have Let's Encrypt Me! 🥳

In this publication, I will:

succinctly explain what is SSL, TLS.
Explain how you can create your digital cert and install it on the webserver that you are most likely using - Nginx or Apache.

To try to help the most people I can, this guide will only be focused on Ubuntu. If you need any help feel free to reach me here or through Twitter! Same reason for Lighttpd ❤️

A refresher on terminology:

SSL - Secure Sockets Layer is a cryptographic protocol responsible for encrypting data (secure connections) between a client and a server.
TLS - Transport Layer Security is also a cryptographic protocol responsible for encrypting data and securing connections between endpoints.

The main difference is that TLS is an updated version of SSL with stronger encryption algorithms like RSA, DSA, etc.

So whenever you are talking or see something saying anything regarding SSL, they are probably referring to TLS.

Let's Encrypt Me

Let's Encrypt me is a non-profitable certificate authority (CA) that easily provides SSL/TLS certificates for free and public use. It is run by the Internet Security Research Group (ISRG).

In layman terms, through ACME protocol, they can easily verify that you own your domain and automatically issue digital certs that you can use to enable https for your websites or applications.

Certbot is the software used to validate your domain and request the digital certificates to be installed on your server.

Add Certbot repository to Ubuntu.

sudo add-apt-repository ppa:certbot/certbot

Depending on the software that you are using, it might be preferable to check for working repositories, because default repositories provided by your Distro could be outdated.

Install Certbot Package

//If you have apache
sudo apt install python-certbot-apache 

//If you have nginx
sudo apt install python-certbot-nginx

Execute Cerbot

//If you have apache
sudo certbot --apache -d vascoabelha.com -d www.vascoabelha.com

//If you have nginx
sudo certbot --nginx -d vascoabelha.com -d www.vascoabelha.com

Instead of vascoabelha.com, here you need to make reference to the name that you defined in your web server configuration, either on:

/etc/apache2/sites-available/[domain].conf (ServerName)
/etc/nginx/sites-available/domain

If it is the first time you are running on your host, you will need to answer some questions. (nothing to worry about)

In the end, you will be presented with:

Please choose whether or not to redirect HTTP traffic to HTTPS, removing HTTP access.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: No redirect - Make no further changes to the webserver configuration.
2: Redirect - Make all requests redirect to secure HTTPS access. Choose this for
new sites, or if you're confident your site works on HTTPS. You can undo this
change by editing your web server's configuration.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate number [1-2] then [enter] (press 'c' to cancel):

I tend to always select number 2. Certbot will then change your Nginx/Apache configuration to add the needed redirects (301) to force your connections to https.

By now, you will have your certs installed and running. If you refresh your website, you will see the shiny 🔒 followed up by https 🥳!!

👏Everything is set and running👏

Last but not least, these certificates are valid for 90 days! Yet you don't need to worry, Certbot runs twice a day and will make sure your certificate is valid and issues another one if needed (less than 30 days of validity).

If you would like to test if the certification refresh process is working fine, run:

sudo certbot renew --dry-run

If in step 3, you provided a real email, you will also be notified in case the renewal process fails!

See you around!

Simplify the development of Portfolio Website by planning your Stack and Design

Vasco Abelha — Wed, 22 Jul 2020 10:40:44 +0000

Hey everyone! 👋

This is my first post ever and I know that this is just another drop in the ocean of guides in "how to build your own webpage/portfolio".
Nevertheless, I hope that I can shed a different light and approach in the planning of your portfolio. This is also for me a first step in giving back to the dev community, the help and guidance I have received so far.

Context

Over the last month, I have built my personal website and tried to keep it simple but aesthetically pleasing.

During the development, I documented every decisions and steps I made in order to help me solidify some concepts. Since there are already many and good guides in how to build portfolios, I will abstain from delving in coding breakdown/guides and will focus on a more generic perspective. Besides I don't think it would be valuable for the reader to be imposed with a design and code that he is not comfortable with. Our personal/portfolios websites should be a representation of ourselves, preferences and likes.

In this post, I will describe what I used for building my webpage (JAMStack) and reasons for choosing this architecture. There will also be a short section regarding the design of your website and some tips to help you stand out!

So why JAMStack?

My decision with going with JAMStack can be quickly summarized in wanting:

A Fast, Cheaper, Secured and Simpler way of developing and maintaining a website.

JAMStack is a term used to describe a new architecture for building websites and applications where

JAM → J = JavaScript | A = API | M = Markup

JavaScript

JavaScript role is building dynamic content on your website, fetching data, etc. This does not mean that everything in a JAMStack is built with Javascript.
In fact, you have different SSG (Static Site Generators) built on different languages. Eg: Hugo in Golang, Jekyll in Ruby, Pelican in Python, etc.

For a list of available SSG you can go to:

https://www.staticgen.com/

API

Where usually we have Server-side functions to access/consume/query data, in JAMStack these API/services are accessed and consumed as you want. JAMStack is agnostic to API. We can easily use 3rd party services as well as building our own specific APIs.

Markup

This is where the fun is! Everything that you see in your site is served through HTML Static Files created during the building phase. These files have already the structure and information needed to be presented when accessing the page.

Unlike dynamic website based on templates, you don't need to wait for the server to query the api/database to present and render the webpage, since the website and its respective pages were prebuilt at the time of the build. The frontend is decoupled from the backend and thus the website can "live on its own" - serverless.

In the end, we have:

A more secure website since it does not need to interact with a backend;
Faster and more responsive website since everything is built on static html files, we don't need to query or wait for anyone to render the page;
Scalable due to the CDN (Content Delivery Network) for the website and the strict separation/modularity, between frontend/website and backend;
Cheaper because you have companies that can even host your static files for free!

With this in mind, and after checking the available static site generators, I ended up choosing the GatsbyJS due to:

better SEO features;
documentation - this was a decisive factor. Gatsby.JS documentation is clear and more than sufficient to guide you in every step of the way, from preparing, developing, to building and deploying a website;
larger dev community, in which can be seen as: "I will most likely fail somewhere, but at least I have a higher chance of finding someone that had the same problem" 😂.

Setting aside the technological decisions, we can now focus on a more subjective and personal based decision - the design.

Design

Choosing between templates or designing yourself.

Like many of you, I am not a designer or specialised in any kind of UI/UX. Everything that I know is from self-learning and probably not much 😬. However, we can't be hostage of our own shortcomings, so what can we do to fill the gaps?

Well we can either use an already built template/theme available 😞 or we can do some research on what does a good design entails and view some examples as inspiration 🤩.

Here is a list of some good websites/resources that I tend to use when I am in need of inspiration:

https://dribbble.com/
https://webframe.xyz/
https://land-book.com/
https://www.behance.net/
https://www.designnotes.co/
https://edu.gcfglobal.org/en/beginning-graphic-design/ (I believe this is a good tutorial to help you grasp some of the basic concepts of graphic design)

Now if you ask me if I would use a theme/template or build one from scratch, I would undoubtedly choose the latter, due to the fact that I believe in the importance of showing ourselves, our uniqueness through what we design and build. Even though a prebuilt template would most likely save me time, I tend to see everything that I do as an opportunity to invest in myself and learn new things and comparing both options I probably wouldn't learn a lot with using a template.

Things to take in consideration during the design and development.

Before coding, I would advise you to create some Mockups of your website. This will not only force you to organise your ideas on what and how do you want to show, but also help delineate a plan, a series of steps needed to effectively assist and guide you during the coding sessions.
For this you can use tools like Figma, Adobe XD, Sketch, etc.

Here are some tips to help you start designing your portfolio/personal websites:

Fonts play an important role. It sets the tone of your design. If you are stuck in choosing a pair of fonts with good contrast for your website, you could use a tool like https://fonts.google.com/ or https://fontjoy.com/ to help you decide.
In terms of colours, you will probably be fine with 2-3 colours that work well together. If you are interested in learning more, you can search for colour theory. There are also websites that help you building pallets of colours like : http://colormind.io/
On top of linking your social media, a Contact Form is also an important tool to build bridges between you and your visitors. Nowadays, depending on the stack that you are using, it is super simple to build and deploy a reasonable Contact Form - E.g. Netlify Forms and Formspree.
In addition to publishing your work in the website, you should also have a hyperlink to a detailed CV. This may speed and ease the process of recruiters or agencies that are searching for candidates, and if they end up in your website, then you are probably a suitable candidate for what they are searching 🤔
This isn't related to design, but if you have the possibility, you should buy a domain. Having a custom domain name will improve your credibility and professionalism in the eyes of the visitors.

Conclusion

For this post, I have covered everything that I wanted:

JAMStack;
Decisions that made me choose a JAMStack Architecture and GatsbyJS;
Resources to help you start designing your website;
What approach to have when building your portfolio in order to stand out, save time and reduce indecisiveness → No Templates, Mockups First and Code later;
General tips to take in consideration while designing and building your portfolio.

If there is interest, I can follow-up this post, with one more focused on a review of the existing CSS Frameworks and what I ended up choosing, as well as SEO explanation and guide to help boost the visibility of your webpage.

If you wanna chat or discussing anything feel free to reach me at Twitter !

Take care 👋 !