DEV Community: Mark Witt

How to Read Scientific Papers from arxiv.org on Kobo E-Readers

Mark Witt — Fri, 11 Aug 2023 16:49:56 +0000

Reading scientific papers on arxiv.org directly from your Kobo E-Reader without needing a USB connection sounds like a dream, right? This guide will walk you through the process, step-by-step.

The Challenge with PDFs

Many scientific papers on arxiv.org are in PDF format. Directly saving these PDFs to Pocket means they won't appear on your Kobo E-Reader at all. Thus, to ensure the content is accessible and optimized for reading, we need a different approach.

Step-by-step Guide:

1. Set Up a Pocket Account on Your Kobo E-Reader

Turn on your Kobo E-Reader.

Tap the More option, usually represented by three dots (...).
Navigate to Settings.
Scroll and find the Accounts section.
Choose Pocket, then Sign in or create an account.
Follow the prompts to set up an account. Remember, the Firefox login option won't work here.

2. Access Your Pocket Account on a Computer or Mobile Device

Go to Pocket’s website.
Use the credentials you set up on your Kobo to log in.

3. Install the Pocket Browser Extension

Head to your browser's extensions or add-ons marketplace.
Search for and install the Pocket extension.
Once installed, you'll gain the ability to add pages to Pocket via the browser's context menu (right-click).

4. Make the arxiv PDF Reader-Friendly

Go to the arxiv page of your desired paper.
In the URL, change the "x" in "arxiv" to "5".

For example:

Original URL: https://arxiv.org/abs/1706.03762

Modified URL: https://ar5iv.org/abs/1706.03762
This change converts the PDF into a responsive webpage, optimized for e-readers.

5. Store the Converted Page in Pocket

Right-click on the page.
Select the Save to Pocket option from the context menu.

6. Access and Read the Paper on Your Kobo E-Reader

On your Kobo E-Reader, from the home screen, tap the More tab on the right.
Tap My Titles (next to the Pocket logo. You should now see a list of all articles you've saved, including the recently added paper.
Tap on the desired paper to read it in a format optimized for the Kobo.

Conclusion

By utilizing ar5iv's rendering tool and the Pocket extension, you can enjoy a seamless experience reading scientific papers from arxiv.org on your Kobo E-Reader. This method bypasses the need for USB transfers, making the entire process wireless and hassle-free.

Automating email verification for online accounts using JavaScript

Mark Witt — Mon, 29 Nov 2021 20:09:35 +0000

What this is good for

To automate the creation of online accounts, we need to perform the same http requests that a user does when signing up programmatically. If it is a website, you can use the DevTools to inspect the network traffic and look it up. If it is an app, you can use an emulator and a tool like mitmproxy to monitor http requests.
Then you can write a quite simple script which creates hundreds of user accounts in a matter of seconds.

put a gentle timeout in between http requests so your script will not be considered as a Denial of Service attack

Most online services require users to have an account linked to a valid email address. To verify that the user really has access to the provided email address, companies use an email verification system in form of a special link sent via email which contains some id and has to be clicked in order to verify.

I am going to show you how to automate that with some simple JavaScript.

Domain and Mandrill Setup

First, We need our own domain to create as many email addresses as we want.
I am using NameCheap in this guide but you can use whatever you are comfortable with.

if you are a student, you can use GitHub student developer pack to get a free .me domain

We want to use MailChimp/Mandrill's Inbound API which has a good free tier to POST any incoming email to a Webhook we will setup later. Again, there's other services such as SendGrid which do the same job, so feel free to use anything you want to.

Follow the instructions to verify an Email Domain at Mailchimp!

We will have to setup some DNS records.

Navigate to https://mandrillapp.com/inbound, add your domain and click "View Setup Instructions". You will need to add 2 DNS records of type "MX" which will set Mandrill as the Mailserver for your domain.

Click "Test DNS Settings"." If it works, proceed:

Let's start coding

Create a new folder and call it what you want
cd into it and npm init (assuming you have node.js installed

We will need 3 dependencies and 1 optional devDependency:

express to setup a server that listens for incoming http requests
cheerio to parse the HTML structure of an incoming email and filter out the link we need to click to verify our email address by an attribute (in this case the text Content of the link but it could be any html attribute)
puppeteer to access the link href inside an automated browser environment (these verification systems use redirects and JavaScript so sending a simple GET request will not do in most cases)
nodemon: this devDependency automatically reruns our app when a file in the project changes. Run:

  npm install cheerio express puppeteer
  npm i -D nodemon

Let's make an app.js file in our project directory and create a simple express app:

   const express = require("express");
   const app = express();
   const cheerio = require("cheerio");
   const puppeteer = require("puppeteer");
   app.use(express.urlencoded());
   /* this middleware enables us to access the http body
   (containing our emails) coming from Mandrill */
   app.get("/", (req, res) => res.sendStatus(200));
   /*adding a get route shows to Mandrill that the url "exists"
   by sending an "OK" status code. */
   app.post("/", (req, res) => {
     console.log(req.body);
     // let us just console.log the body for now...
     req.sendStatus(200);
   });
   app.listen(process.env.PORT || 3000);
   /*for local development, our server will run on port 3000
   When deployed, the PORT environment will be created by
   Heroku */

In package.json, add two npm scripts:

   "scripts": {
       "start": "node app.js",
       "dev": "nodemon start"
     },

Run npm run dev to start a local server. Notice that it reruns everytime you change something and save.

build a tunnel using ngrok

ngrok is a super cool free utility that we can use to tunnel our localhost server to the world wide web.

Download ngrok and unzip it into your project directory

add a .gitignore file:

ngrok
#on windows: ngrok.exe
node_modules

run ngrok http 3000
copy the full .ngrok.io address from your command line (keep in mind that it will change if you restart ngrok)
Go to Mandrill Inbound ->

Select your Domain -> Routes and add a new Route.

Enter an asterik (*) in the first field to forward all email addresses to your webhook. Paste your ngrok url in the second field.
Save the webhook, select it in the dashboard and click "send test". If everything is working, you will get something logged to your console.
Use JSON.parse(unescape(req.body.mandrill_events) to get a valid object that you can explore. You can iterate over emails using forEach.

app.post("/", (req, res) => {
  console.log(req.body);
  const mandrillEvents = JSON.parse(unescape(req.body.mandrill_events));

  mandrillEvents.forEach((mandrillEvent) => {
    const html = mandrillEvent.msg.html;
    parse(html); //implemented in next step
  });
  res.sendStatus(200);
});

Extracting the desired url using cheerio

We now have the html content of our email as a string.
Let us use cheerio to parse it and extract our link:

const LINK_TEXT = "verify email address"; //adjust this
function parse(html) {
  const $ = cheerio.load(html, {
    withDomLvl1: true,
    normalizeWhitespace: true,
    decodeEntities: true,
  });
  /* creates a traversable Document tree from your html string
      Now, let us iterate over every anchor tag and see
      if it is the link we are looking for */
  $("a").each((i, link) => {
    const href = link.attribs["href"];
    if (link.childNodes[0].data) {
      if (link.childNodes[0].data.includes(LINK_TEXT)) clickLink(href); //will be implemented in the next step
    }
  });
}

Notice that the code you need may differ from mine depending on your email html structure. Analyze it by saving the html on your file system and opening it in your web browser. The cheerio library has a very similar syntax to jQuery.

"Clicking the link" using puppeteer

Puppeteer is a library which allows you to run an automated Chromium instance.

Let us create a clickLink function to open the url provided url.

async function clickLink(href) {
  const browser = await puppeteer.launch({ headless: false });
  /* setting "headless" to false enables us 
  to actually see what is going on behind the scenes*/
  const page = await browser.newPage();
  await page.goto(href);
  console.log("Puppeteer is at " + href);
}

Deployment using Heroku

Create a Heroku account and sign it
Download and install the Heroku CLI
Run heroku login and follow the instructions
cd into the project directory and run git init && heroku create
For technical reasons which I am not the right person to explain, we need to install the node buildpack and the puppeteer buildpack for Heroku:
heroku buildpacks:add jontewks/puppeteer && heroku buildpacks:add heroku/nodejs
The puppeteer buildpack requires us to run puppeteer with the argument "--no-sandbox". At the same time, we want it to run in "headless" mode on the server (without gui).

Let's create two constants devOptions and prodOptions:

const devOptions = {
  headless: false,
};
const prodOptions = {
  args: ["--no-sandbox"],
  //"headless" defaults to tru
};

Inside the heroku dyno, the environment variable NODE_ENV is setto "production". We can use it to run puppeteer with differentoptions depending on if we are running it locally or inside Heroku.
Change the first line inside clickLink:

const browser = await puppeteer.launch(
  process.env.NODE_ENV === "production" ? prodOptions : devOptions
);

We're ready to rumble. Open your terminal and run

  git add .
  git commit -am "whatever message you want to put here i don't care"
  git push heroku master

Type heroku info -s and copy the Web URL from your terminal. Paste it as the webhook URL inside Mandrill Inbound Dashboard. Now, everything should work as expected.

If it doesn't work or you have anything else you want to tell me, feel free to hit me up anytime on Twitter or via E-Mail.

This is my first blog post and I am always grateful to hear any feedback or suggestions from you guys 🤠

Creating a custom VSCode Terminal Profile for using Rosetta on an M1 Mac (Apple Silicon)

Mark Witt — Mon, 29 Nov 2021 18:29:45 +0000

Why do this?

Many programs and packages still do not support the ARM architecture used on the new Apple M1 chip.
If you are a developer dealing with older codebases, you may have already had issues installing npm dependencies or running specific tools.

Luckily, Apple provides us with the Rosetta emulation layer.
It enables us to emulate the x86 architecture to run all the programs made for older Macs, with a little performance decrease when compared to apps optimized for M1.

Rosetta Terminal

When installing build tools like CocoaPods, you have to run your whole terminal session with Rosetta enabled.

A prevalent practice to achieve this is to create a copy of the Terminal.app, click "Get Info" and enable the flag "Open using Rosetta". Now you have a second terminal that is always running as Rosetta.

How to do this in VSCode

I very much enjoy using the integrated Terminal in VSCode because I don't have to switch windows so often and can continue coding while running processes like building my app or installing dependencies.

How to create a custom terminal profile:

Open Visual Studio Code
Use cmd+shift+p to open the Command Palette.
Type "Open Settings (JSON)" and press enter

In the settings.json file, search for the property terminal.integrated.profiles.osx, it should look similar to this:



"terminal.integrated.profiles.osx": {
    "bash": {
    "path": "bash",
    "args": ["-l"],
    "icon": "terminal-bash"
    },
    "zsh": {
    "path": "zsh",
    "args": ["-l"]
    },
    "fish": {
    "path": "fish",
    "args": ["-l"]
    },
    "tmux": {
    "path": "tmux",
    "icon": "terminal-tmux"
    },
    "pwsh": {
    "path": "pwsh",
    "icon": "terminal-powershell"
    }
}


- Add this property:

    ```


    "terminal.integrated.profiles.osx": {
     ....,
      "rosetta": {
        "path": "arch",
        "args": ["-x86_64", "zsh", "-l"],
        "overrideName": true
      }
    }

Save the file

Tada! You now have a dedicated rosetta Terminal profile in VSCode

Click the dropdown arrow at the right side of your VSCode terminal. You should see the option "rosetta".

If it doesn't work or you have anything else you want to tell me, feel free to hit me up anytime on Twitter.