DEV Community: Shayan

Automate your Notion backups

Shayan — Sat, 17 Apr 2021 13:32:35 +0000

If you've read my last blog post, then you know I wanted to leave Notion for good.

The problem is the other solutions are just not quite there yet for me. I really tried, but I'm probably just too accustomed to Notion and its features.

One of the described issues in that post with Notion was the performance - or the lack of it. I feel like in the last few days the team behind Notion did some backend improvements (also a database upgrade). It definitely feels snappier now. Searching is faster, databases load quicker and you can actually use templates now without waiting forever for them to be set up in the page you're trying to create.

But this post is not really about this, so let's not get into too much detail and focus on our topic.

In the settings of Notion there is an Export all workspace content button. And for backup purposes I did those regularly, but depending on the size of your workspace it can take a while, plus it's a pretty manual process that you have to constantly remind yourself about.

And I'm the type of guy that tries to automate everything away so I can focus on the important things in my life (plus it's fun to do for an engineer, let's be honest).

For this reason I have written a small open source command line application in Go that can do just that: back up your Notion workspace or just specific parts of it - automatically. And thanks to Go it's platform independent, so you can even run it on macOS or Windows (there are pre-built binaries in the repository, if you don't want to build it yourself).

You can find it on Github at

5hay / notionbackup

A small utility command line application that can recursively download Notion pages

The easiest way to run is by downloading a pre-built binary, but since it's open source you can of course build it yourself.

Then you have to set at least 2 environment variables and you're good to go.

The necessary env vars are

NOTION_TOKEN
- The token_v2 cookie for Notion that is used for authenticating
- You should find enough search results if you just google "Notion token_v2"
  - Here's a quick explanation on how to retrieve that token in case you can't find anything: https://www.redgregory.com/notion/2020/6/15/9zuzav95gwzwewdu1dspweqbv481s5
NOTION_PAGEID
- This is the Notion page ID you want to set up for the automatic backup
- On every page you can hit those 3 dots in the corner and click Copy link
  - Alternatively, these 3 dots are also in the sidebar for every page if you hover with your mouse over them
- The page ID can look something like this
  
  https://www.notion.so/username/PageTitle-3514g811b36849b3zi8322fdac38287f
  - You can either set this env variable to the whole URL or just the ID part at the end after the last -.

On Linux based systems you can put them into your .*rc files (.bashrc, .zshrc, ...), so they are always available.

export NOTION_TOKEN=someLongAlphanumericValue
export NOTION_PAGEID="https://www.notion.so/username/......"

After that, source your .*rc file and you're already set up.

source ~/.bashrc

There are 2 other environment variables you can set for this application, but they have their defaults and don't need to be necessarily set (check the Github repository if you want to set them anyway).

Now you can execute the binary application by calling ./notionbackup (assuming you have already cd'd into that folder).

That's pretty much it. Once the application has finished downloading your backup (it's gonna be a .zip file) you can do whatever you want with it, e.g. integrate it into your backup solution and forget about it.

You could also automatically unzip it after every export so your backup solution can deduplicate the files better. There are endless possibilities. The important thing is just to have a backup in case something happens.

I keep it pretty simple.

I have set up a cron job for once per week that does a backup of my workspace and then pushes the file into my encrypted Google Drive folder with rclone (great tool if you're not familiar yet). The contents of my encrypted Google Drive folder are also replicated into other backup systems, but that's irrelevant for this post.

I hope you found this blog post and the tool useful. If there's only one thing to learn from this post, it's to always have backups for the worst case scenario.

If you want to follow me or just chat, you can find me on Twitter: @5hay

Set up your own cross platform, open source and privacy respecting Roam Research

Shayan — Mon, 08 Mar 2021 15:19:39 +0000

Introduction

A few weeks ago I stumbled upon Roam Research. At first glance it's
just another note-taking tool like Notion and Evernote.
But after a few minutes I realized how powerful this type of note-taking could be.

This is how Roam Research describes their own product:

A note-taking tool for networked thought.

As easy to use as a document. As powerful as a graph database. Roam helps you organize your research for the long haul.

Basically, what Roam Research does differently is the ability to have bi-directional links between notes and see the relation between these notes in a graph.

Why this is such a powerful feature and to get a pretty good understanding of the power of such a tool, I suggest you watch this YouTube video by Nat Eliason:

There's a problem though with Roam and the other apps I've mentioned: No privacy because these services can see the contents of all my notes.

I've been using Notion for the past 2+ years now and it's such a good product that I tried to ignore this problem. But due to the fact that lately Notion struggles with (performance) issues and the discovery of Roam, I decided that it's time to find a way for me to get back full control of my data without storing my most private data on servers I have no control over.

The Alternatives

There are a couple of other note taking apps that try to replicate the functionality of Roam Research.

Notion
- Yes, Notion itself can be a (although basic) alternative to Roam as they've implemented the backlinking functionality into Notion in September 2020
- Doesn't solve our problem though
Obsidian
- Privacy friendly as the data is only stored locally
- App is closed source though which I don't like
- No official way to take notes on mobile yet
Athens
- Open source, can be selfhosted (though in very early stage)
- I don't want to use Java-ish stuff
Logseq
- Open source, can be selfhosted (in early stage)
- more focused on you storing all your notes on Github, which gets us back to my initial problem
- Again, Java world
Foam
- Open Source, selfhosted
- Notes are just Markdown files
- You can use Git to version control your notes and have a history of the changes you make to the notes
- built on Visual Studio Code (the majority of us are probably using it anyway). This fact alone probably makes you realize how much potential this has
- full control of the data
- Early development stage, but already very functional

You probably already guessed it. In this post we're going to look at Foam as well as how to install and configure it so that it becomes platform independent and be used on every relevant platform: Windows, Mac, Linux, iOS and Android. Because as you know, VSCode is not available on mobile.

Installation and Configuration

You probably know VSCode as the Desktop application that is used by so many developers around the world. VSCode is open source and is built with JavaScript
(TypeScript to be more precise). So there are ways to make it run on the Web (read: in the browser). One such way is the project code-server. It's literally VSCode that you know from Desktop running in the browser (pretty cool, right?).

Now you should already have a pretty good understanding about how we're going to accomplish the cross platform functionality.

We're going to use Docker (and docker-compose) to install code-server. If you're not familiar with Docker yet, there are already countless posts about that topic on this website and the Web in general. It's out of scope for this post.

I have code-server (and Foam) installed on my server, but it will also without a problem run on something like a Raspberry Pi that you have lying at home.

I'm going to assume you have Docker and docker-compose
already installed.

So let's finally start with the steps required to get Foam installed and configured.

Create a new folder for this project on your server (I'm going to call it code-server)
In that folder create the following folder structure:
1. mkdir -p config/.config
2. mkdir {notes,code}
Create the docker-compose.yml file in the just created code-server folder with the following content:

   version: "2.1"

   services:
     code-server:
       image: ghcr.io/linuxserver/code-server
       restart: unless-stopped
       container_name: code-server
       ports:
         - 8443:8443
       volumes:
         - ./config:/config
         - ./notes:/notes # where our notes will be stored
         - ./code:/code # optional, if you want to also use this VSCode instance as your dev environment
       environment:
         - PUID=1000
         - PGID=1000
         - TZ=Europe/Berlin
         - PASSWORD=MyPassword #optional, password protect your VSCode instance
         - SUDO_PASSWORD=MyRootPassword #optional, for root access on the terminal of the Docker container

         # open-vsx.org extension marketplace
         # the default extension marketplace in code-server is not that great (very outdated extensions and slow)
         # this one is much better, but has also some outdated stuff
         - SERVICE_URL=https://open-vsx.org/vscode/gallery
         - ITEM_URL=https://open-vsx.org/vscode/item

         # Microsoft extension marketplace
         # gray area, Microsoft officially only allows its own products to use this marketplace
         #- SERVICE_URL=https://marketplace.visualstudio.com/_apis/public/gallery
         #- ITEM_URL=https://marketplace.visualstudio.com/items

         # optional, if you also want to use this browser based VSCode instance as your dev environment (I can highly recommend you test it out).
         # For other available mods check out the image's Docker Hub page
         # https://hub.docker.com/r/linuxserver/code-server
         - DOCKER_MODS=linuxserver/mods:code-server-golang|linuxserver/mods:code-server-nodejs|linuxserver/mods:code-server-php8|linuxserver/mods:code-server-python3

(Optional) You can also run it behind a reverse proxy like Traefik or nginx, but this it out of scope for this post. Check out Nginx Proxy Manager for a very easy to manage way to use nginx as your reverse proxy.
Pull the Docker image and start the code-server container (you must be in the code-server directory you created in the first step)

docker-compose up -d

(you can stop the container with docker-compose down)

You should now have a running version of VSCode in the browser on http://localhost:8443 (give it some time, especially if you're using the DOCKER_MODS). You can check the logs of this container
by running docker logs -f code-server to see once it's ready.

If you used the env var PASSWORD then you should log in with this password (there are also ways to store your password as a hash in the compose file).

The difficult part is already done, now we only need to get Foam up and running in our browser based VSCode and configure some settings.

For this you can check out the Foam Getting Started guide.

But basically, you need to either git clone or just download the
zip file and unzip it in the folder you want to use for your notes (the notes subfolder in this example).

After opening this folder within your code-server instance (that's why we've created the notes subfolder and added it as a volume to our container), VSCode should display a message about recommended extensions to install for Foam. Install (or review before installing) them.

Other extensions I'd recommend:

GitLens
- better Git integration
GitDoc
- Allows you to automatically and periodically commit (and push) the changes to your notes to a Git repository of your choice
Prettier
- Automatically format your notes, e.g. wrap long lines on Save etc.
Web Clipper
- A web clipper like Notion has that turns websites into Markdown and stores them in notes.
Vim
- If you're a vim enthusiast like me and need your vim keybindings pretty much everywhere

Here are the global VSCode settings that I've set for myself, feel free to use them to get a good starting point.

{
  "workbench.startupEditor": "newUntitledFile",
  "terminal.integrated.shell.linux": "/bin/bash",
  "files.autoSave": "off",
  "window.autoDetectColorScheme": true,
  "editor.tabSize": 2,
  "editor.matchBrackets": "always",
  "editor.minimap.enabled": true,
  "breadcrumbs.enabled": true,
  "editor.renderControlCharacters": true,
  "workbench.sideBar.location": "left",
  "workbench.colorTheme": "Default Dark+",
  "gitdoc.autoCommitDelay": 300000,
  "workbench.editor.enablePreview": false,
  "workbench.editor.enablePreviewFromQuickOpen": false,
  "foam.openDailyNote.directory": "journal",
  "prettier.enable": true,
  "prettier.printWidth": 100,
  "prettier.proseWrap": "always",
  "editor.formatOnSave": true,
  "[markdown]": {
    "editor.defaultFormatter": "esbenp.prettier-vscode"
  },
  "vscodeMarkdownNotes.newNoteDirectory": "WORKSPACE_ROOT"
}

If you're done with all these steps, you can start getting to know and using Foam by opening the getting-started.md file in the notes folder.

Tip: After you got a good understanding of Foam and deleted the test files (like readme.md, getting-started.md, etc.), you can git init that folder. With the GitDoc extension (see my
recommended extensions) every change will be periodically commited (and optionally pushed). That way you have unlimited revisions of your private knowledge base.

Conclusion

Now you're all set with a fully working, cross platform VSCode and your own privacy respecting and open source Roam Research alternative.

Coding on an Android tablet / iPad in VSCode (with vim)? Coding on every computer without having to re-configure your environment? Of course you can now also take your notes on every platform. The
possibilities are endless.

Thank you very much for reading. I hope you're going to have as much fun with your cross platform dev environment and powerful as well as privacy friendly note taking app as me.

I'm a Berlin based Software Engineer, mainly working with C++, Go and JS in the medical industry. If you want to follow or chat with me, you can find me on Twitter @5hay.