DEV Community: alephalpha0

Building A CLI Flow - 01

alephalpha0 — Wed, 11 Nov 2020 16:45:31 +0000

Building a cli program flow in order to archive a single webpage or article.

The problem to be solved:

I have a desire to be able to, through the command line, download a single webpage into a self-contained .html file with the option to convert it to (or save alongside) other formats such as PDF, EPUB, or even md if possible. There are two main use-cases I want to offer: 1) get the entire webpage, strip out the ads/useless shit, and save the new version locally & 2) for journal/website articles, blog entries, or tutorials: simply save the main content and discard every other element from the source.

The initial idea for solution:

A bash script that takes in a URL and, through menu redirection, outputs the desired file(s) into a standard directory.

Why not use currently available methods:

I've come across quite a few programs and scripts that seem like they would be perfect for what I want to do, but unfortunately I'm using termux on an Android and most of them do not work for one reason or the other in this enviromment.

CLIs that I have tried with no luck to get working on Termux:

hget - hget
monolith - monolith
erised - erised
percollate - percollate
ferret - ferret

The semi-focused detailing of my initial mind thoughts:

So, I feel like my grasp of scripting and pipelines is decent enough to cobble together a functioning bash script using programs and sources at my disposal that should take care of what I want to do. Namely, a locally saved, single file .html and have the option/ability to convert to PDF, md, or EPUB if desired. In my mind, there will be several working pieces of this script that could and should be broken down into smaller modules\functions .

I'm designing and planning on a menu-driven interface so creating menu-building and menu-displaying functions will be two individual tasks. Directory checking, creation, and manipulation should be another set of tasks. Error handling, logging and graceful failures are another set. And finally, the actual downloading and parsing into files will be a set of functions.

I'm aware that bash scripting might not be the optimal route to go with this idea, due to processing times, data handling and manipulation, and the fact that other languages come with the advantage of libraries and support modules that cover just about every detail and "feature" I could possibly come up against. However, it will serve as a basis for laying out the flow and main ideas of what I want my script to do as well as help prototype and get a working product to use as proof of concept.

From the get-go, there are several options that I can and will be utilizing to achieve my end-goal. I'm hoping that using and offering several options of methods will not over-complicate the road ahead. The ability to write code that can be easily extended and has the ability to easily have options added to it is something that I am working towards. The thought that I should focus on one (one method for obtaining the webpage, one method for processing the downloaded data, one method for saving, and one method for file conversion) is there, and it has validity. However, I am blockheaded when it comes to ambitions, and throwing caution to the wind is the name of the game right now.

Options in consideration for scraping/downloading/getting source:

wget
cURL
httpie
ferret
w3m

This is as far as I have gotten, planning and layout wise. The next part of this session of brain-vomitting will cover the logic trains, process flow, and possible ways of brainstorming the layout of the script.

TODO:

research the methods of using ferret to scrape websites and obtain the contents.
look into flags and options of wget cURL httpie and w3m.
Sketch out the menus and brainstorm ways for it to work.
For processing... HTML2JSON? HTML2MD? HTML2?????
File conversion should be PANDOC, but of course, the perfect tool doesn't work under termux.

Originally posted on my blog and on a gist

Mini conceptual breakthrough re: ln

alephalpha0 — Sun, 23 Aug 2020 05:01:52 +0000

mind. blown.

i do not use the typical laptop or desktop setup to muck around on the linux cli, to use text editors [micro, my babe], or futz around in image and video editors. i rely mainly on my android phones rocking termux and other various apps.

one of the pains in my keister lately, as far as getting the files to my site host's server goes, has been in the quite convoluted steps i take to do a quite simple task. these are mainly, mostly in fact, done as a direct result of a) i am scared of ssh/sftp via a cli still, and b) im self teaching on every aspect of my growth so far. the process is roughly:

commit the updates via git
rsync copy the files from the termux app storage into android storage proper /data/data/com.termux/... to the usual /storage/emulated/0
use TotalCommander as visual ftp client and send those badboys up to the server.

Now the point of this post. ln and hard links. I was reading some articles and the die.net MAN pages on ln tonight and it finally sank in, what makes a hard link so incredibly valuable. especially in arenas like where i dwell.

a file name simply points the OS to the data bits and bytes that are written on the disk that make up what i conceptualize as a .html or .mp3 file. if the file name is erased (this happens when i delete the file) only the pointer to the specific bits is removed. the bits an bytes remain and are written over, etc etc. there is nothing that says that multiple file names cannot point to and access the same range of bytes from the OS. that multiple file pointers cant reference the same data.

😲😲😲😲😲😲

i can, through termux, access my /storage/emulated/0/ file system and create hard links, via ln, to the html, liquid, css, etc files i edit and manipulate in termux and be able to upload them from android os. takes out constant rsync and mkdir and cp and mv shenanigans. makes my workflow leaner, tighter, more tiger like.
📱🔗🔗⛓️🔗⌨️

🐯

main article that brought on the ah hah moment

zshLoadOrder

alephalpha0 — Fri, 07 Aug 2020 12:22:08 +0000

ZSH shell startup/shutdown files.

This is really just for my own reference, as ZSH loads its config in its own order and I can never seem to remember it. The global files (which shouldn't really be tampered with by users) will be mentioned and put into larger font than the user files. However, I won't be expounding upon them, as that is outside the purpose of this writing.

`/etc/zshenv` above all else. The alpha, the omega, the genesis.

.zshenv (req to be in `$HOME`)

Here there be environmental variables.
Place here anything I'm going to export so that it can be referenced later.
Good candidates include XDG_* and changes to program defaults.
Set up your $PATH & $PAGER & etc etc etc.

Quality of life: export ZDOTDIR="$XDG_CONFIG_HOME/zsh"

`/etc/zprofile` ONLY if the shell is a login shell.

.zprofile (`$ZDOTDIR/.zprofile`) ONLY if shell is login.

Any commands you want happening on every login.
Start daemons, set a new MOTD, start a SSH tunnel.
The possibilities are endless.

`/etc/zshrc` ONLY if the shell is interactive.

.zshrc (`$ZDOTDIR/.zshrc`) ONLY if the shell is interactive.

Used for shell options and various commands.
There should be a loooong list of setopt and unsetopt going on here.
Shell modules and the ZSH prompt go here, as well as history config.

Source your scripts and set your aliases.

`/etc/zlogin` ONLY if shell is login.

.zlogin (`$ZDOTDIR/.zlogin`) ONLY if shell is login.

Almost same purpose as .zprofile but you know, not really different. Use only one or the other if you're flying by the book.

.zlogout (`$ZDOTDIR/.zlogout`) ONLY if shell is login.

Can be used to execute commands when a shell exits.
Good time to clean up caches or tmp files/folders.
Logout of sessions, set the environment back to how you like it when you login.

Send an email to yourself of temp logs and an email to your mom telling her you love her.

DEV Community: alephalpha0

Building A CLI Flow - 01

Building a cli program flow in order to archive a single webpage or article.

The problem to be solved:

The initial idea for solution:

Why not use currently available methods:

CLIs that I have tried with no luck to get working on Termux:

The semi-focused detailing of my initial mind thoughts:

Options in consideration for scraping/downloading/getting source:

TODO:

Mini conceptual breakthrough re: ln

mind. blown.

zshLoadOrder

ZSH shell startup/shutdown files.

/etc/zshenv above all else. The alpha, the omega, the genesis.

.zshenv (req to be in $HOME)

/etc/zprofile ONLY if the shell is a login shell.

.zprofile ($ZDOTDIR/.zprofile) ONLY if shell is login.

/etc/zshrc ONLY if the shell is interactive.

.zshrc ($ZDOTDIR/.zshrc) ONLY if the shell is interactive.

/etc/zlogin ONLY if shell is login.

.zlogin ($ZDOTDIR/.zlogin) ONLY if shell is login.

.zlogout ($ZDOTDIR/.zlogout) ONLY if shell is login.

References, sources & points of interest.