DEV Community: Shreyas Ragavan

Setting up Continuous Integration (CI) for docker containers

Shreyas Ragavan — Wed, 22 Jan 2020 06:30:00 +0000

This blog post takes you through the process of setting up Continuous Integration for building docker images via Dockerhub and Github, and via Github Actions. It also contains a condensed summary of important notes from the documentation.

Goal: Gain an overview of CI and actually use it to get automated builds of the docker images that built for my datascience toolbox.

Essentially I want to be able to a status check the docker containers that I am maintaining. Eventually I want to setup a series of checks that the libraries and software tools that I use are working as expected. Though dockerhub enables containers to be built on a commit, I would also like a CI/CD pipeline to be setup in order to understand how it actually works.

Pre-requisites : a dockerhub account and some dockerhub image to work off with. The dockerfile and related source code should be available in a github repository.

The github repository I will use is shrysr/sr-ds-docker and the dockerhub image shrysr/shiny. Within the github repository, the shiny folder contains all the files needed to build the shiny image. Note here that the rbase image is required for the shiny image to build.

Plan `[3⁄3]`

Complete reading the Github documentation on github actions.

setup a github hosted runner
[-] Setup a self-hosted runner : ~~Lower priority~~ Postponed because it is better to understand how a runner works with github code before allowing any github code to run on my VPS.

Create a CI integration between Dockerhub and Github
Expand the CI setup to the datascience docker containers.

Setting up a Github Runner `[0/1]`

A github runner is essentially creatd by using the Actions tab. There is a marketplace of Actions that can be used for free. Actions already exist for many popular workflows like building a docker image and pushing it to some registry.

Apparently the first action has to be a checkout of the repository. Without this step, the process will not work. I spent a long time in

Specify build context to a specific folder. i.e do not use build . because then the context and paths will not work, thus the COPY type functions won’t work.

Apparently Github will reocognise yaml files only within the .github/workflows location, though I may be wrong. If their autosuggestion for the action setup is used, the folder is created automatically. However, thankfully,any YAML file created in this folder will be run by Github actions. Refer to the notes below regarding the API limitations. Since it appears that this YAML file cannot be used for say Travis or CircleCI, it may be good to have these within the github folder anyhow.

Getting started was as simple as using the actions tab when the repository is opened. A basic YAML template is offered and was actually sufficient for me to quickly get started.

The build context is specified by the location of the file in this case, and the tag can be specified. Currently, I’m using an ephemeral container.

An idea for a test would be like : run an ubuntu docker container, and then call the shiny container within it. Get the container running, and then also devise some output via R scripts. One way could be load a bunch of packages. Another way could be to get a shiny app to run and provide some kind of temporary output. This has to be refined further.

name: Docker Image CI

on: [push]

jobs:

  build:

    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v2
    - name: shiny
      run: docker build . --file /shiny/Dockerfile --tag my-image-name:$(date +%s)q

The build details can be found by clicking on the individual jobs. The raw log can be downloaded for verbose logs to enable a good text search. During a live build process, there is some lag between the log update and the webpage refresh, however it seems within tolerable limits as of now.

Notes about build and queue

The free build time is rather long and the configuration of the servers is unknown. It is probably a lot faster to build the images locally and push them to dockerhub. However, minor image updates and small image builds are quite quick to take place. The key is getting a single successful build off a github commit after which the context is established and new layers need not take the same amount of time.

All this being said, the queue time in dockerhub is very long compared to the queue time of builds via actions on Github. The free tier is actually quite generous for a lot of room to play and experiment. It would also appear that the github computer servers are faster than Dockerhub.

Setting up CI via Dockerhub and Github

The pre-requisite is of course that you have a docker image in your repository.

This process is relatively simple. Login to your Dockerhub account and click on the fingerprint like icon to reach the account settings. Use the linked accounts tab and setup github with your login credentials.

Next, select your docker image repository and click on the Builds tab and click on setup automated builds.

Now you have the option to select a github repository and settings are available to point to a particular branch or a particular Dockerfile as well.

Note the option Enable for Base Image for the Repository Links. This can be set enabled if your image depends on another image. Suppose that base image is updated, then a build will be triggered for your image.

The source option can be set to a named branch or a tag and the docker tag must also be specified. The build context helps if you have multiple docker configurations stored together.

Note also below that Environment variables can be specified thus enabled a more customised deployment of the image. The variable can be used to specify things like the username and password, Rstudio version or r-base version, etc. Docker image tags are typically used to demarcate these more easily.

Here’s how the build activity looks like on Dockerhub:

General notes

Overview of components

For any code to run, there has to be a server or a computer to run it. This is called a runner and one can be created on a self-hosted platform or there are services with different tiers on which the runners can be placed.

Reference: Github docs

GitHub Actions enables you to create custom software development life cycle (SDLC) workflows directly in your GitHub repository. GitHub Actions is available with GitHub Free, GitHub Pro, GitHub Team, GitHub Enterprise Cloud, and GitHub One

Workflows run in Linux, macOS, Windows, and containers on GitHub-hosted machines, called ‘runners’. Alternatively, you can also host your own runners to run workflows on machines you own or manage. For more information see, “About self-hosted runners.”

You can create workflows using actions defined in your repository, open source actions in a public repository on GitHub, or a published Docker container image. Workflows in forked repositories don’t run by default.

You can create a workflow file configured to run on specific events. For more information, see “Configuring a workflow” and “Workflow syntax for GitHub Actions”.

GitHub Marketplace is a central location for you to find, share, and use actions built by the GitHub community.

On API limits

Most of these limits are not applicable to self-hosted runners. Exception to the above : “You can execute up to 1000 API requests in an hour across all actions within a repository.”

There are some limits on GitHub Actions usage. Unless specified, the following limits apply only to GitHub-hosted runners, and not self-hosted runners. These limits are subject to change.

You can execute up to 20 workflows concurrently per repository. If exceeded, any additional workflows are queued.

Each job in a workflow can run for up to 6 hours of execution time. If a job reaches this limit, the job is terminated and fails to complete.

The number of concurrent jobs you can run across all repositories in your account depends on your GitHub plan. If exceeded, any additional jobs are queued.

You can execute up to 1000 API requests in an hour across all actions within a repository. This limit also applies to self-hosted runners. If exceeded, additional API calls will fail, which might cause jobs to fail.

In addition to these limits, GitHub Actions should not be used for:

Content or activity that is illegal or otherwise prohibited by our Terms of Service or Community Guidelines.

Cryptomining

Serverless computing

Activity that compromises GitHub users or GitHub services.

Any other activity unrelated to the production, testing, deployment, or publication of the software project associated with the repository where GitHub Actions are used. In other words, be cool, don’t use GitHub Actions in ways you know you shouldn’t.

You can execute up to 1000 API requests in an hour across all actions within a repository. This limit also applies to self-hosted runners. If exceeded, additional API calls will fail, which might cause jobs to fail.

GitHub plan	Total concurrent jobs	Maximum concurrent macOS jobs
Free	20	5
Pro	40	5
Team	60	5
Enterprise	180	15

Self-hosted runners versus github hosted runners

Self-hosted runners can be physical, virtual, container, on-premises, or in a cloud. You can use any machine as a self-hosted runner as long at it meets these requirements:

You can install and run the GitHub Actions runner application on the machine. For more information, see “Supported operating systems for self-hosted runners.”

The machine can communicate with GitHub Actions. For more information, see “Communication between self-hosted runners and GitHub.”

GitHub-hosted runners offer a quicker, simpler way to run your workflows, while self-hosted runners are a highly configurable way to run workflows in your own custom environment.

GitHub-hosted runners:

Are automatically updated.

Are managed and maintained by GitHub.

- Provide a clean instance for every job execution.

Self-hosted runners:

Can use cloud services or local machines that you already pay for.

Are customizable to your hardware, operating system, software, and security requirements.

Don’t need to have a clean instance for every job execution.

Depending on your usage, can be more cost-effective than GitHub-hosted runners.

Notes on setting up :Using an .env file

If setting environment variables is not practical, you can set the proxy configuration variables in a file named .env in the self-hosted runner application directory. For example, this might be necessary if you want to configure the runner application as a service under a system account. When the runner application starts, it reads the variables set in .env for the proxy configuration.

An example .env proxy configuration is shown below:

https_proxy=http://proxy.local:8080no_proxy=example.com,myserver.local:443

Self-hosted runner security : With respect to public repos - essentially, this means that some repo of code on github is able to run on your computer. Therefore unless the code is trusted and vetted, it is dangerous to allow this. Forks can cause malicious workflows to run by opening a pull request.

It is safer to use public repositories with a github hosted runner.

We recommend that you do not use self-hosted runners with public repositories.

Forks of your public repository can potentially run dangerous code on your self-hosted runner machine by creating a pull request that executes the code in a workflow.

This is not an issue with GitHub-hosted runners because each GitHub-hosted runner is always a clean isolated virtual machine, and it is destroyed at the end of the job execution.

Untrusted workflows running on your self-hosted runner poses significant security risks for your machine and network environment, especially if your machine persists its environment between jobs. Some of the risks include:

Malicious programs running on the machine.

Escaping the machine’s runner sandbox.

Exposing access to the machine’s network environment.

Persisting unwanted or dangerous data on the machine.

CI

Continuous Integration: enables commits to trigger builds and thereby enhances error discovery and rectification process.

The CI server can run on the same server as the runner. Therefore it can be github hosted or a self-hosted CI server.

Github analyses a repository when CI is setup and recommends basic templates depending on the language used.

When you commit code to your repository, you can continuously build and test the code to make sure that the commit doesn’t introduce errors. Your tests can include code linters (which check style formatting), security checks, code coverage, functional tests, and other custom checks.

Building and testing your code requires a server. You can build and test updates locally before pushing code to a repository, or you can use a CI server that checks for new code commits in a repository.

You can configure your CI workflow to run when a GitHub event occurs (for example, when new code is pushed to your repository), on a set schedule, or when an external event occurs using the repository dispatch webhook.

Oddmuse-curl exploration

Shreyas Ragavan — Mon, 20 Jan 2020 17:11:00 +0000

oddmuse-curl is kensanata’s tool to use Emacs to edit oddmuse wikis. I think it works rather well for my needs. Alternatively, I can also use the org exporter to export org mode content and manually copy this into a wiki page via the web browser. CLI options also exist to update the wiki, but I don’t think I need that functionality.

Borrowing off the documentation of oddmuse-curl, I was able to set this up, with some additional settings. It always helps to read the source code, and I’m finding of late that elisp code often seem to have a detailed explanation, comments and etc that make the code a pleasure to read.

One always hears - keep the code well commented, and to use caution in describing the idea rather than the syntax. However, it is not often to find such examples. It is probably also worth noting that most of them also seem to prefer to not have their readme’s tangling the source code. The .el files are probably edited directly in Emacs. This is not surprising because the lispy mode (and etc) provide very good support for writing lisp in Emacs.

So to begin with : Clone kensanata’s package from github to the desired location:

git clone https://github.com/kensanata/oddmuse-curl.git ~/scimax/user/external_packages/

Next here is my config. There are some additional options that kensanata’s readme does not mention, however all these options can be found by reading the source code.

(use-package oddmuse-curl
:load-path "~/scimax-personal/external_packages/oddmuse-curl/"
:defer nil
:ensure t
:config
;; user name
(setq oddmuse-username "shrysr")

;; Wiki names
(setq oddmuse-wikis
      '(("EmacsWiki" "https://www.emacswiki.org/emacs" utf-8 "abcd" nil)
    ("sr" "https://shrysr.ragavan.co" utf-8 nil nil)))

;; Directory where files will be downloaded from the wiki for editing.
(setq oddmuse-directory "~/my_org/01_wiki/oddmuse/")

;; adding an oddmuse-odd for files in the fiki directory
(add-to-list 'auto-mode-alist '("~/my_org/01_wiki/oddmuse" . oddmuse-mode))

;; autoload modes
(autoload 'oddmuse-edit "oddmuse-curl"
  "Edit a page on an Oddmuse wiki." t)

;; Not yet sure what this does and how it related to version control.
(add-to-list 'vc-handled-backends 'oddmuse)
(defun vc-oddmuse-registered (file)
  "Handle files in `oddmuse-directory'."
  (string-match (concat "^" (expand-file-name oddmuse-directory))
                (file-name-directory file)))

;; Since I work primarily with org before the wiki - I would rather note have the mode initialised.
;; (oddmuse-mode-initialize)

;; I would like to be able to call the wiki when desired and so the curl package is initialised.
(require 'oddmuse-curl)
)

All I have to do is use M-x oddmuse-go to select the wiki I want and then start editing the page required. In the beginning it is a little intimidating to consider that a page has to be selected. However the search option can be easily used for anything relevant. It might also make sense to place useful links within a single page like the page which can be visited as a bookmark.

A graphic overview of the 'binary' with respect to R packages

Shreyas Ragavan — Sun, 19 Jan 2020 04:28:00 +0000

Recently there was a question as to what a Binary is, building off a question posted on the Rstudio community forum. I’ve always found these aspects interesting, and a little hard to keep track of the connections and flow - So I’ve made a flowchart that will help me remember and hopefully explain what is happening to a noob.

In this process, I was able to remember One of the first documents I really enjoyed reading when I started learning how to use Linux. I would recommend that article for anybody starting out. The document is meant for people with a non-technical background, but I think it is technical enough.

So: Lets pretend the binary is a capsule to be swallowed by the computer to gain superpowers :D : The capsule is in a sense off-the-shelf and made for your System. When the capsule is not available, it has to be manufactured (compiled) by your machine locally for which it needs certain tools and dependencies, etc and this varies from package to package, and possibly between hardware architectures as well and more. Please feel free to let me know if there are any discrepancies in the flowchart !

Binaries can be built and maintained if you desire. There should be people maintaining their own binaries or frozen versions as well. The question is - who is going to maintain them and how many binaries can you build.

Implementing HTTPS : Let's Encrypt

Shreyas Ragavan — Sat, 09 Nov 2019 14:41:00 +0000

What is Let’s Encrypt?

Let’s Encrypt is a Certificate Authority (CA). A certificate from a CA is required to enable HTTPS.

Certbot’s documentation summarises it well:

Certbot is part of EFF’s effort to encrypt the entire Internet. Secure communication over the Web relies on HTTPS, which requires the use of a digital certificate that lets browsers verify the identity of web servers (e.g., is that really google.com?). Web servers obtain their certificates from trusted third parties called certificate authorities (CAs).

How Let’s Encrypt works

To certify my domain, I need to demonstrate control over my domain. i.e one has to run a software tool to generate this certificate (periodically) on the server. being able to do this demonstratesa control over the domain.
- Similar to domain control, there are other certificates for different purposes as well. See the excerpt from the ACME protocol below:

Different types of certificates reflect different kinds of CA verification of information about the certificate subject. “Domain Validation” (DV) certificates are by far the most common type. For DV validation, the CA merely verifies that the requester has effective control of the web server and/or DNS server for the domain, but does not explicitly attempt to verify their real-world identity. (This is as opposed to “Organization Validation” (OV) and “Extended Validation” (EV) certificates, where the process is intended to also verify the real-world identity of the requester.)

Let’s Encrypt’s documentation mentions that the software above will use the ACME protocols to generate the cert, and there are different approaches to do so, depending on the availability of shell access (or not) to the server.
ACME stands for Automatic Certificate Management Environment : The introduction in the RFC demonstrates how ACME automates a significantly manual procedure combining ad-hoc protocols.

…protocol that a certificate authority (CA) and an applicant can use to automate the process of verification and certificate issuance. The protocol also provides facilities for other certificate management functions, such as certificate revocation.

Since I have shell access to my VPS, I will focus on this approach.
There are multiple ACME clients to choose from, and Certbot is ‘recommended’ (by the EFF). On a superficial glance, GetSSL looks interesting as an alternative.

At this point, I will proceed with Certbot, because I’ve not yet found any particular reason not to.

On Certbot `[0/1]`

The Certbot website provides customized instructions for the OS and server. The main requirement(s) is having an online HTTP website with an open port 80, hosted on a server. I can go ahead since I’ve got these.

Certbot will run on the web server (not locally) periodically and will help in automating the process of certificate management.

Setting up Certbot (on debian)

wget https://dl.eff.org/certbot-auto
sudo mv certbot-auto /usr/local/bin/certbot-auto
sudo chown root /usr/local/bin/certbot-auto
sudo chmod 0755 /usr/local/bin/certbot-auto

Checking that the above was actually done with a simple:

ls -al /usr/local/bin/cert*

Next, a one-command certificate setup is possible (with nginx)

Note that this command may require additional dependencies to be installed, and will need a bunch of user input as well, and so should not be run in a dumb terminal.

sudo /usr/local/bin/certbot-auto --nginx

This will:

Install necessary dependencies and the certbot plugins (authenticator, installer) for nginx.

Noted the option of --no-boostrap for debian. I’m not sure, but this probably has to do with addressing the dependencies for different debian versions.

For reference, the following packages were checked/installed:

ca-certificates is already the newest version (20190110).
ca-certificates set to manually installed.
gcc is already the newest version (4:8.3.0-1).
libffi-dev is already the newest version (3.2.1-9).
libffi-dev set to manually installed.
libssl-dev is already the newest version (1.1.1c-1).
openssl is already the newest version (1.1.1c-1).
openssl set to manually installed.
python is already the newest version (2.7.16-1).
python-dev is already the newest version (2.7.16-1).
python-virtualenv is already the newest version (15.1.0+ds-2).
virtualenv is already the newest version (15.1.0+ds-2).
virtualenv set to manually installed.

Suggested packages:

augeas-doc augeas-tools
The following NEW packages will be installed:
  augeas-lenses libaugeas0

An email address has to be entered for ‘urgent’ communication regarding the certificate, and optionally can be shared with the EFF (which was a trifle annoying (as a part of an installation process), though I said yes).

I had to enable https with UFW to complete the test successfully. sudo ufw allow https. Earlier, only HTTP had been enabled.

Automatic certificate renewal by setting up a cron job.

echo "0 0,12 * * * root python -c 'import random; import time; time.sleep(random.random() * 3600)' && /usr/local/bin/certbot-auto renew" | sudo tee -a /etc/crontab > /dev/null

deciphering the cron job, and verifying it is as expected. For now, I’ve not run this command because I want to know what it is doing first.

As an alternative to a ‘one-step’ installation, getting just the certificate will mean nginx’s configuration will have to done manually. This is probably a good choice to ‘learn more’.

sudo /usr/local/bin/certbot-auto certonly --nginx

I need to verify this, but it appears nginx’s main configuration is at /etc/nginx/nginx.conf , and a quick peek showed me that the user was still set as ‘www-data’, which was used as the initial setup of the nginx test website. This was changed subsequently. Perhaps this is why I am unable to get Wordpress plugins write access.

At the end of it all, I received a link, through which it appears I can get a detailed ‘SSL Report’.

Congratulations! You have successfully enabled https://s.ragavan.co

You should test your configuration at:
https://www.ssllabs.com/ssltest/analyze.html?d=s.ragavan.co

This report appears to be quite important, but I could not make much sense of it, and it needs to be re-visited. As such, I see that the ACME challenges need to be understood to comprehend these results.

Short Peek under the hood.

A skim of the extensive documentation of Certbot shows that certbot relies on 2 types of plugins to function.

authenticators: plugins to obtain a certificate, but not install (i.e edit the server configuration). Used with the certonly command.
Installers: used to modify the server’s configuration. Used with the install command.
Authenticators + installers : can be used with the certbot run command.

These plugins use ‘ACME Protocol challenges’ to prove domain ownership. Section 7 (as of today) of the internet draft of the standard provides an overview, and the challenges are described in detail in the draft.

There are few types of identifiers in the world for which there is a standardized mechanism to prove possession of a given identifier. In all practical cases, CAs rely on a variety of means to test whether an entity applying for a certificate with a given identifier actually controls that identifier.

Challenges provide the server with assurance that an account key holder is also the entity that controls an identifier. For each type of challenge, it must be the case that in order for an entity to successfully complete the challenge the entity must both:

Hold the private key of the account key pair used to respond to the challenge

Control the identifier in question

Conclusions

HTTPS via Let’s Encrypt is setup for my website.
Had a brief introduction into the methodology/philosophy behind Let’s Encrypt.
Brief exploration of ACME and it was quite interesting to go through the draft standard, though it will take a lot more effort to fully comprehend all the tests. I think it is likely that I have visit this in more detail as I make progress in learning about encryption.
Learned about the existence of ‘Internet Standards’. These are documented by one or more documents called RFC’s (Request for Comments) and revised until deemed satisfactory to become a standard.

Federal R&D Spending on climate change

Shreyas Ragavan — Fri, 01 Nov 2019 20:27:00 +0000

Summary

This is a short exploration into the tidy tuesday dataset focused on the Federal R&D budget towards global climate change. The data has been extracted from a TidyTuesday dataset, which in return is moderately cleaned dataset from publicly available data. The analysis will show that NASA’s budget dwarfs the money going into other departments, and that the median spend towards climate change has been increasing since the year 2000.

Useful links:

Github Repo of Tidy Tuesday explorations
Tidy tuesday dataset: link
Data Dictionary link
Viz posted on Twitter to participate in TidyTuesday.
Tools used: ESS, Org mode

Download R script : this is the entire script below.

Loading libraries

The easypackages library allows quickly installing and loading multiple packages. Note: Uncomment the appropriate line if this library needs to be installed.

# Loading libraries
                                        # install.packages("easypackages")
library("easypackages")
libraries("tidyverse", "tidyquant", "DataExplorer")

All packages loaded successfully

Reading in the data

Since this is a small dataset, the data can be read in directly from Github into memory.

# Reading in data directly from github
climate_spend_raw <- readr::read_csv("https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2019/2019-02-12/climate_spending.csv", col_types = "cin")

Exploring the data

We have 6 departments, and the remaining departments are lumped together as ‘All Other’.

The data is available for the years 2000 to 2017.

The above can be found using the unique function.

climate_spend_raw$department %>% unique()
climate_spend_raw$year %>% unique()

[1] "NASA" "NSF" "Commerce (NOAA)" "Energy"
[5] "Agriculture" "Interior" "All Other"

 [1] 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014
[16] 2015 2016 2017

Some Notes on the data:

We have the following columns:
- name of the department. (chr)
- year (int)
- spending (double)
The data is relatively clean. However some manipulation is required to summarise the department wise spending.

An overview of missing data can be easily scrutinised using the plot_intro command, and actual numbers can be extracted using introduce. These functions are from the DataExplorer package.

##plot_str(climate_spend_raw, type = 'r')
plot_intro(climate_spend_raw)
##introduce(climate_spend_raw)

There are no missing values or NA’s.

For a quick look at the outliers, we can use a boxplot, using DataExplorer’s functions.

variance_climate_spend <- plot_boxplot(climate_spend_raw, by = "year")

Figure 1: It can be seen above that there are not many outliers. Subsequent visualisations will show that NASA is the most significant outlier. The median spending has been increasing over the years.

Data Conditioning

Note: this initial conditioning need not have involved the date manipulation, as the year extracted from a date object is still a double.

climate_spend_conditioned <- climate_spend_raw %>%
  mutate(year_dt = str_glue("{year}-01-01")) %>%
  mutate(year_dt = as.Date(year_dt)) %>%
  mutate(test_median = median(gcc_spending)) %>%
  mutate(gcc_spending_txt = scales::dollar(gcc_spending,
                                           scale = 1e-09,
                                           suffix = "B"
                                           )
         )

Applying some summary statistics to calculate the total spend per department, per year.

# Total spend per department per year
climate_spend_dept_y <- climate_spend_conditioned %>%
  group_by(department, year_dt = year(year_dt)) %>%
  summarise(
    tot_spend_dept_y = sum(gcc_spending)) %>%
  mutate(tot_spend_dept_y_txt = tot_spend_dept_y %>%
           scales::dollar(scale = 1e-09,
                          suffix = "B")
         ) %>%
  ungroup()

Lets see how much money has been budgeted in each department towards R&D in climate change from 2000 to 2017.

climate_spend_conditioned %>%
  select(-c(gcc_spending_txt, year_dt)) %>%
  group_by(department) %>%
  summarise(total_spend_y = sum(gcc_spending)) %>%
  arrange(desc(total_spend_y)) %>%
  mutate(total_spend_y = total_spend_y %>% scales::dollar(scale = 1e-09,
                                                        suffix = "B",
                                                        prefix = "$")
       )

Department	Total Spend from 2000-2017
NASA	$25.77B
Commerce (NOAA)	$5.28B
NSF	$5.26B
Energy	$3.32B
Agriculture	$1.63B
All Other	$1.54B
Interior	$0.86B

It is clear from here that the outlier department is NASA. Further exploration would be needed to understand the function of each department and the justification of this expenditure and the skew. For example, one might think the Interior department would not be able to produce R&D superior to NASA/NSF.

Function to plot a facet grid of the department spending

By using a function to complete the plot, the plot can be easily repeated for any range of years. It can also work for a single year.

The function below takes the following arguments:

The range of the years we want to look into , example 2005-2010
The number of columns in the facet wrap plot.
The caption that consititues the observation from the plots and anything else.

The title of the plot includes the year range that is input above.

climate_spend_plt_fn <- function(
                                 data,
                                 y_range_low = 2000,
                                 y_range_hi = 2010,
                                 ncol = 3,
                                 caption = ""
                                 )
{

  plot_title <- str_glue("Federal R&D budget towards Climate Change: {y_range_low}-{y_range_hi}")

  data %>%
  filter(year_dt >= y_range_low & year_dt <= y_range_hi) %>%
  ggplot(aes(y = tot_spend_dept_y_txt, x = department, fill = department ))+
  geom_col() +
  facet_wrap(~ year_dt,
             ncol = 3,
             scales = "free_y"
             ) +
  #scale_y_continuous(breaks = scales::pretty_breaks(10)) +
  theme_tq() +
  scale_fill_tq(theme = "dark") +
  theme(
    axis.text.x = element_text(angle = 45,
                               hjust = 1.2),
    legend.position = "none",
    plot.background=element_rect(fill="#f7f7f7"),
    ) +
  labs(
    title = plot_title,
    x = "Department",
    y = "Total Budget $ Billion",
    subtitle = "NASA literally dwarfs all the other departments, getting to spend upwards of 1.1 Billion dollars every year since 2000.",
    caption = caption
  )

}

Visualizing department-wise spending over the years

Calling the function and passing in the entire date (year) range of 2000-2010. Note that for a single year, have both the arguments y_range_low and y_range_high equal to the same year.

climate_spend_plt_fn(climate_spend_dept_y,
                     y_range_low = 2000,
                     y_range_hi = 2010,
                     caption = "#TidyTuesday:\nDataset 2019-02-12\nShreyas Ragavan"
                       )

Figure 2: R&D Budget towards Climate Change from year 2000-2010 across departments.

climate_spend_plt_fn(climate_spend_dept_y,
                     y_range_low = 2011,
                     y_range_hi = 2017,
                     caption = "#TidyTuesday:\nDataset 2019-02-12\nShreyas Ragavan"
                       )

Figure 3: R&D Budget towards Climate Change from year 2011-2017 across departments.

Some Concluding statements

NASA has the highest R&D budget allocation towards climate change, and one that is significantly higher than all the other departments put together. The median spending on R&D towards climate change has been increasing over the years, which is a good sign considering the importance of the problem. Some further explorations could be along the lines of the percentage change in spending per department every year, and the proportion of each department in terms of percentage for each year.

mosh - for better access to my VPS

Shreyas Ragavan — Thu, 01 Aug 2019 15:42:00 +0000

Mosh is short for mobile shell, and is useful as an alternative to SSH, especially for poor network conditions, and where one has to frequently switch networks. It works via the UDP port, which has to be specifically enabled. I learnt of mosh through the guys in the #emacs.

I’ve faced frequent trouble due to network issues over SSH connections, with the lag hampering my ability to type, in general, and it is particularly inconvenient to respond on IRC/Weechat. I’m hoping mosh will alleviate the issue.

UDP needs to be enabled for mosh to work. I used UFW on the ports 60000:61000 for this.

sudo ufw allow 60000:61000/udp

Essentially, a mosh server runs on both the machines (VPS and local machine), and these perform the background job of syncing commands and output with each other. This reduces the lag in typing, among other advantages. The initial connection of Mosh, including authentication is via SSH, after which the UDP protocol is used.

Installing Mosh:

On debian - mosh is directly available as a package. Run apt-get update and then install mosh.

apt-get install "mosh"

The mosh-server has to be run on both the machines. It may be a good idea to include this in .bashrc, or in the list of start-up programs. This command will start up the mosh-server and detach the process (into the background).

mosh-server

This is where I ran into trouble. A UTF-8 environment has to be specified for mosh to run, and it appears that the locales of the two connecting machines have to match (?). On Debian, this is relatively easy with dpkg

sudo dpkg-reconfigure locales

I chose the en_USA.UTF-8 option. The existing locale configuration can be viewed with locale.

locale

LANG="en_CA.UTF-8"
LC_COLLATE="en_CA.UTF-8"
LC_CTYPE="en_CA.UTF-8"
LC_MESSAGES="en_CA.UTF-8"
LC_MONETARY="en_CA.UTF-8"
LC_NUMERIC="en_CA.UTF-8"
LC_TIME="en_CA.UTF-8"
LC_ALL=

Sometimes, additional settings for the locale are defined in locations like ~/.bashrc. This should be something like :

export LANGUAGE=en_US.UTF-8
export LANG=en_US.UTF-8
# export LC_ALL=en_US.UTF-8

The above can be used for explicitly setting the preference. The Debian wiki dissuades end-users from using LC_ALL, but that is easiest way. My initial settings were with en_CA.UTF-8. While this is also UTF-8, for some reason, mosh still threw out locale errors. In any case, I wanted all my computers to uniformly use the en_US version.

Did mosh make a difference?

It’s only been a few hours, but the difference can already be felt. Mosh clearly indicates when the connection has been lost and there is no lag in typing. Further experimentation is necessary to understand its behavior, but atleast, I can type out a message in peace without lag.

[2019-07-31 Wed] After 3+ days of using mosh, I am happy to note that the experience of engaging with my vps over a terminal has significantly improved. There were few instances of really poor network connection, and mosh would clearly indicate the disconnection, and also allow a safe exit if required. I can switch computers and jump right in, without bothering to restore the SSH connection.

'Archaic', text based email clients rock

Shreyas Ragavan — Sun, 14 Jul 2019 02:16:39 +0000

TLDR - for the busy folks
1. Goals:
2. Summary:
Multiple email accounts. Lack of a unified interface.
Creating sync channels via mbsync
Text based email client! Speed + simplicity
Why mu4e rocks [for me] - the perks
Quirksx
Multiple levels of filters are still necessary.
Takeaways
Links and References

The post below inspired me to complete this languishing draft of my current email setup, and the benefits I've gained from using a text based email client in Emacs.

How I unified my email accounts in 2019

Mohammed Ismail Ansari ・ Jul 12 '19

#watercooler #productivity

Hope you find it entertaining. In any case, the links and reference section will certainly prove useful. This article is also posted on my personal website.

TLDR - for the busy folks

Goals:

Unification of email accounts while preserving separate individual components.
Local backup of email.
Potential to extend system to a personal server
Email access from Emacs !
Hopefully improve overall productivity with reduced context switching.

Summary:

Started with 2 Gmail accounts and 1 MSN account.
Switched to a paid account with Fastmail.
Used Fastmail's tools to transfer email from both Gmail and MSN accounts.
Setup forwarding for all new emails to Fastmail.
Decided between retaining copies of emails in Gmail/MSN or deleting them once forwarded.
Used customised settings in mu4e to manage Email from within Emacs.
Occasionally rely on web browser / iOS app. Fastmail's interface is clean and very fast.
Goals Achieved !! Live with the quirks and enjoy the perks.

Look at the Links and References section for almost all the resources I relied on.

A portion of my mu4e configuration is available on my website. The personal filters and configuration are placed in an encrypted file.

My mbsync configuration is also shared for reference.

Multiple email accounts. Lack of a unified interface.

Some years back, I found that I had 2 Gmail accounts, and an MSN account. I discarded age old Yahoo and rediffmail accounts which were luckily not used much (and God knows how many more I made as a kid).

Gmail's interface felt just about tolerable, but inconvenient. The idea of viewing ads tailored to the content of emails had become disconcerting. Their Inbox app was interesting, but did not work smooth enough. MSN's web interace and apps always felt cumbersome, though updates over the years, this has improved significantly.

Useful emails could be email digests that contain a wealth of links, discussions, articles and information. Or perhaps email digests of product and technology news that are useful to retain as an archive of reference.

It would be nice to be able to process these links in a systematic manner, and have them available with a fast search system that is also integrated with a task management system.

My solution was to switch to forwarding all my emails to a single Fastmail account. It's been an excellent experience over 2+ years.¹,²

Creating sync channels via `mbsync`

My mbsync configuration is posted as a public gist. It is reasonably self explanatory, and shows how separate channels were made grouping together folders, by specifying a pattern. This took some time, but was finally very satisfying to know as a fine grained control technique.

I started out using offlineimap. I found mbsync to be significantly faster.

Text based email client! Speed + simplicity

Imagine being engrossed with your code or engineering notebook and the need for shooting off an urgent brief email arises. What if this could be done with a few key-presses on an email client, right from the terminal or the code editor that you are already engrossed in?

How about adding an email as a task in your organiser with a deadline / planned date?

What if I had the option to setup separate channels of mail transfer, such that I can sync the inbox or a custom group of folders alone when I am pressed for bandwidth or space?

Practical solutions will need to cater to a lot more situations.

The good news is: usually anything you need is possible (or already implemented) using Emacs.

I use mu4e, which uses the indexer mu as it's back-end. There are other popular options like notmuch and mutt. I have briefly experimented with mutt, which has a fast email search capability, but has to be coupled with another front-end to be used within Emacs or elsewhere. The philosophy and system behind notmuch (leveraging the Gmail tag based approach) differ from mu4e.

Over a few years of using this system, I have found that text and terminal based email clients offer a speed and integrity that is extremely pleasing.

Why mu4e rocks [for me] - the perks

The ability to create custom search filters (called bookmarks) that can be accessed with easy shortcuts. This is an example of bookmark configuration:

(setq mu4e-bookmarks
      `( ,(make-mu4e-bookmark
       :name  "Unread messages"
       :query "flag:unread AND NOT flag:trashed"
       :key ?u)
     ,(make-mu4e-bookmark
       :name "Today's messages"
       :query "date:today..now"
       :key ?t)
     ,(make-mu4e-bookmark
       :name "Last 7 days"
       :query "date:7d..now"
       :key ?w)
     ,(make-mu4e-bookmark
       :name "Messages with images"
       :query "mime:image/*"
       :key ?p)
     ,(make-mu4e-bookmark
       :name "Finance News"
       :query (concat "from:etnotifications@indiatimes.com OR "
              "from:newsletters@valueresearchonline.net"
              "from:value research")
       :key ?f)
     ,(make-mu4e-bookmark
       :name "Science and Technology"
       :query (concat "from:googlealerts-noreply@google.com OR "
              "from:reply@email.engineering360.com OR "
              "from:memagazine@asme.org"
              "from:action@ifttt.com"
              "from:digitaleditions@techbriefs.info")
       :key ?S)
         ))

This is how it looks:

Complete keyboard based control, and using it with Emacs means the ability to compose email from anywhere and build all kinds of workflows. Examples:

Hit Control+x and m (C-x m) in Emacs parlance, and I have a compose window open.
There are built-in workflows and functions in starter-kits like Scimax, which enable you to email an org-heading or buffer directly into an email, with the formatting usually preserved, and as intended.

I often use yasnippet to insert links to standard attachments like my resume. This essentially means being able to attach files with a 1-2 key strokes.

While Mu4e may be a programmatic solution with no pleasing GUI - it allows one to search a large number of emails with glorious ease. This is particularly more effective on a SSD drive, rather than the conventional Hard disk.

One has to experience the above to know the dramatic impact it makes in getting closer in speed to your thoughts, using a customisable system. Emails can be easily captured or added as tasks into Org mode documents as a part of task and project management.

Using the mu4e and mbsync, I've devised a 'sane inbox' which is bereft of the noise, like annoying digests, social media updates and so on. The idea was to dedicate focused blocks to rapidly process email, all within Emacs.

I have tried using Todoist extensively in the past, along with their integration with Gmail. This approach is a reasonable solution, if one is open to using different applications.

Quirks

mu4e is a text based email interface. It can be set such that the rendered HTML is displayed in the mu4e-view buffer for each email, which enables graphics and pictures (if any). However, the render is not perfect at all times. The HTML parsing engine can be specified. Thus, heavy HTML emails are unlikely to render correctly, to the extent of being a nuisance.

Such emails can be viewed in the browser of your choice with merely 2 key presses, 'a' and then 'v', with cursor in the body of the email. This could be Firefox, or w3m or any other browser of your choice.³

Email syncing frequency is set in mu4e. This update process takes a few seconds, and it is not as seamless as a web app. Notifications for new email can be configured on the mode line or through pop-ups in Emacs. However, the experience with working synced emails is good.

Multiple levels of filters are still necessary.

Situations where I do not have access to Emacs will need me to use the iOS app or the web interface. Therefore the inbox in the web interface here cannot be 'insane', and a higher level of filters need to be implemented in Fastmail itself.

For example all Linked in group and job updates have their own folders. These folders are all sub-folders of the Archive. They never reach the inbox at all. These emails often remain unread, or if necessary, I can focus on bunches of them at a time.

By grouping all such incoming mails into sub-folders within the Archive folder, I can use a single channel for all the relatively unimportant mail.

Takeaways

Using an 'archaic' text based email client (mu4e) has significantly boosted the speed with which I can handle my emails and focus on tasks. The simple interface and speed enables better focus.
While there are many articles and plenty of guidance on this topic, it takes time and patience to get this working the way you need it to. However, once it is setup, it does become rather comfortable to use.
Context switching is expensive on the brain and dents productivity.
Integrating email with time and project management is important. mu4e integrates well with Org mode. Beyond tasks, it is also a good reference, and I can easily attach notes, summaries etc to these emails.

Links and References

These are the links and references I've used in setting up and troubleshooting my email setup.

These could be organized better, and some links may be repeated. All put together, these should give you all you need to get hooked up!

Some of the links have additional comments, and many are tagged with dates, as a reference to when I collected the link. Sometimes, this is fun to reflect on!

A Complete Guide to Email in Emacs using Mu and Mu4e, <2017-03-08 Wed 10:04>
Reading IMAP Mail in Emacs on OSX | Adolfo Villafiorita, <2016-11-27 Sun 08:17>
[ ] Excellent link talking about mu4e and notifications Handling Email with Emacs – malb::blog, <2016-08-01 Mon 18:37>
Which email client (mu4e, Mutt, notmuch, Gnus) do you use inside Emacs, and why? : emacs <2016-05-31 Tue 07:32>
emacs-fu: introducing mu4e, an e-mail client for emacs - Emacs and mu4e stuff <2016-04-20 Wed 13:02>
Emacs as email client with offlineimap and mu4e on OS X / KG / Hacks. Thoughts. Writings. - nice blog related to Emacs and linux <2016-04-21 Thu 22:44>
EOS: Mail (Email) Module - explaining multiple email setup in mu4e <2016-04-27 Wed 07:56>
The Ultimate Emailing Agent with Mu4e and Emacs – Emacs, Arduino, Raspberry Pi, Linux and Programming etc, <2016-08-17 Wed 13:19>
Varun B Patil | EOM a.k.a End of Mail a.k.a Emacs + offlineimap + mu4e - multiple accounts <2016-04-19 Tue 12:19>
Master your inbox with mu4e and org-mode | Pragmatic Emacs <2016-03-26 Sat 14:56>
notmuch - email setup My personal mail setup — Articles — WWWTech <2017-06-13 Tue 16:09>
Search-oriented tools for Unix-style mail | Mark J. Nelson, <2017-05-10 Wed 16:29>
- interesting comparison of mu and notmuch, going beyond superficial differences, but not too much depth either.
Mutt with multiple accounts, mbsync, notmuch, GPG and sub-minute updates | French to English translator, <2017-04-28 Fri 07:19>
- interesting link, author profile and content available on-line.
Assorted Nerdery - Notmuch of a mail setup Part 2 - notmuch and Emacs, <2017-04-27 Thu 18:41>
Mutt, mu4e and notmuch links
mu4e specific Links <2016-04-19 Tue 21:48>

Footnotes

¹ Fastmail allows for a variety of interesting features like aliases, easy email transfer (from a different email provider like Gmail or MSN), responsive technical support, and many more aspects, and much more. They have their own implementation of the IMAP protocol, called JMAP, which is significantly faster.

² While there are many advantages in Gmail and many swear by it's search capabilities - it is worth noting that Fastmail's ad-free interface and search just feels a lot quicker than Gmail, and I can find my way around the settings better than I used to with Gmail.

³ You may be surprised to see the ease in browsing a good number of websites on a text based web browser. Besides the added advantage of being within Emacs - a surprising number of websites can be viewed functionally on w3m. It works fine for quick searches on Google (which like anything else, can be done within a few key strokes in Emacs).

Why bother with Emacs and Workflows?

Shreyas Ragavan — Thu, 11 Jul 2019 15:43:28 +0000

I've written several posts on different ways and tools available to aid productivity, and probably a lot about Emacs. My background is in computational physics, and not in programming, and yet Emacs has been an indispensable driver of my daily workflow for the past 3 years.

The fact is that knowing Emacs (or Vim), or having a custom configuration is not a wildly marketable skill, nor is it mandatory to achieve spectacular results. An Emacs configuration suits personal workflows and style, which may be borderline peculiar to another person. Such a dependence on customised tools would also drastically reduce your speed while using a new IDE or text editor.

So : why add Emacs to the ever-growing to-do list? The question is more pertinent considering that mastery of a 'text editor' is not something you can freely talk about and frequently expect empathetic responses or even a spark like connection. Emacs would be considered by many to be an esoteric and archaic software with a steep learning curve that is not beginner friendly.

However …..

This article elucidates many points where Emacs can help PHB's (Pointy Haired Boss). The internet abounds with several examples on how org-mode and Emacs have changed lives for the better. Here is another cool article by Howard Abrams on using Emacs as his (only) window manager, in place of a desktop environment.

Watching an experienced person handle his tools emphasises the potential art form behind it, especially when compared to the bumbling of an amateur. Yes, the amateur may get the job done given enough time, and depending on his capabilities - even match the experienced professional's output (eventually).

However, as experience is gained, the workflows and steps to achieve an optimal result become more lucid. I've experienced an exponentially increasing and compelling need to implement specific preferences to achieve the required optimized results faster and with fewer mistakes.

It is therefore obvious that the workflow and tools used must allow the provision to evolve, customise and automate. This is particularly true with respect to the world of data science and programming. I don't think there is anything better than Emacs with respect to customisation.

Imagine the following:

having a combination of scripts or snippets in different languages to jumpstart a project, which is available with a few keypresses? (Yasnippet)¹
Maintaining a blog with a single document, with articles updated automatically on a status change. (ox-hugo)
working with multiple R environments in a single document. (Org-babel, ESS)²
Different Window configurations and processes for different projects that can be called with a few keypresses (hint : keyboard macros)
An integrated git porcelain that can actually help you learn git so much faster (magit)
Intimately integrating email with tasks, projects, documentation and workflows (mu4e, Org-mode)
A customised text editor available right in your terminal (Use Emacsclient launched off a daemon within a terminal)
Not requiring to use the mouse for navigation!³

Now : imagine the consolidated effect of having all the above at your disposal, in a reasonably streamlined state. Then, considering the cumulative effect over multiple projects! The above is just a shallow overview of the possibilities with Emacs.

Investing in learning Emacs, has the serious potential to spawn exponential results in the long run.

Footnotes

¹ Articles on using Yasnippet: — Using Emacs Yasnippet against repetitive boileplate code || Tweaking Emacs Yasnippet || Expanding snippets

² Links to using R with Emacs: Using R with Emacs and ESS || Using R with Emacs || Tips from R Coders who use ESS || Why I use Emacs for R programming

³ See this article of a non-technical user's experiment with not using the mouse, reporting significant gains in speed and productivity. I've experienced this myself after gaining basic proficiency in moving around Emacs.

Rapidly accessing cheatsheets to learn data science with Emacs

Shreyas Ragavan — Sat, 02 Feb 2019 17:24:00 +0000

Matt Dancho’s course DSB-101-R is an awesome course to step into ROI driven business analytics fueled by Data Science. In this course, among many other things - he teaches methods to understand and use cheatsheets to gain rapid level-ups, especially to find information connecting various packages and functions and workflows. I have been hooked to this approach and needed a way to quickly refer to the different cheatsheets as needed.

Favio Vazquez’s ds-cheatsheets repo, akin to the One Ring to Rule them All (with respect to Cheatsheets of course), combined with Emacs (Projectile + Helm packages) make it almost a breeze to find a specific cheatsheet quickly, by just typing in a few words. ¹

The built-in Hydras in Scimax make it very easy to do the above with a few key presses. All I do is F12 >> p >> ww, start typing in “ds-” and choose the project and then start typing in the name of the PDF file I’m looking for. Check out the animation below.

Rapidly switching to a cheatsheet PDF

The above concept applies to switching to any file in any git based project that is added to Projectile’s lists.

The next aspect to consider was switching between maximized buffer of the opened cheatsheet PDF and the current code buffer. As it goes in Emacs, “there’s probably a package for that..” ! My solution was to use one of the various frame/window configuration packages in Emacs to save the position and orientation of the buffers and rapidly switch between the maximised PDF frame and the split code and interpreter frames.

Facilitating the above was in fact already available in Scimax, where a frame or window configuration can be saved into a register that is valid for that session. Persistent saving of window configuration across sessions (i.e Emacs restarts) is a little more complex, but it is still possible with some tweaking. Winner-mode is also an interesting option to switch rapidly between window configurations.

To some extent, it is also possible that launchers like the Alfred app could be set or programmed to search in particular locations. This is a less hacky and still a functional option for Mac users. ^{^}

Literate Programming - Emacs, Howard Abrams and Library of Babel

Shreyas Ragavan — Fri, 25 Jan 2019 21:44:00 +0000

I’m an admirer of Howard Abrams, especially because his posts and videos show the awesome power of doing things in Emacs, and the importance of writing clean and logical code. Watching his videos and reading his posts make me feel like I was born yesterday and I am just getting started. But more importantly, they also fire up my imagination regarding the possibilities out there and the potential to create glorious workflows.

Howard’s tutorial on Literate Programming, combined with his Literate Devops with Emacs video are among the best ways to get started with understanding the power of using Org Mode and Org-Babel to create complex, inter-connected, multi-language programs / documents / research that are of course well documented (this being one basic tenet of literate programming). Essentially, Org Mode and Org-Babel enable a high quality programming environment in a single Org mode buffer or document. The said environment is significantly more feature rich compared to Jupyter notebooks, especially being supported by it’s foundation in Emacs.

Though I’ve been using Org files for a while now for all my programming explorations, I’ve been bothered about my sub-par workflows. I could not easily reference other code blocks and snippets and recipes for a new document or project. It was inefficient and time consuming to locate the necessary snippet and re-write or re-paste the code in the new source blocks. I was not making much progress plodding through the vast documentation of org-babel.

Therefore, I was thrilled to discover the Library of Babel through Howard’s tutorial, which can be used to add files to a global library that is accessible from anywhere! Did I mention that it involves hitting barely 3 keys, and any number of arguments can be passed to these source blocks? I’m not sure such a feature is available with any other IDE.

In addition, the above tutorial clearly elucidates how different languages can be combined together, and the video elucidates typical Devops procedures, which are easily taken care of with appropriate arguments and headers to the source code blocks. For example, all the source code blocks could be tangled into appropriately named and located script files using a single argument. These tutorials tied up bits and pieces of info in my head from various sources and was invaluable in enhancing my understanding of using Emacs and Org-Babel

The Library of Babel can be made persistent across sessions by loading a specified org-file from which the named source code blocks are automatically read in. It is surprising that the internet does not seem to contain more references and examples using the Library of Babel. Perhaps there are some caveats that I am yet to encounter. One question that arises is whether the Library of Babel is automatically updated when the source code block is updated.

Searching the awesome-lists on Github

Shreyas Ragavan — Fri, 25 Jan 2019 21:38:00 +0000

Discovered the glorious awesome lists today on Github. They are available through a simple search on github, and contain curated lists of resources of all kinds on a multitude of topics.

As one might expect, there is a lot of common ground between these lists, including topics and links.

How could one search for a keyword through all these repositories? I have always wanted search for particular keywords or code snippets in my Emacs configuration files, or in other files in a particular location. This is especially to verify if a bit of code or note is already available, in another location. Something that looks like this ;):

Searching for ‘datascience’ with emacs-helm-ag through a bunch of awesome-lists and other local repositories.

An answer had been available in Howard’s cool blog post on why one should learn Emacs - in a footnote (!), in which he’s mentioned ackand ag (the silver searcher). ¹. It is even possible to edit in line with each search.

The silver searcher github page provides clear examples of how it’s significantly faster than ack (and similar tools). Further exploration led me to the emacs-helm-ag package, which is a helm interface to the silver searcher. Implementing emacs-helm-ag was as simple as adding it to my list of packages, and adding a basic setup to my helm configuration.¹

As of now, I add packages to Scimax using this bit of code that I’ve obviously borrowed from the internet, and this case - I’m afraid I did not note the source.

;; Setting up use packages
;; list the packages you want
(setq package-list '(diminish org-journal google-this ztree org-gcal w3m org-trello org-web-tools ox-hugo auto-indent-mode ob-sql-mode dash org-super-agenda ox-hugo workgroups2 switch-window ess ess-R-data-view interleave deft org-bookmark-heading writeroom-mode evil evil-leader polymode helm-ag))

;;fetch the list of packages available
(unless package-archive-contents
  (package-refresh-contents))

;; install the missing packages
(dolist (package package-list)
  (unless (package-installed-p package)
    (package-install package)))

;; Remember to start helm-ag. As per the Silver searcher github site, the helm-follow-mode-persistent has to be set before calling helm-ag.

(custom-set-variables
 '(helm-follow-mode-persistent t))

(require 'helm-ag)

This is how it looks in action >> Sweet !!

Notice the search across multiple files. So I’ve called require org capture perhaps more times than necessary.

This is my first animated gif in a blog post! It was tricky! I used the free GIPHY capture app on the Mac store. ^{^}

Notes from the movie Whiplash

Shreyas Ragavan — Sun, 20 Jan 2019 02:37:00 +0000

Whiplash: Wikipedia

Whiplash is a fascinating movie on many levels regarding a topic that interests me deeply… How to progressively perform, and strive to become the very best in a chosen field. Personally, I found each step of the movie riveting and would recommend it to anybody who would find the above question even mildly interesting. The movie’s climax was immensely interesting, inspiring and supported by great acting. At any rate, the movie induced a blog post !

The story revolves around the mind and life of a student who wants to be among the greats in his field, and the way he deals with an abrasive, abusive and unorthodox teacher whose intentions are to bring out the best in a student. No movie is perfect - while some points in Whiplash do appear extreme and therefore relatively unrealistic - the overriding message and theme conveyed certainly rings out clearly, in an engaging plot.

I could relate to the following pointers from the movie:

Leverage stress to achieve new levels of insight and performance

The belief of the teacher, that the best performance or attributes hidden inside a person can come out only via repeated, unexpected and stressful prodding. I’m not sure if this works as shown in the movie, but I have found unexpected insights at times of extreme stress, that have were taken forward to habits that changed my life.

Weathering criticism

The mental conditioning required to weather and beat intense, sharp, depressing criticism along with verbal and physical abuse from a mentor or teacher and use the same as a motive force for self-improvement and eventually superlative performance. Though there are examples of extreme abrasiveness in leaders like Steve Jobs - such an approach would not be tolerated by most people today.

I know other stories of people working under such mentors, striving to learn and gain their approval and eventually winning the same. These efforts paid off by resulting in skills, thinking patterns and a superior mental conditioning. Finding such a mentor at the formative stage is probably the best thing to happen to anybody.
An effective strategy to find mentors is to shadow people on Linked in and learn from their profiles and activity. Some of them may be willing to connect and invest time in mentoring.
Another possibility to find like minded people and mentors would be to join the communities of on-line courses, like Datacamp and Dataquest, who have lively channels in Slack for paid members.

Getting back up after a fall

Everybody breaks. Just as the promising student in Whiplash breaks. But the champions among us rally, to stage a comeback and performance that make history.

Regularly surpassing the level of deliberate knowledge of your own performance, and thus improvement by exactly being able to measure your performance and pinpoint mistakes. This point is portrayed in a very interesting manner in Whiplash, where the teacher expects the student to know exactly what mistake is being made.

Be Great, not Good

Rejecting the ‘Good’ or ‘Good enough’ feedback from anybody. The goal is to be Great, not good. The goal should be to strive to set the precedent and not just follow a beaten track. The pinpoint focus should be on progressive improvement to become the best, and that entails never being satisfied and to be ruthless in rooting out flaws.

Achieving Balance : mind + body + surroundings

Great performance is about that perfect balance between the body, mind and environment to leverage the best result possible. I view the scene where the student survives a car crash, just to reach a performance and then not being able to perform, as a good example of overreaching, without strengthening the core, and thus inviting instability.

Go off the beaten track and Lose yourself

It was the ending of Whiplash that truly drove me to comprehend the points so far. It is twisted, unexpected and led me to truly enjoy the movie and appreciate that: despite the above points being reasonably discernible - the human mind and nature is exceedingly complex. Stability and reasoning are not the only keystones to the foundation of greatness. There has to be a healthy mix of some form of abnormal obsession thrown in, to make a stellar performance what it is. However, can this be practically repeated on a regular basis?

Learning velocity and Flow

There are several bodies of research work available today that can be studied to get closer to consciously stimulating a great performance. One such example is:

Unlocking the Talent Code With Dan Coyle on the Unmistakable Creatives podcast provides an insight in line with the points seen above, into what constitute outliers and performers. The interesting concept of ‘Learning velocity’ is explained by Dan with a lucid example. It is surmised that progress and maximum learning to become better occurs at the boundary line dividing what we know at the moment, and the unknown skills that beckon.

That point sems to be an amalgamation of several factors, that are typically present when someone is in ‘flow’. Perhaps this flow can be described as a heightened sense of what is, and what should be and the energy to strive and achieve what should be.. It certainly does feel logical to think that we become better by pushing that boundary.

DEV Community: Shreyas Ragavan

Setting up Continuous Integration (CI) for docker containers

Plan [3⁄3]

Setting up a Github Runner [0/1]

Notes about build and queue

Setting up CI via Dockerhub and Github

General notes

Overview of components

On API limits

Self-hosted runners versus github hosted runners

- Provide a clean instance for every job execution.

CI

Oddmuse-curl exploration

A graphic overview of the 'binary' with respect to R packages

Implementing HTTPS : Let's Encrypt

What is Let’s Encrypt?

How Let’s Encrypt works

On Certbot [0/1]

Short Peek under the hood.

Conclusions

Federal R&D Spending on climate change

Summary

Loading libraries

Reading in the data

Exploring the data

Data Conditioning

Function to plot a facet grid of the department spending

Visualizing department-wise spending over the years

Some Concluding statements

mosh - for better access to my VPS

Did mosh make a difference?

'Archaic', text based email clients rock

Contents

How I unified my email accounts in 2019

Mohammed Ismail Ansari ・ Jul 12 '19

TLDR - for the busy folks

Goals:

Summary:

Multiple email accounts. Lack of a unified interface.

Creating sync channels via mbsync

Text based email client! Speed + simplicity

Why mu4e rocks [for me] - the perks

Quirks

Multiple levels of filters are still necessary.

Takeaways

Links and References

Footnotes

Why bother with Emacs and Workflows?

Footnotes

Rapidly accessing cheatsheets to learn data science with Emacs

Rapidly switching to a cheatsheet PDF

Literate Programming - Emacs, Howard Abrams and Library of Babel

Searching the awesome-lists on Github

Notes from the movie Whiplash

Leverage stress to achieve new levels of insight and performance

Weathering criticism

Getting back up after a fall

Be Great, not Good

Achieving Balance : mind + body + surroundings

Go off the beaten track and Lose yourself

Learning velocity and Flow

Plan `[3⁄3]`

Setting up a Github Runner `[0/1]`

On Certbot `[0/1]`

Creating sync channels via `mbsync`