DEV Community: Shubhojyoti

How I created my Gatsby blog with WordPress as the backend

Shubhojyoti — Fri, 02 Aug 2019 19:37:31 +0000

Recently, I created my blog again after junking my writings on my other domains. My previous experience with Static Site Generators (SSG) was Hugo. This time I used Gatsby as my primary work deals with JavaScript, so it was a quick development process for me. For the backend, I used WordPress.

I plan to write a multi-post series as I have lots of notes on how I went about it. There are still some CSS polishing required. But overall, the site is ready. This part has loads of rambling, so feel free to skip it.

Why do I use SSGs? - A history.

I don’t know about you, but I love static sites. There are no headaches involved. Just plain HTML and CSS with a dash of JavaScript if required. That is how I learnt about developing websites. It was later that I realised maintaining them is a headache. A single change in the theme required changing and keeping track of all the files. It was 2005 when I discovered WordPress. I was going to start college during the summer, and I wanted to use something for my journal. I stuck with WordPress for the next 8-10 years. My main issue was it was slow even though my blog received barely any traffic. I used the recommended caching plugins, which improved it. But it was always an extra step.

Then I found Hugo. I moved my [Enjoyable Recipes] blog in 2017. It was a mess. I learned Go language, even though it was not required. The build was super fast, and I loved the speed with which the site loaded. However, there were multiple breaking changes in later versions. Shifting mentally from JavaScript development at work to Golang was too much for me at that time. So during the current redesign, I considered a JavaScript-based SSG. I studied Gatsby and Eleventy. Ultimately I decided on Gatsby because I was working on two React projects at that time so this was a natural fit.

The Backend CMS

I started with markdown files, moved to Contentful, finally settling on WordPress. I love writing in markdown. My Hugo blog is entirely in markdown as there is no other option. For the current site, I had the following requirements. These are solely my requirements. YMMV.

Markdown Support - Contentful has native Markdown support. WordPress requires a plugin. Advantage Contentful.
Image Management - I didn’t want to check-in images to my repository. Also, I wanted automatic resizing wherever necessary. Netlify has support via Large Media. But I did not want to add LFS support. In case I decide to leave Netlify, I wanted the images resized with me. I considered Cloudinary too who have a generous free plan. But I did not want to add another provider. Contentful has automatic image management, which is fantastic. WordPress has Media Management and image resizing options via the plugin system. Advantage WordPress and Contentful.
Metadata for Images - Advanced Custom Fields (ACF) plugin for WordPress quickly add custom fields, and metadata support to posts, pages and images. Contentful only allows adding title, description and alt text. We need to create a separate content type for images and add extra fields. Advantage WordPress.
Categorization - Contentful supports categories and tags but in a round-about way. Everything needs to have a separate content type. WordPress has in-built support. Just type and press enter. Advantage WordPress.
Post Series Support - This is slightly difficult in Contentful and easier in WordPress. In Contentful, you can create references to other posts and then can link them in HTML. WordPress too does not support this natively. But it is easier (for me) using custom taxonomies. Tie between WordPress and Contentful.

Considering the above requirements, WordPress was a better fit for me. However, I loved Contentful. Its interface is a beauty. Their free plan, their image support and no need to manage installations make it an excellent CMS. I also considered Ghost. But they have no image resize support yet, so it was out.

Finally, I write my posts in markdown. I manage these in a private repository so that I never lose them if my WordPress installation gets corrupted.

Conclusion

I know that was loads of rambling. In the next part, I will write about my WordPress setup and finally begin the finale - Gatsby integration. What do you use for your website?

Originally published at Shubho.Dev

How I use JavaScript Promises

Shubhojyoti — Thu, 25 Jul 2019 17:38:24 +0000

Originally published at Shubho.Dev

Async programming in JavaScript was scary for me. The only async paradigm I was comfortable with was jQuery's $.ajax. However, I went full vanilla JavaScript for the past 8 years, and when I started working with NodeJS, I had to learn Promises. I haven't dabbled much with third-party libraries like Bluebird. I have the most experience with native Promise.

My main issue with Promise (or asynchronous paradigm in general) used to be when I wanted to execute statements after the Promise statement began. It took some time to realise that once a Promise statement fires, there is no way to cancel it. Another issue was Promise chaining. This one was a kicker. My earlier functions with Promises always looked like friends of callback hell. After all these years and working on a couple of big projects, I can safely say I love Promises. Even though async/await is the new fad, I still love Promises.

So here is how I use Promises to make my coding life simpler.

Create a Promise skeleton

Whenever I create a new function that returns a Promise, I create the skeleton first. Since the function cannot return anything other than Promise based values, I always wrap all the statements of the function within the Promise.

function sample() {
    return new Promise(function(resolve, reject) {
        // The function body
    });
}

The above sample function wraps its entire statement within the Promise, returning immediately. You can either resolve() or reject() the output you want from the body. This way, I never make a mistake of not returning a Promise. It also helps me in creating Promise chains. Whenever in a chain, I realise I need a new function, I create the skeleton name it appropriately and finish the main chain. Then I come back one by one and finish the individual functions.

Promise chains - Points to remember

Promise chaining is tricky. If we are not careful, we can have a new type of callback hell. An example:

function promiseCallback() {
    return new Promise((resolve, reject) => {
        aNewFunction()
            .then((values) => {
                someOtherFunction(values)
                    .then((someOtherValue) => {
                        // Do something
                        resolve(someOtherValue);
                    })
                    .catch((err1) => {
                        // Error in inner function
                        reject(err1);
                    });
            })
            .catch((err) => {
                // Error in outer function
                reject(err);
            });
    });
}

In the above sample aFunction() and someOtherFunction() are two functions returning Promises. If you see carefully, the sequence looks like a callback hell. The inner then and catch the chain, and outer ones are independent. We cannot handle errors in a common catch block, and we need to be careful that the inner functions are always the last line within their outer then() otherwise we can't control the execution flow.

A better way with chains:

function promiseCallback() {
    return new Promise((resolve, reject) => {
        aNewFunction()
            .then((values) => {
                return someOtherFunction(values);
            })
            .then((someOtherValue) => {
                // Do something
                resolve(someOtherValue);
            })
            .catch((err) => {
                // Error in outer function
                reject(err);
            });
    });
}

Returns within the then chain can only have three types:

Promise - A then function in a chain can return a Promise. It's result is passed to the next then.
Scalar Value - A then function in a chain can return a value like a String or a Number. This value is passed to the next then as is and the chain can continue.
Throw - A then function can throw an error, which moves the execution to the catch block.

As long as all your returns within a then follow the above three types, you shouldn't have issues following your Promise chain.

Note
Remember to always resolve() or reject() in the last then or catch of the chain.

When to create a new Promise function

Within a Promise chain, if there are multiple if-else conditions, and each condition can lead to different Promise results, it is an excellent time to create a new function that returns a Promise. This way, the Promise chain returns a single statement calling the new function.

Handling a scalar value or a Promise function in one step

Assume we have a function which gets the marks attained by a student using his roll number. However, the function either takes a roll number as an input or the name of the student. The marks can be attained from the DB only using the roll number. Here is some pseudo-code.

function getMarks(obj) {
    let rollNumberPromise = null;
    if ('rollNumber' in obj) {
        rollNumberPromise = Promise.resolve(obj.rollNumber);
    } else if ('studentName' in obj) {
        rollNumberPromise = getRollNumberFromName(obj.studentName);
    }

    if (!rollNumberPromise) {
        reject('Nothing worked');
    }

    rollNumberPromise
        .then((rollNumber) => {
            return get_marks_from_db(rollNumber);
        })
        .then((marks) => {
            resolve(marks);
        })
        .catch((err) => {
            reject(err);
        });
}

function getRollNumberFromName(studentName) {
    return new Promise(function(resolve, reject) {
        fn_to_get_roll_number_from_db(studentName)
            .then((rollNumber) => {
                resolve(rollNumber);
            })
            .catch((err) => {
                reject(err);
            });
    });
}

function fn_to_get_roll_number_from_db(studentName) {
    return new Promise(function(resolve, reject) {
        // some code
    });
}

function get_marks_from_db(rollNumber) {
    return new Promise(function(resolve, reject) {
        // some code
    });
}

getMarks(obj) takes an Object as an input. We create a local variable rollNumberPromise. If the rollNumber is already present, we save the value in the variable using Promise.resolve(). This creates a Promise which resolves when called with the value. If student’s name is sent, then we save the call to the function getRollNumberFromName(studentName) to the local variable. Calling rollNumberPromise.then() returns a rollNumber whether it is received from the DB or sent directly as input to the function. Using it this way ensures that getMarks() has a single Promise chain, rather than an if-else condition based on whether the input passed was a number or a name.

Invoke a Promise at the end

As mentioned before, once a Promise, once invoked, cannot be cancelled. Any statements which do not depend on the Promise output and which can be carried out independently without an async call should complete before you start a Promise chain in your function. Once a Promise chain begins, any subsequent steps must be within the then chain. The only exception to this is when you do not care of the Promise value, and you want the Promise to execute in the background while your primary function keeps running.

Conclusion

Promises are difficult. However, with practice and following some rules, it makes working with them a charm. I strictly follow the above rules, and I never go wrong with Promises these days. Find out what you are comfortable with and create your own rules.

Wordpress HTML to Markdown for Gatsby

Shubhojyoti — Mon, 22 Jul 2019 16:34:37 +0000

I am currently in the process of creating my blog using WordPress as the backend and Gatsby for the frontend. One of the most enticing features of Gatsby is plugins. Almost every feature you might want on your blog is available as a plugin, or you can create one for yourself. As a developer who has dabbled with WordPress plugins (but is not proficient in PHP) and knows JavaScript, I feel creating plugins for Gatsby is way easier. Of course, that is a biased opinion coming from me.

Gatsby source plugin for WordPress

Gatsby has many official plugins. Their structure is similar, but Gatsby does provide some standard terminology to make it easy to recognize the purpose for it. https://www.gatsbyjs.org/docs/naming-a-plugin/.

Initially, I decided to use Contentful for my backend, the plugin being gatsby-source-contentful (see how naming it following the standard convention helps). The Contentful plugin provides all the posts as a Markdown node in GraphQL, and as a result, all “transformation” plugins for “Remark” can be used on them. Now the transformation plugins for “Remark” for “transforming” markdown data are fantastic. And working on the Contentful data using them is a pleasure.

For getting data from WordPress into Gatsby, we use a “source” plugin gatsby-source-wordpress. I will discuss my reason for using WordPress in another post. But the main issue I faced with this plugin was it queries the data from the WordPress REST API and then creates the GraphQL schema for use within Gatsby. But the WordPress REST API by default returns the content only as HTML. So even if you write your posts as Markdown using some WordPress plugin (I use [WP Githuber MD]), the REST API will return the final content. However, this makes sense for WordPress as the output for their themes are always HTML. But I needed Markdown as I wanted to use those transformer plugins and they only work on the Markdown nodes. There are multiple Github issues on them like here https://github.com/gatsbyjs/gatsby/issues/6799. Even if a WordPress Markdown plugin exposes a separate REST endpoint, the Gatsby source plugin needed to support these. I didn’t want to find such a plugin or hack the official source plugin for Gatsby. 😀

Turndown - Convert HTML to Markdown

So I wanted to look for a solution which can convert HTML to Markdown. Since I am always a DIY guy, I started reading on ASTs and started writing a conversion from HTML to Markdown by myself. I spent three days and had a working version. But there were lots of bugs. I realized this was silly of me. There must be some package already. Enter [Turndown]. It was awesome. The conversion was almost perfect. So I junked my conversion library and instead went to write a local Gatsby plugin that takes a WordPress Post (or Page) node and creates a Markdown node out of it using Turndown.

The plugin `gatsby-transformer-wordpress-markdown`

I named the plugin as per the Gatsby naming standards. The folder “gatsby-trasformer-wordpress-markdown” goes under the plugins folder of your root Gatsby project.

The folder has 3 files:

├── gatsby-node.js
├── index.js
└── package.json

index.js only contains a line // noop .

package.json contains the name of the plugin and the turndown package as a dependency yarn add turndown and yarn add turndown-plugin-gfm.

The main workhorse is the gatsby-node.js.

In my main gatsby-config.js, I call the plugin as follows:

module.exports = {
    siteMetadata: {
       ...
    },
    plugins: [
        ...
        {
            resolve: `gatsby-transformer-remark`,
            options: {
                plugins: [
                    {
                        resolve: `gatsby-remark-reading-time`
                    },
                    {
                        resolve: `gatsby-remark-embed-gist`,
                    },
                    {
                        resolve: `gatsby-remark-prismjs`,
                        options: {
                            classPrefix: "language-",
                            aliases: {
                                javascript: 'js'
                            },
                            inlineCodeMarker: '>>',
                            showLineNumbers: false,
                            noInlineHighlight: false,
                            showLanguage: true
                        }
                    }
                ]
            }
        },
        ...
        {
            resolve: `gatsby-transformer-wordpress-markdown`,
            options: {
                turndownPlugins: ['turndown-plugin-gfm']
            }
        }
    ],
};

I haven’t added any tests as such as this is my local plugin. I might need to clean it up a bit. But here are a couple of points:

The plugin needs to tie in during the onCreateNode lifecycle of Gatsby build. In the current case, during the creation of a WordPress Post or Page node, the above plugin executes.
Turndown, by itself has a plugin system. I am using the turndown-plugin-gfm plugin. The plugin enables GitHub specific markdowns like tables in the Markdown Output. Line nos 26-35 are options you can pass to the local plugin. I am using all the defaults from the main turndown package.
For each WordPress Post and Page node created, the plugin extracts the HTML content, runs TurndownService against it and creates a Markdown child node of type MarkdownWordpress.
Since a new node of mediaType text/markdown is created, the gatsby-transformer-remark and its sub-plugins are run over them.

Caveats

In pure markdown nodes, the Markdown content is as you have written. However, note that in this case, WordPress has already created a HTML out of your post, and you are converting it back to Markdown. So if you use any special Markdown syntax, they will be lost. I did work around some of them as they were specific to my use case (I will write more on these in a future post), but YMMV.

Add a no-js class in Gatsby

Shubhojyoti — Tue, 16 Jul 2019 20:55:18 +0000

I have been working with GatsbyJS for the past week. I am trying to set up my new blog. My old blog used to be on WordPress. But after working with Hugo for my recipe blog, I fell in love with static site generators. I learnt HTML/CSS using static HTML sites. My first website contained folders of HTML pages. I loved that approach. However, I outgrew it when I started journaling. Copying the HTML template every time and then replace the latest writing was a pain. When I wanted a design refresh (which was often), it was painful. And that is how I discovered WordPress.

My initial foray with Static Site Generators (SSG) was Hugo. It had a steep learning curve for me, especially since I didn’t know template syntax in Golang. But I credit Hugo for spiking my interest in Golang, and I studied the language due to it (though you do not need to know Go to learn using Hugo).

However, the language I am most comfortable in is JavaScript. I was late to the React game (Angular is my first love). I started using it two months back. Once I got comfortable enough to code in React, I wanted to create my blog in Gatsby - static site, JavaScript, and React. I love it.

Problem Statement

Add a “no-js” class to the document head in a Gatsby site.

Why do it?

I have this terrible habit of checking websites by disabling JavaScript. Not because I want to find issues with the site but mostly to find the fresh ways people make their site. Some sites create wonderful fallbacks for their site, and it’s great to see the way they enhance the experience with JavaScript. And others make fantastic and stunning websites without JavaScript. With my Gatsby site, I wanted to see how my site will behave by disabling JavaScript.

Note - Do not disable JavaScript from your Developer Tools. That will not be an actual test of the site behavior as the service workers are still loaded. To altogether disable JavaScript, do it from the browser settings.

So I disable JavaScript and fire up my Gatsby site gatsby build; gatsby serve. Everything looks good. It does display that “This app works best with JavaScript” that Gatsby automatically adds. But other than that, all is well. However, some JavaScript-only functionality will not work. e.g., I had a theme switcher. It works with CSS Variables, but the actual switching and remembering of the old settings are JavaScript. So the switcher is there, but the user clicks on it, and nothing happens. Wouldn’t it be nice if all JavaScript functionality doesn’t even show up? Many sites handle this by adding a no-js class to the <head> element.

Issues galore

The no-js class is added to the HTML by default. JavaScript is used to replace it. The JS code is the following three liner.

window.addEventListener('DOMContentLoaded', function() {  
document.documentElement.className =  
document.documentElement.className.replace("no-js","js");  
});

I am using the react-helmet library to manage the header. Using the htmlAttributes property of <Helmet> I am already managing the class property. So I added the no-js class there. And I put the above JS code in a separate JS file, and I call it in gatsby-browser.js file. Does it work? No. Even though my JS file is sourced, react-helmet is dynamic. It fires and keeps the header updated even when I am replacing it from my static HTML. And worse, the no-js class remains every time so all my JavaScript-only behavior is gone even when it is enabled. So I cannot club no-js class to react-helmet as I am already managing it there.
I put the no-js as a body class. But, you guessed it, I am managing the body classes too using react-helmet. So that’s a no go.
As mentioned here [Customize Gatsby HTML] I copy over the html.js. I inlined the script in the <head> tag and manually add the class to the <head>. Doesn’t work as react-helmet removes the class. 😢

The Solution

So finally I figured it out. Since I want to continue using react-helmet to manage my head classes, I needed some other property to hold this value. So I add a data-html-class="no-js" to the <head> in html.js file. And then I inline the script, but instead of replacing class, I do a setAttribute. Here is how the template looks (only showing the HTML template) from html.js. The relevant parts are line number 3 and line numbers 21-28.

export default function HTML(props) {
  return ( 
    <html data-body-class="no-js" {...props.htmlAttributes}>
      <head>
        <meta charSet="utf-8" />
        <meta httpEquiv="x-ua-compatible" content="ie=edge" />
        <meta
          name="viewport"
          content="width=device-width, initial-scale=1, shrink-to-fit=no"
        />
        {props.headComponents}
      </head>
      <body {...props.bodyAttributes}>
        {props.preBodyComponents}
        <div
          key={`body`}
          id="___gatsby"
          dangerouslySetInnerHTML={{ __html: props.body }}
        />
        {props.postBodyComponents}
        <script dangerouslySetInnerHTML={{
            __html: `
            window.addEventListener('DOMContentLoaded', function() {
                document.querySelector('html').setAttribute('data-body-class', 'js')
            });
            `
        }}/>
      </body>
    </html>
  )
}

Now in CSS, I can have this.

html[data-body-class='no-js'] .only-js {
    display: none;
}

Whichever elements are JavaScript-only, I add the class only-js to them, and their display property is set to none when JavaScript is disabled.

Conclusion

HTML is awesome. JavaScript is awesome. Gatsby is awesome. I am in love.

Elastic Search Update by Query with Ingest Pipeline

Shubhojyoti — Sun, 14 Jul 2019 22:22:49 +0000

I should start with a confession. I love Elasticsearch. It feels fantastic to feed it any document and then search even the precise terms out of it. However, when I started creating and managing a cluster at work, it was a nightmare. I was a beginner, so lots of concepts took time to sink in. Most of the tutorials I found online were of version 1.x-2.x but ES was already at 5.x and just about started the 6.x line. The initial cluster was created on 2.x version and to upgrade it to latest 6.x line required an upgrade first to 5.x line and then finally to the 6.x line.

Since it was still just 1 month into the data feeding the data was not that huge, and the upgrades took 2-3 days (due to the reindexing involved). Our cluster is both index and query heavy. However, indexing is on a best-effort basis. The entire pipeline allows the system to index at various times four to five times. So as I was reindexing and upgrading my cluster, new data was still coming in the old cluster. Once the reindexing completed, I did another reindexing with only the new data. Also, any further data during this reindex was just thrown off. Because each indexing pipeline has around 5 tries at various intervals, eventually the dropped data came back into the new system.

The above process is easy when the overall data load in the cluster is less. Currently, our cluster now has around 15TB worth of data spanning a year, and it supports 3 different systems. Reindexing these days is out of the question. Most of the indices are hot and continuously being used.

Recently I had a requirement of updating a field of a couple of indices with a different value. Now the good thing was it didn’t need a change in the mapping. The field type did not change. It was just the data that needed to be updated.

As an example, assume the field name to be “jobIdentifier”. Most of the documents had this value as alphanumeric. However, some of the documents had entries like “213452-2-12932-<Alphanumberic>”. The requirement was to normalize these values to just <Alphanumberic>. Simple enough, right?

Updating Elasticsearch field value using regular expressions

The straightforward way to do this is regular expressions using Painless scripting language. From Elasticsearch docs,

Painless is a simple, secure scripting language designed specifically for use with Elasticsearch.

Using Painless scripting language for the above requirement, we can write the regular expression like this.

{
    "script": {
        "lang": "painless",
        "source": "ctx._source.jobIdentifier = /[0-9]+-[0-9]+-[0-9]+-(.*)/.matcher(ctx._source.jobIdentifier).replaceAll('$1')"
    }
}

So in the regular expression, it matches field “jobIdentifier” with the value of type [0-9]+-[0-9]+-[0-9]+-(.*) and replaces it with the grouped pattern .*. Thereby removing the preceding 3 numbers along with the hyphen separator. So a value like 213452-2-12932-Some_random_value becomes Some_random_value.

However, due to StackOverflow issues in Java, regular expressions in Painless scripting are disabled by default in the latest versions of Elasticsearch. [https://github.com/elastic/elasticsearch/pull/20427] The issue and its related links describe the fix in great detail. It convinced me to look into other solutions to the above issue. I didn’t want to enable regular expressions in scripting unless necessary.

Ingest Pipeline and Update by Query

Ingest nodes in Elasticsearch are used to pre-process documents before they are indexed. By default, all nodes in a cluster are ingest nodes. They can be separated if the ingest process is resource-intensive. Pipelines define the pre-processor. They contain a "description" and a "processor". The processor is a series of steps that are carried out on the document. They can contain Logstash grok patterns and scripting like Painless.

First, let us see the pipeline definition.

curl -X PUT "localhost:9200/_ingest/pipeline/my-pipeline-id" -H 'Content-Type: application/json' -d'
{
    description": "Used to update in place - Remove the alphanumeric ids from jobIdentifier in my_index index",
    "processors": [
        {
            "grok": {
                "field": "jobIdentifier",
                "patterns": [
                    "%{NEWFIELDIDS:eID}-%{NEWFIELDIDS:eSEQID}-%{NEWFIELDIDS:e2ID}-%{GREEDYDATA:REST}"
                ],
                "pattern_definitions": {
                    "NEWFIELDIDS": "[0-9]+"
                },
                "ignore_missing": true
            }
        },
        {
            "script": {
                "lang": "painless",
                "inline": "jobIdentifier = ctx.REST"
            }
        },
        {
            "script": {
                "lang": "painless",
                "inline": "ctx.remove('eID'); ctx.remove('eSEQID'); ctx.remove('e2ID');  ctx.remove('REST')"
            }
        }
    ]
}

The documents go through three processors in the pipeline.

"grok": {
    "field": "jobIdentifier",
    "patterns": [
        "%{NEWFIELDIDS:eID}-%{NEWFIELDIDS:eSEQID}-%{NEWFIELDIDS:e2ID}-%{GREEDYDATA:REST}"
    ],
    "pattern_definitions": {
        "NEWFIELDIDS": "[0-9]+"
    },
    "ignore_missing": true
}

Match the field “jobIdentifier”. Create a pattern named “NEWFIELDIDS” which has 1 or more numbers. If the field has a sequence of “Numbers-Numbers-Numbers-Other data”, then each number set is assigned a field name (eID, eSEQID, e2ID). Match the rest of the pattern until the end and assign this to field “REST”. This match is done using the inbuilt type “GREEDYDATA” from the Logstash grok patterns. At the end of this step, add the four new fields (eID, eSEQID, e2ID, REST) to the document. However, as mentioned before, there can be documents where the above field pattern might not match. In such cases, the pipeline stops. We do not want that. Hence we add “ignore_missing” as "true".

"script": {
    "lang": "painless",
    "inline": "jobIdentifier = ctx.REST"
}

Now we run a Painless script. We take the newly created field “REST” and reassign the “jobIdentifier” field as this value.

"script": {
    "lang": "painless",
    "inline": "ctx.remove('eID'); ctx.remove('eSEQID'); ctx.remove('e2ID');  ctx.remove('REST')"
}

Finally, we remove the newly created fields in step 1 (eID, eSEQID, e2ID, REST) as we do not need them anymore.

The explanation looks enormous, but we basically matched the pattern, updated the field, and then removed the extra new fields created.

Finally using the newly created pipeline (identified by “my-pipeline-id”), we run an Update by Query on the index. We match the documents whose fields match the requisite pattern and pass them through the pipeline. We can use multiple indices in the field to run documents from multiple indices. We use wait_for_completion as false, so the query returns immediately and the update happens in the background.

curl -X POST "localhost:9200/my_index/_update_by_query?pipeline=my-pipeline-id&wait_for_completion=false"
{
    "query": {
        "bool": {
            "must": [
                {
                    "regexp": {
                        "jobIdentifier": "[0-9]+-[0-9]+-[0-9]+-(.*)"
                    }
                }
            ]
        }
    }
}

Conclusion

We might not need the expensive regular expressions in scripting while matching/updating fields in Elasticsearch. Instead of enabling it blindly in our cluster, we can first try a pipeline and "update by query", which can certainly meet our needs in most common cases.

Networking Basics - RIP

Shubhojyoti — Sun, 07 Jul 2019 17:07:20 +0000

One of the first routing protocol that you would ever learn is the Routing Information Protocol (RIP). It is an Interior Gateway Protocol (IGP), and it uses a distance-vector algorithm called Bellman-Ford Algorithm. Here are my brief notes on what the protocol is and how it works.

The protocol has two versions RIPv1 and RIPv2. Both are in use today and have significant differences. UDP is used to transmit updated. The UDP port is 520. The IPv6 version is called RIPng.

Neighbors exchange their complete routing table at regular interval of 30 seconds (default). This is called the update timer. There is no session formation between RIP neighbors. Once RIP is enabled on a network subnet, it immediately starts sending the RIP updates.

Differences between RIPv1 and RIPv2

RIPv1	RIPv2
Update Message is sent to local subnet broadcast address viz. 255.255.255.0	Update Message is sent to multicast address 224.0.0.9
No authentication support is available	Support available for both plain text and MD5 authentication.
VLSM support not present. Hence RIPv1 is classful. Only the default subnet masks of Class A (/8), Class B (/16) and Class C (/24) addresses are used. These subnet masks are not sent in the updates. When the routes are received, the masks are derived from the addresses.	Supports VLSM. The subnet mask is carried in the updates and hence RIPv2 is classless. Also RIPv2 can still come up when a network is discontiguous due to this feature.
Doesn’t allow a router to advertise a different next-hop router than itself.	Supports Next Hop field in the Update message. Hence a router can specify another router as a next-hop.

RIP Timers

RIP uses four different timers which help in maintaining the routing table and helps in avoiding loops. All these timers are per route. These are:

Update timer (30 secs) – This is the interval which states how often the update for a specific route must be sent.
Invalid timer (180 secs) – This is the interval which states that once an update is received for a route how long to wait till the route is considered invalid or non-reachable.
Hold down timer (180 secs) – If a route gets a worse metric and is updated, the hold down timer is started. During this time no other metric for the route is accepted which is equal or worse than the current metric.
Flush or Garbage timer (240 secs) – This is the interval which states how long to wait since the last valid update before removing the route from the routing table.

Whenever a valid update is received from a neighbor, the update, invalid and flush timer resets to 0. The hold down timer resets to 0 whenever a metric value decreases.

RIP uses hop count as the metric. So in case, a prefix is received from multiple paths, the prefix with a smaller no of hops will be preferred. RIP allows up to 15 hops. 16 hops and above are considered to be infinite, and those routes are deemed to be unreachable. This limit in the number of hops prevents loops. Unlike other protocols like OSPF and ISIS, which are link state protocols and whose implementation itself ensures loop-free network, RIP uses some specific concepts which help in loop prevention. This makes RIP a great protocol in terms of implementation, but this also introduces a more substantial convergence time.

Loop Prevention Concepts in RIP

Split Horizon – Even though in RIP the entire routing table is sent to the neighbors, RIP omits the routes which has been advertised by the neighbor to whom the update is being sent. In other words, if the outgoing interface of a route matches the interface out of which the new update is to be sent, then that route is omitted.
Route Poisoning – When a route’s reachability fails, the update is sent with the metric of that route as 16 (infinite) thereby specifying to the neighbors that the route is unreachable.
Poison Reverse – In this case, the route with an infinite metric is advertised out the same interface through which it is received with the same infinite metric. Split Horizon with Poison Reverse is used together in most of the cases.

Originally published on Shubho.Dev

Featured Image courtesy Jordan Harrison at Unsplash

Backup NextCloud data to Dropbox

Shubhojyoti — Fri, 05 Jul 2019 23:04:03 +0000

Dropbox is the primary cloud sync provider in my family. All our personal documents, family photographs, code projects etc. live there. Folder common to both my wife and I are shared between our accounts. Everything else remains separate. We also backup the data to external hard drives. My personal MacBook also uses Backblaze. We overdo our backup strategy due to a data loss incident couple of years back.

We also have Nextcloud on a Debian system in our home network. It suffices as our media centre. I wanted to create a backup system for the Nextcloud installation. I didn’t want to use S3 or anything else at this point as our Nextcloud instance is not a huge data hog. So I am using my existing Dropbox instance for this. Nextcloud allows Dropbox to be used as an external storage. But I use my local HDD as the storage for Nextcloud and then rsync it to a Dropbox folder.

Initial setup for Dropbox

On the host machine (where Nextcloud is installed), install Dropbox. This process will vary based on your operating system.
Create a folder “NEXTCLOUD-BKP” in Dropbox. Make it a shared folder if required.
Using the Selective Sync feature remove all other folders except the newly created one. Since the primary purpose of my Debian system is Nextcloud, I am only syncing the backup folder for Dropbox and not any other data.
This step is specific for my setup. My Nextcloud instance uses Apache web server and runs as www-data user. But my Dropbox is installed as the main logged in user. In order to enable me to copy the data owned by www-data user, add the logged in user to www-data group. usermod -a -G www-data <LOGGED_IN_USER>. Make the permissions of the “NEXTCLOUD-BKP” folder to g+rwx. chmod g+rwx ~/Dropbox/NEXTCLOUD-BKP. Ensure that the Dropbox and “NEXTCLOUD-BKP” folder has “x” bit set to enable the www-data user to copy to its subfolders. https://askubuntu.com/questions/455000/group-permissions-allow-but-still-get-permission-denied

Enable maintenance mode on Nextcloud

This locks the sessions of existing logged in users to prevent inconsistency in the database. It also prevents new logins. php $NEXTCLOUD_PATH/occ maintenance:mode --on.

Copy the Nextcloud data using rsync

You can either copy the whole Nextcloud folder or just the data, config and themes folder.

rsync -Aavx $NEXTCLOUD_PATH/data ~/Dropbox/NEXTCLOUD-BKP
rsync -Aavx $NEXTCLOUD_PATH/config ~/Dropbox/NEXTCLOUD-BKP
rsync -Aavx $NEXTCLOUD_PATH/themes ~/Dropbox/NEXTCLOUD-BKP

Backup Nextcloud database

My Nextcloud database is MariaDB. If you use PostgreSQL or SQLite, then this step will be different for you.

# Database name is nextcloud
/usr/bin/mysqldump --defaults-extra-file=$MYSQL_CREDENTIALS_FILE --single-transaction --databases nextcloud > ~/Dropbox/NEXTCLOUD-BKP/database/nc-db-bkp_`date +"%Y-%m-%d-%H-%M"`.bak

The $MYSQL_CREDENTIALS_FILE provides the username and password for the next cloud database. https://dev.mysql.com/doc/refman/5.5/en/option-files.html nextcloud is the name of the database.

Update permissions of the backup folders created/updated

This step will depend on your setup and might be completely optional. Since I have different user running nextcloud, I need to do this for Dropbox to read the newly created files.

chmod -R 777 $DROPBOX_NEXTCLOUD_PATH/data
chmod -R 777 $DROPBOX_NEXTCLOUD_PATH/config
chmod -R 777 $DROPBOX_NEXTCLOUD_PATH/themes
chmod -R 777 $DROPBOX_NEXTCLOUD_PATH/database

Since my installation is within my home network and is behind a firewall and opens only the required ports, I feel safe doing the above. YMMV.

Disable maintenance mode on Nextcloud

Once all the above steps are done, disable the maintenance mode. php $NEXTCLOUD_PATH/occ maintenance:mode --on.

Setup a cron

I run the script once a day at 9PM. 0 21 * * * nextcloud-backup.sh. The script must run as the www-data user or the user running the nextcloud process. Here is the complete script.

#!/usr/bin/env bash

# Can set the below three as environment variables or uncomment to set actual paths
NEXTCLOUD_PATH=<NEXTCLOUD_INSTALLATION_PATH>
DROPBOX_NEXTCLOUD_PATH=<PATH_TO_NEXTCLOUD_BACKUP_FOLDER_IN_DROPBOX>
MYSQL_CREDENTIALS_FILE=<MYSQL_CREDENTIALS_FILE_CONF_FORMAT>

PHP_PATH=`which php`
MYSQLDUMP_PATH=`which mysqldump`
# Enable Maintenance mode
$PHP_PATH $NEXTCLOUD_PATH/occ maintenance:mode --on

# Backup data, config and themes from nextcloud directory.
# Can sync the entire nextcloud directory if needed
rsync -Aavx $NEXTCLOUD_PATH/data $DROPBOX_NEXTCLOUD_PATH
rsync -Aavx $NEXTCLOUD_PATH/config $DROPBOX_NEXTCLOUD_PATH
rsync -Aavx $NEXTCLOUD_PATH/themes $DROPBOX_NEXTCLOUD_PATH

# Backup MySQL
$MYSQLDUMP_PATH --defaults-extra-file=$MYSQL_CREDENTIALS_FILE --single-transaction --databases nextcloud > $DROPBOX_NEXTCLOUD_PATH/database/nc-db-bkp_`date +"%Y-%m-%d-%H-%M"`.bak

# Update permissions of the backup path to 777 (Because the user running nextcloud is different.)
chmod -R 777 $DROPBOX_NEXTCLOUD_PATH/data
chmod -R 777 $DROPBOX_NEXTCLOUD_PATH/config
chmod -R 777 $DROPBOX_NEXTCLOUD_PATH/themes
chmod -R 777 $DROPBOX_NEXTCLOUD_PATH/database

# Disable Maintenance mode
$PHP_PATH $NEXTCLOUD_PATH/occ maintenance:mode --off

This provides a good setup for me and my Nextcloud data is backed up to Dropbox. Please note that the mysqldump is not incremental. So you might need to delete older files if required. I keep 5 days backup only as the actual data folder is completely overwritten by rsync. You can update the rsync commands by creating an archive and saving instead of actual copy.

Originally published on Shubho.Dev

Featured Image courtesy Dirk Wouters at Pixabay

DEV Community: Shubhojyoti

How I created my Gatsby blog with WordPress as the backend

Why do I use SSGs? - A history.

The Backend CMS

Conclusion

How I use JavaScript Promises

Create a Promise skeleton

Promise chains - Points to remember

When to create a new Promise function

Handling a scalar value or a Promise function in one step

Invoke a Promise at the end

Conclusion

Wordpress HTML to Markdown for Gatsby

Gatsby source plugin for WordPress

Turndown - Convert HTML to Markdown

The plugin gatsby-transformer-wordpress-markdown

Caveats

Add a no-js class in Gatsby

Problem Statement

Why do it?

Issues galore

The Solution

Conclusion

Elastic Search Update by Query with Ingest Pipeline

Updating Elasticsearch field value using regular expressions

Ingest Pipeline and Update by Query

Conclusion

Networking Basics - RIP

Differences between RIPv1 and RIPv2

RIP Timers

Loop Prevention Concepts in RIP

Backup NextCloud data to Dropbox

Initial setup for Dropbox

Enable maintenance mode on Nextcloud

Copy the Nextcloud data using rsync

Backup Nextcloud database

Update permissions of the backup folders created/updated

Disable maintenance mode on Nextcloud

Setup a cron

The plugin `gatsby-transformer-wordpress-markdown`