DEV Community: Patrick Sevat

Optimize your git clone / fetch strategy for CI pipelines

Patrick Sevat — Sat, 26 Sep 2020 21:33:38 +0000

Over the last week I've been working hard on migrating our Frontend Monorepo from a on-premise Jenkins pipeline to a cloud based solution.

One of the big obstacles I had to overcome was dealing with the sheer size of our monorepo. If we include the whole history (all branches on remote, all tags, full history), the size of our repo amounts to more than 1GB.

Downloading that much data on each run is a waste of bandwith and time, so I've dived into the git documentation and found several possibilities that I will share with you.

You only need to checkout / build a single branch

`git clone --single-branch`

This simple flag (--single-branch) makes sure you only fetch the history of your main branch. The more active branches live on your remote, the bigger the benefit and with more than 400 active contributors in our repo, that helps!

`git fetch --no-tags`

Another simple flag! This one omits the tags from the git history. Tags are often version numbers such as <package-name>@1.2.67. Although not a huge data saver, it helps. Be careful though, some CI pipelines need tags, so double check if you need it.

`git fetch --depth=<n>`

Oh yes, now we are talking. This flag restricts the history to the latest commits. Really useful if you only care about the newest state of a certain branch. For example if you know you need to build a feature branch (or your main branch) and you don't need all it's history! Potential huge data savings 🤤.

Note: If your depth is quite large, the server will have to do a lot of calculations increasing transfer time. A depth of 50000 increased time by 300%. A depth of 1 cut time with 90%.

`git fetch --depth=1 origin <full_SHA1>`

Used when you need a single commit that is not at the tip of a branch, without pulling in all history of that branch. See this StackOverflow question for details

Combining the flags `git clone --no-tags --single-branch --depth=1`

If you just need the latest state of a single branch this is your command! You can also modify this command to checkout a specific branch by adding --branch=<branch-name> if you are building a branch that is not your main.

You need to more than one branch (but not all)

This is where things become a bit more tricky. In this scenario git clone might not be our best bet because it only allows for cloning all branches or one branch.

This was also the scenario which was most tricky for me, because I needed to determine a single "ancestor" commit where a feature branch started to branch of the main branch.

`git clone --depth=<n>`

This command fetches all active branches with limited history. This might work for you if you are interested in the most recent state of two (or more branches).

Downsides of this command is that it still fetches all branches and that determining "ancestor" commits is difficult/impossible (depends on depth, but can still be inaccurate).

Custom git config

This one might seem scary, but can actually be really effective. You have full control over the number of branches you want to track, if you would like tags and still control depth of the history

Steps

Initialize an empty git repository by running git init

Add the following snippet to your .git/config file

# .git/config
[core]
        repositoryformatversion = 0
        filemode = true
        bare = false
        logallrefupdates = true
[remote \"origin\"]
        # replace with your own repo!
     url = git@github.com:patricksevat/foo.git

        # list the branches that you want to track
     fetch = +refs/heads/main:refs/remotes/origin/main
        fetch = +refs/heads/development:refs/remotes/origin/development
        fetch = +refs/heads/feature:refs/remotes/origin/feature

# Replace main with master or whatever your main branch name is
[branch \"main\"]
        remote = origin
        merge = refs/heads/main

Now you can git fetch with whatever flags you like.
Do not add any flags for full history of the listed branches, or use --no-tags or --depth=1 to limit history.
Don't forget to run git checkout with one of your fetched branches or you'll have an empty working directory!

Conclusion

I hope these commands help you in optimizing your own git CI strategy. If you have any useful tips or improvements please leave them in the comments!

How to use Web Components in HTMLTemplateElement in Angular

Patrick Sevat — Sun, 19 Jul 2020 22:14:56 +0000

TLDR

Use [innerHTML] on the <template> tag
Use DomSanitizer.bypassSecurityTrustHtml() to make sure your Custom Elements are not stripped from your provided HTML string

See code example at the end

Within the Rabobank Design System (based on Web Components) we sometimes come across with unusual specs. Our department relating to wholesale banking (large business customers) came up with the requirement of a select dropdown containing thousands of bank accounts.

We also envisaged different use cases than just thousands of bank accounts. Maybe thousands of checkboxes with labels and icons. To allow for future use cases we wanted to leverage the <template> component, also known as HTMLTemplateElement.

The cool thing about the <template> tag is that its contents are not actually rendered by the browser as long as they reside within <template>. This would give us the flexibility we need for our component.

Our components worked fine in plain HTML / JS but once we appended the components within our <template> tag to the DOM using Angular, it started double rendering! 😠

When inspecting the <template> element we also noticed that in Angular it did not yield a new DocumentFragment as it does in plain HTML...

This means the <template> tag was not recognized as such and because our Web Components used slots, those slots where rendered and then re-rendered upon appending to the DOM.

Unfortunately, searching Google for angular + template only yields results for ng-template, but after searching for Angular HTMLTemplateElement we got to this StackOverflow question, which point us to the [innerHTML] syntax.

After trying binding to the innerHTML property we noticed that the double rendering stopped, but the Web Components within the <template> tag were not rendering as they should, but with an example consisting of HTML5 elements (span, div, p...) it did render as expected.

There were two possible explanations:

The Web Components were not registered correctly.
- A quick inspection of the CustomElementRegistry showed that they were registered
There is some sort of sanization in play that strips out "invalid" elements because of the usage of innerHTML

The sanitation turned out to be the culprit. By using the DomSanitizer we were able to mark our HTML as safe and get our code working as expected.

Code example

/* some-component.component.ts */
import { DomSanitizer } from '@angular/platform-browser';

@Component({
    selector: 'some-component',
    templateUrl: './some-component.component.html',
})
export class SomeComponent {
    constructor(private _sanitizer: DomSanitizer) {}

    templateHtml = this._sanitizer.bypassSecurityTrustHtml(`
        <webcomponent-bank-account>
            <span slot="label">John Doe</span>
            <span slot="balance">€ 100.000</span>
            <p slot="account-number">1234567890</p>
        </webcomponent-bank-account>
    `);
}

<!-- some-component.component.html -->
<template #webComponentTemplate [innerHTML]="templateHtml">
</template>

Wrapping up: useful plugins for Gatsby

Patrick Sevat — Thu, 27 Feb 2020 21:42:40 +0000

Illustration by Katerina Limpitsouni undraw.co/illustrations

Typescript in Gatsby

My starter template did not provide Typescript out of the box, luckily there's a plugin for that 🎉. Just install gatsby-plugin-typescript and add it to your plugin array. You can now either rename your .js(x) files to .ts(x) with no further config needed.

Search functionality in Gatsby

This would be one of the domains where beforehand I'd expect GraphQL to excel, however, GraphQL is not available in the browser... d'oh!

To get search working browser side there are 2 approaches:

API based using Algolia and its Gatsby plugin
Creating an index and querying in the browser ("local search"). I took this approach and used the @gatsby-contrib/gatsby-plugin-elasticlunr-search plugin.

❗ Be aware that for websites with lots of websites the created index can get bloated really, really fast. Keep an eye on your bundle size!

Pagination in Gatsby

For my "all blogs page" I wanted to add pagination. For this to work you do not need an external plugin, but you can follow the approach described on the Gatsby docs. A quick gist from gatsby-node.js:

exports.createPages = ({ actions, graphql }) => {
  const { createPage } = actions

  return graphql(`
    # big GraphQL query
  `).then(result => {
    const posts = result.data.allMarkdownRemark.edges;
    const blogs = posts.filter(post => post.node.frontmatter.templateKey === 'blog-post');
    const blogsPerPage = 6
    const numPages = Math.ceil(blogs.length / blogsPerPage)

    Array.from({ length: numPages })
    .forEach((_, i) => {
      createPage({
        path: i === 0 ? `/blog` : `/blog/${ i + 1 }`,
        component: path.resolve("./src/templates/blog-overview.tsx"),
        context: {
          limit: postsPerPage,
          skip: i * postsPerPage,
          numPages,
          currentPage: i + 1,
        },
      })
    });

    // more code
  })
}

Styling SVG in Gatsby

Using SVG files works out of the box in Gatsby via Webpack. However, the drawback is that Webpack converts smaller SVG files into a data-URI. This works fine, but the downside is that you can no longer target the SVG file in css, for example to change the fill property.

To overcome this I've installed gatsby-plugin-react-svg plugin which I've configured to include this RegEx /\.inline\.svg$/;

import Paperclip from '../img/paperclip-logo.inline.svg';

const FooComponent = () => {
  return <Paperclip title="an paperclip icon" />
}

Image optimalization in Gatsby

One of the most useful plugins that came with the Gatsby + Netlify CMS starter template are the gatsby-transformer-sharp and gatsy-plugin-sharp plugins.

These plugin autogenerate images in the newer webp format which takes less size while retaining quality. Furthermore, it can also generate smaller versions of images if you know that full-size is not needed (for example in a card component). And for places where you do want larger sizes, the plugins can also create an srcset which allows browsers to pick the most appropriate size.

PWA support in Gatsby

We all want that 100 in Lighthouse, right? To get the 100 in PWA we must enable two plugins: gatsby-plugin-manifest and gatsby-plugin-offline. Further reading can be done here.

Big thanks to all readers ❤! I'd love to hear your thoughts in the comments or via Twitter!

This blog is part of a series on how I migrated away from a self-hosted Drupal blog to a modern JAM stack with Gatsby and Netlify CMS.

So... what database does Gatsby use?

Patrick Sevat — Thu, 27 Feb 2020 21:42:26 +0000

Recently I started migrating an older Drupal based blog to a combination of Gatsby and Netlify CMS (source), both of which I had no prior experience with. In this blog series I'll talk you through the experiences, hurdles and solutions. In part 5 I'll answer one of the questions that kept nagging me during development: what database does Gatsby use to query from GraphQL?

GraphQL queries from databases right?

In my previous understanding, one of the defining features of GraphQL is being able to query efficiently from multiple datasources. I always understood these datasources can be REST APIs, other GraphQL APIs or databases such as SQL and MongoDb.

Thus arised my question, what database does GatsbyJS use?

No database, but Redux

Gatsby actually does not use a database at all. Under the hood everything that is processed by the plugins gets transformed into Nodes and added to a nodes collection in Redux.

Other collections that exist in Redux are: pages, components, schema, jobs, webpack and metadata.

The combination of plugins and actions (such as those defined in gatsby-node.js like createPage()) will populate these collections.

Redux stores these collections in-memory during build / dev time and GraphQL queries against these collection based on an automatically generated GraphQL Schema.

The in-memory approach also means that data is not persisted! A regular database would write its data to a database file on a system. In-memory databases recreate the whole database every time. Gatsby does provide a caching system to speed up subsequent builds though.

The schema itself is also stored in Redux and used to execute GraphQL queries that query other Redux collections.

That's quite some Redux-ception!

There is still one problem to be solved. Redux collections store plain Javascript, but GraphQL queries are written in using the GraphQL Syntax. The solution is using sift.js. Further reading can be found here.

The in-memory approach does have some drawback at scale (100k+ documents)...

Next up, LokiJS

To address the scaling issues, the Gatsby team are investigating replacing Redux with the LokiJS library. This storage / retrieval mechanism is still hidden behind a feature flag. LokiJS is still an in-memory database, but promises better performance.

If you'd like to know more, take a peek in the source code and follow the code 🔎.

That's it! Hopefully you understand now where the data comes from that is being queried by GraphQL!

This blog is part of a series on how I migrated away from a self-hosted Drupal blog to a modern JAM stack with Gatsby and Netlify CMS.

How to retrieve all fields of a Netlify CMS Relation in Gatsby

Patrick Sevat — Thu, 27 Feb 2020 21:42:10 +0000

Illustration by Katerina Limpitsouni undraw.co/illustrations

Recently I started migrating an older Drupal based blog to a combination of Gatsby and Netlify CMS (source), both of which I had no prior experience with. In this blog series I'll talk you through the experiences, hurdles and solutions. In part 4 I'll explain Netlify CMS relations and how to make sure you can retrieve all fields of the linked relation.

❗ This blogs assumes basic understanding of Gatsby and GraphQL queries, read this blog first if you do not feel comfortable yet.

Setting up a relation in Netlify CMS

If you have read my previous blog, you've set up your content collections and fields by now.

One of the widgets I really like is the relation widget. This allows you to link two seperate entities. If you are familiar with SQL it's a bit like a join. In this blog I'll link my blogs collection with my authors collection.

The configuration is rather simple:

# ./static/admin/config.yml
collections:
  - name: "blog"
    label: "Blog"
    folder: "src/pages/blog"
    fields:
      - {label: "Template Key", name: "templateKey", widget: "hidden", default: "blog-post"}
      - {label: "Title", name: "title", widget: "string"}
      - {label: "Body", name: "body", widget: "markdown"}
      - label: "Author"
        name: "author"
        widget: relation
        collection: "author"
        displayFields: ["name"]
        searchFields: ["name"]
        valueField: "name"
     # other fields omitted for brevity
  - name: "author"
    label: "Authors"
    folder: "src/authors"
    identifier_field: "name"
    fields:
      - {label: "Template Key", name: "templateKey", widget: "hidden", default: "author"}
      - {label: "Name", name: "name", widget: string}
      - {label: "Job title", name: "job_title", widget: string}
      - {label: "Profile picture", name: "profile_picture", widget: image, allow_multiple: false}
     # other fields omitted for brevity

Everywhere you query your blogs using GraphQL in Gatsby you will now receive a field with the author's name. But what if you do not only need the name, but also want to display the author's job title and profile picture? Well... that doesn't work out of the box.

Retrieving all fields of a `relation`

On my blog pages I want to show all fields relating to its author, so I had to work around this limitation of Netlify CMS. In this section I'll show two approaches to make this work. If you know a better way, let me know in the comments!

Populating browser side

The first, more naive route I took was to enhance the graphql query on each page where I also retrieved blogs, by quering the allMarkdownRemark collection again (under the alias authors) to retrieve all authors.

A shortened version of that query:

query BlogPost {
  allMarkdownRemark(id: { eq: $id }) {
    html
    frontmatter {
      title
      author
      # other fields omitted for brevity
    }
  }
  authors: allMarkdownRemark(filter: {frontmatter: {templateKey: {eq: "author"}}}) {
    nodes {
      frontmatter {
        name
        job_title
        profile_picture
      }
    }
  }
}

Now that all the authors are available you can retrieve the author of the blog in the browser. This could be done during render of your component:

// src/templates/blog.js
const BlogPost = (props) => {
  const post = props.data.markdownRemark;
  // ❗ Not very efficient: providing all authors then looking one up. 
  const author = props.data.authors.nodes
    .find(author => author.frontmatter.name === post.frontmatter.author);

  return (
    <div>
      <h1>{ post.frontmatter.title }</h1>
      <AuthorComponent {...author} />
      <div dangerouslySetInnerHTML={{ __html: post.html }} />
    </div>
  )
};

export default BlogPost

This author retrieval could be optimized by utilizing a [useMemo](https://reactjs.org/docs/hooks-reference.html#usememo) hook.
However, this approach has still several downsides:

this goes against the GraphQL principle of querying only what you need
all the unnecessary data is added to your javascript chunks, bloating your file size
if you have many authors or finding even more relations in the browser, this will cause a performance penalty sooner or later.

Populating server side (using `gatsby-node`)

A better solution would be to enrich the blog posts during server-side generation. This would increase build time slightly on during page generation at the advantage of better performance on the browser side. A trade-off I'm happy to make!

The gist of this approach is to understand that within Netlify CMS all content types are saved in Markdown. As I explained in a previous blog page creation is done is done in gatsby-node.js. In that file, allMarkdownRemark content is retrieved. The emphasis here is on all content.

To create blogs we filter on a certain templateKey, but that also means that with the right query we can make all authors available during page creation time:

exports.createPages = ({ actions, graphql }) => {
  const { createPage } = actions

  return graphql(`
    {
      allMarkdownRemark(limit: 1000) {
        edges {
          node {
            # used to retrieve individual blog data in page queries
            id
            frontmatter {
              templateKey
              # the author fields in Blog that represents a relation
              author
              # field on the Author content type
              name 
              # field on the Author content type
              job_title
              # field on the Author content type
              profile_picture 
            }
          }
        }
      }
    }
  `).then(result => {
    const posts = result.data.allMarkdownRemark.edges;

    const blogs = posts.filter(post => post.node.frontmatter.templateKey === 'blog-post');
    const authors = posts.filter(post => post.node.frontmatter.templateKey === 'author');

    posts.forEach(edge => {
      const id = edge.node.id
      createPage({
        path: edge.node.fields.slug,
        tags: edge.node.frontmatter.tags,
        component: path.resolve('src/templates/blog.js'),
        // additional data can be passed via context
        context: {
          id,
          author: authors.find((author) => author.node.frontmatter.name === edge.node.frontmatter.author)
        },
      })
    })
  })
}

The context field can be used during page creation to provide extra data to the render props of the React Component that is used to generate the page. We can now refactor our Blog template.

// src/templates/blog.js
const BlogPost = (props) => {
  const post = props.data.markdownRemark;
  const author = props.pageContext.author.node;

  return (
    <div>
      <h1>{ post.frontmatter.title }</h1>
      <AuthorComponent {...author} />
      <div dangerouslySetInnerHTML={{ __html: post.html }} />
    </div>
  )
};

export default BlogPost

That's it! If you know a different or more efficient way of retrieving full relations, drop your knowledge in the comments!

This blog is part of a series on how I migrated away from a self-hosted Drupal blog to a modern JAM stack with Gatsby and Netlify CMS.

A beginners guide to understanding your Gatsby starter template

Patrick Sevat — Thu, 27 Feb 2020 21:41:51 +0000

Illustration by Katerina Limpitsouni undraw.co/illustrations

Recently I started migrating an older Drupal based blog to a combination of Gatsby and Netlify CMS (source), both of which I had no prior experience with. In this blog series I'll talk you through the experiences, hurdles and solutions. In part 3 I'll explain what's happening in your Gatsby starter template. Although a starter kit is a good place to start, for those with no prior knowledge of Gatsby and GraphQL it might look like magic 🧙‍♂️. So let's try and demystify!

For my project I used the Gatsby + Netlify CMS starter, although much of the information will also be applicable to other starters.

Page creation

The most basic entity for Gatsby is a page. If you were to create a "vanilla Gatsby" project, every .js file with a React Component within the src/pages directory would be compiled to an HTML page. This still holds true for your starter project, even though you might find an index.md, index.yml or index.json file in your template.

So, how does the conversion from markdown file to JavaScript happen?

This process happens in the gatsby.node.js file. Let's take a closer look:

// gatsby.node.js (parts omitted for brevity)

// 1️⃣
exports.createPages = ({ actions, graphql }) => {
  const { createPage } = actions

  // 2️⃣
  return graphql(`
    # you'll find a biggg GraphQL query here
  `).then(result => {
    // ℹ your datastructure might differ
    const pages = result.data.allMarkdownRemark.edges;

    pages.forEach(edge => {
      const id = edge.node.id
      // 3️⃣
      createPage({
        component: path.resolve(`src/templates/page.js`),
        // additional data can be passed via context
        context: {
          id,
        },
      })
    })
  })
}

What we see is that:

the createPages() method of the Gatsby Node API is being utilized to programmatically create new pages.
This method accepts an Promise. The Promise returned here is a GraphQL query. How and where GraphQL gets its data will be explained in the next section. For now we will assume that this query returns some raw data that we have saved in our markdown/yaml/json file.
For each block of raw page data we create a new page. This is done by the createPageAction. This causes Gatsby to create a new page based on the component defined. In our case the component is a JavaScript template file which contains a React Component. As we already know, Gatsby is capable of converting React Components into (static) pages with HTML, CSS and JS.

Data sourcing in Gatsby

Now that we have high-level knowledge on how pages are created, let's do an inspection on how the page data is created.

We know the source of the page data: index.md, we also know the output: it is part of a GraphQL response. The big question is how is this tranformation done?

The answer can be found in Gatsby's plugins. The plugins are defined in gatsby.config.js:

module.exports = {
  siteMetadata: {
    // ... metadata
  },
  plugins: [
    // several plugins omitted for brevity
    {
      resolve: 'gatsby-source-filesystem',
      options: {
        path: `${__dirname}/src/pages`,
        name: 'pages',
      },
    },
    {
      resolve: 'gatsby-transformer-remark',
    },
    {
      resolve: 'gatsby-plugin-netlify-cms',
      options: {
        modulePath: `${__dirname}/src/cms/cms.js`,
      },
    },
    'gatsby-plugin-netlify', // make sure to keep it last in the array
  ],
}

The first plugin that does its work in Gatsby is very often the gatsby-source-filesystem. It reads the files in a certain directory, in this case src/pages/ and reads all files recursively (though there is an ignore option). The source-filesystem plugin then creates an intermediate entity, called a File which can be picked up by other plugins for further processing.

The File contains metadata on the extension and the path to find this file.

After the File is created, subsequent plugins can process it further. In our example, gatsby-transformer-remark filters all Files with extensions md and
markdown.

Now that the transformer-remark has knowledge about the file's location, it can read the file, parse it and make the processed result available under a new GraphQL type: MarkdownRemark.

We now know the process that happen under the hood in processing from file to data queryable via GraphQL.

A short intro to GraphQL in Gatsby

Learning GraphQL is beyond the scope of this article and worthy of a full recommended course. So I'll keep it short and to the bare minimum to get going.

Revisiting `gatsby.node.js`

In the previous section we learned that there is a new MarkdownRemark type added to GraphQL. We can see this when we revisit the query we skipped earlier in gatsby.node.js:

exports.createPages = ({ actions, graphql }) => {
  return graphql(`
    {
      allMarkdownRemark(limit: 1000) {
        edges {
          node {
            id
            frontmatter {
              title
              body
            }
          }
        }
      }
    }
  `).then(result => {
    // create pages
  })
}

We can now understand that allMarkdownRemark will return an array of markdown page data. It will do so in this datastructure:

const data = {
  allMarkdownRemark: {
    edges: [{
      node: {
        id: '123',
        frontmatter: {
          title: 'My awesome blog about Gatsby + Netlify CMS',
          body: 'Lorem ipsum'
        }
      }
    }]
  }
};

Querying at the page level

If you'd want to query a single page, you'd query by the id field.
The id field is added to the page context (scroll up to createPage) and used in the query defined in the template.

// src/templates/page.js

export const query = graphql`
  query PageByID($id: String!) {
    markdownRemark(id: { eq: $id }) {
      html
      frontmatter {
        date(formatString: "MMMM DD, YYYY")
        title
      }
    }
  }
`

const Page = (props) => {
  return (
    <div>
      { props.data.markdownRemark.frontmatter.title }
      <br />{ props.data.markdownRemark.frontmatter.date }
      <div dangerouslySetInnerHTML={{ __html: props.data.markdownRemark.html }} />
    </div>
  )
};

export default Page

A couple things stand out in this snippet:

There is an exported const called query. This constant will be picked up by Gatsby, the query will be executed and passed as props.data to the component.
We see the $id parameter being injected into the query. As stated before, any parameter is being looked up via the page context. The page context is set during page creation in gatsby-node.js
Rather than using allMarkdownRemark which returns an array we use markdownRemark as Query type here. This makes sure a single object is returned

That's it, you know all you need to know to go play around with pages and queries! Try to create some new content, content types or fiddle around with queries such as sorting by date.

🧪 Tip: if you are going to play around with the GraphQL be sure to turn on the GraphQL Playground by changing your npm develop script to "develop" : "GATSBY_GRAPHQL_IDE=playground gatsby develop". This allows you to use the more intuitive Prisma GraphQL Playground

This blog is part of a series on how I migrated away from a self-hosted Drupal blog to a modern JAM stack with Gatsby and Netlify CMS.

A beginners guide to setting up Netlify CMS

Patrick Sevat — Thu, 27 Feb 2020 21:40:54 +0000

Illustration by Katerina Limpitsouni undraw.co/illustrations

How it works

Netlify CMS is a headless Content Management System. Think Wordpress, but only the admin portal. Another key difference is that it does not use any database.

Wait... what?! Yes, you read it right. Netlify CMS is part of the Git-based headless CMS group. It works by committing markdown files to your repo. Alternatively you could also store your content as JSON, yml or toml.

Using Git as your content database comes with some downsides and you might think that content editing is reserved to persons with access to the Git repo, but fortunately, Netlify CMS provides a good way to manage your content for non-devs which I will discuss in the next section.

An impression of the admin interface of Netlify CMS

To create some content you can login to the admin interface (defaults to /admin) and start editing in the User Interface. When you click on Publish, Netlify CMS will create a commit on the master branch of your repo, adding or modifying the Markdown file.

The commit on master is picked up by Netlify (the platform, not the CMS) which is registered as a Github App. Netlify schedules a new production build and as soon as that is finished, your app will be updated with new content!

❗ This means that before you start working on your site, you have to remember to pull the latest data from master.

If you are confused what the difference between Netlify and Netlify CMS is, I recommend reading this link.

Authentication

The first thing I wanted to know when considering Netlify CMS was the authentication part. For my use case the content editors should not have to create a Github account.

Fortunately, with Identity you can invite editors to the admin interface. They can sign up using an email/password combination or by linking an external OAuth provider (currently Google, Github, Bitbucket and GitLab).

Customizing fields

If you first open up the admin interface you'll be met with some basic options for creating and editing different content types. You can manage for individual pages (such as the home page or the about page) and for sets of pages such as blogs.

You'll find that individual pages will have different fields in the CMS. A home page might have a field with highlighted blogs. That field will be missing on the about page. In other words, the fields of these pages are not uniform.

Blogs on the other hand have the same set of fields, although some may be optional. This distinction is the difference between Collection types. Individual pages belong to the file content type, whereas sets of similar content belong to the folder content type.

You can have multiple of either collection types. You could have a file for the home page and another for the contact page. On the other hand you can have seperate folder content types for blogs and authors.

The configuration for these content types can be found in the static/admin/config.yml file. In basic form the content types would look like this:

# other CMS configuration

collections:
  - name: "blog"
    label: "Blog"
    folder: "src/pages/blog"
    fields:
      - {label: "Title", name: "title", widget: "string"}
      - {label: "Body", name: "body", widget: "markdown"}
  - name: "author"
    label: "Authors"
    folder: "src/authors"
    fields:
      - {label: "Name", name: "name", widget: string}
      - {label: "Profile picture", name: "profile_picture", widget: image, allow_multiple: false}
  - name: "pages"
    label: "Pages"
    files:
      - file: "src/pages/index.md"
        label: "Home"
        name: "index"
        fields:
          - {label: "Home title", name: title, widget: string}
      - file: "src/pages/contact/index.md"
        label: "Contact"
        name: "contact"
        fields:
          - {label: "Contact title", name: "title", widget: "string"}

Adding new fields is rather easy. You just add a new line with the proper configuration, commit and push the changes and it will show up in the admin panel.

There are many field types to choose from. In Netlify CMS field types are defined by the widget property. You got various options including textareas, file uploads and select fields. All available widgets can be viewed here. If the widget you would like is not present, there's even the possibility to create a custom widget.

You can now set up your CMS as you see fit. Don't worry if you need or want to iterate on it. You can add fields later on as well. If you decide to rename them, you'll have to edit the markdown files to update the fields there as well.

When inspecting the Markdown file you might encounter some syntax that you have not seen before. It looks like the Markdown is enhanced with some metadata:

---
title: "An awesome blog"
---
**The body of the blog post**

Typing `Markdown` as you *know it*

This metadata format is known as frontmatter. Essentially, it is yaml syntax. All fields are saved as frontmatter data, except the body field if the body field uses the markdown widget.

Previewing your content in the admin interface

Even though you added some new fields to your content types and you filled them in, they will not show up immediately in the preview pane. To get that working you must register a previewTemplate. If you use any of the Netlify CMS starters the template is already created for you. All you need to do is retrieve the new data and update your rendered component(s) with this new data.

A local CMS

One last tip: if you are developing locally you might want to test with some mocked data (looking at you Foo Blog 🙄) or you might be working on the preview templates mentioned in the previous section.

Either way, it is possible to run a local version of the CMS using a beta feature that sets up a proxy. This will make sure that any content you create is not pushed to the repo.

My final configuration

You can find the CMS configuration I ended up with in my Github repo.

You might notice that I created a relationship between the Blog content type and the Author content type. The how and why of that relationship is the subject of the next blog!

This blog is part of a series on how I migrated away from a self-hosted Drupal blog to a modern JAM stack with Gatsby and Netlify CMS.

How to choose your JAM stack

Patrick Sevat — Thu, 27 Feb 2020 21:40:15 +0000

Illustration by Katerina Limpitsouni undraw.co/illustrations

Over the last months I have migrated my Drupal and Wordpress websites to the JAMstack (Javascript, API and Markup).

This blog series will focus on my first endeavour with Gastby and Netlify CMS, but this episode will also focus on my experience with other libraries and providers in the ecosystem.

3 reasons to migrate to the JAM stack

No more server management! Upgrading LAMP stacks, maintaining Linux servers requires a lot of attention and knowledge and it is not my strong suit
For small sites, you can host for free or very cheap using providers like Netlify, Vercel and others
Learning new tools is FUN!

JAM stack tool considerations

You can find plenty of blogs or websites discussing the benefits and various tools available to you. To help you make a decision I'd recommend you consider these points. I'll share my personal findings for each point.

Easy deploy

I want to spend as little time configuring build/copy steps, exposing (public) folders and setting up SSL certificates as possible. I just want to define one npm script to execute and let the service do the rest. There are a couple options here such as Heroku, Netlify and Vercel. Due to the availability of a complementary CMS, form submission handling and a good starter template, I ended up with Netlify. But also give a good look at Vercel!

CMS for non-tech users

My users require a CMS with a admin interface which they can use. I also do not want to self-host my CMS elsewhere (headless CMS like Wordpress API). You can explore options on headlessCMS.org. Consider if you have a preference for a git-based or API-driven / SaaS CMS.

Some good options: Contentful (SaaS & free tier), Netlify CMS (git-based, so free), Strapi (self-hosted)

Extendable CMS (custom fields)

Make sure that your CMS supports extention of fields. Have you thought about comments, attachments, tags? Make sure your CMS supports adding these later on. Even better, your CMS should support creating new entities. For example I created an Author entitity so my users can place content on behalf of the actual authors.

Modern techniques

New tools not only drive your learnings, but also bring benefits such as faster loading. See if you can utilize new optimizations such as GraphQL, webp image formats, serverless functions, CDN edge servers and variable fonts. Don't forget the advances on the developer experience as well. Tools such as dependabot, Cypress.io and TravisCI can make updating a breeze.

Big community / ecosystem

Whenever choosing your JAM stack tools, you can make your life easier by choosing tools that are welcoming to newcomers. The documentation of Gatsby can be daunting at first due to its sheer size, but it does an amazing job of gradually learning you everything you need to know. To make things even easier there are plenty of starter templates for you to pick from. Gatbsy also has an amazing ecosystem of plugins that provide you with all kinds of functionality.

Another recommendation in the JavaScript space would be to consider 11ty, it is less mature than Gatsby but still has quite a community and plugins. The same goes for Vuepress.

Not too steep learning curve

Although learning new technologies, frameworks and techniques is really fun, you should also keep it manageable.

Certain techniques are easier to pick up than others. Below an overview of the learning curves for all tech that I've encountered in my journey. When describing difficulty I assume decent knowledge of HTML/CSS/JS/JS frameworks, but no prior experience with particular technology

Name	Difficulty	Comment
GraphQL	8 / 10
React	6 / 10
Gatsby	9 / 10	Steep learning curve if stepping outside of templates, only try if comfortable with React or GraphQL
Svelte	5 / 10	Easy to pick up, ecosystem not fully mature
Sapper	5 / 10	Same as Svelte
Vue	5 / 10	Easy to pick up
Vuepress	7 / 10	Takes a bit to understand once going outside of themes

My experiences

CMS

My biggest doubt when I embarked on this journey was that it might be difficult to find a CMS flexible enough to adjust to all the custom fields I needed, while at the same time be friendly to non technical editors.

Netlify CMS proved to handle both. Fields are extremely extendable. Netlify CMS also supports custom content types. Adding relations between content types was a challenge though (more on that in a next post).

Non tech users can create and edit content with the help of Identity. In the free tier that means your content editors can log-in using a Google account.

To be honest, Netlify was the first CMS I considered, also due to the ease of deployment and availability of start templates. I'm pretty sure other CMS systems would also satifsy these requirements so let me know your experiences!

Update, july 2020
Even though I experimented with other site generators (see next section), I stuck with Netlify CMS as my content system!

Static Site Generation (SSG)

Update, july 2020
This post originally only covered Gatsby, since then I've also tried other static site generators. This section is now updated.

Gatsby

Gatsby is one of the big names within the JAMstack space.
Strengths are:

extensive, newbie-friendly documentation,
newbie-friendly starter-kit and templates
extensible plug-ins leading to a very lively and mature ecosystem

A potential downside is the learning curve if you are not already familiar with React or GraphQL. If you only need to learn either of those technologies, Gatsby is a great choice. If you are new to both React and GraphQL, choosing Gatsby might be too much to chew.

Nonetheless, the docs do a great job at introducing all the concepts of both React and GraphQL. The rest of this blog series is also aimed at making Gatsby more understandable.

The starter template I used is the Gatsby + Netlify CMS Starter. It proved a useful starting point, but my customizations were quick to follow as you can read in the rest of this series

Vuepress

Vuepress is a static site generator within the Vue ecosystem.

Strengths include:

Built upon Vue, which is very easy to learn and has an awesome ecosystem. For example, the Vuetify component library / design system is one of the best across all JS frameworks.
Has many out-of-the-box themes. This resembles Wordpress theme experience, allowing you to get up and running very fast
Is extensible enough to create your own theme (for example using Vuetify)

Downsides:

The ecosystem of Vuepress is less mature. For example, at the moment of writing there is no community-provided plugin for Netlify CMS relations (although I plan to write one, in the mean time check here

Sapper

Sapper is a static site generator within the Svelte community.

Strengths include:

Lightweight builds, interesting if performance is paramount for you
Svelte is easy to learn

Downsides:

The ecosystem of Svelte is less mature than React (Gatsby) and Vuepress (Vue). I ran into this when looking for component libraries. Smelte is a perfect example: it is inspired by Vuetify, but less extensive. On the other hand it is way smaller and more performant.

Although I enjoyed experimenting with Sapper / Svelte, my personal preference is to lean towards something more batteries-included and take the performance penalty. But that is a trade-off every developer has themselve.

This blog is part of a series on how I migrated away from a self-hosted Drupal blog to a modern JAM stack with Gatsby and Netlify CMS.

DEV Community: Patrick Sevat

Optimize your git clone / fetch strategy for CI pipelines

You only need to checkout / build a single branch

git clone --single-branch

git fetch --no-tags

git fetch --depth=<n>

git fetch --depth=1 origin <full_SHA1>

Combining the flags git clone --no-tags --single-branch --depth=1

You need to more than one branch (but not all)

git clone --depth=<n>

Custom git config

Steps

Conclusion

How to use Web Components in HTMLTemplateElement in Angular

Code example

Wrapping up: useful plugins for Gatsby

Typescript in Gatsby

Search functionality in Gatsby

Pagination in Gatsby

Styling SVG in Gatsby

Image optimalization in Gatsby

PWA support in Gatsby

So... what database does Gatsby use?

GraphQL queries from databases right?

No database, but Redux

Next up, LokiJS

How to retrieve all fields of a Netlify CMS Relation in Gatsby

Setting up a relation in Netlify CMS

Retrieving all fields of a relation

Populating browser side

Populating server side (using gatsby-node)

A beginners guide to understanding your Gatsby starter template

Page creation

Data sourcing in Gatsby

A short intro to GraphQL in Gatsby

Revisiting gatsby.node.js

Querying at the page level

A beginners guide to setting up Netlify CMS

How it works

Authentication

Customizing fields

Previewing your content in the admin interface

A local CMS

My final configuration

How to choose your JAM stack

3 reasons to migrate to the JAM stack

JAM stack tool considerations

Easy deploy

CMS for non-tech users

Extendable CMS (custom fields)

Modern techniques

Big community / ecosystem

Not too steep learning curve

My experiences

CMS

Static Site Generation (SSG)

Gatsby

Vuepress

Sapper

`git clone --single-branch`

`git fetch --no-tags`

`git fetch --depth=<n>`

`git fetch --depth=1 origin <full_SHA1>`

Combining the flags `git clone --no-tags --single-branch --depth=1`

`git clone --depth=<n>`

Retrieving all fields of a `relation`

Populating server side (using `gatsby-node`)

Revisiting `gatsby.node.js`