DEV Community: Cleve Littlefield

RSS/Atom and Site Map for Svelte/Sapper Blog - Part 3

Cleve Littlefield — Sat, 16 May 2020 09:46:58 +0000

I feel like a boomer when I tell people I still use an RSS reader. I still find it is the best way to keep up with interesting people writing about technology. I often share my blog roll with other programmers when they ask and point them to my reader of choice, RSS Owl. So yes, I expect my blog to support RSS even in this day and age.

When I started with a template Svelte/Sapper Blog project, I was slightly surprised that it does not have a way to generate RSS/Atom out of the box. It took a bit of searching to see how others were implementing RSS and found this on David Lacourt's blog, which references a solution from the Svelte man Rich Harris himself..

Pretty straight forward, with a regular server-side route, just returning XML over JSON. I have to admit coming other technologies that I assumed this needed to be more complicated with a model and XML translation middleware. I worked with "frameworks" so much that a straightforward solution eluded me for the moment. I also blame working predominantly with Kubernetes and infrastructure for the last few years.

Since I static gen my site with sapper export and that works by traversing all links, I have a link to my feeds from one my pages, I used the privacy policy page.

While RSS would do the trick on its own, let's also do ATOM to be thorough. The real secret here is ATOM expects the date to be in RFC3339 format pulled from here.


import { getPosts } from './_posts.js';
import { siteUrl } from '../../stores/_config.js';

function toRFC3339(date) {

  function pad(n) {
      return n < 10 ? "0" + n : n;
  }

  function timezoneOffset(offset) {
      var sign;
      if (offset === 0) {
          return "Z";
      }
      sign = (offset > 0) ? "-" : "+";
      offset = Math.abs(offset);
      return sign + pad(Math.floor(offset / 60)) + ":" + pad(offset % 60);
  }

  return date.getFullYear() + "-" +
      pad(date.getMonth() + 1) + "-" +
      pad(date.getDate()) + "T" +
      pad(date.getHours()) + ":" +
      pad(date.getMinutes()) + ":" +
      pad(date.getSeconds()) + 
      timezoneOffset(date.getTimezoneOffset());
}

function renderXmlAtomFeed(posts) {
  return `<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">

  <title>Clever Dev Codes</title>
  <link href="${siteUrl}"/>
  <link rel="self" href="${siteUrl}/blog/atom" />
  <updated>${toRFC3339(new Date())}</updated>
  <author>
    <name>Cleve Littlefield</name>
  </author>
  <id>urn:uuid:DBEF5693-FACC-47DB-9063-3AFF0FD30733</id>
  ${posts.map(post => `
  <entry>
    <title>${post.title}</title>
    <link href="${siteUrl}/blog/${post.slug}"/>
    <id>urn:uuid:${post.id}</id>
    <updated>${toRFC3339(new Date(post.publishedDate))}</updated>
    <content type="xhtml">
      <div xmlns="http://www.w3.org/1999/xhtml">
        ${post.html}
      </div>
    </content>
    <summary type="xhtml">
      <div xmlns="http://www.w3.org/1999/xhtml">
        ${post.excerpt}
      </div>
    </summary>
  </entry>`).join('\n')}

</feed>`;
}

export async function get(req, res) {

  res.writeHead(200, {
    'Cache-Control': `max-age=0, s-max-age=${600}`, // 10 minutes
    'Content-Type': 'application/atom+xml'
  });

  const posts = await getPosts();
  const feed = renderXmlAtomFeed(posts);
  res.end(feed);

}

How about a site map? Similar concept:


import { getPosts } from "./blog/_posts.js";
import { siteUrl } from "../stores/_config.js";

function renderSitemapXml(slugs) {
  return `<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:news="http://www.google.com/schemas/sitemap-news/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml" xmlns:mobile="http://www.google.com/schemas/sitemap-mobile/1.0" xmlns:image="http://www.google.com/schemas/sitemap-image/1.1" xmlns:video="http://www.google.com/schemas/sitemap-video/1.1">
${slugs
  .map(
    slug => `
  <url>
    <loc>${siteUrl}/${slug}</loc>
    <changefreq>daily</changefreq>
    <priority>0.7</priority>
  </url>
`
  )
  .join("\n")}
</urlset>`;
}

export async function get(req, res) {
  res.writeHead(200, {
    "Cache-Control": `public, max-age=0, must-revalidate`,
    "Content-Type": "application/xml"
  });

  const posts = await getPosts();
  const slugs = posts.map(post => `blog/${post.slug}`);
  const feed = renderSitemapXml([...slugs, "", "/blog"]);
  res.end(feed);
}

Exporting Squidex Assets for Svelte/Sapper blog - Part 2

Cleve Littlefield — Tue, 25 Feb 2020 02:33:00 +0000

Squidex supports uploading image assets, then setting a focal point. Then when you download those assets with a width/height it will downscale those images automatically (you can even supply a quality parameter).

By default those images though are pinned to the https://cloud.squidex.io/ domain, that just will not work for me. I like to control all the things.

I really wanted to be able to experiment during dev but during export I wanted to download and include these assets with my static site. I whipped up some code to do just that.

Before I just show you the code, let's walk through what it is doing.

Process markdown, adding syntax highlighting using prismjs
If the current NPM command is a sapper export, let's parse the resulting HTML using cheerio
Let's pull out any images, if they are sourced form Squidex, let's download those images, making sure to include the width and height if those attributes are present.
Let's save the file to the sapper relative export path and then rewrite the URL to the local path


import prism from "prismjs";
import marked from "marked";
import fs from "fs";
import path from "path";
import request from "request";
import cheerio from 'cheerio';
require('prismjs/components/prism-jsx.min');

const renderer = new marked.Renderer();
renderer.code = (code, language) => {
  const parser = prism.languages[language] || prism.languages.html;
  const highlighted = prism.highlight(code, parser, language)
  return `<pre class="language-${language}">
    <code class="language-${language}">${highlighted}</code>
    </pre>`;
}

export default function renderMarkdown(text) {
    const html = marked(text, { renderer: renderer }).replace(/^\t{3}/gm, '');
    if (process.env.npm_lifecycle_script.startsWith('sapper export') === false) {
      return html;
    }

    const dom = cheerio.load(html);
    // const dom = parse(html, { pre: true });
    // return dom.innerHTML;    
    const images = dom("img");
    const prefix = `https://cloud.squidex.io/api/assets/${process.env.SQUIDEX_PROJECT}/`;

    for (let i = 0; i < images.length; i++) {
      const img = images[i];
      const src = img.attribs.src;
      const width = img.attribs.width;
      const height = img.attribs.height;
      let imageSpec = getImageSpec(width, height);
      const assetUrl = `${src}${imageSpec}`;
      const index = src.indexOf(prefix);

      if (src && index > -1 ) {
        const relativePath = getRelativePath(src, prefix);
        const localPath = path.resolve(`./__sapper__/export${relativePath}`);
        fs.mkdirSync(path.dirname(localPath), { recursive: true});
        download(assetUrl, localPath);
        img.attribs.src = relativePath;
      }
    }

    return dom.html();
}

function getRelativePath(src, prefix) {
  const queryPartIndex = src.indexOf('?');
  return '/assets/' + src.slice(0, queryPartIndex).replace(prefix, "");
}

function getImageSpec(width, height) {
  let imageSpec = "";
  if (width) {
    imageSpec += `&width=${width}`;
  }
  if (height) {
    imageSpec += `&height=${height}`;
  }
  return imageSpec;
}

function download(uri, filename){
  request(uri).pipe(fs.createWriteStream(filename));
};

Now all we need to do is change line 35 of src/routes/blog/_posts.js :.

    const html = renderMarkdown(post.text);

Run this using yarn export and any images you have included in your Squidex posts will be downloaded and optimized automatically.

Svelte, Sapper, and Squidex Headless CMS, Part 1

Cleve Littlefield — Thu, 20 Feb 2020 09:50:35 +0000

I have experienced many different web platforms since I pivoted to the web around 2003. First ASP.Net, then ASP.Net MVC, then AngularJS, then React. Imagine how many frameworks boom and bust cycles I also managed to skip!

I remember how excited I was for each new renaissance to lift us out of the past dark ages. Gone was the complexity that weighed us down, now that the new hotness is here! In truth, each of these did provide a productivity boost, but then a plateau would inevitably come.

Now there is Svelte. The feeling playing with Svelte is both very similar to this prior experience and altogether different at the same time. It just seems... simple. Straightforward. The learning curve, at least for me, is way lower. And it generates impossibly small code, so everything is super fast. It must be too good to be true.

Like all magnificent things, it's very simple.
Natalie Babbitt, Tuck Everlasting.

Well, let's explore that and other topics here, in my brand new space, with Yet Another How I Bootstrapped this Blog post. To give it a fresh spin, let's wire it up to a headless CMS. I shopped around and ultimately landed on Squidex because I liked the feature set but mainly because the API was REST-y, and I prefer that over GraphQL. Let's give it a spin.

I created a Squidex account (there is a free account option at https://squidex.io/ and clicked the New Blog Sample. Now let's get a feel for their API.
Head over to settings in the project you created above and grab a clientid and secret. I generated a new one as the default one has editor permissions, and to export our website, we only need read permissions.

Another one of Cleve's new favorite things ^TM is the REST client extension for VS Code. The REST client extension feels like an even more developer-friendly Postman. I like UIs, but this has way less clicking and window management and can be checked into source control to share with others.

Here is a a .http file for getting posts from Squidex. project, squidex_clientid and squidex_secret are setup as REST extension environment variables. X-Flatten removes these iv extra depth fields that Squidex adds by default.

# @name authorization
POST https://cloud.squidex.io/identity-server/connect/token
Content-Type: application/x-www-form-urlencoded

grant_type=client_credentials&client_id={{squidex_clientid}}&client_secret={{squidex_secret}}&scope=squidex-api

@accessToken = {{authorization.response.body.$.access_token}}

###
GET https://cloud.squidex.io/api/content/{{project}}/posts/
Authorization: Bearer {{accessToken}}
X-Flatten: true

What is X-Flatten? At least for Squidex is removes the iv field, which feels superfluous. Invariant? Meh.

Now let's create our Svelte/Sapper project. Assuming you have npm/npx installed:

npx degit "sveltejs/sapper-template#rollup" my-blog

Create a src/routes/_squidex.js file (borrowed config from @querc/squidex-client). Unfortunately, that library didn't work for me because of auth issues and was more complicated than I needed.

import fetch from "node-fetch";

function defineProperty(obj, key, value) {
  if (key in obj) {
    Object.defineProperty(obj, key, {
      value: value,
      enumerable: true,
      configurable: true,
      writable: true
    });
  } else {
    obj[key] = value;
  }
  return obj;
}

function convertFromJson(json) {
  const result = Object.assign({}, json);
  result.created = new Date(json.created);
  result.lastModified = new Date(json.lastModified);
  return result;
}

class SquidexClientConfiguration {
  constructor() {
    defineProperty(this, "url", "https://cloud.squidex.io");

    defineProperty(this, "clientId", void 0);

    defineProperty(this, "clientSecret", void 0);

    defineProperty(this, "project", "");
  }
}

class ConfigurationManager {
  static buildConfiguration(options, ...extraOptions) {
    if (options === undefined) {
      throw new Error(
        "Configuration options are required"
      );
    }

    if (options.clientId === undefined) {
      throw new Error("`clientId` is required");
    }

    if (options.clientSecret === undefined) {
      throw new Error("`clientSecret` is required");
    }

    if (options.project === undefined) {
      throw new Error("`project` is required");
    }

    return Object.assign(
      {},
      new SquidexClientConfiguration(),
      options,
      ...extraOptions
    );
  }
}

export class SquidexClient {
  constructor(options) {
    defineProperty(this, "config", void 0);
    defineProperty(this, "token", void 0);
    this.config = ConfigurationManager.buildConfiguration(options);
  }

  async getAuthenticationToken() {
    if (!this.token) {
      await this.initializeToken();
    }

    return this.token;
  }

  async initializeToken() {
    const authorizationResponse = await fetch(
      `${this.config.url}/identity-server/connect/token`,
      {
        headers: {
          "Content-Type": "application/x-www-form-urlencoded"
        },
        method: "POST",
        body: `grant_type=client_credentials&client_id=${this.config.clientId}&client_secret=${this.config.clientSecret}&scope=squidex-api`
      }
    );

    if (!authorizationResponse.ok) {
      const errorText = await authorizationResponse.text();
      throw new Error(`Could not obtain Squidex token. ${errorText}`);
    }

    const json = await authorizationResponse.json();
    this.token = `Bearer ${json["access_token"]}`;
  }

  async query(schema) {
    const token = await this.getAuthenticationToken();
    const response = await fetch(
      `${this.config.url}/api/content/${this.config.project}/${schema}`,
      {
        headers: {
          Authorization: token,
          "X-Flatten": "true"
        }
      }
    );

    if (!response.ok) {
      const errorText = await response.text();
      throw new Error(errorText);
    }
    const data = await response.json();
    return data.items.map(x => {
      return convertFromJson(x);
    });
  }
}

Let's go ahead and run yarn add node-fetch since we depend on it here.

Now we have to modify how the starter template fetches data. To be honest, I do not come from the NodeJS/Nuxt inspired/JAMstack world, so I found how Sapper mixes server and client together to be confusing at first. But after reading and playing with the code a few times, I got the hang of it, starting by reading https://www.codingwithjesse.com/blog/statically-generating-a-blog-with-svelte-sapper/.

Basically inside src/routes/blog/index.svelte there is a preload call to fetch blog.json. The preload returns an object with a posts variable. Since there is a local variable called posts Sapper wires them together. This part also seemed a little weird and hidden magic for me, but c'est la vie, let's move on.

This calls src/routes/blog/index.json.js which is a server side route, which gets data from src/routes/blog/_posts.js. Similarly src/routes/blog/[slug].svelte calls src/routes/blog/[slug].json.js which gets data from src/routes/blog/_posts.js.

Now let's modify the src/routes/blog/_posts.js to fetch from Squidex. We are doing is caching our results, so we only get the data once. In a future post, we will add some more features like calculate reading time and process markdown.

import { SquidexClient } from "../_squidex.js";

let posts;
let lookup;

export async function getPostLookup() {
  if (!lookup) {
    const items = await getPosts();
    lookup = new Map();
    items.forEach(item => {
      lookup.set(item.slug, JSON.stringify(item));
    });
  }

  return lookup;
}

export async function getPosts() {
  if (posts) {
    return posts;
  }
  var client = new SquidexClient({
    clientId: process.env.SQUIDEX_CLIENT_ID,
    clientSecret: process.env.SQUIDEX_SECRET,
    project: process.env.SQUIDEX_PROJECT
  });

  const items = await client.query("posts");
  posts = items.map(item => {
    const post = item.data;
    const id = item.id;
    const title = post.title;
    const slug = post.slug;
    const html = post.text;

    return {
      id,
      title,
      slug,
      html
    };
  });

  return posts;
}

We need to change the two endpoint files to use this data:

Change src/routes/blog/index.json.js:

import { getPosts } from "./_posts.js";

export async function get(request, response) {

  const posts = await getPosts();
  response.writeHead(200, {
    'Content-Type': 'application/json'
  });
  response.end(JSON.stringify(posts));
}

and change src/routes/blog/[slug].json.js to use our cached lookup:

import { getPostLookup } from './_posts.js';

export async function get(req, res, next) {

    const lookup = await getPostLookup();
    const { slug } = req.params;

    if (lookup.has(slug)) {
        res.writeHead(200, {
            'Content-Type': 'application/json'
        });

        res.end(lookup.get(slug));
    } else {
        res.writeHead(404, {
            'Content-Type': 'application/json'
        });

        res.end(JSON.stringify({
            message: `Not found`
        }));
    }
}

Now run yarn dev, and you should see the sample blog post from Squidex.

Stay tuned, in future posts in this series we will:

Render markdown, processing image assets, and add reading time
Add the RSS/Atom feed
Create a 404 page
Implement a tag cloud and recent posts
Play with styling and layout
Implement SEO and OG markup
Create a static about me page