Importing Toots from Mastodon to WordPress Posts Using Google Apps Script

Ayumi Sato — Fri, 26 Jul 2024 05:05:45 +0000

I have been archiving my online writings on a private WordPress site since 2003. This includes diaries originally written in HTML, posts from other external platforms, and content created using MovableType - all of which have been migrated to WordPress. Although I don't post regularly and not everything is archived, I still have an impressive collection spanning 15 years, totaling approximately 2,500 articles.
Recently, I wanted to import my past toots from Mastodon into WordPress in a similar manner. I was able to achieve this using Google Apps Script. By importing my toots into my WordPress environment, I can effectively manage my data and easily search through the content.
If you are looking to do something similar with your own Mastodon toots and WordPress site, I hope this article will be a helpful resource.

Objectives

Import past toots from my Mastodon account into WordPress.
Use Google Apps Script to avoid setting up a new environment.
Set the toot’s post date as the article’s post date in WordPress.
Use the first 20 characters of the toot’s body as the WordPress article title.
If the toot’s body is empty, set the WordPress article title as "no title".
Categorize the articles as "Toots".
Attach images from toots to the WordPress article body.
Convert hashtags in toots to WordPress tags.
Exclude boosts and replies.

What Was Not Done

Video attachments.
Other necessary or unnecessary things I didn't think of.

Requirements

To fetch data from Mastodon, you need an access token.

Obtaining the Mastodon Access Token

Log in to your Mastodon instance.
Select "Development" from "User Settings" and click "New Application".
Enter an application name and select the necessary permissions (scope). For this task, "read" permission is sufficient.
Click "Submit".
Note the access token from the application details page.

Google Apps Script Code

Write the following code in Google Apps Script. Set the URL of the Mastodon instance, the obtained access token, and the URL of the WordPress site.
Execute the main() function. It may take a few minutes to complete. If a security warning appears during execution, please allow it.

// Main function: Convert Mastodon posts to WordPress format and save as an XML file
function main() {
  const instanceUrl = "https://mastodon.example.com"; // Set the URL of the Mastodon instance
  const accessToken = "your_token_here"; // Set the obtained access token
  const siteUrl = "https://wordpress.example.com"; // Set the URL of the WordPress site

  // Fetch Mastodon posts
  const posts = fetchMastodonPosts(instanceUrl, accessToken);

  // Convert fetched posts to WordPress format XML
  const wxrContent = createWxrFile(posts, siteUrl);

  // Create and save the XML file to Google Drive
  const blob = Utilities.newBlob(wxrContent, "application/xml", "mastodon_posts.xml");
  const file = DriveApp.createFile(blob);
  Logger.log("File created: " + file.getUrl());
}

// Function to fetch Mastodon posts
function fetchMastodonPosts(instanceUrl, accessToken) {
  const headers = { Authorization: "Bearer " + accessToken };
  const options = { method: "get", headers: headers };

  // Get user ID
  const userId = getUserId(instanceUrl, options);

  // Fetch all posts
  return fetchAllPosts(instanceUrl, userId, options);
}

// Function to get user ID
function getUserId(instanceUrl, options) {
  const userResponse = UrlFetchApp.fetch(`${instanceUrl}/api/v1/accounts/verify_credentials`, options);
  return JSON.parse(userResponse.getContentText()).id;
}

// Function to fetch all posts
function fetchAllPosts(instanceUrl, userId, options) {
  let posts = [];
  let url = `${instanceUrl}/api/v1/accounts/${userId}/statuses`;

  while (url) {
    const response = UrlFetchApp.fetch(url, options);
    const data = JSON.parse(response.getContentText());
    posts = posts.concat(data);

    // Get the URL of the next page
    url = getNextPageUrl(response);
  }

  return posts;
}

// Function to get the URL of the next page from response headers
function getNextPageUrl(response) {
  const links = response.getHeaders()["Link"];
  if (links && links.includes('rel="next"')) {
    return links.match(/<(.*)>; rel="next"/)[1];
  }
  return null;
}

// Function to strip HTML tags
function stripHtmlTags(str) {
  if (!str) return "";
  return str.toString().replace(/<[^>]*>/g, "");
}

// Function to convert HTML content to WordPress blocks
function convertToWordPressBlocks(htmlContent) {
  return htmlContent
    .replace(/<p>(.*?)<\/p>/g, (match, content) => `<!-- wp:paragraph -->\n<p>${content}</p>\n<!-- /wp:paragraph -->\n`)
    .replace(/<img src="(.*?)" alt="(.*?)" \/>/g, (match, src, alt) => `<!-- wp:image -->\n<figure class="wp-block-image"><img src="${src}" alt="${alt}" /></figure>\n<!-- /wp:image -->\n`);
}

// Function to format date for RSS
function formatPubDate(date) {
  const myDate = new Date(date);
  const days = ["Sun", "Mon", "Tue", "Wed", "Thu", "Fri", "Sat"];
  const months = ["Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"];
  return `${days[myDate.getUTCDay()]}, ${myDate.getUTCDate().toString().padStart(2, "0")} ${months[myDate.getUTCMonth()]} ${myDate.getUTCFullYear()} ${myDate.getUTCHours().toString().padStart(2, "0")}:${myDate.getUTCMinutes().toString().padStart(2, "0")}:${myDate.getUTCSeconds().toString().padStart(2, "0")} +0000`;
}

// Function to format date for WordPress
function formatDateToWordPress(date) {
  const myDate = new Date(date);
  return `${myDate.getFullYear()}-${String(myDate.getMonth() + 1).padStart(2, "0")}-${String(myDate.getDate()).padStart(2, "0")} ${String(myDate.getHours()).padStart(2, "0")}:${String(myDate.getMinutes()).padStart(2, "0")}:${String(myDate.getSeconds()).padStart(2, "0")}`;
}

// Function to create WordPress eXtended RSS (WXR) file
function createWxrFile(posts, siteUrl) {
  let xml = '<?xml version="1.0" encoding="UTF-8" ?>\n';
  xml += '<rss version="2.0" xmlns:excerpt="http://wordpress.org/export/1.2/excerpt/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:wfw="http://wellformedweb.org/CommentAPI/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:wp="http://wordpress.org/export/1.2/">\n';
  xml += "<channel>\n";
  xml += "<wp:wxr_version>1.2</wp:wxr_version>\n";
  posts.forEach((post, index) => {
    xml += createWxrItem(post, index, siteUrl);
  });
  xml += "</channel>\n";
  xml += "</rss>\n";
  return xml;
}

// Function to create WXR item for each post
function createWxrItem(post, index, siteUrl) {
  // Skip replies and reblogs
  if (post.in_reply_to_id !== null || post.reblog !== null) {
    return "";
  }

  let content = post.content;
  const strippedContent = stripHtmlTags(post.content);
  const title = post.spoiler_text ? `⚠️${post.spoiler_text}` : post.content ? strippedContent.substring(0, 20) : "no title";
  const postDate = formatDateToWordPress(post.created_at);
  const postPubDate = formatPubDate(post.created_at);

  // Extract hashtags
  const hashtags = extractHashtags(content);

  // Convert hashtags to links
  content = convertHashtagsToLinks(content, siteUrl);

  // Add spoiler text if present
  if (post.spoiler_text) {
    content = `<p>${post.spoiler_text}</p>${content}`;
  }

  // Add media attachments if present
  if (post.media_attachments.length > 0) {
    post.media_attachments.forEach((media) => {
      const alt = media.description ? media.description : "";
      if (media.type === "image") {
        content += `\n\n<img src="${media.url}" alt="${alt}" />`;
      }
    });
  }

  // Construct WXR item
  let xmlItem = `
<item>
<title><![CDATA[${title}]]></title>
<content:encoded><![CDATA[${convertToWordPressBlocks(content)}]]></content:encoded>
<excerpt:encoded><![CDATA[]]></excerpt:encoded>
<pubDate><![CDATA[${postPubDate}]]></pubDate>
<dc:creator><![CDATA[dummy]]></dc:creator>
<wp:post_id>${index + 1}</wp:post_id>
<wp:post_date><![CDATA[${postDate}]]></wp:post_date>
<wp:post_date_gmt><![CDATA[${postDate}]]></wp:post_date_gmt>
<wp:post_modified><![CDATA[${postDate}]]></wp:post_modified>
<wp:post_modified_gmt><![CDATA[${postDate}]]></wp:post_modified_gmt>
<wp:post_type>post</wp:post_type>
<wp:status><![CDATA[publish]]></wp:status>
<category domain="category" nicename="toots"><![CDATA[Toots]]></category>
`;

  // Add hashtags as WordPress tags
  hashtags.forEach((tag) => {
    xmlItem += ` <category domain="post_tag" nicename="${tag}"><![CDATA[${tag}]]></category>\n`;
  });

  xmlItem += " </item>\n";
  return xmlItem;
}

// Function to extract hashtags from content
function extractHashtags(content) {
  const regex = /<a href="[^"]*" class="mention hashtag" rel="tag">#<span>([^<]+)<\/span><\/a>/g;
  const hashtags = [];
  let match;
  while ((match = regex.exec(content)) !== null) {
    hashtags.push(match[1]);
  }
  return hashtags;
}

// Function to convert hashtags to WordPress links
function convertHashtagsToLinks(content, siteUrl) {
  return content.replace(/<a href="[^"]*" class="mention hashtag" rel="tag">#<span>([^<]+)<\/span><\/a>/g, function (match, tag) {
    const tagUrl = `${siteUrl}/tag/${encodeURIComponent(tag)}/`;
    return `<a href="${tagUrl}" class="hashtag">#${tag}</a>`;
  });
}

Download the WXR File

After running the script, an XML file will be created in Google Drive. Download this file.

Import into WordPress

From the WordPress admin panel, go to "Tools" → "Import" and select "WordPress". Import the WXR file.
Before importing into the production environment, be sure to verify that the import works correctly in a test environment.

Import External Images into WordPress

Initially, images in imported articles are linked directly to the Mastodon instance. To import these images into WordPress, I used the Auto Upload Images plugin. Although it seems outdated, I couldn't find another plugin with the same functionality.

Using the Auto Upload Images Plugin

Install and activate the plugin.
From the WordPress admin panel, select "Tools" → "Replace External Images".
Select the target articles from the post list.
Click the "Replace" button to upload the images.

Set Featured Images

After importing images, set featured images as needed. To streamline this process, I used the XO Featured Image Tools plugin.

Using XO Featured Image Tools

Install and activate the plugin.
From the WordPress admin panel, select "Tools" → "Featured Image".
Select the target posts and click "Create Featured Image from Image".
Featured images will be generated automatically.

Impressions

When I decided to migrate my posts, I searched for tools but couldn't find any that met my requirements. Some tools were outdated or didn't fully import toots as WordPress posts.
Since no suitable tools were available, I decided to create my own script with the help of Perplexity. The most challenging part was generating a minimal WordPress eXtended RSS (WXR) file for the migration.
I couldn't find the WXR specifications, so I exported articles from WordPress and wrote the script by mimicking the exported content.
For now, the script works in my environment. Whether it will work in other environments or continue to work in the future is uncertain, but I plan to use it as needed.
Next, I plan to create a script using Google Apps Script to continuously import new posts from Mastodon.

Official & Unofficial STUDIO Keyboard Shortcuts List

Ayumi Sato — Sun, 14 Aug 2022 06:16:00 +0000

You can use some keyboard shortcuts on the STUDIO, the online website builder. Keyboard shortcuts reduce mouse and trackpad operations, and you can build pages faster. This article introduces official shortcuts and also unofficial ones, which I have found.

For Windows, replace ⌘ with Ctrl.

Officially announced shortcuts

The shortcuts listed in the STUDIO U (Japanese article) are as follows.

Box editing actions

It works on the editor canvas or layer panel.

Action	Keys
Copy	⌘ + C
Cut	⌘ + X
Paste	⌘ + V
Delete	delete
Group	⌘ + G
Ungroup	⌘ + shift + G
Create list	⌘ + L
Create symbol	⌘ + J

Box selection actions

It works when selecting some element on the editor canvas or layer panel.

Action	Keys
Select multiple items	shift + click
Select child box	return (enter)
Select parent box	esc
Select next box	tab
Select previous box	shift + tab

By mastering selections, you can increase your work efficiency by applying the same state to multiple elements in a single operation. Also, selecting parent and child boxes from the keyboard allows you to go to the desired layer without opening and closing layers.

Editor action

Action	Keys
Undo	⌘ + Z
Redo	⌘ + shift + Z
Zoom in/Out	+ / -
View in 100%	⌘ + 0
Enable / Disable content editing mode	⌘ + 0

Undo shortcuts will let you return to the previous state instantly. In 100% view, the canvas will be displayed the same size as the actual website, reducing impression differences.

Unofficial shortcuts

The following shortcuts are not on the official shortcut list above, but I have confirmed that they work. I discovered these shortcuts by pressing keys that I thought might work.

Box editing actions

Action	Keys
Duplicate	⌘ + D
Duplicate by D&D	option (alt) + D&D

Duplicate

Duplicate by drag&drop

Panel actions

Opens/closes each panel.

Action	Keys
Open the Pages Panel	⌘ + 1
Open the Layers Panel	⌘ + 2
Open the Add Panel	⌘ + 3
Open the Styles Panel	⌘ + 4
Open the Box Settings Panel	⌘ + 5

Layer actions

By clicking the layer toggle > with holding down the option key, it will open and close all the layers under it. Knowing this shortcut made all the difference in my working speed!

Action	Keys
Open/close all Layers	option (alt) + click

Page actions

Go back to the page that was opened before the page you are currently opening. These might be Google Chrome's shortcuts.

Action	Keys
Previous Page	⌘ + ←
Next Page	⌘ + →

Richtext editing

You can use these shortcuts in CMS article bodies and rich text boxes. In addition to the shortcuts below, you can use markdown as well.

Action	Keys
Make the selection bold	⌘ + B
Italicize the selection	⌘ + I
Underline the selection	⌘ + U
Make the selection strikethrough	⌘ + shift + S
Add the link to the selection	⌘ + K

If you know any other shortcuts, please let me know!

Original Post: STUDIOの公式＆非公式キーボードショートカット一覧 (Japanese article)

DEV Community: Ayumi Sato

Importing Toots from Mastodon to WordPress Posts Using Google Apps Script

Objectives

What Was Not Done

Requirements

Obtaining the Mastodon Access Token

Google Apps Script Code

Download the WXR File

Import into WordPress

Import External Images into WordPress

Using the Auto Upload Images Plugin

Set Featured Images

Using XO Featured Image Tools

Impressions

Official & Unofficial STUDIO Keyboard Shortcuts List

Officially announced shortcuts

Box editing actions

Box selection actions

Editor action

Unofficial shortcuts

Box editing actions

Duplicate

Duplicate by drag&drop

Panel actions

Layer actions

Page actions

Richtext editing