DEV Community: Artur Piszek

Live demo of your WP Plugin with GH releases and WP Playground

Artur Piszek — Sat, 04 Jan 2025 16:37:44 +0000

I have been relying on Github actions to publish my plugins to the .org repositiory for some now. It feels magical – you publish a release and it builds, zips and uploads the plugin wherever it needs to go.

With WP Playground, you can take that process to another level – the moment you hit publish, new version of your plugin will be immediately available for testing right in browser, without installing and exposing existing sites. This means anybody can try out your plugin before downloading, without installing it with zero downsides, in seconds.

What is WordPress Playground?

Playground is a way to run an entire instance of WordPress in the browser, with no external backend. It is destroyed on reload, but perfect for trying out changes before installing on your site

Action to build plugin on release

I’m assuming you are using Github as a sane person. This Github action will run your plugin build process, zip it, and upload it as a release asset. Remember to change your zip file name.

name: Build and Upload Plugin ZIP
on:
  release:
    types: [published]

jobs:
  build-and-zip:
    runs-on: ubuntu-latest

    steps:
    - name: Checkout Repository
      uses: actions/checkout@v4

    - name: Set up Node.js
      uses: actions/setup-node@v3
      with:
        node-version: 20 # Use Node.js v20

    - name: Install Node.js Dependencies
      run: npm install

    - name: Set up PHP and Composer
      uses: shivammathur/setup-php@v2
      with:
        php-version: '8.2' # Adjust PHP version as needed
        extensions: mbstring, zip

    - name: Install Composer Dependencies (Production Only)
      run: composer install --no-dev --optimize-autoloader

    - name: Build Plugin
      run: npm run build

    - name: Create Plugin ZIP
      run: npm run plugin-zip

    - name: Upload Release Asset
      uses: actions/upload-release-asset@v1
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      with:
        upload_url: ${{ github.event.release.upload_url }}
        asset_path: ${{github.workspace}}/wp-personal-os.zip
        asset_name: wp-personal-os.zip
        asset_content_type: application/zip

You will also have to make sure your package.json has npm scripts:

    "scripts": {
        "build": "wp-scripts build",
        "plugin-zip": "wp-scripts plugin-zip",

Remember to give your Github actions write access:

Link to latest plugin zip

There is undocumented URL for the “release asset of the latest release” on Github:

https://github.com/USER/PROJECT/releases/latest/download/ASSET.zip

Replace user, project and asset accordingly

Create your first release

Create your first Github release for the action to run

Your Playground Blueprint

With this link, you can now create a playground blueprint that will install the plugin to a temporary WordPress site that runs in the browser. You can also add additional steps that will create sample data for the best demo experience. Here is the very basic blueprint:

{
  "steps": [
    {
      "step": "installPlugin",
      "pluginData": {
        "resource": "url",
        "url": "https://github.com/USER/PROJECT/releases/latest/download/ASSET.zip"
      }
    }
  ],
  "login": true
}

I highly recommend playground builder to work on these blueprints and debug them.

Once you have your blueprint, you can create a special link and drop it in the README of your plugin, like here

Link will load playground
It will install latest release of your plugin
redirect the user to the proper page so they can see what your plugin is all about

The post Live demo of your WP Plugin with GH releases and WP Playground appeared first on Artur Piszek.

WordPress Data Views: Basic setup

Artur Piszek — Sun, 22 Dec 2024 18:30:35 +0000

WordPress has some sweet, sweet new APIs. One of them I am super excited about is Data Views. Data Views is the API to introduce modern table views into WordPress and I was excited to try it out for my Personal OS plugin.

Example: Custom taxonomy

In my example, I wanted to create a “Bucketlist” page in my WP-Admin to render a custom taxonomy that would hold my Bucketlist. We will need 3 elements:

We will need to hook into PHP to create a new WP-Admin page and render React there
React code to display Data Views
A build process to compile this React code.

React Gutenberg code

I will start with the actual Gutenberg React code. I put my code together using this fantastic tutorialand the code from WordPress Storybook

import { Icon, __experimentalHStack as HStack } from '@wordpress/components';
import domReady from '@wordpress/dom-ready';
import { useState, useMemo, createRoot } from '@wordpress/element';
// Per https://github.com/WordPress/gutenberg/tree/trunk/packages/dataviews :
// Important note If you're trying to use the DataViews component in a WordPress plugin or theme and you're building your scripts using the @wordpress/scripts package, you need to import the components from @wordpress/dataviews/wp instead of @wordpress/dataviews.
import { DataViews, filterSortAndPaginate } from '@wordpress/dataviews/wp';
import { __ } from '@wordpress/i18n';
import { useEntityRecords } from '@wordpress/core-data';
import './style.scss';
import { trash, flag } from '@wordpress/icons';

function NotebookAdmin() {
    const [view, setView] = useState( {
        type: 'table',
        search: '',
        page: 1,
        perPage: 100,
        fields: ['name', 'description', 'flags', 'count'],
        layout: {},
        filters: [],
        sort: {
            order: 'asc',
            orderby: 'name',
        },
    } );

    // Our setup in this custom taxonomy.
    const fields = [
        {
            label: __( 'Name', 'your-textdomain' ),
            id: 'name',
            enableHiding: false,
            enableGlobalSearch: true,
            type: 'string',
        },
        {
            label: __( 'Description', 'your-textdomain' ),
            id: 'description',
            enableSorting: false,
            enableGlobalSearch: true,
            type: 'string',
        },
        {
            label: __( 'Flags', 'your-textdomain' ),
            id: 'flags',
            header: (
                <HStack spacing={ 1 } justify="start">
                    <Icon icon={ flag } />
                    <span>{ __( 'Flags', 'your-textdomain' ) }</span>
                </HStack>
            ),
            type: 'array',
            render: ( { item } ) => {
                return item.meta?.flag?.join( ', ' ) || '';
            },
            enableSorting: false,
        },
        {
            label: __( 'Count', 'your-textdomain' ),
            id: 'count',
            enableSorting: true,
            enableGlobalSearch: false,
            type: 'number',
        },
    ];

    // We will use the entity records hook to fetch all the items from the "notebook" custom taxonomy
    const { records } = useEntityRecords( 'taxonomy', 'notebook', {
        per_page: -1,
        page: 1,
        hide_empty: false,
    } );

    // filterSortAndPaginate works in memory. We theoretically could pass the parameters to backend to filter sort and paginate there.
    const { data: shownData, paginationInfo } = useMemo( () => {
        return filterSortAndPaginate( records, view, fields );
    }, [view, records] );

    return (
        <DataViews
            getItemId={ ( item ) => item.id.toString() }
            paginationInfo={ paginationInfo }
            data={ shownData }
            view={ view }
            fields={ fields }
            onChangeView={ setView }
            actions={ [
                {
                    id: 'delete',
                    label: __( 'Delete', 'your-textdomain' ),
                    icon: trash,
                    callback: async ( items ) => {
                        // Implement delete functionality
                        console.log( 'Delete items:', items );
                    },
                },
            ] }
            defaultLayouts={ {
                table: {
                    // Define default table layout settings
                    spacing: 'normal',
                    showHeader: true,
                },
            } }
        />
    );
}

domReady( () => {
    const root = createRoot( document.getElementById( 'bucketlist-root' ) );
    root.render( <NotebookAdmin /> );
} );

In your plugin you have to import the components from @wordpress/dataviews/wp instead of @wordpress/dataviews. I expect this is a recent change and it will be further evolve. You also need to add

“@wordpress/dependency-extraction-webpack-plugin”: “^6.14.0”

To your devpependencies (check at the end of this post)

If you you import directly from @wordpress/dataviews you will get errors like these (click to expand)

column-header-menu.js:105 Uncaught TypeError: Cannot read properties of undefined (reading 'Group')
    at HeaderMenu (column-header-menu.js:105:44)
    at renderWithHooks (react-dom.js?ver=18.3.1:15496:20)
    at updateForwardRef (react-dom.js?ver=18.3.1:19255:22)
    at beginWork (react-dom.js?ver=18.3.1:21685:18)
    at HTMLUnknownElement.callCallback (react-dom.js?ver=18.3.1:4151:16)
    at Object.invokeGuardedCallbackDev (react-dom.js?ver=18.3.1:4200:18)
    at invokeGuardedCallback (react-dom.js?ver=18.3.1:4264:33)
    at beginWork$1 (react-dom.js?ver=18.3.1:27500:9)
    at performUnitOfWork (react-dom.js?ver=18.3.1:26606:14)
    at workLoopSync (react-dom.js?ver=18.3.1:26515:7)
react-dom.js?ver=18.3.1:18714 The above error occurred in the <AddFilterMenu> component:

    at AddFilterMenu (http://localhost:8901/wp-content/plugins/personalos/modules/bucketlist/js/build/admin.js?ver=5a41136…:352:3)
    at div
    at FiltersToggle (http://localhost:8901/wp-content/plugins/personalos/modules/bucketlist/js/build/admin.js?ver=5a41136…:742:3)
    at div
    at http://localhost:8901/wp-includes/js/dist/components.js?ver=490baf0…:14024:47
    at UnforwardedView (http://localhost:8901/wp-includes/js/dist/components.js?ver=490baf0…:14903:3)
    at UnconnectedHStack (http://localhost:8901/wp-includes/js/dist/components.js?ver=490baf0…:32221:23)
    at div
    at http://localhost:8901/wp-includes/js/dist/components.js?ver=490baf0…:14024:47
    at UnforwardedView (http://localhost:8901/wp-includes/js/dist/components.js?ver=490baf0…:14903:3)
    at UnconnectedHStack (http://localhost:8901/wp-includes/js/dist/components.js?ver=490baf0…:32221:23)
    at div
    at DataViews (http://localhost:8901/wp-content/plugins/personalos/modules/bucketlist/js/build/admin.js?ver=5a41136…:2415:3)
    at BucketlistAdmin (http://localhost:8901/wp-content/plugins/personalos/modules/bucketlist/js/build/admin.js?ver=5a41136…:14023:87)

Consider adding an error boundary to your tree to customize error handling behavior.
Visit https://reactjs.org/link/error-boundaries to learn more about error boundaries.
3
react-dom.js?ver=18.3.1:18714 The above error occurred in the <ForwardRef(HeaderMenu)> component:

    at HeaderMenu (http://localhost:8901/wp-content/plugins/personalos/modules/bucketlist/js/build/admin.js?ver=5a41136…:3894:3)
    at th
    at tr
    at thead
    at table
    at ViewTable (http://localhost:8901/wp-content/plugins/personalos/modules/bucketlist/js/build/admin.js?ver=5a41136…:4363:3)
    at DataViewsLayout (http://localhost:8901/wp-content/plugins/personalos/modules/bucketlist/js/build/admin.js?ver=5a41136…:1526:69)
    at div
    at DataViews (http://localhost:8901/wp-content/plugins/personalos/modules/bucketlist/js/build/admin.js?ver=5a41136…:2415:3)
    at BucketlistAdmin (http://localhost:8901/wp-content/plugins/personalos/modules/bucketlist/js/build/admin.js?ver=5a41136…:14023:87)

Consider adding an error boundary to your tree to customize error handling behavior.
Visit https://reactjs.org/link/error-boundaries to learn more about error boundaries.
react-dom.js?ver=18.3.1:26972 Uncaught TypeError: Cannot read properties of undefined (reading 'Item')
    at add-filter.js:32:31
    at Array.map (<anonymous>)
    at AddFilterMenu (add-filter.js:31:1)
    at renderWithHooks (react-dom.js?ver=18.3.1:15496:20)
    at mountIndeterminateComponent (react-dom.js?ver=18.3.1:20113:15)
    at beginWork (react-dom.js?ver=18.3.1:21636:18)
    at beginWork$1 (react-dom.js?ver=18.3.1:27475:16)
    at performUnitOfWork (react-dom.js?ver=18.3.1:26606:14)
    at workLoopSync (react-dom.js?ver=18.3.1:26515:7)
    at renderRootSync (react-dom.js?ver=18.3.1:26483:9)

PHP end

In PHP, you have to:

set up a WP-Admin page
enqueue your script
enqueue your style

add_action( 'admin_menu', array( $this, 'add_admin_menu' ) );
add_action( 'admin_enqueue_scripts', array( $this, 'enqueue_admin_scripts' ) );

public function add_admin_menu(): void {
    add_menu_page(
        'Bucketlist',
        'Bucketlist',
        'read',
        'bucketlist',
        array( $this, 'render_admin_page' ),
        'dashicons-list-view'
    );
}

public function render_admin_page(): void {
    ?>
    <div class="wrap">
        <h1><?php echo esc_html( get_admin_page_title() ); ?></h1>
        <div id="bucketlist-root"></div>
    </div>
    <?php
}

public function enqueue_admin_scripts( string $hook ): void {
    if ( 'toplevel_page_bucketlist' !== $hook ) {
        return;
    }

    $asset_file = plugin_dir_path( __FILE__ ) . 'js/build/admin.asset.php';
    $asset = require $asset_file;

    wp_enqueue_script(
        'bucketlist-admin',
        plugin_dir_url( __FILE__ ) . 'js/build/admin.js',
        $asset['dependencies'],
        $asset['version'],
        false
    );

    wp_enqueue_style(
        'bucketlist-admin',
        plugin_dir_url( __FILE__ ) . 'js/build/style-admin.css',
        array(),
        $asset['version']
    );
}

Build code

This is my package.json. Remember to add ”@wordpress/dependency-extraction-webpack-plugin”: “^6.14.0″ in this specific version.

Otherwise, wp-scripts will treat dataviews as WordPress dependency and it won’t render in your app!

{
    "name": "bucketlist",
    "version": "1.0.0",
    "description": "Bucketlist module for PersonalOS",
    "scripts": {
        "build": "wp-scripts build js/src/admin.js --output-path=js/build",
        "start": "wp-scripts start js/src/admin.js --output-path=js/build"
    },
    "dependencies": {
        "@wordpress/components": "^25.0.0",
        "@wordpress/data": "^9.0.0",
        "@wordpress/dataviews": "^4.10.0",
        "@wordpress/element": "^5.0.0"
    },
    "devDependencies": {
        "@wordpress/scripts": "^30.7.0",
        "@wordpress/dependency-extraction-webpack-plugin": "^6.14.0"
    }
}

For style tweaks and the rest you can refer to this fantastic walkthrough

The post WordPress Data Views: Basic setup appeared first on Artur Piszek.

Building sync engines in WordPress

Artur Piszek — Fri, 21 Jun 2024 13:50:11 +0000

Let’s say you want to synchronize some data in your WordPress with an external service. How do you do that? For my little “Second Brain” WordPress plugin, I have implemented 2 sync services so far:

Readwise
Evernote

Both of them synchronize data with a custom post type called “notes“. Here is what I learned

WP-Cron

All sync software has some kind of background service that looks for changes and syncs the detected ones. We do not have operating system access in WordPress to fire an actual process, but we have wp-cron.

WP-Cron is a job management system designed for periodic or scheduled maintenance actions that need to be performed: like publishing scheduled posts, checking plugin updates etc.

So here is what we are going to do:

We are going to create an hourly cron job for our sync.
Each job run will work through a batch – a limited number of elements (I gravitated towards 100 of updates)
Whenever this job finishes a run, it will check if more updates are waiting to sync. If so,
- It will cancel the next run (in one hour)
- Schedule itself in 1 minute
- Unschedule any “regular” – hourly sync events so that at any time, there is only one sync event waiting

We are doing all these because:

We don’t want any of these runs to be too compute and memory-intensive. WordPress is running on a variety of hosts with different limitations, and we don’t want our run to be killed
We don’t know how many updates in total we are expecting. I have 8000+ notes in my Evernote account, so the initial sync with Evernote will take quite some time. That’s why we redo the job every minute if there are pending updates
We don’t want to poll any external service too often because rate limits are a pain to manage

New cron job

I have a little abstraction layer of modules in my code, but essentially, each service has its own module.

class External_Service_Module extends POS_Module {
    public $id = 'external_service';
    public $name = 'External Service';

    public function get_sync_hook_name() {
        return 'pos_sync_' . $this->id;
    }

    public function register_sync( $interval = 'hourly' ) {
        $hook_name = $this->get_sync_hook_name();
        add_action( $hook_name, array( $this, 'sync' ) );
        if ( ! wp_next_scheduled( $hook_name ) ) {
            wp_schedule_event( time(), $interval, $hook_name );
        }
    }

    public function sync() {
        $this->log( 'EMPTY SYNC' );
    }
}

So here is a simplified sync method for my Evernote module. You can see the full code here.

class Evernote_Module extends External_Service_Module {
    public $id = 'evernote';
    public $name = 'Evernote';
    public $description = 'Syncs with evernote service';
    public $advanced_client = null;

    public function __construct( \POS_Module $notes_module ) {

        $this->register_sync( 'hourly' );
    }

    /**
     * Sync with Evernote. This is triggered by the cron job.
     *
     * @see register_sync
     */
    public function sync() {
        $this->log( 'Syncing Evernote triggering ' );
        $usn = get_option( $this->get_setting_option_name( 'usn' ), 0 );

        $sync_chunk = $this->advanced_client->getNoteStore()->getFilteredSyncChunk( $usn, 100, $sync_filter );
        if ( ! $sync_chunk ) {
            $this->log( 'Evernote: Sync failed', E_USER_WARNING );
            return;
        }
        if ( ! empty( $sync_chunk->chunkHighUSN ) && $sync_chunk->chunkHighUSN > $usn ) {
            // We want to unschedule any regular sync events until we have initial sync complete.
            wp_unschedule_hook( $this->get_sync_hook_name() );
            // We will schedule ONE TIME sync event for the next page.
            update_option( $this->get_setting_option_name( 'usn' ), $sync_chunk->chunkHighUSN );
            wp_schedule_single_event( time() + 60, $this->get_sync_hook_name() );
            $this->log( "Scheduling next page chunk with cursor {$sync_chunk->chunkHighUSN}" );
        } else {
            $this->log( 'Evernote: Full sync completed' );
        }
        // ACTUALLY PROCESS ITEMS HERE
    }

}

I have a very similar sync code for Readwise:

<?php

class Readwise extends External_Service_Module {
    public $id = 'readwise';
    public $name = 'Readwise';
    public $description = 'Syncs with readwise service';

    public function __construct( $notes_module ) {
        $this->register_sync( 'hourly' );
    }

    public function sync() {
        $this->log( '[DEBUG] Syncing readwise triggering ' );

        $query_args = array();
        $page_cursor = get_option( $this->get_setting_option_name( 'page_cursor' ), null );
        if ( $page_cursor ) {
            $query_args['pageCursor'] = $page_cursor;
        } else {
            $last_sync = get_option( $this->get_setting_option_name( 'last_sync' ) );
            if ( $last_sync ) {
                $query_args['updatedAfter'] = $last_sync;
            }
        }

        $request = wp_remote_get(
            'https://readwise.io/api/v2/export/?' . http_build_query( $query_args ),
            array(
                'headers' => array(
                    'Authorization' => 'Token ' . $token,
                ),
            )
        );
        if ( is_wp_error( $request ) ) {
            $this->log( '[ERROR] Fetching readwise ' . $request->get_error_message(), E_USER_WARNING );
            return false; // Bail early
        }

        $body = wp_remote_retrieve_body( $request );
        $data = json_decode( $body );
        $this->log( "[DEBUG] Readwise Syncing {$data->count} highlights" );

        //phpcs:ignore WordPress.NamingConventions.ValidVariableName.UsedPropertyNotSnakeCase
        if ( ! empty( $data->nextPageCursor ) ) {
            // We want to unschedule any regular sync events until we have initial sync complete.
            wp_unschedule_hook( $this->get_sync_hook_name() );
            // We will schedule ONE TIME sync event for the next page.
            //phpcs:ignore WordPress.NamingConventions.ValidVariableName.UsedPropertyNotSnakeCase
            update_option( $this->get_setting_option_name( 'page_cursor' ), $data->nextPageCursor );
            wp_schedule_single_event( time() + 60, $this->get_sync_hook_name() );
            $this->log( "Scheduling next page sync with cursor {$data->nextPageCursor}" );
        } else {
            $this->log( '[DEBUG] Full sync completed' );
            update_option( $this->get_setting_option_name( 'last_sync' ), gmdate( 'c' ) );
            delete_option( $this->get_setting_option_name( 'page_cursor' ) );
        }

        foreach ( $data->results as $book ) {
            $this->sync_book( $book );
        }
    }

}

Two-way sync

In the case of Evernote, we can have a two-way sync. When you update something in Evernote, the above job will trickle the updates to WordPress. If you update something in WordPress, it should reflect in Evernote.

The main trap is an update loop. We want to avoid a loop where you update something in WordPress, Evernote picks up the change and updates there, so WordPress gets the notification, updates in Evernote…

So here is what we are going to do:

We are going to hook into save_post_ hook. This means we don’t have to really do sync. We assume WordPress is online, which means that we can push updates synchronously when they happen, not after some time, hopefully reducing conflicts.
We are going to update the edited post with the returned Evernote content immediately on save. That way:
- Even if Evernote changes the data on their end, we are sure that after post saving the data is similar on both ends
- We can calculate the bodyHash and other indicators to know if data got out of sync
Since we are going to update WordPress end again after pushing data to Evernote, we have to remember to unhook save_post_ hook to prevent recursion.
This is not relevant to WordPress, but because Evernote has a limited syntax, we have to strip out and convert some data.

Here is the simplified code. You can read the whole thing here.

/**
* This is hooked into the save_post action of the notes module.
* Every time a post is updated, this will check if it is in the synced notebooks and sync it to evernote.
* It will then receive the returned content and update the post, so some content may be lost if it is not handled by evernote
*
* @param int $post_id
* @param \WP_Post $post
* @param bool $update
*/
public function sync_note_to_evernote( int $post_id, \WP_Post $post, bool $update ) {
    $guid = get_post_meta( $post->ID, 'evernote_guid', true );

    if ( $guid ) {
        $note = $this->advanced_client->getNoteStore()->getNote( $guid, false, false, false, false );
        if ( $note ) {
            $note->title = $post->post_title;
            $note->content = self::html2enml( $post->post_content );
            $result = $this->advanced_client->getNoteStore()->updateNote( $note );
        }
        return;
    }

    // edam note
    $note = new \EDAM\Types\Note();
    $note->title = $post->post_title;
    $note->content = self::html2enml( $post->post_content );

    $result = $this->advanced_client->getNoteStore()->createNote( $note );
    if ( $result ) {
        $this->update_note_from_evernote( $result, $post );
    }

}

/**
* This is called when a note is updated from evernote.
* It will update the post with the new data.
* It is triggered by both directions of the sync:
* - When a note is updated in evernote, it will be updated in WordPress
* - When a note is updated in WordPress, it will be updated in evernote and then the return will be passed here.
*
* @param \EDAM\Types\Note $note
* @param \WP_Post $post
* @param bool $sync_resources - If true, it will also upload note resources. We want this in most cases, EXCEPT when we are sending the data from WordPress and know the response will not have new resources for us.
*/
public function update_note_from_evernote( \EDAM\Types\Note $note, \WP_Post $post, $sync_resources = false ): int {
    remove_action( 'save_post_' . $this->notes_module->id, array( $this, 'sync_note_to_evernote' ), 10 );
    // updates updates 
    add_action( 'save_post_' . $this->notes_module->id, array( $this, 'sync_note_to_evernote' ), 10, 3 );
    return $post->ID;
}

Gutenberg blocks as sync items

Evernote’s core primitive is a note. But in the case of Readwise sync, I was stuck between 2 primitives:

Individual highlight
A book/article/podcast

I decided to store the books/articles as post type (of note), but keep highlights’ individual existence as blocks of the Readwise type. Each block is tracked individually.

Each instance of the block is appended in PHP – I published how to do it in a separate tiny tutorial.

You can see how the block is implemented in this pull request:

https://github.com/artpi/PersonalOS/pull/1

Subscribe to my newsletter to get updates my Personal OS plugin and my other musings:

The post Building sync engines in WordPress appeared first on Artur Piszek.

Prompt hacking is Oxygen

Artur Piszek — Thu, 05 Oct 2023 14:34:11 +0000

If Communication is Oxygen, and Prompt hacking is essentially communication, then…

1: Communication is Oxygen

At Automattic, we like to say that communication is the oxygen of a distributed company. Why is that exactly?

For remote work to work, you have to provide sufficient context so your coworkers are on the same page. They cannot grab you across the desk to clarify what you meant, and any follow-up question can take up to a full day to be answered.

We try to share context early, and preemptively. We prepare information for each other to be easily surfaced and reached. We stuff the context window.

2: Prompt hacking is Async Communication.

With the release of ChatGPT, prompt hacking became a thing that captured the imagination of journalists around the world: “If only I describe my task in this weird way, AI will do it.“

Now, with the initial hype cycle stabilizing a bit, they are deeming it dead and a bygone fad.

For me, it was never it’s own thing. When I try to write a prompt, it is no different than describing a Github issue:

Describe the task to be done in sufficient detail
Try to be mindful of the receiver’s context (window)
Share as much information as possible to avoid follow-up questions
Describe the shape of the result
Sidenote: I use please a lot because I don’t see the prompt as something different than any other “delegation”.

I am trying to set up the other party for success and make it easier for them to deliver the result I would like to see.

3: Prompt hacking is oxygen

This post started as an interesting title, but it got me thinking about the implications:

Async communication is essentially zero-shot prompting. How can we use our async work expertise to elevate our prompts?
What other learnings from Remote work setup can make us better at working with AI models?
Can learnings from prompt hacking make us better at managing?

We’re still early.

The post Prompt hacking is Oxygen appeared first on Artur Piszek.

The AIKEA Effect

Artur Piszek — Sun, 03 Sep 2023 14:54:06 +0000

Just like the IKEA Effect, but for the AI products. Try not to overenginner when working with LLMs.

The IKEA Effect

The IKEA effect is a cognitive bias in which consumers place a disproportionately high value on products they partially created. The name refers to Swedish manufacturer and furniture retailer IKEA, which sells many items of furniture that require assembly.

Wikipedia

There is substantial evidence that people value things (including software) after they put in at least some effort, regardless of the “objective” end state.

Low-hanging fruit from the Tree of Knowledge

The current models (GPT-3.5, GPT-4, LLAMA2) are very good at working “out of the box”. So much so, that we really cannot work fast enough at integrating them. There is a broad class of “ synthesize these documents in 2-3 paragraphs ” that provides immediate business value. Subclasses of this solutions include:

Q&A mechanism for dataset X, where you enable an AI model to query your dataset and formulate an answer to your question
- Think of it as Bing Chat for your proprietary dataset
- You could also use AI for a vector search throughout your dataset, but existing search algorithms also work.
Synthesize a number of random document chunks in a format Y
- A dashboard that will surface key points
- Scan huge documents for relevant passages
Turn unstructured data into a structured format to (finally) connect one system to the other.

While tackling this low-hanging fruit has obvious business value, it is also technically not very challenging. You plug a model, craft a prompt, and you’re done. And when it works – you feel a little bit guilty.

The typical AI Developer

With the obvious business opportunities, what should companies do? Obviously, they put their best people on it.

But their best people are used to solving complicated and challenging problems. It’s in their blood to go after hard things. This is how they became “one of the best people”.

They do what they are best at: try to solve a complicated engineering problem where one might not exist. Where just shoving your problem into prompt would suffice, they immediately launch into:

Talking about training and fine-tuning models
Start building infrastructure where you could call OpenAI
Using Embeddings where existing search algorithms are enough
Build complicated pipelines using the fresh findings from AI papers

The engineering also serves a bit as a reassurance that we understand the whole stack – at least I’ll understand the parts I built.

The moat of OpenAI Wrappers

The early crop of “AI Writing Assistants” (jasper.ai, copy.ai, etc.) is not doing so well now that bigger players are catching up to them. This is often used as an illustration that it’s not worth grabbing this low-hanging fruit because this is not a viable long-term strategy.

But the fact that these companies were able to gather users, funding, and revenue just by being a wrapper over somebody else’s API is impressive.

Yes, in the age of Open Source LLMs the tech will probably not be a moat. But distribution, marketing, and speed of execution definitely can.

The multiple dividends on speed in the AI world

Not only do we not know the full extent of LLM capabilities, but we also don’t know what customers are comfortable with, what interfaces they prefer, and what improvements wait just around the corner.

Speed is always a force multiplier, but in the AI world, a shorter feedback loop is a much bigger advantage than your perfect architecture.

This is where your “best of the best” AI team can really shine. The low-hanging fruit from the tree of knowledge will keep coming for some time before we get kicked out of the garden.

The post The AIKEA Effect appeared first on Artur Piszek.

Adding ChatGPT to your Slack in multiplayer mode

Artur Piszek — Tue, 07 Mar 2023 11:14:41 +0000

I just deployed ChatGPT as a Slackbot at Automattic, and let me tell you: it’s so much better and cheaper than the ChatGPT App. Here is how it works:

You can ping @gpt in any public or private channel,
It will respond in a Slack thread.
The state is kept in that slack thread, and it counts as a continuous conversation.

Once I had the (obvious in retrospect) idea of keeping conversations as separate threads, it came together in about 3 hours of coding. The result is surprisingly awesome, as having AI help integrated with the tool we are already using (Slack) reduces cognitive load and incentivizes use.

It’s more convenient.

ChatGPT is a useful tool, but the mental load of deciding that you want to consult the AI overlord and the need to retype (or copy-paste) your message is sometimes too much of a cognitive load.

With GPT Slackbot, you can:

Ask it for advice under a message somebody else started
Consult it quickly where you already are without switching tabs
Return to past conversations easily.

My coworkers found it immediately helpful:

I enjoyed using this in a Slack thread more than the ChatGPT interface. And Slack is searchable, which is way better! Thank you for doing this !

Also, I like that @gpt replies to older threads, without the need to retype the context of the initial chat, which I found incredibly useful!

My coworkers

It’s cheaper

ChatGPT is priced at about $1 for 75 000 words, and you are billed per use without a subscription (like ChatGPT Plus). With the pay-per-use pricing, any member of your organization can play without committing to a subscription.

It’s more whimsical

But where it really shines is the interaction of:

Multiplayer mode, as anybody can jump in on the thread
Public use in the public channels, as people can easily learn from each other

It reminds me of the Midjourney discord bot, where you learn from other people’s prompts while waiting for your own, turning the wait time into a collaboration.

Implementing your own ChatGPT Slackbot

Consider yourself warned: You are going to see some PHP. This uses the recently released ChatGPT API.

SlackBot setup

First, you need to create a Slackbot App that will be able to be pinged and has appropriate permissions to respond:

Create a new App on Slack here. Make sure you select the Bot app – I chose “From Scratch”
Add Events API subscription for the app_mentions:read event
Add scopes to read public and private channels and one to post in channels ( channels:history, groups:history, chat:write )
Once you add all the scopes, install the app to your workspace. You need admin permissions to do that or help of a workspace admin.
Copy the xoxb-.... token and save it in your project as SLACK_TOKEN

Here are all the scopes our app has:

Webhook

Once you have the Slack part figured out, you can build code to handle it. This is a very simplified code assuming that you already implemented webhook verification. (You can use it without verification, but it’s a bad idea).

This webhook will be triggered each time you ping @gpt:


$payload = json_decode( file_get_contents( 'php://input') );

// Respond to url_verification challenge
if ( 'url_verification' === $payload->type && isset( $payload->challenge ) ) {
    print wp_kses( $payload->challenge, array() );
    return;
}

if (
    ! isset( $payload->type, $payload->event ) ||
    $payload->type !== 'event_callback' ||
    $payload->event->type !== 'app_mention'
) {
    return;
}

$parent_thread_id = '';
if ( isset( $payload->event->thread_ts ) ) {
    //This is a response in thread, so we need to pass the original top message.
    $parent_thread_id = sanitize_text_field( $payload->event->thread_ts );
} else {
    $parent_thread_id = sanitize_text_field( $payload->event->ts );
}
$channel = sanitize_text_field( $payload->event->channel );

queue_async_job(
    (object) array(
        'channel' => $channel,
        'thread' => $parent_thread_id,
    ),
    'slack_gpt_respond'
);

As you can see, all that it does after sanitizing some data is to trigger a job in a job system. This is because Slack expects your webhook to respond within 3 seconds.

The “Respond to url_verification challenge” code block makes sure you can submit this webhook in the slack interface. Now you are ready to paste the URL where this code runs as Slack event handler.

The job

This job:

Retrieves the backscroll of the slack thread from the Slack API
Changes the format of the messages to one used by ChatGPT API
Sends the entire list to ChatGPT API
Posts the response in the Slack thread.

That way, I can use Slack as the state, which requires no additional database and reduces complexity.

// This is the actual job triggered by your job system
function slack_gpt_respond( $job ) {
    $data = $job->data;
    if ( isset( $data->thread, $data->channel ) ) {
        $chatgt_response = get_chatgpt_response( $data->thread, $data->channel );
        slack_gpt_respond_in_thread( $data->thread, $data->channel, $chatgt_response );
    }
}

// We use WordPress hooks as the job system
add_action( 'wpj_slack_gpt_respond', 'slack_gpt_respond' );

function slack_gpt_respond_in_thread( $ts, $channel, $response ) {
    $data = [
        'channel' => $channel,
        'thread_ts' => $ts,
        'text' => $response,
    ];

    $res = wp_remote_post(
        'https://slack.com/api/chat.postMessage',
        array(
                'headers' => array(
                    'Content-type' => 'application/json; charset=utf-8',
                    'Authorization' => 'Bearer ' . SLACK_TOKEN,
                ),
            'body' => json_encode( $data ),
        )
    );

}

function slack_gpt_retrieve_backscroll( $thread, $channel ) {
    $response = wp_remote_get(
        "https://slack.com/api/conversations.replies?channel={$channel}&ts={$thread}",
        array(
                'headers' => array(
                    'Content-type' => 'application/json; charset=utf-8',
                    'Authorization' => 'Bearer ' . SLACK_TOKEN,
                ),
        )
    );
    $body = wp_remote_retrieve_body( $response );
    $data = json_decode( $body );

    return array_map( 'slack_message_to_gpt_message', $data->messages );
}

function slack_message_to_gpt_message( $message ) {
    $text = preg_replace( '#\s?<@[0-9A-Z]+>\s?#is', '', $message->text );
    $role = isset( $message->bot_id ) ? 'assistant' : 'user';
    return [
        'role' => $role,
        'content' => $text,
    ];
}

function get_chatgpt_response( $ts, $channel ) {
    $backscroll = slack_gpt_retrieve_backscroll( $ts, $channel );

    $api_call = wp_remote_post(
            'https://api.openai.com/v1/chat/completions',
            array(
                'headers' => array(
                    'Content-Type' => 'application/json',
                    'Authorization' => 'Bearer ' . OPENAI_TOKEN,
                ),
                'body' => json_encode( [
                    'model' => 'gpt-3.5-turbo', // ChatGPT
                    'messages' => $backscroll,
                ] ),
                'method' => 'POST',
                'data_format' => 'body',
                'timeout' => 120,
            )
        );
    if ( is_wp_error( $api_call ) ) {
        return;
    }

    $result = json_decode( $api_call['body'] );
    if ( is_wp_error( $results ) ) {
        return;
    }

    return $results->choices[0]->message->content;
}

The post Adding ChatGPT to your Slack in multiplayer mode appeared first on Artur Piszek.

Transcribing your iOS Voice Memos to Markdown with Whisper

Artur Piszek — Sun, 23 Oct 2022 14:53:27 +0000

Open AI has recently introduced an Open-Source library to transcribe voice recordings, and it immediatelly caught my eye. I like automating things, and transcribing memos is a great example of a high leverage automation.

Recording voice or video is much faster and easier than writing text
Text is much easier to parse and consume than video or audio.

When I record a voice memo for myself and a bot transcribes it, I have something easy to produce, but also searchable and easily read.

As my current note-taking app is Logseq, it auto-imports those markdown files to my notes database.

Installing Whisper

If you have Python, pip3 and ffmpeg ready, you should be able to run a command like:

pip3 install git+https://github.com/openai/whisper.git

Then you can run it on any audio or video file:

whisper /path/to/file

More instruction in the GH repo.

iOS Voice Memos

When iCloud sync is enabled, the voice memos from your phone sync to your laptop via icloud. This will be perfect!

The recordings are stored in a directory like this: (replace artpi with your username)

/Users/artpi/Library/Application Support/com.apple.voicememos/Recordings/

Unusually accessible for an Apple product, but a win for us! Let’s put it all together. The following code will:

Read all .m4a files from /Users/artpi/Library/Application Support/com.apple.voicememos/Recordings/
Transcribe
Save each as /Users/artpi/GIT/logseq/pages/RECORDING_NAME.md file
When re-run, it will skip the files already saved.


<?php
//https://piszek.com/?p=5874

function sync_apple_voice_memos( $dir, $save_dir, $whisper_dir ) {
    foreach ( glob( dir ) as $filename ) {
        $name = basename( $filename, ".m4a" );
        $file_to_save = $save_dir . '/Voice Memo - ' . $name . '.md' ;
        // Only proceed if file not already transcribed
        if ( ! file_exists( $file_to_save ) ) {
            $lines = [];
            exec( "{$whisper_dir} \"{$filename}\"", $lines );

            $lines = array_slice( $lines, 2 ); // Some default headers.
            $lines = array_map( function( $line ){
                return preg_replace( '#\[[0-9.:]+ --> [0-9.:]+\]\s+(.*?)$#is', '- \\1', $line );
            }, $lines );

            if ( $lines ) {
                file_put_contents( $file_to_save, implode( "\n", $lines ) );
            }
        }
    }
}

sync_apple_voice_memos(
    '/Users/artpi/Library/Application Support/com.apple.voicememos/Recordings/*.m4a',
    '/Users/artpi/GIT/logseq/pages',
    '/opt/homebrew/bin/whisper'
);

More Logseq imports

Here are some other importer ideas for Logseq:

I wrote a bunch of scripts to import everything to my @logseq, and it is glorious.

I have an overview of every day, including today.

It prevents me from opening a bunch of apps to check notifications and puts me in a more proactive frame of mind.

Here is what syncs hourly:

— Minimum Viable Polish (@artpi) August 2, 2022

The post Transcribing your iOS Voice Memos to Markdown with Whisper appeared first on Artur Piszek.

Running Tumblr on WooCommerce

Artur Piszek — Tue, 13 Sep 2022 11:20:58 +0000

Tumblr has recently released features for creators to earn a living with their art. Whenever you turn those features on, we provision an entire WordPress site, complete with WooCommerce (and WooCommerce Payments) plugins to facilitate billing. This post explains why (in case it’s not obvious).

Your new WooCommerce is hiding behind this toggle

Read more about Post+ and Tipping on Tumblr.

When I am not writing blog posts on this site, I work at WordPress.com (actually, the company is called Automattic) on helping creators earn a living without sacrificing artistic freedom. When we bought Tumblr, we wanted to extend that option to Tumblr creators as well. Only, Tumblr isn’t running WordPress.

Or is it?

At Automattic, we also work on WooCommerce – the most extensible eCommerce platform on the web. Why not extend it to power Tumblr features as well? Actually, there are lots of reasons not to do it, but we did it anyway.

How does it work?

Whenever you enable Tumblr monetization features, we:

Create a new WordPress site on WordPress.com infrastructure to serve as a payments backend
Install WooCommerce, and WooCommerce-Subscriptions plugins to power the payments
Create a Stripe account for you via WooCommerce-Payments
Create products on that site with the price point you set on Tumblr
Link that backend site to your Tumblr blog

From now on, your Tumblr blog will use that site as its billing API.

A design I originally drew many months ago, still accurate.

Why?

At Automattic, we like to eat our dogfood. And eat the dog food bowl. And eat the table under which the dog food is served. It is really surprising how the dog has managed to survive.

The above design is definitely inefficient from data perspective: Yes , we do have some duplication. Yes , we create 38 new SQL tables each time somebody decides to try out tipping on Tumblr.

But it is very efficient from an organizational perspective. We already have composable pieces for each requirement. Even if any one piece is not fitting exactly right, we would have to make it so.

WordPress.com (WPCOM)

WordPress.com is a giant multisite (It’s one installation of WordPress running all the blogs – read more about multisites). Our systems teams made it run on bare metal, effectively turning WordPress into “serverless”. New WordPress sites have a marginal cost for us. The ones with WooCommerce are slightly more problematic, but that is something we have to solve anyway.

WooCommerce

WooCommerce is an eCommerce platform based on WordPress distributed as a plugin. It is designed with extensibility in mind to suit any use case, including our billing system (dubbed “Tumblrpay”).

Checkout is the only interface of WooCommerce that we expose

Hundreds of open source contributors and our colleagues are making sure WooCommerce is reliable, secure, and a perfect solution for any payment-related use case. It powers a thriving ecosystem of extensions that make it do exactly what you want, although some use cases are easier than others.

By far, the biggest challenge of our work was making it run on the giant multisite of WordPress.com. We had to improve security, reliability, stability, and data handling. We are streamlining database structure and plugin APIs and fixing bugs that are only uncovered at this scale.

WooCommerce Payments

WooCommerce Payments is an Automattic-owned payment gateway based on Stripe. It leverages Automattic’s infrastructure and experience from running WooCommerce at scale to provide the most convenient merchant experience of any WooCommerce payment method.

Our teams are monitoring fraud, paying attention to the payment flow and a set of processes to ensure security, stability, and maintenance on the payment side.

But the main reason we made it handle Tumblr web payments and renewals is because it’s fully vertically integrated with the rest of our business and existing workflows.

Alternative designs

I am pretty confident that this is not how you would imagine monetization features on Tumblr to work. The design had the following goals:

Use each component of the existing ecosystem for its strenghts.
Find gaps in those, and improve each component so other efforts (including the Open Source users) can benefit.
Leave minimal footprint. No new dashboards, no new systems. Ideally, each subsequent billing need should be a configuration, not a custom project to be maintained.

The alternative architectures we were discussing did not meet these goals:

Combined backend for all Tumblr blogs, running on top of Stripe Connect (the obvious SAAS-like architecture with 4-5 bigger tables):
- Does not utilize our infrastructure
- Does not contribute to other products
- Requires custom management UIs and new dashboards for Fraud, Systems, and Accounting teams
- Creates issues with In-App-Purchases
Running “One Big WooCommerce” – a design where it’s still WooCommerce under the hood, but are payments are aggregated on one store, instead of per-merchant:
- Cannot use WooCommerce Payments, requiring a custom payment gateway
- Requires us to write custom logic for splitting revenue across “merchants”

In-App Purchases

Running hundreds of thousands of WooCommerce sites is easy compared to the gymnastics we had to perform to enable In-App-Purchases in the app.

We are effectively running a Marketplace on top of the Apple IAP API, and it’s not easy. I will describe it in a seperate post – subscribe to know where it’s ready:

https://piszek.com/subscribe/

Some lessons learned

This is a random assortment of lessons we learned:

Just because you find a clever way to do it, it is not always worth it. You have to design for ease of maintenance in the future
Real problems are org-chart problems. Everything technical is solvable, but organizational issues are much harder to untangle.
Chekhov’s function: If you deploy a function to a shared codebase, chances are somebody is going to use it without the same precautions as you would take. Sanitize your inputs.
It’s always the cache.
Tumblr users really want to share porn, while both Apple and Stripe really do not want that.
Running 80 000 database updates per minute is a bad idea.
Complexity grows at an exponential rate. If you have 3 untested features, when something breaks, you don’t know what you can depend on. Even if you build on a stable foundation, it’s best to do it one story at a time.
When your wp_users table is bigger than 300 million, than don’t create a user that owns more than 100 000 sites. As a role on each site is stored as a user_meta, that giant entry will create cache timeouts.
Composer autoloader filenames are incompatible with WordPress coding standards.
WooCommerce boot order is a delicate beast.
Your job is to code yourself out of the job

Kudos

I want to give huge huge huge kudos to my amazing teammates who made this challenging architecture a reality:

And the entire product team using this infrastructure to build amazing features like crabs, so you can give crabs to other Tumblr users. It has been a joy to work with them.

The post Running Tumblr on WooCommerce appeared first on Artur Piszek.

Synchronizing WordPress posts with Github README.md

Artur Piszek — Mon, 15 Aug 2022 15:36:46 +0000

I treat my WordPress blog as my “digital home” – THE place to collect my thoughts, describe projects, and publish ideas.

It works very well for writing, but my coding projects feel better on Github. They also appreciate the documentation and descriptions I attach to them.

Over time, my projects would get out of sync with their WordPress counterparts. I would add new features, change behavior or describe new use cases and would have to manually change that in WordPress, which I naturally forgot to do.

I wrote this simple snippet to embed any Github .md file inside WordPress posts or pages. Content would be refreshed from Github every hour and displays inside the post like it was written on WordPress.

Here is an example page. Almost the entire content is from the associated README.md

How to sync your WordPress posts with any markdown file on Github

Use the attached code to introduce the GitHub markdown handler to your WordPress blog. Remember to also attach Parsedown.php!
Create an amazing project solving the world’s most pressing problem, write a .md file describing it
Copy the URL to Github markdown file
Paste it to WordPress block editor, and see how it turns into the content from the file.

Here is the code:

Sort by Surprising

Artur Piszek — Wed, 29 Jun 2022 09:25:00 +0000

In the project management land, surprises are very costly. In the best case, they are introducing work you have not budgeted for, extending your timeline.

In the worst case, you discover a dependency on external vendors, or internal teams. And that is blocking you right now! Your team cannot proceed and can only sit idle and observe the roadblock removed.

Since sitting empty-handed is corrosive, drains morale and kills momentum, you undertake some filler work or some other project in the meantime. By the time your dependencies get resolved, you are deep in something else, which you have to switch back from.

Your total cost of the surprise is:

Cost_Of_a_Surprise = Time_Spent_Waiting_on_Others + Time_Spent_Integrating_Their_Solution + Time_Switching_Back_From_Filler_Work

Simple solution: Frontload the surprises

There are many project management frameworks and philosophies, but my simple heuristic is to discover surprises, so work can be put in parallel as soon as possible.

When kicking off the project, the natural instinct of engineers is double down on what they know, share that brilliant idea on how to deal with the X requirement, or use the technology Y they have been playing with in spare time. This compounds the problem, because it postpones dealing with uncertainty. The least known area holds the most surprises, and is usually the most underestimated.

I fight these urges, starting any project with the least known area first.

We will make slower progress at the beginning,
We cannot jump at those exciting ideas from the kickoff,
We will discover undetected dependencies which we can start delegate right away
We can adjust timeline much sooner

In plain SQL, SELECT tasks FROM project ORDER BY surprising DESC;

The post Sort by Surprising appeared first on Artur Piszek.

The price of free time: programmer’s guide to helping a Non-profit

Artur Piszek — Sat, 22 Jan 2022 17:03:15 +0000

This piece was originally published on Piszek.com, where I write about Remote Work, being an effective professional, payment systems, psychology and solarpunk.

Congratulations! You have decided to help out a Non-Profit. Full of energy, and good intentions, you have embarked on a journey to use your professional skills to help a cause.

It’s a win-win: Surely, with a better website / CRM / tech, they will be able to help a few more people. You, on the other hand, will meet interesting folk, do something purposeful (as opposed to optimizing button colors at your day job), and learn a few things.

Here is what you need to know to not go insane:

The benefits of helping a Non-profit

You probably have personal reasons to help a Non-profit. Working on hard problems with friends is one of the most fulfilling things you can do with your life. If you are not working on a world-changing startup and you need a respite from the drudgery of corporate existence, a Non-Profit may be your best next bet – the purpose and mission are plentiful.

Non-profits are also a great place to meet interesting, like-minded people. Working side-by-side you can make real friends and create deeper connections, than you would build by exchanging the latest plots of TV shows over coffee at work.

But there also are powerful benefits directly translating to your career.

My entire programming journey started from helping a Non-profit – a scout team I was a part of. I made my first website in 1998, graduated to building one for dad’s business, and later launched a WordPress web agency. Now I work at WordPress.com, periodically reporting to the creator of WordPress himself. During that journey, I helped my high school, a local TEDx chapter, and a non-profit supporting remote work.

Working on projects is the best way to learn – you get to experiment with real-world problems and you get to try out different approaches and fail; building that tacit knowledge that makes one an expert.

Since you are not paid for your contributions, there is a shared understanding of what can be expected of you in a Non-Profit. You have a mandate to play a little, try out things your way, and goof off. To further boost learning, it feels more like play than work, encoding the knowledge much more effectively.

The traps

As with everything in life, the downsides are directly correlated to the upsides. Yes, in a Non-Profit, you can be a bit unpredictable and inexperienced. It does not feel like work and you get a breather from a corporate feel of a professional workplace.

But guess what – other people get to do that too. If you have just reserved a weekend to finish that signup page, and the people preparing the copy (texts) decided to be unprofessional – it suddenly becomes a problem.

The Hero’s (that’s you) Journey

Let’s assume you volunteered to create a website for your favorite Non-profit. Don’t be surprised, if the whole process goes like this:

You start full of energy and ideas.
The non-profit is eager to launch a new website because they have project X coming up.
Project X is the most important thing, and the website (meaning you) is a blocker.
You jump straight into work! You cannot be a blocker, right? You ramp up and are ready to implement the most important piece.
The texts and promotional materials are not ready, despite previous promises.
You try to work around these requirements – project X is most important, right?
You get a call. It seems that the “About the Team” page is most important now.
Let’s do a photoshoot for the Team!
You still don’t have materials for project X, but you got 10 pages of UI corrections, including a bigger logo, different button colors, and some creative ideas about the slider.
You start implementing those changes, still have no materials about project X.
Wait, there are changes to the changes now. Can you revert to the old button color?
Sometime, after a few weeks, we finally got the Project X page to work. The placeholder photos you chose are still there. “About the Team” page that got 3 meetings, photoshoot, and 12 hours of your time has gotten a total of 100 visitors this month.

Things to watch out in a Non-profit

The price of free time

Professional environments have learned a long time ago, that time is money. If everybody is salaried, the easiest way to turn a profit is to stop wasting people’s time. The correlation is clear and obvious.

I do realize that corporate environments waste mindblowing eons of their employees’ time. This is due to the scale. Big organisms being less nimble is a law of physics called inertia.

Non-profits, however, have a peculiar relationship with money. They are called Non-Profits. Duh! They get funded through donations, grants, and sometimes sales – but they are incentivized not to run a tight operation. Volunteers’ time is treated as free, so wastefulness is not controlled. It’s up to you to say no, which is hard because non-profits attract precisely the people least likely to defend their time.

It’s everybody else’s side-gig, too

As I mentioned – you can learn, and experiment with new techniques and approaches. But other people do too. If your work depends on graphic design, don’t be surprised when the designer comes up with something out-of-the-box, which naturally will be harder for you to implement, than the run-of-the-mill website.

Other people, like you, will cut corners. The designer has a family to feed, probably a day job and the thing called life. She can’t check every resolution, think about dimensions of headlines when you cram 100 characters in a title and give the proper attention to everything.

Last, but not least – without salary, recognition becomes the currency. Don’t be surprised, that “about the team” is treated as the most important page on the entire website (even if the visitors don’t care) – this is the equity paid to volunteers. Being paid with recognition also drives some folk to seek more of that compensation – they will contribute to discussions, where they have not much expertise nor understanding. These are perfect bikeshedding conditions. Beware.

Non-profits are passion-driven

Most non-profits have a mission to fix a particular problem in the world. Hunger, poor education, lack of equality, climate change – these are all areas society is failing at and non-profits are stepping in to help.

Many people are driven to work on these problems because they feel strongly about putting up with the collective screwups of society. Non-profits tend to attract people who approach most of the problems with passion and purpose, with no patience for tedious reasoning.

This leads to:

Passion-driven-project-management

Urgency is the sole method of prioritization. Things are made urgent to ensure their completion, not because they actually are time-sensitive.
Since urgency=priority, the priorities are fluid over time.
Yesterday’s priority is forgotten today because somebody who feels more strongly comes in with more passion.
Flashy things are more important than fulfilling the initial purpose. If you are working on a website, prepare for multiple CTAs.

Non-profit survival techniques

These techniques helped me stay sane while working within a few organizations.

Find a senior member of the organization to “report to”. Ideally somebody with corporate experience, and some tenure inside the Non-Profit. You don’t want to report to a committee.
Never agree to do anything ASAP. Chances are, that before you get to it – the original request will change or be forgotten. Save yourself the revert. Bonus points for batching change requests into sprints.
They will promise you texts, materials, and whatever else you’ll need. You WILL NOT get them on time. Plan accordingly.
Record yourself changing stuff in the interface – this will be a good v1 for documentation so that everyone else can implement tiny changes themselves
If you are creating a website – for goodness sake, use WordPress. It will save you from reinventing the wheel.
- With WP, you have ready tutorials to send people to, so you don’t have to fix every typo yourself. Chances are, that other folks have WP experience too.
- The next person dealing with the system will know what to do with it.
Use a ready template, don’t work with an empty canvas. Yes, it will be less original than a custom-made design, but you will be able to get off the ground and focus on what’s important – content and functionality. You have no idea how many tiny details come together to make a template work. Implementing custom design without an hourly rate will lead to an endless back-and-forth on every detail. It costs them nothing to throw in another change. The constraints of an existing template work in your favor.
Every statement you hear will be over-hyped – it’s a function of passion-driven project management. You have to do the mental math of halving the emotional charge of all statements.
Remember to have fun. Despite unreasonable requests, the people you are working with are probably quite awesome. Don’t forget that, and schedule some time to meet them as people – not vendors of website updates.

Working in a Non-Profit is a process of realizing that the corporate environment has its advantages and lessons to teach you as well.

Coming to work on Monday to a well-oiled machine, where every cog (including you) is humming nicely, where the work flows seamlessly through the paths of well-established processes, where everything has its place is a refreshing experience. Of course, sometime around Wednesday you are sick of it all, yearning for the freedom and creativity you get to enjoy in your organization.

This post was first published on Piszek.com. Check out my site where you can find more articles like this one.

Your own Roam Research Quicknote REST API with Firebase

Artur Piszek — Thu, 13 Jan 2022 18:33:02 +0000

I love Roam Research (have written a lot about it), but the biggest struggle for me is the lack of the proper REST API. I have tried faking it with Puppeteer, but I needed something quick for capturing the notes on the go.

Now I can capture the notes from a variety of apps through IFTTT, or iOS shortcuts (including Siri on my watch), and I have been doing so for more than a year. As automation is important for me, this script has helped include roam into my workflows.

Hey Siri, make a note in Roam

Cool, huh?

How does it work?

iOS shortcut or IFTTT makes a POST request to Firebase DB
Every 10 minutes Roam checks that DB, fetches new notes and deletes them in DB.
Any new note shows up in my current daily page, with “#Inbox” tag

From what I have been able to gather, this is similar to what phonetonote is doing, without the need for you to manage your own DB.

What do you need to make this work for you?

You need 3 things: A Firebase project, a custom script in your Roam graph, and iOS shortcuts to start getting your data in the graph.

Firebase

Firebase is a Google Cloud offering that lets you store JSON in the cloud. With the usage you will be generating, you will be able to operate the project for free.

You need to set up a Firebase Realtime Database project
Once you set it up, you need to generate a database secret ( Project Settings -> Service Accounts -> Database Secrets)

Roam custom plugin

This custom Roam plugin will pull data from this database – you can customize path/to/quicknotes to whatever you like.

Read here on how to install custom Roam plugins

const fireBaseUrl = 'https://YOURPROJECT.firebaseio.com/path/to/quicknotes';
const fireBaseToken = '';

function zeroPad( data ) {
    if ( data < 10 ) {
        return '0' + data;
    }
    return '' + data;
}

function getTodayUid() {
    const d = new Date();
    return zeroPad( d.getMonth() + 1 ) + "-" + zeroPad( d.getDate() ) + "-" + d.getFullYear();
}
function importFromFirebase() {
    window.fetch( fireBaseUrl + '.json?auth=' + fireBaseToken )
        .then( response => response.json() )
        .then( data => {
            console.log( 'Firebase Import:', data );
            if ( ! data ) {
                return;
            }
            Object.keys( data ).forEach( key => {
                const entry = data[key];
                if ( ! entry.string ) {
                    console.warn( 'The payload needs at least a string', entry );
                    return;
                }
                entry.string += ' #Inbox';
                window.roamAlphaAPI.createBlock( {
                    "location": {"parent-uid": getTodayUid(), "order": 0 }, 
                    "block": entry
                } );
                window.fetch( fireBaseUrl + '/' + key + '.json?auth=' + fireBaseToken, { method: 'DELETE' } );
            } );
        } );

    window.setTimeout( importFromFirebase, 10 * 60 * 1000 ); // Check for more notes every 10 minutes.
}

window.setTimeout( importFromFirebase, 60 * 1000 ); // We run this a minute after Roam starts.

Start POSTing your data

Now you need to start throwing your data into the database and see it in your Roam graph!

iOS shortcuts

Here is an iOS shortcut action that will save my data in this database. You can include it in your workflows (just remember to change Text to your token and YOURPROJECT to your project).

IFTTT

You can always include it in IFTTT to provide similar functionality, or make it save all your liked Youtube videos for example:

Enjoy!