DEV Community: Vince Fulco (It / It's)

weaviate

Vince Fulco (It / It's) — Sun, 15 Dec 2024 14:32:23 +0000

duckdb

Vince Fulco (It / It's) — Sun, 24 Nov 2024 09:14:40 +0000

DBOS-Cloud: Minimal Effort Change Data Capture (CDC) Tool

Vince Fulco (It / It's) — Sun, 28 Jul 2024 08:01:13 +0000

This is a simple DBOS example focusing on remote deployment to DBOS Cloud, their hosted solution with a generous free tier for devs.

The github repo sets up: 1) a workflow with an http api endpoint that receives events from Supabase when an INSERT is made to a table, 2) the same workflow archives the event payload to a DBOS hosted Postgres table, and 3) additional endpoints as utilities to view and delete all events while in development.

Prerequisites

Make sure you have node.js 21.x
Sign up for DBOS Cloud (https://www.dbos.dev/dbos-cloud)
Have a Supabase account and a table you are inserting in to.

Getting Started

Clone the repository and navigate to the project directory
Install the dependencies

Be sure not to commit / hard code your secrets to a public repo! This setup is meant for local development and direct deployment to the dbos-cloud service.
To deploy to DBOS-cloud, login with this, "npx dbos-cloud login" and follow the instructions to match the uuid given in the console to the one in the browser, then standard login user/password applies.
Next provision a d'base instance: "npx dbos-cloud db provision database-instance-name -U database-username"
Register your app with the d'base instance: "npx dbos-cloud app register -d database-instance-name"
To use secrets in DBOS, add your variables in the cli like this:

export PGPASSWORD=put-the-password-you-created-when-you-setup-the-remote-database-here

These will be picked up at build time and inserted into the dbos-config.yaml fields: ${PGPASSWORD}.

You will notice on line #56 of operations.ts, there is a @PostApi decorator which sets up a url with a randomly generated endpoint. This will be the receiver of events from Supabase. I created the randomly generated endpoint to protect the api (sometimes called protection by obfuscation).

**Note: this is not a recommended method! It was done as a quick and dirty dev hack. For real production use, you will want to use DBOS' authentication / authorization features **

Take the base url that DBOS returned when the deploy finished and add the randomly generated endpoint you create (instructions are in the code).

It should look something like this: https://foo-bar-dbos.cloud.dbos.dev/OBstAqG6qOv7cWXCqgg. You'll use this in the next step.

The Supabase settings

On the Supabase side, set up the trigger to publish changes externally by: 1) choosing the project you want to use, then on the left hand side menu go to "Database" --> "Webhooks". 2) create a new webhook, give it a name, choose the table to watch, and the events (in our case insert, but update and delete are also available). Make sure the http request method is set to POST, then enter the URL created by DBOS-Cloud plus the randomly generated stub. Hit "Create Webhook".

Staying in the Supabase console, go to the "Table Editor", choose your table, and add a new row with the required inputs.

Almost immediately, you should be able to query in Insomnia / Postman or curl given your specific DBOS supplied url, https://foo-bar-dbos.cloud.dbos.dev/getEventData, and see the event.

For reference, here is the format of a sample event body sent to DBOS-Cloud:

{

    "type": "INSERT",

    "table": "fooData",

    "record": {

        "id": "tempId-e71m1dtxan6b5c0dqxkbhoez",

        "email": "joe@company.com",

        "lastName": "Blow",

        "firstName": "Joe",

        "created_at": "2024-07-25T06:03:10.792595+00:00",

        "updated_at": "2024-07-25T06:03:10.792595"

},

    "schema": "public",

    "old_record": null

}

Record fields will vary based on how your table is structured. For simplicity purposes, in this example, I archive the JSON object in total vs. parsing the records further.

Reference Docs (From Official Repo)

To add more functionality to this application, modify src/operations.ts, then rebuild and redeploy it.
For a detailed tutorial, check out our programming quickstart.
To learn how to deploy your application to DBOS Cloud, visit our cloud quickstart
To learn more about DBOS, take a look at our documentation or our source code.

Resources to learn more --

Awesome-dbos

The first article in this series uses cron and Postmark email service provider.

The second article in this series aggregates and archives data from Supabase tables.

DBOS-Cloud: Fast and Free Automatic Data Aggregator

Vince Fulco (It / It's) — Sat, 20 Jul 2024 03:03:09 +0000

This is a simple DBOS app example focusing on remote deployment to DBOS Cloud, their hosted solution with a generous free tier for devs.

The github repo sets up a cron job which: 1) performs a "SELECT COUNT" on a Supabase Postgres database table and 2) an "INSERT" to a second table. Consider it a starter for a poor man's data lake aggregating data in the receiving table.

Note: This is not the fastest, most compact and recommended way to create this workflow. DBOS-Cloud has terrific in-app full Postgres database functionality. Check out the transactions section for more. I built the script this way since I had pre-existing data stored on Supabase.

Prerequisites

Make sure you have Node.js 21.x
Sign up for DBOS Cloud (https://www.dbos.dev/dbos-cloud)
Have a Supabase account and get your project url (https://foobar.supabase.co/rest/v1) and project API anon key. Check out the Supabase docs to get them.
Make sure to have two tables set up, the first can have any data in it as we will be counting rows, and the second table should have the columns "id", "count", and "created_at".

Getting Started

Clone this repository and navigate to the project directory

Install the dependencies
To deploy to DBOS-cloud, login with this, "npx dbos-cloud login" and follow the instructions to match the uuid given in the console to the one in the browser, then standard login user/password applies.
Next provision a d'base instance: "npx dbos-cloud db provision database-instance-name -U database-username"
Register your app with the d'base instance: "npx dbos-cloud app register -d database-instance-name"

Secrets Handling

To use secrets in DBOS, add your variables in the cli like this:

export PGPASSWORD=password-when-you-setup-the-remote-database-here
export SUPABASE_URL=https://foobar.supabase.co
export SUPABASE_ANON_KEY=long-foo-bar-key
export SOURCE_TABLE=source-table-name
export DESTINATION_TABLE=destination-table-name

These will be picked up at build time and inserted into the dbos-config.yaml fields: ${PGPASSWORD}, ${SUPABASE_URL}, ${SUPABASE_ANON_KEY}, ${SOURCE_TABLE}, ${DESTINATION_TABLE} respectively. Don't use quotation marks wrapping the values. The PGPASSWORD is for the basic user (not admin) Postgres service, which won't be used here.

Be sure not to commit / hard code your secrets to a public repo! This setup is locally developed and directly deployed to the dbos-cloud service.

Get ready to deploy!

And finally deploy your app: "npx dbos-cloud app deploy"

After 5 minutes, assuming you have data in your source table, check your destination table for entries.

If you are not getting expected results:

To delete the app online,npx dbos-cloud app delete [application-name] --dropdb. Remove the '--dropdb' parameter if you want to retain the database table. If you don't delete it, the cron script will continue to run.

There are so many more benefits to mention re: workflow orchestration, observability, debugging, monitoring and overall speed, security, and costs which can be explored further here: https://docs.dbos.dev/

Reference Docs (From Official Repo)

To add more functionality to this application, modify src/operations.ts, then rebuild and redeploy it.
For a detailed tutorial, check out our programming quickstart.
To learn how to deploy your application to DBOS Cloud, visit our cloud quickstart
To learn more about DBOS, take a look at our documentation or our source code.

Resources to learn more --

Awesome-dbos

The first article in this series using cron and Postmark email service provider.

DBOS-Cloud: Fast and Free Automatic Supabase Table Copier

Vince Fulco (It / It's) — Sat, 20 Jul 2024 02:52:49 +0000

This is a simple DBOS app example focusing on remote deployment to DBOS Cloud, their hosted solution with a generous free tier for devs.

Prerequisites

Make sure you have Node.js 21.x
Sign up for DBOS Cloud (https://www.dbos.dev/dbos-cloud)
Have a Supabase account and get your project url (https://foobar.supabase.co/rest/v1) and project API anon key. Check out the Supabase docs to get them.
Make sure to have two tables set up, the first can have any data in it as we will be counting rows, and the second table should have the columns "id", "count", and "created_at".

Getting Started

Clone this repository and navigate to the project directory

Install the dependencies
To deploy to DBOS-cloud, login with this, "npx dbos-cloud login" and follow the instructions to match the uuid given in the console to the one in the browser, then standard login user/password applies.
Next provision a d'base instance: "npx dbos-cloud db provision database-instance-name -U database-username"
Register your app with the d'base instance: "npx dbos-cloud app register -d database-instance-name"

Secrets Handling

To use secrets in DBOS, add your variables in the cli like this:

export PGPASSWORD=password-when-you-setup-the-remote-database-here
export SUPABASE_URL=https://foobar.supabase.co
export SUPABASE_ANON_KEY=long-foo-bar-key
export SOURCE_TABLE=source-table-name
export DESTINATION_TABLE=destination-table-name

Be sure not to commit / hard code your secrets to a public repo! This setup is locally developed and directly deployed to the dbos-cloud service.

Get ready to deploy!

And finally deploy your app: "npx dbos-cloud app deploy"

After 5 minutes, assuming you have data in your source table, check your destination table for entries.

If you are not getting expected results:

To delete the app online,npx dbos-cloud app delete [application-name] --dropdb. Remove the '--dropdb' parameter if you want to retain the database table. If you don't delete it, the cron script will continue to run.

Reference Docs (From Official Repo)

To add more functionality to this application, modify src/operations.ts, then rebuild and redeploy it.
For a detailed tutorial, check out our programming quickstart.
To learn how to deploy your application to DBOS Cloud, visit our cloud quickstart
To learn more about DBOS, take a look at our documentation or our source code.

Resources to learn more --

Awesome-dbos

The first article in this series using cron and Postmark email service provider.

DBOS-Cloud Simple and Robust Workflow Orchestration

Vince Fulco (It / It's) — Wed, 10 Jul 2024 02:50:18 +0000

This is a toy DBOS app example focusing on remote deployment to DBOS Cloud, their hosted solution with a generous free tier for devs.

The github repo sets up two simple HTTP API endpoints that: 1) sends an email using Postmark ESP when you hit /sendemail/:friend/:content endpoint and 2) inserts a record into a postgres d'base instance and retrieves the records when you visit the /emails endpoint.

Prerequisites

Make sure you have node.js 21.x
Sign up for DBOS Cloud (https://www.dbos.dev/dbos-cloud)
Have a Postmark account, username / password from the server (https://postmarkapp.com/), and a specific sending email address setup

***Since this uses nodemailer, you can easily swap out Postmark for your email service provider of choice.

Getting Started

Clone this repository and navigate to the project directory
Install the dependencies

Be sure not to commit / hard code your secrets to a public repo! This setup is meant for local development and direct deployment to the dbos-cloud service.

To deploy to DBOS-cloud, login with this, "npx dbos-cloud login" and follow the instructions to match the uuid given in the console to the one in the browser, then standard login user/password applies.
Next provision a d'base instance: "npx dbos-cloud db provision database-instance-name -U database-username"
Register your app with the d'base instance: "npx dbos-cloud app register -d database-instance-name"
To use secrets in DBOS, add your variables in the cli like this:

export POSTMARK_USER=api-key-from-postmark-server-here
export POSTMARK_PASSWORD=api-key-from-postmark-server-here
export PGPASSWORD=put-the-password-you-created-when-you-setup-the-remote-database-here
export SENDER_EMAIL=sender@foobar.com
export RECIPIENT_EMAIL=receiver@foobar.com

These will be picked up at build time and inserted into the dbos-config.yaml fields: ${PGPASSWORD}, ${POSTMARK_USER}, ${POSTMARK_PASSWORD}, ${SENDER_EMAIL}, ${RECIPIENT_EMAIL} respectively.

And finally deploy your app: "npx dbos-cloud app deploy"

After a minute or two, you'll get back an api endpoint that looks like: https://<username>-<app-name>.cloud.dbos.dev/.

Test the endpoint:

First, visit https://<username>-<app-name>.cloud.dbos.dev/sendemail/friend/content replacing 'friend' and 'content' with your own choices. Hitting this sends an email using Postmark and inserts a record into the d'base named 'postmark'.

Then visit https://<username>-<app-name>.cloud.dbos.dev/emails to retrieve the records from the d'base.

To delete the app online,npx dbos-cloud app delete [application-name] --dropdb. Remove the '--dropdb' parameter if you want to retain the database table.

Some caveats:

The endpoint is not secure, and anyone can send an email if they guess the assigned endpoint. You should add some sort of authentication to the endpoint to prevent abuse.

Here are some instructions for that: https://docs.dbos.dev/tutorials/authentication-authorization

When building workflows with DBOS, consider creating the respective standalone functions, then wrapping them in an Httpapi decorator / function, then wrap all that in a @workflow which will add a bunch of cool built-in features.

-- granular observability
-- guaranteed only once execution
-- asynchronous execution
-- precise management and observability of workflows

There are so many more benefits to mention re: debugging, monitoring and overall speed, security, and costs which can be explored further here: https://docs.dbos.dev/

Reference Docs (From Official Repo)

To add more functionality to this application, modify src/operations.ts, then rebuild and redeploy it.
For a detailed tutorial, check out our programming quickstart.
To learn how to deploy your application to DBOS Cloud, visit our cloud quickstart
To learn more about DBOS, take a look at our documentation or our source code.