DEV Community: Datner

The Effect Tax

Datner — Mon, 13 May 2024 08:36:33 +0000

It's been over a year now since I took the effect pill and I'll probably never develop the same way as I did before. I love effect, and I am absolutely a power-user.

But at my core, I am a front-end oriented developer. I care about things like bundle size and perceived performance, and I took this into consideration when adopting effect into my stack. Although the latter point was never a concern to me with effect (I've seen complex, front-end applications running at 120 FPS powered by Effect and React), the former point seemed to be just a fact I'd have to contend with -- effect is large.

A couple of weeks ago I inherited a create-react-app project. It had many of the classic problems seen in medium-to-large React projects - AnyScript, class components, esoteric 1-off dependencies preventing updates, a million lint violations of the == kind, multiple overlapping component libraries as well as multiple overlapping styling solutions.

My task was to add four pages to the app as well as a whole bunch of features. Using the existing material was absolutely out of the question - I would have had to create my own lib from scratch if I wanted to have any shred of confidence in the app.

I started hacking, and it didn't even take a full hour before I had to wrangle untyped garbage from the legacy side of the fence for a simple HTTP call. Here were some of the problems I found almost immediately:

It wasn't parsed (i.e. there was no zod or other schema solution)
It wasn't even typed Promise<Something>, it was proudly typed as Promise<any> and then just consumed the untyped result
I wasn't there at the dawn of creation of the app, so I had no idea what the shape of the result even was - I could only follow function names and see the properties being arbitrarily accessed.

So I did research to find out what the shape was supposed to be and created a little utility:



const UnsafeTypeId: unique symbol = Symbol.for("@types/Unsafe");

/**
 * Brands `A` as an unsafe type.
 * An unsafe type is a type that was cast but never validated.
 *
 * @example
 * ```ts
 * const response = await axios.get(...)
 *
 * const data = response.data as Unsafe<Todo>
 * ```
 *
 * @author Datner<contact@datner.me>
 */
export type Unsafe<A> = A & {
  readonly [UnsafeTypeId]?: "Unsafe";
};

Now I can at least give a proper shape to the result while also acknowledging to future developers that the shape of the type wrapped by Unsafe is not actually validated.

Then I encountered another situation that prompted me to implement a different utility ... and then another ... and then another ... and on and on it went. I found myself implementing infrastructure and utilities just so I would have the basic essentials to write usable software.

First I tried just coping with it, then I tried to just let go and get the project done, but these issues continued to linger in the back of my mind ... and then it dawned on me - I was just re-implementing effect.

Worse, I'm re-implementing effect by trying to glue libraries together that were never designed to work with one another.

Oh the axios client needs the current tenant? Ok, then it needs access to the context. But it can change in the lifetime of the application and also needs the bearer token, but it might need revalidation or even a re-log. Ah, but I make too many requests for the same thing so I also need some caching. It's been nearly a month now and I've only been building infrastructure! The boring kind too! Plus I hate the result and it's full of bugs I won't see until they pop up in production!

You gotta understand, effect does not start and end with Effect or any of the other fancy modules like Stream, Schema, or PubSub. It's packed full of useful toolbelt utils like you'll get with lodash, but with full support and interop with each other and ofc everything strictly typed.

So I gave up. Whatever the effect "tax" is, it's worth it. From what we've seen, the "tax" on bundle size is about 50kb. Depending on your use-case, this may seem like a lot or very little. But considering that I added around 20k LOC to the project, as well as some new dependencies, for me it wasn't even a consideration.



$ npm i effect @effect/schema @effect/platform @effect/platform-browser

I began replacing all my imitations with the utils from effect.

To give a visual example, here is a combine function I created for a particular useQueries query from the great @tanstack/react-query (a dependency which I also added a week ago).



import { Array, Predicate, Option, pipe } from "effect"

// further down....
const combine = (
  queries: [
    UseSuspenseQueryResult<User[], Error>,
    UseSuspenseQueryResult<Recipient[], Error>,
  ],
) => {
  const [users, recipients] = queries;

  const initialRows = pipe(
    users.data,
    Array.filter((_) => Option.isSome(_.emailAddress)),
    Array.appendAll(recipients.data),
    Schema.decodeSync(Schema.Array(NormalizedRow)),
    Array.dedupeWith((a, b) => a.Email === b.Email && a.fromIDP),
  );

  const isRecipient = Predicate.or(RecipientRow.isRecipient, (_) =>
    recipients.data.some((r) => r.recepientId === _.id),
  );
  return {
    pending: queries.some((_) => _.isPending),
    initialRows,
    initialSelected: Array.filterMap(initialRows, (_) =>
      isRecipient(_) ? Option.some(_.id) : Option.none(),
    ),
  };
};

This function takes the result of two collections that have some possible overlap, cleans up users from members that don't have an email, concatenates it to the recipients into a nice wholesome (User | Recipient)[] that is unusable, uses an @effect/schema NormalizedRow schema to homogenize it into an actually useful RecipientRow[], and remove duplicate rows, keeping only the User-originated ones.

This utility also then creates a derived collection that is just the ids of the RecipientRows that either represent a Recipient or represent a User that has a corresponding Recipient (remember we deduped them).

You could glue a solution that does this using lodash, zod, and some hand-crafted stuff, but it won't be this elegant considering the definition above. Especially the normalization part and the types. In this snippet it looks like it's trivial -- it's not. This function was over 100 LOC before.

Recently, I have seen effect being referred to as "a different language" or "hard to learn" or "unreadable", including commentary from people who have never even used the library. Effect is a complete toolkit for developing enterprise-grade applications. Not an RxJS simulator or a cool way to use Result from Rust.

I also have a ton of Effect-effect usage. With all the bells and whistles! From a quick search here's a partial list of modules I use in this project

Effect
Layer
Context
Predicate
Array
Cause
Option
Config
Scope
ManagedRuntime
GlobalValue
String
HttpClient
ClientRequest
ClientResponse
Schema
Data

"Oh no! Egad! Thats too much to learn!" - You, right now, totally ignoring that you probably have learned all of these concepts, and much more, piecemeal, from scratch on every project according to whatever 30-or-so packages are included in the project

Don't worry dear reader, you don't have to learn any of these, you can be productive
with effect even when using just a single module, and learn to use the rest at your own pace. For me its the missing standard library that is internally consistent and 100% interoperable.

Cut to yesterday, I've finally finished implementing all the features I needed. 80 changed files (mostly new files), around 12K LOC added by yours truly (trimmed a lot of fat thanks to effect, but react theres still a lot of React code and CSS).

So, nows the money time. How much did I pay for my hubris?

)

Net 50kb.

effect, @effect/schema, @effect/platform, @effect/platform-browser, and react-query to boot.

80 files, 4 new routes, nearly all new code.

This was not an isolated case, as effect makes its way into more and more areas of your application, its size amortizes because you inevitably trim a lot of "fat" from your application. I removed almost all the custom utilities I had previously built and was able to uninstall several packages.

For example, my production server clocks in at just under 90kb.

Grief and effort was saved from implementing and fixing bugs in tools that I get for free just by using effect.

Additionally, effect is 100% tree-shakeable, so I didn't end up with unnecessary code in my final application bundle.

Like MooTools, knockoutjs, jquery, express, angularjs, react, redux, nextjs, rxjs, and many other powerful abstraction that changed how we write JavaScript code, I can say without hesitation that the effect "tax" was 100% worth it.

Tauri + oauth2

Datner — Wed, 19 Apr 2023 16:52:24 +0000

So you want to do native application authentication.

Before we begin, I recommend you don't just skim the code snippets, many decisions are made due to my own restrictions, security concerns, and lack of skill with rust.

I am not going to explain the mechanisms used in depth, as it would put both you and I to sleep.

Auth flow

A quick read through the fascinating and very stimulating oauth0 specification that I am sure you read in full will tell you how native app auth is recommended. Machine to Machine auth is problematic, and some flows are more secure than others.
I chose to authenticate through the browser, secured using CSRF and PKCE.

Meaning the flow is as follows:

Determine that user needs to authenticate
Call the /authorize endpoint to receive the auth_url with a PKCE challenge
Open the users browser allowing them to log in
Catch callback from idp
Check CSRF integrity
Exchange the code from the callback with a token from /oauth/token
????
🎉 Profit! 🎉

Setup

I will be using Auth0 as my authentication provider. But should work just the same with any provider.

First you should have a project.

$ pnpm create tauri-app

You can pick whatever, we're going to touch only the rust side for added security 😉

Some thing to remember, unlike web apps, we can't hide any secrets from a malicious (or curious, not gonna judge) actor. We also do not work in an isolated environment, so we should assume that all outward communication is entirely public. This means NO CLIENT SECRET my dudes, I mean it.

We need a client id, authorization url, and a token url. You get those from your auth provider.
For example, in auth0 I need to create a new Native Application. Then inside I can take the Client Id and the Domain. Those are not secret, I put them in my env vars, but they can live wherever you want

OAUTH2_CLIENT_ID=<my client id>
OAUTH2_AUTH_URL=https://<domain>/authorize
OAUTH2_TOKEN_URL=https://<domain>/oauth/token

note that these endpoints are the standard, but they might be different for you.

now to make my life easier, I'm going to use a few crates. If you're a 🦀 and know better, go ahead, I'm not your dad.

[dependencies]
tauri = { version = "1.2", features = ["window-close", "window-hide", "window-maximize", "window-minimize", "window-show", "window-start-dragging", "window-unmaximize"] }
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"
tokio = { version = "1", features = ["full"] }
axum = { version = "0.6.12", features = ["headers"] }
oauth2 = "4.3"
reqwest = { version = "0.11", default-features = false, features = ["rustls-tls", "json"] }
open = "4.0.2"

Because this post isn't long enough, I'm going to break down the usage:

tokio -- Async
axum -- The server framework
oauth2 -- The oauth2 library
reqwest -- http request libary, playes well with oauth2
open -- to open the browser without hassle

Implement auth flow

Now that everything is in place, first lets create a struct to describe our auth dependencies

#[derive(Clone)]
struct AuthState {
    csrf_token: CsrfToken,
    pkce: Arc<(PkceCodeChallenge, String)>,
    client: Arc<BasicClient>,
    socket_addr: SocketAddr
}

and initiate them in the main method.

Side note: I'm not going to include all my imports. Just follow your heart (LSP auto-import)


#[tauri::command]
async fn authenticate() {
    todo!("we'll get here soon enough")
}

fn main() {
    let (pkce_code_challenge,pkce_code_verifier) = PkceCodeChallenge::new_random_sha256();
    let socket_addr = SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 1)), 9133); // or any other port
    let redirect_url = format!("http://{socket_addr}/callback").to_string();

    let state = AuthState {
        csrf_token: CsrfToken::new_random(),
        pkce: Arc::new((pkce_code_challenge, PkceCodeVerifier::secret(&pkce_code_verifier).to_string())),
        client: Arc::new(create_client(RedirectUrl::new(redirect_url).unwrap())),
        socket_addr
    };

    tauri::Builder::default()
        .manage(state)
        .invoke_handler(tauri::generate_handler![authenticate])
        .run(tauri::generate_context!())
        .expect("error while running tauri application");
}

Now, if you're using a more compliant provider than auth0, you can use a more robust and secure method to get the socket_addr like the oauth2 spec recommends

fn get_available_addr() -> SocketAddr {
    let listener = TcpListener::bind("127.0.0.1:0").unwrap();
    let addr = listener.local_addr().unwrap();
    drop(listener);

    addr
}

fn main() {
    // ...
    let socket_addr = get_available_addr();
    // ...
}

This will make sure you are getting an unknowable available port. Making your server harder to mess with.

You will notice that I lower the PkceCodeVerifier to a String. This is because it does not implement Clone or Copy out of security concerns and it makes it really hard to pass around. For me this is safe enough, but you're free to do this your way.

the create_client function is exactly what it says on the tin. My implementation looks like this:

fn create_client(redirect_url: RedirectUrl) -> BasicClient {
    let client_id = ClientId::new(env!("OAUTH2_CLIENT_ID", "Missing AUTH0_CLIENT_ID!").to_string());

    let auth_url = AuthUrl::new(env!("OAUTH2_AUTH_URL", "Missing AUTH0_AUTH_URL!").to_string());

    let token_url = TokenUrl::new(env!("OAUTH2_TOKEN_URL", "Missing AUTH0_TOKEN_URL!").to_string());

    BasicClient::new(client_id, None, auth_url.unwrap(), token_url.ok())
        .set_redirect_uri(redirect_url)
}

now back to authenticate, we need to get the state from tauri. It does expose some State helper, but I found that passing around an AppHandle around is much more convenient to work with

#[tauri::command]
async fn authenticate(handle: tauri::AppHandle) {
    let auth = handle.state::<AuthState>();
}

Now with the state, we can create our auth url. Don't forget to add all the data you need like scopes and such.

#[tauri::command]
async fn authenticate(handle: tauri::AppHandle) {
    let auth = handle.state::<AuthState>();

    // The 2nd element is the csrf token.
    // We already have it so we don't care about it.
    let (auth_url, _) = auth
        .client
        .authorize_url(|| auth.csrf_token.clone())
        // .add_scope(...)
        .set_pkce_challenge(auth.pkce.0.clone())
        .url();
}

Before we open the browser with our newly-created url, we should spawn a server to actually listen to the callback. For that I defined a run_server function

async fn authorize() -> impl IntoResponse {
  todo!("woo hoo!")
}

async fn run_server(
    handle: tauri::AppHandle,
) -> Result<(), axum::Error> {
    let app = Router::new()
        .route("/callback", get(authorize))
        .layer(Extension(handle.clone()));

    let _ = axum::Server::bind(&handle.state::<AuthState>().socket_addr.clone())
        .serve(app.into_make_service())
        .await;

    Ok(())
}

and spawn it

#[tauri::command]
async fn authenticate(handle: tauri::AppHandle) {
    // ...
    let server_handle = tauri::async_runtime::spawn(async move { run_server(handle).await });
}

Now it's time to open the browser with the auth link using the open crate we added or some other method.

#[tauri::command]
async fn authenticate(handle: tauri::AppHandle) {
    // ...
    open::that(auth_url.to_string()).unwrap();
}

lets get back to authorize to implement it.
The request we are expecting to receive is a GET request to the /callback endpoint with code and state (<- CSRF).

Thankfully, axum makes my life a bit easier

#[derive(Deserialize)]
struct CallbackQuery {
    code: AuthorizationCode,
    state: CsrfToken,
}

async fn authorize(query: Query<CallbackQuery>) -> impl IntoResponse {
    todo!("very cool, thanks axum")
}

We will also need the AppHandle we used earlier to access the state

async fn authorize(
    handle: Extension<tauri::AppHandle>,
    query: Query<CallbackQuery>
) -> impl IntoResponse {
    let auth = handle.state::<AuthState>();
}

Now we just gotta check the CSRF token, and exchange our code with the actual token, don't forget to attach the PKCE verifier.

async fn authorize(
    handle: Extension<tauri::AppHandle>,
    query: Query<CallbackQuery>
) -> impl IntoResponse {
    let auth = handle.state::<AuthState>();

    if query.state.secret() != auth.csrf_token.secret() {
        println!("Suspected Man in the Middle attack!");
        return "authorized".to_string(); // never let them know your next move
    }

    let token = auth
        .client
        .exchange_code(query.code.clone())
        .set_pkce_verifier(PkceCodeVerifier::new(auth.pkce.1.clone()))
        .request_async(async_http_client)
        .await
        .unwrap();

    "authorized".to_string()
}

Ok now what? You tell me. You have the token!
you can use keyring to store the token, you can use a channel to broadcast back to authenticate that you got a token so it can close the server (which is what I did). The world is your authenticated oyster :)

Closing words

Despite the length of this post, this solution is not complete! You, the reader, will have to implement token refresh and usage, you will have to implement server control, you will have to configure your oauth2 provider with the correct parameters. But for the sake of brevity I kept it to the absolute bare essentials.

If there is a missing piece that you think is absolutely essential or an optimization feel free to drop a comment.

Upload to S3 without losing you sanity

Datner — Mon, 23 May 2022 06:05:26 +0000

TL;DR scroll to The Actual Code header

The story

if you're like me, and you hate, and I mean HATE, working with user assets, then congratulations. You're a sane person!

I've spent days trying to find a safe, lean, intelligent, and clean (or slic (I just made that acronym up)) way to handle uploads, specifically to S3.

Now, I know there are a ton of ways. I've heard them all. I've combed through 100s of articles and most of the aws docs.
Maybe I'm dumb, or lack respect for the old ways, from the offered solutions there are hacking tools like multer and multiparty into your app, even if it doesn't actually support it like a nextjs application doesn't. Or generating hidden elements in a form sourced from some dark magic like it's 1999. Or the worst one, trying to wrangle a file to a readable steam and then to a byte array and pray that the next person reading your code was bored enough to memorize file-related api or is a huge js nerd.

Now, lets differentiate between uploading a file and uploading a file.

This isn't about getting a File inside an <input /> in a fancy way. For that there are many amazing solutions like uppy and react-dropzone that I really suggest you use over making some home-brew solution.

We're talking about what do to with the damn file once you have it. Let take the simpler use case of uploading a single file because I'm not your savior when it comes to galleries, Sorry 🤷‍♂️

So lets understand the problem. Why can't you just upload the asset directly to S3? It's technically possible, they even document how to do and and all sorts of neat stuff! Well sure, if you don't mind me snatching your credentials and uploading whatever I want, go ahead! Theres an exposure risk involved.

Usually, you'd just swallow the bitter pill and just pray that someone somewhere was motivated enough to write a sane solution recently enough as to not make it too outdated or even just broken.

But we don't want to do that.
So how do we get the file safely to where they should without getting it through our server? Well thats simple, you create a very temporary safe portal for it to go directly to your S3 bucket!

How? Well, if you go snooping around, you'll see that sometimes around 53BC, the aws-sdk package had an S3 module with a createPresignedPost method. What's that method? It creates a url and fields combo, that if used together in the alloted time, could be used to upload assets without going through your server. Wow! 😃
Too bad we don't have that any more. 🤦‍♂️

Or do we? 🤔
Snooping a bit more, we find that aws-sdk split itself into clients. Ok, that gets us @aws-sdk/client-s3. Looking even further, there's a @aws-sdk/s3-presigned-post. Ok back on track! 🥳

This is enough talk. I just wasted so much time on this I wanted to express myself and my journey.

The Actual Code

You need to first create yourself a server, lets say you already have one. It should have an endpoint to return the url and fields we need.
Add the required packages:

$ yarn add @aws-sdk/client-s3 @aws-sdk/s3-presigned-post

Supply the aws credentials however you prefer, I'll use an environment variable

AWS_ACCESS_KEY_ID=<ACCESS_KEY_ID>
AWS_SECRET_ACCESS_KEY=<SECRET_ACCESS_KEY>

and create an endpoint

// controllers/createUploadUrl.ts

import { createPresignedPost } from "@aws-sdk/s3-presigned-post"
import { S3Client } from "@aws-sdk/client-s3"

// fill your region and anything else you care about
const s3Client = new S3Client({ region: 'eu-central-1' })

// idk what framework you're using, adapt!
const handler = async (req: Request, res: Response) => {
  const { url, fields } = await createPresignedPost(s3Client, {
    Bucket: 'my-own-unique-bucket',
    Key: `my-app/assets/${req.body['fileName']}`,
    Fields: { acl: 'public-read' },
    Conditions: [{ acl: "public-read" }, { bucket: "my-own-unique-bucket" }, ["starts-with", "$key", "my-app/assets/"]

 return { url, fields }
}

Note in particular the Conditions field. That field will make sure that no surprising changes will be made to the request once you return the url and fields back to the frontend. Add that together with a self-destruct timer and not using your hidden credentials, and you have yourself a pretty darn secure portal for your clients files to fly through!

Now back in the frontend, it's just a matter of


async function sendAsset(file: File) {
  // use fetch or axios or whatever to reach your endpoint
  const { url, fields } = await createUploadUrl({
    name: `logo.${file.name.split(".").pop()}`,
  })
  // FormData accepts only a form element in the constructor, so we gotta build it ourselves
  const formData = new FormData()
  Object.entries(fields).forEach(([field, value]) => {
    formData.append(field, value)
  })
  // now the important part!
  formData.append('file', file) // Note, it has. to be called file!

  // This is also very important! Undocumented too 😢
  const headers = new Headers({'Content-Length': `${file.size + 5000}`})

  await fetch(url, {
    method: "POST",
    headers,
    body: formData
  })
}

bada bing bada boom. No hacks around it.
I hope this helps someone, I've struggling with this for too long

Tell me in the comments how you handle asset uploading 🥰

Configuring eslint and prettier in LunarVim 🌙. Bonus: Tailwindcss 🌊

Datner — Sun, 17 Apr 2022 17:01:58 +0000

Congratulations, you decided to move your web-dev activities to neovim. Specifically LunarVim. Great choice!

LunarVim does most of everything you would expect automatically 🤖 but it does not decide a formatter or linter for you.

lets keep it short and simple:

eslint

to add eslint as a linter for your project. Make sure it is installed as a devDependency or is installed globally if you're about that life.

next, while inside LunarVim, enter <leader> L c (<leader> is usually the spacebar unless you changed it) to reach the config.lua where you config, using lua, your editor 😃

now at the bottom, add the following lines

local linters = require "lvim.lsp.null-ls.linters"
linters.setup {
  { command = "eslint", filetypes = { "typescript", "typescriptreact" } }
}

just make sure that you didn't declare a local variable linters before, save, and voila! You're back in the abusive relationship that is working with eslint!

prettier

to add prettier as a formatter for your project, similar to eslint you need to have it installed somewhere reachable, and in the config.lua (<leader> L c), similar to the linters, add formatters as so:

local formatters = require "lvim.lsp.null-ls.formatters"
formatters.setup {
  {
    command = "prettier",
    filetypes = { "typescript", "typescriptreact" },
  },
}

for the keen-eyed, you can see that the config comes with an example for using prettier, so thats cool!

Bonus: TailwindCSS 🌊

Man I love me some tailwind. I love the intellisense of it and I don't like compromising. So if you're like me, and happened to use tailwind in your project. The setup is rather simple if undocumented.

first install the tailwind lsp using :LspInstall tailwindcss. Usually that is enough, but not in this case. lvim comes with configurations for tailwind, but you gotta tell him to actually set tailwind up!

the easiest way is to first install the tailwind language server globally like yarn global add @tailwindcss/language-server

then go back to your config.lua (using <leader> L c) and adding the line

require("lvim.lsp.manager").setup "tailwindcss"

Note: there is a way to use a local (read: devDeps) version of the language-server, but it's a tad convoluted and needs to be set up per-project. So Yeah 🤠

Hope that helps anybody

P.S Try to close and open your LunaVim if either of these don't immediately start up