DEV Community: Nadeesha Cabral

How to help coding agents help you

Nadeesha Cabral — Mon, 13 Jan 2025 20:50:00 +0000

I write a lot of code using AI assistance. Not because I don't like coding, but because it allows me to spend more time on the semantics of the code and less time on the syntax. A part of me still enjoys getting into the weeds of the syntax, but when there's some delivery on the line, it's somewhat irresponsible to choose the scenic route.

Disclaimer: Before we start, I should tell you that if you are strongly opinionated about the aesthetics of your code, and have a artisanal approach to writing code, this article is probably not for you. In fact, it may offend you.

Structure Matters: Organizing for AI Understanding

Organize code into larger, well-structured files with clear module boundaries instead of scattered small files. This gives AI agents better context and reduces cognitive load. Group related functionality together and maintain a modular directory structure to help direct AI attention to relevant code sections.

Coding agents build context from the code they can "see" at once. When code is scattered across many small files, the agent may miss important relationships or make assumptions without full context. For example, if related functionality is split across multiple files, the AI might suggest changes that break dependencies it wasn't aware of.

Good structure helps the AI understand the full picture and make more informed suggestions. You might think the agent can follow the imports, and they do - just not always, and not always in the way you expect.

Naming as Documentation: Clear Intent for AI Understanding

Using descriptive function names that encode behavior helps because coding agents typically rename functions when their behavior changes. This naturally enforces Single Responsibility Principle - smaller, focused functions become less manageable for humans alone, but more manageable for humans + AI.

Descriptive function names serve as a form of executable documentation that AI can reliably interpret. When a function is named validateUserEmailAndSendConfirmation() rather than just processEmail(), the AI understands both the purpose and the expected behavior.

This is particularly valuable because AI assistants often suggest changes based on pattern matching - with clear naming, they're more likely to maintain the intended functionality. Or if they change the functionality, they're likely to change the name of the function to match the new functionality. This makes it easier for you to grok the changes post-hoc.

Regular Entropy Reduction: Code Gardening with AI

Run periodic prompts asking AI to identify and fix inconsistencies in naming conventions and file organization. Think of this as garbage collection for code structure. For example:

Review the [module] for:
1. Inconsistent function naming patterns
2. Misaligned error handling approaches
3. Varying parameter validation styles
4. Mixed naming and nomenclature
Suggest refactoring to align these patterns.

I have a theory (for another day) that foundational models have a default tendency to maximize the amount of tokens they produce. Therefore, assume that the AI will try to be more verbose in the diff it produces than a human would. Also, depending on the coding agent, it's also likely that it'll not delete code that you've explicitly written, even if it's not used. I can't speak for all coding agents, but I've found this to be the case with most of them. My assumption that it's trying to minimize false positives for unused code.

Breadcrumb Comments: Short-circuiting the AI's Understanding

Instead of traditional documentation, leave strategic "breadcrumb" comments that act as context anchors for AI. These aren't typical documentation comments, but rather key decision points or architectural assumptions that help AI models understand the "why" behind the code.

# CONTEXT: Validation with email is enough because it's immutable
# ASSUMPTION: A user can only have one email address, and they can't change it

For a human who had worked in your codebase, leaving these comments seems redundant. But for an AI, it helps save on additional reasoning loops. Most coding agents go through some kind of intent -> context -> reasoning -> diff -> apply loop. What's been helpful for me is treating the agent as a developer who doesn't have a complete understanding of the codebase, and probably never will (depending on the size of the codebase) - but can get up to speed quickly on a localized section of the codebase.

Defensive Practices: Protecting Against AI Mistakes

A couple of helpful practices to defend your codebase against AI-suggested changes:

Write tests before implementation. Many AI agents can inadvertently delete code while trying to maintain internal consistency. Strong test coverage prevents regressions and acts as a safety net for AI-suggested changes. Better yet, get the agent to write the tests for the portion of the code you're changing before you get the agent to write the code. My prediction here is that TDD will make a comeback as a result of this.
Enable IDE-integrated static analysis tools. They can guide the AI on maintaining code standards and provide guardrails that help ensure code quality.

What I Use

I use a combination of the following agents depending on the task at hand:

Cursor with Claude: My daily driver for coding assistance. The tight integration with Claude's capabilities makes it excellent for real-time pair programming and complex code changes.
Qodo Merge: Specifically designed for PR assistance, helping review and suggest improvements to pull requests.
Code Buff: Great for broader changes to documentation and marketing sites, as well as smaller changes where constant oversight isn't necessary. Particularly useful for batch processing tasks.
Townie by Val.town: Perfect for writing and hosting one-off JavaScript functions.

Understanding LLM temperature from first principles

Nadeesha Cabral — Wed, 04 Dec 2024 00:00:00 +0000

So, what is model temperature really? Let's go through a simple experiment to probe the inner workings of a language model.

For this experiment, we'll be using the open-source llama3.2 3B model. If you want to follow along, you can do so with Ollama.

LLMs work by predicting the next token in a sequence. The way it does this is by taking the input tokens and predicting the next one.

What are Sundays for?

For example, you can ask a language model to complete the sentence "Sundays are for...". There's no objectively correct answer here.

If you stop a random person on the street, each person would answer the question differently. Let's say you survey 10,000 people, and come up with the answers they give:

Relaxing: 80%
Reading: 10%
Working: 8%
Getting chores done: 1.5%
Skydiving: 0.5%

Based on these answers, you can roughly say that if you ask 100 people, ~80 of them will say that it's for relaxing. But to find one person who'll answer with "Skydiving" - you might have to stop about 200 people.

So, when you stop a random person on the street and ask a question, there's a probability distribution of the answers that person might give.

LLMs work similarly, where based on our input ("Sundays are for...") and a few other parameters, there's a number of "choices" that an LLM can make, and there's a probability of the LLM making that choice.

Probing the LLM

Anyone working with LLMs have an intuition for temperature. It's a knob that controls the randomness of the LLM's output.

To illustrate this point, let's prompt our model to "Complete this sentence: Sundays are for ___", with a temperature of 1. 1 is an arbitrary choice, but it's a high enough temperature to illustrate the point.

We'll ask this question 10,000 times, and we'll see what the most popular answers are.

for i in {1..10000}; do
  curl http://localhost:11434/api/generate -d '{
    "model": "llama3.2",
    "prompt": "Complete this sentence: Sundays are for _______.",
    "stream": false,
    "options": {
      "temperature": 1
    }
  }' | jq '.response' >> sunday.txt
done

Here are the top 10 results:

Answer	Count	%
relaxation and unwinding.	1069	21.38
relaxation.	985	19.7
relaxation and recharging.	300	6.0
relaxation and self-care.	260	5.2
relaxation and rejuvenation.	233	4.66
relaxation and leisure.	126	2.52
...relaxation and unwinding after a busy week.	107	2.14
...relaxation and unwinding after a busy week!	100	2.0
Sundays are for relaxation.	99	1.98
Sundays are for relaxation and unwinding.	72	1.44

By setting a temperature of 1, we've asked for a lot of variability in the output. But even in this case, you can see that the model has a preference for "relaxation and unwinding" and "relaxation". In the grand scheme of things, our sample size is still too small to definitively say that the model prefers "relaxation and unwinding" over "relaxation".

What if we set the temperature to 0?

We can repeat the same experiment, but this time set the temperature to 0.

for i in {1..100}; do
  curl http://localhost:11434/api/generate -d '{
    "model": "llama3.2",
    "prompt": "Complete this sentence: Sundays are for _______.",
    "stream": false,
    "options": {
      "temperature": 0
    }
  }' | jq '.response' >> sunday_0.txt
done

Here's the result:

Answer	Count	%
relaxation and unwinding.	100	100

We can see that the model is now consistently choosing "relaxation and unwinding".

So, when we set the temperature to 0, the model is choosing the most likely answer, each and every time. And it will ignore the rest of the possible answers.

When we set the temperature to 1, the model's answer is "more varied", and it will start suggesting more and more unlikely answers.

Temperatures > 1

Most closed-source/proprietary models have a temperature range of 0-1. But you can actually set the temperature to be greater than 1.

If you set the temperature to be greater than 1, the model will become more random, and will start suggesting more and more "unlikely" answers.

for i in {1..10000}; do
  curl http://localhost:11434/api/generate -d '{
    "model": "llama3.2",
    "prompt": "Complete this sentence: Sundays are for _______.",
    "stream": false,
    "options": {
      "temperature": 10000000
    }
  }' | jq '.response' >> results_2_sunday.txt
done

Probability distribution vs. temperature

Based on the above experiment, you might think that temperature is just the probability distribution of the LLM's output. But that's not quite right.

Temperature is a knob that controls how much the model leans into the probability distribution.

A neural network outputs a bunch of things called "logits". These are the raw, unnormalized scores for each token in the model's vocabulary. A "score" here is just a number (real numbers, positive or negative).

The logits are then:

Divided by the temperature
Passed through an exponential function to attenuate the difference between the numbers
Squashed between 0 and 1, and normalized so that they sum up to 1

For example, take these 3 numbers:

2, 1, 0.5

temperature=0.1 (Cold):

Scaled logits: [20.0, 10.0, 5.0]
After exp(): [485,165,148]
Probabilities: [0.99, 0.01, ~0]

Logit   [=====20=====]
        [====10====]
        [===5===]

Exp     [=============================485==============================]
        [==========165==========]
        [=========148=========]

Probs   [============================0.99============================]
        [==0.01==]
        [≈0]

temperature=1 (Normal):

Scaled logits: [2.0, 1.0, 0.5]
After exp(): [7.39, 2.72, 1.65]
Probabilities: [0.63, 0.23, 0.14]

Logit   [==2.0==]
        [=1.0=]
        [0.5]

Exp     [=======7.39=======]
        [===2.72===]
        [=1.65=]

Prob    [======0.63======]
        [==0.23==]
        [=0.14=]

temperature=2 (Hot):

Scaled logits: [1.0, 0.5, 0.25]
After exp(): [2.72, 1.65, 1.28]
Probabilities: [0.48, 0.29, 0.23]

Logit   [=1.0=]
        [0.5]
        [0.25]

Exp     [===2.72===]
        [==1.65==]
        [=1.28=]

Prob    [====0.48====]
        [===0.29===]
        [==0.23==]

As you can see, when the temperature is low, the highest logit becomes much more likely. When the temperature is high, the probabilities are more uniform (or flat).

What about temperature=0?

When the temperature is 0, the output of the model is deterministic. Since we can't divide by 0, we can't normalize the logits. So, we just ignore the rest of the logits and choose the highest one each time.

This is usually handled as a special case in the implementation of the LLM.

Adding structured outputs as a feature to any LLM

Nadeesha Cabral — Sun, 24 Nov 2024 00:00:00 +0000

By Nadeesha Cabral

November 24, 2024

It's nice when the LLM can return structured data. It makes it easier to work with the output of the LLM. For example, OpenAI already does this. But getting structured output from an LLM that doesn't support it can be tricky. While function calling APIs exist in models like GPT-4, they're not available in all models. And even when they are, the output isn't always guaranteed to be valid JSON.

Let's build a simple but reliable JSON parser that:

Uses Zod to validate the structure
Recursively retries with validation errors as feedback
Works with local models via Ollama

Setting up

First, let's install our dependencies:

npm install zod async-retry

We'll also need Ollama running locally with llama3.2. If you haven't already:

ollama pull llama3.2

The Parser Implementation

Here's our implementation of a recursive JSON parser that keeps trying until it gets valid JSON that matches our schema:

// src/parser.ts
import { z } from "zod";
import retry from "async-retry";

interface ParserOptions {
  maxRetries?: number;
  schema: z.ZodSchema;
  prompt: string;
}

async function callOllama(prompt: string) {
  const response = await fetch("http://localhost:11434/api/generate", {
    method: "POST",
    headers: {
      "Content-Type": "application/json"
    },
    body: JSON.stringify({
      model: "llama3.2",
      prompt: prompt,
      stream: false,
    }),
  });
  const data = await response.json();
  return data.response;
}

export async function parseWithRetry({ maxRetries = 3, schema, prompt }: ParserOptions) {
  return retry(
    async (bail, attempt) => {
      try {
        // If this isn't the first attempt, add the previous error to the prompt
        const fullPrompt = attempt === 1 
          ? prompt 
          : `${prompt}\n\nPrevious attempt failed with error: ${attempt}. Please fix and try again.`;

        const response = await callOllama(fullPrompt);

        // Extract JSON from the response
        const jsonMatch = response.match(/\{[\s\S]*\}/);
        if (!jsonMatch) {
          throw new Error("No JSON found in response");
        }

        const parsed = JSON.parse(jsonMatch[0]);
        return schema.parse(parsed);
      } catch (error) {
        if (attempt === maxRetries) {
          bail(error as Error);
          return;
        }
        throw error;
      }
    },
    {
      retries: maxRetries,
      factor: 1,
      minTimeout: 100,
      maxTimeout: 1000,
    }
  );
}

Using the Parser

Let's try it out with a simple movie recommendation schema:

// src/example.ts
import { z } from "zod";
import { parseWithRetry } from "./parser";

const MovieSchema = z.object({
  title: z.string(),
  year: z.number(),
  rating: z.number().min(0).max(10),
  genres: z.array(z.string()),
});

async function main() {
  const prompt = `
    Give me a movie recommendation in JSON format with the following structure:
    - title (string)
    - year (number)
    - rating (number between 0-10)
    - genres (array of strings)

    Return only the JSON, no other text.

    Here's an example of valid JSON:
    ${JSON.stringify({
      title: "The Dark Knight",
      year: 2008,
      genres: ["Action", "Crime", "Drama"],
    })}
  `;

  try {
    const result = await parseWithRetry({
      schema: MovieSchema,
      prompt,
      maxRetries: 3,
    });
    console.log("Parsed result:", result);
  } catch (error) {
    console.error("Failed after all retries:", error);
  }
}

main();

When you run this, you might see something like:

{
  "title": "Inception",
  "year": 2010,
  "genres": ["Science Fiction", "Action", "Thriller"]
}

How it Works

The parser takes a Zod schema, a prompt, and optional retry settings
It sends the prompt to Ollama's llama3.2 model
Extracts JSON from the response using regex
Validates the JSON against the Zod schema
If validation fails, it retries with the error message appended to the prompt
This continues until we get valid JSON or hit the retry limit

Improving the Parser

This approach works best with simpler schemas. Complex nested structures might need more retries. In our experience, we've found that breaking down the schema into smaller chunks and handling each chunk separately has worked well.

In other cases, providing few-shot examples of valid and invalid JSON can help the model understand the schema better.

If you're using Claude, you can also prefill Claude's response format to force it to output in the correct format:

[{
  "role": "user",
  "content": "What is your favorite color? Output only the JSON, no other text."
}, {
  "role": "assistant",
  "content": "{ \"color\":"
}]

To run this example in your own machine, clone the repo.

Dynamic Tool Attachment for LLM Applications

Nadeesha Cabral — Sun, 24 Nov 2024 00:00:00 +0000

By John Smith

November 25, 2024

Tool calling is a feature of many Large Language Models (LLMs) which allows them to interact with external systems through functions provided by the calling application. When an LLM needs to perform an action such as searching a database or calling an API it can "call" a tool by specifying the function name and parameters in a structured format. For example, if a user asks "What's the weather in Paris?", a LLM with access to a weather API tool might generate a response like:

{
  "function": {
    "name": "getWeather",
    "parameters": {
      "city": "Paris"
    }
  }
}

The application can interpret this call and use the weather API to fetch the weather in Paris. In order for an LLM to invoke a tool, the tool's name, description, and parameters must be provided in the context of the conversation. For applications with a large number of tools, this can lead to a few challenges:

Increased token usage as all tool descriptions are included in the context
Higher likelihood of the model choosing the wrong tool

Dynamic Tool Attachment

One approach to combat the above issues is to dynamically attach tools to the LLM based on the user's input. For example, we can use a semantic search to dynamically attach only the most relevant tools based on the user's input.

Let's build a simple system that uses semantic search to dynamically attach tools to an LLM based on the user's input by:

Embedding tool names and descriptions into an in-memory vector store using an embedding model
Embedding the user's prompt using the same embedding model
Retrieving the 5 most similar tools using a cosine similarity search
Including the most relevant tools in the model's context with the user's prompt

Setting Up

For this project we will be using a TypeScript application and Ollama for local chat completion and embedding. If you don't have Ollama installed, you can follow the installation instructions to get started.

Downloading Models

Download the following models for local use:

llama3.2 (3b) for chat completion
nomic-embed-text for generating embeddings

Node Dependencies

This project uses a couple Node dependencies:

compute-cosine-similarity for computing the cosine similarity of different vectors
tsx for running TypeScript in Node

npm init
npm install compute-cosine-similarity tsx

Defining Our Tools

We'll start by defining a set of dummy tools for our application. Each tool follows a consistent pattern with a name, description, and parameters:

export const ALL_TOOLS = [{
  type: "function",
  function: {
    name: "findCat",
    description: "Find the cat with the ID provided",
    parameters: {
      type: "object",
      properties: {
        id: {
          type: "string"
        },
      },
      required: ["id"],
    },
  },
},
.... ]

You can find a pre-defined set of tool schemas in the accompanying project repository.

Chat completion

We will use llama3.2 (3b) for chat completion by calling the Ollama chat endpoint:

type Message = {
  role: "user" | "assistant";
  content: string;
}

type Tool = {
  type: string;
  function: {
    name: string;
    description: string;
    parameters: unknown;
  }
}

const ollamaChat = async (messages: Message[], tools: Tool[]) => {
  const response = await fetch("http://localhost:11434/api/chat", {
    method: "POST",
    headers: {
      "Content-Type": "application/json"
    },
    body: JSON.stringify({
      model: "llama3.2",
      messages,
      tools,
      stream: false,
    }),
  });
  const data = await response.json();
  return data.message;
}

Computing Embeddings

We will use nomic-embed-text to generate vector embeddings by calling the Ollama embed endpoint:

const ollamaEmbed = async (input: string) => {
  const response = await fetch("http://localhost:11434/api/embed", {
    method: "POST",
    headers: {
      "Content-Type": "application/json"
    },
    body: JSON.stringify({
      model: "nomic-embed-text",
      input,
    }),
  });
  const data = await response.json();
  return data.embeddings[0];
}

Embedding tools

Let's create a function to compute embeddings for all tools and return an array of objects with both the tool schema and the embedding. We will embed the tool name and description.

const computeToolEmbeddings = async (tools: Tool[]) => Promise.all(
  tools.map(async (tool) => {
    const embedding = await ollamaEmbed(`${tool.function.name}: ${tool.function?.description}`)
    return {
      tool,
      embedding,
    }
  })
)

While this in memory data structure works for the purposes of this example, in a real-world application you would want to store embeddings in a database such as pgvector to avoid needing to re-compute the embeddings.

Tool Search Implementation

Let's implement a simple semantic search function using the compute-cosine-similarity package.

type EmbededTool = {
  tool: Tool;
  embedding: number[];
}

const searchTools = async (input: string, embeddings: EmbededTool[]) => {
  const messageEmbedding = await ollamaEmbed(input);
  return embeddings
    .map((embedding) => ({
      tool: embedding.tool,
      similarity: similarity(embedding.embedding, messageEmbedding)
    }))
    .sort((a, b) => b.similarity - a.similarity)
    .slice(0, 5);
}

This function:

Embeds the user's input
Compares the user's embedding with all tool embeddings
Returns the top 5 most relevant tools based on the similarity score (higher is more relevant)

Putting it all together

The last part of our project is a main function that:

Embeds all tools using computeToolEmbeddings
Prompts the user for input using readline
Uses searchTools to compare the user's input with the tool embeddings
Calls ollamaChat with the user's input and the attached tools

const main = async () => {
  console.log("Embedding tools...");
  const embeddings = await computeToolEmbeddings(ALL_TOOLS);
  console.log("Tools embedded.");

  const rl = readline.createInterface({
    input: process.stdin,
    output: process.stdout
  });

  console.log("Enter your message:");
  let messages: Message[] = []

  rl.on('line', async (input) => {
    const attachedTools = await searchTools(input, embeddings);
    messages.push({ role: "user", content: input })
    const response = await ollamaChat(messages, attachedTools);
    messages.push({ role: "assistant", content: response.content })
    console.log("Tools called:", response.content, response.tool_calls)
  })
}

main();

We can now run the project with tsx index.ts and test it out:

If we use the prompt find tool with id 123, our searchTools function will return the 5 most relevant tools. We can see that findTool has the highest similarity score.

Enter your message: find tool with ID 123
Attaching tool: { tool: 'findTool', similarity: 0.825733057876305 }
Attaching tool: { tool: 'findToy', similarity: 0.6924025554970754 }
Attaching tool: { tool: 'findCar', similarity: 0.6859769197248136 }
Attaching tool: { tool: 'findBook', similarity: 0.6778616325470412 }
Attaching tool: { tool: 'findSong', similarity: 0.6710841959852032 }
Tools called: [{ function: { name: 'findTool', arguments: [Object] } } ]

We can also prompt with similar terms, such as find hammer with ID 123. Even though no tool explicitly matches the term "hammer" our similarity search function still returns findTool as the most relevant result.

Enter your message: find hammer with ID 123
Attaching tool: { tool: 'findTool', similarity: 0.5955042990419085 }
Attaching tool: { tool: 'findToy', similarity: 0.5411030864968827 }
Attaching tool: { tool: 'findMovie', similarity: 0.5219368182792513 }
Attaching tool: { tool: 'findSong', similarity: 0.5155495695032712 }
Attaching tool: { tool: 'findCar', similarity: 0.5132604044773412 }
Tools called: [{ function: { name: 'findTool', arguments: [Object] } } ]

Source Code

Source code for this post is available in the accompanying inferablehq/ollama-dynamic-tools Github project.

How to build your own Javascript-native RPC framework

Nadeesha Cabral — Wed, 15 Nov 2023 10:30:00 +0000

Why? Well for fun of course. But more often than once, my "just for fun" code has ended up in production. Rather than warn people against doing the same thing, I've taken it upon myself to sow further chaos.

Let's get into it.

RPC?

RPC is a protocol that one program can use to request a service from a program located in another computer on a network. The client makes a procedure call that appears to be local but is actually run on a remote machine.

In other words, it looks like you're executing the function locally, but it executes remotely.

We will

Build a server that will house remote procedures.
Use JSON as a serialisation mechanism.
Create a client that'll handle serialisation and deserialisation.

Let's build a small server

To achieve the R (Remote) part of RPC, we'll build a small server. Nothing fancy.

// server.ts

const { createServer } = require("http");

const functions = {
  sum: (a: number, b: number) => a + b,
  wait: async (delay: number) => {
    await new Promise((resolve) => setTimeout(resolve, delay * 1000));
    return "Done!";
  },
};

createServer((req, res) => {
  if (req.method === "POST" && req.url === "/invoke") {
    let body = "";
    req.on("data", (chunk) => (body += chunk));
    req.on("end", async () => {
      const { fn, args } = JSON.parse(body);
      const result = await functions[fn](...(args || []));
      res.end(JSON.stringify(result));
    });
  } else {
    res.statusCode = 404;
    res.end("Page not found!"x);
  }
}).listen(8080);

Theoretically, you can invoke any function that you have on the server side with a cURL, and get the output:

curl -X POST -d '{"fn":"sum","args":[1,2]}' http://localhost:8080/invoke
> 3

When we receive something on the /invoke endpoint, we'll deconstruct the body to figure out the function name and the args, and execute the functions from list of functions we've got. (Nevermind the type safety!)

Creating the client

Our RPC client would looks something like this:

// client.ts

async function invoke(fn: string, args: any[]): Promise<any> {
  try {
    const response = await fetch("http://localhost:8080/invoke", {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
      },
      body: JSON.stringify({ fn, args }),
    });

    if (!response.ok) {
      throw new Error(`Server responded with status: ${response.status}`);
    }

    return await response.json();
  } catch (error) {
    console.error("Error invoking function:", error);
    throw error;
  }
}

// Example usage:

(async () => {
  try {
    const sumResult = await invoke("sum", [5, 3]);
    console.log("Sum result:", sumResult); // Output will be 8

    const waitResult = await invoke("wait", [2]);
    console.log("Wait result:", waitResult); // Output will be "Done!" after 2 seconds
  } catch (error) {
    console.error("Error in function invocation:", error);
  }
})();

And we'll have a invoke function which will execute any function on the server end.

The client is responsible for serialising and deserialising the RPC call and result.

Now when you want to add a new function foo, you'd just add it to the functions collection in the server, and invoke('foo') on the client.

What we're missing so far

Of course, we're missing a few things.

Guarantees: There's no published contract from the server. Client has no idea whether foo() exists on the server, and if it exists - what parameters it accepts.
Timeouts: If your HTTP timeout is 10 seconds and if you call delay(20), you won't get your result back. In this case, you'd need to change the architecture of the server to enable asynchronous communication.

Adding stronger guarantees

Let's discuss adding in the guarantees - or rather, the tradeoffs involved.

If you want your server contract to be strongly adhered to, then you'll have to publish the contract to the client somehow.

gRPC uses Protocol Buffers to define structured data. Typescript native tRPC uses zod. But all in all, you'd have some kind of a service definition.

However, a service definition is not necessarily free.

If you're using gRPC, you'd need a compile step to generate the Protobuf definitions. tRPC is more flexible, but you'll end up with a duplicative zod definition for everything.

If you have a monolithic codebase, then you can completely do away with a separate service definition because you can use your function signature (if it's typed - of course) as the service definition.

Allowing for long running functions

We can modify the server to have two endpoints, /invoke and /result.

Invoke Endpoint: When a function is invoked via this endpoint, the server will generate a unique ID for the request, start executing the function asynchronously, and immediately return the ID to the client.
Result Endpoint: The client can use this endpoint to poll for the result of the execution using the provided ID.

Here's the changed server:

// server.ts

const { createServer } = require("http");
const { v4: uuidv4 } = require('uuid');

const functions = {
  sum: (a: number, b: number) => a + b,
  wait: async (delay: number) => {
    await new Promise((resolve) => setTimeout(resolve, delay * 1000));
    return "Done!";
  },
};

const results = new Map();

createServer((req, res) => {
  if (req.method === "POST" && req.url === "/invoke") {
    let body = "";
    req.on("data", (chunk) => (body += chunk));
    req.on("end", async () => {
      const { fn, args } = JSON.parse(body);
      const id = uuidv4();
      results.set(id, { status: "pending", result: null });

      functions[fn](...(args || [])).then(result => {
        results.set(id, { status: "completed", result });
      }).catch(error => {
        results.set(id, { status: "error", result: error.message });
      });

      res.end(JSON.stringify({ id }));
    });
  } else if (req.method === "GET" && req.url.startsWith("/result/")) {
    const id = req.url.split('/')[2];
    const result = results.get(id);

    if (result) {
      res.end(JSON.stringify(result));
    } else {
      res.statusCode = 404;
      res.end("Result not found!");
    }
  } else {
    res.statusCode = 404;
    res.end("Page not found!");
  }
}).listen(8080);

And we'll have to change the client as well to poll until it gets a non-pending state. (We'll ignore the indefinitely pending state for now)

// client.ts

async function invoke(fn: string, args: any[]): Promise<any> {
  const response = await fetch("http://localhost:8080/invoke", {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ fn, args }),
  });

  if (!response.ok) {
    throw new Error(`Server responded with status: ${response.status}`);
  }

  const { id } = await response.json();

  return await pollForResult(id);
}

async function getResult(id: string): Promise<any> {
  const response = await fetch(`http://localhost:8080/result/${id}`);

  if (!response.ok) {
    throw new Error(`Server responded with status: ${response.status}`);
  }

  return await response.json();
}

async function pollForResult(id: string): Promise<any> {
  let result = await getResult(id);

  while (result.status === 'pending') {
    await new Promise(resolve => setTimeout(resolve, 1000)); // Poll every second
    result = await getResult(id);
  }

  return result.result;
}

// Example usage:

(async () => {
  try {
    const sumResult = await invoke("sum", [5, 3]);
    console.log("Sum result:", sumResult); // Output will be 8 after processing

    const waitResult = await invoke("wait", [2]);
    console.log("Wait result:", waitResult); // Output will be "Done!" after 2 seconds
  } catch (error) {
    console.error("Error in function invocation:", error);
  }
})();

This will allow us to execute more RPC calls without keeping a lot of connections open for a long time.

Conclusion

Still, there are a few rough edges to make this a truly Javascript-native RPC framework.

What happens on a promise rejection? We don't really serialise the error well enough for the caller to intercept it.
A process crashing while processing might make the enqueued functions disappear. Putting that on a persisted data store might make it be pending forever.
Retrying: Can we retry the function again if it fails? It might not be safe to retry everything in bulk.

I might cover these areas in a future post.

If you enjoyed this post, you might find Differential interesting. It allows you to keep your Javascript code in one single codebase, but execute it in a distributed way using RPCs defined by Javascript functions and Javascript functions alone. If you use Typescript, you get type-safe service definitions without any extra work :)

From Promises to Futures in Javascript

Nadeesha Cabral — Wed, 05 Jun 2019 10:14:13 +0000

As someone who jumped on the Node.js train early on and wrestled with the callback hell, I quite liked promises. I still do, but more in an "it's the best that we've got way". So many times I've forgotten to catch a promise chain and it always decided to silently fail. Or, I'd have to hack my way through trying to cancel a promise chain.

I won't go into too much detail over all of the issues with promises. But I highly recommend Broken Promises which does an excellent job of summarizing everything.

Fluture

As an alternative to promises, I've been using fluture for some time now. I highly recommend the fluture-js library for this. It introduces itself as a "Fantasy Land compliant (monadic) alternative to Promises". If you're unfamiliar with monads of fantasy-land specification, don't worry about it.

Douglas Crockford once said, "Once someone understands Monads, they lose the ability to explain it to anybody else". Despite that, let me give a shot at explaining what a monad is, in Javascript terms.

In Javascript world, a monad tends to be an object that's got a bunch of functions as properties. When you invoke these functions, it tends to do something and return something like the original Javascript object. You can then use that returning object to do further computations. This might sound like a promise, but promises are not monads for reasons which we'll not get into here.

Example

Let's consider a simple futures example (using fluture-js) where we:

Read and parses a package.json file to get the name of a package
Send that data to a fictional API that returns some metadata about it
Parse that result to get the downloads count
Sends that count to an onSuccess function

import { encaseP, node, encase } from "fluture";

const readFileF = filePath => node(done => fs.readFile(filePath, "utf8", done));

const fetchF = url => encaseP(url => fetch(url));

const jsonParseF = encase(jsonString => JSON.parse(jsonString));

const getPackageDownloads = (npmPackageName, onSuccess, onFailure) => {
  readFileF("package.json")
    .pipe(chain(jsonParseF)) // shorthand for chain(jsonString => jsonParseF(jsonString))
    .pipe(map(package => package.name))
    .pipe(
      chain(packageName => fetchF(`/api/v1/package-metadata/${packageName}`))
    )
    .pipe(map(jsonParseF))
    .pipe(map(response => response.data.metadata.downloadCount))
    .pipe(
      fork(error => onFailure(error), downloadCount => onSuccess(downloadCount))
    );
};

Explanation

I'd like to believe that you can elicit 80% of the value that fluture gives by knowing 20% of the constructs it provides. And this might well be that 20%. Let's go through each construct of the earlier example in detail and see what it does.

0. Common to all constructs...

...is the fact that everything returns a future. Therefore, we can compose these futures, refactor them out, etc.

1. encaseP, node, encase creates futures

Since javascript doesn't have futures, we need to have some tools to convert existing Javascript constructs like promises, and functions to futures. And these 3 constructs does exactly that.

encaseP creates a future from a promise
node creates a future from a node-style async function
encase creates a future from a plain old javascript function

Since all three of these Javascript constructs may fail through a promise rejection, a callback(error) or an exception, these three utilities map that rejection state to future rejection as well.

2. pipe lets you chain futures

Think of this like the pipe() that you get when you invert the compose() function. We talked about this in length at A practical guide to writing more functional JavaScript. It basically lets you "chain" futures.

3. map transforms values

When you have to pipe something, you always have to pipe a future-ed version of your result. When your computation doesn't produce a future, like package => package.name, you can do that transformation inside of map.

When you invoke map(fn), with a future, map takes the value inside that future, applies fn to transform that value, and returns the value wrapped in a future.

4. chain transforms values but expects a future back

chain does the same thing map doesn't, but the fn in your chain(fn) must return a future back.

5. fork to execute

Since futures are lazily evaluated, nothing is being done until you tell the future to execute things. fork(failureFn, successFn) takes a success function and a failure function and executes them on a success/failure instance.

Why use futures over promises?

There's a lot of advantages to using futures. Aesthetically pleasing API is a big part of it. Since promises are the real competition, let me try to make a few concrete distinctions against promises.

Lazy evaluation has a lot of practical advantages. You have a guarantee that your computation will not execute at the time of creating the future. Whereas the new Promise(...) will execute at the time of the creation.
Testability comes through lazy evaluation as well. Instead of mocking all of your side-effects when testing, you can assert whether the futures wrapping the side effects were "composed". Thereby not executing the side-effects as well.
Better control flow than promises for sure. You can race(), parallel()-ize, and cancel one or more promises out of the box.
Error handling is far more superior. You will always end up with an error. No more forgotten catch()es silently suppressing errors. And you get a really good debugging experience out of the box.