DEV Community: Lukas Klinzing

How to use LLM for efficient text outputs longer than 4k tokens?

Lukas Klinzing — Mon, 10 Jun 2024 12:09:16 +0000

Hey community,

I wanted to share a nifty tool that could save us a lot of time and cost when we're making edits to large texts. It's called LLM Patcher, and it's built with Next.js, the Vercel AI SDK, and OpenAI.

The Problem: Time and Cost Inefficiencies

You know how painful it can be when we ask an LLM to make changes to a large document? It always sends back the entire modified text. For big files, this means:

Time Drain: We end up waiting for the whole text to stream back, which is super slow.
Cost Spike: It’s expensive because we’re processing and transferring more data than needed.

LLM Patcher is designed to tackle these issues head-on.

Demo

How LLM Patcher Works

Here’s a quick rundown of how it streamlines the process:

Input and Query: You feed it the text and a find-and-replace query.
Text Segmentation: It splits the text into lines and sentences.
Identifier Prefixing: Each segment gets a unique identifier (like <l1s1> for line 1, sentence 1).
Find-and-Replace Execution: The LLM processes each segment to apply the changes.
Streaming Changes: Instead of the whole text, it streams back just the changes in a diff format (e.g., <r:l1s1> string to find || string to replace).

What LLM Patcher Is and Isn’t For

What It’s Great For:

Finding and replacing text in large documents.
Editing and patching texts efficiently.
Specific workflows like:
- Fixing typos or word replacements.
- Anonymizing texts by replacing names.
- Handling sensitive info by replacing it with placeholders.
- Enhancing texts with SEO keywords.

What It’s Not Designed For:

Generating text from scratch.
Summarizing or translating text.
Sentiment analysis.
Acting as a chatbot.
General-purpose LLM tasks.

Getting Started

Setting it up locally is straightforward:

Clone the repo and navigate to the project directory.
Set up your environment variables as defined in .env.example for OpenAI integration.
Install dependencies and start the dev server:



   pnpm install
   pnpm dev

The app should now be live on localhost:3000.

Heads Up: Don’t commit your .env file – it contains sensitive info.

Final Thoughts

LLM Patcher could be a real game-changer for a lot of workflows, especially when dealing with large text edits. It’s efficient, cost-effective, and tailored for find-and-replace operations. Check out the LLM Patcher Repository to see it in action. You can also checkout out the live demo inside the repo!

Let me know if you want to dive deeper into this or have any questions about integrating it into our current projects.

New Prompt & Confirmation Dialog API for React

Lukas Klinzing — Wed, 14 Jul 2021 22:44:18 +0000

In React the declarative nature rules how we write our code today. With this rules in mind, showing modals, prompts or confirmation dialogs come with some additional cost of rethinking how state, effects must work.

Confirmation dialog in React the old way

Let's say we want to delete an item from a list, but before, that, we want to ask the user if he really wants to do that.

The user clicks a delete button
the click handler sets a state, which indicates that a deletion action was issued and that a confirmation is requested
the component needs to render a confirmation dialog
the user confirms the delete action by clicking on DELETE in the confirmation dialog
the dialog sets the state to "confirmed"
An effect is listening to the state and if we are in a deleting state and a confirmed state, we can actually execute some remote logic.

We could model the state as following

type State = 
  | null 
  | DELETE_REQUEST 
  | DELETE_CONFIRMED 
  | DELETE_PENDING
  | DELETED

our code might then look like

const [state, setState] = useState<State>(null);

const handleDelete = () => {
  if (state !== null) return;
  setState('DELETE_REQUEST');
};

const handleConfirm = () => {
  setState('DELETE_CONFIRMED');
};

useEffect(() => {
  if (state === 'DELETE_CONFIRMED') {
    setState('DELETE_PENDING');
    deleteRemote().then(() => {
      // setting to null resets to initial state.
      setState(null);
    });
  }
}, [state]);

return (
  <>
    {state === 'DELETE_REQUEST' && (
      <ConfirmDialog onConfirm={handleConfirm} />
    )}
    {state === 'DELETE_PENDING' && <Loading />}
    {state === null && <Button onClick={handleDelete}>Delete now</Button>}
  </>
);

Now this works usually perfectly fine. We have distinct state and we can easily and safely transition from one to another.

Still this is quite some boilerplate and when multiple asynchronous steps are involved, I usually do not want to put every single asynchronous step in a declarative way into states and react on them in a useEffect.

I believe an asynchronous prompt is possible in react and will solve many issues.

Prompting dynamic content

A lot of React programmers think of JSX as a magical data structure. It is not. It is just an object holding the component function and the props. It is basically an instruction of how to render things.

We can pass this jsx around the same way as anything else.

One thing I do dislike about prompts in react is, that if you have let's say 10 prompts and you would like to show them all after each other, than you would need to do a lot of state management and complex rendering.

because of this, I play around with this approach

// 👍 Consider this API
const [child, setChild] = useState();
setChild(<strong>Why is he putting JSX into state?</strong>)

// instead of this api
const [show, setShow] = useState(false);
setShow(true)
return (show && <strong>as usual</strong>)

What does this do? Nothing special, just putting the instruction how to render stuff, into state. returning child will work as usual.

This kind of unorthodox approach has one benefit.
You could define what to render in a callback.

So if we slightly change this, we can do something like this:

const [promptContent, confirm] = useState();
const onClick = () => {
   confirm(<strong>Are you sure?</strong>)
}

return promptContent

Now, when onClick is triggered, promptContent will contain the element. This is very powerful...

Asynchronous prompts

Now you think, well I can maybe render from within a callback, but this does not help a lot.

What if I tell you, that you can

be in a callback
render a prompt inside of the callback
waiting for the user confirmation
hide the prompt while still in the callback
continue the callback and do remote operations.

So I say, we can do all this, in one single callback.

The key to it is that we need a hook which will additionally manage a promise.

const useConfirm = () => {
  const promiseRef = useRef();
  const [child, setChild] = useState();
  const confirm = (jsxElement) => {
    setChild(child);
    // we return a promise and store
    // the resolve 
    return new Promise((resolve, reject) => {
       promiseRef.current = {resolve, reject}
    });
  }

  const handleConfirm = () => {
     // we hide the prompt by setting the jsx to null
     setChild(null)
     // we resolve the promise
     promiseRef.current.resolve()
  }
}

Now you could do

const onClick = async () => {
   if (await confirm(...)) {
      // ...
   }
}

Wrapping up

I understand that this draft and idea are in a very unpolished state. But if you want to know more, leave me a comment.

Currently I am working on a library that will allow this kind of API and you will be able to bind it to the most popular UI libraries out there.

You can follow me for more updates at https://twitter.com/theluk246

useMountedEffect: asynchronous useEffect on potentially unmounted components

Lukas Klinzing — Sun, 11 Jul 2021 13:05:19 +0000

When we use effects, or more precisely the useEffect hook, than because very often we do want to execute something asynchronously. In most scenarios we are doing this when a component gets mounted.

Problem

Component gets mounted
useEffect gets executed
we set the state that we are starting to load something
we fetch data from the server
ups... the user navigated away
our component gets unmounted
our async function resolves finally
happy that we got data from server, we call setState to update our component
error: component shouts at us because we set state on something that does not exist anymore

Solution

So what is the solution here? Well, React tells us already, when it unmounted the component. We just need to listen to what React tells us, but unfortunately it is not that straight forward and looks also a little bit weird.

But because every hook can ask for a callback for the event of unmounting, we can abstract the handling of state whether we are mounted or not into a custom hook.

useMountedEffect

this hook basically is the same as the useEffect but has a 3 different points.

it can be asynchronous
the callback passed in will receive a isMounted function
you cannot return a unmount funciton by yourself unfortunately

Here is how it looks

So What happens here?

We define the arguments of the hook.

It is very similar to the original useEffect. It has the dependency array, but the callback you pass in, will receive a isMounted function. You can call this function whenever you want to check if the component is mounted or not.

We create two Refs using useRef

I use ref's for everything that does not require reactivity and if I am only interested in the latest version of some value.

Here we do not want to "react" if your callback changes, or if we get unmounted/mounted. We want to only react to changes, if the dependency array changes.

The hook stores the current state whether it is mounted or not. To get this state, we define a function that resolves with the mounted reference.
The hook calls the current callback. The callback can change but we call it only if the dependency array changes, so it works basically exactly like useEffect.
the callback now should check whether it should set a state based on the information isMounted() returns.

Example

const MyComponent = ({ id }) => {
  const [status, setStatus] = useState();
  const [data, setData] = useState();

  useMountedEffect(async (isMounted) => {
    setStatus("LOADIGN");
    try {
      const data = await fetchMyData({ id });
      // only set state if we are still mounted
      if (isMounted()) {
        setData(data);
        setStatus("DONE")
      }
    } catch (e) {
      if (isMounted()) {
        setStatus("ERROR")
      }
    }
  }, [id])
}

useLocal: the useState hook for a properly-synced state

Lukas Klinzing — Sun, 11 Jul 2021 12:28:56 +0000

As you might know, the React hook useState takes one argument, which will initialize its internal state to that given value. When React 16 with Hooks got released, I found it a little bit confusing, but it perfectly makes sense.

No matter what, there are still situations, where you need to update the state based on the incoming props. Usually that is done using a useEffect hook, where you listen on the incoming prop to change, and then update your local state.

This is a great way of handling it, because you, as the owner of the component can perfectly control, if the parent prop change is actually what you want. It could even break your component if the parent decides to update a prop during a critical state of your component.

But there are also situations, where not much can happen. There are simple scenarios where you basically want to hold a local version and not propagate it up the tree until a certain condition happen. During that time, you still want to allow the parent controlling component to update the local value, if that is what it wants.

Here is an example how to allow the parent controlling component set (and update, if necessary) the from and to values. It can happen that for example for whatever reason, there is another component, that can set a date range, for that, we would want that from and to can be updated.

const RangeDatepicker = ({ value: { from, to }, onChange }) => {
   const [from, setFrom] = useLocal(from)
   const [to, setTo] = useLocal(to)
   const update = () => {
    if (from && to) {
      onChange({ from, to });
    }
   }

   useEffect(update, [from, to])

   return <MyCalendarRangePicker from={from} to={to} onChangeFrom={setFrom} onChangeTo={setTo} />
}

the easiest implementation of such a hook (in Typscript) looks like this

One further improvement would be, to pass in some conditional function that checks, if it is allowed to update the local state.

You can visit the gist here

https://gist.github.com/theluk/13b7a17455b599699b7d34775cbf8273