DEV Community: Mingming Ma

My contributions to ChatCraft summary (v1.1.0 - v2.0.0)

Mingming Ma — Sat, 20 Apr 2024 04:18:56 +0000

Recently, I contributed to the open source project Chatcraft, and participated in the development of 10 release versions from v1.1.0 to v2.0.0. I would like to write a blog post to summarize my contributions.

New Feature contributions

1. Enables users to upload images to ChatCraft and work with vision-related models (related release: `v1.2.0`, `v1.3.0`, `v1.4.0`, `v1.6.0`, `v1.9.0`, `v2.0.0`)

Release note:

v1.2.0: first release
v1.3.0: supports mobile
v1.4.0: adds image compression
v1.6.0: supports multiple image compression options
v1.9.0: removes workaround hardcode
v2.0.0: supports GPT-4 Turbo

Demonstration

(Please click on the GIF file to trigger the animation if it doesn't start automatically.)

Note, although the message time displayed is current, here is the v1.2.0 commit branch being used to show the initial version functionality.

The latest release v2.0.0 is as follows, since the new model of gpt-4-turbo as of 04-09 also supports "Vision", I have adapted it accordingly.

(Please click on the GIF file to trigger the animation if it doesn't start automatically.)

Users can also add images by using drag and drop and copy-paste methods.

drag and drop	copy-paste

(Please click on the GIF file to trigger the animation if it doesn't start automatically.)

To implement this feature, I studied the newly emerged vision-related API from the openai-node project and integrated it into the codebase, added new UI components to display user images, and made modifications to the model, database, and optimization operations such as image compression. For example, the integration of new API can be found in the changes from original type CreateChatCompletionRequestMessage to new type ChatCompletionMessageParam as below:

@@ -90,16 +96,28 @@ export class ChatCraftMessage {
       date: this.date.toISOString(),
       type: this.type,
       text: this.text,
+      imageUrls: this.imageUrls,
     };
   }

-  toOpenAiMessage(): OpenAI.Chat.Completions.CreateChatCompletionRequestMessage {
+  toOpenAiMessage(model: ChatCraftModel): OpenAI.Chat.Completions.ChatCompletionMessageParam {
     const text = this.text;
+
+    const content: OpenAI.Chat.Completions.ChatCompletionContentPart[] = [{ type: "text", text }];
+    if (model.supportsImages && this.imageUrls.length > 0) {
+      this.imageUrls.forEach((imageUrl) => {
+        content.push({
+          type: "image_url",
+          image_url: { url: imageUrl },
+        });
+      });
+    }
+
     switch (this.type) {
       case "ai":
         return { role: "assistant", content: text };
       case "human":
-        return { role: "user", content: text };
+        return { role: "user", content };
       case "system":
         return { role: "system", content: text };
       case "function":
@@ -119,6 +137,7 @@ export class ChatCraftMessage {
       chatId,
       type: this.type,
       text: this.text,
+      imageUrls: this.imageUrls,
     };
   }

This new type suggests a more flexible structure that can accommodate multiple types of content within a single message. This is crucial for integrating capabilities for the image processing and handling, which are essential for models that support vision features.

The image compression function is completed client-side, which can reduce the cost of the user's API token. Users can make related settings in the settings modal.

As a side effect, I use it to do quick image compress.

This feature has been the biggest challenge I've faced in the open-source community so far. In my previous post, 'A 55-commit PR,' I also wrote about my feelings after releasing. Thanks again to the ChatCraft community for their help and support.

2. Enables users to generate images with the DALL·E 3 model (related release:`v1.5.0`, `v1.7.0`, `v1.8.0`, `v2.0.0`)

Since the image generation model kind of independent than the language models, my approach is to use ChatCraft's command interaction, and I added an /image command, usage:

/image [layout-option]<prompt>

layout=[l\|landscape\|p\|portrait]

Release note:

v1.5.0: first release
v1.7.0: add loading info
v1.8.0: fix for issue: generated image not available after some time
v2.0.0: fix for issue: the loading spinner does not disappear when image generation fails

Demonstration

(Please click on the GIF file to trigger the animation if it doesn't start automatically.)

3. Supports rendering of mathematical expressions (LaTeX) (`v1.1.0`)

Demonstration

(a night mode example)

Improved UI contribution

Allows for the hiding and showing of long chat. (`v1.1.0`, `v1.2.0`)

It works great when the message is super long. During the development, I encountered a Minified React error issue. If you are interested, you can check my post 'Minified React error #426' to see a possible solution. An interesting update in v1.2.0 is we don't fold super long final message to give the user better experience.

Demonstration

(Please click on the GIF file to trigger the animation if it doesn't start automatically.)

Bug fixes contributions

1.dollar symbol render conflict vs Latex rendering issue (`v1.1.0`)

Bug	Fix

2.no label code block rendering issue (`v1.1.0`)

Bug	Fix

3.Html no syntax highlight issue (`v1.2.0`)

Bug	Fix

4.encode & in Markdown URLs issue (`v1.2.0`)

Bug	Fix
command failed, converting that to `&`

5.non-OpenAI model uses OpenAI logo (`v1.8.0`)

Bug	Fix

6. Disable message editing on shared chats (`v1.9.0`)

Bug	Fix

Reflections

My contributions to ChatCraft from versions v1.1.0 to v2.0.0 have been both challenging and rewarding. Throughout this journey, I have been deeply involved in enhancing the functionality and user experience of ChatCraft. These contributions have not only improved the platform but have also provided me with invaluable learning experiences.

My work on ChatCraft has been a profound learning curve and a testament to the power of community and collaboration in the open-source space. I am grateful for the support from the ChatCraft community, which has been instrumental in overcoming challenges and achieving milestones. I look forward to engaging more with the community and continuing to contribute to the evolution of ChatCraft.

Thoughts about AI on UI design

Mingming Ma — Mon, 15 Apr 2024 12:31:36 +0000

In PR 575 I fixed a issue of ChatCraft so that user now can only see the copy button in the shared chat.

Before	After

And then we have a different opinions about the Options button at the end of the shared chat.

Should we keep it as previous:

or can we just use the Duplicate... option as the new button in a shared chat?

It has not been updated for now, but it reminds me of an issue: tools similar to chatGPT are not well-suited for this kind of design work. Despite being quite helpful with coding throughout the semester in my studies, which enabled me to build my iOS app and fix issues in ChatCraft.

I'm wondering if there is an AI tool that can help design UI, since I know that DALL-E 3 often generates bizarre images and not suitable for creating design mockups. There are some tools like Uizard that can generate general outlines, although I have not tried it myself, I highly doubt it can handle the issue mentioned above. I believe design skills will become increasingly important to developers, and I need to start practicing as soon as possible.

Base64 data of DALL·E 3 response

Mingming Ma — Sun, 07 Apr 2024 16:58:31 +0000

This week, I submitted a PR to ChatCraft that set DALL·E 3 generated image's format to Base64 data. Previously, we used a URL format, but it was only viable for an hour due to OpenAI's requirements.

The `response_format` parameter

The format in which the generated images are returned. Must be one of url or b64_json. URLs are only valid for 60 minutes after the image has been generated.

Usage examples

Suppose you have an <img> tag in your component, and the src attribute is populated from a variable called imgSrc:



<img src={imgSrc} alt="..." />

We can pass the src with the Base64 encoded image data rather than url



const imgSrc = "data:image/jpeg;base64,/9j/4AAQSkZJRgAB...";

Now, let's take a look at the API. Here's an example of a url response:



const response = await openai.images.generate({
  model: "dall-e-3",
  prompt: "a white siamese cat",
  n: 1,
  size: "1024x1024",
});
const imgSrc = response.data[0].url;

To obtain the response from OpenAI in Base64 format, we can update the code as follows:



const response = await openai.images.generate({
  model: "dall-e-3",
  prompt: "a white siamese cat",
  n: 1,
  size: "1024x1024",
  response_format: "b64_json",
});
const imgSrc = `data:image/jpeg;base64,${response.data[0].b64_json}`;

I hope you find it helpful. See you in the next post!

Create toasts from outside of React Components

Mingming Ma — Mon, 01 Apr 2024 04:25:41 +0000

This week, I submitted a PR to ChatCraft that added an alert function using the Standalone Toasts feature of Chakra UI. I previously used the useToast hooks, but I found that the Standalone Toasts are the solution when hooks can't be used. I'd like to share how I implemented it.

Standalone Toasts

Firstly, let's see the examples from Chakra UI document

import * as ReactDOM from 'react-dom/client'
import { createStandaloneToast } from '@chakra-ui/react'

const { ToastContainer, toast } = createStandaloneToast()

// render the ToastContainer in your React root
const rootElement = document.getElementById('root')
ReactDOM.createRoot(yourRootElement).render(
  <>
    <App />
    <ToastContainer />
  </>,
)

toast({
  title: 'An error occurred.',
  description: 'Unable to create user account.',
  status: 'error',
  duration: 9000,
  isClosable: true,
})

At first glance, I found it a little confusing because it seems like we need to add the ToastContainer somewhere.

// render the ToastContainer in your React root
const rootElement = document.getElementById('root')
ReactDOM.createRoot(yourRootElement).render(
  <>
    <App />
    <ToastContainer />
  </>,
)

Actually, if your project already has @chakra-ui/react installed, then we don't need to import it separately. What we need becomes to :


import { createStandaloneToast } from '@chakra-ui/react'

const { toast } = createStandaloneToast()

toast({
  title: 'An error occurred.',
  description: 'Unable to create user account.',
  status: 'error',
  duration: 9000,
  isClosable: true,
})

How I did in the PR

//utils.ts
/**
 * Only meant to be used outside components or hooks
 * where useAlert cannot be used.
 */
import type { AlertArguments } from "../hooks/use-alert";
export const utilizeAlert = async () => {
  const { createStandaloneToast } = await import("@chakra-ui/react");
  const { toast } = createStandaloneToast();

  const info = ({ id, title, message }: AlertArguments) => {
    toast({
      id,
      title,
      description: message,
      colorScheme: "blue",
      status: "info",
      position: "top",
      isClosable: true,
      duration: 3000,
    });
  };

  const loading = ({ id, title, message }: AlertArguments) => {
    const fallbackId = new Date().toISOString();
    toast({
      id: id ?? fallbackId,
      title,
      description: message,
      colorScheme: "blue",
      status: "loading",
      position: "top",
      isClosable: true,
      duration: null,
    });
    return id ?? fallbackId;
  };

  const closeLoading = (id: string) => {
    toast.close(id);
  };

  return {
    info,
    loading,
    closeLoading,
  };
};

As you can see, I wrap different kinds of toasts in the utilizeAlert function. Then we can show the alert by:

import { utilizeAlert } from "./utils";

const nonComponentFunction () {
    const { info , loading, closeLoading } = await utilizeAlert();

    //show info alert
    info({title: "this is info alert"})

    //show loading alert
    const alertId = loading({
      title: `Generating image, please wait.`,
    });

    //some stuff

    //close loading alert
    closeLoading(alertId);
}

Standalone Toasts are a good design pattern that allows users to create toasts outside of a component. Sometimes it's good to have more options, and I think that's one of the reasons why Chakra UI is so popular. So that's it. I hope you found this helpful. Thanks for reading!

This week's learning: object parameter ; fieldset

Mingming Ma — Sun, 24 Mar 2024 19:28:44 +0000

In this week, I submitted a PR to ChatCraft proposing a new feature that allows for multiple options in the compression settings of attached images.

This enhancement aims to provide a more customizable user experience, allowing users to balance image quality and file size according to their specific needs. By offering more control over image compression, we can ensure that the images shared on our platform are optimized for both visual appeal and efficient data usage.

I would like to share some insights from the work I did on this pull request. Specifically, I'd like to focus on two aspects: object parameters and fieldset.

Object parameters

In my initial approach to implementing the compression settings feature, I used individual variables to represent each of the user's selected options. Here's what that code looked like:

export const compressImageToBase64 = (
  file: File,
  compressionFactor: number = 1,
  maxSizeMB: number = 20,
  maxWidthOrHeight: number = 2048
): Promise<string> => {
  // ...
}

While this approach worked, it had some limitations. If you only want to pass one option to that function, you would still need to pass values for the other options as well, even if they are just default values. This can be inconvenient and can make the code more verbose than necessary. It made it more difficult to add new options in the future, since each new option would require a new parameter to be added to the function signature.

To address these limitations, we can refactor the code to use an object parameter instead:

interface ImageCompressionOptions {
  compressionFactor?: number;
  maxSizeMB?: number;
  maxWidthOrHeight?: number;
}
export const compressImageToBase64 = (
  file: File,
  options: ImageCompressionOptions = {}
): Promise<string> => {
  const { compressionFactor = 1, maxSizeMB = 20, maxWidthOrHeight = 2048 } = options;

This is what I did in the PR, however, you can continue refactor like bellow's example:

interface EmailOptions {
  from: string; // Sender email address
  to: string; // Recipient email address
  subject?: string; // Email subject (optional)
  body: string; // Email body content
  cc?: string[]; // Carbon copy recipients (optional)
  bcc?: string[]; // Blind carbon copy recipients (optional)
}

function sendEmail({
  from,
  to,
  subject = "No Subject",
  body,
  cc = [],
  bcc = [],
}: EmailOptions) {
  console.log(`Sending email from ${from} to ${to}`);
  console.log(`Subject: ${subject}`);
  console.log(`Body: ${body}`);
  if (cc.length > 0) {
    console.log(`CC: ${cc.join(", ")}`);
  }
  if (bcc.length > 0) {
    console.log(`BCC: ${bcc.join(", ")}`);
  }
  // Assume the email is sent securely here
}

// usage
sendEmail({
  from: 'noreply@example.com',
  to: 'jane.doe@example.com',
  subject: 'Welcome!',
  body: 'Thank you for joining our service.',
  cc: ['team@example.com'],
  bcc: ['audit@example.com']
});

Object parameters are a powerful way to pass complex data structures between functions or methods, allowing for more flexible and modular code. By using objects as parameters, we can encapsulate related data and behavior, making our code more organized and easier to maintain.

`fieldset`

The <fieldset> tag is used to group related elements in a form. You can find the example from Chakra UI to use this tag.

<FormControl as='fieldset'>
  <FormLabel as='legend'>
    Favorite Naruto Character
  </FormLabel>
  <RadioGroup defaultValue='Itachi'>
    <HStack spacing='24px'>
      <Radio value='Sasuke'>Sasuke</Radio>
      <Radio value='Nagato'>Nagato</Radio>
      <Radio value='Itachi'>Itachi</Radio>
      <Radio value='Sage of the six Paths'>Sage of the six Paths</Radio>
    </HStack>
  </RadioGroup>
  <FormHelperText>Select only if you're a fan.</FormHelperText>
</FormControl>

I attempted to use it to group multiple sliders together in this PR. Using the <Fieldset> component can make your forms more accessible and easier to use, especially for users who rely on assistive technologies like screen readers.

That's it, thanks for reading!

Adding new feature through small changes

Mingming Ma — Sun, 17 Mar 2024 17:46:50 +0000

This week, I made a PR for ChatCraft that allows users to generate images (by the DALL·E 3 model). Currently, the way to generate images is by using the 'image' command, for example: /image treasure map,

In contrast to my previous PR that introduced support for the vision model, this PR did not require significant changes. This allows us to steadily enhance it without causing many disruptions. And thanks to the valuable feedback, here are the following up PRs need to be submitted : such as changing the size of the generated image (issue#499), adding a loading info when generating images (issue#504), etc. I will write about some of the problems or thoughts they encountered in subsequent blog posts.

In general, this feeling is great. Small changes make the system more stable, and modifications based on review feedback are also easier, which benefits both. Moreover, I found that in this PR, despite being small changes, the reviewer proposed lots of suggestions or new ideas, This leaves lots of room for subsequent development work and allows for a greater sense of accomplishment. Therefore, I say this feeling is great.

Currently, this feature relies on command-based interaction, but most users may not use this type of interaction. One of the challenges for the future is how to make users know this feature. As I ponder over this challenge, I eagerly anticipate sharing my thoughts and progress with you in the next blog post. Stay tuned!

Learning Cache

Mingming Ma — Sun, 10 Mar 2024 05:57:58 +0000

Recently, ChatCraft enabled python support via WASI which is really amazing! (You can check out this PR by WangGithub0 for more details)

In that PR, I found

WASI.start(fetch(url), {
        args: ["python", "-c", code],
        stdout: (out) => {
          console.log(out);
        },
        stderr: (err) => {
          console.error(err);
        },
      })
        .then((result) => {

seems each time run the fetch, but after the first time running, which is time-consuming, the later ones are significantly speeds up loading times.

After some research, I got to know it is related to the cache, I would like to share what I learned in this post.

How Caching Works with WebAssembly

Fetching and Compiling: The first time a WebAssembly module is requested, the browser fetches it, compiles it into machine code, and then executes it. This process can be time-consuming, especially for large modules.
Cache Storage: After the initial compilation, both the raw and compiled versions of the module can be stored in the browser's cache.
Subsequent Requests: For subsequent requests, if the browser detects that the module is already in its cache and hasn't changed (based on cache headers and validation tokens), it can skip the fetching and compilation steps, directly loading the module from cache. This significantly speeds up loading times.
Cache-Control Headers: The effectiveness of caching can be influenced by HTTP Cache-Control headers sent by the server. These headers can instruct the browser on how and when to cache resources, including WASM modules.
Versioning and Updates: To ensure users receive the latest version of a WASM module, developers must manage caching carefully. Changing the URL of the module or using cache-busting techniques can force the browser to fetch a new version.

Cache-Control Headers

The caching behavior is determined by the HTTP Cache-Control header and other caching headers sent by the server along with the resource. These headers can specify directives such as:

max-age: Indicates how long the resource is considered fresh and can be served from the cache without revalidation.
no-cache: Instructs the browser to revalidate the resource with the server before using the cached copy.
no-store: Tells the browser not to cache the resource at all.
must-revalidate: Indicates that once a resource becomes stale, it must be revalidated with the server before it can be used.
public or private: Specifies whether the response is intended for a single user (private) or can be cached by shared caches (public).

And developers can control caching behavior in the fetch request by using the cache option in the Request object to specify how the request should interact with the browser's cache. For example:

default: The browser will follow its default caching algorithm.
reload: The browser will bypass the cache and download the resource from the server.

See more at Request: cache property

@antonz/python-wasi headers

Now let's see the headers of the used lib @antonz/python-wasi using Thunder client in VS code)

We can also see the MIME type to content-type application/wasm, it looks like being set here in the source code

Image Compression: Part 2

Mingming Ma — Sun, 25 Feb 2024 07:16:37 +0000

In this blog post, I continue to share some discoveries about image compression. I'm quite curious about how Google Gemini handles uploaded images. So, I compared Chatcraft and Google Gemini, using a 20MB image file. After selecting the image and before sending the prompt, I observed the Network tab in the developer tools, as shown in the following images:

You can see that in the console resources, ChatCraft shows 3.2 MB, while Google Gemini displays 21.3 MB. ChatCraft shows 3.2 MB because it utilizes image compression in browser, performed locally. While Gemini displays 21.3 MB, I believe Gemini doesn't perform local image compression but instead conducts compression on the server side. I observed that after refreshing the page, the image is downloaded through a GET request, and at this point, you can find the image has been compressed to around 77KB.

Next, I attempted to inspect the payload through the POST request of the prompt to understand how the image is transmitted. But I spent much amount of time and still couldn't figure it out. I was able to identify the text part in the payload, but the subsequent string is encoded in a way I couldn't understand. It might involve an authentication token. You can see that the payload also automatically includes the filename of the image. I wonder if Gemini uses it to assist in generating responses.

If I make any new discoveries, I will update this blog post. Of course, if you know the meaning of the payload, please let me know. I would greatly appreciate it. Thank you for reading.

Going down the rabbit hole of the canvas

Mingming Ma — Sat, 17 Feb 2024 15:15:30 +0000

Recently, I encountered an issue on ChatCraft that required me to adjust the size of an image. If it needs to be done on the frontend (browser), it involves the use of canvas. In simple terms, this process can be explained using the example below.



//an example of how you might use a canvas to 
//compress an image using JavaScript:
function compressImage(image, outputFormat, quality, callback) {
  // Create a canvas element
  const canvas = document.createElement('canvas');
  const ctx = canvas.getContext('2d');

  // Set the canvas size to the desired output size
  canvas.width = image.width / 2; // Example: reduce width by half
  canvas.height = image.height / 2; // Example: reduce height by half

  // Draw the image onto the canvas
  ctx.drawImage(image, 0, 0, canvas.width, canvas.height);

  // Convert the canvas to a Blob (binary large object)
  canvas.toBlob(function(blob) {
    // The blob is a compressed version of the image
    callback(blob);
  }, outputFormat, quality);
}

// Usage example
const image = new Image();
image.src = 'path/to/your/image.jpg'; // Load your image here
image.onload = function() {
  compressImage(image, 'image/jpeg', 0.7, function(compressedBlob) {
    // Do something with the compressed image blob
    console.log('Compressed image size:', compressedBlob.size);
  });
};

I was fascinated by this approach, prompting me to commence my exploration into the depths of canvas.

In the example above, let's see these two lines:



const ctx = canvas.getContext('2d');
ctx.drawImage(image, 0, 0, canvas.width, canvas.height);

First, the canvas.getContext('2d') is using the HTMLCanvasElement's method: HTMLCanvasElement.getContext(), :

A drawing context lets you draw on the canvas. Calling getContext with "2d" returns a CanvasRenderingContext2D object, whereas calling it with "webgl" (or "experimental-webgl") returns a WebGLRenderingContext object. This context is only available on browsers that implement WebGL.

So, ctx.drawImage(image, 0, 0, canvas.width, canvas.height) is using the CanvasRenderingContext2D's method drawImage()



drawImage(image, dx, dy)
drawImage(image, dx, dy, dWidth, dHeight)
drawImage(image, sx, sy, sWidth, sHeight, dx, dy, dWidth, dHeight)

ctx.drawImage(image, dx, dy) - Draws the image at its natural size at the specified position on the canvas.
ctx.drawImage(image, dx, dy, dWidth, dHeight) - Draws the image, scaling it to fit the specified width and height at the specified position on the canvas.
ctx.drawImage(image, sx, sy, sWidth, sHeight, dx, dy, dWidth, dHeight) - Draws a specified area of the source image to a specified area on the canvas, allowing for both cropping and scaling.

Then, I found this gem: an article The HTML5 Canvas Handbook Although it was written in 2013, I found it still applicable today. It is easier for beginners to understand various aspects compared to directly reading the MDN. I highly recommend it.

In the 2.2.11 Drawing images section, you can find straightforward examples. The author provides a series of images to demonstrate the usage of drawImage.

Next, I tried to find the source code for drawImage, and I found the CanvasRenderingContext2D.cpp from the Blink, which is the browser engine developed as part of the Chromium project.



void CanvasRenderingContext2D::drawImage(const CanvasImageSourceUnion& imageSource, float x, float y, ExceptionState& exceptionState)
{
    CanvasImageSource* imageSourceInternal = toImageSourceInternal(imageSource);
    FloatSize sourceRectSize = imageSourceInternal->elementSize();
    FloatSize destRectSize = imageSourceInternal->defaultDestinationSize();
    drawImage(imageSourceInternal, 0, 0, sourceRectSize.width(), sourceRectSize.height(), x, y, destRectSize.width(), destRectSize.height(), exceptionState);
}
void CanvasRenderingContext2D::drawImage(const CanvasImageSourceUnion& imageSource,
    float x, float y, float width, float height, ExceptionState& exceptionState)
{
    CanvasImageSource* imageSourceInternal = toImageSourceInternal(imageSource);
    FloatSize sourceRectSize = imageSourceInternal->elementSize();
    drawImage(imageSourceInternal, 0, 0, sourceRectSize.width(), sourceRectSize.height(), x, y, width, height, exceptionState);
}
void CanvasRenderingContext2D::drawImage(const CanvasImageSourceUnion& imageSource,
    float sx, float sy, float sw, float sh,
    float dx, float dy, float dw, float dh, ExceptionState& exceptionState)
{
    CanvasImageSource* imageSourceInternal = toImageSourceInternal(imageSource);
    drawImage(imageSourceInternal, sx, sy, sw, sh, dx, dy, dw, dh, exceptionState);
}

And all three functions will be calling:



void CanvasRenderingContext2D::drawImage(CanvasImageSource* imageSource,
    float sx, float sy, float sw, float sh,
    float dx, float dy, float dw, float dh, ExceptionState& exceptionState)
{
    if (!drawingCanvas())
        return;
    RefPtr<Image> image;
    SourceImageStatus sourceImageStatus = InvalidSourceImageStatus;
    if (!imageSource->isVideoElement()) {
        image = imageSource->getSourceImageForCanvas(&sourceImageStatus);
        if (sourceImageStatus == UndecodableSourceImageStatus)
            exceptionState.throwDOMException(InvalidStateError, "The HTMLImageElement provided is in the 'broken' state.");
        if (!image || !image->width() || !image->height())
            return;
    } else {
        if (!static_cast<HTMLVideoElement*>(imageSource)->hasAvailableVideoFrame())
            return;
    }
    if (!std::isfinite(dx) || !std::isfinite(dy) || !std::isfinite(dw) || !std::isfinite(dh)
        || !std::isfinite(sx) || !std::isfinite(sy) || !std::isfinite(sw) || !std::isfinite(sh)
        || !dw || !dh || !sw || !sh)
        return;
    FloatRect srcRect = normalizeRect(FloatRect(sx, sy, sw, sh));
    FloatRect dstRect = normalizeRect(FloatRect(dx, dy, dw, dh));
    clipRectsToImageRect(FloatRect(FloatPoint(), imageSource->elementSize()), &srcRect, &dstRect);
    imageSource->adjustDrawRects(&srcRect, &dstRect);
    if (srcRect.isEmpty())
        return;
    if (shouldDisableDeferral(imageSource) || image->imageForCurrentFrame()->isTextureBacked())
        canvas()->disableDeferral();
    validateStateStack();
    draw(
        [this, &imageSource, &image, &srcRect, dstRect](SkCanvas* c, const SkPaint* paint) // draw lambda
        {
            drawImageInternal(c, imageSource, image.get(), srcRect, dstRect, paint);
        },
        [this, &dstRect](const SkIRect& clipBounds) // overdraw test lambda
        {
            return rectContainsTransformedRect(dstRect, clipBounds);
        }, dstRect, CanvasRenderingContext2DState::ImagePaintType,
        imageSource->isOpaque() ? CanvasRenderingContext2DState::OpaqueImage : CanvasRenderingContext2DState::NonOpaqueImage);
    validateStateStack();
    bool isExpensive = false;
    if (ExpensiveCanvasHeuristicParameters::SVGImageSourcesAreExpensive && image && image->isSVGImage())
        isExpensive = true;
    if (imageSource->elementSize().width() * imageSource->elementSize().height() > canvas()->width() * canvas()->height() * ExpensiveCanvasHeuristicParameters::ExpensiveImageSizeRatio)
        isExpensive = true;
    if (isExpensive) {
        ImageBuffer* buffer = canvas()->buffer();
        if (buffer)
            buffer->setHasExpensiveOp();
    }
    if (imageSource->isCanvasElement() && static_cast<HTMLCanvasElement*>(imageSource)->is3D()) {
        // WebGL to 2D canvas: must flush graphics context to prevent a race
        // FIXME: crbug.com/516331 Fix the underlying synchronization issue so this flush can be eliminated.
        canvas()->buffer()->flushGpu();
    }
    if (canvas()->originClean() && wouldTaintOrigin(imageSource))
        canvas()->setOriginTainted();
}

This is a lengthy function, but we can observe one crucial call:



 drawImageInternal(c, imageSource, image.get(), srcRect, dstRect, paint);

And in drawImageInternal function, we can see:



void CanvasRenderingContext2D::drawImageInternal(SkCanvas* c, CanvasImageSource* imageSource, Image* image, const FloatRect& srcRect, const FloatRect& dstRect, const SkPaint* paint)
{
    int initialSaveCount = c->getSaveCount();
    SkPaint imagePaint = *paint;
    if (paint->getImageFilter()) {
        SkMatrix ctm = c->getTotalMatrix();
        SkMatrix invCtm;
        if (!ctm.invert(&invCtm)) {
            // There is an earlier check for invertibility, but the arithmetic
            // in AffineTransform is not exactly identical, so it is possible
            // for SkMatrix to find the transform to be non-invertible at this stage.
            // crbug.com/504687
            return;
        }
        c->save();
        c->concat(invCtm);
        SkRect bounds = dstRect;
        ctm.mapRect(&bounds);
        SkRect filteredBounds;
        paint->getImageFilter()->computeFastBounds(bounds, &filteredBounds);
        SkPaint layerPaint;
        layerPaint.setXfermode(paint->getXfermode());
        layerPaint.setImageFilter(paint->getImageFilter());
        c->saveLayer(&filteredBounds, &layerPaint);
        c->concat(ctm);
        imagePaint.setXfermodeMode(SkXfermode::kSrcOver_Mode);
        imagePaint.setImageFilter(nullptr);
    }
    if (!imageSource->isVideoElement()) {
        imagePaint.setAntiAlias(shouldDrawImageAntialiased(dstRect));
        image->draw(c, imagePaint, dstRect, srcRect, DoNotRespectImageOrientation, Image::DoNotClampImageToSourceRect);
    } else {
        c->save();
        c->clipRect(dstRect);
        c->translate(dstRect.x(), dstRect.y());
        c->scale(dstRect.width() / srcRect.width(), dstRect.height() / srcRect.height());
        c->translate(-srcRect.x(), -srcRect.y());
        HTMLVideoElement* video = static_cast<HTMLVideoElement*>(imageSource);
        video->paintCurrentFrame(c, IntRect(IntPoint(), IntSize(video->videoWidth(), video->videoHeight())), &imagePaint);
    }
    c->restoreToCount(initialSaveCount);
}

In the above function, this line



        c->scale(dstRect.width() / srcRect.width(), dstRect.height() / srcRect.height());

tells us it will use SkCanvas's scale method. You will notice that the SkCanvas is actually coming from third party lib Skia



//CanvasRenderingContext2D.cpp
#include "third_party/skia/include/core/SkCanvas.h"

Skia is an open-source 2D graphics library written in C++ and serves as the graphics engine for Google Chrome and ChromeOS, Android, Flutter, and many other products. Here is the source code of scale function:



void SkCanvas::scale(SkScalar sx, SkScalar sy) {
    if (sx != 1 || sy != 1) {
        this->checkForDeferredSave();
        fMCRec->fMatrix.preScale(sx, sy);

        this->topDevice()->setGlobalCTM(fMCRec->fMatrix);

        this->didScale(sx, sy);
    }
}

If you like, you can continue dig out how the scaling transformation is applied to the matrix in preScale function.

In terms of the pixel grid on a computer screen or canvas, the scaling transformation affects how the drawing commands are mapped to the actual pixels. Simply terms in general:

Original Coordinates: Each shape or image has coordinates that determine its position and size on the canvas. These coordinates are based on a logical coordinate system, not directly tied to the physical pixels.

Transformation Matrix: When a scaling transformation is applied, the transformation matrix is updated to include the scaling factors. This matrix is used to transform the logical coordinates of the shapes or images.

Rasterization: During rasterization, the transformed logical coordinates are converted to pixel coordinates. The rendering engine uses the transformation matrix to determine where and how to place each shape or image on the pixel grid.

Pixel Mapping: If an object is scaled up, more pixels are used to represent it, making it appear larger on the screen. Conversely, if an object is scaled down, fewer pixels represent it, making it appear smaller.

Interpolation: When scaling images, interpolation algorithms (like nearest-neighbor, bilinear, or bicubic interpolation) determine the color values of the new pixels based on the original pixels to create a smooth transition and minimize artifacts.

I stopped further tracking of the source code. As an end of this blog post, I would like to share my finding of the React Native Skia which brings the Skia Graphics Library to React Native. You can get more information from a Shopify Engineering blog Getting Started with React Native Skia. Hope it helps, see you next post.

A 55-commit PR

Mingming Ma — Sat, 10 Feb 2024 07:38:24 +0000

Since last November, I have been working on a feature that enables ChatCraft to support the use of the gpt-4-vision-preview model.
Fortunately, this PR has been merged after nearly 3 months.

I want to say, this process wasn't easy, to the extent that after completion, I felt calm. Bringing me a bit of surprise is that I simultaneously received the Pull Shark x2 badge. The feeling is quite subtle. Are you a Christian? I feel like this is a little gift from God to me, as if saying, 'Hey, you're doing not bad, and I'll help you keep track of it, and keep on coding!'

I am fortunate to have this opportunity at ChatCraft. I enjoy coding, but it seems I have a problem-solving-driven personality. This inclination makes me more proactive in community projects, as they often represent the expectations of others. The reason this PR eventually has 55 commits is that I aimed to complete this feature in one go. However, it is evident that my strategy wasn't mature enough. Fortunately, the ChatCraft community is fantastic. I received ample support and assistance, ultimately leading to the acceptance of this PR. I think there won't be another open-source project that will allow me to submit such massive changes in one PR.

Finally, I'd like to share that life is a wonderful journey. Whether you're a beginner, a professional, or someone who, like me, comes from a different background but enjoys coding, I extend a heartfelt invitation for you to join the ChatCraft community. This community is not just for coders, but for anyone who is interested in learning, experimenting, and being creative. It marks the beginning of another beautiful chapter in life.

A trick to know inline-code in react-markdown

Mingming Ma — Sat, 03 Feb 2024 05:34:06 +0000

If you are using react-markdown lasted version, you may find that inline property is removed.

I have faced same situation as I need to deal with different markdown rendering for the inline code and Fenced Code Blocks to solved this issue.

Take a look at the example code shows Syntax Highlighting to code blocks from this blog post, which has used inline property before it is removed:


    <Markdown
      remarkPlugins={[remarkGfm]}
      rehypePlugins={[rehypeRaw]}
      components={{
        code({ node, inline, className, children, ...props }: any) {
          const match = /language-(\w+)/.exec(className || '');

          return !inline && match ? (
            <SyntaxHighlighter style={dracula} PreTag="div" language={match[1]} {...props}>
              {String(children).replace(/\n$/, '')}
            </SyntaxHighlighter>
          ) : (
            <code className={className} {...props}>
              {children}
            </code>
          );
        },
      }}
    >
      {markdown}
    </Markdown>

Since now we don't have inline property in react-markdown, you may say we can still use the match logic to detect. While it is good at most time when you specify the language next to the backticks. Without specifying the language, then the match

const match = /language-(\w+)/.exec(className || '');

is same for inline code and code blocks, because className is undefined in both situation.

So here is how I resolved this puzzle, I found when you give code block syntax to react-markdown, the String(children) always has the new line '\n' no matter language is given or not. For the inline code syntax, String(children) does not has the new line '\n', which also aligns with its literal meaning. So here is this trick:

        code({ className, children, ...props }: any) {
          const match = /language-(\w+)/.exec(className || '');

          return String(children).includes("\n") || match ? (
            <SyntaxHighlighter> ... </SyntaxHighlighter>
          ) : (
            <code> ... </code>
          );

Fun fact: I got to know the inline property "solution" when I asked chatgpt how to distinguish inline code and code block, and googled out the above mentioned post from that.

That's it. Hope it helps. See you next time.

Minified React error #426

Mingming Ma — Sat, 27 Jan 2024 17:10:49 +0000

In this blog post, I would like to address an issue I encountered in this week. This is the first time I met this kind of problem using React and first time know the startTransition of React.

How it comes

This error related to the ReactMarkdown component and the useDisclosure hook in the @chakra-ui/react.

<ReactMarkdown children={children}>
 components={{
        code({children, ...props }) {
 // ...
   return (
    // some rendering based on the children, such as
    // <code>{children}</code>
    // <SyntaxHighlighter>{String(children)}SyntaxHighlighter>
    // <Preview children={Array.isArray(children) ? children : [children]} ></Preview>
    )
  }
</ReactMarkdown>

 const { isOpen, onToggle } = useDisclosure();

Here is the original workflow:

a button is toggled -> useDisclosure catch the changes -> ReactMarkdown's updates rendering (based on the isOpen status)

<Button onClick={() => onToggle()}>
    {isOpen ? "Show Less" : "Show More..."}
</Button>

<ReactMarkdown>
    {isOpen ? veryLongText : shortText }
</ReactMarkdown>

Error occurs

Now when we have a very very long string veryLongText, the update will gives you the error:

A component suspended while responding to synchronous input. This will cause the UI to be replaced with a loading indicator. To fix, updates that suspend should be wrapped with startTransition.

This indicates that a component has paused or stopped functioning while it was trying to handle synchronous input. Synchronous input refers to actions or events that are processed immediately, without waiting for other tasks to complete.

It suggests that we can wrap the updates or changes that cause the suspension of the component with a function called startTransition

How to solve

Here is the updated to solve the error by using startTransition


  const { isOpen, onToggle: originalOnToggle } = useDisclosure();

  const onToggle = () => {
    startTransition(() => {
      originalOnToggle();
    });
  };

// no changes to Button and ReactMarkdown code
<Button onClick={() => onToggle()}>
    {isOpen ? "Show Less" : "Show More..."}
</Button>

<ReactMarkdown>
    {isOpen ? veryLongText : shortText }
</ReactMarkdown>

We use the startTransition function telling React to optimize and batch updates. This means that when the button is clicked, and the onToggle function is called, the state update triggered by originalOnToggle() is treated as a low-priority update. Here is the full diff if you are interested:

diff --git a/src/components/Message/MessageBase.tsx b/src/components/Message/MessageBase.tsx
index 25b83d82..f7f27042 100644
--- a/src/components/Message/MessageBase.tsx
+++ b/src/components/Message/MessageBase.tsx
@@ -1,5 +1,6 @@
 import {
   memo,
+  startTransition,
   useCallback,
   useState,
   useEffect,
@@ -33,6 +34,7 @@ import {
   useClipboard,
   Kbd,
   Spacer,
+  useDisclosure,
 } from "@chakra-ui/react";
 import ResizeTextarea from "react-textarea-autosize";
 import { TbDots, TbTrash } from "react-icons/tb";
@@ -111,6 +113,17 @@ function MessageBase({
   const isNarrowScreen = useMobileBreakpoint();
   const messageForm = useRef<HTMLFormElement>(null);
   const meta = useMemo(getMetaKey, []);
+  const { isOpen, onToggle: originalOnToggle } = useDisclosure();
+  const isLongMessage = text.length > 5000;
+  const displaySummaryText = !isOpen && (summaryText || isLongMessage);
+
+  // Wrap the onToggle function with startTransition, state update should be deferred due to long message
+  // https://reactjs.org/docs/error-decoder.html?invariant=426
+  const onToggle = () => {
+    startTransition(() => {
+      originalOnToggle();
+    });
+  };

   useEffect(() => {
     if (settings.countTokens) {
@@ -334,6 +347,14 @@ function MessageBase({
         <CardBody p={0}>
           <Flex direction="column" gap={3}>
             <Box maxWidth="100%" minH="2em" overflow="hidden" px={6} pb={2}>
+              {
+                // only display the button before message if the message is too long
+                isLongMessage ? (
+                  <Button size="sm" variant="ghost" onClick={() => onToggle()}>
+                    {isOpen ? "Show Less" : "Show More..."}
+                  </Button>
+                ) : undefined
+              }
               {editing ? (
                 // eslint-disable-next-line jsx-a11y/no-noninteractive-element-interactions
                 <form onSubmit={handleSubmit} ref={messageForm} onKeyDown={handleKeyDown}>
@@ -381,8 +402,12 @@ function MessageBase({
                   </VStack>
                 </form>
               ) : (
-                <Markdown previewCode={!hidePreviews} isLoading={isLoading} onPrompt={onPrompt}>
-                  {summaryText || text}
+                <Markdown
+                  previewCode={!hidePreviews && !displaySummaryText}
+                  isLoading={isLoading}
+                  onPrompt={onPrompt}
+                >
+                  {displaySummaryText ? summaryText || text.slice(0, 250).trim() : text}
                 </Markdown>
               )}
             </Box>

The startTransition function is specifically designed to work with the React Concurrent Mode (or just Concurrent Mode), which is an experimental set of features in React for improving the performance and user experience of complex user interfaces. I'm very glad to know this feature. Hope it helps. See you next post!

DEV Community: Mingming Ma

My contributions to ChatCraft summary (v1.1.0 - v2.0.0)

New Feature contributions

1. Enables users to upload images to ChatCraft and work with vision-related models (related release: v1.2.0, v1.3.0, v1.4.0, v1.6.0, v1.9.0, v2.0.0)

Release note:

Demonstration

2. Enables users to generate images with the DALL·E 3 model (related release:v1.5.0, v1.7.0, v1.8.0, v2.0.0)

Release note:

Demonstration

3. Supports rendering of mathematical expressions (LaTeX) (v1.1.0)

Demonstration

Improved UI contribution

Allows for the hiding and showing of long chat. (v1.1.0, v1.2.0)

Demonstration

Bug fixes contributions

1.dollar symbol render conflict vs Latex rendering issue (v1.1.0)

2.no label code block rendering issue (v1.1.0)

3.Html no syntax highlight issue (v1.2.0)

4.encode & in Markdown URLs issue (v1.2.0)

5.non-OpenAI model uses OpenAI logo (v1.8.0)

6. Disable message editing on shared chats (v1.9.0)

Reflections

Thoughts about AI on UI design

Base64 data of DALL·E 3 response

The response_format parameter

Usage examples

Create toasts from outside of React Components

Standalone Toasts

How I did in the PR

This week's learning: object parameter ; fieldset

Object parameters

fieldset

Adding new feature through small changes

Learning Cache

How Caching Works with WebAssembly

Cache-Control Headers

@antonz/python-wasi headers

Other helpful links:

Image Compression: Part 2

Going down the rabbit hole of the canvas

A 55-commit PR

A trick to know inline-code in react-markdown

Minified React error #426

How it comes

Error occurs

How to solve

1. Enables users to upload images to ChatCraft and work with vision-related models (related release: `v1.2.0`, `v1.3.0`, `v1.4.0`, `v1.6.0`, `v1.9.0`, `v2.0.0`)

2. Enables users to generate images with the DALL·E 3 model (related release:`v1.5.0`, `v1.7.0`, `v1.8.0`, `v2.0.0`)

3. Supports rendering of mathematical expressions (LaTeX) (`v1.1.0`)

Allows for the hiding and showing of long chat. (`v1.1.0`, `v1.2.0`)

1.dollar symbol render conflict vs Latex rendering issue (`v1.1.0`)

2.no label code block rendering issue (`v1.1.0`)

3.Html no syntax highlight issue (`v1.2.0`)

4.encode & in Markdown URLs issue (`v1.2.0`)

5.non-OpenAI model uses OpenAI logo (`v1.8.0`)

6. Disable message editing on shared chats (`v1.9.0`)

The `response_format` parameter

`fieldset`