DEV Community: Ryo Kuroyanagi

"nvm use" does not work with nvm-windows

Ryo Kuroyanagi — Tue, 31 Dec 2024 16:55:18 +0000

TL;DR
If "nvm use" command does not work with nvm-windows on your windows PC, please create C:\nvm4w folder. (When you used installer with the default settings.)

Phenomena

I installed nvm with the installer (v1.2.1). I tried a simple nvm use 20.18.1 (or whatever version you were targeting) and was met with a frustrating error.

It seems that the command succeeded. It says Now using node v20.18.1 (64-bit). But it does not activate the version actually.

If the command succeeded, nvm list should show the following output.

PS C:\Users\myusername> nvm list

    22.12.0
  * 20.18.1 (Currently using 64-bit executable)

But it just showed

PS C:\Users\myusername> nvm list

    22.12.0
    20.18.1

Solution

By checking Internet, I found that it creates C:\nvm4w and a symlink in it when the command succeeds. I did not see the folder on my PC.

So, I created C:\nvm4w by myself and nvm use command worked.

My setup

OS Edition: Windows 11 Home
OS Version: 24H2
OS Build: 26100.2605

How to realize Real-Time Speech with Dify API

Ryo Kuroyanagi — Sat, 24 Aug 2024 06:10:28 +0000

Dify is an open-sourced SaaS platform for building LLM workflows online. I'm using the API to create conversational AI experience on my app. I was struggling with getting TTS streams as the API response and play it. Here I demonstrates how to process the audio streams and play it correctly in real-time. In short, please check my code.

I'm using the API endpoint https://api.dify.ai/v1/chat-messages for text chat. It returns audio data in the same stream as the text response if we enabled Text to Speech feature in our Dify apps.

Press ADD FEATURE button and add Text to Speech feature.

You can check the response from API with the following curl command.



curl -X POST 'https://api.dify.ai/v1/chat-messages' \
--header 'Authorization: Bearer YOUR_API_KEY' \
--header 'Content-Type: application/json' \
--data-raw '{
    "inputs": {},
    "query": "What are the specs of the iPhone 13 Pro Max?",
    "response_mode": "streaming",
    "conversation_id": "",
    "user": "abc-123",
    "files": []
}'

I demonstrate in TypeScript / JavaScript but you can apply the same logic to your programming language.

Anatomy of streamed data

First, let's understand what kind of data Dify is using for the streams.

Streamed data format

Dify is using the following text data format. It is like JSON lines but it is not the same exactly.



data: {"event": "workflow_started", "conversation_id": "065fb118-35d4-4524-a067-a70338ece575", "message_id": "3f0fe3cf-5aa1-4f7c-8abe-2505bf07ae8f", "created_at": 1724478014, "task_id": "dacb2d5c-a6f5-44b5-b5a6-de000f24aeba", "workflow_run_id": "50100b30-e458-4632-ad7d-8dd383823376", "data": {"id": "50100b30-e458-4632-ad7d-8dd383823376", "workflow_id": "debdb4fa-dcab-4233-9413-fd6d17b9e36a", "sequence_number": 334, "inputs": {"sys.query": "What are the specs of the iPhone 13 Pro Max?", "sys.files": [], "sys.conversation_id": "065fb118-35d4-4524-a067-a70338ece575", "sys.user_id": "abc-123"}, "created_at": 1724478014}}

data: {"event": "node_started", "conversation_id": "065fb118-35d4-4524-a067-a70338ece575", "message_id": "3f0fe3cf-5aa1-4f7c-8abe-2505bf07ae8f", "created_at": 1724478014, "task_id": "dacb2d5c-a6f5-44b5-b5a6-de000f24aeba", "workflow_run_id": "50100b30-e458-4632-ad7d-8dd383823376", "data": {"id": "bf912f43-29dd-4ee2-aefa-0fabdf379257", "node_id": "1721365917005", "node_type": "start", "title": "\u958b\u59cb", "index": 1, "predecessor_node_id": null, "inputs": null, "created_at": 1724478013, "extras": {}}}

data: {"event": "node_finished", "conversation_id": "065fb118-35d4-4524-a067-a70338ece575", "message_id": "3f0fe3cf-5aa1-4f7c-8abe-2505bf07ae8f", "created_at": 1724478014, "task_id": "dacb2d5c-a6f5-44b5-b5a6-de000f24aeba", "workflow_run_id": "50100b30-e458-4632-ad7d-8dd383823376", "data": {"id": "bf912f43-29dd-4ee2-aefa-0fabdf379257", "node_id": "1721365917005", "node_type": "start", "title": "\u958b\u59cb", "index": 1, "predecessor_node_id": null, "inputs": {"sys.query": "What are the specs of the iPhone 13 Pro Max?", "sys.files": [], "sys.conversation_id": "065fb118-35d4-4524-a067-a70338ece575", "sys.user_id": "abc-123", "sys.dialogue_count": 1}, "process_data": null, "outputs": {"sys.query": "What are the specs of the iPhone 13 Pro Max?", "sys.files": [], "sys.conversation_id": "065fb118-35d4-4524-a067-a70338ece575", "sys.user_id": "abc-123", "sys.dialogue_count": 1}, "status": "succeeded", "error": null, "elapsed_time": 0.001423838548362255, "execution_metadata": null, "created_at": 1724478013, "finished_at": 1724478013, "files": []}}

data: {"event": "node_started", "conversation_id": "065fb118-35d4-4524-a067-a70338ece575", "message_id": "3f0fe3cf-5aa1-4f7c-8abe-2505bf07ae8f", "created_at": 1724478014, "task_id": "dacb2d5c-a6f5-44b5-b5a6-de000f24aeba", "workflow_run_id": "50100b30-e458-4632-ad7d-8dd383823376", "data": {"id": "89ed58ab-6157-499b-81b2-92b1336969a5", "node_id": "llm", "node_type": "llm", "title": "LLM", "index": 2, "predecessor_node_id": "1721365917005", "inputs": null, "created_at": 1724478013, "extras": {}}}

...

In the response, Dify pushes text answer and audio data.

Example line of text answer



data: {"event": "message", "conversation_id": "aa13eb24-e90a-4c5d-a36b-756f0e3be8f8", "message_id": "5be739a9-09ba-4444-9905-a2f37f8c7a21", "created_at": 1724301648, "task_id": "0643f770-e9d3-408f-b771-bb2e9430b4f9", "id": "5be739a9-09ba-4444-9905-a2f37f8c7a21", "answer": "MP"}

Example line of audio data



data: {"event": "tts_message", "conversation_id": "aa13eb24-e90a-4c5d-a36b-756f0e3be8f8", "message_id": "5be739a9-09ba-4444-9905-a2f37f8c7a21", "created_at": 1724301648, "task_id": "0643f770-e9d3-408f-b771-bb2e9430b4f9", "audio": "//PkxABhvDm0DVp4ACUUfvWc1CFlh0tR9Oh7LxzHRsGBuGx155x3JqTJiwKKZf8wIcxpMzJU0h4zhgyQwwwIsgWQMAALQMkanBTjfCPgZwFsDOGGIYJoJoJoJoPQPQLYEgAOwM4SMXMW8TcNWGrEPEME0HoIQTg0DQNA0C5k7IOLeJuDnDVi5nWyJwgghAagQwTQQgJAGrDVibiFhqw1YR8HOEjBUA5AcgagQwTQTQQgJAAtgLYKsQ8hZc0PV7OrE4SgQgFIAsAQAwA6H0Uv4t4m4m49Yt4uYOQHIBkAyAqAkAuB0Mm6UeKxDGRrIODkByBqBNBCA1ARwHIEgBVg5wkY41W2GgdEVDFBNe+HicQw0ydk7HrHrIWXM62d48ePNfCkNATcTcNWGrCRhqxDxcwMYBwBkByCGC4EILgoJTQUDeW8W8TcTchZ1qBWIYchOBbBCA1AhgSMJGGrFzLmh6fL+LeBkAyAZAcgSAXAhB0Kxnj4YDkJwXA6FAzwj8IIJoJoPQXA6EPOcg4R8FOBnCRljRAwlwoh4EUwLhFTCVA+MR0R8wyxOhgAwwDgJjBUABMM0hMxBgnTPtMrMBEEcwJQCzIXIdMZMG821DmjDKHJAwLDKHRMQsJkwbwVRoFs//PkxEx5dDnwAZ7wANHgEUFJHGCUCQp3LWCQQYGAATI5QzwHBJF4UFktpfATT2l0goAGNADLOU64HAMCQCK50szABAIkDS2/j8gl6l6Di7QgBEiAfMEADBnyZBgeAWCMK4xvBbhoRZj1M+ktsNMTrMNcHEwHQEzAjAHMGQAQwRQZTBHALMGMDkzhh2jGhLtMgsMMwfhOzCnGLMMcKgwOw8pqHMoGtvdDzos0AIAiXIsBAmGsRFtYcBABmB0AUYjQfhhDAfjoCrETAGArMOAJ4iAAMCMFkwXwh5fffuhpYMhyP2bl3MVAJQrSYQDsna7G2+fx/GvyAwUQbTAdAFCAHVKyIAduTXHZZXDjNS57/VeVJ5+JBJ+0kATkCSells8/NBt/2/5Dj1s+chDBYSINutNS9FQwDwBWHjgASKRgAAJOyYC4Ao0CMNAKBgB6KK1hYBkAAHROM9mLsknb8avTcB0MerV6jl7llE70egOerRh9WcP/FoHqtVsO/In2f+G2tsdnH+L/KSSvBQB4OATam27Yi4jiBgBFOpq15bTQU6k1G4LoWo1mMAwDQwlBEzEnKsMkA7c5JYuTOzK2MvAbEysSPTM+dOOn1XEzGgIzXzmPODVvs1cyNTJxQ9MsAWwy//PkxDlz7DIMAd7gAek5EwnjcjX9QVN1N0czFyijQKOmMi4IYw8RvzFvCHMHYBQwdQlTRxVNvm8ycGjLYlMTAQ=="}

We can distinguish JSON lines of audio data by checking the event property. Audio JSON has tts_message as the value. The audio mp3 binary is stored in the audio property of the JSONs in base64 format.

Problems in handling data

The first problem that we have when we play TTS audio real-time is the JSON lines are split into packets and each packet is not valid JSON data as it is.

Example packet which is cut in the middle



gMkhx2XCjT6Y0rKnDuvOnora378v6wGEMscxTGVK4ZLfbI+7cFjtUZxDCk3joo9En2RVbx1oIiz1VZYxKB2wq4pmSLWo55pbOoqtN0G2aY/LsNwomtvPH4M2zxBRpLsxKBJTIV6xF7IPaFQuq3CcZ/lDUQafC3mgavJHUWs7L+O8zuxIoahyH40TEFNRTMuMTAwqgGNTDg1JPDM5yHt0ZFFRiVTGYHgakOZhxJkgZMggAwCIxUTGFwQZQRRhIemGCABSONDpTQgEAIFxj8UmDhOYQAIMAgYaSQKmQwcXeBAYAAXEAKR8MIEABGIEBwyuFzQiVNXqcycmDT86Pug89ZUjiFYO6Oc2+BWXmEAqaDCRgUCGGA2Y7CgAEZMMgg1GDACCDwq3O9NNq+JiIOOBciCJyXYkWGCQjCmSOmVSFU2KGxxgYbMYBoacYBcpK+OM/OuxIngNUGJTg02CgJGVCxyfPr6FZIJGmmkBwQwxIxgQzgILC2X//PkxONtxDoABOafcMeL9NfW0rYzVsTJRAHVPD6hrLVnqxDJ4zpZFsVCg0ywkiWoUs6MADVREAIAki0xhwxeJYYrCpuLXb1ayPaFT4FeqU0lzVHUJZxJyqDqVo3kLOh0sE6Jc4oTjbk/LGfxuk7MpgOBmYISXTKcbDkVrMV5zohMIalUZJYoCkJrZVLSH1CPjrcz7OhCyxF9W2RKJKIT1A=="}

data: {"event": "tts_message", "conversation_id": "9ed2e63a-8527-41ff-851f-bf449e7f1096", "message_id": "706bf92a-eca4-4ec8-a04e-a54af25c8cca", "created_at": 1724491999, "task_id": "5f3ca6e2-b8bc-4cb7-946b-b5e0c1a85e99", "audio": "CWNnU8iypDSsX0myFoS4rzmeqmdtaHk4PJWJpIPUalRYjLJCh6iSBcnNXlOcJxsxdkPY4CoVTnHVq7TqEpqqMOhMQU1FMy4xMDCqqhKNPkjR+Ex2kM2MTDCcwcfAmod6hmLu5lhwZkkGBKphA0cQ3GAKxrVmaEhmrIhmaGZ6aDSUaYKBw2ZkImIggABC6xFVmcFBiwCKp4jBGBiIFGEwWYJMhl0qGlgAYLO5oiAmxcuaGnByxCGBfAaWdh06dmmWMZtVJnIfGDQYYMDZEIDDxhMAmAymZBkPhgOEQAAwsCggMNGM0QCjGoYMaisxohTIwnMDlYzGNhqKFj5FGLwGDDg4aY00bEcYMkY0AiiJHQxsb42YMmJGg5qVjwq+BVUKhDklRVOZRoc6EckebYSZGuaYCaUwAiaAIaHyKWp9PU7/8+TE9HH8OfgE3zUMkw9t3VLwKPo7oWpJKoegyTC0JiIoyZFUQQKL9GCIkpQKBxIxXGQQqFUAdpXyQ6QTdXrtv9bf5jbqpBO8uXbZV0vs/eEtbEqOGnZXDNrxVC5al8hhlIVmsWnnKaA9jIJM+MMxWA5Q8DswxSxunbC6sD0sCuA5Uom12td2qo+61VVYHYc4qche5qDOmhKtjzjPMqVOV0YZnGlTVuKqflkYak8F5/YLmGjMpvyN23tgW08zQoQ8yporCXVgmClh3UeyB387NsRcV2JEorHm5UagMxPQC8FMQU1FMy4xMDBVVVVVVVVVVVVVVVVVVVVVCVZncYm5NgeRQJt1kGKVgbuHRpBsmQhYYOBRh4jjQuMUBI0KOTC5EMII8xGXTEggBAXMOgYDIMOF7LTGZEL0DgCMKhwwCTjBxHMKAoxANggMF8AABgMEjFpIMil0z8ujsrQM+JM3N1z0ynM+gkx80jmakA2PM4hQkVRo8BlRCOozaMxKYnsVQafpEMLvmTEGWADaIWeB2JDw1LAtChEZoAEGjMjDjLzGCQ7CXjGlhiAZAkNSeIhQFFAIMXmMwk0CwgVFZ90kS6zLgIGVETefOwEzSBxK4lgWcECaCIvzEFXqZPGzJnT/8+TE5m5cOfQM5rNIgRVXhpQx6MVpN0dplrRXEQFOCAjUxcGtOqIgUmXKC4DrLpRxlkaUMcp1mAwMXhUqIglO3LWO1lMZyJp/2XwLStuwhTymDcGuwoSAk7XWcq3JuyBcsjhSlMPMiY22zOGlt2a09y6ELmUMqhp0l2KSZCzlrTGlN3CZa11uDDVi0zoslVgcZzXKeB+3rgXcNNduQbEIizl624P/R2k+3Bi89YkuXWBuPA0P1nISOcBOaTNcWAdZbDG1VmyqvnX3nJhaUMqrsRgZVKXM5bktR5tQHFYHYg=="}

The packet is starting from the middle of a JSON line. We have to combine multiple packets to get valid JSONs lines.

The second problem is the audio data chunk in a JSON is not a valid audio data. The data is cut in the middle of mp3 frames.

Implementation

To handle the split data of JSON and mp3, we have to do some smart way. The flow of the process is following:

First, we have to get valid JSON data and split into JSONs while receiving packets. When we got a packet with \n at the end, we can say the concatenation of the packets received so far is not cut in the middle. The pseudo code is like this.



let packets = []
stream.on('data', (bytes) => {
   const text = bytes.toString()
   packets.push(text)
   if (text.endsWith('\n')) {
      // Extract audio data from the packets.
      const audioChunks = extractAudioChunks(packets.join(''))
      // Clear the packet array
      packets = []
   }
})

Second, we have to split the audio chunks into mp3 frames. We concat the audio chunks into a binary and find each mp3 frames in it.



const mp3Frames = []
const binaryToProcess = Buffer.concat([...audioChunks])
let frameStartIndex = 0
for (let i = 0; i < binaryToProcess.length - 1; i += 1) {
  const currentByte = binaryToProcess[i]
  const nextByte = binaryToProcess[i + 1]
  // MP3 frame header always starts with eleven 1 bits. Checking 2 bytes.
  // It is a beginning of mp3 frame if current byte is 0xff and the beginning of the next byte is 111.
  // MP3 Spacification
  // http://www.mp3-tech.org/programmer/frame_header.html
  if (currentByte === 0xff && (nextByte & 0b11100000) === 0b11100000) {
    mp3Frames.push(binaryToProcess.subarray(frameStartIndex, i))
    frameStartIndex = i
  }
}

This is not the full implementation of splitting into mp3 frames. In the actual process, we have to consider cases that we have remainder bytes when we extracted mp3 frames from the audio binary and use the remainder as the beginning of the audio bytes in the next iteration.

Play the frames

I used fluent-ffmpeg for decoding and speaker for playing the decoded PCM audios. To play the TTS audio immediately after it received, I used stream for creating the decoding-playing pipeline.



class Mp3FrameReadable extends Readable {
    _read(size: number) {}
}

const mp3FrameStream = new Mp3FrameReadable()
const speaker = new Speaker()
ffmpeg(mp3FrameStream)
    .audioFrequency(44100)
    .audioChannels(2)
    .format('s16le')
    .pipe(speaker)

// Push a mp3 frame immediately after it is extracted from packets.
mp3FrameStream.push(frame)

Please check my GitHub repo for the full implementation. Hope this helps.

Using xLua in Unity

Ryo Kuroyanagi — Sun, 04 Aug 2024 14:51:30 +0000

This is my note of my testing of xLua. I'm using "Unity 2022.3.27" and URP.

Installation

From the release page, download the source code zip file. The latest build today is "Tag:v2.1.16_with_ohos".

Put all the files and the folders in "Assets" folder into your Unity project (some meta files, XLua, Plugin folder).

In "Examples" folder, there's "ExampleGenConfig.cs" and it causes an error.

Assets\XLua\Examples\ExampleGenConfig.cs(112,36): error CS0246: The type or namespace name 'MemberInfo' could not be found (are you missing a using directive or an assembly reference?)

You can delete the file to use xLua.

Build

Windows

I just built an app for windows in the way we build.

Android

Switch to Android platform. Go to [XLua top menu] -> [Generate Code] to generate platform specific codes. Then, build an APK file.

Detecting visionOS by JavaScript

Ryo Kuroyanagi — Sat, 27 Jul 2024 10:56:45 +0000

As of July 2024, you can use the following code to determine if a browser is on visionOS or not. (This does not identify browser brands; I only checked major browsers.)

function isVisionOS () {
  return navigator.userAgent.includes("(Macintosh;") &&
    !!navigator.xr &&
    document.ontouchstart !== undefined;
}

Breakdown

The function is using 3 criteria.

1. Whether the user agent string contains "(Macintosh;" or not.

Safari on macOS, iPadOS, visionOS has user agents like following.

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.0 Safari/605.1.15

By using the following code, we can determine if the browser is on macOS, iPadOS, or visionOS.

navigator.userAgent.includes("(Macintosh;")

2. Whether the browser supports XR or not.

According to the MDN page, Safari for macOS, iPadOS does not support XR. However, Safari on visionOS supports.

You can check if XR is supported with the following code:

!!navigator.xr

So far, it seems like criteria No.1 and No.2 are enough to identify visionOS. However, Chrome on macOS supports XR. So we need No.3.

3. Whether it is a touch device or not.

Finally, we can exclude macOS by using the following code.

document.ontouchstart !== undefined

OS	Browser	UserAgent	navigator.xr	Touch enabled
visionOS (2.0)	Safari	✅	✅	✅
visionOS (2.0)	Firefox (128.3)	✅	✅	✅
Windows 11	Chrome (127.0.6533.72)	❌	✅	❌
Windows 11	Edge (127.0.2651.74)	❌	✅	❌
Android 12 (Pixel 6)	Chrome (126.0.6478.188)	❌	✅	✅
iOS (18.0)	Chrome (127.0.6533.77)	❌	❌	✅
iOS (18.0)	Safari	❌	❌	✅
iPadOS (17.5.1)	Chrome (127.0.6533.77)	✅	❌	✅
iPadOS (17.5.1)	Safari	✅	❌	✅
macOS (Sonoma 14.5)	Chrome (126.0.6478.114)	✅	✅	❌
macOS (Sonoma 14.5)	Safari	✅	❌	❌

How to create awaitable prompt as React Component

Ryo Kuroyanagi — Sat, 06 Jul 2024 14:23:07 +0000

Motivation

JavaScript browser API has prompt() function which is a synchronized function for getting text input from user. We sometimes uses that kind of input UI components. However, the natively implemented UI component cannot be customized. I wanted to make it with customized UI and make it awaitable like const value = await prompt();.

Implementation

Like public react component libraries, I implemented use hook function. I'm exposing only the usePrompt() because I do not want developers to care about the UI implementation and want them to focus on using it as a capsulized feature.

TyepScript implementation.



import styles from "./style.module.scss"
import { useState, useRef, useCallback } from "react"
import { createPortal } from "react-dom"

type Props = {
  open: boolean
  value: string
  onChange: (value: string) => void
  onClose: (value: string | null) => void
}

export function Prompt({
  open,
  value,
  onChange: onValueChange,
  onClose
}: Props) {
  const onChange = useCallback((e: React.ChangeEvent<HTMLInputElement>) => {
    onValueChange(e.target.value)
  }, [onValueChange])
  const onOkClick = useCallback(() => {
    onClose(value)
  }, [value, onClose])
  const onCancelClick = useCallback(() => {
    onClose(null)
  }, [onClose])
  return createPortal((
    open && (
      <div className={styles.cover}>
        <div className={styles.frame}>
          <div>
            <input type="text" value={value} onChange={onChange} />
          </div>
          <div>
            <button onClick={onCancelClick}>CANCEL</button>
            <button onClick={onOkClick}>OK</button>
          </div>
        </div>
      </div>
    )
  ), document.body)
}

export function usePrompt() {
  const [open, setOpen] = useState<boolean>(false)
  const [value, setValue] = useState<string>("")
  const onCloseRef = useRef<(value: string | null) => void>()
  const onClose = useCallback((value: string | null) => {
    setOpen(false)
    if (onCloseRef.current) {
      onCloseRef.current(value)
    }
  }, [setOpen, onCloseRef])
  const onChange = (value: string) => {
    setValue(value)
  }
  return {
    open: async (value: string) => {
      setOpen(true)
      setValue(value)
      return new Promise<string|null>((resolve) => {
        onCloseRef.current = (value: string | null) => {
          resolve(value)
        }
      })
    },
    elem: (
      <Prompt open={open} value={value} onClose={onClose} onChange={onChange}/>
    )
  }
}

Style in SASS



.cover {
  align-items: center;
  background-color: rgba(0, 0, 0, 0.5);
  display: flex;
  justify-content: center;
  height: 100dvh;
  left: 0;
  position: fixed;
  top: 0;
  width: 100dvw;
}

.frame {
  background-color: white;
  padding: 16px;
}

How to use



import { usePrompt } from "./Prompt"
import { useState } from "react"

function App() {
  const { open, elem } = usePrompt()
  const [value, setValue] = useState<string>("")
  const onOpenClick = () => {
    open("Initial value").then((value) => {
      setValue(value || "cancelled")
    })
  }
  return (
    <>
      <div>
        <button onClick={onOpenClick}>Open prompt</button>
      </div>
      {value && <div>Input value: {value}</div>}
      {elem}
    </>
  )
}

You can check my git repo if you want. Hope this helps!

Legacy pages/api works in App Router Next.js project

Ryo Kuroyanagi — Fri, 02 Jun 2023 11:30:34 +0000

TL;DR

pages/api directory works even in projects that are using App Router.

Why I wrote this doc?

I tried to use Auth0 library in my Next.js project and I'm using App Router. I faced a problem that I can not use Auth0 library in the manner of app directory API routing because the library provides only a handler for the legacy pages/api. However, I noticed that even in the project, I can use the legacy pages/api directory.

This is my example code.

Hope this helps.

How to use VSCode for coding on Unity

Ryo Kuroyanagi — Sun, 12 Mar 2023 04:18:06 +0000

TL;DR

Install C# Plugin for VS Code
Install .NET Core and .NET Framework
Generate project files (.csproj) by Unity preference menu

Background & Overview

Always I start using new PC, I get stuck with setting up VSCode for Unity programming. This is a quick note for people who are willing to set up VS Code for Unity. In the following sections, I explain how to enable intellisense on VSCode. Please note that I do not explain how to set up debugging (inserting breakpoint etc.).

Prerequisite

I assume that you have Unity and VSCode on your PC. I'm using Unity 2021 now but the steps may not change event for different Unity versions.

Step 1: Install C# VS Code plugin.

In your extensions search on VS Code or you can find the plugin on this page.

Step 2: Install .NET Core / .NET Framework

At this moment, we can download the SDK from on this page. The URL may change in the future.
I installed .NET Core 7. We need .NET Framework 4.7.1 for that core. I installed .NET Framework 4.8.1 first but it did not work and error messages told me I need ver 4.7.1. You may use another version of .NET Core,

Step 3: Generate C# project files.

On Unity editor, go to preference of editor [Edit] --> [Preferences] --> [External Tools]. Set External Script Editor as Visual Studio Code, check all checkboxes in Generate .csproj files for: section and then click Regenerate project files button.

Wrap-up

That's it! Hope this make your life easier.

Your own AI knowledge base chatbot with OpenAI without Fine-Tuning

Ryo Kuroyanagi — Sun, 22 Jan 2023 10:05:14 +0000

Overview / Motivation

ChatGPT is a strong AI chatbot but it cannot answer based on our own knowledge base. Everyone is wondering if we had our own AI but it still difficult to make because we have to train machine learning models usually. I explain how to make our own AI chatbot quickly. The chatbot is not perfect but it should be a cost efficient way and it is a way that even people without ML experience can try.

How it works

The GPT-3 models available on OpenAI APIs, can find answers for questions from a document. By using that, we can extract answers from our documents. If we create an API that receives questions and answers retrieved from our documents, it should be our own AI chatbot.

Experiments on ChatGPT

The prompt template is like this.

Please give me answers for my questions based on the article.

Article: 
[YOUR KNOWLEDGE BASE SENTENCES]

Question:
[QUESTION]

The point is giving all knowledges to GPT-3 model every time you ask questions. That's simple, Right?!

Build the system

To maintain our chatbot's knowledge, we have to have a way to update knowledge easily. For this chatbot, I used Google Document to store the data and used Google Apps Script as a web server.(Please check reference to make an API endpoint with Google Apps Script on Google Sheets.)

Step 1. Create a document

Please create a Google Document and write down your knowledge that AI uses. The content can be anything but I think what are difficult to find on web is appropriate. Because, if we can find on web easily, searching on the web can be faster than using your chatbot.

My ancle phone number
XXX-XXXX-XXXX

Your tasks that are not urgent
1. task 1
2. task 2

Please note that OpenAI has limits of the length of prompt. So if the knowledge base document is too long, it fails. To avoid this, you have to call OpenAI API twice. The flow is like the below.

Find a doc that appropriate for a question.
Find answers on the doc

I'm using the idea to increase knowledge that AI can search around but I do not explain in this article because it's a bit complicated.

Step 2. Call OpenAI API from Apps Script

Let's make a JavaScript function to find answers from a Google Document that works on Apps Script platform. I created a Google Sheet and wrote the script as Apps Script of the sheet.

/**
 * @param {string} docId ID of a Google Document.
 * @param {string} q Question text.
 */
function searchDoc(docId, q) {
  // Opens a doc
  const doc = DocumentApp.openById(docId);
  // Gets the content as text not as structured data.
  const text = doc.getBody().getText();
  const data = {
    "model": "text-davinci-003",
    "prompt": `Please give me answers for my questions based on the article.\n\nArticle: ${text}\n\nQuestion: ${q}`,
    "max_tokens": 200,
    "temperature": 0.7
  }
  const options = {
    "method" : "post",
    "payload" : JSON.stringify(data),
    headers: {
      "Content-Type": "application/json",
      "Authorization": "Bearer [YOUR API KEY]",
    }
  };
  const res = UrlFetchApp.fetch("https://api.openai.com/v1/completions", options).getContentText()
  return JSON.parse(res).choices[0].text
}

It's basically easy because it's just calling the completion API. However, you have to use Apps Script specific function: UrlFetchApp instead of XMLHttpRequest in usual scripting. I'm using Davinci in the code but you can use other models if you want to lower the running cost. The price of other models are much lower than Davinci. Please check how many tokens you consumes per request on your account page of OpenAI. Your transaction history is available.

I do not talk how to receive text of questions as a parameter of GET requests. Please check a reference at the end of this article.

Step 3. Connect to Slack

I'm using this solution to check my companies rules like how to submit bill, how to apply vacation on our HR system. I'm using Slack so it is the best entry point for me. It depends on your own case.

To get a message from Slack and proxy to the Apps Script API server, it's easier to use Zapier than building a server by owrselves. The flow is like the below

Set a trigger which fires on I send a message on a specific channel on Slack
Get the message content and proxy to the API endponit
Create a new message on the channel as a chatbot.

Please note that making HTTP requests from Zapier is a limited feature for subscribers. However, you can try the function for 7 days even with their free plan.

Conclusion

That's basically it. Easy right? I hope it makes your life easier!

References

Build An API in Google Sheets and Google Apps Script

[Cloud Build] Failed to trigger build: generic::invalid_argument: invalid build: invalid image name

Ryo Kuroyanagi — Thu, 11 Aug 2022 15:11:00 +0000

TL;DR If you see the error message in this article title when you run Cloud Build trigger, the Docker image name is too long to handle for GCP.

Details

The default inline cloudbuild.yaml which is auto-generated when we create Cloud Run service, defines docker image names as like the following.

$_GCR_HOSTNAME/$PROJECT_ID/$REPO_NAME/$_SERVICE_NAME:$COMMIT_SHA

However, the image name can be so long like

asia.gcr.io/your-long-project-id/repository-name/service-name:87caefacca50d6d64288732070b522a92690dde0

Seems like GCP has the limit for handling Docker image name and it is 100 characters at max. So, if the default definition does not work for your case, you can remove an element from the template like $REPO_NAME. If the Docker image name gets shorter than 100, the build pipeline should work.

Hope this shorten your debug time !

Script to check if docker booted for Github Actions

Ryo Kuroyanagi — Mon, 25 Apr 2022 02:33:43 +0000

Background

I use Docker (docker-compose) to built system for my local development and testing on Github Actions. In my local, I can check by my eyes if my containers have booted. However, in Github Actions, I need a way to make sure that my docker containers have booted and run test for the containers after their boot. Simply I came up with the following Github Actions setting. But usually that does not work because the completion of docker-compose up -d does not mean the program / servers in the containers are ready.

jobs:
  all:
    name: My test
    runs-on: ubuntu-18.04
    steps:
      - name: Docker compose up
        run: docker-compose up -d
      - name: Test
        # run test for my containers. e.g. Access API servers
        run: ...

Solution: Script to check health

I wrote the following bash script to check my target container health status. My docker-compose has healthcheck setting (see this article to learn how to setup). It checks the health every second and finishes its process after the container gets healthy. If the program did not get healthy within the timeout, the script exits with 1.

# Name of the target container to check
container_name="$1"
# Timeout in seconds. Default: 60
timeout=$((${2:-60}));

if [ -z $container_name ]; then
  echo "No container name specified";
  exit 1;
fi

echo "Container: $container_name";
echo "Timeout: $timeout sec";

try=0;
is_healthy="false";
while [ $is_healthy != "true" ];
do
  try=$(($try + 1));
  printf "■";
  is_healthy=$(docker inspect --format='{{json .State.Health}}' $container_name | jq '.Status == "healthy"');
  sleep 1;
  if [[ $try -eq $timeout ]]; then
    echo " Container did not boot within timeout";
    exit 1;
  fi
done

This is the command to run the script.

healthcheck.sh <container_id|name> <timeout_in_sec>

A realistic command should be like this. Waits 120 seconds or less until api container gets ready. Then run test.sh (Please change whatever command you want).

healthcheck.sh api 120 && test.sh

Finally, my Github Actions got the form like this.

jobs:
  all:
    name: My test
    runs-on: ubuntu-18.04
    steps:
      - name: Docker compose up
        run: docker-compose up -d
      - name: Test
        run: healthcheck.sh api 120 && test.sh

Appendix: Fix the name of a container

Without an explicit name, docker-compose names a container like api_1 to fix the name as you want, you must specify a name as container_name in docker-compose.yml.

services:
  api:
    container_name: api

Frequently used 2 healthcheck recipes for docker-compose dependency resolution

Ryo Kuroyanagi — Mon, 25 Apr 2022 01:56:00 +0000

These days, many developers are using docker-compose to build system on our local machines. If each container does not depend on each other, the system is simple. However, we usually have dependent and dependency containers within our docker-compose networks. I'm sharing the most basic cases of dependent-dependency pattern and how to boot containers in a order that we expect.

Basics: Docker healthcheck

Docker has healthcheck functionality. The rules can be written in our Dockerfile or docker-compose. In docker-compose, it is written like following. The server container relies on db container. db should have healthcheck section and the condition. server should have db in the depends_on and the condition underneath to make sure that it boots after the db container gets health status.

services:
  server:
    image: your_server_image
    depends_on:
      db: 
        condition: service_healthy
  db:
    image: your_db_image
    healthcheck:
      test: ... # Command to check health.
      interval: 5s # Interval between health checks.
      timeout: 5s # Timeout for each health checking.
      retries: 20 # Hou many times retries.
      start_period: 10s # Estimated time to boot.

Case 1: Server depends on MySQL

Maintain server and MySQL containers in the same docker-compose runtimes is a very common pattern. The server must be booted after the MySQL container gets ready if you want to do something like table migrations when the server container booted. Pinging with mysqladmin to MySQL server in the db container, we can check if MySQL is ready. The MySQL server runs on localhost in the contianer, the host -h parameter must be localhost. The password is the one you set in the environment section with the MYSQL_ROOT_PASSWORD parameter.

  db:
    image: mysql:8.0.26
    ports:
      - 3316:3306
    environment:
      MYSQL_DATABASE: yourdb
      MYSQL_ROOT_PASSWORD: pass
    healthcheck:
      test: ["CMD", "mysqladmin" ,"ping", "-h", "localhost", "-uroot", "-ppass"]
      interval: 5s
      timeout: 5s
      retries: 20

Case 2: Server depends on another server

If we have more than 1 server, the servers may depends on others. e.g. In an image detection service system, one main server for user access and another server for image detection by machine learning. For the case, self-pinging should be the simplest solution. For the server runs on localhost and 8080 port in the container, curl command to ping should be like the below.

  image_detection:
    image: your_image
    ports:
      - 8082:8080
    healthcheck:
      test: "curl -f localhost:8080"
      interval: 5s
      timeout: 5s
      retries: 20

Chmod +x by Git on Windows

Ryo Kuroyanagi — Sat, 23 Apr 2022 14:44:50 +0000

TL;DR The following Git command is chmod +x equivalent. You can run it on your Windows machine and the permission is reflected to git repo once you commit the change.

git update-index --chmod=+x script.sh

Background

I've been a mac user for long time since I started coding. However, I'm Windows system recently for machine learning / 3DGC. Windows OS has the different file permission management system and it's not compatible to Linux / macOS.

I use Github Actions to run tests and build on Ubuntu. Sometimes bash script is used to pack complicated commands into scripts used in Actions. I found that the bash scripts can be run on Windows with Git for Windows terminal even the Linux execution permission is not set. However, once I push my scripts and try to run it on Ubuntu on Github Actions, it does not work because of a lack of permission.

Change permission and check if it has changed

As I put at the top of this article, the following command gives the execution permission to a script.

git update-index --chmod=+x script.sh

git ls-tree head command shows the permissions like the below when a script file script.sh is created on Windows. It says the permission is 644.

100644 blob <hash> script.sh

After we have used the chmod equivalent command, Git automatically stages the change. Once, we commit the change, the permission changes to like the below. Now we see 755 permission of the file.

100755 blob <hash> script.sh

That's it ! Hope this helps who are working on cross platforms.