DEV Community: Yosuke Hanaoka

Implement Face Detection App on iOS with React Native + Expo within 10 Minutes!

Yosuke Hanaoka — Tue, 15 Apr 2025 04:34:55 +0000

Ever wanted to build a face detection app using React Native + Expo? Here's how you can go from zero to working face detection in just 10 minutes — including real-time face bounding boxes and face status like yaw, pitch, and eye openness!

Let’s dive in. 💪

🧱 Step 1: Set up your environment

First, create a new Expo project with TypeScript. I specify the project name as "face-detection".

npx create-expo-app@latest --template blank-typescript

Then install the required packages:

cd face-detection
npx expo install \
  react-native-vision-camera@4.6.4 \
  react-native-vision-camera-face-detector@1.7.2 \
  @shopify/react-native-skia@1.5.0 \
  react-native-worklets-core \
  react-native-reanimated

🧠 Step 2: Paste the full sample code

Open the project with VSCode

 code .

Paste the sample code in App.tsx

// App.tsx
import React, {useEffect, useState, useRef} from "react"
import {StyleSheet, View, Text, useWindowDimensions } from "react-native"
import {Camera as VisionCamera, useCameraDevice, useCameraPermission } from "react-native-vision-camera"
import {Camera, Face, FaceDetectionOptions} from 'react-native-vision-camera-face-detector';
import {useSharedValue, useAnimatedStyle, withTiming} from 'react-native-reanimated';
import Animated from 'react-native-reanimated';

export default function App() {
  const {hasPermission} = useCameraPermission()
  const {width, height} = useWindowDimensions();
  const [faceStatus, setFaceStatus] = useState<{ yaw: string; pitch: string; eye: string } | null>(null);
  const device = useCameraDevice('front')

  useEffect(() => {
    (async () => {
      const status = await VisionCamera.requestCameraPermission();
      console.log(`Camera permission: ${status}`);
    })();
  }, [device]);

  const aFaceW = useSharedValue(0);
  const aFaceH = useSharedValue(0);
  const aFaceX = useSharedValue(0);
  const aFaceY = useSharedValue(0);

  const drawFaceBounds = (face?: Face) => {
    if (face) {
      const {width, height, x, y} = face.bounds;
      aFaceW.value = width;
      aFaceH.value = height;
      aFaceX.value = x;
      aFaceY.value = y;
    } else {
      aFaceW.value = aFaceH.value = aFaceX.value = aFaceY.value = 0;
    }
  };

  const faceBoxStyle = useAnimatedStyle(() => ({
    position: 'absolute',
    borderWidth: 4,
    borderLeftColor: 'rgb(0,255,0)',
    borderRightColor: 'rgb(0,255,0)',
    borderBottomColor: 'rgb(0,255,0)',
    borderTopColor: 'rgb(0,255,0)',
    width: withTiming(aFaceW.value, {duration: 100}),
    height: withTiming(aFaceH.value, {duration: 100}),
    left: withTiming(aFaceX.value, {duration: 100}),
    top: withTiming(aFaceY.value, {duration: 100})
  }));

  const faceDetectionOptions = useRef<FaceDetectionOptions>({
    performanceMode: 'accurate',
    landmarkMode: 'all',
    contourMode: 'none',
    classificationMode: 'all',
    trackingEnabled: false,
    windowWidth: width,
    windowHeight: height,
    autoScale: true,
  }).current;

  const handleFacesDetection = (faces: Face[]) => {
    try {
      if (faces?.length > 0) {
        const face = faces[0];

        // You can add your own logic here!!
        drawFaceBounds(face);
        setFaceStatus({ 
          yaw: face.yawAngle > 15 ? "Right" : face.yawAngle < -15 ? "Left" : "Center",
          pitch: face.pitchAngle  > 15 ? "Up" : face.pitchAngle < -10 ? "Down" : "Center", 
          eye: face.leftEyeOpenProbability > 0.7 && face.rightEyeOpenProbability > 0.7 ? "Open" : "Close" 
        });
      } else {
        drawFaceBounds();
      }
    } catch (error) {
      console.error("Error in face detection:", error);
    }
  }

  if (!hasPermission) return <Text>Camera permission is required to use this feature.</Text>
  if (device == null) return <Text>Camera device not found.</Text>

  return (
    <View style={StyleSheet.absoluteFill}>
      <Camera
        style={StyleSheet.absoluteFill}
        device={device}
        isActive={true}
        faceDetectionCallback={handleFacesDetection}
        faceDetectionOptions={faceDetectionOptions}
      />
      <Animated.View style={[faceBoxStyle, styles.animatedView]}>
        <Text style={styles.statusText}>Yaw: {faceStatus?.yaw}</Text>
        <Text style={styles.statusText}>Pitch: {faceStatus?.pitch}</Text>
        <Text style={styles.statusText}>Eye: {faceStatus?.eye}</Text>
      </Animated.View>
    </View>
  )
}

const styles = StyleSheet.create({
  animatedView: {
    justifyContent: 'flex-end',
    alignItems: 'flex-start',
    borderRadius: 20,
    padding: 10,
  },
  statusText: {
    color: 'lightgreen',
    fontSize: 14,
    fontWeight: 'bold',
  },
});

This will draw a green animated rectangle around your face and display the current yaw, pitch, and eye status (open/close).

📱 Step 3: Build and run on iOS

Generates native iOS/Android project files for using native modules. Then, installs iOS dependencies.

npx expo prebuild
npx pod-install

Run npm start first to launch the Metro Bundler. This ensures your JavaScript code is correctly loaded into the app. Then open the project in Xcode and click Run to build the app. There is no need to edit any code in Xcode. The reason we opened Xcode is simply to install the app onto your physical device.

npm start
open ios/facedetection.xcworkspace

🔧 In Xcode:
Select your physical device from the device dropdown at the top.

Click the Run ▶️ button in the top-left corner. This will build the app and install it directly onto your iPhone. Once installed, you'll see the camera view launch on your device with face detection and bounding box animations.

🛡️ Note:
If this is your first time installing the app on your device, you'll need to manually trust your developer certificate.

On your iPhone, go to:
Settings > General > VPN & Device Management
Tap your Apple ID under "Developer App" and select "Trust".

🙌 That's it!

You now have a fully functional real-time face detection app built with React Native + Expo — in just 10 minutes.

Developing a Sign Language Learning App with Kaggle’s Top Model and Customized MediaPipe Gesture Model

Yosuke Hanaoka — Mon, 06 Jan 2025 06:51:20 +0000

Introduction

This app is designed as a learning tool that allows users to check their proficiency in American Sign Language (ASL) through AI-powered quizzes after engaging with ASL learning content on YouTube. For ASL recognition, I utilized a customized MediaPipe Gesture Recognizer and a top-performing model from a Kaggle ASL competition (Unfortunately, it was not developed by me).

I created this app as a Proof of Concept (PoC) to demonstrate the idea's technical feasibility. While the UI/UX is still in its initial stage and has big room for improvement, the focus of this project was on showcasing the underlying technical capabilities.

Also, the gesture classification model used in this project references the models created by @hoyso48 and @ohkawa3. I was truly impressed by the exceptional quality. I sincerely appreciate the generosity in sharing both the models and the Jupyter Notebook on Kaggle.

GitHub Repo: https://github.com/yoshan0921/asl-practice-app

About the App

This app offers two main features for learning and practicing American Sign Language (ASL):

1. Finger Spelling Practice

Learn and practice finger spelling for the alphabet (A-Z).

2. Basic Vocabulary Practice

Access and practice approximately 200 essential ASL vocabulary words.

This interactive format allows users to actively reinforce their ASL skills through practice and immediate feedback. For a detailed demonstration, please refer to the following video.

App Demo

1. Finger Spelling Practice

2. Basic Vocabulary Practice

Technology Stack

The app was developed using the following technologies:

Front-end: React, TypeScript, MUI, MediaPipe
Back-end: Flask, Python, TensorFlow

Architecture Overview

The following diagram illustrates the key components and their interactions:

1. Finger Spelling Recognition

The front-end runs a custom gesture recognition classification model directly to process user gestures.
The model outputs the classification results in text format, which are then displayed on the front-end interface.

2. Basic Vocabulary Recognition

The front-end uses MediaPipe’s Holistic model to capture the xyz coordinates of 543 landmarks frame by frame.
These landmark data are transmitted to the back-end in real-time using Socket.IO and are stored sequentially.
Once sufficient landmark data has been accumulated, the back-end executes a basic vocabulary gesture recognition classification model.
The classification model processes the accumulated data and returns the recognition results in a list format, which are then sent back to the front-end for display.

Insights and Challenges

1. Real-time Performance Achieved

The app successfully achieved sufficient real-time performance for both the Finger Spelling and Basic Vocabulary features, ensuring smooth and responsive interactions for users during gesture recognition.

2. Technology Choice: REST API vs. Socket.IO

The Kaggle ASL competition model used in this project was designed for implementation in TFLite format, which could have been executed directly in the browser. However, for this Proof of Concept (PoC), Flask and Python were chosen to implement the gesture recognition functionality as a REST API to prioritize ease of data processing and development efficiency.

By the way, when the frame data accumulated on the back end is visualized, it appears as follows. This serves as the input data for the gesture recognition classification model.

Challenges with REST API:

Initially, the front-end recorded and accumulated frame data, sending it to the back-end in bulk via a POST request, but the server-side preprocessing required before executing the model introduced a slight delay, resulting in a noticeable lag between performing gestures and receiving recognition results.

Solution with Socket.IO:

To address this issue, Socket.IO was used instead of the REST API, transmitting frame data to the server incrementally in real-time and enabling stepped pre-processing, which successfully eliminated the lag observed in the REST API implementation.

Scalability Concern:

While Socket.IO proved effective, it may face performance issues under heavy server load or high concurrent connections due to increased processing demands on the back end, a scalability risk that was not fully tested within the scope of this project and remains an area for future investigation.

3. MediaPipe Model Integration Challenges

The app uses MediaPipe’s Gesture Recognition model for Finger Spelling and the Holistic model for Basic Vocabulary in the front-end.

Error Encountered:

When loading the Gesture Recognition model after the Holistic model, an error occurred despite ensuring cleanup through the useEffect lifecycle by calling the model instance's close() method.

Failed to load MediaPipe gesture model: RuntimeError: Aborted(Module.arguments has been replaced with plain arguments_ (the initial value can be provided on Module, but after startup the value is only looked for on a local variable of that name)) at abort (holistic_solution_si…wasm_bin.js:9:17640) at Object.get (holistic_solution_si…_wasm_bin.js:9:7759) at vision_wasm_internal.js:9:2905 at async createGestureRecognizer (Quiz.tsx:49:35)

Workaround:

The error was eventually resolved by explicitly setting window.Module to undefined to clear the previous state.

Unresolved Root Cause:

While the workaround fixed the issue, the exact cause of the error and why the solution worked remain unclear. This aspect requires further investigation for a robust understanding.

Unresolved issues

1. Dynamic Gestures (J and Z) for Finger Spelling

The current gesture models for fingerspelling do not accommodate dynamic gestures such as ‘J’ and ‘Z.’ It seems that addressing this limitation would require developing the models based on LSTM (Long Short-Term Memory) or Transformer architectures. However, this challenge has not yet been addressed.
MediaPipe's Gesture Recogniser provides a customization tool that makes it relatively easy to create classification models, which I have used in this case.
The actual code for how this customization tool works can be found in the GitHub repository below. According to the code, the create method in gesture_recognizer.py uses TensorFlow Keras to build the neural network for the classification model. However, it’s my understanding that dynamic (time series considerations) gestures are not supported by default and require customisation, such as adding layers.

2. Playback of Learning Content

The playback of learning content is currently implemented as a simple iframe embedding of YouTube videos, which leads to usability challenges in terms of controls.
YouTube provides a Player API Reference for iframe Embeds, which offers more granular control over video playback. However, it was put on hold because it could not be implemented as expected.

Planned Features

User login functionality
Improvement of learning content display
Management of learning progress and records
Optimization and personalization of quiz questions
Addition of new quiz methods (e.g., entering an address using Finger Spelling)

References

Fitness App Development with Real-Time Posture Detection using MediaPipe and React

Yosuke Hanaoka — Wed, 04 Dec 2024 00:03:32 +0000

Introduction

I recently worked with my teammates on a college project to develop a fitness app using machine learning. So I would like to leave some notes on the technical points.

The app we developed has been deployed here. I was responsible for implementing the dashboard and workout screens/features.

App URL: https://www.bodybuddy.me
GitHub: https://github.com/vinsouza99/BodyBuddy.git

What I have learned

MediaPipe offers multiple pre-trained models for image processing. The pose detection model used in this project takes webcam footage as input and outputs the coordinates of 33 body landmarks, such as shoulders, knees, elbows, and heels. This model is also available as a JavaScript library and can be executed directly in the browser.

It is crucial to appropriately define which landmarks to track and the conditions under which a count is triggered to ensure a seamless user experience. Additionally, detection can become unstable if parts of the body go out of frame or if the user is too close to the camera, causing significant fluctuations in the coordinates. To address this, it is necessary to design mechanisms that account for such scenarios and prevent miscounts.

Furthermore, pose detection consumes substantial computational resources. For example, while it ran smoothly on a recent Mac, some stuttering was observed on slightly older Windows PCs. Therefore, it is recommended to clearly define the supported environments for reliable operation.

Technology Stack

Javascript
React
Chart.js
MediaPipe
Node.js
Express.js
Sequalize
Supabase (Authentication, Storage, Postgres)

Demo

Code Description

Below is an overview of the processes performed during the workout screen execution. The flow is quite simple: the system uses the webcam feed as input and employs MediaPipe's machine-learning model to detect posture in real-time. Specifically, the MediaPipe model estimates the coordinates of 33 body landmarks (such as shoulders and knees) from the video feed.

Next, these coordinates are overlaid on a canvas, with the landmarks visualized as dots and lines. Furthermore, logic is executed to calculate angles formed by specific landmarks or to measure the distance travelled by certain points. Based on these calculations, the system increments the count for movements (e.g., squats or push-ups) or sends alerts for incorrect posture.

This entire process is repeatedly executed frame by frame using requestAnimationFrame, enabling real-time analysis of movements.

Here are some key sections of the code, explained in a bit detail.

import {
  PoseLandmarker,
  FilesetResolver,
  DrawingUtils,
} from "@mediapipe/tasks-vision";

This code imports essential modules from @mediapipe/tasks-vision to enable pose detection and visualization. The PoseLandmarker is used to detect 33 body landmarks from images or videos in real-time. The FilesetResolver manages and loads necessary resources, such as models and WASM files, required for MediaPipe tasks. Lastly, DrawingUtils provides functionality to render the detected landmarks and their connections onto a 2D canvas for visualization.

useEffect( () =>{
    const createPoseLandmarker = async () => {
      try {
        const vision = await FilesetResolver.forVisionTasks(
          "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@0.10.0/wasm"
        );

        const poseLandmarker = await PoseLandmarker.createFromOptions(vision, {
          baseOptions: {
            modelAssetPath: "https://storage.googleapis.com/mediapipe-models/pose_landmarker/pose_landmarker_lite/float16/1/pose_landmarker_lite.task",
            delegate: "GPU",
          },
          runningMode: "VIDEO",
          numPoses: 1,
        });
        poseLandmarkerRef.current = poseLandmarker;
      } catch (error) {
        console.error("Error loading PoseLandmarker:", error);
      }
    };
    createPoseLandmarker();
});

Within useEffect, call createPoseLandmarker, which loads the required resources using FilesetResolver and creates a PoseLandmarker instance using the vision resource and the specified options. The PoseLandmarker instance includes methods for processing video or webcam frames in real-time. The created instance is stored in poseLandmarkerRef and can be used in subsequent processes.

const predictPosture = async () => {
  if (!poseLandmarkerRef.current) {
    console.error("PoseLandmarker is not initialized.");
    return;
  }

  const videoElement = videoRef.current;
  const canvasElement = canvasRef.current;
  const canvasCtx = canvasElement.getContext("2d");
  const drawingUtils = new DrawingUtils(canvasCtx);

  canvasElement.width = videoElement.videoWidth;
  canvasElement.height = videoElement.videoHeight;
  clearCanvas();

  try {
    const results = await poseLandmarkerRef.current.detectForVideo(
      videoElement,
      performance.now()
    );

    if (!results?.landmarks?.length) return;

    drawingUtils.drawConnectors(
      results.landmarks[0],
      PoseLandmarker.POSE_CONNECTIONS,
      { color: "#00FF00", lineWidth: 2 }
    );

    drawingUtils.drawLandmarks(
      results.landmarks[0],
      { radius: 2, color: "#FF0000" }
    );

    const {count = 0, alert = "", calorie = 0, score = 0 } = 
      exerciseCounter?.processPose(results.landmarks[0]) || {};

    // *** The actual code is omitted as it is very long. ***
    // Add some processing using exerciseCounter's response
    // For example:
    // - Displaying count, calories burned, score earned
    // - Showing alert when user's posture is not proper

  } catch (error) {
    console.error("Error during pose detection:", error);
  }

  if (webcamRunning) {
    animationFrameIdRef.current = window.requestAnimationFrame(predictPosture);
  }
};

This code represents the main part of posture detection. It is an asynchronous function that detects posture in real-time from webcam footage and performs additional processing based on the detected landmarks. First, it checks if the PoseLandmarker required for posture detection has been initialized. If it is not initialized, an error is logged, and the process is terminated. Next, it sets up the webcam feed and canvas, adjusts the canvas size based on the video dimensions, and clears any previous drawings.

The function then uses the detectForVideo method to perform posture detection and retrieve the detected landmarks. You can get the 33 landmark data as the return value of detectForVideo. These landmarks are used to draw points and connections on the canvas, visually representing the body’s shape. Additionally, although omitted in the code above, it performs extra processing based on landmarks, such as counting movements, identifying errors in posture, and calculating metrics like calories burned and scores.

If an error occurs during posture detection, it is logged into the console to ensure the process continues safely. Finally, the function utilizes requestAnimationFrame to repeatedly call itself, enabling continuous real-time processing of video frames. This workflow allows for real-time posture detection and feedback based on webcam footage.

[
    [
        {
            "x": 0.4226442873477936,
            "y": 0.24005821347236633,
            "z": -0.21633145213127136,
            "visibility": 0.9985895752906799
        },
        {
            "x": 0.4282989799976349,
            "y": 0.22500008344650269,
            "z": -0.1938890814781189,
            "visibility": 0.9979572296142578
        },
        {
            "x": 0.4316709041595459,
            "y": 0.2257886528968811,
            "z": -0.19421276450157166,
            "visibility": 0.9979007244110107
        },
        {
            "x": 0.4349389374256134,
            "y": 0.22702443599700928,
            "z": -0.1943206638097763,
            "visibility": 0.9980199337005615
        },
        {
            "x": 0.41542500257492065,
            "y": 0.22407883405685425,
            "z": -0.20759187638759613,
            "visibility": 0.9975939393043518
        },
        {
            "x": 0.40978389978408813,
            "y": 0.22423318028450012,
            "z": -0.20759187638759613,
            "visibility": 0.9974191188812256
        },
        {
            "x": 0.40484046936035156,
            "y": 0.22473159432411194,
            "z": -0.2076997607946396,
            "visibility": 0.9975274205207825
        },
        {
            "x": 0.432990700006485,
            "y": 0.24122315645217896,
            "z": -0.07725352048873901,
            "visibility": 0.9970536231994629
        },
        {
            "x": 0.39492490887641907,
            "y": 0.2370660901069641,
            "z": -0.13616472482681274,
            "visibility": 0.9975370168685913
        },
        {
            "x": 0.4275425970554352,
            "y": 0.2611766755580902,
            "z": -0.1707993447780609,
            "visibility": 0.9989877343177795
        },
        {
            "x": 0.41113120317459106,
            "y": 0.2596157491207123,
            "z": -0.18806269764900208,
            "visibility": 0.9990490078926086
        },
        {
            "x": 0.4555538594722748,
            "y": 0.3702540993690491,
            "z": 0.013790455646812916,
            "visibility": 0.9987744688987732
        },
        {
            "x": 0.3487405776977539,
            "y": 0.3495810627937317,
            "z": -0.14501219987869263,
            "visibility": 0.9992875456809998
        },
        {
            "x": 0.46265295147895813,
            "y": 0.5064929723739624,
            "z": 0.07860222458839417,
            "visibility": 0.774071455001831
        },
        {
            "x": 0.3323880136013031,
            "y": 0.49474000930786133,
            "z": -0.1699361801147461,
            "visibility": 0.9865716695785522
        },
        {
            "x": 0.4919297993183136,
            "y": 0.5954675674438477,
            "z": 0.0021107152570039034,
            "visibility": 0.8174288272857666
        },
        {
            "x": 0.4082949161529541,
            "y": 0.5761870741844177,
            "z": -0.22895526885986328,
            "visibility": 0.9741412401199341
        },
        {
            "x": 0.5029866695404053,
            "y": 0.6259068250656128,
            "z": -0.012617086060345173,
            "visibility": 0.7789857983589172
        },
        {
            "x": 0.4148033857345581,
            "y": 0.6108712553977966,
            "z": -0.27060312032699585,
            "visibility": 0.9540517926216125
        },
        {
            "x": 0.5001327991485596,
            "y": 0.6205637454986572,
            "z": -0.05518879368901253,
            "visibility": 0.7929456830024719
        },
        {
            "x": 0.4252701997756958,
            "y": 0.5970830917358398,
            "z": -0.26628729701042175,
            "visibility": 0.9539660811424255
        },
        {
            "x": 0.4919487535953522,
            "y": 0.6099179983139038,
            "z": -0.016885722056031227,
            "visibility": 0.8022459149360657
        },
        {
            "x": 0.4270872175693512,
            "y": 0.5896612405776978,
            "z": -0.22722893953323364,
            "visibility": 0.94629967212677
        },
        {
            "x": 0.4176270067691803,
            "y": 0.6360388398170471,
            "z": 0.04059586301445961,
            "visibility": 0.9976218342781067
        },
        {
            "x": 0.3522874414920807,
            "y": 0.6387805938720703,
            "z": -0.04035309702157974,
            "visibility": 0.9984330534934998
        },
        {
            "x": 0.4253230690956116,
            "y": 0.8206213712692261,
            "z": 0.061069127172231674,
            "visibility": 0.8570517301559448
        },
        {
            "x": 0.36017337441444397,
            "y": 0.8413680791854858,
            "z": -0.017398227006196976,
            "visibility": 0.9662834405899048
        },
        {
            "x": 0.4192449450492859,
            "y": 0.9211388826370239,
            "z": 0.2934770882129669,
            "visibility": 0.6253237724304199
        },
        {
            "x": 0.3750242292881012,
            "y": 1.0136358737945557,
            "z": 0.12494354695081711,
            "visibility": 0.8725647330284119
        },
        {
            "x": 0.41356179118156433,
            "y": 0.9699124097824097,
            "z": 0.3161352276802063,
            "visibility": 0.3453260064125061
        },
        {
            "x": 0.3750942647457123,
            "y": 1.0250861644744873,
            "z": 0.13540945947170258,
            "visibility": 0.4877343773841858
        },
        {
            "x": 0.44083380699157715,
            "y": 1.0195741653442383,
            "z": 0.24557125568389893,
            "visibility": 0.6147988438606262
        },
        {
            "x": 0.3841099143028259,
            "y": 1.0800800323486328,
            "z": 0.014930106699466705,
            "visibility": 0.788257360458374
        }
    ]
]

The value that we can find in result.landmark is like the above form. There will be 33 points of dataset and those can be mapped as below.

Image from MediaPipe official page

const loadExerciseCounter = (selectedExercise) => {
  if (!selectedExercise) return null;

  const CounterClass = exerciseCounterLoader[selectedExercise.exercise_id];
  if (!CounterClass) {
    console.error(
      "Exercise counter is not implemented for:",
      selectedExercise.name
    );
    return null;
  }
  return new CounterClass();
};

Regarding the exercise counter class, the app dynamically loads the appropriate counter class for the exercise selected by a user. A base baseCounter class is prepared and specific exercises, such as squats or push-ups, implement their own logic by extending the baseCounter class. This design makes it easy to add new exercises.

const TOP_ANGLE_THRESHOLD = 170;
const BOTTOM_ANGLE_THRESHOLD = 100;

#processCount(leftShoulder, leftHip, leftKnee, leftAnkle) {
  this.shoulderHipKneeAngle = calculateAngle(leftShoulder, leftHip, leftKnee);
  this.hipKneeAnkleAngle = calculateAngle(leftHip, leftKnee, leftAnkle);

  // Judge Squat Up (top position)
  if (this.shoulderHipKneeAngle > TOP_ANGLE_THRESHOLD && !this.up) {
    this.up = true;
    this.down = false;
    console.log("Top position reached.");
  }

  // Judge Squat Down (bottom position)
  if (this.up && this.hipKneeAnkleAngle < BOTTOM_ANGLE_THRESHOLD && !this.down) {
    this.down = true;
    this.successCount += 1;
    console.log("Bottom position reached. Count:", this.successCount);
  }

  // Reset Up state when user returns to the top position
  if (this.down && this.shoulderHipKneeAngle > TOP_ANGLE_THRESHOLD) {
    this.up = false;
    console.log("Reset to start position.");
  }
}

This is an excerpt from the main logic of the squat counter class.　The method calculates the angles formed by key body landmarks (shoulder, hip, knee, and ankle) to determine the user's position and count the repetitions.

The logic starts by calculating two critical angles: the shoulder-hip-knee angle and the hip-knee-ankle angle. These angles are used to detect the top and bottom positions of the squat. When the shoulder-hip-knee angle exceeds the defined TOP_ANGLE_THRESHOLD (170 degrees), and the user is not already marked as "up," the system identifies the user as having reached the top position. At this point, it resets the "down" state to prepare for tracking the downward motion.

Next, the system detects the bottom position when the hip-knee-ankle angle drops below the BOTTOM_ANGLE_THRESHOLD (100 degrees) while the user is marked as "up." Upon reaching this position, the system increments the squat count, logs the event, and marks the user as "down."

Finally, when the user returns to the top position with a shoulder-hip-knee angle exceeding the TOP_ANGLE_THRESHOLD after being marked as "down," the system resets the "up" state, allowing the cycle to repeat for the next squat.

export const calculateAngle = (a, b, c) => {
  const radians =
    Math.atan2(c.y - b.y, c.x - b.x) - Math.atan2(a.y - b.y, a.x - b.x);
  let angle = Math.abs((radians * 180.0) / Math.PI);
  if (angle > 180.0) angle = 360 - angle;
  return angle;
};

The angle calculation is performed by the above code.

_isLandmarkUnstable(landmark) {
  if (!this.previousLandmarks) {
    this.previousLandmarks = landmark;
    return false; // First frame, no comparison possible
  }

  let unstable = false;
  landmark.forEach((landmark, index) => {
    const prevLandmark = this.previousLandmarks[index];
    const dx = landmark.x - prevLandmark.x;
    const dy = landmark.y - prevLandmark.y;

    const distance = Math.sqrt(dx * dx + dy * dy);

    if (distance > this.movementThreshold) {
      unstable = true;
    }
  });

  // Update previous landmarks for the next comparison
  this.previousLandmarks = landmark;

  return unstable;
}

Lastly, this is a mechanism to prevent false detections. MediaPipe has an issue where landmark coordinates can become significantly unstable if the user is too close to the camera or if parts of their body are outside the frame. To avoid incorrect counting under such conditions, I added the logic to ignore results when coordinates move too quickly or excessively.

That's all for this blog post. This time I used the pre-trained models provided by MediaPipe, but next time I would like to incorporate my own customised models into the application.

Machine Learning Model Deployment as a Web App using Streamlit

Yosuke Hanaoka — Tue, 27 Aug 2024 08:47:44 +0000

Introduction

A machine learning model is essentially a set of rules or mechanisms used to make predictions or find patterns in data. To put it super simply (and without fear of oversimplification), a trendline calculated using the least squares method in Excel is also a model. However, models used in real applications are not so simple—they often involve more complex equations and algorithms, not just simple equations.

In this post, I’m going to start by building a very simple machine learning model and releasing it as a very simple web app to get a feel for the process.

Here, I’ll focus only on the process, not the ML model itself. Alsom I’ll use Streamlit and Streamlit Community Cloud to easily release Python web applications.

TL;DR:

Using scikit-learn, a popular Python library for machine learning, you can quickly train data and create a model with just a few lines of code for simple tasks. The model can then be saved as a reusable file with joblib. This saved model can be imported/load like a regular Python library in a web application, allowing the app to make predictions using the trained model!

App URL: https://yh-machine-learning.streamlit.app/
GitHub: https://github.com/yoshan0921/yh-machine-learning.git

Technology Stack

Python
Streamlit: For creating the web application interface.
scikit-learn: For loading and using the pre-trained Random Forest model.
NumPy & Pandas: For data manipulation and processing.
Matplotlib & Seaborn: For generating visualizations.

What I Made

This app allows you to examine predictions made by a random forest model trained on the Palmer Penguins dataset. (See the end of this article for more details on the training data.)

Specifically, the model predicts penguin species based on a variety of features, including species, island, beak length, flipper length, body size, and sex. Users can navigate the app to see how different features affect the model's predictions.

Prediction Screen
Learning Data/Visualization Screen

Development Step1 - Creating the Model

Step1.1 Import Libraries

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score
import joblib

pandas is a Python library specialized in data manipulation and analysis. It supports data loading, preprocessing, and structuring using DataFrames, preparing data for machine learning models.
sklearn is a comprehensive Python library for machine learning that provides tools for training and evaluating. In this post, I will build a model using a learning method called Random Forest.
joblib is a Python library that helps save and load Python objects, like machine learning models, in a very efficient way.

Step1.2 Read Data

df = pd.read_csv("./dataset/penguins_cleaned.csv")
X_raw = df.drop("species", axis=1)
y_raw = df.species

Load the dataset (training data) and separate it into features (X) and target variables (y).

Step1.3 Encode the Category Variables

encode = ["island", "sex"]
X_encoded = pd.get_dummies(X_raw, columns=encode)

target_mapper = {"Adelie": 0, "Chinstrap": 1, "Gentoo": 2}
y_encoded = y_raw.apply(lambda x: target_mapper[x])

The categorical variables are converted into a numerical format using one-hot encoding (X_encoded). For example, if “island” contains the categories “Biscoe”, “Dream”, and “Torgersen”, a new column is created for each (island_Biscoe, island_Dream, island_Torgersen). The same is done for sex. If the original data is “Biscoe,” the island_Biscoe column will be set to 1 and the others to 0.
The target variable species is mapped to numerical values (y_encoded).

Step1.4 Split the Dataset

x_train, x_test, y_train, y_test = train_test_split(
    X_encoded, y_encoded, test_size=0.3, random_state=1
)

To evaluate a model, it is necessary to measure the model's performance on data not used for training. 7:3 is widely used as a general practice in machine learning.

Step1.5 Train a Random Forest Model

clf = RandomForestClassifier()
clf.fit(x_train, y_train)

The fit method is used to train the model.
The x_train represents the training data for the explanatory variables, and the y_train represents the target variables.
By calling this method, the model trained based on the training data is stored in clf.

Step1.6 Save the Model

joblib.dump(clf, "penguin_classifier_model.pkl")

joblib.dump() is a function for saving Python objects in binary format. By saving the model in this format, the model can be loaded from a file and used as-is without having to be trained again.

Sample Code

Development Step2 - Building the Web App and Integrating the Model

Step2.1 Import Libraries

import streamlit as st
import numpy as np
import pandas as pd
import joblib

stremlit is a Python library that makes it easy to create and share custom web applications for machine learning and data science projects.
numpy is a fundamental Python library for numerical computing. It provides support for large, multi-dimensional arrays and matrices, along with a collection of mathematical functions to operate on these arrays efficiently.

Step2.2 Retrieve and encode input data

data = {
    "island": island,
    "bill_length_mm": bill_length_mm,
    "bill_depth_mm": bill_depth_mm,
    "flipper_length_mm": flipper_length_mm,
    "body_mass_g": body_mass_g,
    "sex": sex,
}
input_df = pd.DataFrame(data, index=[0])

encode = ["island", "sex"]
input_encoded_df = pd.get_dummies(input_df, prefix=encode)

Input values are retrieved from the input form created by Stremlit, and categorical variables are encoded using the same rules as when the model was created. Note that the order of each data must also be the same as when the model was created. If the order is different, an error will occur when executing a forecast using the model.

Step2.3 Load the Model

clf = joblib.load("penguin_classifier_model.pkl")

"penguin_classifier_model.pkl" is the file where the previously saved model is stored. This file contains a trained RandomForestClassifier in binary format. Running this code loads the model into clf, allowing you to use it for predictions and evaluations on new data.

Step2.4 Perform prediction

prediction = clf.predict(input_encoded_df)
prediction_proba = clf.predict_proba(input_encoded_df)

clf.predict(input_encoded_df): Uses the trained model to predict the class for the new encoded input data, storing the result in prediction.
clf.predict_proba(input_encoded_df): Calculates the probability for each class, storing the results in prediction_proba.

Sample Code

Step3. Deploy

You can publish your developed application on the Internet by accessing the Stremlit Community Cloud (https://streamlit.io/cloud) and specifying the URL of the GitHub repository.

About Data Set

Artwork by @allison_horst (https://github.com/allisonhorst)

The model is trained using the Palmer Penguins dataset, a widely recognized dataset for practicing machine learning techniques. This dataset provides information on three penguin species (Adelie, Chinstrap, and Gentoo) from the Palmer Archipelago in Antarctica. Key features include:

Species: The species of the penguin (Adelie, Chinstrap, Gentoo).
Island: The specific island where the penguin was observed (Biscoe, Dream, Torgersen).
Bill Length: The length of the penguin's bill (mm).
Bill Depth: The depth of the penguin's bill (mm).
Flipper Length: The length of the penguin's flipper (mm).
Body Mass: The mass of the penguin (g).
Sex: The sex of the penguin (male or female).

This dataset is sourced from Kaggle, and it can be accessed here. The diversity in features makes it an excellent choice for building a classification model and understanding the importance of each feature in species prediction.

Creating Maps in Streamlit Apps using Folium

Yosuke Hanaoka — Mon, 12 Feb 2024 06:48:08 +0000

Introduction

In this blog post, I will show you how to use Folium to create maps in a Streamlit application. In the sample app, I will use the following two data provided by the CITY OF VANCOUVER OPEN DATA PORTAL.

Electric vehicle charging stations (CSV format)
Local area boundary (GeoJSON format)

What I Made

MarkerCluster Example
GeoJson Example

Code & Explanation

Preparation

Importing libraries

import csv
import pandas as pd
import streamlit as st
import folium
from folium import plugins
from streamlit_option_menu import option_menu

I think everyone can understand csv, pandas, streamlit and forlium. So, I don't need to explain about them.

I used streamlit_option_menu to create a menu in the sidebar. I know that there is a component called streamlit-forlium but I didn't use it this time.

Page Setting

st.set_page_config(
    page_title=None,
    page_icon=None,
    layout="wide",
    initial_sidebar_state="auto",
    menu_items=None,
)

The layout was specified as wide because I wanted to display the map as large as the full width of the screen.

Creating a Map with MarkerCluster

map1 = folium.Map(location=[49.255, -123.13], zoom_start=12)
marker_cluster = plugins.MarkerCluster().add_to(map1)

data = read_csv_data(datafile1)
for station in data:
    location = [station["latitude"], station["longitude"]]
    folium.Marker(
        location,
        popup=folium.Popup(
            "<b>Operator:</b><br>" + station["operator"] + "<br>"
            "<b>Address:</b><br>" + station["address"],
            max_width=450,
        ),
    ).add_to(marker_cluster)

Firstly, I need to do the basic settings for the map with "folium.Map()". The center position of the map (latitude and longitude) and zoom are specified as the minimum settings. Shifting the values a little bit at a time, I looked for a value where Vancouver would just fit.

Next, generate a layer of marker clusters by "plugins.MarkerCluster()” and add them to the map I just created. MarkerCluster is one of folium's many Plugins. Using this, markers placed on the map can be grouped together when the map is zoomed out. The number of markers in the cluster is displayed instead of the number of markers in the map. When the map is zoomed in, the markers are displayed one by one.

Finally, marker information contained in the CSV file is retrieved one by one and added to the marker cluster layer.

Creating a Map with GeoJson

map2 = folium.Map(location=[49.255, -123.13], zoom_start=12)

popup = folium.GeoJsonPopup(
    fields=["name"],
    aliases=["Area Name:"],
)

folium.GeoJson(
    datafile2,
    popup=popup,
).add_to(map2)

The basic map settings are the same. Then add GeoJSON data to the map. GeoJSON data itself is provided by CITY OF VANCOUVER OPEN DATA PORTAL, and I didn't edit the file.

We can display a popup by specifying the argument named popup in folium.GeoJson. folium has a function called "GeoJsonPopup" that retrieves all specified information from a JSON file and displays it in a popup.

Displaying a Map on the Screen

st.header("EV Charging Stations in the Vancouver", divider=True)
st.components.v1.html(folium.Figure().add_child(map1).render(), height=500)

st.header("Local Area Boundary in the Vancouver (GeoJSON)", divider=True)
st.components.v1.html(folium.Figure().add_child(map2).render(), height=500)

Streamlit's Components API is used here. Please check the following url for details. (Components API Reference)

There are two types of Streamlit components: static components and bi-directional components. This time, my goal is solely to render a map from a Python visualization library (forlium); I used st.components.v1.html

It is very easy to use and is as follows.

Function signature
st.components.v1.html(html, width=None, height=None, scrolling=False)
Parameters
html (str)	The HTML string to embed in the iframe.
width (int)	The width of the frame in CSS pixels. Defaults to the app's default element width.
height (int)	The height of the frame in CSS pixels. Defaults to 150.
scrolling (bool)	If True, show a scrollbar when the content is larger than the iframe. Otherwise, do not show a scrollbar. Defaults to False.

The HTML returned from the forlium’s render method is passed as the first parameter, the width is not specified but left to the screen size, and the height is set to 500px.

As explained in the Streamlit documentation, the result is incorporated as an iframe.

By the way, I didn't use bi-directional component in this blog post, but it is something not simply controlled and displayed only from the Streamlit (Python) side but instead receives input values from the component's UI, passes them to the Streamlit (Python) side for processing and then returns the results to the component side.

I want to try bi-directional component next time.

That's all for this blog post!

Context Manager and the "with" Statement in Python

Yosuke Hanaoka — Thu, 14 Dec 2023 04:48:02 +0000

Introduction

According to the Python official document, the context manager is defined as below.

A context manager is an object that defines the runtime context to be established when executing a with statement. The context manager handles the entry into, and the exit from, the desired runtime context for the execution of the block of code. Context managers are normally invoked using the with statement (described in section The with statement), but can also be used by directly invoking their methods.

Typical uses of context managers include saving and restoring various kinds of global state, locking and unlocking resources, closing opened files, etc.

Source: Python Official Documentation

In this blog post, I will confirm how the context manager in Python works with code examples. There are two ways to implement a context manager: class-based and function-based.

How Class-Based Context Manager Works

To define a context manager, we need to implement the .__enter__ and .__exit__ special methods in our classes.

Method	Description
.__enter__(self)	Enter the runtime context and return either this object or another object related to the runtime context. The value returned by this method is bound to the identifier in the as clause of with statements using this context manager.
.__exit__(self, exc_type, exc_val, exc_tb)	Exit the runtime context and return a Boolean flag indicating if any exception that occurred should be suppressed. If an exception occurred while executing the body of the with statement, the arguments contain the exception type, value and traceback information. Otherwise, all three arguments are None.

Source: Python Official Documentation

Code Example: Class-Based Context Manager

Example 1: Normal end

class ContextManager(object):

    def __init__(self):
        print("call __init__")

    def __enter__(self):
        print("call __enter__")
        return self

    def __exit__(self, exc_type, exc_value, traceback):
        print("call __exit__")
        if exc_type is None:
            print('exit normally')
        else:
            print(f'exc_type={exc_type}')
            return True  # Surpress the exception

    def work(self):
        print("start work")
        # raise Exception() # Exception occurred
        print("end work")


if __name__ == '__main__':
    try:
        with ContextManager() as cm:
            cm.work()
    except Exception as e:
        print("Exception is propagated to the caller")

Execution result:

call __init__
call __enter__
start work
end work
call __exit__
exit normally
yosuke@Yosuke-Hanaoka python-sandbox %

Example 2:
An exception occurs in the "with" block and .__exit__() return True.

    def work(self):
        print("start work")
        raise Exception() # Exception occurred
        print("end work")

Execution result:
The exception is suppressed in .__exit__() and NOT propagated to the caller.

call __init__
call __enter__
start work
call __exit__
exc_type=<class 'Exception'>
yosuke@Yosuke-Hanaoka python-sandbox %

Example 3:
An exception occurs in the "with" block and .__exit__() return False

    def __exit__(self, exc_type, exc_value, traceback):
        print("call __exit__")
        if exc_type is None:
            print('exit normally')
        else:
            print(f'exc_type={exc_type}')
            return False  # Propagate the exception

Execution result:
The exception is NOT suppressed in .__exit__() and propagated to the caller.

call __init__
call __enter__
start work
call __exit__
exc_type=<class 'Exception'>
Exception is propagated to the caller
yosuke@Yosuke-Hanaoka python-sandbox %

How Function-Based Context Manager Works

Decorating an appropriately coded generator function with @contextmanager, we can automatically get a function-based context manager that provides .__enter__() and . __exit__().

The processing before and after yield will be .__enter__() and .__exit__(). If there is an object to return by __enter__, we can return with yield. The object returned by yield is bound to the identifier in the as clause of with statements using this context manager.

If any exception occurs in the "with" statement, it can be caught in the except clause of the context manager function, as we can see below.

Code Example: Function-Based Context Manager

Example 1: Normal end

from contextlib import contextmanager

@contextmanager
def context_manager_func():
    print('__enter__')
    try:
        yield "test"
    except Exception as e:
        print("Exception is caught")
    finally:
        print('__exit__')


if __name__ == '__main__':
    try:
        with context_manager_func() as cm:
            print("start work")
            # raise Exception() # Exception occurred
            print("end work")
    except Exception as e:
        print("Exception is propagated to the caller")

Execution result

__enter__
start work
end work
__exit__
yosuke@Yosuke-Hanaoka python-sandbox %

Example 2:
An exception occurs in the "with" block.

if __name__ == '__main__':
    try:
        with context_manager_func() as cm:
            print("start work")
            raise Exception("")
            print("end work")
    except Exception as e:
        print("Exception is propagated to caller")

Execution result:
The exception is caught in the except clause of the context manager function.

__enter__
start work
Exception is caught
__exit__
yosuke@Yosuke-Hanaoka python-sandbox %

Example 3:
An exception occurs in the "with" block, and the context manager function doesn't catch it.

@contextmanager
def context_manager_func():
    print('__enter__')
    try:
        yield "test"
    # except Exception as e:
    #     print("Exception is caught")
    finally:
        # Always done last, whether an exception occurs or not.
        print('__exit__')

Execution result:
The exception is NOT suppressed in the context manager function and propagated to the caller.

__enter__
start work
__exit__
Exception is propagated to caller
yosuke@Yosuke-Hanaoka python-sandbox %

Reference:

Python 3.12.0 documentation: Statement Context Managers
Python 3.12.0 documentation: Context Manager Types
PEP 343 – The “with” Statement
Context Managers and Python's with Statement

Boost Python Performance: A Guide to Asyncio, Threading, and Multiprocessing

Yosuke Hanaoka — Mon, 11 Dec 2023 15:29:31 +0000

Introduction

Concurrency is one of the approaches that can drastically improve the performance of our Python programs, which can be achieved in Python using numerous methods and modules. In this blog post, I would like to summarize my understanding and share the results of my attempts to speed up Python programs using the following three basic libraries.

asyncio: Coroutine-Based Concurrency
threading: Thread-Based Concurrency
multiprocessing: Process-Based Concurrency

What is Concurrency?

Before we get to the main topic of concurrent programming, please let me clarify the definition of the word "Concurrent". There are a variety of slightly different understandings of the word “concurrent" on the Internet, but after reading various explanations by various people, I understand it as follows, and I use the term concurrent with this thought in this blog post.

Concurrency is the ability to manage multiple tasks at the same time. The multiple tasks are executed in overlapping time periods, but not necessarily simultaneously, and they can be interleaved or executed in overlapping time periods.

Parallelism is a subset of concurrency and the ability to manage multiple tasks simultaneously. The multiple tasks are required to execute simultaneously, and the purpose of parallelism is to increase computational performance.

What limits the speed of our Python program?

In reducing the overall execution time of a program, it is important to understand what is currently influencing or limiting the execution time. This is because the reason for this will change the effective approach. In general, there are two main types of causes that limit program execution time: 1) CPU-bound and 2) I/O-bound.

CPU-bound refers to a situation where the execution time of a program depends on the computation speed of the CPU. For example, suppose there is a program that performs large-scale scientific calculations. If the program doesn't perform input/output (I/O) to/from the disk but takes a considerable amount of time to complete processing, the processing speed of this program is dependent on the CPU computation speed. Other examples include compression/decompression, encryption, and image conversion processes. The simplest solution is to use higher-frequency cores. Another solution is to rewrite the program so that it processes multiple works in parallel by multiple cores instead of one core, thereby reducing the overall program execution time.

I/O-bound refers to a situation where the execution time of a program depends on the processing speed of I/O. For example, suppose there is a program that searches for a given document from a large amount of data stored on disk. If the faster the disk is, the less time it takes to search, then the processing speed of this program depends on the speed of the I/O, not the CPU. Other examples include Web API calls, network latency, etc. Since the CPU is not working during I/O processing, it is CPU-waiting time. Therefore, by having other processing take place during this waiting time, the overall execution time of the program can be shortened.

Basic Types of Concurrent Programming in Python

asyncio: Coroutine-Based Concurrency

A coroutine is a unit that allows execution to be suspended and resumed. A single thread may execute many coroutines in an event loop. Unlike threads and processes, where the operating system controls when a thread or process is suspended and when it is resumed and executed, coroutines themselves control when they are suspended and resumed. Coroutines are defined and used via the async/await syntax in Python.

The asyncio is a library for writing single-threaded asynchronous I/O processing code using the async/await syntax. Asynchronous I/O processing is, simply put, "not waiting for one I/O process to finish before processing another. asyncio is appropriate for I/O-bound activities.

threading: Thread-Based Concurrency

A thread is the smallest unit of CPU utilization within a process. Basically, a CPU can only execute one thread in parallel on one core. At least one thread called the MainThread is included in a process. Any additional threads that we create within the process will belong to that process.

The threading is a library for launching multiple threads in the same process and writing multi-threaded concurrent code. Due to Global Interpreter Lock (GIL), no matter how many threads are created in the same process, there is always only one thread running at a time in the same process while the other threads wait to acquire the lock to execute. threading is appropriate for I/O-bound activities.

multiprocessing : Process-Based Concurrency

A process is a program being currently executed by OS. For example, when you run a Python source code, the interpreter compiles the source code into byte code; the OS executes this byte code and begins processing it as described in the source code. This is called a process.

The multiprocessing is a library for launching multiple processes and writing parallel processing code. Since GIL exists on a per-process basis, multiprocessing allows threads in each process to execute the bytecode in parallel. multiprocessing is appropriate for CPU-bound activities.

Comparison Table

Libraries	Num. of Thread	Num. of Process	Num. of Core	Concurrency Type
asycio	1	1	1	Concurrency
threading	N	1	1	Concurrency
multiprocessing	1/Process	N	N	Parallelism

The impact of GIL on multi-thread programming

Python's Global Interpreter Lock (GIL) is, simply put, a mutex (lock) that allows only one thread with a lock to execute bytecode in the same process, even when there are multiple threads, and the other threads are kept in a waiting state. In other words, multiple threads cannot run simultaneously within a single process, and Python multithreading can't be parallelism.

Python uses GIL to manage memory safely. On the other hand, it can be a performance bottleneck in CPU-bound and multi-threaded code.

Code Example

Sorry for the lengthy lead-in. Finally, I’ve reached the main part. This time, I used the following code for testing each library. I defined a function that processes CPU-bound, waits for I/O processing, and CPU-bound in sequence. Regarding CPU-bound, this function simply executes numerous loops to load. worker() is for synchronous, threaded, and multiprocessing, and worker_async() is for asyncio.

Then, I used asyncio, threading and multiprocessing libraries to run work() and work_async(). To measure processing time, logs are output before and after CPU and I/O bound. Additionally, logs are also output at 25%, 50%, and 75% progress of CPU bound (loop processing). But, for readability, log output is omitted from the following code display. The full version of the code can be found at the Github Gist link, concurrent_test.py .

The I/O bounds should actually be verified using such as the following libraries, but this time, I will simplify that part by replacing it with time.sleep() as an example.

File I/O -> built-in modules or aiofiles
Network I/O (HTTP) -> requests or aiohttp
Network I/O (DB) -> sqlite or aiosqlite

Code Example Execution Result

Below is a summary of the results of the code example run. The actual output log can be found at the Github Gist link, console.log

In the case of programs including I/O processing waits, asyncio executes the next CPU process during the I/O processing wait time so that processing is performed without making the CPU wait. Compared to synchronous, the overall processing time is reduced by 6 seconds (three times the 2-second I/O processing wait).

Regarding threading, the CPU time behaviour of threading in the above table is only to show the characteristics and is not exact, but as can be seen from the log output, the loop processing progress of each Thread is side by side as in the case of CPU bound, so it's assumed to be processed in this way.

The problem of this threading case is that there is a long CPU wait in the middle of the processing. Personally, I expected I/O wait time will be used a little more effectively by shifting the three CPU=bound processings as in the asyncio case. However, it didn't happen.

So I lowered CPU_LOAD to 10**4 and re-ran. In that case, the threading took the same form as in asyncio. In short, instead of three CPU-bound processes proceeding concurrently, it completed only one CPU-bound processing first and then started the next CPU-bound processing while waiting for I/O processing. What criteria does threading use to switch processing? I would like to explore this in-depth separately.

In multiprocessing, the main thread of each process is executed concurrently. It is not constrained by GIL as in the above three cases. Therefore, even in this example, the entire process is completed in about one-third of the time compared to synchronous processing.

Conclusion

If we want to accelerate CPU-bound processing, we must adopt multiprocessing because GIL exists in Python. Anything other than multiprocessing is not an option.
If we want to speed up I/O bound processing, we should apply asyncio or threading to utilize the waiting time.
As to whether to select asyncio or threading, asyncio can be the first choice because it's possible to process with one thread, but there are some processing that asyncio can't handle, so I think that threading is a candidate in that case.
Moreover, as shown in my code example above, there might be cases where threading can't use CPU efficiently at all compared to asyncio.
As far as the above results are concerned only, multiprocessing is the fastest. We need to consider more complex cases, such as exchanging data between processes. Plus, it can't use I/O processing waits efficiently.
The first thing we need to consider is to decide if we should use these libraries. In some cases, they will not have much effect and will only complicate the code. Then, once we have determined that we need to apply these libraries for concurrent programming, the next step is to examine whether the program is CPU-bound or I/O-bound and select the appropriate library.