DEV Community: Yuiko Koyanagi

I built “Mozary”: An iOS App That Turns Your Photos Into a Zoomable Mosaic

Yuiko Koyanagi — Fri, 13 Mar 2026 15:42:56 +0000

Mozary is an iOS app that turns hundreds of your photos into a single zoomable mosaic.

Zoom out → it looks like one image
Zoom in → individual photos appear (and you rediscover forgotten moments)

App Link: https://apps.apple.com/us/app/mozary/id6759013945
Try it out!

#buildinpublic #mobileapp #computervision #imageprocessing #productdevelopment | Yuiko Koyanagi

I’ve been building a photo mosaic app that creates an image from images. You pick a target photo, and the app composes it using many smaller photos—so from a distance it looks like one picture, but when you zoom in, you can still see each individual photo clearly. I put a lot of work into the processing pipeline, and the generation is genuinely fast (the video isn’t sped up). Now it’s in app review—excited to share more once it’s live! #buildinpublic #mobileapp #computervision #imageprocessing #productdevelopment

linkedin.com

The idea (why I built it)

I take a lot of photos, but I rarely revisit them.

I wanted a way to turn “a pile of photos” into something you actually want to look at.

During a trip, I saw a huge mosaic wall and thought: what if my own memories could become a mosaic artwork?

That became Mozary.

How it works (user flow)

Choose a target image (what the mosaic becomes)
Choose photos from your library to use as tiles (up to 1000)
Generate (usually 5–20s, up to ~40s for 1000 photos)
Pinch to zoom, then export as an image/video

The engineering part: making the viewer buttery smooth

The hardest part wasn’t generating the mosaic.

It was rendering it interactively.

A mosaic like 200×150 already means 30,000 tiles.

Naively drawing them every frame is a fast path to jank (or crashes).

Borrowing an idea from Google Maps: LOD (Level of Detail)

Google Maps doesn’t draw every tiny detail at every zoom level.

It switches what it renders depending on zoom.

Mozary uses a similar LOD approach (3 levels):

Zoomed out: draw a single pre-rendered image

Instead of looping through 30,000 tiles, I pack each tile color into a pixel and render one image.

Same look, dramatically cheaper.
Zoomed in: render only what’s visible, in 256×256px tiles

I issue render requests only for the region currently on screen.
Keep the main thread free: heavy rendering runs on a background thread

So pan/zoom stays responsive.

Result: the mosaic scrolls/zooms smoothly even at large sizes.

Color matching: RGB → Lab

Mosaic quality is mostly about color matching.

I first tried RGB distance, but it often chose tiles that were numerically close yet visually off.

RGB isn’t perceptually uniform.

Switching to Lab color space improved matches immediately:

Convert each photo’s average color RGB → Lab
For each grid cell, pick the tile with the smallest Euclidean distance in Lab

Simple change, big visual improvement.

Optional: Color grading when your photo set lacks colors

Sometimes your selected photos don’t cover the target’s color palette.

So I added an optional “color grading overlay”:

overlay the target’s hue + saturation
keep the tiles’ luminance (so the photo texture stays)

It’s subtle when your set already has enough color, but helpful when it doesn’t:

Speed tricks that mattered

Two practical optimizations made generation fast enough for real use:

Parallelize photo analysis (average color computation)
Downsample for analysis (I use ~40×40px instead of full-res)

Even if photos are several MB, the color info needed for matching is captured at small size—saving both memory and time.

What I want to improve next

Reduce repeated use of the same tile photo (more even distribution)

That's it!

If you try it, I’d love feedback—especially on export quality and performance on older devices.

App Link: https://apps.apple.com/us/app/mozary/id6759013945

Hosting Applications with AWS CDK: An Introduction to Infrastructure as Code

Yuiko Koyanagi — Wed, 16 Oct 2024 10:18:27 +0000

Introduction

Until now, I have been manually building the infrastructure for my applications. This time, I challenged myself to manage the infrastructure using Infrastructure as Code (IaC).

By utilizing AWS CDK, I was able to codify complex infrastructure settings, manage them simply, and enable efficient deployments. In this article, I will introduce how to host an application using IaC with AWS CDK.

Overview of the Application Architecture

The application I created this time has the following structure. I designed it to efficiently run both the frontend and backend using Docker images on AWS.

Frontend: React + TypeScript
Backend: NestJS + TypeScript Below is a diagram of the application's architecture:

Infrastructure Design

First, I needed to create a secure network configuration using a VPC (Virtual Private Cloud). Therefore, I set up public and private subnets within the VPC.

I placed an ALB (Application Load Balancer) in the public subnet and the backend and frontend servers managed by ECS (Elastic Container Service) Fargate in the private subnet. This ensures that the servers are not directly exposed to the outside, enhancing security.

Users can access the application via the Internet Gateway. Access is received through the ALB placed in the public subnet. The ALB is designed to route requests to the appropriate services and perform load balancing.

For the database, I decided to use DynamoDB to store data. The ECS tasks in the private subnet are configured to access DynamoDB via VPC endpoints. This allows communication with DynamoDB without going through the internet, improving communication security and performance.

The Docker images used by the servers are stored in ECR (Elastic Container Registry). ECS uses these images to run the application. Additionally, by obtaining images from S3 and ECR via VPC endpoints, it is possible to use resources within the private network.

Implementation

Next, I will translate the above infrastructure design into code.

Setup

First, set up the authentication information using AWS CLI:

aws configure

Then, proceed with the CDK setup:

npm install -g aws-cdk
mkdir cdk && cd cdk  # Create a directory of your choice and move into it
cdk init app --language typescript
cdk bootstrap aws://${AWS_ACCOUNT_ID}/${AWS_REGION_NAME}

This automatically generates the files necessary for CDK settings.

Prepare Docker images for both the frontend and backend and store them in ECR.

Writing the CDK Content

Modify cdk-stack.ts Since the CdkStack class is already created, write the above design into its constructor. I want to create various environments such as testing and production, so I add environment as an argument to the constructor for future use.

export class CdkStack extends cdk.Stack {
  constructor(
    scope: Construct,
    id: string,
    environment: string, // Added
    props?: cdk.StackProps
  ) {

This CdkStack class is called in cdk.ts, so modify the code in cdk.ts to make environment usable:

#!/usr/bin/env node
import * as cdk from "aws-cdk-lib";
import "source-map-support/register";
import { CdkStack } from "../lib/cdk-stack";

const app = new cdk.App();
const environment = app.node.tryGetContext("env"); // Obtain from command line during deployment
new CdkStack(app, `${environment}CdkStack`, environment);
app.synth();

From here, write the content inside the constructor of the CdkStack class in cdk-stack.ts.

First, create a VPC. Since we need private and public subnets this time, write as follows:

const vpc = new ec2.Vpc(this, `${environment}AppVpc`, {
  vpcName: `${environment}AppVpc`,
  maxAzs: 2,
  natGateways: 1,
  subnetConfiguration: [
    {
      name: `${environment}AppPublicSubnet`,
      subnetType: ec2.SubnetType.PUBLIC,
      cidrMask: 24,
    },
    {
      name: `${environment}AppPrivateSubnet`,
      subnetType: ec2.SubnetType.PRIVATE_ISOLATED,
      cidrMask: 24,
    },
  ],
});

Regarding the Internet Gateway, since we are creating a public subnet, it is automatically created even though it is not explicitly mentioned in the code.

Next, create VPC endpoints that the private subnet will use:

// VPC Endpoint for DynamoDB
vpc.addGatewayEndpoint(`${environment}DynamoDbEndpoint`, {
  service: ec2.InterfaceVpcEndpointAwsService.DYNAMODB,
});

// VPC Endpoint for S3 (necessary for obtaining Docker images)
vpc.addGatewayEndpoint(`${environment}S3Endpoint`, {
  service: ec2.GatewayVpcEndpointAwsService.S3,
});

// VPC Endpoint for ECR (endpoint to send requests)
vpc.addInterfaceEndpoint(`${environment}EcrApiEndpoint`, {
  service: ec2.InterfaceVpcEndpointAwsService.ECR,
  subnets: { subnetType: ec2.SubnetType.PRIVATE_ISOLATED },
});

// VPC Endpoint for ECR Docker (endpoint to obtain Docker images)
vpc.addInterfaceEndpoint(`${environment}EcrDockerEndpoint`, {
  service: ec2.InterfaceVpcEndpointAwsService.ECR_DOCKER,
  subnets: { subnetType: ec2.SubnetType.PRIVATE_ISOLATED },
});

// VPC Endpoint for CloudWatch Logs (for logging)
vpc.addInterfaceEndpoint(`${environment}CloudWatchLogsEndpoint`, {
  service: ec2.InterfaceVpcEndpointAwsService.CLOUDWATCH_LOGS,
  subnets: { subnetType: ec2.SubnetType.PRIVATE_ISOLATED },
});

All connections are with the private subnet, but for DynamoDB and S3, we can use addGatewayEndpoint (gateway-type endpoints), so specifying the subnet is unnecessary.

Next, create ECS, which is necessary for container management:

const cluster = new ecs.Cluster(this, `${environment}AppCluster`, {
  clusterName: `${environment}AppCluster`,
  vpc: vpc,
});

Create an IAM role for Fargate to use:

const taskExecutionRole = new iam.Role(
  this,
  `${environment}TaskExecutionRole`,
  {
    // Ensure only ECS tasks can assume this role
    assumedBy: new iam.ServicePrincipal("ecs-tasks.amazonaws.com"),
  }
);

// Attach necessary access permissions
taskExecutionRole.addManagedPolicy(
  iam.ManagedPolicy.fromAwsManagedPolicyName(
    "service-role/AmazonECSTaskExecutionRolePolicy"
  )
);

Create a Fargate task definition for the frontend:

const frontendTaskDef = new ecs.FargateTaskDefinition(
  this,
  `${environment}AppFrontendTaskDef`,
  {
    family: `${environment}AppFrontendTaskDef`,
    memoryLimitMiB: 512,
    cpu: 256,
    executionRole: taskExecutionRole,
  }
);

Since we are using ECR, specify the image repository for the frontend (unnecessary if using Docker Hub):

const frontendRepository = ecr.Repository.fromRepositoryName(
  this,
  `${environment}frontendRepository`,
  "app-react-nginx-image"
);

Add container information to the task definition created above. This time, we specify the image repository:

const frontendContainer = frontendTaskDef.addContainer(
  `${environment}AppFrontendContainer`,
  {
    containerName: `${environment}AppFrontendContainer`,
    image: ecs.ContainerImage.fromEcrRepository(
      frontendRepository,
      "latest"
    ),
    logging: new ecs.AwsLogDriver({
      streamPrefix: "Frontend", // Prefix for logging
    }),
  }
);

frontendContainer.addPortMappings({
  containerPort: 80,
  protocol: ecs.Protocol.TCP,
});

The containerPort refers to the port for access from the ALB to Fargate, so it can remain as 80 regardless of whether the app itself is hosted with HTTPS.

Next, launch a Fargate service for the frontend server:

const frontendService = new ecs.FargateService(
  this,
  `${environment}AppFrontendService`,
  {
    serviceName: `${environment}AppFrontendService`,
    cluster: cluster,
    taskDefinition: frontendTaskDef,
    assignPublicIp: false,
    vpcSubnets: {
      subnetType: ec2.SubnetType.PRIVATE_ISOLATED,
    },
  }
);

// Configure auto-scaling
const scalingFrontend = frontendService.autoScaleTaskCount({
  minCapacity: 1,
  maxCapacity: 5,
});

// Auto-scale when CPU utilization exceeds 50%
scalingFrontend.scaleOnCpuUtilization("CpuScalingFrontend", {
  targetUtilizationPercent: 50,
});

// Auto-scale when memory utilization exceeds 70%
scalingFrontend.scaleOnMemoryUtilization("MemoryScalingFrontend", {
  targetUtilizationPercent: 70,
});

With this, the settings for the frontend server are complete. Next, proceed to set up the backend server. Since it is almost the same as the frontend server, explanations will be brief.

const backendTaskDef = new ecs.FargateTaskDefinition(
  this,
  `${environment}AppBackendTaskDef`,
  {
    family: `${environment}AppBackendTaskDef`,
    memoryLimitMiB: 512,
    cpu: 256,
    executionRole: taskExecutionRole,
  }
);

const backendRepository = ecr.Repository.fromRepositoryName(
  this,
  `${environment}backendRepository`,
  "app-nestjs-image"
);

const backendContainer = backendTaskDef.addContainer(
  `${environment}AppBackendContainer`,
  {
    containerName: `${environment}AppBackendContainer`,
    image: ecs.ContainerImage.fromEcrRepository(
      backendRepository,
      "latest"
    ),
    logging: new ecs.AwsLogDriver({
      streamPrefix: "Backend",
    }),
  }
);

backendContainer.addPortMappings({
  containerPort: 3000,
  protocol: ecs.Protocol.TCP,
});

const backendService = new ecs.FargateService(
  this,
  `${environment}AppBackendService`,
  {
    serviceName: `${environment}AppBackendService`,
    cluster: cluster,
    taskDefinition: backendTaskDef,
    assignPublicIp: false,
    vpcSubnets: {
      subnetType: ec2.SubnetType.PRIVATE_ISOLATED,
    },
  }
);

const scalingBackend = backendService.autoScaleTaskCount({
  minCapacity: 1,
  maxCapacity: 5,
});

scalingBackend.scaleOnCpuUtilization("CpuScalingBackend", {
  targetUtilizationPercent: 60,
});

scalingBackend.scaleOnMemoryUtilization("MemoryScalingBackend", {
  targetUtilizationPercent: 75,
});

Next, create the ALB:

const loadBalancer = new elbv2.ApplicationLoadBalancer(
  this,
  `${environment}AppApplicationLoadBalancer`,
  {
    vpc,
    loadBalancerName: `${environment}AppApplicationLoadBalancer`,
    internetFacing: true,
  }
);

Connect the ALB to resources in the private subnet:

const FrontendTargetGroup = new elbv2.ApplicationTargetGroup(
  this,
  `${environment}AppFrontendTG`,
  {
    targetGroupName: `${environment}AppFrontendTG`,
    vpc,
    port: 80,
    protocol: elbv2.ApplicationProtocol.HTTP,
    targets: [frontendService],
    healthCheck: {
      path: "/",
    },
  }
);

const backendTargetGroup = new elbv2.ApplicationTargetGroup(
  this,
  `${environment}AppBackendTG`,
  {
    targetGroupName: `${environment}AppBackendTG`,
    vpc,
    port: 3000,
    protocol: elbv2.ApplicationProtocol.HTTP,
    targets: [backendService],
    healthCheck: {
      path: "/v1/health", // Specify the API for health checks
    },
  }
);

Describe the ALB's request handling:

const listener = loadBalancer.addListener(`${environment}HttpListener`, {
  port: 80,
  open: true,
  defaultAction: elbv2.ListenerAction.forward([FrontendTargetGroup]),
});

listener.addTargetGroups(`${environment}BackendTargetGroups`, {
  targetGroups: [backendTargetGroup],
  priority: 1,
  conditions: [elbv2.ListenerCondition.pathPatterns(["/v1/*"])],
});

This time, we are using a single ALB to distribute traffic to the frontend and backend. By default, requests are forwarded to the frontend, but if there is access with /v1, it is set to forward to the backend Fargate. If you are using two ALBs, the addTargetGroups configuration is unnecessary.

Finally, create the necessary DynamoDB table. In DynamoDB, only the PartitionKey is mandatory, but you can include SortKey, Index, and TTL as needed.

const usersTable = new dynamodb.Table(this, `${environment}UsersTable`, {
  tableName: `${environment}-users`,
  partitionKey: {
    name: "account_id",
    type: dynamodb.AttributeType.STRING,
  },
});
usersTable.grantReadWriteData(backendTaskDef.taskRole);

Now that we have written the code necessary for IaC, we just need to execute and verify it.

Execution

First, check for syntax errors in the CDK code above:

cdk synth --context env=dev

If there are errors, they will appear as follows, so fix them as needed:

Error: Validation failed with the following errors:
  [undefinedCdkStack/undefinedAppFrontendTargetGroup] Target group name: "undefinedAppFrontendTargetGroup" can have a maximum of 32 characters.
  [undefinedCdkStack/undefinedAppBackendTargetGroup] Target group name: "undefinedAppBackendTargetGroup" can have a maximum of 32 characters.

If you can execute without errors, proceed to deploy. If the Docker image is not functioning properly, Fargate will repeatedly start up, encounter errors, shut down, and restart. If it is taking an excessive amount of time, please check the AWS Management Console.

cdk deploy --context env=dev

With this, hosting the application using IaC is complete.

Conclusion

This was my first attempt at IaC, and by managing the infrastructure as code, I found it very beneficial to build infrastructure for multiple environments such as production and development without human errors.

If this article was helpful in any way, I would be encouraged if you could press the like button! Thank you for reading to the end.

How to Automatically Adjust Colors in High Contrast Mode

Yuiko Koyanagi — Thu, 22 Aug 2024 14:10:05 +0000

Introduction

I recently received a bug report where an SVG icon was not displaying correctly in high contrast mode. In this article, I’ll share the solution that worked for me.

Solution

In high contrast mode, I used the CanvasText system color to automatically adjust the icon's color.

.icon {
  mask-image: url(svg-link);
  background-color: currentColor;
  ...
}

@media (forced-colors: active) {
  .icon::before {
    background-color: CanvasText;
  }
}

In my case, I initially used currentColor to inherit the color from the parent element. However, in high contrast mode, I wanted to set the background-color to CanvasText universally within the child element, so I applied this change.

What is CanvasText?

CanvasText refers to the text color used for application content or documents. It automatically adjusts to provide the best contrast against the system's background color.

By using CanvasText, you ensure that text and icons remain visible even when the user enables high contrast mode. Additionally, since CanvasText adapts based on the system's theme, it works well with both dark and light modes.

In my case, the icon's background-color was initially set to black. However, when the background turned black in high contrast mode, the icon became invisible. Changing the color to white made it visible again, but to handle this consistently across all scenarios, I opted to use the system color CanvasText.

References

https://developer.mozilla.org/en-US/docs/Web/CSS/system-color

OpenAI: AI reviews your code automatically!

Yuiko Koyanagi — Thu, 22 Dec 2022 09:07:51 +0000

Hi there!

Have you ever felt that reviews are a hassle?

I have launched an app which automatically reviews your code with using OpenAI, AI Code Reviewer!

[Note] I sold this Application and the owner is not me anymore.

I took first place in the Product hunt 🎉🎉🎉

https://www.producthunt.com/posts/ai-code-reviewer

How to use

It's very easy.
Just paste the code you would like reviewed and press the button.

For example, I purposely paste the wrong code which has type mistakes and have it reviewed: const [count, setCount] = useState<string>(0);

Good! AI said, "The code is missing a type annotation for the count variable. The type should be set to number instead of string. ".

Also try to have a Python code reviewed!.
Here is the code that I forgot to add the colon.

def hello()
  print('hello')

The reviews is:

This code has a syntax error. The print statement should be indented so that it is part of the hello() function. The correct code should look like this:

def hello(): print('hello')

Good! Yes, this code has a syntax error and now you got a good review.

Like that, AI Code Reviewer helps you write the right code.

If you have any questions, you can also ask AI Chat Bot.
Try to ask about useEffect.

I asked, "How should I use useEffect? What should I watch out for?", and AI answered, "Using useEffect can be tricky, so it's important to understand when and how to use it properly. You should consider the following: when you use useEffect, the function you pass to it will be called after the render is committed to the screen, which can lead to unexpected behavior if you're not careful. Additionally, useEffect runs after every render, so you should be mindful of how often you're running the effect and the impact it could have on performance. Finally, you should be aware of any potential side-effects that could be caused by your useEffect code."

Looks good!
You can ask anything to AI.

Check it out at: https://ai-code-reviewer.com
Youtube DEMO: https://youtu.be/V7K4IZV_k-E

How this application is structured

It consists of the following

Next.js
Chakra UI
TypeScript
OpenAI
Vercel

I started out using React, but switched to Next.js after reading the following section of the documentation.

Important note: When using Javascript, all API calls should be made on the server-side only, since making calls in client-side browser code will expose your API key. See here for more details.

As for OpenAI, I was able to set it up easily because the official site shows us a sample repository of setups written in Next.js.

https://github.com/openai/openai-quickstart-node

All you have to do is create an API using Next.js API Routes and hit it, as shown in the official sample code below.

import { Configuration, OpenAIApi } from "openai";

const configuration = new Configuration({
  apiKey: process.env.OPENAI_API_KEY,
});
const openai = new OpenAIApi(configuration);

export default async function (req, res) {
  const completion = await openai.createCompletion({
    model: "text-davinci-002",
    prompt: generatePrompt(req.body.animal),
    temperature: 0.6,
  });
  res.status(200).json({ result: completion.data.choices[0].text });
}

function generatePrompt(animal) {
  const capitalizedAnimal =
    animal[0].toUpperCase() + animal.slice(1).toLowerCase();
  return `Suggest three names for an animal that is a superhero.
Animal: Cat
Names: Captain Sharpclaw, Agent Fluffball, The Incredible Feline
Animal: Dog
Names: Ruff the Protector, Wonder Canine, Sir Barks-a-Lot
Animal: ${capitalizedAnimal}
Names:`;
}

You can try it!

That's it!

Thanks for reading.
This is my first time to use OpenAI, and I am happy to develop this app :D

Please send me a message if you need.

yuiko.dev@gmail.com
https://twitter.com/toffy_dev

If you think this article is useful, please buy me a coffee :)
https://www.buymeacoffee.com/yuikoito

Thank you!

Making eyes bigger in images with Amazon Rekognition and OpenCV

Yuiko Koyanagi — Tue, 28 Dec 2021 12:31:12 +0000

Hey guys, I have developed an application with Amazon Rekognition and OpenCV, that makes your eyes bigger in an image.

In this article, I will explain how to develop this application.

DEMO→https://eyedol.vercel.app/
github→https://github.com/yuikoito/eyedol

How to use

All you have to do is upload the image you want to make the eyes bigger and choose how big you want your eyes to be.
If you don't set it lower than "much" it will look very strange, so I don't recommend it.

If you choose "a little" or "some" you'll get a much better result.

Technology used

Frontend
- Next.js + TypeScript
- Chakra UI
Backend
- Python
- OpenCV
- Amazon Rekognition
Infrastructure
- AWS

The way to implement this is to use Amazon Rekognition to check where the eyes are, then use OpenCV to cut them out, expand them, composite them, and finally blur the boundaries.

So if you're looking sideways or wearing glasses, I don't think it will work very well.

This was the first time I used Amazon Rekognition, and it was easy to use.
However, the fee is much higher than I expected, so I added a Buy Me a Coffee link at the bottom right of the app.
link at the bottom right of the app.
If you can afford it, please buy me a coffee🙏.

I thought the same thing could be done with TensorFlow, so I'll try it with TensorFlow next time.

In order to make it as natural as possible, I adjusted the position of the eyes so that they were cropped as close to the edge as possible, and I also struggled a bit with the part where I blurred the border.

Implementation details

The implementation details are as follows

Recognize eyes with Amazon Rekognition
Crop the image using OpenCV, enlarge it, and composite it.
Blur the boundary to make it look natural after merging.

Recognize eyes with Amazon Rekognition

The code for detecting the location of the eyes is as follows.

import cv2
import boto3
from botocore.config import Config

config = Config(
   retries = {
      'max_attempts': 10,
      'mode': 'standard'
   }
)

def rekog_eye(im):
    #post amazon_rekognition
    client = boto3.client('rekognition','ap-northeast-1',config=config)
    result, buf = cv2.imencode('.jpg', im)

    #get face information by rekognition_image
    faces = client.detect_faces(Image={'Bytes':buf.tobytes()}, Attributes=['ALL'])

    leftEyeLeft = faces['FaceDetails'][0]['Landmarks'][11]
    leftEyeRight = faces['FaceDetails'][0]['Landmarks'][12]
    leftEyeUp = faces['FaceDetails'][0]['Landmarks'][13]
    leftEyeDown = faces['FaceDetails'][0]['Landmarks'][14]
    rightEyeLeft = faces['FaceDetails'][0]['Landmarks'][15]
    rightEyeRight = faces['FaceDetails'][0]['Landmarks'][16]
    rightEyeUp = faces['FaceDetails'][0]['Landmarks'][17]
    rightEyeDown = faces['FaceDetails'][0]['Landmarks'][18]
    EyeList = [leftEyeLeft, leftEyeRight, leftEyeUp, leftEyeDown, rightEyeLeft, rightEyeRight, rightEyeUp, rightEyeDown]

    EyePoints = {}
    h, w, ch = im.shape
    for eyepoint in EyeList:
        EyePoints[eyepoint['Type']] = {'X': int(eyepoint['X']*w), 'Y': int(eyepoint['Y']*h)}

    return EyePoints

I used boto3, a library that makes it easy to use various AWS services.
Then create a boto3 client and specify the service to be used (in this case, rekognition) and select the region.
In config, I specify how many times I want to try. In this case, the number is 10, so even if Rekognition doesn't work properly on the first try due to server error or network error, it will try again on the second try, and so on up to 10 times.

client = boto3.client('rekognition','ap-northeast-1',config=config)

Then, post the image byte sequence to the detection_faces of rekognition to detect the face.

faces = client.detect_faces(Image={'Bytes':buf.tobytes()}, Attributes=['ALL'])

Name the necessary landmarks in a way that makes them easy to use.
For example, the left edge of the left eye is called leftEyeLeft, and the right edge of the left eye is called leftEyeRight here.
Since I want to cut and spread the image at the very edge, I detect four points for each eye.

The detected locations are put into an array for easier use.

EyeList = [leftEyeLeft, leftEyeRight, leftEyeUp, leftEyeDown, rightEyeLeft, rightEyeRight, rightEyeUp, rightEyeDown]

Finally return them.

    EyePoints = {}
    h, w, ch = im.shape
    for eyepoint in EyeList:
        EyePoints[eyepoint['Type']] = {'X': int(eyepoint['X']*w), 'Y': int(eyepoint['Y']*h)}

Crop the image using OpenCV, enlarge it, and composite it

First, prepare functions to convert images from base64 to cv2 and vice versa.

import numpy as np
import base64

def base64_to_cv2(image_base64):
    """base64 image to cv2"""
    image_bytes = base64.b64decode(image_base64)
    np_array = np.fromstring(image_bytes, np.uint8)
    image_cv2 = cv2.imdecode(np_array, cv2.IMREAD_COLOR)
    return image_cv2


def cv2_to_base64(image_cv2):
    """cv2 image to base64"""
    image_bytes = cv2.imencode('.jpg', image_cv2)[1].tostring()
    image_base64 = base64.b64encode(image_bytes).decode()
    return image_base64

The code for cutting and merging is as follows.

def handler(event, context):
    ##############Extracting the request body from the request payload############
    # convert event['body'] to dict
    try:
        base_64ed_image = event.get('myimg', 'none')
        magnification = float(event.get('magni', 1.4)
        blur_num = int(event.get('blur', 3)) ##for blur
        im = base64_to_cv2(base_64ed_image)
        EyePoints = rekog_eye(im)
        bityouseix = 20
        bityouseiy = 5
        leftTop = min([EyePoints['leftEyeUp']['Y'], EyePoints['leftEyeDown']['Y'], EyePoints['leftEyeRight']['Y'], EyePoints['leftEyeLeft']['Y']])
        leftBottom = max([EyePoints['leftEyeUp']['Y'], EyePoints['leftEyeDown']['Y'], EyePoints['leftEyeRight']['Y'], EyePoints['leftEyeLeft']['Y']])
        leftRight = max([EyePoints['leftEyeUp']['X'], EyePoints['leftEyeDown']['X'], EyePoints['leftEyeRight']['X'], EyePoints['leftEyeLeft']['X']])
        leftLeft = min([EyePoints['leftEyeUp']['X'], EyePoints['leftEyeDown']['X'], EyePoints['leftEyeRight']['X'], EyePoints['leftEyeLeft']['X']])
        rightTop = min([EyePoints['rightEyeUp']['Y'], EyePoints['rightEyeDown']['Y'], EyePoints['rightEyeRight']['Y'], EyePoints['rightEyeLeft']['Y']])
        rightBottom = max([EyePoints['rightEyeUp']['Y'], EyePoints['rightEyeDown']['Y'], EyePoints['rightEyeRight']['Y'], EyePoints['rightEyeLeft']['Y']])
        rightRight = max([EyePoints['rightEyeUp']['X'], EyePoints['rightEyeDown']['X'], EyePoints['rightEyeRight']['X'], EyePoints['rightEyeLeft']['X']])
        rightLeft = min([EyePoints['rightEyeUp']['X'], EyePoints['rightEyeDown']['X'], EyePoints['rightEyeRight']['X'], EyePoints['rightEyeLeft']['X']])

        leftEye = im[leftTop:leftBottom+bityouseiy, leftLeft-bityouseix:leftRight+bityouseix]
        leftEye = cv2.resize(leftEye, (leftEye.shape[1], int(leftEye.shape[0]*magnification)))
        rightEye = im[rightTop:rightBottom+bityouseiy, rightLeft-bityouseix:rightRight+bityouseix]
        rightEye = cv2.resize(rightEye, (rightEye.shape[1], int(rightEye.shape[0]*magnification)))

        im[leftTop:leftTop+leftEye.shape[0], leftLeft-bityouseix:leftLeft+leftEye.shape[1]-bityouseix] = leftEye
        im[rightTop:rightTop+rightEye.shape[0], rightLeft-bityouseix:rightLeft+rightEye.shape[1]-bityouseix] = rightEye


        return {'status':200, 'message':'OK', 'img': cv2_to_base64(im)}
    except Exception as e:
        return {'status':500, 'message':str(e)}

In the following section, the coordinates for cropping are calculated again.
The Y-coordinate of EyeUp is not directly used as the Top because it takes into account the case where the eye is slanted.
(However, since it is just stretched vertically after cropping, it will be distorted.)

So, if necessary, I think it would be more natural to stretch the image diagonally when the right side is the maximum value, but since this is a crappy application, I didn't go this far.

        leftTop = min([EyePoints['leftEyeUp']['Y'], EyePoints['leftEyeDown']['Y'], EyePoints['leftEyeRight']['Y'], EyePoints['leftEyeLeft']['Y']])
        leftBottom = max([EyePoints['leftEyeUp']['Y'], EyePoints['leftEyeDown']['Y'], EyePoints['leftEyeRight']['Y'], EyePoints['leftEyeLeft']['Y']])
        leftRight = max([EyePoints['leftEyeUp']['X'], EyePoints['leftEyeDown']['X'], EyePoints['leftEyeRight']['X'], EyePoints['leftEyeLeft']['X']])
        leftLeft = min([EyePoints['leftEyeUp']['X'], EyePoints['leftEyeDown']['X'], EyePoints['leftEyeRight']['X'], EyePoints['leftEyeLeft']['X']])
        rightTop = min([EyePoints['rightEyeUp']['Y'], EyePoints['rightEyeDown']['Y'], EyePoints['rightEyeRight']['Y'], EyePoints['rightEyeLeft']['Y']])
        rightBottom = max([EyePoints['rightEyeUp']['Y'], EyePoints['rightEyeDown']['Y'], EyePoints['rightEyeRight']['Y'], EyePoints['rightEyeLeft']['Y']])
        rightRight = max([EyePoints['rightEyeUp']['X'], EyePoints['rightEyeDown']['X'], EyePoints['rightEyeRight']['X'], EyePoints['rightEyeLeft']['X']])
        rightLeft = min([EyePoints['rightEyeUp']['X'], EyePoints['rightEyeDown']['X'], EyePoints['rightEyeRight']['X'], EyePoints['rightEyeLeft']['X']])

Then cut it out according to its coordinates.

        leftEye = im[leftTop:leftBottom+bityouseiy, leftLeft-bityouseix:leftRight+bityouseix]
        rightEye = im[rightTop:rightBottom+bityouseiy, rightLeft-bityouseix:rightRight+bityouseix]

Resize the cropped left eye and right eye.

        leftEye = cv2.resize(leftEye, (leftEye.shape[1], int(leftEye.shape[0]*magnification)))
        rightEye = cv2.resize(rightEye, (rightEye.shape[1], int(rightEye.shape[0]*magnification)))

Then synthesize it.

        im[leftTop:leftTop+leftEye.shape[0], leftLeft-bityouseix:leftLeft+leftEye.shape[1]-bityouseix] = leftEye
        im[rightTop:rightTop+rightEye.shape[0], rightLeft-bityouseix:rightLeft+rightEye.shape[1]-bityouseix] = rightEye

Blur the boundary to make it look natural after merging

To blur the boundaries, I used OpenCV's Gaussian filter.

Reference:

http://labs.eecs.tottori-u.ac.jp/sd/Member/oyamada/OpenCV/html/py_tutorials/py_imgproc/py_filtering/py_filtering.html

Specify the blur area (image), kernel size, and standard deviation.

def mosaic_area(src, x, y, width, height, blur_num):
    dst = src.copy()
    for i in range(blur_num):
        dst[y:y + height, x:x + width] = cv2.GaussianBlur(dst[y:y + height, x:x + width], (3,3),3)
    return dst

By the way, I wanted to control how much blur is applied from the client side, so I used blur_num to specify how many times this blur should be applied.

Then, incorporate the above function in the handler function above.

            im = mosaic_area(im, leftLeft-bityouseix-int(bityouseix/2), leftTop, bityouseix, leftEye.shape[0]+bityouseiy, blur_num)
            im = mosaic_area(im, leftRight+int(bityouseix/2), leftTop, bityouseix, leftEye.shape[0]+bityouseiy, blur_num)
            im = mosaic_area(im, leftLeft-bityouseix, leftTop+leftEye.shape[0]-int(bityouseiy/2), leftEye.shape[1], bityouseiy, blur_num)
            im = mosaic_area(im, rightLeft-bityouseix-int(bityouseix/2), rightTop, bityouseix, rightEye.shape[0]+bityouseiy, blur_num)
            im = mosaic_area(im, rightRight+int(bityouseix/2), rightTop, bityouseix, rightEye.shape[0]+bityouseiy, blur_num)
            im = mosaic_area(im, rightLeft-bityouseix, rightTop+rightEye.shape[0]-int(bityouseiy/2), rightEye.shape[1], bityouseiy, blur_num)

In this case, there is no need to blur the upper part of the eye because nothing has been changed, but the left, right, and lower parts of the eye have been widened and composited so that the boundaries are visible. Since we are performing blurring at each boundary, we are using the blur filter 6 times above to blur the (left, right, bottom) x both eye parts.

After deploying this to Lambda, you can use APIGateway to make it into an API and you are done.
There is a little trick to using OpenCV in Lambda, so if you need it, please read this.

That's it!

Then finish!

Thanks for reading.
This is my first time to use Amazon Rekognition, and I am happy to make such a funny API :D

Please send me a message if you need.

yuiko.dev@gmail.com
https://twitter.com/yui_active

If you think this article is useful, please buy me a coffee :)
https://www.buymeacoffee.com/yuikoito

I made an app to enjoy Halloween🎃🦇🕸

Yuiko Koyanagi — Mon, 01 Nov 2021 19:00:22 +0000

Hey guys,

It was Halloween day, but in this self-restraint mode, it's hard to go out and enjoy dressing up in costumes.
So I made an app that lets you enjoy Halloween online!

Halloween has already passed, but you can even enjoy this app!

URL: https://ohmyface.ai/

How to use

The usage is as shown in the following slide.

First, when you visit to https://ohmyface.ai/, you can choose a favorite style from this week's popularity ranking or the overall ranking.

This ranking is based on the number of styles converted, and is updated daily at 9pm (JTC).

Once you have selected a style, all you have to do is just upload the image you want!
The loading process will run automatically and the image will be converted.

Don't worry, the uploaded images are not retained by the server.

Composition

The configuration is as follows.

Backend...python
Infrastructure...AWS (API gateway, Lambda, dynamo DB, s3)
Frontend...Next.js + TypeScript + Chakra UI

In order to develop the ranking, I created an API to post which style the image was converted in, along with a timestamp, then store the data in dynamo.

In order to aggregate the data, I create a Lambda function that returns the style data sorted by the number of conversions, and save the results in another table in dynamo.
That Lambda function is run once a day by EventBridge.

Then, I also created an API to access the dynamo table where the aggregate results are stored and retrieve the data, and call it from the client side to display the ranking order.

For the frontend, I used the usual Next.js + TypeScript, but this time I used Chakra UI for the first time.
It was very easy to use, and I'll use again for sure!

That's it!

Halloween is over, but I think it's an app that can be enjoyed even after Halloween, so I'd be happy if you could play it!

Contact

Please send me a message if you need.

yuiko.dev@gmail.com
https://twitter.com/toffy_dev

Released an app to train facial expressions: Multilingual in Next.js, dynamic OGP, facial expression recognition in face-api.js

Yuiko Koyanagi — Wed, 15 Sep 2021 14:39:33 +0000

Hi guys, I wrote Developed the app which trains your facial expressions: face-api.js + Next.js + TypeScript last week, and this week I updated the app, which you can train your face expressions and released it.

The main updates are as follows

Support for multiple languages
Added the ability to select the difficulty level
Dynamic OGP support
Allowing users to move on if they succeed
Overall design update

The entire code is below.

https://github.com/yuikoito/face-expression-challenge

URL: https://face-expression-challenge.vercel.app/

Enable to select the difficulty level.

I thought it would be undesirable to have the same difficulty level every time, so I made it possible to choose the difficulty level.

You can choose between easy, normal, hard, and devil.

In addition to changing the time for each difficulty, I also changed the threshold value to make the judgment itself tougher.

In the part where the expression is obtained, the threshold is used as shown below. (I used detectSingleFace from now on)

      const detectionsWithExpression = await faceapi
        .detectSingleFace(video, new faceapi.TinyFaceDetectorOptions())
        .withFaceExpressions();
      if (detectionsWithExpression) {
        const Array = Object.entries(detectionsWithExpression.expressions);
        const scoresArray = Array.map((i) => i[1]);
        const expressionsArray = Array.map((i) => i[0]);
        const max = Math.max.apply(null, scoresArray);
        const index = scoresArray.findIndex((score) => score === max);
        const expression = expressionsArray[index];
        if (
          expression === subject &&
          // Don't make a decision unless it's above the specified threshold here.
          Array[index][1] >= levelConfig[level].threshold
        ) {
          clearInterval(intervalHandler);
          setIsMatch(true);
          setStage("result");
        }
      }

When succeeded, move on to next subject.

Originally, I gave the subject for 1.5 seconds and then judged it after 1.5 seconds, but I decided to make users move on to next subject when succeeded.

So, after 1.5 seconds of presentation, I added a 3-second time limit, and if the facial expressions matched well within that time, you can move on to the next topic.

Even if no match is found, if no match can be made within 3 seconds, the next step is taken.

Dynamic OGP Support

The results are now dynamic OGP, which makes it easier to understand when they are shared.

It's a simple structure with a background image and text.
To display the background image, I imported and used loadImage from within canvas.

  const backgroundImage = await loadImage(
    path.resolve("./images/background.jpg")
  );
  ctx.drawImage(backgroundImage, 0, 0, WIDTH, HEIGHT);

Multilingual support

Next.js has i18n built-in since v10, so it can be multilingualized without importing anything.

The dictionary files are prepared as a TypeScript file, and the files called useTranslate.ts in which it is determined which one to load depending on the language.

import { useRouter } from "next/router";
import { JaTexts } from "../locales/ja";
import { EnTexts } from "../locales/en";

const useTranlate = () => {
  const { locale } = useRouter();
  return locale === "ja" ? JaTexts : EnTexts;
};

export default useTranlate;

Then don't forget to set the following settings in next.config.js.

  i18n: {
    locales: ["en", "ja"],
    defaultLocale: "en",
  },

This is the first time I noticed that only the top page is automatically rolled back to the file in that language...

I wanted to make the OGP part multilingual as well, depending on whether the share URL is in English or Japanese, so I made it so that when you share, the locale parameter is left in the share URL. (If the URL is in Japanese, the share URL will be /ja/share...) )

After that, the overall design was improved and released.

That's it!

This article is the 17th week of trying to write at least one article every week.

If you'd like, please take a look at my previous weekly posts!
See you soon!

🍎🍎🍎🍎🍎🍎

Please send me a message if you need.

yuiko.dev@gmail.com
https://twitter.com/yui_active

🍎🍎🍎🍎🍎🍎

Developed the app which trains your facial expressions: face-api.js + Next.js + TypeScript

Yuiko Koyanagi — Tue, 07 Sep 2021 01:28:38 +0000

Hey guys,
I developed the app which trains your facial expressions.

Do you ever lose your expression when you are alone at home?
I have lost mine　lol.

So I felt a sense of crisis about it, so I made an app that can help you train your facial expressions.

Let's try now.

DEMO: https://face-expression-challenge.vercel.app/
github: https://github.com/yuikoito/face-expression-challenge

I have only added minimal functionality at this time, so the content of the code may change drastically in the future.

Functions

The functions are simple.
When you press the start button, a countdown starts, and then a question appears.
The subject is given randomly, and the judging starts 1.5 seconds after showing the subject. Then the judging is done based on the facial expressions you make during that time.

The expressions to be judged are the following seven patterns, as described in Recognize facial expressions and change face to Emoji using face-api.js with Next.js+TypeScript

I'll omit how to introduce the face-api.js model, as it is still written in the above article.

Check if the subject and the facial expression match

In order to check whether the subject and the facial expression match, I put the subject as an argument into the function that recognizes facial expressions in the video (faceDetectHandler).
Then, the facial expression is detected every 0.1 second.

  const faceDetectHandler = (subject: string) => {
    const video = webcamRef.current.video;
    const intervalHandler = setInterval(async () => {
      const detectionsWithExpressions = await faceapi
        .detectAllFaces(video, new faceapi.TinyFaceDetectorOptions())
        .withFaceExpressions();
      if (detectionsWithExpressions.length > 0) {
        detectionsWithExpressions.map((detectionsWithExpression) => {
          const Array = Object.entries(detectionsWithExpression.expressions);
          const scoresArray = Array.map((i) => i[1]);
          const expressionsArray = Array.map((i) => i[0]);
          const max = Math.max.apply(null, scoresArray);
          const index = scoresArray.findIndex((score) => score === max);
          const expression = expressionsArray[index];
          if (expression === subject) {
            setIsMatch(true);
          }
        });
      }
    }, 100);
    setIntervalHandler(intervalHandler);
  };

Then, if the expression and the subject match, setIsMatch(true) to set the flag to true.

However, now isMatch will always be true once it is set to true, so I reset it back to false when I call faceDetectHandler.
The calling part looks like the following.

  const drawSubject = (expression: string) => {
    // Count how many times to have drawn the subject in order to stop the process when the game count reaches 5 times.
    setGameCount((gameCount) => gameCount + 1);
    // Reset to false
    setIsMatch(false);
    faceDetectHandler(expression);
    const canvas = canvasRef.current;
    // Import subject image and show it
    const ctx = canvas.getContext("2d");
    const image = document.createElement("img");
    image.onload = () => {
      coverCanvas(ctx, canvas);
      ctx.drawImage(
        image,
        (canvas.width - 300) / 2,
        (canvas.height - 300) / 2,
        300,
        300
      );
    };
    image.src = `/emojis/${expression}.png`;
    // If the faceDetectHandler already moved before, the last interval should be stopped.
    if (intervalHandler) {
      clearInterval(intervalHandler);
    }
    // Do clearRect 1.5s after showing the subject.
    setTimeout(() => {
      ctx.clearRect(0, 0, canvas.width, canvas.height);
    }, 1500);
    // Then, judge 1.5s after it
    setTimeout(() => {
      setStage("judge");
    }, 3000);
  };

As for the flow of starting the game, submitting a subject, making a judge, and ending the game, the state management is managed by stage in a messy way.

  const [stage, setStage] = useState<
    "isNotStart" | "ready" | "start" | "judge" | "finish"
  >("isNotStart");

  useEffect(() => {
    const canvas = canvasRef.current;
    const ctx = canvas.getContext("2d");
    if (stage === "judge") {
      judge(ctx, canvas);
    }
    if (stage === "start") {
      const expression =
        ExpressionTypes[Math.floor(Math.random() * ExpressionTypes.length)];
      drawSubject(expression);
    }
    if (stage === "finish") {
      setTimeout(() => {
        coverCanvas(ctx, canvas);
        drawText(ctx, canvas, `${point}／5`);
      }, 1500);
    }
  }, [stage]);

Next functions I will develop in the future

There are a few problems that I am planning to fix before the official release.

First, the facial expression recognition runs immediately after the subject is given, so in terms of UI, the subject is given (1.5 seconds) -> judgment starts (1.5 seconds), but in reality, the judgment starts when the subject is given, so it feels like you should make that expression within 3 seconds. That's not good.

Also, the judging of facial expressions is pretty lax.
This is because the method of judging expressions is based on the highest score among the current expressions, as shown below.

          const Array = Object.entries(detectionsWithExpression.expressions);
          const scoresArray = Array.map((i) => i[1]);
          const expressionsArray = Array.map((i) => i[0]);
          const max = Math.max.apply(null, scoresArray);
          const index = scoresArray.findIndex((score) => score === max);

In face-api.js, you can get the scores of all the expressions as shown below, and if the highest score (neutral in this case) is close to 1, as in the example below, it's fine, but if all the scores are between 0.1 and 0.2, I think it's too miscellany to assume that it's that expression.

{
  angry: 0.00012402892753016204
  disgusted: 0.00000494607138534775
  fearful: 2.4963259193100384e-7
  happy: 0.00011926032311748713
  neutral: 0.9996343851089478
  sad: 0.00010264792217640206
  surprised: 0.000014418363207369111
}

So, I'm thinking of adding a rule that says if the estimated expression is not more than 0.5, it won't be judged, and see how it goes.

I'm also planning to add a share function, dynamic OGP, and some kind of message at the final judgment.

I'm hoping to release it sometime next week, so please play with it after the release if you like!

That's it!

This article is the 16th week of trying to write at least one article every week.

If you'd like, please take a look at my previous weekly posts!
See you soon!

🍎🍎🍎🍎🍎🍎

Please send me a message if you need.

yuiko.dev@gmail.com
https://twitter.com/toffy_dev

🍎🍎🍎🍎🍎🍎

Try AR: Using AR.js to Display a wolf

Yuiko Koyanagi — Tue, 31 Aug 2021 12:50:55 +0000

Hey guys! I tried AR!

In this article, I will explain how to develop this application.

DEMO: https://yuikoito.github.io/ar-sample/
github: https://github.com/yuikoito/ar-sample

Setup

I read the following document in order to create it.

https://ar-js-org.github.io/AR.js-Docs/

There are three main types of AR.js.

Image Tracking. Recognizes a specific image and displays a model on top of it.
Location Based AR. Display a model at a specific location.
Marker Tracking. Displays the model at a specific marker location.

The type I use this time is Marker Tracking.

Depending on which type you use, there are slight differences in what you need to import.
Since I use Marker Tracking now, I only need to import the following.

  <script src="https://aframe.io/releases/1.0.4/aframe.min.js"></script>
  <script src="https://raw.githack.com/AR-js-org/AR.js/master/aframe/build/aframe-ar.js"></script>

Now it looks like I can use AR. Easy!

Use an original marker

AR.js has a pre-set marker called hiro and if you want to use, just write as the follows.

<a-marker preset="hiro">

In this case, I wanted to use my own marker, so I visited to this site to create one.

You can change the image whatever you want, but now I use this one.

Set the downloaded marker under the assets folder.

<a-marker preset="custom" type="pattern" url="assets/pattern-marker.patt">
  // Set the model to be displayed in this.
</a-marker>

Display an original 3D model

Even if you don't have your own model, you can use a-text to display text, or a-box to display a box, so it will be fun enough to just play with the pre-prepared ones.

This time, I downloaded a nice wolf model from Free3D, so I'll display it.

This is gltf format model, so I'll specify the id as follows.

<a-entity gltf-model="#wolf"></a-entity>

The model should be loaded in a-assets beforehand.

      <a-assets>
        <a-asset-item
          id="wolf"
          src="assets/Wolf-Blender-2.82a.glb"
        ></a-asset-item>
      </a-assets>

Now I can display it. Very easy.

The whole code.

  <body>
    <a-scene
      embedded
      arjs="sourceType: webcam; debugUIEnabled: false; detectionMode: mono_and_matrix; matrixCodeType: 3x3;"
    >
      <a-assets>
        <a-asset-item
          id="wolf"
          src="assets/Wolf-Blender-2.82a.glb"
        ></a-asset-item>
      </a-assets>

      <a-marker preset="custom" type="pattern" url="assets/pattern-marker.patt">
        <a-entity
          id="model"
          gltf-model="#wolf"
          position="0 0 1"
          scale="1 1 1"
          rotation="0 -90 60"
        >
        </a-entity>
      </a-marker>
      <a-entity camera></a-entity>
    </a-scene>
  </body>

That's it!

Thanks for reading.

This is my first time to use AR, but it was really easy.
I will try more :D

🍎🍎🍎🍎🍎🍎

Please send me a message if you need.

yuiko.dev@gmail.com
https://twitter.com/yui_active

🍎🍎🍎🍎🍎🍎

Create a coloring book in canvas: Developed an app that you can create and enjoy your own original coloring from an image.

Yuiko Koyanagi — Mon, 23 Aug 2021 18:26:30 +0000

Hey guys,

I developed an app that you can create and enjoy your own original coloring book from an image.

Look this!

You can also play coloring in this app.

The part about turning the image into a coloring book is almost the same as this article I wrote before.

So, in this article, I will explain how to develop this coloring functions using canvas.

DEMO→https://nurie-maker.com/
github→https://github.com/yuikoito/nurie-creator

※I'm using Nuxt.js here. Please replace it as necessary.

Draw a background image on the canvas

First, we need to prepare a canvas element and draw a background image on the canvas.

It takes a little time to draw the background image on the canvas, and I want to make sure to emit loading during that time.
Therefore, I wait here for the image to be fully loaded in Promise.

<script>
export default {
  async asyncData({ params }) {
    return {
      url: `${process.env.BASE_URL}/nurie/${params.id}`,
      image: `${process.env.AWS_IMAGE_URL}/nurie/${params.id}.jpg`,
      twitterImage: `${process.env.AWS_IMAGE_URL}/nurie/${params.id}.jpg`,
    }
  },
  data() {
    return {
      canvas: null,
      ctx: null,
      noPicture: require('@/assets/img/noPic.png'),
      overlay: true,
    }
  },
  mounted() {
    this.init()
  },
  methods: {
    async init() {
      this.canvas = this.$refs.canvas
      this.ctx = this.canvas.getContext('2d')
      const wrapper = this.$refs.wrapper
      await this.loadImage(this.image).then((res) => {
        const scale = wrapper.clientWidth / res.naturalWidth
        this.canvas.width = res.naturalWidth * scale
        this.canvas.height = res.naturalHeight * scale
        this.ctx.drawImage(res, 0, 0, this.canvas.width, this.canvas.height)
      })
      this.overlay = false
    },
    loadImage(src) {
      return new Promise((resolve) => {
        const img = new Image()
        img.src = src
        img.onload = () => resolve(img)
        img.onerror = () => {
          img.src = this.noPicture
        }
      })
    },
  }
}
</script>

Now we can draw an image on canvas.

Use Retina Canvas Support

If you look at the image drawn above, you will see that the resolution is a little rough, so we will fix that part.

Specifically, when loading an image in canvas, load it at twice the size and set the canvas style to 1/2. The actual canvas drawing size will be the same as above, but the resolution will be finer and the image will be drawn beautifully.

      await this.loadImage(this.image).then((res) => {
        this.ctx.scale(2, 2)
        const scale = wrapper.clientWidth / res.naturalWidth
        this.canvas.width = res.naturalWidth * scale * 2
        this.canvas.height = res.naturalHeight * scale * 2
        this.canvas.style.width = this.canvas.width / 2 + 'px'
        this.canvas.style.height = this.canvas.height / 2 + 'px'
        this.ctx.drawImage(res, 0, 0, this.canvas.width, this.canvas.height)
      })

Implement a drawing function.

When a user clicks on the screen, the event can be detected by mousedown on a PC, mousemove for dragging, and mouseup (or mouseout) for the timing when the dragging is finished.

<canvas
  ref="canvas"
  @mousedown.prevent="dragStart"
  @mouseup.prevent="dragEnd"
  @mouseout.prevent="dragEnd"
  @mousemove.prevent="draw"
>
</canvas>

※The lineWidth and currentColor can be changed by the user.

    dragStart() {
      this.ctx.beginPath()
      this.isDrag = true
    },
    draw(e) {
      const x = (e.clientX - this.canvas.getBoundingClientRect().left) * 2
      const y = (e.clientY - this.canvas.getBoundingClientRect().top) * 2
      if (!this.isDrag) {
        return
      }
      this.ctx.lineCap = 'round'
      this.ctx.lineJoin = 'round'
      this.ctx.lineWidth = this.lineWidth
      this.ctx.strokeStyle = this.currentColor
      if (this.lastPosition.x === null || this.lastPosition.y === null) {
        this.ctx.moveTo(x, y)
      } else {
        this.ctx.moveTo(this.lastPosition.x, this.lastPosition.y)
      }
      this.ctx.lineTo(x, y)
      this.ctx.stroke()
      this.lastPosition.x = x
      this.lastPosition.y = y
    },
    dragEnd() {
      this.ctx.closePath()
      this.isDrag = false
      this.lastPosition.x = null
      this.lastPosition.y = null
      this.isErase = false
    },

Change color of the line

I used vue-color to change color.

yarn add vue-color

Create vueColor.js under plugins.
In this case, we are using a design called Sketch, so we are only loading Sketch.

import Vue from 'vue'
import { Sketch } from 'vue-color'

Vue.component('Sketch', Sketch)

Add in nuxt.config.js.

  plugins: [{ src: '@/plugins/vueColor.js', mode: 'client' }],

Then draw this component in the page.
The drawing is fine as it is, but you will get a warning, so move it inside <client-only>.

  <client-only>
    <Sketch :value="colors" @input="updateValue"></Sketch>
  </client-only>

Set a default value for colors.
When a user change color, you will get the color by the following function.

    updateValue(e) {
      this.currentColor = e.hex
    },

This library is easy and useful!

Download the canvas

Then, we will implement a function to download the canvas when the button is clicked.

    download() {
      let link = document.createElement('a')
      link.href = this.canvas.toDataURL('image/jpeg')
      link.download = 'nurie-' + new Date().getTime() + '.jpg'
      link.click()
    },

... However, we can't download canvas at this point.

Execution of toDataURL fails due to CORS.

In fact, this time, for the background image of canvas, I was referring to the URL stored in s3 from uuid, so the error occurred due to CORS in that part.

image: `${process.env.AWS_IMAGE_URL}/nurie/${params.id}.jpg`

Since the image is loaded as shown above, the domain of the current URL and the domain of the image URL are different, causing the following error.

tainted canvases may not be exported

Then I used @nuxtjs/proxy to fix this problem.

yarn add @nuxtjs/proxy

Add nuxt.config.js.

  modules: ['@nuxtjs/dotenv', '@nuxtjs/proxy'],
  proxy: {
    '/nurie/': {
      target: AWS_IMAGE_URL,
      changeOrigin: true,
      secure: false,
    },
  },

Now, you can replace ${process.env.AWS_IMAGE_URL}/nurie/${params.id}.jpg with the URL of your site.
Change the image part to the following.

image: `${process.env.BASE_URL}/nurie/${params.id}.jpg`,

Enable to play coloring on the phone

As it is, we can't do coloring on the phone, so we will also add events for the phone.
On the phone, you can detect events with touchstart, touchmove, and touchend.

<canvas
  ref="canvas"
  @mousedown.prevent="dragStart"
  @touchstart.prevent="dragStart"
  @touchend.prevent="dragEnd"
  @mouseup.prevent="dragEnd"
  @mouseout.prevent="dragEnd"
  @mousemove.prevent="draw"
  @touchmove.prevent="spDraw"
>
</canvas>

The events that can be detected on a smartphone differ from those on a PC only for the part where the line is drawn, so adjust the following.
The point is that on a PC you can get the location of the touch by e.clientX, etc., but on a smartphone it is e.changedTouches[0].clientX. (In the case of a single finger)

    spDraw(e) {
      if (e.changedTouches.length == 1) {
        this.draw(e.changedTouches[0])
      }
    },

Finish!

That's it!

Thanks for reading.
I am very happy if you enjoy it!

🍎🍎🍎🍎🍎🍎

Please send me a message if you need.

yuiko.dev@gmail.com
https://twitter.com/toffy_dev

🍎🍎🍎🍎🍎🍎

Look back on these three months: I release an app and write an article every week

Yuiko Koyanagi — Mon, 16 Aug 2021 05:16:45 +0000

Hey guys,

I've been challenging myself #tryAtleast1Article1week to release an app and write an article in both English and Japanese per week since the latter half of May.

I'm planning to continue this challenge for a little while longer, but anyway, I've completed this challenge for three months, so I would like to look back!

Week 1: React + TypeScript: Face detection with Tensorflow

There were not many examples of using TensorFlow.js with React and TypeScript, so I got stuck at first.
But now I know how to run it with TypeScript, so actually this was good choice for me to use TensorFlow.js with TypeScript.

https://dev.to/yuikoito/face-detection-by-using-tensorflow-react-typescript-3dn5

This was the first time I wrote an article in English, and I was happy to see that it became the 4th most trending article.

https://dormoshe.io/newsletters/ag/typescript/42?utm_source=twitter&utm_campaign=twitter

Week 2: UI Components website Released!

https://dev.to/yuikoito/ui-components-website-released-2g8e

I'm not very good at animations, so I've been keeping all the animations I used in a notepad so I can copy and paste them, but I thought it was getting a bit hard to see, so I decided to put them together.

Pretty convenient, isn't it?

Week 3: I made 18 UI components for all developers

I added 18 components on UI Components, which I release last week.

Week 4: Image Transformation: Convert pictures to add styles from famous paintings

I made this app because I was interested in image processing.

I was happy to see that someone wrote an article about it.

https://momosiri.nagoya/33835

Week 5: Developed an app to transcribe and translate from images

I developed this app because there are only a few applications to translate from pictures.

https://dev.to/yuikoito/developed-an-app-to-transcribe-and-translate-from-images-32gp

There are many apps to translate but for almost all of apps, we have to choose which language we want to translate.
So, I thought it was good to translate automatically to user language.

I think it's more useful.

Week 6: Generate Open Graph images with Next.js and TypeScript on Vercel

Actually this is even not an app, but I wanted to try open graph images with Next.js and TypeScript.

https://dev.to/yuikoito/generate-open-graph-images-with-next-js-and-typescript-on-vercel-110j

Week 7: OpenCV in Lambda: Created an API to convert images to Pokémon ASCII art style using AWS and OpenCV

I was looking at Pokémon images, and I thought it would be fun to use this to convert images. Then I developed.

https://dev.to/yuikoito/opencv-in-lambda-created-an-api-to-convert-images-to-pokemon-ascii-art-style-using-aws-and-opencv-287l

Using OpenCV in Lambda is the part that I struggled with.

Week 8: Developed a web app to convert images to Pokémon ASCII art style

I made a web app using the API I made last week.

https://dev.to/yuikoito/developed-a-web-app-to-convert-images-to-pokemon-ascii-art-style-mb7

Week 9: Using OpenCV: Developed a web app to convert images to manga style

https://dev.to/yuikoito/using-opencv-developed-a-web-app-to-convert-images-to-manga-style-23p0

I think it's pretty fun, so please play with it if you like.

If you want to replace the background, you need to use an image without anything in the background, because it does not extract people.

I'm almost new to image processing, but I'm getting used to it.

Week 10: TensorFlow + Next.js + TypeScript: Remove background and add virtual background image with web camera

I just changed the background, but to my surprise, it became the first place for trending! Yey.

https://dormoshe.io/newsletters/ag/typescript/50?utm_source=twitter&utm_campaign=twitter

Also, this article triggered three inquiries.

I struggled quite a bit when I ran TensorFlow.js with React+TypeScript in the first week, but this time I didn't struggle that much and was able to do it easily, so I was happy to feel my growth a little bit.

Week 11: Use Pose detection of TensorFlow with Next.js and TypeScript: Let's become pictograms with Pose detection #Tokyo2020

For this challenge, my friend, Tommy contacted me and we decided to develop a pictogram application together.

https://dev.to/yuikoito/tensorflow-next-js-typescript-let-s-become-pictograms-with-pose-detection-tokyo2020-17ia

We developed for two days from morning to midnight, connected to Discord all the time.

It was much fun and I would love to do again.

Week 12: Recognize facial expressions and change face to Emoji using face-api.js with Next.js+TypeScript

I've been having a lot of online meetings lately, and it's a pain to have to put on makeup every time.
But it's also a pain to not be able to see my facial expressions, so I created an app that solves that problem.

https://dev.to/yuikoito/recognize-facial-expressions-and-change-face-to-emoji-using-face-api-js-with-next-js-typescript-21n9

If there was a demand for it, I was going to make it a Chrome extension so that I could use it in actual meetings.
What do you think? :D

That's it!

It was really hard for me to release and write an article every week. lol

Many times I wondered why am I doing this painstaking work, but sometimes it makes me happy to get a response, so I plan to continue for a little while longer.

It was my first challenge to write an article in English, but I'm happy to see that I'm getting some inquiries.

Thank you for reading. See you next week!

Contact

Please send me a message if you want to offer a job or ask me something.

yuiko.dev@gmail.com
https://twitter.com/toffy_dev

Recognize facial expressions and change face to Emoji using face-api.js with Next.js+TypeScript

Yuiko Koyanagi — Mon, 09 Aug 2021 09:16:15 +0000

Hey guys,

I have developed an application with face-api.js, that recognize facial expressions and change the face to Emoji realtime!

In this article, I will explain how to develop this application.

DEMO→https://face2emoji.vercel.app/
github→https://github.com/yuikoito/face2emoji

Why I developed

I've been having more and more online meetings lately, and there are times when I don't want to show my face, but I want to convey my expression.

I personally don't want to show my face, but, of course I understand facial expressions tell many things.

Then, I was wondering if it would be great if I could show only my facial expressions! lol

You can't use this application as a real web conference, but if enough demand, I will make it a Chrome extension so that it can be used for actual web conferences.

Now, just try how it works and tell me your thoughts :)

App Structure and Function

I'm using Next.js + TypeScript to create the application.
The facial expression recognition is done using face-api.js, a module based on TensorFlow.js.
And the hosting is vercel.

The function is simple: when the app is launched, it recognizes the face and facial expression, and places an emoji on your face.
The emoji used are apple emoji.

With face-api.js, 7 patterns of facial expressions can be obtained, and for each of these, the following emojis are set for each expression.

OK, now it's time to explain how to develop this application.

Setup and Install face-api.js

$ yarn create next-app <app-name>
$ cd <app-name>
$ touch tsconfig.json
$ yarn add --dev typescript @types/react

Then, rename index.js and _app.js to index.tsx, _app.tsx.

Install webcam.

$ yarn add react-webcam @types/react-webcam

Now we are ready to install face-api.js.

$ yarn add face-api.js

As stated in the README, face-api.js needs to load models.
So, copy the weights folder from face-api.js github and place it under public.
Rename from weights to models.

Load the models.
Here, we are using tinyFaceDetector and faceExpressionNet.
With ssdMobilenetv1, the accuracy will be better, but it will be much heavier, so some devices will not work.

  const loadModels = async () => {
    const MODEL_URL = "/models";
    await Promise.all([
      faceapi.nets.tinyFaceDetector.load(MODEL_URL),
      faceapi.nets.faceExpressionNet.load(MODEL_URL),
    ]);
  };

Let's check if the logs are output correctly.
Try to recognize a face by clicking a button and see the log as follows.

// pages/index.tsx

import * as faceapi from "face-api.js";
import { useRef } from "react";
import Webcam from "react-webcam";

export default function Home() {
  const webcamRef = useRef<Webcam>(null);
  const canvasRef = useRef<HTMLCanvasElement>(null);
  const loadModels = async () => {
    const MODEL_URL = "/models";
    await Promise.all([
      faceapi.nets.tinyFaceDetector.load(MODEL_URL),
      faceapi.nets.faceExpressionNet.load(MODEL_URL),
    ]);
  };

  const faceDetectHandler = async () => {
    await loadModels();
    if (webcamRef.current && canvasRef.current) {
      const webcam = webcamRef.current.video as HTMLVideoElement;
      const canvas = canvasRef.current;
      webcam.width = webcam.videoWidth;
      webcam.height = webcam.videoHeight;
      canvas.width = webcam.videoWidth;
      canvas.height = webcam.videoHeight;
      const video = webcamRef.current.video;
      const detectionsWithExpressions = await faceapi
        .detectAllFaces(video, new faceapi.TinyFaceDetectorOptions())
        .withFaceExpressions();
      console.log(detectionsWithExpressions);
    }
  };

  return (
    <div className={styles.container}>
      <main className={styles.main}>
        <Webcam audio={false} ref={webcamRef} className={styles.video} />
        <canvas ref={canvasRef} className={styles.video} />
        <button onClick={faceDetectHandler}>顔認識</button>
      </main>
    </div>
  );
}

OK, seems good!

The detection part is the face recognition part and contains the coordinates, etc. The expressions part is the facial expressions and contains the score for each expression.

Now all we have to do is check the location of the face and draw an emoji based on the expression with the highest score.

Change emoji according to facial expressions

The drawing part is long, so I put the logic in a separate file.
DetectionsWithExpressions and canvas should be sent from index.tsx.

Change the expressions object to a form that is easy to calculate.

// expressions output
{
  angry: 0.00012402892753016204
  disgusted: 0.00000494607138534775
  fearful: 2.4963259193100384e-7
  happy: 0.00011926032311748713
  neutral: 0.9996343851089478
  sad: 0.00010264792217640206
  surprised: 0.000014418363207369111
}

Then we change like the following.

    const Array = Object.entries(detectionsWithExpression.expressions);
    const scoresArray = Array.map((i) => i[1]);
    const expressionsArray = Array.map((i) => i[0]);

The whole code is as follows.

// utils/drawEmoji.ts

import {
  WithFaceExpressions,
  FaceDetection,
  FaceExpressions,
} from "face-api.js";

export const drawEmoji = async (
  detectionsWithExpressions: WithFaceExpressions<{
    detection: FaceDetection;
    expressions: FaceExpressions;
  }>[],
  canvas: HTMLCanvasElement
) => {
  detectionsWithExpressions.map((detectionsWithExpression) => {
    const ctx = canvas.getContext("2d");
    const Array = Object.entries(detectionsWithExpression.expressions);
    const scoresArray = Array.map((i) => i[1]);
    const expressionsArray = Array.map((i) => i[0]);
    const max = Math.max.apply(null, scoresArray);
    const index = scoresArray.findIndex((score) => score === max);
    const expression = expressionsArray[index];
    const image = document.createElement("img");
    image.onload = () => {
      const width = detectionsWithExpression.detection.box.height * 1.2;
      const height = detectionsWithExpression.detection.box.height * 1.2;
      const x = detectionsWithExpression.detection.box.x - width * 0.1;
      const y = detectionsWithExpression.detection.box.y - height * 0.2;
      ctx.clearRect(0, 0, canvas.width, canvas.height);
      ctx.drawImage(image, x, y, width, height);
    };
    image.src = `/emojis/${expression}.png`;
  });
};

By the way, since multiple people can be detected this time, detectionsWithExpressions is set to multiple, but if you put ctx.clearRect(0, 0, canvas.width, canvas.height); in the above location, the map will flow in order and if there are multiple people, one person will be detected and drawn, and the others will not be drawn.

For this reason, if you want to support multiple people, you may need to prepare a temporary canvas element, draw all the users on it, and then draw them on the main canvas.

Here it is not intended for use by multiple people.

Anyway, now the logic part is done, and all you have to do is to call the above function in index.tsx.

Notes on deployment: Module not found: Can't resolve'fs' support

Since face-api.js is also available in node.js, some use fs. However, of course, fs is not required when running with a browser, so an error will occur at the time of deployment in this part.
Therefore, it is necessary to explicitly state that fs is not used when running in a browser.

This part was reported in the following issue.

https://github.com/justadudewhohacks/face-api.js/issues/154

Then, change next.config.js as follows.

// next.config.js
module.exports = {
  reactStrictMode: true,
  webpack: (config, { isServer }) => {
    if (!isServer) {
      config.resolve.fallback = {
        fs: false,
      };
    }
    return config;
  },
};

If need, read my github.

That's it!

This article is the 12th week of trying to write at least one article every week.

If you'd like, please take a look at my previous weekly posts!
See you soon!

Contact

Please send me a message if you want to offer a job or ask me something.