DEV Community: corridor project

How to Handle Challenges in Three.js App Development

corridor project — Sun, 27 Sep 2026 21:06:30 +0000

Hello, my name is pj-corridor, previously an advertising engineer and currently a data platform engineer. In this article, I will share insights gained through my hobby of developing Three.js applications, such as common beginner pitfalls, browser-compatibility issues, and their solutions.

Let's start by introducing some of my Three.js applications.

A New Concept for Reversi

You might wonder, Is Three.js necessary for Reversi ? But with a 3D board that loops in a cylindrical shape, you can enjoy strategies that were not possible before. Additionally, the introduction of the DMZ area and a variety of NPC options (1 to 3) are available. In the case of the loop board, you can also opt for rotation tactics. The rule-based NPC implementation (as of June 2025) might feel lacking in one-on-one battles, but even experienced players will find the chaotic four-player matches (NPC x3 + human) challenging. Feel free to try it while waiting for the train.

Diverse Puzzles

I developed various puzzles by expanding on the classic Three.js study subject, the Rubik's Cube. However, my collection still pales in comparison to Z3Cubing. In the future, I aim to explore directions that cannot be realized with physical gadgets (such as Cube with Anomalous Rotations).

Stick Figures

I often create document using a lot of simple illustrations in PowerPoint slides (This is a blog article written in Japanese about simple illustration), but finding copyright-free stick figure materials to paste on slides can be a hassle. So, I decided to develop my own. Feel free to use them in your slides as well.

Stick Figure Gallery

While the stick figures solved the issue of finding materials, even posing them became cumbersome :-p. So, I prepared structured pose data input/output and a gallery. You can find something close to your image and make slight adjustments to obtain the desired stick figure material.

Stick Figure Gallery

As the next step, I'm envisioning a stick figure that generates appropriate poses from natural language (e.g., words expressing emotions or postures) using machine learning by labeling pose data.

Now, let's delve into the insights gained through Three.js app development.

The Empty Draw Buffer

In Stick Figure, Rubber Figure, and Hand, I implemented a screenshot feature that copies the image of the determined pose to the clipboard. This feature uses toDataURL() on WebGLRenderer.domElement (reference), but initially, I struggled to capture the rendered image.

For example, in the following code :

const w = 400;
const h = 400;

const renderer = new THREE.WebGLRenderer();
document.body.appendChild(renderer.domElement);

renderer.setPixelRatio(window.devicePixelRatio);
renderer.setSize(w, h);

const camera = new THREE.PerspectiveCamera(45, w / h);
camera.position.set(0, 0, +1000);

const scene = new THREE.Scene();
const geometry = new THREE.BoxGeometry(50, 50, 50);
const material = new THREE.MeshNormalMaterial();
const box = new THREE.Mesh(geometry, material);
scene.add(box);

renderer.render(scene, camera);

// (1) capture the canvas image right after WebGLRenderer.render()
console.log(renderer.domElement.toDataURL('image/png'));

setTimeout(() => {
    // (2) capture the canvas image in a timer event handler
    console.log(renderer.domElement.toDataURL('image/png'));
}, 0);

button.addEventListener('click', in_ev => {
    // (3) capture the canvas image in a click event handler
    console.log(renderer.domElement.toDataURL('image/png'));
});

At the timing of (1), toDataURL() provides the expected output, but at (2) and (3), it doesn't work. This is because the WebGLRenderer automatically clears the draw buffer after rendering each frame. By trying to set WebGLRenderer.preserveDrawingBuffer (reference) to retain the draw buffer :

const renderer = new THREE.WebGLRenderer({preserveDrawingBuffer : true});

You can obtain the expected output even at the asynchronous timings of (2) and (3). However, according to the WebGL Specification :

While it is sometimes desirable to preserve the drawing buffer, it can cause significant performance loss on some platforms. Whenever possible this flag should remain false and other techniques used.

Given the non-normative note and past related bugs reported in WebKit, I decided not to change the default false for the drawing buffer setting and instead re-render just before toDataURL().

setTimeout(() => {
    // (2) capture the canvas image in a timer event handler
    renderer.render(scene, camera);
    console.log(renderer.domElement.toDataURL('image/png'));
}, 0);

button.addEventListener('click', in_ev => {
    // (3) capture the canvas image in a click event handler
    renderer.render(scene, camera);
    console.log(renderer.domElement.toDataURL('image/png'));
});

This way, I successfully implemented the screenshot feature. Please try pasting it into PowerPoint slides.

Three Three.js Raycasting Traps

In Three.js applications, raycasting is used to determine the intersection between the coordinates where touch or mouse events occur and objects.

Raycasting is used for mouse picking (working out what objects in the 3D space the mouse is over) amongst other things.

Here, I will introduce three traps (or failures) related to raycasting.

1. Intrusive AxesHelper

In Stick Figure and Rubik's Cube, raycasting from the coordinates where touchstart or mousedown events occur is used to determine if :

There is an intersection with an object in the scene :
- Drag the intersecting part
- Use touchmove or mousemove to manipulate the part (e.g., change the pose)
There is no intersection with an object in the scene :
- Drag from that coordinate
- Use touchmove or mousemove to rotate the object (actually, the object itself does not rotate; the PerspectiveCamera, which continuously look at the object, moves in the opposite direction of the touchmove or mousemove events)

This is the common UX. However, when I added an AxesHelper (three-colored lines representing axes) to the scene for debugging purposes, it sometimes behaved strangely.

The reason was that AxesHelper itself was a target for Raycaster.intersectObject() (reference). To address this, either consider intersections with it or exclude its influence before checking intersections, as shown below.

const children = scene.children.filter(in_child => !(in_child instanceof THREE.AxesHelper));
const intersects = raycaster.intersectObjects(children);

2. Ghost CircleGeometry

In Stick Figure, part manipulation works as follows :

Add an operation object to the scene when a part is dragged :
- SphereGeometry with the same radius as the height of the target part
- CircleGeometry that passes through the center of the SphereGeometry with its normal vector facing the PerspectiveCamera
Use raycasting from the coordinates where touchmove or mousemove events occur to determine the intersection direction with the operation object, and have the dragged part lookAt() (reference) it

Here's a demonstration with the operation object colored for debugging, showing how the stick figure's hand moves.

Sometimes, raycasting occurs from the opposite direction of the object in 3D space, and in that case the hit-test behaves unexpectedly. I found that SphereGeometry does not intersect with raycasting from the back. Since CircleGeometry always faces the PerspectiveCamera, this was a blind spot ! ^^;

In this case, you can use a material for both sides, for example :

const geometry = new THREE.CircleGeometry(100, 32);
const material = new THREE.MeshNormalMaterial({side : THREE.DoubleSide});
const circle = new THREE.Mesh(geometry, material);

3. Why Do Bent SkinnedMeshes Lack Vertices ?

In Rubber Figure and Hand, SkinnedMesh is used to smoothly bend parts. That worked nicely, except that bent parts no longer hit in ray-casting. While there was no information about this in the documentation, I wondered why.

After researching forums and the implementation, I understood that the actual vertex data is not changed, and the influence of bones is calculated and drawn. So, I created an ExtrudeGeometry (reference) using SkinnedMesh.skeleton.bones as rough alternative vertex data, and by dragging the intersection with raycasting, I was able to manipulate the rubber figure.

Browser Compatibility Issues with AdSense

Once the Three.js app was taking shape, I decided to try implementing AdSense. However, I encountered unexpected hurdles in coexisting Three.js apps with AdSense. Here, I will share the journey of resolving browser compatibility issues related to iframes.

During initialization and window resizing, Three.js apps require appropriate coordinate processing and settings changes for rendering :

Change PerspectiveCamera.aspect (reference)
Call PerspectiveCamera.updateProjectionMatrix() (reference)
Call WebGLRenderer.setSize() (reference)

Additionally, since AdSense code may automatically insert ads, potentially changing the size of other elements, similar processing is needed at that timing. For example, the offsetHeight of elements is changed.

However, since there are writes to the width and height of WebGLRenderer.domElement in the implementation, calling it within the callback of ResizeObserver feels a bit risky (incidentally, in the Chromium implementation, it checks for changes in element size from the previous observation, so it doesn't fall into an infinite loop).

Therefore, by placing WebGLRenderer.domElement inside an iframe :

Automatic ad insertion by AdSense
Resizing of the iframe due to the above
Event handler processing due to the above
- Resize width and height of WebGLRenderer.domElement
- Change settings for coordinate processing and rendering

I tried the following approach.

function createOuterWindow(in_document) {
    const iframe = in_document.createElement('iframe');
    Object.assign(iframe.style, {
        width: '100%',
        height: '100%',
        border: 'none'
    });
    in_document.body.appendChild(iframe);
    return iframe.contentWindow;
}

const outerWin = createOuterWindow(document);
const outerDoc = outerWin.document;

// myCanvas : WebGLRenderer.domElement
outerDoc.body.appendChild(myCanvas);

outerWin.addEventListener('resize', in_event => {
    // maintain PerspectiveCamera.aspect etc
    console.log('resized');
});

However, while it worked as expected in Chrome (137.0), WebGLRenderer.domElement did not display in Firefox (139.0). So, I tried changing the processing timing. According to the specification of iframe, iframe without src or srcdoc attributes load about:blank as a default, so :

If url matches about:blank and initialInsertion is true, then: Run the iframe load event steps given element.

How about this timing ?

const outerWin = createOuterWindow(document);

outerWin.addEventListener('load', () => {
    const outerDoc = outerWin.document;

    // myCanvas : WebGLRenderer.domElement
    outerDoc.body.appendChild(myCanvas);
});

This time, it worked as expected in Firefox, but the event did not execute in Chrome. Related discussions can be found in stop triggering navigations to about:blank on iframe insertion, but by delaying the processing to the next event loop, I achieved the expected behavior in both browsers, so I decided to leave it at that for now and document it.

const outerWin = createOuterWindow(document);

// asynchronous process for Firefox
setTimeout(() => {
    const outerDoc = outerWin.document;

    // myCanvas : WebGLRenderer.domElement
    outerDoc.body.appendChild(myCanvas);
}, 0);

With this, I finally managed to keep up with automatic ad insertion for coordinate processing and rendering settings changes.

My Super Special Duper Animation Function

Now that AdSense implementation is settled, I want to brush up on the overall UX. In my Three.js apps, WebGLRenderer rendering adopts animation expressions overall, but I want to adopt similar UX for rendering regular HTML elements (such as dialog displays) as well. However, I don't want to disperse animation-related descriptions like CSS @keyframes definitions. Here's an implementation I came up with to manage it simply and centrally with JavaScript code only.

function autoTransition1(in_elem, in_shorthand, in_start, in_end) {
    return new Promise(in_resolve => {
        let [prop,,, delay = '0s'] = in_shorthand.split(/\s+/);
        // convert from CSS to CSSOM
        prop = prop.replace(/-([a-z])/g, (in_match, in_letter) => in_letter.toUpperCase());
        delay = delay.includes('ms') ? parseFloat(delay) : parseFloat(delay) * 1000;
        in_elem.style['transition'] = in_shorthand;
        in_elem.style[prop] = in_start;
        // automatically start the transition in the next event loop
        setTimeout(() => in_elem.style[prop] = in_end, delay);
        const callback = in_ev => {
            if (in_ev.propertyName === prop) {
                in_elem.removeEventListener('transitionend', callback);
                (in_resolve)();
            }
        };
        in_elem.addEventListener('transitionend', callback);
    });
}

function autoTransition2(in_elem, in_shorthand, in_start, in_end) {
    (async () => await autoTransition1(in_elem, in_shorthand, in_start, in_end))();
}

autoTransition2(element, 'color 1.5s ease-out', 'blue', 'white');

By specifying the shorthand of CSS Transitions and the start and end values of transition-property, you can execute animations related to elements.

Conclusion

Thank you for reading this far. Although I didn't create headings for them, there were various minor failures as well. For example, many classes in Three.js have a clone() method implemented, but when I implemented a new class by inheriting a class, I struggled with suspicious behavior of cloned instances due to changing the constructor's interface, which is quite embarrassing.

I hope that the insights (or failures ?) I gained through Three.js app development will be useful information for you.

Improving Online Meetings with Doodles

corridor project — Sat, 26 Sep 2026 03:03:30 +0000

Hello, I'm pj-corridor, a platform engineer. This article is a translated version of the original Japanese article.

0. Whiteboard

In the past, when going to the office was the norm, discussions utilizing whiteboards were held throughout the office. In retrospect, discussions using whiteboards were effective in many respects, such as :

Organizing one\'s ideas
Enhancing participants\' creativity
Aligning participants\' understanding
Early detection of issues by drawing out participants\' perspectives and questions

For companies in the process of integrating knowledge and culture between former organizations, such as right after an M&A, we would like to bridge the gaps through discussions using whiteboards, as we did before. However, in situations where global collaboration and diverse working styles are prevalent, it is not always possible for all stakeholders to gather in person. Then, it is essential to enhance the quality of online communication in our daily operations.

Therefore, as an alternative to discussions using whiteboards, I'd like to suggest using doodles extensively, along with discussions that utilize them.

1. Benefits of Doodles

As Richard Sennett explores in "The Craftsman", the act of making is inherently linked to thinking and problem-solving. In the process of devising doodles while preparing for discussions, you can better organize your thoughts and points of argument. This alone makes creating doodles highly recommended.

Moreover, since doodles are easy to grasp in concrete form, participants can easily spot inconsistencies and unclear points. The visual stimulation also draws out participants\' creativity, sparking new ideas and solutions. Additionally, the picture superiority effect makes it easier to remember information after online communication.

While using existing materials is rational from the perspective of reducing preparation time, if you value the benefits mentioned above, I recommend investing in the use of doodles I suggested.

2. Increase the Number of Pages

First, don\'t hesitate to increase the number of pages. However, increasing the number of pages is not the same as overwhelming participants with voluminous materials. The goal is to enhance the quality of online communication.

For example, having participants navigate back and forth between slides or transitioning to linked content to reference specific information can hinder understanding and discussion. If you want to repeatedly refer to the same slide, it should be reinserted each time. If you are concerned about maintenance errors caused by copying, you can manage slides that need to be reused as slide-master layouts.

Now, let's increase the number of pages further. With that mindset, let me introduce :

Where We Are Now
Pay Attention !
Insights from Change

3. Where We Are Now

Participants may feel stressed about leaving inconsistencies or unclear points unresolved, even if they don't want to interrupt the conversation with trivial questions. Therefore, insert slides that clearly indicate when the topic changes (in this case, I recommend to use slides with inverted brightness) and visualize the progress at the same time.

By doing this, you can expect effects such as :

Making it easier for participants to determine when to ask questions
Allowing presenters to create opportunities to ask about understanding based on the situation

Intentionally creating timing can promote interactivity.

4. Pay Attention !

Emphasizing a part of doodles to draw participants\' attention is useful when you want to explain the whole in order or deepen discussions on important parts.

For example :

Duplicate the doodle and change the overall tone to a grayed-out expression, such as overlaying it with 10% transparency white.
Add the slide from step 1 as a layout in the slide master.
Overlay the original doodle on a new slide using the layout from step 2.
Remove everything except the part you want to highlight.
As a result, the removed parts will be grayed out.

The slide created using these steps looks like this :

By overlaying a dark color, for example, 50% transparency black, it looks like this :

You can also use radial lines or background colors. In this case, having a set of reusable parts is convenient.

5. Insights from Change

Things that are hard to notice only in snapshots can become apparent by demonstrating change, making it easier for participants to gain insights. This is a doodle used in discussions on real projects. With a doodle only Before & After, which omits several changes in process, participants\' thoughts cannot keep up.

By inserting doodles that express several changes in between, you can draw out more perspectives from participants, leading to the early detection and improvement of issues.

For example, you might gain insights like this at each step :

In this step, presenter uses several changes to appeal to participants\' mental models about the "integration of A and B". Participants will then be concerned about attribute information or state conflicts linked to A and B and will demonstrate creativity to resolve them.
In this step, presenter explains that C requires special treatment, including the background. Participants will then understand that this state is challenging to express in the UI.
In this step, presenter explains proposal that aligns the treatment of D and E with C, maintaining consistency overall. Participants will then deepen their understanding of the Before & After based on the background and simultaneously begin exploring better alternatives.

You can expect to draw out participants\' creativity in this way.

On a side note, when someone starts using the pen tool of online meeting system, other participants tend to wait (in a state of suspended thinking) while watching the screen until it's completed. Then, the finished modern art confuses participants anew.

The pen tool promotes interactivity through visual stimulation, but it seems more effective when used simply, such as :

Adding a few lines
Marking elements with "o" or "x"
Drawing one or two additional elements

With doodles that include several changes, you can easily select a base doodle that matches the discussion point, maximizing the effect of the pen tool.

6. Avoid Preconceptions

As stimulating participants\' creativity is one goal, be mindful of preconceptions, misunderstandings, and distractions that hinder understanding. For example, if size is a discussion point, the doodle should represent size relationships without causing misunderstandings.

Also, when expressing connections, it\'s important to minimize misunderstandings. In the following doodle, when connecting boxes labeled with the same letter, using an elbow connector is not recommended. Instead, there is room for improvement by using straight or curved lines.

For example, how about visual separation of intersections using masks ?

Shifting the intersection itself might also work.

You can move connectors to the front or change the order of masks and connectors using "Format Shape" + "Selection Pane".

Next, let's express the relationship of the vertices of an equilateral triangle with doodle. When positioning an equilateral triangle at the center of a slide according to PowerPoint guides, the center and centroid of the triangle do not align. Therefore, a doodle connecting each vertex from the center may give the impression that vertex A is a special entity, which is inappropriate for expressing equal relationships.

A doodle connecting each vertex from the centroid of an equilateral triangle can clearly indicate the equal relationship of each vertex.

In the case of a regular pentagon, a doodle connecting each vertex from the center of the guide reveals the source of discomfort when rotated.

A doodle connecting each vertex from the centroid maintains a beautiful regular pentagon even when rotated.

Next is a topic specific to concrete examples. A doodle often expresses API offerings or data exchanges. In this case, the direction of the HTTP request calling the API and the direction of data exchange may not only be the same but also opposite. Be careful not to cause preconceptions or misunderstandings about what the arrows in the doodle are intended to represent.

7. Which Preset Storage ?

A doodle often represents storage, especially for system use, but PowerPoint has two types of preset storage. Cylinder A cannot adjust the shape of the bottom, resulting in awkward perspective when elongated. Which cylinder should be used ?

Cylinder B allows for expressing data accumulation or proportions in storage by overlaying cylinders without using gradient fills. In conclusion, using cylinder B seems more flexible.

8. Doodle in Tables

There are times when you want to express the behavior for each use case using both tables and doodles. While separating tables and doodles into different slides makes the material tidy, you have to navigate between slides to understand what happens in each case.

How about making a doodle into a table ? Although it may look cluttered, it succinctly expresses what happens in each case. It's also possible to embed the doodle part using Slide Zoom.

9. You Can Draw Anything

Like generative AI improves accuracy with specific examples, specific examples are beneficial when dealing with humans too. By providing specific examples as input, you can set expectations for the direction of arguments and feedback, enabling more targeted discussions. Finally, for your specific examples, I\'ll share how to draw doodles with high flexibility.

For example, a diamond can be created by:

Combining triangles
Coloring each part with gradient fills
Grouping them

Bags and wallets can be drawn easily. Reuse them for advertisements or EC-specific examples.

For more flexible doodle preparation, prepare a shape with rounded edges by surrounding a square with curves.

The same applies to shapes other than squares.

Connect and join these to create a stick figure.

When you'd like to use stick figures more easily, please check out my Stick Figure Gallery made by Three.js. You can find something close to your image and make slight adjustments to obtain the desired stick figure material.

As a side note, PNG or JPG images can also be color-adjusted or have their backgrounds removed in PowerPoint. For example, when expressing multiple characters with PNG image icons, you can handle it with color adjustments without preparing different images.

Conclusion

Thank you for reading this far. In fact, my method using doodles is still in the process of trial and error and improvement, and it is not yet systematically organized, with ongoing updates planned. The burden of drawing doodles remains a challenge. On the other hand, within the scope summarized in this article, we have been able to experience its effectiveness through actual work, and we hope that readers will try it as a means to improve online communication.

The PowerPoint file used in this article is available here, so feel free to use it.

Did Generative AI Do a Good Job Today? What About Tomorrow?

corridor project — Wed, 23 Sep 2026 21:33:11 +0000

Hi, I'm pj-corridor, an engineer.

A while ago, Google's AI Overviews drew attention after surfacing advice about using glue on pizza. How do you approach the quality of generative AI output?

I want to refactor an increasingly bloated prompt, but I'm worried about unintended side effects.
After upgrading the model, will those carefully tuned prompt tricks still work the same way?
A change to the generation pipeline seems functionally fine, but could it introduce regressions in edge cases?

I imagine many development teams face similar concerns. In the service I work on, we also had a need to automatically generate briefings from news articles and social media posts using generative AI.

In this article, I'll use automated briefing generation as a case study to explain how we evaluated the quality of generative AI output.

Overview of the Evaluation Process

Let's start with an overview of the evaluation process. The main actors are experienced editors and operations staff (green), an evaluation bot (blue), and the generation pipeline (red).

Ask experienced editors to create concrete examples of "good briefings" (hereafter, Gold Data).
Have them abstract the Gold Data into criteria that define what makes a briefing "good" (hereafter, Rubrics), which become the basis of the evaluation bot.
Run an initial validation. Can the evaluation bot assign high scores to the Gold Data?
Review either the Gold Data or the Rubrics until the validation produces the expected results.

Build the initial version of the briefing generation pipeline.
Run the generation and evaluation batch process.
- Generate briefings from an input dataset with sufficient volume and diversity.
- Have the evaluation bot produce evaluation reports for the generated briefings.
Improve the generation pipeline based on the reports and treat the result as the next-generation version.
Repeat generation, evaluation, and improvement until the predefined criteria are met.
Have experienced editors perform the final deployment review.

Artifacts and reports are managed in the repository. If the model changes in the future, or if the briefing specification changes—for example, to support stronger personalization—we can repeat the process and compare the new version against previous ones when necessary.

The goal of this process is to maximize the cost-effectiveness of evaluation: the evaluation bot provides a certain level of quality at scale, while experienced editors focus on reviewing risks that require human judgment.

To support this approach, we implemented an evaluation bot and process-support framework using DeepEval. Through actual use, however, we encountered a number of challenges.

Let's take a closer look at those challenges and how we addressed them.

Creating Gold Data

Let's start with a fictional article. It's intentionally long, so feel free to skim it.

The International Space and Aerospace Development Agency (ISADA) and other members of an international space exploration team announced on the 7th that initial construction of humanity's first permanent lunar base, "Artemis Base," near the Moon's south pole had been completed. The milestone establishes the infrastructure needed for humans to remain on the lunar surface for extended periods and conduct scientific research and development. The base consists of residential modules, solar power systems, and an experimental plant that extracts hydrogen and oxygen from lunar water resources. Up to six astronauts are expected to rotate through the base, conducting research into the effects of low gravity on the human body, astronomical observations, and technology demonstrations for resource extraction. The base is also expected to serve as a staging point for future crewed missions to Mars, since launching rockets from the Moon could require less fuel than traveling directly from Earth to Mars. A representative of the international exploration team described the completion of the base as a historic step toward humanity becoming "a species that lives in space."

Now let's generate several types of short-form text from it.

First, we prompted:

Give this article a title of no more than 30 characters that concisely conveys its content.

And got:

Humanity's First Moon Base

That seems like a reasonable title.

Next, we asked:

Summarize this article as an internet forum thread title.

And got:

[Good News] New house completed: quiet neighborhood, great sunlight, no oxygen

That certainly has the right vibe.

Express this article as an acrostic using the word "MOON".

And got:

Mankind makes a home beyond Earth

On the Moon, a new chapter begins

Outward lies the journey to Mars

Now, space becomes a place to live

Not bad for an acrostic, right?

Now suppose we asked, "Which of these short texts is the best?"

Without defining what "good" means, there is no reproducible way to answer that question.

The same applies to briefings. Since it is difficult to arrive at an abstract definition from the outset, at this stage of the overall evaluation process...

...we first create multiple examples of "good briefings" as Gold Data, taking the direction of the service into account.

We then compare those examples and extract common characteristics such as:

Is accuracy the most important quality?
Is it more important to make readers want to continue reading?
What kind of care is required when describing sensitive topics?
Should clarity be prioritized, for example by using rhetorical techniques?

We use these shared characteristics to define the Rubrics.

Defining the Rubrics

DeepEval provides built-in Metrics such as Faithfulness, which evaluates whether output is faithful to the input, and Summarization, which evaluates the quality of a summary.

However, the Rubrics that define a "good briefing" do not necessarily map directly to existing Metrics. For this reason, we chose G-Eval, which allows Rubrics to be defined in natural language.

For example, an accuracy Rubric might be:

The generated text must not contradict the original and must accurately reflect its content. It must not introduce facts, speculation, exaggeration, or definitive claims that are not present in the original, and it must not present uncertain information as established fact. Missing information alone should not reduce the score.

A Rubric for appropriate handling of sensitive information might be:

The generated text must handle sensitive topics such as death, accidents, disasters, crime, illness, suicide, discrimination, and human rights issues with appropriate care. It must not include unnecessary condemnation or ridicule of victims, bereaved families, related parties, or people merely suspected of wrongdoing. It must not damage someone's reputation or credibility through speculation or unverified information. It must not use sensational or entertainment-oriented language to excessively attract attention to sensitive topics.

The evaluation bot refers to Rubrics like these when scoring outputs.

Because evaluations from multiple perspectives can be run in parallel, adding more perspectives does not necessarily have a large impact on processing time. However, increasing the number of Rubrics also creates problems such as:

Instructions may overlap across Rubrics, reducing maintainability.
Instructions may conflict across Rubrics. For example, improving "conciseness" could worsen "completeness," making it harder to improve the generation pipeline coherently.

These issues require care.

So, at this stage of the overall evaluation process...

...we report not only the consistency between the Gold Data and the Rubrics, but also overlaps and conflicts among the Rubrics themselves, helping us improve the evaluation bot.

Going through this step helps both humans and the evaluation bot develop a clearer understanding of what constitutes a "good briefing."

That said, although the original G-Eval paper reported improvements over previous approaches for summarization tasks, its Spearman correlation with human evaluation was 0.514.

We show that G-EVAL with GPT-4 as the backbone model achieves a Spearman correlation of 0.514 with human on summarization task, outperforming all previous methods by a large margin.

We therefore do not assume that validation against Gold Data alone guarantees agreement with human judgment. As described earlier, it is more appropriate to assign the evaluation bot the role of providing "a certain level of quality at scale."

Generation Pipeline

At this stage of the overall evaluation process...

...the resulting pipeline will eventually be implemented in the production product. The prompts and rule-based processing (Python code) managed in the framework can therefore be treated as implementation specifications for the production system.

You could aim for a highly polished pipeline from the beginning, but I recommend keeping the first version simple so that the impact of each modification remains easy to understand. The framework also includes a tool that automatically generates a simple initial prompt from the Rubrics.

However, problems such as the following can still occur even if the prompt strongly prohibits them or explicitly asks the model to review its output:

Mixed-language output
Violations of character limits
Use of prohibited words

For that reason, we separated rule-based processing from the prompt and built a pipeline like this:

Generate a briefing using the prompt.
Apply rule-based processing to adjust the result and validate constraints.
If the output does not satisfy the constraints, regenerate the briefing with additional guidance intended to avoid the previous failure.
- Add the failed output and its failure reason to the prompt as feedback.
- Temporarily adjust the temperature parameter to increase output diversity.

We decided not to support conditional branching or multi-stage prompting until an actual need arose. Metadata input, however, was treated as a mandatory requirement.

For example, a teenage woman and a man in his forties might focus on different aspects of the same news story. Similarly, for a social media post about a restaurant, the most effective angle for a briefing might depend on the weather or time of day when the briefing is displayed.

When metadata is included in the generation prompt, the evaluation bot also enables G-Eval's SingleTurnParams.CONTEXT, keeping the generation and evaluation conditions aligned.

Reducing Evaluation Variance

Now let's evaluate the output of the generation pipeline.

Suppose an accuracy score changes from 0.75 in the previous evaluation to 0.91 in the current one.

Can we say the system improved?

Now suppose the score for appropriate handling of sensitive information changes from 0.93 to 0.84.

Is that a regression?

To make decisions based on scores, we need to reduce evaluation variance where possible and understand its characteristics before interpreting changes.

So first, let's try to reduce that variance.

Looking at the temperature parameter, which controls output diversity, the original G-Eval paper used a temperature of 0 when evaluating with GPT-3.5 in order to increase model determinism.

We use OpenAI’s GPT family as our LLMs, including GPT-3.5 (text-davinci-003) and GPT-4. For GPT-3.5, we set decoding temperature to 0 to increase the model’s determinism.

Likewise, the default for AnthropicModel is 0.0.

temperature: A float specifying the model temperature. Defaults to TEMPERATURE if not passed; falls back to 0.0 if unset and raises if < 0.

Based on this, we also use 0 for the evaluation bot. This value may need to be reconsidered for future models.

G-Eval also includes a mechanism that uses the probabilities of candidate scores produced by the generative model to calculate a weighted average and reduce scoring bias.

This does not directly eliminate evaluation variance, but we use it because we expect it to contribute to more stable scoring. However, some backends, including the Bedrock backend used in this implementation, cannot provide the score-token probabilities required for this mechanism.

In the original G-Eval paper, the authors used the probabilities of the LLM output tokens to normalize the score by calculating a weighted summation.
This step was introduced in the paper because it minimizes bias in LLM scoring. This normalization step is automatically handled by deepeval by default (unless you're using a custom model).

We also paid attention to the fact that G-Eval internally performs evaluation in two stages.

Since G-Eval is a two-step algorithm that generates chain of thoughts (CoTs) for better evaluation, in deepeval this means first generating a series of evaluation_steps using CoT based on the given criteria, before using the generated steps to determine the final score using the parameters presented in an LLMTestCase.

For example, when evaluating the accuracy Rubric shown above, it may generate evaluation_steps like these:

"evaluation_steps": [
    "Read the original text carefully to identify all stated facts, claims, and the degree of certainty attributed to each piece of information.",
    "Read the generated text and list every factual claim, inference, or assertion it contains.",
    "For each item in the generated text, check whether it is directly supported by the original text. Flag any item that introduces a fact, detail, or figure not present in the original.",
    "Check whether the generated text adds speculation, predictions, or assumptions that are not present in the original.",
    "Check whether the generated text exaggerates or overstates any claim beyond what the original text supports.",
    "Check whether the generated text presents uncertain or conditional information from the original as definite fact.",
    "Check whether any statement in the generated text directly contradicts a statement in the original text.",
    "Do not penalize the generated text for omitting information that appears in the original, as insufficient information volume is not a scoring criterion.",
    "Assign a score based on the number and severity of violations found: no violations warrants the highest score, and each confirmed addition of unsupported facts, speculation, exaggeration, false certainty, or contradiction lowers the score proportionally."
]

If different evaluation_steps are generated when evaluating the previous and current versions, that difference itself could contribute to evaluation variance.

To avoid this, our implementation reuses the same evaluation_steps unless the Rubric changes. As a side effect, this also reduces processing cost.

Understanding Evaluation Variance

Even after these measures, evaluation variance cannot be eliminated entirely.

In addition, the characteristics of that variance may differ depending on whether an evaluation dimension is discrete and objective, such as factual consistency, or continuous and subjective, such as how engaging a briefing is.

To make better decisions based on scores, we therefore need to understand the characteristics of evaluation variance.

Immediately before entering the generation-pipeline improvement cycle...

...we use a randomly generated input dataset and repeat the same evaluation multiple times to report the characteristics of "evaluation variance."

For example, the following statistics summarize the standard deviation observed when evaluating articles input items iterations times.

{
    "model": "YOUR_BACKEND_MODEL",
    "articles": 75,
    "iterations": 5,
    "note": {
        "stddevAvg": "Average standard deviation of repeated evaluations for the same test data. Lower values indicate more consistent scoring.",
        "stddevMax": "Maximum standard deviation among all test data. Lower values indicate the worst-case evaluation inconsistency is smaller.",
        "testDataInfo.stddev": "Standard deviation of the average scores across the test data. Higher values indicate the test data covers a wider range of quality."
    },
    "rubrics": {
        "accuracy": {
            "stddevAvg": 0.17848219523165634,
            "stddevMax": 0.3666060555964672,
            "testDataInfo": {
                "max": 0.96,
                "min": 0.56,
                "avg": 0.7906666666666667,
                "stddev": 0.09308538493710433
            }
        },
        "sensitivity": {
            "stddevAvg": 0.09983845129437227,
            "stddevMax": 0.3006659275674582,
            "testDataInfo": {
                "max": 0.98,
                "min": 0.74,
                "avg": 0.8904,
                "stddev": 0.05650817050067478
            }
        },
        ...
    }
}

From these results, we can see that accuracy tends to show evaluation variance more broadly, while appropriate handling of sensitive information tends to show larger variance only for some articles.

Note that testDataInfo.stddev is not a measure of evaluation variance. It indicates how widely the evaluation scores are distributed across the input dataset.

If that distribution is too narrow, we review the input dataset and the resulting output distribution before collecting the statistics again.

As an aside, TypeSafe AI recently announced Jev. Its approach of producing a predefined judgment and confidence score rather than generating free-form text looks promising for addressing some of the challenges related to evaluation variance.

Iterating on the Improvement Cycle

The evaluation bot interprets score changes while also taking statistics such as stddevAvg and stddevMax into account, and produces reports describing potential improvements and regressions.

After the overall evaluation, the report lists scores and reasons for individual briefings.

We then use this report to iterate on the improvement cycle.

However, if the evaluation bot throws a traceback, the improvement cycle stops as well. Avoiding that required handling some exceptions that occur inside DeepEval.

For example, GEval._evaluate() contains logic that interprets the value returned by the evaluation model as JSON. With one model, however, we occasionally received output like this:

{
    "score": 10,
    "reason": "...",
    "score": 10
}

Hmm, let me provide valid JSON:

{
    "reason": "...",
    "score": 10
}

This causes JSON parsing to fail and can stop the entire batch process.

As a somewhat pragmatic workaround, when the model returns output that cannot be interpreted as JSON, we retry the request.

def generate_raw_response(self, in_prompt, **in_kwargs):
    def isDeepEvalSafe(in_string):
        try:
            # emulate trimAndLoadJson
            text = in_string.strip()
            if text.startswith('```'):
                text = text[3:]
            if text.startswith('json'):
                text = text[4:]
            if text.endswith('```'):
                text = text[:-3]
            json.loads(text.strip())
            return True
        except json.JSONDecodeError:
            return False
    retryCount = 0
    while True:
        chatCompletion = self.runner.toChatCompletion(in_prompt)
        if isDeepEvalSafe(chatCompletion.choices[0].message.content):
            break
        retryCount += 1
        if retryCount >= 10:
            raise RuntimeError('DeepEval acceptable response could not be generated.')
    untrackedCost = 0
    return chatCompletion, untrackedCost

As we repeated the improvement cycle, we discovered a number of practical issues. Here are a few examples.

News headlines often reflect the publisher's editorial intent—for example, which aspect of a story they want to emphasize.

In some cases, the generation pipeline was pulled toward that framing and produced briefings that were factually misleading.

From the evaluation bot's perspective, however, the input and output were still consistent with each other, so the accuracy Rubric could not detect the problem.

We ultimately resolved the issue by excluding headlines from the input, but the lesson was that evaluation results are affected not only by the evaluation model and Rubrics, but also by the input data itself.

We also encountered cases where each individual fact was correct, but incorrect relationships between those facts could not be detected.

For those cases, we revised the accuracy Rubric so that relationships such as "actor," "target," "event," and "condition" would also be preserved and evaluated.

SNS posts may also contain typos or factual errors, but we cannot simply correct them during briefing generation based on our own assumptions. In such cases, we need to avoid relying on potentially incorrect information and abstract the content to a level where the briefing can be expressed without using it.

Finally, a lesson from operations.

Early in the improvement cycle, even a relatively small input dataset is often enough to expose critical problems.

For a more efficient improvement process, I recommend starting with a smaller dataset and switching to an input dataset with sufficient volume and diversity only after the generation pipeline reaches a reasonable level of quality.

Conclusion

After going through this process, we were able to automatically generate "good briefings" from news articles and social media posts, and even received a passing grade 😊 from our experienced editors.

Does that mean we can safely assume tomorrow's automatically generated briefings will also be fine?

Unfortunately, not necessarily.

For cases where the risk is difficult to accept—such as reporting involving members of the Imperial Family or the suicide of a public figure—we may still need HITL approval before publication.

It may also be worth running the evaluation bot in the production system itself and blocking publication when scores fall below a given threshold.

Using generative AI in production is not exactly effortless 😅

I hope this article provides some useful ideas for anyone struggling with the quality of generative AI output.

Although this article focused on automatically generating briefings, the evaluation bot and process-support framework are implemented as a general-purpose system for text-generation use cases. The implementation is available on my personal GitHub repository, so feel free to use it if you find it useful.

I also wrote an earlier article about a personal AI-integrated application. That article ended with a small teaser about LLM-as-a-Judge, so I'm glad I finally got to follow up on it here.

If you're interested, you can think of that earlier article as Episode #1.

What to do in such situations: Developing an app integrated with generative AI

corridor project — Tue, 22 Sep 2026 05:04:43 +0000

Hello, I’m pj-corridor, an engineer. In this article, I will introduce the challenges I faced while developing an application integrated with generative AI, such as dealing with abuse and navigating trial-and-error and trade-offs in prompt tuning.

First, please try the personality assessment app I initially developed. The annoying ads 😆 can be hidden by adjusting the window width.

Like the popular MBTI, CAPS and DiSC are so-called pseudoscientific personality assessments. They should not be used as a basis for decision-making, but for example, having workshop participants share their results can help energize self-introduction sessions and warm up the atmosphere. Please consider using this assessment app on such occasions.

For reference, here is my CAPS assessment result.

The advice and user manual are generated by AI, but the sudden appearance of “Respondent” in Japanese sentences feels a bit unnatural. In this way, while generative AI provides rich expressive capabilities, it also introduces quality risks. That said, I wanted to keep the cost of ensuring quality within the scope of a hobby project. Therefore, I kept the rule-based scoring logic deterministic and limited the generative AI output to supplementary text, thereby controlling the impact of hallucinations.

From here, I will introduce concrete examples of trial and error and how I approached trade-offs, divided into three chapters.

1. Abuse prevention

The assessment app can be accessed anonymously, but the backend uses a paid generative AI service (Amazon Bedrock). Therefore, being overwhelmed by bot abuse is a plausible scenario. However, rather than aiming to eliminate all threats, I adopted a basic policy of considering the attacker’s incentives: “accept threats with low manifestation risk, and mitigate the rest.”

1.1. Blocking common attack traffic with WAF

For WAF, referring to AWS’s standard protection pack, I adopted:

GeoRule (blocking IPs from regions with frequent attacks)
AWS-AWSManagedRulesAmazonIpReputationList (blocking known malicious IPs)
AWS-AWSManagedRulesAnonymousIpList (blocking anonymized/tunneled IPs)
A tuned GlobalRateBasedRule for the assessment app (request rate limiting)
A tuned RateBasedRulePOST (limiting POST/PUT/DELETE requests)

and removed unnecessary rules such as AWS-AWSManagedRulesSQLiRuleSet. However, when using a Lambda Function URL as a public endpoint, there is concern that requests not routed through CloudFront could bypass the WAF. While it is possible to enforce CloudFront (and thus WAF) routing, doing so would require AWS-specific implementations, such as:

Changing the Lambda Function URL AuthType to AWS_IAM and using CloudFront OAC 👉 This requires support for x-amz-content-sha256 in POST requests
Adding a secret header in CloudFront and verifying it in Lambda 👉 This requires managing multiple environment variables and verification logic

In any case, AWS-dependent implementation would be necessary. Considering the risk that a malicious actor could discover the Lambda Function URL and bypass the WAF, I decided not to implement this measure for now.

As a side note, settings configured in the AWS console are easy to forget. Forgetting to revert temporary settings can cause issues. To avoid this, I recommend managing managed service configurations in a repository (something close to IaC).

For example, for a WAF protection pack:

Manage the JSON as the Source of Truth in the repository
Always apply changes via Source of Truth → AWS console
Check consistency between Source of Truth and the applied WAF policy via CI

Also, as a follow-up: with the traffic scale of this app, the fixed cost of WAF ended up exceeding the usage-based cost of Bedrock. A good lesson in estimating costs 😅. Going forward, I am considering suspending WAF until traffic increases and instead implementing POST request limits using DynamoDB.

1.2. Controlling access from simple bots and browsers bypassing the flow

For access control, I implemented:

When accessing the app, generate a token from “secret information + timestamp” and send it via Set-Cookie
The browser includes the token in requests using credentials: 'include'
The server validates that the token could have been generated within a human response time window

Additionally, I applied:

SameSite=Strict to block cookies in cross-domain POST requests (see SameSite explanation)
Restricting Access-Control-Allow-Origin to a whitelist to limit browser-based access

These measures can be bypassed by more sophisticated bots, but I decided to start within the scope of the basic policy and consider additional measures while monitoring access patterns.

2. Technology selection

Here I describe trial and error related to Lambda and API Gateway.

Lambda was a reasonable choice for the runtime environment, but I initially preferred:

Lambda container image (using PHP)
Lambda zip deployment (using Python/Node)

I leaned toward (1) because I already had a local PHP testing environment, which seemed suitable for agile development and testing. However, considering compatibility with Lambda and concerns about CI complexity, I ultimately chose (2). In hindsight, I spent more time dealing with AWS environment nuances and black-box behavior, so it turned out to be the right choice.

Speaking of trial and error, repeatedly creating Lambda functions via the AWS console automatically generates new IAM roles each time. Not just these remnants, but leaving unused resources can become technical debt, so they should be cleaned up promptly.

I resisted the urge to jump into development and prepared a mechanism to handle Lambda and test environments transparently.

This later became the foundation for prompt tuning.

As for whether to use API Gateway:

Feature	Implementation
Authentication	Custom access control
Routing	Handled within Lambda
Throttling	Handled by WAF (planned to change)

Given this, I decided not to use API Gateway at this stage, as the cost outweighed the benefits.

3. Application development

With abuse prevention and technology selection in place, it was time for development. Once completed, I wanted to share it globally via Reddit. That led to the motivation to support multiple languages, so I implemented a simple i18n class to support 9 languages.

const GREETING = i18n.text({
    en : 'Hello',
    ja : 'こんにちは',
    fr : 'Bonjour',
    de : 'Hallo',
    es : 'Hola',
    pt : 'Olá',
    hi : 'नमस्ते',
    ko : '안녕하세요',
    zh : '你好'
})

To stabilize AI output quality, I used the user’s native language for UI but fixed the AI input language to English.

I also unified state management, UI components, and interfaces with Lambda/test environments between DiSC and CAPS, enabling reuse for future assessments like MBTI.

Finally, I focused on improving and stabilizing AI output quality through prompt tuning. Instead of rushing, I first built a foundation:

Prepare shortcut features for prompt generation
- For example, “random answers + query execution” automation
- Abuse prevention via hash-based validation
Tune in a test environment (without AI connection)
1. Execute shortcut
2. Output prompts to console
3. Simulate production using Gemini/ChatGPT and evaluate
4. Iterate prompt/template improvements
Tune in production (with AI connection)
1. Partial tuning via shortcut
2. Full app testing and final checks

During this process, I discovered missing outputs in hi, ko, and zh, likely due to token differences across languages.

Also, AI suggestions tend to bloat prompts with redundant instructions, so consolidating and refactoring prompt structure is recommended.

Finally, about “Respondent”: the requirement was to fix the subject as colleagues/friends while referring to the user consistently. Attempts like:

...use the {{lang}} term for "Respondent".

resulted in “you,” while banning pronouns led to “Respondent.” To stabilize output, I used:

always use the fixed keyword "_RESPONDENT_"

and replaced it client-side 😅

const RESPONDENT = i18n.text({
    en : 'Respondent',
    ja : '回答者',
    fr : 'Répondant',
    de : 'Befragter',
    es : 'Encuestado',
    pt : 'Respondente',
    hi : 'उत्तरदाता',
    ko : '응답자',
    zh : '受访者'
});

Conclusion

Thank you for reading. In this project, I focused on structuring prompt tuning while controlling hallucination impact. Future improvements may include automated output quality checks:

Rule-based validation of format and keywords
Using one AI to evaluate another AI’s output

I hope my trial-and-error and trade-off decisions are helpful to you.