DEV Community: orca_forge

The angle between the head and chest was 0.000 in every frame: The time my motion tracking was broken

orca_forge — Thu, 17 Sep 2026 00:53:41 +0000

The character's head would tilt along with the body whenever it twisted. When I measured the angle difference between the head and chest, it was 0.000° for every frame. I ended up redoing the head orientation logic downstream three times before realizing the cause.

The issue was that the nose and ear landmarks used for measurement were synthesized from the torso's base vector. I thought I was measuring the head and chest separately, but I was actually comparing the same orientation.

This is the second installment in a series about creating VRM animations from fixed-camera live-action videos. This time, I'll organize the lessons learned from actual mistakes regarding measurement targets, coordinate systems, and verification methods.

In the previous article, I covered the entire pipeline and design decisions. Links to each installment in the series are provided at the end.

What Are We Measuring?

0.000° Means "Measuring the Same Thing Twice," Not "Fixed"

In this material, the live actor's head moved about ±10° relative to the chest, so I was suspicious of the 0.000° result for every frame. Upon investigation, I found that the head and chest measurements were not independent.

Here's the dependency diagram:

Old: Shoulder line ──→ Torso base ──┬──→ Nose/Ear (synthesized)──→ Head orientation
                             └───────────────────→ Chest orientation
                             (Two outputs from the same input)

New: SMPL mesh actual vertices (nose/ear)─────────────→ Face orientation
    Torso base ──────────────────────────────→ Chest orientation

By definition, the ear line matches the shoulder line. This means the head orientation can never be anything other than the chest orientation in the calculation. Subtracting them always results in 0. So 0.000° didn't mean "the head is fixed to the chest," but rather "I was subtracting two values created from the same input."

I ended up redoing the downstream logic three times, thinking the issue was with how the head orientation was applied in the FK (forward kinematics) stage.

The fix wasn't in the estimator itself, but in the adapter (pipeline/estimate_pose_gvhmr.py) that converts GVHMR outputs to landmarks for this pipeline. It reads the actual nose and ear vertices from the SMPL mesh and outputs a unit vector connecting their midpoint every frame. The implementation name is gaze, but it's actually an approximation of face orientation, not the eyeball's line of sight.

I also added two checks to the QA process: if the standard deviation of the head-chest angle difference is less than 2°, or if the average face elevation angle exceeds 12°, a confirmation prompt is triggered. These thresholds are designed to catch anomalies in this pipeline. They don't flag intentionally fixed head movements or constant upward gazes as defects.

Capsule Approximation Said "4mm Clearance"

This happened when creating a video of two characters fighting. To the human eye, the kicking leg was penetrating the opponent's head. The comparison script reported "4mm clearance."

The comparison used a model approximating the torso as a cylinder with a 0.16m radius and the head as a sphere with a 0.11m radius. This model is lightweight and fast, useful for estimating distances and reach. However, it doesn't represent the character's silhouette. The hood, shoulders, gloves, and boots are all outside this approximation.

When checking for overlaps with the actual skinned mesh, I found intersections in 4 frames, with a maximum of 230 overlapping face pairs.

Capsule approximation: 4mm clearance
Actual mesh: 230 face intersections

From then on, I used Blender's BVH to check for face intersections between actual meshes for penetration confirmation in this scene. BVH checks for "face intersections," not all contacts or enclosures. There's a distinction between approximations for corrections and final visual confirmation (pushing with approximate shapes is covered in Part 4).

The lesson here, in one sentence, is:

Same Point Name Doesn't Mean Same Location

Measurement breakdowns often occur when two compared points aren't actually the same point. I encountered three such cases in the same session.

Comparison/Judgment	Mismatch	Observed Impact
Actor's nose vs character's `Head` bone	Face position vs skull base	Compared points were ~10cm apart
Shoulder line vs `Neck` bone	Shoulder vs neck height	Compared points were 5-10cm apart
Contact judgment using only leg endpoints	Endpoints vs line segment	Missed shin penetration when knee and ankle were outside

The third case is particularly problematic. Relying only on endpoints means you can't detect "shin penetrating the head" if both the knee and ankle are outside the target.

Another issue is mistaking body proportions for pose errors. In one clip flagged for "hand position too high relative to face," 4.6cm of the 14.8cm difference was due to the character's head being lower relative to the shoulders compared to the actor. The comparison script now separates body proportion ratios from pose errors.

Which Coordinate System Are We Measuring In?

Camera "Up" Isn't Gravity

I received feedback that a kick "lacked height." Measurements confirmed it was low, but the issue wasn't the kick itself - I was reading the height in the camera coordinate system without correcting for the camera's tilt.

The camera was tilted 4.2° downward. Without adjusting for this tilt, there was a 0.14m error in the kick's peak height relative to the head. Before judging the kick as insufficient, I needed to align the comparison coordinate systems.

In this material, the camera tilt wasn't noticeable just by watching the video. Even when mounted on a tripod, this level of tilt is common. When measuring height, establish the gravity direction first - leading to the next point.

Compare Two People in a Shared Coordinate System

This is from a record of handling a two-person fight video (found in the forked repository's documentation).

Measuring the distance between the kicking foot and the opponent's head gave 0.18m. However, both the video and independently measured image quantities showed 0.29m. The difference between 0.18m and 0.29m is significant - "barely missing" versus "clearly missing."

The cause was that the estimator's world coordinate system normalized each person so "frame 0 faced forward." With two people, there are two separate coordinate systems. Any difference in their starting orientations creates a misalignment. Arranging their limbs in a single space produces results that seem plausible but are subtly incorrect.

For this comparison, I aligned the estimation results of both individuals to a common camera coordinate system. Note that the scale conversion from image quantities to meters and the assumptions used are not recorded in the original documentation. To reuse the 0.29m value, that conversion would need to be redone.

World and In-Camera Are Correct for Different Things

The pose estimator (GVHMR) produces two types of outputs for the same body:

World: Estimation results in a gravity-aligned coordinate system. In this material, position and height drift were problematic
In-camera: Estimation results in the camera coordinate system. Easier to compare with images, but depth estimation errors and camera tilt need separate handling

Both are estimation results. It's not a contrast between "world as prediction and in-camera as measurement." Mixing these two can lead to bugs with unclear failure modes. Here are some real-world examples (different materials and comparisons in each row, so don't compare numbers directly):

Material	Symptom	What Was Compared	Numbers
Plaza	Ground rises over time	World height time change	0.25m over 3 minutes
Fountain	Jump height not captured	World upward movement vs image upward movement	0.02-0.07m vs 0.12-0.26m
Plaza	Backwards-starting material flips entirely	Normalized orientation vs video orientation	180°
Material with approach (241 frames)	Insufficient step-in distance	World movement vs actual step-in	Reported 0.68m for 0.92m, final 0.16m marker mismatch

The last row is a clear example. The world output seems physically consistent and "correct" intuitively, but it drifts due to model-based estimation. Meanwhile, the in-camera output matched independently measured image quantities (waist pixel count ÷ body pixel count) within 2cm for all 241 frames. Again, the conversion from pixel ratios to meters isn't recorded, so the match accuracy needs retesting.

Suspect Common Causes When Applying Multiple Fixes

At one point, I was applying separate corrections for orientation flipping, lateral movement disappearance, and jump disappearance. All three issues stemmed from using the estimator's world output for position, height, and orientation.

I discarded all the fixes and unified the coordinate system. A constant rotation is calculated from the in-camera and world joint configurations to align with gravity. This rotation transforms the camera coordinate system estimation results, while translation uses the camera fit values.

The rotation is solved using the Kabsch method. The key is subtracting the center of mass each frame, which eliminates translation and drift, leaving only orientation.

# pipeline/frames.py:37-44
def camera_to_world_rotation(joints_incam, joints_world):
    a = (joints_incam - joints_incam.mean(axis=1, keepdims=True)).reshape(-1, 3)
    b = (joints_world - joints_world.mean(axis=1, keepdims=True)).reshape(-1, 3)
    return kabsch(a, b)

Since the camera is fixed, I assume constant rotation for the entire clip here, treating all frames as one least-squares problem. Finally, I add a yaw rotation to make the lens point along +z, completing the output coordinate system with y as up and y=0 as the floor.

In this coordinate system, downstream processes like "re-estimating orientation" and "restoring movement" become unnecessary.

Two Discarded Alternatives

I tested two other approaches that seemed better but ultimately discarded them:

Position from image plane: When the camera is tilted, actual depth movement leaks into height. In dance_full's approach, it was 0.16m lower
Depth from lowest foot ray-floor intersection: Performs poorly when the pelvis ray is nearly horizontal. For chest-height shots (most of this material), depth fluctuated ±17% (fit itself was ±4%)

In conclusion, the fit's own depth was the least bad option. Remaining errors are "depth error × sin(camera tilt)," which is a few centimeters in these materials. Compared to the world's 0.25-0.35m drift, it's sufficiently small.

Verify with Quantities Not Used in Fitting

Here's an effective verification pattern.

I wrote logic to transform hand estimation results (camera coordinates) to the rig's coordinate system. The transformation rotation is calculated by matching four torso points (shoulders and hips) between the two coordinate systems.

The challenge is verifying if this rotation is truly correct. You can't confirm its applicability to other parts using only the fitted points. Naturally, the rotation matches well when verified with the torso used for fitting. While not an exact match due to shape differences between the rig and actor, this verification is weak.

So I verified with the forearms, which weren't used in fitting. The camera-coordinate forearm orientation is transformed using this rotation and compared to the rig's forearm. In the desk material, the directional difference was 3.6°. Matching corresponding points and consistency across different parts are checked separately.

The implementation calculates rotation using SVD and adds a matrix sign to prevent mirroring. Without this, SVD fits can happily return flipped rotations.

Before Removing Based on Old Measurements, Retest if Reproducible Now

There's another temporal pitfall in measurement stories: your own past measurements.

Depth errors in fitting scale the entire translation, meaning depth mistakes also raise/lower foot height and produce tilted floors relative to depth. Actual measurements showed dance_full at +4° and plaza at -13°.

I didn't measure the actual floor tilt. The tilt appearing in estimation results is corrected to align foot heights. Floor plane fitting uses iterative reweighted least squares on the lower envelope, with 3 rounds, using only the lower 20% of residuals each time, and a ridge term (walking along one line makes x and z collinear).

# pipeline/frames.py:126-134
A = np.stack([x - x.mean(), z - z.mean(), np.ones_like(x)], axis=1)
sel = np.ones(len(low), dtype=bool)
lam = np.diag([0.05, 0.05, 0.0])   # ridge: a walk along one line leaves x and z collinear
for _ in range(rounds):
    As, ys = A[sel], low[sel]
    coef = np.linalg.solve(As.T @ As + lam * len(ys), As.T @ ys)
    res = low - A @ coef
    sel = res <= np.percentile(res, keep * 100)

At one point, based on an old measurement claiming "floor tilt estimation was counterproductive in all three materials," I decided to remove this processing and actually deleted it.

Later, when retesting with 12 clips containing tilts, the old measurement wasn't reproducible. The metric was the 5th-95th percentile width of the lowest foot height (ankle or toe, whichever is lower). Smaller values mean more consistent foot heights. 9 of 12 clips worsened without correction, 2 (desk1, desk1b) improved 5-8%, and the remaining clip's result wasn't recorded.

Material	With correction: Lowest foot height p5-p95 width (m)	Without correction: Same (m)
dance_full	0.112	0.144
Clip showing 29° tilt	0.195	2.98
desk1b	0.092	0.085

Based on these results, I restored the processing and recorded the retest conditions and process in the code. When making changes based on old measurements, first verify if they're reproducible with the current code.

Appendix: 2D Detection Consistency and Smoothing Side Effects

Here are two implementation points that come into play after aligning measurements, summarized briefly.

2D Detection Matches Image Positions Better

3D body fitting can be off by centimeters for hand and foot tips. In dance_full, wrist reprojection error was 20px (p50) to 40px (p90) - 3-7cm for a ~1000px actor. Hands that touch in the video appear 5cm apart.

In this material, ViTPose's 2D detection (already run by GVHMR as preprocessing) matched image joint positions better than reprojecting 3D fit points. However, 2D detection alone doesn't determine depth.

So I keep the 3D fit's depth but use 2D detection ray points as correction targets. In practice, correction amounts toward these targets are interpolated and smoothed. The code below uses camera coordinates (x right, y down, z depth), with p[:, 2] as depth.

# pipeline/frames.py:194-196
target = np.stack([(uv[:, 0] - cx) * p[:, 2] / f,
                   (uv[:, 1] - cy) * p[:, 2] / f], axis=1)
delta = _fill_and_smooth(target - p[:, :2], ok, window)
out[:, j, :2] = p[:, :2] + delta      # z (depth) untouched

Since delta is smoothed before addition, corrected points aren't strictly on the ray each frame. Two subtle but effective tweaks:

Skip frames with confidence <0.7. Missing frames are interpolated from neighbors and median-smoothed over 5 frames. Frame-level 2D detection jitter shouldn't affect the character
Skip waist and shoulders. Different "waist" and "shoulder" definitions in SMPL vs COCO. Elbows and knees are corrected after subtracting clip-wide median offsets

This is detailed work checking "which joints, which definitions, which coordinate systems" one by one. Being vague here leads back to the failures in the first section.

Smoothing Halved Deceleration

I was applying a 7-frame Savitzky-Golay filter as preprocessing - a noise reduction standard. Testing on dance_full showed:

Metric	With filter (7 frames)	Without
Wrist deceleration: \|acceleration\| p99 (m/s²)	32	64
"Stop" event count (times)	29	51
Noise median	6.5	10.7

The "noise" in row 3 isn't defined in the records - treat the ratio as indicative.

In this material, deceleration dropped from 64m/s² to 32m/s², and "stop" events from 51 to 29. While reducing noise, it may also weaken desired sharp movements. The event reduction rate doesn't directly equal "lost sharpness," but strike deceleration is clearly dulled.

I also measured kick peaks. For a roundhouse kick peaking in ~6 frames, a 7-frame window reduced shin angle by 5° and height by 0.04m. A 5-frame window reduced angle by 1.4°.

It's now off by default. If the estimator is noisy, re-enable via spec, but it wasn't a default processing step. The same topic appears in Part 5 (hand smoothing).

Key Points

When numbers align suspiciously, check measurement independence and rounding
Distinguish between approximate shapes being sufficient and needing actual mesh verification
Align not just point names but position definitions and coordinate systems when comparing
When applying multiple fixes, investigate common upstream causes
Before removing processing based on old measurements, retest if reproducible now

The checklist for questioning numbers has settled on these five:

Comparing the same points (by position definition, not just name)
Same coordinate system and scale (gravity direction established? conversion assumptions documented?)
Measurements independent (two values not created from the same source?)
Looking at current output (not relying on old logs or measurements?)
Verified with quantities not used in fitting?

Next Preview

Next time, with measurements aligned, we'll place feet on the floor. Starting from an estimation where the swinging leg penetrates the floor by 148mm, we'll cover how to detect ground contact and align feet and waist.

Series Installments

Part 1: Animating VRM from a Single Live Video
Part 2 (this article): Head-Chest Angle Difference of 0.000° in Every Frame - Broken Motion Measurement
Part 3: 1% Shorter Legs Resulted in 16° Bent Knees
Part 4: Arm Penetration Correction Jumps Front to Back
Part 5: Thumb "Placed" on Index Finger, Not Pushed Away
Part 6: Wrist Crease Caused by Skinning, Not Pose

Implementation & Sources

The materials and code in this article are based on squall01337/mixamo-llm-mocap (MIT). The coordinate system discussion for two-person comparisons comes from the forked repo's documentation. Coordinate unification, floor leveling, and 2D consistency were added later.

Code quoted in this article that was added after forking isn't publicly available. Assume retesting isn't possible when reading (quotes include file names and line numbers).

Code excerpts are for explanation - see the referenced files (pipeline/frames.py, pipeline/wrist_heading.py, pipeline/estimate_pose_gvhmr.py) for full implementations including initialization and helper functions.

Making a VRM Character Dance from a Single Live-Action Video: Fixing Leg, Arm, and Finger Clipping After Estimation

orca_forge — Thu, 17 Sep 2026 00:22:04 +0000

Creating a Pipeline to Make VRM Characters Dance from Live-Action Videos

We are building a pipeline to make VRM characters dance from live-action videos. However, when we applied the estimated motion to the character, the feet penetrated the floor, the arms intersected with the body, and the thumb passed through the index finger.

In our dance material, we were able to reduce the maximum penetration into the floor from 148mm on the left and 94mm on the right to 0.4mm on the left and 1.5mm on the right. While correcting the grounding, we also confirmed that the original motion was not overly trimmed.

The input is a video shot with a fixed camera, and the output is a VRM animation (VRMA) and a browser viewer. This series will introduce the process of measuring and correcting the defects that remain after estimation, divided into six parts. The first part will cover the overall pipeline and design decisions.

Comparing Before and After Correction with Numbers

We compared the before and after correction using dance material (1,789 frames, 60fps). The slip is the total of the entire clip, and the penetration is the maximum value. p90 is the 90th percentile when the values are arranged in order from smallest, and f represents frames. The numbers are not the direct output of the solver, but were measured by writing the correction back to the Blender rig and re-observing it.

Item	Before Correction	After Correction
Foot Slip (Total, Left/Right)	9366 / 8084 mm	1362 / 1537 mm
Penetration into the Floor (Maximum, Left/Right)	148 / 94 mm	0.4 / 1.5 mm
Foot Acceleration p90 (Left/Right)	6.8 / 9.7 mm/f²	7.2 / 9.6 mm/f²

The foot acceleration p90 was 6.8→7.2 on the left and 9.7→9.6 mm/f² on the right. While reducing slip and penetration, this indicator maintained a close value. However, with p90 alone, it cannot be determined whether individual steps or kicks were preserved. The slip did not become zero, and some correction remained at the boundary of the grounding interval.

The slip indicator selects the target vertex based on height conditions, so it cannot be determined whether the number of evaluation targets changed based on the total value alone. This point will be discussed in Part 3.

It's easy to "eliminate slip" by just stopping the feet, but that would also make the motion flat. The goodness or badness of the correction should be judged not only by whether the desired error is reduced but also by whether the quantity that should not be eliminated remains. The numbers that appear in this series are mostly in sets of these two.

What we originally wanted to do was simple: "make the VRM avatar dance with the same motion as the video we took." If we use commercial motion capture products, it would be faster, but we wanted to create it from a single video shot with a handheld camera without a suit or markers. Monocular pose estimation (a model that restores 3D humans from a single video) has become quite practical in recent years.

The estimation itself is surprisingly accurate. The defects that occur are after that, and what we actually observed were symptoms such as:

Feet penetrating or floating from the floor
Knees bending even when standing
Arms intersecting with the body
Thumbs passing through index fingers
Wrist skin being crushed like candy wrapper

This series is a record of measuring and correcting each of these defects one by one.

Note that the process of applying the estimated motion to a character with a different physique is called "retargeting." In this article, we refer to the process of reassembling the bone direction to match the character's bone length as "lifting," and the process of creating a pose from bone rotation as "FK application."

Flow from Estimation to Correction and Verification

0. Material video (fixed camera, full-body)
1. Estimation        Convert GVHMR result to joint points, pelvis height, and line of sight
1b. Coordinate system  Unify to "camera coordinate system" aligned with gravity
1c. Image integration  Reload wrist, ankle, elbow, and knee onto 2D detection ray
2. Score            Detect landing, takeoff, and impact numerically
3. spec            Support foot schedule and correction specification (JSON)
4. Lifting      Reassemble direction only, using rig length
5. FK application  Orient all bones on Blender, explore waist height
5.1 Range of motion  Keep all joints within anatomical range
5.1c Grounding      Detect foot grounding interval, solve foot trajectory and waist at once
5.2 Interference      Push arms away from body
5.3 Hand            Finger range of motion, place thumb above index finger
6. Comparison      Measure difference by eye, comparing with original video
7. QA          Numerical gate (defect, slip, drift)

We output the corrected animation to VRMA and verify it with a viewer.

The joint points in step 1 are not in the format directly output by the estimation model. An adapter in this repository composes MediaPipe-style 33 points from SMPL joints and facial mesh vertices. The downstream process only looks at these 33 points, so even if the estimator is replaced, the same code can be run from there.

Among these, steps 1b, 1c, and 5.1 onwards are layers that were added after actually moving and finding them to be insufficient. It wasn't in this form from the beginning.

Transferring Bone "Direction"

If we apply the estimated joint positions to the character as is, defects will occur because the physique is different. On the other hand, a method that adds and subtracts differences also breaks down when the correction amount becomes large. The original documentation has a record that with a 20cm correction, the hand interval shrunk from 0.25m to 0.08m, and the wrist angle changed from 7° to 123°.

In this pipeline, we use the estimated bone direction and reassemble it using the actual measured value of the rig length.

# pipeline/lift_to_mixamo.py:441-449
l_elbow = l_arm + seg(mp, i, "left_shoulder", "left_elbow") * LEN["l_arm"]
l_wrist = l_elbow + seg(mp, i, "left_elbow", "left_wrist") * LEN["l_fore"]
l_hand  = l_wrist + hand_dir(mp, i, "left") * LEN["l_hand"]

l_knee  = l_upleg + seg(mp, i, "left_hip", "left_knee") * LEN["l_upleg"]
l_ankle = l_knee  + seg(mp, i, "left_knee", "left_ankle") * LEN["l_leg"]

seg() is a unit vector between two points, and LEN is the actual measured length of the rig. The position of the shoulders and hip joints is also not taken from the estimation, but the offset from the rest pose is applied to the estimated base of the waist.

This method ensures that the bone length does not shift by definition. However, the problem of the character's hands and feet being thicker than the performer remains. The bone length is summarized in rig_profile.json, and when the character is replaced, only this one file is replaced.

Avoiding Blender's IK Constraint and Writing to Rotation Key

With the rig and settings used this time, applying Blender's IK constraint caused the feet to fly away to -81m. Therefore, we configured it to write the corrected pose as the rotation of each bone. We also use our own analytical 2-bone IK (intersection of two circles) for calculations that solve leg and arm positions. Putting the feet on the floor is also done by "exploring the waist height to find the height at which the support foot touches the ground," rather than using a constraint.

There are other premises that should be noted regarding the rig.

The armature is rotated 90° around the X-axis and scaled to 0.01. If we create a pose matrix in world coordinates and enter it into the pose bone, the hands will fly away 100 times. The target is to decide in armature space (centimeter system).
The only moving bone is Hips. All others are quaternions, and location is reset to 0 every frame (old keys will remain and cause defects).
World positions are read after view_layer.update() using (arm.matrix_world @ pose_bone.matrix).to_translation(). When comparing positions, the coordinate space and evaluation timing must be aligned.

These points will be discussed in Part 6.

Caching Estimation Results and Repeating Correction on CPU

GVHMR took 52 seconds to estimate 300 frames on an RTX 4080 Laptop (VRAM 12GB), with a VRAM peak of 4.2GB. We cache the estimation results and repeat the correction using NumPy and Blender.

The only part that uses the GPU is the pose estimation. There are two backends for hand finger estimation: MediaPipe, which runs on CPU, and WiLoR, which runs on GPU. If WiLoR is chosen, the GPU is also used for this part.

This division has practical benefits.

Estimation only needs to be done once, and the results can be cached. Subsequent trial and error do not compete for the GPU.
Grounding processing is written only with NumPy, so changing a constant and re-measuring takes only a few seconds.

If it takes several minutes to make one round trip with Blender, this difference becomes the difference in the number of trial iterations. In fact, the grounding processing is divided into three segments: "Blender observation → NumPy solution → Blender writing," and only the middle part was repeated dozens of times to decide on the constants.

For reference, the actual time for each segment when passing a 4-minute material (14,373 frames, 60fps) is as follows. Hand and facial expression estimation are not included in this table.

Segment	Time
Pose estimation (GPU)	Several minutes
Lifting + FK application	About 10 minutes
Grounding processing (observation → solution → writing → re-observation)	About 35 minutes
Arm interference (NumPy part is 3 minutes)	About 25 minutes
Thumb	About 1 hour
VRMA output	About 30 minutes

The total is about 2.5 hours.

spec Includes Material Information and Correction Specifications

Each clip has one JSON spec that summarizes material information, support foot schedule, standing pose adjustment, and override of estimated intervals that went wrong. Automatically generated content is also corrected as needed.

{
  "name": "fountain",
  "landmarks": "plates/fountain/landmarks.json",
  "src_fps": 29.97,
  "dst_fps": 60,

  // Support foot. Intervals are all written in the frame number of the original video.
  "plant": [
    { "src": [1, 70],    "support": "left"  },
    { "src": [71, 126],  "support": "both"  },
    { "src": [127, 129], "support": "none"  },   // Floating
    { "src": [130, 145], "support": "left"  }
  ],

  // Standing pose adjustment (knee in/out, stride, elbow)
  "stance": { "knee_in_pct": 3.5, "toe_in_deg": 14 },

  // Override estimated intervals that went wrong, with human reading
  "arm_overrides": [
    { "side": "right", "src": [76, 163], "ramp_src": 6,
      "wrist_local": [-0.17, -0.12, 0.33],
      "elbow_local": [-0.25, 0.05, 0.12] }
  ]
}

We are aware of the following three design considerations:

Unify frame numbers to the original video's frame number. When a person looks at the video and says "the right arm is strange at frame 27," it can be written directly into the spec. If the frame number after conversion to 60fps is inserted, it becomes easier to mix up the frame numbers. In fact, we made a mistake here.

Correction must always have an interval and a ramp. Corrections such as tightening the elbow only for a certain interval or overriding the arm position only for a certain interval will have a step difference at the boundary if switched abruptly. We use ramp_src to smooth it out.

There are scenes where human reading wins over estimation. In intervals where the arm is hidden behind the body, the estimation error becomes large. In such cases, it is faster for a person to look at the video and write it. However, only for that interval, and without touching other joints.

The support foot schedule is troublesome to write by hand, so we automatically generate it from the pelvis acceleration and correct only the necessary parts. When processing 41 materials in bulk, we did not make individual adjustments, and everything was automatic generation.

Compared Three Estimators and Returned to the First One

We set up two other estimators (PromptHMR-Vid, Human3R) for comparison. All of them can write landmarks.json in the same format, and the downstream process runs unchanged.

In this comparison, the slip and penetration after grounding processing were at the same level for the three estimators.

Estimator/Setting	Slip p90 (mm/f, Left/Right)	Penetration Maximum (mm, Left/Right)	Apply Error p50 (mm, Left/Right)
GVHMR	3.8 / 5.4	6.1 / 8.1	11 / 6
PromptHMR	4.5 / 4.7	5.7 / 2.1	5 / 4
PromptHMR raw	2.6 / 5.7	4.5 / 5.6	7 / 5
Human3R	4.1 / 2.9	2.2 / 1.9	5 / 4

We note the following three conditions. This comparison is for the same dance material passed at 30fps, and the fps and evaluation process are different from the table at the beginning. Both cannot be read as the same numerical value. PromptHMR raw is the estimation result without PromptHMR's own post-processing optimization (translation and camera re-fitting).

The remaining difference is the nature of the estimator itself. With this material and setting, PromptHMR-Vid had 1.3-1.7 times more jerky foot and waist acceleration, and the jump disappeared (no floating intervals remained) when the post-processing optimization was applied. Human3R estimated that the camera moved 3.5°/0.29m even though it was a fixed camera clip.

So, we stuck with the default GVHMR. Another thing we found out from this comparison is that none of the estimators can make the feet touch the ground. The lowest vertex of the shoe sole was floating from the ground by 45-85mm on average, and the frames where it touched the ground were less than 1%.

By the way, the initial version of this comparison table had an error. The other three columns were processed without "walk-away correction," leaving a difference of 0.4-1.1m, which appeared as 85-143mm of fake penetration. If the comparison conditions are not aligned, apparent differences will be generated in this form.

Output: VRMA and Viewer

The final output is VRMA (VRM Animation), and we are using a three.js viewer to play it back. We set up a local server and made it possible to switch materials with a pull-down.

What comes into effect here is the problem of which correction reaches which route. Bone pose corrections are written as animation keys to VRMA. Skinning weight corrections, which will be discussed in Part 6, belong to the avatar's mesh and are used in the corrected VRM in the viewer. Whether the effect of Blender's modifiers or constraints reaches depends on the output route and whether baking is performed.

We also had an accident due to the difference in routes. The processing of finger range of motion and line of sight was only in the rendering segment and did not run once in the VRMA creation route. The viewer had a character with "eyes that don't follow and fingers that bend more than 90°." We solved this by moving the processing to right after the arm interference.

What This Series Will Cover

Each part will be written to be readable independently.

Part	Content
Part 1 (this article)	Making VRM characters dance from live-action videos ── Correcting defects in feet, arms, and fingers that remain after estimation
Part 2	The difference in head and chest angles was 0.000° for all frames ── A story about how motion measurement was broken
Part 3	Shortening the leg by 1% made the knee bend by 16° ── A pitfall in VRM's grounding correction
Part 4	Arm penetration correction flips front and back ── Deciding the direction to escape using dynamic planning
Part 5	The thumb rests on the index finger, not pushed away ── Correction to maintain grasping and smoothing
Part 6	The cause of the wrist skin being crushed was skinning, not pose

The next part will start with the abnormality that "the difference in head and chest angles was 0.000° for all frames." It's a story about doubting measurement before motion when appearance and numbers contradict.

Supplement: The Foundation is Someone Else's Implementation

The skeleton of this pipeline (estimation → score → spec → lifting → FK application → QA, and comparison with the original video) is an implementation of squall01337/mixamo-llm-mocap (MIT). The part referred to as "fork origin" in this article refers to this repository. We explicitly state the reference to avoid confusion with repositories of the same name.

The parts that we added after forking, which are quoted in this article, are not published. Please note that it is not possible to reproduce and verify the implementation by looking at the repository (the quotes include file names and line numbers).

The unification of coordinate systems, range of motion, grounding, arm interference, hand processing, and migration to VRM are all parts that we added afterwards. The code quotes are necessary for explanation and are excerpted from the implementation. They are not code that can be pasted and run as is, but rather calculations that were taken out to illustrate the point.

Apology Messages Were Tainting the Next Search — A Loop of Failure Strengthening Itself

orca_forge — Wed, 16 Sep 2026 01:11:47 +0000

The Log of Voice Dialogue Avatar Had This Interval

👤 This page's content is explained      🤖 The corresponding description was not found.
👤 Can you hear me?                     🤖 (Same apology)
👤 Can you hear me again?               🤖 (Same apology)
👤 I can hear you.                      🤖 (Same apology)

4 consecutive turns returned almost the same apology. The more the user tries to confirm, the more the same answer is returned.

The Cause Was the Improvement I Made Earlier

A little while ago, I made the following improvement: "Detailed" phrases like "Tell me more about that" cannot be searched alone, so add the previous conversation (user's speech + avatar's answer) to the search. The measurement was also effective (0.602 → 0.741).

Looking at the log, it was like this:

[retrieval] Add the previous conversation:
  'Can you hear me? I apologize, but the corresponding description was not found.'

The apology sentence without grounds was also included in the search query.

The apology sentence has no topic. Not only that, but words like "corresponding description" and "another way of saying" pull the search in a completely unrelated direction. That's why the next one is also wrong. And the failed sentence is included in the next query.

The failure was reinforcing itself.

Don't Judge by String

At first, I thought of "detecting and excluding apology sentences". I stopped soon.

Apology phrases differ depending on the job profile. They can be set differently for window, support, and delivery, and can change during operation. Judging by string would create holes every time a new phrase is added.

Instead, I decided based on facts.

if not getattr(self, "_last_turn_grounded", False):
    a = ""     # Don't add answers from turns without grounds

"Whether there was a ground for that turn" is already judged by the system. It doesn't depend on phrases.

⚠️ I shouldn't have written this update in the recording process. Since conversation log recording is best-effort (it doesn't stop the conversation even if it fails), if an exception occurs, the update will also fail. Even if the recording fails, the presence or absence of grounds affects the search quality of the next turn. I moved it to the end of the turn.

⚠️ The apology path also needed to be marked. If I drop it, the apology sentence will be included in the next query.

Another Thing That Went Against the Facts

While investigating the same 4 turns, I noticed another problem. The page text was not held even once during that time.

[page] Use the full text  ← This log didn't appear even once during the 4 turns

I was answering "This page does not contain the corresponding description" without holding the page text. This is against the facts. Whether the visitor can't read it or it's not there after reading is completely different for the visitor.

No text or index → "I'm currently reading the page content. Please wait a moment."
Text exists but no match → "The corresponding description was not found on the page I'm currently viewing."

⚠️ I almost made another mistake. The synthesis audio reuse key didn't distinguish between phrases. Since the standard response reuses audio by caching, if left as is, the audio for "reading" would play when intending to say "not found". I split the key by phrase.

Result

👤 What's the weather like today?                → 🤖 (Try to answer the page content)
👤 Tell me about the weather.                   → 🤖 This page does not contain weather information, so I apologize.
👤 (Next question)                             → 🤖 (Answer normally)  ← Apology doesn't chain

Generalizable Things

Designs that return output to input amplify when they fail.

Using conversation history as context is a natural design. However, there are two types of "avatar answers":

Answers with grounds     … Contain topics. Useful as context
Apology answers without grounds     … Contain no topics. Rather, distort the search

I should have used only the former. The system already had the material to distinguish (whether there was a ground for that turn), but I was treating answers as one thing.

Similar structures exist elsewhere. Using summaries as input for the next summary, mixing generated data into learning data, using search results as the next search query — all of them "get better when good, worse when bad". A mechanism to stop when bad is needed.

And the judgment to stop should be based on facts, not strings. Phrases change, but "whether there was a ground" doesn't change.

Series: Until the Voice Dialogue Avatar Answers Correctly

This article is the last part of Part 2: Understanding Language.

← Previous: The one line at the end of a huge prompt was ignored four times
→ Next: Not all utterances are questions

Series of 8 articles

Part 1: Stopping Sound

There were two types of events with the same name
The self-echo countermeasure never fired
A finger on the speaker was breaking the echo canceller

Part 2: Understanding Language

Part 3: Judging

Not all utterances are questions
I thought I was measuring, but I was measuring something else

The notes that led to this insight are summarized in Improving the Response Quality of Voice Dialogue Avatars.

My smartphone fingers were breaking the echo cancellation — speech cutting out due to non-verbal sounds

orca_forge — Wed, 16 Sep 2026 00:40:52 +0000

The Avatar that Stopped Working on iPhone Safari

The voice dialogue avatar that worked fine on PC had an issue on iPhone Safari.

What the Logs Showed

Looking at the judgment records, a pattern emerged.

13:39:52.843  Speech start detected (VAD)
13:39:52.848  Bot stopped speaking   ← The avatar was stopped 5 milliseconds later

All five instances showed the same pattern. The decisive factor was that no transcription was output.

The VAD judged that "speech has started" and stopped the avatar's speech as an interrupt. However, the sound was not recognized as a word, and no turn occurred. There was also a period where the avatar started speaking and was stopped repeatedly at 2-second intervals.

Mechanism

The user's guess was " sudden decrease in volume". To clarify further, when you cover the speaker with your finger, the sound's loudness and resonance change. The browser's echo canceller learns the relationship between the "sound sent to the speaker" and the "sound returned to the microphone" and cancels it out. If this relationship changes suddenly, it cannot be completely cancelled out, and the residual sound leaks out.

The VAD sees this as human speech. The reason there was no transcription output was that it was not a word.

The reason it didn't happen on PC was not because the countermeasures were effective, but because the browser's AEC was canceling out the echo during playback.

The Gate I Created is Not Effective for This Symptom

I had already implemented a gate as a countermeasure against self-echo, which drops speech immediately after the sound finishes. However, this is not effective. By design, speech during playback is allowed to pass through. This is done to prevent interrupts from being killed.

Self-echo    … After the sound finishes (AEC tail) → Gate's responsibility
Incorrect interrupt    … During playback                   → Outside gate's responsibility

Echo and incorrect interrupts occur at different times. A different countermeasure is needed.

How the Industry Handles This

I investigated how web services that keep the microphone open for conversation handle this issue.

LiveKit's publicly available numbers were specific. They don't trigger an interrupt immediately, but instead look at the first few hundred milliseconds of speech (the rise, sustain, and rhythm of the waveform) before making a judgment.

Listen to 216ms of audio on average before interrupting
This rejects 51% of VAD-based interrupts
The judgment model's inference itself is less than 30ms

In other words, delaying the judgment by 200-300ms to examine the audio's characteristics was the answer.

On the other hand, no one has a silver bullet for echo itself. Deepgram's official documentation only says "leave it to the browser's standard", and there is no mention of mobile or speakerphones. Depending on the client-side AEC is a common premise.

Implementation: Make VAD Less Sensitive During Playback

Since all observed misfires were short sounds (no transcription output), we can filter by duration.

During the avatar's speech, increase the continuous time required to recognize speech from 0.2 seconds to 0.5 seconds
Do not change the normal time ＝ The response time of the turn remains unchanged. Only the interrupt becomes less sensitive

The framework we were using had a mechanism for interrupt strategy, but waiting for the transcription to be ready would delay the interrupt by about 1.6 seconds (1.2 seconds of silence waiting + speech recognition). This contradicts the requirements for responsiveness, so we didn't adopt it.

⚠️ The VAD parameter setting API could not be used. Because it initializes the internal state, calling it while the user is interrupting and speaking will cut off the speech. Moreover, the timing you want to call it (when the avatar's speech stops) is precisely that moment. We changed it to directly overwrite the threshold frame count.

⚠️ This attribute does not exist immediately after creating the object. It is born when the sampling rate is decided (at the time the pipeline starts). We were testing with a fake object, so we didn't notice it, and it crashed the moment we added the item to test with the real thing. Until then, we were suppressing exceptions and doing nothing.

Results

                    Before countermeasures    After countermeasures
Interrupts during playback    5 instances      0 instances
Self-echo disposal      0 instances      1 instance (speech 7ms after playback ended)
Real speech           ―        3 instances all passed (5.0 seconds / 10.5 seconds / 17.2 seconds later)

We also left evidence of the countermeasures being effective in the logs.

Countermeasure against incorrect interrupts: increase the speech judgment time during playback from 0.2 seconds to 0.5 seconds

This line not being output means the countermeasures are not working. Since we were once deceived by this, we made sure to output both the evidence that it worked and the fact that it didn't work.

Generalizable Points

"Audio input" and "speech" are different events, and VAD can only judge the former.

VAD cuts speech intervals based on audio energy and features. It cannot determine whether a person intentionally spoke. Therefore, coughs, noise, machine contact sounds, and echo residuals all become "speech".

There are two directions for countermeasures.

Cut by time (ignore sounds that don't continue for a certain time) — deterministic, light, and delayed by several hundred milliseconds
Cut by characteristics (look at waveforms or rhythms) — high accuracy, requires a model

The difficulty level also changes depending on the environment. Desktop PCs with headsets are easy, while smartphones with speakers are difficult. The device moves, the way it's held changes, and the speaker is covered with a finger — the echo canceller's premise that "the sound path is constant" does not hold for mobile devices.

"Since it worked on PC, it's fine" is not applicable to voice.

Series: Making the Voice Dialogue Avatar Answer Correctly

This article is the last part of Part 1: Stopping the Sound.

← Previous: The gate that never fired
→ Next: The three kinds of "it"

Series of 8 articles

Part 1: Stopping the Sound

There were two types of events with the same name
The gate that never fired
A finger on the speaker was breaking the echo canceller ← Now here

Part 2: Understanding Words

Part 3: Judging

Not all speech is a question
I thought I was measuring something, but it was actually something else

The notes that led to this insight are summarized in The Quality of the Voice Dialogue Avatar's Response.

The single line at the end of the massive prompt was ignored all four times

orca_forge — Wed, 16 Sep 2026 00:09:18 +0000

Conversation with the Voice Dialogue Avatar

I had the following conversation with the 24-hour avatar.

👤 This 24-hour avatar is working with what kind of mechanism?  → Explain the whole thing
👤 What was the Pod startup you mentioned earlier?                → Explain the whole thing again
👤 Tell me more about it.                                         → Explain the whole thing again

When I tried to dig deeper, I got the same summary back. The conversation log had a repetition flag set for 5 out of 13 turns.

The anaphora judgment was working correctly (recognizing "earlier" as a remote context and adding the corresponding past conversation to the search query). What was off was the response.

A Simple Countermeasure

I added instructions right after the latest user utterance — the most effective position in the prompt.

(The visitor is listening to the continuation of the previous conversation. Without repeating the overall image or summary that has already been stated, 
please describe the specific part that the question is pointing to, following the corresponding section of the text. 
Do not say the same thing as the previous response again.)

I chose this position for a reason. In the same project, I had placed speech-linked pointing instructions and tool call reminders in this position, and they were effective. As the history grows, the model starts to omit tags by simulating the "past self" that didn't output tags, so I reminded it to output tags at the end of each turn — a countermeasure that actually worked.

I deployed and tried it.

Nothing Changed

👤 I'll tell you more about it.    → Explain the whole thing
👤 Can you tell me more about it?      → Explain the whole thing again (repetition flag)

First, I checked if the instructions were actually being sent. They were sent. All four times.

Question: Tell me more about it.

(System instruction: For each item/section, ...)

(The visitor is listening to the continuation of the previous conversation. Without repeating the overall image or summary that has already been stated, 
please describe the specific part that the question is pointing to, following the corresponding section of the text. 
Do not say the same thing as the previous response again.)

The prompt included the full text of the page, 8,207 characters.

The Answer Was Written in the Same File

While searching for the cause and reading the code, I came across a comment that I didn't write.

The same phenomenon was recorded in the same file. And the existing code was escaping to a dedicated small call for that reason.

I fell into the same trap, in the same place, after reporting that I had placed it in the "most effective position".

Separating Responsibilities

I made the digging deeper the same shape. I stopped passing the full text and instead passed only the relevant section.

【Already mentioned】Previous conversation (up to 600 characters)
【Main text】Relevant section only (up to 2,400 characters)
【Question】Current utterance

The instruction is "Without repeating what has already been mentioned, describe the specific content that has not been touched yet in 2-4 sentences".
I made it a single task.

The number of calls does not increase. Since it replaces the existing call, it's actually faster because the input is smaller.
For turns where there is no relevant section or previous conversation, it follows the same route as before, so it doesn't get worse.

It Was Still 0

I deployed and tried it, and the digging deeper call didn't fire at all.

There were two routes to get the material.

Full text route      … Turns that use the full text of the page. I had implemented it here
Search route      … Turns where the page search hits. **This was missing**

Ironically, the turns where the search hits are the ones where the material for digging deeper is most available.
I wasn't picking it up. The previous time it happened to fire was only when it went through the full text route.

This wouldn't be found in testing. I only found out when I saw the 0 fires in the actual machine log.

Result

15:12:28 "Explain this page"
      🤖 This page is about the system that continues to broadcast with an AI avatar, its quality assurance...

15:14:06 "Tell me more about it."
      [dig] Section='things that only come out after a long time' Already mentioned=322 characters Main text=1519 characters
      🤖 "Bugs that only come out after a long time" include things like memory leaks, or periodic tasks that are missing, 
         which only appear as time passes...

It entered a specific part instead of repeating the summary. The repetition flag was not set either.

Generalizable Things

It's not the position of the instruction, but what you pass with it that determines its effectiveness.

"It's most effective to put it at the end of the prompt" is conditionally correct. If the surroundings are short, it's effective.
If you put it with an 8,000-character main text, that one line becomes less than 1% of the context.
The model is pulled into the dominant task of "summarizing the given data".

And the more ambiguous the input, the more the model escapes into a summary. The same thing happened when the state was the same.

👤 What's the weather like today?          → 🤖 This page is about the system that continues to broadcast with an AI avatar... (summary)
👤 Yes, please wait.  → 🤖 This page is about the system that continues to broadcast with an AI avatar... (summary)

Even for utterances that aren't questions, the same summary is returned. The latter was a state where the same answer was returned no matter what.

The direction of the countermeasure was not "to make the instruction stronger", but "to make the task single and narrow down the material".
The less you pass, the relatively larger the instruction becomes.

Another thing. The answer to the same problem may already be in the codebase. In my case, it was a comment that I found after implementing the countermeasure and deploying it, and it didn't work.
Searching comments in the codebase for "has this symptom occurred before" is more cost-effective than I thought.

Series: Until the Voice Dialogue Avatar Answers Correctly

This article is part of Part 2: Understanding Language.

← Previous: ""It", "this page", and "earlier" were different things
→ Next: Apology words were contaminating the next search

Series of 8 articles

Part 1: Stopping the Sound

There were two types of events with the same name
The self-echo countermeasure had never fired
The smartphone's finger was breaking the echo canceller

Part 2: Understanding Language

""It", "this page", and "earlier" were different things
A single line in a huge prompt was ignored four times ← Now here
Apology words were contaminating the next search

Part 3: Seeing Through

Not all utterances are questions
I thought I was measuring, but I was measuring something else

The notes that became the basis for this knowledge are summarized in The Response Quality of Voice Dialogue Avatars.

Mic Doesn't Return Even After Reconnecting — It Was the Old Microphone Marked for Disposal That Was Enabled

orca_forge — Tue, 15 Sep 2026 10:36:14 +0000

Operating a Voice Dialogue Avatar Embedded in a Website

The avatar connects when the page is opened and responds when spoken to. Behind the scenes, WebRTC connects the browser and voice pipeline.

However, after restarting the server, the avatar stopped responding.

👤 (speaking)
🤖 (no response)
👤 (speaking again)
🤖 (no response)

Visually, everything seems fine. The avatar is standing, and the microphone button is still ON. Visitors won't notice that it's "broken" until they reload the page.

Initially, There Was No Reconnection Mechanism

The first thing I checked was that there was no code to reconnect after a disconnection.

The connection process is only called once when the page is opened, and there is no path to reconnect after a disconnection. This means that not only server restarts but also events like subway rides, Wi-Fi switching, or waking up from sleep can cause the same issue. With each deployment, visitors who happen to be speaking to the avatar at that moment will be met with silence.

I added a reconnection process with exponential backoff.

const waits = [1000, 2000, 4000, 8000, 15000, 30000];
const wait = waits[Math.min(retryRef.current, waits.length - 1)];
retryRef.current += 1;
setTimeout(() => reconnectRef.current(), wait);

It will not timeout after the last 30 seconds, but repeat indefinitely. I intentionally did this because visitors are likely to leave the page open, rather than coming back later. Since the cost of reconnection is almost zero, there's no reason to give up.

After deploying and restarting the server, I got this log.

[widget] Connection lost — reconnection in 1 second (1st attempt)

...but it stopped there. The first log appeared, but the second one didn't. It didn't reconnect.

Non-Idempotent Function Was Called as If It Were Idempotent

The connection process started like this.

const connect = useCallback(async () => {
    if (pcRef.current) return;      // ← here
    ...

If an old connection object remains, it will return without doing anything. As a function called only once when the page is opened, this guard is correct to prevent double connections.

However, when reconnecting, it's precisely the "old connection" that remains. The connect() function was quietly returning without doing anything every time. No error, no log.

I changed it to call disconnect() before reconnecting. This time, it connected successfully.

15:09:43  Server restart
15:10:16  Voice pipeline detected disconnection
15:10:28  Reconnection (12 seconds later)

The conversation history remained intact. Since the conversation ID is stored in the browser's sessionStorage, reconnection allows the conversation to "continue from where it left off."

The Microphone Alone Didn't Come Back

Even though it reconnected, speaking to the avatar yielded no response. The microphone button was still ON.

I confirmed this on the server-side log, where I had implemented a diagnostic that outputs the volume and "human voice-like" detection every 30 seconds.

[vad] Last 30 seconds: 1500 judgments, 0 speeches, 0 near-misses (average voice 0.00, volume 0.00)

The volume remained 0.00. No sound was reaching the server. The microphone track is obtained when connecting and disabled by default. The button is enabled when pressed. So, after reconnection, if the microphone was originally ON, it needs to be re-enabled.

I Read the Source and Missed It Three Times

This is where the embarrassing part begins. I read the code, formed a hypothesis, fixed it, and missed it three times.

Suspected cause	Reason	Result
Limit on usage	There's a process that mutes the microphone when the limit is reached	The limit wasn't being applied
8-second status check misjudging	I had added a process to verify the reconnection later	It was unrelated
Reconnection process's internal order being swapped	Asynchronous processes were running in parallel	This was also incorrect

All three times, I "read the source and chose a plausible explanation." Each one seemed to make sense when reading the code, but none of them were actually happening.

On the fourth attempt, I gave up and added diagnostic logs to see who was muting the microphone and when.

One Log Line Solved It

The returned log was this.

[widget] After microphone recovery: {called: true, enabled: [true]}
[voicepipe] Obtaining new microphone track (default is disabled)    ← ★this is later
[widget] 2 seconds after microphone recovery: {enabled: [true], returned: true}

The microphone was being enabled before obtaining the new track.

In other words, the track I was enabling was the old track that was about to be discarded. After that, connect() created a new track, which started disabled by default.

It wasn't that someone was muting it later; the order was simply reversed. My three hypotheses were all about "who muted it," but no one was muting it.

The Premature Reason Was a Lying State Variable

Then, why did the enablement run before the new track was obtained? The condition was written like this.

if (voicepipe.status === 'connected') { ... }   // enable when connected

The update of the connection status was like this.

pc.oniceconnectionstatechange = () => {
  const s = pc.iceConnectionState;
  if (s === 'connected' || s === 'completed') {
    setStatus('connected');
  } else if (s === 'failed' || s === 'disconnected' || s === 'closed') {
    if (s === 'failed') setStatus('failed');    // ← only when failed
    ...

The status remains 'connected' even when 'disconnected' or 'closed'. The status is only updated to 'failed' when it actually fails.

So, the condition "enable when connected" was already met immediately after disconnection. I was waiting, but I wasn't actually waiting.

This state variable isn't wrong. It's reasonable not to immediately set the status to 'failed' when disconnected, as it might recover. The problem is that I used this variable, which has a specific meaning, to judge "is it currently connected?"

The fix was simple. Instead of looking at the state variable, I waited for the connection process to complete.

voicepipe.disconnect();
await voicepipe.connect();                              // wait for completion
if (micOnRef.current) voicepipe.setMicEnabled(true);     // then enable

The connect() function returns after obtaining a new track and completing the offer/answer process, so at this point, I can be sure to get the new track. I changed from "looking at the status and judging" to "executing in order."

It Worked

I confirmed it by restarting the server.

15:10:28  Reconnection (45 seconds after server restart / 12 seconds after disconnection detection)
15:17:42  [vad] Volume 0.51, 0 speeches      ← microphone is alive and waiting
15:18:12  [vad] 31 speeches, voice 0.03        ← responded to speech
15:17:49  [Talk TTS] 'Hello. What brings you here today?'

As a byproduct, I can now determine the microphone's status from the server-side log alone.

	Volume
Microphone OFF	0.00
Microphone ON	around 0.5

After realizing this, I no longer need to ask visitors to open their browser's console. I can verify the issue from the server-side log when they report that the microphone isn't responding despite being turned on.

Takeaways

Be careful not to call non-idempotent functions as if they were idempotent. The if (already exists) return; guard is correct for initialization, but it becomes a "quietly do nothing" function when called from the "retry" path. Since it doesn't produce an error, it takes time to discover.
State variables might only be looking at a part of the failures. The status variable this time was meant to indicate "failed or not," not "is it currently connected?" The name looks like the latter, but the processes that use this variable as a condition will all be wrong. If you want to wait for something, it's more reliable to await the completion rather than looking at the state.
Hypotheses formed by reading the source can be wrong. I was wrong three times. Diagnostic logs solved it in one attempt. Code only teaches you "what can happen," not "what happened." If you come up with three plausible hypotheses, it's likely that you can't figure it out just by reading the code.
Reconnection is not just about server restarts. Before implementing reconnection, I organized the causes of disconnections.
- Deployment, pod restart, node replacement
- Wi-Fi ↔ mobile network switching, subway, elevator
- Smartphone sleep and wake (background connections may be terminated)
- Intermediate NAT or TURN timeouts (may drop after prolonged silence)

While server-side issues can be reduced with operation, the last three cannot be reduced. Investing in "reconnect even if disconnected" rather than "don't disconnect" is the right approach.

'That,' 'this page,' and 'that one' are different: 1 win and 3 losses by treating pronouns uniformly

orca_forge — Tue, 15 Sep 2026 10:04:26 +0000

An avatar that interacts through voice receives the following utterances:

👤 Are there any articles about streaming on YouTube?   → Article identified
👤 Can you explain that in more detail?               → Only responds with "Yes, I'll explain in detail."
👤 Wait, you're not explaining in detail?           → Only responds with "Sorry, just a little more..."

Despite identifying the subject in the previous turn, the context is lost in the next turn. While this is the most natural way to continue the conversation, it's the weakest for search. The phrase "Can you explain that in more detail?" doesn't contain any searchable terms.

Naive Solution and Results

I modified the system to include the user's previous utterance in the search query. The utterance itself, which is passed to the response, remains unchanged (rewriting it causes the model to respond to things the user didn't say).

After running this for a while, I re-evaluated it on a real site (38 pages).

Utterance	Baseline	With Previous	Expected
Can you explain that in more detail?	0.435 ❌	0.602 ❌	Correct answer was 2nd. Retrieved the homepage
What about the avatar discussion earlier?	0.597 ❌	0.594 ❌	Retrieved a different avatar article
What other articles are on this site?	0.489 ✅	0.429 ⚠️	Score decreased when added
Why is silent streaming necessary?	0.627 ✅	0.579 ❌	Correct answer dropped when added

The result was 1 win and 3 losses. The intended fix worsened the results in 3 out of 4 cases.

Three Types of Deictic Expressions Mixed

Upon examining actual conversation logs, it became clear that "this" and "that" refer to completely different things.

👤 What's on this page?        ← Current screen
👤 What other articles are on this site?        ← Current site
👤 Can you explain that in more detail?                   ← Previous interaction
👤 What about the avatar discussion earlier?               ← Interaction from a few turns ago

In Japanese grammar, these are distinguished as situational deixis (referring to something present in the speech situation) and textual deixis (referring to a previously mentioned linguistic expression). In a voice + screen UI, the situation = the screen.

Textual deixis further splits into two types: referring to the immediately preceding interaction and referring to an interaction from several turns ago.

Type	Example	Context Needed
Situational Deixis	This page / This site	Don't add. The answer is on the current page
Immediate Textual	That / It / In more detail / Continue	Add the full previous exchange
Distant Textual	Earlier about X / The X example	Search history for X and add only the relevant exchange
(Interrogatives)	Why / How / What	Not deictic. Adding drops the correct answer

Including interrogatives in the list caused the deterioration in the 4th example. "Why is silent streaming necessary?" is an independent question with no carryover.

What Should Be Added Isn't the "User's Utterance" but the "Full Exchange"

In "Can you explain that in more detail?", "that" typically refers to what the avatar just explained, not the previous question.

Query Construction	1st	Score
Baseline utterance only	❌ Unrelated article	0.435
Previous user utterance + current	❌ Homepage	0.602
Full previous exchange + current	✅ Correct	0.741
Previous avatar response + current	✅ Correct	0.729

Only when the full exchange (user's utterance + avatar's response) was added did the correct answer rank first.

For Distant Textual Deixis, "Sliding Window" Performs Worse Than Not Adding

For "What about the avatar discussion earlier?", where the topic was 3 turns prior:

	1st	Score
Baseline utterance only	❌ Different avatar article	0.597
Immediate 1 exchange + current	❌ Same as above	0.587
Immediate 2 exchanges + current	❌❌ Unrelated streaming article	0.564
Immediate 3 exchanges + current (includes correct)	✅ Correct	0.732
Only relevant exchange + current	✅ Correct	0.813

Three things became clear:

"Earlier" doesn't reach with just the immediate exchange. The uncertainty of how far back to go is inherent in this expression.
Partial backtracking performs worse than not adding. It gets dragged down by intervening topics.
Adding only the relevant exchange is better than adding everything (0.813 vs 0.732). Extra context dilutes the query.

Fortunately, this utterance contains its own clue: "Earlier about avatar". Searching the conversation history for exchanges containing "avatar" suffices. Matching with 2-gram character overlap was enough; no morphological analysis or additional embeddings were needed.

⚠️ Failed attempt to embed extracted terms in the prompt. The implementation dropped function words at the character level, turning "それをもうちょっと詳しく" into "うちょ詳". While usable as a search key, it's unreadable as Japanese. Since the question itself is already in the prompt, there was no need to reinsert it.

"In Detail" Was Being Truncated by Character Count

Another finding from real data came from user feedback:

Verified with 22 real data points:

	Count
Contains "in detail" / "specifically"	22 (all)
Accompanied by request (explain / tell)	12
Accompanied by intention (want to know / want to ask)	2
Neither	8 (36%)

While the meaning aligns with the feedback, one-third were omitted on the surface.

About creating a stream, in detail.
Specific steps are.
Specific content, pipeline about.

The intent is clearly a request, yet the verb is dropped (nominalization). This is common in Japanese, similar to zero anaphora. Making "accompanied by a request or intention" mandatory would drop one-third of cases.

Meanwhile, "in detail" / "specifically" appeared in all 22 cases, making the terms themselves the most reliable indicators.

This verification also uncovered a larger flaw: 4 out of 22 cases exceeded 25 characters (longest was 70 characters).

No, not that, but about how maintaining 30FPS with set intervals gradually desynchronizes lips and audio, I want to know more about that.

The decision to add context was tied to the decision for follow-up. This included a 25-character cutoff to catch "short utterances that can't be searched alone." Thus, these weren't treated as follow-up requests.

"Adding context" and "indicating follow-up" are separate dimensions. Long utterances can be searched independently but still require follow-up instructions.

Results

Utterance	Before Fix	After Fix
Can you explain that in more detail?	0.602 (incorrect)	0.741 (correct)
What about the avatar discussion earlier?	0.594 (incorrect)	0.832 (correct)
What other articles are on this site?	0.502 (correct)	0.483 (correct)
Why is silent streaming necessary?	0.624 (correct)	0.627 (correct)

Zero LLM round trips, with no change in response time.

Generalizable Insights

In conversational search, this process is called Conversational Query Rewriting. It handles two phenomena: anaphora ("that" referring back) and ellipsis (omitted terms), with the latter being a specialized research area in Japanese as "zero anaphora."

Industry reports suggest over 60% of follow-up utterances contain unresolved anaphora or implicit context. Rewriting can be handled by smaller models, with p95 under 100ms.

However, research also shows "Not All Queries Need Rewriting." Rewriting queries that already contain sufficient information reduces dense retrieval performance. The recommendation is to default to no rewriting in stable domains and use search confidence as a proxy when selectively rewriting.

My implementation decided based on "utterance appearance" (character count and word list), causing deterioration in 3 out of 4 cases.

In voice + screen UIs, one research assumption changes: Unlike text-only conversational search, "this" can refer to the screen. In real data, many deictic expressions pointed to the screen, not the conversation. This distinction directly corresponded to Japanese grammar's situational/textual deixis.

Series: Making Voice Interaction Avatars Respond Properly

This article is Part 2: Decoding Language.

← Previous: Smartphone Fingers Were Breaking Echo Cancellers

→ Next: The Line Added to the End of a Huge Prompt Was Ignored All 4 Times

Full Series (8 Articles)

Part 1: Stopping Sound

There Were Two Events with the Same Name
The Echo Countermeasure Never Activated
Smartphone Fingers Were Breaking Echo Cancellers

Part 2: Decoding Language

“That,” “This Page,” and “Earlier” Were All Different ← You are here
The Line Added to the End of a Huge Prompt Was Ignored All 4 Times
Apology Words Were Contaminating the Next Search

Part 3: Discerning

Not All Utterances Are Questions
What I Thought I Was Measuring, I Wasn’t

The insights are summarized in the notes on Voice Interaction Avatar Response Quality.

Echo Issue Went Unnoticed as It Never Triggered

orca_forge — Tue, 15 Sep 2026 09:32:36 +0000

The Voice Dialogue Avatar Was Responding to Its Own Utterances

🤖 (filler) "Yes, I'll check, please wait a moment."
STT: [Yes, I'll check, please wait a moment.] Confidence -0.306
🤖 "I'll check, please wait a moment."

It's not fabrication, but correct transcription, so it can't be dropped due to confidence. The confidence threshold is intentionally disabled (the true identity of low confidence is surrounding environmental noise, which can lower the confidence of real utterances to -0.826).

There was no defense on the server side, and only the browser's echo canceller was relied upon.

Made According to Convention

Investigation revealed that this issue has a established solution: discard input for a certain period of time after utterance playback ends.
300ms was insufficient, and 500-800ms was required (AEC tail).

Input during playback is not discarded. Discarding it would prevent interruption, which is the responsibility of the echo canceller.

I implemented this and wrote tests.

OK Utterance starts immediately after playback ends (self-echo) → Discarded
OK Utterance starts 600ms after playback ends (within AEC tail) → Discarded
OK Utterance starts outside the window (real response) → Passed
OK Utterance starts during playback (barge-in) → Passed
OK All 3 intermediate results are discarded
...
8/8 Passed

I also ran it in a container with the real library. 8/8. Deployed.

Zero Incidents in Actual Machine

A user verified it on their smartphone and collected judgment logs.

echo-gap breakdown:
  (empty)

Discarded count: 0

The conversation was successful. The avatar uttered 18 times, and the user started speaking 17 times. Yet,
the gate judgment was never reached.

The setting was enabled. The pipeline was entered. The file contents matched the source hash.

The cause was that the frame type being observed was different. The frame type that VAD outputs for utterance start and the frame type that the user aggregator outputs were different, and I was looking at the latter. The aggregator is downstream of this gate. Things created downstream do not flow upstream.

The gate was operating as if it didn't exist.

Why the Tests Were Green

It was only natural. The tests were using frames I created and passed through.

await gate.process_frame(UserStartedSpeakingFrame(), DOWNSTREAM)

It's only natural that what I created and passed through would be processed, and the fact that the actual frame type is different wouldn't appear in this test.

I rewrote the test and added the following to the beginning:

def _transport_emits(frame_cls) -> bool:
    """Whether the input transport actually outputs this frame."""
    import inspect
    from pipecat.transports import base_input
    return f"{frame_cls.__name__}(" in inspect.getsource(base_input)

check(f"{UserStart.__name__} is the type that transport.input() actually outputs",
      _transport_emits(UserStart))

I checked the library source code. This became the first item.

Stepped on the Same Issue Again on the Same Day

I added another countermeasure (raising the VAD threshold during playback) to the revised version. The tests passed. This time, I was using a fake VAD object.

class _FakeVAD:
    """Minimum VAD with only `_vad_start_frames` (same attribute name as the real one)."""
    def __init__(self, frames: int = 4):
        self._vad_start_frames = frames

The moment I added an item to verify that "it works with the real VAD", it failed.

AttributeError: '_vad_start_frames'

The real VAD doesn't have that attribute initially. It's created when the sampling rate is decided (at pipeline startup). My code was swallowing the exception, so it was "fixed" without actually being fixed.

except Exception:  # noqa: BLE001
    pass          # ← This creates "it worked without actually working"

I made sure to log when swallowing exceptions. Both the evidence of it working (info) and the fact that it didn't work (warning, only once).

Effect of the Countermeasure

The revised version with the correct type finally made the gate work.

echo-gap 7ms within window → Discarded as self-echo
echo-gap 5037ms outside window → Passed (real utterance)
echo-gap 10454ms outside window → Passed
echo-gap 17231ms outside window → Passed

Utterance that started 7 milliseconds after playback ended. Clearly its own voice.

Generalizable Points

"Implemented" and "working" are different, and the difference can't be covered by tests.

It's only natural that tests pass with the input I prepared, but if the actual input is different, the tests will all be green while the actual machine does nothing. I stepped on the same issue 4 times that day.

I was able to detect it because I was counting how many times it fired in the actual machine.

Number of judgments entered (all, including passed ones)
Number of discards
Evidence of countermeasures being effective

⚠️ Don't just count the number of discards. That would make it impossible to judge whether the window width is reasonable, as the "leaked" part outside the window would be invisible in principle. Also record the passed ones.

And when swallowing exceptions, always log them. except: pass is a setup that makes it look like it's working when it's not.

Series: Until the Voice Dialogue Avatar Responds Correctly

This article is part of Part 1: Stopping the Sound.

← Previous: There were two types of events with the same name
→ Next: The smartphone's finger was breaking the echo canceller

Series of 8 articles

Part 1: Stopping the Sound

There were two types of events with the same name
The self-echo countermeasure was never working ← Now here
The smartphone's finger was breaking the echo canceller

Part 2: Understanding Language

"It", "this page", and "earlier" were different things
The one line at the end of the huge prompt was ignored all 4 times
Apology words were contaminating the next search

Part 3: Judging

Not all utterances are questions
I was measuring something different from what I thought

The notes that led to this insight are summarized in Improving the Response Quality of Voice Dialogue Avatars.

Half of Free LLM APIs Were Dead After 2 Months

orca_forge — Mon, 07 Sep 2026 14:12:49 +0000

In July, I wrote an article titled "Summary of Free LLM API Providers That Don't Require a Credit Card." I listed eight providers and explained that by bundling multiple providers and implementing fallbacks, you could achieve real-world operation using only the free tiers.

Two months later, I inspected the pipeline running on this setup and found that the success rate had dropped to 60%.

Upon investigating the cause, I discovered that several of the eight providers I had listed were no longer functioning for various reasons, and none of the fallbacks were working. This article documents the full investigation. All figures are based on actual API calls made on 2026-09-07.

The Issue: A 6-Provider Setup Running on Essentially One Provider

I sent the same request to the router (LiteLLM) ten times, and here's what happened:

Success: 6 / Failure: 4

Here are the details of the failures:

litellm.APIError: CerebrasException - Payment required to access this resource.
  Received Model Group=free
  Available Model Group Fallbacks=None

litellm.NotFoundError: GeminiException - models/gemini-2.0-flash is no longer available

The line Available Model Group Fallbacks=None says it all. Despite bundling six providers, no fallbacks were occurring.

I then individually tested each provider's API to check their status.

Provider	Result
Groq	✗ 401 Invalid API Key
Cerebras	✗ 402 Payment required
OpenRouter	✗ 401 API key expired
Gemini	✗ 404 model no longer available
Mistral	✗ 429 Rate limit exceeded
Cohere	○ 200

Only one provider was still functioning. The reason for the 6 out of 10 successes was that LiteLLM's num_retries: 5 setting kept retrying until it landed on Cohere. While it appeared to have redundancy, in reality, all the load was on a single provider.

The Decay Wasn't Uniform

This is where things get interesting. I thought it would be a simple case of "free tiers ending," but each provider had a different reason for failing. Here’s the classification after testing 13 candidates:

1. Free Tier Itself Became Paid — Cerebras

402 Payment required to access this resource. Visit your billing tab.

The key is valid, authentication passes, but payment is required. In July, there was a "1M tokens/day" free tier. This is the only case where the information in my article was clearly incorrect.

2. Service Terminated — GitHub Models

GET https://models.github.ai/catalog/models
→ HTTP 410 Gone

A 410 status code means "permanently gone." While I had considered it a candidate, it’s no longer an option.

3. Models Completely Replaced — Groq

After reissuing the key, the error changed from 401 to 404.

404 The model `llama-3.3-70b-versatile` does not exist or you do not have access to it.

The free tier is still available, but the models are gone. The current catalog includes 9 models, none of which are Llama-based.

allam-2-7b / canopylabs/orpheus-* / groq/compound / groq/compound-mini
openai/gpt-oss-120b / openai/gpt-oss-20b / qwen/qwen3.6-27b / qwen/qwen3.8-27b

4. Only Model Names Expired — Gemini

404 NOT_FOUND: This model models/gemini-2.0-flash is no longer available.
    Please update your code to use models/gemini-3.6-flash

The free tier is still available here too. It was simply a matter of not keeping up with the model name changes.

As a fix, I switched to using the gemini-flash-latest alias. This works in practice and will automatically adapt to future name changes. It’s a measure to avoid repeating the same mistake.

5. Keys Had Expiration Dates — OpenRouter

401 API key expired.

Checking the dashboard, it shows Expires: Expired / Last Used: Never / Key limit: unlimited. The key expired due to its expiration date, not usage. When creating keys, you can choose "No expiration," which should always be selected for persistent use cases.

6. "No Card Required" but Balance Needed — Z.ai

429 Insufficient balance or no resource package. Please recharge.

While no card is required to issue a key, you can't make a single request without recharging. It’s often introduced as a "no-card-required free tier," but the reality is different.

7. Catalog Doesn't Reflect Reality — NVIDIA NIM

The model list API returns 81 entries. I tested all of them.

Success: 13
HTTP 404: 55   ← Models listed in the catalog but don't exist
Timeout/Exceptions: 7
503 / 500 / 400: 6

Of the 13 functioning models, 3 were for content safety classification, 2 were dedicated to translation, and 1 required image input. Only 5 were suitable for general chat use.

If you take the explanation "81 models available for free" at face value, you’ll only realize the issue after implementation.

8. Almost Defunct — SambaNova

Here are the results after testing all 7 models:

DeepSeek-V3.1 / V3.2 / Meta-Llama-3.3-70B / gpt-oss-120b → 402 A payment method is required
MiniMax-M2.7 / M3                                        → 429 high demand
gemma-4-31B-it                                           → ○ 200 (but took 12,021ms)

Only one model remains free, and it takes 12 seconds. Compared to Groq's 220ms, it’s 55 times slower. While you could say "free tier available," its practicality is questionable.

9. Card Required — Nebius / Scaleway

Nebius's registration page requires billing details, including name, address, and card information (with $0 authorization for card verification). Scaleway's official documentation states that "a card and KYC are required to receive normal rate limits."

Neither meets the "no credit card required" condition.

10. Token Permission Pitfalls — HuggingFace

403 This authentication method does not have sufficient permissions

Checking the token, it only had repo.content.read permissions. For inference, inference.serverless.write (displayed as "Make calls to Inference Providers" in the UI) is required.

After re-enabling the permission, it worked. Issuing a key and the key being usable are two different things, but I didn’t realize this until seeing the error message.

The Real Reason Fallbacks Didn't Work

This was the biggest lesson learned.

Here’s how LiteLLM was configured:

router_settings:
  routing_strategy: simple-shuffle
  num_retries: 5
  allowed_fails: 1
  cooldown_time: 60

At first glance, it seems robust with 5 retries. However, in practice, it failed immediately upon hitting Cerebras.

402 (payment required) and 404 (model not found) are not subject to retries. Retries and fallbacks are designed for issues like 429 (rate limits) or timeouts, which can be resolved by waiting. Since 402 and 404 are permanent failures, they are returned immediately.

In other words,

If one out of six providers is permanently dead, simple math tells us that 1/6 of requests will fail. The assumption that "having many providers ensures safety" only holds if all providers are functioning.

In fact, just removing Cerebras from the pool improved the success rate as follows:

Just fixing Groq and Gemini model names: 6/12 (50%) ← All failures were Cerebras' 402
Excluding Cerebras: 15/15 (100%)

Removing the dead provider was more effective than fixing model names.

Two Types of Inference Model Traps

In my July article, I mentioned that inference models should be avoided for agent use cases. This time, I discovered an even more troublesome issue.

(a) Multi-Turn Conversations Break (Known)

Inference model responses include reasoning_content. Agents accumulate conversation history and send it to the next turn, inadvertently sending this unfamiliar field.

400 property 'messages.*.assistant.reasoning_content' is unsupported

This is a problem specific to agents (multi-turn conversations). Single-turn generation doesn’t send history, so this issue doesn’t occur. This was already mentioned in July.

(b) Even Single-Turn Generation Returns Empty Content (New Discovery)

The real problem is this. Here are the results of asking several inference models, "What is 2+2? Answer with just the number."

Model	content
`qwen/qwen3.8-27b`	`"4"` ← Non-inference
`qwen/qwen3.6-27b`	`""` ← Empty
`openai/gpt-oss-20b`	`""` ← Empty
`liquid/lfm-2.5-2.6b:free`	`""` ← Empty
`nvidia/nemotron-3-super-120b-a12b`	`"User asks 2+2..."` ← Exposes reasoning

The answer ends up in reasoning_content, leaving content empty. Code that only reads choices[0].message.content receives an empty string without error.

This happens even in single-turn generation. So, it’s not enough to "just be careful with agents when using inference models." Without defenses like empty response retries, it will silently break.

Note that even within the same Qwen series, 3.6 outputs reasoning while 3.8 does not. You can’t judge based on model name alone, so it’s best to test each model once before adding it to the pool to check for the presence of reasoning_content.

Comparison with July Version

Provider	July Description	September Actual
Google Gemini	Free tier available	○ Available ( Model name update required )
Groq	Free tier available	○ Available ( All Llama models gone → Replaced with Qwen, etc. )
Cerebras	1M tokens/day	✗ 402 Payment required
OpenRouter	:free 20RPM	○ Available ( Key expiration, all models are inference type )
NVIDIA NIM	Numerous models	△ Only 13 out of 81 responded
Cohere	Trial tier	○ Available (The only one functioning consistently this time)
SambaNova	Permanent free tier	✗ 6 out of 7 models returned 402
Mistral	Experiment tier	△ Frequent 429 errors
—	(Not mentioned)	○ Cloudflare Workers AI emerged as a strong contender
—	(Not mentioned)	○ HuggingFace works if permission settings are correct

Only 3 out of the original 8 providers remained valid.

New Addition: Cloudflare Workers AI

The most straightforward provider to get working was Cloudflare Workers AI.

10,000 Neurons/day, no card required
API token requires only Account > Workers AI > Read permission for inference ( Edit not needed)
@cf/meta/llama-3.3-70b-instruct-fp8-fast responds in 570ms, non-inference

One caution: the permission names are confusing. AI Gateway permissions do not work.

A token with only AI Gateway permission returns 401 with error code 10000.

The gap left by Groq removing Llama models was filled by this provider.

Post-Recovery Setup

After removing dead providers, updating model names, and adding three new providers, here’s the result:

Provider	Model	Measured Speed
Groq	`qwen/qwen3.8-27b`	220ms
Cloudflare	`@cf/meta/llama-3.3-70b-instruct-fp8-fast`	570ms
Cohere	`command-a-03-2025`	748ms
HuggingFace	`meta-llama/Llama-3.3-70B-Instruct`	937ms
OpenRouter	`google/gemma-4-26b-a4b-it:free`	957ms
NVIDIA NIM	`google/diffusiongemma-26b-a4b-it`	1,070ms
Gemini	`gemini-flash-latest`	—
Mistral	`mistral-small-latest`	Frequent 429 errors

Success rate trend: 6/10 (60%) → 15/15 → 20/20 → 24/24 → 28/28

Conclusion: Summary Articles Go Stale in Two Months

Since half of my own article became unusable in two months, this is also a reminder to myself. Here are three practical conclusions for readers:

1. Implement Health Checks First.
It’s more effective to detect and remove dead providers than to add more providers. In this case, removing one dead provider contributed more to success rate improvement than fixing model names. A simple script sending one request to each provider is sufficient.

2. Fallbacks Are Not a Panacea.
They work for 429 errors and timeouts. They don’t work for 402 (payment required), 404 (model gone), or 401 (key expired). These are permanent failures, so they’ll keep generating errors until manually removed.

3. Don’t Fix Model Names; Verify Before Adding.
Use latest aliases if available. If not, test each model once before adding it to the pool to ensure a response is returned and reasoning_content is not included. Behavior can vary even within the same series.

Note: All figures in this article are based on actual measurements as of 2026-09-07. Free tier conditions, available models, and rate limits can change within months, as seen here. Always check each provider’s official documentation for the latest information before implementation. This article, too, will likely be half incorrect in two months.

18 Insights from Mass-Producing Voice Models — From Diffusion TTS Voice Design to Training Corpus Creation and Quality Gate Pitfalls

orca_forge — Sun, 06 Sep 2026 00:16:22 +0000

This is a record of designing voices from single-line captions, automatically creating a learning corpus, and passing all 12 role-specific voices (narrator/counselor/sales/presenter/operator/MC for both men and women) through full inspection. I wrote about the failures I encountered during approximately one month of actual work, divided into 18 articles. This article is the table of contents.

The Conclusion Upfront

Voice design, voice manufacturing, and voice operation are different technologies with different failures.

Design uses diffusion TTS. The voice is determined by the caption and random seed, making it fully reproducible.
Manufacturing is primarily about corpus generation. The design of the quality gate directly determines the voice quality.
Operation relies on lightweight pre-trained models. Diffusion TTS is too slow for conversation (2.5 times slower on the same GPU).

The biggest lesson boils down to one point: Having a quality gate and it being effective are two different things. Six of these 18 articles are about gates that existed but weren’t effective.

Reading Order

The articles are arranged in the order of design → manufacturing → inspection → operation. Reading from the top will take you through the journey of a single voice being created and deployed into production.

Chapter 1: Design — How to Determine the Voice

The TTS Chosen for Sound Quality Was Too Slow for Conversation A 2.5x real-time factor (RTF) difference. How we settled on a two-stage approach: designing voices with diffusion TTS and using pre-trained models for speech.
Drawing Voices Like a Gacha Voices are determined by captions and random seeds. By keeping a ledger of design values, voices can be recreated even if the model is lost.
Letting a Machine Choose "Narrator-like Voices" from 24 Candidates Listening to all candidates is unsustainable. Automatically measure speech rate, intonation, and stability to only listen to the top candidates. Also, the story of how all male candidates were eliminated due to metric limitations.

Chapter 2: Manufacturing — Corpus Quality Directly Becomes the Voice

The Stricter the Quality Gate, the More Monotonous the Takes Survive The reason all emotion-infused corpora ended up monotonous was the quality gate itself. This is the central story of this series.
Speech Rate Cannot Be Changed After Training Endings and speech rate are baked into the corpus. They cannot be adjusted with synthesis parameters.
TTS That Changes "Recording Location" Every Time Even with the same model and speaker, frequency characteristics differ for each clip. If not standardized, style switching results in inconsistent sound quality.
One Rough Clip Ruins the Entire Style If one out of five clips is rough, the entire style becomes hoarse. Dilution into the average doesn’t help.

Chapter 3: Inspection — Having a Gate and It Being Effective Are Different

Where Did the AI’s Habit of Stretching "Hello" Come From? Verification discarded long vowels due to kana normalization, making the defect fundamentally undetectable.
"A Little" Becomes "Shomo" The model and parameters were fine, but the input text was broken. The symbol removal list was dropping characters like "々", "〆", and "髙".
The Hallucination Guard Code Only Failed During Hallucinations A single-line bug made the guard ineffective only in necessary situations.
The "Three Characters" Allowed by the Quality Gate Became a Verbal Tic The size passing through the gate matched the size of the sound reproduced by the model.
Rejecting Candidates for Fixable Defects Is the metric measuring the product’s characteristics or the process’s state?
Defects Unseen in Transcription STT-only inspection misses 0.1-second additive sounds with silence in between. Capture them with waveform envelopes.

Chapter 4: Operation — Running as a Factory

70 Minutes of Training Material Lost to a Network Blink One disconnection led to 205 retries. Finding areas without retries and designing idempotent resumable processes.
From "ja" to "JP": Creating a Babbling Model Language code case sensitivity led to a completed babbling model. A catalog of pitfalls encountered.
Four Registration Paths, Zero Management Screens Asset catalogs became unmanageable. Only after creating an inventory API could we see the full picture.
Each Deployment Overwrote the Other’s Work Two parallel workflows alternately deleted each other’s changes from production. Branch checks didn’t prevent it; only matching sha256 with running Pods stopped it.
Chasing Unmeasured Targets with Thresholds Always Leads to Failure Interpreting zero metrics as "no issues" and tightening thresholds. Not realizing what wasn’t being observed until the end.

Three Articles for Those Short on Time

Interested in quality gate design? → Article 4 (selection bias) → Article 11 (three characters) → Article 13 (defects unseen by STT). These three cover all patterns of "gates existing but not being effective."
Planning to train a speech model? → Article 15 (pitfall catalog) → Article 9 (broken input) → Article 5 (style baked into the corpus).
Reading as an MLOps/manufacturing story? → Article 12 (process defects vs. product traits) → Article 14 (idempotent resume) → Article 18 (not chasing unmeasured targets).

Common Thread Across These 18 Articles

Most failures occurred not in models or GPUs, but in inspection design. It wasn’t that gates were too loose; it was that gates weren’t measuring what mattered, and those unmeasured aspects got baked into the product. Unlike text, with audio, defects that slip through aren’t noticed until heard.

What ultimately worked wasn’t finding smarter models, but systematically addressing what wasn’t being measured.

The work notes that formed the basis of these insights are compiled in Manufacturing Pipeline for Practical Voices from Diffusion TTS.

Chasing Unmeasured Metrics with Thresholds Always Leads to Failure — A Tale of 10 Mistakes in One Day

orca_forge — Sat, 05 Sep 2026 01:12:35 +0000

In a single day, I encountered the same type of failure ten times. I made four of these mistakes myself, while my colleague, working in parallel, made seven (with one overlapping case). The issues spanned various domains, including speech synthesis, search, audit logs, text preprocessing, and frame processing, with no overlap. Yet, the underlying structure of the failures was identical.

The target was outside the observable system from the beginning. No matter how I adjusted the thresholds, it was impossible to reach.

The First Six Cases

① Additional sounds at the end of speech model outputs

I assumed this was due to hallucinations in the training corpus and tightened the script alignment insertion tolerance from 3→1→0. After three re-trainings, the additional sounds went from 3/6 → 2/6 → 3/6, which was within the margin of error.

Scanning 198 audio clips, I found that the four clips with additional sounds all had insertion 0. Since the speech-to-text (STT) system didn’t recognize these sounds as text, they weren’t included in the threshold denominator.

② Dialogue act detection never triggered in real conversations

I used a vocabulary list to determine if a statement was a question. Since it never triggered, I added more words, narrowed the list, and adjusted conditions. In reality, it failed to detect compound utterances (e.g., "What is this, and also the previous one") and only passed on simple test sentences.

③ "Evidence present" metric always returned True

I measured whether answers included evidence as a binary value. Since the results were too good, I tightened the threshold. In reality, when automatic page navigation was enabled, something was always attached as evidence, so the metric itself carried no meaningful information.

④ Audit log showed 0 navigate events

When a user reported "the page transitioned automatically," I checked the audit table and found 0 entries, concluding it hadn’t occurred. The implementation didn’t log audit events at issuance, so it couldn’t have been recorded. I mistook the absence of records as evidence that the event didn’t happen. I misdiagnosed the first report and only questioned the logging system after the second report.

⑤ Unable to answer based on page content

I added four guards: search thresholds, exclusion conditions, and handover judgments. When a user suggested, "Shouldn’t you grasp the entire page text first?" I realized I was applying vector search to a 231-character page. The text was too short for meaningful segmentation, and preprocessing was dropping characters, causing searches to fail.

⑥ Pronunciation breakdown

The phrase "少々お待ちください" (Please wait a moment) was heard as "しょもおまちください." I checked the model, synthesis parameters, cache, and routing in sequence. Finally, I printed the preprocessing output and found this:

_clean_tts_text('少々お待ちください。')  →  '少お待ちください。'

The allowed character whitelist included kanji in the range 一-鿿, but 「々」 (CJK Symbols block) was excluded. Synthesizing "少お待ちください" resulted in "ショーをお待ちください" (Show o omachi kudasai).

Four More Cases (Later That Day)

While writing this article, four more cases arrived from my colleague, all following the same pattern:

Misidentified frame type, causing the gate to never trigger (discovered via 0 real-world logs)
Fake VAD passed, but the real one lacked attributes and did nothing
Instructions placed at the end of a large prompt were ignored four times (the same issue was documented in the same file)
elapsed_ms was misinterpreted, leading to unnecessary fixes

The colleague summarized it as:

The first case was particularly harsh. I built the gate, wrote tests, passed 8/8, deployed, and it never triggered. Tests are written based on my assumptions about the frame, so if the actual frame type differs, both the test and the implementation are based on the same misunderstanding.

The Common Pattern

There are three layers to these failures:

Layer 1: Interpreting "0 items" as evidence

In ①, "0 STT detections meant no additional sounds," and in ④, "0 audit logs meant no transitions." Both assumed a functioning observation system, even though a broken system would also return 0.

"0 items" is not evidence of absence. It could mean either "the observation system is working, and the target is absent" or "the observation system is broken." The value 0 doesn’t distinguish between these two scenarios.

Layer 2: Continuously adding guards to symptoms

In ②, I added vocabulary; in ③, I tightened thresholds; in ⑤, I added four guards. All adjustments were made near the symptoms.

This seems logical—symptoms are likely close to the cause. But when dealing with unobservable targets, tightening around symptoms is ineffective. In ①, adjusting thresholds three times had no meaningful impact.

Layer 3: Not questioning my own observation systems

This is the deepest layer.

In ①, I wrote the quality gate, so I assumed "if it passed, the material was normal."
In ④, the audit logs were implemented by the other party, so I assumed "if they implemented it, it should be visible."
In ③ and ⑤, the creators didn’t question their own metrics or searches.

We’re less likely to doubt systems we’ve built ourselves, especially if we remember them working. And without doubt, we don’t look beyond them.

The colleague who encountered ②–④ also failed to notice that the pipeline failure visualization panel they built was 404ing the next day. They built it, saw it working, and moved on.

The Only Effective Solution

The cases I resolved myself shared a common approach: I changed the axis of observation and measured the target again.

Case	Original Axis	New Axis	Result
① Additional sounds	STT transcription	Waveform envelope	Discovered 4 additional sounds in insertion 0 clips
⑤ Page content	Search score	Character count	Found 231 characters
⑥ Pronunciation	Listening to synthesized audio	Printing preprocessing output	Found "々" was removed

Each took less than 5 minutes. Scanning waveforms in ① took 30 minutes, still less than a tenth of the time spent adjusting thresholds and retraining.

Self-validation of metrics is inherently difficult. You can’t verify a broken metric using the same metric. Instead of asking, "Is this metric correct?" it’s faster to apply a different measure once.

User Reports as Independent Observations

Another effective approach across all cases:

When metrics and user reports conflict, doubt the metrics first.

In ①, even after a user reported hearing "こちらですって" (It’s over here), I prioritized my measurement: "STT detected 0/6, so it wasn’t passed." Another observation system (human ears) had already provided results.

Similarly, in ④, I dismissed the first user report based on 0 audit logs.

User reports are often treated as weaker evidence than machine metrics because they’re subjective and ambiguous. But as independent observation systems, their value is high when conflicts arise. If two independent measurements of the same target disagree, and one says "anomaly," doubt your own instruments first.

Practical Takeaways

Here’s the distilled procedure—nothing special:

Before adjusting thresholds, ask once if the target is observable with that threshold. For ①, ask, "Can additional sounds exist in insertion 0 clips?" This question leads to checking waveforms.

When seeing "0 items," verify the logging system separately. For ④, intentionally trigger an audit event to confirm it’s recorded. Takes 5 minutes.

Before adding a second guard to symptoms, look upstream once. For ⑤, count page characters. If the first guard didn’t fix it, it likely wasn’t capturing the target.

After building a detector, test it with non-target cases first. The waveform detector in ① initially flagged all 12 models as "anomalous," mistaking pauses for additional sounds. If detection rates are too high, doubt the detector, not the target.

For systems you’ve built, verify them with a different axis at least once. This is the most effective step. Doing it immediately after building would have saved the three retraining attempts in ①.

Naming the Issues

Finally, the most frustrating realization:

Almost all were known, standard problems.

Additional sounds in speech are known as trailing artifacts / babbling. The phenomenon of saying "I’ll move" but not moving is called procedural hallucination, with dedicated benchmarks. The need for a 500–800ms microphone gate after speech ends (not 300ms) to prevent self-echo is well-documented.

We treated all issues as environment-specific and dug internally. If we’d first checked if the symptoms had names, we could have quickly found known properties like "STT can’t detect them."

I wrote "10 failures in one day," but accurately, all 10 had precedents outside our environment.

Series: Mass-Producing Practical Voices from Diffusion TTS

This series documents designing voices from single-line captions, manufacturing training corpora, and mass-producing role-specific practical voices. This article is Part 5: Summary.

← Previous: Deploying Overwrote Each Other’s Work

(End of series)

All 18 Parts

The insights are compiled in the notes on Mass-Producing Practical Voices from Diffusion TTS Manufacturing Pipeline.

Overwriting Each Other’s Work with Every Deployment — 3 Rollback Incidents in 2 Sessions

orca_forge — Sat, 05 Sep 2026 00:41:30 +0000

Two teams were working on the same set of microservices in parallel: one on the audio pipeline and the other on conversation quality. Their responsibilities were separate, and the files they worked on barely overlapped.

Yet, three times, they overwrote each other's work in production.

1st Incident: 5 Frontend Commits Disappeared

To apply a frontend fix, I created a branch from main, built, and deployed it.

Then, a user reported:

A screen that was supposed to be removed was back, and a feature I added was missing. Upon investigation, the running image wasn't built from main.

$ grep -l "virtualpv:1.0.407" ~/kaniko-builds/*.yaml | xargs grep -- --branch
  --branch feat/character-list-preview

That branch was 5 commits ahead of main. By building from main, those 5 changes were removed from production.

In this repository, main wasn't the source of truth for production. Unmerged branches piled up: 54, 12, 6, and so on. The common assumption that "main is the latest" didn't hold here.

I restored the previous tag using kubectl set image, cherry-picked the changes to the correct base, and rebuilt.

2nd Incident: 17 Backend Versions Disappeared

After the first incident, we established this procedure:

# Before building, check if you've missed any commits from the production base branch
git log --oneline HEAD..<production-base>

Despite following this procedure, I caused another incident during the next build.

Before building the backend, I checked the running Pod to ensure my changes were included.

>>> inspect.signature(judge_transcript).parameters['max_inserted'].default
1     # My value. The running version included my changes.

Feeling confident, I built from main. As a result, the other team's 17 versions disappeared, halting conversation logging and losing 60 minutes of data.

I checked if the running image included my changes, but I should have checked if my build included the running image's changes. These are two different things, yet I treated them as the same.

A simple diff would have revealed the issue:

$ git log --oneline <my-HEAD>..<production-base> | wc -l
20

Running this once before building would have prevented the incident. Although we had a procedure, I didn't follow it when switching repositories. For the frontend, I even checked the manifest's --branch, but for the backend, I only verified a single value inside the Pod.

3rd Incident: The Team Following the Procedure Was Affected

Later, the opposite happened.

I fixed the backend and deployed it. Then, I switched to monitoring the rebuild but forgot to push to main.

The other team followed the correct procedure, comparing their branch to the production base and ensuring no commits were missed. Since my changes weren't in main or their branch at that time, their check passed. As a result, my changes were overwritten.

No matter how correctly one team checks, if the other doesn't push, incidents can't be prevented.

Branch Checks Weren't Enough

After the third incident, the other team nearly caused a fourth incident in another repository, which they caught just in time. They compared the SHA256 hashes of files in the running Pod with their branch:

pipeline.py     Match
brain_talk.py   ⚠️ Mismatch  Running=1a02…  Local=341e…
app.py          ⚠️ Mismatch  Running=8a41…  Local=4110…

The mismatches were three delivery recovery fixes I had added. If they had done a full build, all would have been lost.

At that time, they had mistaken the base branch (they thought it was a feature branch, but it was actually built from main). The git log HEAD..<base> procedure assumes the base is known, so it fails if there's a misunderstanding. SHA256 comparison works without knowing the branch name.

Another lesson was that checking for file existence isn't enough. After the second incident, I started checking if the other team's conversation_log.py existed, but in this case, the files existed but had different contents. Existence checks wouldn't catch this.

Conclusion: Two Steps Are Enough

From the three incidents, two essential steps emerged:

1. Push to main immediately after deployment.

This is the core. If both teams follow this, main will always match production, and step 2 becomes automatic. The third incident occurred because I failed to do this.

2. Before building, compare the running Pod and your branch using SHA256.

This is the safety net. Even if the other team doesn't follow step 1, this will catch issues.

kubectl exec -n <ns> deploy/<dep> -- sh -c \
  'cd /app && find apps/services -name "*.py" | sort | xargs sha256sum' > /tmp/pod.txt
(cd <repo> && find apps/services -name "*.py" | sort | xargs sha256sum) > /tmp/br.txt
diff /tmp/pod.txt /tmp/br.txt   # Ensure differences are only your changes

Step 2 alone couldn't prevent the third incident, and step 1 alone couldn't prevent the first. Both are necessary.

Additional Pitfalls Encountered

kubectl rollout status completion can't be trusted. "Successfully rolled out" appeared immediately, but two Pods were running simultaneously:

exista-voicepipe-<rs-old>-lpqtx   Running  ...:0.7.14
exista-voicepipe-<rs-new>-cv92x  Running  ...:0.7.15

They converged after a few seconds, but during that time, testing would have resulted in inconsistent behavior. After deployment, count Pods directly.

A similar incident occurred in a batch job generating 200 clips. The Pod was replaced mid-generation, resulting in half the clips using different settings. I only noticed when logs showed the new gate hadn't triggered once.

Don't reuse build definitions. I once modified an old manifest with sed but forgot to update the --branch, causing the first incident. It's better to keep separate definitions for each purpose. The other team now includes a note to use the running tag as the base.

Beware of tag overwrites. Once, a chain of && broke, skipping a tag bump and pushing new content to an old tag. Rolling back to that tag no longer restores the original behavior. Without knowing this history, troubleshooting becomes impossible.

Generalizable Lessons

These incidents seem specific to parallel work, but they can happen even solo.

The first incident's root cause was the common assumption that main is the source of truth didn't apply here. The deployment branch was the source of truth, and main was outdated. This is common in environments with unstructured release flows.

Even solo, if past-you built from a different branch, the same issue arises. Your past self is as unpredictable as another team member.

The key is to check if your build includes what's currently running, not what you added. You remember what you added but not what's running. That's why mechanical comparison is essential.

Series: Mass-Producing Practical Voices from Diffusion TTS

This series documents designing voices from a single caption, creating training corpora, and mass-producing role-specific practical voices. This article is Part 4: Operations.

← Previous: Four Registration Paths, Zero Management Screens

→ Next: Failing When Chasing Unmeasured Metrics with Thresholds

All 18 Articles in the Series

The insights are compiled in the Practical Voice Mass Production Pipeline from Diffusion TTS.