HarmonyOS

Posted on Jul 15

Synchronized Real-time Text Highlighting with Audio Playback Timeline

#harmonyos #audiorenderer #arkts #wordhighlighting

Read the original article：Synchronized Real-time Text Highlighting with Audio Playback Timeline

Requirement Description

This guide shows how to synchronize visual text highlighting with audio playback timeline using sample-accurate timing in HarmonyOS applications. The implementation provides real-time word-by-word highlighting that precisely follows audio playback progress, creating a synchronized visual-audio experience for users.

Background Knowledge

Real-time text highlighting synchronized with audio playback requires precise timing coordination between audio sample processing and UI animations. The core concepts include;

Sample-Accurate Timing: Using audio sample counts to determine exact playback position.

Progress Calculation: Converting audio samples to text progression using mathematical interpolation.

Word Tokenization: Splitting text content into individual highlightable units.

State Management: Tracking current highlighting position and animation states.

The synchronization works by;

Tracking the number of audio samples played during AudioRenderer callbacks
Calculating text progression percentage based on total audio duration
Mapping text progression to word indices for highlighting

Common use cases for this feature;

Karaoke applications with lyric highlighting
Educational apps with reading assistance
Accessibility features for text-to-speech
Media players with subtitle synchronization
Language learning applications with pronunciation guides.

Example usage on a weather app;

Implementation Steps

Setup Audio Progress Tracking

Initialize sample counters in your audio playback component
Configure AudioRenderer callback to track played samples
Calculate total expected samples from audio duration and sample rate

Implement Text Tokenization

Split your text content into individual words or phrases
Store word boundaries and create indexable word arrays

Create Progress Calculation Logic

Convert played samples to time-based progress percentage
Map progress percentage to word index positions
Implement smooth interpolation for natural highlighting transitions

Synchronize Audio and Visual Updates

Update highlighting state in audio callback functions
Ensure thread-safe communication between audio and UI threads
Handle edge cases like audio seeking or playback interruptions

Code Snippet / Configuration

Component Setup

Initialize component with text tokenization and progress tracking;

@Component
export struct SynchronizedTextHighlight {
  @State private words: string[] = [];
  @State private currentWordIndex: number = 0;

  private playedSamples: number = 0;
  private totalSamples: number = 0;

  aboutToAppear(): void {
    const fullText = "Hope you had a good day! Overcast skies today. 22 degrees and comfortable.";
    this.words = fullText.split(' ').filter(word => word.length > 0);
    this.setupAudioProgressTracking();
  }
  //Rest of the code
}

Progress Calculation

Convert audio progress to word highlighting position;

private calculateProgress(): void {
    const audioProgress = Math.min(this.playedSamples / this.totalSamples, 1.0);
    const targetWordIndex = Math.floor(audioProgress * this.words.length);

    if(targetWordIndex !== this.currentWordIndex) {
      animateTo({duration: 150, curve: Curve.EaseOut}, () => {
        this.currentWordIndex = Math.min(targetWordIndex, this.words.length - 1);
      });
    }
  }

Audio Callback Integration

Track audio playback progress and update highlighting;

private setupAudioProgressTracking(): void {
    this.audioRenderer?.on('writeData', (buffer: ArrayBuffer) => {
      const numSamples = buffer.byteLength / 2;
      this.playedSamples += numSamples;
      this.calculateProgress();

      //Process your audio data here
      const int16Samples = new Int16Array(buffer);
      // ... audio processing code ...
    });
  }

Text Highlightling Component

Display text with synchronized word highlighting and progress bar;

build() {
    Column() {
      Text() {
        ForEach(this.words, (word: string, index: number) => {
          Span(`${word} `) //Notice the extra space here
            .fontColor(this.getWordColor(index))
            .animation({duration: 200, curve: Curve.EaseInOut})
        },(word: string, index: number) => `${word}_${index}`)
      }
      .padding(16)

      Progress({
        value: (this.playedSamples / this.totalSamples) * 100,
        total: 100
      })
        .width('90%')
        .height(4)
    }
  }

private getWordColor(index: number): ResourceColor {
    if(index < this.currentWordIndex) {
      return '#ff1e90ff';  //Highlighted already, blue
    } else if(index === this.currentWordIndex) {
      return '#ffffd700';  //Currently highlighting, gold
    } else {
      return '#ff808080';  //Not yet highlighted, gray
    }
  }

Limitations or Considerations

HarmonyOS API Version: Requires HarmonyOS API 11 or higher for AudioRenderer

Text Length: Very long texts may cause performance issues with excessive Span elements

DEV Community