Audio Description Manager

Read the original article：Audio Description Manager

Requirement Description

Manage audio descriptions for multimedia content to support users with visual impairments.

The system must manage audio descriptions for multimedia content in order to provide accessibility support for users with visual impairments.

Audio descriptions act as an additional narrative layer, filling in the gaps where visual information is essential for understanding.

The solution should ensure that these descriptions are accurately synchronized with media playback and delivered in a seamless manner without disrupting the primary media experience.

Background Knowledge

Audio descriptions (AD) are spoken narrations that describe important visual elements in video or multimedia content.

They are primarily intended for users with visual impairments or blindness, enabling them to perceive visual actions, scene changes, facial expressions, or other non-verbal cues that are essential for full comprehension of the media.

Implementation Steps

• Track Media Playback Position
  • Continuously monitor the current playback timestamp of the media player.
  • This position acts as a reference to determine when to trigger specific audio descriptions.
• Trigger Audio Descriptions at Appropriate Timestamps
  • Compare the current playback time with the timestamps of scheduled audio descriptions.
  • When playback time enters a description’s valid interval (timestamp to timestamp + duration), initiate playback of that description.
  • Ensure that each description only plays once at the correct moment.
• Manage Description Playback Queue
  • Maintain a sorted list of all descriptions for the current media.
  • Support adding new descriptions dynamically (e.g., user-generated or updated annotations).
  • Use a queue-like mechanism to determine the "next" upcoming description based on current playback position.
  • Provide enable/disable functionality so users can toggle audio descriptions on demand.
• Handle Accessibility Integration
  • Check whether screen reader or accessibility mode is enabled (e.g., accessibility.isScreenReaderOpenSync()).
  • Only trigger audio descriptions when accessibility support is active, avoiding unnecessary narration for sighted users.
• Playback of Descriptions
  • Use either TTS engines (for dynamic narration) or pre-recorded audio clips (for high-quality narration).
  • Optionally, adjust volume mixing so the description doesn’t clash with dialogue or background sound.

Code Snippet / Configuration

import { accessibility } from "@kit.AccessibilityKit";

export interface AudioDescription {
  timestamp: number; // in seconds
  description: string;
  duration: number; // in seconds
}

export class AudioDescriptionManager {
  private static descriptions: AudioDescription[] = [];
  private static currentMediaTime: number = 0;
  private static isEnabled: boolean = true;

  public static initialize(descriptions: AudioDescription[]): void {
    AudioDescriptionManager.descriptions = descriptions.sort((a, b) => a.timestamp - b.timestamp);
    AudioDescriptionManager.isEnabled = accessibility.isScreenReaderOpenSync();
  }

  public static updateMediaTime(currentTime: number): void {
    AudioDescriptionManager.currentMediaTime = currentTime;
    AudioDescriptionManager.checkForDescriptions();
  }

  private static checkForDescriptions(): void {
    if (!AudioDescriptionManager.isEnabled) return;

    const upcoming = AudioDescriptionManager.descriptions.find(desc =>
    desc.timestamp <= AudioDescriptionManager.currentMediaTime &&
      desc.timestamp + desc.duration > AudioDescriptionManager.currentMediaTime
    );

    if (upcoming) {
      AudioDescriptionManager.playDescription(upcoming.description);
    }
  }

  private static playDescription(description: string): void {
    // Use TTS or pre-recorded audio
    console.info(`[AUDIO DESCRIPTION]: ${description}`);
  }

  public static addDescription(timestamp: number, description: string, duration: number = 5): void {
    AudioDescriptionManager.descriptions.push({ timestamp, description, duration });
    AudioDescriptionManager.descriptions.sort((a, b) => a.timestamp - b.timestamp);
  }

  public static enable(): void {
    AudioDescriptionManager.isEnabled = true;
  }

  public static disable(): void {
    AudioDescriptionManager.isEnabled = false;
  }

  public static getUpcomingDescription(): AudioDescription | undefined {
    return AudioDescriptionManager.descriptions.find(desc => desc.timestamp > AudioDescriptionManager.currentMediaTime);
  }
}

Usage:

@Entry
@Component
struct Index {
  private currentTime: number = 0;

  aboutToAppear() {
    AudioDescriptionManager.initialize([
      { timestamp: 10, description: "A car enters the scene from the left", duration: 5 },
      { timestamp: 25, description: "The character looks surprised", duration: 3 }
    ]);
  }

  build() {
    Column() {
      // Video player component
      Slider({ value: this.currentTime, min: 0, max: 100 })
        .onChange((value: number) => {
          this.currentTime = value;
          AudioDescriptionManager.updateMediaTime(value);
        })
    }
  }
}

Test Results

Audio descriptions play at appropriate times during media playback, enhancing accessibility for visually impaired users.

Limitations or Considerations

Requires careful timing synchronization and may need to pause main audio during descriptions.

Written by Dogan Evci