How to Build a Talking-Photo Lip Sync Pipeline with Kling and an API

If you have ever tried to turn a still portrait into a short narrated video, the hard part is usually not generating the face motion — it is keeping the mouth movement aligned with the voice without stitching together a fragile video pipeline.

This guide walks through a small, practical pipeline using the Kling Lip Sync API on Ace Data Cloud. The goal is simple: start with a Kling-generated 5s or 10s video, then drive the character's mouth with either an audio file or a short text prompt.

The useful part is that the lip sync step is just one JSON request, so you can wire it into a backend job, an internal content tool, or a prototype for digital-human narration without building a custom video stack.

What you can do

The endpoint is:

POST https://api.acedata.cloud/kling/lip-sync

It accepts application/json and returns application/json. You authenticate with a bearer token:

authorization: Bearer ${API_KEY}
content-type: application/json
accept: application/json

There are two generation modes:

• audio2video: use an existing audio file to drive the mouth movement.
• text2video: provide short text and a voice ID, and let the API generate the speech and lip sync together.

For the source video, you provide exactly one of these:

• video_id: the ID of a Kling-generated video, such as one returned by /kling/videos with image2video.
• video_url: a public URL to a video file.

The video_id path is convenient if you are already using Kling to animate a still image. The docs note that the video must be 5s or 10s and generated within the last 30 days. If you use video_url, the file must be .mp4 or .mov, no more than 100MB, 2–10 seconds long, 720p or 1080p, and between 720 and 1920 pixels in dimensions.

How it works

Think of the API as a second pass after you already have a short character video.

A complete talking-photo flow looks like this:

still image -> /kling/videos image2video -> video_id -> /kling/lip-sync -> video_url

The first step creates motion from an image. The second step replaces or aligns the mouth motion so the subject speaks naturally.

Here is the image-to-video step shown in the docs:

curl -X POST 'https://api.acedata.cloud/kling/videos' \
  -H 'authorization: Bearer ${API_KEY}' \
  -H 'content-type: application/json' \
  -d '{
    "model": "kling-v2-1-master",
    "action": "image2video",
    "start_image_url": "https://cdn.acedata.cloud/4hfydw.jpg",
    "prompt": "look at camera, natural",
    "duration": 5,
    "mode": "pro"
  }'

That request returns a video_id, which becomes the input to the lip sync call.

Option 1: drive the video with audio

Use audio2video when you already have a voiceover, a TTS file, or a recorded narration.

curl -X POST 'https://api.acedata.cloud/kling/lip-sync' \
  -H 'authorization: Bearer ${API_KEY}' \
  -H 'content-type: application/json' \
  -d '{
    "mode": "audio2video",
    "video_id": "895055164389466178",
    "audio_url": "https://cdn.acedata.cloud/6f7d62b18b.wav"
  }'

For this mode, audio_url is required when audio_type is url, which is also the default. Supported audio formats are .mp3, .wav, .m4a, and .aac, with a maximum file size of 5MB.

A practical tip: keep the audio duration close to the video duration. The API can only work with the visual material you give it, so a 12-second narration is not a good fit for a 5-second clip.

Option 2: drive the video with text

Use text2video when you want to produce a short line directly from text.

curl -X POST 'https://api.acedata.cloud/kling/lip-sync' \
  -H 'authorization: Bearer ${API_KEY}' \
  -H 'content-type: application/json' \
  -d '{
    "mode": "text2video",
    "video_id": "895055164389466178",
    "text": "Hi, long time no see. I am doing well, take care of yourself.",
    "voice_id": "genshin_vindi2",
    "voice_language": "en",
    "voice_speed": 1.0
  }'

In text2video, the required fields are:

• text: the line to speak, up to 120 characters.
• voice_id: the voice to use.

Optional voice controls include:

• voice_language: zh or en, defaulting to zh.
• voice_speed: from 0.8 to 2.0, with one decimal place.

This mode is useful for UI previews, short product explainers, onboarding clips, or any workflow where the script is generated dynamically and you do not want to manage separate audio files.

Handling the response

A successful synchronous response includes the task ID, the new Kling video ID, and the final video URL:

{
  "success": true,
  "task_id": "07a3ec65-9f7e-4a09-b7b7-282684082527",
  "video_id": "895055968777281546",
  "video_url": "https://platform2.cdn.acedata.cloud/kling/07a3ec65-9f7e-4a09-b7b7-282684082527.mp4",
  "duration": "4.966",
  "state": "succeed"
}

The returned video_url is the artifact you can show in your app, hand off to an editor, or queue for another processing step. The returned video_id is also reusable for later Kling operations such as another lip sync or extension step.

Async mode for production jobs

For a backend workflow, you probably do not want to hold a request open while a video job runs. The API supports async mode in two ways:

{
  "mode": "audio2video",
  "video_id": "895055164389466178",
  "audio_url": "https://your.cdn/voice.mp3",
  "async": true
}

You can also provide callback_url. Either option returns a task_id immediately. To poll, call /kling/tasks with:

{
  "action": "retrieve",
  "id": "<task_id>"
}

A good production pattern is to store the original request, the task_id, and the current state in your database. Then either process the callback or poll from a worker until the task reaches a terminal state.

A few guardrails I would add

Before sending a request, validate the constraints close to your UI or API boundary:

• Require exactly one of video_id or video_url.
• For text2video, reject text longer than 120 characters.
• For audio2video, check that audio is no larger than 5MB.
• Prefer clear, frontal, single-person videos.
• Keep the script or audio length aligned with the clip length.

These small checks prevent the most common bad requests and make the workflow much easier to debug.

If you want the exact field table and the latest constraints, the full Ace Data Cloud doc is here: https://platform.acedata.cloud/documents/kling-lip-sync-integration