DEV Community: AndrewDDev

How karaoke apps make users fall in love with singing

AndrewDDev — Fri, 10 Jun 2022 02:21:43 +0000

The explosion of online karaoke

With the diversification of people's entertainment activities, online entertainment activities are also diversified. This year, online karaoke applications have sprung up, such as WeSing, Smule, Yokee, StarMaker, KaraDoReMi, SingIt (DingaStar), The Voice Karaoke, Many singing applications such as SingPlay have been loved by a large number of users.

As more and more companies enter, there are more and more ways to play online karaoke, from the initial local recording, to today's Lianmai duet, grabbing wheat, and round wheat singing.

So how do we make users fall in love with singing?

If you want users to fall in love with singing, you first need to make users sing well.

So how do we make our customers sing well?

Four secrets to improve sound effects

Voice change: Through special processing of the sound，By optimizing the sound, the singer can make the voice sound softer while maintaining his own timbre.
Reverberation: Through special processing of the sound, the reverberation effect of different environments is created, making the sound appear as if it is in a concert hall, a cathedral, etc.
Reverberation echo: Through special processing of the sound, it can be matched with voice change and reverberation to achieve a variety of custom sound effects, such as ethereal and robot sounds.
Virtual stereo: Through the deep use of dual-channel technology, each position and angle of the sound source is virtualized to achieve stereo sound, 3D surround sound, and sound discrimination.

The function is only effective for the sound collected by the SDK. The developer can dynamically adjust the voice change, reverberation, reverberation echo, and virtual stereo during a call or live broadcast.

Prerequisites

Before proceeding with voice change/reverb/stereo, please make sure:

Create a project in ZEGOCLOUD Admin Console, and get the AppID of your project.
The ZEGO Express SDK has been integrated into the project. For details, see Integration and Implementation.

Voice Change

Set Preset Voice Changer

Call the setVoiceChangerPreset method to use SDK preset voice changer effect.

ZegoVoiceChangerPreset preset voice changing effects are as follows, developers can choose according to their needs:

The following sample code takes a male voice to a child voice as an example:

ZegoExpressEngine.getEngine().setVoiceChangerPreset(ZegoVoiceChangerPreset.MEN_TO_CHILD);

Set Custom Voice Changer

If the sound changing effect preset by the SDK cannot meet the needs, the developer can call ZegoVoiceChangerParam Method, through the pitch parameter pitch to set a custom voice changer, the value range of this parameter is [-8.0, 8.0], the larger the value, the sharper the sound, the default value is 0.0 (ie no voice change).

ZegoVoiceChangerParam voiceChangerParam = new ZegoVoiceChangerParam();
param.pitch = 2.0f;
ZegoExpressEngine.getEngine().setVoiceChangerParam(param);

Reverberation

Set Preset Reverb

Call setReverbPreset to set reverberation through preset enumeration.

ZegoReverbPreset preset reverb effects are as follows, developers can choose according to their needs:

The following sample code takes the large room mode as an example:

ZegoExpressEngine.getEngine().setReverbPreset(ZegoReverbPreset.LARGE_ROOM);

Set Custom Reverb

If the SDK preset reverb type cannot meet the needs, developers can call ZegoReverbAdvancedParam method, to achieve the reverberation effect required by the developer through the configuration of related parameters (for detailed parameter description, please refer to the API documentation).

ZegoReverbAdvancedParam reverbParam = new ZegoReverbAdvancedParam();
reverbParam.damping = 50.0; // Reverberation damping
reverbParam.reverberance = 50.0; // Reverberation
reverbParam.roomSize = 50.0; // room size
reverbParam.wetOnly = false;
reverbParam.wetGain = 5.0;
reverbParam.dryGain = 5.0;
reverbParam.toneLow = 80.0;
reverbParam.toneHigh = 80.0;
reverbParam.preDelay = 20.0;
reverbParam.stereoWidth = 0.0;
ZegoExpressEngine.getEngine().setReverbAdvancedParam(reverbParam);

After setting the custom reverb parameters, the preset reverb effect set when reverb is enabled will be invalid. If you want to use the SDK preset parameters again, you can use setReverbPreset. Set enumeration method to set.

Reverberation Echo

Call the setReverbEchoParam method and set the relevant parameters to realize the developer The required reverberation echo effect (for detailed parameter description, please refer to the API document).

The following sample code takes the ethereal effect as an example:

ZegoReverbEchoParam echoParam = new ZegoReverbEchoParam();
echoParam.inGain = 0.8f;
echoParam.outGain =1.0f;
echoParam.numDelays = 7;
int[] delay ={230,460,690,920,1150,1380,1610};
echoParam.delay=delay;

float[] decay={0.41f,0.18f,0.08f,0.03f,0.009f,0.003f,0.001f};
echoParam.decay=decay;
zegoReverbEchoParamDatas.add(echoParam);
ZegoExpressEngine.getEngine().setReverbEchoParam(echoParam);

Virtual Stereo

Set the Number of Streaming Audio Channels

If you need to turn on the virtual stereo function, you must first call the setAudioConfig method to set before pushing the stream The audio encoding channel is Stereo dual channel (default is Mono).

The example here uses the preset enumeration structure ZegoAudioConfig to be set to dual-channel.

ZegoAudioConfig audioConfig = new ZegoAudioConfig(STANDARD_QUALITY_STEREO);
ZegoExpressEngine.getEngine().setAudioConfig(audioConfig);

Set Virtual Stereo Parameters

After setting the audio encoding channel to dual channel, call the enableVirtualStereo method, Turn on the virtual stereo through the enable parameter, and set the sound source angle of the virtual stereo through the angle parameter to have the stereo effect. The angle range is 0 to 360, and generally can be set to 90 degrees (that is, straight ahead).

Starting from the 2.15.0 version, the SDK supports the all-around virtual stereo. To enable it, set the angle parameter to -1.

The following shows how to turn on the virtual stereo and set the angle to 90 degrees:

ZegoExpressEngine.getEngine().enableVirtualStereo(true, 90);

The following shows how to enable the all-around virtual stereo:

ZegoExpressEngine.getEngine().enableVirtualStereo(true, -1);

How to implement sound waves in Android by using ZEGOCLOUD SDK

AndrewDDev — Tue, 31 May 2022 03:31:28 +0000

Introduction

In the karaoke scene, it is often necessary to pull multiple streams and display the user who is speaking. We need to identify whether the user is speaking, the volume of the speech (sound wave), and display it on the UI.

The SDK provides the following two methods to achieve this function：

Sound level refers to the volume of a stream. The ZEGO Express SDK provides the ability to capture the sound level of a stream in real time and delivers the captured sound level data to the app client through related callbacks. A typical use case of this feature is that you can visualize the sound level data on your app UI to indicate the current active speaker and their volume. The following picture shows an example of such use cases.

Audio spectrum refers to a series of values, each of which represents an audio signal's energy level at a point in a frequency range. The ZEGO Express SDK provides the ability to capture the audio spectrum data of a stream in real time and delivers the captured audio spectrum data to the app client through related callbacks. A typical use case of this feature is that you can use it to create and display an audio spectrum visualizer on your app UI, especially for those online karaoke apps. The following picture shows an example of audio spectrum visualizers.

Prerequisites

Before you begin to use sound level and audio spectrum in your project, make sure you complete the following steps:

Create a project in ZEGOCLOUD Admin Console, and get the AppID of your project.
The ZEGO Express SDK has been integrated into the project. For details, see Integration and Implementation.

Enable the sound level callbacks

To enable sound level callbacks, call the startSoundLevelMonitor method.

// mSDKEnging is an instance of ZegoExpressEngine
mSDKEnging.startSoundLevelMonitor();

After the above step is completed:

When you start previewing the local video by calling the startPreview method or publishing a local stream by calling the startPublishingStream method, the SDK will trigger the onCapturedSoundLevelUpdate callback at an interval of 100ms to deliver the sound level data of the locally captured audio.

/**
    * The callback to deliver the sound level data of locally captured audio.
    *
    * The callback sends out a notification at an interval of 100 ms.
    * @param soundLevel: The sound level value of the locally captured audio, which is in the range [0.0, 100.0].
    */
public void onCapturedSoundLevelUpdate(double soundLevel){

}

When you start playing remote streams by calling the startPlayingStream method, the SDK will trigger the onRemoteSoundLevelUpdate callback at an interval of 100ms to deliver the sound level data of the remote streams.

/**
    * The callback to deliver the sound level data of remote streams
    *
    * The callback sends out a notification at an interval of 100 ms.
    * @param soundLevels: The sound level data (key-value pairs) of the remote streams, of which the key is a stream ID, and the value is the sound level value of the corresponding stream. Each sound level value is in the range [0.0, 100.0].
    */
public void onRemoteSoundLevelUpdate(HashMap<String, Double> soundLevels){

}

Enable the audio spectrum callbacks

To enable audio spectrum callbacks, call the startAudioSpectrumMonitor method.

// mSDKEnging is an instance of ZegoExpressEngine
mSDKEnging.startAudioSpectrumMonitor();

After the above step is completed:

When you start previewing the local video by calling the startPreview method or publishing a local stream by calling the startPublishingStream method, the SDK will trigger theonCapturedAudioSpectrumUpdate` callback at an interval of 100ms to deliver the audio spectrum data of locally captured audio.

/**
    * The callback to deliver the audio spectrum data of locally captured audio
    *
    * The callback sends out a notification at an interval of 100 ms.
    * @param audioSpectrum: An array of audio spectrum values of the locally captured audio. Each audio spectrum value is in the range [0, 2^30].
    */
public void onCapturedAudioSpectrumUpdate(float[] audioSpectrum){

}

When you start playing remote streams by calling the startPlayingStream method, the SDK will trigger the onRemoteAudioSpectrumUpdate callback at an interval of 100ms to deliver audio spectrum data of the remote streams.

/**
    * The callback to deliver the audio spectrum data of remote streams
    *
    * The callback sends out a notification at an interval of 100 ms.
    * @param audioSpectrums: The audio spectrum data (key-value pairs) of remote streams, of which the key is a stream ID, and the value is an array of audio spectrum values of the corresponding stream. Each audio spectrum value is in the range [0, 2^30].
    */
public void onRemoteAudioSpectrumUpdate(HashMap<String, float[]> audioSpectrums){

}

Obtain data from the callbacks

The SDK delivers the sound level data and audio spectrum data of the remote streams as key-value pairs in a HashMap. In each key-value pair, the key is the stream ID of a remote stream published by another user in the same room, and the value is the sound level value or an array of audio spectrum values that stream.

You can first obtain and save the list of remote streams published by the other users in the current room through the onRoomStreamUpdate callback, and then use each stream ID in the list as an index to get the sound level value or the audio spectrum data of that stream from the HashMap.

The following examples show how to obtain the sound level and audio spectrum data from the callbacks. For how to render the data onto the UI of the app, refer to the sample code mentioned in Section 2 above.

class MyEventHandler extends IZegoEventHandler
{
    @override
    public void onCapturedSoundLevelUpdate(double soundLevel) {
        // Obtain the sound level data of locally captured audio and render it to specific UI controls.
    }
    @override
    public void onRemoteSoundLevelUpdate(HashMap<String, Double> soundLevels) {
        // Obtain the sound level data of remote streams and render it to specific UI controls.
    }
    @override
    public void onCapturedAudioSpectrumUpdate(double[] audioSpectrum) {
        // Obtain the audio spectrum data of locally captured audio and render it to specific UI controls.
    }
    @override
    public void onRemoteAudioSpectrumUpdate(HashMap<String, double[]> audioSpectrums) {
        // Obtain the audio spectrum data (key-value pairs) of the remote streams and render it to specific UI controls.
    }
}

Disable the sound level callbacks

To disable the sound level callbacks, call the stopSoundLevelMonitor method.

// mSDKEnging is an instance of ZegoExpressEngine
mSDKEnging.stopSoundLevelMonitor();

After the above step is completed, the SDK will stop triggering the callbacks onCapturedSoundLevelUpdate and onRemoteSoundLevelUpdate.

Disable the audio spectrum callbacks

To disable audio spectrum callbacks, call the stopAudioSpectrumMonitor method.

// mSDKEnging is an instance of ZegoExpressEngine
mSDKEnging.stopAudioSpectrumMonitor();

After the above step is completed, the SDK will stop triggering the callbacks onCapturedAudioSpectrumUpdate and onRemoteAudioSpectrumUpdate.

How to use ZEGOCLOUD SDK to implement Voice change/Reverb/Virtual stereo in Android

AndrewDDev — Fri, 27 May 2022 06:57:01 +0000

Introduction

In the live broadcast, chat room, and karaoke room scenes, in order to increase the fun and interactivity, players can change the voice to make things funny, reverberate the atmosphere, and use stereo to make the sound more three-dimensional. ZegoExpress SDK provides a variety of preset voice change, reverberation, reverberation echo, and stereo effects. Developers can flexibly set the sound they want. If you need to listen to it, you can enable ear return for testing.

Voice change: By changing the user's pitch, the output sound is sensory different from the original sound, and various effects such as changing male voice to female voice are realized.
Reverberation: Through special processing of the sound, the reverberation effect of different environments is created, making the sound appear as if it is in a concert hall, a cathedral, etc.
Reverberation echo: Through special processing of the sound, it can be matched with voice change and reverberation to achieve a variety of custom sound effects, such as ethereal and robot sounds.
Virtual stereo: Through the deep use of dual-channel technology, each position and angle of the sound source is virtualized to achieve stereo sound, 3D surround sound, and sound discrimination.

The function is only effective for the sound collected by the SDK. Developers can dynamically adjust the voice change, reverberation, reverberation echo, and virtual stereo during a call or live broadcast.

Prerequisites

Before proceeding with voice change/reverb/stereo, please make sure:

Create a project in ZEGOCLOUD Admin Console, and get the AppID of your project.
The ZEGO Express SDK has been integrated into the project. For details, see Integration and Implementation.

Voice Change

Set Preset Voice Changer

Call the setVoiceChangerPreset method to use SDK preset voice changer effect.

ZegoVoiceChangerPreset preset voice changing effects are as follows, developers can choose according to their needs:

The following sample code takes a male voice to a child voice as an example:

ZegoExpressEngine.getEngine().setVoiceChangerPreset(ZegoVoiceChangerPreset.MEN_TO_CHILD);

Set Custom Voice Changer

ZegoVoiceChangerParam voiceChangerParam = new ZegoVoiceChangerParam();
param.pitch = 2.0f;
ZegoExpressEngine.getEngine().setVoiceChangerParam(param);

Reverberation

Set Preset Reverb

Call setReverbPreset to set reverberation through preset enumeration.

ZegoReverbPreset preset reverb effects are as follows, developers can choose according to their needs:

The following sample code takes the large room mode as an example:

ZegoExpressEngine.getEngine().setReverbPreset(ZegoReverbPreset.LARGE_ROOM);

Set Custom Reverb

ZegoReverbAdvancedParam reverbParam = new ZegoReverbAdvancedParam();
reverbParam.damping = 50.0; // Reverberation damping
reverbParam.reverberance = 50.0; // Reverberation
reverbParam.roomSize = 50.0; // room size
reverbParam.wetOnly = false;
reverbParam.wetGain = 5.0;
reverbParam.dryGain = 5.0;
reverbParam.toneLow = 80.0;
reverbParam.toneHigh = 80.0;
reverbParam.preDelay = 20.0;
reverbParam.stereoWidth = 0.0;
ZegoExpressEngine.getEngine().setReverbAdvancedParam(reverbParam);

Reverberation Echo

The following sample code takes the ethereal effect as an example:

ZegoReverbEchoParam echoParam = new ZegoReverbEchoParam();
echoParam.inGain = 0.8f;
echoParam.outGain =1.0f;
echoParam.numDelays = 7;
int[] delay ={230,460,690,920,1150,1380,1610};
echoParam.delay=delay;

float[] decay={0.41f,0.18f,0.08f,0.03f,0.009f,0.003f,0.001f};
echoParam.decay=decay;
zegoReverbEchoParamDatas.add(echoParam);
ZegoExpressEngine.getEngine().setReverbEchoParam(echoParam);

Virtual Stereo

Set the Number of Streaming Audio Channels

The example here uses the preset enumeration structure ZegoAudioConfig to be set to dual-channel.

ZegoAudioConfig audioConfig = new ZegoAudioConfig(STANDARD_QUALITY_STEREO);
ZegoExpressEngine.getEngine().setAudioConfig(audioConfig);

Set Virtual Stereo Parameters

Starting from the 2.15.0 version, the SDK supports the all-around virtual stereo. To enable it, set the angle parameter to -1.

The following shows how to turn on the virtual stereo and set the angle to 90 degrees:

ZegoExpressEngine.getEngine().enableVirtualStereo(true, 90);

The following shows how to enable the all-around virtual stereo:

ZegoExpressEngine.getEngine().enableVirtualStereo(true, -1);

How to use ZEGOCLOUD SDK to implement call quality monitoring in Android

AndrewDDev — Fri, 27 May 2022 06:42:48 +0000

Introduction

The ZEGOCLOUD Express-Video SDK provides the ability to monitor the streaming quality during stream publishing and stream playing. After stream publishing or stream playing is started, the SDK triggers the corresponding streaming quality callback every 3 seconds, and client applications can obtain real-time streaming quality data by listening for these callbacks.

Prerequisites

Before you monitor the streaming information, make sure you complete the following:

Create a project in ZEGOCLOUD Admin Console, and get the AppID of your project.
The ZEGO Express SDK has been integrated into the project. For details, see Integration and Implementation.

Monitor the quality of stream publishing

To monitor the quality of the stream publishing, listen for the onPublisherQualityUpdate callback. After stream publishing is started, this callback will be triggered every 3 seconds to provide the stream publishing quality data.

You can obtain stream publishing quality data through this callback to monitor the health of stream publishing in real time and display the upstreaming network quality on your application's UI.

engine.setEventHandler(new IZegoEventHandler() {
    @Override
    public void onPublisherQualityUpdate(String streamID, ZegoPublishStreamQuality quality) {
    // You can focus on specific quality attributes and report them to your business server, or focus on a specific field of an object for giving friendly notes for users.
    // The parameter 'quality' has many attributes. If you do not want to handle every one of them, you can just focus on the 'level' attribute, a comprehensive quality indicator calculated by the SDK based on other quality attributes.
     Log.v("onPublisherQualityUpdate: streamID=", streamID);
     }
}

The stream publishing quality attributes in details

The stream publishing quality includes the audio and video capture frame rate, bitrate, RTT, packet loss rate, and others of capturing and encoding stages of the stream publishing process.

The ZegoPublishStreamQuality object includes quality attributes of different stages of the stream publishing process. You can find the detailed attribute definitions in this section:

Quality attributes of audio and video capturing

The following attributes reflect the stream publishing quality of audio and video capturing, which is close to the user's subjective perception of the local preview.

audioCaptureFPS: The audio capture frame rate (fps).
videoCaptureFPS: The video capture frame rate (fps).

Quality attributes of video encoding

The stream publishing quality attributes of video encoding of the encoding stage is as follow:

videoEncodeFPS: The target output frame rate (fps) of video encoding.

Quality attributes of stream transmission

The stream publishing quality attributes below reflect the quality of stream transmission (sending), which is related to the encoding bitrate and the current network quality.

audioSendFPS: The actual audio transmission frame rate (fps).
audioKBPS: The actual audio transmission bitrate (kbps).
videoSendFPS: The actual video transmission frame rate (fps).
videoKBPS: The actual video transmission bitrate (kbps).
rtt: The rount-trip time (ms) from the client device to the ZEGO server.
packetLostRate: The client-side upstreaming packet loss rate.

Quality attributes of the total number of bytes

The stream publishing quality attributes below reflect the total number of bytes sent.

totalSendBytes: The total number of bytes sent, including audio, video, SEI.
audioSendBytes: The number of audio bytes sent.
videoSendBytes: The number of video bytes sent.

Quality attributes of encoding information

The stream publishing quality attributes below reflect the encoding information:

videoCodecID: The video codec.
isHardwareEncoder: Whether to enable hardware encoding.

Quality attributes of published stream quality level

The parameter quality has many attributes. If you do not want to handle every one of them, you can just focus on the level attribute, a comprehensive upstreaming network qualityindicator calculated by the ZegoExpressEngine based on other quality attributes.

The following table describes the level fields:

Monitor the quality of stream playing

To monitor the quality of the stream playing, listen for the onPlayerQualityUpdate callback. After stream playing is started, this callback will be triggered every 3 seconds to provide the stream playing quality data.

engine.setEventHandler(new IZegoEventHandler() {
    @Override
    public void onPlayerQualityUpdate(String streamID, ZegoPlayStreamQuality quality) {
     // You can focus on specific quality attributes and report them to your business server, or focus on a specific field of an object for giving friendly notes for users.
     // The parameter 'quality' has many attributes. If you do not want to handle every one of them, you can just focus on the 'level' attribute, a comprehensive quality indicator calculated by the SDK based on other quality attributes.
     Log.v("onPlayerQualityUpdate: streamID=", streamID);
    }
}

The stream playing quality attributes in details

The stream playing quality includes the audio and video frame rate, bitrate, delay, packet loss rate, and others of receiving, decoding, and rendering stages of the stream playing process.

The ZegoPlayStreamQuality object includes quality attributes of different stages of the stream playing process. You can find the detailed attribute definitions in this section:

Quality attributes of stream receiving

The stream playing quality attributes below reflect the quality of stream receiving, which is related to the quality of stream transmission (sending) and the current network quality.

audioRecvFPS: The actual audio receiving frame rate (fps).
audioDejitterFPS: The audio dejitter frame rate (f/s).
audioKBPS: The actual audio receiving bitrate (kbps).
audioBreakRate: The actual received audio break rate (number of breaks / every 10 seconds).
videoRecvFPS: The actual video receiving frame rate (fps).
videoDejitterFPS: The video dejitter frame rate (f/s).
videoKBPS: The actual video receiving bitrate (kbps).
videoBreakRate: The actual received video break rate (number of breaks / every 10 seconds).
packetLostRate: The client-side downstreaming packet loss rate, in percentage, 0.0 - 1.0.
rtt: The round-trip time (ms) from the client device to the ZEGO server.
avTimestampDiff: The difference between the video timestamp and the audio timestamp, used to reflect the synchronization of audio and video, in milliseconds. This value is less than 0 means the number of milliseconds that the video leads the audio, greater than 0 means the number of milliseconds that the video lags the audio, and 0 means no difference. When the absolute value is less than 200, it can basically be regarded as synchronized audio and video, when the absolute value is greater than 200 for 10 consecutive seconds, it can be regarded as abnormal.
peerToPeerDelay: The delay from peer to peer, in milliseconds.
peerToPeerPacketLostRate: The packet loss rate from peer to peer, in percentage, 0.0 - 1.0.

Quality attributes of stream rendering

The stream playing attributes below reflect the streaming quality at the stream rendering stage, which is close to the viewer's subjective perception. They are affected by the audio/video codec, and the values may be lower than the actual audio/video receiving frame rates.

audioRenderFPS|_blank](@audioRenderFPS-ZegoPlayStreamQuality): The audio rendering frame rate (fps).
[videoRenderFPS: The video rendering frame rate (fps).
delay: The delay after the data is received by the local end, in milliseconds.

Quality attributes of the total number of bytes

The stream publishing quality attributes below reflect the total number of bytes received.

totalRecvBytes: The total number of bytes received, including audio, video, SEI.
audioRecvBytes: The number of audio bytes received.
videoRecvBytes: The number of video bytes received

Quality attributes of decoding information

The stream playing quality attributes below reflect the decoding information:

videoCodecID: The video codec.
isHardwareDecode: Whether to enable hardware decoding.

Monitor the status of stream publishing/playing

Callback for updates on stream publishing status

After stream publishing starts, if the status changes, the SDK sends out the event notification through the onPublisherStateUpdate callback. To monitor the status of stream publishing, listen for this callback.

engine.setEventHandler(new IZegoEventHandler() {
    @Override
    public void onPublisherStateUpdate(String streamID, ZegoPublisherState state, int errorCode, JSONObject extendedData) {
        super.onPublisherStateUpdate(streamID, state, errorCode, extendedData);
        // In this callback, when the [state] is [ZegoPublisherState.NO_PUBLISH] and the [errorCode] is not 0, indicates the stream publishing is failed. And the SDK won't retry to publish the stream. At this time, user can set a corresponding UI to indicate the current state.     
        // In this callback, when the [state] is [ZegoPublisherState.PUBLISH_REQUESTING] and the [errorCode] is not 0, indicates the stream publishing retry is ongoing. But if the stream publishing retry still fails after the default reconnection duration, it won't try again and will send out a failure notification.
    }
}

You can determine the user's network status during stream publishing based on the state property.

The following table describes the value of the state property and the corresponding user status:

The parameter extendedData provides extended information for status updates. If you are using the ZEGO CDN, the key of this parameter is flv_url_list, rtmp_url_list, and hls_url_list, which correspond to the stream playing URL of FLV, RTMP, and HLS protocols respectively.

Callback for updates on stream playing status

After stream playing starts, if the status changes, the SDK sends out the event notification through the onPlayerStateUpdate callback. To monitor the status of stream playing, listen for this callback.

engine.setEventHandler(new IZegoEventHandler() {
    @Override
    public void onPlayerStateUpdate(String streamID, ZegoPlayerState state, int errorCode, JSONObject extendedData) {
        super.onPlayerStateUpdate(streamID, state, errorCode, extendedData);
        // In this callback, when the [state] is [ZegoPlayerState.NO_PLAY] and the [errorCode] is not 0, indicates the stream playing is failed. And the SDK won't retry to play the stream. At this time, user can set a corresponding UI to indicate the current state. 
        // In this callback, when the [state] is [ZegoPlayerState.PLAY_REQUESTING] and the [errorCode] is not 0, indicates the stream playing retry is ongoing. But if the stream playing retry still fails after the default reconnection duration, it won't try again and will send out a failure notification.
    }
}

You can determine the user's network status during stream playing based on the state property.

The following table describes the value of the state property and the corresponding user status:

Monitor the first frame of audio and video

Callback on capturing the first frame of audio

To receive the event notification when the first frame of audio is captured, listen for the onPublisherCapturedAudioFirstFrame callback.

After stream publishing starts, the SDK sends out the event notification through this callback when the first frame of audio is captured.

When you enable stream publishing or local preview for the first time, the SDK engine starts to capture the audio data of the local device, and the SDK sends out event notification through this callback.
You can tell whether the audio data has been captured successfully according to the returned event notification. If no notifications are received, it means the audio device is being used or the device is abnormal.

engine.setEventHandler(new IZegoEventHandler() {
    @Override
    public void onPublisherCapturedAudioFirstFrame() {
        super.onPublisherCapturedAudioFirstFrame();
    }
}

Callback on capturing the first frame of video

To receive the event notification when the first frame of video is captured, listen for the onPublisherCapturedVideoFirstFrame callback.

After stream publishing starts, the SDK sends out the event notification through this callback when the first frame of video is captured.

When you enable stream publishing or local preview for the first time, the SDK engine starts to capture the video data of the local device, and the SDK sends out event notification through this callback.
You can tell whether the video data has been captured successfully according to the returned event notification. If no notifications are received, it means the video device is being used or the device is abnormal.

engine.setEventHandler(new IZegoEventHandler() {
    @Override
    public void onPublisherCapturedVideoFirstFrame(ZegoPublishChannel channel) {
    super.onPublisherCapturedVideoFirstFrame(channel);
    }
}

Callback on receiving the first frame of audio

To receive the event notification when the first frame of audio is received, listen for the onPlayerRecvAudioFirstFrame callback.

After stream playing starts, the SDK sends out the event notification through this callback when the first frame of audio is received.

engine.setEventHandler(new IZegoEventHandler() {
    @Override
    public void onPlayerRecvAudioFirstFrame(String streamID) {
        super.onPlayerRecvAudioFirstFrame(streamID);
        AppLogger.getInstance().receiveCallback("onPlayerRecvAudioFirstFrame streamID:%s",streamID);
    }
}

Callback on receiving the first frame of video

To receive the event notification when the first frame of video is received, listen for the onPlayerRecvVideoFirstFrame callback.

After stream playing starts, the SDK sends out the event notification through this callback when the first frame of video is received.

engine.setEventHandler(new IZegoEventHandler() {
    @Override
    public void onPlayerRecvVideoFirstFrame(String streamID) {
        super.onPlayerRecvVideoFirstFrame(streamID);
    }
}

Callback on rendering the first frame of video

To receive the event notification when the first frame of video is rendered, listen for the onPlayerRenderVideoFirstFrame callback.

After stream playing starts, the SDK sends out the event notification through this callback when the first frame of video is received and has been rendered.

You can also use this callback to count the time consumed by capturing/receiving/rendering the first frame or use this callback to update the UI components of stream playing.

engine.setEventHandler(new IZegoEventHandler() {
    @Override
    public void onPlayerRenderVideoFirstFrame(String streamID){
        super.onPlayerRenderVideoFirstFrame(streamID);
    }
}

Monitor the changes on video resolution

Callback for the changes on the captured video resolution

To receive the event notification when the captured video size changes, listen for the onPublisherVideoSizeChanged callback.

After stream publishing starts, the SDK sends out the event notification through this callback when the resolution of the captured video changes.

When you enable stream publishing or local preview for the first time, the SDK engine starts to capture the video data of the local device, and the resolution of the video changes simultaneously.

You can use this callback to remove the UI elements in the view of the local video preview, or adjust the scale of the view of the local preview dynamically based on the resolution returned by the event callback.

engine.setEventHandler(new IZegoEventHandler() {
    @Override
    public void onPublisherVideoSizeChanged(int width, int height, ZegoPublishChannel channel) {
        super.onPublisherVideoSizeChanged(width, height, channel);
    }
}

Callback for the changes on the played video resolution

To receive the event notification when the played video resolution changes, listen for the onPlayerVideoSizeChanged callback.

After stream playing starts, the SDK sends out the event notification through this callback when the resolution of the played video changes. You can adjust the video display based on the final resolution of the video streams.

You won't receive any notifications when playing audio-only streams.
If the stream publisher end triggers the traffic control within the SDK due to network issues, the code distinguishability on the stream publisher side may be dynamically reduced, and SDK sends out notification through this callback.
The SDK sends out notification when the UI of the streams you played is being rendered. You can use this callback to update or switch the UI elements.

engine.setEventHandler(new IZegoEventHandler() {
    @Override
    public void onPlayerVideoSizeChanged(String streamID, int width, int height) {
    }
}

Monitor the status of the stream forwarding via CDN

Callback for updates on the status of CDN URL

To receive event notification when the specified CDN URL changes (adds new URL or removes existing URL), listen for the onPublisherRelayCDNStateUpdate callback.

After the ZEGO RTC Server forwards the stream to CDN, the SDK sends out event notification when the status of streams forwarded to CDN changes, for example, the stream forwarding stops or the forwarding operation retries.

You can tell whether the audio and video streams forwarded to CDN are normal according to this callback:

If the returned result of this callback indicates it is abnormal, locate the cause of the audio and video streams forwarded to CDN and perform a disaster recovery operation accordingly.
If you do not know the cause of the exception, contact ZEGO technical support for further analysis.

engine.setEventHandler(new IZegoEventHandler() {
    @Override
    public void onPublisherRelayCDNStateUpdate(String streamID, ArrayList<ZegoStreamRelayCDNInfo> infoList) {
    super.onPublisherRelayCDNStateUpdate(streamID, infoList);
    }
}

The stream forwarding attributes in details

The stream forwarding information includes the CDN URL, forwarding status, the cause of stream forwarding status, and when the status changes.

The ZegoStreamRelayCDNInfo object includes stream forwarding attributes. You can find the detailed attribute definitions in this section:

The following table describes the value of the state property:

The following table describes the value of the updateReason property:

How to use ZEGOCLOUD SDK to implement Acoustic Echo Cancellation、Automatic Gain Control、Active Noise Control in android

AndrewDDev — Mon, 23 May 2022 12:22:17 +0000

Introduction

To promote audio quality and user experience, ZEGOCLOUD provides the ability to process audio with 3A technologies, including AEC, AGC, and ANS.

AEC (Acoustic Echo Cancellation): A method of removing the echo from the audio signal captured by the microphone. After this feature is enabled, the SDK will filter the collected audio data to remove the echo.
AGC (Automatic Gain Control): A feature that automatically adjusts the receiver's audio gain (volume). After this feature is enabled, the sound will be amplified, but the sound quality will be affected to some extent.
ANS (Active Noise Control): A set of audio technologies that actively cancel the ambient noise. Enabling this feature will make your voice clearer.

Prerequisites

Before you begin, make sure you complete the following steps:

Create a project in ZEGOCLOUD Admin Console, and get the AppID of your project.
The ZEGO Express SDK has been integrated into the project. For details, see Integration and Implementation.

Set up audio 3A processing

Set up the AEC (Acoustic Echo Cancellation)

To implement the AEC feature, do the following:

To enable the AEC feature, call the enableAEC method.
Optional: To enable the AEC feature for the headphones scenario, call the enableHeadphoneAEC method.
To set the AEC mode, call the setAECMode mode.

The following sample code is used for setting the AEC to the medium mode (ZegoAECMode.MEDIUM):

// Enable the AEC feature.
engine.enableAEC(true);
// Enable the AEC when using headphones.
engine.enableHeadphoneAEC(true);
// Set the AEC to medium mode (ZegoAECMode.MEDIUM).
engine.setAECMode(ZegoAECMode.MEDIUM);

Set up the AGC (Automatic Gain Control)

To enable the AGC, call the enableAGC method. After this function is turned on, the SDK can automatically adjust the Microphone Volume to adapt to near and far sound pickups and keep the Volume stable.

// Enable the AGC feature.
engine.enableAGC(true);

Set up the ANS (Active Noise Control)

To implement the ANS feature, do the following:

To enable the ANS feature, call the enableANS method.
Optional: To control or cancel the transient noise, call the enableTransientANS method. This feature can be used to suppress the noise of typing keyboard, table, etc.
To set or adjust the ANS mode, call the setANSMode method. The SDK uses medium mode (ZegoANSMode.MEDIUM) by default.

The following sample code is used for setting the ANS to the soft mode (ZegoANSMode.SOFT):

// Enable the ANS feature.
engine.enableANS(true);
// Enable the transient ANS feature to control the transient noise.
engine.enableTransientANS(true);
// Set the ANS to soft mode (ZegoANSMode.SOFT).
engine.setANSMode(ZegoANSMode.SOFT);

Recommended configurations for 3A

The default and recommended configurations for audio 3A processing in the SDK are as follows:

Method	Description	Default config	Recommended config
`enableAEC`	Enable/Disable the AEC feature.	True	We recommend you use the default settings for general scenarios.
`enableHeadphoneAEC`	Enable the AEC for the headphone scenario.	False	We recommend you enable this for chatting or playing games in groups.
`setAECMode`	Set the AEC mode.	ZegoAECMode.AGGRESSIVE (Aggressive mode)	We recommend you use the default settings for general scenarios.
`enableAGC`	Enable/Disable the AGC feature.	True	In the chatting scenario, use the default settings. In the music radio scenario, disable the AGC feature to keep original sound. In the educational scenarios, such as large classes, small classes, and 1v1 classes, enable the AGC feature.
`enableANS`	Enable/Disable the ANS feature.	True	We recommend you use the default settings for general scenarios.
`setANSMode`	Set the ANS mode.	ZegoANSMode.MEDIUM (Medium mode)	We recommend you use the default settings for general scenarios.