Introduction
In the previous article, we looked at the internal structure of the EIS pipeline through Gyroflow: gyro data acquisition → integration → smoothing → frame warping. But when you actually implement this pipeline, you hit a problem that's easy to overlook.
"I applied EIS correction, but the video looks even more wobbly — like jello."
The culprit is Rolling Shutter. Rolling Shutter is a physical characteristic of the sensor, unrelated to EIS itself. But when EIS transforms frames without accounting for it, the distortion gets amplified. This article covers what Rolling Shutter is, how it conflicts with EIS, and why a separate correction step is necessary.
What Is Rolling Shutter?
Global Shutter vs Rolling Shutter
Image sensors capture a frame in one of two fundamental ways.
Global Shutter exposes all pixels simultaneously — every pixel starts and stops exposure at the same instant. Every pixel in a single frame captures the same moment in time. This method is used in CCD sensors and some industrial CMOS sensors.
Rolling Shutter exposes each row sequentially. The first row begins exposure, then the second row starts slightly later, then the third, and so on. By the time the full frame is read out, there's a time difference of several to tens of milliseconds between the top row and the bottom row.
Nearly all CMOS sensors in smartphones and consumer cameras today use Rolling Shutter. The reason is straightforward — lower manufacturing cost, simpler circuit design, and easier high-resolution implementation.
Distortions Caused by Rolling Shutter
When the camera moves during capture with a Rolling Shutter sensor, each row captures the scene at a slightly different point in time. This produces three characteristic distortions.
1. Skew
When the camera pans horizontally at speed, vertical lines tilt diagonally. The top rows capture the scene at one horizontal position, while the bottom rows capture it after the camera has shifted. Buildings and poles appearing to lean is a classic example.
2. Jello/Wobble
When the camera vibrates during video recording, each row captures a slightly different position. The result is a jello-like wobbling distortion across the entire frame. This is commonly seen in dashcam footage or video shot while running.
3. Partial Exposure
Very brief light sources like a flash or lightning illuminate only certain rows, leaving the rest dark. This shows up as a bright band across part of the frame in still photos.
For video recording and EIS, the most problematic distortions are skew and jello — especially the jello effect.
What Happens When EIS Meets Rolling Shutter
The Core Assumption of EIS: "All Pixels in a Frame Were Captured at the Same Instant"
Recall the EIS pipeline from the previous article. EIS estimates camera rotation from gyro data and warps the entire frame in the opposite direction to cancel out that rotation. There's a critical assumption embedded in this process.
"One frame represents one moment in time."
In other words, EIS applies the same correction value (rotation matrix) to every pixel in the frame. With a Global Shutter sensor, this assumption holds perfectly. With a Rolling Shutter sensor, it doesn't.
What Goes Wrong
Consider a concrete scenario.
Assume the camera is vibrating clockwise. EIS reads the gyro data and determines: "This frame was captured with a 2-degree clockwise rotation, so I'll apply a 2-degree counter-clockwise correction."
But due to Rolling Shutter, the actual situation is different:
- Top rows: captured at 1 degree of clockwise rotation
- Middle rows: captured at 2 degrees of clockwise rotation
- Bottom rows: captured at 3 degrees of clockwise rotation
When EIS uniformly applies "2 degrees counter-clockwise" to the entire frame:
- Top: 1° rotation + (-2° correction) = -1° overcorrected
- Middle: 2° rotation + (-2° correction) = 0° correct
- Bottom: 3° rotation + (-2° correction) = +1° undercorrected
The result is that the top and bottom of the frame shift in opposite directions, creating a new jello effect that wasn't there before, or amplifying an existing one.
This is the real explanation behind "I turned on EIS and the video looks even shakier."
What Is Rolling Shutter Correction (RSC)?
Why Row-Level Correction Is Needed
To solve this problem, the EIS warping stage must not apply a single correction value to the entire frame. Instead, each row (or group of rows) needs its own correction value based on the exact moment that row was captured.
This is Rolling Shutter Correction (RSC).
How RSC Works
The key to RSC is leveraging the temporal resolution of gyro data. As mentioned in the EIS pipeline article, gyro data is typically sampled at 1kHz or higher. At 30fps, one frame spans about 33ms. With a 1kHz gyro, that's 33 gyro samples within a single frame.
If the Rolling Shutter readout time (the total time to sequentially read all rows) is, say, 20ms, then the exact capture time of each row can be calculated, and the camera orientation at that time can be estimated from the corresponding gyro data.
In summary:
- Calculate the capture time of each row within the frame (from readout time and row number)
- Interpolate gyro data at that time to determine camera orientation
- Apply a different correction transform to each row
Effectively, warping is performed on a "per-row" basis rather than a "per-frame" basis.
How Do You Know the Readout Time?
One of the most critical parameters for RSC is the sensor's readout time. This value varies depending on the sensor's row count, clock speed, and resolution mode.
In commercial products, readout time is either specified in the sensor datasheet or passed as part of the frame metadata through the ISP driver. Internally at manufacturers, this value is often hardcoded into the ISP pipeline.
In the open-source Gyroflow, the value can be entered manually by the user, automatically extracted from video metadata, or estimated through a calibration video. Since cameras and sensors vary enormously unlike commercial products, this process is essential.
RSC Implementation in Commercial Products
Smartphones
In smartphone ISPs, EIS and RSC typically operate as a single integrated pipeline. The ISP receives readout time information from the sensor, synchronizes it with gyro data timestamps, and performs row-level correction. All of this runs in real time within the ISP hardware accelerator.
Major mobile ISPs including Qualcomm Spectra, Samsung SLSI, and MediaTek Imagiq all have built-in RSC capabilities. However, RSC quality varies between ISP vendors and depends on the tuning state.
Action Cameras / Drones
Products like GoPro and DJI handle EIS+RSC through proprietary ISPs or dedicated processors. Action cameras face particularly harsh vibration environments, making RSC critically important. GoPro's HyperSmooth and DJI's RockSteady are both pipelines that include RSC.
Post-Processing
Post-processing software like Gyroflow and Adobe Premiere's Warp Stabilizer also support RSC. The advantage of post-processing is unlimited look-ahead since the entire video can be analyzed, but the challenge lies in knowing the readout time accurately.
Practical Issues with Rolling Shutter and EIS
Readout Time Synchronization
RSC precision is directly determined by the synchronization accuracy between gyro data timestamps and sensor readout timing. Even a few milliseconds of sync error can cause corrections to misalign, leaving residual jello effects or creating new distortions.
In commercial products, the gyro sensor and image sensor share a hardware timer, or the ISP aligns gyro data using frame sync signals as a reference. In cases like Gyroflow where external gyro logs are used, this synchronization must be estimated in software, which can reduce precision.
Resolution and Readout Time
Even with the same sensor, readout time changes depending on the resolution mode (4K vs 1080p), binning settings, and frame rate. In 4K mode, more rows need to be read, so readout time increases and Rolling Shutter distortion grows. Conversely, binning reduces the effective row count and shortens readout time.
In practice, these values are managed per sensor mode and must be accurately passed to the ISP. If the value is wrong, RSC applies corrections at the wrong timing, resulting in distortion that's subtle but clearly visible.
Summary
Rolling Shutter is a physical characteristic arising from the sequential row-by-row exposure of the sensor. On its own, it causes skew and jello effects, but the problem becomes more severe when combined with EIS. When EIS assumes "the entire frame was captured at the same instant" and applies uniform correction, the per-row timing differences from Rolling Shutter get amplified into visible distortion.
RSC (Rolling Shutter Correction) solves this by leveraging the high temporal resolution of gyro data to apply individual corrections matched to each row's capture time. Today's major mobile ISPs process EIS and RSC as an integrated pipeline, and post-processing software like Gyroflow also supports RSC.
In the next article — the final installment of the EIS series — I plan to cover EIS from a camera module developer's perspective: its present and future, including AI-based EIS, sensor technology evolution, and the direction of software-defined cameras.
This article was written based on hands-on experience in camera module development. It describes publicly available technical principles and general industry practices, not the internal implementation of any specific product. If you find any errors or have feedback, please leave a comment.
Top comments (0)