DEV Community: Ross Kim

The Present and Future of EIS — A Camera Module Developer's View on AI, Sensors, and Software-Defined Cameras

Ross Kim — Wed, 22 Apr 2026 11:43:32 +0000

Introduction

Over the previous three articles, we have looked at EIS (Electronic Image Stabilization) from multiple angles.

The first article laid out the differences between OIS, EIS, and HIS, positioning EIS within the broader landscape of image stabilization.
The second article took us inside the EIS pipeline through Gyroflow, an open-source project.
The third article examined the collision between Rolling Shutter and EIS, explaining why RSC (Rolling Shutter Correction) is necessary. This final article closes the EIS series. So far we have focused on "how current EIS works." This time we focus on "where EIS is heading."

From a camera module developer's perspective, three major trends are palpable on the ground today — AI-based EIS, sensor technology evolution, and software-defined cameras. Predicting the future always carries the risk of being wrong, so instead of offering the rosy outlooks typical of marketing materials, I'll focus on changes I actually feel in module development work.

AI-Based EIS — From Gyros to Learning

The Limits of Conventional EIS

The EIS we have discussed so far was fundamentally a combination of gyro sensors + traditional computer vision + mathematical models.

Read angular velocity from the gyro to estimate camera orientation
Separate intended motion from shake using a low-pass filter
Warp frames using rotation matrices
Correct row-by-row timing differences with RSC This structure has been validated over many years and is still used in most commercial products. However, it has a few structural limitations.

First, it is weak against translation. Gyros only measure rotation. Vertical bounce while walking, or back-and-forth motion inside a vehicle, cannot be fully estimated from gyro data alone. Even with an accelerometer added, integration drift becomes a major problem.

Second, it does not account for scene context. Gyro-based EIS only knows "how much the camera moved," not "what is in the frame." It applies the same correction whether the subject is centered, near, or far.

Third, crop margin is wasted. A generous crop margin is reserved to handle worst-case shake, but such large shake rarely occurs in practice. As a result, FOV is sacrificed unnecessarily most of the time.

Where AI Steps In

Deep learning-based approaches tackle these limitations in a different way. There are three main directions.

1. Learned motion estimation

Instead of or alongside gyro data, pixel-level motion between frames (optical flow) is estimated using learned models. This provides a motion field that takes into account subject depth and scene structure. Google Pixel's video stabilization is said to have evolved in this direction.

2. Learned frame synthesis

Areas lost to cropping are filled in using information from surrounding frames, or heavily shaken frames are reconstructed from multiple frames. This can reduce crop margins or even retain most of the original FOV. Research tracks like "Motion-Aware Video Frame Interpolation" and "Neural Video Stabilization" fall in this direction.

3. End-to-end learning

A neural network learns directly from input video to stabilized output video, without a traditional pipeline. While many good results have been reported in research, commercial deployment remains limited due to challenges in real-time mobile processing and output consistency.

The Reality of AI EIS on the Ground

Reading academic papers and tech media, AI-based EIS looks like it has already solved everything. The reality in module development is a bit different.

Power and thermals. Running neural inference in real time on mobile devices requires NPUs or GPUs. Keeping them under high load throughout video capture causes serious heat and battery drain. The typical compromise is "high-quality AI EIS only in specific modes."

It does not fully replace the gyro. AI-based motion estimation works well when the scene has many feature points and lighting is sufficient, but performance degrades sharply in low light or against monochromatic backgrounds. Realistic products are moving toward a hybrid of gyro and AI: the gyro guarantees a baseline of stability, while AI adds scene awareness to push quality higher.

Training data bias. Neural network performance depends heavily on training data. It performs well in certain scenes (bright outdoors, urban environments), but in environments not represented in training data (underwater, extremely low light, industrial settings), unpredictable artifacts can emerge. From a camera module supplier's perspective, this translates into increased validation burden.

In summary, AI-based EIS is a clear direction, but rather than "gyro-based EIS fully exiting the stage," the more accurate picture from the field is that "a gyro + AI collaborative structure is becoming the standard."

Sensor Technology Evolution — Hardware for EIS

It is easy to focus solely on software algorithms, but much of EIS's progress owes itself to advances in sensor hardware. Here are a few changes I see on the ground.

Higher-Precision, Smaller Gyro Sensors

The performance of 6-axis IMUs built into smartphones has visibly improved in recent years. Sampling rates now exceed 1–2 kHz, and noise density and drift characteristics continue to improve. This directly lifts the quality of data that EIS depends on.

Perhaps more importantly, hardware synchronization between image sensor and gyro is becoming standard. The readout time synchronization issue discussed in the third article directly affects gyro timestamp accuracy, and recent platforms solve this at the hardware level.

Stacked CMOS and Fast Readout

Stacked CMOS technology, pioneered by Sony, separates the pixel layer and the circuit layer and stacks them. This allows more circuit area without increasing the overall sensor size, enabling significantly faster readout.

Faster readout directly reduces Rolling Shutter distortion. Rolling Shutter distortion is fundamentally proportional to "the time difference between the top and bottom rows of a frame," so reducing that time difference reduces the distortion, even with a Rolling Shutter architecture.

Going further, Stacked sensors with integrated DRAM use high-speed buffering to approximate Global Shutter behavior. They are not true Global Shutter, but they offer a realistic way to significantly mitigate Rolling Shutter's drawbacks.

Global Shutter on Mobile?

In theory, Global Shutter sensors would eliminate Rolling Shutter problems at the root. So will mobile eventually transition to Global Shutter?

From an image-capture camera perspective, not anytime soon. As discussed in the third article, the triple wall of pixel size, low-light performance, and cost remains high. This applies not only to the main camera but equally to the front camera. The primary use cases for front cameras (selfies, video calls) are more often in low-light conditions, and there is little high-speed distortion to solve, so there is weak motivation to switch to Global Shutter. As a result, cameras that users "shoot" with — whether main or front — are likely to continue using Rolling Shutter + RSC.

On the other hand, sensing-oriented cameras follow a different path. Receivers in ToF and structured light systems, tracking cameras in AR/VR headsets, barcode scanners in industrial devices — in the domain of sensors that the system uses to perceive the world, rather than image-makers for the user, Global Shutter is already close to standard. In this domain, low latency and distortion-free instantaneous capture matter more than low-light performance.

In other words, "Global Shutter is entering mobile" is accurate, but the path is sensing, not imaging. In the territory that EIS addresses — user-facing video — Global Shutter is unlikely to become mainstream in the near future.

Gyro-in-Sensor

There is movement toward integrating a gyro sensor within the image sensor package itself. This minimizes the physical distance between image sensor and gyro, allowing vibration phases to match more precisely and hardware synchronization to be cleaner.

From a module developer's perspective, this is welcome — the BOM simplifies, and the sensor vendor takes responsibility for gyro characteristics as well. The trade-off is reduced flexibility in sensor selection.

Summary: Hardware Sets the Ceiling for Algorithms

No matter how refined an EIS algorithm is, there is a clear ceiling if input data quality is poor. Conversely, when sensor and gyro quality improve, the same algorithm yields visibly better results.

From a software developer's standpoint, it's easy to say "the algorithm determines quality." But working in module development, you feel every day that hardware sets the ceiling for algorithms. Even in the age of AI-based EIS, this relationship will not change.

Software-Defined Camera — Boundaries Disappear

What is a Software-Defined Camera?

Software-Defined Camera (SDC) is not a term with a rigid definition, but the concept circulating in the industry can be summarized as:

"A structure in which a camera's final output quality is determined primarily by software processing (especially AI/ML), rather than by hardware such as sensors and lenses."

This concept is not new. Features like Google Pixel's HDR+, Night Sight, and Magic Eraser were early forms of SDC. The reason Pixel — whose hardware specs appeared unremarkable on paper — maintained top-tier photo quality was entirely due to software.

In the context of EIS, what SDC means is that "stabilization is no longer something software does after the sensor captures the frame; the entire capture pipeline is designed with stabilization as a built-in premise."

A Pipeline Without Boundaries

The traditional camera pipeline was divided into these stages:

Sensor readout → RAW → ISP (demosaic, noise reduction, tone mapping) → EIS/RSC → Encoder

Each stage received the output of the previous one and processed it independently, in a block structure. Recent pipelines, however, are rapidly blurring these boundaries.

Multi-frame Fusion. Instead of a single frame, multiple frames are taken in simultaneously, and noise reduction, HDR, super-resolution, and stabilization are all performed together. Here, EIS is no longer an independent stage — it becomes part of the multi-frame alignment process.

Neural ISP. There is a movement toward replacing or augmenting traditional ISP blocks with neural networks. Demosaicing, noise reduction, and detail recovery are being unified into a single network. If this trend expands to include EIS as well, an end-to-end network of "RAW input → stabilized and corrected output" becomes a real possibility.

Computational Viewfinder. The viewfinder image the user sees is already a computed result. It used to be "preview is light, high-end processing applies only to the final capture." Now, the preview is processed close to the final result. EIS has become a feature that must operate from the preview stage.

A Module Developer's Dilemma

The shift toward SDC places camera module developers in a peculiar position.

On one hand, because software compensates for hardware weaknesses, there is more headroom in module design. Optical defects that would have been classified as defects in the past can now be saved by software correction.

On the other hand, optimizing only for the module's "optical performance" is no longer enough. What matters is how well the module meshes with the entire software pipeline. Even with the same image sensor, the final result changes completely depending on ISP tuning, gyro synchronization accuracy, and calibration data quality.

As a result, the value chain of the camera module industry is shifting from "pure hardware supply" toward the integrated provision of "hardware + calibration data + drivers + ISP tuning." This is precisely the change I have been feeling firsthand since starting this series.

Where EIS Is Heading — Closing the Series

Closing this four-part EIS series, I want to organize a few thoughts that run through the whole. What started as a simple story about shake correction turned out to be a story that expanded to cover the entire camera system.

First, EIS is not just a "shake-correction feature." It is an integrated problem that spans the full camera system — gyro, sensor, ISP, lens correction, encoder. That is why this series covered OIS/HIS, the pipeline, Rolling Shutter, and AI.

Second, the evolution of EIS is the interaction of hardware and software. A good algorithm reaches its true potential only on a good sensor, and a good sensor only becomes meaningful when paired with well-tuned algorithms. Discussing EIS while looking at only one side is half an understanding.

Third, the role of the camera module developer is changing. In the past, the core was "supplying optical modules that meet specs." Going forward, competitiveness lies in "providing systems that integrate into a software pipeline." I write this series from the middle of that shift.

There are many topics on EIS that this series did not cover — actual tuning workflows, EIS in automotive cameras, the very different requirements of VR/AR, and more. Future articles may pick these up as the opportunity arises.

The next series will address another pillar of camera module development: ISP tuning. If EIS was "the domain of motion," ISP is "the domain of light and color."

This article is based on practical experience in camera module development. Statements about the future reflect a reading of publicly known technology trends and changes observed on the ground, and are not predictions or descriptions of any specific product roadmap. Comments, corrections, and opinions are always welcome.

Rolling Shutter and EIS — Why Image Stabilization Can Make Your Video Worse

Ross Kim — Fri, 17 Apr 2026 11:47:23 +0000

Introduction

In the previous article, we looked at the internal structure of the EIS pipeline through Gyroflow: gyro data acquisition → integration → smoothing → frame warping. But when you actually implement this pipeline, you hit a problem that's easy to overlook.

"I applied EIS correction, but the video looks even more wobbly — like jello."

The culprit is Rolling Shutter. Rolling Shutter is a physical characteristic of the sensor, unrelated to EIS itself. But when EIS transforms frames without accounting for it, the distortion gets amplified. This article covers what Rolling Shutter is, how it conflicts with EIS, and why a separate correction step is necessary.

What Is Rolling Shutter?

Global Shutter vs Rolling Shutter

Image sensors capture a frame in one of two fundamental ways.

Global Shutter exposes all pixels simultaneously — every pixel starts and stops exposure at the same instant. Every pixel in a single frame captures the same moment in time. This method is used in CCD sensors and some industrial CMOS sensors.

Rolling Shutter exposes each row sequentially. The first row begins exposure, then the second row starts slightly later, then the third, and so on. By the time the full frame is read out, there's a time difference of several to tens of milliseconds between the top row and the bottom row.

Nearly all CMOS sensors in smartphones and consumer cameras today use Rolling Shutter. The reason is straightforward — lower manufacturing cost, simpler circuit design, and easier high-resolution implementation.

Distortions Caused by Rolling Shutter

When the camera moves during capture with a Rolling Shutter sensor, each row captures the scene at a slightly different point in time. This produces three characteristic distortions.

1. Skew

When the camera pans horizontally at speed, vertical lines tilt diagonally. The top rows capture the scene at one horizontal position, while the bottom rows capture it after the camera has shifted. Buildings and poles appearing to lean is a classic example.

2. Jello/Wobble

When the camera vibrates during video recording, each row captures a slightly different position. The result is a jello-like wobbling distortion across the entire frame. This is commonly seen in dashcam footage or video shot while running.

3. Partial Exposure

Very brief light sources like a flash or lightning illuminate only certain rows, leaving the rest dark. This shows up as a bright band across part of the frame in still photos.

For video recording and EIS, the most problematic distortions are skew and jello — especially the jello effect.

What Happens When EIS Meets Rolling Shutter

The Core Assumption of EIS: "All Pixels in a Frame Were Captured at the Same Instant"

Recall the EIS pipeline from the previous article. EIS estimates camera rotation from gyro data and warps the entire frame in the opposite direction to cancel out that rotation. There's a critical assumption embedded in this process.

"One frame represents one moment in time."

In other words, EIS applies the same correction value (rotation matrix) to every pixel in the frame. With a Global Shutter sensor, this assumption holds perfectly. With a Rolling Shutter sensor, it doesn't.

What Goes Wrong

Consider a concrete scenario.

Assume the camera is vibrating clockwise. EIS reads the gyro data and determines: "This frame was captured with a 2-degree clockwise rotation, so I'll apply a 2-degree counter-clockwise correction."

But due to Rolling Shutter, the actual situation is different:

Top rows: captured at 1 degree of clockwise rotation
Middle rows: captured at 2 degrees of clockwise rotation
Bottom rows: captured at 3 degrees of clockwise rotation

When EIS uniformly applies "2 degrees counter-clockwise" to the entire frame:

Top: 1° rotation + (-2° correction) = -1° overcorrected
Middle: 2° rotation + (-2° correction) = 0° correct
Bottom: 3° rotation + (-2° correction) = +1° undercorrected

The result is that the top and bottom of the frame shift in opposite directions, creating a new jello effect that wasn't there before, or amplifying an existing one.

This is the real explanation behind "I turned on EIS and the video looks even shakier."

What Is Rolling Shutter Correction (RSC)?

Why Row-Level Correction Is Needed

To solve this problem, the EIS warping stage must not apply a single correction value to the entire frame. Instead, each row (or group of rows) needs its own correction value based on the exact moment that row was captured.

This is Rolling Shutter Correction (RSC).

How RSC Works

The key to RSC is leveraging the temporal resolution of gyro data. As mentioned in the EIS pipeline article, gyro data is typically sampled at 1kHz or higher. At 30fps, one frame spans about 33ms. With a 1kHz gyro, that's 33 gyro samples within a single frame.

If the Rolling Shutter readout time (the total time to sequentially read all rows) is, say, 20ms, then the exact capture time of each row can be calculated, and the camera orientation at that time can be estimated from the corresponding gyro data.

In summary:

Calculate the capture time of each row within the frame (from readout time and row number)
Interpolate gyro data at that time to determine camera orientation
Apply a different correction transform to each row

Effectively, warping is performed on a "per-row" basis rather than a "per-frame" basis.

How Do You Know the Readout Time?

One of the most critical parameters for RSC is the sensor's readout time. This value varies depending on the sensor's row count, clock speed, and resolution mode.

In commercial products, readout time is either specified in the sensor datasheet or passed as part of the frame metadata through the ISP driver. Internally at manufacturers, this value is often hardcoded into the ISP pipeline.

In the open-source Gyroflow, the value can be entered manually by the user, automatically extracted from video metadata, or estimated through a calibration video. Since cameras and sensors vary enormously unlike commercial products, this process is essential.

RSC Implementation in Commercial Products

Smartphones

In smartphone ISPs, EIS and RSC typically operate as a single integrated pipeline. The ISP receives readout time information from the sensor, synchronizes it with gyro data timestamps, and performs row-level correction. All of this runs in real time within the ISP hardware accelerator.

Major mobile ISPs including Qualcomm Spectra, Samsung SLSI, and MediaTek Imagiq all have built-in RSC capabilities. However, RSC quality varies between ISP vendors and depends on the tuning state.

Action Cameras / Drones

Products like GoPro and DJI handle EIS+RSC through proprietary ISPs or dedicated processors. Action cameras face particularly harsh vibration environments, making RSC critically important. GoPro's HyperSmooth and DJI's RockSteady are both pipelines that include RSC.

Post-Processing

Post-processing software like Gyroflow and Adobe Premiere's Warp Stabilizer also support RSC. The advantage of post-processing is unlimited look-ahead since the entire video can be analyzed, but the challenge lies in knowing the readout time accurately.

Practical Issues with Rolling Shutter and EIS

Readout Time Synchronization

RSC precision is directly determined by the synchronization accuracy between gyro data timestamps and sensor readout timing. Even a few milliseconds of sync error can cause corrections to misalign, leaving residual jello effects or creating new distortions.

In commercial products, the gyro sensor and image sensor share a hardware timer, or the ISP aligns gyro data using frame sync signals as a reference. In cases like Gyroflow where external gyro logs are used, this synchronization must be estimated in software, which can reduce precision.

Resolution and Readout Time

Even with the same sensor, readout time changes depending on the resolution mode (4K vs 1080p), binning settings, and frame rate. In 4K mode, more rows need to be read, so readout time increases and Rolling Shutter distortion grows. Conversely, binning reduces the effective row count and shortens readout time.

In practice, these values are managed per sensor mode and must be accurately passed to the ISP. If the value is wrong, RSC applies corrections at the wrong timing, resulting in distortion that's subtle but clearly visible.

Summary

Rolling Shutter is a physical characteristic arising from the sequential row-by-row exposure of the sensor. On its own, it causes skew and jello effects, but the problem becomes more severe when combined with EIS. When EIS assumes "the entire frame was captured at the same instant" and applies uniform correction, the per-row timing differences from Rolling Shutter get amplified into visible distortion.

RSC (Rolling Shutter Correction) solves this by leveraging the high temporal resolution of gyro data to apply individual corrections matched to each row's capture time. Today's major mobile ISPs process EIS and RSC as an integrated pipeline, and post-processing software like Gyroflow also supports RSC.

In the next article — the final installment of the EIS series — I plan to cover EIS from a camera module developer's perspective: its present and future, including AI-based EIS, sensor technology evolution, and the direction of software-defined cameras.

This article was written based on hands-on experience in camera module development. It describes publicly available technical principles and general industry practices, not the internal implementation of any specific product. If you find any errors or have feedback, please leave a comment.

Understanding the EIS Pipeline Through Gyroflow — A Look Inside Image Stabilization via Open Source

Ross Kim — Mon, 13 Apr 2026 12:16:51 +0000

Introduction

In the previous article, I covered the differences between OIS, EIS, and HIS. For EIS, I explained it roughly as "correcting frames based on gyro data," but what actually happens inside is more complex than that suggests.

EIS is not simply "cropping a shaky video to stabilize it." It is a pipeline that reads data from gyro sensors, estimates camera motion, decides which movements to keep and which to remove, and finally transforms each frame.

In this article, I will reference an open-source project called Gyroflow to examine what each stage of the EIS pipeline actually does. While commercial EIS implementations are proprietary, Gyroflow implements the same principles as open source, making it an excellent learning resource.

What Is Gyroflow

Gyroflow is an open-source video stabilization software that uses gyroscope data. It was originally built for post-processing stabilization of action camera and drone footage.

Here are its key features:

Gyro data-based: Estimates motion vectors from gyro sensor data rather than image analysis (optical flow)
Wide camera support: Can directly read gyro data from major action cameras including GoPro, DJI, Sony, and Insta360
Lens profiles: Includes built-in distortion models for various lenses, including fisheye
GPU acceleration: Supports near-real-time processing speeds via OpenCL/wgpu
Rust-based: Core logic is written in Rust, making the code relatively readable

The source code is available on GitHub, and the community is active. Beyond practical use, it holds significant value as a reference for understanding how EIS works internally.

EIS Pipeline Overview

The EIS pipeline can be broken down into five stages. The diagram below shows the overall flow:

Let's walk through each stage.

Stage 1: Gyro Data Acquisition

A gyroscope sensor measures the camera's angular velocity. In simple terms, it records in which direction and how fast the camera is rotating across three axes (pitch, yaw, roll).

This data is typically recorded alongside video frames at the camera firmware level. Action cameras like GoPro embed gyro data in the video file's metadata. Gyroflow parses this metadata to extract the gyro data.

There are some critical points here:

Timestamp Synchronization

The timestamps of gyro data and video frames must match precisely. Since the gyro sensor's sampling rate (typically 200Hz–8kHz) and the video frame rate (30fps, 60fps, etc.) differ, time synchronization is essential to correctly match gyro data to specific frames.

To illustrate how sensitive this synchronization is: an offset of just 10ms can cause the correction direction to reverse relative to the actual shake, effectively doubling the shake amplitude. At 30fps, one frame is approximately 33ms, so 10ms is only one-third of a frame — yet it is fatal to the result. Gyroflow allows adjusting this offset either automatically or manually, while commercial products typically handle synchronization at the hardware level.

Gyro Data Noise

Gyro sensor data always contains noise. There is drift — a slow accumulation of error — as well as high-frequency noise. Using this noisy data directly results in jittery or unstable corrections. Therefore, preprocessing of the gyro data itself (such as a low-pass filter) is necessary.

Stage 2: Motion Estimation — Camera Orientation Calculation

Integrating gyro data (angular velocity) over time yields the camera's orientation. The direction the camera was pointing at each frame is expressed as a quaternion or rotation matrix.

Angular velocity data → Time integration → Per-frame camera orientation (quaternion)

Several practical issues arise in this process:

Integration Drift

Integrating angular velocity accumulates error. This is gyro drift. It is negligible over short intervals, but over tens of seconds to several minutes of footage, it builds up to significant levels. Gyroflow offers an option to use optical flow-based auxiliary estimation alongside gyro data to correct this drift.

Coordinate System Transformation

The gyro sensor's coordinate system does not always match the camera (image) coordinate system. The axis mapping varies depending on how the sensor is mounted on the circuit board. This is why Gyroflow has an IMU orientation setting. In commercial products, this mapping is fixed in firmware.

Stage 3: Smoothing — Which Movements to Keep, Which to Remove

This stage is the core of EIS and has the greatest impact on output quality.

Camera movements fall into two categories:

Intended movements: Smooth camera motion the operator deliberately performs, such as panning and tilting
Unintended movements: Unwanted shake from hand tremor, walking vibration, etc.

The goal of EIS is to remove the latter while preserving the former. The challenge is that perfectly distinguishing between the two automatically is impossible.

Limitations of a Simple Low-Pass Filter

The most intuitive approach is to apply a low-pass filter (LPF) to the camera orientation data — remove high-frequency shake while passing through low-frequency motion.

However, this method has a fundamental problem:

Set the cutoff frequency low → Hand shake is well suppressed, but fast panning introduces lag (the video feels like it is trailing behind)
Set the cutoff frequency high → Panning response is good, but hand shake remains

Ultimately, a fixed LPF cannot simultaneously achieve "stable yet responsive" results.

Gyroflow's Smoothing Algorithms

Gyroflow provides several smoothing algorithms, notably:

Plain 3D: Gaussian smoothing in quaternion space. Simple but produces basic results.
Horizon lock: A special mode that locks the horizon. Frequently used for drone footage.
Fixed camera: Removes all movement as if the camera were fixed. Tripod effect.
Velocity dampened: Dynamically adjusts smoothing intensity based on camera movement speed. Strong smoothing when stationary, weak smoothing during panning.

Among these, velocity dampened is the most practical. It takes an adaptive approach to overcome the limitations of a simple LPF: "reduce smoothing when the camera is moving fast, strengthen smoothing when stationary."

Commercial EIS products also use similar adaptive smoothing in principle, but the specific implementations vary by manufacturer and are not publicly disclosed.

Stage 4: Frame Warping

Once a "corrected camera orientation" has been obtained through smoothing, the actual video frames need to be transformed.

The frame is rotated/transformed by the difference between the original camera orientation and the corrected camera orientation. This process is called frame warping.

How Warping Works

Simplified, the flow looks like this:

Original frame's camera orientation: Q_original
Smoothed target orientation: Q_smoothed
Correction rotation: Q_correction = Q_smoothed × Q_original⁻¹
Apply this correction rotation to the image

Multiplying by Q_original⁻¹ (the inverse of the quaternion) means "reversing the original frame's rotation." It is an operation that cancels out the original shaky orientation and moves the frame to the desired smooth orientation.

In practice, lens distortion correction is also performed alongside the rotation. If the image is rotated while wide-angle fisheye distortion remains, the edges exhibit a warping artifact. This is why Gyroflow requires a lens profile — the distortion model must be known for accurate warping.

FOV Loss

Rotating/transforming a frame creates empty areas at the edges. These empty areas must be hidden by cropping the video, which reduces the field of view (FOV).

This is the most fundamental trade-off in EIS:

Stronger correction → more cropping → greater FOV loss
Preserving FOV → correction intensity must be reduced → shake remains

Gyroflow allows users to directly adjust this crop ratio and offers a Dynamic Zoom feature that automatically calculates the optimal crop per frame. Commercial products typically use a fixed crop (around 10–15%) or resolution oversampling (4K capture → 1080p stabilized output) to make FOV loss imperceptible.

Stage 5: Output

Encoding the warped frames produces the final stabilized video. Since Gyroflow is a post-processing tool, this occurs as a separate rendering step, whereas commercial real-time EIS executes this entire pipeline simultaneously with each frame capture.

Real-time processing introduces additional constraints:

Look-ahead limitation: Future frames available for smoothing are limited
Computational resources: Processing must complete within GPU/DSP resource budgets
Thermal/power: Must operate within the mobile device's thermal and power budget

Because of these constraints, even when using the same principles, there is a quality gap between post-processing (Gyroflow) and real-time (commercial EIS) output.

Differences Between Gyroflow and Commercial EIS

While Gyroflow helps us understand EIS principles, there are several important differences from commercial EIS:

Aspect	Gyroflow (Post-processing)	Commercial EIS (Real-time)
Processing timing	After recording	Simultaneous with recording
Smoothing look-ahead	Entire video available	A few to tens of frames
Lens distortion correction	Separate lens profile	Built into ISP
Gyro synchronization	Manual/auto estimation	Hardware synchronization
Computational resources	PC GPU (unlimited)	Mobile DSP (limited)
FOV handling	Dynamic crop available	Typically fixed crop

In commercial products, this entire pipeline runs with hardware acceleration inside the ISP (Image Signal Processor), often operating in hybrid mode coordinated with OIS.

Summary

On the surface, EIS can be summarized as "cropping to suppress shake," but internally, a complex pipeline exists:

Gyro sensor data acquisition and time synchronization
Camera orientation estimation through angular velocity integration
Smoothing to separate intended movements from shake
Frame warping that accounts for lens distortion
Managing the trade-off between FOV loss and correction intensity

Gyroflow implements this pipeline transparently as open source, making it an excellent reference for understanding how EIS works. For those interested in reading the actual code, I recommend checking out the GitHub repository.

In the next article, I plan to cover the relationship between Rolling Shutter and EIS — why rolling shutter correction is additionally required in EIS.

This article was written based on hands-on experience in camera module development. The descriptions of Gyroflow reference its publicly available source code and documentation, and do not describe the internal implementation of any specific product. If you find any errors or have feedback, please leave a comment.

OIS vs EIS vs HIS — A Camera Module Developer's Guide to Image Stabilization

Ross Kim — Fri, 10 Apr 2026 06:48:03 +0000

Why Image Stabilization Matters

Smartphone camera megapixel counts keep climbing year after year, but what truly determines video quality lies elsewhere: Image Stabilization.

In video recording, frame-to-frame shake accumulates, making footage uncomfortable to watch no matter how high the resolution. For photos, faster shutter speeds can mitigate the effects of hand shake in bright conditions, but in low light, slower shutter speeds make motion blur an unavoidable problem.

There are three main approaches to solving this: OIS (Optical), EIS (Electronic), and HIS (Hybrid), which combines the two. Let's break down the principles, pros and cons, and real-world differences in how each actually works.

OIS — Optical Image Stabilization

How It Works

OIS physically moves the lens or image sensor to counteract hand shake. When the gyro sensor inside the camera module detects movement, an actuator shifts the lens assembly or sensor in the opposite direction to correct the optical axis.

Implementation Methods

OIS is broadly divided into two approaches:

Lens-shift: Moves a portion of the lens group along the X and Y axes. This is the traditional approach with a long history in the camera industry.
Sensor-shift: Moves the image sensor itself. Apple introduced this with the iPhone 12 Pro Max and has since expanded it across the entire iPhone lineup. Samsung also adopted sensor-shift OIS with the Galaxy S26 Ultra. Chinese manufacturers like Huawei (Pura 70 Ultra, Pura 80 Ultra) and OPPO have also implemented it in their flagships. Since the sensor is lighter than the lens, it offers faster response times and a wider correction range, and adoption is expanding across the flagship market.

Actuator types include VCM (Voice Coil Motor), SMA (Shape Memory Alloy), and ball-guide mechanisms, selected based on tradeoffs between module size, power consumption, and correction range.

Pros

Optical correction means no image quality loss
Stabilizes the light reaching the sensor, making it strong in low-light conditions
Effective for both still photos and video

Cons

Additional components (actuators, gyros, driver ICs) increase module cost and size
Correction range has physical limits (typically around ±1 degree)
Durability concerns from drops and impacts (OIS lock mechanisms required)
Additional power consumption

EIS — Electronic Image Stabilization

How It Works

EIS corrects hand shake through software rather than hardware. It calculates inter-frame shake based on gyro sensor data and applies cropping and warping (geometrically transforming and aligning the image) to each frame to produce stabilized video output.

In simple terms, the actual output uses a narrower area than the full sensor capture, and the position of this output region is adjusted frame by frame according to the detected shake.

Pipeline

A typical EIS processing pipeline works as follows:

Gyro data acquisition — Angular velocity data is read from the IMU (Inertial Measurement Unit)
Motion estimation — Gyro data is combined with (in some cases) image-based motion vectors to estimate camera movement
Target path generation — Intentional panning is separated from unintentional shake. A smoothing filter creates a stabilized target path
Frame transformation — Affine (transformation preserving parallelism, including rotation, scaling, and translation) or homography (transformation that also accounts for perspective) transforms are applied to each frame for correction
Crop and output — The final frame is cropped to remove black borders caused by the transformation

Pros

No additional hardware required, favorable for cost reduction
No impact on module size
Algorithm improvements possible through software updates
No physical correction range limits, enabling compensation for large movements (practically effective up to around ±3 degrees)

Cons

FOV (field of view) loss due to cropping (typically 10–20%)
Resolution degradation from the crop + warp process
Difficult to apply to still photos (requires inter-frame comparison)
Weak at correcting high-frequency vibrations
Cannot reduce motion blur itself in low light (since it's not optical correction)
Combined with rolling shutter distortion, correction results can appear unnatural

HIS — Hybrid Image Stabilization

Concept

HIS combines OIS and EIS. The term "HIS" itself is more of a marketing and technical convenience label used by manufacturers rather than an industry standard, but the concept of using OIS and EIS together is adopted by most flagship smartphones today.

Actual Operating Structure: OIS First-Pass Correction → EIS Post-Processing

It's easy to imagine HIS as OIS and EIS equally sharing the workload, but the actual implementation is closer to a serial pipeline.

OIS corrects first, physically — When the gyro sensor detects shake, the actuator moves the lens or sensor to optically counteract the movement.
EIS reads the OIS correction results and post-processes — Residual micro-shake, rolling shutter distortion, and motion blur remaining after OIS correction are refined through software to produce smooth final video.

The key point is that EIS knows what OIS corrected and by how much. By passing OIS actuator motion data to the EIS pipeline, over-compensation is avoided and the strengths of each system can be leveraged.

The best example of this structure is Google Pixel 2's (2017) Fused Video Stabilization. Google extracts gyro signals and OIS motion data together to precisely estimate camera movement, then synthesizes frames through a 3-stage pipeline with machine learning-based motion filtering. Sony Xperia's Optical SteadyShot also officially employs a structure that drives OIS and EIS simultaneously.

Angle-Based Role Division

As an approach to OIS-EIS collaboration, dividing roles based on correction angle is also discussed.

OIS's physical correction limit is approximately ±1 degree
EIS can provide practical correction up to approximately ±3 degrees

Using this difference, shake within ±1 degree is handled optically by OIS without quality loss, while larger movements exceeding ±1 degree are handled by EIS through software. This minimizes EIS crop amount while securing a wide correction range.

However, the extent to which this approach is implemented in actual commercial products varies by manufacturer, and most do not disclose their detailed internal structures.

Differences by Camera and Mode

One important point to note is that even on the same smartphone, the stabilization method changes depending on the camera and shooting mode.

Still photos → OIS only (EIS cannot be applied as it requires inter-frame comparison)
Video recording (main camera) → OIS + EIS combination often operates
Ultra-wide camera → EIS only (no OIS hardware)
Telephoto camera → Primarily OIS

Even modes marketed as providing powerful video stabilization, such as Samsung's "Super Steady" and Apple's "Action Mode," actually work by switching to the ultra-wide camera and maximizing EIS crop. Rather than sophisticated OIS+EIS collaboration, they are closer to modes that aggressively apply EIS.

Pros

Adding EIS software post-processing on top of OIS optical correction produces smoother video than either alone
Since OIS handles first-pass correction, EIS crop amount can be reduced
Most flagship smartphones adopt this combination, making it effectively the standard configuration for video stabilization

Cons

Synchronization between OIS motion data and the EIS pipeline is challenging to implement
OIS hardware costs are still included
Behavior varies by camera and mode, making it difficult for users to experience consistency
Implementation levels vary significantly between manufacturers, and most do not disclose their detailed structures

Comparison Summary

Attribute	OIS	EIS	HIS
Correction Method	Hardware (Physical)	Software (Digital)	Hardware + Software
Image Quality Impact	No loss	FOV loss, resolution degradation	OIS-level maintained + minimized EIS crop
Low-Light Performance	Excellent	Limited	Excellent
Still Photos	Effective	Ineffective	Same as OIS
Video Stabilization	Effective (range limited)	Effective (quality tradeoff)	Most effective
Module Cost	High	No additional cost	High
Module Size	Increases	No impact	Increases
Implementation Difficulty	Medium	Medium	High
Correction Range	±1° (physical limit)	~±3°	OIS ±1° + extended by EIS

Future Trends

The Rise of AI-Based EIS

While traditional EIS has been based on gyro data and conventional computer vision, deep learning-based motion estimation and frame synthesis technologies have been advancing rapidly in recent years. Google's Pixel series leads this field, evolving toward reducing crop loss by combining inter-frame interpolation with super-resolution.

Expansion of Sensor-Shift OIS

Sensor-shift was pioneered by Apple, followed by Chinese manufacturers like Huawei and OPPO, and Samsung finally adopted it with the Galaxy S26 Ultra. As major flagship manufacturers adopt it one after another, sensor-shift is becoming the new standard in the flagship market. Sensor-shift increases lens design flexibility, creating strong synergy with lens module miniaturization.

Software-Defined Camera

In the long term, the weight of hardware correction will likely decrease, with software correction combining sensor data and AI processing becoming more important. However, the limits of physics — particularly insufficient light in low-light conditions — cannot be fully overcome by software alone, so OIS is unlikely to disappear entirely.

Conclusion

Image stabilization is not a simple feature but a system design challenge where hardware, software, physics, and cost are complexly intertwined. There is no single "best" answer among OIS, EIS, and HIS. The optimal combination varies depending on product positioning, target price, and primary shooting scenarios.

In the next article, I plan to dive deeper into the internal structure of the EIS pipeline, covering gyro data processing, frame transformation, and smoothing algorithms.

This article was written based on hands-on experience in camera module development. What type of image stabilization does your smartphone use? If you find any errors in this article, please point them out in the comments. Feedback and opinions are always welcome.

What It's Like Being a Software Engineer at a Hardware Company — The Reality of SW Development Dragged Along by HW Schedules

Ross Kim — Tue, 07 Apr 2026 14:58:33 +0000

I used to work in software development in the mobile industry, and now I develop software at a camera module company. We make camera modules for smartphones and industrial devices. The core of the product is hardware — lenses, sensors, mechanical design — and software's role is to make that hardware work or to develop test programs.
Working as a software engineer in this kind of environment, you run into some unique situations. Things that would be hard to imagine at a pure software company (though, to be fair, I've never actually worked at one, so I can't say for sure). Here are a few experiences I've put together.
"We Can't Change the Hardware, So Fix It in Software"
This is the phrase you hear the most as a software engineer at a hardware company.
Once, the hardware team physically mounted a display in the wrong orientation. With assembly already complete, tearing it off and reattaching it didn't make sense in terms of cost or time. So the request that came to software was simple: "Just flip the display output direction in software."
This wasn't a one-time incident. When problems come up in hardware design, the cost of fixing them is high, so it's an everyday occurrence for the fix to land on software's plate. From software's perspective, work that was never part of the plan suddenly appears. And yet, the deadline doesn't change.
An Organization That Treats Software Like Hardware
The opposite pattern exists too.
Sometimes a fix that would clearly take just a few lines of code gets met with: "That's already been finalized, so changes aren't possible." Hardware-centric organizations tend to be very conservative about changes after design sign-off. Recutting molds or redesigning boards costs real money, so the mindset itself makes sense.
The problem is when that same standard gets applied to software. Software's essential strength is its flexibility — but when you lock it into the same process as hardware, inefficiency follows. Instead of a straightforward fix, you end up adding workaround logic or tacking on extra features. The code gets more complex with every iteration.
When Hardware Is Delayed, It's the Software Schedule That Shrinks
Product development has deadlines. Delivery dates promised to clients don't move easily.
The development sequence is usually hardware first, software second. You need the sensor and mechanical parts ready before you can load firmware, and tuning and validation have to happen on actual hardware — so this order makes sense.
But when the hardware phase gets delayed due to design changes or component issues, the overall deadline stays the same while the time given to software shrinks. Once the hardware team finishes their part, their stance becomes: "We've handed it off, so figure out the rest within the remaining time." (Or, as mentioned above, it becomes: "Just find a way to fix it in software.")
A software development period that was supposed to be a month can turn into two weeks. But the amount of work doesn't shrink. If this happened once or twice, you could push through with overtime. But when it repeats on every project, it just starts to feel normal.
So Is This Environment All Bad?
I'm not trying to just list complaints.
Working in this kind of environment, you naturally become a software engineer who understands hardware. You learn to read circuit diagrams and develop an instinct for designing software on top of physical constraints. I think that's an experience that's hard to gain in a pure software environment.
And when you keep responding to requests like "solve this hardware problem with software," the range of your problem-solving ability grows. Implementing defined requirements and creatively working around physical constraints are two different kinds of skills.
Being a software engineer at a hardware company is uncomfortable, but it's also a uniquely rewarding environment for growth.

I develop software at a camera module company. I write about real experiences from the field.