DEV Community: Jim Wang

什么是Online Softmax and Flash Attention？

Jim Wang — Sat, 20 Sep 2025 21:11:06 +0000

Softmax是Transformer模型架构中非常重要的一环。它所在的Attention模块虽然所需要的计算量不大，但也是不容忽视的一环。同时由于它本身的数学特性所造成的数据依赖，如果按照其原始方法来进行运算，会耗费大量的计算时间，因为它需要三次完整读取数据。

Online normalizer calculation for softmax 提出了online softmax，通过牺牲计算来节省数据读取次数，将三遍完整读取（3 passes）降低到两遍完整读取（2 passes）。

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness 则应用了类似的思想，更进一步，利用NVIDIA GPU的本地存储，将读取次数减少为一遍。

那么就让我们详细了解一下其中奥秘吧。本文会出现数学公式，但不要慌张，仅仅是简单的数组知识而已。同时也会辅以可以运行的Python代码，以便理解。

Softmax的数学表达

首先，softmax作用在一维vector上，而非多维tensor。

\vec{X} [1:N] ) = \frac {e^{x_i}} {\sum_{j-1}^{N} e^{x_j} }（1）

我们需要（1）先计算每个x的e^x并进行加总得到总和，（2）然后计算除法。

Safe softmax

但是由于e^x的特性，当x相对较大时，e^x就容易溢出，尤其是使用float16甚至更低精度的浮点表达时。因此我们需要对X进行normalization（归一化），并称之为saft softmax。

\vec{X} [1:N] ) = \frac {e^{x_i - \max(\vec{X})}} {\sum_{j-1}^{N} e^{x_j - \max(\vec{X})} } （2）

因此，softmax的三遍读取为（1）统计最大值（2）加总（3）除法

如果用Python来表示的话：

vec_x = [random.random() for _ in range(N)]
# 1st pass
max_x = max(vec_x)
# 2nd pass
sum_ex = sum([math.exp(x - max_x) for x in vec_x])
# 3rd pass
softmax_x = [math.exp(x - max_x) / sum_ex for x in vec_x]

Online softmax

但是，我们其实可以将（1）和（2）合并，在寻找最大值的同时进行加总操作。

如果我们能够构造两个数组M和S，它们的单元数量和X一样：

m_i = \max(\vec{x}[1:i]) \newline s_i = \sum_{j=1}^{i}{e^{x_j - m_i}} （3）

即，数组M保存的是数组X到当前index为止的最大值，而数组S保存的是类似softmax的加总（除数）。数组S的所有数值都和softmax的第二步加总的出来的结果的不一样，除了最后一个。也就是说，如果我们一个个读取数组X，并计算数组M和数组S，在计算到最后一个的时候，数组S的d_N正好等于softmax的除数。这样我们只需要一遍读取就能完成步骤（1）和步骤（2）。我们选择符号m来表示max最大值；选择符号s来表示sum加总。

这种奇思妙想不得不令我们想起dynamic programming。

如果我们展开数组S的计算过程：

s_1 = 1 （4）

s_i = s_{i-1} \times e^{m_{i-1} - m_i} + e^{x_i - m_i} （5）

s_N = \sum_{j=1}^{N} e^{x_j - m_N} （6）

我们可以证明公式6，即当i=N的时候，s_i即为softmax中加总之后的结果。

s_N = s_{N-1} \times e^{m_{N-1} - m_N} + e^{x_N - m_N} \newline = \sum_{j=1}^{N-1}{(e^{x_j - m_{N-1}} \times e^{m_{N-1} - m_N})} + e^{x_N - m_N} \newline = \sum_{j=1}^{N-1}{e^{x_j - m_{N}} } + e^{x_N - m_N} \newline = \sum_{j=1}^{N} e^{x_j - m_N} （7）

如果用Python来表示的话：

vec_x = [random.random() for _ in range(N)]

s_prev = 1 # previous s_{i-1}
m_prev = vec_x[0] # previous m_{i-1}
for i in range(1, N): # starting from the 2nd element
  x = vec_x[i] # read x
  m = max(m_prev, x) # calculate m_i
  s = s_prev * math.exp(m_prev - m) + math.exp(x - m) # calculate s_i
  s_prev = s # save as s_{i-1}

Attention的数学表达

\mathbb{A} = softmax(mask(\frac{\mathbb{Q} \cdot \mathbb{K}^T}{\sqrt{K}})) \cdot \mathbb{V} （8）

其中A, Q, K, V都是二维matrix，A, Q, K, V的shape分别依次是[N,K], [N,K], [M,K], [M,K]。但对于Transformer架构而言，N == M。mask() 函数旨在加载self-attention或者其他种类的masking，但对masking的探讨超越了本文范围，且因为食element-wise操作，所以不改变本文讨论的内容。

为什么要除以常数√K?
同样是为了归一化，这样在进行 Q·Kᵀ 时（1）不会溢出（2）分布过于极端导致softmax溢出（3）在训练过程中造成不稳定。

简化1：1/√K scaling通常是在进行点乘运算前对Q或者K进行，因此在后面的讨论中不再赘述。

简化2：以后的讨论我们还会简化masking的步骤。

经过简化，同时寻找X和Y符号来表示中间步骤的变量，attention可以表达为：

softmax(\mathbb{Q} \cdot \mathbb{K}^T) \cdot \mathbb{V} = softmax(\mathbb{X}) \cdot \mathbb{V} = \mathbb{Y} \cdot \mathbb{V} = \mathbb{A} （9）

选择符号X和x来表示softmax的自变量，与之前对于softmax的讨论统一；选择y来表示softmax的结果（可惜s已经被用了）；选择A来表示attention的结果。

注意：同时我们需要了解，在LLM中，N和M代表的是 batch_size * num_head * seq_length，而K代表的是head_dim。因此N和M是可以运行时动态变量（runtime dynamic variable），而K是编译时静态变量（compile-time static variable）。这个理解对于进行性能优化非常重要。

应用online softmax

如果将online softmax应用在attention上，同时将有如下两遍循环操作：

循环（1）

完全套用online softmax的操作：

x_{n,m} = \mathbb{X}[n,m] = \mathbb{Q}[n,:] \cdot \mathbb{K}^T[:,m] （10）

注意：此处的符号数量限制，为避免误导，将 n, m, k 分别定义为在N, M, K 三个dimension上的index。

可以看出，虽然matrix S是的shape为[N,K]，但是softmax只作用于其inner dimension。也就是说其outer dimension是只是重复的循环操作。因此为了简便，我们只看内循环操作，即矩阵X的一行操作：

x_m = \mathbb{Q}[n,:] \cdot \mathbb{K}^T[:,m] （11）

其中的符号 n 指代外循环变量 for n in range(N) ，同时符号 m 指代内循环变量 for m in range(M)

g_m = max(g_{m-1}, x_m) （12）

s_{m} = s_{m-1} \times e^{g_{m-1} - g_m} + e^{x_m - g_m} （13）

此处由于符号m被使用了，我们将max()的结果从符号m换成符号g（greatest，因为符号x, a, i都被使用了）

循环（2）

将类似思想套用在 Y•V 这个步骤上，则可以得到：

y_m = \frac{e^{x_m - g_M}}{s_M} （14）

\vec{A_0} = 0 \newline \vec{A_m} = \vec{A_{m-1}} + y_m \times \mathbb{V}[m,:] （15）

需要再次指出的是： m 指代内循环变量 for m in range(M)，而 k 指代外循环变量 for k in range(K)。因此（15）计算的是一个vector而不是scalar。这么做的好处就是每次计算出来的 y_m 可以用完就丢弃，同时这个表达方式可以引出我们之后对flash attention的讨论。

如果你跟我一样曾经觉得困惑的话，其实（15）并不难理解，只不过是将矩阵乘法的内外循环互换并将中间结果保存起来。

首先来看矩阵乘法：

\mathbb{A} = \mathbb{Y} \cdot \mathbb{V} \newline \mathbb{A}[n,k] = \mathbb{Y}[n,:] \cdot \mathbb{V}[:,k] = \sum_{m=1}^M(\mathbb{Y}[n,m] \times \mathbb{V}[m,k]) （16）

matrix_a[:,:] = 0 # initialize matrix A to be zeros
for n in range(N):
    for k in range(K):
        for m in range(M):
            matrix_a[n, k] += matrix_y[n, m] * matrix_v[m, k]

如果我们调换循环K和循环M，结果一致，只是计算顺序变了。我们从一个一个计算A的element，变成了先计算A的一行的中间值（partial sum）然后加总。

matrix_a[:,:] = 0 # initialize matrix A to be zeros
for n in range(N):
    for m in range(M):
        for k in range(K):
            matrix_a[n, k] += matrix_y[n, m] * matrix_v[m, k]

那么数学表示则变成：

\mathbb{A}[n,:] = \sum_{m=1}^M( \mathbb{Y}[n,m] \times \mathbb{V}[m,:]) （17）

而（15）中的 y_m 则是当前第n行的Y[n,m]，因此：

\mathbb{A}[n,:] = \vec{A_M} = \sum_{m=1}^M(y_m \times \mathbb{V}[m,:]) = \sum_{m=1}^M(\frac{e^{x_m - g_M}}{s_M} \times \vec{V_m}) （18）

所以循环（2）变成了：

\vec{A_m} = \sum_{m=1}^M(\frac{e^{x_m - g_M}}{s_M} \times \vec{V_m}) （19）

优化循环（2）

我们利用oneline softmax同样的思想，将计算 A_M 的过程转变成一个循环即A_m = func(A_{m-1})，然后构造一个新的A'_m来去掉数据依赖，因为我们只需要保证A'_M == A_M即可，而不需要中间过程相等。

\vec{A'm} = \sum{i=1}^m(\frac{e^{x_i - g_m}}{s_m} \times \vec{V_i}) （20）

当 m=M 时，则有：

\vec{A'M} = \sum{i=1}^M(\frac{e^{x_i - g_M}}{s_M} \times \vec{V_i}) = \vec{A_M} （21）

如果比较（20）和（19），仅仅是将m替换成了i，将M替换成了m，但是本质原理是用循环计算的方式来避免写入内存(avoid materialize matrix)。

如果我们将（20）展开，则可以得到：

\vec{A'1} = \frac{e^{x_1-g_1}}{s_1} \times \vec{V_1} \newline \vec{A'_m} = \vec{A'{m-1}} \times \frac{s_{m-1} \times e^{g_{m-1} - g_m}}{s_m} + \frac{e^{x_m-g_m}}{s_m} \times \vec{V_m} （22）

合并循环（1）和优化后的循环（2）

因为两者的思想和应用方法是一样的，即在循环中计算中间值，仅保证最终结果与目标一致即可，因此我们可以将其合并为一个循环。同时我们将其表达为矩阵乘法形式。以下为同一循环中的四个步骤：

(1) Score

\mathbb{X}[n,m] = \mathbb{Q}[n,:] \cdot \mathbb{K}^T[:,m] （23）

(2) Max

\mathbb{G}[n,1] = \mathbb{X}[n,1] \newline \mathbb{G}[n,m] = max(\mathbb{G}[n,m-1], \mathbb{X}[n,m]) （24）

(3) Sum

\mathbb{S}[n,1] = 1 \newline \mathbb{S}[n,m] = \mathbb{S}[n,m-1] \times e^{\mathbb{G}[n,m-1] - \mathbb{G}[n,m]} + e^{\mathbb{X}[n,m] - \mathbb{G}[n,m]} （25）

(4) Attention

\mathbb{A}1 [n,:] = \frac{e^{\mathbb{X}[n,1] - \mathbb{G}[n,1]}}{\mathbb{S}[n,1]} \times \mathbb{V}[1,:] \newline \mathbb{A}_m [n,:] = \mathbb{A}{m-1} [n,:] \times \frac{\mathbb{S}[n,m-1] \times e^{\mathbb{G}[n,m-1] - \mathbb{G}[n,m]}}{\mathbb{S}[n,m]} + \frac{e^{\mathbb{X}[n,m] - \mathbb{G}[n,m]}}{\mathbb{S}[n,m]} \times \mathbb{V}[m,:] （26）

用Python表示：

matrix_q = [[random.random() for _ in range(K)] for _ in range(N)]
matrix_k = [[random.random() for _ in range(K)] for _ in range(M)]
matrix_v = [[random.random() for _ in range(K)] for _ in range(M)]
matrix_a = [[0.0 for _ in range(K)] for _ in range(M)]

for n in range(N):
  for m in range(M):
    x = 0.0
    for k in range(K):
      x += matrix_q[n,k] * matrix_k[m,k]

    g = max(g_prev, x) if m > 0 else x

    e = math.exp(x - g) if m > 0 else 1.0
    s = s_prev * math.exp(g_prev - g) + e if m > 0 else 1.0

    for k in range(K):
      if m > 0:
        matrix_a[n,k] = (matrix_a[n,k] * s_prev * math.exp(g_prev - g) + e * matrix_v[m,k] ) / s
      else:
        matrix_a[n,k] = e * matrix_v[0,k] / s

    g_prev = g
    s_prev = s

What is ML runtime?

Jim Wang — Tue, 16 Sep 2025 05:10:06 +0000

Popular ML runtime frameworks include ExecuTorch, ONNX runtime, TensoRT. They bridge the gap between model training and model deployment.

ML models are usually trained with PyTorch on a GPU. But directly deploying them in PyTorch format is cumbersome. PyTorch’s inference runtime libtorch is written in C++ and very heavy. Other ML runtime like ExecuTorch, also coming from the same PyTorch team, is built for embedded environment with lightweight and efficiency as its primary goals.

ML runtime is created to be deployed on different hardware backends, GPUs, NPUs, TPUs, CPUs, DSPs and other accelerators. So that models can be delivered in one single format (a DAG graph that describes the relationship between tensors and operations, as well as a binary of weights) and deployed on many different platforms.

思考：研究与工程(Thoughts about Research and Engineering)

Jim Wang — Sat, 30 Aug 2025 19:19:33 +0000

在一个尚处于快速发展和迭代的领域，例如AI，由于各种新方向的不确定性，研究和工程常常很难互相配合甚至互相纠结掣肘。从而导致，要么工程实现犹犹豫豫、方向摇摆不定；要么研究做得差，无法做到引领方向、开拓新领域的作用。

工程实现的主要矛盾是执行效率。所以它不能够常常进行大范围的功能变更，同时要以最终产品（deliverables）作为主要导向。而且最终产品的功能（functionality）和质量（quality）是其主要量化衡量标准。因此工程实现中的流程化是非常重要的手段。良好的流程能够提高大规模组织合作效率，提升可追溯性以及可重复性。

前沿研究的主要矛盾是创新性以及量化评估。创新性很好理解，就是需要跟踪业界的最新发展动态，并在此基础上发展出符合自身情况的创新性想法，并加以实现。量化评估则涉及到一个系统能够快速准确的对不同的创新性想法和实现方法进行量化分析，并得出不同条件下的最优选择结果。因此研究过程中就不能循规蹈矩，需要跳出条条框框（think out of the box）。所以前沿研究是反流程的，而且需要反流程。但是需要注意的是，公司内部的研究团队毕竟和大学不一样，它们需要为最终产生经济效益而服务，并且有较强的时效性。因此能够对创新性想法进行原型化（prototyping）是非常重要的。

研究与工程的共同点就是需要非常强的Ownership意识。研究团队更多的分散作战，类似游击队。每个小团队多则数人，少则一人，他们为自己的方向选择、成本控制以及原型化负责。团队之间的互相配合主要在于分享思想，所以他们的组织架构更适合自下而上的组织方式。工程实现则类似集团军会展，团队规模依据项目不同从数十人、数百人到上千人，需要很好的纪律性以及互相配合，责任分工需要非常明确。所以他们的组织架构更适合自上而下的组织方式。

为促进研究与工程的相互理解和交流，以及比较好的做法就是借助于矩阵架构，鼓励人员内部流动，防止出现结构性的组织壁垒。在矩阵架构里，应该以纵向的项目负责人或者负责团队为主导，在组建项目实施团队的过程中，他们需要“说服”研发人员加入他们的项目。而横向的人员管理（people manager）则应该处于配合角色，主要侧重横向不同领域的人才招聘、团队建设、职业发展和领域内信息交流。由项目负责团队和人员管理团队互相商讨确定项目实施团队的规模、日程计划和日常化项目管理。

思考：建立信任的几种方式 (How to gain trust?)

Jim Wang — Sat, 30 Aug 2025 19:18:03 +0000

这一段时间在学习B站上资深HR专家王新宇的沟通课，其中有一个主题正好切入到了我工作中遇到的困惑。如何建立信任？

为什么？

作为团队中的Senior IC（individual contributor），我的日常工作中百分之五十以上的时间需要跟不同团队的manager或者engineer打交道。为的是获得他们的支持或者是合作。而其中很多时候是需要跟陌生的面孔打交道。
如何能够快速获得其他团队的成员的支持常常基于最基本的利益以及两者之间的信任关系。
利益问题是根本的，与组织架构和项目本身的性质有关，很难短时间内由个人左右。
而快速建立起信任关系则很有可能成为项目成功的关键因素。

为什么说利益问题是根本？假设一个场景，如果合作方能够获得正向激励，那么项目伊始对方就有比较强烈的意愿进行合作，则信任度能够很快建立起来。而相反的，如果合作方会利益受损（他们会因此失去一些scope甚至headcount），那么抵触情绪则会令另一方的建立信任的努力完全成为对牛弹琴。

但是利益格局也不是完全不可改变的，双赢的机会是需要去发掘的。这个时候成为一个好的Story teller就很关键。

怎么做？

首先总结一下，后面再展开：

信守承诺
展现专业度
个人亲近程度

信守承诺

从长期的角度来看，“信守承诺”是唯一可行的建立并维持信任的方法。当然是越重大的事情、越困难的承诺在对方的心目中越有分量。
但是在交往初期，从小事开始信守承诺是快速建立起信任的方法，比如说守时。

展现专业度

在工作关系中，高专业度往往可以简单有效的另对方产生信任。专业程度的考量往往因人而异，是相对的。这也很容易理解，因为对于初入职场的人而言，他们往往对于专业还不甚了解，那么这个“是否专业”的阈值就比较低。而要在顶尖的专家所擅长的领域展现专业度，则是非常困难的。这个时候不妨另辟蹊径，从其他的方向出发，比如说运动、艺术等领域先入手。这个时候如果有一些业务擅长的兴趣爱好就能达到较好的效果。而且如果这个领域也恰好是对方感性的方向，则更加事半功倍。

个人亲近程度

想要快速提升信任度则可以借助于个人亲近程度。可虽然见效快，但很容易饱和，且不容易持久。
对于刚刚展开的工作关系，闲聊有的时候能够快速拉近亲近关系，以此建立初期的相互信任。
所以，有的时候：“沟通的目的就是沟通本身。”

System Performance of Edge AI Applications is Beyond Models

Jim Wang — Sat, 30 Aug 2025 19:15:56 +0000

After working on enabling use-cases for edge AI accelerator for more than 2 years now, here is some of my thinking about system performance.

Thoughts: Much More Beyond Model

One of the biggest challenges we've been facing is how to improve overall system performance. (While the other is how to quickly enable a model from CPU / GPU backend to custom AI accelerator backend.)

According to my observation, system performance of edge AI has the following attributes:

End users care about end-to-end performance that includes latency, power and throughput (in this particular order).
End-to-end performance is much more beyond just the model itself, with extra overhead from:
1. Data loading, ie. from DRAM to on-chip memory.
2. Data transfer, ie. from CPU to GPU over PCIe.
3. Memory copy, ie. user space from/to kernel space.
4. IPC (inter-process communication), ie. interrupting main CPU that runs Linux.
5. Application specific pre/post-processing, ie. tokenization, YUV2RGB and etc.
AI applications are different from typical software applications because:
It has a model loading phase and model size is usually very large.
Once loaded, a model can be reused in later inferences.
Inference may consumes / produces large data blobs, depending on use-cases.
Inference is usually not the final stage. One exception is LLM (large language model).

Lots of effort has been focusing on optimizing the model itself, for performance (accuracy and etc.) and performance (latency and power). However, less focused on optimizing the overall performances as a whole system.

Real-life example

One real-life example: in many of our use-cases over 60% to 80% of the end-to-end latency comes from software stacks above model inference, in some extreme case this number goes above 90%. As you can imagine how much time will be spent on data transfer / memory copy / IPC and post-processing when a high resolution image captured from an embedded camera has to be tiled and transferred back-and-forth over PCIe while being controlled by Android OS and post-processed by software running on GPU.

Project management and politics

Another very important factor is more about project management and politics. An end-to-end use-case usually involves many subsystems that are created / managed by different teams and most likely under different VPs. Every one of them cares about their own subsystems and have their own priorities. The first challenge is not to come up with a technical solution, but to persuade everyone to recognize "end-to-end performance IS a top-priority problem that needs to be fixed". And every steps afterwards are projects of their own: define optimization metrics and targets; define budgets for each components; system modeling; system integration and etc. After all these, we can finally reach to the stage of system performance optimization.

System View

To understand the overall performance of edge AI applications, it's pretty important to establish a system view. It requires you to take a step back and zoom out, to see different components of software and hardware as a system without any bias. Unfortunately this is not easy. Sometimes due to the background of one person, either from hardware or software; sometimes due to their personal favorites of different components.

It's also important to involve experts from different domains into the discussion, so that all aspects are covered and they can provide insights from different angles.

Data-driven decision making

To mitigate all these difficulties, one of the weapon I always use is "data-drive decision making". When you have data, the decision making process can converge much faster. It's hard to find any engineers that would ignore real data either from a modeling system or a prototype, although they may challenge your methods to get those data or they may have different interpretations of the same data.

To get data, you will need:

A clear definition of final metrics, which usually are latency, throughput and power.
Either a modeling system or a prototype.
1. A list of compromises and abstractions in these systems.
2. A set of profiling methods built into your model or prototype.

Practice: Profiling and Optimization

In this section, I'm going to discuss the practices of profiling and optimizing latency, throughput and power of a system.

Latency

Latency is probably the easiest to profile among all three.

For simplicity, cycle counters can be used in software to measure latency. But you shall consider the following factors:

Measure not only the end-to-end latency, but also every stage of your software stack.
Reporting could be causing latency itself, especially if you use printf and low speed interface like UART.
- Instead, you can save intermediate data into memory and dump them out periodically. Some debug system includes an independent MCU that can help with this kind of task.
CPUs that software is running on can adjust their frequencies and also can go to sleep with clock stopped.
- Most scenario, OS provides more sophisticated routines to get wall-clock time.
- For early stage development, you can just disable DVFS (dynamic voltage frequency scaling) and auto-sleeping.
If multiple CPUs are involved, they need to have a common agreement of time.
- Usually a heterogenous SOC provides a low-frequency always-running system timer.
Timer's resolution. To conserve energy, always-on timer usually runs at a frequency of mega-hertz or even kilo-hertz.
- You should consider the following numbers: the latency of an IPC into Linux usually takes 200us to be answered; the latency of an IPC into RTOS usually takes 50us to be answered; single-lane PCIe 4.0 is maxed at 2GB/s, so it will take 600~700us to transfer 1MiB of data.

To optimize latency, you could consider:

Optimize the algorithms
- Other than normal optimization methods, you should look at if any processing heavy functions can be accelerated by using vector processors.
Optimize memory copying
- Even though we will not be able to achieve zero memory copy due to security reason or for maintaining abstraction levels, we surely can try to avoid useless memory copy as much as possible.
Reduce IPC overhead
- Some times due to program / functionality partitioning, IPCs are used excessively between different processes on the same CPU or between different CPUs. Optimizing for the system to combine and reduce these partitions can lead to a more efficient system.
- However, you do need to take into consideration that in production environment if a certain processor is over-subscribed or not. Because putting too much work on one processor can dramatically reduce its overall throughput and affects each workload's latency.
Caching
- Some more fundamental techniques like caching can also be used to help with latency. You should consider cache to be a more general terminology other than CPU cache. For example, keep frequently used models' weights in GPU memory to avoid repetitive reloading from CPU memory.
Parallelization
- Run different parts of the model on different execution engine can improve latency, but will require software support for synchronization. E.g. future and await support in PyTorch.

Throughput

Throughput is usually measured from the end user's point of view, ie. frame per second for a video conference call application.

To optimize throughput, you could consider:

Optimize the latency
- If the latency of each frame is small, overall throughput is improved.
Parallelization
- Parallelizing data transfer and model inference of different models, or the same model but different inferences. E.g. running video enhancement ML models on PCIe attached GPU, you can process frame N on GPU while transferring output of frame N-1 to CPU and transferring input of frame N+1 from CPU.
Context switch
- QoS (quality of service) is another important metric to measure system throughput, and it cares more about the latency of high priority tasks than low priority ones. System needs to support saving context of running low priority task, and restoring its context after high priority task is finished.

Power

The power-related metric shall be considered from three separate angles, max current, TDP (thermal design power) and total energy.

Max current means the maximum transition current the hardware draws, and is limited by the transition current that can be provided by PMIC (power management IC) and power supply mechanism. Higher the max current is required, more expensive PMIC you will need.
TDP is the average power the hardware consumes over a relatively long period of time, e.g. seconds to minutes, and is limited by the cooling mechanism and overall industry design. Higher TDP, better heat sink and cooling design you will need.
Total energy is the energy needed to complete the task, no matter how long it takes, and is limited by the battery the device has. Higher total energy, larger battery you will need.

For edge devices, if it's powered by battery, all three factors are important and needed to be considered.

To optimize power, the most effective weapon is to lower the voltage. That's one of the strongest arguments to push the envelop of the semiconductor process from 28nm to 5nm. Even with the same semiconductor process, choosing lower nominal voltage can give you benefit of low power by trading-off with top frequency and design effort.

To optimize max current, you could consider:

Trade-off latency by lowering the frequency
- Lower the frequency of compute units can help to lower the max current directly, but with sacrifice of overall latency.
Power throttling
- Pacing different tasks or different parts of the same task to run in sequential fashion instead of in parallel. Obviously with sacrifice of overall throughput and latency.
DVFS (dynamic voltage and frequency scaling)

To optimize TDP, you could consider similar mechanisms of optimizing max current, but in a much longer time frame. So instead of maneuvering tasks, you need to consider turning off power domains and functionalities for seconds to minutes until the rush hour has passed.

To optimize energy, you could consider:

Optimize algorithm to find a low-power alternative.
Use specialized hardware accelerator to replace general computing.
Reduce external memory access, but utilize on-chip memory and cache instead.
... (other software, hardware and physical mechanisms)

Software is King, Especially in AI

Jim Wang — Sat, 30 Aug 2025 19:08:37 +0000

"Software is king"

It's a hard statement to make, as a veteran hardware engineer. Lots of pride and ego to swallow. However, it's truly based on my observations in the industry. Allow me to reason it with the following arguments:

(1) Software has direct impact on user-experience

It is the last layer that users interact with. And user-experience is the first and last thing will make you standout in the business world with so many competitors. Software directly decides how your users use your product.

(2) Software has the power and the flexibility

Well designed software can dramatically improve system performance and power consumption. It's a known fact that in a software hardware system, more benefits in performance and power you will be able to extract when you move up the system layers. The trade-off is the techniques used in optimization become more specific and tied to applications when moving up the layers.

It's also a double-edge sword. Bad designed software can harm the performance and power consumption in ways very hard to detect and debug.

(3) Hardware engineers underestimate software's complexity

In a hardware majority company or team, like traditional IC design houses, decision makers that come from the hardware branch of the organization unintentionally underestimate software's complexity. Therefore, either they don't allocate enough resource in software, or start to involve software engineers into the decision making process too late.

This tough reality often sends the situation downhill, and software becomes the absolute bottleneck. The throne in this situation is a hot spot.

Especially in AI

Why do I say it's even more seriously in AI domain?

(1) AI is evolving fast

AI is evolving so fast. Models are improving in terms of not years or months but weeks or days. Different use-cases and business models are booming with so many startups every week.

However, hardware simply cannot catch up. The cycle to build and deploy a hardware solution is usually months. If it involves making custom silicon, then we are talking about years. Therefore the trend is moving the complexity from hardware to software.

With most generalized hardware, fast evolving software can be changed in much faster pace.
With "well designed" software stack, like LLVM compilers, we can move lots of optimization to compile time and fully utilize the power of hardware.

(2) AI is complex in software, simple in hardware

Mathematically, AI is complex in software but simple in hardware, because the atomic operation in deep-learning models is MAC (multiplication-addition-accumulation). The complexity comes from the total number of MACs we need to do. How to run them fast and with less power is the question keeps most AI engineers up in the night.

My Answer

My answer, as a veteran hardware engineer, is to develop myself into system-level position, to understand both software and hardware

Project Planning Process

Jim Wang — Sat, 30 Aug 2025 19:07:03 +0000

Have a Vision: Without a clear vision outlining the final goals, how can we measure our progress? And it's essential to clearly communicate this vision with all team members.
Break Down into Tasks and Milestones: To turn our vision into reality, we must break it down into tasks and establish milestones.
Identify the Critical Path(s): With goals, tasks and milestones in place, we need to identify the most critical path(s). What are the determining factors for them? They could be technical unknowns, resources constraints, or external dependencies. What are the associated risks?
Define the Action Plan(s): Based on the critical path(s), their determining factors, and the risks, how can we mitigate them? A concrete action plan is essential.

It is an iteration process:

while True:
    build()
    measure()
    learn()

The key is maintaining transparency and open communication.

My Tactics With Research Projects

Jim Wang — Sat, 30 Aug 2025 18:58:36 +0000

Research projects often come with a lot of unknowns. Many engineers find this unsettling because it’s less straightforward than math or digital realm. However, I think we should welcome these uncertainties. They reflect the real challenges of real-world task and provide a chance to showcase our capability, making our work more engaging and lively.

Here’s my strategy to handle a research project:

Clarify requirements thoroughly

It’s important to know not just what is needed but also why it is needed. Finding out why can be tough because people might not want to explain their core motivation, which could be personal or office politics related sometimes. But knowing this helps make better decisions.

Check your inventory

Reach out to all your connections to find out what technical stacks are available to you.
Double-check the resources you’re told, use strategy "trust but verify". This part is extremely important because most of the time the people disappoint you are not someone you don't trust, but instead someone you fully trust in the beginning but failed you later.
Ask for schedule or roadmaps for future available resources. Add extra buffer time for each of those resources, depending on two factors (1) how important these deliverables are to their owners (2) how much influence do you have on their owners. If we eliminate the possibility of lying, still we must know that people are busy and have their own priorities. Usually they tell you the schedule or roadmaps based on best scenarios.
For critical external dependencies, always have plan B.
"Inventory" not only includes existing technical stacks, but also team members' expertise. For long-term projects, you may have time to hire experts or ask team members to pick up new skills; for short-term projects, you may have to pick technical stacks that your team is most familiar with.

Brainstorm with your team

Lay out all the facts and assumptions. There must be a clear separation between these two categories.
Talk through different ideas with your team, and very importantly try to bring out ideas from your team members. If you are a senior member, talk last.
Within all the ideas and options, there must be something that require further confirmation or investigation. Take some time to do more investigation, and remember to delegate those well defined tasks to your team members.

Make decisions and move forward

Spend enough time to think over your options, but not too long. Then make decisions together with your team, and very importantly tell them the "why".
Stick to your decision once it’s made, and keep going until you reach an major milestone in the project. Then you can slow down to reflect and diverge some resource to solve technical debts. But before that, block out all the noises and keep executing.

Set milestones and celebrate small wins:

Set clear milestones and check your progress as you go.
- Milestones must be verifiable, even better if you can find a quantitative criteria or automated checks.
- Milestones must be carefully paced, usually once every months.
- Milestones have their importance, some may be major milestones, some may be small checkpoints.
Define milestone and their criteria, publish them to your team member. This is a very good / suitable task for team leaders.
Celebrate every success, big or small, but keep it real to stay focused. Don't fool yourself with marketing materials.

Ask for help

Always be ready to ask for help if you need it, and sometimes prefetch, to anticipate what kind of road blocks there will be and think about solution, is also necessary.
Help others too, which can help build a good reputation.
Find help and find solution to road blocks shall be the top priority tasks for team leaders.

Thank everyone and keep promoting your work

At major milestones, send out emails or make posts to promote your team's work with demos, and thank everyone who helped.
The project doesn’t end when you reach your goals. Keep promoting the work, especially in big companies where getting noticed are key.

ChatGPT翻译版本

研究项目往往充满了很多未知因素。许多工程师发现这令人不安，因为它不如数学或数字领域那样直接。然而，我认为我们应该欢迎这些不确定性。它们反映了现实世界任务的真正挑战，并提供了一个展示我们能力的机会，使我们的工作更加有趣和生动。

以下是我处理研究项目的策略：

彻底澄清需求

重要的是不仅要知道需要什么，还要知道为什么需要它。找出原因可能很困难，因为人们可能不想解释他们的核心动机，这有时可能与个人或办公室政治有关。但了解这一点有助于做出更好的决定。

盘点你的资源

联系你所有的人脉，找出你可以使用的技术栈。
对你被告知的资源进行双重检查，使用"信任但验证"的策略。这部分极其重要，因为大多数时候让你失望的人不是你不信任的人，而是你一开始完全信任但后来让你失望的人。
询问未来可用资源的时间表或路线图。为每个资源添加额外的缓冲时间，这取决于两个因素：(1)这些交付成果对其所有者有多重要 (2)你对其所有者有多大影响力。即使我们排除说谎的可能性，我们仍必须知道人们很忙并有自己的优先事项。通常他们根据最佳情况告诉你时间表或路线图。
对于关键的外部依赖，始终要有备用计划。
"资源清单"不仅包括现有的技术栈，还包括团队成员的专业知识。对于长期项目，你可能有时间雇用专家或让团队成员学习新技能；对于短期项目，你可能必须选择你的团队最熟悉的技术栈。

与你的团队头脑风暴

列出所有事实和假设。这两个类别之间必须有明确的区分。
与你的团队讨论不同的想法，非常重要的是尝试从你的团队成员那里获得想法。如果你是高级成员，最后发言。
在所有的想法和选择中，必须有一些需要进一步确认或调查的内容。花一些时间做更多调查，记住将那些明确定义的任务委派给你的团队成员。

做出决定并向前推进

花足够的时间考虑你的选择，但不要太长。然后与你的团队一起做出决定，非常重要的是告诉他们"为什么"。
一旦做出决定就坚持下去，继续前进直到你达到项目的一个重要里程碑。然后你可以放慢节奏进行反思，并分配一些资源来解决技术债务。但在此之前，屏蔽所有噪音并继续执行。

设定里程碑并庆祝小成就

设定明确的里程碑并在进行过程中检查你的进度。
- 里程碑必须是可验证的，如果你能找到定量标准或自动检查就更好了。
- 里程碑必须仔细安排节奏，通常每月一次。
- 里程碑有其重要性，有些可能是主要里程碑，有些可能是小检查点。
定义里程碑及其标准，并将其发布给你的团队成员。这对团队领导者来说是一个非常好/合适的任务。
庆祝每一个成功，无论大小，但要保持真实以保持专注。不要用营销材料欺骗自己。

寻求帮助

如果需要帮助，随时准备寻求帮助，有时预先获取，预见会有什么样的路障并思考解决方案，也是必要的。
也要帮助别人，这有助于建立良好的声誉。
寻找帮助和寻找路障解决方案应该是团队领导者的首要任务。

感谢每个人并继续推广你的工作

在重要里程碑时，发送电子邮件或发帖通过演示推广你团队的工作，并感谢所有帮助过的人。
项目不会在你达到目标时结束。继续推广这项工作，特别是在大公司中，获得关注是关键。