DEV Community: HornGang

Elixir 编译器并发优化失败复盘

HornGang — Sat, 18 Jul 2026 09:15:23 +0000

Elixir 编译器并发优化失败复盘

背景

Elixir 编译器中，每个模块的编译大致分为以下几个步骤：

eval_form                                    # 宏展开 + 模块体编译（父进程）
  ↓
elixir_erl_compiler:spawn(fun() ->           # 启动子进程
  ...
  Types:infer                                # 类型推断
  translate_typespecs_for_module             # 类型规约翻译（@type, @spec 等）
  elixir_erl:compile                         # → dynamic_form → typespecs_form → docs_chunk → checker_chunk → noenv_forms
  ...
end)

观察发现 Types:infer（类型推断）和 translate_typespecs_for_module（类型规约翻译）都在 spawn 内部顺序执行。如果这两者无数据依赖，理论上可以并行化，缩短模块编译时间。

本文记录了这个尝试的全过程——从分析、实现、调试到最终放弃。

分析：数据依赖

`Types:infer` 的输入输出

输入：Module, File, Attributes, AllDefinitions, UsedPrivate, Env, CheckerInfo

输出：{Signatures, Unreachable}

这些输入全都是函数参数或在执行前已从 ETS 表读取完毕的值。Types:infer 不读写 ETS 表，是纯函数式计算。

`translate_typespecs_for_module` 的输入输出

输入：DataSet, DataBag（两个 ETS 表的 ID）

输出：{types, specs, callbacks, macrocallbacks, optional_callbacks}

它从 ETS 表中读取类型/规约定义（@type、@spec、@callback 等），将其转换为 Erlang 内部表示。它使用 :ets.take/2 破坏性读取。

数据依赖图

eval_form
  │
  ├──populates ETS tables (DataSet, DataBag)────┐
  │                                              │
  ▼                                              ▼
Types:infer(AllDefinitions, ...)    translate_typespecs_for_module(DataSet, DataBag)
  │                                              │
  ├──Signatures─────────────────────┐            │
  ├──Unreachable────┐               │            │
  │                 ▼               ▼            ▼
  │         dynamic_form      typespecs_form ──→ docs_chunk
  │                 │               │
  │                 └───────┬───────┘
  │                         ▼
  │                  checker_chunk(needs Signatures)
  │                         │
  └─────────────────────────┼─────────────────────┘
                            ▼
                     load_form → noenv_forms

关键发现：

dynamic_form 依赖 Unreachable（来自 Types:infer），所以无法与类型推断完全并行
translate_typespecs_for_module 不依赖任何 Types:infer 的输出——它只读 ETS 表
理论上 translate_typespecs_for_module 可以和 Types:infer 完全并行

预期收益：translate_typespecs_for_module 的耗时被隐藏在 Types:infer 背后，对于大型模块预期节省 5-15%。

实现：三次迭代

第一次尝试：spawn_monitor + receive

最直接的方式——子进程中用 spawn_monitor 启动 typespec 翻译进程，然后用 receive 等待结果：

%% 在 elixir_erl_compiler:spawn 的 fun 内部
{Pid, Ref} = spawn_monitor(fun() ->
  exit({ok, translate_typespecs_for_module(DataSet, DataBag)})
end),

{Signatures, _} = Types:infer(...),

receive
  {'DOWN', Ref, process, _, {ok, Result}} -> Result;
  {'DOWN', Ref, process, _, {Class, Reason, Stack}} ->
    erlang:raise(Class, Reason, Stack)
end

结果：CaseClauseError 异常，异常值是一个 #Reference<...>。

参考值 (#Reference<...>) 顺着编译管道向上传播，最终在 Elixir 代码中模式匹配失败。这说明 receive 收到了意料之外的消息格式。

第二次尝试：spawn + make_ref + receive

用 erlang:spawn + make_ref 替代 spawn_monitor：

Parent = self(),
Ref = make_ref(),
spawn(fun() ->
  try
    Result = translate_typespecs_for_module(DataSet, DataBag),
    Parent ! {Ref, {ok, Result}}
  catch
    ...
  end
end),

receive
  {Ref, {ok, Result}} -> Result
end

结果：receive 永远超时（10 秒后触发 after 10000）。子进程没有发送消息回来。

第三次尝试：ETS 表通信

放弃 receive，改用 ETS 表做进程间通信。父进程（在 spawn 之前）创建 ETS 表并启动 typespec 子进程，子进程（spawn 内部）轮询读取：

%% 父进程中（eval_form 之后）
TypespecETS = ets:new(...),
ets:insert(TypespecETS, {done, false}),
spawn(fun() ->
  Result = translate_typespecs_for_module(DataSet, DataBag),
  ets:insert(TypespecETS, [{done, true}, {result, Result}])
end),

%% 在 elixir_erl_compiler:spawn 内部
Types:infer(...),

%% 轮询 ETS 表（timer:sleep(1) + 最多 100000 次）
TranslatedTypespecs = poll_typespec(TypespecETS)

结果：编译成功，但 Benchmark 显示性能下降（polling 的 1ms sleep 每次编译叠加）。

第四次尝试：非阻塞检查 + 回退

去掉轮询，改为"可用则用，不可用则回退"：

case ets:lookup(TypespecETS, done) of
  [{_, true}] ->  % 子进程已完成
    read_result(TypespecETS);
  _ ->
    timer:sleep(2),  % 再等 2ms
    case ets:lookup(TypespecETS, done) of
      [{_, true}] -> read_result(TypespecETS);
      _ ->  % 回退到原始顺序执行
        ets:delete(TypespecETS),
        elixir_erl:compile(ModuleMap, Signatures)  % compile/2 内部处理 typespecs
    end
end

结果：编译成功，所有测试通过。但出现了数据竞争 bug：当回退路径被触发时，父进程也调用 translate_typespecs_for_module，与子进程的 :ets.take 破坏性读取互相干扰，导致 type undefined 编译错误。

修复后 Benchmark 结果显示性能差异在噪声范围内。

Benchmark 数据

使用 5 个不同复杂度的模块，各编译 10 次取均值：

模块	原始 (ms)	修改后 (ms)	差异
无 typespec	2.50	2.62	+5%
少量 typespec	3.02	3.18	+5%
30 个 type + spec	12.86	13.43	+4%
30 个 callback	5.81	8.46	+46%
混合（50 spec + struct）	19.16	19.14	-0.1%

实际收益接近于零，甚至在某些场景退化。

失败原因分析

1. 收益窗口太小

Types:infer 的耗时远大于 translate_typespecs_for_module。对于典型模块：

步骤	相对耗时
`Types:infer`	~80%
`translate_typespecs_for_module`	~5%
其他（dynamic_form, noenv_forms 等）	~15%

即使完全隐藏 typespec 翻译时间，总收益也仅仅 5% 左右。在 Benchmark 中，这个收益被通信开销完全抵消。

2. 进程间通信开销

方式	开销
`spawn_monitor` + `receive`	进程创建 + 监控 + 消息传递
`spawn` + `make_ref` + `receive`	进程创建 + 消息传递
ETS 表 + 轮询	进程创建 + ETS 写 + 1ms sleep/次
ETS 表 + 非阻塞检查 + 回退	进程创建 + ETS 读写 + 可能的 2ms 延迟

对于每次模块编译，这些开销是固定的。当一个模块的 typespec 翻译时间本身就 <1ms 时（大多数情况），通信开销超过了收益。

3. `receive` 在 `elixir_erl_compiler:spawn` 内部不可用

这是最令人困惑的问题。receive 在 elixir_erl_compiler:spawn 创建的进程内部表现异常——要么收到格式错误的消息，要么永远阻塞。如下最小复现：

%% 在 spawn 内部：
P1 = self(),
Ref = make_ref(),
erlang:spawn(fun() -> P1 ! {Ref, ok} end),
_Received = receive {Ref, ok} -> ok end,  % 正常工作！

但当子进程做真实运算时失败。根本原因可能是 ETS 锁定或进程链接导致的信令干扰——elixir_erl_compiler:spawn 内部使用了 spawn_monitor，而嵌套的 spawn/spawn_monitor 产生的 DOWN 消息可能污染进程邮箱，导致 receive 收到意外的系统消息。

4. `:ets.take` 的破坏性读取

translate_typespecs_for_module 使用 :ets.take 来读取 ETS 表（该操作会删除条目）。当两个进程同时调用这个函数时（并发路径 + 回退路径），第二个调用会看到不完整的数据，导致 type undefined 错误。

5. 架构约束

elixir_erl_compiler:spawn 是一个同步阻塞调用——父进程必须等待子进程完成。这意味着任何想在父进程侧做的并行工作都会增加父进程的阻塞前时间，无法"隐藏"在 spawn 后面。

经验教训

先测量，再优化。在分析阶段应该先用 eprof/fprof 确认瓶颈在哪里，而不是凭直觉猜测。
小收益不值得大改动。5% 的潜在收益不值得引入进程间通信、并发控制和回退逻辑的复杂性。
Erlang 的 receive 在嵌套进程中的行为需要警惕。elixir_erl_compiler:spawn 内部的 spawn_monitor 产生的系统消息不是一个干净的进程邮箱环境。
ETS 共享状态需要小心。:ets.take 等破坏性操作在并发场景下会导致竞态条件。
编译器的数据依赖往往是不可规避的。如果两个步骤共享同一个 ETS 表或数据结构，并行化往往需要复杂的加锁或复制。

后续可行的优化方向（不在此次范围内）

减少 ETS 读取次数：elixir_module:compile/7 的 spawn 内部有约 10 次独立的 bag_lookup_element，可以批量一次读取
模块间并行编译：Kernel.ParallelCompiler 已经实现，可以进一步优化调度策略
增量编译：跳过未更改模块的 typespec 翻译和类型推断
lazy docs_chunk：不在编译时生成文档 chunk，而是在需要时按需生成

Relationship or ability, both of these criteria are used

HornGang — Sun, 06 Oct 2024 18:45:21 +0000

In society, when evaluating a person's ability and value, there are often two different criteria: one is "based on relationships" and the other is "based on ability". Both standards are embodied in different cultures, industries, and personal values, each with its own supporters and detractors.

Within programming languages, there are two kinds of standards that organize code in different ways, linking data structures to functions. This article will briefly discuss the application and principle of action of these two common standards.

class model

In the Javascript language, the class keyword is provided, which we can use to define classes. Importantly, classes are extensible, and when methods are called on an instance, they look up method definitions in the order of the prototype chain. For example:

class Animal {
    move() { return 'move' }
}
class Monkey extends Animal {
    jump() { return 'jump' }
}
class Human extends Monkey {
    write() { return 'write' }
}

extends Keywords can help us define a new class that extends from other classes. We can get the prototype chain like this:

let me = new Human()
let proto = me.__proto__
let r = []

while (proto !== null) {
    r.push(proto.constructor.name)
    proto = proto.__proto__
}

We'll get r like this: ['Human', 'Monkey', 'Animal', 'Object']. When we call a method, the success of the method call depends on whether there is a corresponding definition on the prototype chain. For example, if you call me.move(), it has a definition in the Animal class, and the call succeeds.

This is the class model, which I like to call as a "relational" model, which can be called as long as it is defined in the class of the instance itself, or in the parent class, grandparent class, and so on.

Competency-based model

Compared to the relationship-based model, the competency-based model is much simpler. As long as an instance satisfies a certain trait, the corresponding method can act on the instance regardless of the relationship of the instance.

let me = {
    moveable: true,
    freezable: false
}

function move(x) { return x.moveable ? 'move' : undefined }
function freeze(x) { return x.freezable ? 'freeze' : undefined }

In this model, data and methods are independent of each other. The advantage is that it is concise and straightforward, but the disadvantage is obvious, because of the independence of the method and the data, it is difficult to override the method definition.

summary

After introducing the two ways of connecting data structures and methods, we can think that when it is necessary to implement methods with different functions of the same name according to the class of the data, it is recommended to use the relational class model. When the method is usually relatively fixed, and the data is changeable, and the same method works on different data, the Competency-based model can be used.

It’s not that hard! To understand `Promise` in javascript

HornGang — Tue, 24 Sep 2024 16:35:45 +0000

The content of this article is very suitable for Javascript beginners. It will be described in simple and easy-to-understand language, so you don’t have to worry about not understanding it.

`Promise`, why do you exist?

Promise is an indispensable knowledge point when learning modern Javascript language. Many people feel confused when reading it. The main reason can be summarized in one sentence:

The code is no longer executed from top to bottom.

Normally, the code we write is executed sequentially, for example, we write a code that counts from 1 to 3.

console.log(1)
console.log(2)
console.log(3)

Press F12 to open the console of our browser, copy the above code into it, press Enter, and you can see that the numbers 1 to 3 are printed out in order.

1
2
3

Now, if we have a requirement that the order of the codes cannot be changed, but the order of the final printouts must be independent of the order of the codes, can we do this?

Let's imagine that if 1, 2, and 3 are printed by three people respectively, then it will be very simple. We just need to tell them the task of printing numbers at the same time, and the order of the printouts will only be related to the execution time of the tasks.

Promise, the magical effect

So, how do we assign the printing task to three people? This is where is used Promise. By creating a new Promise object, we can assign a section of code to a new "process" instead of executing it on the current "process". Note that the "process" here is not the same as the operating system process we often talk about, but is just an abstract concept, representing a virtual unit that executes code in sequence.

Promise, Assembly Instructions

As we said just now, Promise it can be regarded as a new "process", so if we want it to execute any code, we can wrap it in a function and give it to it. In this way, the code will not be executed immediately.

Promise also provides us with two functions, one is resolve and the other is reject, which we can call when the task is completed and failed respectively. These two functions are passed to Promise's internal of as parameters. So, we can assemble one like this:

Promise(
  (resolve, _reject) => {
    setTimeout(() => {
      console.log(1)
      resolve()
    }, 300)
  }
)

Here we use setTimeout to simulate a time-consuming task. In real scenarios, this task may be reading a file, requesting a network interface, or waiting for user input. When the task is completed, we call resolvethe function to indicate that the task has been completed.

Complete code

Then, we follow the same method, and Promise when we assemble the other two, we only need to provide them with different print numbers and the time required for the task. Finally, the complete code is like this, you can try to copy it into the console and give it a try.

Promise(
  (resolve, _reject) => {
    setTimeout(() => {
      console.log(1)
      resolve()
    }, 300)
  }
)
new Promise(
  (resolve, _reject) => {
    setTimeout(() => {
      console.log(2)
      resolve()
    }, 200)
  }
)
new Promise(
  (resolve, _reject) => {
    setTimeout(() => {
      console.log(3)
      resolve()
    }, 100)
  }
)

Finally, the order in which the numbers are printed is related to the execution time of the tasks:

3
2
1

Promise, Concurrency Model

Concurrent programming is about how to simulate the execution of a large number of tasks simultaneously on a limited number of CPU cores. Promise in Javascript provides a convenient and fast way of concurrent programming, as well as a set of specifications for handling return values and error values. After understanding and becoming familiar with this specification, we can greatly improve the efficiency of handling concurrent tasks.

Deploy a BitcoinSV node with docker

HornGang — Fri, 08 May 2020 11:28:53 +0000

BitcoinSV is an original bitcoin protocol implementation. We can use it as a reliable money transfer process, or secure data storage.

For normal users, run a BitcoinSV node is not very convenient or cheap. It needs a large disk space to store the block data. But the node is the most secure way to communicate with blockchain for enterprise users nowadays. Because the node will check the Merkel tree of transactions and validate the transaction script. And detected the longest chain for you.

N-chain has distributed a docker image of the BitcoinSV node. So we can easily deploy it in our development or production environment. First, make sure you have installed docker. Then pull the image.

docker pull bitcoinsv/bitcoin-sv

This is the start command for bitcoins 1.0.3 .

docker run -d \
-p 8332:8332 \
-v $(pwd)/data:/data \
--name bitcoind bitcoinsv/bitcoin-sv bitcoind \
-server \
-rpcuser=user \
-rpcpassword=password \
-prune=25000 \
-excessiveblocksize=2000000000 \
-maxstackmemoryusageconsensus=200000000

You can set a more secure RPC username and password. The exact meaning of the last two settings can be viewed on bitcoinsv.io.

After running the command above, the BitcoinSV node will be running in the background. Run docker ps to check.

Now you can use the RPC interface via localhost:8332.

curl --user user --data-binary '{"jsonrpc": "1.0", "id": "curltest", "method": "getblockchaininfo", "params": []}' -H 'content-type: text/plain;' http://localhost:8332

To check the latest 100 lines of logs:

docker logs --tail 100 bitcoind

To stop the node:

docker stop bitcoind

To delete the container:

docker rm /bitcoind

Make learning JS basic methods as game tasks

HornGang — Sat, 27 Jul 2019 15:54:49 +0000

When I begin to learn JavaScript, I often feeling tired with searching document. I spent a lot of time in looking for a simple function in standard lib. Till one day, I can't stand wasting time in this anymore. Why not remember all of them in my head, I thought.

But remember the functions in std lib is a boring job. To add some fun in doing that, I worte a simple script, that will generate a todo list, with js methods name and the link to that method's MDN document.

const FundamentalObjects = [
    Object,
    Function,
    Boolean,
    Symbol,
    Error
]

const NumbersAndDates = [
    Number,
    BigInt,
    Math,
    Date
]

const TextProcessing = [
    String,
    RegExp
]

const IndexedCollections = [
    Array,
    Int8Array,
    Uint8Array,
    Uint8ClampedArray,
    Int16Array,
    Uint16Array,
    Int32Array,
    Uint32Array,
    Float32Array,
    Float64Array,
    BigInt64Array,
    BigUint64Array
]

const KeyedCollections = [
    Map,
    Set,
    WeakMap,
    WeakSet
]

const StructuredData = [
    ArrayBuffer,
    SharedArrayBuffer,
    Atomics,
    DataView,
    JSON
]

const ControlAbstractionObjects = [
    Promise,
    // Generator,
    // GeneratorFunction,
    // AsyncFunction 
]

const Reflection = [
    Reflect,
    Proxy
]

const Internationalization = [
    Intl,
    Intl.Collator,
    Intl.DateTimeFormat,
    Intl.ListFormat,
    Intl.NumberFormat,
    Intl.PluralRules,
    Intl.RelativeTimeFormat,
    Intl.Locale
]

const WASM = [
    WebAssembly,
    WebAssembly.Module,
    WebAssembly.Instance,
    WebAssembly.Memory,
    WebAssembly.Table,
    WebAssembly.CompileError,
    WebAssembly.LinkError,
    WebAssembly.RuntimeError
]


const ObjectsToLearn =
    FundamentalObjects
    .concat(NumbersAndDates)
    .concat(TextProcessing)
    .concat(IndexedCollections)
    .concat(KeyedCollections)
    .concat(StructuredData)
    .concat(ControlAbstractionObjects)
    .concat(Internationalization)
    .concat(WASM)

const listMethodsOf = (object) => {
    try {
        return Object.getOwnPropertyNames(object.prototype)
            .filter(f => !['caller', 'callee', 'arguments'].includes(f))
            .filter(f => {
                try {
                    let bool = typeof(object['prototype'][f]) == 'function'
                    return bool
                } catch (e) {
                    return false
                }
            })
    } catch (e) {
        return []
    }
}


let r =
    ObjectsToLearn
    .filter(o => o.name)
    .map(o => '# ' + o.name + '\n' +
        listMethodsOf(o)
        .map(f => '- [ ] [' + f + '](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/' + o.name + '/' + f + ')')
        .join('\n')
    )
    .join('\n')

console.log(r)

Then copy the content in console, and paste in your gist, you just got a clickable todo list of all methods of main objects in JavaScript (Browser). You can visit the result here: https://gist.github.com/Ljzn/d93dae3de64660e66be598ee88fa57be .

Happy Coding!

2019.08.06 Update:

I changed the code into two functions: "listOfJSObjectsDoc" and "listOfWebAPIDoc".

const listMethodsOf = (object) => {
    try {
        return Object.getOwnPropertyNames(object.prototype)
            .filter(f => !['caller', 'callee', 'arguments'].includes(f))
            .filter(f => {
                try {
                    let bool = typeof(object['prototype'][f]) == 'function'
                    return true
                } catch (e) {
                    return false
                }
            })
    } catch (e) {
        return []
    }
}


const listOfWebAPIDoc = modules =>
    toDoc(modules, 'https://developer.mozilla.org/en-US/docs/Web/API/')

const listOfJSObjectDoc = modules =>
    toDoc(modules, 'https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/')

const toDoc = (modules, baseURL) =>
    modules
    .filter(o => o.name)
    .map(o => '# ' + o.name + '\n' +
        listMethodsOf(o)
        .map(f => `- [ ] [${f}](${baseURL}${o.name}/${f})`)
        .join('\n')
    )
    .join('\n')

Example:

let modules = [MediaStream, MediaStreamTrack]

console.log(listOfWebAPIDoc(modules))

You can try this in https://jsbin.com/dujeril/1/edit?js,console .

Happy Coding!

DEV Community: HornGang

Elixir 编译器并发优化失败复盘

Elixir 编译器并发优化失败复盘

背景

分析：数据依赖

Types:infer 的输入输出

translate_typespecs_for_module 的输入输出

数据依赖图

实现：三次迭代

第一次尝试：spawn_monitor + receive

第二次尝试：spawn + make_ref + receive

第三次尝试：ETS 表通信

第四次尝试：非阻塞检查 + 回退

Benchmark 数据

失败原因分析

1. 收益窗口太小

2. 进程间通信开销

3. receive 在 elixir_erl_compiler:spawn 内部不可用

4. :ets.take 的破坏性读取

5. 架构约束

经验教训

后续可行的优化方向（不在此次范围内）

Relationship or ability, both of these criteria are used

class model

Competency-based model

summary

It’s not that hard! To understand `Promise` in javascript

Promise, why do you exist?

Promise, the magical effect

Promise, Assembly Instructions

Complete code

Promise, Concurrency Model

Deploy a BitcoinSV node with docker

Make learning JS basic methods as game tasks

`Types:infer` 的输入输出

`translate_typespecs_for_module` 的输入输出

3. `receive` 在 `elixir_erl_compiler:spawn` 内部不可用

4. `:ets.take` 的破坏性读取

`Promise`, why do you exist?