Every article in the Innovation Wormhole series stands on its own — no prior reading is required. For context: Part 1 introduced the Innovation Wormhole — the idea that AI should transfer proven mechanisms between industries instead of reinventing them — and Part 2 defined a machine-readable protocol for it (the Mechanism Transfer Protocol). This piece takes the hardest question left open there: how do you know, before you build, whether a transfer will actually work?*

Quantifying Transferability: How to Score Cross-Domain Innovation Before You Build It

Every AI system that proposes cross-domain innovations eventually faces the same uncomfortable question:

"Your system says this solution transfers. Says who?"

If the only answer is "the model said so" or "the expert liked it," the score is decoration, not measurement. An innovation pipeline that cannot quantify its own hit rate is not an innovation pipeline — it is a slot machine with good copywriting.

This article defines a measurement discipline for cross-domain transfer: how to build ground truth from historical transfers, which metrics actually matter, how to calibrate scoring weights against realized outcomes, and what an open benchmark for the whole field should look like.

Why Intuition Is Not a Score

Suppose a system scores two candidate transfers at 0.83 and 0.61. What does the gap mean?

Without calibration, nothing. An uncalibrated score cannot be compared across systems, cannot be tracked over time, and cannot be audited. Worse, expert intuition — the traditional alternative — has a documented weakness: the further an analogy is from the expert's home domain, the worse humans get at judging whether it will actually transfer. Design research calls the reverse effect design fixation: anchoring on familiar features of a known solution prevents designers from seeing how it fails in a new context (Jansson & Smith, "Design Fixation," Design Studies, 1991).

The problem is not that humans judge badly. The problem is that we currently have no shared yardstick against which human or machine judgments about transferability can be checked. That is what this article proposes to build.

Core Concepts (Self-Contained Recap)

Two record types from the previous article are enough context for everything below:

• Problem Card — a problem stripped of industry vocabulary: objectives, constraints, failure modes, success metrics, and the physical properties of the system.
• Mechanism Card — a minimal transferable unit of a solution: what it does (transformation), what it needs (assumptions), what proves it (evidence), and where it holds (validity envelope).

A scoring function such as the Transfer Feasibility Score — TFS = (w_s · S_struct + w_f · S_func + w_c · S_causal) − (R_ctx + R_econ) — maps a candidate pair to a number. The entire question of this article is: what justifies those weights?

Three new concepts are defined here:

Definition 1 — Realized Transfer Record (RTR). A machine-readable record of a transfer that was actually attempted, in history or in industry:

RTR = ⟨K, P_target, outcome, assumptions_survived, cost_overrun, time_to_value⟩

where outcome is one of {succeeded, partially_succeeded, failed} and assumptions_survived records which of the mechanism's original assumptions held in the target context.

Definition 2 — Assumption Survival Rate (ASR). The fraction of a mechanism's enabling assumptions that survived the transfer intact:

ASR = |A_survived| / |A_total|

ASR is measured per realized transfer, post hoc. A large collection of ASRs tells you which kinds of assumptions break most often — which is exactly the knowledge a scoring function needs.

Definition 3 — Calibration. A transferability score s is calibrated when, across all transfers assigned a score near s, the observed success frequency is also near s. A system that says "80% likely to work" should be right roughly 80% of the time, or its numbers are theater.

Building Ground Truth from History

You cannot calibrate a predictor without labeled outcomes. The good news: history has already run millions of cross-domain transfer experiments — we just never wrote them down as training data.

Four sources of Realized Transfer Records, in decreasing order of structure:

• Patent reclassification and reassignment. When a technology migrates between industries, it often migrates between patent classes. Cross-domain exploration of patents using functional extraction and similarity analysis is an active research technique (ScienceDirect: causal extraction for cross-domain patent exploration), and analogy-based patent mining has been used to map solutions from mature technologies onto emerging ones (Expert Systems with Applications).
• Documented technology-transfer cases. The microwave oven (radar magnetron → consumer appliance), memory foam (aerospace safety → consumer bedding), and negative-selection intrusion detection (immunology → cybersecurity) are all RTRs with public evidence trails.
• Failed transfers. Commercial post-mortems, abandoned pilots, and products killed by regulatory or economic reality. Iridium's satellite phone network is a canonical record of a technically brilliant transfer whose R_econ term was catastrophically underestimated.
• Collaboration records. Cross-industry programs such as Pumps & Pipes (petroleum engineering ↔ medicine) have documented how transfers actually get facilitated and which ones fail (Springer: Cross-Industry Innovation Does Not Happen by Chance).

An RTR looks like this:

{
  "rtr_id": "RTR-1997-0031",
  "mechanism": "negative selection of detectors against self samples",
  "source_domain": "immunology",
  "target_domain": "network intrusion detection",
  "outcome": "partially_succeeded",
  "assumptions_total": 5,
  "assumptions_survived": 3,
  "asr": 0.6,
  "cost_overrun": "2.1x",
  "time_to_value": "4 years",
  "evidence_refs": ["doi:…", "case study:…"],
  "failure_note": "detector scaling to high-dimensional traffic"
}

Metrics That Matter

Once you have RTRs, four metrics do most of the work:

Metric	Formula	What it tells you
Precision@k	(validated transfers in top-k) / k	Of the recommendations the system is most confident about, how many actually hold up?
Assumption Survival Rate	|A_survived| / |A_total|	Which assumption categories break most often across transfers?
Calibration error	mean(|s_i − y_i|) over realized transfers	Do the scores mean what they claim to mean?
ROI delta	realized ROI − predicted ROI	Is the economic-friction term honest?

Two properties of this metric set matter more than the formulas themselves:

• They decompose. A bad Precision@k with a good calibration error means the ranking is broken but the scores are honest — a retrieval problem, not a scoring problem. Diagnosis requires that separation.
• They are gameable, so the benchmark must hold out data. Any metric published on the same records used for calibration becomes a target, not a measurement (a lesson every ML benchmark community has learned the hard way).

Calibrating the Weights

Here is the structural insight that makes calibration tractable: the TFS is already a linear score. That means calibrating it is not a research moonshot — it is a logistic regression.

Fit, on Realized Transfer Records:

P(success) = sigmoid(w_s·S_struct + w_f·S_func + w_c·S_causal
                     - R_ctx - R_econ + b)

where y (success / partial / failure) comes from the RTR outcomes, and the features S_struct, S_func, S_causal are computed the same way the live system computes them.

Three rules keep the calibration honest:

• Walk-forward validation. Calibrate only on transfers realized before the evaluation period. History leaks badly if you fit on outcomes that overlap your test set.
• Report the weights. The fitted w_s, w_f, w_c are the scientific result. If structural similarity turns out to predict success far better than functional similarity — or the reverse — that is a genuine discovery about how cross-domain innovation works, and exactly the kind of finding the knowledge-graph-assisted design-by-analogy literature's "domain distance" measures point toward (Cambridge AI EDAM).
• Fit per transfer class, not globally. Assumption survival behaves differently in software-to-software transfers than in physical-to-digital ones. One global model hides that.

What Failure Teaches

Benchmark culture has a bias worth naming: nobody's dataset headline features the negatives. But for transferability, failures are the signal.

A system that only ever sees successful analogies will learn that everything transfers — the base-rate blindness that makes uncalibrated idea engines dangerous. Roughly speaking, the failure records are what teach the boundary conditions:

• Design fixation teaches that superficially attractive analogies anchor judgment (Jansson & Smith, 1991).
• Iridium teaches that R_econ is not a rounding error — it can invalidate an entire technically-sound transfer.
• Regulatory mismatches teach that evidence admissible in one regime is worthless in another.

The uncomfortable operational implication: collecting failed transfers deserves the same budget as collecting successes. An organization that logs only its wins has a marketing database, not a calibration set.

TransferBench: An Open Benchmark Proposal

The field needs what information retrieval got from TREC and ML got from ImageNet: a shared, adversarially maintained yardstick. Provisionally — TransferBench — with three tasks and one rule:

• Task 1 — Retrieval. Given a Problem Card, return the k most promising Mechanism Cards from a held-out index. Metric: Precision@k against expert-validated transfers.
• Task 2 — Scoring. Given a candidate pair, output a calibrated transferability score. Metric: calibration error on held-out RTRs.
• Task 3 — Assumption analysis. Given a candidate pair, output which assumptions survive. Metric: ASR prediction accuracy against post-hoc records.
• The one rule — temporal holdout. All evaluation records must be realized transfers that post-date any training data the system has seen. This is the only defense against benchmark contamination.

Current research systems already suggest the shape of entries: facet-recombination ideation tools such as Scideator, analogy retrieval from large repositories such as the U.S. patent database (Analogy Mining, arXiv:1706.05585), and multi-agent literature-mining pipelines that propose cross-domain hypotheses automatically (an overview of automated discovery through analogical reasoning). None of them currently shares a yardstick. A benchmark would let the field tell, for the first time, whether a system is actually better at cross-domain reasoning — or just better at writing plausible prose about it.

Authorization Boundaries: What a Score May Never Do

Measurement changes behavior, and not always for the better. Explicit boundaries:

• AI is autonomous in: computing scores, flagging assumption risks, drafting calibration reports.
• A human gate is mandatory before: acting on any score that conflicts with regulatory constraints; any transfer in a safety-critical domain; any decision where the benchmark was not representative of the target context.
• A score must never be used as: proof of viability, a compliance instrument, or a reason to skip a pilot. A calibrated score is a prior; the pilot is the posterior.

The rule carries over from the previous article unchanged: AI accelerates discovery; humans own applicability. Calibration makes AI's opinions honest — it does not make them decisions.

Technical Challenges and Open Problems

• Ground-truth scarcity. RTRs are expensive to annotate and the interesting ones (failures) are rarely published. Bootstrapping from patent reclassification helps, but survivorship bias is severe.
• Outcome ambiguity. "Partial success" is common in real transfers. Metrics must handle graded outcomes without pretending they are binary.
• Non-stationarity. Industries change; a calibration from 2015 does not price 2026 regulatory risk. Weights need scheduled re-fitting, and the re-fitting schedule itself needs to be published.
• Feature leakage. If the similarity features are computed by the same LLM that knows the outcome's history, "prediction" is quietly contaminated. Task definitions must pin feature computation to pre-outcome information.
• Benchmark gaming. Temporal holdout is necessary but not sufficient; adversarial review of submissions will be needed, as in every mature benchmark.

Why This Matters

The innovation literature has argued for decades that breakthroughs concentrate at intersections of fields — Frans Johansson's The Medici Effect (Wikipedia) being the best-known formulation. The AI systems are now arriving: idea engines that can genuinely propose cross-domain transfers.

What has not arrived is accountability. Without ground truth, calibration, and shared benchmarks, "the AI suggested it" will be indistinguishable from "the AI hallucinated it" — and the field of cross-domain reasoning will deserve the skepticism it gets.

Calibration is how that changes. A transferability score that has been fitted against realized outcomes, walks forward honestly, and publishes its weights, is no longer an opinion. It is an instrument.

The natural companion to this piece is the engineering one: how to actually build the pipeline — the knowledge graph, the indexing stack, and the human-in-the-loop tooling — that produces calibrated Transfer Reports in production. That is the next standalone article in the Innovation Wormhole series; follow the series if you want it in your feed.

Which historical transfer would you want in the first calibration set — a success like negative-selection intrusion detection, or a failure like Iridium? I'd love to hear your thoughts in the comments.

References

1. Scideator — Human-LLM Compound System for Scientific Ideation through Facet Recombination and Novelty Evaluation — arXiv:2409.14634
2. Accelerating Innovation Through Analogy Mining — arXiv:1706.05585
3. Discovering new applications: Cross-domain exploration of patent documents using causal extraction and similarity analysis — ScienceDirect
4. Creating patents on the new technology using analogy-based patent mining — Expert Systems with Applications
5. Knowledge graph-assisted design-by-analogy: promoting product innovation through structured analogical knowledge retrieval — Cambridge AI EDAM
6. Cross-Industry Innovation Does Not Happen by Chance: The Role of Facilitation — Springer Professional
7. Jansson, D. G., & Smith, W. K. (1991). Design Fixation. Design Studies, 12(1) — the foundational study on how prior solutions block creative transfer.
8. The Medici Effect — Frans Johansson — Wikipedia
9. Automated Discovery through Cross-Domain Analogical Reasoning — Lacuna
10. Part 1 of this series — Building an AI-Powered Innovation Wormhole

---

created by Seyed Alireza Alhosseini Almodarresieh
https://almodarresieh.github.io/