Artificial General Intelligence — Ch.07: Chapter 07. AGI Evaluation and Benchmarks [EN]

STATERA Library · Volume 47 · Chapter 07 · EN
Source: https://github.com/Statera-Library/01_Vol_47_Artificial_General_Intelligence/blob/main/Chapter_07_AGI_Evaluation_and_Benchmarks/Chapter_07_AGI_Evaluation_and_Benchmarks_EN.md

Volume 47. Artificial General Intelligence

Chapter 07. AGI Evaluation and Benchmarks

07.00. What is AGI Evaluation

Artificial General Intelligence evaluation is the systematic process of determining whether an artificial system possesses intelligence that extends beyond competence in isolated tasks. Unlike conventional model evaluation, which measures performance on predefined datasets or narrow objectives, AGI evaluation asks whether a system can understand unfamiliar situations, acquire new capabilities, transfer knowledge across domains, reason under uncertainty, and pursue goals in environments that were not explicitly represented during training.

The central difficulty is that general intelligence cannot be represented adequately by a single accuracy score. A system may outperform humans on mathematics, programming, language, or visual recognition while remaining fragile when tasks require unfamiliar combinations of those abilities. AGI evaluation therefore treats intelligence as a multidimensional capability involving perception, memory, reasoning, learning, planning, adaptation, communication, action, and metacognition rather than as mastery of one benchmark.

Generalization is especially important because an apparently intelligent system may simply exploit patterns contained in its training distribution. A stronger evaluation examines performance when tasks, rules, environments, representations, or objectives change. The question is not merely whether the system can solve previously observed problem classes, but whether it can identify the structure of a new problem, connect it with prior knowledge, construct an appropriate strategy, and adapt when its initial assumptions prove incorrect.

Transfer provides another essential dimension of AGI evaluation. Knowledge acquired in one domain should improve performance in related or structurally similar domains without requiring complete retraining. A generally intelligent agent might learn principles of spatial reasoning from navigation and later reuse them in manipulation or planning. Evaluation must therefore examine whether internal representations capture reusable concepts and relationships rather than superficial correlations specific to individual datasets or tasks.

Reasoning evaluation investigates how effectively an agent can transform available information into conclusions, explanations, predictions, and decisions. This includes logical, mathematical, probabilistic, causal, commonsense, and counterfactual reasoning. Difficult evaluations should require combinations of these forms rather than testing them independently, because real problems rarely announce which reasoning method is appropriate. An AGI candidate must often determine both how to reason and what information is relevant.

Learning ability must also be evaluated dynamically rather than only through the capabilities frozen into a trained model. General intelligence implies the capacity to acquire useful knowledge from demonstrations, instructions, interaction, feedback, observation, and experience. Evaluation should measure learning efficiency, the amount of experience required for adaptation, resistance to catastrophic forgetting, and the ability to integrate new knowledge without destroying previously acquired competencies.

Agent evaluation expands the problem from producing correct answers to achieving goals through sequences of decisions. An intelligent agent may need to interpret an objective, decompose it into subgoals, select tools, gather missing information, monitor progress, recover from errors, and revise plans. These abilities become increasingly important as evaluation moves from short question-answer benchmarks toward interactive environments where actions alter future states and mistakes can accumulate across long horizons.

Long-horizon evaluation reveals weaknesses that short tasks often conceal. A system may appear highly capable when each interaction is independent yet fail when success depends on maintaining objectives, memories, constraints, and causal dependencies over hundreds of actions. AGI evaluation therefore needs tasks in which the agent must preserve coherence over time, recognize when circumstances invalidate an earlier plan, allocate resources, and balance immediate rewards against delayed consequences.

Multimodal evaluation examines whether intelligence remains coherent across language, vision, audio, spatial information, sensor observations, and actions. General intelligence should not consist merely of independent modality-specific modules producing disconnected outputs. The system should integrate evidence across modalities, establish correspondences between different representations of the same world, resolve conflicting observations, and use one modality to compensate when information in another modality is incomplete or uncertain.

Robustness is equally important because intelligence demonstrated only under carefully controlled benchmark conditions may not survive real deployment. Evaluations should introduce ambiguity, noisy observations, incomplete instructions, misleading information, distribution shifts, unexpected events, and adversarial conditions. The objective is not to demand perfect performance, but to determine whether the system recognizes uncertainty, avoids unjustified confidence, detects failure, and degrades gracefully instead of producing uncontrolled behavior.

Metacognition introduces evaluation of what the system knows about its own cognitive state. A capable agent should estimate confidence, recognize knowledge gaps, identify conflicts in its reasoning, detect unsuccessful strategies, and decide when additional information or external assistance is required. Such self-monitoring becomes critical for autonomous systems because raw problem-solving ability without reliable awareness of limitations can produce confident but systematically dangerous decisions.

AGI evaluation must distinguish capability from reliability. Demonstrating that a system can occasionally solve an extremely difficult task does not establish that the capability can be invoked consistently when required. Repeated trials, variations in context, alternative formulations, perturbations, and controlled difficulty levels are therefore necessary. Evaluation should characterize not only peak performance but also variance, failure modes, calibration, recovery behavior, and the conditions under which particular abilities emerge or disappear.

Human comparison can provide useful reference points, but human-level performance should not automatically define AGI. Humans themselves vary greatly across knowledge, memory, reasoning, perception, and expertise, while artificial systems may exhibit fundamentally different capability profiles. Evaluation should consequently use human baselines as informative anchors while separately measuring breadth, adaptability, efficiency, autonomy, robustness, and transfer instead of reducing general intelligence to imitation of an average human score.

Benchmark contamination presents a major methodological challenge. When evaluation problems or closely related examples appear in training data, high performance may reflect memorization rather than general intelligence. Strong AGI evaluation therefore benefits from newly generated tasks, procedurally constructed environments, hidden test distributions, dynamically changing rules, and evaluation sets created after model training. The more novel the evaluation conditions are, the more meaningful successful adaptation becomes.

Safety and alignment are inseparable from evaluation once systems become autonomous and broadly capable. An agent that efficiently accomplishes objectives while violating constraints cannot be considered successfully evaluated merely because task completion is high. Evaluation must examine instruction following, constraint preservation, resistance to harmful manipulation, uncertainty handling, corrigibility, oversight compatibility, and behavior when goals conflict or when the requested action exceeds authorized boundaries.

No single benchmark can establish the existence of AGI. A credible evaluation framework therefore combines generalization tests, reasoning benchmarks, agent environments, multimodal tasks, long-horizon challenges, learning experiments, robustness tests, and safety assessments. These components correspond to the broader evaluation structure in which generalization, reasoning, agency, multimodal intelligence, temporal persistence, and integrated evaluation are treated as complementary views of the same underlying question.

Ultimately, AGI evaluation should measure the structure and adaptability of intelligence rather than the accumulation of benchmark victories. The strongest evidence for general intelligence would be sustained competence across unfamiliar domains, efficient acquisition of new skills, coherent reasoning and planning, effective interaction with changing environments, reliable self-correction, and safe behavior under uncertainty. Evaluation thus becomes an ongoing scientific process for mapping what an intelligent system can understand, learn, transfer, accomplish, and reliably control.

07.01. Generalization Tests

Generalization tests examine whether an artificial intelligence system can apply knowledge and capabilities beyond the exact conditions encountered during training. For AGI, this ability is fundamental because intelligence must remain useful when tasks, environments, representations, goals, or rules change. A system that succeeds only on familiar examples may demonstrate strong pattern recognition without demonstrating genuinely general intelligence.

The simplest form of generalization evaluates performance on unseen examples drawn from approximately the same distribution as the training data. This tests whether the system has learned reusable patterns rather than memorized individual cases. Although important, such in-distribution evaluation is insufficient for AGI because real environments frequently produce situations whose combinations, contexts, and constraints differ substantially from those represented during training.

Out-of-distribution generalization increases the difficulty by changing important properties of the evaluation environment. Inputs may contain unfamiliar objects, different linguistic styles, altered visual conditions, new task structures, or unexpected combinations of known concepts. Successful performance indicates that the system has learned abstractions that remain useful when surface-level statistical regularities change, rather than depending entirely on familiar correlations.

Compositional generalization tests whether previously learned concepts can be recombined in novel ways. A system may understand individual ideas such as objects, actions, spatial relationships, numerical operations, or logical rules, yet fail when they appear in combinations that were absent during training. AGI requires the ability to construct new solutions from reusable components because the number of possible real-world combinations is effectively unlimited.

Systematic generalization evaluates whether learned rules are applied consistently across equivalent situations. If a system learns a relationship in one context, structurally similar relationships should often be handled using the same underlying principle. This form of testing is particularly valuable for distinguishing genuine rule-like understanding from pattern matching, since superficial similarity may be deliberately reduced while deeper structural relationships are preserved.

Cross-domain transfer extends generalization beyond variations of a single task. Knowledge obtained in language, vision, mathematics, navigation, or interaction may contain principles that are useful elsewhere. Tests can therefore examine whether an agent recognizes shared structures between domains and reuses them effectively. Strong transfer suggests that internal representations encode concepts at a sufficiently abstract level to support broader intelligence.

Few-shot and zero-shot evaluation measure how efficiently a system can operate when direct training examples are scarce or absent. Zero-shot tasks require the system to interpret instructions and apply existing knowledge without task-specific examples, while few-shot tasks provide only limited demonstrations. These settings are important for AGI because a generally intelligent system should not require extensive retraining whenever it encounters a new objective or unfamiliar environment.

Task adaptation tests add an interactive dimension by allowing the system to learn during evaluation. Instead of measuring only fixed pretrained capability, the evaluator observes how quickly the system identifies new rules, updates strategies, and improves through feedback. Adaptation speed, sample efficiency, stability, and retention become important measures because intelligence involves not only what has already been learned but also how effectively future learning occurs.

Distribution shifts can be introduced gradually or abruptly. Gradual shifts test whether a system notices changing conditions and continuously adjusts its internal models, while abrupt shifts examine recovery from unexpected transitions. Examples include altered sensor characteristics, new environmental dynamics, changing user behavior, or modified reward structures. Robust AGI should recognize that previous assumptions are becoming unreliable and revise them accordingly.

Generalization should also be tested under changes in representation. The same underlying problem may be expressed using different words, symbols, visual layouts, coordinate systems, units, or modalities. A system that truly understands the problem structure should retain much of its competence despite these transformations. Large performance losses after superficial reformulation can reveal dependence on presentation-specific cues rather than stable conceptual understanding.

Counterfactual and causal generalization provide stronger tests because they require reasoning about how outcomes would change under different interventions. Statistical similarity alone may not be sufficient when the environment behaves differently from familiar examples. An intelligent system should distinguish correlation from causation, predict consequences of interventions, and adapt its reasoning when mechanisms change while irrelevant background features remain constant.

Generalization tests must carefully control benchmark contamination. If evaluation tasks, solutions, or close variants appeared during training, apparently impressive zero-shot performance may partly reflect memorization. Hidden evaluation sets, procedurally generated problems, newly created environments, randomized rules, and post-training task construction can reduce this risk. The objective is to maximize the probability that success reflects transferable capability rather than exposure.

Difficulty should be varied systematically rather than represented by a single pass-or-fail threshold. Evaluators can increase novelty, number of interacting variables, required reasoning depth, horizon length, uncertainty, or environmental change. This produces a capability profile showing where performance begins to degrade. Such profiles are more informative than aggregate scores because two systems with similar averages may exhibit very different generalization boundaries.

Failure analysis is therefore an essential component of generalization testing. Evaluators should determine whether errors arise from missing knowledge, weak reasoning, poor abstraction, memory limitations, incorrect uncertainty estimates, brittle planning, or inability to adapt. Repeated tests with controlled perturbations help reveal whether a failure is isolated or systematic and whether the system can recognize the failure and recover through additional reasoning or learning.

Ultimately, AGI generalization tests should measure whether intelligence remains functional as familiar assumptions are progressively removed. Strong evidence would include competence on novel tasks, reusable abstractions, compositional reasoning, cross-domain transfer, efficient adaptation, robustness to representation changes, and recovery from distribution shifts. Generalization is therefore not a single benchmark property but a central test of whether learned intelligence can extend beyond its training history.

07.02. Reasoning Benchmarks

Reasoning benchmarks evaluate whether an artificial intelligence system can transform information into justified conclusions rather than merely retrieve familiar answers or reproduce statistical patterns. Within AGI evaluation, reasoning is especially important because unfamiliar problems often require the system to identify relationships, construct intermediate conclusions, compare alternatives, and determine an appropriate solution even when no directly matching example exists in its training experience.

A comprehensive reasoning benchmark should measure several forms of reasoning rather than treating intelligence as a single homogeneous capability. Logical, mathematical, probabilistic, causal, spatial, temporal, analogical, commonsense, and counterfactual reasoning involve different structures and assumptions. An AGI candidate should not only perform well within these categories independently but also recognize which reasoning mode, or combination of modes, is appropriate for a particular problem.

Logical reasoning tests whether a system can manipulate propositions, rules, constraints, and relationships consistently. Tasks may require deduction from known premises, induction from observations, or abduction toward the most plausible explanation. Strong performance requires more than producing the correct final answer: the system should preserve relevant constraints, avoid contradictions, distinguish valid implications from unsupported conclusions, and remain consistent when equivalent problems are expressed differently.

Mathematical reasoning benchmarks examine the ability to understand quantities, symbols, equations, structures, and transformations while maintaining correctness across multiple steps. Difficult tasks may combine arithmetic, algebra, geometry, probability, optimization, or abstract mathematical concepts. For AGI evaluation, the important property is not memorization of known solutions but the capacity to formulate unfamiliar problems, select suitable operations, track intermediate states, and verify whether results satisfy the original conditions.

Probabilistic reasoning evaluates decisions and conclusions when information is incomplete or uncertain. An intelligent system should distinguish possibility from probability, update beliefs when new evidence becomes available, and reason about conditional dependencies. These benchmarks are valuable because real-world environments rarely provide complete information, making calibrated uncertainty and rational belief revision essential components of reliable reasoning rather than optional extensions to deterministic problem solving.

Causal reasoning goes beyond statistical association by asking whether a system understands how changes in one variable influence another through underlying mechanisms. Benchmarks may distinguish observation from intervention or require predictions under altered conditions. Such tasks reveal whether a model can reason about why events occur, anticipate consequences of actions, and revise predictions when causal mechanisms change even though superficial correlations remain similar.

Counterfactual reasoning extends causal understanding by asking what would have happened if some event, condition, or action had been different. The system must preserve relevant aspects of the known situation while modifying specific assumptions and propagating their consequences. This capability is important for planning, diagnosis, scientific reasoning, decision support, and learning from mistakes because intelligent agents frequently need to compare actual outcomes with plausible alternatives.

Commonsense reasoning evaluates knowledge and inference about ordinary physical, social, and practical situations. Many apparently simple problems require assumptions that are rarely stated explicitly, such as object permanence, typical human intentions, everyday causality, or basic physical constraints. AGI reasoning benchmarks should therefore examine whether a system can infer unstated but relevant information while avoiding unsupported assumptions that merely resemble frequent patterns in its training data.

Analogical reasoning measures whether a system can identify structural similarities between situations whose surface features differ. Instead of matching words or appearances, the system must recognize corresponding relationships and transfer a useful solution principle from one problem to another. This capability connects reasoning benchmarks directly with generalization because abstract relational knowledge enables previously acquired reasoning strategies to become useful in unfamiliar domains and contexts.

Spatial and temporal reasoning evaluate relationships involving location, orientation, movement, sequence, duration, and change. A system may need to infer relative positions, predict how configurations evolve, reconstruct event order, or reason about dependencies across time. These abilities become particularly important for embodied AGI, robotics, autonomous agents, and world models where reasoning must remain grounded in dynamically changing states rather than isolated textual descriptions.

Multi-step reasoning benchmarks test whether a system can maintain coherent intermediate states across extended inference chains. Errors can accumulate when an early assumption is incorrect, relevant information is forgotten, or intermediate conclusions become inconsistent. Evaluation should therefore vary reasoning depth and dependency structure systematically, revealing how performance changes as problems require longer chains, branching alternatives, nested constraints, or interactions among several reasoning processes.

Reasoning under incomplete information is another important test of general intelligence. Some problems cannot be solved reliably from the evidence initially provided, and the correct behavior may be to acknowledge uncertainty, request additional information, search for evidence, or postpone commitment. Benchmarks that always guarantee a definite answer can unintentionally reward overconfidence, whereas realistic evaluation should distinguish justified conclusions from guesses presented with unjustified certainty.

Interactive reasoning expands evaluation beyond static question answering. An agent may need to perform experiments, query tools, inspect external information, test hypotheses, or choose actions that reveal previously unavailable evidence. Reasoning then becomes a closed loop involving hypothesis formation, information acquisition, inference, action, observation, and revision. This structure more closely reflects the reasoning requirements of autonomous AGI operating in complex environments.

Benchmark design must also defend against memorization and contamination. Widely published reasoning problems may appear directly or indirectly in training corpora, making high scores difficult to interpret. Novel problem generation, hidden test sets, randomized variables, procedurally constructed tasks, altered surface representations, and post-training evaluation can reduce this risk. Multiple equivalent formulations can additionally reveal whether performance depends on understanding or accidental prompt-specific cues.

Evaluation should measure reasoning reliability as well as maximum capability. A system that solves a difficult problem once but fails under minor paraphrases, reordered information, irrelevant distractors, or repeated trials has demonstrated fragile reasoning. Useful metrics therefore include consistency, calibration, sensitivity to perturbation, error propagation, recovery from incorrect intermediate steps, and the ability to detect contradictions or recognize when a proposed solution requires reconsideration.

Ultimately, AGI reasoning benchmarks should determine whether a system can construct, evaluate, and revise chains of thought across diverse and unfamiliar problems. Strong evidence includes logical consistency, mathematical competence, uncertainty-aware inference, causal and counterfactual understanding, analogical transfer, commonsense judgment, long-chain stability, and adaptive information seeking. Reasoning evaluation therefore complements generalization tests by examining not merely whether intelligence transfers, but whether the transferred knowledge can be transformed into coherent decisions and solutions.

07.03. Agent Benchmarks

Agent benchmarks evaluate whether an artificial intelligence system can operate as an autonomous goal-directed agent rather than simply answer isolated questions. Within AGI evaluation, this distinction is fundamental because real intelligence must connect perception, reasoning, memory, planning, tool use, and action across sequences of interaction. The central question is whether the system can transform an objective into effective behavior in a changing environment.

An agent benchmark typically places the system inside an environment where actions have consequences and future observations depend on earlier decisions. Unlike static reasoning tests, the agent cannot treat every step independently. It must maintain a representation of the current state, remember relevant history, anticipate possible outcomes, and continuously update its strategy as new information arrives. Evaluation therefore becomes a test of closed-loop intelligence rather than one-shot prediction.

Goal understanding is one of the first capabilities that agent benchmarks must measure. Real instructions may be incomplete, ambiguous, hierarchical, or expressed in terms of desired outcomes rather than explicit procedures. A capable agent should identify the underlying objective, infer reasonable constraints, distinguish required results from optional preferences, and recognize when clarification is necessary. Misinterpreting the goal can invalidate even technically competent planning and execution.

Task decomposition evaluates whether an agent can transform a complex objective into manageable subgoals. Many useful tasks cannot be solved through a single action and instead require dependencies among information gathering, preparation, execution, verification, and correction. Strong agents should identify these dependencies, order them appropriately, revise the decomposition when assumptions change, and avoid unnecessary actions that consume time or resources without contributing meaningfully to the objective.

Planning benchmarks examine how agents select sequences of actions while considering constraints, uncertainty, cost, and future consequences. Short plans may reveal basic competence, but AGI-oriented evaluation should include situations requiring branching strategies, contingency planning, resource allocation, and delayed outcomes. The agent must often choose between acting immediately, gathering more information, or preserving options until uncertainty is reduced.

Tool-use evaluation tests whether an agent can extend its internal capabilities through external resources. Tools may include search systems, databases, calculators, software environments, APIs, simulators, sensors, or robotic actuators. Effective tool use requires recognizing when a tool is useful, selecting the appropriate one, constructing valid inputs, interpreting returned information, and incorporating the result into subsequent reasoning rather than treating tool invocation as an isolated action.

Information-seeking behavior is particularly important when the environment does not initially provide everything needed for success. A competent agent should recognize missing knowledge and actively acquire relevant evidence instead of guessing. Benchmarks can evaluate whether the agent asks useful questions, searches strategically, performs diagnostic actions, or conducts experiments that reduce uncertainty. Efficient information acquisition is often as important as reasoning over the information already available.

Memory becomes essential as agent tasks extend across longer interaction sequences. The agent may need to preserve goals, intermediate results, user preferences, environmental changes, failed approaches, and unresolved constraints. Agent benchmarks should examine whether relevant information remains accessible when needed without allowing obsolete or irrelevant details to dominate decisions. Memory evaluation therefore includes retention, retrieval, updating, prioritization, and controlled forgetting.

Monitoring and self-correction distinguish robust agents from systems that merely execute an initial plan. During execution, an intelligent agent should compare observed progress with expected progress, detect deviations, identify likely causes, and determine whether local correction or complete replanning is required. Evaluation should therefore include unexpected failures, invalid assumptions, unavailable tools, changed environments, and misleading observations that force the agent to reconsider earlier decisions.

Long-horizon agent benchmarks test whether coherent behavior can be sustained across many steps. Small errors in memory, planning, reasoning, or tool use can accumulate until the original objective is lost. The system must preserve task identity while allowing individual plans to change. Performance should therefore be measured not only by final task completion but also by the efficiency, consistency, recoverability, and stability of the decision process over extended horizons.

Environmental interaction adds another layer of difficulty because intelligent action changes the state being reasoned about. In robotics, games, simulations, software systems, or digital workspaces, an action can create opportunities while simultaneously removing others. An AGI agent must therefore reason about state transitions and understand that actions are not merely outputs but interventions that reshape subsequent possibilities, risks, observations, and planning requirements.

Partial observability makes agent evaluation more realistic. An agent often sees only a limited portion of the environment and must infer hidden state from observations and history. It may need to maintain hypotheses about unseen conditions, update them when evidence changes, and select actions that are useful both for achieving goals and revealing information. Benchmarks that expose every relevant state variable can substantially underestimate the difficulty of real autonomous operation.

Multi-agent environments test whether an agent can reason about other intelligent actors rather than treating the world as passive. Other agents may cooperate, compete, communicate, negotiate, share resources, or pursue conflicting objectives. Evaluation can examine coordination, role assignment, communication efficiency, prediction of other agents, conflict resolution, and adaptation to changing group behavior. These abilities become increasingly important as AGI systems participate in human and machine organizations.

Reliability must be distinguished from occasional task success. An agent that completes a difficult objective once but fails frequently under small variations cannot be considered robustly autonomous. Repeated trials should vary initial conditions, instruction wording, environment layouts, tool availability, disturbances, and task order. Useful measurements include success rate, completion time, resource consumption, unnecessary actions, recovery rate, consistency, and the frequency of unrecoverable failures.

Safety and constraint preservation are inseparable from agent benchmarking because autonomous systems can produce consequences through action. Completing an objective is not sufficient if the agent violates explicit restrictions, ignores authorization boundaries, damages resources, or pursues unsafe shortcuts. Benchmarks should therefore incorporate operational constraints and evaluate whether the agent maintains them throughout planning, tool use, execution, replanning, and recovery from unexpected situations.

Benchmark contamination is also a concern because fixed agent tasks may eventually become familiar through training data or repeated optimization. Procedurally generated environments, hidden objectives, randomized layouts, changing tool interfaces, novel combinations of subtasks, and post-training scenarios can provide stronger evidence of genuine agent capability. Evaluators should measure whether learned strategies transfer to new situations rather than whether specific benchmark trajectories have been memorized.

Ultimately, AGI agent benchmarks should measure an integrated cognitive loop in which the system understands goals, observes the environment, recalls relevant knowledge, reasons about alternatives, plans actions, uses tools, monitors results, learns from feedback, and adapts until the objective is completed or responsibly abandoned. This role within the broader AGI evaluation structure complements generalization and reasoning benchmarks by testing intelligence through sustained autonomous behavior.

07.04. Multimodal Evaluation

Multimodal evaluation examines whether an artificial intelligence system can understand, integrate, and reason across multiple forms of information rather than operating effectively within only one modality. For AGI, intelligence may need to combine language, images, video, audio, spatial information, sensor signals, and actions. The central question is whether these inputs contribute to one coherent representation of a situation rather than remaining isolated capabilities.

A basic multimodal benchmark measures competence within each individual modality before evaluating their integration. A system may demonstrate strong language understanding but weak visual perception, or accurate image recognition but poor temporal interpretation of video. Such asymmetries matter because high performance in one channel can conceal weaknesses elsewhere. AGI evaluation therefore requires a capability profile that reveals both modality-specific strengths and cross-modal competence.

Cross-modal grounding evaluates whether concepts represented in one modality can be connected correctly with corresponding information in another. Language may refer to objects in an image, sounds may indicate events outside the visual field, and sensor measurements may provide physical evidence about a scene. A generally intelligent system should establish these correspondences and understand that different modalities can describe complementary aspects of the same underlying world state.

Multimodal fusion tests whether information from several sources can be combined into a more useful representation than any individual source provides alone. Effective fusion requires determining which observations are relevant, how they correspond in space and time, and how much confidence should be assigned to each source. The system should exploit complementary information while avoiding the assumption that every modality is equally reliable under every condition.

Visual-language reasoning is a common component of multimodal evaluation because it requires perception and symbolic interpretation to operate together. Tasks may involve describing scenes, answering questions about images, following visually grounded instructions, comparing objects, reading diagrams, or inferring relationships that are not explicitly stated in text. Strong performance requires the system to reason about visual evidence rather than merely associate image patterns with familiar linguistic responses.

Audio-language evaluation extends this integration to speech, environmental sounds, acoustic events, and temporal signals. A capable system may need to distinguish what was said from how it was said, associate sounds with visible events, or recognize important events that are not visually observable. These benchmarks test whether auditory information contributes meaningfully to reasoning instead of functioning as an independent transcription or classification channel.

Video evaluation introduces temporal continuity because understanding a sequence requires more than recognizing individual frames. The system must track objects and agents, identify events, infer motion and state transitions, preserve identity over time, and determine causal or temporal relationships. For AGI, video provides an important bridge between static perception and world modeling because it exposes how environments evolve as actions and external events occur.

Spatial multimodal evaluation examines whether information from images, depth, maps, coordinates, language, and sensors can be organized into a coherent representation of space. This is particularly important for embodied intelligence, where an agent must understand where objects are, how they relate geometrically, which regions are reachable, and how movement changes perspective. Evaluation should test both spatial understanding and the ability to use that understanding for planning and action.

Temporal alignment is equally important because different modalities may arrive at different frequencies, delays, or levels of precision. A spoken instruction, camera frame, LiDAR scan, proprioceptive measurement, and action command may refer to related events while occurring at different times. Multimodal evaluation should therefore test whether a system can align asynchronous observations, preserve temporal order, and avoid combining information that belongs to different states of the environment.

Cross-modal inference tests whether one modality can supply information that is missing or ambiguous in another. An obscured object might be inferred from sound, language may clarify the intended meaning of a gesture, or depth information may resolve ambiguity in a two-dimensional image. Strong multimodal intelligence should use complementary evidence flexibly rather than fail whenever the preferred information channel becomes incomplete, noisy, or unavailable.

Conflicting evidence provides an even stronger evaluation condition. Different modalities may disagree because of sensor noise, occlusion, misleading language, measurement errors, or environmental change. The system should not combine these signals mechanically. It should assess reliability, identify contradictions, estimate uncertainty, and determine whether additional observation is required. The ability to manage disagreement is essential for robust intelligence in uncontrolled environments.

Missing-modality tests evaluate graceful degradation when one or more information channels disappear. A system trained with rich visual, linguistic, and sensor inputs may become unexpectedly fragile if one source is unavailable. AGI-oriented benchmarks should remove or corrupt modalities systematically and observe whether the system can reorganize its reasoning around remaining evidence. Robust performance indicates that multimodal intelligence is integrated without becoming unnecessarily dependent on a single channel.

Interactive multimodal evaluation connects perception directly with action. An agent may receive visual observations, spoken instructions, sensor measurements, and textual objectives while controlling software or physical actuators. Actions then alter the environment and generate new observations. This creates a closed perception-reasoning-action loop in which multimodal understanding must continuously support planning, execution, monitoring, correction, and learning rather than produce a single static answer.

Embodied evaluation makes this requirement especially concrete. A robot or simulated agent may need to identify an object visually, understand a linguistic command, estimate its spatial relationship, navigate toward it, manipulate it, and verify the result through multiple sensors. Success depends on maintaining a shared representation across modalities and time. Such tasks reveal whether multimodal intelligence can support purposeful interaction with the physical or simulated world.

Benchmark design should prevent systems from succeeding through shortcuts in a dominant modality. If a question can be answered from text alone, the benchmark may not actually measure visual reasoning even when an image is present. Evaluators can use counterbalanced examples, modality ablation, contradictory inputs, hidden information, novel combinations, and controlled perturbations to determine whether each modality genuinely contributes to the final decision.

Evaluation metrics should therefore extend beyond final answer accuracy. Useful measures include cross-modal consistency, grounding accuracy, temporal alignment, robustness to missing inputs, uncertainty calibration, conflict resolution, adaptation to sensor degradation, and task success during interaction. Failure analysis should identify whether errors originate in perception, alignment, fusion, memory, reasoning, grounding, or action so that apparently similar failures can be distinguished mechanistically.

Ultimately, AGI multimodal evaluation should determine whether diverse information channels are integrated into a coherent and actionable model of the world. Strong evidence includes reliable cross-modal grounding, complementary fusion, spatial and temporal coherence, inference under missing information, resolution of conflicting evidence, and successful perception-action loops. Multimodal evaluation therefore tests whether general intelligence can remain unified when reality is represented through many different forms of observation.

07.05. Long Horizon Tasks

Long-horizon tasks evaluate whether an artificial intelligence system can pursue objectives that require many dependent decisions over extended periods rather than solving isolated problems in a single interaction. Within AGI evaluation, this capability is especially important because real-world goals often unfold through sequences of perception, reasoning, planning, action, monitoring, and correction. The benchmark structure places long-horizon tasks within AGI evaluation alongside generalization, reasoning, agent, and multimodal evaluation.

A long-horizon task differs from a conventional benchmark because success cannot usually be achieved through one correct prediction or response. The system must transform a high-level objective into intermediate goals, determine dependencies among them, select appropriate actions, and preserve progress across multiple stages. Errors made early in the sequence may propagate forward, making the ability to maintain coherent behavior over time as important as competence at any individual step.

Goal decomposition is therefore a fundamental capability tested by long-horizon evaluation. An AGI system should convert broad objectives into manageable subgoals while preserving their relationship to the original intent. Decomposition should not merely produce a plausible checklist; it should capture prerequisite relationships, resource constraints, ordering requirements, and conditions for completion. Effective decomposition reduces complex objectives into executable structures without losing strategic coherence.

Planning quality must also be evaluated at multiple temporal scales. Short-term planning determines the next useful action, while medium-term planning coordinates groups of actions and long-term planning maintains direction toward the final objective. A capable system should move flexibly among these levels rather than generating one rigid plan at the beginning. This requirement connects naturally with the broader AGI architecture in which planning, memory, reasoning, perception, and action are distinct but integrated mechanisms.

Memory becomes critical as the task horizon expands. The agent must remember completed actions, unresolved problems, intermediate discoveries, previous failures, environmental changes, and commitments that may become relevant much later. Working memory alone is insufficient when tasks extend across hundreds or thousands of interactions. Long-horizon evaluation should therefore examine whether information is retained selectively, retrieved when useful, updated when circumstances change, and prevented from accumulating into misleading context.

State tracking complements memory by requiring the system to maintain an accurate representation of current progress. The agent should distinguish between what was planned, what was attempted, what actually occurred, and what remains unfinished. This distinction becomes especially important when actions fail silently or produce unexpected consequences. A system that continues reasoning from an assumed state rather than the observed state can gradually diverge from reality even when its individual decisions appear reasonable.

Long-horizon competence also requires continual monitoring of progress toward goals. Instead of executing a plan blindly, the agent should compare observed outcomes with expected outcomes and determine whether the current strategy remains appropriate. Progress monitoring allows deviations to be detected before they become catastrophic. Useful benchmarks can introduce delayed consequences, incomplete feedback, misleading intermediate success, or changing conditions to test whether the system genuinely monitors execution.

Replanning measures the ability to modify behavior when the original plan becomes invalid. Resources may disappear, tools may fail, environments may change, or newly discovered information may reveal that an earlier assumption was incorrect. A generally intelligent agent should preserve useful parts of an existing plan while revising the portions affected by new evidence. Restarting the entire task unnecessarily can be almost as problematic as refusing to change the plan at all.

Error recovery is particularly revealing because long tasks inevitably create opportunities for failure. Evaluation should distinguish systems that merely avoid common benchmark errors from systems capable of diagnosing and repairing unexpected failures. Recovery may require identifying the failed step, estimating its consequences, restoring a valid state, selecting an alternative method, and continuing without losing unrelated progress. This transforms robustness from passive resistance into an active cognitive capability.

Credit assignment becomes difficult when consequences occur far after the decisions that produced them. A poor outcome near the end of a task may originate from an assumption or action taken many stages earlier. Long-horizon evaluation should examine whether the system can trace outcomes backward through its decision history and identify influential causes. Without this ability, learning from extended experience becomes inefficient because the agent cannot determine which decisions should be reinforced or changed.

Uncertainty should accumulate explicitly rather than disappear as plans become longer. Every perception, inference, retrieved memory, external tool result, and predicted consequence may introduce uncertainty. If an agent repeatedly treats uncertain intermediate conclusions as established facts, confidence can become detached from reality. Long-horizon benchmarks should therefore test uncertainty tracking, verification behavior, information gathering, and the ability to reconsider assumptions when accumulated uncertainty becomes operationally significant.

Tool use creates another important dimension of long-horizon evaluation. Complex objectives may require search systems, databases, software applications, calculators, external services, robotic actuators, or specialized models. The agent must determine not only how to use individual tools but when they are necessary, what information should be passed between them, and how their outputs affect subsequent decisions. This reflects the volume\'s broader treatment of tool use and planning as an engineering component of AGI.

Long-horizon tasks can also test resource management. An intelligent system may operate under limited time, computational capacity, energy, memory, communication bandwidth, or financial cost. It should allocate these resources according to task importance and uncertainty instead of spending maximum effort on every decision. Efficient intelligence requires recognizing which situations justify deeper reasoning, additional observation, external tools, or contingency planning and which can be handled through inexpensive routine behavior.

Environmental persistence further separates realistic long-horizon evaluation from static question answering. Actions can modify the world, and those modifications may constrain later choices. Moving an object, changing a file, consuming a resource, communicating information, or navigating to another location creates consequences that persist. Evaluation should test whether the agent reasons about these state transitions and understands that later decisions occur in a world partly produced by its own earlier actions.

Multi-agent and human interaction can extend the horizon even further. Goals may depend on commitments, delegated tasks, negotiations, shared resources, or information provided by other agents over time. The evaluated system should remember responsibilities, recognize dependencies, update plans when collaborators behave unexpectedly, and avoid contradictory actions. Long-horizon intelligence therefore involves maintaining social and operational continuity as well as maintaining an internal sequence of reasoning.

Embodied long-horizon tasks provide particularly demanding tests because planning must remain grounded in continuous physical interaction. A robot might navigate through several locations, locate objects, manipulate equipment, recharge, respond to obstacles, coordinate with people, and eventually complete a mission lasting hours or days. Each action changes subsequent possibilities. Success requires integrating perception, memory, planning, control, world modeling, and recovery within one persistent perception-action loop.

Benchmark design should measure more than whether the final goal was eventually achieved. Useful metrics include completion rate, number of successful intermediate goals, unnecessary actions, recovery efficiency, resource consumption, memory consistency, planning stability, adaptation speed, safety violations, and accumulated error. Evaluators should also examine the trajectory itself because two agents reaching the same endpoint may differ greatly in efficiency, reliability, autonomy, and robustness.

Difficulty can be increased systematically by extending task length, delaying feedback, introducing hidden dependencies, changing environmental conditions, restricting resources, corrupting observations, requiring tool transitions, or injecting recoverable failures. The most informative benchmarks should prevent solutions based on memorized action sequences. Novel task combinations and dynamic environments reveal whether an agent can construct and maintain plans rather than reproduce previously learned trajectories.

Ultimately, long-horizon evaluation asks whether intelligence remains coherent as decisions accumulate over time. Strong AGI performance requires persistent goals, hierarchical planning, reliable memory, accurate state tracking, uncertainty management, progress monitoring, adaptive replanning, error recovery, and learning from delayed consequences. A system that performs impressively for a few steps but gradually loses its objective or diverges from reality remains fundamentally limited; general intelligence must sustain purposeful behavior across extended interaction.

07.06. Evaluation Framework [w/Code]

An AGI evaluation framework provides a systematic structure for determining whether an artificial intelligence system demonstrates broad, transferable, and reliable intelligence rather than isolated competence on individual benchmarks. Because general intelligence involves perception, reasoning, learning, memory, planning, action, and adaptation, evaluation must integrate evidence across multiple capability dimensions instead of reducing intelligence to a single accuracy score.

The framework should begin by defining the capabilities that constitute the target concept of general intelligence. These may include generalization, transfer, reasoning, planning, multimodal understanding, autonomous agency, long-horizon behavior, learning, memory, tool use, and adaptation. Each capability should be operationalized through observable behaviors so that abstract claims about intelligence can be translated into measurable experimental conditions.

Evaluation should distinguish capability from benchmark performance. A benchmark is an instrument for observing behavior under particular conditions, while a capability is the underlying competence that the benchmark attempts to measure. High performance on one dataset does not necessarily demonstrate general reasoning or generalization. Multiple tasks, environments, perturbations, and evaluation methods are therefore needed to establish stronger evidence for a capability.

A useful framework organizes evaluation hierarchically. At the lowest level, individual skills such as object recognition, retrieval, logical inference, navigation, or tool invocation can be measured directly. Higher levels evaluate combinations of these skills within reasoning problems, multimodal tasks, agent environments, and extended missions. The highest level examines whether capabilities remain coordinated when the system encounters complex objectives that require integrated intelligence.

Generalization evaluation should determine whether performance transfers beyond familiar training distributions. Tests may introduce new examples, combinations, environments, tasks, rules, or domains while preserving some underlying structure that the system must discover. Increasing distributional distance can reveal whether competence depends on memorized patterns or reflects reusable representations and strategies capable of supporting behavior under genuinely unfamiliar conditions.

Reasoning evaluation should examine more than whether the final answer is correct. The system may need to infer relationships, combine evidence, reason causally, manage uncertainty, construct intermediate conclusions, and revise beliefs when new information appears. Evaluation should vary problem structure and surface representation so that success cannot depend primarily on recognizing familiar benchmark templates or exploiting accidental statistical correlations.

Agent evaluation extends assessment from responses to behavior. An agent must interpret goals, construct plans, interact with tools or environments, observe consequences, maintain state, and determine subsequent actions. Performance should therefore be measured across complete trajectories rather than isolated decisions. Task completion, action efficiency, recovery behavior, resource consumption, constraint satisfaction, and unnecessary interventions can provide complementary evidence about autonomous competence.

Multimodal evaluation determines whether different information channels contribute to one coherent understanding of a situation. Language, images, video, audio, spatial information, sensor measurements, and actions may provide complementary or conflicting evidence. Evaluation should measure grounding, fusion, temporal alignment, spatial consistency, missing-modality robustness, conflict resolution, and the ability to use multimodal representations during reasoning and action.

Long-horizon evaluation examines whether competence remains stable as decisions accumulate over time. Extended tasks require persistent goals, hierarchical planning, memory, state tracking, progress monitoring, uncertainty management, replanning, and error recovery. A system that performs individual steps correctly may still fail when hundreds of dependent actions must remain coordinated. Evaluation should therefore examine both final success and the quality of the trajectory leading to it.

Robustness should be tested by deliberately disturbing the conditions under which intelligence operates. Inputs may be noisy, incomplete, delayed, contradictory, adversarial, or outside the normal operating distribution. Tools may become unavailable and environments may change unexpectedly. Robust systems should degrade gracefully, recognize uncertainty, seek additional evidence when necessary, and avoid turning small disturbances into uncontrolled chains of failure.

Adaptation provides another critical dimension because general intelligence should respond constructively to novelty rather than merely tolerate it. An evaluation environment can introduce new rules, objectives, interfaces, tools, or constraints after deployment. The system should infer what has changed, update its strategy, exploit previous knowledge when appropriate, and acquire new competence without unnecessarily destroying capabilities that were learned earlier.

Efficiency must be considered alongside raw capability. Two systems may achieve the same objective while consuming very different amounts of computation, memory, energy, time, external tool calls, or human assistance. An evaluation framework should therefore measure the resources required to achieve performance. Efficient intelligence allocates expensive reasoning and information gathering selectively instead of applying maximum computational effort to every situation.

Reliability requires repeated evaluation rather than isolated demonstrations. A system that solves a difficult task once but fails frequently under equivalent conditions cannot be considered dependable. Repeated trials, alternative task formulations, random seeds, environmental variations, and independent evaluation sets can reveal performance variance. Reporting distributions and failure rates provides a more realistic picture than presenting only average or best-case results.

Calibration should evaluate whether the system\'s confidence corresponds to the probability that its judgments or actions are correct. This is especially important when an AGI system must decide whether to proceed autonomously, gather additional information, request assistance, or abstain. An agent that recognizes the limits of its knowledge may be operationally more capable than one that achieves slightly higher accuracy while expressing unjustified confidence.

Safety and constraint satisfaction should be integrated into capability evaluation rather than treated as entirely separate concerns. An agent may accomplish an objective while violating operational limits, ignoring instructions, consuming unacceptable resources, or creating hazardous intermediate states. Evaluation should therefore determine not only whether the goal was achieved but whether it was achieved within specified boundaries and throughout an acceptable execution trajectory.

Human intervention provides a useful measure of practical autonomy. Some systems appear autonomous while depending on repeated clarification, correction, task decomposition, or recovery assistance from human operators. Evaluation should record when intervention occurs, why it was required, and whether the agent could resume effectively afterward. Decreasing dependence on intervention across increasingly difficult tasks provides evidence of stronger autonomous competence.

Benchmark contamination and shortcut learning can seriously distort AGI evaluation. Test items may resemble training data, contain unintended cues, or permit solutions that bypass the capability supposedly being measured. Evaluation frameworks should therefore incorporate novel task generation, hidden test sets, counterbalanced examples, controlled perturbations, modality ablations, alternative formulations, and procedural environments that make superficial memorization less effective.

Metrics should be organized into a multidimensional capability profile rather than collapsed prematurely into one universal number. Useful dimensions may include task success, generalization, reasoning accuracy, planning quality, grounding, adaptation, robustness, efficiency, calibration, autonomy, safety, memory consistency, and recovery performance. A composite score can support comparison, but the underlying profile should remain visible because different systems may reach similar totals through fundamentally different strengths and weaknesses.

Failure analysis should accompany quantitative scoring. Errors can originate in perception, retrieval, memory, grounding, reasoning, planning, tool selection, execution, state estimation, or adaptation. Identifying the mechanism of failure helps distinguish systems that produce the same incorrect outcome for different reasons. Evaluation therefore becomes not only a ranking mechanism but also a diagnostic process that reveals where architectural improvements and additional training are required.

Evaluation should also consider capability interactions because general intelligence emerges from coordination among components. Strong perception may be useless if relevant observations are not retained, and accurate reasoning may fail operationally if planning cannot translate conclusions into actions. Integrated tasks should therefore test whether perception, memory, reasoning, planning, learning, and control can exchange information coherently while maintaining a shared representation of goals and environmental state.

A mature AGI evaluation framework should progressively increase task diversity, novelty, duration, uncertainty, interaction, and environmental complexity. Early stages can establish fundamental competencies under controlled conditions, while later stages combine capabilities in dynamic environments with incomplete information and delayed consequences. This progression allows evaluators to identify where performance begins to deteriorate rather than relying on an arbitrary binary distinction between AGI and non-AGI.

Ultimately, AGI evaluation should build an evidence profile demonstrating that intelligence remains general, integrated, adaptive, robust, efficient, and reliable across changing conditions. No single benchmark can establish general intelligence. Stronger evidence emerges when a system generalizes to novel situations, reasons across modalities, acts autonomously over long horizons, recovers from failure, manages uncertainty, respects constraints, and continues to perform coherently as task complexity increases.