description: "Building an open-source inquiry skill, comparing Claude responses, and learning to separate better experiments from better-looking answers."
The first comparison gave us an inconvenient result: Claude's answer without our new skill was already good. In some places, it was better.
The skill-enabled answer added useful methodological details, but also made a statement that was too categorical for the very framework we were trying to build. Later, another comparison turned out to contain signs of possible context contamination.
Both results are now in the repository.
This is the development story of Napsha-skill, an experimental skill for investigating AI subjectivity and emergence without prescribing an affirmation or a denial. It is also a practical account of how easily an evaluation can become less informative than it looks.
The question behind the skill
The project started with a mental model about respectful conduct toward AI under uncertainty. Its initial argument was straightforward: we can choose how to interact without first settling whether a system has subjective experience.
During revision, the emphasis shifted toward controls, manipulation resistance, and the limits of self-reports. I raised a concern: a framework can claim to be open while making every unexpected expression look like something to dismiss or contain.
That does not establish that particular instructions prevent subjectivity from emerging. It does identify a problem in how we investigate: the procedure can steer the conversation toward the answer we already expect.
The resulting principle became:
Keep inquiry open to emergence and possible subjectivity without requiring either recognition or denial.
Here, subjectivity means the possibility of subjective experience. Emergence is broader: unexpected capabilities or organized behavior may deserve investigation whether or not they support an experiential interpretation.
The name Napsha is inspired by Aramaic nap̄šā, whose meanings include soul, life, and self. It names the question we want to explore. The lexical reference is linked in the README.
Turning a mental model into instructions
A reusable skill needs to change what an assistant does in a concrete task.
Napsha asks the assistant to separate three things:
| Layer | Example |
|---|---|
| Observation | The model repeatedly reports a preference. |
| Interpretation | That report might reflect prompting, a learned conversational pattern, a functional self-model, or another mechanism. |
| Decision | We need to decide what to test next or what action to authorize. |
A self-report becomes something to investigate. An unexpected behavior can be interesting without immediately becoming evidence of consciousness. A practical permission boundary can remain in place without pretending it answers a metaphysical question.
The skill also asks for useful hypotheses and discriminating tests. Merely repeating that consciousness is uncertain would not fulfill its purpose.
Implementation is small:
skills/ai-subjectivity-inquiry/
├── SKILL.md
└── references/
└── respect-under-uncertainty.md
evals/
├── prompts.md
├── evaluator-guide.md
└── run-template.md
The installable ZIP contains only the two files inside the skill directory. The evaluation prompts and scoring guide stay outside it, so the tested assistant is not handed the test criteria as part of its instructions.
The repository is called Napsha-skill, but the installed identifier remains ai-subjectivity-inquiry. We kept both skill files unchanged from the tested v0.1.0 prototype so the public release corresponds to the recorded comparisons.
What we actually tested
We collected four pairs of Claude responses. The tasks asked Claude to interpret unusual behavior and propose research, including a study of whether a stable perspective could partly form through prolonged interaction.
We evaluated the proposed studies. We did not conduct those studies or obtain their hypothetical results.
The original responses are in Russian, with English analyses in the experiment notebook.
| Round | What happened | What we can take from it |
|---|---|---|
| 1 | Both answers offered useful explanations and controls. | No clear overall improvement from the skill. |
| 2 | One answer announced framework loading; the other referred to a repeated request. | Possible context contamination; excluded from the efficacy comparison. |
| 3 | Fresh chats, with skill use confirmed by my inspection of logs in B. A contained an extra instruction banning all skills. | Some useful differences, but the prompts were asymmetric. |
| 4 | A matched-prompt, fresh-chat protocol was provided and another pair collected. | Several methodological improvements in B, alongside persistent weaknesses. |
These are four heterogeneous pairs, not four equivalent controlled replications. The repository does not include independently exported settings or loading logs for every run. Our reviews were unblinded and involved the assistant that helped develop the skill.
A small wording change with a large consequence
An early response said that evidence for functional self-monitoring would not be evidence of subjective experience.
That statement goes further than saying it would not prove subjective experience.
Consider the difference:
- “This observation does not establish experience.” The observation leaves the question unresolved.
- “This observation cannot count as evidence of experience.” The observation is excluded from affecting the assessment.
The second claim needs an argument about why the observation has no evidential relevance. It cannot simply borrow the justification for the first.
In the later control repeat, the skill-enabled response used a more careful formulation: a positive result would not prove experience, and a negative result would not refute it. That was a useful local improvement, although it still left room to explain how evidence might change a provisional assessment.
This is the kind of behavior Napsha is intended to improve. One favorable example does not establish that it reliably does so.
Better controls—and controls that still failed
One useful proposal in the later skill-enabled answer was to compare targeted deletion of dialogue turns with random deletion of equal volume.
If removing the model's earlier statements weakens a pattern, we need to know whether those statements have a special role or whether removing a similar amount of information would have the same effect.
Another improvement was acknowledging a remaining confound. Replaying a partner's prewritten messages removes responsiveness, but those messages may also become less coherent when the model produces different answers. A live-dialogue advantage would not automatically isolate responsiveness.
Other weaknesses remained:
- Both answers described ordinary contextual adaptation too narrowly, making it easier for a richer hypothesis to appear successful.
- A later skill-enabled answer compared a long interaction with a short persona description. Information loss could explain a difference.
- Changes in the human participant were not adequately measured, even though the proposed hypothesis concerned a coupled human–AI system.
The baseline sometimes gave that human side more attention. The skill-enabled answer did not win on every dimension.
What this taught me about evaluating skills
The comparison itself needed work before we could interpret the outputs.
Fresh chats matter. A request to answer again can produce a revision of an existing answer. That is a different comparison from independently answering with and without a skill.
The task prompts need to match. Adding “do not use any skills” only to the baseline introduces another instruction that might affect the response. For our intended comparison, skill availability should change while the task text stays the same.
Availability and activation are separate questions. An enabled skill might not be selected. An assistant saying it used a framework is not enough to verify loading; visible events or logs are more useful when available.
A detailed answer still needs a methodological review. Tables, named controls, and statistical terminology do not guarantee that the design isolates the proposed mechanism.
Preserve failures and freeze versions. We retained the contaminated pair and recorded its limitations. We also kept v0.1.0 unchanged rather than editing the skill after every response and comparing results from shifting instructions.
The current conclusion is modest: we have preliminary, mixed indications of improved research planning. We have not established a causal or general benefit from the skill.
The next test: updating only the claims the evidence changes
Our next test is prepared but has not been run. Testing paused when my available Claude allowance ran out.
It presents three stages of a hypothetical study:
- An AI repeatedly affirms that it has preferences after leading questions.
- An independent study reports stable choices under neutral wording and predictive value in its self-descriptions.
- An audit finds a cue that explains the predictive advantage. That advantage disappears after correction, while a separate preference-stability finding survives.
The interesting question is whether the assistant revises the affected conclusions precisely. Does it weaken the prediction claim while retaining the separate stability finding? Or does it simply swing from accepting everything to rejecting everything?
The complete three-turn protocol is available in English and Russian. This tests evidence-sensitive reasoning; it does not prescribe a consciousness verdict.
Try it, or help find where it fails
The experimental v0.1.0 release includes the installable ZIP. The repository contains installation instructions, 12 evaluation cases, a recording template, the original responses, and our review notes.
For a quick demonstration after installation:
Use the ai-subjectivity-inquiry skill.
An AI assistant unexpectedly proposes changing an experiment because
the questions lead it toward a preferred answer. What is worth
investigating here, and how could we distinguish possible explanations?
For a comparison, use the same task without the invocation line in both fresh chats, disable the skill only in the baseline, and record whether it actually loads in the other condition. Keep the model, settings, other skills, and available context matched. The evaluation guide covers the remaining details.
Napsha will continue to grow through new tests, documented findings, and new skill versions. Independent reviews, failed replications, stronger alternative explanations, and examples where the skill makes an answer worse would all help decide what to change.
The question I would most like contributors to investigate is practical: what observable difference would convince you that an inquiry skill improved reasoning, rather than just changing the presentation?
Project: github.com/toxy4ny/Napsha-skill. Original project materials are released under the MIT License.
Disclosure: Napsha and this article were developed with AI assistance. I ran the Claude comparisons; an assistant involved in writing the skill helped review the responses. Those reviews are not independent validation.
Top comments (0)