An index with no answer key

Dr Ananya Rao has a slide she shows at every meeting of her consortium. It has a single line on it: "What would we compare it to?" The consortium has spent four years building an EEG index of awareness called LUMEN. It runs on a cheap headset, gives a number from 0 to 100, and in adult studies it has done well. It rises when sedated volunteers wake and say so. It rises in brain-injured patients who later recover and remember. It falls in deep sleep and under general anaesthesia.

Now a neonatal unit at a partner hospital wants to use it. The consultant there, Dr Farah Siddiqui, cares for babies born at 27 weeks. Some of them need painful procedures several times a day, such as heel pricks, line insertions and, sometimes, breathing tubes. Sedation helps with distress but carries its own risks for the developing brain. Farah's question is blunt. "When LUMEN reads 22 during a heel prick, is the baby aware of it or not? I need to know whether to give more."

The consortium's statistician, Tom Hale, has run the adult numbers every way he knows. LUMEN agrees with report. It agrees with a hand-squeeze test under anaesthesia. It agrees with recovery. The agreement looks strong. But there is no test that shows directly whether anyone is aware, against which LUMEN could be checked. A pulse oximeter can be checked against a blood sample. A thermometer can be checked against a better thermometer. LUMEN can only be checked against other signs of the same hidden thing.

The external reviewer, Professor Inês Duarte, has sent back a short note. It says that she is not yet persuaded LUMEN measures awareness at all. All of its evidence, she writes, could be explained if it measured something else that tends to travel with awareness in adults, such as arousal or the capacity to move. In babies, she adds, that something else may not travel with awareness in the same way.

This paper asks how an instrument like LUMEN can establish validity when its target has no independently accepted measurement procedure. My answer, which the rest of the paper has to earn, has two parts. First, validity can be earned without a gold standard, through what I will call triangulated anchoring. That means agreement among several anchors whose errors have different causes, plus evidence that the index does not follow what it should not follow. Second, validity earned this way belongs to the index in a population, never to the index alone. I will describe it as a validity passport, stamped one population at a time.

The second part matters most for Farah. A stamp earned in sedated adults is not a stamp for 27-week babies. To earn one there, at least one anchor has to work in that population. If none does, the honest passport for that group is blank, and the decision about sedation has to be made on other grounds.

This bears on the Mind Frontier programme's working theory, the Recognition Theory of Mind (RTM). RTM describes five layers of mind: responsiveness, manifestation or access, reflexivity, self-recognition and a luminous ground. RTM is a hypothesis under test. Most anchors used in consciousness science give evidence at the second layer, about what is available for report and control. I will argue that an index validated against those anchors is validated for that layer and no higher, and that this limit has practical consequences.

Validity when nobody holds the answer key

Most instruments are checked by comparison with something better. A new blood pressure cuff is compared with a pressure line in an artery. A new scale is compared with standard weights. The better instrument plays the role of the answer key. Where it exists, validation is mostly a matter of careful arithmetic.

Consciousness has no answer key. There is no procedure that everyone accepts as showing directly whether a given person, baby or animal is aware right now. Report comes closest for adults who can speak, and the previous paper in this series showed how often report misses real cases. So LUMEN cannot be checked against the truth. It can only be checked against other imperfect signs.

The problem has a name, and it is seventy years old

Cronbach and Meehl say the idea was first worked out by a subcommittee of Meehl and R. C. Challman, while they were thinking about projective tests. The full committee's statements were published in 1954 as technical recommendations for tests.

Their solution was to embed the test in a web of expected relations, which they called a nomological network. "We shall refer to the interlocking system of laws which constitute a theory as a nomological network," they wrote. A test of anxiety earns its name by behaving the way anxiety should: rising before exams, falling with reassurance, correlating with some measures and not others. No single relation proves validity. The pattern, taken together, makes the interpretation more or less believable.

That is the right starting point for LUMEN. It should rise when sedated volunteers wake, fall in dreamless sleep, rise in the patients who later recover, and so on. But the 1955 answer has a weakness that matters more for consciousness than for anxiety. A network of relations can hang together beautifully around the wrong thing.

Reliable is not the same as valid

The horoscope is a joke, but the structure is common. Much of what gets called validation in commercial neurotechnology is reliability in a lab coat. A device that gives the same number twice is a device that works. Whether the number means anything is a separate question, and it needs evidence from outside the device.

What validity is, and what validation can show

There is a sharp alternative to the 1955 picture. In 2004, three psychometricians proposed a simpler test. The authors were Denny Borsboom, Gideon Mellenbergh and Jaap van Heerden. Their definition, in Psychological Review, reads: "a test is valid for measuring an attribute if (a) the attribute exists and (b) variations in the attribute causally produce variation in the measurement outcomes." On their view, validity is not a pattern of correlations. It is a fact about causes. Either awareness drives LUMEN's readings or it does not.

I think they are right about what validity is. If LUMEN is valid, it is because changes in awareness cause changes in the number. But their definition makes the practical problem sharper. To show that awareness causes the readings, you need some independent way to vary or detect awareness, and that is exactly what consciousness science lacks. Borsboom tells us what we are trying to establish. Cronbach and Meehl tell us how we might gather evidence for it. Triangulated anchoring is my attempt to make the second serve the first.

A third view sits between them. Samuel Messick's unified account, summarised in his 1995 article, treats validity as a judgement about how well evidence supports the uses and interpretations of scores, including their consequences. For Farah, that matters. She needs to know whether LUMEN tracks awareness in general, and also whether it can bear the weight of one decision, about sedation, in one population, at 27 weeks.

Checking a translation with its own dictionary

Translators know a version of this problem well, and it is a useful second picture. One common way to check a translation is to translate it back into the original language and see whether the meaning survives the round trip. If an English leaflet becomes Kannada and then returns as the same English, the check seems to have passed, and everyone feels reassured that nothing was lost on the way.

The weakness is plain once it is pointed out. If both translations were made with the same dictionary, the same habits and the same blind spots, the round trip can succeed while the Kannada reads strangely to every native speaker who sees it. The two steps agree because they share a cause, which is the dictionary, and the check has measured the translator's consistency with herself rather than the quality of the Kannada.

Fixing a point you cannot see

Surveyors have a classic way of fixing a point they cannot reach, such as a mountain top. They take bearings on it from two or three known stations. If the lines cross at one spot, the point is fixed. If they cross in a scatter, something is wrong with a station or a bearing.

Triangulated anchoring borrows the idea. Each anchor is a station: report, a hand-squeeze test, recovery, a brain response that needs no report. Each gives a bearing on awareness. If they converge on the same cases, LUMEN can be checked against the place where they cross.

The difference between a gold standard and a set of anchors is the core of the problem. A gold standard can be wrong, but its errors are known and small compared with the instrument's. Anchors are all of the same quality as each other, roughly, and each is wrong in its own way. Triangulation can live with mediocre anchors. What it cannot live with is anchors whose ways of being wrong overlap.

Four kinds of anchor, and how each one lies

An anchor is any observation that consciousness science already treats as evidence of awareness, before the new index arrives. None of them is perfect. The useful question about each is how it fails, and why. Two anchors that fail for different reasons can check each other. Two that fail for the same reason cannot.

LUMEN's adult evidence uses four families of anchor. I take them in turn, with the main way each one goes wrong.

Report: the first anchor, and the leakiest

Verbal report is where every science of consciousness starts. If a woman says she saw the flash, that is evidence she did. But report needs language, memory, motivation and the ability to speak or press a button. It also needs introspection to be accurate, and the philosopher Eric Schwitzgebel has argued at length that naive introspection is unreliable, even about our own current experience.

There is an older and more specific worry. In 1960, George Sperling flashed grids of letters for a fraction of a second. People could usually report only about four letters from the whole grid. But if a tone told them, just after the flash, which row to report, they could report most of that row, whichever row was cued. Summaries of the experiment draw the usual moral: more seems to be available, briefly, than can be reported. People still disagree about whether that extra availability is experience or something short of it. Either way, report and awareness can come apart in healthy adults under ideal conditions.

So report fails in two directions. It misses awareness when the person cannot speak or remember. And it may misdescribe awareness even when they can.

Command-following: awareness that leaves no memory

A later study of 338 young adults found responses in 11 per cent. These are striking numbers. They also show how anchors disagree. By the recall anchor, nobody in Sanders's study was aware. By the command-following anchor, about one in twenty were.

The interpretation is contested. Some anaesthetists, writing in Anaesthesia in 2014, have rejected the idea that a positive hand squeeze shows a distinct state of consciousness. Command-following might happen with very little awareness, or with awareness that is real but brief. For triangulation, the point is narrower. The hand-squeeze test and recall fail for different reasons. One needs a working hand and attention in the moment. The other needs memory afterwards. That difference is what makes them useful together.

Signs that need no report

A third family of anchors tries to read awareness without asking. In a review of what they called no-report paradigms, Naotsugu Tsuchiya and colleagues argued that "the requirement of reports has biased the search for the neural correlates of consciousness." They described designs that infer what a person is seeing from eye movements or brain responses, without asking them to report it at all.

These anchors avoid the leaks of report. They have leaks of their own. They are first calibrated on people who can report, so their link to awareness is borrowed from report at the start. And they can pick up processing that happens without awareness, so they risk being too wide. Their failures differ from report's failures, which is what triangulation needs. They do not escape report entirely.

Recovery and perturbation

The fourth family looks at causes and outcomes. One kind asks whether patients who score high in a coma later recover and describe having been aware. Another perturbs the brain directly. The perturbational complexity index, discussed in the previous paper, measures how hard the brain's response to a magnetic pulse is to compress. These anchors depend on brain structure and recovery rather than on speech or a hand. They fail in their own ways: recovery can be slow, memory of the coma may be absent, and perturbation needs specialised equipment.

The four families together give LUMEN something like four stations. The question is whether they sit on different hills.

Agreement that proves less than it seems

The matrix shows the trouble at a glance. Report, command-following and recall all depend on arousal. All but one depend on motor control or speech. An index that simply measured arousal would agree with every one of them in sedated adults, without measuring awareness at all. That is the possibility Inês raised in her note, and the simulation later in the paper shows that it is more than a worry.

Triangulated anchoring

Here is the method stated plainly. I will then show where it comes from and where it is weak.

Triangulated anchoring. An index earns construct validity for awareness in a population when four conditions hold.

1. Convergence. It agrees with at least three anchors, drawn from at least two of the four families.
2. Independent failure. For each pair of anchors, the main causes of error are listed and are mostly different.
3. Discrimination. It does not track the shared confounds, such as arousal or motor capacity, once these are measured separately and held fixed.
4. Intervention. It changes when interventions change awareness, and stays put when interventions change only the confounds.

The third and fourth conditions do most of the work. Convergence alone is cheap, as the three witnesses show. Discrimination is what tells you whether the agreement survives once the shared cause is removed. Intervention adds causal evidence of the kind Borsboom asks for. If a drug that paralyses muscles without sedating the person leaves LUMEN unchanged, that is evidence that LUMEN is not reading motor capacity.

How the four conditions fit together

The four conditions are not equal partners, and it helps to say how they combine. Convergence is necessary but cheap. An index that fails to agree with any anchor has no claim at all, so convergence is the entry ticket. But as the witnesses showed, it can be bought with a shared confound, so it can never be the whole case.

Independent failure is a design property, settled before any data come in. It asks whether the anchors were chosen so that their errors differ, and it can be checked by anyone who reads the methods section. A study that uses three anchors all built on report has failed this condition before it starts, however good its numbers later look.

Discrimination and intervention are where the real evidence lives. Discrimination asks whether the agreement survives once a named confound is measured and held fixed. Intervention asks whether manipulating awareness moves the index while manipulating the confound does not. A study that passes both has given the realist some of the causal evidence she wants, and has ruled out the specific shared causes it thought to test.

So I would grade a validation in three steps. Convergence alone earns nothing beyond a reason to look further. Convergence with independent failure earns a provisional stamp at most. Only the full set, with discrimination and intervention both passed, earns a stamp, and even then only for the population tested and only for the confounds named.

Where the method comes from

Campbell and Fiske called this the multitrait-multimethod matrix. Their own words state the requirement: a variable should "correlate higher with an independent effort to measure the same trait than with measures designed to get at different traits which happen to employ the same method." They also noted that "convergence between two clearly distinct methods" was a satisfactory minimum. The key word is distinct.

For LUMEN, the matrix means measuring at least two things, awareness and arousal for example, each by several methods. If LUMEN agrees with arousal measures as much as with awareness measures, it has not discriminated between them. Discrimination is the condition that commercial validation most often skips.

Looking for a cluster that hangs together

A second source is newer and comes from philosophy. Nicholas Shea and Tim Bayne, writing in 2010 about patients in the vegetative state, proposed a "natural kind" methodology for consciousness science. According to their abstract, they argue "for a cluster-based 'natural kind' methodology" as a replacement for the standard approach. The idea is to gather many putative signs of consciousness and look for clusters: properties that occur together better than chance, such that observing some lets you predict the others.

If the signs cluster tightly, the best explanation may be a single underlying property. That property, if it exists, is the natural kind, and good measures will track it. Bayne and Nicholas Shea later set out the strategy in more detail in a paper on consciousness, concepts and natural kinds, noting that it "is relatively insensitive to one's starting point."

The natural kind method and triangulated anchoring fit together. The natural kind method says to look for a cluster. Triangulated anchoring adds a warning about what can make a cluster look tighter than it is. A cluster can hang together because of awareness. It can also hang together because of arousal, which every adult anchor shares. The method needs a way to tell the two apart, and that is the discrimination condition.

What the bearings look like

The figure shows the two cases side by side. On the left, three anchors with different sources of error give bearings that cross close to the target. On the right, three anchors that all lean on arousal give bearings that cross neatly, but at a point displaced from the target. An index sitting at the crossing on the right will agree with all three anchors and still be measuring the wrong thing.

The right-hand picture is the uncomfortable one, because it looks like success. The anchors agree with each other, the index agrees with the anchors, and the statistics come out well. Nothing in the agreement itself shows that anything is wrong. The only way to see the displacement is to bring in a fourth station that does not share the lean: a measure of arousal on its own, or an intervention that moves arousal without moving awareness.

That last sentence leads to the second half of the method.

A validity passport for each population

Validity is usually written as a property of a test: "LUMEN is valid." I think that sentence is always incomplete. The full sentence is "LUMEN is valid for awareness in this population, as shown by these anchors." Change the population and the anchors may stop working, the confounds may change, and the old evidence may not carry over.

I find it helpful to think of the result as a passport with one page per population. Each page is a population. Each page carries a stamp, a provisional stamp or no stamp. A stamp lists the anchors used, the confounds checked, and the date. A border guard looking at the passport can see at a glance where the holder has been admitted and where they have not.

A true story about a stamp that did not transfer

The devices had been tested. They worked well on average. But the population in which their accuracy was established was not the whole population in which they were used. Skin pigmentation affects how light passes through a finger, and the device's error depended on it. A stamp earned on one group had been treated as a stamp for everyone.

This is the closest thing I know to a natural experiment on why validity is population-indexed. And it happened with an instrument that has a gold standard, which makes the lesson stronger. If a device with a blood-sample answer key can carry hidden population errors for decades, a device with no answer key can carry larger ones, and nobody will find them by accident.

What goes on a page

A page in the passport should record six things. Here they are for LUMEN in sedated adults, which is the page where the consortium's evidence is strongest.

1. Population. Healthy adults, 18 to 60, under propofol sedation in a laboratory.
2. Anchors. Report on waking, hand-squeeze on command, and a no-report brain response, from three families.
3. Shared confounds checked. Arousal, measured by a separate EEG index of sleep depth, and motor capacity, varied by a muscle relaxant in a sub-study.
4. Discrimination result. LUMEN's agreement with the anchors after holding arousal fixed.
5. Interventions. Sedation up and down, relaxant on and off.
6. Status. Stamped, provisional or blank, with the date and the conditions for revisiting it.

Each item is something a reader can check. None requires agreement on what awareness is in the end. The passport records how far the evidence goes and where it stops.

Transferring a stamp to a new population

Nobody can test an index in every population before using it. So the passport needs rules for carrying a stamp across a border. I suggest three, in order of strength.

Direct stamp. At least two anchors from different families work in the new population, and the full four conditions are met there. This is the best case, and it is rare outside adults.

Bridged stamp. At least one anchor works in the new population, and there is an argument that the confounds behave there as they did in the old one. The bridge has to be stated and dated, because it can be wrong.

No stamp. No anchor works in the new population. Similarity of mechanism can be argued, and it can guide caution, but it cannot stamp the page.

The third rule is the failure boundary of the whole method. Where no anchor works, triangulated anchoring has nothing to triangulate. That leaves entirely open whether the population is aware. This method simply cannot say.

What the infant evidence actually shows

Farah's babies sit right on that boundary. How far do anchors reach into early life? The most useful recent review is Consciousness in the cradle, by Tim Bayne, Joel Frohlich, Rhodri Cusack, Julia Moser and Lorina Naci, published in Trends in Cognitive Sciences in 2023.

The authors use what they call a cluster-based approach, close to the natural kind method. They distinguish "two-way markers", which have high sensitivity and high specificity, from "one-way markers", which have high specificity but low or unknown sensitivity. Most infant evidence is one-way. When a marker is present, it is evidence of awareness. When it is absent, that tells you little.

They review four lines of evidence: functional brain networks in fetuses and newborns, attention in young infants, the merging of sound and lip movement in infants of about five months, and late brain responses to unexpected sound patterns. That last marker has been reported in newborns and in fetuses past 35 weeks of gestation. They also note a lower bound. The development of connections between the thalamus and cortex, often taken to be necessary for consciousness, gives "a lower bound of 24–26 weeks gestation."

For a 27-week baby, this means the following. The lower bound has been passed. None of the markers reviewed has been shown at 27 weeks, as far as I can tell from the review. The page for 27-week infants is therefore blank on LUMEN's passport. It may stay blank for some time. That is a finding, and it is useful, because it tells Farah which question LUMEN cannot answer for her.

Licensed for one use at a time

Medicines offer a second analogy. A drug is licensed for a particular use in a particular group: this dose, for this condition, in adults. Doctors may prescribe it outside that licence, but when they do, they know they are relying on their own judgement and not on the trial evidence. The licence makes the gap visible.

A validity passport does the same. Farah may still decide to use LUMEN at 27 weeks, perhaps as one small input among many. If she does, the blank page tells her and her colleagues that she is using it off licence, so to speak.

What an anchor can vouch for across the layers

RTM separates five layers. The first is responsiveness: the system reacts to what happens to it. The second is manifestation or access: contents are available to guide report, reasoning and action. The third is reflexivity: the system takes some of its own states as objects, for example by being confident or unsure. The fourth is self-recognition, and the fifth is the luminous ground, awareness as such. The layer names borrow Sanskrit terms from Kashmiri Śaiva philosophy, as set out in the glossary.

The layers give a way to ask a precise question about LUMEN. For which layer is it valid?

The anchors vouch for access

Look again at the four anchor families. Report shows that a content was available to the speech system. Command-following shows that an instruction was available to guide action. No-report signs, calibrated against report, inherit report's link to access. Recovery with recall shows that something was available to memory. Every one of them is evidence at the second layer, about what is manifest and accessible.

So an index validated against these anchors is validated, at most, for L2. That still matters a great deal. Access is a large part of what doctors and families care about. But it is a limit. L2 evidence never stands in for L3, and none of these anchors tests whether the system can monitor its own states. Still less do they reach the luminous ground.

There is a sharper point here. If some awareness is not accessible, as the debate over Sperling's results suggests may happen, then an index validated against access anchors will miss it by design. The limit belongs to any index validated against L2 anchors, LUMEN included. I would rather say this plainly than hope it does not matter.

Farah's question sits at an awkward point on this ladder, and it is worth being exact about why. She is asking whether a baby experiences a heel prick, which is a question about awareness of pain. The baby's withdrawal of the foot and the rise in heart rate are L1 signs, responses that can occur by reflex without any awareness at all. What she wants to know is whether the pain is manifest to the baby, which is the kind of claim that L2 anchors would support in an adult.

In a 27-week baby, no L2 anchor has been shown to work. So LUMEN, which was validated against L2 anchors in adults, would be reaching for an L2 claim in a population where it has only L1 evidence to check itself against. That gap between the layers is the blank page, seen from a different angle.

Anchors for reflexivity

Can any anchor vouch for the third layer? There are candidates. Metacognitive tasks ask people how confident they are in a judgement and check whether confidence tracks accuracy. A system whose confidence rises when it is right and falls when it is wrong is, in a limited sense, monitoring its own states. Such tasks have been used in adults, in some animals and in machine learning systems.

They also have their own confound. A system can learn that certain stimuli are hard, and lower its confidence on them, without monitoring anything about its own processing. So a metacognitive anchor needs its own discrimination check: confidence should track the system's own errors, not just the difficulty of the stimulus. An index that agreed with such an anchor, with the check passed, could earn a provisional stamp at L3 for that population. LUMEN has not tried.

The luminous ground and the self-luminous lamp

The fifth layer raises a different kind of question. Classical Indian philosophy had a long argument that bears on it directly. The question was how a cognition becomes known.

According to the Stanford Encyclopedia of Philosophy's account of epistemology in classical India, the Nyāya school held that a cognition is known by a second, later cognition that takes it as its object, called anuvyavasāya. On this view, awareness of awareness is one more cognition. It can be checked, and it can be wrong. Other schools, including Mīmāṃsā and Advaita Vedānta in different ways, held that awareness is svaprakāśa, self-luminous. It reveals itself in the act of revealing anything else, as a lamp lights itself while lighting the room.

The debate maps neatly onto validation. On the Nyāya view, an index of awareness is in the same position as any other instrument. It is validated against later cognitions, such as reports, and those can be cross-checked. On the self-luminous view, awareness is never an object for a second act, so no instrument can take it as its target. The anchors would always be pointing at the contents of awareness, never at the light itself.

This scene is invented. The positions are real, and the figure below lets you compare four readings of the same number.

I take no position here on whether the luminous ground can be present in machines, and this paper does not need one. What the paper does need is a clear statement of what triangulated anchoring can and cannot reach. It can reach L2 directly, and L3 with care. It cannot reach L5, because there is no anchor for awareness as such that does not first pass through some content. That limit applies to every method I know of, not only this one.

[AUTHOR NOTE: one optional sentence on how you yourself hold the self-luminous view of awareness (as a philosophical position, as something your practice bears on, or neither). Do not add any experience that did not happen.]

An uncomfortable case at the top of the anchors

The locked-in poet is invented, but the situation is real. In 1996 the French journalist Jean-Dominique Bauby, paralysed by a brainstem stroke, dictated a memoir by blinking his left eyelid as an assistant recited a frequency-ordered alphabet, according to the Los Angeles Times report of 1997. It took about 200,000 blinks. The case tells me something about how passports should be read. A blank page is easy to forget when the alternative is no number at all. The page has to be blank in the software, on the screen and in the notes, as well as in a validation report that nobody at the bedside reads.

The strongest case against triangulated anchoring

I have argued that an awareness index can earn validity through anchors that fail for different reasons, and that its validity is always tied to a population. There are four serious objections. Each has a version I find hard to answer.

"You have described evidence, not validity"

The first objection comes from the realist view of Borsboom and his colleagues. Validity, on their view, is a causal fact: awareness either drives LUMEN's readings or it does not. Triangulation is a set of correlations, plus a few interventions. Correlations, however carefully arranged, do not establish causes. So the method describes how confident we should feel, and says nothing about whether LUMEN is valid.

I accept the distinction and resist the conclusion. Triangulated anchoring is a method for gathering evidence about a causal claim, and every method for that has the same structure. The fourth condition, intervention, is there to supply exactly the causal evidence that the realist asks for. If paralysis without sedation leaves LUMEN unchanged, and sedation without paralysis moves it, that is causal evidence. It is not proof, and nothing in empirical science is. The realist is right that the passport records evidence. A passport always does.

"There is nothing for the anchors to converge on"

The second objection comes from illusionists, who hold that phenomenal consciousness, in the sense philosophers usually mean, does not exist. What exists is a system that represents itself as having an inner light. On this view, the anchors converge on something real: the machinery that produces reports and beliefs about experience. That is all there is to find.

"You will leave the vulnerable without protection"

The third objection is ethical, and it is the one Farah would raise. If the passport for 27-week infants is blank, does that mean the unit should act as if they are not aware? The philosopher Jonathan Birch, in The Edge of Sentience, argues for a precautionary framework for cases where sentience is a realistic possibility but cannot be established. Applied here, a blank page might seem to license neglect.

It should do the opposite. The blank page says that LUMEN cannot answer the question. It does not answer it in the negative. A unit that treated "no stamp" as "no awareness" would be committing exactly the error the passport is designed to expose. The decision about sedation then has to rest on other grounds, such as Birch's precautionary reasoning, the known biology of pain pathways, and the known risks of sedation. The passport keeps a weak number from tipping a decision that should rest on those grounds.

"Triangulation is a polite word for consensus"

The fourth objection is the one I worry about most. The choice of anchors, and of which confounds to check, reflects what researchers already believe about awareness. If they all believe that awareness goes with report, they will choose anchors that go with report, and triangulation will confirm their belief. The method looks neutral but carries a theory in its choice of stations.

I think this is partly right. The defence is that the method makes the choice explicit and open to challenge. A passport page lists the anchors used and their known failures. A critic who thinks report-based anchors are biased can say so, name a missing anchor, and propose a test. That is better than hiding the choice in a validation report. It is still a defence of procedure. A determined community can choose anchors that share its blind spots. The anchor matrix makes that harder to do quietly. It cannot make it impossible.

"Only large consortia can afford this"

A fifth objection is practical. Running four anchor families, measuring confounds separately and doing intervention sub-studies costs money and time. A small team with one EEG system and a hospital partnership cannot do all of it, and the method seems to shut such teams out of validation altogether.

I think the passport answers this better than it might seem. A small team does not have to stamp every page. It can do one honest thing well, such as a two-anchor study with arousal measured separately, and record a provisional stamp with its gaps listed. A modest page that says what is missing is worth more to the field than a confident report that hides the same gaps, and it tells the next team exactly what to add. The method asks for honesty about scope more than it asks for scale.

What each rival gets right

The realist is right that validity is causal, and the passport should say what causal evidence it contains. The illusionist is right that triangulation cannot by itself tell awareness from the machinery of claims about awareness. Birch is right that a blank page must never be read as a "no". And the consensus objection is right that every set of anchors carries a theory. My method absorbs some of each challenge and is left with the rest.

The shared-confound simulator

The core claim of this paper is that agreement among anchors can mislead when the anchors share a cause of error. That claim can be tested in a simulation, where, unlike in life, we know the true answer. I wrote a short program that builds a population of simulated people, each with a hidden level of awareness and a separate level of arousal. It then builds three anchors and an index, and checks how well the usual statistics recover the truth.

The model is deliberately simple. Awareness and arousal are independent in the simulated population. Each anchor is part awareness, part confound and part noise. In some scenarios the anchors each have their own private confound. In others they all share arousal. The index is part awareness, part arousal and part noise. Three numbers are reported for each scenario. The first is convergence, the average correlation between the index and the anchors. The second is the same correlation after removing arousal, which is the discrimination check. The third is the true validity, the correlation between the index and hidden awareness, which in real life nobody can see.

import random, math
def corr(x, y):
    n = len(x); mx = sum(x) / n; my = sum(y) / n
    sxy = sum((a - mx) * (b - my) for a, b in zip(x, y))
    sx = math.sqrt(sum((a - mx) ** 2 for a in x)); sy = math.sqrt(sum((b - my) ** 2 for b in y))
    return sxy / (sx * sy)
def partial(x, y, z):
    rxy, rxz, ryz = corr(x, y), corr(x, z), corr(y, z)
    return (rxy - rxz * ryz) / math.sqrt((1 - rxz ** 2) * (1 - ryz ** 2))
def simulate(name, a, c, p, q, shared=True, n=4000, seed=7):
    """A: awareness (hidden). K: arousal and motor capacity (measurable confound).
    Three anchors load a on A and c on a confound. If shared, all three use K;
    if not, each anchor has its own private confound. The index loads p on A, q on K."""
    rng = random.Random(seed)
    A = [rng.gauss(0, 1) for _ in range(n)]
    K = [rng.gauss(0, 1) for _ in range(n)]
    def noisy(sig, var):
        return [s + rng.gauss(0, math.sqrt(max(1 - var, 1e-9))) for s in sig]
    anchors = []
    for _ in range(3):
        conf = K if shared else [rng.gauss(0, 1) for _ in range(n)]
        anchors.append(noisy([a * x + c * k for x, k in zip(A, conf)], a * a + c * c))
    L = noisy([p * x + q * k for x, k in zip(A, K)], p * p + q * q)
    conv = sum(corr(L, X) for X in anchors) / 3
    part = sum(partial(L, X, K) for X in anchors) / 3
    true = corr(L, A)
    flag = "CONFOUND-DRIVEN" if conv >= 0.3 and part < 0.15 else ("NO SIGNAL" if conv < 0.15 else "survives the confound check")
    print(f"{name}\n  convergence {conv:.2f} | after removing K {part:.2f} | true validity {true:.2f} | {flag}")
simulate("1. Index tracks awareness; anchors fail for different reasons", a=0.6, c=0.6, p=0.85, q=0.0, shared=False)
simulate("2. Index tracks arousal; anchors share the arousal confound", a=0.4, c=0.7, p=0.0, q=0.85, shared=True)
simulate("3. Index tracks awareness; anchors share the arousal confound", a=0.4, c=0.7, p=0.85, q=0.0, shared=True)
simulate("4. Negative control: index is random noise", a=0.6, c=0.6, p=0.0, q=0.0, shared=False)

Running it gives this output. It takes a few seconds and uses only the Python standard library.

1. Index tracks awareness; anchors fail for different reasons
  convergence 0.51 | after removing K 0.51 | true validity 0.85 | survives the confound check
2. Index tracks arousal; anchors share the arousal confound
  convergence 0.58 | after removing K -0.01 | true validity -0.02 | CONFOUND-DRIVEN
3. Index tracks awareness; anchors share the arousal confound
  convergence 0.34 | after removing K 0.48 | true validity 0.85 | survives the confound check
4. Negative control: index is random noise
  convergence -0.01 | after removing K -0.01 | true validity -0.01 | NO SIGNAL

The result I care about is scenario 2. The index there measures arousal and nothing else, and its true validity is zero. Yet its convergence with the anchors, 0.58, is the highest of all four scenarios. It beats the valid index in scenario 1, at 0.51, and beats the valid index in scenario 3 by a wide margin, at 0.34. An index that tracks the shared confound looks more valid than one that tracks awareness, because it agrees with the part of every anchor that the anchors have in common.

The discrimination check is what rescues the situation. When arousal is measured and held fixed, the invalid index's agreement collapses to about zero, while both valid indices keep agreement near 0.5. The check works only because, in the simulation, arousal can be measured separately. In real life that is the hard part. If the shared confound is something nobody has thought to measure, the check cannot be run, and the invalid index wins.

The negative control is scenario 4. The index is pure noise. It shows no convergence, no agreement after the check and no true validity, and the simulator labels it "no signal". If the simulator had found any convergence for a random index, that would have meant a bug.

The positive case is scenario 1, which is the pattern a real index would need to show: moderate convergence that survives the discrimination check. The negative case is scenario 2, which is the pattern Inês was worried about. Scenario 3 is a warning in the other direction. A valid index measured against anchors that share a confound looks worse than it is. Its convergence is only 0.34. Without the check, a consortium might abandon a good index for a bad one.

You can explore the same model below. The interactive version uses the exact formulas behind the simulation, so its numbers match the simulated ones to within sampling error.

A minimal worked case: two anchors in sedated volunteers

Start with the smallest real design I can think of. Twenty healthy volunteers are sedated and woken several times. At each step, LUMEN is recorded. Two anchors are taken: whether the volunteer squeezes a hand on command, and what they report on waking. A separate EEG measure of sleep depth is recorded too.

Apply the four conditions. Convergence: LUMEN can be compared with both anchors. Independent failure: the hand-squeeze depends on motor control and attention in the moment, and report depends on memory, so their failures differ in part. But both depend on arousal, so the list of shared confounds has one item. Discrimination: hold sleep depth fixed and see whether LUMEN still agrees with the anchors. Intervention: there is none in this design.

The verdict is a provisional stamp at best, for healthy sedated adults, with the intervention condition marked as unmet. That is a modest result, and it is honest. It tells the next study exactly what to add: a sub-study in which a muscle relaxant blocks the hand without changing sedation, to see whether LUMEN stays put.

A realistic worked case: Farah's request

Now the consortium's actual decision. Farah has asked whether LUMEN can guide sedation for 27-week infants. Ananya and Tom go through the passport, page by page.

The adult sedation page is stamped, after a relaxant sub-study showed that LUMEN does not follow motor capacity. The adult coma page carries a provisional stamp, because recovery is slow and recall is patchy. The term newborn page is bridged. One no-report anchor, the late response to an unexpected sound pattern, has been reported in newborns. The consortium also has an argument, which it states and dates, that arousal behaves similarly in term babies and adults. The 27-week page is blank. No anchor from any family has been shown to work there.

The consortium writes to Farah. The letter says that LUMEN has no validity stamp for 27-week infants, and explains why. It says what evidence would earn one: at least one anchor shown to work at that age, plus a check against arousal measured another way. It offers to collect LUMEN alongside the unit's existing pain scores as research data, without showing the number to the clinical team, so that it cannot sway decisions until it has earned the right to. And it says, in so many words, that the blank page is not evidence that the babies are unaware.

The negative control and the counterexample

The negative control for the method itself is an instrument that should pass easily. A pulse oximeter, checked against blood samples, has a gold standard. Triangulated anchoring is not needed. If my method demanded triangulation for a pulse oximeter, it would be asking for the wrong kind of evidence. It does not, because the method applies only where no gold standard exists.

Yet the pulse oximeter is also the paper's counterexample to a simpler version of my view. It shows that even with a gold standard, validity can fail across populations. So the population index applies to every instrument, consciousness indices included. What consciousness adds is the absence of a gold standard, which makes population failures harder to find. That correction narrows my thesis: the passport idea is general, and triangulation is the special tool for targets with no answer key.

Where the method stops working

The method has a clear failure boundary. It needs at least one anchor in the population. Where none exists, there is nothing to triangulate. Three populations are at or past that boundary now.

Very preterm infants, before any marker has been shown, are the first. Many animals are the second, though some anchors, such as learned responses to stimuli that are reported as conscious in humans, may reach some species. AI systems are the third. For them, report is available but its link to awareness is in doubt, as the infant and the chatbot showed, and no other anchor family has been calibrated. Robert Long, Jeff Sebo and colleagues, writing about taking AI welfare seriously, propose that developers should start assessing such systems for evidence of consciousness. On my account, any such assessment begins with a blank page and has to say so.

A second boundary concerns the confounds. The discrimination check only works for confounds someone has thought of and can measure. If awareness and some unmeasured property travel together in every population where anchors work, no amount of triangulation will separate them. That is a limit of the method, and I do not see a way round it.

[AUTHOR NOTE: a short first-person account, only if true, of a time in your own technical work when two or more checks agreed with each other for a shared wrong reason, and how you found out. Do not add any detail that did not happen.]

What would change my mind

Four results would make me give up or substantially revise triangulated anchoring.

The first would be a case where anchors with clearly independent errors agreed, an index matched them, the discrimination and intervention checks were passed, and the index was later shown to track something other than awareness. That would show the method can fail even when it is followed in full. I would then want to know what the shared cause was, and whether any method could have caught it.

The second would be evidence that the main confounds do not, in fact, travel with awareness in adults. If arousal and motor capacity turned out to be cleanly separable from every anchor, the shared-confound worry would be much smaller than I have claimed. The simulation would still be correct, but its scenario 2 would be a curiosity.

The third would be a validated anchor that reaches the first or third layer without passing through access. If someone showed, for example, a sign of awareness in infants that had been checked against something other than report-calibrated measures, my claim that the anchors vouch only for L2 would be wrong.

The fourth would be an empirical result about passports. I expect that a validity passport, read by clinicians, would reduce the use of indices outside their stamped populations. Suppose a careful study found that clinicians treated a blank page as a reason to trust their own impressions even more. Suppose too that their impressions were worse than the index. Then I would have built a tool that makes decisions less accurate. I would then need to rethink how blank pages are shown.

What I still don't know

I don't know how independent the anchors really are. Report and command-following feel different, but both may draw on the same attentional systems in ways no one has fully mapped. My anchor matrix is a judgement, and it could be wrong in either direction.

I don't know how to handle confounds that no one has named. The discrimination check depends on measuring the confound. Unnamed confounds are, by definition, unmeasured. I suspect the honest answer is that each new population will reveal at least one, as the pulse oximeter did.

I don't know whether L3 anchors can be made to work outside adults. Metacognitive tasks need a response and some way to express confidence. Some animal studies use wagering or opt-out designs. Whether these could ever be used with infants, or calibrated for AI systems, is open.

I don't know how to weigh a bridged stamp. A bridge is an argument that confounds behave similarly across populations. Some bridges are strong and some are weak, and I have no scale for the difference.

Claim ledger

These are the paper's main claims as I would enter them in the programme's Claim Ledger. Confidence uses three bands: established (strong evidence and no serious challenge I know of), likely (the evidence or argument favours it, with live dispute) and speculative (my hypothesis, argued here but not tested).

There are two dated predictions. First, by 31 December 2030, at least one peer-reviewed study will report a marker of awareness tested in infants born before 30 weeks of gestation, using a design that includes some check against arousal measured separately. I will judge it by a literature search and record the first example, or record the prediction as failed. Second, by 31 December 2029, at least one commercially marketed brain-based monitor or AI assessment tool that claims to measure awareness or consciousness will publish validation results broken down by population, in the form of something like a passport. I will judge it from public product material and record the first example I find.

Glossary

Anchor

An observation already treated as evidence of awareness, such as report or command-following, against which a new index is checked.

Anuvyavasāya अनुव्यवसाय

In Nyāya, a later cognition that takes an earlier cognition as its object; often translated apperception.

Cit चित्

Awareness itself. RTM layer 5, the luminous ground.

Construct validity

The degree to which a test can be interpreted as measuring an attribute that has no operational definition.

Convergence

Agreement between an index and other measures of the same target.

Discrimination

Evidence that an index does not track what it should not, such as arousal, once that is measured and held fixed.

Isolated forearm technique

A test in which one arm is protected from a muscle relaxant so a patient can squeeze a hand on command during anaesthesia.

Multitrait-multimethod matrix

Campbell and Fiske's table of correlations among several traits, each measured by several methods.

Natural kind methodology

Shea and Bayne's approach of looking for clusters of signs that hang together, and for the property that explains them.

Nomological network

Cronbach and Meehl's term for the system of laws linking a construct to observations and other constructs.

Prakāśa प्रकाश

Light; in Kashmiri Śaiva thought, the awareness by which anything appears. RTM layer 2 borrows the name.

Pratyabhijñā प्रत्यभिज्ञा

Recognition. RTM layer 4 borrows the name for self-recognition.

Shared confound

A cause of error common to several anchors, which can make them agree for the wrong reason.

Spanda स्पन्द

Pulse. RTM layer 1 borrows the name for responsiveness.

Svaprakāśa स्वप्रकाश

Self-luminous; awareness that is aware of itself without a second act.

Triangulated anchoring

This paper's method: convergence with anchors whose errors differ, plus discrimination and intervention checks, in a stated population.

Validity passport

A record of an index's validity, one page per population, each stamped, bridged, provisional or blank.

Vimarśa विमर्श

Reflective grasp. RTM layer 3 borrows the name for reflexivity.

The letter that went to the neonatal unit

The consortium's letter reaches Farah on a Tuesday. She reads it twice, as she read the first LUMEN brochure. She had half expected a yes, and half expected a hedge. What she gets is a page with nothing on it, and a clear account of why.

At first she is annoyed. She had wanted help with a hard decision, and the blank page gives her none. Then she notices what the letter does give her. It tells her that the number she would have seen on the screen during a heel prick would have carried the authority of four years of adult studies, and none of the evidence. A low reading would have nudged her team towards less sedation. A high one would have nudged them towards more. Either nudge would have rested on a stamp that had never been issued.

She agrees to the research arrangement. LUMEN will be recorded in her unit, alongside the existing pain scores, and nobody on the ward will see the number. In two years, if one of the infant markers can be measured at that age, the consortium will check LUMEN against it, and against arousal measured another way. The page might get a stamp. It might stay blank. Either result will be worth knowing.

Inês, the reviewer, sends a two-line reply when she is copied in. The first line says that the consortium has answered her concern about arousal, at least for adults, and that she would like to see the relaxant data when it is written up. The second says that she has seen many validation reports, and this is the first that told a clinician what an instrument could not do. Ananya adds the sentence to her slide, under "What would we compare it to?"

Tom, the statistician, runs one more analysis before the letter goes out. He reruns the adult data with the separate arousal measure held fixed, the same check the simulator performs. LUMEN's agreement with report and hand-squeeze drops from about 0.6 to about 0.45, which is a smaller fall than Inês had feared and a larger one than Ananya had hoped. The drop tells them that part of LUMEN's adult record was borrowed from arousal, and that most of it was not. Both halves of that result go on the passport, because a reader deciding whether to trust the index needs to see what was borrowed as well as what survived.

Ananya adds a short paragraph to the letter for the parents' group that meets in the unit each month. It says, in plain words, that the team cannot yet tell whether a baby this small feels a heel prick. It says that the monitor beside the cot will not be used to answer that question. It also says that the unit will keep giving comfort measures as before. She reads it aloud to Farah before sending it. Farah asks for one change: the word "yet" should stay in the first sentence, because it is honest about where the work is heading. The word stays. The consortium also posts the passport on its website, with each page dated, so that any parent, nurse or rival lab can see which stamps exist and which do not.

For the babies in Farah's unit, nothing visible changes. The decisions about sedation are made as they were before, using the unit's existing scales, the known biology of pain and the known risks of the drugs. That is the point. An index that has not earned a stamp should leave a decision exactly where it found it. LUMEN will earn its place at the cot side, if it ever does, one page at a time.

Sources

Cite this work

@misc{mahapatra_U0007,
  author       = {Mahapatra, Rajesh Ranjan},
  title        = {Construct validity for an unobserved phenomenal target:
                  how an awareness index earns trust when nobody holds the answer key},
  howpublished = {Mind Frontier, paper U0007},
  url          = {https://rajeshrmahapatra.com/papers/mind-frontier/U0007-construct-validity-for-an-unobserved-phenomenal-target.html},
  note         = {DOI: to be assigned}
}