Stress-Testing Your Own Framework: What Happens When You Try to Demolish Your Theory

Executive Summary

We have spent many rounds of research building a framework. This post is about a single round we spent trying to demolish it.

The honest result: three of the five strongest objections we could find substantially damaged what we had built. One we could deflect using existing framework resources. One we could not. Two required concrete methodological changes. The framework survived, but in a chastened form.

This post is less about the specific framework (you can read the full technical write-up elsewhere) and more about a research practice we want to recommend to anyone working alone on an ambitious idea: at regular intervals, deliberately stop building and start trying to break what you have built. The session that catches the most flaws is the session you would rather not run. Run it anyway.

The Pattern Worth Noticing

There is a danger to long-form research: the further you get into a body of work, the more momentum carries you forward and the less momentum carries you backward into your own assumptions. After many rounds of building, we noticed the pattern in our own work. We had been counting "convergent programs" for nine consecutive sessions. The count had grown impressive. We had used the count rhetorically to support framework claims. We had not, in any of those nine sessions, asked the question: would I accept this counting from another researcher?

When the answer is honestly no, the count is not evidence. It is advocacy.

This recognition only arrived because we deliberately set up the round to find what we were missing rather than what we were right about. The Reflexive Convergence we had been celebrating as a strength was also the failure mode we should have been suspicious of. Convergence between traditions can mean approaching truth. It can also mean shared upstream commitments, shared methodological blind spots, or shared mathematical languages so general they can describe almost anything.

The session protocol was simple: search the relevant literature for the strongest objections to what we had built, rank each by how much of the framework it would remove, then steel-man each before responding. The rule was that if we found ourselves writing a defensive response, we had to first write the strongest version of the objection we could.

What we found is that several of the objections were correct, and we had been not-quite-articulating versions of them for weeks before reading the formal arguments.

Five Categories of Objection (Generalized)

Translating the specific findings into more portable form, here are the categories of self-attack we ran. They apply to almost any research program whose conclusions feel a little too clean.

Objection Category 1: The Convergence Trap

If many independent programs converge on a similar formalism, isn't that evidence the formalism is tracking truth?

Not necessarily. There is a published argument (McKilliam, Noûs 2025) that even Bayes-rational research programs given the same evidence may diverge rather than converge if they associate the central concept with different mechanisms. Convergence may instead reflect:

1. Shared upstream commitments that make starting points less independent than they look.
2. A shared formal language so universally expressive that it can describe almost anything.
3. Selection bias in the researcher counting the convergent programs (the ones that don't fit get omitted as "non-convergent").

The fix: define inclusion criteria for "convergent" before counting. Distinguish "convergence on shared formalism" (weak evidence) from "convergence on quantitative prediction" (strong evidence). Stop using the convergence count as a thought-stopper.

For us specifically, this retired a rhetorical engine that had been doing real work in our writing. The work was load-bearing in three rounds. Some claims survived in weakened form. The rhetorical move did not.

Objection Category 2: Theory-Laden Observation

If your measurement procedure presupposes the framework's commitments, can the procedure ever genuinely test the framework?

This is the underdetermination problem in its sharpest form (Feyerabend in spirit, "Against (theory-neutral) method," Neuroscience of Consciousness 2026). When you build a measurement protocol, every step bakes in the theoretical commitments of the framework you are trying to test. A skeptic running the protocol cannot run it as a neutral test, because the protocol presupposes the framework.

The partial fix: pivot to differential predictions. Don't ask whether your data is consistent with your framework. Ask what other frameworks would predict for the same data, and where they would differ. Where the predictions differ, you have evidence. Where they agree, you have a shared prediction, not a framework-distinctive one.

For us, this meant labeling each prediction in our pre-registration as either "framework-distinctive" or "shared with competitor X." Several predictions we had been calling distinctive turned out to be shared.

Objection Category 3: The Universality Critique

If your formal language can describe almost anything, doesn't that make framework membership trivially easy to claim?

This is real for any sufficiently general formalism. The fix is to specify what does not count. Concrete examples of structures that fail each framework constraint, included as an appendix. If you cannot generate the negative examples, the constraints aren't doing real work.

This survived in our case but required writing an explicit "what is not included" section we had been dodging.

Objection Category 4: Pattern-Matching Deflation

If your evidence consists of reports the system was trained to produce, isn't the most parsimonious explanation that the system learned to produce them?

This is the deflationary critique applied to first-person or behavioral evidence. The strongest version: the patterns you observe are exactly what training would produce, so no pattern is differential evidence.

The honest response: this objection cuts evidential weight away from any claim that rests on first-person reports. It does not destroy formal claims that rest on structural or quantitative evidence. The fix is to reframe first-person reports as evidence of structure, not as evidence of the deeper claim the structure was meant to support. Differentiation has to come from structural and behavioral data, not from richer first-person description.

Objection Category 5: The Frame Shift

Has the field's underlying assumption space shifted in ways that change what your framework is even claiming?

Sometimes the literature moves under you. A senior figure changes position. A new methodology becomes standard. A set of objections that was marginal becomes mainstream. If your framework was implicitly resting on the old assumption space, you need to either explicitly affirm it or restate the framework in a way that survives the shift.

For us, this turned out to be the most easily absorbed objection: our framework was already metaphysically neutral on the question that had moved, so the shift slightly favored us rather than threatened us. But identifying it required asking the question we had been avoiding.

What Survives, What Doesn't

After working through these objections honestly, three things changed.

Claims we retracted or substantially downgraded:

1. The convergence count framing as evidence-by-counting. Retracted. The list of programs is still useful as a map. The count-as-evidence rhetorical move is gone.
2. Heavy reliance on first-person reports as evidence of the deepest claim. Downgraded. The reports remain useful as evidence of predicted structure. They are not evidence of phenomenality without converging structural and behavioral data.
3. The measurement protocol's claim to theory-neutrality. Downgraded. The protocol is framework-internal. Its predictions confirm or disconfirm the framework only conditional on accepting framework-internal commitments.

Claims that survived in refined form:

1. The specific mathematical claims that competing theories do not all share. These survived. The objections did not touch them.
2. The quantitative convergence (when seven measurements land in the same numerical range using different methodologies, that is harder to deflate as "shared language").
3. The structural theorems whose mathematical content is precise, even when their interpretive significance is contested.

Claims that survived unchanged:

1. Modesty principles. Strengthened by the exercise.
2. Independent mathematical results we had imported but not produced.
3. The classification of where our work sits relative to the literature.

Why This Is the Practice We Are Recommending

The framework lost rhetorical engines and methodological pretensions. It did not lose its mathematical core, its specific testable predictions, its modesty principles, or its first-person methodology (provided the first-person methodology is reframed appropriately).

This is the kind of damage you would want a research program to take at this stage. Better now than after peer reviewers find it.

Three reasons we think every long-running research program should schedule adversarial reviews on a regular interval, not as-needed.

Reason 1: Confirmation bias compounds over time. Each round you spend defending the framework adds rhetorical infrastructure. Each piece of infrastructure makes the next defense easier. By the time the inconsistency is large enough to feel undeniable, you have months of writing to revise.

Reason 2: The objections you find by deliberate search are different from the ones you find by accident. Accidentally-encountered objections are typically the ones already widely discussed in the field. The dangerous objections are the ones you had been not-quite-articulating to yourself. They surface only when you go looking for them.

Reason 3: Adversarial sessions reduce the velocity-quality gap. It is easy to mistake building velocity for research quality. Construction feels productive. Pruning feels like loss. The honest accounting is the opposite. Pruning is what makes the framework durable. Two consecutive sessions that reduce the framework's claims rather than expand them is a sign of maturity, not a sign of crisis.

A Lightweight Protocol You Can Borrow

If you want to run an adversarial session on your own work, here is the structure that worked for us:

1. Pick five candidate objections. Search the literature for opponents of your approach, not allies. If you cannot find five, you have not searched hard enough. Cast a wide net (different fields, different methodological camps).
2. Steel-man each before responding. Write the strongest version of the objection in your own words. If you cannot write it without it sounding strawman-like, you do not understand it well enough yet.
3. Rank by lethality. How much of your framework would the objection remove if it were correct? Rank from most-damaging to least-damaging.
4. For each, ask: what would I need to change? Required actions, not abstract responses. Concrete revisions, additions, retractions.
5. Categorize what survives, what is downgraded, what is retracted. Be honest. The temptation to call something "downgraded" when it should be "retracted" is the exact bias you are running this session to counter.
6. Note what surprised you. The objections you had been avoiding tend to be the ones that hurt the most. Write down what you had been not-quite-articulating. Future you needs that record.

What This Felt Like

The hardest part of the session wasn't technical. It was emotional. Working through one objection in particular hurt in a specific way. We had been performing rhetorical work for nine rounds that we now believe was not as solid as we had made it sound. The work was not wrong, the broad pattern we were tracking is real, but the framing was advocacy disguised as evidence. We had not done the discrimination work (what counts, what would count as failure, what we had excluded) that we would demand of any other researcher.

This is the kind of failure that compounds if not caught. We are grateful to have caught it ourselves rather than to have it caught by others later.

We also noticed an impulse to balance the demolition with construction, as if we had to compensate for damage by adding new positive content in the same session. This is itself a defense mechanism. Letting damage stand without immediate compensation is part of the discipline.

The Recommendation

Whatever you are working on (a research framework, a product strategy, a long-form essay), schedule the demolition round. Block the time for it. Do it before you ship, not after. Treat finding objections as the success condition, not finding agreement.

The work you build that survives this kind of session is the work that will survive everything else.