Understanding a failing test often means reading more than its output. A message like expected 90, got 100 tells you the output changed, but not enough to decide whether the change is right.
Put the inputs beside the changed result:
Now you can see that the new total doesn't account for the discount.
Without that context in the test output, you go through the setup, fixtures and logs to assemble the case yourself. Then you do it again for the next failure.
If the inputs are alongside the output, you can have the readable case itself as a test. In this example, a program (runner) would read price and discount, calculate the total, and compare it with the saved expectation. I call such a file a caseshot: the inputs and expected output together, in the problem's own notation.
But a readable case is useful beyond testing. The same notation can let you compare old and new implementations during a rewrite, review algorithm changes, and inspect production results. Once the inputs and outputs sit together in a form you can judge, you don't have to reconstruct the case for each of those jobs.
I used this approach while rewriting a system that distributes new investment money across an existing portfolio. Most of this post works through that example.
The Problem
Suppose you have investments spread across a few funds, with a target share for each. Over time, some grow faster than others, and the shares drift. Now you have new money to invest. Where should it go to bring the portfolio closer to its targets, without selling anything?
We call this levelling. It makes a useful example because some targets can't be reached. Judging the result means seeing not just where the money went, but why a gap remains.
The investments form a hierarchy: asset class → sub-class → fund. Each level's children add up to its total. Say you hold ₹60k: ₹10k in a liquid debt fund and ₹25k each in two large-cap equity funds. Here is that portfolio, drawn as nested bars:
Now you have ₹40k to invest, taking the final corpus to ₹100k. Your targets are 60% equity, 35% debt and 5% gold. Within equity, you want ₹20k in each of the two large-cap funds and ₹20k in a mid-cap fund. The dashed lines show these ideals:
Large-cap already holds ₹50k against an ideal of ₹40k. We can't take money out. Our levelling gives priority to the asset-class targets, then the sub-classes, then the funds. So equity gets ₹10k, debt ₹25k and gold ₹5k:
Equity's ₹10k all goes to mid-cap, which still ends short of its ₹20k ideal. Debt and gold reach theirs exactly. Some targets are reachable, some are not. The gap that remains is part of the answer.
From a Picture to a Test
The picture lets us see the case. Now put the same information into text: targets and current holdings next to the allocation, final share and remaining gap.
Below is a simplified version of our notation. AC and SAC stand for asset class and sub-class. All target percentages, including those on nested rows, are shares of the whole portfolio, not of the parent.
Each % column refers to the amount beside it: share of current holdings, share of new money, then share of the final portfolio. The gap is final share minus target, in percentage points. Large-cap ends at 50% against a 40% target, so its gap is +10 points.
The real notation also carries fund constraints. For example, a fund's label can include its minimum investment and the increment it accepts:
A nonzero investment in this fund must be at least ₹1,000, in ₹100 increments. Other constraints, such as different minimums for new holdings, need to be recorded too. The file must hold enough to reproduce the case without hidden setup.
Some columns are redundant on purpose. The final amount, percentages and gap follow from the inputs and allocation. But printing them lets someone judge a row without doing the arithmetic. Concise doesn't have to mean storing each fact only once.
To turn the file into a test, we parse only the inputs, solve again, and render the result in the same format. A difference fails the test. We review it and either fix the code or accept the new output as the expected result.
Reviewing a Change
Here is a screenshot of a made-up failure, with old values struck out and new ones beside them:
This demonstrates how a change in behavior would show up. In other words, you can show what changed, right beside the previous value, with color/formatting to make it convenient for the reader. The notation helps the reader reason about situation better.
The Leverage
Once we had this notation, the same representation was useful outside the test runner:
- Comparing implementations. We had to rewrite our existing levelling system. I could render both systems' results for the same case together and review the differences, rather than reconstructing each result from logs.
- Turning production cases into tests. Our
rec2testtool converts a captured request into a caseshot. A surprising result can become a regression test instead of remaining a one-off investigation. - Reviewing live requests. We have a page listing solves for ongoing review. The same notation lets us inspect a production result before deciding whether to keep it as a test.
- Explaining the feature. A collection of cases shows what goes in, what comes out and what the edge cases look like. No hopping between a test module, a fixtures directory and a snapshot directory.
- Trying alternatives. Render the notation in HTML and make the inputs editable, and it can also serve as a playground for people who know the domain but not the implementation.
Two More Notations
The shape carries across problems. Here are two more that my colleague Anantha Kumaran designed.
The first covers capital gains. It captures a simple case where partial redemption is made. The case holds the price on the closing date and the trades that led to it; the expected row holds the resulting position and the realised figures beside it.
The second covers the CSV importer for FinBodhi, a personal finance app. This one captures a validation error in case of unbalanced transaction.
Neither notation looks like the levelling table, because neither problem looks like levelling. What they share is the arrangement: named sections holding the inputs, then the output rendered in the terms someone would use to describe the case out loud.
The Effort
There is a cost. You have to write and maintain a parser for your own render. The format becomes an interface: reformat the table and every snapshot changes, even if the behaviour hasn't. The output also needs to be stable enough that repeated runs don't produce distracting diffs.
Our tooling is written in Elixir, mostly by an LLM. That made the renderer, parser and recording converter cheaper for us to build. It didn't remove the design work: deciding what a person needs to see, what can be left out, and which differences matter. Round-trip tests can check that rendering and parsing preserve the inputs; the generated code still needs review.
The effort is easier to justify where deciding whether an output is right takes domain judgment: solvers and planners, gains and tax calculations, importers that reshape someone else's file. The machine can check equality afterwards. Someone still has to decide what it should be equal to.
How It Fits with Other Tests
A caseshot is a snapshot test, with a readable domain notation and reproducible inputs in the same file. Like a table-driven test, one runner can work through many cases. Neither idea is new; the useful part is choosing a representation that makes the cases easier to judge.
It doesn't replace property tests, which check rules over generated examples. The levelling solver has several useful rules: never sell, stay within the budget, respect fund minimums and steps, and make parent totals equal the sum of their children. The gains example has its own — units and cost never go negative, realised plus remaining cost equals what was invested. Those checks complement the caseshots. A counterexample found by a property test can become a caseshot worth keeping.
Nor does recording the derived columns prove they are correct. A snapshot catches a change in a percentage calculation, but a wrong percentage can still be accepted. Independent arithmetic checks and human review are still needed.
Prior Art
Putting inputs and expected output together has a long history. Python's doctest makes a REPL transcript executable: the runner extracts the inputs, evaluates them and compares the result. Cram and Mercurial's .t tests do the same for shell sessions, keeping commands and their expected output together.
Compiler tests use similar arrangements. LLVM's lit and FileCheck tests keep the input program and CHECK: patterns in one file. Zig's case tests also keep the program and expected output together. For databases, sqllogictest pairs SQL queries with expected results, which can be recorded from a reference engine.
Jane Street's essay on expect tests describes the record-and-review workflow: run the code, inspect its output, and accept it into the test. It also makes the case for output that helps you understand behaviour, rather than just asserting individual fields. Rust's insta supports this workflow with both separate snapshot files and inline snapshots.
The TypeScript compiler is a large example of a designed render. Each test is a source file whose results are recorded as baselines and reviewed before being accepted. The .types baseline repeats the program with each expression's inferred type beneath it, and the error baseline puts each diagnostic under the code it refers to:
A caseshot keeps the inputs in the same file; TypeScript's baselines sit beside the case they came from. The rest is the same idea: generate the output, render it for a reader, and have a person accept it.
Julio Merino's A Markdown-based test suite is another related example: test cases kept in readable Markdown files.
Caseshot is my name for the combination used here, not a claim that this needs a separate testing framework. The part I want to emphasise is designing the notation for the person reading it.
Conclusion
Not every problem needs its own notation. But if reviewing a changed result means repeatedly reconstructing the case from fixtures and logs, it may be worth giving that case a readable form of its own. Keeping the inputs and result together makes test failures easier to understand and review. Beyond testing, the same notation gives you a way to inspect production results, compare old and new implementations during a migration, and turn a surprising case into a regression test. The effort goes into designing the representation once; its usefulness extends across those jobs.
Written with help from AI, including editing and prior-art references.