Measuring a model
of one.
A model built to be private cannot be watched in production. With no egress there is no dashboard, no population A/B test, no confusion matrix — and its single user is the only witness. So it has to be evaluated on the device, before it ships, or its fidelity claims cannot be checked at all.
This page is the programme built around that problem — and an open invitation. It is one person's work so far, which is its main weakness. If any of it is close to what you work on, I would rather do it with you than alone: co-authorship, joint supervision, a student project, or just an argument about whether the method holds. How to start a conversation is at the bottom.
The paper.
methodologyMeasuring a Model of One: A Falsifiable, On-Device Evaluation Methodology for Private Personal Models
Karthikeyan NG · independent researcher
The problem it addresses
Privacy and observability are in direct tension, and almost nobody states it. The moment a model is built so that nothing leaves the device, every ordinary method of knowing whether it works stops being available: no server-side logs, no population A/B test, no held-out set you can inspect after the fact. The user is the only witness, and the user cannot audit a model of themselves — they are the thing being modelled.
The usual answer is to give up the privacy. This paper's answer is to move the evaluation: run it on the device, before shipping, against baselines strict enough that the result can come back negative.
What the method actually requires
- Every metric is lift over an explicit baseline. Not accuracy — accuracy on a skewed personal corpus flatters everything. Lift over the do-nothing answer, stated with the baseline next to it.
- A permutation null and five negative controls. Destroy the link between label and text and the metric must collapse to chance. If it does not, the metric was measuring the corpus, not the model.
- Staged legs. Synthetic pipeline checks gate what is worth testing on humans; population-scale human corpora come next; a full-protocol single subject last. Each leg can stop the next.
- The instrument ships with the claim. A fidelity claim that cannot be re-run by the person it is about is a marketing number.
How it is laid out
- 1 Introduction
- 2 The Personal Model — what is being claimed, and what would falsify it
- 3 The substrate, and its drift
- 4 DailyVox: an existence proof — the shipping testbed
- 5 What is not yet shown — the limits, stated before the results
- 6 The instrument, and what it measured
- 7 Discussion and limitations
- 8 Ethics and privacy
- 9 Conclusion
A preprint, not yet peer-reviewed, and hosted here rather than on a repository — so treat it as a working document. Corrections are welcome and will be credited; if you think the method is wrong, that is the most useful email this page could produce.
Karthikeyan NG. "Measuring a Model of One: A Falsifiable,
On-Device Evaluation Methodology for Private Personal Models."
Preprint, July 2026.
https://getdailyvox.com/paper/measuring-a-model-of-one.pdfWhat it found.
including the failuresLeading with the losses, because an instrument that cannot catch its own maker's model is not an instrument.
- The deployed Twin failed first contact with a real diary. On the author's own journal, keyword-derived traits scored below a do-nothing baseline — lift −25.2%. The cause was a written-versus-spoken register shift that synthetic validation could not have exposed. The app was shipping; the harness said it should not have been trusted.
- Some layers carry signal, others do not. Across 1,473 writer-self-labelled entries and 15,809 held-out utterances: OS valence reads people at r = +0.59. The keyword emotion layer scores 43.5% against an 85.0% majority baseline — worse than always guessing the common answer.
- The repair is named and not yet takeable. A learned emotion probe beats the keyword layer on transfer to real diaries (50.1% vs 43.8%), but trains on research-licence-only data. The first licence-clean retrain fails the deployment gate by over-firing on neutral text.
- An honest null. A 105 KB openness head reads 1,307 writers' self-reported openness at r = +0.13 — statistically real, practically worthless, and bounded by how much text a person writes rather than by the head.
Three of those four are the product losing. That is the result.
Open problems.
pick oneStated as problems rather than plans, with what a good contribution to each would actually look like. None of them require access to anyone's diary.
A licence-clean emotion head that survives the gate
ML · tractableThe probe that beats the shipped keyword layer trains on data whose licence forbids commercial use. The first clean retrain over-fires on neutral text, which on a journal is the majority class and the one that matters — a model that calls an ordinary Tuesday "sad" is worse than no model.
A contribution looks like: a head trained only on permissively-licensed corpora that beats 43.8% transfer accuracy and holds its neutral precision, under about 200 KB, running on the phone's own embeddings.
Long-horizon memory that can be scored
eval · openRetrieval over a week is not memory. The hard question is whether a model connected two entries months apart because the connection is there, or because it confabulated one — and existing long-context benchmarks are built for documents, not for one person's decade.
A contribution looks like: a benchmark over a personal longitudinal corpus that distinguishes a real connection from a plausible one, with a null that collapses when the timeline is shuffled.
Code-mixed, non-Anglo on-device NLP
NLP · regionalThe app's name recogniser is model-free — a heuristic over capitalisation evidence — and it finds 100% of the non-Anglo person names in the test corpus where a learned model finds 97.7%, precisely because it has no prior about what a name looks like. That result points at a gap: sentiment and entity work for code-mixed diaries, on-device, without a cloud translation hop.
A contribution looks like: a Tamil-English (or any code-mixed pair) evaluation set of journal-register text, and a sentiment or entity approach that runs in a phone's memory budget.
Aggregating evidence without egress
systems · openIf a product's whole claim is that nothing leaves the device, how do you ever learn anything across devices? Federated approaches assume a coordinating server this architecture refuses to have. There may be no good answer; saying so rigorously would also be a result.
A contribution looks like: a protocol where a participant can contribute an aggregate measurement without a server ever holding their text — or a clear argument that the trade cannot be made.
The consented cohort
blocked · needs peopleEvery number above comes from one author's diary plus public corpora. A cohort is the gate between "it works for me" and a finding. The protocol is written, the consent form exists, the harness is ready — and it is stopped, because one person cannot recruit, review and consent a cohort in evenings.
A contribution looks like: an institution with an ethics process, or a supervised group willing to run the protocol. This is the one that cannot be solved by writing code.
Where it goes after that.
longer horizonThe five above are close enough to attempt. These are the directions behind them — further out, less defined, and the ones I would most want to argue about with someone who reads this literature properly. Several may turn out to be bad ideas, which is itself worth establishing.
Memory that has a middle
The Twin currently goes from raw entries to a statistical persona with nothing between. There is no layer of atoms ("she started therapy in March") and no layer of scenarios ("the year of the move"), so every question re-derives everything at query time. Building that middle is an architecture question and an evaluation question at once: what is the right granularity for a memory of a person, and how would you know you had it?
Personality depth without fine-tuning
Weight-level personalisation on a phone is not practical yet. Persona conditioning — steering a shared model with a measured profile and the person's own phrasings — might get most of the way. The open question is whether a conditioned model is meaningfully more that person than a well-retrieved one, or whether retrieval is doing all the work and the conditioning is theatre.
Drift: a person changes, does the model?
A twenty-year diary is not one person. Someone at 22 and the same person at 42 are different subjects, and a model averaged over both describes neither. Detecting the boundaries, and deciding whether a Personal Model should track the current self or remember the former one, is unsolved and probably the most interesting thing on this page.
A mirror that can disagree
Personal models have an obvious failure mode: agreeing with you. A Twin that reflects a distorted self-image back approvingly is worse than no Twin, particularly in this domain. Measuring sycophancy in a system whose ground truth is the user's own words — where the user is both subject and judge — is a genuinely hard evaluation problem.
Ambient signals, and their ethics
Photos, music, movement and sleep are on the phone already, and each carries mood signal. Every one of them also widens what the model knows past what the person chose to say. The research question and the ethics question are the same question, and the current answer — a review-and-discard queue, nothing learned without being seen — is a design choice that has never been tested against alternatives.
Cross-lingual personal models
Most people who journal in more than one language switch within a thought, not between entries. A personal model that treats the switch as noise loses the part where the switch is the signal. This runs alongside the code-mixed work in §03 but is the deeper version of it.
What data exists.
and what does not- Public corpora, licence-clean. The population-scale numbers — 1,473 self-labelled entries, 15,809 held-out utterances, 1,307 writers' openness self-reports — come from published datasets. Reproducible.
- Research-licence-only corpora. The emotion probe's training data. Usable for a paper, not for a shipped model. This constraint is the whole of open problem 01.
- One author's diary. Twenty years, and the only longitudinal personal corpus in the programme. Not shareable, and its non-shareability is the point of the product.
- No user data. At all. The app has no telemetry and no server. There is no dataset of DailyVox users' journals, there never has been, and building one would require every participant to individually choose to send an export.
Let's do it together.
open invitationEverything above is one person's work, and that is the honest limitation of all of it. A single author with a single diary can build an instrument and can catch his own model failing — but cannot produce a finding, cannot peer-review himself, and cannot see the parts of the literature he has not read. I would genuinely rather do this with other people.
There is no funding, no institutional sponsor and nothing to pay anyone, so this is not a job offer. What there is:
- A shipping testbed. A real app, on the App Store, with real users — not a notebook and a Kaggle set.
- An instrument that already exists. TwinEval runs, with baselines, nulls and negative controls in place. You would not be starting from an empty repository.
- Problems that are actually open. Not homework with a known answer at the back.
- Authorship, stated up front. If your work is in a paper, your name is on it. That gets agreed in writing before anyone starts, not after.
- An engineer who will do the engineering. If the interesting part for you is the method, the shipping and the plumbing are mine.
Shapes this could take
Any of §03 or §04 is a paper with two names on it. The methodology work in particular needs someone who will attack it.
Several of these are the right size for a final-year or master's project, with a shipping app as the testbed and a real result at the end.
The one thing that cannot be solved by code. An institution with an ethics process unblocks the entire bottom of the ladder.
If the method has a hole in it, that is worth more than a collaboration and takes one email. Corrections get credited.
Reasonable next steps: read the paper, look at the architecture, take the source and research kit — then write and say which part you would want.
On ethics, so it is settled before anyone asks: work involving participants runs on informed consent, no compensation, data kept local to the participant's own machine, deletion honoured immediately, withdrawal honoured any time before publication, and aggregate numerics only in anything published. Those terms predate anyone asking and are not negotiable.