Voice cloning on iPhone: three runtimes, one blind test, and why we stopped

We wanted the DailyVox Twin to answer in your own voice. Not a voice that sounds a bit like you. Your voice, copied from journal audio you already recorded, with no setup work asked of you.

We spent three weeks on it and then stopped. This post is the full record: what we measured, how we measured it, what the numbers were, and the test that ended the project. We are writing it up the way we would write up an experiment, because that is what it was.

What we were trying to build

Since version 1.9, DailyVox can read its answers out loud. It uses a voice that is already on your iPhone. That works, and it needs no setup, but the accent is only approximated. An Indian English speaker gets an Indian English voice, not their voice.

The goal was to replace that with a copy of your real voice, built from thirty seconds of audio you had already recorded in the app. No enrollment screen. No reading 150 sentences aloud.

The memory limit, and why it decides everything

iOS does not let an app use as much memory as it likes. A background process called jetsam watches how much memory each app holds and kills apps that go over a limit. On a 6 GB iPhone, a foreground app gets somewhere around 250 MB before it is at risk.

The number jetsam actually looks at is called phys_footprint. This is not the same as the size of the model file on disk, and it is not the same as the number Xcode shows you in the memory gauge. So we built a small measuring tool inside the app that reads phys_footprint directly, and used that number for every test below.

Everything else in this post is a comparison against that 250 MB line.

Why this particular model

Two kinds of model can make a computer sound like you, and only one of them works for this.

The first kind copies timbre, which is the colour of your voice. Is it deep or light, rough or smooth. Models like OpenVoice do this, and they are small and fast. We tried one early on. The result had our voice colour painted on top of a British speaking rhythm, and it did not sound like the right person.

The reason is that an accent is not mostly timbre. An accent lives in how you shape each sound and which sounds you pick, and those decisions depend on the sounds that came before them. To copy that, a model has to generate speech one small step at a time, with each step looking back at the steps before it. That is called an autoregressive model.

Kyutai Pocket TTS is autoregressive and has 109.5 million parameters, which is small for this class of model. That combination is why it was the candidate. It is also why it is expensive to run, because looking back at previous steps means holding a lot of state in memory.

Method: three ways to run it

The same model can be packaged for a phone in different ways. Each package makes different trade-offs, so we measured all three.

Runtime 1: FluidAudio (Core ML)

An open source Swift package that converts Pocket TTS to Core ML, Apple's own machine learning format. It is Apache 2.0 licensed, supports iOS 17 and later, and installs through Swift Package Manager. It has a working voice cloning function, which matters, because another iOS port we checked ships that function as an empty stub that raises an error when called.

Model files, English pack, on disk:

int8 means the model's numbers are squeezed from 16 bits down to 8 bits each, which roughly halves the size and usually costs a little quality. Only one of the four parts of this model has an int8 version available, so the savings are limited. The largest single file is the conditioning step at 254.3 MB, and it stays at full size.

Runtime 2: sherpa-onnx (ONNX Runtime)

An open source toolkit that runs the model through ONNX Runtime, a cross-platform engine. This was our planned choice, and on paper it deserved to be:

Real-time factor is compute time divided by audio length. 0.15 means one second of computing produces about seven seconds of speech, so this is comfortably fast enough to speak while it thinks.

Runtime 3: an earlier candidate

Before either of these we had measured a different model, chatterbox-turbo, at 953.7 MB. We had also listened to MOSS-TTS-Nano and rejected it by ear, and its published files are 728 MB with no smaller version available. Both were already out on size.

Results: memory

All figures are phys_footprint, measured on a Mac with the same code paths the phone would use. Mac numbers are not phone numbers, and we say more about that under limitations. The 250 MB line is the iPhone budget.

Read the first column carefully. FluidAudio is over budget after simply loading the model, before it has done any work at all.

Our own design document had estimated 685 MB would be about 159 MB. That estimate was wrong by a factor of four, and every estimate in this project has been wrong in the same direction. This is the argument for measuring rather than projecting.

Results: app size

Memory is what happens while the app runs. App size is what every user downloads, including users who never turn the feature on. So we measured that too.

The two code libraries needed come to 63 MB of static archives. That number overstates the cost, because a linker only pulls in the parts you actually call and then discards unused code. To find the true cost we wrote a tiny program that calls only the speech functions, linked it with dead code stripping on, and stripped the symbol table:

LINKED (with -dead_strip): 34.3 MB
AFTER strip:               22.3 MB

So the honest figure is about 22 MB of extra binary.

The app roughly doubles for everyone, and adds 125 MB of storage for anyone who uses the feature.

The flaw in how we had been testing

Here the story stops being about memory.

In late July we had produced a cloned clip, listened to it, and approved it. We called it A92. We compared it against the OpenVoice output, and later against MOSS-TTS-Nano, and A92 sounded more like the right person both times. A92 became the standard. Every runtime after that was judged by whether it matched A92.

Every one of those comparisons put one synthetic clip next to another synthetic clip.

That design can only tell you which clip is less bad. It cannot tell you whether either clip is good enough to ship. We had three weeks of engineering sitting on top of an approval that had never once been compared against a real recording.

When we finally played the newest output next to A92, the honest reaction was that neither one was clearly good. That is when we designed a test that could actually answer the question.

The blind test

Design

A forced choice test. The listener gets two clips saying exactly the same sentence. One is a genuine recording of them. The other is the model imitating them. They must pick the real one. There is no "not sure" option, which is the point: it forces a decision and makes the result countable.

We ran 8 trials.

Controls

The hard part of this kind of test is that the listener can cheat without meaning to. If the real clips are louder, or clearer, or longer, they will pick the real one every time for reasons that have nothing to do with the voice. So each of those clues was removed:

That last row was a real bug in the first version of the test. Every genuine clip was exactly 7.00 seconds long, because that was the window size in the script. A listener could have found the real clip just by noticing that one of them always felt the same length. That would have been a flaw in our tooling being measured as a property of the model.

Making sure the model had not heard the audio before

A cloning model is given a reference clip to copy from. If you then test it on that same audio, you are testing memory rather than ability. So the test clips had to come from audio the model had never been given.

To guarantee that, we took the thirty second reference clip and cross-correlated it against the full source recording to find exactly where it came from. Cross-correlation slides one signal along another and measures how well they line up at each offset. The best match tells you the position.

It sat at exactly 92.0 to 122.0 seconds of a 224.2 second recording. Every genuine clip for trials 3 through 8 was then taken from outside that window, between 12 and 203 seconds.

This also corrected a misunderstanding we had carried for two weeks. The clip was labelled A_92, and we had assumed the 92 meant a 92 second source. It was the start offset. The other variants we had been treating as different clip lengths were simply different thirty second windows of one recording.

Procedure

1. Cut a segment of genuine audio from outside the reference window.
2. Transcribe it with Whisper small.en to get the exact words.
3. Ask the cloned model to say those same words.
4. Match sample rate and loudness on both clips.
5. Assign each to slot A or B in an uneven, non-guessable order and write down the answer key.
6. Listen and choose, writing all answers down before opening the key.

Results: the listening test

8 correct out of 8.

If someone were purely guessing, each trial is a coin flip, so the chance of getting all 8 right is 0.5 multiplied by itself 8 times, which is 1 in 256, or p = 0.0039. In plain terms: a result this clean would happen by luck about 4 times in 1000. The identification was real, not luck.

So the cloned voice is reliably distinguishable from a genuine recording, even on audio the model had never been given, with loudness, audio quality, wording and length all controlled.

It also means the July approval of A92 was a relative judgement. Better than OpenVoice and better than MOSS-TTS-Nano is not the same as good. There had never been a demonstrated standard for anything to reach.

Limitations, honestly stated

A result is only as good as the things that could be wrong with it. Four things could be wrong with this one.

The bar may be stricter than the product needs. "Impossible to tell from a real recording" is the hardest test available. The useful product question is closer to "does this sound more like me than a generic voice", which is easier, and the clone might well pass it. We chose the hard test without first asking which question the product needed answered. The memory numbers close the project regardless, but if memory had been fine, this test would have been harder than the decision deserved.

One listener. Eight trials with one person, who is also the person whose voice it is. Being the owner of the voice probably makes identification easier than it would be for a stranger, so this does not tell you whether your friends could pick it out.

Mac, not iPhone. The memory numbers come from a desktop. macOS has no jetsam pressure, so it never forces the system to release memory the way a phone does. The iPhone figures would likely be lower. They would have to be about four times lower to matter, and nothing suggested that.

Reference length differed between runtimes. FluidAudio's encoder is capped at 10 seconds of reference audio, while sherpa-onnx accepts the full 30. That was a genuine confound in the first comparison, and it is why we did not close the whole project on the FluidAudio result alone.

Two findings we are keeping

Giving the model a written transcript of the reference clip, instead of an empty string, tightened the speaking pace from a spread of 149 to 223 words per minute down to 167 to 199. That is a genuine improvement in output quality. It did not change the verdict.

And sherpa-onnx remains the best runtime we found for this class of model. The plumbing works, it is fast, and it is properly licensed. If a smaller accent-preserving model appears with a sherpa export, a new candidate can be tested in an afternoon without opening Xcode at all.

What changed as a result

The blind test took about an hour, including finding the source recording and writing the loudness-matching script. It answered a question that three separate runtime integrations across three weeks could not.

So the rule now is that any future voice candidate gets the forced choice test against real audio, with the format clues controlled, before the port compiles rather than after it.

Until a model appears that can pass it inside the memory budget, the DailyVox Twin keeps using a voice your phone already has. The accent stays approximate. That is written into the source code as a deliberate stopping point rather than a gap someone should quietly fill later.