What’s the question underneath?
Every so often, a major report puts institutional language to something a lot of people have been feeling without quite having words for it. The new MIT Technology Review Insights report, Privacy-Led UX in the AI Era, produced in partnership with Usercentrics, is one of those moments.1 Its core finding, that privacy-led UX is becoming a prerequisite for AI adoption rather than a competitive edge, names a shift that founders and product teams in privacy-adjacent categories have been feeling in their churn, their support tickets, and their user research for a couple of years now.
Before we get to what the report says, it’s worth sitting for a moment with what it’s responding to.
What is the thing people are feeling?
It looks like this. The woman who keeps two cycle-tracking apps installed and trusts neither of them. The user who stops logging symptoms the week things get personal. The hot flash recorded in a paper journal because the app feels like the wrong place to put it. The friend who tells you she loves the wellness product you built and then quietly opens her phone to show you she’s been keeping the real notes in her camera roll.
None of these people can tell you, in technical terms, what they’re protecting themselves from. They don’t have the vocabulary for data minimization or training-set boundaries or schema-level retention policy. What they have is an unsettled feeling: a pre-rational sense that handing this particular kind of data to this particular kind of system is wrong, even when they can’t quite say why.
That feeling is the market signal the report puts numbers to. 77% of global consumers say they don’t fully understand how their data is collected and used. 82% have abandoned a brand over privacy concerns. 59% are uncomfortable with their data being used to train AI models, and 81% are concerned about AI’s access to their data, even while 32% use AI daily. Perhaps most usefully: 73% of US consumers say they’d share more data if they had genuine visibility and control over it.
The report is rigorous, and its TRUST framework (Translate, Reduce, Unify, Secure, Track) is sound. It treats consent as an ongoing relationship rather than a launch artifact: plain language at the moment a user actually needs it, equal visual weight for accept and decline, consistency across touchpoints, end-to-end governance of data flows, measurement that goes past opt-in rate. It lives, deliberately, at the layer where most companies can act on it. The design layer.
What I want to do here is gently turn the report on its side and ask about the layer underneath.
A different question
The layer underneath isn’t a different framework. It’s a different question.
The question the report’s framework answers is roughly: how do we design consent so it actually serves users? That’s a worthwhile question and a hard one, and the framework does it justice.
The question underneath is one most products in the AI era haven’t sat with for very long: are the things people would want to consent to or refuse even happening in our system in the first place?
That can sound like a semantic distinction. I don’t think it is. It opens up a fork in the road, and the two paths look almost identical from outside for the first year or two.
One path leads to better consent dialogs. Clearer toggles. Onboarding that doesn’t hide where data flows. A “manage your data” page that actually lets you manage your data. Permission prompts that name the specific use rather than the category. Each of these is a real improvement, and the discipline of designing them well puts useful pressure on the underlying decisions. None of them are nothing.
But none of them change what the system does.
The data still flows where it flows. The model still trains on what it trains on. The third-party SDKs still phone home with what they phone home with. The UX is the place a person encounters the system, but it isn’t the system. And in AI products, the gap between those two has become large enough to drive a company through.
The other path is the one the report’s finding points at if you take it all the way. It treats privacy-led UX as the visible surface of a system designed so that the things users would object to are structurally not happening. Not “disclosed clearly.” Not “managed responsibly.” Not happening.
That’s a different commitment, and it changes things downstream that policy can’t reach. What you collect. Where it lives. Who can ask for it. What survives a server compromise, a subpoena, a board change, an acquisition. What questions a Series B investor can ask, and what answers exist because the architecture made them exist rather than because someone wrote them in a doc.
Here’s a useful way to feel out which path a company is actually on. Ask:
I’d offer that less as a verdict and more as a diagnostic. Most teams I talk to assume they’re on the second path, because they care about the second path. The architecture tends to have its own opinions about that.
Where the difference becomes visible
For a while, the two paths produce similar artifacts. Both yield thoughtful onboarding flows. Both yield privacy pages that read well. The difference shows up at three moments.
When a breach happens. The first company explains how committed it is to user privacy and announces credit monitoring. The second company explains that the data wasn’t there to take.
When a regulatory shift happens. The first company adjusts its policy and adds three new consent prompts. The second company doesn’t have to adjust, because the regulated behavior wasn’t possible in the system.
When the acquisition happens. The first company’s user data becomes a balance-sheet asset, and the privacy policy is rewritten by the acquirer. The second company’s user data isn’t an asset, because it isn’t sitting in a database to be sold.
Notice what these three moments have in common. They’re all places where the gap between what the policy promised and what the architecture allowed gets exposed. They’re also the moments users feel hardest. Most people I’ve talked to about this can name the brand that disappointed them at one of these three crossings. It’s not abstract.
This is also where the report’s framing around agentic AI lands hardest. As the report notes, when AI systems begin acting on a user’s behalf, the traditional consent moment may never occur. Agent-generated data flows require infrastructure that goes well beyond the cookie banner. The TRUST framework asks where consent should live. The architectural question is whether the data exists in a form that consent could meaningfully cover at all.
What it actually costs
So why doesn’t everyone build this way? Two real reasons. One of them is the one most of the industry hasn’t reckoned with out loud yet.
The engineering cost is real but bounded. Data minimization at the schema level, least authority across the request path, ephemeral processing in encrypted environments rather than always-on retention. These are well-understood patterns at this point. They work. They cost more operational complexity than the path of least resistance, which is “call a hosted model API and store the response in your own database.” But the techniques themselves are not exotic.
The business model cost is the one that gets quieter at conferences.
If the data isn’t centralized, it can’t be the asset. If the inference runs in environments that hold nothing after the computation completes, the retained corpus can’t be the moat. If the patterns surface against a user’s own history rather than the aggregated behavioral exhaust of millions of strangers, the network effect doesn’t compound through extraction. None of that is impossible to monetize. It’s a different shape of monetization, with different unit economics, and it requires a different kind of story to investors than “we’re building a data network.”
That’s the part I’d love to see more of the field talk about openly. Privacy-led UX is relatively cheap. Privacy-led architecture is a strategic choice that closes some doors permanently, the doors a lot of the venture playbook has been quietly assuming would stay open.
What the MIT/Usercentrics report makes harder to ignore, by virtue of where the consumer numbers are pointing, is that in this category those doors were going to close anyway. The question isn’t really whether to walk through them. It’s whether to walk through on purpose, while there’s still time to design what’s on the other side, or to be pushed through later under regulatory pressure.
Where I’m writing from
I should say where I sit in this. I’ve been building privacy-architected wellness software for women in midlife, and the architecture is the product, not a wrapper around it.
What that looks like in practice: we don’t run “everything on-device.” That’s a different design point with its own honest trade-offs, and it’s not the one we made. What we built is server-side pattern analysis that runs in fully encrypted ephemeral instances, spun up to compute against an individual woman’s data on demand or on a schedule, and then destroyed. The instance holds her data long enough to do the work and no longer. There is no retained corpus, no aggregated body-data warehouse, no per-user record sitting in a database that could be queried, breached, subpoenaed, or sold.
That’s more infrastructure than calling a hosted model API. It is meaningfully more expensive than the alternative. And it produces a privacy guarantee that doesn’t depend on a privacy policy, because the system literally cannot do the things a privacy policy would otherwise have to promise it won’t do.
I’m not offering this as the only valid answer. There are honest trade-offs in any architecture, and there are categories where the surface answer is genuinely sufficient. Wellness data for women in midlife is not, in my view, one of them. The unsettled feeling I described at the top is too loud, and the asymmetry between what this kind of data reveals and what a user can recover if it leaks is too steep.
What I’d ask anyone building in an AI-saturated, sensitive-data category to sit with, coming out of this report, is the question underneath. Not just “how do we make our consent flow better.” Better consent flows are good. But: what is actually happening inside the system, and would the user, fully informed, consent to it? If the answer is no, the design layer can soften it. It can’t solve it. The architecture is where that question gets answered structurally, or doesn’t.
The report draws the surface beautifully. The architecture is the thing the surface is reflecting back.
Footnotes
-
An accompanying article is published at technologyreview.com. Fair warning: the cookie consent flow you’ll meet there is somewhat unreasonable given the topic of the article. ↩
Continue reading
- Start hereA Different Kind of VesselWhat if there were a privacy-first federated AI model for women's wellness — and what if we, collectively in femtech, owned it?
- Who Wants Neurological Data on Midlife Women?Recent research has moved midlife symptom data from the wellness category into the neurological one. Almost no one is talking about what that means for the data women have already generated, or for where they generate it next.
- There Is No Privacy Paradox.A reaction to recent ChatGPT-influencer research. The privacy paradox isn't a paradox — it's the sound of an architecture working exactly as designed.
