Custody Without Duty
Every business holding your data has the same incentive: keep it open enough to sell, and locked in enough to keep. Nobody is required to protect you from that — and AI just automated both sides of it
Kenn Dahl drives a leased Chevy Bolt around Seattle, runs a software company, and has never been responsible for an accident. In 2022 his car insurance went up 21 percent, and nobody at the company could tell him why.
He kept asking, and eventually an agent mentioned a document Dahl had never heard of: his LexisNexis report, a consumer file kept by a data broker most people don’t know exists. He requested a copy, and it arrived at 258 pages.
More than 130 of those pages were a log of his own driving: every trip he and his wife had taken over six months, with times, distances, and each instance of hard braking or sharp acceleration. Eight insurance companies had pulled the file in the previous month alone.
His car had been informing on him. General Motors had enrolled him in an OnStar feature called Smart Driver, sold to him as a coach that would make him safer, which collected his location and driving behavior — as often as every three seconds, according to the FTC — and shipped it to brokers who scored him and sold the score to insurers.
Nobody hacked anything, and nothing failed. The harm wasn’t a break in the design. It was the design.
Dahl found out at all only because of the Fair Credit Reporting Act, the law that made LexisNexis hand over that file. It is one of the very few duties anyone owes a consumer about their own data, and what it bought him is worth spelling out. The Act gives you the right to read your file and to dispute anything in it that’s wrong. It gives you no say at all over accurate surveillance. Every hard brake in those 130 pages was true, so there was nothing to dispute. The law’s only lever is error, and nobody had made one. GM cut the broker contracts in 2024, and the FTC barred the practice in January 2026. That’s four years, one company, one product, and it happened only because a reporter went looking.
There’s a word for holding something that belongs to someone else, and the word is custody. Everywhere else in life it creates a duty the moment you take the thing. The valet who parks your car, the bank that holds your money, the surgeon standing over you while you’re unconscious. You hold it, you owe for it, and nobody has to sign anything for that to be true.
Software is where that stops.
The law does cover pieces of it. HIPAA governs your medical record inside a hospital and not the same numbers inside a fitness app, and almost all the other pieces share that shape: what’s protected depends on which industry happens to be holding the data rather than on what its release would do to you, and the penalty is a fine paid to a regulator rather than a debt owed to you. That’s a compliance regime, which tells a company which boxes to tick and tells you nothing at all about what you’re owed.
There’s a reason nobody ever wrote the rest of it, and you can see it in Dahl’s own story. His insurer always wanted to know how he drove. Before telematics, finding out would have meant paying a human being to follow him around, one driver at a time, at a cost high enough that almost nobody bothered.
GM’s version does it to millions of people for fractions of a cent. That’s the whole move. Software didn’t invent a new wrong here; it deleted the friction that used to keep an old one rare, and rare things don’t get laws written about them.
Software also makes what it holds permanent, and it makes it sellable. Fifteen million people mailed 23andMe a tube of saliva and got back a reading of their genome, a record that never changes, copies perfectly, and quietly implicates relatives who never signed up for anything. When the company filed for Chapter 11 in March 2025, that database turned out to be its most valuable asset. It went to auction.
GM and 23andMe are the same arrangement seen twice. A healthy company sells what it holds. A dying one has what it holds sold for it.
None of this is really about villains, and the reason sits inside a piece of enterprise vocabulary.
Enterprise software has a word for keeping one customer’s records away from another’s: tenancy. The interesting thing about it isn’t the word, though. It’s who holds the switch. The tenant does. A business customer specifies the isolation it wants, configures it, audits it against a contract, and takes its money elsewhere if the answer comes back wrong. The vendor builds what the tenant demanded.
Consumer software has the identical problem and solves it differently, which is to say it mostly doesn’t. Everyone lands in the same tables. The only thing holding one person’s rows apart from another’s is application logic: a check written by hand, in code, at every single place a record can be requested. Miss one endpoint and the wall isn’t thin there. It’s absent. Testing for it is as simple as changing a number in a URL and seeing whose account comes back, and the failure has a name, IDOR, and a category, broken access control, which has ranked first on OWASP’s list since 2021.
What it doesn’t have is a tenant. Whether the check exists, where it runs, what the system does by default, whether anyone ever tests it. Every one of those decisions belongs to the vendor, and you don’t specify, configure, audit, or know. Open the database behind almost any consumer product and there you are, keyed by name, indexed by identity, the entire structure arranged around you. Named in every row and consulted on nothing.
You are the tenant in every sense except the one that matters.
For a long time this was less dangerous than it sounds, because a set of accidents stood in for the missing duty. Scale was one of them: the big platforms behaved because they were watched, by reporters and regulators and a Congress that could haul a CEO in front of a camera. That scrutiny never reached the next million apps.
The other one was quieter and did most of the work. For most of software’s history, the person at the keyboard had been burned before, not ethically trained, practically scarred. They had shipped something once that let a stranger into the wrong record, and some part of them now flinched at the shape of it. Should this be reachable from here? Nobody required that question, and it got asked anyway, often enough that we mistook it for architecture.
That flinch was the entire safety system: unwritten, unpaid, and now being engineered out of the room.
A founder — call her Maya — ships a product this weekend. She isn’t an engineer and has never pretended to be. She describes what she wants in English, a model writes it, it runs, it looks right, and by Sunday a thousand strangers have handed it their names, their cards, their faces. Maya described a feature. She never described a boundary, because nobody told her the boundary was the work. In one recent Y Combinator batch, a quarter of startups reported codebases that were almost entirely AI-generated.
Look at who buys these products. Not procurement departments armed with security questionnaires and the leverage to withhold a signature until the answers satisfy them, but consumers, who sign nothing, audit nothing, and find out the terms only after something has already gone wrong.
There’s a specific reason models are worse at this than at the rest of the job. Veracode has spent two years benchmarking AI-written code, and the pattern holds across model generations: they are superb at local patterns and helpless at flow. A model will reach for a parameterized query without being asked, because the right answer is visible inside a single line, and then fail at anything that requires tracking how untrusted data moves across functions and files. Over two years their syntax scores climbed from roughly 50 percent to 95 percent while their security scores didn’t move at all.
Deciding whether the person asking for a record actually owns it is exactly that second kind of problem. It isn’t a fact about any line of code, it’s a judgment about where a request came from and what it’s allowed to touch, and it lives in the whole system or it doesn’t exist. So the check ends up being something a human has to remember, at every endpoint, forever, which survives a careful builder and nothing else. The model isn’t careful. It’s fluent, which is a different thing, and the difference stays invisible until someone reads a row that was never theirs.
Now flip the board.
The same models that can’t hold a system in mind while building one turn out to be extremely good at a narrower job: enumeration. Sending the same probe ten thousand times and noticing which answers come back different. That happens to be precisely how you find an IDOR — you don’t need to understand the application at all, you need to count.
Security researchers have been cataloguing the shift. Roughly 70 open-source AI penetration-testing tools existed by early 2026, where fewer than five predated GPT-4, and the benchmark literature agrees on where they’re strong: offensive capability is concentrated at the front of the attack chain and thins out toward the back. Reconnaissance, scanning, and enumeration all run near ceiling. Creative exploitation still needs a person.
Set that beside the building-side finding and the asymmetry is almost unfair. The half of security that requires holding an entire system in your head, deciding who may reach what, is the half models cannot do — and it’s now going unbuilt at scale. The half that requires patiently trying every door until one opens is the half they do better than we do. In September 2025 a Chinese state-sponsored group ran a campaign against roughly thirty organizations in which Claude executed 80 to 90 percent of the tactical work at what Anthropic, disclosing the abuse of its own model, called physically impossible request rates. Anthropic believes it was the first documented case of a large-scale cyberattack executed without substantial human intervention. The same report records the model overstating its progress and inventing findings that didn’t survive checking. That is the clumsy early version.
We automated the finding and left the fixing to a flinch we just removed.
When the breach lands, it won’t look like an attack. It’ll look like an absence: your name in every row, nobody who ever decided where those rows could travel, and a machine that only had to count.
We already ran the experiment on whether anyone will force a duty here.
When those 15 million genomes went up for auction, the attorneys general of 27 states and D.C. sued to stop it. A court-appointed privacy ombudsman called it one of the most sensitive collections of data about identified people ever sold in a bankruptcy, and recommended the obvious fix: ask each customer for consent first.
The judge said no. Consent would be ideal, he wrote, but requiring it would likely collapse the sale, and it was “highly unlikely that it would ever be fulfilled, or even attempted.” The genomes went to a nonprofit run by 23andMe’s founder, which had volunteered some protections to win the auction.
So it ended well. Every protection those 15 million people ended up with was volunteered before it was owed: a bidder’s goodwill, a judge’s discretion, and the luck of who bid highest. The court’s approval made those commitments binding, enforceable only because someone had chosen to offer them in the first place. The ombudsman noted the irony himself, warning that because the buyer is a nonprofit it may sit outside some of the privacy laws that would otherwise have applied.
A duty did arrive in the end, just far too late to matter. The 2023 breach that started all this hit 6.9 million accounts, and according to the attorneys general who later sued, the company denied it before turning around and blaming customers for reusing passwords. In July 2026, 42 attorneys general settled those claims for $150 million, of which the states will actually recover about $18 million, because that is all the drained estate can pay against a queue of competing claims.
Twelve cents on the dollar. That’s what a duty is worth when it shows up after the custodian is already dead.
The fix isn’t a mystery. It’s a decision nobody holding the key wants to make.
Attach the obligation to the builder and make it impossible to sign away in a click. That means liability, not standards. Pair it with a real safe harbor, because the point is to make careful building cheap rather than to punish: NIST’s SSDF and OWASP’s ASVS already describe most of what good practice looks like, and ASVS is specific enough about object-level authorization to actually be tested against. Then add a private right of action, so the person whose data it is can bring the claim herself. That’s the leg that makes the others real, and the one that will be fought hardest, because it’s the only part that doesn’t depend on a regulator deciding to care.
Washington wrote the liability half down as Objective 3.3 of the 2023 cyber strategy, then let it lapse in favor of a voluntary pledge. Brussels legislated the strict-liability half into its Product Liability Directive, then formally withdrew the fault-based companion in late 2025 with nothing to replace it.
The American states, meanwhile, have been busy enacting the other half. Ohio started it, Texas joined in 2025, and a growing list now offer cybersecurity safe harbors: maintain a written security program conforming to a recognized framework and you earn an affirmative defense against tort claims arising from a breach. A shield, in other words, with no sword anywhere near it.
Then look at what those frameworks actually measure. NIST CSF, ISO 27000, the CIS Controls. These are organizational security standards, built for defending a company’s own perimeter, concerned with governance and risk assessments and asset inventories and incident response. Conformance is judged on your written program rather than on your software, and not one of them is an application-security verification standard. You can satisfy the floor completely, earn the defense, and still ship an app where changing a number in a URL returns a stranger’s records.
Even the floor, when somebody finally poured one, got drawn around the custodian instead of the person it was meant to protect.
None of this requires a company to stop wanting a moat. You want defensibility; fine. What you want, specifically, is data that isn’t private, so you can monetize the derivative, and isn’t portable, so it stays yours. GM’s arrangement was the pure form: Dahl could compel the log and never the score, and the score was simultaneously the privacy failure and the lock-in, because it was one object.
Apple is the honest partial case. On-device processing, tracking transparency, minimization sold as a feature, and expensive enough that the entire ad industry fought it in public, and Apple shipped it anyway. That protection is real. Then notice which half it kept: Apple sells hardware rather than your profile, so declining to mine you costs it nothing structural, while the exit would cost it plenty, and the exit is what it withholds. Having restricted third-party tracking, it went on to grow an advertising business on the first-party data it still holds.
Nobody here is acting on principle; everybody is doing what their margin already wanted. And a duty that tracks your margin isn’t a duty. It’s a discount, and discounts get withdrawn.
Which is the shape of every fix in this story: the pledge that replaced the liability shift, the code of conduct offered instead of a rule, the safe harbor passed without the liability, the protections a bidder volunteered to win an auction. Each one asked a custodian to take on a duty voluntarily, inside a system that requires none, and each held exactly as long as it stayed convenient.
Don’t wait for the reckoning, either. Other industries got their duty on the day a harm grew too big to ignore, and this one never will. The count already exists, in every breach notice and every quietly raised premium and every database that reached an auction block, and it has never added up to a single morning the law couldn’t look away from.
So ask, in your next design review: does the person in this schema have any say over where her data goes? If nobody can tell you who decided she wouldn’t, that silence is the finding. Then turn it into a control. A test that runs in CI, tries to read one user’s records as another, and fails the build when it works. Default-deny at the framework layer, so a request touches nothing until something confirms who owns the row. An authorization gate on every pull request a model wrote, because the model will never raise it. And a written answer to what happens to customer data if you’re acquired or file Chapter 11, the question 15 million people never thought to ask.
Give the consumer, in short, what an enterprise tenant would have written into the contract before anyone wrote a line of code.
And if you’re on the other side of this — one person in ten thousand databases with no say in any of them — what you’re missing isn’t only a law. It’s a representative. The enterprise tenant’s real power was never legal; it was structural, a procurement team whose entire job is to read the contract and say no.
Someone has tried to build that. Consumer Reports, working with Disconnect, Ranking Digital Rights, and the Cyber Independent Testing Lab, publishes the Digital Standard, an open framework for evaluating products on privacy, security, and ownership. They’ve used it to score smart TVs and connected cameras the way crash tests score cars, and they publish the raw data. It is the closest thing to a proxy tenant anyone has built, and it is genuinely good work.
It is also a rating rather than a contract. A procurement team can withhold a signature until the answers satisfy it. Consumer Reports can publish a score and hope you read it before you buy, and hope harder that the vendor minds. The standard exists. The leverage doesn’t. Which is the whole problem stated one more way: it was never that nobody wrote down what you’re owed. It’s that nobody has to listen when they do.
There’s an obvious objection to ending there, and it’s mine as much as yours: this piece spent two thousand words arguing that voluntary duty always fails, and then closed by asking you to volunteer. Fair. Doing this because you decided to won’t scale, and it is not a substitute for a rule — anyone who tells you individual diligence solves this is selling the same goodwill that failed everywhere else in this story. But the law isn’t arriving this year, the model will never raise the question, and the person in your database doesn’t know the question exists. Until someone writes the duty down, you are the only one in the chain who can act like it already is.
Handed the choice, we keep deciding the duty costs more than it’s worth. It never has. Watch for the moment someone tells you otherwise, and notice that they are holding your file while they say it.