hckrnws

> "can you really trust accounting software not to make mistakes? Won’t there be edge-cases in mortgage underwriting that software might miss, but an experienced underwriter would catch?" The proof is in the pudding: the world runs on software now.

Indeed.

https://en.wikipedia.org/wiki/Post_office_scandal

I worked as a PM for a startup that makes software to help review appraisals, and there's absolutely no question that stuff gets missed by humans in appraisal review as well as the rest of the underwriting process.

Every organization is built around tradeoffs between time (which is very much money here) and quality. You route appraisals to people of different skill levels depending on the likelihood of issues and the likelihood that those issues matter. With appraisals, for example, if buyer's putting down 40% where only 20% is required, you check only the super critical stuff, because even if you're meaningfully off on the appraised value it doesn't actually matter that much.

There are a ton of these tradeoffs of whether you should review something as well as whether you should have your most senior people review it. On top of that, reviewers are judged on speed, so naturally they're making tradeoffs of accuracy vs. time themselves on each review.

Using AI means you're able to review things that wouldn't even have been looked at otherwise, because there's no longer a tradeoff of time/cost vs. accuracy. Even the world's best reviewer can't catch things they're not looking at.

See also the Australia Robodebt scheme, which also claimed many lives: https://en.wikipedia.org/wiki/Robodebt_scheme

Wow, those two articles are as Kafka-esque as it gets. What a nightmare it must have been for the people caught up in them.

Why did they defend Horizon though? The Wiki article offers a POV from someone who was functionally powerless:

> We have to be careful, that we are not creating a cottage industry that damages the brand and makes clients like the DWP and the DVLA think twice. The DWP would not have re-awarded the Post Office card account contract, which pays out £18 billion a year, in the last month if they thought for a minute that this computer system was not reliable

I know that's something that someone said, but is it true? So what if a lot of people say that? Nobody knows who or what leads to sales or not sales. If sales were all that mattered, they wouldn't do the IT upgrade at all.

People use shitty software all the time.

> The new Horizon project became the largest non-military IT contract in Europe.

Also... really doubt that is true.

The Horizon IT report's first volume "will focus on redress (compensation) and the human impact of the Horizon scandal." Okay. But why did people feel so strongly about the technology in the first place? Who gives a fuck about bugs?

>Another classic detail of insurance markets is that insurers need to diversify the risks that they underwrite: for example, if you provide flood insurance, then you wouldn’t want to write all your policies in a single town by the river: when one house gets flooded by a storm, chances are that all the houses get flooded, and you go out of business. That’s concentration of risk, and insurers strive to avoid it.

If I have insured a whole population by the river, then I'm heavily incentivized to sell an additional insurance policy to someone else by that same river, after all, if there is a flood, I cannot become more bankrupt, and in the other case, I collect one more premium.

Not really. Insurance is heavily regulated. The insurer would be required to buy more reinsurance to cover that additional risk, until the regulator was satisfied.

>insurers strive to avoid it.

This is not true, and nothing you have said contradicts my argument to that effect.

Gave a Waymo engineer a ride in my FSD Tesla around the Bay recently. The conversation was very illuminating. It seems to me there should be pretty clear corollaries in more traditional engineering markets. Does Black and Veatch take out E&O insurance? Did John McShain take out E&O for the Pentagon? Did Tishman take out E&O on the World Trade Center? Did the University of California take out E&O on the Manhattan Project? Does Boeing or Lockheed take out E&O on aircraft?

> Did the University of California take out E&O on the Manhattan Project?

Interestingly, see [1] for

“a teletype from General Groves to Oppenheimer from February 1944, instructing the latter as to what to tell Underhill [UC’s secretary and finance officer] about the hazards to be insured against at an unspecified site”

[1]: https://blog.nuclearsecrecy.com/2012/03/28/weekly-document-i...

> The conversation was very illuminating

this sounds very interesting! any more details?

I don't think the adverse selection problem is anywhere near as serious as the author seems to think--insurance companies aren't selling superior knowledge of their client's situation--they're selling against their client's massively different risk appetite.

I expect home insurance to cost more than it pays out (both in median and mean terms) but I take the negative-value deal to protect against rare financially ruinous outcomes.

Quality underwriting and minimizing adverse selection gives an insurance company a massive advantage over competitor insurance companies but it doesn't make or break the market its own.

I'm also not sold on model provider diversity being the measure of risk diversity-surely most of the risk is coming from application errors and not failures of "safety" tuning of models (which are mostly about preventing LLMs from saying things you wouldn't want in the newspaper--I assume AI E&O isn't interested in ensuring reputation risk)

Ultimately this will be like everything else in law (insurance is law by other means) and will be a product of court rulings which means it’s a whack-a-mole game for lenient jurisdictions (like money laundering).

The jurisdiction that hears the case in whatever “justice” system hears it, will set the precedent for all others to reference, based on their alignment with the jurisdiction that uses state power to enforce a resolution.

I expect people will host or make remotely available systems which fall outside of the acceptable limits for whatever regional jurisdiction has their laws.

As usual, pirates and the powerful will steer around those.

Real-time bidding a la online ad impressions.

{Task, model, coverage} --> bid.

It can be circular Ai with insurer Ai doing the evaluation and bidding.

It wouldn't be too hard to set up a marketplace like that, but how do you know your bot is good enough to not accidentally bankrupt the business it's bidding on behalf of? Ads have this wonderful property that individual ads are pretty cheap. If you mis-bid to show an ad to someone not interested it won't cost you very much. Don't do it too often of course, but a few mistakes here and there are fine. In addition, they are bid on and delivered within a second or so at most.

Insurance is very different. Nobody is looking to insure the unit test they vibe coded late Friday afternoon, rather it would be the multi million dollar "we replaced all our accountants with a chatGPT based system" decisions. Getting one of those decisions wrong will absolutely be a problem for your AI-insurance company. In addition, in most cases you won't even know if you were right or wrong until many years later so you have to keep reserves locked up for much much longer.

This would add exponential complexity to the already difficult challenge of insuring AI

The fact that the tech E&O insurance market is small just says to me that we're not doing enough to police and punish harmful behavior by tech companies. The risk for companies is low because we're not willing to make things more dangerous for them.

What a coincidence for those two people who were involved in this.

I don't know. Lots of words to say, "Tort reform for everyone is the right answer, but it hurts my bottom line, so I can't say the intellectually honest and obvious thing."