The Horizon scandal

The tumbrils roll

At last, at last, after a quarter of a century, Government and Parliament in the UK are taking the scandal which ruined the lives of 736 innocent people seriously. Sort of. There will be large amounts of taxpayers' money doled out in compensation, spread around like balm on a wound; the Boss Lady has elected to return her gong; the in-so-many-ways fatuous and deeply unpleasant Ed Davey may be finally brought down by his own arrogance and stupidity, the loss of his gong a minor problem compared to the public trashing of his already shaky reputation.

Nevertheless, let's be happy that the tumbrils have at last begun to roll. Whether all the miscreants in this tragedy will end up in them is doubtful.

The many human tragedies the scandal left behind are now occupying the attention of the media – but only after a television dramatisation brought this awful mess into their field of view. The few stalwarts who have campaigned more or less unheard for many years to get this terrible wrong righted can finally feel some satisfaction that they have been proved right all along and that perhaps the day of reckoning has finally arrived. But why did it take so long?

If we leave the awful human stories that are currently filling the media aside, the scandal becomes even bleaker. From beginning to end we follow a chain of events, each link of which is in itself a scandal.

Speaking Geek

The software behind the Horizon system delivered by Fujitsu appears to have had a flaw so elementary that we have to wonder how any trained and experienced programmer could write such deficient code. Software developers work in teams, teams have supervisors and software should be subjected to rigorous testing. We geriatric coders still remember the adage we learned when we were crashing code in Fortran and Algol: How do you reduce the quantity of bugs in a program? Don't put them in in the first place.

A quarter of a century has gone by and a detailed technical explanation of the problems has yet to be published. We therefore don't yet know what the defects were that caused Horizon to malfunction. According to nerd gossip, the main problem with the Horizon software was that is didn't manage transactions properly.

The usual rule applies with defective software, everything works fine – until it doesn't. Working back from that failure to manage transactions correctly, we can infer the nature of the root problem with reasonable confidence.

Let us say a customer in the sub-postoffice paid £20 for an item or as a payment onto their Post Office bank account. The client software in the Post Office branch would send a transaction record to the server. The client should generate a transaction ID in order to distinguish this transaction from all the other transactions being processed.

The server then receives the transaction details. I don't know what plausibility tests were applied to the received data, but if the data was formally correct the server would book the transaction accordingly and close it. If not it would discard the corrupt transaction and send a message to the client saying 'please resend'.

If the data is correct the server would then send a confirmation of receipt back to the client, which would then display some sort of 'transaction completed' message. So far, so good. This is what must have happened most of the time, otherwise every sub-postmaster in the country would be behind bars.

But many datalinks at that time, particularly to rural sub-postoffices, were slow and unstable. If there were errors in the transmission of the data the confirmation response was either not received or garbled and thus not understood by the client.

The sub-postmaster, staring at a non-responsive screen or some message about a transmission error, with one eye on the queue of customers in front of the counter still waiting for their stamps or their pensions, would do what humans do and click the send button once more – perhaps even give it a really good clicking.

In a well-designed system the client should resend the transaction with the same ID. That ID should be maintained by the client until a server confirmation of the transaction is received and understood. Any competent database programmer would understand this crucial distinction: the ID represents the transaction, not a particular instance of a transmission.

It is difficult to imagine a database table without at least one field that allowed only unique values. The crucial error in the Horizon software seems to have been that the transaction ID was set by the server on receiving the data. Every time it received transaction data it set an ID, even when the data was the sort of duplicate transaction we have described. In contrast, where the client is responsible for generating the ID, the server would simply discard the duplicate record. The Horizon error would not be possible.

The forensic accountant looking at the chronological table of transactions years after the fact would have to look very closely to notice that certain 'separate' transactions contained identical data. Readers should remember that this system is receiving a flow of transactions from clients all over the country. Between the first record a particular client sent and its duplicate caused by a resend at busy periods there might be very many other records from other clients. In other words, the records would rarely be next to each other in the database table.

However, an investigator with access to the Horizon system could sort this out by listing the transactions for a particular sub-postoffice or even a particular terminal. A few minutes – no more – and a few SQL instructions would be all that was required to display the problem. As far as I know no one performed this simple task and, if they did, no one reported the results.

The result of this error was that when the sub-postmaster re-sent the transaction, the server would book this as a further transaction. Thus as far as the Post Office accounts system was concerned £40 (£20 + £20) had been handed over to the sub-postmaster by the customer, but at close of business the till only contained £20. The conclusion of the Post Office: the thieving sub-postmaster had pocketed £20. If the sub-postmaster had given the send button a good clicking, he or she might end up 'stealing' hundreds of pounds. Over a few months or even years, that can mount up into big money.

This should not happen: it is terrible systems design. Some of the first commercial uses for the early computers that began to appear in the 60s were for payroll and accounting. Databases were developed very quickly and a key element of even the very earliest databases was transaction integrity, particularly over the very rocky communication paths that existed then. I have suggested one solution, there are many others.

Bad enough. But no one caught this elementary error; no supervisor checked the code thoroughly; no unit tests turned up the problem, or if they did, no one fixed it. It's the old story: consciously or subconsciously, no programmer wants to break their own code. In addition, it is also difficult to do testing of the behaviour of a system using bad transmission routes.

But, that said, the vulnerabilities in the use of transactional databases have been known clearly for so long that any expert would have anticipated them without needing to think. Note also that this error is not a bug – all the code seems to work as it should – it is a design error.

Sanity check 1

Most shockingly of all, when the software began showing hundreds of sub-postmasters as thieves and fraudsters, no one seems to have performed that most useful design task, the 'sanity check'. In this case, the check is extremely simple: whereas perhaps one cheat might turn up every few years in the old system, now hundreds of sub-postmasters with impeccable records up until then were supposed to be pocketing money. Something has to be wrong with the system, the alternative is insane.

Sanity check 2

If you wanted to defraud your employer (or the tax people), the good old-fashioned way was to accept cash from the customer and only pretend to put it in the till. Sophisticated cash registers, receipts and card transactions have put an end to that problem in the modern world.

The sanity check in this case is to consider the simple question: Why would anyone attempt to steal money in a way that is so obviously detectable? Even the dimmest criminal mind would be aware that simply not putting the cash for a computerised transaction in the till would lead to easy detection. Something has to be wrong with the system, the alternative is insane.

The blind leading the blind

Despite such obvious problems, no one thought of checking the code or was forced to check the code. The Fujitsu managers didn't check or didn't want to check, the Post Office managers didn't check the Fujitsu managers, the Post office hierarchy didn't check anyone or anything, the government apparatchiks didn't check, the Minister 'responsible' didn't check and didn't seem to care.

Fujitsu's behaviour is incomprehensible. It is one thing to try to avoid making too much fuss about some badly designed software, fixing it quietly and doing what they could to avoid too much reputational damage. But what they did was to circle the wagons around their defective code for sixteen years, from 1999 to 2015, the year that the post office stopped prosecuting their sub-postmasters. The Chief Architect of the system maintained on oath again and again that the system was operating perfectly.

How did this shambles happen? Because those in the managerial and administrative path knew nothing about computing; they could neither formulate the question nor understand the reply they would have received. The Germans say 'Vertrauen ist gut, Kontrolle ist besser' – 'Trust is good, checking up is better'. Everyone trusted and no one checked. They didn't check because they couldn't.

My not so learned friends

We are nowhere near the end of the chain of disasters in this tragedy. The next link is the courts system.

The adversarial procedure of a trial is supposedly a good way at getting at the truth. Too often though we find that the process can be very hit and miss. In this scandal it failed abysmally again and again.

The defence counsel can be a duffer without the intellectual capacity to get to the core problem, the Horizon accounting system; the magistrate or judge almost certainly lacks the technical understanding to preside over such a technical case, or – more likely – is also a duffer; the prosecution is not there to help the defendant's case.

The courts relied on the 'expert testimony' of Gareth Jenkins, a Fujitsu employee who was the chief architect of the Horizon system. There was no chance that this was independent expertise. He only appeared in the flesh once, in a case in October 2010. As we have already noted, he asserted that the system was working perfectly, an opinion he repeated again and again.

When no one else in the courtroom has the slightest idea of the technical issues involved, the expert witness can state all manner of things without fear of challenge and has a clear run. Even if an alert defence counsel were to hire their own expert witness, unless Fujitsu allowed access to their system and behaved cooperatively, that expert witness for the defence would have nothing at all to say. Fujitsu had the Horizon system completely under their bell jar.

We have to give the defence in these cases some credit for the fact that any defence which relied on computer error would be hopeless. Before 1999, prosecutors using evidence obtained from computers had to demonstrate that the computer system was working correctly. A ruling introduced in that year made the prosecutor's task much easier by reversing the burden of proof. From then on courts were to take it as given that a computer system was functioning correctly. Computer systems were to be considered to be technical 'black boxes' analogous to speed traps or breathalysers. No judges needed to lose sleep pondering the design of transactional databases.

From 1999 on the defence had to demonstrate conclusively that the computer system had faults – in practice an impossible task when access to the computer system was under the control of the prosecution. What could the defence do? The only hope was to bring on a few character witnesses and suggest a guilty plea to obtain a lenient sentence.

Because of this ruling it seems that Jenkins was never pressed with a technically knowledgeable cross-examination. His testimony was never examined forensically: it was simply used as a boiler-plate submission in subsequent trials. All he had to say was 'Horizon works correctly as designed'. At the instigation of the Post Office prosecutors, his subsequent witness statements were even hardened up to make them even more damning.

In subsequent enquiries Jenkins has been uncooperative and refractory, avoiding answering questions and giving minimalist answers, dodging and prevaricating.

He has still to be cross-examined in person, which means that, a quarter century after the start of the scandal, we are still no nearer to learning the nature of the system defects that led to this human disaster. Everything we have proposed so far has been speculation.

Whether, when he finally does appear, he will be given a proper, technically competent forensic grilling is also still to be seen. He is currently demanding immunity from prosecution when he testifies, which, in itself, is the wrong foot to start on. Perhaps he is an idiot after all.

It is still shaming that it never occurred to these highly remunerated lawyers for the Post Office to question the technical issues behind the case they were prosecuting. In the adversarial system they are not agents for the defence – their only task is to get a guilty verdict. It was even worse in this case because, by some arcane ruling, the Post Office rather than the CPS could prosecute its own cases, which seems to have boosted the vigour of the prosecutions, in which private prosecutors were earning their keep. The relaxed 'you win some, you lose some' ethic of the CPS had no place here. Even more distasteful was the Post Office's practice of paying bonuses to their investigators for every successful prosecution, effectively setting bounty hunters onto the defenceless sub-postmasters.

In the legal mincer

Thus the poor sub-postmasters were always guilty as charged. Some were forced to minimise their punishment by pleading guilty to a crime they did not commit. There were even sub-postmasters who, to avoid prosecution, paid back substantial amounts they had not even stolen. At least four people took their own lives. Some were even subjected to disgusting vigilante campaigns in their local communities: once someone has been found guilty of a crime, they are effectively defenceless.

As the numbers of successful prosecutions grew, so did the momentum for guilt. It needed only one case to demonstrate how flawed the prosecution case was to lead to the results of the others being appealed. That one case never happened to my knowledge and if it did it was ignored. There's another sanity check in there: how can you arrest, charge and try 736 separate individuals and find them all, without any exception that I have found, guilty. In a sane world you would expect some percentage of the suspects to be found not guilty.

The justice machine ground on for sixteen years, years in which it ruined the lives of 736 innocent people. Finally, in 2020, six former sub-postmasters had their names cleared after the Court of Appeal quashed their wrongful criminal convictions. The court quashed 39 more convictions in 2021. At the time of writing 93 cases have been overturned. Only 643 cases to go. Thirty out of the 93 have been awarded compensation. Only 706 left to be compensated. Around 70 claimants died before their claims were settled.

Had the scandal not erupted as a result of the TV dramatisation, the Court of Appeal might be grinding on still, working its way slowly through those who still have the energy to appeal their conviction. At the present rate there are only another seven years of judicial churning to go before everyone has their convictions quashed. For many of these, compensation will still be a distant dream.

At the time of writing, the government has been shamed into trying to find a way to quash all the remaining convictions at a stroke. If they manage to organize something, let us not praise them or forgive them. They could have done it at least ten years ago when the scale of the miscarriage of justice was clear to everyone who cared to look.

The trusty sword of media truth

And what of our valiant media giants, who are currently clutching their pearls at the enormity of the scandal? They are also technologically clueless and assume that their readers are equally so. They may be right in that. Corrupt sub-postmasters on the make and hero sub-postmasters traduced are better human interest stories than nerdy errors in software.

Down the years a few people understood the problem, but it took a soppy television dramatisation to wake the wider media up. In fairness, the media woke up a little bit following the Appeal Court session in 2020. Praise is particularly due, however, to Computer Weekly, which has followed the scandal doggedly since the beginning.

Afterthoughts

Snow's 'Two Cultures'

How can we explain or understand such a long chain of terrible failures? In a 1959 lecture the novelist and physical chemist C.P. Snow identified what he called the 'Two Cultures'. One culture consisted of scientists, who didn't seem to bother much with literature; the other culture was that of the literary types.

The two cultures were not balanced: there are of course many scientists with broad interests in some aspect or other of the humanities, whereas the vast majority of the 'literary intellectuals' have as good as no idea about science and technology. It is suggestive that it was the tech journal Computer Weekly that most consistently kept up with the scandal. Unfortunately, 'literary intellectuals' don't read that journal.

Snow's concept of the 'Two Cultures' fizzled out not very long after he proposed it, largely because, I would suggest, the 'literary intellectuals' control the culture in society. They are the ones who write books, scribble for magazines and contribute to the media generally. They are the ones with their hands on the levers of the state, who have never been happy with being identified as scientific dummies.

In the present context we see the complete inability of the literary culture to cope with issues that are fundamentally scientific or technological. Science and technology are never settled, they are battlefields of claim and counter-claim. The Oxbridge PPEs that infest government have no clue; the denizens of the law courts are equally at sea; the average reporters or pundits in the media can only parrot what someone else has told them – when a scientific item lands on their desks, they have no analytical skills to measure competing claims (the latter being the essence of science).

Think on: The economically ruinous Net Zero project, the evil consequences of which are becoming more apparent every day, is the spawn of Ed Miliband's Climate Change Act 2008, which was passed almost unanimously by the UK parliament. I seem to remember that at the time there were only two or three MPs with a science background.

Let's not forget the COVID shambles, which showed the UK government's disastrous zig-zag responses to the competing claims of its scientific advisors.

And now comes the Horizon scandal, which only woke up the cultural and political elite in the UK when they watched a TV dramatisation.

All of these instances show that Snow was bang on the money, then and now.

Big state, little person

The powerlessness of the innocents in this saga should serve as an example of the many ways that the administrative-judicial machine has gained the upper hand in Britain.

The strategy for the little guy is the same as it always was in totalitarian regimes of all flavours: keep your head down and hope the big people don't notice you. If they do, you are in for it – you will learn what helplessness feels like: your careful life, your house, your job, your savings, your good standing will disappear. You may spend time in prison. People you have known a long time, perhaps even your relatives and your children, would suddenly rather not be seen in your company. The computer was wrong, you tell them. Really? they say.

What cash amount will compensate you for that?

After the dramatisation of the human face of the Horizon scandal there has been a huge wave of sympathy and outrage in the general population. Does this reflect the pent up rage of the citizen against an ever more oppressive machine?

The memories of the COVID days are still fresh: the scare tactics of the government 'nudge unit'; the repressive and often heartless restrictions that were imposed so rigorously, bossily and insensitively on a helpless population. Horizon has gouged at that scab, a wound that is still a long way off from healing.

Most citizens have felt the force of the administrative state at some time in their lives: parking fines and all kinds of minor traffic violations enforced with delighted rigour; fines for dropping an ice cream; fines and penalties for an ever growing list of this and that, including the odd swear word; even schools can fine parents for – well, all sorts of things. In the surveillance state the citizens will all be wearing hoodies soon.

The generally useless councils that now have CEOs and cabinets and task forces and 'tzars' are a constant irritation. The bank accounts of citizens, once sacred places, can now be raided by public and private organisations, seemingly at will. And, to cap it all, the fatuous NetZero campaign has bombarded fretting Britons with one imprecation after another, its unreachable deadlines and projected bans a constant cause for worry. We used to joke, 'they can't touch you for it'. Now they can, for ever more 'its'.

Watching the postmasters have their lives ruined by the same kind of implacable forces that every citizen not insulated by piles of cash now feels has clearly struck a nerve in the populace. Even the normally dopey political class has seen this and has, within days, gone from tedious inertia to a bustle of activity.

Update 28.03.2024

January's explosion of public outrage at the Horizon scandal was unsustainable by the media over more than a week or two. The Establishment responded: politicians expressed shock; the compensation procedure was accelerated from a standing start to a judicial stroll; the nobs involved were to be named and shamed (with almost no consequences) and so on; Fujitsu has been rewarded with some more juicy contracts from the UK government.

The inquiry continues its glacial progress – at some point in the distant future, when all concerned have changed jobs or have taken their substantial pensions, a report will be published. Said report will have a news lifetime of about a week and then the whole matter can be forgotten about.

One recent revelation caught my attention today: some of the IT people in the Post Office knew that Fujitsu had permanent and complete access to the Horizon system – well, stap me vitals! The surprise which anyone with a technical background in IT feels at this news is that anyone is surprised. Every system has 'superusers' and designated classes of users beneath that. Fujitsu could not have created, supervised and maintained Horizon without such access.

The 'revelation' is anyway nugatory, since no one to my knowledge has suggested that Fujitsu's staff were manipulating the system to their own financial advantage or the disadvantage of others. It is difficult to imagine how a Fujitsu employee could benefit from creating fictional deficits in the accounts of subpostmasters. The only value the admission has is a very narrow legal one: it nails the lie that only the evil subpostmasters had access to the system, thus proving beyond a reasonable that they had to be the thieves: guilty as charged.

As such, the point is a distraction, one of the many that we hear in this scandal. The truth is that the software of the Horizon system was defective. In what way it was defective, the inquiry, after years of grinding forward, has yet to discover.