The Horizon scandal revisited

We wrote about the UK's Horizon scandal nearly three years ago. 736 innocent and upstanding people had their lives ruined and their reputations trashed by the cumulative incompetence and ignorance of software developers, business executives, politicians and lawyers.

During the intervening years the Horizon Inquiry has trudged its lawerly way through the issues. Two exemplary independent websites have tracked its work and delivered comprehensive and comprehensible commentaries: Post Office Scandal and The Post Office Horizon IT Scandal. If you want further comment on the mountain of detail that the Inquiry has thrown up then they are the places for you. The mainstream media have lost interest, just as they have lost interest in the Grenfell Tower Fire Inquiry and the Rape Gangs Inquiry – 'Go, go, go, said the bird: human kind / Cannot bear very much reality'. (©T.S. Eliot, 'Burnt Norton').

My interest in the issue nearly three years ago was in the defective software the lay behind this mess. As the scandal unfolded, the public and the courts were assured that the Horizon system was 'working perfectly'. No one outside Fujitsu, the system contractor, had access to the code, so I was forced to speculate on the nature of some of the bugs that caused 736 innocent people to be prosecuted for theft. There is a little bit of cold comfort for me in that my speculations have turned out to be completely correct. In fact we know now that the development and introduction of the Horizon System was a shambles from beginning to end.

Just a short while ago, on 25 August 2026, a technical report was published in the Journal of Cybersecurity and Privacy: The Horizon Scandal as Socio-Technical Failure: A Systematic Analysis Through Cyber Security and Digital Forensics Frameworks by Harjinder Singh Lallie (Cyber Security Centre, WMG, University of Warwick). The report is open access – and should be read by those interested in the details.

The study identifies 'five interconnected failure categories', most of which we touched on three years ago:

• software defects and poor system design;
• deficient patch governance;
• inadequate audit logging and compromised evidence integrity;
• investigative failures and prosecutorial conflict of interest;
• a systemic absence of technical expertise and independent oversight.

In short, the Horizon System was anything but a system which was operating perfectly. 'At its core, the scandal reflects a profound breakdown in cyber security principles.' Connoisseurs of IT and management stupidity should read the entire report. What follows here is just a very meagre overflight of the mess.

Computers are infallible

Readers with good memories will recall that one of the key hindrances to postmasters mounting a defence was a legal ruling a few years previously that courts should assume that complex devices such as speed cameras and breath testers were in all cases working correctly and that legal arguments suggesting otherwise had no legal standing.

This ruling doomed the cases of the Horizon postmasters: their lawyers could not argue that computer system errors caused the crime; they were only able to field mitigation arguments – the accused was a fine, upstanding member of the community with an unblemished record etc.

We wonder at the low quality of the brainpower within the legal profession that gave this nonsense legal respectability, for it is, of course, based on a category error that confuses the operation of hardware and software.

Hardware works or it doesn't. When it works it does so in consistent ways, except in very rare cases such as an intermittent contact error. It is reasonable to assume, unless there is good evidence to the contrary, that hardware is infallible.

Software, however, written by humans, in its nature is extremely fallible – as fallible as its human authors. One need only consider the amount of updating required to remove bugs (and often to introduce others) throughout the life of every commercial software in order to keep it on the straight and narrow. We expect software in its nature to be error-prone.

By the category error of mixing up hardware and software, the great brains of the legal profession are to blame for the years of misery and injustice that it caused. The briefest moment of reflection will tell us that treating the operation of the Horizon System as the equivalent of the operation of a speed camera is absurd, an absurdity that the great brains of the legal profession clung to for years.

Horizon I, the pilot system

The first Horizon system was a pilot that was rolled out in a few branches beginning in 1996. This system contained 'critical flaws, including software bugs, unexplained shortfalls, and user interface issues that branch staff could not resolve'. This is not surprising, since the development team consisted of two people who sort of knew what they were doing and another six who should not even have been hired, so low was their level of technical competence.

The operation of the pilot was abysmally bad: 'pilot project flaws and software defects to third-party integration issues and poor oversight, both technological and governance failings were deeply embedded'. What is the worst thing you can do with a rackety software system? Suppress or ignore the failings, give it further tasks with a massive feature creep then roll it out nationwide.

Note that this occurred before the rollout of the final Horizon System. Everyone who had anything to do with the system and those who supposedly supervised them and those who supposedly supervised the supervisors should have known what a mess the system was. If they didn't, they were incompetent; if they did, they later stood by in silence as one postmaster after another was investigated und put up before the beak, ending up ruined by this system that 'worked perfectly'.

Horizon II, the final system

In a normal software development the prototype serves as a proof of concept, a test of design assumptions and a test of the quality and robustness of the underlying code. Even non-specialists are familiar with the traditional version numbering: v0.x for prototypes, v1.x etc. for production code. After fixing all the things that went wrong in the prototype you asymptotically approach the point at which you can speak of the final system, one which will work as specified and not get anyone sent to prison.

At least, that's how it is supposed to go. Horizon II, after the first prototype, broke the procedure by changing existing specifications dramatically and introducing new requirements to such an extent that Horizon II was effectively a fresh start. The most serious change – the change that led to the disaster we are dealing with now – was that, whereas Horizon I operations were executed on the local client, Horizon II was a distributed system with thousands of clients talking to a central system, a situation that is a much greater programming challenge.

Your author still bears the mental scars acquired during the development of distributed systems during the pre-internet age, systems which relied on rackety, slow modems from Deutsche Telekom, their operation documented in a 12 cm thick ringfile. We were not allowed to use the US modems, which were much faster, one tenth the size and one twentieth the price. When almost every transaction is messed up in some way the programmer quickly learns the key professional requirement: pessimism allied with scepticism. The authors of the present paper note:

A combination of unchecked scope creep and unaddressed design flaws resulted in a wide array of software faults. These included critical bugs such as synchronisation failures, transaction isolation issues, and silent crashes in key modules like CABSProcess. Other faults – such as the Dalmellington and Callendar Square bugs – generated phantom or duplicate transactions, while audit functions frequently failed to record complete end-of-day activity.

In almost 'Skynet' style, 'Out-of-hours transactions were being generated by the Horizon system – not directly input by sub-postmasters at branch terminals'!

All this was made much worse by the fact that the Horizon System 'suffered from inadequate error feedback, meaning it frequently failed to provide sub-postmasters (SPMs) with clear, timely alerts regarding faults, inconsistencies, or data discrepancies. This lack of visibility left branch staff unaware of system-generated errors that directly impacted their accounts'. So if Skynet had carried out overnight transactions on your Sub-Post Office account you would know nothing about it until the cops came to call.

Nor would subpostmasters know when updates were carried out and what had been changed. Even worse, even developers had no way of evaluating the effects of new updates.

Prior to 2009, logs were minimal, and key actions in the system, such as transaction entries or system errors, were not adequately tracked. This absence of detailed logs made it difficult to trace and verify actions within the system, which allowed discrepancies to go unnoticed for extended periods.

Even worse, audit trails and error feedbacks were not only largely ignored, but they seem to have been largely suppressed:

Audit Record Queries (ARQs) were requests made by the Post Office to Fujitsu for detailed transaction log data from specific SPM branch terminals…

…

PEAKs (Problem Evaluation and Assessment Kits) and KELs (Known Error Logs) were internal fault management tools used by Fujitsu to track and manage problems within the Horizon system …

During the Horizon scandal, neither PEAKs nor KELs were disclosed to affected sub-postmasters or the courts. KEL dsed5628Q provides a concrete illustration: raised in January 2008 following a CABSProcess timeout that caused an undetected £465.73 trading discrepancy at branch FAD005948, the call was closed without a fix on grounds of rarity, with Fujitsu’s own developer noting that the underlying error-handling problem was ‘endemic’ to the EPOSS codebase. Neither the KEL nor the existence of the defect was disclosed to the affected branch or to the courts. This omission withheld critical evidence that could have demonstrated that errors in the system were known and recurring, and not the fault of the individuals being prosecuted.

Of course, since the legal belief was that only the postmasters had access to the system, they were the only people left standing in this gruesome system. But Fujitsu engineers, as we suspected three years ago, always had superuser access to their system – it is inconceivable that they didn't. They not only could change anything, but their actions and the results those actions were not fully logged, meaning that no one would ever know what had been done.

The Horizon scandal illustrates a failure mode in which the Post Office simultaneously occupied the roles of system operator, investigator, prosecutor, and alleged victim. In such circumstances, organisational incentives may actively favour the minimisation of defects, resistance to scrutiny, and suppression of contradictory evidence. Existing frameworks provide only limited protection against this scenario because they govern the mechanics of assurance and compliance rather than the incentive structures within which those mechanisms operate.

This limitation is evident in several of the most consequential aspects of the scandal. The suppression of expert concerns raised by Second Sight, Jason Coyne, and Fujitsu personnel; the withholding of PEAKs and KELs from defence teams; the continued reliance upon system-generated evidence despite known reliability concerns; and the use of private prosecutions without independent oversight are not failures readily addressed through conventional security controls. Rather, they reflect organisational decisions taken despite the existence of information indicating that serious problems existed.

Every technical and managerial aspect of the Horizon System is a noisome mess; the legal system – prosecutors, defenders, judges – fermented the stinking mess; the Post Office and government ministers made it all much worse; the media was totally out of its very shallow intellectual depth in IT matters.

It is, in my opinion, highly unlikely that those involved will be punished in any way. Large quantities of taxpayer cash will be thrown – albeit grudgingly – at the victims and after another couple of years the whole mess will be laid ad acta. The powers that be are unlikely to want their dismal performance being scrutinised in successive trials for years to come.