Scrapbook for September
Richard Law, UTC 2026-09-01 02:01
19.09.2026 – That UK traffic control shambles
Nearly three years ago – how time flies! – we poked around in the noisome innards of the Horizon Scandal. As far as I know, no conclusion has yet been reached by that tedious enquiry into an IT shambles that ruined the lives of 736 upstanding innocents.
One of our own principal conclusions is that the overpaid panjandrums who run organizations operating software processes have no idea – no idea at all – about the operation, the quality or the integrity of the software, the operation of which is in fact their core business. This blissful ignorance is shared by the judiciary, politicians, civil servants and the know-nothings in the mainstream media.
The initial explanation of the meltdown in the software behind the UK's traffic control system on 8 September, which led to the disruption of more than 330'000 travel plans, is turning out to be yet one more example of the disastrous combination of rickety code and ignorant, incompetent managers. The report was published on 16 September. In this case, Martin Rolfe, the wildly overpaid CEO of NATS, claimed the meltdown was caused by a 'a small subsection of coding' (recte 'code'). We have someway to go yet, so we will leave the silly 'small subsection' uncommented. Here is the narrative according to NATS:
The preliminary investigation has determined that the incident was caused by a legacy and previously unknown defect in a software module within the NAS which processes requests for the allocation or reassignment of squawk codes. A squawk code is an identification code transmitted (squawked) by the aircraft during flight, enabling air traffic control systems to match the aircraft’s altitude, speed and direction with its intended flight plan so that flight progress can be monitored.
The incident was triggered by a valid manual request for a squawk code. This manual request was made correctly and there was nothing abnormal or invalid about the associated flight plan.
While this request was being processed, the NAS received a message for a higher priority activity to be undertaken which resulted in the squawk code allocation being paused while the system processed the higher priority message. Switching between different activities in response to prioritised requests is a normal function of the system; however, when the processing of the squawk allocation request resumed, the software defect meant it did not resume correctly and the resulting output was corrupted
The twee cuteness of this text is a soothing lullaby for the non-technical reader: 'a legacy and previously unknown defect' – put in enough soothing words and it all sounds so reasonable: 'valid manual request', 'nothing abnormal or invalid'. However, alarms are already ringing in the head of any software professional reading this, whose first thought will be: 'If everything is so normal and valid, why did the entire system crash?'
Before we go any deeper, the first question is: why isn't this system capable of multithreading? It has for many years been an everyday coding task to run multiple processes simultaneously. Every modern program language can do this and offers features for running multiple processes concurrently and without interference. The generation and assignment of a squawk code appears to be a completely trivial process in software terms.
If, for whatever reason, a multithreaded solution has not been chosen then the old-school solution is to insert incoming requests into a queue. The current operation is protected until it completes gracefully, after which the next operation is taken from the queue in accordance with its priority. Multiple queues are conceivable. In this scenario the squawk code generation process cannot be interrupted by the arrival of a new request – an incoming request is parked in the queue(s) and only fetched when the squawk code generator is ready. The operation is a pull, not a push. I cannot imagine how any programmer could come up with a code generator that is interruptible in the way that NATS describes it:
1. The defect existed in a specific subsection of code within a software module, with an exposure window estimated as approximately one millisecond.
2. For the fault to occur, a higher-priority request had to arrive during that exact millisecond while the original request was part-way through updating a value.
3. Had the higher-priority request arrived even one millisecond earlier or later, the update would have completed normally.
I have no idea what an 'exposure window' is: I cannot imagine writing code that had such a vulnerability. I have also no idea why NATS is talking of an 'update'. What this goobledegook really means is that it was only a matter of time before this defect manifested itself. It is not, as implied, just bad luck that the request arrived when it did; sooner or later it would have occurred, being a classic accident waiting to happen.
All that is bad enough, but much worse is the downstream cascade that brought the entire system crashing down (programmers speak of 'borking' the system). There is a standard terminology for this: in a robust and well-programmed system, any failure or irregularity that occurs has to be dealt with 'gracefully'. This means that it has to catch the problem, alert to its existence, log it comprehensively and then recover in a controlled fashion.
A process should not emit garbage, nor should it accept garbage. Packets of data exchanged by the modules of the system should be in a self-validating format. There is no excuse for any error triggering a chain reaction which brings the system down along with all the systems connected to it. This is the key point that is not addressed in the fairy tale supplied by NATS.
Already in 1980, the programming language ADA was introduced for embedded and real-time critical systems, particularly in the military. Its design enforced a robust programming style with checks at every level. Nearly fifty years after its development it is still used in safety-critical applications. I remember reading somewhere that a variant of ADA is used in NAS systems, in which case it would be one more proof that even the most robust software is vulnerable to unsupervised idiots.
No amount of fairy tale explanations can get past this. The only conclusion is that the NAS software was written by incompetents, tested and validated by incompetents working for incompetent managers.
In the world of software creation there are coders and there are coders with imagination – the latter being much rarer and much more valuable than the former. On the basis of the present piece of incompetent coding we can be sure that this software is riddled with such errors ('legacy and previously unknown defects') that are 'waiting to happen'.
NATS prefaced its report with a letter from the CEO. This gave an even more anodyne summary of the fault. The company's PR people knew that lazy media reporters, who know almost nothing about software engineering, would lap up the tidbits in the summary and not even bother confronting the nonsensical guff in the attached 'report'.
09.09.2026 – Switzerland: government by wishful thinking
So, we finally find out, about a year after the fact, that it wasn't Trump's hard-nosed dealmakers cheating poor little Switzerland out of their 'fixed-price' contract for F-35 fighter jets.
According to the just-published findings of the Geschäftsprüfungskommission (GPK), 'Parliamentary Surveillance Commission', a 'fixed price' contract never existed – with the result that the price of the jets rose serenely over the course of the ten years in which the Swiss government fretted about this and that.
Egg on faces time all round, but particularly for Viola Amherd, the Bundesrat, 'Federal Counsellor', who was a small-town lawyer before she took over the responsibility for the Swiss Department of Defence. The fixed price fantasy seems to have popped into her and her advisors' heads. Oh well, never mind, she'll get by somehow on her very substantial pension and, though she left the job under a cloud, need fear no accountability.
In matters of defence, the Swiss can sleep soundly in their beds knowing that Amherd's successor, Martin Pfister, a historian by training, comes from the same party, Die Mitte, and is just as clueless about what he is supposed to be doing.
Update
The Neue Zürcher Zeitung has produced a detailed review of the findings of the enquiry into the F-35 affair (German only). The conclusion is essentially as we presented it: over the course of the years, credulous ministers, parliamentarians, civil servants, lawyers, military and procurement specialists all believed in the myth of a 'fixed-price' contract even though the Americans repeatedly warned them that no such thing was possible since it violated American law. The procurement order was signed off by various hands, most of whom don't seem to have read or understood this six billion+ CHF order. In particular they were all relying on each other and a single 'specialist', Darko Savic, who seems to have just told them what they wanted to hear.
The article concludes by listing the key players who, with only one exception, have all moved on to pastures new. So funktioniert die Eidgenossenschaft, 'That's how the Swiss Federation works', concludes the reporter, Christina Neuhaus, who thus sounds a little more surprised than I am.
0 Comments
Server date and time:
Browser date and time:
Input rules for comments: No HTML, no images. Comments can be nested to a depth of eight. Surround a long quotation with curly braces: {blockquote}. Well-formed URLs will be rendered as links automatically. Do not click on links unless you are confident that they are safe. You have been warned!