So western Sweden’s healthcare operator RVG tried to roll out a new application for medical records, prescriptions and so on at a single hospital. It didn’t go very well and they had to reverse the changes and return to using the old software. Notes dropping negation(“He did not have a heart attack” became “He did have a heart attack”) and prescriptions sometimes disappeared into nothingness. It was a bit of a scandal. Some people argue that the region needs to abandon Millennium entirely but after spending several billion swedish kronor that’s not palatable to RVG.
I don’t see Millennium being a problem in other places where it is used so it doesn’t seem to be fundamentally broken. Rather the issues that have popped up with the introduction of Millennium speaks to the region being woefully inept at making this kind of switch and I would expect them to fail just as hard in implementing a different solution. I don’t see the point in spending more time and more money just to fail again.
I think I disagree with RVG on the role of this kind of software as I view it as high integrity or life-critical. Admittedly it is less life-critical than say embedded software in a ventilator. With Millennium you have more time to rectify things and there are humans involved in the process. Note, RVG haven’t said outright that they don’t view Millennium as life-critical, I just don’t see how they could possibly do that and then go on to roll software into production use when it is so clearly broken.
Why is Millennium broken? This has not been made entirely clear and I think it’s valuable to point out that any software could be flawless but if users don’t know how to actually use it, then the end result is something that doesn’t work for five cents. This is one part that has been made fairly obvious, that the intended users of Millennium were given very inadequate training so they sort of fell at the first hurdle here. Even if the software had been great and the infrastructure perfect, the poor user training doomed the whole rollout.
Now this isn’t unique to RVG. It’s more common than not that some company tries to switch over to a new software system and it’s a huge mess for reasons pertaining to the software or the training or some other issue. Largely we don’t care. Maybe the prospective customers of that company are annoyed but they just take their business elsewhere. But among healthcare providers? Why would you go full steam ahead into this kind of débâcle? Even if they don’t view the software as life-critical surely “minimally functional” should be a requirement, no? And this doesn’t just apply to the quality of the software itself; it doesn’t matter why something doesn’t work if it doesn’t work. Training and infrastructure are two things which can scuttle software just fine and you aren’t left with a functional system even if you can then prove that the software itself was good.
I see three main issues on a technical basis. Software modification is a necessary part of this whole process because there’s no “one-size-fits-all” software that works for all healthcare providers in various countries. It seems Oracle have been tasked with doing the modifications so that makes me think proper programmers have done the work, which is a good thing! But software developers understand computers and not the domain for the software is aimed. So you need tight feedback from end users to developers to avoid misunderstandings* but I’m not convinced they did this and I’m also less than convinced that they did proper testing. Even if you don’t view the software as life-critical, it must surely be a bad sign if you can reliably find bugs by just sitting down to use the software for a few hours?
The other technical issue is that of infrastructure. Did they load-test things? Did they make sure they had enough bandwidth? Doing a realistic simulation seems like a good first step followed by ramping up traffic and the number of requests beyond that realistic level to see how things fare. Eventually of course things will stop working but does the software simply not work or does it sometimes yield incorrect results if load or traffic exceeds some threshold? Because you would want this type of software to be sort of atomic, either it works correctly or not at all.
Third is the issue of training. In a lot of rollouts the assumption is(tacitly or otherwise) that people will learn by doing. That’s a terrible idea for healthcare. People need to be confident what each button does. “I wonder what this thing here does? <click>” is fine for non-production environments but even with distinctly non-life-critical applications this way of doing things is not advisable in production.
But… It seems to me that the bigger issue is administrative, not technical. Because it’s not at all difficult to see how a new system like Millennium might at some point fail in all these regards. On day two of the project software quality is going to be bad, the infrastructure insufficient and training non-existent. So anyone on day two saying “Let’s roll out the software” is promptly fired because obviously the product isn’t close to being ready for live use. This is well understood by people who do this sort of stuff(or anyone really). They are aware of how you start out with things not being ready and you have a dual process of improving things and determining how close you are to production use.
Months ago some doctors received some small amount of training on Millennium. It didn’t go well and some doctors quit to work with people who have the flu or sprained an ankle or something because they didn’t want to provide life-critical healthcare and be required to use Millennium. They were quite aware that they would have to use Millennium, but without peoples’ lives being on the line they were okey with the software being jank. This is the sort of thing that’s a major warning sign. That the rollout was attempted at the time that it was, could only have happened if you were diligently trying to avoid feedback. Even the most cursory attempt to see if the system was fit for purpose would have made it very clear that it wasn’t ready for production use.
This isn’t new. Boeing had this as well with the 737 Max. They had their own engineers literally say “I won’t let my family fly on one of these planes” to each other. So the company didn’t need to spend an extra $100 million to find out that the plane wasn’t safe, they only needed to say “Hey, safety is more important than deadlines or budgets. If anyone knows anything that isn’t safe, please speak up.” Only by diligently avoiding feedback could they put the product into production use with flaws that ended up killing lots of people.
Whether you make airliners or you provide healthcare, putting your shoulder into not collecting feedback, not being a hardass who insists on things being fit for purpose isn’t acceptable. The quality of the underlying software in the case of RVG is only tangentially relevant if you intently avoid feedback and any information that says “We might have a problem here”.
Now, it should be said in defence of people at Boeing and RVG that this type of group-think isn’t necessarily the result of laziness or incompetence. Did people running Boeing have an incentive to meet deadlines? Did they have an incentive to maintain safety? Did anyone at RVG working on the Millennium-project have the role of being a hardass who won’t let the product go anywhere near production environments until tests for software correctness, infrastructure viability and end user training had been achieved to the requisite extent? I don’t see why you would have a person do that and then ignore doctors actually resigning over the quality of the software product. Do I think they should have applied at least some bare-minimum tests on the system working? Yes. But would that make sense when you ignore huge warning signs? You can’t simultaneously try to make the software fit for purpose and then also ignore clear indications that it isn’t. So RVG seems to suffer more from the administrative issue of promoting group-think over reality than from technical issues. Are there technical issues? Yes, to different extents, but if they hold on to their commonly agreed upon conclusion even when reality contradicts it, they have an even bigger problem.
* This issue of requirements elicitation is the biggest problem plaguing software development going back decades but it seems somewhat intractable. Programmers have to interpret things literally because that how computers work. People who work at a bank of a hospital tend not to be so literal and will say about a feature things like “everyone should have access” when what they mean is “everyone with an active employment at this company and its subsidiaries should have access”. You have no idea how off-the-rails software development can go with requirements that aren’t literal.
Update 2025-03-29:
KPMG did an external audit of sorts about the whole debacle of Millennium and this meant that people in a position of authority couldn’t say “No comments until the external audit is complete”. The audit concludes that the software had not been tested in a realistic environment and that functional issues meant that by the project’s own rules the rollout couldn’t go ahead, but it did anyway. Deadlines were prioritized over functionality and safety.
Two arguments have been made by politicians and civil servants at the top of the organization. One is more suited to a situation where a few bugs had slipped by and been detected in the production system, not that a woefully incomplete and unready system was rolled out for production use in an actual hospital treating actual patients. The second is more relevant I think, namely that politicians argue that they asked all the pertinent questions and had no reason to distrust the information provided to them by civil servants.
This is where the rubber meets the road! Dodging the seriousness of the situation makes it fairly clear that people in a position of authority don’t see the failure of the rollout as a serious problem but hopes that they can continue to down-play the issue until it goes away. They may be right about that, I have no idea what voters will do. They may dismiss the issue as a quirky mishap. But the argument that politicians asked precisely if the system was ready to be rolled out and if the safety of patients was assured – but were given incorrect information – is significant.
Let us assume for the time being that this is a true representation of how things transpired. That means that multiple civil servants in charge the Millennium projects studiously withheld crucial information about the various failings from the politicians who had the ultimate responsibility for the project. This raises the question: How are civil servants promoted and appointed to various positions? How does a civil servant get to a position where their task is both to be in charge of the development of projects and provide politicians with relevant information about that project yet actually withholding crucial information from politicians? It doesn’t seem that any project can be expected to achieve anything constructive under such conditions.
I agree with much of the criticism lodged against various top civil servants of the project because it’s clear that information and feedback has been restricted and kept from bubbling up to politicians. If you create an environment that prioritizes deadlines above all else and “shoot the messenger” whenever there’s something going wrong, you will end up presiding over disaster. But that’s just treating a specific problem and not addressing the issue of how that problem came about in the first place. Why would people be chosen to suddenly provide information to politicians and encourage feedback? Because the rollout was a disaster? To spend years and billions of swedish kronor promoting “no bad news” and “no feedback” was always going to lead to disaster. That the disaster is now manifest isn’t some surprising quirk, it was inevitable under those circumstances.
So it makes sense to reorganize the civil servant part of the projects governance, you can’t realistically expect people who have created an environment of “no bad news” to now create the opposite environment. But without politicians owning up to why they set up such a situation and abided by it for years, there’s little hope that the new management will be any different than the previous one. So the news that a bunch of people are being sidelined and other people given their positions isn’t surprising, it seems necessary. But while it seems required it is not sufficient.
Now, let’s consider if the statements from politicians in this matter can be accepted at face value. I would like to argue that we have seen that even when we do accept their statements without question, the politicial leadership of the region still comes off looking very incompetent. But here’s the issue; the politicians could have just taken a stroll at some random hospital – maybe even the one slated for spearheading the rollout – and talked to the nurses and doctors to find out that the software wasn’t ready and that training was insufficient. The multitude of issues were reported in multiple news organizations online and on paper. You had to try very hard to avoid information about Millennium not being ready. So why would you choose people to run the project that encouraged feedback and provided politicians with accurate information, when you actively avoid information that indicates that there is a (in this case quite huge) problem? Why would they choose more communicative managers of the project now?
So it would seem to me that the political leadership needs to do two things at this point:
- Acknowledge that the rollout being reversed after three days wasn’t just a “whoops, an unexpected problem was found” but a a figurative train-wreck that endangered the lives of multiple patients.
- That the issues of the system and its status of not being ready was something that staff and reporters had been haranguing both civil servants and politicians about for months.
I assume that any criticism was dismissed as people with an axe to grind trying to make themselves look more important, but people who just resigned can’t really be dismissed on that account. This isn’t about a few heads rolling and then we can pretend like everything’s resolved. A few people I think need to be removed from the project but a few firings does not a solution make. It wouldn’t surprise me to see calls for politicians to resign but addressing the issues properly is more important than posturing about someone being forced to resign, which is how things typically go.