1999 throwback - the Millennium Bug

I remember 2 things;

1) Vastly overpaid “consultants“ inserting flõppy discs in workstations and , when done, attaching a “Y2K Compliant” sticker to each machine.

2) Claims of legacy COBOL coders being offered a whole year’s salary equivalent to be on site 12 hours either side of 01-01-2000 00:00:00
1: Many of those "overpaid consultants" were doing the work at a time when the number of people available was low compared to the number of places that needed it done, and often it wasn't just "insert a floppy" but to first work out what systems needed updating - depending on the software and hardware in use that could be a dozen very different programs (or more) and either a bios update or OS patch to work around a hardware issue.
So the company using them might have been charged a lot by an outsourcing company for what seemed like a very quick fix, but that would have been after the actual work needing to be done had been identified, the patches sourced (in some cases written specifically) and tested - the "insert floppy" stage was usually the final step in something that in a big business might have involved hundreds of man hours of planning and testing, then however long it took to update and confirm for each of the machines (in some cases hundreds/thousands).

2: How many legacy coders do you think there were at the time?
And how many of them knew the systems they were watching?
Most of those claims were in regards to systems like banking where the total number of people intimately familiar with the (often ancient) code running in the background could be counted in the low single digits for a major bank, so not only were you paying for skills and knowledge that in many cases was exceptionally rare, but at a time when many people even in low end jobs would be looking at double or triple their normal overtime rate, and a lot of those people had long since retired, meanwhile the systems they were being paid to babysit could cost many times what they were being paid in total for every minute of downtime.
It really doesn't take long for a banks system to be down for the cost in lost business and regulatory fines to hit 7 or 8 digits ;)

From what I remember every instance of people being paid "a years salary" was for people being expected to be ready to deal with either life critical systems, or financial systems where downtime costs a fortune, and where the number of people who could troubleshoot a problem and fix it on a live system in a hurry was very limited. I remember reading stories from people involved where there might have only been two or three people alive who had worked on the original code well enough to fix it, and in a funny turn of events they could pretty much name their price to in some instances do things like fly in from another country to spend the time sitting watching what was going on having also reviewed the code and refreshed their knowledge in the preceding weeks or months.

Oddly enough I seem to remember several banks starting to (very slowly) update a lot of their really old* systems to more modern code and native hardware in the years after 2000, it was almost like something had happened to focus the minds of management and the accountants on the age and supportability of the systems their companies relied on.



*There were/are systems that are running basically the same code from the 1970's on emulators where they hadn't been replaced due to the complexity of the systems added to it and the risks involved, as as long as they could run the code faster on new hardware (even with an emulation layer to look like the original hardware) it wasn't deemed a major problem. IIRC one of the biggest outage in UK banking history came when a bank tried to split the computer systems of two of it's subsidiaries and had a booboo.
 
I personally didn't know anyone who got paid a fortune to fix this. Just heard stories about such people.

We just fixed any applications long in advance and had no problems with Y2K. None of the applications or code team I was on had any issues AFAIK. There were a few hard coded dates unrelated to Y2K that were discovered and fixed.

But running out of codes and IDs isn't that unusual these days with older software and databases that have been poorly designed.
 
1: Many of those "overpaid consultants" were doing the work at a time when the number of people available was low compared to the number of places that needed it done, and often it wasn't just "insert a floppy" but to first work out what systems needed updating - depending on the software and hardware in use that could be a dozen very different programs (or more) and either a bios update or OS patch to work around a hardware issue.
So the company using them might have been charged a lot by an outsourcing company for what seemed like a very quick fix, but that would have been after the actual work needing to be done had been identified, the patches sourced (in some cases written specifically) and tested - the "insert floppy" stage was usually the final step in something that in a big business might have involved hundreds of man hours of planning and testing, then however long it took to update and confirm for each of the machines (in some cases hundreds/thousands).

2: How many legacy coders do you think there were at the time?
And how many of them knew the systems they were watching?
Most of those claims were in regards to systems like banking where the total number of people intimately familiar with the (often ancient) code running in the background could be counted in the low single digits for a major bank, so not only were you paying for skills and knowledge that in many cases was exceptionally rare, but at a time when many people even in low end jobs would be looking at double or triple their normal overtime rate, and a lot of those people had long since retired, meanwhile the systems they were being paid to babysit could cost many times what they were being paid in total for every minute of downtime.
It really doesn't take long for a banks system to be down for the cost in lost business and regulatory fines to hit 7 or 8 digits ;)

From what I remember every instance of people being paid "a years salary" was for people being expected to be ready to deal with either life critical systems, or financial systems where downtime costs a fortune, and where the number of people who could troubleshoot a problem and fix it on a live system in a hurry was very limited. I remember reading stories from people involved where there might have only been two or three people alive who had worked on the original code well enough to fix it, and in a funny turn of events they could pretty much name their price to in some instances do things like fly in from another country to spend the time sitting watching what was going on having also reviewed the code and refreshed their knowledge in the preceding weeks or months.

Oddly enough I seem to remember several banks starting to (very slowly) update a lot of their really old* systems to more modern code and native hardware in the years after 2000, it was almost like something had happened to focus the minds of management and the accountants on the age and supportability of the systems their companies relied on.



*There were/are systems that are running basically the same code from the 1970's on emulators where they hadn't been replaced due to the complexity of the systems added to it and the risks involved, as as long as they could run the code faster on new hardware (even with an emulation layer to look like the original hardware) it wasn't deemed a major problem. IIRC one of the biggest outage in UK banking history came when a bank tried to split the computer systems of two of it's subsidiaries and had a booboo.

IT departments habit not keeping applications up-to-date and enough people trained up still goes on.
 
I was still using routines created in quick basic well into this century. They worked, so nobody recoded them.

If was a bit of a ball ache having a non networked windows 95/98 machine to run them in DOS.
 
Last edited:
There's an old saying in business, usually said by IT illiterates "Why do we bother spending so much on IT when everything works".

The Millennium Bug is probably the final form of this stupidity. Something very bad was going to happen so a lot of people worked extremely hard to ensure it didn't happen, were for the most part successful, and afterwards people who didn't understand what was going on complained that the people who prevented the issue had wasted a massive amount of time/money.
 
Still doing something similar with VBA and other such code that won't run in the cloud.

We still have code and applications stuck together with spit and chewing gum. I've been landed with a process that uses flat delimited files to move data between systems. Someone (a team) created this within the last 24 months, it's like I'm back when I started in the 90s.
 
There's an old saying in business, usually said by IT illiterates "Why do we bother spending so much on IT when everything works".

The Millennium Bug is probably the final form of this stupidity. Something very bad was going to happen so a lot of people worked extremely hard to ensure it didn't happen, were for the most part successful, and afterwards people who didn't understand what was going on complained that the people who prevented the issue had wasted a massive amount of time/money.
Yup

I've seen many stories of IT departments being cut back because accounts don't see why they're spending so much money on something that is obviously not needed as they don't have many IT problems...

It's like people who don't see the need to get their car/boiler etc serviced because it's running fine at the moment forgetting that one of the main reasons you have them serviced is to spot/prevent problems before your heating fails on the first cold morning or the brakes fail because the fluid is low or the pads are too worn.
 
There's an old saying in business, usually said by IT illiterates "Why do we bother spending so much on IT when everything works".

I heard similar in IT, except it's 2 sayings:

"Everything is working fine. Why are we paying you this much?"
"Nothing is working properly. Why are we paying you this much?"

Damned if you do, damned if you don't.
 
There's an old saying in business, usually said by IT illiterates "Why do we bother spending so much on IT when everything works".

The Millennium Bug is probably the final form of this stupidity. Something very bad was going to happen so a lot of people worked extremely hard to ensure it didn't happen, were for the most part successful, and afterwards people who didn't understand what was going on complained that the people who prevented the issue had wasted a massive amount of time/money.
100%. One of the systems I was supporting at the time was a funds clearing application with literally billions upon billions passing through it every night, the amount spent on it to make the necessary changes and testing would have been a drop in the ocean compared to what the impact would have been to thousands of companies if that system had failed. But yes the majority did see the whole thing as being a massive waste and I think to this day it's done no favours when it comes to IT budgets in major corporations; IT budgets constantly being cut and without a big doomsday event like Y2K on the horizon to support IT investment its probably going to be a case of luck rather than decent investment that avoids a serious outage.
 
I remember the ancient PCs still working at school after the holidays. And those weird electronic register things they had.
 
I heard similar in IT, except it's 2 sayings:

"Everything is working fine. Why are we paying you this much?"
"Nothing is working properly. Why are we paying you this much?"

Damned if you do, damned if you don't.

Yea and then companies outsourced everything and found out why.

Now they are being back their own IT department as they don't want to wait a week for a tech to show up.
 
Last edited:
I can't confirm or deny that some companies in 1998 and 1999 offered retainers for staff not to resign before mid 2000. I suspect that if that did happen then they may not be able to discuss it for contractual reasons.
 
I remember the ancient PCs still working at school after the holidays. And those weird electronic register things they had.

In many cases it was a nothing burger as the software simply didn't use just two digits for the year. In other cases it was a potentially serious issue but mitigated as there was plenty of awareness of it and the fixes were typically quite small and simple.

End result was that the serious stuff that needed to be fixed was fixed as money and resources went into this issue and there was plenty of awareness of it, stuff that was missed was typically less serious stuff. There had also been some alarmist hype in the press that seems to have carried on well after the problem was already being worked on and people who needed to be aware of it were aware - end result is (because the serious stuff got fixed) some might have been left with the impression that there wasn't a serious problem at all.

Anyway this bakery in Indiana apparently still uses Commodore64 computers, still going strong after the millennium too. :)

tDb7IYM.jpeg
 
Anyway this bakery in Indiana apparently still uses Commodore64 computers, still going strong after the millennium too. :)
It always amuses me to see things like that, usually the story behind it is either they simply don't need anything better, or quite commonly they'll have some equipment hooked up to those ancient machines that uses a custom interface and replacing the computer might mean replacing several million dollars of associated equipment or recreating some software that there isn't something as easy/good to replace it off the shelf :)
IIRC there are a load of places that are still using things like heavy metalworking equipment (milling etc) that rely on 3.5" drives for their interface or even 5.25" ones, to the point where there are people/companies that make drive emulators that plug into the original drive cable and let you select a disc image that is then read exactly like the original disc drive as far as the machine is concerned.

It's like some of the traditional carpet makers are/were* still using the original punchcard pattern interfaces (what the computers evolved from) from the 1800's, often with the original wooden punchcards for the patterns, there was no need to replace the machines that wove the carpets as they are easily repaired and do the job for that sort of traditional high quality carpet, and replacing them would cost millions for something that likely wouldn't last anything like as long.


*I remember watching something about it a while back and some of the machines were something like 100 years old, although probably a bit of triggers broom.
 
Yea you see it in manufacturing. Specialist machinery which was custom made and can't be replaced anymore etc.

It's no problem as it's not connected to anything. Plus these old computers last forever (and are repairable) as back then the components were good quality and weren't made in China. If it was replaced with something modern, you'd be buying new stuff every 5 years because it would go wrong, or be "smart", require and internet connection for licencing etc. and full of security holes.
 
Last edited:
So December 31st 1999.

I was the manager of a high end private hire car firm in central London that only did account work at the time for the likes of JP Morgan / Chase Manhatton Bank , ABN , Lehman Brothers , Bank of America , Godman Sachs and UBS - We did work for all of these banks that night transferring in and out I.T workers.

If I remember correctly we were charging 3x standard rates and 2 of the banks each had 10 cars on standby in their basement at £500 car. All the I.T guts we drove all said they were being paid ridiculously for 48 hours and they all had absolutely zero issues, I'm sure most of the leg work was done prior to that anyway.
 
Last edited:
Yea you see it in manufacturing. Specialist machinery which was custom made and can't be replaced anymore etc.

It's no problem as it's not connected to anything. Plus these old computers last forever (and are repairable) as back then the components were good quality and weren't made in China. If it was replaced with something modern, you'd be buying new stuff every 5 years because it would go wrong, or be "smart", require and internet connection for licencing etc. and full of security holes.

One place I worked was using old hardware of 1990s vintage we used to search eBay for parts. Horrible reliability. You'd lose a power supply and part of the system would limp along for months. Eventually they virtualized it, but only a few years ago. That was so expensive only then they decided to replace those system completely. Madness..

There was also a high profile hack of the Irish health services who had a lot of old systems,


"...The Irish Independent reported today that almost 30,000 computers running Windows 7 software are still in operation within the HSE.

Smyth told members of the Joint Committee on Transport and Communications that systems should not use out of date software “where possible”.

He said all of the HSE Windows 7 machines will be upgraded at some point.

He said around half of the Windows 7 computers are connected to “very large” hardware such as MRI machines or X-ray devices which he said cannot support newer versions of Windows software. ..."


In addition....

"...the National Cyber Security Centre which is responsible for the state's cyber security, had only 25 members of staff, a budget of €5 million a year, no dedicated premises, and that its position of Director had been vacant for a year due to its salary of €89,000 a year..."
 
Last edited:
Yea you see it in manufacturing. Specialist machinery which was custom made and can't be replaced anymore etc.

It's no problem as it's not connected to anything. Plus these old computers last forever (and are repairable) as back then the components were good quality and weren't made in China. If it was replaced with something modern, you'd be buying new stuff every 5 years because it would go wrong, or be "smart", require and internet connection for licencing etc. and full of security holes.
I'm not sure if it's so much that the parts were so much better (I'm sure people will be remembering some of the old computers and the fun you had with them*...), but a lot of them were much more simple and easy to maintain.

Things like CPU's that at most needed a heatsink the size of an old 50p so you had no chance of a failure due to overheating when a fan bearing fails after 3 years of continuous use (or less in a lot of manufacturing environments), and interface cards that might have been fully custom but because they were built to work on an ISA bus using off the shelf parts (except maybe a bit of firmware on a common EEPROM that could be copied off, or replaced if the company kept the file) you could actually pay a specialist company (or even a good hobbyist) to make you up a replacement interface card if need be with them looking at the failed one and reverse engineering it because they could actually see and trace the connections.

One of the problems with wanting more and more performance is that it does come at a cost of far more things that can go wrong, and when they do you're less likely to be able to fix it in a workshop or with normal off the shelf parts.

I watch a few youtube channels of people who restore old machines and it's fun watching them work through issues and in some cases rebuild failed boards, but part of the reason they can usually do that is that if a part fails it's often something they can actually replace with a stock part or it was a bit that was available to the likes of field techs, and the process of replacing it involves maybe a 2 or 4 layer PCB not a 16 layer one (which also means if the board is cracked due to say poor handling or decades of thermal stresses you can potentially bridge the crack to repair the board as opposed to it being junk).


*Poor quality solder joints and design choices made by companies that relied on the fact their customers could do the final QC/build checks at home or bodge up fixes involving blue tack and dropping the machine from a few inches onto a desk...
 
Back
Top Bottom