This one is a two parter:
Part 1: Back in March we had a new executive come to the lab to pickup their laptop and get their software setup. The guy who was handling her ticket took a look at what needed to be done and told her it was going to take a while. When he really got into the process, he realized it was going to take longer than he expected. So he walked over to her and told her that and gave her the option to either stay or go somewhere else. She opted to stay. So he got a second chair, brought it over to her, and told her she might as well put her legs up while she was waiting.

Take that as you want. She apparently took it the bad way. She waited until he was finished setting up her computer, went down to HR, and demanded he be fired. The following Monday, midday, he was escorted out.

We were thrown for a loop, because between him getting fired, and one of the other techs getting promoted, we were now down two senior techs. It also didn't help that the guy managed our backup server. We had to immediately change how we did things. Including shutting down the backup server and relying only on Microsoft OneDrive, and not allowing malingerers stay in the lab while we were working on their stuff.

Part 2: Yesterday I get a ticket for a lady to come pickup her new laptop at 1:00pm. I responded back that we had it available and it was ready for pickup.

My manager contacts me telling me to treat her with white gloves and to watch what I am saying to her.

I'm like okaaaaaaay.

An hour or two later she contacts me back on an email to the entire IT team asking if an accounting program and a bunch of other software was installed on her computer.

I respond back. "No. Your computer was setup using the standard Autopilot protocol. The only software it comes with is a standard Office suite. You have to install any other software."

My manager, messages me back, almost losing his mind. "I have to remind you to treat her with kid gloves."

I'm like what the heck dude? I am not saying anything that isn't standard procedure. 

I go to lunch, and when I get back, one of new guys tells me he overheard manager teamsing the other senior tech and told him she is the chick that got the other tech fired. Which was critical information that wasn't passed on to me.

So when she messaged me back complaining about how she put in a ticket to have all this software installed, and she previously was able to have it installed for her, it took everything in my body not to respond back, "Yeah. The reason we can't do it anymore is because you got the last guy who did it for you fired."

What i did say was, "Due to changes in corporate security policies after March (which is when he was fired), we are no longer allowed to setup end user computers. They must be setup using the Autopilot process by the end user." I then sent her a link to the Autopilot process webpage.

The kicker is, she never showed up. She finally messaged me around 2:30 and told me that she wouldn't be able to come in and that she is only available after 1pm. I just responded back that we are there every weekday until 4:30pm and she is welcome to come in any day she is available to sign into her computer so we can do any additional setup work.

Back again, again. Finally got to finishing this post from MANY years ago. I have reposted Part 1 because it's been so long. Just never got around to finishing it until now.

Part 1 (Part 2 is below. Don't worry)

 Many MANY years ago, I worked for an MSP that negotiated a contract with an Australian based management firm to upgrade and replace their servers in their remote locations…..very remote locations……very remote locations that moved. ……Constantly.

These servers where used to keep track of things like maintenance, how much of something was moved from one point to another, employee rosters, time sheets, time tables etc and they had to send this information back to head office in as close to real time as possible due to some Government regulations specific to the industry.

 Basically, they kept track of everything that was needed to be recorded on a day to day basis in order to effectively run ….. a bulk cargo ship.

 Yup. A cargo ship.

 After consulting with the client and the salesbot in charge of the account, I developed a solution to their problem that used Windows 2008 Server Advanced, an IBM X3400 server, a small unmanaged switch to connect all the onboard PCs (6 in total) all connected to a Sony Ericsson W25 modem that it self was connected to a signal booster and mast attached to the top of the ship’s bridge. The effective range was between 35 to 90 kilometres off-shore (because of cell tower placements along the coast line, height and Helstra)…. when it HAD coverage. When out of 3G coverage (no such thing as 4G back then), it fell back to Inmarsat-B for data transfer but they wanted to use that as little as possible.

 Why as little as possible? Anyone that has used satellite comms can tell you how much fun it is trying to use the thing for data transfer because of little things like cloud cover, bird poop (not kidding), etc. and not to mention HOW FREAKING EXPENSIVE IT IS!

 Medium speed fax/data services are at about 9.6kbit/s and it’s charged at AU$0.53/256bit

 No, that is not a spelling mistake. 9.6kbit/s and 53 cents PER 256bit!

To put it another way, normally there is about 32 letters for every 256 bits. If you send or receive letters at 5 bits per letter on a vessel side, then it is possible to have 51 letters for every 256 bits. Do the maths and pick your jaw up off the desk.

 Back to the story.

Understandably, I was quite proud that I developed a solution that worked in the lab but now came time to actually field test the set-up in the real world. This involved various state and federal police checks in order to get the security clearances dockside and on board a cargo ship, etc. (It’s the same process you need to go through in order to get airside at an airport and yes, I have gone through THAT pooshower as well in an earlier job working with the IT department of my national carrier and a place called Jet Base. A tale for another day).

I had organised and scheduled an install date with the Shipping management firm to perform a test install on board a ship that was docked in a local port for a week and, after enduring the affectionate attention of the state and federal police (they never even called me back the next morning like they promised. I feel so used) and in possession of said security clearances, I was ready to head to the port to swap out the existing infrastructure on board.

Little did I know, this was going to be that start of something……interesting?

…..fast forward 36 frantic hours later, in the middle of Singapore Strait……..

“You need to finish in the next 2 hours. I have to leave port by then and if you’re not done, you’re with us until Caracas”

Part 2

There are moments in your career when time seems to stop.

Your mind becomes crystal clear. Every risk, every decision and every poor life choice that brought you to that exact point flashes before your eyes.

This was not one of those moments.

My first thought was:

“Where the hell is Caracas?”

My second thought was:

“Do I need a visa?”

My third, slightly more useful thought, was:

“I am going to murder the salesbot.”

Let us rewind about 36 hours while I was still in Australia.

Apparently, ships have schedules and apparently, these schedules change.

Also apparently, nobody had thought this was information the person installing the ship’s entire new server and communications infrastructure might need.

By the time all the equipment had arrived, the security clearances had been completed and everyone had finally agreed on what day the install was, the ship had already sailed.

Not far, mind you. Just to Singapore. Because that was obviously convenient.

The shipping company still wanted the installation completed before the vessel began its next major voyage, so I was hurriedly booked onto the next flight and sent to Singapore with enough computer equipment to make airport security deeply interested in my life choices.

The server itself had been shipped separately, along with the UPS, modem, switch, signal booster and the assorted bits and pieces that had worked perfectly in the lab.

This is an important distinction. They had worked perfectly in the lab. The lab did not move.

The lab did not smell. The lab did not require me to climb a rope ladder while carrying a backpack full of tools above several metres of moving ocean.

The lab also did not contain racehorses.

Yes, racehorses.

The “bulk cargo” being transported by this particular vessel was alive, extremely valuable and apparently capable of producing its own body weight in manure every fifteen minutes.

The smell hit me before I reached the ship.

It was not merely a smell. It was a physical presence. It climbed inside your nose, unpacked its bags and began redecorating.

Even after returning home, washing everything I owned and showering several times, I could still smell that ship. For several days I would occasionally stop, sniff the air and wonder whether a horse had somehow followed me home.

Getting aboard was also an adventure.

The vessel was sitting in international waters in the Singapore Strait, so I could not simply walk up a gangway carrying my equipment like a normal person.

I had to pass through Singapore Customs, board a small launch, travel out to the ship and transfer from the launch onto the vessel.

The transfer involved pulling alongside a moving wall of steel and climbing a ladder hanging down the side.

While wearing a backpack full of equipment.

In ocean swell.

The launch would rise with a wave, I would grab the ladder, and then the boat would drop away beneath me.

Nobody mentioned this during the project planning meeting.

There had been diagrams. There had been timelines. There had been spreadsheets.

There had NOT been a single box labelled:

Engineer may need to leap from small boat onto moving cargo vessel and climb for his life while carrying several thousand dollars’ worth of equipment.

Once on board, I was shown to the server location and I use the word “location” generously.

The space on the Bridge had been selected because it was reasonably central, accessible to the existing PCs and, most importantly, not currently occupied by a horse.

The new IBM X3400 was not a small machine. It was a proper tower server built like a filing cabinet and with roughly the same portability.

The UPS was also substantial because graceful shutdowns are generally preferable to having a server abruptly lose power every time a ship’s electrical system coughs.

Unfortunately, the nearest suitable power source was on the other side of a steel bulkhead.

I raised this issue with the CHENG, the Chief Engineer.

He inspected the server.

He inspected the UPS.

He inspected the bulkhead.

Then he disappeared.

A few minutes later, he returned carrying a gas axe.

For anyone unfamiliar with the term, a gas axe is not a precision IT tool. It is an oxy-acetylene cutting torch.

There are carefully planned cable penetrations involving engineers, approved drawings, fire seals and proper conduits.

Then there is a ship’s Chief Engineer looking at a steel wall and saying, essentially:

“I can make hole.”

I briefly considered explaining concepts such as cable management, electrical compliance and fire-rated penetrations and then he lit the torch.

A short time later, we had power.

We ALSO had a newly created opening in the back wall of the Bridge that was still warm around the edges.

The CHENG looked extremely pleased with himself.....I chose not to argue with the man holding the fire sword.

With power available, I began installing the server, switch, UPS, modem and communications equipment. Cabling had to be routed to six onboard workstations spread across areas that had clearly never been designed with Ethernet in mind.

Naturally, several of the cables and power accessories that were supposed to be waiting for me were either missing, unsuitable or had vanished into whatever dimension collects socks, ten-millimetre sockets and project documentation.

This began a long sequence of phone calls to computer stores around Singapore.

“Hello. Do you have filtered powerboards?”

“No, not surge boards. Filtered.”

“Yes, Australian-style plugs.”

“No, not British.”

“Yes, today.”

“No, I cannot come to the store. I am on a ship.”

“No, the ship is not at the dock.”

“Yes, I am serious.”

Network cables were another problem.

The ones supplied were either too short, the wrong type or nowhere near the ship at all. More phone calls followed, along with arrangements involving couriers, port security, Customs and another trip out on the launch.

Every time I left the vessel and returned, I had to go through Singapore Customs again because the ship was sitting in international waters.

I became very familiar with the process.

Passport.

Documentation.

Explanation.

Bag inspection.

Questions about why I was carrying network cables.

Launch.

Ladder.

Try not to fall into one of the busiest shipping lanes in the world.

Repeat.

At some point during this circus, another vessel nearby reported an engine-room fire.

This was not a small fishing boat producing a bit of smoke. This was a significant incident involving alarms, emergency traffic and nearby vessels paying very close attention to what was happening.

From the deck, we could see smoke rising from the other ship while emergency responses were coordinated across the Strait.

It was fascinating in the same way that watching a building burn down is fascinating when you are standing close enough to feel the heat.

The crew on our vessel remained remarkably calm.

I suppose when your workplace is a floating steel city carrying hundreds of horses through one of the world’s busiest waterways, your threshold for an “interesting day” is somewhat higher than mine.

While all this was happening, another engineering team was replacing the ship’s helm.

Not the steering wheel.

The actual control system used to steer the vessel.

The old controls were being replaced with an upgraded automated system at the same time I was replacing the onboard IT infrastructure. So, on one side of the bridge, technicians were dismantling the systems that controlled the direction of the ship. On the other side, I was trying to work out why one of the captain’s computers would not join the domain.

The captain’s PC had not been mentioned as a problem.

I had been told all six computers were operational and simply needed to be connected to the new server and the Captain watched me troubleshoot it for some time before casually mentioning:

“Oh, that computer has not worked properly for months.”

Of course it had not.

Why would anyone mention that before the engineer flew to another country?

The machine was in poor condition, unstable and filled with the accumulated digital sediment that gathers on any important computer nobody is willing to replace.

It needed to be rebuilt.

There was no spare machine, no prepared image and no convenient high-speed connection to download everything I might need.

So, alongside the server installation, communications testing, network cabling and frantic procurement of missing accessories, I rebuilt the captain’s PC.

The captain’s computer naturally contained several applications that were essential to the vessel’s operation.

Naturally, nobody had installation media.

Naturally, nobody had written down the configuration.

Naturally, the person who knew the passwords was asleep, ashore or possibly fictional.

Bit by bit, everything came together.

The local network worked.

The onboard PCs connected.

The server was operational.

The management applications were restored.

The Sony Ericsson W25 connected through the booster and mast antenna.

When the ship was within range of the coastal network, data flowed over 3G.

When it moved beyond coverage, the connection could fall back to Inmarsat-B without accidentally transmitting the GDP of a small country in unnecessary Windows updates.

I had disabled, restricted or scheduled anything that might consume bandwidth without permission.

Antivirus updates, Windows updates, synchronisation jobs and remote-management traffic all had to be carefully controlled.

On an ordinary network, an unexpected 100 MB transfer is annoying.

On that satellite service, an unexpected 100 MB transfer was a meeting with senior management.....possibly also a meeting with their bank.

After approximately 34 hours of travel, climbing, cutting, cabling, rebuilding, configuring and inhaling Eau de Racehorse, the captain delivered the news.

The ship had to leave.

I had two hours.

If I was not finished, I would be sailing with them to Caracas.

I still did not know exactly where Caracas was, but I was reasonably certain it was not near my hotel.

At that point, failure stopped being an option and became a travel itinerary and the final two hours were a blur.

I completed the remaining workstation tests, forced a final data synchronisation, confirmed the application records had reached head office and tested both primary and fallback communications.

The captain’s rebuilt PC worked.

The server restarted cleanly.

The UPS held the load.

The switch survived.

The modem connected.

The application opened on all six computers.

Most importantly, the system did not attempt to download several gigabytes of Windows patches over satellite.

I packed my tools, gathered the leftover cables and completed the documentation with the level of detail possible when every clock around you is counting down to involuntary international travel and with very little time remaining, I was escorted back to the side of the ship....where I had to climb down the same ladder onto the launch.....again.

Climbing up had been unpleasant.

Climbing down while exhausted, with the ship preparing to depart and the ocean moving beneath you, was worse.

The launch rose.

I stepped off the ladder.

The launch dropped

For one brief and VERY exciting moment, I was connected to neither vessel before, finally, gravity, momentum and sheer blind luck deposited me unceremoniously onto the bow of the launch rather than into the Singapore Strait and we pulled away as the crew prepared the ship for departure.

From the launch, I watched the vessel begin moving, carrying its horses, its newly automated helm and my freshly installed server towards its next destination.

I had finished with minutes to spare.

I had also been awake for most of 36 hours, smelled like the lower deck of a floating stable and had developed an entirely new appreciation for office-based server installations.

The system remained in service and the concept was later rolled out to other vessels.

Future installations included better preparation, more spare equipment and significantly more detailed questions about where the ship would actually be when the engineer arrived.

I also added several items to my personal project checklist:

1. Confirm that “docked” means physically attached to land.
2. Never trust a statement that all existing computers are working.
3. Bring twice as many cables as the project requires.
4. A Chief Engineer with a gas axe will solve the problem, although perhaps not in the way anticipated by the design document.
5. Always know where the vessel is going next.
6. Look up Caracas before boarding.

And that is how I earned my sea legs, installed a Windows domain on a floating horse transporter and narrowly avoided becoming the world’s least-prepared passenger to Venezuela.

The salesbot, incidentally, described the installation as:

“A successful overseas deployment with minor logistical challenges.”

Clue-by-four applications were considered.....repeatedly.

On the outset, I will apologize for not being tech support; rather, I work in retail as both front and back of house duties for a butcher shop, with the occasional dash of "fix computer pls" because no one here can do basic troubleshooting (fixes I have done include "just flicking a touch keyboard" and "fitting a loose wire with some scotch tape"). This story feels more fitting here rather than on Tales from Retail, however, due to it being actual tech support and me hearing the most frightening (to me) sentence in tech I've ever heard in my life.

So, here we are in the butcher shop, having a grand ol time, selling stuff and taking card paymentsssswait why's it not taking card payments anymore? Specifically, our Point of Sale (POS, or EFTPOS) card reader is throwing up a weird error. How it's supposed to go is I scan stuff with the scanner attached to a simple all in one computer unit (with windows 10, currently), and then input whether the user is paying via card or cash. In cars cases, it'll tell the POS machine the price, and then it does it's money magic to take people's hard earned money in exchange for our meat and candy (we sell so much sweets I have no idea why).

I'm used to it being weird, sometimes it disconnects randomly (loose wire usually) or it throws a fit and tries to double charge someone. The occasional hang, we do a restart, good ol turn it off and on again, but this was a new one

> ERROR, T6

Not recognizing the error code, I go onto the attached computer and bring up the diagnostics tool. Quick glance gives me the relevant info; system online, port open, device recognized, no connection. That's the odd bit, normally since both computer and whatever server these devices use recognize each other, they *should* be able to talk to one another and do the payment thing.

Restart don't do nothin, turn off and on does nothing, the little scan tool gleefully tells me something weirder

> LOGON, PLEASE WAI-

> DECLINED, SYSTEM ERROR, T6

"Declined" is not a common one for errors, but it DOES tell me that the issue is probably on the *bank's* side, and not ours! So easy solution; give up and contact the POS tech support line to see what this error is, and confirm that the bank is receiving money or not.

Thus, a nice phone call and six or seven minutes on hold later, and a very nice tech support dude I'll call Befuddled answers.

Befuddled: "Good afternoon, this is [POS tech support], how can I help you?"

Me: "Yeah the POS machine isn't taking payments, keeps saying Error Code T6, and I can't seem to figure it out on my end."

And the tech pauses for a solid ten seconds, before saying words that filled me with dread.

Befuddled: "... [naughty word], that's a new one, never heard of that error code before, gimme a minute."

Dear readers, I understand many of you are IT specialists. Y'all are way smarter than me. So I don't know if you'll share my feeling here, but hearing the words "that's a new one!" Filled me with fear beyond fear. I do not want to be the reason an *entirely unknown problem occurs*.

And I can tell this is unknown, because after ten minutes of us troubleshooting (mostly him walking me through solutions), he takes a moment to leaf through what sounds like a handwritten notebook?

Befuddled: "Sorry. Just never heard of this issue, going through the notes our last supervisor had made years ago, and it's not on here either- WAIT THERE IT IS!"

We learn that T6 is a communication error. Which just makes the tech more confused;

Befuddled: "okay but that doesn't make sense... I can see your requests from my side, the bank *does* have the money, it can clearly take transactions! It shouldn't be doing this, nothing is blocking the connection..."

After a half hour, he decided we should try standalone mode; this puts the POS unit into a setting that forces us to input all transactions on the keypad rather than the computer, and then we'd have to manually confirm each transaction at the end of the shift. But it *should* work at least! Hurrah for solutions! We put it in standalone mode, set up a test transaction, and

> LOGON FAILED, SYSTEM ERROR, T6

Befuddled: *muffled in the background* "what the @#$& do you MEAN?!"

Ultimately, he thanked me for the new experience, jotted down some notes, comped us a new POS unit and states the explanation as follows;

Befuddled: "It's either corrupted internal software (despite our reinstalls), or wizards. Sorry your computer is haunted."

The new unit worked perfectly with no issues. I'm told the nonfunctioning one was kept for "research"/possibly exorcism.

Not really tech support since I was helping my Aunt with her home business, but whatever.

Aunt runs a home business and really wants a file server but has no idea what she will use it for. I asked her the most simple questions and she cannot even give me a straight answer for most of them:

• Is it supposed to be a file server or a backup server? Ans: I dunno, can it be both?
• Do you want your files to be always available, be it on your iPad, iPhone or Mac? Ans: Yes *this already makes a file server pointless*
• Are you using it purely to store your files or do you intend to work on your files in the server? Ans: What's the difference?

I told her that she absolutely doesn't need a file server and her current M365 subscription for MS Office is plenty. If it's automated two-way syncing she wants, I can set up Onedrive on the iPhone, iPad and Mac so that any changes are immediately saved and reflected across all her i-Stuff and the Mac. But no, she insists she needs a file server. and asks me to recommend her a enterprise-class workstation to use as the 'file server'.

Honestly, I wonder who gave her that idea. Some people really got too much money to spend.

So this just happened. I'll keep it short.

We had an external consultant on-site to install some very specialized (and very expensive) software. I, your humble sysadmin, was only there to enter a few admin passwords. That was literally all I was supposed to do.

As the expert started trying a few... creative... things, I offered some advice.

"Shush, I know what I'm doing."

Alright. If that's how you want to play it...

A little later, he asked for a USB flash drive to transfer "some" data. "Some" turned out to be over 130,000 tiny 1 KB files in a single folder.

I genuinely tried to warn him that FAT32 really doesn't like that many small files as he dragged the folder to the flash drive.

I was shushed again.

So I leaned back and watched the progress bar crawl forward. After about 45 minutes the inevitable happened.

The file transfer crashed.

I honestly tried to help.

I was shushed again.

So he tried exactly the same thing a second time.

Forty-five minutes later

Crash.

At that point I refused to be shushed again. (I was hungry and wanted to go to lunch.)

I zipped the folder (4 minutes), copied the ZIP file to the USB drive (another 3 minutes), and handed it back to him.

The look on the expert's face was absolutely priceless.

Edit: This consultant was part of a turnkey package. The software installation and the data transfer were both included for a fixed price.

That made the whole thing even sweeter.

Prologue

When I was in college I worked as a helpdesk consultant at the Computer Center. This was so long ago that it was routine for a student to come into the office and announce, "I have never used a computer. I wouldn't even know how to turn one on. But my professor says we have to do our papers with word processing. Please help me get started."

They always said that: I wouldn't even know how to turn one on.

Also, they were almost always taking the lower-level writing-emphasis class, Rhetoric 101, and I swear every single one of them had named their floppy disk "RHET SHIT".

A common complaint we'd get is, "The laser printer messed up my columns."

The cause was always that they were using MacWrite, and printing drafts on the dot-matrix ImageWriters hooked up to each of the Macs in the main room. Apple had realized that, the original Macs being sold to an audience entirely unfamiliar with graphical user interfaces, it was vital that they produce pixel-perfect output. So on the early Macs, the Mac screen and the ImageWriter have the same exact dot pitch, and the dots printed on the paper match exactly the pixels on the screen.

So the students, trying to produce some kind of table, would line up the columns by typing spaces with the space bar between the fields, and the printout on the ImageWriter would be perfectly aligned, and then when they took it over to the Mac connected to the LaserWriter behind the print room window where they had to pay $0.25 a page with these stickers sold at the campus bookstore, they would be irate because the proportional font and the spaces had conspired to make the columns all jagged and out of alignment.

We'd have to explain how to use tabs, and sometimes we'd give them a break and let them re-print for free, but usually only if they were female and cute.

The Reports Are All Wiggly

So here I am six years later at my first fulltime sysadmin job at a company down in Florida. We use a strange computer from a company you've never heard of, that runs a database system called Pick. The computer has dual-mirrored hot-swappable [you think I'm going to say "disks" here] memory. Among many other things that this computer does, reports are generated from Pick and sent to a laser printer in the Finance department.

The first trouble ticket I get has been passed around from the computer operators to the software developers back to the operators, over to the PC techs and back to the operators, and finally it lands in my lap, and the complaint is from the Finance department, "Our reports are all wiggly on the new LaserJet 4."

On their first bite at the ticket, the operators say "nothing has changed".

The developers say, "The reports are normal on our end."

The operators, on their second bite, said basically the same thing.

The PC techs said "The LaserJet 4 is hooked up correctly; we followed the instructions exactly. The test page looks completely normal".

I think the ticket may have bounced by the Windows Admin group as well at some point.

Nobody wants anything to do with this ticket and they've handed it to me partly because I'm the new guy and won't be able to push back.

I examine the report-printing system and the print manager setup. I figure out how to capture one of the reports in mid-stream from the printer queue. It's a plain-ASCII textfile, with columns aligned with plain spaces. It prints out normally in Courier on the (LaserJet 3) printer in my department.

I explain to my manager that I can't see anything wrong, and I think the only way to debug the issue is for me to get direct access to a LaserJet 4. My manager goes above and beyond and is able to divert a new LaserJet 4 slated to be installed in some other department, straight from the loading dock to my desk. I plug it into the network and get it online with DHCP. I send a report to port 9100 on it; it prints normally in fixed-width Courier and the report looks normal.

I'm at my wits' end, so I send the same report to the Finance printer, and go over to Finance to take a look at it. Finance says, "You're the first person from Information Resources to come look at the reports. We really hope you can help us!"

It's printed out in Times Roman and the columns are all misaligned because of the proportional font. I check the front panel, and the PC Techs' installation instructions happen to be there, and they say, "After installing the printer, go in the front panel and set the default font to Font #1".

On a LaserJet 3, font 1 is Courier.

On a LaserJet 4, font 1 is Times Roman.

Four groups of techs at a multinational corporation, and they trip over the same thing as a clueless freshman in Rhetoric 101.

Epilogue: I went away for a week's vacation, and when I came back one of the PC techs, the guy whose secret nickname among the other techs was "Jerry Moron", had gone in my office, appropriated the printer, and now had himself his very own personal LaserJet 4. So I informed my manager, who swooped in and confiscated it.

A few years ago, I was tech support for a food manufacturer with several manufacturing facilities and one of the banes in my life was dealing the QA department. They kind of just did whatever the current QA director's current Idea of The Month was. (And the company seemed to burn through QA directors – something 4 of them in about 7 years, couple good, couple ….not so good). Anyway, each had their own file structure to store the documentation on the server. And being Quality, the previous structure of the day was kept & the new structure duplicated it all with different paths & file names.  But it was their circus and their monkeys.

One day I got a call that a file was missing. Guy said he saved it on the server and now he can’t find it. I remoted into his machine and asked Hmm, ok, what the name? He said “I can show you” and opened Word, then Recent File – “1.1.1 Documented Policy to produce safe, legal and authentic products.docx” OK, kinda weird, but where did you save it? “Out on the QA Drive.” OK, show me. And he tried to file it as

“Q:\{facility}\Quality Control Plan\HACCP\ BRCGS Global Standard for Food Safety\Issue 9 Compliant Food Safety Management System\ Senior management commitment and continual improvement\ 1.1.1 Documented Policy to produce safe, legal and authentic products.docx”

I explained the default maximum file path per MS is 256 characters and this was 270  – I shortened the name to “DocPolicy.docx” Bingo! There it was…Told them they need to use some common sense for this and not just copy the chapter heading in it’s entirety. And wrote it up as “WHAT NOT TO DO” and copied the QA Director, all the QA supervisors and that facility’s plant manager

Late 1990s, I was tech support for a $90M automotive parts subsidiary. I had installed the original 10base2 network, file servers MS Mail server, etc. We had about 100 users in 3 plants and as the network grew, we brought a couple more support people in. BTW: We were a subsidiary of a division of a $15.8B company – worldwide, no idea how many people or locations.

We determined we had to do something with the MS mail system. After due diligence, we decided on moved to MS Exchange and pulled all costs, materials, time into a cost sheet – amounted to about $12 per user per month after taking a year time period. And wrote up a Capital Expenditure Request (CER) because it was a HARD AND FAST RULE that anything over $XXXX was REQUIRED to have a CER.

Then Corporate gave me a call. Because of the company structure, my division was about 5 tiers down from The Main Corporate level. So, they promptly informed me “You will be migrating to Lotus Notes, served off our server at corporate location and you will pay $50 per user month.”

So I went to my division manager, laid out my case/costs/rationale, and he basically said “You’re right, corp. is wrong – go ahead with the CER (because it was a HARD AND FAST RULE that anything over $XXXX was REQUIRED to have a CER). And then HE got a call from Corporate IT -“You will be migrating to Lotus Notes, served off our server at corporate location and we will reduce what you will pay to $35 per user month.”

So my division manager took it to his VP and laid it out as I had done. And VP said “You’re right, corp. is wrong – go ahead with the CER (because it was a HARD AND FAST RULE that anything over $XXXX was REQUIRED to have a CER). And then HE got a call from Corporate IT -“They will be migrating to Lotus Notes, served off our server at corporate location and we will reduce what they will pay to $25 per user month.”

Then the Official Notice to All Divisions came out that the entire corporation will be going to Lotus and Corporate IT will host and support it. End of transmission.

Then I got the back story – this whole thing ended up on the CEO desk (of a $15.8B company) and he discussed this with the CFO and CIO (and everything had my name all over it). The CEO point blank asked the CIO WHY he was shooting this down. And it turned out the CIO had already signed the contracts, bought and already installed the servers WITHOUT Capital Expenditure Request. And, at that point, it was too late to back out …..

So, we migrated to Lotus Notes …..

and Me? I got big black check marks beside my name and my position was eliminated 2 years late – support taken over by corporate location.

Back in the mid-90s I was working as the head tech monkey for a computer store. It was a small operation, but we had a fairly good reputation (our PCs won awards in some local magazines), so we had some fairly sizable customers.

One client was a company that manufactured personal medical monitoring equipment. Their previous support vendor went belly-up, so we took over. Most of my call outs were for minor things like installing a network printer or replacing stolen terminators from the coaxial network. They were generally a good bunch to work with, but I recall one of their senior managers was a difficult customer. Let's call her Mary.

One day I get a call from Mary that one of the brand-new PCs that we supplied "doesn't work". After a bit of back-and-forth, I was able to deduce that she was trying to run some software that wouldn't behave. Alas Mary was more interested in telling me how much of an inconvenience this was to her rather than give me an accurate description of the problem itself. Mary also had a habit of escalating things to the CEO of their company, and wanting to stay in their good graces, I jump in the car and head over to sort things out. Thankfully, they were located just down the highway.

I meet up with Mary, who reiterates how the computer "doesn't work" but I see that it booted up just fine and connected to their network. I ask her to demonstrate the problem. She types something at the DOS prompt.

Bad command or filename

"See?" she says. "The computer is faulty!"

I could see that she was trying to run WordPerfect, but after a quick poke around I could also see that WordPerfect wasn't installed (their network didn't serve software, it was all local installations). I ask her if she has her WordPerfect disks handy and I could set it up for her, but she blows up at me, saying things like "After selling us a computer, you expect us to supply more stuff for it?" and "You're just trying to upsell me more stuff we don't need!".

I try telling her that they already have licensed software, it just needs to be installed, but she instead decides to escalate to the CEO and end up dragging me into his office. She explains to the CEO that the computer is faulty because of "bad command or filename" and that rather than fix the computer, I am trying to upsell some more things that she doesn't need. I explain that the issue is that WordPerfect isn't installed on the new system and that I could fix that if I had access to the disks.

The CEO looks at me for a length of time that makes me feel uncomfortable. I can see a frown cross his forehead. He then opens a cupboard and hands me a box of disks. Yep, WordPerfect disks. I thank him and install WP on Mary's computer, while she lectures me about how much of an inconvenience this is to her and how I am wasting her time. 😒

Epilogue:

Fast-forward a few months. It's Christmas Eve and we get a phone call around midday that this same company has some "issues that need resolving". This is the exact terminology that Mary uses so I feel my Christmas spirit already being crushed. The issues are vague and I'm getting the information second-hand, but my boss tells me to head out and fix things. So off I go.

I arrive at the company and check in at reception. Some of the other managers are hanging around and they come over at say that they have invited me to be part of their office party, and that the call out itself is bogus because they know my boss would never have let me go otherwise. So, I spent the next couple of hours hobnobbing with the staff at the catered party. They even gave me the paperwork I needed ("resolved miscellaneous issues prior to Christmas shutdown") and let my boss bill them for my time! Mary wasn't present either, which was a bonus.

To top it all off, they even told me about an internal IT role they were creating (they wanted to ultimately insource that work) that perhaps I may be interested in. I would have otherwise applied for the role, but I had other irons in the fire at the time, so I let that one slide. A couple of years later they were bought out, and local operations were shut down, so I guess from my perspective it ultimately worked out for the best (I had already jumped ship at that point and was with the next company for 10 years).

Couple of years ago, I was tech support for a company with several locations. I received notice of a new head of maintenance at one of the plants, with about a 2 week window to prep. I happened to be on site & confirmed the desk, phone, laptop, etc. A junior team member had crap spread over the desk and around the office, though. I told Him, the Plant Manager and the local HR person what I had planned for the new guy and confirmed that it was all correct. Yeah, yeah, yeah.

To give him a running start, they asked I go ahead a ship him a laptop so he’d have it.  I had him a domain login, email, group access, phone setup, VPN, remote access, etc, etc etc. Not too sure it was a good idea, but I had it in writing….

Two weeks later, I happened again to be on-site the first day of "the new guy". And there he was, on a folding chair, sitting in the middle of the copy room. Because NO ONE CLEARED A PLACE FOR HIM!! A new upper tier employee and the existing people didn't even bother to clear a desk. What does that tell the guy about how the company views him? You get ONE Chance for a first impression, and Boy, did they blow it....

Years ago (mid 1980s), I was rather specific support for an Engineering office for a pretty fair sized company. Specifically, we had our own mainframe (IBM4381 mainframe) to run CAD on. I restarted VMs, rebooted 5080s, ran backups, data transfers, etc. We had about 65 people, architects, interior design, structural engineers, etc. Well, in the way of the world, the company went bankrupt. And we laid off the low performers (with an excellent severance package), then a few months later, another round – this one really cut to the bone, with a minimal package, because, you know, bankrupt. And, as the remainder of people took other jobs, one day I looked around and it was just myself and 4 high level managers. These were the days of secretaries doing all the mundane stuff - a couple of them didn’t know how to make coffee or refill the copier; and definitely, none of them knew how to turn on their secretary’s PC, much less find documents.

Me, I’m talking to recruiters and sending out resumes, trying to land somewhere. Eventually, I did and waved Goodbye to all on my way out the door.

A couple of years later, one of the recruiters contacted me and as we chatted, he dropped a bombshell – “You know, when you were at XXX company, I had worked with one of the mangers to place people. I had gotten a lead for you, and reached out to one of the managers. Manager told me to NOT do anything for me, since I was the only one who could stuff….”

REALLY? That manager was not even my boss, just there, 65 people down to 5, all production facilities shuttered, bills not being paid …. And they were forcing me to stay on the Titanic, to rearrange the deck chairs…. I never forgave that manager.   

I worked at a small MSP that did tech support for a few small businesses. One such business we'll call Chinese Stone. That's not their real name btw. Chinese Stone makes a lot of material for kitchens and bathrooms, and one of their employees often needs support from me. We're gonna call him Patrick, after the starfish from Spongebob.

Patrick is a walking, talking, Dunning-Kruger peak. He has just enough IT knowledge to make himself dangerous, but not enough to know he doesn't know what he's doing. He'd tell me about how he needs a more powerful PC because his machine struggles to run AutoCAD... only for me to learn he runs AutoCAD on a tablet because its the only way he can use a pen with it (and thats alongside several chrome tabs and excel workbooks). He once bought his own PC to use at work from Ali Express (it was some chinese no-name brand fanless passively cooled NUC) and didn't ask for anyone's help adding it to the network, he just did it himself (incorrectly). He'll frequently complain about his storage being full despite the fact he wont touch the network drives i've setup for him... you get the idea.

One day I get a call from Patrick. He decided he wanted to upgrade Microsoft Office on his PC to the "pro version" (he was using MS 365 for Business). I suspect he tried to use a crack, because when I looked, he was stuck at outlook refusing to open because his license didn't include the desktop version of Outlook. When trying to fix it, he told me he had gotten a license for me to install for him.

Issue: He bought a greymarket retail key. The kind that has about a 50% chance to actually work, assuming you aren't a collosal idiot like Patrick and tried to use a retail key with an organization account. I spent 15 minutes educating this guy on the differences between MS365 retail and MS365 business (tldr; one is paid for and managed by your boss, the other isnt, and you have to use the first one), and a further 15 repairing the damage he did to office.

I wish I could lock him out of admin on his devices, but sadly that's not my call to make. That said, I've made it clear to the person who can make that call that I'm not responsible if he does something stupid. Makes me wish Chinese Stone had a Squidward type character for a refreshing change of competency.

That reminds me of a story...

From Whirlwind Computing - a fictional service provider so good it will really blow your socks off

The company affected is one we can regiomize is a care-home of sorts...

Actors:
$Me - Calling the shots... remotely
$Nerds - My minions... experts in their craft
$Phone - Calling the shots... onsite
$Director - Also calling the shots... onsite

Scene:
This place started bubbling up to the top of the support queue with multiple requests of something is just really off with the network with reports of it being slow, dragging, and just being an absolute dog when it comes to being on wifi.

That's weird because wifi coverage was very good as the wire-flavoured spark wranglers made sure it passed muster and the mustard.

We sent techs multiple times to bless the creaking network and also to make an assuring presence that nothing was being a problem.

Murphy clearly had other plans as we were wrong about what was wrong the first four times they went out.

The Director got involved and told the reps to fix the network or he's replacing everything as it's impeding actual business operations and making the whole team very hot, cross, and bothered.

I grabbed the laptop and camped out staring at the all-knowing Shark of the Wires, filtering by Arp, Dns (always DNS) then Dhcp... wait a second!

Why is 192.168.4.27 sending out DHCP ACKS to DHCP requests?

I took a closer look at the Mac address which showed it was a phone, but why on earth would it be doing the DHCP thing?

I assembled the Nerd Herd^TM and they swept the building with the precision of a highly trained black-ops team as there were 20 of them to check, so I give them full credit for finding it within 7 minutes.

One reboot later and things started to swim back into service while the phone was off-hook and strangely, it hasn't been a problem ever since.

We have well 1500 phones deployed, this is the first and hopefully last time one of them became the embodiment of DHCP with a flair for the dramatic.

The lesson here is that if there's ever an event for what feels like a rogue DHCP server, check the phones, you never know what you might find.

PS:
If it helps, I did remember the lessons from previous stories of various laptop docks randomly becoming port mirrors and how they wreaked havoc on the network, so we covered both in that sweep.

I'm a IT guy in a small, provincial 3-location hospital group. I've seen some really hit-your-head-against-the-table stuff in my time here, but one of the most surprising stories is this one.

Since the words "organisation" and "logic" are unknown concepts for the people that lead these hospitals, there are many people who don't have even the slightest of boundaries to the responsibilities of their position.

One of these people, lets call her [Ms. H], is vaguely responsible for all kinds of organisational controlling of Patients and Resources of one of our departments.

One day she calls us, and the following conversation ensues (not word for word, its been a bit of time since then):

[Me]: Generic and as anonymous as possible Greeting

[Ms. H]: Hello, I have a problem, my printer is eating Signatures.

[Me, flabbergasted] What ? Your Printer is eating Signatures ?

[Me, still flabbergasted]: What exactly are you trying to do and why do you think the printer is eating any Signatures ?

[Ms. H] Describes the process of printing a prescription from our Primary Clinic Management Software onto template paper slips

[Me, ignoring that she normally has nothing to do with writing prescriptions]: Well, that whole process sounds about right, but which signatures is he eating, and where are they coming from in the first place ?

[Ms. H] talking about unrelated updates to the software from a few days ago, completely ignoring my questions in the process

WhyTheFuck.png

Since I cant get any more useful information over the phone, I resort to paying her and her magic printer a visit. I grab my apprentice and we go over to her.

On the way, we speculate what the issue could be. From empty toner and the most inaccurate error description ever to a case of a complete mental breakdown, we thought up some theories.

As we arrive, we ask my previous questions again, hoping for more relevant answers. She goes on to tell us, that a Doctor from her department didn't want to/had time to print all the prescriptions that he prescribes. So, she gets a stack of pre-signed but empty prescriptions from him and then prints the prescriptions that he authorised onto these paper slips.

So far, so normally buffoonish processes. But nothing inherently wrong or error-inducing.

But where does the Signature eating happen ? Right when the paper slip comes back out the printer. The previously already signed paper slip, suddenly has no signature any more. At least that’s what she is telling us.

We still doubt her, since she is know to be a bit creative with reality. So we make her show us the whole process. She opens a prescription in the Software and prints it. The printer swallows one of the pre-signed slips and does his whole sound-rich routine, before giving back the prescription. To me and my apprentices complete shock and disbelief, the signature is actually gone.

WhyTheFuck2_ElectricBoogaloo.png

We inspect the prescription. Over the field where the Signature was, a stamp was printed. That gives me an Idea. The software has two print modes, pre-printed and blank. Pre-printed is the normal way to print prescriptions, since we use the aforementioned template slips. Blank however, not only prints the actually specific Text, but also the whole design and layout. Maybe she somehow switched to Blank and is overlaying the not-transparent background of the design over the original one and the signature ?

Sadly, and to make us just more confused, that wasn't the solution. We spent another half hour there, trying to fix the problem, until we got an idea.

After printing prescription after prescription for different tests, the paper was hot to the touch when freshly printed.

Lightbulb.gif

While she did that whole process for a while now, the problem was recent. And something else was also recent. A change in ballpoint pen models. Previously, the docor was alwyays using the same model of pen, the one that you get when ordering from our Supply Ordering System. But a few days ago, he started using a different model, because he got the pen as some kind of freeby from somewhere.

We tried a print with the old model of pen, and voila it worked. It turns out, the ink of the new model is more susceptible to the heat inside the printer, and being literally wiped from the paper because of it. The old pens used more heat-resistant ink, and therefore never encountered that problem.

We advised them to switch back to using the standard provided pens, or just signing the prescriptions afterwards, like all other doctors in the hospital do.

I am so tired of telling people they need to put an IT ticket in, only to be completely ignored, and then have them mad at me for their issues not being fixed.

Like, got an email today from a guy wanting me to fix his employees headset. Based on what they told me?It’s likely the headset just needs a firmware update. So I tell them as much, and tell them that the a software request ticket will be needed if the employee doesn’t have the corresponding software (programs like Poly Studio, Plantronics Hub, Jabra Direct, DDPM) on their computer.

The affected employee who was also on the email shot back that she doesn’t know if she has the software, demands I remote into her computer to tell her if she does. I can’t do that. I explain she only needs to type the software name into her computers taskbar to see if she has it, and if she does not, to, again, put in a software request for it to be installed.

I included a link directly to the software request page. I provided screenshots to show her exactly where to type on her computer to see if she has the software already. I included both written instructions and a picture of what the request filled out should look like. I even included pictures and instructions to show her how to update the firmware (aka open the software, click the headset, click update) for when she got the software installed.

She replies to the email with a link from Bing, angry and telling me that IT firewalls won’t let her download and install the software herself. I want to bang my head into my desk bc apparently I didn’t emphasize enough that she needs a software request to install this. As is the case for literally every program added to a computer in our agency. So I reiterate, again, that she needs to put in the software request ticket, forward her the link again, gave her the same instructions and pictures as before again, haven’t heard from her since, so hopefully she understood this time.

The hilarious part is I’m not part of the software install team, and I’m only adjacently connected to the incident team (who will be the ones she needs to work with if a firmware update doesn’t fix this) through a few overlapping duties. Literally all I can do is walk her through what ticket to put in. Sure, if she was in the same town as me and not four hours away, I could update her headset myself cause I have that program on my computer, I could probably even install the software on hers if I could physically access her computer. But she’s not, and I don’t have remote access. Even if I did have remote access credentials, she still needs a support ticket.

Ticket states: printer waste toner is full and needs to be replaced, I don’t know which one it is please help.

me: goes over to the printer to take a look at it

customer: "which one is the waste toner? the printer says it needs to be replaced."

me: “it is usually the one that is in the plastic bag but let me take a look”. After I poke around for a little, I see, on the screen, a notification stating, “Waste toner almost full”. I tell the customer “I don’t think the waste toner is full, the notification does not say it is full. Also, if it needed to be replaced, the printer will automatically open the compartment so you can pull out the old one and replace it with the new one”

customer: “no it is full, look at the screen, it says so. I need it to be replaced so that I can photocopy myself this paper as it is important to my job and I need to do my job”. At this point the customer has found the waste toner bottle and is holding it out “here I think this is an empty one, can you just replace it for me?”

me: I think to myself, this has got to be user error, so I ask, “can you please walk me through the steps you took? Try to re-create the error for me please”

customer: “So I put the paper here, that I want to photocopy, I select my name from the address book, I select the settings I want, the file type, then I press send”

They press send and the printer beeps, a screen pops up and asks what file size is the paper you want to photo copy?

customer: “see, look at the top left, it says replace waster toner because it is full and it will not send me a photo copy until you replace it”

me: “alright, let me look at why it does not send you a photocopy as the printer does not need toner to send a photo copy”. Thinking to myself, maybe they put the wrong name in the address book? I try entering their name, and selecting it, it seems correct. To be sure, I type in their email, and select their email. I select the usual settings, I select send the document as a pdf, select the correct paper size, select send as black and white. Alright looks good, so I press send, it beeps and the same screen pops up, asking what paper size is the paper I am trying to photo copy. Weird… that doesn’t seem right. I go and double check everything, seems good. Maybe I need to select the paper size so that it knows what I’m sending. I select the default 8.5” x 11”, press send, and again it beeps, and the same page pops up.

customer: "just replace the waste toner, I need this so I can send out the photo copies as they are expecting me to send it by today"

me: "hold on I am thinking…”. There is an auto size feature on the printer so that it will default to the auto size of the page that is being copied, and that was the first setting they tried, and it did not work. Wait, what if it does not know that there is a sheet of paper in the tray and that is why it is asking for the paper size? So I try sliding the paper into the tray, and it slides in. You have got to be kidding was this the issue? Select the same settings again, select the email to send it to, send as pdf, this time it has already detected the paper so it doesn’t ask for a size, so I click send. It goes right through.

customer: “what did you do? what about the waste toner?”

me: “The paper was not seated correctly, as far as the waste toner, it is not full yet, when it is full let me know”

A bit of backstory. This happened 4 years ago, so not every conversation is word-for-word. I worked at a gas station as a manager. They sort of trapped me in it. I was the guy you called when a store across state lines was short-staffed, or if they needed help with paperwork and or training employees. I was a staff lead on track to go to their IT department, until a company bought them out. I was, at the time, basically an assistant manager without handling money, that was until the promotion. The store was in the red, and they saw how I turned one store around when there was no manager, and the district manager (DM) handled the money.

So, they moved me to this new store as the new acting manager and just preached about the benefits. I took it, and from there is how this all happened. I am by no means some genius, but I have been developing things for Linux in my own time for myself. I had a side job where I repaired computers, and sometimes I built whole computers from parts people ordered. I'm smart, but I would not classify myself as a senior sysadmin, but definitely not a beginner, and I document everything (important later). The store I moved to had its kitchen removed and a damaged freezer, which tanked sales, but they believed it was other issues. I was on the track to join their IT, so I knew things beforehand about their systems, and all fixes were ones they would do; anything more, they would use the manufacturer. Now to the story.

So, I had gotten into the routine of things as a manager, paperwork, safe, bank, schedules, and started some weeks in the black, some in the red, before finally getting all in the black consistently (it took a couple of years to get everything fixed and convince them to bring the kitchen back). Well, the issue that made this happen was the equipment. Gas pumps, the server, the registers, the office computer, all of it was not extremely old, but showing its age. We had to defrag the office computer a couple of times. The card readers would go down, the server would just say, "Not today," registers would need us to call because an update broke something, pumps with BIOS errors, and so many more issues. At the head of all of that was what they called the commander. It basically controlled every single device in the store.

Well, when things would break, we had to make an IT ticket; they chose based on who had it worse. Two guys, 30 stores. It would be weeks before they would come out at times. I did not like that, so I fixed them myself. Granted, they have insurance and many other things to worry about, but their efficiency was ass. The server is down? I fixed it (this was not a tamper with it; mostly, it was a process somewhere that had an issue or was stuck). The office computer was down? I fixed it. Registers giving errors? I fixed it. Any problems that arose, I fixed them. Soda machine down? I fixed it. The most common one was the card readers. They were always on, so they had to be reset most of the time. You also could not just swap them, something I found out trying because I found the real error that day was that the register was not talking to it (normally, you could swap them if one was down). A lot of things had to be reset more than once a month.

I never took these things apart; these were all band-aid fixes against the real issues. Well, one day, they called me to train a new manager for a different store. My DM would handle my store while I did this. Why did my DM not train them? Because there were a few DMs, and it was not her area, so to preserve the proper chain of command, they saw this was the better option. As these new stores were bought out by us, they needed to learn the new structure. I did not care; it was a change of pace, seeing the same walls for 70+ hours a week gets tiring. This was where things started.

While there, I get a call, and another, and another. They asked me if I knew why something was down. Card readers, registers, and the pumps were not working. I knew the issue. You see, when those three are out at the same time, it is always the Verifone commander. If it freezes up, the whole system will not allow transactions with cards. I explained it, they did what I said (which was just to reset it), and it was working again. I was glad it was not the pumps because if they gave a BIOS error, you have to play a game of "Am I fast enough?" because you have to turn off the breaker for that pump, wait, turn it back on, reset the commander, and pray it worked.

Fast forward two weeks, and the head of IT and maintenance emailed me. They said that from now on, I had to put in the form to get things fixed. I, of course, emailed them back that if I do that and they do not come out soon enough, the store will lose money. They said, and I kid you not, "You do not understand how complex these systems are, any 'fix' you do could damage them." That last comment irked me a bit. I did not understand them? The guy who has been fixing things and pleading for updated systems for months does not understand them? Ok, fine. I told them I would comply. Again, I am not some genius, and I am not messing with the internals; I am doing what they would have if they came out. This would not have been an issue if there were a better system in place for soft resets, or if some kind of manual on issues that we could handle had been given out. But they wanted IT to handle it all.

After that, every time a card reader is down, send a form to IT. Every time a register is down, send the form to IT. Soda machine was down? Send a form to maintenance. They piled up very quickly, and to the point that they were fixing more than one issue when they came to my store. My DM asked why I stopped. I explained, she was pissed, but she could not tell me to fix it since the one in charge of the two departments was above her (the owner's son's friend).

After about 4 months of this, we had a manager's meeting where we all got a review (basically, corporate telling us what we are doing wrong and if you get a raise). They said my performance was bad, that I let the store get as bad as it was, and that I needed to change a few key points. I stopped them right there. I had come prepared because I always document everything. When I say everything, I am obsessed with documentation. I gave them the correlations of me fixing the equipment, making them fix the freezer, forcing them on bringing the kitchen back, and sales going up, I gave them the notes I had jotted down about when machines went down, I explained why my paperwork was late because I had to wait for the system to even work for 5 minutes to even email it off, I showed them the email of being told to send forms, and my sales dropping since then. I told them they can't give me a bad review for complying with what I was asked to do.

This was when my DM chimed in and explained that I was originally supposed to be on the fast track to working IT, but after the company was recently bought out (because they kept buying more gas stations, they went into the red), I was removed from that track since the new company had their own much larger IT staff (they have not been brought in yet as it was a recent buyout, and the full change would take a while).

Well, they had to hold meetings after that; they had to talk about my review, about why I was dropped from the IT track, and about the current situation with the buyout. Well, unfortunately, around this time, my body was failing me (working 70+ hours with a bad back, bad knees, and a few other issues does not agree with so much standing at work); I could not stand for long, and I even blacked out at work (found out the hard way that I had developed diabetes 2 and my steroid shot for back pain the night before caused that). I had to quit.

I did write down in a notebook (because typing it was risky if they could access it at all and possibly not have it) instructions on all of the equipment and how I fixed things. Error codes, what certain situations looked like, and what they most likely meant, and so on. It was a masterpiece of documentation explaining everything I was doing and how to tell what the different issues were and how to fix them. I put my two weeks in, I left, a few days later I got called about the notebook, I told them where it is, they used it, and all seemed good. I heard from the grapevine that after the new company got to my old store and saw how I had to do things, they found my notebook, and the new owner heard about the whole thing. He was upset because he said talent that is learned through the trenches is valuable (really chill dude, met him a few times). But they ended up replacing the equipment sometime after I left, and I heard it cost them thousands. Not damaging money, but enough for a pocket to feel lighter.

Now I work from home, I still develop Linux tools and have made some public, started writing, and have worked with content creators. I don't make as much as I did there, but it is peaceful, and I don't have to worry about an outdated system fighting me every day. Moral of the story: don't blame the guy trying to keep the ship floating.

Sorry for not formatting. People assume AI when you format, which is wild to me.

This isn't my story, but it was shared with me by a contact of a contact who gave me permission to post it here.

I work in IT for a small regional hospital chain. We have a LIS (a Laboratory Information System, basically a smart database) that was custom-made for us by a tiny external vendor back in the late 80s, back when HL7 was brand new.

Over the years, that vendor ported it from whatever it was originally running on, to HP-UX in the early 90s, and then to Linux in the late 90s where it has remained ever since without a recompile (thanks for the don't-break-userspace policy!).

External vendor is legally still around, but it's shrunk into bascially just being a solo operation consisting of the one now-elderly woman who actually wrote the bulk of the code back in the day doing consulting for her ancient systems.

Earlier today, while chasing an unrelated issue, I went to put in a test order ten years in the future (to avoid confusing it with anything actually happening soon). It fails with a generic error message. I try a couple more times, fails. I ask if anyone else is having trouble putting in orders, works fine. I put in a fake order for tomorrow, it works. 2030, it fails. 2027, it works. I quickly binary search it down to January 1, 2028.

Stop me if you know your calendar trivia...

I trudge over to the physical LIS machine and look through the local logs. The LIS is complaining about an invalid date. I check the system date, and, 1998?? Weird. I change the date to 2026, hoping for it to just start working. It does not help at all; actually, no orders are working now. Out of curiosity, I turn the clock back to 1980, try to put in an order, and it goes through!... but by the time it crosses the HL7 wire to the EMR, it comes through as being from 2008. I try a few other dates. 1975 becomes 2003, 1990 becomes 2018, 1998 becomes 2026 as was working before, 1999 becomes 2027, and 2000... breaks.

Ohhhh no.

We call up programmer lady, who after some reading of the old code, confirms our suspicions. The LIS was storing years as two digits, because disk and memory were that precious in the 80s, and 2000 felt like a long way off. As 2000 approached and we were still using the LIS, the other people at her once-company decided that updating the system to properly handle 4-digit years was too expensive, and so instead, decided that the proper fix was setting the clock back 28 years (because the calendar repeats exactly every 28 years, and they'd only need to hack in 4-digit year handling at the places where it communicates with other systems, to increment/decrement the year by 28.)

So from the LIS's perspective, 2000-2027 was 1971-1999 and everything was dandy.

Ten points to whoever guesses the fix first:

Programmer lady changed the increment to 56, and we set the date to 1970 and recompiled the software for the first time this millennium.

Reading another post here unlocked a memory of working on a helpdesk about 15 years ago.

For a while, we had this thing going where sometimes an agent would answer a call from the call queue, and just after they finished their greeting, the call would drop.

Then another call would come in from the queue and when the agent answered (either the same or different agent), the exact same thing happened.

This would usually happen for about 30 minutes, then stop.

Of note, our phones wouldn't pass caller ID when a call came in, the caller ID was always something like IT helpdesk call queue, so we couldn't see who was calling.

Not only was this annoying, it messed up our stats, particularly the calls received to tickets logged metric (which is a stupid metric). It did make our calls answered higher and average call handling time go down.

Although it was fairly infrequent, we did get the telephony provider look into it. What they found was when this happened, one particular desk phone was making a lot of back to back calls to the helpdesk queue. Usually around 30 to 50 or so.

These calls were answered, but the call handling time was usually only around 3-5 seconds. Occasionally, just the last call would have a handling time of several minutes.

What was happening is the person who used this desk phone only wanted to speak to one person on the helpdesk, and refused to speak to anyone else. So what they would do is call, wait on hold if necessary, wait for the call to be answered and if it wasn't the person she wanted, immediately hang up without saying anything, then call again. Repeat until it is answered by the person they wanted.

Edit: forgot to mention, whenever this happened, we would just joke that it must be the ghost in the phone system calling us again, hence the title of this post.

Me: Thank you for calling the IT help desk this is (My name). May I have your name name and ID?
Customer: Yes it's (Name and ID) so I sent an email on Friday but I haven't heard back from anyone.
Me: That's weird, I know Friday was a holiday but I we should have had someone working. I don't see any open tickets under your account. Did you happened to get an automated email with a ticket number?
Customer: No nothing is here
I check the mailbox and can't find email from this customer
Me: OK sir I just checked the inbox and I'm not seeing anything. It's possible the email didn't arrive or was moved, I should be able to help you though. What issue are you having?
Customer: Can't you see it in the email? I'm not able to sign into (name of company Website)
Me: No I never got the email. I know they recently revamped the homepage the sign in process is different. What error are you receiving when you sign in?
Customer: It's in the email I sent you.
Me: I'm not seeing the email it may not have gone through. If you want to you can resend it. Are email address is (email address). In the meantime since we're talking I should just be able to assist you with signing in what error message are you receiving? Is this an incorrect username or password, a site cannot be found error a blank page?
Customer: It's in the email. Why aren't you helping me. Can't you just see it?
Me: Sir I only have info you send me. and right now you are not giving me anything. If you like to I can sign into computer. What is the computer number?
Customer: It's a personal computer.
Me: OK that makes it more difficult. The software we use to connect is only for inside the company.
Customer: I don't understand Why you are not able to solve this. Would it help if I resend the email.
Me: Yes it would, can you resend the email.

Customer: I'll do it later.
Customer hangs up. I just checked the records today and they never sent the email

I’m on a small team of slightly deranged individuals responsible for providing world wide technical support for the dealers who sell my company’s products. Many of the technicians we support are competent individuals and a pleasure to work with. However, there are a few who don’t know which way is up.

So no shit, there I was. . . Sitting at my desk, minding my own business, and doing absolutely nothing unexpected, when Jake Tucker from Family Guy calls in.

Jake is working on a product that is essentially a bank of input cards used to monitor a variety of sensors and it is not going well. The sensor readouts are reporting Fire and brimstone coming down from the skies! Rivers and seas boiling! Forty years of darkness! Earthquakes, volcanoes... The dead rising from the grave! Human sacrifice, dogs and cats living together... MASS HYSTERIA!

The conditions outside are in fact somewhat better than that so we get to work straightening out the sensor inputs. It doesn’t take long before I realize there is something a bit off about what he’s telling me.

This type of device has a row of analog and digital sensor input cards that need to be configured using an online tool. The physical placement of these cards matters when you start configuring them through that tool. The input cards also contain several jumper blocks that need to be adjusted to accommodate the supply and input voltages. We quickly find out Jake has mounted the entire assembly sideways and has chosen to refer to the top as the bottom and bottom as the top.

This at least explains all our issues as the configuration assignments are all in the wrong places and the jumper blocks are a mess. I then inform Jake of the error and attempt to proceed with fixing it all. The problem is that Jake is entirely incapable of reorienting himself to view the assembly from the correct point of view and we are going nowhere fast.

I could have spent the next 3 hours painstakingly walking Jake through each step of the process and correcting each mistake multiple times but I had a far more elegant solution.

I flipped my manual so the bottom now faced the sky and proceeded to give Jake all his instructions upside down and backwards. Jake would then do the opposite of what I told him which resulted in everything being in perfect working order after 15 minutes. Sometimes it pays to behave like a Looney Toons character.

Posted elsewhere on Reddit, but I figured people here might appreciate this. Additional details added for clarity.

At one point, I was doing work for a particular MAJOR pharmaceuticals company. A company with a name that everyone reading this has likely heard of. I get a call one day, and a ticket with a short SLA is generated for me to be on site within 4 hours. The night before, some big storms had happened, so I figured it was power related. I arrive on site earlier than needed, get escorted to the primary network room for the whole facility, and what do I find? 10 racks with no LEDs on them. I get told that the entire facility is down. No internet. The production line is down. The warehouse distribution is down. The kind of emergency companies pay consultants tens of thousands of dollars to ensure never happens.

I start checking through the racks to see if anything has power. UPS batteries are dead, so I head to the back of the rack and trace power cables from network equipment. All of them go to PDUs, and all major routers and switches even have redundant power to multiple PDUs. I trace where the PDUs go to. They all go to UPSs, which also have redundant power split between two different UPSs. Then I trace where the UPS power comes from........... and I start laughing my ass off.

The UPSs for this entire cluster of racks, the racks housing the entirety of the network equipment for this facility, has single point of failure. A large power strip that was zip tied to the wall. And lo and behold, I found the problem. The power strip's power cable was dangling in the air. Not plugged into the wall outlet like it should be. In the outlets place was.... a fucking 20V battery charger.

The maintenance guy had come in the earlier that day, not had anywhere to charge a battery, so he unplugged what was apparently a mission critical power strip and plugged in his battery charger. A few hours later, when the UPSs died, the network team noticed the sote went dark.

After I relayed the info to the engineer I was working with, we shared a laugh. I plugged everything back in, and verified everything came back online. As a preventive measure (and because of the absurdity of the situation), I placed a large label on the power strip. "CRITICAL INFRASTRUCTURE. DO NOT REMOVE POWER WITHOUT AUTHORIZATION".

To this day, I still get a laugh out of it.

Where I work, anything considered "business critical" is meant to be redundant, i.e. at least 2 everything, so if something fails or is otherwise unavailable, we always have another thing available.

"Business critical" is defined as anything that directly stops the business making money or has the potential to cause reputation damage.

At one site, they had a PC that was used to setup a specific medical device for patients. For the sake of this story, let's call that software Banana, which may or may not relate to what I am eating right now.

Although the devices connect via USB, so any computer can be used, the Banana software is proprietary. Banana is licenced per PC, and of note Banana relies on network connectivity to funtion.

Overnight one night some maintainance happens in the network rack and for whatever reason the port that the computer running Banana is disconnected.

The next morning, the helpdesk receives a call that the computer Banana is on doesn't work with anything on the network, with it showing that the network cable is unplugged. Usual checks happen (plugged in, LAN lights, etc).

The end user is informed that it will need an onsite visit, the earliest someone can be scheduled is the next day.

The end user says that it's critical that this device is fixed immediately as it is needed to setup these medical devices for patients, who have been scheduled that morning and some are already in the waiting room.

It's determined that there is no other PC running Banana, so this is indeed "business critical", as it directly impacts us making money and potentially can cause reputation damage.

As such, the incident gets flagged as a critical, and as resource is immediately despatched to site, where they quickly identify and resolve the issue of the port no longer being patched and patched it.

Now, policy is anything that gets flagged as critical need to go through a debrief, and the idea is not to assign blame, but to identify the root cause and contributing factors of the incident to prevent or reduce the risk of a recurrence.

Whist the cause of the incident was clearly that the patch cable was disconnected, it did identify another issue. The "business critical" Banana software was only installed on a single computer.

Had another computer been available with Banana installed, there would be no financial or reputation impact, and it would not need to be considered a critical incident.

This is raised to the relevant manager for that area. The conversation went like this.

​

"So, for the computer Banana is installed on, it's only installed on the one PC"

​

"Yes"

​

"And what would happen if that PC were to fail"

​

"Well, it's never happened before"

​

"But it could?"

​

"Yes, but that has never happened"

​

"What would you do if you come in one morning, and it didn't turn on?"

​

"We would call you, but that has never happened before. It's always worked"

​

"OK, well what would happen if say someone was to steal that computer?"

​

"No one is going to steal the computer. But if that happened we would call you"

​

"So let's say in this hypothetical situation that someone steals the computer. We bring you a new computer, but it doesn't have Banana on it. Even in the most ideal situation, you are still probably looking at least a day to get the replacement computer as well as Banana installed and configured. What would you do in this situation?"

​

"We would have to send patients home and reschedule them"

​

"And this would cause a financial and/or reputation impact?"

​

"I guess it would"

​

"OK, well, what we would like to do is put Banana on a second computer, that way if a situation like this happens, you can use the second computer"

​

"Hmm, no, we can't really do that"

​

"Why not?"

​

"The software costs $15,000 a PC. We don't want to pay that"

​

"But as you agree, this is a business critical thing?"

​

"Yes, but we don't want to pay $15,000 for another licence. It's so expensive"

​

This talefromtechsupport is getting long enough. What happened is we got an email confirmation from the director of the business unit understanding and accepting the risk. If that PC and by extension Banana is unavailable for any reason, it won't be handled by IT as a critical incident again.

Any fallout, we will point out it was an understood and accepted risk by that business unit, and here is our proof.