Ladies and gentlemen: the story you are about to read is true. Only the names have been changed to protect the innocent.
I had a conversation yesterday with a customer who we’ll call Joe. A few months ago, we saved Joe’s bacon when a failing server brought his business to its knees.
Joe has developed a FileMaker application to manage almost all of the data and information that are pertinent to his business. He developed this application twelve years ago in FileMaker Pro 3. Over time, he’s upgraded, extended, fed and watered his FileMaker application – keeping it healthy.
Joe’s a pretty savvy guy. He’s never asked for professional advice with his FileMaker application. He’s also never sought expert counsel regarding his IT infrastructure which consists of a couple Apple Xserves and six to 10 workstations. It’s nothing extravagant, just enough to keep his business moving forward.
Perfect…Until I received a frantic call from Joe.
He explained that his Xserve had crashed. Evidently, a drive was full and Joe couldn’t figure out why. He also could not run his business. He was frozen with no way to generate revenue. Crippled. Dead in the water. You get the picture.
Fortunately, we respond to phone calls like this regularly. Oliver Block spent the better part of that day working with Joe. Through a methodical and well practiced approach to troubleshooting, Oliver located the unlikely culprit. A log file. You know, one of the files on your server that ‘normal people’ never look at. The ones you’ve opened for a second just to see what’s there and found an unimaginably mundane list of repetitive repeating repetition.
This particular log involved the faxing service. Someone probably enabled this this service accidentally, not really really knowing what it did and thinking, “what can it hurt?” The log file showed that no fax had ever been sent. In fact, it showed that no fax line could be detected. It had been attempted to find a fax line millions of times. Each time, a bit of text landed in the log file saying “I tried. It’s not there.” Four or five years later this file had grown to 35 GB. Joe was amazed. He didn’t even know the fax log existed. He didn’t know the fax was being attempted. He wasn’t watching ALL of the log files.
The fact is, even the most cautious, over protective IT guy shouldn’t watch ALL of the log files on server. Knowing which logs to monitor and how to identify a problem in the ones you’re not paying attention to, that’s the trick.
A 35 GB file alone won’t crash most servers. However, combined with an ever expanding FileMaker database housing hundreds, if not thousands, of images there is a clash of the titans.
There is a happy ending to this story. Oliver was able to completely recover Joe’s FileMaker application. Joe is considering re-architecting the system to streamline image handling using 360Works SuperContainer plugin.
The moral(s) of the story:
It’s tough to know what you don’t know. You’ll pay for mistakes so why not pay to make sure there aren’t any?
Technology failure is typically more costly than technology maintenance. What forgotten piece of technology will fail your business?
Your servers were not made by Ron Popeil. They are not ‘Set it and forget it!’ technology. Who’s tending your critical infrastructure?