I just spent 5 hours today, plus 4 hours yesterday, working on a problem backing up the data on one of our clustered servers. The last 3 of those hours was spent on a WebEx session with the backup software vendor. You have to love
WebEx, even if Cisco owns them now.
Ultimately the problem goes back to the fact that the cluster was misconfigured. Yesterday I was getting grief from the software engineer involved, who, frankly, has the ethics of a four year old: He wants what he wants when he wants it; his story changes to suit the conversation; and nothing is ever his fault. Fun guy to work with when the chips are down, let me tell you.
I got completely fed-up with it this morning, and called him out for it. This, of course, got me an admonishment from my manager. This is where maybe the what-have-you-done-for-me-lately culture will work to my advatage, as we were eventually successful in getting the backup done, so crisis mostly averted.
I say mostly because it turns out that the software we are using doesn't have the capacity to backup a file opened for writing correctly, so we will have to have downtime for the backup. Not a popular option, but the only one supported by the vendor. They've been using this product for 8 years, and I'm supposedly the first person to discover this.
Except that it'll probably go down as my fault, too.