Sat, 31 Jan 2004
Ok, so it goes like this. Crappy old third party SCSI drive fails in the staff mail server. No problem, we'll just replace it. Ok, no spares available in a decent time frame from any local suppliers. (It's an old 18G with an SCA connector). Right, next plan. Lets just remove the partner disk of the failed one from the mirror set, and reduce the size of the mirror. Afterall, The single 75G volume served both Staff and Students until a month ago, so staff wont miss ~18G or so.
Tonight rolls around, when I'm supposed to do this work. Ok, so I try putting a spare 36G into the slot where the 18G failed. BAD idea:
For some reason, it cut power to the entire disk enclosure. Although, there are two enclosures, and half of the mirror set is in each. (4 18's striped in each enclosure, then mirrored.) No problem, after futzing with power in the enclosure, things come back. Good good. Right, New plan. I use the silly Solaris DiskSuite tools to remove the failed submirror from the mirror, and recreate a new stripeset with the 3 good disks. We'll copy to that, then do some fun data shuffling. No problem.
Start rsync. Wait. Wait some more. And some more. Go get food... Right ok, I'm back, and what do I find? but it copying the old student spool, sitting in cyrus/PRE_STUDENTMOVE/ - Mike, you're a dumbass. Lets see if I can get by for the rest of the evening without screwing up too badly again... *sigh*
UPDATE 2004/02/01: Finally...
Still had to wait ages on rsync though....