ISU had a power outage at about noon today (12 May). Power is back now, but one of the important fileservers is currently recovering. Will update the news item with progress made! Thanks for your patience.
Update 2:45 PM: Most everything is back up, except for the import and near term archive file server. This server remains in a XFS filesystem repair mode and it is unclear when it will finish. I have put a bandaid in place to at least get realtime data flowing again, but we remain in a degraded state :( Thanks for your patience
Update 8:15 PM: Recovery of the main file server continues at a glacial pace... I am optimistic it will be successful, but have no idea when yet. In the meantime, lots of flat file archive remains inaccessible. Will update again with progress in a few hours.
Update 13 May 2 AM: The filesystem check continues...
Update 13 May 8 AM: The filesystem check continues. I have put in place a temporary backup of the archive so at least something exists instead of nearly nothing. This all is totally suboptimal and makes me horribly depressed. Sorry that I have yet to post a daily feature today, just not feeling up to it yet. Now if this XFS filesystem check would ever finish! Once the dust settles, I am going to redo all this to prevent such troubles in the future. Augh!
Update 14 May 5:30 AM: The filesystem check failed last night due to memory exhaustion. I am trying a different trick now and will (im)patiently wait to see if it can work. The problem with just restoring the system from backups is that it would take many weeks to finish, so will try the filesystem repair once again before giving up!
Updated 17 May 2022: The fileserver came back to live this morning and I have mostly completed a sync to get it back up to real-time. Oye, I would like to never have this happen again so will figure out some changes to prevent it in the future.