Dear NERSC Users, The maintenance work that was started yesterday is taking longer than expected, so we’ve extended the maintenance period through the end of the day (though we hope to return the system sooner). This maintenance included several major changes, such as an upgrade to the Slingshot Host Software to v2.1, a BIOS update for all the CPU nodes, and numerous improvements to the DVS and Lustre File Systems. However, these updates required an upgrade to the internal Perlmuttter management system, consisting of 27 nodes that provide all of the user-facing services like Slurm, DVS, and CPE. Upgrading the management system takes a long time, because each management node must be individually upgraded in a process that takes about an hour to complete in the best case. These nodes are intended to be upgraded in the background while the system is up, but during the preparation for this maintenance, we discovered a misconfiguration that meant that some file systems could become inaccessible during the upgrade of a management node. Because of this misconfiguration, we weren’t able to upgrade as many of the management nodes ahead of time as we’d originally planned, and instead had to move the lengthy process into the maintenance window to avoid disrupting user jobs. We have identified the cause for the issue and believe we should be able to avoid it in the future. There will be one additional disruptive maintenance in early October, and from that point on we expect that the majority of Perlmutter maintenance activities will be capable of being carried out in a rolling fashion, essentially eliminating full outages in most cases. We appreciate your patience as we make these important improvements to system reliability. Regards, -Rebecca -- Rebecca Hartman-Baker, Ph.D User Engagement Group Leader National Energy Research Scientific Computing Center | Berkeley Lab rjhartmanbaker@lbl.gov | phone: (510) 486-4810 fax: (510) 486-6459 Pronouns: she/her/hers _______________________________________________ Users mailing list Users@nersc.gov