Virtual Machine outage
Incident Report for DiscoverIT @ Discovery Building
Resolved
All services are now online.
Posted Feb 21, 2020 - 14:17 CST
Update
regenbio.morgridge.org, ganglia.discovery.wisc.edu, pages.discovery.wisc.edu, WordPress sites, and Git/Subversion server are back online.
Posted Feb 20, 2020 - 16:47 CST
Update
wid-docs, dataverse, the science festival site, and filelocker are now online as well.
Posted Feb 20, 2020 - 12:44 CST
Update
lists.discovery.wisc.edu, data.tinyearth.wisc.edu, redmine-ext-1, discovery.wisc.edu, and elog-1 are all back up.
Posted Feb 20, 2020 - 10:49 CST
Update
mail.morgridgeinstitute.org is now up, with other services to follow beginning with wid-docs, lists.discovery.wisc.edu, and data.tinyearth.wisc.edu.
Posted Feb 20, 2020 - 10:10 CST
Identified
We're preparing to move the affected VMs to another hosting option and will update soon when that operation is complete

A more detailed list of affected services is below:

3d-csi-shiny
discovery.wisc.edu website
elog-1
filelocker.discovery.wisc.edu
jotools-1
lists.discovery.wisc.edu
mail.morgridgeinstitute.org
merlin-wp WordPress sites
pages.discovery.wisc.edu
redmine-ext-1 external facing redmine sites
regenbio.morgridge.org
data.tinyearth.wisc.edu
vcs-1 git/svn services
wid-docs file storage
wisconsin science festival site
Posted Feb 20, 2020 - 09:51 CST
Investigating
Maintenance on our virtual machine environment exposed underlying problems with the VM cluster, resulting in wider outages. Various services including Morgridge mail and the wid-docs share are affected.
Posted Feb 20, 2020 - 08:06 CST
This incident affected: Communications (Email, Calendar, and Mail Lists), Servers, Hardware, and Software (Virtual Servers, Web and Database Hosting), and Data Storage (Document Storage).