There appears to be an issue with access to TitanConnect and the portal/email this morning. The problem looks like it began after midnight. We are locating staff to resolve the issue.
Calls have been made to several staff but no response yet. We will keep on trying to track down the network staff to address this and will update with details as soon as they are known.
IT staff have been onsite for about an hour. The issue is related to redundant cooling systems failing. Facilities personnel have been contacted and management is sending people to look at this within the next 30 minutes.
If they are unable to resolve this, all units will need to be powered down until temps cool.
At this time the temps are 103F in the server room - well above acceptable standards.
Oddly, systems started shutting down around midnight - which means the server room was over 90F at that time.
Excessive heat is a significant problem for technology. Please stay tuned while we await repairs.
Issue somewhat resolved. Everything is back up except the legacy mainframe system. No system operator available to bring up or test.
Apparently, the storm tripped the electricity which tripped the air units. Both air units were off and the heat in the room rose to 90 degrees around midnight last night. Servers and racks started taking themselves down including the management unit which notifies ITS staff immediately of issues. Around 5:08 pm e-mail was back online.
Messages sent during this time would have queued up on the sender side and would be delivered by now.
There is a need to investigate the way the air units are set up such that they don't both go down so easily. It doesn't happen often but even once is more than enough.
Thank you for your patience during this process.
Some of our systems temp thresholds are higher than others and therefore some services remained available throughout this time.
If "Servers and racks started taking themselves down including the management unit which notifies ITS staff immediately of issues." why did it take 15+ hours to have someone onsite to deal with this?
UDM is largely not a 24*7 operation and IT staffing work hours are limited to 7:30-6:00 weekdays. While the expectation that IT services should be available 24*7, the reality is they must be operate on auto-pilot for most of the time. The generosity of the staff that do keep an eye on things after hours and come in on their own time is generous.
If you think there should be 24*7 on-site support, let your representative on the Shared Governance committee know that. There are reps that represent students, faculty and staff.
It's Monday morning 6/21 at 7am. I cannot access (Outlook/Exchange) email. Tried remotely through IE ("HTTP/1.1 503 Service Unavailable"), and on site with Outlook (lower right corner of Outlook status bar says "Disconnected").
I was able get voicemail this am, but messages were all from prior to 6/19.
Exchange mail is now working 100%. One of the 4 data-stores was down, and though the server stayed up during the recent problems, we hadn't noticed the one store was down.
All messages that were queued up should be delivered shortly.
Hello, Same problem as yesterday. No access to OL/Exchange email, outlook states "Disconnected." Same message as yesterday when I try to access through IE (either at home or on-site).
Exchange Mail has a similar problem to this past weekend's problem. About 25% of our Exchange users cannot access their mailboxes. This issue may take considerable time to fix and at this time we have no estimate.
17 comments:
voice mail is down as well
Would it be possible to get an update as to when email will be restored? Thank you.
Calls have been made to several staff but no response yet. We will keep on trying to track down the network staff to address this and will update with details as soon as they are known.
Is this outage going to impact incoming e-mail that was received during down time?
Incoming e-mail may be queued up on the sending servers and once things are back up it should flow in at that time.
IT staff have been onsite for about an hour. The issue is related to redundant cooling systems failing. Facilities personnel have been contacted and management is sending people to look at this within the next 30 minutes.
If they are unable to resolve this, all units will need to be powered down until temps cool.
At this time the temps are 103F in the server room - well above acceptable standards.
Oddly, systems started shutting down around midnight - which means the server room was over 90F at that time.
Excessive heat is a significant problem for technology. Please stay tuned while we await repairs.
The voice-mail system is housed in this room and it is without power at this time due to the heat in the room.
Issue somewhat resolved. Everything is back up except the legacy mainframe system. No system operator available to bring up or test.
Apparently, the storm tripped the electricity which tripped the air units. Both air units were off and the heat in the room rose to 90 degrees around midnight last night. Servers and racks started taking themselves down including the management unit which notifies ITS staff immediately of issues. Around 5:08 pm e-mail was back online.
Messages sent during this time would have queued up on the sender side and would be delivered by now.
There is a need to investigate the way the air units are set up such that they don't both go down so easily. It doesn't happen often but even once is more than enough.
Thank you for your patience during this process.
Some of our systems temp thresholds are higher than others and therefore some services remained available throughout this time.
If "Servers and racks started taking themselves down including the management unit which notifies ITS staff immediately of issues." why did it take 15+ hours to have someone onsite to deal with this?
UDM is largely not a 24*7 operation and IT staffing work hours are limited to 7:30-6:00 weekdays. While the expectation that IT services should be available 24*7, the reality is they must be operate on auto-pilot for most of the time. The generosity of the staff that do keep an eye on things after hours and come in on their own time is generous.
If you think there should be 24*7 on-site support, let your representative on the Shared Governance committee know that. There are reps that represent students, faculty and staff.
Hello,
It's Monday morning 6/21 at 7am. I cannot access (Outlook/Exchange) email. Tried remotely through IE ("HTTP/1.1 503 Service Unavailable"), and on site with Outlook (lower right corner of Outlook status bar says "Disconnected").
I was able get voicemail this am, but messages were all from prior to 6/19.
Exchange mail is now working 100%. One of the 4 data-stores was down, and though the server stayed up during the recent problems, we hadn't noticed the one store was down.
All messages that were queued up should be delivered shortly.
Is the mailborder server still down? I have been unable to access it since Friday.
Based on your comments I just reset mailborder and you can now log in to use it.
Hello,
Same problem as yesterday. No access to OL/Exchange email, outlook states "Disconnected." Same message as yesterday when I try to access through IE (either at home or on-site).
Exchange Mail has a similar problem to this past weekend's problem. About 25% of our Exchange users cannot access their mailboxes. This issue may take considerable time to fix and at this time we have no estimate.
Exchange is fully up again, and we are keeping a close eye on it. Please report any more problems as you encounter them.
Post a Comment