Saturday, December 12, 2009

Email Issue - Should Be Resolved

We just learned at 10:30 pm of issues related to incoming e-mail and the network and webservices area were quick to respond to address the issue.  Mail appears to be flowing through at this time.  The mail system for external incoming e-mail appears to have stopped around 11:55 am today.

 

 

6 comments:

Anonymous said...

Incoming mail actually stopped a couple of days ago, not just this morning. Are we going to eventually see messages that vanished Thursday and Friday?
-Will Wittig.

Robert N. Brown (law school) said...

Reports of non-receipt of e-mails were submitted on this blog as early as abt 10:30 AM yesterday, a full 12 hours before you indicate you were aware of the problem. I posted a note and submitted a help desk ticket at abt 6pm last night. I am puzzled why it took until 10:30PM for ITS to discover the problem. An apparent deficiency w/ this system is that there seems no way for ITS users to report a problem other than to post a comment on a topic posted by ITS. Thus those of who tried to reach ITS abt yesterday's e-mail problem had to resort to posting comments in response to a different issue. It would be much better for users to report a problem thru this system rather than being confined to submitted a help desk ticket, particularly since the help desk is unavailable when the system is down. Can you please respond both to how the delay in discovery occurred as well as to whether users can directly report a problem via this blog.

UDM-ITS said...

Blogger is configured to notify us when a comment is posted to the message. We need to investigate with them as to why the notifications stopped coming through. That is why we did not respond to the comments until later in the day when an internal e-mail was sent from the Controller on the matter.

The e-mail system never really stopped working and messages were trickling through; thus, no notification thresholds were reached because of the ongoing activity.

Our logs show messages moving in and out of the system up until 9:30 pm 12/12. At 10:30 pm, it looked like we were only down for the past hour.

The network team investigation showed issues started Saturday morning (and not Thurs or Fri) contrary to others experience.


The issue that started on Saturday morning was related to technical problems with the Load Balancers and the LDAP authentication servers.

Internal e-mail continued to operate just fine and e-mail on the @students.udmercy.edu domain worked just fine as well. Some internet mail passed through for addresses that were in the caching buffers however some would have been queued on the sender's side until the fix was in at 10:30 pm on 12/12. Within hours, all of the queued mail was delivered.

A readjustment to the load balancers is scheduled for Monday to avoid the chances of this particular problem reoccuring.

12 hours out on a weekend at the end of a semester is not a good thing; things could have been worse.

Gilbert Sunghera SJ said...

I sent a help desk ticket on Friday regarding this issue as well. I noticed that the forwarding feature on my Titan Connect was activated (I had suspended forwarding on this account several months prior). I am wondering if this outage may have something to do with the recent request to update our email passwords.
I am still missing all my prior email messages.

UDM-ITS said...

Gilbert, what e-mail to update your password? TC requires a password change every 90 days which you will detect when logging in. If you were responding to an e-mail to submit your user code and password to "save" or "update" or "protect" your account, you may want to change your password one more time. There are a lot of phishing scams that request such info and sometimes those come through the UDM spam filter spoofing as if it were a "webmail" administrator.

I would be concerned with your account since a feature is now on you turned off. We did not turn that on for you.

Denise @ Law Fin. Aid said...

Has the problem of sending e-mails to hotmail addresses been resolved?

Thank you.