On the evening of Friday, Feb. 15, students looking to get in a bit of work before the start of the weekend were met with a challenge when the network went down for several hours. The issues persisted from 9:20 to 11:20 p.m., during which time wired connections, wireless (eduroam) connections, and telephone service were all lost.
According to Chief Information Technology Officer Joel Cooper, ITS has been working on several persistent issues this semester that have been causing network instability.
“We identified a bug in the network operating system running on some of our core network switches,” said Cooper.
ITS ran emergency maintenance on the network, an “upgrade of the network operating system running on the core switches” in order to address the bug.
The prolonged outage however, was unexpected.
“We expected a 10 to 20 minute loss of Internet connectivity during the process, nothing more,” said Cooper.
Maintenance began at 7:30 p.m., and around two hours later the network went down. ITS followed precautions to ensure the campus was aware of the issues, by emailing students and faculty about the outage and in coordination with PubSafe sending out a message via LiveSafe regarding the issue.
Stories about experiences during the outage have swirled throughout the student body in the last week. Paul Seth ’22 was in the SCCS computer lounge playing DotA 2, a multiplayer online battle arena game in which he is ranked 150th in the Americas competitively, when the network went down at a critical point. “The game started and it seemed like we were in a pretty good spot, but then I saw my hero [player character] stop moving and everything froze in place. I thought it was just one of those brief hiccups that the network experiences occasionally, but it wasn’t.” Seth slowly began to realize that this was a more serious matter than usual.
ITS resumed working with Juniper on the issues on Monday, February 18, and has requested from Juniper a network engineer to be on site at the next such maintenance. ITS has also announced changes to future scheduling of such maintenance. “We’ll make sure the schedule impacts services as little as possible (think: between midnight and 5:00 a.m.). By planning outages in this timeframe, we hope to avoid interrupting significant campus events,” concluded Cooper in the ITS blog post.
“Ten seconds later I alt-tabbed into Google Chrome and attempted to load up xkcd [an online webcomic], my go-to test for network connectivity, and when that didn’t work I became slightly more concerned, so I opened my eduroam-connected phone to load up a YouTube video and when that didn’t work I knew it wasn’t just an isolated issue, that everything was down.”
This was clearly an issue for Seth.
“You can be disconnected from a DotA match for five minutes before you are assessed as ‘abandoned’ and you lose. Your teammates can pause the game, but your opponents can unpause.” Seth said. “Eventually after realizing the cause was lost and eduroam would stay down, I left.”
Bilal Soukouna ’22 was also negatively affected by the outage, in a way that might be more familiar to most students.
“I was working on my semantics homework, I had already finished and all I needed to do was to submit it to Moodle,” Soukouna explained. “Moodle wouldn’t load, and the project was due in two hours, so I began freaking out. I tried using my phone’s hotspot to connect to Moodle but that still didn’t work because the whole server was down. There was no way I could put it onto Moodle.”
Realizing he would not be able to submit the assignment, Bilal emailed his professor.
“It was rough, and it was stressful for no reason. I think that was the second outage that week, it was terrible. I understand an outage every once in a while but for it to happen multiple times a week is kind of insane.”
Chief Information Technology Officer Joel Cooper said that the instability has been caused by new network switches that were added in anticipation of the opening of the BEP. “The devices aren’t getting along perfectly with everything else, and infrequent, random instability, particularly noticeable to wifi users, is the result,” said Cooper.
ITS staff will continue working hard to resolve the issues. Cooper has assured that further work on the network will be conducted during spring break in order to minimize the effect of any possible disturbances.
