Reflections of Hurricane Sandy Six Months Later

Just days before Halloween in 2012, the residents of New Jersey faced the high winds, storm surge, and flooding from Hurricane Sandy.  We asked Sheri Prupis from NJEDge to give us an update on how New Jersey’s colleges and universities weathered the storm.  She also provides lessons learned for those of us who were lucky this time, but need to be prepared for the next emergency.

My days, like most of us, are filled with technology-related activities.  I am the Vice President for Academic and Community Engagement for New Jersey’s State Research & Education Network, NJEDge.Net.  Every day I am up to my eyeballs in networking, academic technologies, virtualization, consortium activities, and more.  Even on days when I am away, I check in several times to look at the status of my projects.  I believe in the work that my colleagues and I do, and I wouldn’t have it any other way.

But who would have imagined that six months ago, on October 29, 2012 at 8:30pm, all of it would come to a screeching halt in New Jersey.  Hurricane Sandy hit and hit hard. All our priorities changed.

We knew for a week that the hurricane was coming.  NJEDge members (higher education institutes, K-12, research organizations) began to prepare.

Colleges and universities sent students home, closed their campuses, brought in extra fuel, and

worked with faculty on learning continuity strategies; all of this to keep learning, teaching, and research going, even if not on campus.

New Jersey was hit hard by Hurricane Sandy.  We sustained economic losses up to $300 billion.  Our higher education community by the shore was hit the hardest, but practically each institution had some malfunction.  Luckily for them it was not the NJEDge support system.

With NJEDge’s shared services, cloud services and remote hosting, most members kept their local networks up and their LMS’s running.  NJEDge’s statewide network, which provides Internet and Internet2 performed as designed and never went down. We remain robust.  We never lost connectivity, because redundancy was built in.  Internet access was not a problem, but power and communications on each campus was severely tested.

Lessons Learned
Several lessons were learned for the NJ Higher Education NJEDge.Net members.

Redundancy, Redundancy, Redundancy
Redundancy is not limited to networking:

• Application servers, authentication services, and web services need to be available in more than one location.
• Fuel resources have to be available from multiple vendors.
• Staff must be able to work in multiple modes.  Institutions must pay for air cards in mobile devices so that staff can perform vital tasks.
• Instruction must be available in multiple media (text, DVD, Internet, face-to-face).

Don’t Forget the People
Most important was the lesson to not forget people and their struggles:  so many of our members invited their local communities in as soon as the institution regained power.   Nearly half of the members made accommodations regarding exams, scheduling, and due dates – and still managed to end the semester on time.

Having the network running, having the LMS available simply isn’t enough.  Our faculty, staff, and students were worried about parents, children, and spouses.  They were looking for power, food, and places to live.  A few of them are still in temporary housing.  What was amazing was realizing that we had, at each of our institutions, a working community.  Using social media, our staff, faculty, and students were able to share information and resources and help one another.

The single most important role that NJEDge and our members had was providing a communications channel between individuals looking for resources of all kinds.

Next Steps
Social media was used at many New Jersey institutions to keep students informed of the situation on their campus.  While electricity was out and schools closed for days, smart phones kept communication afloat.

It is heartening to learn that faculty and students from many schools pitched in helping residents find temporary housing and to provide provisions and clothing.  Many of the residents along the shore lost everything.  Several schools created fund drives to help the victims of the storm and many used social media to get the word out and even to solicit contributions.

While we hope never to experience another hurricane as damaging as Sandy, we need to set up contingency measures.  New Jersey state agencies, research and hospital institutions, and the higher education community, along with K-12 districts represent a sizeable population to protect.  We welcome any examples of helpful technology protection colleagues can offer.

Sharing Our Stories
This blog posting is created so that we can share stories within the community, and derive contingency measures that can be put in place for the next super storm.

We reached out to our members asking for their experiences.  What follows are their stories.

Brookdale Community College
NJEDge and Brookdale Community College were in frequent contact working to keep mission-critical services up during the storm and its aftermath. In particular, Brookdale’s website was all-important conduit between college officials and our students, faculty, and staff.  Through the website we were disseminating information about when each campus would re-open and providing information about help and resources to members of the campus community affected by the storm. It was imperative that it remain available.

Our first call to NJEDge was Monday night at 11:30 PM during the worst part of the storm. All of us were at home without electricity when we received notice from our monitoring service that the college website was down. As damaging winds battered our homes, an NJEDge employee came to our aid to help orchestrate the changes needed to switch our website to Brookdale’s cloud-based backup site in Chicago.

The next day, the Office of Information Technology staff drove around downed trees and dangling power lines to arrive on campus, at which time we were able to find and correct a minor problem with our generator. As soon as we had brought our Lincroft data center back online, NJEDge assisted with DNS changes needed to restore service. Unfortunately, our return to the Internet was short lived, as twelve hours later our generator failed again, this time burning up an electrical part that was on backorder and unavailable anywhere in the country. Again, NJEDge quickly, and cheerfully, orchestrated the communication with Cogent which was needed to bring up our emergency site.

Although everyone was frustrated by the slow pace of electrical service restoration, many of in the college community remarked that they were pleased that Brookdale’s emergency website was available and up-to-date.

Fairleigh Dickinson University
Fairleigh Dickinson University (FDU) included in its preparation for Hurricane Sandy the rental of four large diesel generators and three 7,500 gallon trucks with diesel fuel to service the Student and Recreation Centers on each campus.  These generators were in addition to the permanently installed units that service the data centers and other select building life safety systems.  However, Hurricane Sandy prevented the normal delivery from FDU’s supplier as the supplier was flooded from the storm.  The University opted to not utilize the prearranged fuel as it had potentially been contaminated by brackish water during the flood.   In order to keep operations on keel, FDU arranged for local diesel suppliers to purchase out-of-state fuel.

The out-of-state supply was not ample to service the generator need.  As the Metropolitan Campus burns low sulfur diesel fuel in lieu of #2 fuel oil in its oil boilers, for emissions compliance, approximately 15,000 gallons was redeployed from the 20,000 gallon tank from one of the lecture halls to the generators on both campuses.

The College at Florham campus was out of power for nine days and the Metropolitan campus for seven days.  This was further complicated by students opting to stay on-campus instead of returning home.

FDU was well prepared for the storm and kept strategic buildings and services open with the generators and creative fueling. What we learned from the experience is to have multiple fuel sources with multiple depot availability.

Richard Stockton University of New Jersey
Stockton was closed Sunday-to-Tuesday during Super Storm Sandy. Monday was an instructional day. Tuesday was an advising day. Faculty had to make alternative plans for contacting their students. Some used telephone, email, Wimba Classroom, or set up appointments later in the week.

Approximately 200 students and employees were hard hit by the storm, many losing their homes. Assistance funds were set up with students and employees being very generous in their contributions.

From a technology standpoint, our data center is equipped with a UPS (uninterupted power supply) and generator. Even though the campus lost power our systems remained up throughout the storm. Power outages in the region were localized. Those who were not affected by the storm surge and power outages were able to continue with online work as usual. Student and faculty with the Blackboard Mobile Learn app were able to access the learning management system using smart phones and tablets.

Richard Stockton maintained some measures at working “against the tide.”

• Sending the residential students home was a good idea and helped ease the burden of feeding and housing the small number remaining.  Had we not sent the students home, it would have been necessary to feed 2,800 students for the duration of the storm.
• Since the storm Physical Plant has purchased 150 cots in the event a situation like this occurs again. This will alleviate the need for personnel to move mattresses to the Campus Center, which became very labor intensive.
• The need for additional emergency generator power became evident.  Although power was lost only for a short period of time, any longer would have become problematic.  The President has asked Administration and Finance to submit additional generator information and cost to him for consideration.

Georgian Court University
Classes were cancelled as soon as news of the magnitude of the storm was forecasted.  Georgian Court  (GCU) shut down its homepage, activated campus security by mass notification to students through all the emergency methods by texting, using the university’s Facebook and Twitter accounts as communication center.

When the campus lost power, our core system went down. Bringing it back up took a few hours, and this happened twice in the days immediately after Hurricane Sandy. Each time it went down, it took time to bring the servers back up in the right sequence, thus affecting a number of operations (e-mail, campus Internet access, ANGEL, the GCU portal). After the second loss of power, the university turned to its portable generators to power the bunker where our servers are kept.

For the IT staff, it was also difficult to determine how well our communications were getting through to members of the campus community.  We didn’t know what other systems (home phones, e-mail, cell phones) were available to our faculty, staff, and students.

With the hurricane directly overhead, the GCU campus lost power.  After four days of continual operation the Data Center generator ran out of Diesel fuel. That’s when the real IT work began. Once fuel arrived, the staff were tasked with bringing all services back up ASAP, which took over 6 hours. The next hurdle came three days later (one full week with no power to campus) when the generator started to spit and sputter requiring immediate maintenance. Luckily the IT staff had enough time to shut all the servers down properly. Shortly after shutdown, the “all clear” was given and the IT staff began the startup ritual for bringing services online. This time only the essential servers and services were brought up. These included Active Directory, Exchange, and SharePoint, each to support communications of staff, faculty, and students. Crucial data systems such as the File servers and ERP system were left in the safer off-state site until campus power was fully restored a few days later.

We saw definite benefits during the outage by having our main web site and student LMS hosted off-campus. For those that remained on-campus during the storm, they were challenged with outages to the IP surveillance system, the IP-based building card access system, and our small deployment of VOIP sets. It came down to power, whereas the GCU buildings with generators and utility power (there were a couple) had network connectivity, wireless, and telecommunications.

William Paterson University
William Patterson University survived the storm well. The damage was mostly from fallen trees. The University had well-defined disaster plans and a standing committee, which immediately went into action.

From a technical standpoint, the WPU Data Center ran on the natural gas generator power so all computer and network systems remained operational. One of our Internet circuits went down for a few hours affecting external telephone services to campus, but all Internet traffic routed to a secondary circuit did not lose data connectivity. We had a minor disruption of on-campus telephone service because an older generator supporting the telephone equipment did not start-up automatically when the power failed. Cell service was acceptable and campus radio communications were not disrupted, so there were no serious communication problems as a result of the loss of telephone service.

Our alert system, ConnectEd worked fine. as well as our web site postings and WPU e-mail announcements. Not all network switches and routers are on generators, so in those buildings, access to the campus network was unavailable until power was restored.  We have plans to have all network equipment on generators and move to complete VoIP which would provide better communications resiliency.

New Jersey Institute of Technology
In the final days of October, it became clear that the NJIT campus in Newark was in the path of Hurricane Sandy. For the IT team at NJIT, preparing for one of the most disruptive storms in recent history meant being ready for a loss of network connectivity, a prolonged power outage that could last days and even possible water damage to their data center environment. They decided that it was necessary to take proactive measures to ensure the availability of critical IT services. Deemed the most important piece to protect was the university’s web presence.

“The NJIT website is the face of the university and all of its college websites,” explained Kevin Byron, director of core systems for NJIT. “It is the primary communication tool among all members of the NJIT community. To have the website up [during a natural disaster like Sandy] was critical.”

The NJIT team decided that the best way to ensure the availability of the university’s website was to relocate their web presence. They sought a solution they could implement quickly with the storm approaching. They needed to act. And they needed to act soon.

Byron and his colleague, Matt Hoskins, senior enterprise architect for NJIT, reached out to NJEDge.Net, the statewide, member-focused provider of network and related services and a trusted strategic partner to the higher education community in New Jersey.

Critical Communication
On Monday, October 29, as Sandy reached the coast of New Jersey and moved inland, New Jersey Institute of Technology was prepared. Even with a loss of power in the NJIT data center environment, there was no lag, no dead air on their website. And they were able to maintain their web presence throughout the duration of the storm and its aftermath. This proved to make all the difference for NJIT students, faculty, and staff.

The NJIT campus lost power for approximately 22½ hours, beginning Monday night and continuing into Tuesday evening. And while power was restored, Newark and its surrounding areas were in such disarray from the storm that classes were cancelled for five days and the campus was closed until the morning of Saturday, November 3.

During the storm, NJIT staff were able to communicate campus-wide status updates via a text ribbon at the top of the NJIT webpage. These updates included class cancellations as well as services availability and restorations.

“That text ribbon had a 200 character maximum, but during the storm those were the most important 200 characters on the entire site,” Byron said.

The updates included a link to the NJIT SOS Blog, which contained more detail than could be communicated effectively through the text ribbon. NJIT SOS is a WordPress blog, so the infrastructure was logically distinct from the NJIT campus and available.

In the days following the power outage on campus, the number of hits on the NJIT SOS Blog skyrocketed. The ribbon and NJIT webpage were clearly important to a great number of people. Byron and Hoskins observed that most users who accessed the blog and website during the storm did so using smartphones.

A Sound Solution

On Friday, November 2, pleased with the restoration on campus, the NJIT website was returned to its primary location on the NJIT campus datacenter environment in Newark. All agreed that it was a successful undertaking and were pleased with the ease with which it was accomplished.

New Jersey Homeland Security
NJEDge played a part in keeping communication fluid between the Office of the Governor with national and state officials throughout Hurricane Sandy.  Through our virtual colocation and redundancy abilities, we were able to provide Vidyo videoconferencing telepresence to the different departments– the Office of Homeland Security and Preparedness, the New Jersey State Police, Regional Operations Intelligence Center, Office of Emergency Management and the National Guard – to be informed in real time.

We welcome any examples of ways to protect students, faculty, staff, and technology that colleagues can offer.

Sheri

Sheri Prupis
Vice President for Community and Academic Initiatives
NJEDge.Net
Sheri.Prupis@NJEDGE.NET

Copyrighted photo used with permission of Richard Stockton College of New Jersey:
http://www.flickr.com/photos/stockton_edu/8168506558/in/set-72157631964433643