Eerie Loud Sound Just Shut Down a Bank’s Data Center for 10 Hours
We have had natural calamities stopping Internet and data centers but this is first time that a loud sound completely destroying hard drives and shutting the data center completely for a whopping ten hours.
The eerie incident happened in ING Bank’s main data center located in Bucharest, Romania. The trigger for the incident was a failed fire extinguishing test which set a fire in the data center premises. But what happened next was a rare and shocking phenomenon never witnessed before. People heard a loud sound of inert gas being released and the next thing that happened shocked the data center employees as well as service engineers. The loud sound also destroyed dozens of hard drives running in the data center. It also took the bank’s data center completely offline and now ING Bank is relying solely on its backup data center, located within a couple of miles’ proximity.
“The drill went as designed, but we had collateral damage”, ING’s spokeswoman in Romania told Motherboard’s Andrada Fiscutean. The spokesperson confirmed that the loud noise was due to an inert gas.
The immediate effect of the big bang was that local ING Bank customers were unable to use debit cards and to perform online banking operations on Saturday between 1PM and 11PM. “Our team is investigating the incident,” she told Motherboard.
Usually, data centers hold such mock fire drills to prepare for any unforeseen eventuality. Most data centers use Helium based fire extinguishing systems to protect their sensitive equipment as Helium does not chemically damage electronics, and the gas only slightly decreases the temperature within the data center.
Normally during mock drills, the inert gas stored in cylinders is released at high velocity out of nozzles uniformly spread across the data center. However, ING Bank service engineers released the inert gas at a higher pressure which a loud sound when rapidly expelled through tiny holes. The same effect can be seen when you release air at high pressure from a balloon at home.
The bank monitored the sound and it was very loud. “It was as high as their equipment could monitor, over 130dB”.
Sound also means vibration coming at high velocity and the sensitive data center hard drives were damaged in the incident. The HDD cases started to vibrate, and the vibration was transmitted to the read/write heads, causing them to go off the data tracks.
“The inert gas deployment procedure has severely and surprisingly affected several servers and our storage equipment,” ING said in a press release.
This is the first time that sound has caused hard drives to fail as there have been a very few studies to determine the same. One of the first such experiments was made by engineer Brendan Gregg, in 2008, while he was working for Sun’s Fishworks team. He recorded a video in which he explains how shouting in a data center can result in hard drives malfunction.
The ING Bank incident immediately drew attention of IBM researchers who are also investigating data center sound-related inert gas issues. “[T]he HDD can tolerate less than 1/1,000,000 of an inch offset from the center of the data track—any more than that will halt reads and writes”, experts Brian P. Rawson and Kent C. Green wrote in a paper. “Early disk storage had much greater spacing between data tracks because they held less data, which is a likely reason why this issue was not apparent until recently.”
Siemens also published a white paper a year ago saying that its tests show that “excessive noise can have a negative impact on HDD performance”. Researchers said this negative impact may even begin at levels below 110dB.
