Portland technology company Ruby suffered a serious outage on May 8 that left 14,000 of its small business customers without virtual receptionist service for nearly four days.
The outage stemmed from a cyberattack on Hillsboro-based data center operator Opus Interactive, which hosted Ruby's services.
The ransomware attack took out most of Opus’ cloud customers in Hillsboro and Dallas, Texas, said Opus CEO Shannon Hulbert. This included Oregon's campaign finance reporting system. The hosting company worked with customers and specialists to restore service within four days. It doesn’t appear any customer data was compromised.
“We hardened (systems and processes) even further than before,” said Hulbert. She declined to offer details of exactly what the company is doing differently.
"The 26 year old organization worked round the clock with data forensics specialists and restoration consultants in an effort to bring customers back online as safely and quickly as possible. Forensics has shown no evidence of customer data exfiltration," Hulbert said in an email.
Ruby has since migrated its business to Amazon's AWS and is conducting vendor audits to help further protect it, and its customers, from any similar incidents in the future.
“We plan for cyberattacks,” Ruby CEO Kate Winkler wrote on LinkedIn following the incident. “We have fully redundant systems, with automatic failover, geographic separation of environments, system-level and device-level anti-virus monitoring, multi-factor authentication, encryption on all our data, and a detailed cyber policy accompanied by employee cyber security training. We know to immediately shut down vulnerable systems, contact the FBI, and engage cybersecurity and forensics professionals.”
In a follow-up interview, Winkler said there is plenty that it and other businesses can do to help shield from the fallout of an attack. Ruby plans to launch a series of interviews with cybersecurity experts for Ruby customers to help everyone better prepare for disasters, whether they are natural disasters or cyberattacks.
Security is about risks and though for some businesses the risk is low, it's not zero.
“That .01% chance happened,” Winkler said. “That is the (business security) lens we now have.”
Here’s some of Ruby’s big takeaways following the cyberattack on their cloud vendor host:
Monitor, monitor monitor
Ruby runs complex infrastructure so it has internal systems in place that sit outside the company's hosting environment to monitor activity. Ruby employees are paged whenever the system identifies something unusual. That's what happened on May 8. Employees began receiving messages of "system not responding." Ruby's team was able to quickly discern that the problem resided with the host, which allowed Ruby to reach out to Opus Interactive for details.
Communication
Roughly 14,000 small businesses rely on Ruby for virtual receptionist service and other telecommunications services. Communication guidelines are part of Ruby's disaster planning. Winkler said those guidelines include informing customer about what the company can control and what they are doing about it. In the case of the outage, this meant fairly detailed updates on the company’s status page about how Ruby was rebuilding its systems on Amazon Web Services and the timing of that effort.
Have a backup system ready to go
Ruby has not only migrated its business to AWS, it is setting up a backup with Microsoft’s Azure cloud. Ruby works with a number of vendors who each have their own licenses and paperwork. Winkler said getting all vendor documentation in place with any backup hosting service will allow for a quick restoration of service should the primary host have an outage.
Know your vendors
Ruby is auditing its vendors so it can fully understand their systems and infrastructure, and to assess their partners' preparedness and protocols when a cyberattack or other disaster strikes.
Have cybersecurity insurance
Cybersecurity insurance is important, Winkler said, adding that if a cyberattack does occur, companies should contact their insurer immediately. Many insurance companies can provide access to resources to address the issue immediately and forensic teams that can help investigate what went wrong.
