A hack attack that crippled operations at a pharmaceutical company earlier this year likely could have been prevented with proper security procedures around access controls, experts said.
This week, a former employee of Shionogi & Co. Ltd., pleaded guilty to charges that he accessed the company’s network through Wi-Fi at a McDonald’s early this year. He used software -- thought to be a vSphere client -- he had installed before leaving the company. The employee, Jason Cornish, then deleted 15 hosts worth of virtual machines (VMs) -- an estimated 88 in all, which represented most of the Japanese-based company’s U.S. infrastructure, according to an FBI press release.
The attack “effectively froze Shionogi’s operations” for days, leaving employees “unable to ship product, cut checks, or communicate by e-mail,” according to the FBI. As a result, Shionogi lost about $800,000 responding to the attack, conducting damage assessments and restoring the network.
Rule no. 1: Cut access rights
Much of this could have been avoided had proper virtualization security procedures been followed from the get-go, experts said.
For example, Cornish’s access privileges should have been revoked the minute he left the company. Not doing so was “just very bad security practice,” said Jay Weinshenker, owner of Austin, Texas-based Weinshenker Consulting. “Actually revoking his access…seems like a no-brainer.”
“Systems admins have a huge amount of power and I'm not sure if corporate management realizes just how much damage could be done by a disgruntled employee with admin rights to everything within their network,” said Graham Gillies, director of IT at a West Coast law firm. “Quite frankly, I am surprised that there haven't been a lot more instances of a sysadmin utterly destroying a corporation's infrastructure and data to the point where it's a total loss.”
Shionogi reps declined to comment when reached on Thursday.
Locking down the virtual infrastructure
This attack could have been carried out in a physical environment, too, but experts expect to see more such attacks on virtual infrastructures, because they tend to use centralized management.
“It’s not an issue caused by [virtualization], it’s an easier attack because I can now manage everything from one central location,” said Edward Haletky, CEO of The Virtualization Practice LLC. “With a lot more people [who don’t understand] security doing virtualization, we’re finding a prevalence of people putting their consoles and management devices actually out on the Internet, which means they could come under attack at any time, and it’s trivially easy to break inside the management layer of modern hypervisors.”
Other than revoking the former employee’s network access, there are other ways to make virtual environments more secure, Haletky said.
For example, management networks need to be separated from the rest of the infrastructure. Virtualization admins also often forget to include third-party management tools, including backup, in such networks, rather than just vCenter, Haletky said.
“Let’s say I only have access to the backup server; what if I just restored a bunch of stuff on top of that VM? So now you have old data -- it’s just as deadly as deleting it, and probably harder to find,” he said.
Another issue users must grapple with is that while they can take inventory of known assets in the virtual environment, detecting rogue management tools is another story, especially since these tools may lurk inside hosted hypervisors such as VMware Workstation.
Virtualization admins need a means of discovering not only which management interfaces are deployed, but that they are deployed in the right spot, Haletky said. Unfortunately, he doesn't know of any tool for doing this. Instead, "quite frankly, a full audit is required of your systems...to know where everything is."
Finally, multiple management systems which all perform their own discretionary role-based access controls can also create security loopholes. Discretionary access controls mean that subjects and users within the environment are capable of creating and setting permissions, which means if you bypass the application setting the permissions, those permissions do not apply.
Highly secure environments, such as government agencies, use mandatory access controls, a low-level policy-driven means of controlling access to different subsystems below that are used by applications such as VMware vCenter. Mandatory access controls are built into the operating system and therefore cannot be bypassed.
But Haletky said he doesn’t expect mandatory access controls to become mainstream anytime soon within the virtual environment. “It would require a fair amount of rework for the hypervisor vendors VMware, Microsoft, Citrix and Red Hat and their respective management ecosystems and third parties, as they have centralized their access control within their management tools and not the hypervisor or an operating system.”
Beth Pariseau is a senior news writer for SearchServerVirtualization.com. Write to her at email@example.com.