Recovering from a breach

Oh, fuuuuuuuddggge

Those infamous words were uttered in super slow motion by Ralphie in A Christmas Story.  It was an emotional, knee jerk reaction to an incident, really an accident, that he knew would result in a loss of confidence from his father.  Ralphie was in a pickle, would spend some time, relatively, digging out of it and he knew it.

Those emotions (and verbal reaction) are likely not much different when it comes to finding out that you just had a breach of data protection. A cold sweat comes over you, thousands of thoughts ranging from “why me” to “what did I miss” and “how did this happen” go through your head.  You know you have to do two things:  get the bad guys out and find out what happened.

If you read last week’s blog, you’ll know that there are things you can do beforehand to not only limit your exposure, but also to prepare for this day.  Some of you may be prepared for a breach of data protection, some of you may not. Either way, the purpose of this article is to guide you through some thoughts and considerations on what to do when you find out you’ve been breached, what is going to happen in the investigation process and how to recover from a data breach.

The initial reaction to an IT security breach is to stop it at all costs.  Kick the bad guy out and hopefully keep him out.  Realize though, that some actions taken to kick out the bad guy may result in covering up evidence that will help you determine what happened.  With some of the changes to the disclosure laws, it is no longer valid to say “I don’t know whether data went out the door, therefore I’m not going to disclose what happened to those potentially affected”.  Instead the laws are changing to “if you can’t prove that data didn’t go out the door, you have to disclose to those potentially affected.”  (Standard disclaimer, I’m not an attorney, didn’t even stay at a Holiday Inn Express last night – although my wife has threatened to send me to one on a couple of occasions –, I’m just speaking from my experience and interpretation in reading the changes.)  With those changes, it might be very important to control how you kick the bad guys out so you don’t make it difficult to figure out what they did.

If you are a small shop and your connection to the internet is for email, paying bills and browsing the web (honestly, I was reading ESPN.com to determine whether we can sell our widget at the Super Bowl), your first act may be to pull that plug to your internet connection.  Disconnecting your network from the internet is the quickest way to stop the flow of data.  If your business relies on 24×7 connection to the internet, such as if you are selling goods, this solution may not fit your situation.

If you are pretty sure that it is just one workstation or server, you can disconnect the network cable on that machine leaving everyone else on the network.  Realize though that you have to be pretty sure it is just that one machine and that the bad guy isn’t hiding, silently, on another computer, ready to wake up that zombie after you cut the head off of the first zombie.

The next question is what to do next?  Well, if you have a data breach plan already created, follow that plan.  But if you haven’t, you need to understand that every step you may take may negatively impact the ability for someone to perform the data breach investigation.  Let’s discuss some of those actions:

Turning off the infected computer.  Quite often I see this step taken when one finds a data breach.  My suggestion is not to turn it off.  If you can turn that machine off without it adversely affecting your operations, I would instead recommend unplugging the network cable.  Turning of a computer can wipe away the work that is being done.  It can remove traces of the intrusion which would be necessary to investigate the incident, specifically those traces that are in memory.  If your infected machine is a virtual one, another option instead of turning it off is to suspend it.

Cleaning off the malware.  While tempting, this step can be the worst thing that you can do.  Of course if the infected machine is mission critical, you may not be able to avoid this action.  Realize, though, that removing the evil files means that the forensic analyst hired to determine what happened may not be able to do so.  There are thousands of different types of malware, some even custom made, and each one of them does different things.  The malware binary can be examined to determine what data file(s) and registry keys it touches.  It is in those data files and registry keys that you often find what data went out the door and to where.  Those data file and registry keys are maybe a handful among millions of other ones.  Without the malware binary, finding the relevant registry keys and data files might be more difficult than finding the needle in the hay stack.

The other item to seriously consider is that with each ticking moment relevant evidence needed to perform the post breach investigation could be dissolving away.  Items such as:

Log files:  servers, routers and other devices that log network and other activity have only so much space.  Quite often they are set up with circular logging, meaning that the oldest data is the first to go.  How old is the oldest data?  That depends on how much space you have and how the logging is configured but it can be anywhere from a few months to just a couple of days.

Temporary Files:  computers store all sorts of data in temporary files.  Sometimes it is by plan, sometimes by accident.  The bad guys may not intend to leave traces of what they did in temporary files, but it can happen.  The quicker you can start preserving, the better.

Memory:  capturing information left behind in physical memory and page files can be crucial in determining what happened to your systems.  This data is highly volatile and may only be around for hours, not days.

Once you get past Day 0 in your IT security breach (or maybe just Hour 0) you may want to consider turning the matter over to a forensic team.  That forensic team may be in-house or may be an outside firm.  But either way, they are the ones tasked with combing through the trillions of bytes of data to find those nuggets which will tell the story of what happened.

The team is usually going to start with preservation of data.  That digital data preservation will include servers and workstations.  Careful consideration should also be directed towards preservation of:

•    Firewall logs
•    Event logs from Active Directory servers, whether infected or not
•    Memory of infected computers
•    VPN logs
•    Web server logs
•    Log collecting applications, such as Syslog

The analysis may take place over a period of days, weeks or longer.  However, it is not uncommon for the forensic team to provide updates along the way.  The updates may not only include results of the analysis but also discovery of additional devices that were infected and used in the attack.

When the team provides the results, you have to take them with the perspective of what they are analyzing.  Realize that the goal of the intruder, or hacker, is to get into your network and exploit your data without being found.  Yes, when you delete data it doesn’t go away right away.  But what if there is no data created?  That statement is a little extreme but it was made to prove a point.  The hacker will do everything possible to remain quiet and hidden.  Therefore the forensic analysis may not be a completely clear picture.  Results will be given with percentages and caveats.  The results are greatly affected by how long it was from when the breach occurred to when the data was preserved as well as what may have been done to “clean up” before the forensic team was brought in.

The one question I get a lot from clients is “can you find out who did it”.  Unfortunately, unless it was an inside job, answering the “who” will be very difficult.  The attack may come from overseas or it may come from a computer which itself was compromised. As you peel back the layers of the computer forensic analysis you may find a trail of compromised computers.

I hope that this post has been helpful and that you took away some good nuggets of information on how to handle breaches from the cleanup through the analysis.  Next week we will go over remediation considerations, answering the “how do I keep this from happening again”.

by Greg Kelley, EnCE, DFCP, Chief Technology Officer at Vestige Digital Investigations