Wednesday, April 18, 2012

Tools in the Toolbox - Triage

Imagine that you are on an Incident Response team. Imagine that your local Incident Response capabilities have adequate staff and a slowly maturing process. Imagine that your company has multiple remote locations. Imagine that in those remote locations your Incident Response capabilities are not as mature and there is a resource shortage. Imagine that there is a language barrier between local and remote staff. Imagine the slow response time due to incorrect files sent for analysis. Imagine the headaches and frustration of trying to troubleshoot a security incident when you do not have physical access to the box and you have to rely on others to collect the data you need. 

This was not something I had to imagine, it was something that I have been living for awhile now. It was a problem that needed to be changed, and with this tool it has allowed our investigations to become more effective. 

This is my story: 

The initial process that we utilized was very ineffective and time consuming. To get the correct usable data would usually require a business day turnaround, unless multiple machines showed the same indicators.  If multiple machines were impacted we still could spend 4 hours waiting on the data before we could analyze, and then only have a small segment of data to analyze.

The following questions were posed to the team:
  1. How can we rapidly get consistent data across every incident from remote locations?
  2. How can we standardize the collection process across all locations, so primary incident handler receive actionable data?
  3. Can we do this with a free toolset?
  4. Can we customize this based on our needs?
  5. Simplified Interface for data collection?

The team discussed some options on remote collection capabilities but based on the way the internal network is structured limited us in usability. During this time I stumbled across a post by Mike Ahrendt about his capstone project called Triage. According to Mike, the functionality of Triage is:
 The script is designed to perform basic triage commands, as well as acquire evidence automatically on the system.  I designed the script to be ran from a flash drive, but you can really run it from anywhere.  All reports and evidence will be collected in the script directory under an Incident folder with a time stamp ("mm-dd-yy Incident").  

The capabilities of the script promising so I downloaded and started working with the script to see if it could meet our needs.  Initial opinion of Triage showed that it had promise, but there were some issues with application. The biggest concerns I had with Triage was the need to run with Administrator credentials, inability to log actions taken, multiple steps to run the program from the GUI, being prompted for the EULA for a Sysinternal tool, and the folder naming convention for an incident.

While my concerns were there, knowing them allows us to still use the tool as long as we understood the behavior of Triage. I submitted a feature enhancement request to Mike on these concerns and he said that when he had time he would look into them.

We started using Triage locally for our Malware Incident Response, making sure that the tool was useful in what we did. We expected that using Triage would shave off some of our local response time and give us some useful data. We were pleasantly surprised that Triage actually trimmed our response time by 80%, and provided more usable data than we had previously acquired and analyzed. I noticed that my first step in analysis with Triage was to hit the Autorun.txt to find suspicious registry entries, previously my first step was attempting to hit all the known autorun locations in a registry dump file. On most incidents I could confirm an infection within 30 minutes; this was down from the normal 2hrs that it usually took. 

With the capabilities of Triage being proven in our decreased response time we started to analyze the capabilities of allowing the remote analyst to use. The process of deploying this tool to remote analyst required us to document the process required to use it. In testing Triage at a few remote sites we decided on a few code improvements that would need to be implemented for wide scale usage.

The Triage code was then enhanced to handle these improvements. The following were changed:

We removed the GUI from the code because we collected the same data and it was an extra couple of steps that were no longer needed. We changed the naming standards on the Incident folder to the following format ComputerName_Date_Incident, this allowed us to run multiple Triage analysis and store the information in a central place, without renaming every incident file as we upload it. Created a Command Line modification to the registry in order to bypass the EULA that is prompted by one of the Sysinternals tools. With the addition of the registry change I implemented a custom cmd.exe that was detailed in the Malware Analyst Cookbook.

  1. As of this posting, our customization to the Triage application has allowed us to do the following:
  2. It has increased response efficiency by standardizing the collection process of files by remote analyst. 
  3. It has decreased incorrect files being sent to us for analysis.
  4. It has decreased the analysis time when looking for suspicious and unknown autoruns.
  5. It allows for a training tool for new malware analysts and what to look at.
  6. It allows for keeping historic snap shots of suspicious machines that we were unable to find IOC’s on.
  7. Response time has dropped to about 1hr with getting files from remote analysts.

Right before I posted this, I gave Mike a heads up, and he got busy and started cleaning up the code and making some changes, so part of my wish list has been implemented.  If you haven’t taken a look at Triage I would highly recommend it.

Next Steps
  • Getting Robocopy to work in XP. I have the exe, just need to test it. 
  • Have Triage execute on machines when our AntiVirus Solution fires off.
  • Compress and move the Incident file off the host machine and onto a network share.
  • Modify Triage to use an INI file for variables, so that customization can happen without recompiling.
  • Create an Error tracking log.
  • Integrate into RegRipper to parse out Registry Settings.
Currently I am testing the next release of Triage, so expect it to be updated within the next week or so. 

Wednesday, April 11, 2012

BSidesIowa, a Reflection of a crazy idea.

I have lived in Iowa almost 6 years. In that time I have seen Kansas City, Omaha, Chicago and the Twin Cities get some love by various Security Conferences or Incredible training events. I have been envious of the fact but never really did anything about it. I attended my first every security conference, GFIRST, in Nashville last year and fell in love with the idea.

I started to question why we didn’t have and event where we bring the local community together and share ideas, topics and presentations. I started looking at my options and somehow convinced myself that a security conference would be EASY to do. I looked around for anything that might give an idea of how to do it. I settled upon the idea of a Security BSides event.

I submitted my requests and I remember playing phone tag with Jack. When we finally connected Jack told me that there would be days I that I regretted, it would be a thankless job, and unless the first year was successful than the likelihood of convincing sponsors that we could do a second year would be difficult.
I am not saying that there were not days that I wanted to give up on BSides, but somehow everything seemed to click and fall into place. Our Sponsors – Symantec, Checkpoint, IOActive and The Chad Carden group stepped up and covered the basic expenses for the conference. Without the generosity of Iowa State, and the College of Engineering to host the event for free, and cover the cost of staff we wouldn’t have had the shirts.

The door prize sponsors support that we received was incredible. Between NoStarch Press, Wiley Tech, Syngress and Mike Sikorski graciously donating prizes we were able to give away 30 different door prizes. This allowed us to give our attendees a 1 in 3 chance of getting something. With these numbers and the prizes given, I am going to have to keep going big every year for door prizes. I am not sure my future attendees will mind.

Then there were the Speakers. We covered a wide range of topics and hopefully something for everyone. I felt that I took a risk with the keynote and it paid off. We also had to scramble the week before the conference to find someone to fill a slot. Finally we even asked a freshman college student to present. Unfortunately due to organizational duties I missed a lot of the talks, so I will not provide feedback on them. Philip Polstra did a recap on his blog.

Finally we have my volunteers. Without Matt, Baily, Michael, Barry and Kala (sp??) helping the morning of, I am not sure if I would have been able to pull this off. Matt helped me a lot on the front end, with making sure the website working and with pushing myself when I was ready to throw in the towel. Baily kept pushing me to communicate with my volunteers and was irreplaceable with the support provided the day of the event. Michael and Kala helped me with check-ins and setting up the morning of the event which proved very helpful in keeping my Sanity. Barry volunteered to pick up Phil, and provided some support at check-in. Without my volunteers I know that this wouldn’t have ran as smooth as it did.

What Did I learn?

If you build it they will come. When I started organizing this event I was hoping for 30-40 registered guests, and maybe half showing up. I was humbled by the 175 registered guests and 95 attendees.
I learned that Central Iowa has a strong community of InfoSec professionals and students that are interested in sharing ideas.

Organizing a conference is A LOT of work. Setting goals, budgets and having a good core team is very important. The group of volunteers I had on Saturday made everything run smoothly.

Whatever you do someone will not be happy. There were plenty of negative feedback on individual topics, but overall most of it was very positive.

Try to get all Sponsorship funding in hand before the event. I ended up having to wait for a couple of checks. I had to give trust in my sponsors as they gave it to me.

What can I do better?

Breakfast was the normal conference carb and sugar heavy donuts and bagels, with coffee. I really wish we had a selection of fruit, and a wide selection of beverages. With limited funding I had to be frugal.
Breaks between each talks, I ran a very tight ship and had no breaks. This is something I need to plan on next year.

Promotion of the event, this was a grass root promotion. We had limited exposure outside of the Des Moines and ISU Campus areas. Next year will be better.

You need Signage for location and refreshments. If you make it difficult to find the venue or refreshment options people may not enjoy it as much.

What about next year?

BSidesIowa will probably be later in the year. Possibly May. This is because Matt and I will hopefully be graduating and will be busy getting things done in March/April.

We will be in a different location next year, while the Auditorium in Howe Hall was great, we have been informed we will not be able to utilize the space next year.

Hopefully we can span two days. One day filled with talks, and the other day filled with CDC and Training.
We are currently looking at options for a better opportunity for networking to the attendees.

Wider range of door prizes, although I am completely satisfied with what we received, I would love to have some interesting geek items to win.

I have learned a lot about myself and the infosec community that is here. I am glad to be part of it, and I know that with their support we will be able to strengthen the community here.

BSidesIowa 2.0….