Defender Spotlight Tyler Hudak

Interviewer: Can you tell us about a favorite incident that you responded to?

Tyler: We were responding to a customer being compromised. Their Jenkins server had been compromised, and they were able to determine this through a System Administrator logging in and finding 4-5 copies of Kali Linux. The attacker had literally installed their own operating system!

Interviewer: I know you as a badass reverse engineer. Lately, you’ve mentioned browser history a few times when we’ve talked. Why?

Tyler: There’s a lot of information that’s useful for both external actors and insider threat events. Actors will use time-stomping, and other methods of covering their trails, but for some reason they commonly fail to erase their tracks when using the browser.

For example, in one incident, we looked at the browser history and saw a LOT of activity before the date of the compromise. It appears that the attacker had installed a Chrome plugin which required them to log in through Google. When they did so, it imported their entire browsing history from their profile: every website they’ve ever visited.

It is also extremely helpful to find the root cause of the incident. If it’s a phishing attack, 9 out of 10 times, we’ll find it in the browser history.

Interviewer: How do you communicate with stakeholders and affected parties throughout the incident response process? Does it change?

Tyler: As a consultant, it depends upon how you are brought on. If an insurance provider or external legal firm is how you were brought in, we make sure to include them on the CC line through any emails. Sometimes those companies want to see everything before sending it to the victim organization. Written communication is key in multi-party communication.

When talking to CIOs and CISOs, I’m not going to talk about how to reverse engineer a malware sample. I talk about containment, risk and other business-impacting topics!

It is incredibly important to stick to the facts. When there are early-career people, they will sometimes get in trouble by speculating or thinking out loud. The audience often takes it as fact. One example was when I was responding to: we saw traces of MimiKatz in memory. While I asked another analyst to validate, I had a call with the client. Mistakenly, I mentioned “we saw traces of MimiKatz, the attacks may have used it.” The client ran with it and started rebuilding the Active Directory environment! Turns out the area of memory that the traces of MimiKatz were found in belonged to the customer’s endpoint security product and it was the signature the endpoint security tool used to identify MimiKatz. There was wasted effort in the remediation phase. Whoops

Interviewer: We worked together on an incident response team as internal employees of the company. What is different as a external consultant?

Tyler: You’re not always sure of the communication flow path. Internally, it’s well known through institutional knowledge of how to communicate. The organization structure is inherently known. As a consultant, it’s a struggle to build out the communication flow on the fly. Every lawyer wants something different. We establish an Incident Commander role, in which information should flow through. It is up to the Incident Commander to determine the customer’s unique communication path.

Consultants also get a much wider view of the current threat landscape and variety of attacks. We are continually expected to always be an expert in all the varieties of attacks. In a majority of organizations, internal members are able to go deeper into a specific silo. Threat hunting and insiders know their environment better. Consultants have limited time, limited exposure. Consultants are constrained by the number of hours, and always have to be effective. Internal responders can explore, experiment (threat hunt!).

Interviewer: Are there common issues you see during the remediation phase?

Tyler: How long do we have to go through all the issues?! For example, do your readers know the process their company would take when performing a domain-wide password reset? Service accounts are the biggest issue. Many times, it’s impossible to find the service account owner. Trying to resolve this can take 6-12 months. 

Backups have gotten better, but immutable backups are a necessity. I’m a big fan of offline tape shipped to be stored under a mountain somewhere. AWS S3 is not enough.

Companies have the tendency to want to restore as quickly as possible. I agree with it and that’s the point, but there’s a balance. It’s crucial to understand dwell time (how long the attacker has compromised the organization) so you’re not restoring the actor’s access. We typically recommend restoring into a separate, clean network while the compromised network is isolated. This restores business operations while the investigation can continue.

Tyler: It is incredibly easy to get burnt out. When you have downtime, take the downtime. No one gets into Incident Response for stress. You are going to miss holidays, birthdays, and more. You will work evenings and weekends. Take the downtime! 

Take your vacation, don’t have vacation left over at the end of the year. Have a hobby that’s not tech related (Tyler is big into Role-Playing Games). I’ve found for myself and others that it helps and refreshes you. Spend time with your family. There are very few incidents that are so critical that you can’t step away for a half hour to eat dinner, read a bedtime story to your kids.