One of my biggest frustrations is the idea of 24/7 SOC, meaning you have employees working around the clock. This frustration goes back to SOC analysts being treated as an entry level position instead of one of the most important security positions in the company. I believe 24/7 SOC can be a thing and can be done right, but if done wrong it can cause SOC turnover and quality goes down. One of my first questions is, why is a 24/7 SOC even needed? When I research answers, I find three major reasons:
- Criminals work around the clock. Organizations can’t afford to allow their network infrastructure to be undefended when the IT personnel goes home after a day of work.
- Shorter response times–analysts can catch the criminals in real time.
- Regulatory compliance and customer trust.
Now, before I go too far down the road for each reason, I know each company has their own reasons for having a 24/7 SOC. Some do it well; I believe most struggle. The Managed Security Service Providers might do this better than most companies just because this is their business. They want to achieve quality and effectiveness, avoid burnout, and not overstaff. I can’t help but read Expel.io blogs and think they’re doing it right. Unfortunately, from my experience I haven’t seen companies do what Expel.io has done in practice. To look back at the reasons above, I’m having a hard time agreeing completely with any of the above reasons, but let’s look at each now.
- “Criminals work around the clock so SOC needs to, too.”
I can’t disagree with this; criminals may work in different time zones so 9 to 5 might be different for somebody across the world. If you only have staff that work 9 to 5 in one time zone, to have coverage beyond that, especially a third shift like midnight to 9, from my standpoint, doesn’t make sense. It doesn’t make sense to me because how are attackers attacking organizations? I believe the most prominent one is phishing emails. Even if an attacker sends a phishing email at 2am, when is the user going to look at it? Probably closer to the 8am mark. From this standpoint maybe having a shift from 6am to 3pm and then 3pm to midnight makes sense. It covers most of the user activity.
- “Shorter response times–analysts can catch the criminals in real time.”
This is what I believe is the biggest disagreement I have. There are several reasons why I don’t believe this is the case.
I find this reason interesting because the dwell time globally is still 24 days (down from 56 days previously but it’s been suggested this is due to ransomware attacks). If you have a 24/7 SOC, how is this possible? Is it because companies just don’t have a way to detect attackers? I think it’s more due to mistakes or assumptions being made.
One of the first things you must consider is sleep deprivation and the ability to function at 100%. Now working third shift you might think that the analyst just needs to sleep during the day and stay up during the night. I can say it’s not for everybody. I worked third shift for almost a year, and I never got used to it. The CDC has stated that sleep deprivation is the equivalent to drinking alcohol. Being awake for 17 hours is the same as BAC .05 and 24 hours being the same as .10. Would you expect your analyst to properly identify a successful attack on your network if they were drunk? If the analyst is in this state and triages this alert wrong, it could have serious consequences for an organization. I’ve personally seen this twice. Once when the analyst had seen a virus alert but dismissed it because the antivirus caught it and another time when the analyst didn’t think there was a risk. In both cases the fresh 1st shift went through the 3rd shift alerts and found the mistakes. So did having eyes on glass help cut the response time? Absolutely not. If nothing else it puts the company at more risk because in essence these alerts were already “worked”.
The second thing about having a 24/7 team is the lack of work or interest. I believe my two examples above were the cause of the analysts not digging deeper on alerts. As the working staff goes home for the day, less and less phishing emails are reported and fewer alerts come in. It’s just how it is. People with an empty queue start to get bored or even uninterested in what they’re doing. Some find this time to be productive with updating playbooks or even studying for a certification. I’ve also heard stories of analysts playing video games and sleeping. You can’t expect somebody to be totally engaged at 2am looking at an empty queue.
Working late shifts is not healthy. I know, personally, that day after day this sleep schedule takes a toll on your body and mind. I’ve seen talented folks leave organizations that require a 3rd shift. I can’t blame them. Why work 3rd shift when you can work 1st or 2nd someplace else and typically make more money? So not only are you putting a huge burden onto your analyst but you could easily lose talent because of this. Also, rotating your staff every three months doesn’t help. This just makes everybody upset at one point or another.
- “Regulatory compliance and customer trust.”
The third reason and the biggest reason why most organizations have a 24/7 SOC is because of the need for regulatory compliance and customer trust. It’s ALL about compliance and not about security. Where things are done not to be secure, but because of compliance. For example an audit may say that you need to log all traffic. So a company will buy a SIEM (Security Information and Event Management), stand it up and proceed to throw every bit of information into it and may or not have somebody look at it full time. If ever audited on it, you indeed can indicate that you’re monitoring all traffic! But are you really? No, because it’s not tuned, the amount of data going in is actually making the SIEM slow and not responsive, and the logs aren’t parsed. You pass the audit, but you don’t stop the hackers. I’ve seen millions of dollars wasted on products that are never maintained or even used. So compliance will tell a company you need people to look at alerts. They immediately think, why not have somebody look at them all the time? Then we can tell customers that “data is protected because we have a team on 24/7.” This will build “customer trust”.
I do believe there are ways to combat some of the issues I stated above. I’ve heard about finding people that enjoy the third shift, having quality control in place to identify issues, or even encouraging threat hunting to build expertise for the later shifts. But, when it comes to a 24/7 SOC my biggest issue really comes down to that third shift or working from 11pm to 8am. Having your employees spread over several different time zones that only consist of 1st and 2nd shift is a suitable answer to me. If a company really “needs” a third shift and has weighed the pros and cons then fine. I feel most companies just do it because they’re told or believe they need to. They don’t truly evaluate what kind of pressure this puts on their people and don’t have a plan to combat their fatigue. At the end of the day I would rather have an alert/fresh analyst with a team around them to analyze an alert than a sleep deprived/uninterested analyst that might triage the alert incorrectly.
Editor: Emily Domedion