It’s an everyday morning in the office of Jim, the CTO of a promising start-up. He’s reached the last few minutes of his usual 30-minute routine to ease into another hectic day, which, as always, is a quick scan of the news, both general and around the tech start-up scene. One particular headline catches his attention. One of his company’s competitors has fallen victim to a cybersecurity attack and it looks as though the result might mean it’s the end of the road for them.
While Jim is by no means a vindictive individual, it’s dog-eat-dog out there in Start-upland and he allows himself a little grin. Plus, the competitor’s co-founders had shown open hostility to Jim and his company. Sure, they were competitors but they’d clearly never heard of friendly or even respectful competition.
But before Jim was able to luxuriate in the downfall of a nemesis, his smile turned to the brow crease of concern. “What if we’re attacked”? Yes, we’ve got a great head of cyber security in Bill, who takes care of the company’s infrastructure and runs occasional penetration tests. “But are these tests enough when we have weekly deployments”? The fact is they are often overdue and lag new deployments, potentially leaving a window of vulnerability.
Jim’s situation is common in a DevOps environment where security departments often play second fiddle during the development and deployment process. The kind of watertight security procedures and documentation that DecSecOps involves can be considered a bottleneck to the agile principles, DevOps, QA and rapidly going to market.
Which, as Jim’s competitors have seen to their cost, can be a mentality with disastrous consequences if a successful hacker attack IS launched. The reality is a well-integrated DevOps security process needn’t slow things down and have a negligible impact on budget when compared to the cost of damage control after the event.
The rest of Jim’s story will demonstrate how a DevOps manager can efficiently implement continuous security standards at little additional cost.
Let’s see how DevOps becomes DevSecOps.
The first step in turning DevOps into DevSecOps is a change in mindset at the top, such as Jim was frightened into. The priority for IT managers is the delivery of competition-shredding features and a smoothly running application within the challenging environment of finite time and budget.
So Jim has to make sure he meets his end goals while still insulating his team’s work within a completely secure DevOps environment.
“What I need is a process that means the application and infrastructure are continuously secure. There shouldn’t be ANY way to bypass it, so a quick fix is out of the question. And comprehensive documentation is key for security responsible Bill or any other auditor that might ever look at it. That’s even important as a USP to be communicated to customers concerned about the news-worthy spate of high profile security breaches that have compromised personal data”.
Jim saw a clear parallel with the code quality issues they had gone through a couple of years ago as they were still fine tuning their DevOps processes. Bugs in production had wreaked havoc, leading to customer dissatisfaction. They stopped that from happening by integrating QA in the development process. Automated tests at commit or build time plus a rigorously-documented process for manual testing before delivery.
Wouldn’t the same approach ensure best practise DevOps security?
“I need the whole team to buy into upgrading our DevOps to DevSecOps”, thought Jim. A meeting was called and in attendance was Bill from security, Joanne from Sales and Igor, the head of their DevOps team.
If you know the enemy and know yourself, you need not fear the result of a hundred battles. If you know yourself but not the enemy, for every victory gained you will also suffer a defeat. If you know neither the enemy nor yourself, you will succumb in every battle.” ― Sun Tzu, The Art of War
Bill, whose pastimes outside of cyber security were military history and amateur theatre had a flair for the dramatic. So, on learning the DevSecOps theme of the meeting and that he was expected to contribute, had prepared a background presentation slide with a fitting quote. “Everyone loves a good quote”, thought Bill. That will impress.
“Let’s start from the beginning,” said Bill. "First, we have to understand potential threats. What are the major security risks and what they can result in? Our first step should be all stakeholders considering the importance of data and the effect of particular threats to the business if this data were to be compromised. Threat modelling techniques should be our approach here”:
“If you want me to handle this, please, put it into the backlog,” said Igor, the DevOps ‘eager beaver’.
“Is it as a feature?” asked Joanne.
“No, a threat is more like an anti-feature,” shot back Bill, his flair for the dramatic tickled by the elegant response he had summoned.
“I propose employing agile software development methods and retesting any negative user stories to double check we are avoiding them here”.
A good roadmap is the necessary basis for any secure infrastructure and DevSecOps is no exception. With the team now focused on the development process, infrastructure like firewalls - despite their importance - were not covered in their discussion.
But Bill, newly enthused by the elevated importance with which he was being treated as a result of Jim’s security push, was on it. He suggested the development of architecture for a range of DAST (Dynamic Application Security Testing) technologies. They allow for the detection of security vulnerability in an app during runtime and not only isolate problems in the source code, but also those that only become apparent during use.
With the team’s strategy filled out, the time has come for Jim, with input from the others, to set down their DevOps infrastructure security checklist. So they got started:
1. Encrypting passwords (passwords should ALWAYS be stored in an encrypted format in the database so they cannot be stolen).
2. SSL connections used for communication
3. Automated configuration and set-up of servers
4. Regular back-ups
5. Control access on cloud providers
6. Upgrade strength of SSH Configuration
Many teams sigh at this kind of checklist because the check points seem so basic and obvious. While that’s true, with one these simple steps are often forgotten. Why do you think every well run gas station or restaurant has a cleaning checklist up in the bathroom that staff members have to sign once they’ve completed their hourly cleaning task? Because it’s not simple to tell them they have to check in every half hour to make sure it’s in good condition? Of course not. But experience shows a simple checklist provides accountability and works – the bathrooms stay in good condition for clients. Super simple but super effective. Your DevSecOps checklist should also embody simple and effective.
Of course, keeping on top of the team’s new checklist would mean a little additional development effort but Igor got this ready to go with some simple OWASP frameworks and templates.
But Jim wasn’t finished there:
“We need to make our application even securer against attacks. If any malicious intruder does manage to penetrate our external defences, we need to ensure that’s as far as they get. Our web applications have to double-check requests are valid and authenticated. Other checks are about data malformation (for example, a remote procedure call, Session Initiation Protocol [SIP], and so on)”.
At this point, the meeting room’s whiteboard was rapidly filling up. Bill was glad he’d put his Art of War quote up on the projector. It would have been scrubbed off the whiteboard to make room and they needed its inspirational push. “I knew my enemy”, thought Bill, proud of his foresight.
Everything was falling nicely into place and a rare air of unanimous approval permeated the room as they all leaned back, slightly smug at a job well done. The brief lull was interrupted when Igor brought up the overlooked problem of legacy application integration that had been done.
“No one knows the code and that means no one knows potential threats!”
This time it was Joanne’s turn to pipe up with an elegant solution.
“Why not move the legacy application into a container with very limited access, protecting the other applications in the event that weak spot is compromised?”
The others agreed that would minimise the risk.
“It is called Runtime Application Self-Protection (RASP),” said Joanne proudly. She’d been on a training course recently. Joanne, unlike many in the company, took training courses seriously. She had a special pencil case and notepad just for them. She then digitalised her notes. Joanne’s smart. Be like Joanne!
“RASP is a security technology that blocks security attacks. Unlike firewalls, which need the help of network information to detect threats, RASP is more efficient and reinforces the security of software by monitoring its inputs and blocking any that could potentially instigate attacks.”
Code was next on the agenda and now Igor took centre stage. Igor wasn’t a natural on centre stage. Igor wasn’t a natural at much. Except coding. And looking scruffy. But it was his coding skills rather than dress sense and personal grooming that he was ready to step up to defend!
“Of course I write secure code!” blustered Igor.
But when it came to objectively demonstrating its security, he had to admit on reflection, this was difficult to prove without the shadow of a doubt. Of course, he checks the strength of passwords, uses encrypted connections to the backend, but, hey, these hackers are genius nowadays. He knows, some of his uni friends had gone over to the Dark Side. And what’s about the juniors in his team? They weren’t all coding jedi like him. Some of them still struggled to put their Star Wars shirts on the right way around and he was relying on them to fend off the Sith hordes…
After a bit of brow furrowing, Igor suggested a simple solution.
“Let’s delegate this element of our DevSecOps to security automation software and do the checks right away. There are tools that check the code for Static-Code-Analysis-Testing (SAST) while the coding is being done.”
By integrating the static code analysis during development and commit time into the IDE (Integrated Development Environment), the developer receives immediate feedback on the security level of their application. This will dramatically improve their learning curve around secure coding. Common threats like SQL injections or Cross-Site Scripting are checked.
“I’ll teach those youngsters to master The Force”, announced Igor, to quizzical looks from the rest of the group.
The next security threat to be addressed was the potential Trojan Horse that third-party software the company used could offer attackers.
“What about the software we use and haven’t coded ourselves,” Jim asked. “We use a lot to speed up development and reduce costs. Unfortunately, they are often the backdoor attackers exploit to get past strong DevOps security systems.”
It becomes obvious that static code testing will also need to be actioned to check any code not written inhouse.
“And open source software?” continued Jim. “It significantly reduces development effort.”
“This code can be a potential security risk,” agreed Igor. “But by checking this code for known vulnerabilities (for example, with OWASP Dependency-Check), we can detect flaws and fix them, as soon they became known.”
The OWASP utility works by scanning your code and dependent open-source component libraries to see if they contain any known OWASP flaws. It checks all open source code against a continuously updated database of known vulnerabilities in open-source software.
“Great job, team!” announced Jim proudly. “We’ve detailed all of the potential risks, created an architecture which mitigates against them and checked new and legacy code for known vulnerabilities. Have we missed anything?”
The team had covered almost everything involved in an impregnable DevSecOps system but additional external security vulnerability scanning, suggested by Igor was also put on the list as a final flourish.
In addition to the white box testing in the development phase, automated security and vulnerability scanning checks, Jim decided on running tests which simulate the way hackers work. These tests would be integrated into the deployment process to cut out the risk they might ever be forgotten or skipped under time pressure. A report would be proof for management.
They were almost done but Jim had one final point on his agenda.
“The last thing we have to ensure, and by no means the least, is maintaining DevSecOps along every step of the app’s lifecycle. The production phase is no exception when it comes to running all of the tests we’ve agreed on. They have to be actioned and carefully monitored. Any incident, no matter how insignificant it might seem, is to be reported to the architecture specialists and the development team so they can improve the security level of the application.”
Jim returned to his office content. Their new approach, was confident, was the right way to proceed when deploying to production. He was running a tight ship and on his watch there would be no shortcuts that could leave them vulnerable.
“Now, we have a tamper-proof, ‘DevSecOps best practise’ deployment process and checklist,” he thought. “And should anything happen despite our best efforts, we will be able to see immediately where the problem has occurred and how we can patch it. Plus, our GDPR officer Helen might even finally break out a smile, secure in the knowledge there will be no transgression of the new EU law. And I can relax because all activities are properly defined within a clear process. Hackers, bring it on!”
He quickly practised his scary face for the theoretical hordes of hackers in the office mirror, made himself a coffee and got on with his day.