Over the last few weeks, I’ve been working to inject security tooling into the CI (continuous integration) pipeline. Coming from a background that’s primarily red-teaming, offensive security, and even a bit of compliance, there have been some learning curves for adapting security tooling to fit your organization’s needs. Given I’m a rather small security team, “shifting left” was the primary factor in making sure we had adequate security tooling coverage while also empowering developers to remediate their own issues.
While this article won’t be a comprehensive guide or even a tool review for that matter, it will serve to provide the questions I found myself asking when securing our delivery pipeline.
Consider your “requirements”
A big topic to begin with, but if you don’t have any requirements for security tooling then you’re going to have a hard time deciding on what tool set to go with. Ask yourself the following kinds of questions:
- What is driving the need for “tooling” in the first place? (aside from having good security practices being a no-brainer)
- Is this a customer requirement?
- Is this a compliance requirement?
- Have there been systemic issues coming from the CI we should be accounting for?
- What’s the current state of your security tooling?
- What do you want your coverage to look like?
- Do you need SAST? (Static Analysis Security Testing)
- Do you need package management and vulnerability detection?
- How about DAST? (Dynamic Analysis Security Testing)
- Are you running containers and need to monitor the container workload?
- Who is going to own the tooling and remediation?
- Will requirements change over time?
Survey your environment
What do you have at your disposal in your environment? No, I’m not talking about “prod”, “dev”, or “canary” – what tooling does your company currently employ and what processes do your engineers follow?
- Do you use GitLab or GitHub?
- How are you using their current CI feature set?
- What pipelines are developers familiar with that you could plug into?
Collect data on your current state (if available)
If you have tooling in place, run it. Build some vulnerable applications, or pluck applications off the shelf like OWASP’s Juice Shop, to see how well your current tooling works against real-world threats.
Another option, if applicable, is to survey your engineers to see what they think of the tooling. Try to evaluate what they think of the user experience presently.
Research what tools mesh with your current, and potential future, tech stack
Admittedly this is the most cumbersome part of the process because you’re going to need a pretty deep understanding of your tech stack. Not just that you use LAMP or MEAN, but you’re going to need specific versions and what, potentially, has been a blindside at your organization.
For example, some vulnerability scanners for packages work great with Node (NPM) but if you’re using Yarn then there could be some compatibility issues – then there are versioning issues if the scanners only support up to a specific version of Yarn. Fairly new languages like Golang also have a complicated story around compatibility. Most companies are on the cutting edge of Go, however, security tooling has yet to catch up and it’s up to you to determine if you can warrant the lack of support.
The build process is also another area of concern if you have an automated pipeline. Companies love to move fast so having a keen eye for “how” they move fast in the build pipeline will let you know what tools you need to deploy to monitor it. For example, scanning Docker images or the like.
Trial, rinse, repeat
Now that you have some basic requirements and what stack you’re using, it’s time to trial, but there are some caveats.
- Get a temperature check on budgeting and finance
- No point in evaluating a tool that costs $100,000/year if there is no budget in place for it
- Read tooling docs to check for compatibility – supported languages, package managers, versions, etc.
- Plug the tools into your CI and see how portable they are
- Some tooling works great in the CI if you can containerize it, others not so much, so it’s important to test as close to “production-ready” as possible
- Consider the developer experience
- Cutting-edge tools are amazing on their own, but worthless if they lack the accuracy or context to help developers understand and fix their issues
- How malleable is the tool if we need to make changes?
- Commercial tools will lead you at the mercy of the organization implementing new features whereas open-source tools can be forked
- What support for the tooling do you need?
- A “pro” of open-source tooling is that you can fix your own issues, but this is also a downside since it will require your company’s resources to do so (if you have resources to allocate for maintenance)
On a personal anecdote, I really recommend checking out the open-source community for some great security tooling. Not only are they great in times where budgets are of concern, but they offer a few other benefits such as:
- Public visibility into the source code – know how it works or build features on top of it
- Community-driven features and improvements
- Often times the tools are more extensible and customizable
- More malleable than commercial tools to fit your own needs
Measure so you can improve
As you are trialing your tools, think about reporting and metrics. Wouldn’t you want to know how successful your tooling is? If the tools you are looking into can’t derive measurable results, such as lower vulnerabilities over time, they aren’t worth the effort you’re putting into.
I recommend spending adequate time thinking about “how” you’re going to quantify that your tooling has lowered your risk. That which cannot be measured, simply cannot be improved.
Prepare to iterate be agile
Whatever tool set you choose, prepare to modify how they work on the fly. If you’re using a tool like Semgrep for your SAST needs, prepare to backlog additional rules you want to write to accommodate the changing landscape of both your company and the security world as new risks emerge.
In my own testing, a lot of commercial tools were not extensible. Not only is that unfavorable in my book, but in smaller, quickly changing companies this poses huge risks because the tooling might have a hard time keeping up with your company.
I know these notes are short and frantic, but if there’s any takeaway it’s this:
No tool you choose will be a “one-size-fits-all” solution…. and you’re going to have to be okay with that.
Consider your organization, your team, your developers, and yourself during this process. If you’re like me, you’re in a small team supporting a lot of engineers so maintenance is a worry. However, in bigger and more mature companies, the issue also might be how well the tooling fits in your legacy tech stacks. Either way, focus on your specific needs to derive what’s going to make the best impact.