At Black Hat a few weeks ago, it felt like every other conversation I had was about AI finding vulnerabilities. AI models can read through codebases and flag weaknesses in about as much time as it used to take an analyst to read through the first page of a scan output. As someone who spent many years doing pen testing and running security programs, the speed is undeniably impressive.
What I keep coming back to, though, is what happens after.
When I ran security teams, findings piled up so fast we could never fix them all, and no CISO would tell you that’s any different today. The natural next question, then, is whether AI can fix what it finds. And if it can, how would you even know the fix worked? Vendors are heading there, and tools will write the patch or open the pull request and, in some shops, those changes push with barely a glance from a human. When your backlog is 1000s of findings deep and your team is a few folks, moving any ticket to “resolved” feels good and like you’re making progress.
But here’s the worry. A closed ticket tells you someone or something made a change. Whether the change truly removed the exposure is a different question that most programs never come back to or sufficiently verify. It’s a problem that’s much older than AI, to be fair. In my pentest practitioner days, we’d go back and retest environments, and the number of “fixed” issues that were still exploitable never stopped being surprising. We’d have patches that passed the exact check the scanner ran, just not much else. Other times it’d be a config change that was overwritten in a subsequent deployment because the template underneath it wasn’t touched, and no one found out until we did.
AI is raising the odds on all of that. Models writing fixes tend to only know what’s right in front of them, so they usually don’t know the compensating controls another team put in previously or the pipeline that’s going to roll back the change in a few days. And, for better or worse, AI is fast, so it can close a lot of tickets without regard for the context of the environment to which those patches are being deployed.
What to measure when the ticket closes
Should CISOs trust AI to fix vulnerabilities? This all comes down to trust, which must be built over time.
The first thing I’d be looking at is whether the fix worked on a system or application that has minimal impact to the business if it goes down. Then verify that the fix sustains itself over time.. Retest on a schedule, maybe weeks or months out, and test the way an attacker would, instead of rerunning the same check the patch was written to pass. Pay attention to those results, because if the same kind of vuln keeps turning up on the same asset or the same codebase, it means there is something more systemic in the environment and change management process. And most dashboards don’t reflect recurrence rates like what’s being described.
Next, you want to determine if your risk has declined. Closed-ticket counts aren’t the right barometer; you can knock out 100 low-severity items and leave your exposure right where it was. Or, to make it more positive, a single vulnerability fix on an internet-facing system with a known and active exploit can have a bigger impact on your security posture than everything else you did all quarter. Either way, you need an accurate perspective on the right prioritization and that takes tying each finding to how important the asset is and whether an attacker can realistically reach it (then watching whether that picture of risk posture improves with time).. If you just had a quarter of heavy remediation work and exposure is holding steady, something is not right with the process.
Someone human also has to own the outcome. Plenty of times, AI will write a fix and it’ll be a good one. But when the board or a regulator comes knocking about whether a critical exposure was handled, “the AI agent took care of it” is not what they’re going to want to hear. You need a record of what changed and whether it was verified, all tied to the finding from the day it was discovered.
Where does the time go?
The best thing AI does for most security teams is hand them back hours. If discovery, reporting, and initial remediation are getting faster, which they are, then putting some of that time into validation is a smart strategy. Retesting and risk measurement tend to be among the first things cut when backlogs grow, but that’s the only way to know if the rest of your program is working. AI is going to keep getting better at finding things and the pile will keep growing. Security teams that pair that speed with more follow-through will come out ahead. Those still grading themselves on closed ticket quantities are destined to find out, invariably at the worst possible time, that “resolved” and “safe” can be quite different.
_____
About Dan DeCloss
Dan DeCloss is the founder and current executive at PlexTrac, the offensive security validation platform that’s part of Brinqa. Dan started his career in the Department of Defense and then moved on to the private sector where he worked for various companies including Telos, Veracode, Mayo Clinic, and Anthem. Dan’s background is in application security and penetration testing, involving hacking networks, websites, and mobile applications for clients. Prior to founding PlexTrac, Dan was the director of cybersecurity for Scentsy.
Click Here For The Original Source.
