As enterprises delegate more responsibility to AI agents, how do they prioritize security risk at scale?
Part of that challenge lies in the infrastructure these systems depend on. Digital certificates help establish trust between services and devices. Weak keys and misconfigured certificates can allow attackers to impersonate trusted systems, intercept sensitive communications, or gain unauthorized access.
In our experiments, running the full suite of deterministic checks on roughly one million certificates took over 24 hours.
I’m excited to share X-amine509: Predicting the Practical Risk Level of Enterprise X.509 Certificates, a paper I led with Shubh Patel, JD Kilgallin, and Caleb Shorter.
Our research question
Can machine learning help enterprises triage certificate risk at scale?
We collected over 1 million certificates from Fortune 500, .gov, and .edu domains and scored them against 177 defect checks drawn from RFC 5280, CA/Browser Forum requirements, and NIST guidance, using severity weights assigned by our team.
We then trained models to predict those scores from certificate attributes. ML prioritizes the queue; deterministic tools identify specific defects and provide findings for remediation.
A few results stood out
- On 201,976 held-out certificates, our Decision Tree assigned 99.76% to the correct severity tier and achieved 98.90% recall for critical-tier certificates.
- It reached approximately 3.7 million predictions per second on a single machine in our inference benchmark.
- On another 571,374 certificates collected thirteen months later, it retained 98.06% critical-tier recall without retraining.
Choosing the right model meant looking beyond average accuracy
We compared 25 models and examined how their performance held up over time. On newer certificates, Random Forest had lower average prediction error but missed more critical-tier certificates than our Decision Tree.
That distinction shaped our recommendation: prioritize the combination of speed and critical-tier recall that the security workflow actually needs.
The results also clarified how to deploy this approach: use ML to prioritize detailed analysis, while retaining targeted deterministic checks for the most consequential defects regardless of the predicted score.
My broader takeaway is that useful ML in security needs to support a concrete operational decision and hold up as the environment changes.
Thank you to Shubh Patel, JD Kilgallin, Caleb Shorter, and the Keyfactor Analytics and AI team for making this work possible.
Full paper: arxiv.org/abs/2609.09402
If you’re building AI agents or working on enterprise security, I’d love to hear how you’re thinking about trust and risk as these systems scale.