Understanding Key Security Indicators (KSIs)
FedRAMP 20X is sort of around the corner, switching from the old approach of framework reliance to the new evidence-based verification method. In plain English, instead of checking the Implemented box on the SSP, you now have to provide machine-readable evidence. There are 46 KSIs, and every one will require persistent evaluation. Start preparing now, because this process will become exponentially more expensive year over year. I have warned people about the inflation for over a decade; I have always been proven right.
What if you don’t care about FedRAMP? If you want/need to be NIST compliant and/or you are taking your security posture seriously, you should start to implement at least a limited set of indicators now, targeting a full implementation by the end of 2027. This method of compliance testing is going to be the norm going forward.
What exactly are KSIs
KSIs persistently measure security outcomes and produce machine-readable evidence.
You can (not necessarily should) transform any NIST control into a measurable KSI.
You can map every KSI to one or more NIST controls.
KSI vs NIST control
NIST controls specify the process that should be implemented.
KSIs specify the required outcome of that process.
NIST controls are usually documented only once and tested on a cadence.
KSIs are persistent, point-in-time assessments.
NIST controls are described in human-readable language.
KSIs are machine-readable results intended to be evaluated by an automated audit process.
Why Persistent and not Continuous
Because SOME people actually give a shit about keeping Scope and, more importantly, Cost reasonable and have fought for years to limit the language to a much more appropriate term.
A really simple KSI
KSI-CNA-RVP — Reviewing Protections:
“The effectiveness of protection against denial of service attacks and other unwanted activity for machine-based information resources is persistently reviewed.”
How would you handle this before? Tick the Implemented Box. Ask ChatGPT to stretch “We bought Azure DDoS Protection” into a full paragraph. Ta-da!
Now you have to actually work for it by persistently, or at least daily, checking that:
- Every public endpoint has an approved DDoS protection mechanism.\
- Every HTTP endpoint is behind the approved CDN/WAF.\
- Origins are not directly internet-accessible.\
- WAF policies are enabled and attached.\
- Required managed rule groups are enabled.\
- Rate limits exist for sensitive endpoints.\
- API gateway quotas and throttling are configured.\
- DDoS protection plans are active.\
- Email filtering and anti-spam policies are enabled.\
- Logging is enabled for WAF, CDN, gateway, load balancer and email security systems.\
- No unauthorized public IPs or listeners have appeared.\
- Infrastructure-as-code configuration matches deployed configuration.
And that’s on top of active testing by:
Periodically validate the controls through authorized tests, such as:\
- Controlled load tests\
- Rate-limit tests\
- WAF rule tests\
- Bot simulation\
- Email anti-spam test messages\
- Origin bypass tests\
- Failover and scaling tests\
- Tabletop exercises with the DDoS provider\
- API quota exhaustion tests in a non-production or safely constrained environment\
PUT THE GUN DOWN!!!!
It’s not that bad. First of all, you should already have done everything in the first list at least once. The new change is that you now have to set up a process to check these settings daily. YES, YOU READ IT RIGHT THE FIRST 2 TIMES, DAILY This is the Find Out stage of pissing off the Federal Government.
You should also be doing at least a large part of the 2nd list. I hope. The change now is that it all needs to be on a precise cadence, where ANY finding immediately triggers a remediation process.
This is where Microsoft is going to make you buy Front Door Premium, WAF, Defender with some of that sweet DDoS protection, etc… And I hope you noticed that IaC line - NO CLICKOPS FOR YOU!
I can write a single Python script that gets all this information since for as much complaining as I do about them, Microsoft provides easy access to all the necessary CLIs and APIs.
Recommendations for those who will need FedRAMP
First of all, this is not going to be a team effort, and I really recommend you don’t tackle the KSIs one at a time. Instead, run 2-4 small teams, each sprinting to complete one KSI automation.
Start with the low-hanging fruit to test the process and perfect your pipeline. You need to merge the various results into coherent JSON. Use the Validator tool often!
As with all things FedRAMP, don’t fight the intended process - you are expected to use Agile. (I can’t stress how this little thing can derail the process)
Recommendations for those simply wanting a better security posture
Keep your hands off the FedRAMP KSIs even if you are a Cloud Service Provider. Focus on whatever the current framework is—NIST, CSA, whatever—and carefully begin to pivot each control from description to actual outcome.
Start with Access Control, as this is the most critical area and one that should already have sufficient control coverage.
Once you commit to implementing a comprehensive set of controls, deploy a permanent pipeline.
Think something like:
Azure Resource Graph
Azure Policy
Log Analytics / Sentinel
Front Door / WAF / DDoS APIs
API Management
Exchange Online
│
▼
Scheduled Azure Container Apps Job
├─ Collect configuration and telemetry
├─ Run each FedRAMP KSI test
├─ Generate normalized test-result JSON
├─ Calculate daily and 30-day metrics
├─ Generate FedRAMP-required JSON
└─ Validate JSON against pinned FedRAMP schemas
▼
Azure Storage
├─ Raw evidence
├─ Daily results
├─ Historical summaries
└─ Current submission package
▼
Log Analytics / Sentinel alerts
Azure DevOps or GitHub source control
Did I mention how much Microsoft LOVES this? This is how they keep growing Azure Revenue 30% quarter over quarter.
Container Apps?
Your collector will likely need: Python packages Azure SDKs Microsoft Graph or Exchange modules JSON Schema validation Multiple queries that may run for several minutes Versioned execution environments Potential command-line utilities Repeatable testing locally before deployment
This is basically the perfect use for a Container. It can also be:
Tested on a workstation. Scanned in the build pipeline. Stored in Azure Container Registry. Executed as a scheduled Container Apps Job. Reproduced later by an assessor.
As an added bonus, they support retries, execution timeouts, schedules, manual execution, and Log Analytics!
But I guess you can also use an Azure Function. Like an asshole.
Is all this really needed?
Yes, it really is. The current audit process is not working and this was a major part of Revision 5 of 800-53.
Try to see this as a milestone on a long term roadmap where the win condition is to automate 100% of the audit and verification process. It’s unreasonable to ask someone to check the same setting every single day, 365 times per year. Not just unreasonable but completely unnecessary.
When broken down, this is not a complicated process and is one that should lead to substantial savings once implemented. You can also squeeze out a little value by using the data for report enrichment (Look Boss! 999.99% Uptime!!).
AI considerations
Can AI, the agentic kind help?
This needs to be broken down into two separate sections
Using AI to implement KSI
Obviously yes. If your current human resources can’t handle the scripting and don’t have the architectural knowledge ChatGPT or Claude can greatly help you.
For Azure, Microsoft provides a set of AgentSkills that will greatly enhance the accuracy of the responses. Careful prompting should provide about 60% and you will have to carefully work through the 40% yourself. Were you expecting 90 or even 100%? Not as of July 26.
Start small and use the AI as a learning tool, not a replacement for talent. Start with a simple KSI (like the one above) and make this a learning exercise where everything is carefully documented and understood. Use it to help develop a plan, one that can be corrected and turned into a process.
Co-Work/Work Projects are excellent for this
I recommend a single over-reaching project with sub-folders representing different KSIs. (I really need project nesting in agents ASAP!)
At least for now, you should initialize Agents.md/Claude.md instead of relying on build in memory. Make sure to draft a full plan.
Install whatever vendor AgentSkills into the project (Don’t install them globally). Microsoft, Google and AWS all have these. Make sure you install the Well-Architected Framework skills.
Using AI to provide the persistent monitoring needed for KSI
I strongly urge against the temptation. Those of us that actually work with agents like Hermes, Co-Work and OpenClaw, those of us that know how the “sausage is made” understand that in the end all this will end up as a python/powershell script and a cron job. Using an inference engine for this is
a) expensive b) extraordinary wasteful c) needlessly dangerous d) very unreliable
AI brings nothing to the table that even remotely outweighs any of the mentioned risks.