After a fix took production down.
Someone closed a port on a Friday and the payments API went quiet. Since then, every security fix needs three approvals, and most never get them. The findings are piling up because the last fix hurt.
You will recognise this if
- A security fix caused an outage, and the post-mortem still comes up in meetings.
- Engineers ask what uses a resource, and nobody can answer it quickly.
- Fixes wait months for a change window that never quite arrives.
- Nobody is sure how a change would be undone if it went wrong.
The fixes that break things are usually the same few.
Almost every security fix is safe. The ones that cause outages tend to remove something a system still depends on. The dependency can usually be seen beforehand, if somebody looks.
| The fix | What it can break | What NxSCORE reads first |
|---|---|---|
| Delete an old access key | A nightly job or a vendor integration that still signs in with it | When the key was last used, and whether any key on the user was used in 90 days |
| Close SSH or RDP to the internet | The way admins reach a server in an emergency | Which network interfaces and instances the security group is attached to |
| Require IMDSv2 on an instance | Older software that still calls the instance metadata the old way | Calls made without a token over the last 14 days |
| Block public access on a bucket | A website or a download link served straight from the bucket | Whether the bucket is set up to host a website |
The answer to “what uses this?” is often already in your account. Reading it takes minutes. Not reading it is how outages happen.
Before the change, find out what uses it.
Trust comes back one safe fix at a time. It starts with every fix arriving with the evidence for it, so nobody has to take it on faith.
- 01
A usage check before anyone suggests the fix.
For each failing resource, up to 20 per finding, NxSCORE reads how it is used: what a security group is attached to, when a key was last used, whether an instance still makes old-style metadata calls, whether a bucket serves a website.
It also reads who created the resource, from the last 90 days of CloudTrail. The question “whose is this?” often answers “is it safe?”.
Is it safe to fix?Checked 4 resources- sg-0a41c9 admin-ssh, prod-paymentsCheck first
Attached to 2 instances: pay-api-1, pay-api-2. Make sure admins reach these through Session Manager or a VPN first.
- sg-07be12 old-jumpbox, prod-paymentsSafe to fix
Not attached to any network interface.
- Access key for svc-export, data-lakeIn use
Last used 3 days ago. Move the caller to a role first.
- Load balancer listener, core-bankingNot checked
No automatic usage check for this kind of resource. Your analyst is asking the owner.
- 02
A verdict, with its reason in plain words.
The reason sits beside the verdict. “Attached to pay-api-1 and pay-api-2” is something an engineer can check in a minute. “Medium risk” is not.
When there is no automatic check for a kind of resource, the verdict says so. A missing answer is never treated as a safe one.
VerdictsWhat each means- Nothing depends on itSafe to fix
The kit is written and goes to the owner.
- Something may depend on itCheck first
Says what it is attached to, and what to confirm before the change.
- Something is using itIn use
Says what, and what has to move first.
- No automatic check yetNot checked
Never guessed. Your analyst asks the owner.
- 03
The effort, stated before the work is planned.
Every issue says how much work it is. A fix that needs downtime is labelled as needing a change window, so it is planned as one, and not discovered as one on the night.
Starting with the safe, single-setting fixes is how most teams rebuild confidence. The engineers see how the kits behave before the harder changes come round.
Effort, on the Priorities page- A single change, nothing depends on it.One setting
- A script across many resources. No downtime expected.Fix kit
- The fix needs downtime. Plan it in a change window.Change window
- Something else has to move first, such as a caller to a role.Needs a plan
The change itself: check, apply, roll back.
A fix kit is a script your engineers can read end to end. It does nothing until they say so, and it carries its own undo.
- 04
Check mode first, and it changes nothing.
Run with no options, a kit only checks. It lists every change it would make, and every resource it left out and why.
It only touches the account it is run in. A resource in another account is skipped and listed, never reached across to. Any warning your engineers need is at the top, under Before you apply.
nxscore-fix-admin-ports.shExample fix kit in check mode$ bash nxscore-fix-admin-ports.sh Check mode. Nothing will be changed. Account 4127-0093-5521, ap-south-1 Before you apply: make sure admins reach these instances through Session Manager or a VPN. would remove 0.0.0.0/0 on port 22 from sg-07be12 would remove 0.0.0.0/0 on port 22 from sg-0c3d88 skipped sg-0a41c9: attached to pay-api-1, pay-api-2 skipped 1 group in another account Run with "apply" to make these changes. - 05
Apply asks first. The undo is in the same file.
Apply shows what it is about to do and waits for someone to type yes. Anything else stops it with nothing changed.
Rollback is in the same script, written at the same time as the fix. Nobody has to work out the undo at two in the morning.
nxscore-fix-admin-ports.shApply, then undo$ bash nxscore-fix-admin-ports.sh apply 2 changes in account 4127-0093-5521. Type yes to continue: yes removed 0.0.0.0/0 on port 22 from sg-07be12 removed 0.0.0.0/0 on port 22 from sg-0c3d88 To undo: bash nxscore-fix-admin-ports.sh rollback Then rescan in NxSCORE. - 06
A guardrail tried against your own history first.
A guardrail for the whole organisation can do more harm than any single fix. Before one is proposed, NxSCORE replays the last 90 days of your CloudTrail management events against it.
Your team sees who it would have blocked and when, with examples, and can move that pipeline first. Where the replay cannot read a condition, it says the result is uncertain.
What would this have blocked?Last 90 daysGuardrail Nobody can turn off S3 Block Public Access Who it would block 1 role: ci-legacy-deployer, in 2 accounts Recent examples PutPublicAccessBlock, 19 Oct and 2 Nov Everyone else Not affected
Then prove the fix held.
After the change, closed only when it is seen to be fixed.
The last part of a safe fix is knowing it worked, and knowing straight away if it did not.
- 07
Rescan that one control, straight away.
Once the change is made, your engineer presses I’ve fixed it, rescan. NxSCORE checks that one control in that account, and the finding closes only if the rescan passes.
The person who made the fix is never the one who confirms it.
Security groups open on port 22, prod-paymentsRescan passedFixed by infra team, change CHG-20418 Confirmed by NxSCORE rescan, 4 minutes later Still failing sg-0a41c9, check first - 08
If it is rolled back, the record says so.
Rolling back is the right call when a change hurts. NxSCORE does not hide it. The next assessment sees the problem again, and the finding is recorded as having come back.
Your analyst reads what happened and writes it into the next verdict, so the second attempt starts from what the first one learned.
HistoryExample- FixedKit applied on 3 November. Rescan passed.
- Came backRolled back on 4 November. Seen again by the next assessment.
- Next verdictIn use. The batch job that broke now named as the caller.
- 09
Every prevention option says what it could break.
When an issue keeps coming back, your analyst writes a prevention plan: a guardrail, an account default, a template fix, a pipeline check, or a detective rule where blocking would be too much.
Each option comes with what it could break, how to test it and how to roll it back, written down before anyone agrees to it.
Prevention option: organisation guardrailWhat it could break Pipelines that turn S3 Block Public Access off and on again How to test it Attach it to one test account first and run a deployment How to roll back Detach the policy. Nothing else changes
Who does what.
An outage usually has more than one cause. Here, the evidence is read by one person, the decision made by another, and the change by the people who know the system.
Your NxSCORE analyst
- Runs the usage check and writes the verdict with its reason
- Asks the owner whatever the evidence cannot answer
- Writes the fix kit, with its check, apply and rollback
- Replays a guardrail against your own history before it is proposed
Your engineers
- Read the check output before anything changes
- Run the kit in their own window, with their own credentials
- Roll back from the same file if they need to
Your security team
- Decides the findings marked for review, with the owners
- Agrees each prevention plan, knowing what it could break
- Decides the exceptions where a fix is not worth the risk
NxSCORE’s access to your cloud is read-only. Every change on this page is made by your people, through your own change process. Read every permission.
Questions teams ask about this.
Can NxSCORE guarantee a fix will not break anything?
No, and nobody honestly can. What it does is read the evidence first, say what it found, mark anything in use for review instead of for fixing, and make every change one your engineers can check before and undo after.
Does NxSCORE make the change in our cloud?
No. NxSCORE’s access is read-only. It writes the script, the command or the policy, and your engineers run it in your change process. If something goes wrong, the person who can undo it is already at the keyboard.
What evidence does the usage check read?
On AWS: what a security group is attached to, when access keys were last used, whether instances still call metadata without a token, whether a bucket hosts a website, and who created the resource, from the last 90 days of CloudTrail. Where there is no automatic check, the verdict says so and your analyst asks the owner.
What happens if we roll a fix back?
The next assessment sees the problem again and the finding is recorded as having come back. Nothing is hidden, and your analyst uses what happened to write the next verdict and the next kit.
How do we know a guardrail will not block real work?
For a service control policy on AWS, NxSCORE replays the last 90 days of your CloudTrail management events against it and shows who it would have blocked, with examples. Your team decides with that in front of them.
We have a strict change process. Does this fit it?
It is built for one. A kit is a file your engineers can attach to a change request, with its check output, its apply step and its rollback. The Priorities page says up front which fixes need a change window.
Start with the fix your team is most wary of.
Connect one account with the read-only role. Your analyst runs the usage checks on the findings nobody wants to touch, and shows your engineers the kit for one of them, in check mode, before anyone changes anything.