KVM: SEV: Forcefully invalidate SNP VMSA if its backing gmem page is zapped

In the Linux kernel, the following vulnerability has been resolved:

KVM: SEV: Forcefully invalidate SNP VMSA if its backing gmem page is zapped

Wire up a gmem_invalidate_range() call for SNP VMs, and use it to force
vCPUs to reload/recheck their guest-provided VMSA if the backing gmem
page is being invalidated, e.g. is being PUNCH_HOLE'd. Use the same core
logic to handle invalidations as VMX does for the APIC-access page, as the
two concepts are nearly identical: shove the physical address of a page
into the vCPU's control structure:

1. Snapshot the invalidation sequence counter
2. Grab the pfn (from guest_memfd in this case)
3. Acquire mmu_lock for read
4. Re-request reload if retry is needed, otherwise commit the change.

Note, the re-request action in #4 is necessary as KVM's retry logic is
fuzzy, i.e. can get false positives. If the guest_memfd page has been
dropped, at some point a subsequent reload will fail to get a PFN from
guest_memfd, and KVM will fail KVM_RUN. If the retry was due to a false
positive, KVM will retry until there are no relevant MMU notifier events
(and will retry in the "outer" loop, i.e. will drop locks and resched as
needed).

Note #2! Take care to invalidate the VMSA when a relevant memslot is
DELETED or MOVED, as invalidations in response to PUNCH_HOLE are predicated
on memslot bindings (KVM doesn't know what GFN range(s) to invalidate
without a binding). And more importantly, the VMSA mapping requires a
memslot, i.e. must be invalidated if its memslots disappears, regardless of
the state of the underlying guest_memfd inode.

Failure to invalidate the vCPU's control.vmsa_pa (which is checked by
pre_sev_run()) can prevent KVM from properly freeing the page as firmware
will reject the RMPUPDATE to reclaim the page with FAIL_INUSE if the vCPU
is actively running, i.e. if VMSA page is in-use. That in turn leads to an
RMP #PF on the next use, as the page will still be assigned to the SNP VM.

SEV-SNP: RMPUPDATE failed for PFN 78d198, pg_level: 1, ret: 3
SEV-SNP: PFN 0x78d198, RMP entry: [0xfff0000000144001 - 0x000000000000000f]
CPU: 3 UID: 0 PID: 31345 Comm: sev_snp_vmsa_pu Tainted: G U O
Tainted: [U]=USER, [O]=OOT_MODULE
Hardware name: Google, Inc. Arcadia_IT_80/Arcadia_IT_80, BIOS 34.86.0-102 01/25/2026
Call Trace:
<TASK>
dump_stack_lvl+0x54/0x70
rmpupdate+0x12c/0x140
rmp_make_shared+0x3b/0x60
sev_gmem_invalidate+0xe0/0x170 [kvm_amd]
delete_from_page_cache_batch+0x1d8/0x220
truncate_inode_pages_range+0x120/0x3d0
kvm_gmem_fallocate+0x19a/0x270 [kvm]
vfs_fallocate+0x1bc/0x1f0
__x64_sys_fallocate+0x48/0x70
do_syscall_64+0x10a/0x480
entry_SYSCALL_64_after_hwframe+0x4b/0x53
RIP: 0033:0x496c7e
</TASK>
------------[ cut here ]------------
SEV: Failed to update RMP entry for PFN 0x78d198 error -14
WARNING: arch/x86/kvm/svm/sev.c:5160 at sev_gmem_invalidate+0x126/0x170 [kvm_amd], CPU#3: sev_snp_vmsa_pu/31345
CPU: 3 UID: 0 PID: 31345 Comm: sev_snp_vmsa_pu Tainted: G U O
Tainted: [U]=USER, [O]=OOT_MODULE
Hardware name: Google, Inc. Arcadia_IT_80/Arcadia_IT_80, BIOS 34.86.0-102 01/25/2026
RIP: 0010:sev_gmem_invalidate+0x12b/0x170 [kvm_amd]
Call Trace:
<TASK>
delete_from_page_cache_batch+0x1d8/0x220
truncate_inode_pages_range+0x120/0x3d0
kvm_gmem_fallocate+0x19a/0x270 [kvm]
vfs_fallocate+0x1bc/0x1f0
__x64_sys_fallocate+0x48/0x70
do_syscall_64+0x10a/0x480
entry_SYSCALL_64_after_hwframe+0x4b/0x53
RIP: 0033:0x496c7e
</TASK>
irq event stamp: 20689
hardirqs last enabled at (20699): [<ffffffff8e76092c>] __console_unlock+0x5c/0x60
hardirqs last disabled at (20708): [<ffffffff8e760911>] __console_unlock+0x41/0x60
softirqs last enabled at (20722): [<ffffffff8e6cd74e>] __irq_exit_rcu+0x7e/0x140
softirqs last disabled at (20717): [<ffffffff8e6cd74e>] __irq_exit_rcu+0x7e/0x140
---[ end trace 0000000000000000 ]---
BUG: unable to handle page fault for address: ffff99
---truncated---