ocfs2: synchronize heartbeat callbacks with o2net teardown

In the Linux kernel, the following vulnerability has been resolved:

ocfs2: synchronize heartbeat callbacks with o2net teardown

Patch series "ocfs2: harden heartbeat teardown races".

This series fixes two OCFS2 heartbeat/o2net teardown races found by
KASAN.

This patch (of 2):

Heartbeat callbacks stay registered while configfs local-node teardown
enters o2net_stop_listening(). A node-down event can still run through
o2net_disconnect_node() and o2net_set_nn_state() while teardown is
destroying o2net_wq, so the later queue/flush operations can hit a dead
workqueue. KASAN has caught this as a slab-use-after-free in
__queue_work() with the call chain:

KASAN slab-use-after-free in __queue_work+0x56/0xa90
Read of size 4
Call trace:
dump_stack_lvl+0x66/0xa0
print_report+0xce/0x630
__queue_work+0x56/0xa90
srso_alias_return_thunk+0x5/0xfbef5
__virt_addr_valid+0x19f/0x330
kasan_report+0xe0/0x110
__queue_delayed_work+0x58/0x1e0
queue_delayed_work_on+0xb4/0xc0
o2net_set_nn_state+0x467/0x840
o2net_disconnect_node+0x7b/0xe0
o2net_hb_node_down_cb+0x54/0x60
o2hb_run_event_list+0x236/0x2d0
o2hb_check_slot+0xad4/0xbc0
lock_release+0xc8/0x290
o2hb_check_slot+0x9ea/0xbc0
trace_hardirqs_on+0x18/0x130
o2hb_do_disk_heartbeat+0x646/0xb30 (fs/ocfs2/cluster/heartbeat.c:1079)
__lock_acquire+0x466/0x2260
lockdep_hardirqs_on_prepare+0xea/0x1a0
ktime_get_with_offset+0xe9/0x230
o2hb_thread+0x14e/0x770
kthread+0x1ad/0x1f0
ret_from_fork+0x3c9/0x540
__switch_to+0x2e9/0x730
ret_from_fork_asm+0x1a/0x30
Allocated by task stack:
kasan_save_stack+0x33/0x60
kasan_save_track+0x14/0x30
__kasan_kmalloc+0xaa/0xb0
__kmalloc_noprof+0x292/0x760
__alloc_workqueue+0x736/0xc60
alloc_workqueue_noprof+0xb1/0x110
o2net_start_listening+0xe5/0x430
o2nm_node_local_store+0x184/0x310
configfs_write_iter+0x18a/0x210
vfs_write+0x469/0x810
ksys_write+0xd2/0x170
do_syscall_64+0x115/0x6a0 (arch/x86/entry/syscall_64.c:87)
entry_SYSCALL_64_after_hwframe+0x77/0x7f
Freed by task stack:
kasan_save_stack+0x33/0x60
kasan_save_track+0x14/0x30
kasan_save_free_info+0x3b/0x60
__kasan_slab_free+0x5f/0x80
kfree+0x313/0x590
rcu_core+0x4f4/0x1320
handle_softirqs+0x156/0x660

queue_delayed_work_on
o2net_set_nn_state
o2net_disconnect_node
o2net_hb_node_down_cb
o2hb_run_event_list

Keep heartbeat callbacks registered so quorum state still tracks node
state, but stop them from driving o2net reconnect/disconnect work once
local teardown starts. Mark the transport offline before destroying
o2net_wq, wait for any in-flight heartbeat callback to finish, and delay
bring-up replay until the new local node is published through
o2nm_this_node().

The replay also has to stay serialized with heartbeat callback delivery.
Otherwise a live-node snapshot can be copied, a real hb_down callback
can install -ENOTCONN for a peer, and the stale replay can call
o2net_hb_node_up() for that same peer and queue reconnect work even
though heartbeat is already down.

The buggy scenario involves two paths, with each column showing the order
within that path:

local-node teardown: heartbeat node-down callback:
1. configfs local-off enters 1. o2hb_run_event_list() invokes
o2net_stop_listening(). o2net_hb_node_down_cb().
2. teardown heads for 2. the callback reaches
destroy_workqueue(o2net_wq). o2net_disconnect_node() and
o2net_set_nn_state().
3. teardown destroys and NULLs 3. the callback flushes or queues
o2net_wq. work through o2net_wq.