lets start with your kernel, you are obviously missing apparmor patches and probably some default config options.
here is an older rpi kernel i worked on using the pi foundation tree as upstream and adding the needed configs and apparmor patches for an initial pi4 kernel (5.1 though):
try to make sure your kernel uses the same configs, there is also this thread which is more specifically for 5.4 kernels:
