Well the issue is found, checked, reproduced. To me, this is a bug. If your continuity camera on a mac mini m1 does not work, check out below.
The solution is to plug in another usb webcam. The continuity camera will work straight away. If I unplug the webcam, this also disconnects the continuity cam.
Thanks to this thread.