u/Seba_dev

This was a bit harder than I expected because dynamically changing the physics world (like the heightmap) causes some issues. Updating physics objects every frame is not very efficient and seems to break ongoing raycasts/collisions.

So I settled for doing custom raycasting logic on the heightmap itself, which can be heavily simplified.
For the collisions (when slamming into the ground for example), I still keep a small patch of physics heightmap underneath the player. And moving it with the player as well.
Not sure if this will scale well, but works for now.