Unreal 5, weird replication issue [HELP]
Im running a listen server setup. I have a weird issue. Game starts, players spawn and get teleported to their spawn locations. Now the client goes and picks up the weapon, it attaches to him, but then I have a problem with starting location of line trace (which is the socket on the attached actor, though whole actors location is not right) was wrong. - Im testing on a create/join session with advanced sessions plugin, this does not happen with just putting 2 players on the map.
Anyway, line trace start location is wrong UNTIL the server player rotates the mouse( - camera controlls player rotation as well), that fixes the vector mismatch and start location for line trace starts working for all clients.
Really confused on how this could be a thing and I have no idea whats causing it. Spent a whole day debugging to come to this conclusion. This bug carries over to the fully compiled game with Steam sessions so its not some PIE issue.
Any help would be appreciated.