Stop hardcoding your scraper logic: use the browser's Copy as cURL first
My two cents. Most people spend bunch of time reverse engineering request headers when the browser will just hand them to you. Next time you find an API call in the Network tab, right click it and hit Copy as cURL. Paste it into your terminal and it works instantly, cookies, headers and all. From there you can import it directly into Postman or use a tool like curlconverter to turn it into clean Python requests code in seconds. The browser already did the hard work of figuring out what the server needs. There's no reason to reconstruct that by hand