
Inconsistent buffering behavior, have tried everything
My Plex server has very odd buffering behavior. Streams sometimes work perfectly and sometimes choke under the same conditions. I have run many many tests that I think conclude the issue is not with the server itself.
When the streams works well, it will spike to a high bitrate then back down to zero repeatedly. This is how I expect it to work. Example: https://ibb.co/WNsDRtLs
When the stream works poorly, it will have the same initial spike but then afterwards flatten to a bitrate much lower than that of the file, anywhere between 3 and 15 mbps. Example: https://ibb.co/DDw5L6Rj
These two screenshots were taken for the same movie, same client, same location, approximately 20 minutes apart. This same behavior repeats itself across iOS, Apple TV, Chromecast, Mac, and Windows clients. The behavior persists no matter if the stream is direct play, transcode, local, or remote. It is not a matter of how overloaded the server is because the issues will happen even with no other processes running.
Plex Media Server is running in a Docker container on TrueNAS. 7th generation i5, GTX 1070, media is stored on a ZFS HDD pool. 1gbps wired connection for the server and a few clients. The docker container is on the host network.
What I have done/found (all below tests have been applied to one movie for consistency, but these problems occur with almost every movie/show):
- HDD read speed testing showed that the pool can sustain read speeds of ~300MB/s for the entire movie's length
- iperf3 testing between the server and every affected client shows connection speeds of several hundred mbps
- FTP can transfer the movie just fine saturating the link between client(s) and server.
- Tried with Plex's remote access feature (not Relay) + forwarding port 32400, also tried using Caddy as a reverse proxy + custom domain specified in the appropriate spot in settings
- Tried swapping out the network switch, then after that didn't work tried swapping the router.
- Tried two different network cards on the server itself
- Tried connecting the HDD pool to two different HBA cards as well as directly to the motherboard's SATA slots
- Added a fan to my HBA card
- For a brief second, I thought it was credit/intro detection. I moved the schedule so those only happen between 2 and 5 am in my time zone.
I also tried many different movies with many different audio/video codecs but they all wind up with this issue.
At this point I just don't know what to do. There is no obvious failure mode anywhere and scouring through the logs after every stutter instance hasn't helped either. What drives me crazy is how I can do the exact same thing twice and get vastly different results. Asking here is the last option I can think of. Thank you in advance