u/cos

Situation:

• A scattered bunch of VM instances running different services, where each of those services exposes prometheus metrics on a local port.

• A central metrics store that accepts metrics via prometheus remote wrote.

To get metrics from the services on these scattered VMs, into the central metrics store, we could run one prometheus server with a config to scrape all those ports, and that prometheus could remote write to the central metrics store.

However, if we prefer to have each host run an agent to collect metrics locally and remote write them to the central store, what agent would you choose in that situation?

We could run a prometheus on each VM to do that, but there may be better alternatives. I know fluentbit can do it too, or we could use Victoria Metrics' vmagent. I'm not familiar with all the options but I've heard of a few others that sound like they could fill this niche as well.

Some things we care about:

• Resilience to temporary failures of the remote write endpoint.
• Resilience to temporary network issues causing latency to the remote write endpoint.
• Reslience to locally scraped services being slow to respond to scrape queries.
• Agent also sends good metrics about itself, so we can easily monitor, troubleshoot, and capacity-plan the small fleet of agents.

What would you choose as the agent, and why?