u/satyendra3339

Hey folks,

I ran into an issue with my setup where Zenarmor was sending a ton of small _bulk requests to Elasticsearch. Even though I was using SSD, it still resulted in lots of small disk writes, higher IOPS, and unnecessary load on the cluster.

Instead of tuning ES endlessly, I tried a different approach — I built a small proxy that sits in between and batches _bulk requests in memory before forwarding them to Elasticsearch.

👉 https://github.com/codifierr/es-bulk-proxy

What it does:

Buffers incoming _bulk requests
Merges them into larger batches
Sends fewer, bigger writes to Elasticsearch
Passes through all read requests unchanged (so dashboards still work normally)

It’s super lightweight and runs as a single container. No disk usage, just in-memory buffering.

Basic usage:

docker run -d \
  -p 8080:8080 \
  -e ES_URL=http://your-es:9200 \
  ssingh3339/es-bulk-proxy

Then just point your client (in my case Zenarmor) to this instead of Elasticsearch.

For me, this significantly reduced write amplification and smoothed out ingestion.

Curious if anyone else has dealt with similar issues or has suggestions to improve this approach. Happy to get feedback!

Zenarmor made 35.7K request which is converted to 361 bulk ES requests

I built a small Elasticsearch proxy to reduce small bulk writes (helped my Zenarmor setup a lot)