r/elasticsearch

hi folks;

i am currently in the process of uplifting my team's terrafrom elasticsearch version from 0.13.1 to 0.14.1, however it is a bit of an arduous process when it comes to fleet. we are testing out using spaced agents so have upped the version to enable smoother implementation of the spaceing, but shifting our approach to integration policies is a bit of a nightmare.

we have been using file()/templatefile() with json.tmpl files for the bulk of the inputs for our elasticstack_fleet_integration_policy resources which has been great, however with 0.14.0+ introducting a new approach to defining inputs for integration templates, it seems that this approach no longer works.

when using very bulky integrations such as the system integration, we have gone from being able to have a small suite of template files to having to explicitly define almost all of the streams and their variables, most of which we wish to keep as the default settings anyway or have disabled. i have attempted to covert this as follows

resource "elasticstack_fleet_integration_policy" "system-integration" {
  name = "example_system"
  integration_name = "system"
  version = "2.1.0"
  ...
  input {
    input_id = "system-system/metrics"
    enabled = true
    streams_json = templatefile("system-metrics.json.tmpl", {
      tags = var.system.tags
    })
  }
  ...
}

to

resource "elasticstack_fleet_integration_policy" "system-integration" {
  name = "example_system"
  integration_name = "system"
  version = "2.1.0"
  ...
  inputs = {
    "system-system/metrics" = {
      enabled = true
      streams = {
        "system.core" = {
          enabled = true,
          vars = jsonencode({
            "tags": var.system.tags
          })
        },
        ...
      }
    }
    ...
  }
}

however, when running a plan of the changes, it appears to destroy the bulk of the existing resource and doesn't appear to recognise that by not defining a value in vars, we want to use the default rather than not use the value at all.

so my overall questions is: is there a way of using file() or templatefile() in the same way with 0.14.1, or is it going to have to be a case of converting the existing template files to be variables or something of that ilk?

thanks so much - i'm pretty new to terraform so any help or advice would be really apprecited! <3

Many e-commerce search implementations are "intent-blind"—they match tokens, but they don't understand context. In this second part of my PRISM series, I’m digging into the architecture of how we solved this using a middleware approach.

The Highlight: We’re using Elasticsearch Percolator queries to do real-time policy lookups. Instead of searching for products, we search for intent first, then rewrite the query on the fly.

What’s in the video:

• Moving search logic out of the app layer and into a governed index.
• Implementing Hard Filters vs. Soft Boosts via the PRISM engine.
• The Math: How we’re influencing $BM25$ ranking using multiplicative boosting.

Some API's require a start/end date for reports. For example:

"data":[

{

"start": "2015-11-16T14:49:18+0000",

"end": "2015-11-16T14:49:18+0000"

}

]

}

Is there a way to feed date/time in without it being hard coded?

Asking as a former employee (non-technical)

I worked at Elastic for four years purely on the finance side, so I know the business reasonably well but I’m nowhere near qualified to assess the product depth and AI threat that you guys may.

The share price has taken a pretty significant hit recently, largely driven by fears around AI disruption the concern being that tools like Claude could eat into what Elastic does. But from what I’ve read, the counter-argument is that AI actually benefits Elastic, particularly around vector search and the way LLMs need to retrieve and search data.

I’m sitting on a meaningful amount of vested RSU’s that I never sold and I’m genuinely trying to work out whether to hold. I’m not looking for financial advice just an honest technical perspective from people who actually use this stuff day to day.

Is Elastic getting pulled into the AI stack at your companies, or being worked around? Really would appreciate the insight.

In my org we want to setup a SIEM solution. When i researched i got to know about the tool name Wazuh. I think it is the best one in market due to its ability and its open-source too.

But we are already using Aws opensearch, all are application logs get into it. My senior told me to explore about it as well. So here I want to know if someone had worked on that or have pros & cons list of it.

Is opensearch security analytics is better than wazuh?

this started as a weekend experiment that got out of hand. i watch a ton of technical youtube content for work and the search on youtube is terrible for finding specific things people said. it only matches titles and descriptions. if someone explained a concept perfectly in minute 47 of a talk you're never finding it unless you remember which video it was.

so i started pulling transcripts and indexing them in elasticsearch. the idea was simple. get the text, index it, search it. basic stuff. but the results are way better than i expected.

for pulling transcripts i use transcript api. setup was:

npx skills add ZeroPointRepo/youtube-skills --skill youtube-full

transcripts come back with timestamps so i index each one as a document with the video metadata plus the full text. i also split them into paragraph-sized chunks as nested documents so search results can point you to a specific segment.

the mapping is nothing special. standard analyzer on the transcript text, keyword fields for channel and tags. date field for publish date so i can filter by time range. i added a custom analyzer with synonym expansion for tech terms so searching "k8s" also matches "kubernetes" and stuff like that.

where elasticsearch really shines is the highlighting. when you search across 2000 transcripts and get back highlighted snippets showing the exact sentence where your term appears with the surrounding context, it's insanely useful. way better than just knowing "this term is in this video somewhere." combine that with the timestamp from the chunk and you can jump straight to that moment.

i also set up aggregations by channel and by year so i can see things like "which channels talk about observability the most" or "how has discussion of rust changed over time across all these talks." that wasn't even the original goal but it's become one of the most useful parts.

2000+ videos indexed, search latency under 50ms. the cluster is tiny since it's just text.

Hey guys.

Serious question.

Currently I am owning a system with 6 logatash servers and 20 nodes. I am integrating different pipelines for different customers. Soon planning to introduce Kafka into the system. I am using gitlab runners to deploy changes in production and Ansible scripts in order to make changes to the infrastructure.

The storage taken by the indexes in the nodes is about 250 terabytes.

Objectively speaking, is this a big system?

If I am to look for a different job, how could I position myself? Am I an Observability engineer? Am I a Site Reliability Engineer?

I noticed that there are SO many job names on LinkedIn, that I really find it hard to know what I am.

Any input is appreciated. Thank you

Search for "steak" on some e-commerce sites and you'll sometimes get steak knives. Here's why — and how we're fixing it.

Most e-commerce search engines are "intent-blind" because traditional search lacks a functional bridge between merchandising goals and technical execution.

In this video, I introduce PRISM (Policy, Rules, & Intent Synthesis Middleware), an Elastic Services accelerator that gives retailers a Governed Control Plane. PRISM decouples business logic from engineering sprints, letting you fix relevance errors and optimize results in seconds.

What's covered:
• The Architecture
• The "Steak" Test — fixing a real-world relevance error in 30 seconds
• SKU & Product ID — auto-detecting technical intent vs. general search
• Complex Queries — handling "Fruit high in Vitamin C under $4" without AI hallucinations
📖 Full 8-part architectural deep dive on Elasticsearch Labs, linked to from my personal blog: https://alexmarquardt.com/
🌐 Elastic Services: https://www.elastic.co/consulting/contact
💼 Connect on LinkedIn: https://www.linkedin.com/in/alexandermarqu...

I'm new to elastic and I want to deploy elastic SIEM on-prem but I need a help in sizing how can I size the GB I require, if any one had such an experience or sheet to follow

Been using elastic self hosted for years for observability, on recent calls with our CSM he mentioned that self hosted platinum licenses won't be sold anymore, which means to us probably doubling the price we pay for licenses if we have to bump to Enterprise.

What's your take on this? The license cost alone will be on par on what we pay currently for our actual workloads.

NewRelic did something like that years ago which made us move to Elastic

I’m trying to set up Elastic on an Ubuntu VM running in VirtualBox (Oracle), but I’ve hit a wall and could use some help.

I’m running into an architecture issue where the system shows ARM64 instead of AMD64, and it’s causing problems with the installation.

Is Elastic even supported on ARM64 in this kind of setup, or should I be using a different VM/image?

So recently I was required at my job to get Elastic certifications. I am fairly new to Elastic so while going through the syllabus I found some topics that are out of date. Like with the current push on AI features from Elastic they should add features which they are pushing for elastic in the exam to and I feel like a update is needed for the syllabus.

Even though it's been out for nearly a year, I haven't been able to find any reviews or impressions on EASE. Is anyone using it? And how is your experience with it?

My colleague published a plugin that adds ES|QL support to IntelliJ IDEA: https://plugins.jetbrains.com/plugin/28898-elasticsearch-es-ql

It includes features like syntax highlighting and query validation, which can be useful if you don’t want to rely only on Kibana or Dev Tools when writing queries.

Sharing in case it’s relevant to others here:

https://www.elastic.co/search-labs/blog/esql-plugin-intellij-idea

Hi all! I'm a master's student working on a final research project on Elastic (search, observability, or security). Looking for people who have used or evaluated Elastic to fill out a <10 min survey.

All responses are anonymous and confidential. Thanks in advance!