OpenSearch Engineering for AWS-Managed Search and Log Analytics

OpenSearch (Apache-2.0) is the right choice when you are on AWS, want to avoid Elastic's SSPL licensing for self-managed deployments, or need tight integration with AWS services (S3, CloudWatch, IAM). Elasticsearch (Elastic licence or SSPL) is preferable when your team already uses Elastic Cloud, Elastic APM or Kibana features that have no OpenSearch equivalent. The REST API is compatible for most query and indexing operations through the 7.10 baseline, so application-level migration is usually straightforward for that subset.

The 7.10 baseline means core search, aggregation and kNN APIs are compatible, but proprietary Elastic features added after 7.10 — EQL refinements, certain Kibana-specific APIs, Elastic APM wire format — have no direct OpenSearch equivalent. We start every migration with an API surface audit against your application code and current index templates, identify the gaps, and map each to an OpenSearch alternative before writing a line of migration script. Most REST-based applications migrate cleanly in one to three weeks of engineering time.

AWS manages hardware provisioning, OS patching, OpenSearch version upgrades (with your approval), automated snapshots to S3, multi-AZ replication and basic CloudWatch metrics. You — and we, on your behalf — handle index design, shard sizing, mapping templates, ISM policies, security plugin configuration, fine-grained access control, custom dashboards and application-layer query optimisation. Choosing AWS OpenSearch Service eliminates the server operations burden but does not remove the need for search engineering expertise.

The OpenSearch k-NN plugin adds a knn_vector field type backed by FAISS or NMSLIB approximate nearest-neighbour graphs stored in JVM off-heap memory. At query time, a knn query returns the k most similar document vectors to a query embedding. In production we combine kNN with BM25 using a hybrid query and a normalisation processor pipeline, which consistently outperforms either approach alone for semantic document retrieval. Instance sizing must account for the kNN graph memory alongside the standard query cache allocation.

For many teams, yes. OpenSearch Ingestion (Data Prepper) ingests logs from Fluent Bit, Logstash or direct HTTP sources, applies field extraction and enrichment, and writes to time-series indices. OpenSearch Dashboards provides visualisation, anomaly detection and alerting comparable to the ELK stack. The security analytics plugin adds threat-detection rules in Sigma format. For regulated environments it delivers the continuous monitoring NIS2 requires at a fraction of the cost of commercial SIEM platforms, with no per-event licensing.

The security plugin layers multiple access control mechanisms: cluster-level permissions, index-level permissions, field-level masking (hash or anonymise specific fields), and document-level security (filter clauses applied automatically per role). Multi-tenancy in OpenSearch Dashboards isolates visualisations and index patterns between teams. We configure dedicated service-account roles for each application, separate read and write principals, and route audit events to an index that operations or compliance staff can query without cluster-admin access.

Cost on AWS OpenSearch Service is dominated by instance count and EBS storage. The primary levers are: right-sizing shard count to 20-50 GB per shard (over-sharding wastes master-node heap), enabling UltraWarm for indices queried infrequently (S3-backed, ~90% cheaper than EBS), moving cold data to the cold tier or deleting it via ISM after the retention window, and using force-merge on read-only historical indices to reduce segment count. We deliver a cost audit with specific ISM policies and instance recommendations before any change is applied to production.