The Wider World
Loki is one position in a market of log stores, and an honest survey starts with the opposite position: Elasticsearch and its fork OpenSearch, which index every word of every line and answer arbitrary-term searches in milliseconds. The capability is real, the workloads that need it are real, and so is the bill.
This topic is the map — what the full-text world offers and charges, where the SaaS platforms sit, and why Harborline, specifically, needs neither. The selection logic outlives any one tool: you will choose a log store more than once in a career, and the criterion that survives is query shape, not feature list.
The Inverted Index
Elasticsearch tokenizes each incoming line and records, for every term, exactly which documents contain it — an inverted index, the same structure as the index at the back of a book. Once it exists, any string is findable across all services and months of history in milliseconds, with no label and no time range required to narrow the hunt.
The bill arrives in three currencies. The index commonly adds 50–100% to the raw data size. It wants fast local disks, because every ingested line updates it. And the cluster serving it needs real capacity planning — shards, replicas, JVM heap. This is the exact cost Loki's founding decision refuses to pay, paid here in full, in exchange for the one query shape Loki grinds through.
The Stack Around It
The store rarely ships alone. The classic stack is Elasticsearch for storage and query, Kibana for search and dashboards, and an ingest layer — Filebeat tailing files, Logstash or Fluent Bit parsing and routing. OpenSearch is AWS's 2021 fork, created when Elastic moved Elasticsearch off its open-source license: API-compatible for common workloads, OpenSearch Dashboards standing in for Kibana, and the engine behind Amazon OpenSearch Service. One ancestor, two ecosystems, now evolving separately.
Where Full-Text Search Earns Its Cost
Security and audit work is search-heavy by nature. "Every occurrence of this IP address across all systems in the last 90 days." "Every action taken with this account token." "Every host that ever logged this malware hash." Each of these is an arbitrary term over a long window with no service label to narrow it — the shape the previous topics showed Loki scanning everything to answer, and the shape the inverted index exists for. SIEM platforms sit on full-text indexes for precisely this reason: the hunt is the workload, and the 50–100% surcharge on every gigabyte is what the hunt costs.
SaaS Logging
Datadog Logs, Splunk, and Grafana Cloud sell the zero-ops version. Pricing centers on ingest — around $0.10 per GB, with indexing and retention tiers layered on top — and Splunk's per-GB/day licensing is the industry's cautionary tale about what unbounded ingest does to a budget. What you buy is genuine: no cluster, no compactor, no capacity planning, someone else's pager when the store degrades.
What changes is the value of the previous topic's hygiene. On self-hosted Loki, an unsampled INFO firehose costs some object-storage dollars; at ingest pricing, the same firehose is a metered line item. Agent-side sampling, health-check exclusion, and retention tiers become billing controls, and the difference between a reasonable invoice and a four-figure surprise is configured in Alloy, not negotiated with the vendor.
Why Harborline Stays on Loki
Run the selection logic on Harborline honestly. Five services, two app hosts, and query patterns that are always service-scoped and time-bounded — last Saturday's entire investigation was {job="bookings"} plus a two-and-a-half-hour window. Per-request correlation rides on request_id in structured metadata. There is no security team running arbitrary-term hunts, and there is one obs-01 box writing chunks at object-storage prices. Every query Harborline actually runs lands on the cheap side of Loki's trade.
The honest exit condition: a SIEM requirement — a compliance mandate, dedicated security hires, 90-day audit-window searches — is a new workload that justifies a second, specialized store running beside Loki. It is not evidence that Elasticsearch should have held the application logs all along. And no store on this map fixes what the logs still cannot say about Saturday: hundreds of connection pool timeout lines name the error, but not where the time goes inside each slow request. That answer needs a third signal, and it is Chapter 8's to give.
- Running a 3-node Elasticsearch cluster — shard planning, JVM heap tuning, index lifecycle management — for five services whose every real query starts with "in
bookings, in the last hour," which a single-binary Loki answers from object storage. - Shipping full-volume logs to a $0.10/GB SaaS without agent-side filtering — 26 million daily INFO lines from
searchbecome a four-figure monthly line item for data nobody queries. - Leaving an Elasticsearch or OpenSearch endpoint on the public internet without authentication — open clusters leaking hundreds of millions of records is a recurring breach headline, and log stores are dense with exactly the data attackers want.
- Letting the log store absorb the metrics workload because Kibana can chart log-derived numbers — dashboards built on log aggregations cost orders of magnitude more per query than Prometheus counters, and they break the moment log volume gets sampled.
- Choosing the log store by feature list instead of query pattern — "Elasticsearch can do everything Loki does" is true and irrelevant; the question is what your actual queries look like and what each engine charges for them.
- Choose by query shape: label-scoped, time-bounded operational grep goes to Loki; arbitrary-term, long-window investigation goes to Elasticsearch or OpenSearch — and be honest about which one your incidents actually produce.
- Cap SaaS ingest at the agent — Alloy-side sampling plus exclusion of health checks and DEBUG — because under ingest pricing, every line dropped before the wire is money kept.
- Keep each signal in its purpose-built store — metrics in Prometheus, logs in the log store — even when one platform advertises that it can hold both.
- Add a search-optimized store when a search-heavy requirement actually lands (SIEM, compliance audit), run it beside Loki for that workload, and resist migrating operational logs to it out of tidiness.
Knowledge Check
What does the inverted index buy, and what does it cost?
- Faster ingestion, at the cost of slower queries
- Millisecond search for any term; 50–100% extra storage plus a cluster to operate
- Cheaper storage, since indexed data compresses better than raw chunks
- Nothing Loki lacks — both engines keep a full-text index and differ only in defaults
Which workload justifies Elasticsearch or OpenSearch over Loki?
- Any deployment with more than a handful of services producing logs
- Dashboards that graph error rates over time
- High log volume, since Loki cannot ingest at scale
- Arbitrary-term searches over long windows — security and audit hunts
OpenSearch in one sentence?
- AWS's 2021 fork of Elasticsearch, born from a licensing dispute
- A rewrite of Elasticsearch that replaces the inverted index with label-based streams
- Elastic's own managed cloud offering of Elasticsearch
- An older name for Kibana's dashboard layer
Why does log hygiene matter more, not less, once logs ship to an ingest-priced SaaS?
- It doesn't — the vendor handles sampling and retention for you
- Because SaaS platforms reject unstructured lines at the API
- Every line shipped is billed, so sampling and exclusion turn directly into money
- Because retention is fixed at 30 days on all SaaS platforms and cannot be tuned
You got correct