Fed Up with Expensive ELK Based Log Management Solutions

Lucene based log management systems were great before we had scalable cloud NoSQL databases. A full text search engine was also very useful for unstructured log files.

Today we have multiple scalable database options. In particular BigQuery was built by Google in house to store and analyze petabytes of event based data. We also have lots of ways to easily structure log data.

It doesn't make any sense to keep log data in a Lucene based datastore anymore. Mostly because it gets prohibitively expensive very quickly and you end up having to ship that data somewhere else anyways if you want to analyze it over the long term.

With Logflare and the native BigQuery backend you pay for the storage and query costs yourself so you can keep as much as you like. And BigQuery is very inexpensive if you keep your queries under control allowing you keep orders of magnitude more data.

Not only can devops inspect and alert on events in real-time but the product owner or analyst can query and build dashboards based on years of high resolution data.

Bring Your own Backend™

Use your own BigQuery tables with Logflare and pay for your storage and query costs directly. Pay us only for managing the pipeline and throughput.

Keep all your data in your own Google Cloud Platform account.

Schemaless BigQuery

Using Logflare to insert events into BigQuery makes it easy because Logflare dynamcially manages the underlying BigQuery table schema.

When we see new fields in your payload metadata, we detect the data type, merge the current schema and the new fields and update the BigQuery schema for you automatically.

Streaming or Batched Inserts

Coming soon! We can insert data into BigQuery by streaming or batching files for bulk loading. You pick.

Dead Simple Setup

Just add your project ID to your Logflare account and your data immediately starts inserting into your new tables in your BigQuery account.

Read the guides