Apache Pinot is a realtime distributed OLAP datastore, which is used to deliver scalable real time analytics with low latency. It can ingest data from offline data sources (such as Hadoop and flat files) as well as online sources (such as Kafka). Pinot is designed to scale horizontally.
These presentations on Pinot give an overview of Pinot:
- Building realtime applications using Pinot @ DataCouncil
- Pinot: Enabling Real-time Analytics Applications @ LinkedIn's Scale - ApacheCon 2019 (Sep 2019)
- Pinot: Realtime OLAP for 530 Million Users - Sigmod 2018 (Jun 2018)
- Open Source Analytics Pipeline at LinkedIn (Sep 2016, covers Gobblin and Pinot)
- Introduction to Pinot (Jan 2016)
- Pinot: Realtime Distributed OLAP Datastore (Aug 2015)
Looking for the ThirdEye anomaly detection and root-cause analysis platform? Check out the Pinot/ThirdEye project
- A column-oriented database with various compression schemes such as Run Length, Fixed Bit Length
- Pluggable indexing technologies - Sorted Index, Bitmap Index, Inverted Index, Star-Tree Index
- Ability to optimize query/execution plan based on query and segment metadata
- Near real time ingestion from Kafka and batch ingestion from Hadoop
- SQL like language that supports selection, aggregation, filtering, group by, order by, distinct queries on fact data
- Support for multivalued fields
- Horizontally scalable and fault tolerant
Because of the design choices we made to achieve these goals, there are certain limitations present in Pinot:
- Pinot is not a replacement for database i.e it cannot be used as source of truth store, cannot mutate data
- While Pinot supports text search, its not a replacement for search engine i.e relevance is not supported
- Query cannot span across multiple tables - Use Presto-Pinot connector to achieve joins and other features
Pinot works very well for querying time series data with lots of Dimensions and Metrics. Example - Query (profile views, ad campaign performance, etc.) in an analytical fashion (who viewed this profile in the last weeks, how many ads were clicked per campaign).
More detailed instructions can be found at Quick Demo section in the documentation.
# Clone a repo
$ git clone https://github.com/apache/incubator-pinot.git
$ cd incubator-pinot
# Build Pinot
$ mvn clean install -DskipTests -Pbin-dist
# Run the Quick Demo
$ cd pinot-distribution/target/apache-pinot-incubating-<version>-SNAPSHOT-bin
$ bin/quick-start-batch.sh
Please refer to Kubernetes Readme to deploy Pinot using Helm and load demo data set.
Pinot also provides k8s integration with interactive query engine Presto and data visualization tool Apache Superset.
- Ask questions on Apache Pinot Slack
- Please join Apache Pinot mailing lists
dev-subscribe@pinot.apache.org (subscribe to pinot-dev mailing list)
dev@pinot.apache.org (posting to pinot-dev mailing list)
users-subscribe@pinot.apache.org (subscribe to pinot-user mailing list)
users@pinot.apache.org (posting to pinot-user mailing list) - Apache Pinot Meetup Group: https://www.meetup.com/apache-pinot/
Check out Pinot documentation for a complete description of Pinot's features.
Pinot community has contributed libraries to interact with Apache Pinot with other languages.
- python-pinot-dbapi/pinot-dbapi - Python DB-API and SQLAlchemy dialect for Pinot
- fx19880617/pinot-client-go - A Golang Query Client for Pinot
Apache Pinot is under Apache License, Version 2.0