Log management with ELK
•Download as PPTX, PDF•
7 likes•3,114 views
ELK (Elasticsearch, Logstash, Kibana) is an open source toolset for centralized logging, where Logstash collects, parses, and filters logs, Elasticsearch stores and indexes logs for search, and Kibana visualizes logs. Logstash processes logs through an input, filter, output pipeline using plugins. It can interpret various log formats and event types. Elasticsearch allows real-time search and scaling through replication/sharding. Kibana provides browser-based dashboards and visualization of Elasticsearch query results.
Report
Share
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (12)
Similar to Log management with ELK
Similar to Log management with ELK (20)
More from Geert Pante
More from Geert Pante (10)
Recently uploaded
Recently uploaded (20)
Log management with ELK
- 1. Log Management with ELK ELASTICSEARCH, LOGSTASH, KIBANA FOR CENTRALIZED LOGS
- 2. Purpose Centralized Log Management ◦ Collect, Parse and Filter using Logstash ◦ Store, Index and Search using Elasticsearch ◦ Visualize using Kibana Full open source stack ◦ Use for free ◦ Support plan from Elasticsearch company
- 3. Elasticsearch Real-time search engine ◦ Based on Apache Solr/Lucene ◦ Pure Java ◦ Document database ◦ Advanced text indexing ◦ Fuzzy search ◦ Replication/Sharding for true scalability
- 4. Logstash JRuby Based log processor Pluggable event pipeline ◦ Input plugins ◦ Filter plugins ◦ Codec plugins ◦ Output plugins DevOps Comunity ◦ Mix of developers, operations and system administrators
- 5. Kibana Browser based dashboard for ElasticSearch Visualization of query results ◦ Time Charts ◦ Filter any field ◦ Compare subsets
- 6. Logstash pipeline Define input, filters and outputs Simple configuration file Ruby syntax
- 7. Logstash not just for logs Interpretes different log formats ◦ Syslog messages ◦ Log4j with full details ◦ Apache log files Other event types too ◦ Ganglia server monitoring events ◦ SNMP events ◦ Windows EventLog Pre-proces before sending ◦ lumberjack
- 8. Logstash Inputs Rsyslog via TCP/UDP Log4j appender JMX Listener Logstash-forwarder File tails SNMP Ganglia …
- 10. Grok Filter Readable regex Predefined patterns for common log data Extract to properties ◦ Indexed properties
- 11. Metrics Filter Aggregate metrics ◦ Event rate using sliding windows ◦ 1 min ◦ 5 min ◦ 15 min ◦ Min/max/stddev/percentiles
- 12. Logstash Outputs
- 13. Statsd output Node.js based ◦ Counters ◦ Timers ◦ Graphite frontend
- 14. Alert outputs Send alerts ◦ Email ◦ Pagerduty ◦ XMPP/Jabber ◦ Hipchat ◦ Nagios Use treshold from metrics filter
- 15. ElasticSearch output Auto-creates new index per day Index all recognized fields Full text index, customizable indexer, mapper
- 16. Kibana dashboard
- 17. Scalability Easy deployment using chef/puppet/docker