Without the right tools to aggregate and parse your log data, finding and understanding the information you’re looking for is nearly impossible
There are endless uses for logs because the logs themselves are endless. Application logs, security logs, BI logs, Lincoln logs (Ok, maybe not)…
Setting aside security, BI and childhood fun for a moment, even looking specifically at uses for application logs alone, there are countless uses. Production monitoring, Performance monitoring, Support, Troubleshooting, QA resolution, and more. It really all comes down to how and what you log. All of the tools here can be used to get a better understanding and more value out of your logs, but they also have their own strengths and weaknesses. In order to get the most out of your tooling, it’s important to select it based on your needs and goals.
Logging Best Practices (Getting the Most Out of Your Tooling)
First and foremost, we need to be aware of what our goals are. Why are we adding log statements to our code in the first place? Do we want to use them for application monitoring? Support and troubleshooting? Security? Depending on what your goals are, your entire approach to logging and the tools you will need may be change.
Once the purpose of the logs is settled, it’s important to structure them so that they are understandable both to yourself and the members of your team and to whichever logging tool you choose to use. JSON and KVP (Key Value Pair) are both good choices.
Logs generate a massive amount of data, and this data may be coming from multiple environments across many servers. To ensure that this data isn’t lost and can be used effectively, they should be consolidated and centralized to a single storage location. Of course, this can be costly which is why, in almost all situations, TRACE, INFO and DEBUG-level logs are turned off in production.
Finally, we need to be aware of the limitations of our logged data. Our first warning sign is that those log-levels that we’re turning off in production (TRACE, INFO, DEBUG) makeup close to two-thirds of all of our logs. That’s a lot of missing information.
Logs have been used for troubleshooting and support for ages, but considering the limited context they provide for application errors, they’re much better suited for other purposes. These log aggregation and analysis tools are most helpful for security and BI purposes or for identifying trends in user events and activities.
And a final note, don’t think that logs are the only data you can get your hands on. These tools all integrate with others which allows you to combine the data parsing power of your logging tools with more meaningful application data.
6 Recommended Log Management Tools
- Splunk
- Power-house enterprise solution
- On-premise (w/option now for SaaS)
- Elastic
- Logstash for logs
- Elastic for search
- Kibana for visualization
- Complex set up
- Sumo Logic
- SaaS competitor to Splunk
- Enterprise-worthy
- Loggly
- More for developers and DevOps (less enterprise-y)
- Parses data from App servers
- PaperTrail
- Straight-forward log aggregator
- Without all the bells and whistles
- GrayLog
- Also for developers
- Open source
- Newer to the space but working hard to be enterprise-ready
- Can handle an extensive range of data formats
1. Splunk
Splunk is the biggest tool in the log management space. It’s well-established, full-featured, and enterprise-class. It’s unique in this space as an on-premises tool (although they have come out with a Cloud version as well).
When to use it: Enterprise companies with lots of feature needs and a variety of data that needs analyzing.
Pricing:
-Enterprise: $50-173/GB/month, depending on data volume
-Cloud: Contact for pricing info
-Light: $87/GB/month for up to 20GB/day (up to 5 users only)
Pros:
Most feature-rich (more than 500 apps)
Built-in search and visualization tools
Good for security, BI and infrastructure monitoring
Cons:
Complex setup and maintenance
High cost to support real-world applications
2. Elastic
Elastic (formerly ELK – ElasticSearch, Logstash, Kibana) is an open source project made up of many different tools for application data analysis and visualization. Logstash, specifically, was made for the collection and management of log files. Beyond log aggregation, it includes ElasticSearch for indexing and searching through data and Kibana for charting and visualizing data. Together they form a powerful log management solution.
When to use it: If you want an open source tool. If you’re interested in implementing the entire Elastic stack or at least see value separately in using ElasticSearch or Kibana and want the interactive benefits that come from combining these tools.
Pricing: Free
Pros:
Open source tools come with a lot of control
Uses 3 mature components to form 1 powerful solution
Quick and easy setup for an open source solution
Cons:
Components need to be treated essentially as individual products
Logstash filters are written in Ruby, Kibana is pure Javascript, and ElasticSearch has its own REST API as well as JSON templates
3. Sumo Logic
Sumo Logic was founded as a SaaS version of Splunk, going so far as to imitate some of Splunk’s features and visuals early on. Since then, Sumo Logic has developed into a full-fledged enterprise-class log management solution in its own right. Sumo Logic is the most enterprise-focused of the cloud-native log analyzers.
When to use it: If you’re an enterprise-type company but are willing to sacrifice some features for the benefits of SaaS, Sumo Logic is worth exploring. It’s also good if you have a strong focus on security. It’s not just developer-oriented as a tool either, with benefits for security teams and business purposes.
Pricing:
-Enterprise: $150/GB/month for minimum 3GB
-Professional: $90/GB/month for minimum 3GB
-Free: Free up to 500MB
Pros:
Easy setup compared to on-prem solutions
Advanced analytics and machine learning for logs, metrics and external data
Baselines are tracked for anomaly detection purposes
Cons:
Less extensive feature list than Splunk
4. Loggly
Loggly is a robust log analyzer, focusing on simplicity and ease of use. It’s targeted for developers and DevOps – making it less enterprise-focused.
When to use it: Primary use cases are for troubleshooting and customer support scenarios. It’s a good tool for a DevOps team.
Pricing:
-Enterprise: Starting at $349/month
-Pro: Starting at $199/month, up to 5 users
-Standard: Starting at $79/month, up to 3 users
-Free: Single User
Pros:
Very developer friendly
Easily create custom performance and DevOps dashboards
Cons:
Best suited for parsing app data for smaller Dev or DevOps teams
Will not perform as full-blown infrastructure/security/analytics solution
5. PaperTrail
PaperTrail is a simple way to look and search through logs from multiple machines, in one consolidated easy-to-use interface. It’s a SaaS tool designed to enhance the logs you already collect or generate.
When to use it: If you want a simple and straightforward tool without lots of extra bells and whistles. If you want a stripped down and basic log analyzer that is good for looking at log files in aggregation and doesn’t try to be anything more.
Pricing:
Ranges from $7/month for 1GB to $395/month for 50GB with option to customize a plan to suit volume requirements
Pros:
Simple way to look at log files from multiple machines in a singular view in the cloud
Quick setup
Very affordable
Cons:
Mostly text-based, UX is similar to looking at an actual log
No advanced integrations, predictive, or reporting capabilities
6. Graylog
Graylog is an open-source log analyzer backed by MongoDB as well as ElasticSearch (similar to Logstash) for storing and searching log errors. It’s mainly focused on helping developers detect and fix errors in their apps, but they’ve also released an official enterprise-ready platform.
When to use it: Graylog is more targeted towards developers than other open source log management tools. Plus, if you want a log management tool that aims to be both enterprise-ready and is open source, Graylog definitely deserves to be in the mix with Elastic/Logstash.
Pricing: Free
Pros:
Can handle an extensive range of data formats
Fine grained control for authentication and user permissions
Alerting for streams you designate makes for quick analytics
Uses a REST API for distributing and being sent data
Cons:
Not particularly management friendly on the dashboard front
Reporting functionality is a bit lacking
As a newer tool, particularly on the enterprise front, it’s not as mature and fully vetted as other options
Final Thoughts
That was a lot of information to absorb. Let’s just sum that all up with the basic use cases for each tool, shall we?
Splunk is the best “out-of-the-box” tool for enterprise companies where money is less of a concern. Elastic is the strongest open-source project with complex setup and maintenance being the downside. Sumo Logic is basically the SaaS version of Splunk but it’s less expensive and has a less-extensive feature list. Loggly is a solid solution for smaller Dev and DevOps teams focusing on monitoring and troubleshooting. PaperTrail is a simple and affordable tool for viewing log files from multiple machines in a singular view in the cloud. Graylog is a solid alternative to Logstash within the Elastic Stack framework.
One last final thought… These log management tools help you understand your log files, but only based on the information that your log files contain. In cases where an error occurs in production, the logs most likely won’t contain the data that you need (the source code and variable state, for example).
| Published on Java Code Geeks with permission by Tali Soroker, partner at our JCG program. See the original article here: 6 Log Management Tools You NEED to Know (And How to Use Them) Opinions expressed by Java Code Geeks contributors are their own. |
