If you spend lot of time in Linux environment, it is essential that you know where the log files are located, and what is contained in each and every log file.
When your system is running smoothly, take some time to learn and understand the content of various log files, which will help you when there is a crisis and you have to look though the log files to identify the issue.
/etc/rsyslog.conf controls what goes inside some of the log files. For example, following is the entry in rsyslog.conf for /var/log/messages.
$ grep "/var/log/messages" /etc/rsyslog.conf *.info;mail.none;authpriv.none;cron.none /var/log/messages
In the above output,
- *.info indicates that all logs with type INFO will be logged.
- mail.none,authpriv.none,cron.none indicates that those error messages should not be logged into the /var/log/messages file.
- You can also specify *.none, which indicates that none of the log messages will be logged.
The following are the 20 different log files that are located under /var/log/ directory. Some of these log files are distribution specific. For example, you’ll see dpkg.log on Debian based systems (for example, on Ubuntu).
- /var/log/messages– Contains global system messages, including the messages that are logged during system startup. There are several things that are logged in /var/log/messages including mail, cron, daemon, kern, auth, etc.
- /var/log/dmesg– Contains kernel ring buffer information. When the system boots up, it prints number of messages on the screen that displays information about the hardware devices that the kernel detects during boot process. These messages are available in kernel ring buffer and whenever the new message comes the old message gets overwritten. You can also view the content of this file using the dmesg command.
- /var/log/auth.log – Contains system authorization information, including user logins and authentication machinsm that were used.
- /var/log/boot.log – Contains information that are logged when the system boots
- /var/log/daemon.log – Contains information logged by the various background daemons that runs on the system
- /var/log/dpkg.log – Contains information that are logged when a package is installed or removed using dpkg command
- /var/log/kern.log – Contains information logged by the kernel. Helpful for you to troubleshoot a custom-built kernel.
- /var/log/lastlog– Displays the recent login information for all the users. This is not an ascii file. You should use lastlog command to view the content of this file.
- /var/log/maillog /var/log/mail.log – Contains the log information from the mail server that is running on the system. For example, sendmail logs information about all the sent items to this file
- /var/log/user.log – Contains information about all user level logs
- /var/log/Xorg.x.log – Log messages from the X
- /var/log/alternatives.log – Information by the update-alternatives are logged into this log file. On Ubuntu, update-alternatives maintains symbolic links determining default commands.
- /var/log/btmp (lastb command; shows all bad login attempts) /var/log/wtmp (displays all users logged in and out since the file is created…last command;login attempts)– This file contains information about failed login attemps. Use the last command to view the btmp file. For example, “last -f /var/log/btmp | more”
- /var/log/cups– All printer and printing related log messages
- /var/log/anaconda.log – When you install Linux, all installation related messages are stored in this log file
- /var/log/yum.log – Contains information that are logged when a package is installed using yum
- /var/log/cron– Whenever cron daemon(or anacron) starts a cron job, it logs the information about the cron job in this file
- /var/log/secure– Contains information related to authentication and authorization privileges. For example, sshd logs all the messages here, including unsuccessful login.
- /var/log/wtmp or /var/log/utmp– Contains login records. Using wtmp you can find out who is logged into the system. who command uses this file to display the information.
- /var/log/faillog– Contains user failed login attemps. Use faillog command to display the content of this file.
Apart from the above log files, /var/log directory may also contain the following sub-directories depending on the application that is running on your system.
- /var/log/httpd/(or) /var/log/apache2– Contains the apache web server access_log and error_log
- /var/log/lighttpd/– Contains light HTTPD access_log and error_log
- /var/log/conman/– Log files for ConMan client. conman connects remote consoles that are managed by conmand daemon.
- /var/log/mail/– This subdirectory contains additional logs from your mail server. For example, sendmail stores the collected mail statistics in /var/log/mail/statistics file
- /var/log/prelink/– prelink program modifies shared libraries and linked binaries to speed up the startup process. /var/log/prelink/prelink.log contains the information about the .so file that was modified by the prelink.
- /var/log/audit/– Contains logs information stored by the Linux audit daemon (auditd).
- /var/log/setroubleshoot/– SELinux uses setroubleshootd (SE Trouble Shoot Daemon) to notify about issues in the security context of files, and logs those information in this log file.
- /var/log/samba/– Contains log information stored by samba, which is used to connect Windows to Linux.
- /var/log/sa/– Contains the daily sar files that are collected by the sysstat package.
- /var/log/sssd/– Use by system security services daemon that manage access to remote directories and authentication mechanisms.
Viewing huge log files for trouble shooting is a mundane routine tasks for sysadmins and programmers. In this article, let us review how to effectively view and manipulate huge log files using 10 awesome examples.
Example 1: Display specific lines (based on line number) of a file using sed command
View only the specific lines mentioned by line numbers.
Syntax: $ sed -n -e Xp -e Yp FILENAME
- sed : sed command, which will print all the lines by default.
- -n : Suppresses output.
- -e CMD : Command to be executed
- Xp: Print line number X
- Yp: Print line number Y
- FILENAME : name of the file to be processed.
The example mentioned below will print the lines 120, 145, 1050 from the syslog.
$ sed -n -e 120p -e 145p -e 1050p /var/log/syslog
In the following example, you can view the content of var/log/cron from line number 101 to 110.
- M – Starting line number
- N – Ending line number
Syntax: sed -n M,Np FILENAME $ sed -n 101,110p /var/log/cron
Example 2: Display first N lines of a file using head command
This example displays only first 15 lines of /var/log/maillog file. Change 15 to 10 to display the first 10 lines of a log file.
Syntax: head -n N FILENAME
$ head -n 15 /var/log/maillog
Example 3: Ignore last N lines of a file using head command
This example shows how to ignore the last N lines, and show only the remaining lines from the top of file. The following example will display all the lines of the /var/log/secure except the last 250 lines.
Syntax: head -n -N FILENAME $ head -n -250 /var/log/secure
Example 4: Display last N lines of the file using tail command
This example displays only last 50 lines of /var/log/messages file. Change 50 to 100 to display the last 100 lines of the log file.
Syntax: tail -n N FILENAME $ tail -n 50 /var/log/messages
Example 5: Ignore first N-1 lines of the file using tail command
This example shows how to ignore the first N-1 lines and show only the remaining of the lines. The following example ignores the 1st four lines of the /etc/xinetd.conf, which contains only the comments.
Syntax: tail -n +N FILENAME
$ tail -n +5 /etc/xinetd.conf
defaults
{
instances = 60
log_type = SYSLOG authpriv
log_on_success = HOST PID
log_on_failure = HOST
cps = 25 30
}
includedir /etc/xinetd.dExample 6: View growing log file in real time using tail command
This is probably one of the most used command by sysadmins.To view a growing log file and see only the newer contents use tail -f as shown below.The following example shows the content of the /var/log/syslog command in real-time.
Syntax: tail -f FILENAME $ tail -f /var/log/syslog
Example 7: Display specific lines (based on line number) of a file using head and tail command
The example below will display line numbers 101 – 110 of /var/log/anaconda.log file
- M – Starting line number
- N – Ending line number
Syntax: cat file | tail -n +N | head -n (M-N+1) $ cat /var/log/anaconda.log | tail -n +101 | head -n 10
- cat : prints the whole file to the stdout.
- tail -n +101 : ignores lines upto the given line number, and then start printing lines after the given number.
- head -n 10 : prints the first 10 line, that is 101 to 110 and ignores the remaining lines.
Example 8: Display lines matching a pattern, and few lines following the match.
The following example displays the line that matches “ Initializing CPU” from the /var/log/dmesg and 5 lines immediately after this match.
# grep "Initializing CPU#1" /var/log/dmesg Initializing CPU#1 [Note: The above shows only the line matching the pattern] # grep -A 5 "Initializing CPU#1" dmesg Initializing CPU#1 Calibrating delay using timer specific routine.. 3989.96 BogoMIPS (lpj=1994982) CPU: After generic identify, caps: bfebfbff 20100000 00000000 00000000 CPU: After vendor identify, caps: bfebfbff 20100000 00000000 00000000 monitor/mwait feature present. CPU: L1 I cache: 32K, L1 D cache: 32K [Note: The above shows the line and 5 lines after the pattern matching]
Example 9: Displaying specific bytes from a file.
The following example explains how to display either the top 40 or the last 30 bytes of a file.
Display first 40 bytes from syslog.
$ head -c40 /var/log/syslog
Display last 30 bytes from syslog.
$ tail -c30 /var/log/syslog
Example 10: Viewing compressed log files
After a specific time all the system log files are rotated, and compressed. You can uncompress it on the fly, and pipe the output to another unix command to view the file as explained below.
- Display the first N lines of a compressed file.
$ zcat file.gz | head -250
- Display the last N lines of a compressed file.
$ zcat file.gz | tail -250
- Ignoring the last N lines of a compressed file.
$ zcat file.gz | head -n -250
- Ignoring the first N lines of a compressed file.
$ zcat file.gz | tail -n +250
- Viewing the lines matching the pattern
$ zcat file.gz | grep -A2 'error'
- Viewing particular range of lines identified by line number.
$ zcat file.gz | sed -n -e 45p -e 52p
