LINUX-20 SYSTEM LOGGING (RHEL-7) [PART-3]

LINUX-20 SYSTEM LOGGING (RHEL-7)

                           (PART-3)

RHEL7 SYSTEM LOGGING-P1
RHEL7 SYSTEM LOGGING-P2
RHEL7 SYSTEM LOGGING-P3
RHEL7 SYSTEM LOGGING-P4
RHEL7 SYSTEM LOGGING-P5
RHEL7 SYSTEM LOGGING-P6

………………LOGROTATE…………………

Log means, to capture any or specific type of activity, log generation and capturing is a continuous process. If this is true then also the log files will grow accordingly. So there must be a requirement to rotate, compress or remove such log files.

Here comes the Logrotate…..

[root@rhel7-server ~]# cat /etc/logrotate.conf

[root@rhel7-server ~]# ls -l /etc/logrotate.d

total 68

-rw-r--r--. 1 root root 178 Feb  5  2014 chrony

-rw-r--r--. 1 root root  71 Jan 25  2014 cups

-rw-r--r--. 1 root root 194 Mar 20  2014 httpd

-rw-r--r--. 1 root root 172 Mar 19  2014 iscsiuiolog

-rw-r--r--. 1 root root 165 Mar 24  2014 libvirtd

-rw-r--r--. 1 root root 162 Mar 24  2014 libvirtd.lxc

-rw-r--r--. 1 root root 163 Mar 24  2014 libvirtd.qemu

-rw-r--r--. 1 root root 106 Mar 26  2014 numad

-rw-r--r--. 1 root root 136 Jan 27  2014 ppp

-rw-r--r--. 1 root root 408 Jan 27  2014 psacct

-rw-r--r--. 1 root root 115 Apr  4  2014 samba

-rw-r--r--. 1 root root  71 Mar 25  2014 subscription-manager

-rw-r--r--. 1 root root 210 Mar 26  2014 syslog

-rw-r--r--. 1 root root  32 Dec  9  2013 up2date

-rw-r--r--. 1 root root 188 Mar  7  2014 vsftpd

-rw-r--r--. 1 root root 100 Jan 28  2014 wpa_supplicant

-rw-r--r--. 1 root root 100 Apr  8  2014 yum

[root@rhel7-server ~]# more /etc/logrotate.conf

# see "man logrotate" for details

# rotate log files weekly

weekly

# keep 4 weeks worth of backlogs

rotate 4

# create new (empty) log files after rotating old ones

create

# use date as a suffix of the rotated file

dateext

# uncomment this if you want your log files compressed

#compress

# RPM packages drop log rotation information into this directory

include /etc/logrotate.d è include all other configuration from directory /etc/logrotate.d

# no packages own wtmp and btmp -- we'll rotate them here

/var/log/wtmp {

    monthly

    create 0664 root utmp

        minsize 1M

    rotate 1

}

/var/log/btmp {

    missingok

    monthly

    create 0600 root utmp

    rotate 1

}

# system-specific logs may be also be configured here.

File is self-explanatory about its functions. For further details recommended to check,

[root@rhel7-server ~]# man logrotate

We can create new service logs to logrotate………………………..

[root@rhel7-server ~]# vi /etc/logrotate.d/ssh.conf

/var/log/ssh.log {

    daily

    missingok

    rotate 2

    notifempty

    compress

    size 20M

    create 0600 root root

}

========================

daily      logrotate will attempt to rotate the logs daily

rotate 2   means only 2 rotated logs should be kept. The oldest file will be

removed on the third run.

size=20M   Log file is rotated only if it grow bigger than 20M

compress   Old versions of log files are compressed with gzip(1) by default

missingok  Don’t output error if logfile is missing

notifempty Don’t rotate log file if it is empty

create     Creates a new log file with permissions 600 where owner and group is root user

Take a dry-run/test of our configuration………..

[root@rhel7-server ~]# logrotate -d /etc/logrotate.d/ssh.conf

reading config file /etc/logrotate.d/ssh.conf

Handling 1 logs

rotating pattern: /var/log/ssh.log  20971520 bytes (2 rotations)

empty log files are not rotated, old logs are removed

considering log /var/log/ssh.log

  log does not need rotating

Check the log timing details…

[root@rhel7-server ~]# ls -ltr /var/log

total 3220

drwx------. 2 root   root         6 Jan 27  2014 ppp

drwx------. 2 root   root         6 Jan 27  2014 speech-dispatcher

drwxr-xr-x. 2 chrony chrony       6 Feb  5  2014 chrony

drwx------. 2 root   root         6 Mar 20  2014 httpd

drwxr-xr-x. 2 root   root         6 Apr  2  2014 qemu-ga

======================o/p removed==========================

-rw-------. 1 root   root     22447 Jan 15 18:32 secure

-rw-rw-r--. 1 root   utmp     87936 Jan 15 18:32 wtmp

-rw-r--r--. 1 root   root   1460292 Jan 15 18:32 lastlog

-rw-r--r--. 1 root   root      8829 Jan 15 18:32 commands.log

-rw-r--r--. 1 root   root      3426 Jan 15 18:32 alllocal6

-rw-------. 1 root   root    943627 Jan 15 18:42 messages

-rw-r--r--. 1 root   root    202478 Jan 15 18:42 cron

Now force logrotate to rotate all logs with -f option…………

[root@rhel7-server ~]# logrotate -f /etc/logrotate.conf

This command will rotate all logs defined in /etc/logrotate.d directory.

[root@rhel7-server ~]# ls -ltr /var/log

total 2104

drwx------. 2 root   root         6 Jan 27  2014 ppp

drwx------. 2 root   root         6 Jan 27  2014 speech-dispatcher

drwxr-xr-x. 2 chrony chrony       6 Feb  5  2014 chrony

drwx------. 2 root   root         6 Mar 20  2014 httpd

drwxr-xr-x. 2 root   root         6 Apr  2  2014 qemu-ga

======================o/p removed==========================

-rw-------. 1 root   root     22447 Jan 15 18:32 secure-20170115

-rw-rw-r--. 1 root   utmp     87936 Jan 15 18:32 wtmp-20170115

-rw-r--r--. 1 root   root   1460292 Jan 15 18:32 lastlog

-rw-------. 1 root   root    943627 Jan 15 18:42 messages-20170115

-rw-r--r--. 1 root   root    202478 Jan 15 18:42 cron-20170115

drwxr-xr-x. 2 lp     sys       4096 Jan 15 18:42 cups

-rw-------. 1 root   root         0 Jan 15 18:42 ssh.log

drwxr-xr-x. 2 root   root      4096 Jan 15 18:42 rhsm

-rw-------. 1 root   root         0 Jan 15 18:42 secure

-rw-------. 1 root   root         0 Jan 15 18:42 maillog

-rw-r--r--. 1 root   root         0 Jan 15 18:42 cron

-rw-------. 1 root   root         0 Jan 15 18:42 spooler

-rw-------. 1 root   root       151 Jan 15 18:42 messages

-rw-rw-r--. 1 root   utmp         0 Jan 15 18:42 wtmp

-rw-------. 1 root   utmp         0 Jan 15 18:42 btmp

-rw-r--r--. 1 root   root      8985 Jan 15 18:42 commands.log

-rw-r--r--. 1 root   root      3582 Jan 15 18:42 alllocal6

Notice the changes in log files…??

Want to force the logrotate to execute on particular log file…??

[root@rhel7-server ~]# cat /var/log/ssh.log

this is my ssh log

[root@rhel7-server ~]# logrotate -f /etc/logrotate.d/ssh.conf

[root@rhel7-server ~]# ls -l /var/log/ssh.*

-rw-------. 1 root root   0 Jan 15 18:47 /var/log/ssh.log

-rw-------. 1 root root  37 Jan 15 18:47 /var/log/ssh.log.1.gz

-rw-r--r--. 1 root root 253 Jan 15 18:32 /var/log/ssh.log-20170115.gz

Want several log rotation in one file…??

[root@rhel7-server ~]# vi /etc/logrotate.d/ssh.conf

/var/log/ssh.log {

    daily

    missingok

    rotate 2

    notifempty

    compress

    size 20M

    dateext

    dateformat -%d%m%Y

    create 0600 root root

}

/var/log/commands.log {

daily

rotate 2

compress

size 20M

create 0600 root root

}

[root@rhel7-server ~]# logrotate -d /etc/logrotate.d/ssh.conf

reading config file /etc/logrotate.d/ssh.conf

Handling 2 logs

rotating pattern: /var/log/ssh.log  20971520 bytes (2 rotations)

empty log files are not rotated, old logs are removed

considering log /var/log/ssh.log

  log does not need rotating

rotating pattern: /var/log/commands.log  20971520 bytes (2 rotations)

empty log files are rotated, old logs are removed

considering log /var/log/commands.log

  log does not need rotating

[root@rhel7-server ~]# ls -ltr /var/log/{ssh,commands}*

-rw-r--r--. 1 root root 253 Jan 15 18:32 /var/log/ssh.log-20170115.gz

-rw-------. 1 root root  37 Jan 15 18:47 /var/log/ssh.log.1.gz

-rw-r--r--. 1 root root  20 Jan 15 19:05 /var/log/commands.log.1.gz

-rw-------. 1 root root 306 Jan 15 19:06 /var/log/ssh.log-15012017

-rw-r--r--. 1 root root   0 Jan 15 19:06 /var/log/ssh.log

-rw-r--r--. 1 root root   0 Jan 15 19:07 /var/log/commands.log

TESTING LOGROTATE……………..

logrotate – Run Status

Validate the logrotate run status for each configuration file and know the status of each one of them

[root@rhel7-server /]# cat /var/lib/logrotate.status

logrotate state -- version 2

"/var/log/yum.log" 2017-1-7-14:49:2

"/var/log/cups/page_log" 2016-11-21-19:0:0

"/var/log/cups/error_log" 2016-11-21-19:0:0

"/var/log/up2date" 2016-11-21-19:0:0

logrotate Manually: To run manual logrotate

[root@rhel7-server /]# logrotate -f /etc/logrotate.d/ssh.conf

logrotate Manually: With verbose option

[root@rhel7-server /]# logrotate -v -f /etc/logrotate.d/ssh.conf

reading config file /etc/logrotate.d/ssh.conf

Handling 2 logs

rotating pattern: /var/log/ssh.log  forced from command line (2 rotations)

empty log files are not rotated, old logs are removed

considering log /var/log/ssh.log

  log does not need rotating

rotating pattern: /var/log/commands.log  forced from command line (2 rotations)

empty log files are rotated, old logs are removed

considering log /var/log/commands.log

error: stat of /var/log/commands.log failed: No such file or directory

set default create context

logrotate Manually: With debug option

[root@rhel7-server /]# logrotate -d /etc/logrotate.conf

OR…

[root@rhel7-server /]# logrotate -d /etc/logrotate.conf >/logrotate-debug.txt

Flags in one go:

[root@rhel7-server ~]# logrotate -vdf /etc/logrotate.conf

For more details, please check

[root@rhel7-server ~]# man logrotate

HOW LOGROTATE REMEMBERS THE LAST RUN………

logrotate stores information about when it last rotated each log file.

[root@rhel7-server ~]# cat /var/lib/logrotate.status

logrotate state -- version 2

"/var/log/yum.log" 2017-1-7-14:49:2

"/var/log/cups/page_log" 2016-11-21-19:0:0

"/var/log/mytask.log" 2017-1-16-14:33:12

"/var/log/cups/error_log" 2016-11-21-19:0:0

Left side is the location of log file and at right side, the date when it was last rotated

So every time we need to force the logrotate to run…??

No… logrotate uses crontab to work. It's scheduled task, not a daemon, so no need to reload its configuration.

Let’s check it……..

[root@rhel7-server /]# ls -l /etc/cron.daily

total 20

-rwxr-xr-x. 1 root root 332 Apr  8  2014 0yum-daily.cron

-rwx------. 1 root root 180 Jul 31  2013 logrotate çç

-rwxr-xr-x. 1 root root 618 Mar 17  2014 man-db.cron

-rwxr-x---. 1 root root 192 Jan 26  2014 mlocate

-rwx------. 1 root root 256 Mar 25  2014 rhsmd

Well… it is scheduled to run daily. Good….

But what if system went down at that time…??

Curiosity is good, but up to this extent…??

Anyways… no issues, check

[root@rhel7-server /]# cat /etc/anacrontab

1       5       cron.daily              nice run-parts /etc/cron.daily

We already learned the way of working of “anacron” earlier.

PROBLEM WITH LOGROTATE…??

Check the status of logrotate,

[root@rhel7-server ~]# logrotate -d /etc/logrotate.conf

reading config file /etc/logrotate.conf

including /etc/logrotate.d

reading config file chrony

reading config file cups

reading config file httpd

reading config file iscsiuiolog

reading config file libvirtd

reading config file libvirtd.lxc

reading config file libvirtd.qemu

reading config file mytask.conf

error: mytask.conf:6 argument expected after size ççç

reading config file numad

reading config file ppp

reading config file psacct

reading config file samba

olddir is now /var/log/samba/old

reading config file ssh.conf

reading config file subscription-manager

reading config file syslog

reading config file up2date

reading config file vsftpd

reading config file wpa_supplicant

reading config file yum

error: /etc/logrotate.conf:26 unexpected } ççç

removing last 21 log configs

Found 2 error messages, these messages are solely related to config files.

Check the config files carefully and rectify the config issues.

Check the logrotate in debug mode,

[root@rhel7-server ~]# logrotate -d /etc/logrotate.d

rotating pattern: /var/log/mytask.log  weekly (2 rotations)

empty log files are rotated, old logs are removed

considering log /var/log/mytask.log

error: stat of /var/log/mytask.log failed: No such file or directory ççç

rotating pattern: /var/log/commands.log  20971520 bytes (2 rotations)

empty log files are rotated, old logs are removed

considering log /var/log/commands.log

error: stat of /var/log/commands.log failed: No such file or directory ççç

Found 2-2 entries from both commands having issue….

These can be further checked by,

[root@rhel7-server logrotate.d]# logrotate -d /etc/logrotate.d/mytask.conf

reading config file /etc/logrotate.d/mytask.conf

error: /etc/logrotate.d/mytask.conf:6 argument expected after size

Handling 1 logs

rotating pattern: /var/log/mytask.log  weekly (2 rotations)

empty log files are rotated, old logs are removed

considering log /var/log/mytask.log

error: stat of /var/log/mytask.log failed: No such file or directory

Two error messages,

1^st one indicating some configuration issue, let’s check it…

[root@rhel7-server logrotate.d]# vi mytask.conf

/var/log/mytask.log {

daily

weekly

rotate 2

nocompress

size ççç No size defined

}

2^nd error indicating no such file or directory in /var/log, let’s check it…

[root@rhel7-server ~]# ls -l /var/log/mytask*

ls: cannot access /var/log/mytask*: No such file or directory

Really there is no file, let’s create it…

[root@rhel7-server ~]# touch /var/log/mytask.log

[root@rhel7-server ~]# echo "test msg..1111..222..333" >/var/log/mytask.log

[root@rhel7-server ~]# cat /var/log/mytask.log

test msg..1111..222..333

Now file is created, let’s take a dry run

[root@rhel7-server ~]# logrotate -d /etc/logrotate.d/mytask.conf

reading config file /etc/logrotate.d/mytask.conf

Handling 1 logs

rotating pattern: /var/log/mytask.log  10485760 bytes (2 rotations)

empty log files are rotated, old logs are removed

considering log /var/log/mytask.log

  log does not need rotating

All good, now run it forcefully…

[root@rhel7-server ~]# logrotate -f /etc/logrotate.d/mytask.conf

Command successful, now again check the status…

[root@rhel7-server ~]# logrotate -d /etc/logrotate.d/mytask.conf

reading config file /etc/logrotate.d/mytask.conf

Handling 1 logs

rotating pattern: /var/log/mytask.log  10485760 bytes (2 rotations)

empty log files are rotated, old logs are removed

considering log /var/log/mytask.log

error: stat of /var/log/mytask.log failed: No such file or directory

OMG… file again gone…??

HOW……??

[root@rhel7-server ~]# ls -l /var/log/mytask*

-rw-------. 1 root root  0 Jan 16 14:26 /var/log/mytask.log.1

-rw-r--r--. 1 root root 25 Jan 16 14:24 /var/log/mytask.log.2

Rotated files are here, means whenever I create a file via “touch” it starts logging and when I force to rotate, they are rotated as well. But new files are not created.

Now time to consider the configuration…

[root@rhel7-server ~]# vi /etc/logrotate.d/mytask.conf

/var/log/mytask.log {

daily

weekly

rotate 2

nocompress

size 10M

create 0600 root root ççç Added this at end

}

Now, again do the same exercise…..

[root@rhel7-server ~]# touch /var/log/mytask.log

[root@rhel7-server ~]# echo "test msg2---88888888888888" >/var/log/mytask.log

[root@rhel7-server ~]# cat /var/log/mytask.log

test msg2---88888888888888

[root@rhel7-server ~]# logrotate -d /etc/logrotate.d/mytask.conf

reading config file /etc/logrotate.d/mytask.conf

Handling 1 logs

rotating pattern: /var/log/mytask.log  10485760 bytes (2 rotations)

empty log files are rotated, old logs are removed

considering log /var/log/mytask.log

  log does not need rotating

[root@rhel7-server ~]# logrotate -f /etc/logrotate.d/mytask.conf

[root@rhel7-server ~]# logrotate -d /etc/logrotate.d/mytask.conf

reading config file /etc/logrotate.d/mytask.conf

Handling 1 logs

rotating pattern: /var/log/mytask.log  10485760 bytes (2 rotations)

empty log files are rotated, old logs are removed

considering log /var/log/mytask.log

  log does not need rotating

[root@rhel7-server ~]# ls -l /var/log/mytask.log

-rw-------. 1 root root 0 Jan 16 14:33 /var/log/mytask.log

Now consider the 2^nd error coming from /var/log/commands.log. This log is configured in ssh.conf

[root@rhel7-server ~]# logrotate -d /etc/logrotate.d/ssh.conf

reading config file /etc/logrotate.d/ssh.conf

Handling 2 logs

rotating pattern: /var/log/ssh.log  20971520 bytes (2 rotations)

empty log files are not rotated, old logs are removed

considering log /var/log/ssh.log

  log does not need rotating

rotating pattern: /var/log/commands.log  20971520 bytes (2 rotations)

empty log files are rotated, old logs are removed

considering log /var/log/commands.log

error: stat of /var/log/commands.log failed: No such file or directory

This error is also same as we had seen in previous example.

Check for another error related to permissions,

[root@rhel7-server ~]# ls -ld /testdir1

drwxrwxrwx+   2 root root       100 Jan  4 19:18 testdir1

[root@rhel7-server ~]# cat /etc/logrotate.d/test.conf

/testdir1/test.log {

create 600 root root

daily

rotate 2

nocompress

size 10M

}

[root@rhel7-server ~]# logrotate -d /etc/logrotate.d/test.conf

reading config file /etc/logrotate.d/test.conf

Handling 1 logs

rotating pattern: /testdir1/test.log  10485760 bytes (2 rotations)

empty log files are rotated, old logs are removed

considering log /testdir1/test.log

error: skipping "/testdir1/test.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.

Now… change the permission,

[root@rhel7-server ~]# ls -ld /testdir1

drwx------+ 2 root root 115 Jan 16 15:03 /testdir1

[root@rhel7-server ~]# logrotate -d /etc/logrotate.d/test.conf

reading config file /etc/logrotate.d/test.conf

Handling 1 logs

rotating pattern: /testdir1/test.log  10485760 bytes (2 rotations)

empty log files are rotated, old logs are removed

considering log /testdir1/test.log

  log does not need rotating

Now… consider for another permission related issue…

[root@rhel7-server ~]# cat /etc/logrotate.d/test.conf

/testdir1/test.log {

create 640 user1 user1

daily

rotate 2

nocompress

size 10M

}

[root@rhel7-server ~]# su - user1

Last login: Mon Jan 16 14:58:27 IST 2017 on pts/3

[user1@rhel7-server ~]$ /usr/sbin/logrotate -d /etc/logrotate.d/test.conf

reading config file /etc/logrotate.d/test.conf

Handling 1 logs

rotating pattern: /testdir1/test.log  10485760 bytes (2 rotations)

empty log files are rotated, old logs are removed

considering log /testdir1/test.log

error: stat of /testdir1/test.log failed: Permission denied

This can be rectified by correcting the permission on parent dir.