LINUX-23 SYSTEM LOGGING (RHEL-7) (PART-6)

LINUX-23 SYSTEM LOGGING (RHEL-7)

                          (PART-6)

RHEL7 SYSTEM LOGGING-P1
RHEL7 SYSTEM LOGGING-P2
RHEL7 SYSTEM LOGGING-P3
RHEL7 SYSTEM LOGGING-P4
RHEL7 SYSTEM LOGGING-P5
RHEL7 SYSTEM LOGGING-P6

-------JOURNAL-2--------

We almost covered journal, except few things.

·         The configuration file.

·         How to make journal logs persistent.

H  How a regular user can access journal logs.

H  How to determine journal log size.

First let’s check how to make the journal logs persistent, so that we can check previous boot logs also.

[root@rhel7-client ~]# mkdir -p /var/log/journal

[root@rhel7-client ~]# ls -ld /var/log/journal

drwxr-xr-x. 3 root root 45 Jan 25 19:14 /var/log/journal

[root@rhel7-client ~]# ls -ltr /var/log/journal

total 0

drwxr-xr-x. 2 root root 27 Jan 25 19:14 7d3eade1a0b140cca61f2d12bd64b4d0

[root@rhel7-client /]# vi /etc/systemd/journald.conf

#Storage=auto

Storage=persistent çç change here from “auto” to “persistent”

[root@rhel7-client ~]# systemctl restart systemd-journald

[root@rhel7-client ~]# killall -USR1 systemd-journald

now the logs are written under /var/log/journal, but this is also not for infinite time.

·         Journal has built-in log rotation facility.

·         Journal is limited to 10% max of the file system on which it is residing.

·         Journal stops growing if the total FS size is less than equal to 15%.

·         Values are configurable.

Whichever comes earlier, either journal reached to 10% of FS size or there is only 15% free of total FS size, then journal drop old messages to make room for newer logs.

[root@rhel7-client ~]# journalctl -b -1

Failed to look up boot -1: Cannot assign requested address

[root@rhel7-client ~]# last |grep reboot

reboot   system boot  3.10.0-121.el7.x Wed Jan 25 19:36 - 19:41  (00:04)

reboot   system boot  3.10.0-121.el7.x Wed Jan 25 19:25 - 19:36  (00:10)

reboot   system boot  3.10.0-121.el7.x Sat Jan 21 11:10 - 19:25 (4+08:14)

[root@rhel7-client ~]# journalctl -b -0

-- Logs begin at Sat 2017-01-21 11:10:38 IST, end at Wed 2017-01-25 19:42:16 IST. --

Jan 25 19:36:29 rhel7-test2 systemd-journal[143]: Runtime journal is using 6.1M (max 49.5M, leaving 74.3M of free 489.1M, current limi

Jan 25 19:36:29 rhel7-test2 systemd-journal[143]: Runtime journal is using 6.1M (max 49.5M, leaving 74.3M of free 489.1M, current limi

Jan 25 19:36:29 rhel7-test2 kernel: Initializing cgroup subsys cpuset

[root@rhel7-client ~]# journalctl -b -1

-- Logs begin at Sat 2017-01-21 11:10:38 IST, end at Wed 2017-01-25 19:42:52 IST. --

Jan 25 19:25:50 rhel7-test2 systemd-journal[143]: Runtime journal is using 6.1M (max 49.5M, leaving 74.3M of free 489.1M, current limi

Jan 25 19:25:50 rhel7-test2 systemd-journal[143]: Runtime journal is using 6.1M (max 49.5M, leaving 74.3M of free 489.1M, current limi

Jan 25 19:25:50 rhel7-test2 kernel: Initializing cgroup subsys cpuset

[root@rhel7-client ~]# journalctl -b -2

-- Logs begin at Sat 2017-01-21 11:10:38 IST, end at Wed 2017-01-25 19:43:03 IST. --

Jan 21 11:10:38 rhel7-test2 systemd-journal[143]: Runtime journal is using 6.1M (max 49.5M, leaving 74.3M of free 489.1M, current limi

Jan 21 11:10:38 rhel7-test2 systemd-journal[143]: Runtime journal is using 6.1M (max 49.5M, leaving 74.3M of free 489.1M, current limi

Jan 21 11:10:38 rhel7-test2 kernel: Initializing cgroup subsys cpuset

[root@rhel7-client ~]# journalctl -b -3

Failed to look up boot -3: Cannot assign requested address

0 for last boot

1 for 2^nd last boot

2 for 3^rd last boot

…so on…

CONFIGURATION FILE:

/etc/systemd/journald.conf

[root@rhel7-client /]# cat /etc/systemd/journald.conf

#  This file is part of systemd.

#

#  systemd is free software; you can redistribute it and/or modify it

#  under the terms of the GNU Lesser General Public License as published by

#  the Free Software Foundation; either version 2.1 of the License, or

#  (at your option) any later version.

#

# See journald.conf(5) for details

[Journal]

#Storage=auto

#Compress=yes

#Seal=yes

#SplitMode=login

#SyncIntervalSec=5m

#RateLimitInterval=30s

#RateLimitBurst=1000

#SystemMaxUse=

#SystemKeepFree=

#SystemMaxFileSize=

#RuntimeMaxUse=

#RuntimeKeepFree=

#RuntimeMaxFileSize=

#MaxRetentionSec=

#MaxFileSec=1month

#ForwardToSyslog=yes

#ForwardToKMsg=no

#ForwardToConsole=no

#TTYPath=/dev/console

#MaxLevelStore=debug

#MaxLevelSyslog=debug

#MaxLevelKMsg=notice

#MaxLevelConsole=info

We can control the log grow by altering following parameters,

SystemMaxUse:         Maximum disk space

SystemKeepFree:       Amount of space that should be kept available

SystemMaxFileSize:    How large individual journal files can become

                      before being rotated

RuntimeMaxUse:        Maximum disk space that can be used within volatile

                      storage

RuntimeKeepFree:      The amount of space that should be kept available

                      for others in volatile storage

RuntimeMaxFilesize:   How large individual journal file can become

                      in volatile storage (within /run filesystem)

                      before being rotated

For detailed info it is suggested to navigate,

[root@rhel7-client /]# man journald.conf

Sometimes unclean reboot causes corruption in journal logs, it can be found by,

[root@rhel7-client ~]# journalctl --verify

PASS: /var/log/journal/7d3eade1a0b140cca61f2d12bd64b4d0/system@2d4274fe941e42ffa8e290efda7fd9ae-0000000000002671-000546ffb82195ba.journal

PASS: /var/log/journal/7d3eade1a0b140cca61f2d12bd64b4d0/system@2d4274fe941e42ffa8e290efda7fd9ae-00000000000030c1-0005470d985a2bf8.journal

PASS: /var/log/journal/7d3eade1a0b140cca61f2d12bd64b4d0/system@2d4274fe941e42ffa8e290efda7fd9ae-0000000000003ae5-0005470dce76ed7a.journal

PASS: /var/log/journal/7d3eade1a0b140cca61f2d12bd64b4d0/system.journal

PASS: /var/log/journal/7d3eade1a0b140cca61f2d12bd64b4d0/user-42.journal

PASS: /var/log/journal/7d3eade1a0b140cca61f2d12bd64b4d0/user-1002.journal

DETERMINE JOURNAL SIZE:

[root@rhel7-client ~]# grep -v "#" /etc/systemd/journald.conf

[Journal]

Storage=persistent

SystemMaxUse=20M

[root@rhel7-client ~]# du -sh /var/log/journal

21M     /var/log/journal

[root@rhel7-client ~]# journalctl --disk-usage

Journals take up 20.0M on disk.

After some reboots with above config,

[root@rhel7-client ~]# du -sh /var/log/journal                                  53M     /var/log/journal

[root@rhel7-client ~]# journalctl --disk-usage                                  Journals take up 44.0M on disk.

It’s not working to limit the size of journal logs.

Now the new config below and reboot to create logs,

[root@rhel7-client ~]# grep -v "#" /etc/systemd/journald.conf

[Journal]

Storage=persistent

SystemMaxUse=20M

SystemMaxFileSize=10M

[root@rhel7-client ~]# journalctl --disk-usage

Journals take up 32.0M on disk.

[root@rhel7-client ~]# du -sh /var/log/journal

33M     /var/log/journal

CAN REGULAR USER’S HAVE ALSO JOURNAL LOGS…??

[user1@rhel7-client ~]$ journalctl

No journal files were found.

No logs yet, let’s do some acticvity.

[user1@rhel7-client ~]$ sftp 192.168.234.142

[user1@rhel7-client ~]$ journalctl

-- Logs begin at Sat 2017-01-28 14:21:06 IST, end at Sat 2017-01-28 14:21:13 IST. --

Jan 28 14:21:06 rhel7-client sshd[3044]: subsystem request for sftp by user user1

Jan 28 14:21:07 rhel7-client sftp-server[3045]: session opened for local user user1 from [192.168.234.142]

Jan 28 14:21:13 rhel7-client sftp-server[3045]: opendir "/home/user1"

Jan 28 14:21:13 rhel7-client sftp-server[3045]: closedir "/home/user1"

It mean that user can also have access to journal logs provided it should be generated by them only.

WHAT IF A REGULAR USER WANTS TO ACCESS ENTIRE JOURNAL LOG…??

By default, Journal users without root privileges can only see log files generated by them, unless they are part of “adm” group.

[root@rhel7-client ~]# id user1

uid=1002(user1) gid=1002(user1) groups=1002(user1),5000(admins1),5001(monster)

[root@rhel7-client ~]# usermod -a -G adm user1

[root@rhel7-client ~]# id user1

uid=1002(user1) gid=1002(user1) groups=1002(user1),4(adm),5000(admins1),5001(monster)

It works only when persistent storage is enabled for Journal.

Can “user1” now view journal logs…??

[user1@rhel7-client ~]$ journalctl -u crond

-- Logs begin at Sat 2017-01-28 14:21:06 IST, end at Sat 2017-01-28 14:21:13 IST. --

[user1@rhel7-client ~]$ journalctl

-- Logs begin at Sat 2017-01-28 14:21:06 IST, end at Sat 2017-01-28 14:21:13 IST. --

Jan 28 14:21:06 rhel7-client sshd[3044]: subsystem request for sftp by user user1

Jan 28 14:21:07 rhel7-client sftp-server[3045]: session opened for local user user1 from [192.168.234.142]

Jan 28 14:21:13 rhel7-client sftp-server[3045]: opendir "/home/user1"

Jan 28 14:21:13 rhel7-client sftp-server[3045]: closedir "/home/user1"

Still the same old info for user1.

Rebooted several times but still same log info.

Now what to do…??

[root@rhel7-client ~]# chown root:systemd-journal /var/log/journal

[root@rhel7-client ~]# ls -ld /var/log/journal

[root@rhel7-client ~]# chmod 2755 /var/log/journal

[root@rhel7-client ~]# usermod -a -G systemd-journal user1

[root@rhel7-client ~]# id user1

uid=1002(user1) gid=1002(user1) groups=1002(user1),4(adm),190(systemd-journal),5000(admins1),5001(monster)

[root@rhel7-client ~]# systemctl restart systemd-journald

[root@rhel7-client ~]# su - user1

Last login: Sat Jan 28 15:00:31 IST 2017 on pts/0

[user1@rhel7-client ~]$

[user1@rhel7-client ~]$ journalctl

-- Logs begin at Thu 2017-01-26 19:26:43 IST, end at Sat 2017-01-28 15:00:53 IST. --

Jan 26 19:26:43 rhel7-client systemd-journal[432]: Time spent on flushing to /var is 574.800ms for 1763 entries.

Jan 26 19:26:42 rhel7-client systemd[1]: Started Trigger Flushing of Journal to Persistent Storage.

Jan 26 19:26:42 rhel7-client systemd[1]: Started Tell Plymouth To Write Out Runtime Data.

Jan 26 19:26:43 rhel7-client systemd[1]: Started Create Volatile Files and Directories.

Jan 26 19:26:43 rhel7-client systemd[1]: Started Security Auditing Service.

Jan 26 19:26:43 rhel7-client systemd[1]: Starting Update UTMP about System Reboot/Shutdown...

Now it is working.

So we need to make the /var/log/journal dir under “systemd-journald” group with sticky bit enabled, and also the user should be part of that group.

[root@rhel7-client ~]# man systemd-journald.service

[root@rhel7-client ~]# journalctl --vacuum-size=20M

journalctl: unrecognized option '--vacuum-size=20M'

[root@rhel7-client ~]# journalctl --list-boots

journalctl: unrecognized option '--list-boots'

I think due to older version above commands are not working, I did not verified the version/update issue so I cannot be sure why they are not running. Need to verify.

[root@rhel7-client ~]# journalctl --version

systemd 208

+PAM +LIBWRAP +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ