LINUX-18 SYSTEM LOGGING (RHEL-7) [PART-1]

LINUX-18 SYSTEM LOGGING (RHEL-7)

                  (PART-1)

RHEL7 SYSTEM LOGGING-P1
RHEL7 SYSTEM LOGGING-P2
RHEL7 SYSTEM LOGGING-P3
RHEL7 SYSTEM LOGGING-P4
RHEL7 SYSTEM LOGGING-P5
RHEL7 SYSTEM LOGGING-P6

It’s very easy to hit,

[root@rhel7-server ~]# tail -f /var/log/messages

[root@rhel7-server ~]# tail -f /var/log/cron

…………etc……etc……etc….

And get all info,

But what info and how…??

“rsyslogd” is daemon which is responsible to collect logs.

In RHEL-7, an enhanced logging daemon is introduced called “system-journeld”.

Let’s see the conventional first.

“RSYSLOGD”

To retrieve “system logs” the daemon “rsyslogd” is dependent upon it’s configuration file.

/etc/rsyslog.conf

Default port used by this daemon is 514

Service provided by “rsyslogd” is “rsyslog” which is further controlled by “systemctl”

Controlling rsyslog service……..

[root@rhel7-server ~]# systemctl status rsyslog

rsyslog.service - System Logging Service

   Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled)

   Active: active (running) since Fri 2017-01-13 13:47:36 IST; 1h 18min ago

 Main PID: 654 (rsyslogd)

   CGroup: /system.slice/rsyslog.service

           └─654 /usr/sbin/rsyslogd -n

Jan 13 13:47:36 rhel7-server systemd[1]: Started System Logging Service.

[root@rhel7-server ~]# systemctl stop rsyslog

[root@rhel7-server ~]# systemctl start rsyslog

[root@rhel7-server ~]# systemctl status rsyslog

rsyslog.service - System Logging Service

   Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled)

   Active: active (running) since Fri 2017-01-13 15:07:48 IST; 2s ago

 Main PID: 4392 (rsyslogd)

   CGroup: /system.slice/rsyslog.service

           └─4392 /usr/sbin/rsyslogd -n

Jan 13 15:07:48 rhel7-server systemd[1]: Started System Logging Service.

Intersection of /etc/rsyslog.conf……………..

If we see the file carefully, then we can find that the file is distributed in three major parts…

1.  #### MODULES ####

2.  #### GLOBAL DIRECTIVES ####

3.  #### RULES ####

#### MODULES ####

# The imjournal module below is now used as a message source instead of imuxsock.

$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)

$ModLoad imjournal # provides access to the systemd journal

#$ModLoad imklog # reads kernel messages (the same are read from journald)

#$ModLoad immark  # provides --MARK-- message capability

# Provides UDP syslog reception

#$ModLoad imudp

#$UDPServerRun 514

# Provides TCP syslog reception

#$ModLoad imtcp

#$InputTCPServerRun 514

Input Modules — Input modules gather messages from various sources.The name of an input module always starts with the im prefix, such as imfile and imjournal.

Output Modules — Output modules provide a facility to issue message to     various targets such as sending across a network, storing in a database, or encrypting. The name of an output module always starts with the om prefix, such as omsnmp, omrelp, and so on.

Parser Modules — These modules are useful in creating custom parsing rules or to parse malformed messages. The name of a parser module always  starts with the pm prefix, such as pmrfc5424, pmrfc3164, and so on.                                          

Things to consider are modules “imjournal”, “imuxsock” & "imklog"

ModLoad is a directive which loads these modules on demand.

imuxsock # provides support for local system logging (e.g. via logger command) imuxsock Provides the ability to accept syslog messages via local Unix sockets. Most importantly, 
this is the mechanism by which the syslog(3) call delivers syslog messages to rsyslogd.

imjournal # provides access to the systemd journal. The journal provides imuxsock with a copy of all “classical” syslog messages, however, it does not provide structured data.

imklog # reads kernel messages (the same are read from journald). imklog generates some messages of itself (e.g. on problems, startup and shutdown) and these do not stem from the kernel.
Historically, in Linux, these too have "kern" facility. Thus, on Linux platforms the default is "kern" while on others it is "syslogd".
inmark # mrking log files

#### GLOBAL DIRECTIVES ####

# Where to place auxiliary files

$WorkDirectory /var/lib/rsyslog

# Use default timestamp format

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,

# not useful and an extreme performance hit

#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog.d/

$IncludeConfig /etc/rsyslog.d/*.conf

# Turn off message reception via local log socket;

# local messages are retrieved through imjournal now.

$OmitLocalLogging on

# File to store the position in the journal

$IMJournalStateFile imjournal.state

This Global portion controls the behavior of “rsyslogd” daemon, with help of six active directives.

1.

# Where to place auxiliary files

$WorkDirectory /var/lib/rsyslog

Specifies the location for the storage of files.

2.

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

To save the captured messages in traditional way/Use default timestamp format.

3.

#$ActionFileEnableSync on

File syncing capability is disabled by default. This feature is usually not required, not useful and an extreme performance hit

4.

$IncludeConfig /etc/rsyslog.d/*.conf

Include all config files in /etc/rsyslog.d/

5.

$OmitLocalLogging on

Turn off message reception via local log socket; local messages are retrieved through imjournal now.

6.

$IMJournalStateFile imjournal.state

File name to store the position in the journal.

#### RULES ####

Most imp part of file, which state that what info should be logged to which location.

Each line contains two fields… called….

SELECTOR (WHAT)                                 ACTION (WHERE)

#kern.*                                         /dev/console

*.info;mail.none;authpriv.none;cron.none        /var/log/messages

authpriv.*                                      /var/log/secure

mail.*                                         -/var/log/maillog

cron.*                                          /var/log/cron

*.emerg                                         :omusrmsg:*

uucp,news.crit                                  /var/log/spooler

local7.*                                        /var/log/boot.log

ACTION: (WHERE)

The action field of a rule describes what to do with the message. In general, message content is written to a kind of "log‐file".

But also other actions might be done, like writing to a database table or forwarding to another host.

SELECTOR: (WHAT)    

Selector field specifies a pattern of facilities and priorities belonging to the specified action.

SELECTOR is further divided in to two parts,

Facility (left): system process categories that generate messages.

Priority (right): severity associated with the messages.

omusrmsg stands for "User Message Output Module" and combined with '*' will send a console alert to all logged in users.( wall feature)

SELECTOR                        ACTION

Facility.Priority

Supported facilities are:

Auth, authpriv, cron, daemon, kern, lpr, mail, news, syslog, user, uucp and local0 to local7, * means all of them

Specifies the subsystem that produces a specific syslog message.

Numeric Keyword	Facility	Description
0	kern	kernel messages
1	user	user-level messages
2	mail	mail system
3	daemon	system daemons
4	auth	security/authorization messages
5	syslog	messages generated internally by syslogd
6	lpr	line printer subsystem
7	news	network news subsystem
8	uucp	UUCP subsystem
9	cron	clock daemon
10	security	security/authorization messages
11	ftp	FTP daemon
12	ntp	NTP subsystem
13	logaudit	log audit
14	logalert	log alert
15	clock	clock daemon (note 2)
16	local0	local use 0 (local0)
17	local1	local use 1 (local1)
18	local2	local use 2 (local2)
19	local3	local use 3 (local3)
20	local4	local use 4 (local4)
21	local5	local use 5 (local5)
22	local6	local use 6 (local6)
23	local7	local use 7 (local7)

Supported priorities/severities are:

Emer, alert, crit, error, warning, notice, info, debug and none, * means all of them.

Specifies a priority of a syslog message

Numerical Code	Severity	Description
0	emerg	system is unusable
1	alert	action must be taken immediately
2	crit	critical conditions
3	error	error conditions
4	warning	warning conditions
5	notice	normal but significant condition
6	info	informational messages
7	debug	debug-level messages

Priority starts from 7 to 0, 0 is highest. So if we consider a selector “cron.warn”

Means all cron related messages with priority equal to “warn” and higher.

What is meaning of Higher/Greater…??

Cron.warn messages will contain

Cron.warn

Cron.err

Cron.crit

Cron.alert

Cron.emerg

In same way if we say “cron.info” means reported severity will be from “info” to “emerg”

Let’s make it simple……..

Something is coming from somewhere and someone branch them based on their nature and send to respective location.

          SELECTOR                                ACTION

a.] *.info;mail.none;authpriv.none;cron.none        /var/log/messages

   (1.2)   (1.2)          (1.2)     (1.2)

b.] authpriv.*                                      /var/log/secure

   (1.2)

c.] mail.info                                         -/var/log/maillog

(1.2)

d.] cron.*                                          /var/log/cron

(1.2)

1 =Facility

2 =Priority/Severity

. =Facility and priority are separated by dot(.)

Taking a, b, c, d for our explanation,

First consider entry (a)

*.info;mail.none;authpriv.none;cron.none        /var/log/messages

• Here * is Function, (.) is separator and “info” is Priority.
• * means all functions, and whenever any “info” or greater priority message  will come with respect to attached facility it will reported to /var/log/messages
• In same way “mail” is Function and “none” is Priority….
• But why there is Priority “none” set for mail-authpriv-cron….??
• Answer is in entry b, c & d, where mail-authpriv-cron are mentioned independently with separate reporting locations.
• To avoid messing with /var/log/messages or to prevent /var/log/messages to become a garbage file, it is directed to record all facilities with priority equal to “info” or greater, except mail-authpriv-cron.

Why…??

Bcoz mail-authpriv-cron are already mentioned for different locations.

mail.info

Matches messages from the mail facility with security levels of 'info' or higher.

Validation of rsyslog.conf file……………

[root@rhel7-server ~]# rsyslogd -N 5

rsyslogd: version 7.4.7, config validation run (level 5), master config /etc/rsyslog.conf

rsyslogd: End of config validation run. Bye.

[root@rhel7-server ~]# rsyslogd -N 3

rsyslogd: version 7.4.7, config validation run (level 3), master config /etc/rsyslog.conf

rsyslogd: End of config validation run. Bye.

I made some changes in file,

[root@rhel7-server ~]# grep aaaaa /etc/rsyslog.conf

authpriv.aaaaaaabbbbbb                           /var/log/secure

[root@rhel7-server ~]# rsyslogd -N 3

rsyslogd: version 7.4.7, config validation run (level 3), master config /etc/rsyslog.conf

rsyslogd: unknown priority name "aaaaaaabbbbbb"

rsyslogd: End of config validation run. Bye.

ACTION………..

root,user6      list of users, separated by comma

*               message to all logged on users

/               file (can be a printer, a console, a tty, ...)

-/              file, but don't sync after every write

|               named pipe

@               other syslog hostname

CONFIGURATION……………..

[root@rhel7-server ~]# vi /etc/rsyslog.conf

local6.err      /var/log/errorandabove

local6.=err     /var/log/onlyerror

local6.*        /var/log/alllocal6

(Tab separated entries)

[root@rhel7-server ~]# systemctl restart rsyslog

Verify all entries by “logger” command……..

The logger command can be used to generate syslog test messages.

Where should these logged…??

[root@rhel7-server ~]# logger -p local6.debug "try debug logging"

[root@rhel7-server ~]# logger -p local6.crit "try crit logging"

[root@rhel7-server ~]# logger -p local6.emerg "try info logging"

[root@rhel7-server ~]# logger -p local6.warn "try warning logging"

[root@rhel7-server ~]# cat /var/log/errorandabove

Jan 15 11:34:44 rhel7-server root: try crit logging

Jan 15 11:34:49 rhel7-server root: try info logging

[root@rhel7-server ~]# cat /var/log/onlyerror

cat: /var/log/onlyerror: No such file or directory

[root@rhel7-server ~]# logger -p local6.err "try error logging"

[root@rhel7-server ~]# cat /var/log/onlyerror

Jan 15 11:37:15 rhel7-server root: try error logging

[root@rhel7-server ~]# cat /var/log/alllocal6

Jan 15 11:34:05 rhel7-server root: root [14283]: systemctl restart rsyslog [0]

Jan 15 11:34:36 rhel7-server root: try debug logging

Jan 15 11:34:36 rhel7-server root: root [14283]: logger -p local6.debug "try debug logging" [0]

Jan 15 11:34:44 rhel7-server root: try crit logging

Jan 15 11:34:44 rhel7-server root: root [14283]: logger -p local6.crit "try crit logging" [0]

Jan 15 11:34:49 rhel7-server root: try info logging

Jan 15 11:34:49 rhel7-server root: root [14283]: logger -p local6.emerg "try info logging" [0]

Jan 15 11:35:56 rhel7-server root: try warning logging

Jan 15 11:35:57 rhel7-server root: root [14283]: logger -p local6.warn "try warning logging" [0]

Jan 15 11:36:20 rhel7-server root: root [14283]: cat /var/log/errorandabove [0]

Jan 15 11:36:55 rhel7-server root: root [14283]: cat /var/log/onlyerror [1]

Jan 15 11:37:04 rhel7-server root: try warning logging

Jan 15 11:37:04 rhel7-server root: root [14283]: logger -p local6.err "try warning logging" [0]

Jan 15 11:37:15 rhel7-server root: try error logging

Jan 15 11:37:15 rhel7-server root: root [14283]: logger -p local6.err "try error logging" [0]

Jan 15 11:37:17 rhel7-server root: root [14283]: cat /var/log/onlyerror [0]

Check the o/p and match with the “severity/priority” hierarchy to understand better that where the logs are going with respect to

logger -p local6.<severity>