System Logging


One integral part of any UNIX system are the logging

facilities. The majority of logging in Linux is provided by two

main programs, sysklogd and klogd, the first providing logging

services to programs and applications, the second providing

logging capability to the Linux kernel. Klogd actually sends most

messages to the syslogd facility but will on occasion pop up

messages at the console (i.e. kernel panics). Sysklogd actually

handles the task of processing most messages and sending them to

the appropriate file or device, this is configured from within

/etc/syslog.conf. By default most logging to files takes place in

/var/log/, and generally speaking programs that handle their own

logging (most httpd servers handle their logging internally) log

to /var/log/program-name/, which allows you to centralize the log

files and makes it easier to place them on a separate partition

(some attacks can fill your logs quite quickly, and a full /

partition is no fun). Additionally there are programs that handle

their own interval logging, one of the more interesting being the

bash command shell. By default bash keeps a history file of

commands executed in ~username/.bash_history, this file can make

for extremely interesting reading, as oftentimes many admins will

accidentally type their passwords in at the command line. Apache

handles all of its logging internally, configurable from

httpd.conf and extremely flexible with the release of Apache

1.3.6 (it supports conditional logging). Sendmail handles its

logging requirements via syslogd but also has the option (via the

command line -X switch) of logging all SMTP transactions straight

to a file. This is highly inadvisable as the file will grow

enormous in a short span of time, but is useful for debugging.

See the sections in network security on Apache and Sendmail for

more information.



General log security

Generally speaking you do not want to allow users to see the

log files of a server, and you especially don't want them to

be able to modify or delete them. Generally speaking most log

files are owned by the root user and group, and have no

permissions assigned for other, so in most cases the only user

able to modify the logs will be the root user (and if someone

cracks the root account all bets are off). There are a few extra

security precautions you can take however, the simplest being to

use the "chattr" (CHange ATTTRibutes command) to set

the log files to append only. This way in the event of a problem

like a /tmp race that allows people to overwrite files on the

system they cannot significantly damage the log files. To set a

file to append only use:

chattr +a filename

only the superuser has access to this function of chattr. If

you set all your log files to append only you must remember that

log rotation programs will fail as they will not be able to zero

the log file. Add a line to the script to unset the append only


chattr -a filename

and add a line after the log rotation script to reset the

append only flag. If you keep log files on the system you may

also wish to set them immutable so they cannot be tampered with

as easily, to set the file immutable simply:

chattr +i filename

and this will prevent any changes (due to /tmp races, etc.) to

the file unless the attacker has root access (in which case

you're already in a world of hurt).

chattr -i filename

only the root user has access to the immutable flag.



System logging

One feature of Linux (and most unices) is the syslog and klog

facilities which allow software to generate log messages that are

then passed to alog daemon and handled (written to a local file,

a remote server, given to aprogram, and so on).



sysklogd / klogd

In a nutshell klogd handles kernel messages, depending on your

setup this can range from almost none to a great deal if for

example you turn on process accounting. It then passes most

messages to syslogd for actual handling (that is it places the

data in a physical file). The man pages for sysklogd, klogd and

syslog.conf are pretty good with clear examples. One exceedingly

powerful and often overlooked ability of syslog is to log

messages to a remote host running syslog. Since you can define

multiple locations for syslog messages (i.e. send all kern

messages to the /var/log/messages file, and to console, and to a

remote host or multiple remote hosts) this allows you to

centralize logging to a single host and easily check log files

for security violations and other strangeness. There are several

problems with syslogd and klogd however, the primary ones being

the ease of which once an attacker has gained root access to

deleting/modifying log files, there is no authentication built

into the standard logging facilities.

The standard log files that are usually defined in syslog.conf



The first one (messages) gets the majority of information

typically; user logins, TCP_WRAPPERS dumps information here, IP

firewall packet logging typically dumps information here and so

on. The second typically records entries for events like users

changing their UID/GID (via su, sudo, etc.), failed attempts when

passwords are required and so on. The maillog file typically

holds entries for every pop/imap connection (user login and

logout), and the header of each piece of email that goes in or

out of the system (from whom, to where, msgid, status, and so

on). The spooler file is not often used anymore as the number of

people running usenet or uucp has plummeted, uucp has been

basically replaced with ftp and email, and most usenet servers

are typically extremely powerful machines to handle a full, or

even partial newsfeed, meaning there aren't many of them

(typically one per ISP or more depending on size). Most home

users and small/medium sized business will not (and should not in

my opinion) run a usenet server, the amount of bandwidth and

machine power required is phenomenal, let alone the security


You can also define additional log files, for example you

could add:

kern.* /var/log/kernel-log

And you can selectively log to a separate log host:

*.emerg @syslog-hostmail.* @mail-log-host

Which would result in all kernel messages being logged to

/var/log/kernel-log, this is useful on headless servers since by

default kernel messages go to /dev/console (i.e. someone logged

in at the machines). In the second case all emergency messages

would be logged to the host "syslog-host", and all the

mail log files would be sent to the "mail-log-host"

server, allowing you to easily maintain centralized log files of

various services. The default syslogd that ships with most Linux

distributions is horribly insecure, log files are easily tampered

with (or outright destroyed), and logging across the network is

completely insecure as well as dangerous for the servers

involved. I do not advise using syslog if you actually have a

need for reliable logging (i.e. the ability to later view log

files in the event of a break-in).

The default file permissions on the log files are usually read

/ write for root, and nothing for anyone else. In addition to

this you can (and should) set the files append only (remember to

take logrotate into account though, it needs to zero the files).

This will prevent any deletion / modifications to the log files

unless root unsets the append only attribute first.



modular syslog

The major problem with syslog however is that tampering with

log files is trivial (setting the log files append only with

"chattr +a" helps, but if an attacker gains root, they

can unset the attribute). There is however a secure version of

syslogd, available at

(these guys generally make good tools and have a good reputation,

in any case it is open source software for those of you who are

truly paranoid). This allows you to cryptographically sign logs

to ensure they haven't been tampered with. Ultimately,

however, an attacker can still delete the log files so it is a

good idea to send them to another host, especially in the case of

a firewall to prevent the hard drive being filled up.



next generation syslog

Another alternative is "syslog-ng" (Next Generation

Syslog), which seems much more customizable then either syslog or

secure-syslog, it supports digital signatures to prevent log

tampering, and can filter based on content of the message, not

just the facility it comes from or priority (something that is

very useful for cutting down on volume). Syslog-ng is available





Nsyslogd supports tcp, and SSL for logging to remote systems.

It runs on a variety of UNIX platforms and you can download it




Log monitoring

Log files are not much good unless you actually check them

once in a while, this is an almost impossible task for most of us

however due to the sheer volume of log files. There are a variety

of tools to automate these tasks however.



Psionic Logcheck

Psionic Logcheck will go through the messages file (and

others) on a regular basis (invoked via crontab usually) and

email out a report of any suspicious activity. It is easily

configurable with several 'classes' of items, active

penetration attempts which is screams about immediately, bad

activity, and activity to be ignored (for example DNS server

statistics or SSH rekeying). Psionic Logcheck is available from:





colorlogs will color code log files allowing you to easily

spot suspicious activity. Based on a config file it looks for

keywords and colors the lines (red, cyan, etc.), it takes input

from STDIN so you can use it to review log files quickly (by

using "cat", "tail" or other utilities to

feed the log file through the program). You can get it at:





WOTS collects log files from multiple sources and will

generate reports or take action based on what you tell it to do.

WOTS looks for regular expressions you define and then executes

the commands you list (mail a report, sound an alert, etc.). WOTS

requires you have Perl installed and is available from:




swatch is very similar to WOTS, and the log files

configuration is very similar. You can download swatch from:



Kernel logging

The lowest level of logging possible is at the kernel level.

Generally speaking users cannot disabled of avoid this type of

logging, and also are usually not even aware it exists (a

definite advantage).




Information is available




Shell logging

A variety of command shells have built in logging





I will also cover bash since it is the default shell in most

Linux installations, and thus its logging facilities are

generally used. bash has a large number of variables you can

configure at run time or during it's use that modify how it

behaves. Everything from the command prompt style to how many

lines to keep in the log file.


name of the history file, by default it is



maximum number of commands to keep in the file, it rotates them

as needed.


the number of commands to remember (i.e. when you use the up

arrow key).

The variables are typically set in /etc/profile, which

configures bash globally for all users, however, the values can

be over-ridden by users with the ~username/.bash_profile file,

and/or by manually using the export command to set variables such

as export EDITOR=emacs. This is one of the reasons that user

directories should not be world readable; the .bash_history file

can contain a lot of valuable information to a hostile party. You

can also set the file itself non world readable, set your

.bash_profile not to log, set the file non writeable (thus

denying bash the ability to write and log to it) or link it to

/dev/null (this is almost always a sure sign of suspicious user

activity, or a paranoid user). For the root account I would

highly recommend setting the HISTFILESIZE and HISTSIZE to a low

value such as 10. On the other hand if you want to log users

shell history and otherwise tighten up security I would recommend

setting the configuration files in the user's home directory

to immutable using the chattr command, and set the log files

(such as .bash_history) to append only. Doing this however opens

up some legal issues, so make sure your users are aware they are

being logged and have agreed to it, otherwise you could get into

trouble. Don't forget to set /hom/username/.bash_history append

only (chattr +A).

Post a Comment

You might also like :

Related Posts with Thumbnails