Monday, August 17, 2009

System Logging

One integral part of any UNIX system are the loggingfacilities. The majority of logging in Linux is provided by twomain programs, sysklogd and klogd, the first providing loggingservices to programs and applications, the second providinglogging capability to the Linux kernel. Klogd actually sends mostmessages to the syslogd facility but will on occasion pop upmessages at the console (i.e. kernel panics). Sysklogd actuallyhandles the task of processing most messages and sending them tothe appropriate file or device, this is configured from within/etc/syslog.conf. By default most logging to files takes place in/var/log/, and generally speaking programs that handle their ownlogging (most httpd servers handle their logging internally) logto /var/log/program-name/, which allows you to centralize the logfiles and makes it easier to place them on a separate partition(some attacks can fill your logs quite quickly, and a full /partition is no fun). Additionally there are programs that handletheir own interval logging, one of the more interesting being thebash command shell. By default bash keeps a history file ofcommands executed in ~username/.bash_history, this file can makefor extremely interesting reading, as oftentimes many admins willaccidentally type their passwords in at the command line. Apachehandles all of its logging internally, configurable fromhttpd.conf and extremely flexible with the release of Apache1.3.6 (it supports conditional logging). Sendmail handles itslogging requirements via syslogd but also has the option (via thecommand line -X switch) of logging all SMTP transactions straightto a file. This is highly inadvisable as the file will growenormous in a short span of time, but is useful for debugging.See the sections in network security on Apache and Sendmail formore information.
General log security
Generally speaking you do not want to allow users to see thelog files of a server, and you especially don't want them tobe able to modify or delete them. Generally speaking most logfiles are owned by the root user and group, and have nopermissions assigned for other, so in most cases the only userable to modify the logs will be the root user (and if someonecracks the root account all bets are off). There are a few extrasecurity precautions you can take however, the simplest being touse the "chattr" (CHange ATTTRibutes command) to setthe log files to append only. This way in the event of a problemlike a /tmp race that allows people to overwrite files on thesystem they cannot significantly damage the log files. To set afile to append only use:chattr +a filename
only the superuser has access to this function of chattr. Ifyou set all your log files to append only you must remember thatlog rotation programs will fail as they will not be able to zerothe log file. Add a line to the script to unset the append onlyattribute:chattr -a filename
and add a line after the log rotation script to reset theappend only flag. If you keep log files on the system you mayalso wish to set them immutable so they cannot be tampered withas easily, to set the file immutable simply:chattr +i filename
and this will prevent any changes (due to /tmp races, etc.) tothe file unless the attacker has root access (in which caseyou're already in a world of hurt).chattr -i filename
only the root user has access to the immutable flag.
System logging
One feature of Linux (and most unices) is the syslog and klogfacilities which allow software to generate log messages that arethen passed to alog daemon and handled (written to a local file,a remote server, given to aprogram, and so on).
sysklogd / klogd
In a nutshell klogd handles kernel messages, depending on yoursetup this can range from almost none to a great deal if forexample you turn on process accounting. It then passes mostmessages to syslogd for actual handling (that is it places thedata in a physical file). The man pages for sysklogd, klogd andsyslog.conf are pretty good with clear examples. One exceedinglypowerful and often overlooked ability of syslog is to logmessages to a remote host running syslog. Since you can definemultiple locations for syslog messages (i.e. send all kernmessages to the /var/log/messages file, and to console, and to aremote host or multiple remote hosts) this allows you tocentralize logging to a single host and easily check log filesfor security violations and other strangeness. There are severalproblems with syslogd and klogd however, the primary ones beingthe ease of which once an attacker has gained root access todeleting/modifying log files, there is no authentication builtinto the standard logging facilities.
The standard log files that are usually defined in syslog.confare:/var/log/messages/var/log/secure/var/log/maillog/var/log/spooler
The first one (messages) gets the majority of informationtypically; user logins, TCP_WRAPPERS dumps information here, IPfirewall packet logging typically dumps information here and soon. The second typically records entries for events like userschanging their UID/GID (via su, sudo, etc.), failed attempts whenpasswords are required and so on. The maillog file typicallyholds entries for every pop/imap connection (user login andlogout), and the header of each piece of email that goes in orout of the system (from whom, to where, msgid, status, and soon). The spooler file is not often used anymore as the number ofpeople running usenet or uucp has plummeted, uucp has beenbasically replaced with ftp and email, and most usenet serversare typically extremely powerful machines to handle a full, oreven partial newsfeed, meaning there aren't many of them(typically one per ISP or more depending on size). Most homeusers and small/medium sized business will not (and should not inmy opinion) run a usenet server, the amount of bandwidth andmachine power required is phenomenal, let alone the securityrisks.
You can also define additional log files, for example youcould add:kern.* /var/log/kernel-log
And you can selectively log to a separate log host:*.emerg @syslog-hostmail.* @mail-log-host
Which would result in all kernel messages being logged to/var/log/kernel-log, this is useful on headless servers since bydefault kernel messages go to /dev/console (i.e. someone loggedin at the machines). In the second case all emergency messageswould be logged to the host "syslog-host", and all themail log files would be sent to the "mail-log-host"server, allowing you to easily maintain centralized log files ofvarious services. The default syslogd that ships with most Linuxdistributions is horribly insecure, log files are easily tamperedwith (or outright destroyed), and logging across the network iscompletely insecure as well as dangerous for the serversinvolved. I do not advise using syslog if you actually have aneed for reliable logging (i.e. the ability to later view logfiles in the event of a break-in).
The default file permissions on the log files are usually read/ write for root, and nothing for anyone else. In addition tothis you can (and should) set the files append only (remember totake logrotate into account though, it needs to zero the files).This will prevent any deletion / modifications to the log filesunless root unsets the append only attribute first.
modular syslog
The major problem with syslog however is that tampering withlog files is trivial (setting the log files append only with"chattr +a" helps, but if an attacker gains root, theycan unset the attribute). There is however a secure version ofsyslogd, available at guys generally make good tools and have a good reputation,in any case it is open source software for those of you who aretruly paranoid). This allows you to cryptographically sign logsto ensure they haven't been tampered with. Ultimately,however, an attacker can still delete the log files so it is agood idea to send them to another host, especially in the case ofa firewall to prevent the hard drive being filled up.
next generation syslog
Another alternative is "syslog-ng" (Next GenerationSyslog), which seems much more customizable then either syslog orsecure-syslog, it supports digital signatures to prevent logtampering, and can filter based on content of the message, notjust the facility it comes from or priority (something that isvery useful for cutting down on volume). Syslog-ng is availableat:
Nsyslogd supports tcp, and SSL for logging to remote systems.It runs on a variety of UNIX platforms and you can download itfrom:
Log monitoring
Log files are not much good unless you actually check themonce in a while, this is an almost impossible task for most of ushowever due to the sheer volume of log files. There are a varietyof tools to automate these tasks however.
Psionic Logcheck
Psionic Logcheck will go through the messages file (andothers) on a regular basis (invoked via crontab usually) andemail out a report of any suspicious activity. It is easilyconfigurable with several 'classes' of items, activepenetration attempts which is screams about immediately, badactivity, and activity to be ignored (for example DNS serverstatistics or SSH rekeying). Psionic Logcheck is available from:
colorlogs will color code log files allowing you to easilyspot suspicious activity. Based on a config file it looks forkeywords and colors the lines (red, cyan, etc.), it takes inputfrom STDIN so you can use it to review log files quickly (byusing "cat", "tail" or other utilities tofeed the log file through the program). You can get it at:
WOTS collects log files from multiple sources and willgenerate reports or take action based on what you tell it to do.WOTS looks for regular expressions you define and then executesthe commands you list (mail a report, sound an alert, etc.). WOTSrequires you have Perl installed and is available from:
swatch is very similar to WOTS, and the log filesconfiguration is very similar. You can download swatch from:
Kernel logging
The lowest level of logging possible is at the kernel level.Generally speaking users cannot disabled of avoid this type oflogging, and also are usually not even aware it exists (adefinite advantage).
Shell logging
A variety of command shells have built in loggingcapabilities.
I will also cover bash since it is the default shell in mostLinux installations, and thus its logging facilities aregenerally used. bash has a large number of variables you canconfigure at run time or during it's use that modify how itbehaves. Everything from the command prompt style to how manylines to keep in the log file.
HISTFILEname of the history file, by default it is~username/.bash_history
HISTFILESIZEmaximum number of commands to keep in the file, it rotates themas needed.
HISTSIZEthe number of commands to remember (i.e. when you use the uparrow key).
The variables are typically set in /etc/profile, whichconfigures bash globally for all users, however, the values canbe over-ridden by users with the ~username/.bash_profile file,and/or by manually using the export command to set variables suchas export EDITOR=emacs. This is one of the reasons that userdirectories should not be world readable; the .bash_history filecan contain a lot of valuable information to a hostile party. Youcan also set the file itself non world readable, set your.bash_profile not to log, set the file non writeable (thusdenying bash the ability to write and log to it) or link it to/dev/null (this is almost always a sure sign of suspicious useractivity, or a paranoid user). For the root account I wouldhighly recommend setting the HISTFILESIZE and HISTSIZE to a lowvalue such as 10. On the other hand if you want to log usersshell history and otherwise tighten up security I would recommendsetting the configuration files in the user's home directoryto immutable using the chattr command, and set the log files(such as .bash_history) to append only. Doing this however opensup some legal issues, so make sure your users are aware they arebeing logged and have agreed to it, otherwise you could get intotrouble. Don't forget to set /hom/username/.bash_history appendonly (chattr +A).
Post a Comment

You might also like :

Related Posts with Thumbnails