For many systems administrators, the task of system security is very daunting and time consuming. I discovered early in my career that even though my work as a systems administrator never seemed to end, I could not neglect the responsibility of system monitoring and security. I started with some of the freeware options, such as Portsentry from Psionic:
http://www.psionic.com/
and Saint from World Wide Digital Security:
http://www.wwdsi.com/
and Sys Admin magazine March 2001:
http://www.sysadminmag.com/articles/2001/0103/0103b/a2.htm
I really like both of these tools, however, I wanted to have more control and flexibility with my system security and notification.
As I sifted through my never-ending supply of logs each day, I thought there must be a better way. I then realized that I was going about this task all wrong. I was neglecting my own knowledge, skill, and expertise in favor of other programs.
Through experience, I found out that when a system is cracked most crackers follow a certain modus operandi. They usually change binaries so their presence and activities are undetected. I also found out that specific files are altered and hidden directories can and usually are created.
I decided to go about the protection and detection of my system in the following ways. Nonetheless, I realize that, much like someone who wishes to enter a building, there is only so much you can do to prevent a cracker from entering your system. If the perpetrator wants on your system bad enough, they will get in. I implemented a four-tier warning program created from simple UNIX commands and anchored by Portsentry.
Tier One
I wanted to be alerted if anyone scanned my inet services or, in other words, to be alerted if someone was “checking the doors” for security holes. As most administrators do, I implemented TCP Wrappers, with a twist. I added the following to my /etc/hosts/deny:
echo -e "\n\
TCP Wrappers\: Connection Refused\n\
By\: $(uname -n)\n\
Process\: %d (pid %p)\n\
Host\: %c\n\
Date\: $(date)\n\
" /bin/mail -s "From tcpd@$(uname -n). %u@%h -> %d."
admin@somewhere.com)
This script will generate an immediate notification to an account of your choice, preferably on a different system. Here’s a detailed explanation of how the script works:
• ALL: — ALL means to deny every and any IP address unless over ruled by the /etc/hosts.allow file.
• spawn — Spawn or create the actual notification.
• echo -e “\n\ — Starts the notification with a new line and then create a heading of “Connection Refused”.
• By — Uses the command uname to identify the system.
• Process — Will identify which process your system is being probed for or by (ftp, portmap, etc.).
• Host — Tries to detect the host the intrusion originates from.
• Date — The date of the attempt and the last line simply emails an account of your choice the notification in the proper format. An example follows:
TCP Wrappers: Connection Refused
By: workstation1.somewhere.com
Process: portmap (pid 321)
Host: xxx.xx.xx.xx
Date: Wed Dec 13 04:51:22 IST 2006
This script provided me with instant notification of any attempts on my system and also a log of this attempt should I need it in the future. When administering a large number of systems or a system that is probed continually, it’s wise to set up an account on a system specifically for handling security issues.
Tier Two
I also wanted to track changes in specific files and binaries using md5cksum or cksum as a fingerprint. You will want to run the script the first time on a system that has never been on a network or that you know has never been tampered with:
/usr/bin/md5sum /etc/passwd > /root/ck_sum.txt
/usr/bin/md5sum /etc/services >> /root/ck_sum.txt
/usr/bin/md5sum /bin/ps >> /root/ck_sum.txt
/usr/bin/md5sum /bin/netstat >> /root/ck_sum.txt
/usr/bin/md5sum /bin/ls >> /root/ck_sum.txt
/usr/bin/md5sum /usr/bin/top >> /root/ck_sum.txt
/usr/bin/md5sum /etc/inetd.conf >> /root/ck_sum.txt
/usr/bin/md5sum /usr/bin/md5sum >> /root/ck_sum.txt
/usr/bin/md5sum /bin/rpm >> /root/ck_sum.txt
/usr/bin/md5sum /bin/ck_sum.scp >> root/ck_sum.txt
echo "Report from someworkstation" > /root/ck_sum.out
/usr/bin/diff /root/ck_sum.txt /root/ck_sum.org >>
/root/ck_sum.out
mail -s "check sum from someworkstation" admin@somewhere.com
< /root/ck_sum.out
The script starts by running md5sum on what I consider my important binaries and files, including the script itself. The output looks like this;
849813023732aef68bb4df674d628f2f /etc/passwd
63a1913bc0d1d39927b40628d6a98ecd /etc/services
6d16efee5baecce7a6db7d1e1a088813 /bin/ps
b7dda3abd9a1429b23fd8687ad3dd551 /bin/netstat
f482ae701e46005a358a01c139f1ae74 /bin/ls
6afa152b929c0ab6ddd0a321254f139f /usr/bin/top
3509dee17211e59dec058e30a50c79e3 /etc/inetd.conf
908162ab85e1e3668a235e223aad7d0e /usr/bin/md5sum
eb3fc42223173db0888cee7aec96f413 /bin/rpm
146fc45ab3af038bf592b271556b7cd5 /bin/ck_sum.scp
The first time I run the script, I save the file as ck_sum.org (for original) and save it on another system, just in case.
The line echo "Report from someworkstation" > /root/ck_sum.out starts the log with a nice heading. I then use the basic UNIX command diff to compare the ck_sum.org to the file I just created, ck_sum.out. I again mail the log to a system used for log checking. If nothing has been altered, the file should only contain:
"Report from someworkstation"
Remember that any administration on the system may cause changes in the files your are checking. For instance, when I add a user, my /etc/passwd file changes. I will then need to recreate ck_sum.scp by re-running my script.
Tier Three
I also wanted to track changes in all files. I realize most of the systems I administer have a relatively small number of users. However, I run this even on the systems that have many users and just omit the home or user directories. I call this script file_check, because I use the script for just that purpose. I again implement basic UNIX commands, this time focusing on the find command (man find).
I actually have two different scripts: the first checks for changes daily, and the second checks weekly. However, you can use any time frame that fits your needs. Here is the script for checking files that were altered in a 24-hour period followed by the breakdown of the script:
#Script to find changes in files on the system
#Omitting the /proc dir
find / -mtime -1 -exec ls -ld {} \; grep -v /proc >
/tmp/1days 2>/dev/null &
# find starting at the root dir "/"
# -mtime is the time the file was last modified
# -1 is the time less than one day of modified time
# -exec ls -ld, execute listing of files in long format and
# also list directories but not content.
# {}
# grep -v /proc omits the /proc dir
# > /tmp/1days pipes the output to this file
# >/dev/null pipes any errors to /dev/null
To use the script for a time period other than a single day, simply change the “-1” to whatever time period you wish. For example, here is my command when I look for changes that have happened in the last week:
find / -mtime -7 -exec ls -ld {} \; grep -v /proc >
/tmp/1days 2>/dev/null &
I use cron to schedule the run time of the script and to email the files to me. Then I check for any files I do not think should have changed in the last day.
In tier three, I also start checking the logs without examining each log individually. I use a script called log-checker, which uses the UNIX command grep, or egrep if you prefer to query logs for keywords (man grep or man egrep). The entire script is only five lines long:
#!/bin/sh
grep refused /var/log/secure > /root/secure.txt
/usr/bin/last >> /root/secure.txt
grep auth /var/log/messages >> /root/secure.txt
mail admin@someworkstation.com < /root/secure.txt
I simply grep for keywords such as “refused”, “auth”, and “connection” in my logs, dumping the output into a file called secure.txt:
/bin/grep refused /var/log/secure > secure.txt
/bin/grep auth /var/log/messages >> secure.txt
/bin/grep connection /var/log/messages >> secure.txt
I also add /usr/bin/last >> secure.txt for good measure to see who has been on the system since the last log. Finally, I have the log emailed to myself, mail admin@somewhere.com <>
Tier Four
In the fourth tier, I installed Portsentry to anchor my entire security logging methodology. I edited portsentry_config.h by uncommenting the line starting with WRAPPER_HOSTS_DENY, and adding the path to the hosts.deny file to enable the use of the program with TCP Wrappers.
Although I like programs like Portsentry, Saint, etc., I appreciate that with a few simple UNIX commands, I can keep a eye on my systems and have several tiers of protection.