a look at the hidden part of the logs

At the risk of contributing to the arctic chill as winter gets started in the northern hemisphere, let’s analyze the stakes and key factors of a log management strategy.

"What were you doing every the night from September 30 to April 1?” the detective asks a petrified Inuit systems administrator.

"If you don’t mind me saying so, if you’re only asking to see the logs, you’ll only see the tip of the iceberg!" the administrator replies.

consider the iceberg

Managing logs within an IT infrastructure is, above all, strongly tied to our conceptions. To borrow the analogy recently used by two colleagues (thanks Eric and Stéphane), the experienced observer should see this:

• at the top and clearly visible is a monitoring system
• lower on the ice, hardly visible under the cold water, is a remediation system
• lastly, all the way at the bottom of the ocean, lies a reporting system

what's with this triple-decker ice cream cone?

The three layers mentioned above correspond to real log management needs today:

1. monitoring: It’s useless to set up a log collection infrastructure for your systems (servers and/or networks) if you’re not equipped to oversee it. You’ll need to immediately look for adequate software solutions because once you reach a certain volume of data, it’s difficult for operators to quickly calculate that “the IP address abcd tried 1,396 times to connect via Secure Shell (SSH) to the root account of machine a.b.c.d over the last three minutes.”

2. remediation: Once information is collected and pre-analyzed, analysis and action is needed, but this can’t happen without people on your team. Did the system recognize that someone tried to connect via SSH to the root account of your infrastructure’s most critical system? Great, what should you do now? It’s highly likely that you would prefer a person to take the appropriate action so that your last automated system doesn’t get automatically blacklisted because the wrong password was configured.

3. reporting: You've spent a fortune setting up the latest firewalls, intrusion detection systems, and intrusion prevention systems. Now you hope to get at least a minimal return on your investment: at best a noticeable decrease in system intrusions, at worst a slick color dashboard in 3D that makes you feel good about your choice and helps you face the next technical team meeting with a smile. Time to wake up: the “reporting” module wasn’t included and your whole system needs a good set of processes to function correctly.

the cold cycle

You’ve probably already noticed that three steps are not enough to set up complete log management. Many variables can alter your material and human costs:

Without going into detail about each one, below are several important steps for managing your logs:

• logging: if your components (materials, software) are not stocking anything, you’ve already completed your log management goal
• collection: sorting and labeling individual logs is necessary to identify their source (component or sub-component name)
• centralization: the primordial step from a security point of view, externalizing all your logs in a single place (unless you want to read blank pages after an attack)
• archiving: in some cases, authorities require professionals to store IT service activities. Also, analyzing an attack may require you to look through logs from six months ago
• restitution: it’s pointless to flood your security surveillance team with information that is redundant or serves no real purpose for their operations (logs of changes in users’ screen resolution are rarely analyzed)
• correlation: this is a step we mentioned above. A system for calculating in real time how often certain logs appear (remember the 1,396 attempts in three minutes) will simplify your life enormously
• analysis: also see above. This is where your return on investment will be highest, since your ability to make decisions based on the information at hand is of utmost importance
• reporting: whether you’re an information systems security officer or not, you’ll greatly improve your indicator reviewing method by implementing a regular, effective reporting process for the steps mentioned above

questions for the iceman

Think you’re ready to launch a log management project for your infrastructure? Here are a few questions to ask yourself concerning some of the steps outlined above:

logging

• do we have Network Time Protocol services to timestamp our logs?
• does each log display the name of the generating component?
• does each log display the IP address of the generating component?
• is each log tagged with a priority label (info, warning, debug, etc)?

collection

• are our log files archived and compressed on a daily basis?
• are our log files saved for a rolling period of X days?
• do we have a real-time log generation system (ex: syslog)?
• do we have a delayed log generation system (for Windows logs, for example)?
• have we set up a surveillance mechanism for sent logs?
• have we set up a surveillance mechanism for disk space used by logs?

centralization

• what volume of logs is expected on a daily basis?
• is the centralization system accessible from a domain name system or IP address?
• does the centralization system offer a load distribution mechanism?
• what protocol is used by the centralization system (e.g., syslog, User Datagram Protocol port 1514)?

archiving

• are centralized logs archived and compressed on a daily basis?
• are centralized logs stored for a rolling period of N months?
• are centralized logs stored according to a single, consistent index?
• what index levels are used for centralized logs?

conclusion

I hope that this article has helped point you in the right direction regarding this topic. To learn more, I recommend reading "Guide to Computer Security Log Management" published by the National Institute of Standards and Technology.