Saturday, December 23, 2006



Are you prepared to make the best decisions and responses to security incidents in your business?

Picture this scenario: you arrive to work on Monday morning to find problems on your network. You can't log in, some servers seem to be down, and the phone is going crazy. You consider the possibility of two causes :

some fault has occurred and needs fixing you network is ( or has been ) under attack

What you do next is critical. Every decision you now make means a huge difference to the productivity of your business. This paper presents a short list of what needs to be done to address this situation: you have a security problem, it may be malicious hackers, or it could an entirely accidental problem.

Security Incidents . A definition
An IT security incident is a violation of IT security policy, acceptable use policy or of standard procedures

Which covers a lot of events, but most frequently amongst small to medium enterprises:

malware attacks

  • virus ( becoming rare )
  • worms
  • trojan horses

Denial of Service ( DoS )

  • as a side-effect of malware attack
  • as a deliberate, intelligent attack

Intruders, intelligent agent attacks

  • insiders
  • outsiders
  • ex-insiders


  • advertising - SPAM
  • scams: phishing, Nigerian, stock market
  • malware-carrying: trojans

Operational incidents

  • system failures: crashes, environmental failure
  • operator error

Prevention is always best, in other words: cheapest. Prevention is generally more economical, less stressful, and incurs less downtime however it requires a very full understanding of the security landscape and prevention systems such as anti-malware tools, and good processes. But eventually, the inevitable happens and the cost of recovery is directly related to the amount of fore-planning applied.

It has been estimated that in 2006, New Zealand businesses will lose a total amount of $140-240 million due to security incidents (Ref). This is a very high figure (dodgy statistics warning!) which must be close to the entire IT security revenue figure for New Zealand.

Detecting Incident Occurrence Detection is not always easy, often detection is through observing the side-effects of a security incident (eg. High network traffic). Major incidents are frequently announced by statements such as "that's funny", or "Uh oh" ...

Intrusion detection tools are generally good analysis tools, sometimes useful for tactical alerting. Always good to have in your toolkit.

Assessing the situation
Assess the situation as soon as possible after detection. Ask yourself (and anyone else in the vicinity) the following questions:

  • what is the business impact of the incident?
  • if the affected system(s) are isolated then what is the impact to business?
  • how much effort will it take to resolve?
  • this estimate can be re-evaluated later
  • estimate for repair-in-place
  • estimate for offline rebuild from scratch
  • estimate to recover from backups ( you do have backups, don't you? )
  • is there enough people with enough expertise available?
  • if not, think about calling someone - either additional internal resources, or external providers
  • document everything include dates and times, include contact information

Identifying the people to handle the incident
An incident team for a small to medium enterprise is almost always two people. One will be the technical lead who will perform the bulk of the remedial work, and the other will be a backup person reporting to management and recording the actions taken. Further people may be involved depending on the size of the organisation, usually in the reporting chain rather than in direct involvement. These may include:

  • the IT manager
  • the CEO/CIO/CTO
  • Media relations
  • Legal
  • Law enforcement

Always get help if you feel the situation is getting out of hand. For example, in one case an incident involving malware infection dragged on for two weeks before someone was called in to diagnose the problem and resolve it within an hour. Get equipment and software tools if required. For example:

  • new workstations, servers, switches/routers, firewalls
  • IDS tools
  • Anti-virus, Anti-spam tools
  • Preparation is the key to saving time here. It is always good to have an incident toolkit ready with software tools on a CD or USB stick.

Forming a plan for resolution
Make a plan, not necessarily written, but at least well communicated. Include the following basic steps:

1. Contain the problem
Potentially by isolating the affected systems

2. Do No Damage
make backups of the affected system(s)
decide if preservation of evidence or critical data is required, if so then backups and correct procedures are critical

3. Resolve the problem
rebuild systems from scratch?
load backup data?
re-secure the affected system(s) as soon as possible

These steps may be reiterated as the resolution process continues. During resolution the following must be continuously performed:

  • record all actions taken, and document as much as practical
  • updated estimated RTO times
  • report to management and other interested parties

Keep other people in the organisation up to date with expected RTO times and any issues that are outstanding. Discretion needs to be applied here : avoid public dissemination of sensitive information ( such as what type of system failed ), and be very cautious as to attributing blame.

Return to Operation
After the incident has been resolved the systems can be returned to operation, but only after they have been tested and re-secured to prevent any re-occurrence. Identify and mitigate all vulnerabilities that were exploited After return to operations, monitor the system closely to make sure all systems are restored to normal.

Report to management and other interested parties that the system has been restored.

Preventing Reoccurrence
This is by far the most crucial area, unfortunately it is often overlooked. Many problem would not occur if prevention measures had been put in place following the first occurrence.

Review the Causes

  • Was the incident caused by a failure in one or more of the following?
  • Technical controls?
  • Environmental systems?
  • Human factors?
  • Management & budget constraints?

Review Resolution

  • Could the incident have been detected earlier?
  • Could the incident have been resolved quicker?
  • Did you need more resources?
  • Was reporting adequate?
  • Would simulations and rehearsals help in resolving similar incidents?
  • Would it help to have better incident response plans?

Create a Final Report
... and keep it short and to the point. 1-2 pages is the norm. In the final report detail the incident causes, how it was detected, the handling process, and provide an executive summary.

The key is to prevent re-occurrence.

Think about how you would handle each of the following hypothetical situations:

Scenario 1
The company's web server slows down then stops responding. High network traffic causes you to suspect a Denial of Service attack. Points to address:

  • How will you identify the source of the traffic?
  • Should you shutdown the corporate web server?
  • How will you stop the traffic?
  • How will you prevent it from reoccurring?

Scenario 2
The organisation starts receiving complaints about spam originating from its network The administrator identifies the source as the company.s web server Points to address:

  • How would you validate the source of the spam?
  • How would you respond to complaints regarding the spam?
  • How would you isolate and repair the web server?
  • How would you prevent reoccurrence?

Scenario 3
A new Trojan/worm is released on the Internet - it distributes itself through Email attachments and downloads malicious code from an Internet FTP server. A number of people in your organisation have opened such attachments. Points to address:

  • How do you identify infected systems?
  • How could you prevent the malware entering the enterprise before antivirus signatures
    were updated?
  • How would you prevent further spread?

Scenario 4
An employee at your site mentions that his bank account has been cleaned out by hackers in Estonia. He has been using online banking from his company workstation. Points to address:

  • Could you confirm whether or not his password was stolen?
  • How would you report this incident?
  • Would you warn other employees?

Scenario 5
The database administrator finds a strange directory on a database server, created by root 6 weeks ago. Points to address:

  • How will you identify the source of the attack?
  • How will you preserve the system?
  • How will you identify what has changed on the system?
  • How will you recover the system?

Scenario 6
A network intrusion detector identifies that someone has downloaded malware from a webmail account.

  • How will you identify the person involved?
  • How will you make sure their workstation is unaffected?
  • How will you prevent it from reoccurring?

1. A survey conducted by the Employers and Manufacturers Association (Northern). Press release 6 December 2005.