Security in a Grid environment

Protecting administrative credentials

carlos.fuentes@rediris.es (Carlos Fuentes Bermejo) — Tue, 14 Mar 2006 13:35:47 +0200

Credentials to access systems or Grid services should be well protected to prevent attackers from gaining access to the system or one of its services.

For instance, for Grid services, the certificate private key (generally stored in userkey.pem or *.p12 files) is the secret part of the information representing the identity of its owner. This information is secret and must remain readable only by its owner. If the private key becomes known to an attacker, he/she will have the ability to impersonate the owner of the certificate on the Grid. While protecting private keys is under the responsibility of their owners, when allowed, site administrators are encouraged to periodically search for publicly readable private keys on their hosts. Unprotected and publicly readable private keys should be sent to the relevant CA for revocation.

Also, SSH bruteforce attacks are very common and it is recommended to use SSH keys authentication instead of password authentication to authenticate against remote SSH servers.
Indeed, once the SSH public key is stored on the remote server, it is possible to authenticate against it by using the relevant SSH private key, which is protected by a passphrase.

Of course, again, this mechanism is efficient only if the private key is protected by a good passphrase!

Updating the CRLs

kouril@ics.muni.cz (Daniel Kouril) — Tue, 12 Feb 2008 12:14:00 +0200

Even though digital certificates have limited lifetime, information in a certificate can become invalid even before the certificate expires, e.g., when he corresponding private key is compromised or lost, the affiliation of certificate owner is changed, etc. In such a case, it is necessary to label the certificate as invalid so it cannot be accepted anymore and the issuing CA must revoke the certificate. In order to make information about revoked certificates widely available, each standard CA regularly publishes certificate revocation lists -- CRLs identifying certificates that have been revoked by the CA. Each machine in a Grid must maintain a local copy of the CRLs of all CAs it trusts. Proper checking of the CRLs must always be an integral part of certificate verification.

The gLite distribution contains a cron script (fetch-crl) that automates the periodical retrieval of CRLs from all CAs. Always make sure the script is installed and enabled. CRLs installed on a local system are automatically checked by most gLite services. However, other services must be explicitely configured to do the CRL checks, therefore, always consult documentation, especially when configuring a third-party application, such as Apache.

Containing user jobs

Romain.Wartel@cern.ch (Romain Wartel) — Wed, 28 Sep 2005 14:56:15 +0200

After a job is completed, it is important to ensure the environment is correctly cleaned from all residual user activity (ex: processes and files), to prevent an attacker to implant a more permanent backdoor in the system.

In particular, SEs, CEs, WNs and RBs all grant some level of user level access via a batch job or with gridftp, and for example, it is important to prevent the use of cron jobs.

Also, user processes can escape their process trees and in some conditions survive the batch system cleaning mechanism.

More details are available at http://goc.grid.sinica.edu.tw/gocwiki/Blocking_batch_jobs_from_creating_ssh_back_doors and at http://www.sysadmin.hep.ac.uk/wiki/ProcessesOnBatchNodes.

Configuring a system-level firewall

Romain.Wartel@cern.ch (Romain Wartel) — Wed, 28 Sep 2005 14:50:18 +0200

Even if a site firewall is offering an important protection, a personal firewall will add an additional security layer.
It enables the system to be protected against:
- The other systems of the local area network. It is very likely that an attacker compromising a system will try to attacker other machines on the same network.
- An incorrect configuration of the site firewall.
- An incorrect configuration of the system, which can be running unnecessary network services.
- A malicious user running an illegitimate service.

Incoming connections on a Linux system can be filtered using various methods, such as iptables. More information about iptables implementation is available here.

Using SSH keys

Romain.Wartel@cern.ch (Romain Wartel) — Wed, 28 Sep 2005 14:48:34 +0200

It is recommended to use SSH keys authentication instead of password authentication to authenticate against remote SSH servers.
Indeed, once the SSH public key is stored on the remote server, it is possible to authenticate against it by using the relevant SSH private key, which is protected by a passphrase.

This improves the experience and the security of both the user and the SSH server by:
Preventing the real password being sniffed and obtained. The SSH private key's passphrase is useless without the SSH private key itself
Making SSH bruteforcers useless if password authentication is blocked
* Authenticate against multiple servers using the same passphrase

Of course, this mechanism is efficient only if the SSH private key is protected by a good passphrase!

Disabling root login with password

Romain.Wartel@cern.ch (Romain Wartel) — Wed, 28 Sep 2005 14:47:55 +0200

System administrators should use SSH keys authentication instead of password authentication especially to login as root.
Once this process is in place, it is possible to disable root logins with password. This means that even if the root password is null or discovered, no remote root login will be possible.

To implement this, simply add this line in your /etc/ssh/sshd_config:

PermitRootLogin without-password

To apply the change, the SSH daemon needs to be restarted.

Applying security patches

Romain.Wartel@cern.ch (Romain Wartel) — Wed, 28 Sep 2005 11:30:09 +0200

Failling to apply security patches is one of the main causes of security incidents.

It is important to apply the security patches on a system on a regular basis:
- It can be very difficult to apply the updated packages when to the list is too long.
- It will prevent know vulnerabilities being exploited to compromise your system.

During the past few years, the amount of time between the publication of a security advisory and the relevant malicious code to exploit it has been continuously reduced. There are now more and more exploits released the same day as the advisory.

Grid participants should therefore aim at reducing the time needed to apply security patches as much as possible.

Performing system backups

Romain.Wartel@cern.ch (Romain Wartel) — Wed, 28 Sep 2005 11:29:15 +0200

Beyond the benefits regarding disaster recovery and business continuity, backups very often have a critical role during a security incident. Backups enable the investigator to find when and in what manner a part of the filesystem has changed and to retrieve information which has been lost of changed during a security incident.

Experience shows that it is also useful to test if the backup/restore procedures are actually working.

Disabling and uninstalling unneeded services

Romain.Wartel@cern.ch (Romain Wartel) — Wed, 28 Sep 2005 11:27:20 +0200

If a system is found compromised, it is extremely likely that the attacker has used a network service. Each network service running on a system is a potential entry point for an attacker.

Therefore, it is recommended to reduce the number of entry points by disabling and uninstalling unneeded services on the system.
The following command will list the services that are listening on the network:
/bin/netstat -eap |grep LISTEN

It is important to uninstall these unneeded services to make sure they will not be re-activated by mistake or by another process.

World-writable files or directories

rumler@cc.in2p3.fr (Rolf Rumler) — Mon, 02 Apr 2007 12:00:00 +0200

While world-writable directories might be useful to provide users with additional memory for their programs in a dedicated area (typically /tmp, /scratch, etc.), it important to periodically check that no unexpected world-writable file or directory resides on the system, in particular at the locations included in the PATH environment variable of the users (ex: /usr/bin). The use of world-writable files is in general strongly discouraged, as they may be abused to cause various instabilities on the affected system (partition full, missing data, etc.), but in some case may also enable a malicious user to trick other users (including root) into executing arbitrary code.

Central syslog server

Romain.Wartel@cern.ch (Romain Wartel) — Wed, 28 Sep 2005 14:52:26 +0200

If a system is compromised, it is very likely that the attacker will delete the log files that have been created during the intrusion. Unfortunately, system logs are often critical during the post-incident analysis, and the logs on the compromised system cannot be trusted as the attacker could have changed them.

This problem can -at least partly- be solved by using a central syslog server. Such a server can gather the system logs of many nodes on the network. As a result, the attacker may be able to change the logs on the compromised system, but it will be very difficult to change them on the central syslog server.

A central syslog server is therefore a great asset to understand how and in what way a system has been comprised. It can very often make recovery of the service quicker after a security incident and can help prevent the same attack happening again.

It is possible to find some information about implementing a central syslog server on the EGEE SEE ROC Wiki and here and here.

Available free space

Alessandra.Forti@cern.ch (Alessandra Forti) — Fri, 30 Mar 2007 12:00:00 +0200

Partitions completely full should be avoided. The consequences of lack of space are different depending on what partitions become full, but there are few partitions (or they can be directories filling their partitions) whose filling might partially or completely stop the machine from working. More information is available here.

System metrics

Romain.Wartel@cern.ch (Romain Wartel) — Wed, 28 Sep 2005 14:51:45 +0200

It is important to monitor system metrics, as it helps ensuring the continuity of the service and could sometimes reveal an intrusion.
For instance, it can be critical to be notified as soon as a particular service is down.

They are very common tool to enable system monitoring, such as http://www.nagios.org.

System performance

Romain.Wartel@cern.ch (Romain Wartel) — Wed, 28 Sep 2005 11:23:38 +0200

Monitoring system performance and review its evolution on a regular basis can be very benificial.
This enables system managers to monitor the past and current usage of their service, but also to anticipate future usage in many occasions.
There is also a great security interest in monitoring system performance. Many attacks will indeed impact system performance, such as network or disc usage. Even if monitoring system perfomance will most of the time not be enough to spot a system compromise, it often gives very useful information to understand how and when a system compromise happened.

There are several tools available to monitor system performance. Ganglia is one of the most well known tools, especially for clusters and groups of computers. It allows users to analyse the evolution of many factors to detect performance issues or bottleneck from a Web page.

Ganglia is available here.

System patching status

Romain.Wartel@cern.ch (Romain Wartel) — Wed, 28 Sep 2005 10:50:26 +0200

Pakiti provides a monitoring and notification mechanism to check the patching status of Linux systems.

Security patches are usually available from Yum or apt-get repositories for Red Hat Linux Legacy products, Fedora Core and Scientific Linux, and from the Red Hat Network via the up2date tool for Red Hat Enterprise Linux.

Once installed on a client host, Pakiti will check every night if new patches are available and report them to the Pakiti Server. No patch will be installed automatically.

As a result, the Pakiti Web page provides a list of the registered systems and the list of the pending patches for them. This helps the system administrator keep multiples machines up-to-date and prevent unpatched machines to be kept silently on the network.

Pakiti is available here.

Backups

Romain.Wartel@cern.ch (Romain Wartel) — Wed, 28 Sep 2005 11:35:47 +0200

Backups have a critical role in disaster recovery, business continuity, and during a security incident analysis. It it therefore recommended to check the backup/restore procedures on a regular basis.

This enables the service manager to make sure that the service can really be restored at any time.

Network services

Romain.Wartel@cern.ch (Romain Wartel) — Wed, 28 Sep 2005 11:34:52 +0200

Even if the system has been carefully configured, it is always a good idea to check what services are really listening on the network. Ideally, this test is performed from another host in order to see what services are remotely available.

An excellent tool to do this remotely is Nmap. More infomation about Nmap can be found here Quite accurate information can be obtained with the following command:/usr/bin/nmap -vv -sS -sV -sR -P0 -p 1-65535 remote_server>

Remote vulnerabilities

Romain.Wartel@cern.ch (Romain Wartel) — Wed, 28 Sep 2005 11:33:47 +0200

Testing know vulnerabilities on your systems, during both the development and the production phases, is definitely a great asset.
Several vulnerability scanners, such as http://www.nessus.org/ are quite simple to use. Nessus performs vulnerability testing against remote systems for a large range of service.

It is possible to unselect dangereous tests such as Denial of Service. Nessus needs to fetch the latest vulnerability tests on a regular basis to be really efficient.
Running Nessus scans against network routers can sometimes be very helpful.

Follow the incident response policies

Romain.Wartel@cern.ch (Romain Wartel) — Wed, 28 Sep 2005 11:32:21 +0200

Any unauthorised access, use or change on a system, or the use of a system in violation of the existing policies, is a security incident.

Grid participants must report security incidents by following the relevant Incident Response policies.
Usually, it is recommanded to follow the local Incident Response policy and the Grid Incident Response policy.

For guidance, the following policies are used by OSG and EGEE/LCG:
- OpenScienceGrid Security Incident Handling and Response
- EGEE/LCG Security Incident Response Policy

Responding promptly to a security incident by providing relevant and accurate information can greatly reduce the impact it can have on a service.

Business continuity plan

Romain.Wartel@cern.ch (Romain Wartel) — Wed, 28 Sep 2005 11:31:34 +0200

A critical area of the system documentation is the business continuity plan.
This document is part of the system risk analysis, in the event of a major disaster.

The aim of this document is to provide detailed procedures that need to be followed in order to maintain a service when a major disaster occurs. For instance, hot spare backup machines, backup tapes or off-site mirror can help restoring a service prompty when required.
Ideally, a business continuity plans contains enough details to be followed and fully executed by non-expert staff.

It is far from easy to define a reasonable business continuity plan. Every aspect of the plan needs to be carefully defined in order to ensure that it can be used at anytime.
Therefore it needs to be defined and agreed in advanced in order to be fully effective.

System documentation

Romain.Wartel@cern.ch (Romain Wartel) — Wed, 28 Sep 2005 11:31:06 +0200

Writing and keeping system documentation up-to-date is an ongoing task, which enables the system manager to have an overview of the system at anytime regarding:
- The purpose of the system
- The hardware configuration
- The filesystem layout
- The network configuration, including firewall rules
- The various groups and users
- The recovery procedures
- The business continuity plan
- The system backup
- The list of people having privileged access
- The logs configuration
- The change control procedures
- The system monitoring
- The system patching
- The files integrity checking
- The dependency on other systems

If a security incident occurs, having an online and up-to-date system documentation can greatly reduce the impact of the incident on the system and on the organisation it belongs to.

Host-level IDS

Romain.Wartel@cern.ch (Romain Wartel) — Wed, 28 Sep 2005 14:59:51 +0200

Host level Intrusion Detection Systems (IDS), such as Tripwire usually drastically improve the accuracy of forensics during a security incident.
Indeed, such tools generally generate a list of hashes of the main files on a system. It is then easy to check what has been changed on the system between two snapshots.

In order to be trusted, the Tripwire database has to be store on a read only media, such as a floppy disk or an USB token with a physical write protection.

Rootkit checkers

Romain.Wartel@cern.ch (Romain Wartel) — Wed, 28 Sep 2005 11:39:00 +0200

Running tools to find installed rootkits can help the system administrators to detect intrusions on their systems.

A root kit is a collection of tools used by an intruder after breaking into a computer system. These tools can help the attacker to hide and maintain his access to the system and use it for malicious purposes.

Chkrootkit is a common unix-based program which checks a system for known rootkits. It works by using several mechanisms, including comparison of file signatures to known rootkits, and checking for suspicious activities (ex: processes listed in the proc filesystem but not in the output of the 'ps' command).

Chkrootkit should be run on a regulary basis on the systems to detect intrusions immediately to take the approriate actions.

More details are available from:
http://goc.grid.sinica.edu.tw/gocwiki/Installing_and_using_chkrootkit

Network-level IDS

Romain.Wartel@cern.ch (Romain Wartel) — Wed, 28 Sep 2005 11:37:48 +0200

Monitoring the network can provide the system managers with an alert and report mechanism in order to detect signatures of an attack.
Network-level IDS analyse the network traffic in real time in order to detect suspicious network patterns. It can then trigger an alert that will enable the system manager to take relevant actions.
As a result, such tools help with detecting and preventing security incidents.

The network-level IDS should be installed close to a main network node, where the traffic level is high. It is also important to keep the network patterns up-to-date.

The most famous network-level IDS is snort. It also interesting to consider other IDS, such as Bro IDS, or the Prelude Hybrid IDS.

Centralised IDS

Romain.Wartel@cern.ch (Romain Wartel) — Wed, 28 Sep 2005 11:36:27 +0200

Host-based IDS such as Tripwire are quite efficient for a few stand alone systems, but it is unfortunately not a very scalable solution.

Several very advanced IDS are now available to be deployed on a large farm of computers. These IDS offer host-based IDS functionalities on a distributed set of hosts, by using a central service, which performs file integrity checking and host-based intrusion detection.

For instance, Osiris and Samhain are both well known IDS that can be deployed on a large number of systems.

If you have any interest in deploying this type of IDS, please contact Romain.Wartel@cern.ch