How could we know that a server is hacked? This is a general question about security, but how do you actually answer it?

Your answers might be:

• You notice something strange on the server.
• Intruders contact you because they want you to know about it.

These answers are correct, but when will you notice it? Your system may have already received serious damages, which is impossible to recover. When you want to protect a valuable system, passive action is not the prefer outcome. Then, how to act more actively on the intruder detection?

To get a clearer picture, imagine a server as your house; how could you know if someone sneaks into your house? You may use simple and classic style without electronic devices by putting a paper on the door. And when you come back to your house, you may notice a new location of the paper if someone opens the door. Otherwise, you can use some electronic devices such as door sensor. The door sensor will trigger the alert when anyone opens the door.

However, if you would like your friends to visit your house, the alarm will also ring every time when your friends open the door, too. To allow only the authorized friends to enter the house without triggering the alarm, you may install a video camera in front of the door in order to ask every visitor to show their face on a camera. If the person is authorized, the alarm will be disabled.

However, what if your visitors turn into foes later after entering your house? They might steal something from your house. Then, you have to install security cameras everywhere in your house. And you have to observe every videos to find out whether someone does something bad in your house. You may use a super intelligent software to run analysis on every video frame and tell you if your visitors do something that is not allowed.

The procedure above is similar to what you should implement on the servers.

How these things work on the servers

We do not have video recorders on the servers, but we have logs. When something bad happens, the first thing we investigate is the logs. Logs are extremely important to any security team. Logs collect everything, including authentication, executed commands, and network logs. This information helps security teams identify intruders and what they do on the system.

So, how does it work on a server? Will you force your colleagues to read every single line of logs 24 hours and call everyone when something unusual is suspected? Of course not, we use a super intelligent software to run analysis on every line of logs just like how we do it on security camera.

Here is how we implement the process on servers:

1. Make sure everything is actually logged.

You can use Linux built-in audit subsystems to log all activities or write down your own custom auditd daemon to log everything to files.

You need to make sure that the audit commands cannot be stopped by anybody. We do use log monitoring system to notify when the audit commands is stopped, for example, to trigger alert when no log is received from a specific server in a certain period of time. If you use Graylog for log monitoring, I have wrote about the Graylog plugin, which can trigger alert when no log is sent from the server. Follow the link here to download it.

2. Monitor and analyze all logs.

You can use any software to monitor logs. At Omise we use Graylog. This software can centrally collect logs from distributed applications, parse and extract data from log messages, as well as, identify the problem and trigger alerts when suspected content is found. Since it is open source, we can develop and contribute our improvement back to the community.

A general overview of Omise Graylog infrastructure:

We added a rule to Graylog pipeline to process logs in streams. A pipeline is processing steps applied to streams. So you can categorize your logs with streams, evaluate though pipeline, and trigger alerts.

The example of the rule to capture an open session as following:

rule “session opened for”

when

(has_field(“application_name”) && ($message.application_name == “su” || $message.application_name == “sshd” || $message.application_name == “sudo”)) &&

contains(to_string($message.message), “session opened”)

then

let fields = regex(“session opened for user (.+) by([^\p{Punct}]+)”, to_string($message.message), [“session_opened_for”, “session_opened_by”]);

set_fields(fields);

end

This pipeline rule will add a name of user who trying to open a session. Please also note that Graylog is built on the Java, so you can use Java regular expression syntax on pipeline rule, too.

3. When something unusual is suspected, send a message to the related person via Slack.

We use Slack for real-time notifications to send an interactive message to the destined users. After a log was analyzed from a step above, we configured Graylog to send notification to Slack. However, if we just send an alert to Slack channel, how long will the team notice a problem? And who will be responsible for that problem?

We solve this problem by using a name of person who attempts to open a session and mentioning this person on Slack. To know if the person is actually responsible of the issue, follow the steps below.

4. If the person performs an action, then, they need to acknowledge it on slack.

So we added an <em>Acknowledge</em> button to the interactive message. The interactive messages ensure us not to miss anything. When the mentioned person receives an alert, he replies the bot by clicking on <em>Acknowledge</em> button and the bot will update the message with the name of user who clicks on the button. The security team will know that this alert message is not triggered from intruders and by whom the issue is being taken care.

5. If they don’t, they need to let the security team know.

Security is hot. We cannot sit and wait until it becomes our nature. The first person who needs to cool it down is the person who is accountable for trying to do something. If the person finds out that he is not the one who does a bad thing or someone takes his account, he needs to click a <em>It is not me!!</em> button. The bot will send a notification to security team immediately.

How does it look?

To solve this problem, we decided to create our own Graylog Slack plugin. The plugin can mention the person who is found on logs, add link back to logs, and add <em>Acknowledge</em> button to the message. Download our plugin here.

Our example case and final result:

1. Siri executed a <em>sudo</em> command on <em>xg1esha</em> server.
2. Graylog received a message and sent the alert message to Slack.
3. Siri received a Slack notification and clicked <em>Acknowledge</em> button.
4. Bot updated a message with acknowledged status.
5. If Siri was not the one who executed it. Siri needed to click <em>It’s not me!!</em> button. Then the security team would take an action and might disable Siri’s account.

Securing your logs.

Graylog supports Syslog inputs with TLS encryption and client/server side authentication. All logs sent to Graylog are encrypted during all times of transit, clients will verify if they are talking to the correct server, and the server will verify if the clients are authorized (via certificates). To avoid detection, this is necessary in order to avoid logs being intercepted and changed. Even within a trusted local network, you would still want to encrypt your logs.

TLS encryption and authentication is a built-in feature of Rsyslog and Syslog-ng. If you use other systems, such as JournalD, you will need to rely of GELF or other inputs.