Category: OSSIM (Page 1 of 2)

Alienvault: File Integrity Monitoring on Linux Part 2

So based on our previous article you have so far set up OSSEC (or HIDS in Alien-speak) in your Linux host which you want to monitor. The next thing to do is to configure FIM to work.

To recap, we have a running CENTOS7 system running in our lab and we finally got our ossec to be communicating with the Alienvault server. You can verify connectivity either through the CLI logs, or using the USM Interface. Now the HIDS can be used for a lot of things – it’s obviously a Host IDS (hence the name), but it’s also a log forwarder as well, so for Linux systems, it doubles up as a security logger, so you don’t need to configure separate plugins to log, for instance SSH denied attempts. If you don’t have the HIDS, you have to forward logs from rsyslog then setup Alienvault plugin for SSH to normalise SSH logs and create those events. HIDS does this for you. Try it. You can attempt multiple logins with wrong password and you should see an event called “SSHD Authentication Failed.”

But for this article, we will be focusing on File Integrity Monitoring or FIM for short. FIM in Alienvault USM is utilising OSSEC inbuilt integrity checking process called Syscheck. Syscheck runs periodically and depending on how many files/directories it is checking can run from 10 minutes to much longer. By default, syscheck in Alienvault executes very 20 hours – if that’s too long for you , you can shorten it in the configuration.

Let’s jump straight in.

In Alienvault (Server if you are using Standard), go Environment -> Detection and on HIDS tab, click on Agent. In the lower tabs, click on SYSCHECKS.

Over here is where you configure the Syschecks on the Agents and you can modify the frequency.

Because we are using Linux, we are going to ignore the portion where Windows Registry is being configured and go straight to: ”

FILES/DIRECTORIES MONITORED

Under files/Directories, put in a sample directory you need to monitor, for instance

/etc/pkf

Don’t worry, out of the box, standard directories being monitored are

/etc

/usr/bin

/usr/sbin

/bin

/sbin

We have in some cases clients insisting on us putting in /var/log in there to inform them of changes occurring in this directory. According to them, log files are key and they need to know if these log files are being changed.

Um, yes. Agree on the first part. But /var/log changes almost every nanosecond. Syscheck is not going to be of much use here. They are probably thinking about log archives as opposed to the current log folder. Anyway, we digress.

So go ahead and put in your own directory in there under agents and then restart HIDS from Alienvault, and also for good measure restart the agent as well (you can go Agent Control -> Click on the clock symbol under the Agent Name to restart). To check, you can click on Agent.Conf tab and you will find something similar to:

<agent_config>
    <syscheck>
      <frequency>1200</frequency>
      <auto_ignore>no</auto_ignore>
      <alert_new_files>yes</alert_new_files>
      <scan_on_start>yes</scan_on_start>
<directories realtime="yes" report_changes="yes" check_all="yes">/etc/pkf</directories>
    </syscheck>
  </agent_config>

So it looks all set up. If you have restarted HIDS and also the agent, you should be able to verify on the agent itself if the configuration has been uploaded. On the client, go to

/var/ossec/etc/shared

Look into agent.conf file and you should be able to see the same thing as the configuration above. Also, you can go to

/var/ossec/logs

and look into ossec.log file and you should be able to see something like

ossec-syscheckd: INFO: Monitoring directory: '/etc/pkf'.
ossec-syscheckd: INFO: Directory set for real time monitoring: '/etc/pkf'.

So there you have it. You can do some testing now.

So we will go into the local directory of our CENTOS and go ahead to create a few random files. The first thing you notice is that even if in our config there was:

<alert_new_files>yes</alert_new_files>

We still do not get any alerts once we create new files in the directory. This is because OSSEC doesn’t check new files in realtime (just changes to files), and we will need to wait for our syscheck to run, or you can go ahead and restart the agent from the Alienvault GUI. For good measure, change a few things about the files as well.

You might notice a strange thing happening here.

Going into the SIEM, you might not find any events relating to integrity issues in your host. This doesn’t seem to be an isolated incident, if you head over to the Alienvault forum, you will see many people having the same issue: We have enabled FIM and we can’t find anything on the SIEM or any events!

If you check on the agent itself, and you click on the “modified files”

You will see a raw list of all the files modified and you will see that /etc/pkf/filename is there listed as well, so it means OSSEC is working and syscheck is working. Another way to verify is to head over to your Alienvault Server and go to

/var/ossec/logs/alerts 

grep pkf alerts.log

Basically I am doing a grep on anything that identifies the files or directories I am looking at to see if alerts are generated. You should change the grep to something related to your filename/directory name. You should be able to see that alerts are generated.

So what gives?

Plugins.

Apparently for some strange reason, some Alienvault setup by default does not have the proper plugins enabled to read the integrity alerts log of ossec. This is very strange, as FIM is touted as a feature for Alienvault, but we need to still work further to get it up and running. So go ahead to your Alienvault GUI:

Configuration -> Deployment

Click on System Detail of your Alienvault setup

Click on Sensor Configuration in the menu on the right side

Go to “Collection”

You notice you have Alienvault_HIDS and Alienvault_NIDS enabled. However, in some cases, Alienvault_HIDS-IDM plugin might be missing and can’t be found under “Plugins Available” column. IDM Is for identity management and it needs to be enabled for FIM to properly work.

The plugin that makes this happen is

ossec-idm-single-line.cfg

In our case, the plugin file was there in /etc/ossim/agent/plugins, but it wasn’t in the ossim database as a “Plugins Available” option. This generally means that it wasn’t (for some reason) written into the ossim-db. So head over to the directory in Alienvault:

/usr/share/doc/ossim-mysql/contrib/plugins

You will see that there is an ossec.sql.gz in there, so go ahead and unzip it and run

cat ossec.sql | ossim-db

alienvault-reconfig

Wait for the reconfig to occur then head back to the GUI of Alienvault, all the way back to the sensor configuration->collection and you will be able to see Alienvault_HIDS-IDM available for selection.

Go ahead and select it there, and then reconfig and now you can try to run the FIM test again.

a) Create a new file

b) Restart the agent (to simulate the syscheck being run)

c) Check SIEM , filter Data Sources to Alienvault HIDS, you should find

AlienVault HIDS: File added to the system.

d) Go to the host and edit the new file and change it

e) Go back and check SIEM and you will find

AlienVault HIDS: Integrity checksum changed.

The last event should be immediate and need not have any restart of the agent. Unless of course, we noticed if the change occurred during the time syscheck is running, if so the event will occur once syscheck finishes. It’s not perfect, but it will have to do.

Congratulations, you have FIM events up and running for Alienvault! If you need further assistance in any Alienvault matters, drop us an email at alienvault@pkfmalaysia.com and we will look into it.

 

Alienvault: File Integrity Monitoring on Linux Part 1

If you have been deploying or troubleshooting Alienvault long enough, you would know a few things: Alienvault is one of the most flexible SIEMs in the market. It has the most varied security features, and covers almost the entire spectrum of our PCI-DSS needs – from IDS, to SIEM, to File Integrity Monitoring, to vulnerability scaring to a partridge in a pear tree.

One of the products working under the Alienvault hood is OSSEC, which is a opensorce host based IDS. Sometimes, its interchangeable to HIDS, which is Host IDS, but really, the latter is simply the type; while the former is the actual name itself. For the sake of this article, we will interchange both terms.

OSSEC runs well with Windows, where Alienvault can do an auto deployment given the correct setup and credentials. However, it’s on Linux boxes that sometimes we get a bit concerned. Not because the product doesn’t work, but simply because the setting up of the installation. There is no auto deployment, so we need to set it up manually, and this might mean downloading the correct packages in the first place.

After this, we are going to look at a specific function of HIDS – File Integrity Monitoring or FIM for short.

Firstly, let’s get started. We have set up a simple CentOS 7 box in our lab in the same network as Alienvault, and we are going to install HIDS on this box as an AGENT. This will then talk to the Alienvault USM which is the server.

So let’s assume you have your agent system network setup (please ensure your DNS is set properly, you should be able to work this out in CentOS 7 either through the network tools or editing resolv.conf).

 yum groupinstall "Development Tools" -y

The CentOS development tools are very useful tools which is a bundle, used primarily for building and compiling software from source code. “Yum” here while making you think of going for a teh tarik is a command found in almost all red-hat based distros to run installations. It’s used for update, installations etc. In the old days before YUM, we would use RPM (which is really what YUM is using), but we would have to manually track down dependencies and it really sucks because to install an RPM package might mean to install a whole bunch of stupid libraries or updating stuff and you are basically running around the internet looking for RPMs like Where’s Wally. It looks awful now, but back in the days, RPM was heavensent. We didn’t need to do “tar”, configure, make, “make install” anymore!

Anyway, the -y argument behind simply automates the command by answering yes to the prompts. So once you run that, fingers crossed, everything runs ok and you get

Complete!

Which means everything is ok.

The next is to get the kernel-devel package.

yum install kernel-devel -y

This is a package that allows us to install a kernel driver later. It’s not the full kernel source, so it shouldn’t take too long before you see the “complete!”.

At this point you are ready to install OSSEC. If there are any issues, then troubleshooting is obviously required.

First, we need to locate the version of HIDS that can work with Alienvault. You might think heading to the latest HIDS in https://ossec.github.io/downloads.html might be the answer, but for Alienvault, we would recommend to get the 2.8.3 version. You can find it here:

https://bintray.com/ossec/ossec-hids/download_file?file_path=ossec-hids-2.8.3.tar.gz

So, go to a installation directory (optional) like /usr/src and run

curl -OL https://bintray.com/ossec/ossec-hids/download_file?file_path=ossec-hids-2.8.3.tar.gz

We used curl here because for some reason wget wasn’t installed. the -OL is supposed to handle the redirected links for that particular site and supposedly to rename it to a proper remote file name. It doesn’t do the rename though (don’t know?) and we wind up with a file called “download_file?file_path=ossec-hids-2.8.3.tar.gz”. Just rename it if you are into aesthetics to ossec-hids-2.8.3.tar.gz.

So now lets do an extraction

tar –xzvf ossec-hids-2.8.3.tar.gz

We now have a folder called ossec-hids-2.8.3. Go into this folder and then run

./install.sh

Once you run, you will be given a series of questions. Default should be fine for most, and you should just select ‘agent’ and also key in the server (Alienvault) IP address. Now if you are running a separate Alienvault setup (non-AIO), then this IP address is actually the address of your SENSOR. Not Alienvault Server. So don’t get mixed up. The Sensor is the Server. Hm.

So everything ready, fingers crossed, just go ahead and install. There will be a lot of text filling your screen but the important thing is that there is no ERROR or WARNING (well warning ain’t bad), but at the end you should have a welcome note stating

 Thanks for using the OSSEC HIDS.
 If you have any question, suggestion or if you find any bug, contact us 
at contact@ossec.net or using our public maillist at ossec-list@ossec.net 
(http://www.ossec.net/main/support/ ).

Press enter and you should be out of the installation. Congratulations!

You are not done yet. You still need to get Alienvault to talk to your box. The steps are as follows:

a) Generate an Agent Key from Alienvault

Go to your Alienvault AIO or your Server (since a standard sensor has no GUI, remember?).

Environment->Detection->Agents

Click “Add Agent”

Select the host from the list (It should be there automatically, but if it’s not just add it there through the asset list).

So now the agent has been created but you should see it as “Disconnected” from the list. Click the little Key sign that says “Extract Key”.

You should see something like

Agent key information for '2' is:

MiBIb3N0LTE5Mi0xNjgtMC01MCAxOTIuMTY4LjAuNTAgMDBmYzI0MzUyNzg4N.....etc
The garbled message is the key. So go ahead and highlight and copy it.

b) Import the key into the agent system

Go back to your agent system and head over to /var/ossec/bin and run

./manage-agents

Type in ‘I’ to import

Paste the whole key into the screen and confirm adding it.

Quit and then restart by going

/var/ossec/bin

And

./ossec-control restart

c) Restarting HIDS on the server

On the server head over to

Environment->Detection->HIDS Control

On the right side, click “Restart” the HIDS and you should be fine.

d) Check the Agent Logs

Head back to the agent system and check the logs

cd /var/ossec/logs
more ossec.log

You should (hopefully!) see

INFO: Connected to the server (192.168.0.xxx:1514).

where xxx is your server IP address.

Back in the USM server you will be able to see that now the agent is “Active”.

In the next article we will see if we can get the FIM to work.

Deploy HIDS agent in a Checkpoint Environment

avlogo

In our years of deploying and implementing SIEM, if we get 10 ringgit everytime a customer says: “Nothing wrong on our side, your SIEM is the problem” and then after hours of troubleshooting, the customer responds: “Ooops sorry, seems like our problem, we blocked your traffic” – we would all be as rich as Jack Ma by now.

The issue here is not so much that SIEMs are notoriously difficult to deploy – they are not, but its more that other devices need to talk to it. Deploying SIEM is easy, the finetuning is the problem. Because products need to get logs to the SIEM – either through the good old Syslog via UDP/TCP 514 or through agents installed on the systems, or through third party applications like Snare and NXLog – both for the naughty Windows machine where no syslog facility is found for some strange, Bill Gatesy reason.

It’s also because most Administrators guard their systems like pitbulls and any request for them to send logs over is generally greeted by a dumbfounded “Why are you so stupid” look, or “Over my dead body” look, or simply you get ignored completely. Oh, and also the “I want to go home at 5 pm today, so don’t mess with my jam” look. Whatever. The war between Administrators and Implementers has been going on longer than the blood feud between Lycans and Vampires.

We have been through deployments where countless of hours were spent just trying to convince our customer: “We are NOT getting anything on the interface. No packets. It’s being blocked.” And customer: “No, there are no firewalls between.” Five hours later, customer: “Oh yeah, there is a small firewall between and it’s blocking. Cheers”. On our SIEM, it’s easy. If our interface doesn’t see it, it ain’t there. That’s it. Do a tcpdump and grep the IP or port and boom, we know it’s not our problem.

The issue here is that “Not our problem” generally gets translated to “It’s now your problem” and we end up troubleshooting for our clients how to fix THEIR systems and devices. I’ve called my pals at Fortigate, Bluecoat, Cisco, Juniper etc so many times, asking them about issues that my client should be doing it themselves.

So anyway – we had this issue whereby we deployed a fair bit of HIDS (Host IDS, aka OSSEC) agents in a fairly large environment. It basically traverses through a firewall (Checkpoint). That Checkpoint firewall dropped UDP 1514 connection between agent and our AlienVault server. Port 1514 is the port that our HIDS uses to communicate between agent and server.

Firstly, establish whether we are getting those traffic or not in the interface. If we are not, then the problem could be on their end. When we do a tcpdump for udp 1514 on that specific host, we are able to observe some traffic from the server and vice versa. In our case even with that, the connection cannot be established. Bascially, our agent is able to reach the server but the server tries to respond but the traffic disappears. Therefore, the agent is orphaned.

For this case, after troubleshoot and checking, we found out that Checkpoint is dropping the UDP 1514 traffic responses from the alienvault server. So the communication between the HIDS agent and the server is never established. The log shows that UDP traffic is dropped with the following message:

Message_Info: Violated unidirectional connection

Great. UDP traffic is a stateless protocol. According to Checkpoint, by default, a reply to a UDP packet is not allowed. The Security Gateway can mark a connection in the Connections Table to allow traffic to pass only in one direction (hence the term ‘unidirectional’). If a UDP connection uses a bi-directional communication method, this would create a violation.

Therefore, the workaround to this is to allow Checkpoint to respond to this bi-directional communication. It’s pretty straightforward for Checkpoint actually.

You will need to add or edit a service object in Checkpoint. Again, we are Alienvault implementers but we end up mucking through Checkpoint firewalls just to get our job done!

So on a checkpoint edge box, it’s basically

Main Menu->Network->Network Services tab

You can now edit or add.

Go ahead and add a new UDP service, fill in the requirements, and then you will have an option for advanced, click OK.

In the Advanced UDP service property, ensure “Accept Replies” is clicked.

Now go ahead and use this service under the new or existing policy rule that has been set up for Agent->Server communication for HIDS.

Ta-da, done!

If you have any questions on Alienvault USM deployment, drop us a note at alienvault@pkfmalaysia.com.

AlienVault Troubleshooting: NFSEN cannot start

One of the issues we faced was that our NFSEN suddenly barfed when restarted. This is highly annoying because everytime we reconfigure AlienVault, it has to hang at NFSEN service restart because it couldn’t get it up. I don’t know why.

Eventhough we don’t use netflow much in our environment, it was still a pain for us so we tried to troubleshoot it and finally resolved it.

The issue was when we click on Environment>Netflow we saw these errors

ERROR: nfsend connect() error: Connection refused!

ERROR: nfsend – connection failed!!

NFSEN

Obviously this was irritating. Under Configuration>Deployment>Sensors, we clicked on our AIO and scroll to the bottom, we saw that the netflow collection configuration was not running.

I think it could be because we didn’t set any interface to be ‘monitoring’. We went ahead and set it using the alienvault-setup menu and assigned eth1 to be monitoring. Strangely we couldn’t assign it in the GUI under Configuration>Deployment>AIO>Sensor Configuration and Detection. We only had option for Eth0 (our management) and ETH5 (our logging interface).

Anyway, once we set an interface to monitoring we still couldn’t start nfsen through the GUI or even through the command line under /etc/init.d/nfsen start/stop.

It kept giving this error

Use of uninitialized value $pid in scalar chomp at /usr/bin/nfsend line 765.
Use of uninitialized value $pid in kill at /usr/bin/nfsend line 767.
Use of uninitialized value $pid in concatenation (.) or string at /usr/bin/nfsen

Which made as much sense as greek.

In any case, at least it gave a clue that /usr/bin/nfsend might be complaining because nfsen wasn’t up in the first place. So we went ahead and
VirtualUSMAllInOne:/usr/bin# ./nfsen start
Starting nfcapd:(564D89B81691003B6E98F73F9FFA258C)[25330]
Starting nfsend.

This apparently didn’t through any errors and nfsend seems started! Do a ps -ef and grep nfsen and you have a nice PID allocated.

VirtualUSMAllInOne:/usr/bin# ps -ef | grep nfs
www-data 25330 1 0 23:12 ? 00:00:00 /usr/bin/nfcapd -w -D -p 555 -u www-data -g www data -B 200000 -S 7 -P /var/nfsen/run/p555.pid -I 564D89B81691003B6E98F73F9FFA258C -l /var/cache/nfdump/flows//live/564D89B81691003B6E98F73F9FFA258C
www-data 25332 1 0 23:12 ? 00:00:00 /usr/bin/perl -w /usr/bin/nfsend
www-data 25333 25332 0 23:12 ? 00:00:00 /usr/bin/nfsend-comm
root 25339 22649 0 23:12 pts/0 00:00:00 grep –color=auto nfs

So we stopped it again but this time with the init.d script.

VirtualUSMAllInOne:/usr/bin# /etc/init.d/nfsen stop
Stopping Nfsen: nfsenShutdown nfcapd: (564D89B81691003B6E98F73F9FFA258C)[25330]. .
Shutdown nfsend:[25332]..

And started it again using the init.d script

VirtualUSMAllInOne:/usr/bin# /etc/init.d/nfsen start
Starting Nfsen: nfsenStarting nfcapd:(564D89B81691003B6E98F73F9FFA258C)[26383]
Starting nfsend.

Now we checked back our netflow on the gui and it works.

I don’t know if anyone else is facing this or has an explanation to this, but it might or might not have anything to do with our interface not being set to monitoring. You can try this out if you are facing this issue.

 

OSSIM Part 2: Typical Setup

From the previous post, you have successfully installed OSSIM into a VM running ESXi 5.1. Congratulations.

Go ahead and access the web IP address of the OSSIM (you do remember it, don’t you??!)

You are greeted with the same screen as AlienVault – setting the admin account. You should never lose the root password, the admin password can be reset.

Once that is done, relogin again with the new admin password and go through the wizard.

Let’s start with the interface. Go ahead and configure one for Logging and the other for monitoring (no IP). Assign another IP to it. For now, we didn’t do any scanning or other setup, the whole idea was just to see what OSSIM is offering.

In case you messed up and only set up 2 network interfaces, don’t worry. Just add a new network interface into the VM and power up the OSSIM again.

You would want to reconfigure it to have that new interface so go to configuration and wait for your OSSIM to load up. The annoying thing about AlienVault is that the Getting Started Wizard is literally ‘Getting Started’. You don’t have a way to invoke that wizard again so you generally have to reconfigure your network devices the hard way. There are two ways:

SSH into your OSSIM and run alienvault-setup if not already in the setup menu. Go to Configure Sensor > Configure Network Monitoring and select the new ETH as your network monitor. Then you need to apply changes and wait for OSSIM to rebuild

Second option is GUI>Configuration>Deployment>Click on the OSSIM installation

On the top right, click on Sensor configuration and then on ‘Detection’. You will see listening interfaces there. Go ahead and select the NIC to add to listening interfaces. You don’t need an IP address for monitoring. Apply Changes.

It’s just annoying, and we really wish OSSIM would just allow us to run the getting started wizard again.

If you need to set up a logging and monitoring role, you just need to go to the alienvault-setup, setup the network interfaces under system preferences and give it an IP. Immediately gets a logging and monitoring role. There shouldn’t be more than one interface per subnet. The question here is, can your management interface also be the logging interface. Yes of course, but it’s best not to.

Now, again, we wish OSSIM would be a little more clear on this. They already have an awesome GUI, but you would think running the wizard again would be a simple thing to do. Nope, it’s not. You have one shot at it.

So now, you have an interface to manage, to log and to monitor.Go ahead and have a look at it under the deployment components.

Once this is done, you are basically good to go to start OSSIM!

« Older posts

© 2020 PKF AvantEdge

Theme by Anders NorenUp ↑