Jump to content

Syslog-NG


rev.dennis
 Share

Recommended Posts


Logging your system messages to a remote server is a good security practice. With all servers logging to a central syslog server, it becomes easier to correlate events across your company. It also makes covering up mistakes or malicious activities harder because the purposeful deletion of log files on a server cannot simultaneously occur on your logging server, especially if you restrict the user access to the logging server.


 


syslog-ng can be used to collect local syslog messages & monitor log files on your servers and then forward them to a collector.


syslog-ng OSE is an open source alternative to the standard syslog daemon that's commonly found on UNIX and UNIX-like (*nix) systems. It uses the basic syslog protocol, but extends it with content-based filtering, flexible configuration options and adds important features, such as using TCP (as well as TLS), which is much more reliable than UDP.


 


Installation - Using a Package Manager

Depending on your Linux distribution you can use yum or APT (do this with root or sudo privileges):



# apt-get install syslog-ng

You will most likely need to enable Extra Packages for Enterprise Linux (EPEL)



# yum install syslog-ng


Installing and Starting syslog-ng

 

You can install syslog-ng using standard Linux procedures.The syslog-ng and rsyslog packages cannot be installed at the same time. You have to uninstall one in order for the other to work. Here’s how you can install syslog-ng using RPM package files.

  1. Uninstall rsyslog using the rpm command. There are some other RPMs that rely on rsyslog so you will have to do this while ignoring any dependencies with the –nodeps flag.   $ rpm -e --nodeps rsyslog
  2. Install syslog-ng using yum.  $ yum -y install syslog-ng
  3. Start the new syslog-ng daemon immediately and make sure it will start on the next reboot.
 

Systems using sysvinit:

$ chkconfig syslog-ng on

$ service syslog-ng start

Starting syslog-ng: [  OK  ]

$

 

Systems using systemd:

$ systemctl enable syslog-ng.service

$ systemctl start syslog-ng.service

Starting syslog-ng: [  OK  ]

$

 

Your new syslog-ng package is now up and running and ready to go!

 

Configuring syslog-ng Clients

 

Clients logging to the syslog-ng server don't need to have syslog-ng installed on them, a regular syslog client configuration will suffice.

If you are running syslog-ng on clients, then you’ll need to modify your configuration file. Let’s look at the below example – Syslog-ng Sample Client Configuration.

Syslog-ng Sample Client Configuration



source s_sys {
   file ("/proc/kmsg" log_prefix("kernel: "));
   unix-stream ("/dev/log");
   internal();
};


destination loghost { 
   udp("loghost.hosangit.com"); 
};


filter notdebug { 
   level(info...emerg); 
};


log { 
   source(local);
   filter(notdebug);
   destination(loghost); 
};

The s_sys source comes default in many syslong-ng.conf files, we have just added some additional parameters to make it work. Here the destination syslog logging server is defined as loghost.hosangit.com. We have also added a filter to the log section to make sure only the most urgent messages, info level and above (not debug), get logged to the remote server. After restarting syslong-ng on your client, your syslog server will start receiving messages.



 


 

Logging your system messages to a remote server is a good security practice. With all servers logging to a central syslog server, it becomes easier to correlate events across your company. It also makes covering up mistakes or malicious activities harder because the purposeful deletion of log files on a server cannot simultaneously occur on your logging server, especially if you restrict the user access to the logging server.


 


syslog-ng can be used to collect local syslog messages & monitor log files on your servers and then forward them to a collector.


syslog-ng OSE is an open source alternative to the standard syslog daemon that's commonly found on UNIX and UNIX-like (*nix) systems. It uses the basic syslog protocol, but extends it with content-based filtering, flexible configuration options and adds important features, such as using TCP (as well as TLS), which is much more reliable than UDP.


Installation - Using a Package Manager

Depending on your Linux distribution you can use yum or APT (do this with root or sudo privileges):



# apt-get install syslog-ng

You will most likely need to enable Extra Packages for Enterprise Linux (EPEL)



# yum install syslog-ng

Configure either using our Configure-syslog script or manually.

Compiling From Source


Download syslog-ng source code & eventlog source code. Install both eventlog & syslog-ng. Eventlog is a generic event logging library developed by Balabit. Once you've unzipped both packages (evenlog_x.x.xx.tar.gz & syslog-ng-x.xx.tar.gz), do this in each of those directories:



$ ./configure
$ make
$ sudo make install

Check your syslog-ng version


 


You'll need to know which version of syslog-ng you've got installed. We recommend running on the latest, but at least version 3.2 for best results.


 


Configure syslog-ng


The /etc/syslog-ng/syslog-ng.conf file

 

The main configuration file for syslog-ng is the /etc/syslog-ng/sylog-ng.conf file but only rudimentary help on its keywords can be found using the Linux man pages.



$ man syslog-ng.conf


Simple Server Side Configuration for Remote Clients

Below is a sample syslog-ng.conf file and outlines some key features. The options section that covers global characteristics is fully commented, but it is the source, destination and log sections that define the true strength of the customizability of syslog-ng.

 

Sample syslog-ng.conf File



options {


          # Number of syslog lines stored in memory before being written to files
          sync (0);


          # Syslog-ng uses queues
          log_fifo_size (1000);


          # Create log directories as needed
          create_dirs (yes);


          # Make the group "logs" own the log files and directories
          group (logs);
          dir_group (logs);


          # Set the file and directory permissions
          perm (0640);
          dir_perm (0750);


          # Check client hostnames for valid DNS characters
          check_hostname (yes);


          # Specify whether to trust hostname in the log message.
          # If "yes", then it is left unchanged, if "no" the server replaces
          # it with client's DNS lookup value.
          keep_hostname (yes);


          # Use DNS fully qualified domain names (FQDN) 
          # for the names of log file folders
          use_fqdn (yes);
          use_dns (yes);


          # Cache DNS entries for up to 1000 hosts for 12 hours
          dns_cache (yes);
          dns_cache_size (1000);
          dns_cache_expire (43200);


        };




# Define all the sources of localhost generated syslog
# messages and label it "d_localhost"
source s_localhost {
          pipe ("/proc/kmsg" log_prefix("kernel: "));
          unix-stream ("/dev/log");
          internal();
};


# Define all the sources of network generated syslog
# messages and label it "d_network"
source s_network {
          tcp(max-connections(5000));
          udp();
};


# Define the destination "d_localhost" log directory
destination d_localhost {
           file ("/var/log/syslog-ng/$YEAR.$MONTH.$DAY/localhost/$FACILITY.log");
};


# Define the destination "d_network" log directory
destination d_network {
          file ("/var/log/syslog-ng/$YEAR.$MONTH.$DAY/$HOST/$FACILITY.log");
};


# Any logs that match the "s_localhost" source should be logged
# in the "d_localhost" directory


log { source(s_localhost);
      destination(d_localhost);
};


# Any logs that match the "s_network" source should be logged
# in the "d_network" directory


log { source(s_network);
       destination(d_network);
};


In our example, the first set of sources is labeled s_localhost. It includes all system messages sent to the Linux /dev/log device, which is one of syslog's data sources, all messages that syslog-ng views as being of an internal nature and additionally inserts the prefix "kernel" to all messages it intercepts on their way to the /proc/kmsg kernel message file.

 

Like a regular syslog server which listens for client messages on UDP port 514, syslog-ng also listens on TCP port 514. The second set of sources is labeled s_network and includes all syslog messages obtained from UDP sources and limits TCP syslog connections to 5000. Limiting the number of connections to help regulate system load is a good practice in the event that some syslog client begins to inundate your server with messages.

The above example also has two destinations for syslog messages, one named d_localhost, the other, d_network. These examples show the flexibility of syslog-ng in using variables. The $YEAR, $MONTH and $DAY variables map to the current year, month and day in YYYY, MM and DD format respectively. Therefore the example:

/var/log/syslog-ng/$YEAR.$MONTH.$DAY/$HOST/$FACILITY.log

refers to a directory called /var/log/syslog-ng/2005.07.09 when messages arrive on July 9, 2005. The $HOST variable refers to the hostname of the syslog client and will map to the client's IP address if DNS services are deactivated in the options section of the syslog-ng.conf file. Similarly the $FACILITY variable refers to the facility of the syslog messages that arrive from that host.

 

Using syslog-ng in Large Data Centers

Below example is a sample syslog-ng.conf file snippet that defines some additional features that may be of interest in a data center environment.

 

More Specialized syslog-ng.conf Configuration



options {


          # Number of syslog lines stored in memory before being written to files
          sync (100);
};




# Define all the sources of network generated syslog
# messages and label it "s_network_1"
source s_network_1 {
          udp(ip(192.168.1.201) port(514));
};


# Define all the sources of network generated syslog
# messages and label it "s_network_2"
source s_network_2 {
          udp(ip(192.168.1.202) port(514));
};


# Define the destination "d_network_1" log directory
destination d_network_1 {
          file ("/var/log/syslog-ng/servers/$YEAR.$MONTH.$DAY/$HOST/$FACILITY.log");
};


# Define the destination "d_network_2" log directory
destination d_network_2 {
          file ("/var/log/syslog-ng/network/$YEAR.$MONTH.$DAY/$HOST/$FACILITY.log");
};


# Define the destination "d_network_2B" log directory
destination d_network_2B {
          file ("/var/log/syslog-ng/network/all/network.log");
};


# Any logs that match the "s_network_1" source should be logged
# in the "d_network_1" directory


log { source(s_network_1);
      destination(d_network_1);
};


# Any logs that match the "s_network_2" source should be logged
# in the "d_network_2" directory


log { source(s_network_2);
      destination(d_network_2);
};


# Any logs that match the "s_network_2" source should be logged
# in the "d_network_2B" directory also


log { source(s_network_2);
      destination(d_network_2B);
};


In this case we have configured syslog to:

  • Listen on IP address 192.168.1.201 as defined in the source s_network_1. Messages arriving at this address will be logged to a subdirectory of /var/log/syslog-ng/servers/ arranged by date as specified by destination d_network_1. As you can guess, this address and directory be used by all servers in the data center.
  • Listen on IP address 192.168.1.202 as defined in the source s_network_2. Messages arriving at this address will be logged to a subdirectory of /var/log/syslog-ng/network/ arranged by date as specified by d_network_2. This will be the IP address and directory to which network devices would log.
  • Listen on IP address 192.168.1.202 as defined in the source s_network_2. Messages arriving at this address will also be logged to file /var/log/syslog-ng/all/debug.log as part of destination d_network_2B.This will be a single file to which all network devices would log. Server failures are usually isolated to single servers whereas network failures are more likely to be cascading involving many devices. The advantage of searching a single file is that it makes it easier to determine the exact sequence of events.
  • As there could be many devices logging to the syslog-ng server, the sync option is set to write data to disk only after receiving 100 syslog messages. Constant receipt of syslog messages can have a significant impact on your system’s disk performance. This option allows you to queue the messages in memory for less frequent disk updates.

     
 


Link to comment
Share on other sites

  • 4 weeks later...
Guest dennis

First install some packages necessary to build syslog-ng. Replace “x86_64” with your own architecture for the devel packages (if in doubt, uname -m will print your architecture):



yum install gcc



yum install glib2-devel.x86_64



Add /usr/local/lib to ld.so.conf:



echo /usr/local/lib > /etc/ld.so.conf.d/local.conf



Create a working directory:



mkdir ~/workdir



Download and install eventlog:



cd ~/workdir



wget http://www.balabit.com/downloads/files/eventlog/0.2/eventlog_0.2.12.tar.gz



tar xzvpf eventlog_0.2.12.tar.gz



cd eventlog-0.2.12/



./configure



make



make install



Go back to the working directory:



cd ~/workdir



Download libdbi from sourceforge: http://sourceforge.net/projects/libdbi/files/libdbi/libdbi-0.8.4/



Compile and install it:



tar xzvpf libdbi-0.8.4.tar.gz



cd libdbi-0.8.4



./configure –disable-docs



make



make install



Go back to the working directory:



cd ~/workdir



Download libdbi-drivers from sourceforge: http://sourceforge.net/projects/libdbi-drivers/files/libdbi-drivers/libdbi-drivers-0.8.3-1/



Compile and install it:



tar xzvpf libdbi-drivers-0.8.3-1.tar.gz



cd libdbi-drivers-0.8.3-1



Configuring is a bit more tricky, as it needs switches and additional development packages. The following example is for PostgreSQL and MySQL, use ./configure –help to see options for other databases. Beware, that sqlite only works with a CVS snapshot of libdbi. First install the development packages for the database and architecture you use:



yum install postgresql84-devel.x86_64



yum install mysql-devel.x86_64



Then configure, compile and install it:



./configure –with-dbi-incdir=/usr/local/include/dbi –with-dbi-libdir=/usr/local/lib –with-mysql –with-mysql-dir=/usr –with-mysql-libdir=/usr/lib64/mysql –with-mysql-incdir=/usr/include/mysql/ –with-pgsql –with-pgsql-dir=/usr –disable-docs



make



make install



Using your self compiled database server will need slightly different parameters.



Finally download, configure and install syslog-ng:



cd ~/workdir



wget http://www.balabit.com/downloads/files/syslog-ng/sources/3.2.4/source/syslog-ng_3.2.4.tar.gz



tar xzvpf syslog-ng_3.2.4.tar.gz



cd syslog-ng-3.2.4/



PKG_CONFIG_PATH=/usr/local/lib/pkgconfig/ ./configure



make



make install



ldconfig



The output of syslog-ng -V should have the following line:



Enable-SQL: on



Compiling on RHEL 6 should be very similar, most likely the version number of PostgreSQL will be something different. On the other hand, hopefully there will be a libdbi enabled syslog-ng package in EPEL soon.



Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

 Share

×
×
  • Create New...