1 <!doctype linuxdoc system>
3 <!-- $Id: ulogd.sgml 5846 2005-07-12 15:33:23Z laforge $ -->
7 <title>ULOGD - the Userspace Logging Daemon</title>
8 <author>Harald Welte <laforge@gnumonks.org></author>
9 <date>Revision $Revision: 5846 $, $Date: 2005-07-12 17:33:23 +0200 (Tue, 12 Jul 2005) $</date>
12 This is the documentation for <tt>ulogd</tt>, the Userspace logging daemon.
13 ulogd makes use of the Linux >= 2.4.x packet filter subsystem (iptables) and
14 the ULOG target for iptables.
23 I want to provide a flexible, almost universal logging daemon for my netfilter
24 ULOG target. It is not optimized in any way, the goal is to keep as simple as
25 possible. These are my thoughts about how the architecture which is most
26 capable of doing that:
29 <tag>Interpreter plugins</tag>
30 It should be possible to add plugins / runtime modules for new protocols, etc.
31 For example the standard logging daemon provides source-ip, dest-ip,
32 source-port, dest-port, etc. Logging for various other protocols (GRE,
33 IPsec, ...) may be implemented as modules.
35 <tag>Output plugins</tag>
36 ... describe how and where to put the information gained by logging plugins.
37 The easiest way is to build a line per packet and fprint it to a file.
38 Some people might want to log into a SQL database or want an output
39 conforming to the intrusion detection systems communication draft from the
46 The major clue is providing a framework which is as flexible as possible.
47 Nobody knows what strange network protocols are out there :) Flexibility
48 depends on the communication between the output of the logging plugins
49 and input of the output plugins.
51 Rusty advised me to use some kind of type-key-value triples, which is in fact
54 One issue is, of course, performance. Up to ulogd 0.3, ulogd did several
55 linked list iterations and about 30 malloc() calls _per packet_. This
56 changed with the new >= 0.9 revisions:
58 <item>Not a single dynamic allocation in the core during runtime.
59 Everything is pre-allocated at start of ulogd to provide the highest
61 <item>Hash tables in addition to the linked lists. Linked lists are only
62 traversed if we really want to access each element of the list.
69 First you will need a recent 2.4.x kernel. If you have a kernel >=
70 2.4.18-pre8, it already has the kernel support for ULOG (ipt_ULOG.o).
72 If you have an older kernel version (between 2.4.0 and 2.4.18-pre6), you
73 can use the patch-o-matic system of netfilter/iptables, as described in
74 the following section.
76 <sect1>ipt_ULOG from netfilter/iptables patch-o-matic
78 You only need to read this chapter if you have a 2.4.x kernel <=
81 In order to put the ipt_ULOG module into your kernel source,you need the latest
82 iptables package, or even better: the latest CVS snapshot. A description how to
83 obtain this is provided on the netfilter
84 homepage <URL URL="http://www.netfilter.org/">.
86 To run patch-o-matic, just type
90 in the userspace directory of netfilter CVS.
93 <sect2>Recompiling the source
95 Download the ulogd package from <URL URL="http://ftp.netfilter.org/pub/ulogd/"> and
98 If you want to build ulogd with MySQL support, type './configure --with-mysql'. You may also have to specify the path of the mysql libraries using '--with-mysql=path'. To build ulogd without MySQL support, just use './configure'.
100 To compile and install the program, call 'make install'.
102 <sect2>Using a precompiled package
104 I also provide a SRPM, which should compile on almost any rpm-based distribution. It is available at <URL URL="http://ftp.netfilter.org/pub/ulogd/">
106 Just download the package and do the usual 'rpm --rebuild <file>'.
109 <sect1>iptables ULOG target
112 Just add rules using the ULOG target to your firewalling chain. A very basic
115 iptables -A FORWARD -j ULOG --ulog-nlgroup 32 --ulog-prefix foo
118 To increase logging performance, try to use the
122 option (where 1 < N <= 50). The number you specify is the amount of packets
123 batched together in one multipart netlink message. If you set this to 20, the
124 kernel schedules ulogd only once every 20 packets. All 20 packets are then
125 processed by ulogd. This reduces the number of context switches between kernel
128 Of course you can combine the ULOG target with the different netfilter match
129 modules. For a more detailed description, have a look at the netfilter
130 HOWTO's, available on the netfilter homepage.
131 <sect2>ULOG target reference
134 <tag>--ulog-nlgroup N</tag>
135 The number of the netlink multicast group to which ULOG'ed packets are sent.
136 You will have to use the same group number in the ULOG target and ulogd in
137 order to make logging work.
138 <tag>--ulog-cprange N</tag>
139 Copyrange. This works like the 'snaplen' parameter of tcpdump. You can specify
140 a number of bytes up to which the packet is copied. If you say '40', you will
141 receive the first fourty bytes of every packet. Leave it to <tt>0</tt>
142 <tag>--ulog-qthreshold N</tag>
143 Queue threshold. If a packet is matched by the iptables rule, and already N
144 packets are in the queue, the queue is flushed to userspace. You can use this
145 to implement a policy like: Use a big queue in order to gain high performance,
146 but still have certain packets logged immediately to userspace.
147 <tag>--ulog-prefix STRING</tag>
148 A string that is associated with every packet logged by this rule. You can use
149 this option to later tell from which rule the packet was logged.
152 <sect2>ipt_ULOG module parameters
154 The ipt_ULOG kernel module has a couple of module loadtime parameters which can
155 (and should) be tuned to accomodate the needs of the application:
157 <tag>nlbufsiz N</tag>
158 Netlink buffer size. A buffer of the specified size N is allocated for every
159 netlink group that is used. Please note that due to restrictions of the kernel
160 memory allocator, we cannot have a buffer size > 128kBytes. Larger buffer
161 sizes increase the performance, since less kernel/userspace context switches
162 are needed for the same amount of packets. The backside of this performance
163 gain is a potentially larger delay. The default value is 4096 bytes, which is
165 <tag>flushtimeout N</tag>
166 The flushtimeout determines, after how many clock ticks (on alpha: 1ms, on
167 x86 and most other platforms: 10ms time units) the buffer/queue is to be
168 flushed, even if it is not full. This can be used to have the advantage of a
169 large buffer, but still a finite maximum delay introduced. The default value
170 is set to 10 seconds.
174 modprobe ipt_ULOG nlbufsiz=65535 flushtimeout=100
176 This would use a buffer size of 64k and a flushtimeout of 100 clockticks (1 second on x86).
180 ulogd is what this is all about, so let's describe it's configuration...
181 <sect2>ulogd configfile syntax reference
183 All configurable parameters of ulogd are in the configfile, typically located
184 at '/etc/ulogd.conf'.
186 The following configuration parameters are available:
189 The netlink multicast group, which ulgogd should bind to. This is the same as
190 given with the '--ulog-nlgroup' option to iptables.
192 The main logfile, where ulogd reports any errors, warnings and other unexpected conditions. Apart from a regular filename, the following special values can be used; ``syslog'' to log via the unix syslog(3) mechanism. ``stdout'' to log to stdout.
194 This specifies, how verbose the logging to logfile is. Currently defined
195 loglevels are: 1=debug information, 3=informational messages, 5=noticable
196 exceptional conditions, 7=error conditions, 8=fatal errors, program abort.
198 This option is followed by a filename of a ulogd plugin, which ulogd shold load
199 upon initialization. This option may appear more than once.
201 Size of the netlink socket receive memory. You should set this to at least the
202 size of the kernel buffer (nlbufsiz parameter of the ipt_ULOG module). Please
203 note that there is a maximum limit in /proc/sys/net/core/rmem_max which you
204 cannot exceed by increasing the ``rmem'' parameter. You may need to raise the
205 system-wide maximum limit before.
207 Size of the receive buffer. You should set this to at least the socket receive buffer (rmem).
209 <sect2>ulogd commandline option reference
211 Apart from the configfile, there are a couple of commandline options to ulogd:
214 Print a help message about the commandline options.
215 <tag>-V --version</tag>
216 Print version information about ulogd.
217 <tag>-d --daemon</tag>
218 For off into daemon mode. Unless you are debugging, you will want to use this
220 <tag>-c --configfile</tag>
221 Using this commandline option, an alternate config file can be used. This is
222 important if multiple instances of ulogd are to be run on a single machine.
225 <sect>Available plugins
227 It is important to understand that ulogd without plugins does nothing. It will receive packets, and do nothing with them.
229 There are two kinds of plugins, interpreter and output plugins. Interpreter
230 plugins parse the packet, output plugins write the interpreted information to
231 some logfile/database/...
233 <sect1>Interpreter plugins
235 ulogd comes with the following interpreter plugins:
238 Basic interpreter plugin for nfmark, timestamp, mac address, ip header, tcp
239 header, udp header, icmp header, ah/esp header... Most people will want to load
240 this very important plugin.
241 <sect2>ulogd_PWSNIFF.so
243 Example interpreter plugin to log plaintext passwords as used with FTP and
244 POP3. Don't blame me for writing this plugin! The protocols are inherently
245 insecure, and there are a lot of other tools for sniffing passwords... it's
247 <sect2>ulogd_LOCAL.so
249 This is a 'virtual interpreter'. It doesn't really return any information on
250 the packet itself, rather the local system time and hostname. Please note that
251 the time is the time at the time of logging, not the packets receive time.
253 <sect1>Output plugins
255 ulogd comes with the following output plugins:
257 <sect2>ulogd_OPRINT.so
259 A very simple output module, dumping all packets in the format
268 to a file. The only useful application is debugging.
269 <p>The module defines the following configuration directives:
272 The filename where it should log to. The default is
273 <tt>/var/log/ulogd.pktlog</tt>
276 <sect2>ulogd_LOGEMU.so
278 An output module which tries to emulate the old syslog-based LOG targed as far
279 as possible. Logging is done to a seperate textfile instead of syslog, though.
281 The module defines the following configuration directives:
283 <tag>file</tag>The filename where it should log to. The default is
284 <tt>/var/log/ulogd.syslogemu</tt>
285 <tag>sync</tag>Set this to 1 if you want to have your logfile written
286 synchronously. This may reduce performance, but makes your log-lines appear
287 immediately. The default is <tt>0</tt>
290 <sect2>ulogd_MYSQL.so
292 An output plugin for logging into a mysql database. This is only compiled if
293 you have the mysql libraries installed, and the configure script was able to
294 detect them. (that is: --with-mysql was specified for ./configure)
297 The plugin automagically inserts the data into the configured table; It
298 connects to mysql during the startup phase of ulogd and obtains a list of the
299 columns in the table. Then it tries to resolve the column names against keys of
300 interpreter plugins. This way you can easily select which information you want
301 to log - just by the layout of the table.
304 If, for example, your table contains a field called 'ip_saddr', ulogd will
305 resolve this against the key 'ip.saddr' and put the ip address as 32bit
306 unsigned integer into the table.
309 You may want to have a look at the file '<tt>doc/mysql.table</tt>' as an
310 example table including fields to log all keys from ulogd_BASE.so. Just delete
311 the fields you are not interested in, and create the table.
314 The module defines the following configuration directives:
317 Name of the table to which ulogd should log.
319 Name of the mysql database.
321 Name of the mysql database host.
323 TCP port number of mysql database server.
325 Name of the mysql user.
330 <sect2>ulogd_PGSQL.so
332 An output plugin for logging into a postgresql database. This is only compiled
333 if you have the mysql libraries installed, and the configure script was able to
334 detect them. (that is: --with-pgsql was specified for ./configure)
337 The plugin automagically inserts the data into the configured table; It
338 connects to pgsql during the startup phase of ulogd and obtains a list of the
339 columns in the table. Then it tries to resolve the column names against keys of
340 interpreter plugins. This way you can easily select which information you want
341 to log - just by the layout of the table.
344 If, for example, your table contains a field called 'ip_saddr', ulogd will
345 resolve this against the key 'ip.saddr' and put the ip address as 32bit
346 unsigned integer into the table.
349 You may want to have a look at the file '<tt>doc/mysql.table</tt>' as an
350 example table including fields to log all keys from ulogd_BASE.so. Just delete
351 the fields you are not interested in, and create the table.
354 The module defines the following configuration directives:
357 Name of the table to which ulogd should log.
359 Name of the database.
361 Name of the mysql database host.
363 TCP port number of database server.
365 Name of the sql user.
367 Password for sql user.
372 An output plugin that can be used to generate libpcap-style packet logfiles.
373 This can be useful for later analysing the packet log with tools like tcpdump
376 The module defines the following configuration directives:
379 The filename where it should log to. The default is:
380 <tt>/var/log/ulogd.pcap</tt>
382 Set this to <tt>1</tt> if you want to have your pcap logfile written
383 synchronously. This may reduce performance, but makes your packets appear
384 immediately in the file on disk. The default is <tt>0</tt>
387 <sect2>ulogd_SQLITE3.so
389 An output plugin for logging into a SQLITE v3 database. This is only compiled
390 if you have the sqlite libraries installed, and the configure script was able to
391 detect them. (that is: --with-sqlite3 was specified for ./configure)
394 The plugin automagically inserts the data into the configured table; It
395 opens the sqlite db during the startup phase of ulogd and obtains a list of the
396 columns in the table. Then it tries to resolve the column names against keys of
397 interpreter plugins. This way you can easily select which information you want
398 to log - just by the layout of the table.
401 If, for example, your table contains a field called 'ip_saddr', ulogd will
402 resolve this against the key 'ip.saddr' and put the ip address as 32bit
403 unsigned integer into the table.
406 You may want to have a look at the file '<tt>doc/sqlite3.table</tt>' as an
407 example table including fields to log all keys from ulogd_BASE.so. Just delete
408 the fields you are not interested in, and create the table.
411 The module defines the following configuration directives:
414 Name of the table to which ulogd should log.
416 Name of the database.
418 Size of the sqlite buffer.
422 <sect2>ulogd_SYSLOG.so
424 An output plugin that really logs via syslogd. Lines will look exactly like printed with traditional LOG target.
427 The module defines the following configuration directives:
430 The syslog facility (LOG_DAEMON, LOG_KERN, LOG_LOCAL0 .. LOG_LOCAL7, LOG_USER)
432 The syslog level (LOG_EMERG, LOG_ALERT, LOG_CRIT, LOG_ERR, LOG_WARNING, LOG_NOTICE, LOG_INFO, LOG_DEBUG)
436 <sect> QUESTIONS / COMMENTS
438 All comments / questions / ... are appreciated.
440 Just drop me a note to laforge@gnumonks.org
442 Please note also that there is now a mailinglist, ulogd@lists.gnumonks.org.
444 <URL URL="http://lists.gnumonks.org/mailman/listinfo/ulogd/">.
446 The preferred method for reporting bugs is the netfilter bugzilla system,
447 available at <URL URL="http://bugzilla.netfilter.org/">.