1 ULOGD - the Userspace Logging Daemon
2 Harald Welte <laforge@gnumonks.org>
3 Revision $Revision: 5846 $, $Date: 2005-07-12 17:33:23 +0200
6 This is the documentation for ulogd, the Userspace logging daemon.
7 ulogd makes use of the Linux >= 2.4.x packet filter subsystem (ipta-
8 bles) and the ULOG target for iptables.
9 ______________________________________________________________________
20 2.2 ipt_ULOG from netfilter/iptables patch-o-matic
22 2.3.1 Recompiling the source
23 2.3.2 Using a precompiled package
26 3.1 iptables ULOG target
28 3.1.2 ULOG target reference
29 3.1.3 ipt_ULOG module parameters
31 3.2.1 ulogd configfile syntax reference
32 3.2.2 ulogd commandline option reference
35 4.1 Interpreter plugins
37 4.1.2 ulogd_PWSNIFF.so
45 4.2.6 ulogd_SQLITE3.so
48 5. QUESTIONS / COMMENTS
51 ______________________________________________________________________
55 \e[1m1.1. CONCEPT
\e[0m
57 I want to provide a flexible, almost universal logging daemon for my
58 netfilter ULOG target. It is not optimized in any way, the goal is to
59 keep as simple as possible. These are my thoughts about how the
60 architecture which is most capable of doing that:
63 \e[1mInterpreter plugins
\e[0m
64 It should be possible to add plugins / runtime modules for new
65 protocols, etc. For example the standard logging daemon
66 provides source-ip, dest-ip, source-port, dest-port, etc.
67 Logging for various other protocols (GRE, IPsec, ...) may be
68 implemented as modules.
71 \e[1mOutput plugins
\e[0m
72 ... describe how and where to put the information gained by
73 logging plugins. The easiest way is to build a line per packet
74 and fprint it to a file. Some people might want to log into a
75 SQL database or want an output conforming to the intrusion
76 detection systems communication draft from the IETF.
80 \e[1m1.2. DETAILS
\e[0m
82 The major clue is providing a framework which is as flexible as
83 possible. Nobody knows what strange network protocols are out there
84 :) Flexibility depends on the communication between the output of the
85 logging plugins and input of the output plugins.
87 Rusty advised me to use some kind of type-key-value triples, which is
88 in fact what I implemented.
90 One issue is, of course, performance. Up to ulogd 0.3, ulogd did
91 several linked list iterations and about 30 malloc() calls _per
92 packet_. This changed with the new >= 0.9 revisions:
94 o Not a single dynamic allocation in the core during runtime.
95 Everything is pre-allocated at start of ulogd to provide the
96 highest possible throughput.
98 o Hash tables in addition to the linked lists. Linked lists are only
99 traversed if we really want to access each element of the list.
102 \e[1m2. INSTALLATION
\e[0m
105 \e[1m2.1. Linux kernel
\e[0m
107 First you will need a recent 2.4.x kernel. If you have a kernel >=
108 2.4.18-pre8, it already has the kernel support for ULOG (ipt_ULOG.o).
110 If you have an older kernel version (between 2.4.0 and 2.4.18-pre6),
111 you can use the patch-o-matic system of netfilter/iptables, as
112 described in the following section.
115 \e[1m2.2. ipt_ULOG from netfilter/iptables patch-o-matic
\e[0m
117 You only need to read this chapter if you have a 2.4.x kernel <=
120 In order to put the ipt_ULOG module into your kernel source,you need
121 the latest iptables package, or even better: the latest CVS snapshot.
122 A description how to obtain this is provided on the netfilter homepage
123 <http://www.netfilter.org/>.
125 To run patch-o-matic, just type
132 in the userspace directory of netfilter CVS.
137 \e[1m2.3.1. Recompiling the source
\e[0m
139 Download the ulogd package from <http://ftp.netfilter.org/pub/ulogd/>
142 If you want to build ulogd with MySQL support, type './configure
143 --with-mysql'. You may also have to specify the path of the mysql
144 libraries using '--with-mysql=path'. To build ulogd without MySQL
145 support, just use './configure'.
147 To compile and install the program, call 'make install'.
150 \e[1m2.3.2. Using a precompiled package
\e[0m
152 I also provide a SRPM, which should compile on almost any rpm-based
153 distribution. It is available at
154 <http://ftp.netfilter.org/pub/ulogd/>
156 Just download the package and do the usual 'rpm --rebuild <file>'.
159 \e[1m3. Configuration
\e[0m
161 \e[1m3.1. iptables ULOG target
\e[0m
163 \e[1m3.1.1. Quick Setup
\e[0m
165 Just add rules using the ULOG target to your firewalling chain. A very
169 iptables -A FORWARD -j ULOG --ulog-nlgroup 32 --ulog-prefix foo
173 To increase logging performance, try to use the
180 option (where 1 < N <= 50). The number you specify is the amount of
181 packets batched together in one multipart netlink message. If you set
182 this to 20, the kernel schedules ulogd only once every 20 packets. All
183 20 packets are then processed by ulogd. This reduces the number of
184 context switches between kernel and userspace.
186 Of course you can combine the ULOG target with the different netfilter
187 match modules. For a more detailed description, have a look at the
188 netfilter HOWTO's, available on the netfilter homepage.
190 \e[1m3.1.2. ULOG target reference
\e[0m
193 \e[1m--ulog-nlgroup N
\e[0m
194 The number of the netlink multicast group to which ULOG'ed
195 packets are sent. You will have to use the same group number in
196 the ULOG target and ulogd in order to make logging work.
198 \e[1m--ulog-cprange N
\e[0m
199 Copyrange. This works like the 'snaplen' parameter of tcpdump.
200 You can specify a number of bytes up to which the packet is
201 copied. If you say '40', you will receive the first fourty
202 bytes of every packet. Leave it to 0
204 \e[1m--ulog-qthreshold N
\e[0m
205 Queue threshold. If a packet is matched by the iptables rule,
206 and already N packets are in the queue, the queue is flushed to
207 userspace. You can use this to implement a policy like: Use a
208 big queue in order to gain high performance, but still have
209 certain packets logged immediately to userspace.
211 \e[1m--ulog-prefix STRING
\e[0m
212 A string that is associated with every packet logged by this
213 rule. You can use this option to later tell from which rule the
217 \e[1m3.1.3. ipt_ULOG module parameters
\e[0m
219 The ipt_ULOG kernel module has a couple of module loadtime parameters
220 which can (and should) be tuned to accomodate the needs of the
224 Netlink buffer size. A buffer of the specified size N is
225 allocated for every netlink group that is used. Please note
226 that due to restrictions of the kernel memory allocator, we
227 cannot have a buffer size > 128kBytes. Larger buffer sizes
228 increase the performance, since less kernel/userspace context
229 switches are needed for the same amount of packets. The
230 backside of this performance gain is a potentially larger delay.
231 The default value is 4096 bytes, which is quite small.
233 \e[1mflushtimeout N
\e[0m
234 The flushtimeout determines, after how many clock ticks (on
235 alpha: 1ms, on x86 and most other platforms: 10ms time units)
236 the buffer/queue is to be flushed, even if it is not full. This
237 can be used to have the advantage of a large buffer, but still a
238 finite maximum delay introduced. The default value is set to 10
244 modprobe ipt_ULOG nlbufsiz=65535 flushtimeout=100
248 This would use a buffer size of 64k and a flushtimeout of 100 clock-
249 ticks (1 second on x86).
254 ulogd is what this is all about, so let's describe it's
257 \e[1m3.2.1. ulogd configfile syntax reference
\e[0m
259 All configurable parameters of ulogd are in the configfile, typically
260 located at '/etc/ulogd.conf'.
261 The following configuration parameters are available:
264 The netlink multicast group, which ulgogd should bind to. This
265 is the same as given with the '--ulog-nlgroup' option to
269 The main logfile, where ulogd reports any errors, warnings and
270 other unexpected conditions. Apart from a regular filename, the
271 following special values can be used; ``syslog'' to log via the
272 unix syslog(3) mechanism. ``stdout'' to log to stdout.
275 This specifies, how verbose the logging to logfile is. Currently
276 defined loglevels are: 1=debug information, 3=informational
277 messages, 5=noticable exceptional conditions, 7=error
278 conditions, 8=fatal errors, program abort.
281 This option is followed by a filename of a ulogd plugin, which
282 ulogd shold load upon initialization. This option may appear
286 Size of the netlink socket receive memory. You should set this
287 to at least the size of the kernel buffer (nlbufsiz parameter of
288 the ipt_ULOG module). Please note that there is a maximum limit
289 in /proc/sys/net/core/rmem_max which you cannot exceed by
290 increasing the ``rmem'' parameter. You may need to raise the
291 system-wide maximum limit before.
294 Size of the receive buffer. You should set this to at least the
295 socket receive buffer (rmem).
297 \e[1m3.2.2. ulogd commandline option reference
\e[0m
299 Apart from the configfile, there are a couple of commandline options
303 Print a help message about the commandline options.
305 \e[1m-V --version
\e[0m
306 Print version information about ulogd.
308 \e[1m-d --daemon
\e[0m
309 For off into daemon mode. Unless you are debugging, you will
310 want to use this most of the time.
312 \e[1m-c --configfile
\e[0m
313 Using this commandline option, an alternate config file can be
314 used. This is important if multiple instances of ulogd are to
315 be run on a single machine.
318 \e[1m4. Available plugins
\e[0m
320 It is important to understand that ulogd without plugins does nothing.
321 It will receive packets, and do nothing with them.
323 There are two kinds of plugins, interpreter and output plugins.
324 Interpreter plugins parse the packet, output plugins write the
325 interpreted information to some logfile/database/...
327 \e[1m4.1. Interpreter plugins
\e[0m
329 ulogd comes with the following interpreter plugins:
331 \e[1m4.1.1. ulogd_BASE.so
\e[0m
333 Basic interpreter plugin for nfmark, timestamp, mac address, ip
334 header, tcp header, udp header, icmp header, ah/esp header... Most
335 people will want to load this very important plugin.
337 \e[1m4.1.2. ulogd_PWSNIFF.so
\e[0m
339 Example interpreter plugin to log plaintext passwords as used with FTP
340 and POP3. Don't blame me for writing this plugin! The protocols are
341 inherently insecure, and there are a lot of other tools for sniffing
342 passwords... it's just an example.
344 \e[1m4.1.3. ulogd_LOCAL.so
\e[0m
346 This is a 'virtual interpreter'. It doesn't really return any
347 information on the packet itself, rather the local system time and
348 hostname. Please note that the time is the time at the time of
349 logging, not the packets receive time.
352 \e[1m4.2. Output plugins
\e[0m
354 ulogd comes with the following output plugins:
357 \e[1m4.2.1. ulogd_OPRINT.so
\e[0m
359 A very simple output module, dumping all packets in the format
371 to a file. The only useful application is debugging.
373 The module defines the following configuration directives:
376 The filename where it should log to. The default is
377 /var/log/ulogd.pktlog
380 \e[1m4.2.2. ulogd_LOGEMU.so
\e[0m
382 An output module which tries to emulate the old syslog-based LOG
383 targed as far as possible. Logging is done to a seperate textfile
384 instead of syslog, though.
386 The module defines the following configuration directives:
389 The filename where it should log to. The default is
390 /var/log/ulogd.syslogemu
393 Set this to 1 if you want to have your logfile written
394 synchronously. This may reduce performance, but makes your log-
395 lines appear immediately. The default is 0
398 \e[1m4.2.3. ulogd_MYSQL.so
\e[0m
400 An output plugin for logging into a mysql database. This is only
401 compiled if you have the mysql libraries installed, and the configure
402 script was able to detect them. (that is: --with-mysql was specified
406 The plugin automagically inserts the data into the configured table;
407 It connects to mysql during the startup phase of ulogd and obtains a
408 list of the columns in the table. Then it tries to resolve the column
409 names against keys of interpreter plugins. This way you can easily
410 select which information you want to log - just by the layout of the
414 If, for example, your table contains a field called 'ip_saddr', ulogd
415 will resolve this against the key 'ip.saddr' and put the ip address as
416 32bit unsigned integer into the table.
419 You may want to have a look at the file 'doc/mysql.table' as an
420 example table including fields to log all keys from ulogd_BASE.so.
421 Just delete the fields you are not interested in, and create the
425 The module defines the following configuration directives:
428 Name of the table to which ulogd should log.
431 Name of the mysql database.
434 Name of the mysql database host.
437 TCP port number of mysql database server.
440 Name of the mysql user.
446 \e[1m4.2.4. ulogd_PGSQL.so
\e[0m
448 An output plugin for logging into a postgresql database. This is only
449 compiled if you have the mysql libraries installed, and the configure
450 script was able to detect them. (that is: --with-pgsql was specified
454 The plugin automagically inserts the data into the configured table;
455 It connects to pgsql during the startup phase of ulogd and obtains a
456 list of the columns in the table. Then it tries to resolve the column
457 names against keys of interpreter plugins. This way you can easily
458 select which information you want to log - just by the layout of the
462 If, for example, your table contains a field called 'ip_saddr', ulogd
463 will resolve this against the key 'ip.saddr' and put the ip address as
464 32bit unsigned integer into the table.
467 You may want to have a look at the file 'doc/mysql.table' as an
468 example table including fields to log all keys from ulogd_BASE.so.
469 Just delete the fields you are not interested in, and create the
473 The module defines the following configuration directives:
476 Name of the table to which ulogd should log.
478 \e[1mdb
\e[22mName of the database.
481 Name of the mysql database host.
484 TCP port number of database server.
487 Name of the sql user.
490 Password for sql user.
493 \e[1m4.2.5. ulogd_PCAP.so
\e[0m
495 An output plugin that can be used to generate libpcap-style packet
496 logfiles. This can be useful for later analysing the packet log with
497 tools like tcpdump or ethereal.
499 The module defines the following configuration directives:
502 The filename where it should log to. The default is:
506 Set this to 1 if you want to have your pcap logfile written
507 synchronously. This may reduce performance, but makes your
508 packets appear immediately in the file on disk. The default is
512 \e[1m4.2.6. ulogd_SQLITE3.so
\e[0m
514 An output plugin for logging into a SQLITE v3 database. This is only
515 compiled if you have the sqlite libraries installed, and the configure
516 script was able to detect them. (that is: --with-sqlite3 was specified
520 The plugin automagically inserts the data into the configured table;
521 It opens the sqlite db during the startup phase of ulogd and obtains a
522 list of the columns in the table. Then it tries to resolve the column
523 names against keys of interpreter plugins. This way you can easily
524 select which information you want to log - just by the layout of the
528 If, for example, your table contains a field called 'ip_saddr', ulogd
529 will resolve this against the key 'ip.saddr' and put the ip address as
530 32bit unsigned integer into the table.
533 You may want to have a look at the file 'doc/sqlite3.table' as an
534 example table including fields to log all keys from ulogd_BASE.so.
535 Just delete the fields you are not interested in, and create the
539 The module defines the following configuration directives:
542 Name of the table to which ulogd should log.
544 \e[1mdb
\e[22mName of the database.
547 Size of the sqlite buffer.
549 \e[1m4.2.7. ulogd_SYSLOG.so
\e[0m
551 An output plugin that really logs via syslogd. Lines will look exactly
552 like printed with traditional LOG target.
555 The module defines the following configuration directives:
558 The syslog facility (LOG_DAEMON, LOG_KERN, LOG_LOCAL0 ..
559 LOG_LOCAL7, LOG_USER)
562 The syslog level (LOG_EMERG, LOG_ALERT, LOG_CRIT, LOG_ERR,
563 LOG_WARNING, LOG_NOTICE, LOG_INFO, LOG_DEBUG)
565 \e[1m5. QUESTIONS / COMMENTS
\e[0m
567 All comments / questions / ... are appreciated.
569 Just drop me a note to laforge@gnumonks.org
571 Please note also that there is now a mailinglist,
572 ulogd@lists.gnumonks.org. You can subscribe at
573 <http://lists.gnumonks.org/mailman/listinfo/ulogd/>.
576 The preferred method for reporting bugs is the netfilter bugzilla
577 system, available at <http://bugzilla.netfilter.org/>.