growinglogfiles check problem

[anonymous]

growinglogfiles alerts changed checksums for logfiles that are just growing (but growing fast). to reproduce, run something close to this (wait a while until the file gets larger):

while true; do echo "kala" >> kala; echo "leib" >> kala; echo "prantsusmaa" >> kala; echo "odekolonn" >> kala; sync; done

define: [GrowingLogFiles] file=/root/kala (or wherever you made the file)

then run check or update for a couple of times while the /root/kala file is being appended to: ./samhain --foreground -p crit -l none -s none -t update ALERT : [2007-12-28T11:36:43-0500] msg=<START>, program=<Samhain>, userid=<0>, path=</etc/samhainrc>, hash=<24F0130C63AC933216166E76B1BB925FF373DE2D49584E7A>, path=</var/lib/samhain/samhain_file>, hash=<24F0130C63AC933216166E76B1BB925FF373DE2D49584E7A> CRIT : [2007-12-28T11:36:44-0500] msg=<POLICY MISSING>, path=</emul> CRIT : [2007-12-28T11:36:45-0500] msg=<POLICY [GrowingLogs] C--------->, path=</var/log/auth.log>, chksum_old=<97205EE75EC9E9E17900016C07F51FB1E8761E78CAC6119D>, chksum_new=<3AF00F4F023D890486533936B340E42E756E06552D14A75C>, CRIT : [2007-12-28T11:36:45-0500] msg=<POLICY [GrowingLogs] C--------->, path=</root/kala>, chksum_old=<A6430F4375CA03536F85A7177C4B272C84DA3DA7B107350F>, chksum_new=<50A573F0B81CBDB5A5DBEE0766669562565659EDDCE8BA69>, ALERT : [2007-12-28T11:36:46-0500] msg=<EXIT>, program=<Samhain>, status=<None>

[anonymous]

i browsed the source and i'm willing to speculate that the problem may arise from differences in filesize between measuring the file size and calculating the new checksum. the problem wouldn't arise for normal files as the checksum is calculated only once. but for growinglogfiles it is calculated twice (for checking and for updating) providing a big enough window where the file may change in the middle of the check.

[siim@…]

Can't attach files for some reason, posting patch here:

--- samhain-2.4.1a/src/sh_unix.c        2007-10-30 22:17:30.000000000 +0200
+++ samhain-2.4.1a-patched/src/sh_unix.c        2008-01-03 14:55:29.675140792 +0200
@@ -3372,7 +3372,7 @@
              char hashbuf[KEYBUF_SIZE];
              sl_strlcpy(fileHash,
                         sh_tiger_generic_hash (theFile->fullpath, 
-                                               rval_open, TIGER_NOLIM, 
+                                               rval_open, buf.st_size, 
                                                alert_timeout, 
                                                hashbuf, sizeof(hashbuf)),
                         KEY_LEN+1);
@@ -3412,7 +3412,7 @@
              char hashbuf[KEYBUF_SIZE];
              sl_strlcpy(fileHash, 
                         sh_tiger_generic_hash (theFile->fullpath, rval_open, 
-                                               TIGER_NOLIM,
+                                               buf.st_size,
                                                alert_timeout,
                                                hashbuf, sizeof(hashbuf)),
                         KEY_LEN + 1);

This patch generates the hash of the file up to the offset that is later saved into theFile->size.

I added some debugging output that gave me the size of of the file when the hash was generated. As my growinglogfile only contained 'a' characters, it was easy to verify that the hashes were those of slightly bigger files (by increasing the file until the checksum matched) than what was recorded in the database. Presumably the files are written into between lstat and checksuming (i was writing one char at a time and the the matching checksums were of the files up to 1000 bytes bigger).

Originally the problem arose on a busy logserver - the growinglogfiles (not rotated) were reported changing in checksums.

I tried (as an alternative approach) to save fbuf.st_size into theFile->size for regular files, but that did not help. I tried investigating the checksuming code but it was overwhelmingly complicated (for me) and i could not verify if the files were somehow opened separately for checksuming nor find any other clue why the files would grow between the lstat and checksuming.

[siim@…]

I think I figured it out: if the file is appended to and the inode stays the same, the checksum code reading the file will read all the file (without requiring a reopen). So the only other option than this simple patch would be to modify checksuming code to return the size of the file as it was when the checksum was generated and save that value into theFile->size (and later to the database).

If you find that changing the checksuming code would be wiser you would have to do it yourself though, as I don't feel confident looking at those sh_tiger* files :)

[rainer]

Samhain does not use st_size for a reason: there are special files (e.g. in the Linux /proc filesystem), for which stat() incorrectly reports zero length. Will change checksumming code to return checksummed file size.

[rainer]

Fixed in changeset [151]. Checksum functions modified to return length of file hashed.