| 1 | <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
|
|---|
| 2 | <html>
|
|---|
| 3 | <head>
|
|---|
| 4 | <title>HOWTO client+server troubleshooting</title>
|
|---|
| 5 | <style type="text/css">
|
|---|
| 6 | <!--
|
|---|
| 7 |
|
|---|
| 8 | html { background: #eee; color: #000; }
|
|---|
| 9 |
|
|---|
| 10 | body { background: #eee; color: #000; margin: 0; padding: 0;}
|
|---|
| 11 |
|
|---|
| 12 | div.body {
|
|---|
| 13 | background: #fff; color: #000;
|
|---|
| 14 | margin: 0 1em 0 1em; padding: 1em;
|
|---|
| 15 | font-family: serif;
|
|---|
| 16 | font-size: 1em; line-height: 1.2em;
|
|---|
| 17 | border-width: 0 1px 0 1px;
|
|---|
| 18 | border-style: solid;
|
|---|
| 19 | border-color: #aaa;
|
|---|
| 20 | }
|
|---|
| 21 |
|
|---|
| 22 | div.block {
|
|---|
| 23 | background: #b6c5f2; color: #000;
|
|---|
| 24 | margin: 1em; padding: 0 1em 0 1em;
|
|---|
| 25 | border-width: 1px;
|
|---|
| 26 | border-style: solid;
|
|---|
| 27 | border-color: #2d4488;
|
|---|
| 28 | }
|
|---|
| 29 |
|
|---|
| 30 | div.warnblock {
|
|---|
| 31 | background: #b6c5f2; color: #000;
|
|---|
| 32 | background: #ffffcc; color: #000;
|
|---|
| 33 | margin: 1em; padding: 0 1em 0 1em;
|
|---|
| 34 | border-width: 1px;
|
|---|
| 35 | border-style: solid;
|
|---|
| 36 | border-color: #FF9900;
|
|---|
| 37 | }
|
|---|
| 38 |
|
|---|
| 39 | table {
|
|---|
| 40 | background: #F8F8F8; color: #000;
|
|---|
| 41 | margin: 1em;
|
|---|
| 42 | border-width: 0 0 0 1px;
|
|---|
| 43 | border-style: solid;
|
|---|
| 44 | border-color: #C0C0C0;
|
|---|
| 45 | }
|
|---|
| 46 |
|
|---|
| 47 | td {
|
|---|
| 48 | border-width: 0 1px 1px 0;
|
|---|
| 49 | border-style: solid;
|
|---|
| 50 | border-color: #C0C0C0;
|
|---|
| 51 | }
|
|---|
| 52 |
|
|---|
| 53 | th {
|
|---|
| 54 | background: #F8F8FF;
|
|---|
| 55 | border-width: 1px 1px 2px 0;
|
|---|
| 56 | border-style: solid;
|
|---|
| 57 | border-color: #C0C0C0;
|
|---|
| 58 | }
|
|---|
| 59 |
|
|---|
| 60 |
|
|---|
| 61 | /* body text, headings, and rules */
|
|---|
| 62 |
|
|---|
| 63 | p { margin: 0; text-indent: 0em; margin: 0 0 0.5em 0 }
|
|---|
| 64 |
|
|---|
| 65 | h1, h2, h3, h4, h5, h6 {
|
|---|
| 66 | color: #206020; background: transparent;
|
|---|
| 67 | font-family: Optima, Arial, Helvetica, sans-serif;
|
|---|
| 68 | font-weight: normal;
|
|---|
| 69 | }
|
|---|
| 70 |
|
|---|
| 71 | h1 { font-size: 1.69em; margin: 1.4em 0 0.4em 0; }
|
|---|
| 72 | h2 { font-size: 1.44em; margin: 1.4em 0 0.4em 0; }
|
|---|
| 73 | h3 { font-size: 1.21em; margin: 1.4em 0 0.4em 0; }
|
|---|
| 74 | h4 { font-size: 1.00em; margin: 1.4em 0 0.4em 0; }
|
|---|
| 75 | h5 { font-size: 0.81em; margin: 1.4em 0 0.4em 0; }
|
|---|
| 76 | h6 { font-size: 0.64em; margin: 1.4em 0 0.4em 0; }
|
|---|
| 77 |
|
|---|
| 78 | hr {
|
|---|
| 79 | color: transparent; background: transparent;
|
|---|
| 80 | height: 0px; margin: 0.6em 0;
|
|---|
| 81 | border-width: 1px ;
|
|---|
| 82 | border-style: solid;
|
|---|
| 83 | border-color: #999;
|
|---|
| 84 | }
|
|---|
| 85 |
|
|---|
| 86 | /* bulleted lists and definition lists */
|
|---|
| 87 |
|
|---|
| 88 | ul { margin: 0 1em 0.6em 2em; padding: 0; }
|
|---|
| 89 | li { margin: 0.4em 0 0 0; }
|
|---|
| 90 |
|
|---|
| 91 | dl { margin: 0.6em 1em 0.6em 2em; }
|
|---|
| 92 | dt { color: #285577; }
|
|---|
| 93 |
|
|---|
| 94 | tt { color: #602020; }
|
|---|
| 95 |
|
|---|
| 96 | /* links */
|
|---|
| 97 |
|
|---|
| 98 | a.link {
|
|---|
| 99 | color: #33c; background: transparent;
|
|---|
| 100 | text-decoration: none;
|
|---|
| 101 | }
|
|---|
| 102 |
|
|---|
| 103 | a:hover {
|
|---|
| 104 | color: #000; background: transparent;
|
|---|
| 105 | }
|
|---|
| 106 |
|
|---|
| 107 | body > a {
|
|---|
| 108 | font-family: Optima, Arial, Helvetica, sans-serif;
|
|---|
| 109 | font-size: 0.81em;
|
|---|
| 110 | }
|
|---|
| 111 |
|
|---|
| 112 | h1, h2, h3, h4, h5, h6 {
|
|---|
| 113 | color: #2d5588; background: transparent;
|
|---|
| 114 | font-family: Optima, Arial, Helvetica, sans-serif;
|
|---|
| 115 | font-weight: normal;
|
|---|
| 116 | }
|
|---|
| 117 |
|
|---|
| 118 | -->
|
|---|
| 119 | </style></head>
|
|---|
| 120 |
|
|---|
| 121 | <body>
|
|---|
| 122 | <div class="body">
|
|---|
| 123 | <p style="text-align: center; background: #ccc; border: 1px solid #2d5588;"><a
|
|---|
| 124 | style="text-decoration: none;"
|
|---|
| 125 | href="http://www.la-samhna.de/samhain/">samhain file integrity
|
|---|
| 126 | scanner</a> | <a style="text-decoration: none;"
|
|---|
| 127 | href="http://www.la-samhna.de/samhain/s_documentation.html">online
|
|---|
| 128 | documentation</a></p>
|
|---|
| 129 | <br><center>
|
|---|
| 130 | <h1>Samhain client/server: What can go wrong, and how can you fix it ?</h1>
|
|---|
| 131 | </center>
|
|---|
| 132 | <br>
|
|---|
| 133 | <hr>
|
|---|
| 134 | <div class="warnblock">
|
|---|
| 135 | <ul>
|
|---|
| 136 | <li>Almost all problems can only be diagnosed correctly by checking the
|
|---|
| 137 | <b>server</b> logs</li>
|
|---|
| 138 | <li>If the server does not write logs, <b>fix this first</b>. For debugging,
|
|---|
| 139 | stop the server, then run it in the foreground with
|
|---|
| 140 | <tt>yule -p info --foreground</tt></li>
|
|---|
| 141 | </div>
|
|---|
| 142 | <p>
|
|---|
| 143 | This document aims to explain how to diagnose and fix common problems that
|
|---|
| 144 | may result from misunderstanding or misconfiguration when setting up
|
|---|
| 145 | a client/server samhain system. This document is divided in several sections
|
|---|
| 146 | more or less corresponding to the different stages when a client
|
|---|
| 147 | connects to a server. Each section starts with a brief explanation that
|
|---|
| 148 | should provide a basic understanding of what is going on.
|
|---|
| 149 | </p>
|
|---|
| 150 | <p>
|
|---|
| 151 | This document does not discuss <i>how</i> to setup a client/server (for
|
|---|
| 152 | this, look into the manual and/or the HOWTO-client+server).
|
|---|
| 153 | </p>
|
|---|
| 154 |
|
|---|
| 155 | <h2><a name="sect1">Table of Contents</a></h2>
|
|---|
| 156 | <p>
|
|---|
| 157 | <a href="#sect1">Connecting to the server</a><br>
|
|---|
| 158 | <a href="#sect2">Authentication</a><br>
|
|---|
| 159 | <a href="#sect3">Downloading config/database files</a><br>
|
|---|
| 160 | <a href="#sect4">Other connection problems</a><br>
|
|---|
| 161 | </p>
|
|---|
| 162 |
|
|---|
| 163 | <h2><a name="sect1">Connecting to the server</a></h2>
|
|---|
| 164 |
|
|---|
| 165 | <p>
|
|---|
| 166 | Client/server connections are always initiated from the client. The port
|
|---|
| 167 | is compiled in (there is a configure option to change the default).
|
|---|
| 168 | The default port is 49777.
|
|---|
| 169 | </p>
|
|---|
| 170 |
|
|---|
| 171 | <h3>Problem #1</h3>
|
|---|
| 172 | <p>
|
|---|
| 173 | The client reports: <b>Connection refused</b>. The server reports nothing.
|
|---|
| 174 | </p>
|
|---|
| 175 | <p>
|
|---|
| 176 | The server is down, listens on the wrong port, or network failure.
|
|---|
| 177 | </p>
|
|---|
| 178 |
|
|---|
| 179 | <h3>Problem #2</h3>
|
|---|
| 180 | <p>
|
|---|
| 181 | The client reports: <b>Connection error: Connection reset by peer</b>, and
|
|---|
| 182 | later also <b>Session key negotiation failed</b>. The server reports:
|
|---|
| 183 | <b>msg="Refused connection from ..." subroutine="libwrap"</b>.
|
|---|
| 184 | </p>
|
|---|
| 185 | <p>
|
|---|
| 186 | The server is compiled with libwrap (TCP Wrapper) support, and the
|
|---|
| 187 | client is either in <tt>/etc/hosts.deny</tt>, or you have set <i>yule: ALL</i>
|
|---|
| 188 | in <tt>/etc/hosts.deny</tt>, and forgot to put the client in
|
|---|
| 189 | <tt>/etc/hosts.allow</tt>.
|
|---|
| 190 | </p>
|
|---|
| 191 | <p>
|
|---|
| 192 | To fix: make proper entries in <tt>/etc/hosts.allow</tt> and/or
|
|---|
| 193 | <tt>/etc/hosts.deny</tt>. There is no need to restart/reload the server.
|
|---|
| 194 | </p>
|
|---|
| 195 |
|
|---|
| 196 |
|
|---|
| 197 | <h2><a name="sect2">Authentication</a></h2>
|
|---|
| 198 | <p>
|
|---|
| 199 | The client has a password that is used to authenticate to the server.
|
|---|
| 200 | This password is located within the binary, and is set with the
|
|---|
| 201 | <tt>samhain_setpwd</tt> helper application, as explained e.g. in the
|
|---|
| 202 | manual or in the Client+Server HOWTO.
|
|---|
| 203 | </p><p>
|
|---|
| 204 | The server has a list of clients that are allowed to connect, and the
|
|---|
| 205 | verifiers corresponding to the passwords of these clients.
|
|---|
| 206 | </p>
|
|---|
| 207 | <p>
|
|---|
| 208 | Upon successful authentication, client and server will negotiate
|
|---|
| 209 | a <b>session key</b> that is used for signing further messages
|
|---|
| 210 | from the client.
|
|---|
| 211 | </p>
|
|---|
| 212 |
|
|---|
| 213 | <h3>Problem #1</h3>
|
|---|
| 214 |
|
|---|
| 215 | <p>
|
|---|
| 216 | If the password is wrong, the client will report
|
|---|
| 217 | <b>Session key negotiation failed</b>. The server will
|
|---|
| 218 | report: <b>Invalid connection attempt: Session key mismatch</b>
|
|---|
| 219 | </p>
|
|---|
| 220 | <p>
|
|---|
| 221 | To fix: make sure that the password has in fact been set, that you are
|
|---|
| 222 | using the correct executable for the client (the one where the password is
|
|---|
| 223 | set), and that the entry in the server config file is the one generated
|
|---|
| 224 | for this password (also look out for double entries for this client).
|
|---|
| 225 | </p>
|
|---|
| 226 |
|
|---|
| 227 | <h3>Problem #2</h3>
|
|---|
| 228 |
|
|---|
| 229 | <p>
|
|---|
| 230 | If the client name (as resolved on the server) is wrong, the client
|
|---|
| 231 | will report
|
|---|
| 232 | <b>Session key negotiation failed</b>. The server will
|
|---|
| 233 | report: <b>Invalid connection attempt: Not in client list</b>,
|
|---|
| 234 | <i>and</i> it will tell you in the same error message
|
|---|
| 235 | what name it has inferred for the connecting
|
|---|
| 236 | client (example): <b>client="client.mydomain.com"</b>.
|
|---|
| 237 | </p>
|
|---|
| 238 | <p>
|
|---|
| 239 | The fix depends on the nature of the problem. In principle, it should be
|
|---|
| 240 | sufficient to change the name of the client in the config file entry, which
|
|---|
| 241 | isn't really a solution if e.g. the server thinks the client is 'localhost'.
|
|---|
| 242 | </p>
|
|---|
| 243 | <p>
|
|---|
| 244 | There are two different ways to determine the client name.
|
|---|
| 245 | Unfortunately, judging
|
|---|
| 246 | from customer feedback as well from common sense, both do not work very well
|
|---|
| 247 | with a messed up local DNS (including /etc/hosts files) and/or
|
|---|
| 248 | überparanoid or misconfigured firewalls (in case of connections
|
|---|
| 249 | across one).
|
|---|
| 250 | </p>
|
|---|
| 251 | <ul>
|
|---|
| 252 | <li>
|
|---|
| 253 | <p>
|
|---|
| 254 | <i>First method: Determine client name on client, and
|
|---|
| 255 | try to cross-check on server</i>
|
|---|
| 256 | <p>
|
|---|
| 257 | <p>
|
|---|
| 258 | This does not work for a number of people because (1) the
|
|---|
| 259 | <tt>/etc/hosts</tt> file on the client machine has errors
|
|---|
| 260 | (yes, there are plenty machines with a completely
|
|---|
| 261 | messed up <tt>/etc/hosts</tt> file), (2) the
|
|---|
| 262 | server cannot resolve the client address because the local DNS is
|
|---|
| 263 | f***ed up, or (3) the client machine has multiple network interfaces, and
|
|---|
| 264 | the interface used is not the one the client name resolves to.
|
|---|
| 265 | </p>
|
|---|
| 266 |
|
|---|
| 267 | <p>
|
|---|
| 268 | If the client uses the wrong interface on a multi-interface machine,
|
|---|
| 269 | there is a config file option
|
|---|
| 270 | <tt>SetBindAddress=</tt><i>IP address</i>
|
|---|
| 271 | that allows to choose the interface the client will use for
|
|---|
| 272 | outgoing connections.
|
|---|
| 273 | </p>
|
|---|
| 274 | <p>
|
|---|
| 275 | If you want to download the config file from the server, you
|
|---|
| 276 | should instead use the corresponding command line option
|
|---|
| 277 | <tt>--bind-address=</tt><i>IP address</i>
|
|---|
| 278 | to select the interface.
|
|---|
| 279 | </p>
|
|---|
| 280 |
|
|---|
| 281 | <p>
|
|---|
| 282 | If you encounter problems, you may (1) fix your
|
|---|
| 283 | <tt>/etc/hosts</tt> file(s), (2) fix your local DNS, or
|
|---|
| 284 | (3) switch to the second method.
|
|---|
| 285 | </p>
|
|---|
| 286 | <p>
|
|---|
| 287 | Errors in name resolving/cross-checking can be avoided by setting a
|
|---|
| 288 | very low severity (lower than the logging threshold), e.g.
|
|---|
| 289 | </p>
|
|---|
| 290 | <p>
|
|---|
| 291 | <tt>SeverityLookup=</tt><i>debug</i>
|
|---|
| 292 | </p>
|
|---|
| 293 | <p>
|
|---|
| 294 | in the <i>Misc</i> section of the server configuration,
|
|---|
| 295 | if you prefer running <i>unsafe</i> at any speed
|
|---|
| 296 | instead of fixing the problem (you have been warned). Doing so will
|
|---|
| 297 | allow an attacker to pose as the client.
|
|---|
| 298 | </p>
|
|---|
| 299 | </li>
|
|---|
| 300 | <li>
|
|---|
| 301 | <p><i>Second method: Use address of connecting entity as
|
|---|
| 302 | known to the communication layer</i></p>
|
|---|
| 303 | <p>
|
|---|
| 304 | This has been dropped as default
|
|---|
| 305 | long ago because it may not always be the
|
|---|
| 306 | address of the client machine.
|
|---|
| 307 | To enable this method, use
|
|---|
| 308 | </p>
|
|---|
| 309 | <p>
|
|---|
| 310 | <tt>SetClientFromAccept=</tt><i>true</i>
|
|---|
| 311 | </p>
|
|---|
| 312 | <p>
|
|---|
| 313 | in the <i>Misc</i> section of the server configuration
|
|---|
| 314 | file. If the address cannot be resolved, or reverse lookup of the
|
|---|
| 315 | resolved name fails, <i>no</i> error message will be issued,
|
|---|
| 316 | but the numerical address will be used.
|
|---|
| 317 | </p>
|
|---|
| 318 | </li>
|
|---|
| 319 | </ul>
|
|---|
| 320 |
|
|---|
| 321 |
|
|---|
| 322 | <h2><a name="sect3">Downloading config/database files</a></h2>
|
|---|
| 323 |
|
|---|
| 324 | <p>
|
|---|
| 325 | The client does <i>not</i> tell the server the path to the requested
|
|---|
| 326 | file - it just requests a config or a database file. It's entirely the
|
|---|
| 327 | responsibility of the server to locate the correct file and send it.
|
|---|
| 328 | </p>
|
|---|
| 329 | <p>
|
|---|
| 330 | The server has a <i>data directory</i>, which by default would be
|
|---|
| 331 | <tt>/var/lib/yule</tt>. Here the config/database files should be placed.
|
|---|
| 332 | </p>
|
|---|
| 333 | <p>
|
|---|
| 334 | Configuration files: <tt>rc.</tt><i>client.mydomain.tld</i> or
|
|---|
| 335 | simply <tt>rc</tt>
|
|---|
| 336 | (this can be used as a catchall file).
|
|---|
| 337 | </p>
|
|---|
| 338 | <p>
|
|---|
| 339 | Database files: <tt>file.</tt><i>client.mydomain.tld</i> or
|
|---|
| 340 | simply <tt>file</tt>
|
|---|
| 341 | (this can be used as a catchall file).
|
|---|
| 342 | </p>
|
|---|
| 343 |
|
|---|
| 344 | <h3>Problem #1</h3>
|
|---|
| 345 |
|
|---|
| 346 | <p>
|
|---|
| 347 | If the server cannot access the configuration (or database) file, either
|
|---|
| 348 | because it does not exist or the server has no read permission, the
|
|---|
| 349 | client will report <b>File download failed</b>. The server will
|
|---|
| 350 | report: <b>File not accessible</b>, <i>and</i> it will tell you in the
|
|---|
| 351 | same report the path where it would have expected the file (example):
|
|---|
| 352 | <b>path="/var/lib/yule/rc.client.mydomain.com"</b>
|
|---|
| 353 | </p>
|
|---|
| 354 | <p>
|
|---|
| 355 | To fix: put the file in the correct location, make sure the permissions
|
|---|
| 356 | are ok.
|
|---|
| 357 | </p>
|
|---|
| 358 |
|
|---|
| 359 |
|
|---|
| 360 | <h2><a name="sect4">Other connection problems</a></h2>
|
|---|
| 361 |
|
|---|
| 362 | <p>
|
|---|
| 363 | The server has a table with client names and their session keys. If
|
|---|
| 364 | another client process accesses the server from the same host,
|
|---|
| 365 | it will negotiate a fresh session key for that host. As a consequence,
|
|---|
| 366 | the session key of the first client process will become <i>invalid</i>.
|
|---|
| 367 | </p>
|
|---|
| 368 | <p>
|
|---|
| 369 | Also, the server keeps track of the status of a client. If a client
|
|---|
| 370 | process does not announce its termination to the server, the server
|
|---|
| 371 | will not expect a <i>startup</i> message, and issue a warning for any
|
|---|
| 372 | such message.
|
|---|
| 373 | </p>
|
|---|
| 374 |
|
|---|
| 375 | <h3>Problem #1</h3>
|
|---|
| 376 |
|
|---|
| 377 | <p>
|
|---|
| 378 | The client reports: <b>Invalid connection state</b>. The server reports:
|
|---|
| 379 | <b>Invalid connection attempt: Signature mismatch</b>. This is a sign that
|
|---|
| 380 | a client has tried to connect using an invalid session key. Most probably,
|
|---|
| 381 | another instance of the client is/was started on the respective host.
|
|---|
| 382 | </p>
|
|---|
| 383 | <p>
|
|---|
| 384 | To fix: if you need to have concurrent access to the server,
|
|---|
| 385 | suspend the first process with SIGUSR2 before starting the second. Use
|
|---|
| 386 | SIGUSR2 again to wake up the first process. Give the process a second or two
|
|---|
| 387 | to return into the main event loop and go into suspend mode. Do not just use
|
|---|
| 388 | SIGSTOP/SIGCONT: it is important that the client tells the server that
|
|---|
| 389 | it will go into suspend.
|
|---|
| 390 | </p>
|
|---|
| 391 |
|
|---|
| 392 | <h3>Problem #2</h3>
|
|---|
| 393 |
|
|---|
| 394 | <p>
|
|---|
| 395 | The server reports:
|
|---|
| 396 | <b>Restart without prior exit</b> for a client.
|
|---|
| 397 | This is a sign that
|
|---|
| 398 | a client has re-started without informing the server about a previous
|
|---|
| 399 | termination.
|
|---|
| 400 | </p>
|
|---|
| 401 | <p>
|
|---|
| 402 | This would happen if the client was killed with SIGKILL, or if it terminated
|
|---|
| 403 | within the routine to send a message to the server (the routine is
|
|---|
| 404 | not re-entrant). You may want to investigate messages logged via another
|
|---|
| 405 | logging facility (e.g. the client's local logfile). Of course it <i>may</i>
|
|---|
| 406 | also be a segfault, which would be reported via syslog.
|
|---|
| 407 | </p>
|
|---|
| 408 |
|
|---|
| 409 | </div>
|
|---|
| 410 | </body>
|
|---|
| 411 | </html>
|
|---|
