| 1 | <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
|
|---|
| 2 | <html>
|
|---|
| 3 | <head>
|
|---|
| 4 | <title>HOWTO client+server troubleshooting</title>
|
|---|
| 5 | <style type="text/css">
|
|---|
| 6 | <!--
|
|---|
| 7 |
|
|---|
| 8 | html { background: #eee; color: #000; }
|
|---|
| 9 |
|
|---|
| 10 | body { background: #eee; color: #000; margin: 0; padding: 0;}
|
|---|
| 11 |
|
|---|
| 12 | div.body {
|
|---|
| 13 | background: #fff; color: #000;
|
|---|
| 14 | margin: 0 1em 0 1em; padding: 1em;
|
|---|
| 15 | font-family: serif;
|
|---|
| 16 | font-size: 1em; line-height: 1.2em;
|
|---|
| 17 | border-width: 0 1px 0 1px;
|
|---|
| 18 | border-style: solid;
|
|---|
| 19 | border-color: #aaa;
|
|---|
| 20 | }
|
|---|
| 21 |
|
|---|
| 22 | div.block {
|
|---|
| 23 | background: #b6c5f2; color: #000;
|
|---|
| 24 | margin: 1em; padding: 0 1em 0 1em;
|
|---|
| 25 | border-width: 1px;
|
|---|
| 26 | border-style: solid;
|
|---|
| 27 | border-color: #2d4488;
|
|---|
| 28 | }
|
|---|
| 29 |
|
|---|
| 30 | div.warnblock {
|
|---|
| 31 | background: #b6c5f2; color: #000;
|
|---|
| 32 | margin: 1em; padding: 0 1em 0 1em;
|
|---|
| 33 | border-width: 1px;
|
|---|
| 34 | border-style: solid;
|
|---|
| 35 | border-color: #FF9900;
|
|---|
| 36 | }
|
|---|
| 37 |
|
|---|
| 38 | table {
|
|---|
| 39 | background: #F8F8F8; color: #000;
|
|---|
| 40 | margin: 1em;
|
|---|
| 41 | border-width: 0 0 0 1px;
|
|---|
| 42 | border-style: solid;
|
|---|
| 43 | border-color: #C0C0C0;
|
|---|
| 44 | }
|
|---|
| 45 |
|
|---|
| 46 | td {
|
|---|
| 47 | border-width: 0 1px 1px 0;
|
|---|
| 48 | border-style: solid;
|
|---|
| 49 | border-color: #C0C0C0;
|
|---|
| 50 | }
|
|---|
| 51 |
|
|---|
| 52 | th {
|
|---|
| 53 | background: #F8F8FF;
|
|---|
| 54 | border-width: 1px 1px 2px 0;
|
|---|
| 55 | border-style: solid;
|
|---|
| 56 | border-color: #C0C0C0;
|
|---|
| 57 | }
|
|---|
| 58 |
|
|---|
| 59 |
|
|---|
| 60 | /* body text, headings, and rules */
|
|---|
| 61 |
|
|---|
| 62 | p { margin: 0; text-indent: 0em; margin: 0 0 0.5em 0 }
|
|---|
| 63 |
|
|---|
| 64 | h1, h2, h3, h4, h5, h6 {
|
|---|
| 65 | color: #206020; background: transparent;
|
|---|
| 66 | font-family: Optima, Arial, Helvetica, sans-serif;
|
|---|
| 67 | font-weight: normal;
|
|---|
| 68 | }
|
|---|
| 69 |
|
|---|
| 70 | h1 { font-size: 1.69em; margin: 1.4em 0 0.4em 0; }
|
|---|
| 71 | h2 { font-size: 1.44em; margin: 1.4em 0 0.4em 0; }
|
|---|
| 72 | h3 { font-size: 1.21em; margin: 1.4em 0 0.4em 0; }
|
|---|
| 73 | h4 { font-size: 1.00em; margin: 1.4em 0 0.4em 0; }
|
|---|
| 74 | h5 { font-size: 0.81em; margin: 1.4em 0 0.4em 0; }
|
|---|
| 75 | h6 { font-size: 0.64em; margin: 1.4em 0 0.4em 0; }
|
|---|
| 76 |
|
|---|
| 77 | hr {
|
|---|
| 78 | color: transparent; background: transparent;
|
|---|
| 79 | height: 0px; margin: 0.6em 0;
|
|---|
| 80 | border-width: 1px ;
|
|---|
| 81 | border-style: solid;
|
|---|
| 82 | border-color: #999;
|
|---|
| 83 | }
|
|---|
| 84 |
|
|---|
| 85 | /* bulleted lists and definition lists */
|
|---|
| 86 |
|
|---|
| 87 | ul { margin: 0 1em 0.6em 2em; padding: 0; }
|
|---|
| 88 | li { margin: 0.4em 0 0 0; }
|
|---|
| 89 |
|
|---|
| 90 | dl { margin: 0.6em 1em 0.6em 2em; }
|
|---|
| 91 | dt { color: #285577; }
|
|---|
| 92 |
|
|---|
| 93 | tt { color: #602020; }
|
|---|
| 94 |
|
|---|
| 95 | /* links */
|
|---|
| 96 |
|
|---|
| 97 | a.link {
|
|---|
| 98 | color: #33c; background: transparent;
|
|---|
| 99 | text-decoration: none;
|
|---|
| 100 | }
|
|---|
| 101 |
|
|---|
| 102 | a:hover {
|
|---|
| 103 | color: #000; background: transparent;
|
|---|
| 104 | }
|
|---|
| 105 |
|
|---|
| 106 | body > a {
|
|---|
| 107 | font-family: Optima, Arial, Helvetica, sans-serif;
|
|---|
| 108 | font-size: 0.81em;
|
|---|
| 109 | }
|
|---|
| 110 |
|
|---|
| 111 | h1, h2, h3, h4, h5, h6 {
|
|---|
| 112 | color: #2d5588; background: transparent;
|
|---|
| 113 | font-family: Optima, Arial, Helvetica, sans-serif;
|
|---|
| 114 | font-weight: normal;
|
|---|
| 115 | }
|
|---|
| 116 |
|
|---|
| 117 | -->
|
|---|
| 118 | </style></head>
|
|---|
| 119 |
|
|---|
| 120 | <body>
|
|---|
| 121 | <div class="body">
|
|---|
| 122 | <p style="text-align: center; background: #ccc; border: 1px solid #2d5588;"><a
|
|---|
| 123 | style="text-decoration: none;"
|
|---|
| 124 | href="http://www.la-samhna.de/samhain/">samhain file integrity
|
|---|
| 125 | scanner</a> | <a style="text-decoration: none;"
|
|---|
| 126 | href="http://www.la-samhna.de/samhain/s_documentation.html">online
|
|---|
| 127 | documentation</a></p>
|
|---|
| 128 | <br><center>
|
|---|
| 129 | <h1>Samhain client/server: What can go wrong, and how can you fix it ?</h1>
|
|---|
| 130 | </center>
|
|---|
| 131 | <br>
|
|---|
| 132 | <hr>
|
|---|
| 133 | <p>
|
|---|
| 134 | This document aims to explain how to diagnose and fix common problems that
|
|---|
| 135 | may result from misunderstanding or misconfiguration when setting up
|
|---|
| 136 | a client/server samhain system. This document is divided in several sections
|
|---|
| 137 | more or less corresponding to the different stages when a client
|
|---|
| 138 | connects to a server. Each section starts with a brief explanation that
|
|---|
| 139 | should provide a basic understanding of what is going on.
|
|---|
| 140 | </p>
|
|---|
| 141 | <p>
|
|---|
| 142 | This document does not discuss <i>how</i> to setup a client/server (for
|
|---|
| 143 | this, look into the manual and/or the HOWTO-client+server).
|
|---|
| 144 | </p>
|
|---|
| 145 |
|
|---|
| 146 | <h2><a name="sect1">Table of Contents</a></h2>
|
|---|
| 147 | <p>
|
|---|
| 148 | <a href="#sect1">Connecting to the server</a><br>
|
|---|
| 149 | <a href="#sect2">Authentication</a><br>
|
|---|
| 150 | <a href="#sect3">Downloading config/database files</a><br>
|
|---|
| 151 | <a href="#sect4">Other connection problems</a><br>
|
|---|
| 152 | </p>
|
|---|
| 153 |
|
|---|
| 154 | <h2><a name="sect1">Connecting to the server</a></h2>
|
|---|
| 155 |
|
|---|
| 156 | <p>
|
|---|
| 157 | Client/server connections are always initiated from the client. The port
|
|---|
| 158 | is compiled in (there is a configure option to change the default).
|
|---|
| 159 | The default port is 49777.
|
|---|
| 160 | </p>
|
|---|
| 161 |
|
|---|
| 162 | <h3>Problem #1</h3>
|
|---|
| 163 | <p>
|
|---|
| 164 | The client reports: <b>Connection refused</b>. The server reports nothing.
|
|---|
| 165 | </p>
|
|---|
| 166 | <p>
|
|---|
| 167 | The server is down, listens on the wrong port, or network failure.
|
|---|
| 168 | </p>
|
|---|
| 169 |
|
|---|
| 170 | <h3>Problem #2</h3>
|
|---|
| 171 | <p>
|
|---|
| 172 | The client reports: <b>Connection error: Connection reset by peer</b>, and
|
|---|
| 173 | later also <b>Session key negotiation failed</b>. The server reports:
|
|---|
| 174 | <b>msg="Refused connection from ..." subroutine="libwrap"</b>.
|
|---|
| 175 | </p>
|
|---|
| 176 | <p>
|
|---|
| 177 | The server is compiled with libwrap (TCP Wrapper) support, and the
|
|---|
| 178 | client is either in <tt>/etc/hosts.deny</tt>, or you have set <i>yule: ALL</i>
|
|---|
| 179 | in <tt>/etc/hosts.deny</tt>, and forgot to put the client in
|
|---|
| 180 | <tt>/etc/hosts.allow</tt>.
|
|---|
| 181 | </p>
|
|---|
| 182 | <p>
|
|---|
| 183 | To fix: make proper entries in <tt>/etc/hosts.allow</tt> and/or
|
|---|
| 184 | <tt>/etc/hosts.deny</tt>. There is no need to restart/reload the server.
|
|---|
| 185 | </p>
|
|---|
| 186 |
|
|---|
| 187 |
|
|---|
| 188 | <h2><a name="sect2">Authentication</a></h2>
|
|---|
| 189 | <p>
|
|---|
| 190 | The client has a password that is used to authenticate to the server.
|
|---|
| 191 | This password is located within the binary, and is set with the
|
|---|
| 192 | <tt>samhain_setpwd</tt> helper application, as explained e.g. in the
|
|---|
| 193 | manual or in the Client+Server HOWTO.
|
|---|
| 194 | </p><p>
|
|---|
| 195 | The server has a list of clients that are allowed to connect, and the
|
|---|
| 196 | verifiers corresponding to the passwords of these clients.
|
|---|
| 197 | </p>
|
|---|
| 198 | <p>
|
|---|
| 199 | Upon successful authentication, client and server will negotiate
|
|---|
| 200 | a <b>session key</b> that is used for signing further messages
|
|---|
| 201 | from the client.
|
|---|
| 202 | </p>
|
|---|
| 203 |
|
|---|
| 204 | <h3>Problem #1</h3>
|
|---|
| 205 |
|
|---|
| 206 | <p>
|
|---|
| 207 | If the password is wrong, the client will report
|
|---|
| 208 | <b>Session key negotiation failed</b>. The server will
|
|---|
| 209 | report: <b>Invalid connection attempt: Session key mismatch</b>
|
|---|
| 210 | </p>
|
|---|
| 211 | <p>
|
|---|
| 212 | To fix: make sure that the password has in fact been set, that you are
|
|---|
| 213 | using the correct executable for the client (the one where the password is
|
|---|
| 214 | set), and that the entry in the server config file is the one generated
|
|---|
| 215 | for this password (also look out for double entries for this client).
|
|---|
| 216 | </p>
|
|---|
| 217 |
|
|---|
| 218 | <h3>Problem #2</h3>
|
|---|
| 219 |
|
|---|
| 220 | <p>
|
|---|
| 221 | If the client name (as resolved on the server) is wrong, the client
|
|---|
| 222 | will report
|
|---|
| 223 | <b>Session key negotiation failed</b>. The server will
|
|---|
| 224 | report: <b>Invalid connection attempt: Not in client list</b>,
|
|---|
| 225 | <i>and</i> it will tell you in the same error message
|
|---|
| 226 | what name it has inferred for the connecting
|
|---|
| 227 | client (example): <b>client="client.mydomain.com"</b>.
|
|---|
| 228 | </p>
|
|---|
| 229 | <p>
|
|---|
| 230 | The fix depends on the nature of the problem. In principle, it should be
|
|---|
| 231 | sufficient to change the name of the client in the config file entry, which
|
|---|
| 232 | isn't really a solution if e.g. the server thinks the client is 'localhost'.
|
|---|
| 233 | </p>
|
|---|
| 234 | <p>
|
|---|
| 235 | There are two different ways to determine the client name.
|
|---|
| 236 | Unfortunately, judging
|
|---|
| 237 | from customer feedback as well from common sense, both do not work very well
|
|---|
| 238 | with a messed up local DNS (including /etc/hosts files) and/or
|
|---|
| 239 | überparanoid or misconfigured firewalls (in case of connections
|
|---|
| 240 | across one).
|
|---|
| 241 | </p>
|
|---|
| 242 | <ul>
|
|---|
| 243 | <li>
|
|---|
| 244 | <p>
|
|---|
| 245 | <i>First method: Determine client name on client, and
|
|---|
| 246 | try to cross-check on server</i>
|
|---|
| 247 | <p>
|
|---|
| 248 | <p>
|
|---|
| 249 | This does not work for a number of people because (1) the
|
|---|
| 250 | <tt>/etc/hosts</tt> file on the client machine has errors
|
|---|
| 251 | (yes, there are plenty machines with a completely
|
|---|
| 252 | messed up <tt>/etc/hosts</tt> file), (2) the
|
|---|
| 253 | server cannot resolve the client address because the local DNS is
|
|---|
| 254 | f***ed up, or (3) the client machine has multiple network interfaces, and
|
|---|
| 255 | the interface used is not the one the client name resolves to.
|
|---|
| 256 | </p>
|
|---|
| 257 | <p>
|
|---|
| 258 | If the client uses the wrong interface on a multi-interface machine,
|
|---|
| 259 | there is a config file option
|
|---|
| 260 | <tt>SetBindAddress=</tt><i>IP address</i>
|
|---|
| 261 | that allows to choose the interface the client will use for
|
|---|
| 262 | outgoing connections.
|
|---|
| 263 | </p>
|
|---|
| 264 | <p>
|
|---|
| 265 | If you want to download the config file from the server, you
|
|---|
| 266 | should instead use the corresponding command line
|
|---|
| 267 | <tt>--bind-address=</tt><i>IP address</i>
|
|---|
| 268 | to select the interface.
|
|---|
| 269 | </p>
|
|---|
| 270 |
|
|---|
| 271 | <p>
|
|---|
| 272 | If you encounter problems, you may (1) fix your
|
|---|
| 273 | <tt>/etc/hosts</tt> file(s), (2) fix your local DNS, or
|
|---|
| 274 | (3) switch to the second method.
|
|---|
| 275 | </p>
|
|---|
| 276 | <p>
|
|---|
| 277 | Errors in name resolving/cross-checking can be avoided by setting a
|
|---|
| 278 | very low severity (lower than the logging threshold), e.g.
|
|---|
| 279 | </p>
|
|---|
| 280 | <p>
|
|---|
| 281 | <tt>SeverityLookup=</tt><i>debug</i>
|
|---|
| 282 | </p>
|
|---|
| 283 | <p>
|
|---|
| 284 | in the <i>Misc</i> section of the server configuration,
|
|---|
| 285 | if you prefer running <i>unsafe</i> at any speed
|
|---|
| 286 | instead of fixing the problem (you have been warned). Doing so will
|
|---|
| 287 | allow an attacker to pose as the client.
|
|---|
| 288 | </p>
|
|---|
| 289 | </li>
|
|---|
| 290 | <li>
|
|---|
| 291 | <p><i>Second method: Use address of connecting entity as
|
|---|
| 292 | known to the communication layer</i></p>
|
|---|
| 293 | <p>
|
|---|
| 294 | This has been dropped as default
|
|---|
| 295 | long ago because it may not always be the
|
|---|
| 296 | address of the client machine.
|
|---|
| 297 | To enable this method, use
|
|---|
| 298 | </p>
|
|---|
| 299 | <p>
|
|---|
| 300 | <tt>SetClientFromAccept=</tt><i>true</i>
|
|---|
| 301 | </p>
|
|---|
| 302 | <p>
|
|---|
| 303 | in the <i>Misc</i> section of the server configuration
|
|---|
| 304 | file. If the address cannot be resolved, or reverse lookup of the
|
|---|
| 305 | resolved name fails, <i>no</i> error message will be issued,
|
|---|
| 306 | but the numerical address will be used.
|
|---|
| 307 | </p>
|
|---|
| 308 | </li>
|
|---|
| 309 | </ul>
|
|---|
| 310 |
|
|---|
| 311 |
|
|---|
| 312 | <h2><a name="sect3">Downloading config/database files</a></h2>
|
|---|
| 313 |
|
|---|
| 314 | <p>
|
|---|
| 315 | The client does <i>not</i> tell the server the path to the requested
|
|---|
| 316 | file - it just requests a config or a database file. It's entirely the
|
|---|
| 317 | responsibility of the server to locate the correct file and send it.
|
|---|
| 318 | </p>
|
|---|
| 319 | <p>
|
|---|
| 320 | The server has a <i>data directory</i>, which by default would be
|
|---|
| 321 | <tt>/var/lib/yule</tt>. Here the config/database files should be placed.
|
|---|
| 322 | </p>
|
|---|
| 323 | <p>
|
|---|
| 324 | Configuration files: <tt>rc.</tt><i>client.mydomain.tld</i> or
|
|---|
| 325 | simply <tt>rc</tt>
|
|---|
| 326 | (this can be used as a catchall file).
|
|---|
| 327 | </p>
|
|---|
| 328 | <p>
|
|---|
| 329 | Database files: <tt>file.</tt><i>client.mydomain.tld</i> or
|
|---|
| 330 | simply <tt>file</tt>
|
|---|
| 331 | (this can be used as a catchall file).
|
|---|
| 332 | </p>
|
|---|
| 333 |
|
|---|
| 334 | <h3>Problem #1</h3>
|
|---|
| 335 |
|
|---|
| 336 | <p>
|
|---|
| 337 | If the server cannot access the configuration (or database) file, either
|
|---|
| 338 | because it does not exist or the server has no read permission, the
|
|---|
| 339 | client will report <b>File download failed</b>. The server will
|
|---|
| 340 | report: <b>File not accessible</b>, <i>and</i> it will tell you in the
|
|---|
| 341 | same report the path where it would have expected the file (example):
|
|---|
| 342 | <b>path="/var/lib/yule/rc.client.mydomain.com"</b>
|
|---|
| 343 | </p>
|
|---|
| 344 | <p>
|
|---|
| 345 | To fix: put the file in the correct location, make sure the permissions
|
|---|
| 346 | are ok.
|
|---|
| 347 | </p>
|
|---|
| 348 |
|
|---|
| 349 |
|
|---|
| 350 | <h2><a name="sect4">Other connection problems</a></h2>
|
|---|
| 351 |
|
|---|
| 352 | <p>
|
|---|
| 353 | The server has a table with client names and their session keys. If
|
|---|
| 354 | another client process accesses the server from the same host,
|
|---|
| 355 | it will negotiate a fresh session key for that host. As a consequence,
|
|---|
| 356 | the session key of the first client process will become <i>invalid</i>.
|
|---|
| 357 | </p>
|
|---|
| 358 | <p>
|
|---|
| 359 | Also, the server keeps track of the status of a client. If a client
|
|---|
| 360 | process does not announce its termination to the server, the server
|
|---|
| 361 | will not expect a <i>startup</i> message, and issue a warning for any
|
|---|
| 362 | such message.
|
|---|
| 363 | </p>
|
|---|
| 364 |
|
|---|
| 365 | <h3>Problem #1</h3>
|
|---|
| 366 |
|
|---|
| 367 | <p>
|
|---|
| 368 | The client reports: <b>Invalid connection state</b>. The server reports:
|
|---|
| 369 | <b>Invalid connection attempt: Signature mismatch</b>. This is a sign that
|
|---|
| 370 | a client has tried to connect using an invalid session key. Most probably,
|
|---|
| 371 | another instance of the client is/was started on the respective host.
|
|---|
| 372 | </p>
|
|---|
| 373 | <p>
|
|---|
| 374 | To fix: if you need to have concurrent access to the server,
|
|---|
| 375 | suspend the first process with SIGUSR2 before starting the second. Use
|
|---|
| 376 | SIGUSR2 again to wake up the first process. Give the process a second or two
|
|---|
| 377 | to return into the main event loop and go into suspend mode. Do not just use
|
|---|
| 378 | SIGSTOP/SIGCONT: it is important that the client tells the server that
|
|---|
| 379 | it will go into suspend.
|
|---|
| 380 | </p>
|
|---|
| 381 |
|
|---|
| 382 | <h3>Problem #2</h3>
|
|---|
| 383 |
|
|---|
| 384 | <p>
|
|---|
| 385 | The server reports:
|
|---|
| 386 | <b>Restart without prior exit</b> for a client.
|
|---|
| 387 | This is a sign that
|
|---|
| 388 | a client has re-started without informing the server about a previous
|
|---|
| 389 | termination.
|
|---|
| 390 | </p>
|
|---|
| 391 | <p>
|
|---|
| 392 | This would happen if the client was killed with SIGKILL, or if it terminated
|
|---|
| 393 | within the routine to send a message to the server (the routine is
|
|---|
| 394 | not re-entrant). You may want to investigate messages logged via another
|
|---|
| 395 | logging facility (e.g. the client's local logfile). Of course it <i>may</i>
|
|---|
| 396 | also be a segfault, which would be reported via syslog.
|
|---|
| 397 | </p>
|
|---|
| 398 |
|
|---|
| 399 | </div>
|
|---|
| 400 | </body>
|
|---|
| 401 | </html>
|
|---|
