source: trunk/docs/HOWTO-client+server-troubleshooting.html@ 223

Last change on this file since 223 was 91, checked in by rainer, 18 years ago

Slightly improved fix for samhain_hide module.

File size: 12.4 KB
Line 
1<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
2<html>
3<head>
4<title>HOWTO client+server troubleshooting</title>
5<style type="text/css">
6<!--
7
8html { background: #eee; color: #000; }
9
10body { background: #eee; color: #000; margin: 0; padding: 0;}
11
12div.body {
13 background: #fff; color: #000;
14 margin: 0 1em 0 1em; padding: 1em;
15 font-family: serif;
16 font-size: 1em; line-height: 1.2em;
17 border-width: 0 1px 0 1px;
18 border-style: solid;
19 border-color: #aaa;
20}
21
22div.block {
23 background: #b6c5f2; color: #000;
24 margin: 1em; padding: 0 1em 0 1em;
25 border-width: 1px;
26 border-style: solid;
27 border-color: #2d4488;
28}
29
30div.warnblock {
31 background: #b6c5f2; color: #000;
32 background: #ffffcc; color: #000;
33 margin: 1em; padding: 0 1em 0 1em;
34 border-width: 1px;
35 border-style: solid;
36 border-color: #FF9900;
37}
38
39table {
40 background: #F8F8F8; color: #000;
41 margin: 1em;
42 border-width: 0 0 0 1px;
43 border-style: solid;
44 border-color: #C0C0C0;
45}
46
47td {
48 border-width: 0 1px 1px 0;
49 border-style: solid;
50 border-color: #C0C0C0;
51}
52
53th {
54 background: #F8F8FF;
55 border-width: 1px 1px 2px 0;
56 border-style: solid;
57 border-color: #C0C0C0;
58}
59
60
61/* body text, headings, and rules */
62
63p { margin: 0; text-indent: 0em; margin: 0 0 0.5em 0 }
64
65h1, h2, h3, h4, h5, h6 {
66 color: #206020; background: transparent;
67 font-family: Optima, Arial, Helvetica, sans-serif;
68 font-weight: normal;
69}
70
71h1 { font-size: 1.69em; margin: 1.4em 0 0.4em 0; }
72h2 { font-size: 1.44em; margin: 1.4em 0 0.4em 0; }
73h3 { font-size: 1.21em; margin: 1.4em 0 0.4em 0; }
74h4 { font-size: 1.00em; margin: 1.4em 0 0.4em 0; }
75h5 { font-size: 0.81em; margin: 1.4em 0 0.4em 0; }
76h6 { font-size: 0.64em; margin: 1.4em 0 0.4em 0; }
77
78hr {
79 color: transparent; background: transparent;
80 height: 0px; margin: 0.6em 0;
81 border-width: 1px ;
82 border-style: solid;
83 border-color: #999;
84}
85
86/* bulleted lists and definition lists */
87
88ul { margin: 0 1em 0.6em 2em; padding: 0; }
89li { margin: 0.4em 0 0 0; }
90
91dl { margin: 0.6em 1em 0.6em 2em; }
92dt { color: #285577; }
93
94tt { color: #602020; }
95
96/* links */
97
98a.link {
99 color: #33c; background: transparent;
100 text-decoration: none;
101}
102
103a:hover {
104 color: #000; background: transparent;
105}
106
107body > a {
108 font-family: Optima, Arial, Helvetica, sans-serif;
109 font-size: 0.81em;
110}
111
112h1, h2, h3, h4, h5, h6 {
113 color: #2d5588; background: transparent;
114 font-family: Optima, Arial, Helvetica, sans-serif;
115 font-weight: normal;
116}
117
118 -->
119</style></head>
120
121<body>
122<div class="body">
123<p style="text-align: center; background: #ccc; border: 1px solid #2d5588;"><a
124 style="text-decoration: none;"
125 href="http://www.la-samhna.de/samhain/">samhain file integrity
126 scanner</a>&nbsp;|&nbsp;<a style="text-decoration: none;"
127 href="http://www.la-samhna.de/samhain/s_documentation.html">online
128 documentation</a></p>
129<br><center>
130<h1>Samhain client/server: What can go wrong, and how can you fix it ?</h1>
131</center>
132<br>
133<hr>
134<div class="warnblock">
135<ul>
136 <li>Almost all problems can only be diagnosed correctly by checking the
137 <b>server</b> logs</li>
138 <li>If the server does not write logs, <b>fix this first</b>. For debugging,
139 stop the server, then run it in the foreground with
140 <tt>yule -p info --foreground</tt></li>
141</ul>
142</div>
143<p>
144This document aims to explain how to diagnose and fix common problems that
145may result from misunderstanding or misconfiguration when setting up
146a client/server samhain system. This document is divided in several sections
147more or less corresponding to the different stages when a client
148connects to a server. Each section starts with a brief explanation that
149should provide a basic understanding of what is going on.
150</p>
151<p>
152This document does not discuss <i>how</i> to setup a client/server (for
153this, look into the manual and/or the HOWTO-client+server).
154</p>
155
156<h2><a name="sect1">Table of Contents</a></h2>
157<p>
158<a href="#sect1">Connecting to the server</a><br>
159<a href="#sect2">Authentication</a><br>
160<a href="#sect3">Downloading config/database files</a><br>
161<a href="#sect4">Other connection problems</a><br>
162</p>
163
164<h2><a name="sect1">Connecting to the server</a></h2>
165
166<p>
167Client/server connections are always initiated from the client. The port
168is compiled in (there is a configure option to change the default).
169The default port is 49777.
170</p>
171
172<h3>Problem #1</h3>
173<p>
174The client reports: <b>Connection refused</b>. The server reports nothing.
175</p>
176<p>
177The server is down, listens on the wrong port, or network failure.
178</p>
179
180<h3>Problem #2</h3>
181<p>
182The client reports: <b>Connection error: Connection reset by peer</b>, and
183later also <b>Session key negotiation failed</b>. The server reports:
184<b>msg=&quot;Refused connection from ...&quot; subroutine=&quot;libwrap&quot;</b>.
185</p>
186<p>
187The server is compiled with libwrap (TCP Wrapper) support, and the
188client is either in <tt>/etc/hosts.deny</tt>, or you have set <i>yule: ALL</i>
189in <tt>/etc/hosts.deny</tt>, and forgot to put the client in
190<tt>/etc/hosts.allow</tt>.
191</p>
192<p>
193To fix: make proper entries in <tt>/etc/hosts.allow</tt> and/or
194<tt>/etc/hosts.deny</tt>. There is no need to restart/reload the server.
195</p>
196
197
198<h2><a name="sect2">Authentication</a></h2>
199<p>
200The client has a password that is used to authenticate to the server.
201This password is located within the binary, and is set with the
202<tt>samhain_setpwd</tt> helper application, as explained e.g. in the
203manual or in the Client+Server HOWTO.
204</p><p>
205The server has a list of clients that are allowed to connect, and the
206verifiers corresponding to the passwords of these clients.
207</p>
208<p>
209Upon successful authentication, client and server will negotiate
210a <b>session key</b> that is used for signing further messages
211from the client.
212</p>
213
214<h3>Problem #1</h3>
215
216<p>
217If the password is wrong, the client will report
218<b>Session key negotiation failed</b>. The server will
219report: <b>Invalid connection attempt: Session key mismatch</b>
220</p>
221<p>
222To fix: make sure that the password has in fact been set, that you are
223using the correct executable for the client (the one where the password is
224set), and that the entry in the server config file is the one generated
225for this password (also look out for double entries for this client).
226</p>
227
228<h3>Problem #2</h3>
229
230<p>
231If the client name (as resolved on the server) is wrong, the client
232will report
233<b>Session key negotiation failed</b>. The server will
234report: <b>Invalid connection attempt: Not in client list</b>,
235<i>and</i> it will tell you in the same error message
236what name it has inferred for the connecting
237client (example): <b>client=&quot;client.mydomain.com&quot;</b>.
238</p>
239<p>
240The fix depends on the nature of the problem. In principle, it should be
241sufficient to change the name of the client in the config file entry, which
242isn't really a solution if e.g. the server thinks the client is 'localhost'.
243</p>
244<p>
245There are two different ways to determine the client name.
246Unfortunately, judging
247from customer feedback as well from common sense, both do not work very well
248with a messed up local DNS (including /etc/hosts files) and/or
249&uuml;berparanoid or misconfigured firewalls (in case of connections
250across one).
251</p>
252<ul>
253 <li>
254 <p>
255 <i>First method: Determine client name on client, and
256 try to cross-check on server</i>
257 <p>
258 <p>
259 This does not work for a number of people because (1) the
260 <tt>/etc/hosts</tt> file on the client machine has errors
261 (yes, there are plenty machines with a completely
262 messed up <tt>/etc/hosts</tt> file), (2) the
263 server cannot resolve the client address because the local DNS is
264 f***ed up, or (3) the client machine has multiple network interfaces, and
265 the interface used is not the one the client name resolves to.
266 </p>
267
268 <p>
269 If the client uses the wrong interface on a multi-interface machine,
270 there is a config file option
271 <tt>SetBindAddress=</tt><i>IP address</i>
272 that allows to choose the interface the client will use for
273 outgoing connections.
274 </p>
275 <p>
276 If you want to download the config file from the server, you
277 should instead use the corresponding command line option
278 <tt>--bind-address=</tt><i>IP address</i>
279 to select the interface.
280 </p>
281
282 <p>
283 If you encounter problems, you may (1) fix your
284 <tt>/etc/hosts</tt> file(s), (2) fix your local DNS, or
285 (3) switch to the second method.
286 </p>
287 <p>
288 Errors in name resolving/cross-checking can be avoided by setting a
289 very low severity (lower than the logging threshold), e.g.
290 </p>
291 <p>
292 <tt>SeverityLookup=</tt><i>debug</i>
293 </p>
294 <p>
295 in the <i>Misc</i> section of the server configuration,
296 if you prefer running <i>unsafe</i> at any speed
297 instead of fixing the problem (you have been warned). Doing so will
298 allow an attacker to pose as the client.
299 </p>
300 </li>
301 <li>
302 <p><i>Second method: Use address of connecting entity as
303 known to the communication layer</i></p>
304 <p>
305 This has been dropped as default
306 long ago because it may not always be the
307 address of the client machine.
308 To enable this method, use
309 </p>
310 <p>
311 <tt>SetClientFromAccept=</tt><i>true</i>
312 </p>
313 <p>
314 in the <i>Misc</i> section of the server configuration
315 file. If the address cannot be resolved, or reverse lookup of the
316 resolved name fails, <i>no</i> error message will be issued,
317 but the numerical address will be used.
318 </p>
319 </li>
320</ul>
321
322
323<h2><a name="sect3">Downloading config/database files</a></h2>
324
325<p>
326The client does <i>not</i> tell the server the path to the requested
327file - it just requests a config or a database file. It's entirely the
328responsibility of the server to locate the correct file and send it.
329</p>
330<p>
331The server has a <i>data directory</i>, which by default would be
332<tt>/var/lib/yule</tt>. Here the config/database files should be placed.
333</p>
334<p>
335Configuration files: <tt>rc.</tt><i>client.mydomain.tld</i> or
336simply <tt>rc</tt>
337(this can be used as a catchall file).
338</p>
339<p>
340Database files: <tt>file.</tt><i>client.mydomain.tld</i> or
341simply <tt>file</tt>
342(this can be used as a catchall file).
343</p>
344
345<h3>Problem #1</h3>
346
347<p>
348If the server cannot access the configuration (or database) file, either
349because it does not exist or the server has no read permission, the
350client will report <b>File download failed</b>. The server will
351report: <b>File not accessible</b>, <i>and</i> it will tell you in the
352same report the path where it would have expected the file (example):
353<b>path=&quot;/var/lib/yule/rc.client.mydomain.com&quot;</b>
354</p>
355<p>
356To fix: put the file in the correct location, make sure the permissions
357are ok.
358</p>
359
360
361<h2><a name="sect4">Other connection problems</a></h2>
362
363<p>
364The server has a table with client names and their session keys. If
365another client process accesses the server from the same host,
366it will negotiate a fresh session key for that host. As a consequence,
367the session key of the first client process will become <i>invalid</i>.
368</p>
369<p>
370Also, the server keeps track of the status of a client. If a client
371process does not announce its termination to the server, the server
372will not expect a <i>startup</i> message, and issue a warning for any
373such message.
374</p>
375
376<h3>Problem #1</h3>
377
378<p>
379The client reports: <b>Invalid connection state</b>. The server reports:
380<b>Invalid connection attempt: Signature mismatch</b>. This is a sign that
381a client has tried to connect using an invalid session key. Most probably,
382another instance of the client is/was started on the respective host.
383</p>
384<p>
385To fix: if you need to have concurrent access to the server,
386suspend the first process with SIGUSR2 before starting the second. Use
387SIGUSR2 again to wake up the first process. Give the process a second or two
388to return into the main event loop and go into suspend mode. Do not just use
389SIGSTOP/SIGCONT: it is important that the client tells the server that
390it will go into suspend.
391</p>
392
393<h3>Problem #2</h3>
394
395<p>
396The server reports:
397<b>Restart without prior exit</b> for a client.
398This is a sign that
399a client has re-started without informing the server about a previous
400termination.
401</p>
402<p>
403This would happen if the client was killed with SIGKILL, or if it terminated
404within the routine to send a message to the server (the routine is
405not re-entrant). You may want to investigate messages logged via another
406logging facility (e.g. the client's local logfile). Of course it <i>may</i>
407also be a segfault, which would be reported via syslog.
408</p>
409
410</div>
411</body>
412</html>
Note: See TracBrowser for help on using the repository browser.