Aikido, programming, system administration, and other things I find interesting

Installing munin-async

With the latest versions of munin-async, this article is out of date. Instructions for munin 2.0.17 on Ubuntu 12.04 are here

When using munin, one often runs into one of two problems:

  •  there are so many nodes to update, the update takes more than the update interval
  • some servers may be connected over flaky lines, so an update may be lost due to timeout

With version 2.0, the designers of munin have started addressing those problems. Today we look at one part of that solution, munin-async. Note that I am using the packages from Debian experimental. Your experience on other OSs may vary. You can tell that munin-async in version 2.0.1 is not quite ready for prime time yet. Here are the steps I needed to take in order for the client to collect munin-async data from the various servers:

Install munin-async on the monitored machines AND the graphing server

The munin-async debian package contains both the client AND the server scripts for async work. This is not consistent, since previously all the data fetching scripts were in the munin package, and all the data serving scripts were in the munin-node package. It also means that you have to install munin-async (creating the munin-async user, with it’s own entry in passwd file and it’s shell set to /bin/bash) on the server, not just on the clients. I don’t like leaving that open. (on remote machine and on server)

apt-get install munin-async

If your distro isn’t including munin-2.0.1 packages yet, you should download them from debian experimental, and install them with

dpkg --install --force-overwrite munin-async_2.0.1-1_all.deb munin-common_2.0.1-1_all.deb munin-node_2.0.1-1_all.deb munin-plugins-*

The force-overwrite is needed due to this error message:

dpkg: error processing munin-common_2.0.1-1_all.deb (--install): trying to overwrite '/usr/share/munin/plugins/plugin.sh' which is also in package munin-node 0:1.4.4-1ubuntu1

Correct the /etc/init.d script

The /etc/init.d/munin-async file is looking for a script called munin-async-server instead of munin-asyncd

--- /etc/init.d/munin-async.orig 2012-07-15 01:10:56.000000000 -0700
+++ /etc/init.d/munin-async 2012-07-15 01:12:23.000000000 -0700
@@ -16,7 +16,7 @@
 # PATH should only include /usr/* if it runs after the mountnfs.sh script
 PATH=/sbin:/usr/sbin:/bin:/usr/bin
 DESC="Munin asynchronous server"
-NAME=munin-async-server
+NAME=munin-asyncd
 DAEMON=/usr/share/munin/$NAME
 DAEMON_ARGS=""
 PIDFILE=/var/run/munin/$NAME.pid

Start munin-asyncd on servers where data is to be collected

(on remote machine) service munin-async start

Prepare the master for using ssh to connect to servers

Change the shell of the munin user to bash so you can do these changes as the munin user:

vipw
su - munin
cd /var/lib/munin
mkdir .ssh
cd .ssh
ssh-keygen

(tell ssh-keygen to create a key with an empty passphrase and place it in /var/lib/munin/.ssh) (on the remote machine)

mkdir /var/lib/munin-async/.ssh

(on the server)

scp /var/lib/munin/.ssh/id_rsa.pub root@example.net:/var/lib/munin-async/.ssh/authorized_keys
chown -R munin:munin /var/lib/munin/.ssh

(on the remote machine)

chown -R munin-async:munin-async /var/lib/munin-async

(on the server, test the connection)

ssh munin-async@example.net
exit

Note that you need to check the connection for EVERY host from which you intend to collect data in the async manner. munin is NOThandling this dialogue:

The authenticity of host 'example.net (2600:more:fool:you:f9b)' can't be established.
RSA key fingerprint is 61:aa:aa:aa:aa:aa:aa:aa:aa:aa:aa:aa:aa:aa:aa:aa.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'example.net,2600:moore:fool:you:f9b' (RSA) to the list of known hosts.

So you need to log in “by hand” first, from the user munin, in order to record the key. Or you need to copy the key from antoher known_hosts file, which may be tricky. Now change the shell of munin back to /bin/false, for security.

vipw

Change the system definition in /etc/munin/munin.conf

(or, as I prefer to do it, in /etc/munin/munin-conf.d/hostlist.conf ).

[async.my-machine.net]
 address ssh://munin-async@example.net /usr/share/munin/munin-async --spooldir /var/lib/munin/spool --spoolfetch
 use_node_name yes

I am using async in the definition name merely so that I can compare the data from the two collection methods.

Security enhancement

To prevent your monitored server being compromised if someone manages to break into your munin collection server, you should edit the /var/lib/munin-async/.ssh/authorized_keys file and add

no-port-forwarding,no-agent-forwarding,no-X11-forwarding,command="/usr/share/munin/munin-async --spooldir /var/lib/munin/spool --spoolfetch"

to the beginningof the relevant line.

Adding plugins

When you add a plugin, it won’t be visible unless you first restart munin-node and THENmunin-async.

Troubleshooting tips

If you haven’t logged in to the host “by hand” or added it’s keys to known_hosts some other way, the fetch will fail. The only log in the munin-update file will say something like

Socket read from async.example.net failed.  Terminating process. at /usr/share/perl5/Munin/Master/UpdateWorker.pm line ...

Another possible cause of mysterious failure to fetch data from the remote host (that does not give a clear error message) is munin-asyncd not running on the target server, or having no prefetched data yet.

Additional ideas

Balint Deaksuggested in a post on the munin-users mailing list: What I would add to this is that if you have many hosts, or hosts are added on a daily basis, it may be annoying to always remember to log in to each new box and say “yes” at the prompt.

If you create a config file for ssh in the $HOME/.ssh/config for the user that runs the master (defaults to ‘munin’) and tell ssh not to check the host key when authenticating, then no prompt will be displayed even for new or unknown hosts.
Add something like:
Host *
 UserKnownHostsFile=/dev/null
 StrictHostKeyChecking=no
I don’t think this makes the setup less secure, but it would make the automation of adding new hosts to the system easier.
Regards,
Balint

 

Related Posts

Why is my munin slow and how to speed it up

At $work we are monitoring a network of hundreds of servers, and that means that we end up recording hundreds of thousands of variable values every five minutes. After a while, the server started slowing down, taking more than 300 seconds to collect the data. Since it has a whole-system lock, that means the next […]

Read More

A munin plugin to monitor each CPU core separately

Monitoring each core separately may seem like a waste – after all, we have an overall CPU usage already available under “system” in munin, isn’t that enough? It turns out that it isn’t. Sometimes, when using top on a multicore/multicpu machine, you can see a process pegged at 100%, while other processes are comfortably using […]

Read More

3 Comments

  • Ricardo Lopes on September 11, 2013

    Hi, can we have a quick chat? Im having a proxy issue and i really need to put munin working with async since the normal mode is giving me holes 🙂

    • Matija on September 11, 2013

      Sure, I’ll contact you via the email you provided.

  • […] Munin authors have moved on in their development and if you are going to be installing munin-async on any system, I strongly recommend that you use the latest version (2.0.17 at the time of writing), which fixes a number of problems I described in a previous article on Installing munin-async. […]

Leave a Reply

Your email address will not be published. Required fields are marked *