Setting up a Munin server and nodes

In this example, we are going to set up a munin server in its own LXC container instance. This container will run unprivileged under an ordinary user. We are using Devuan because regular Debian with systemd will not run in an unprivileged container AFAIK.

1. lxc-create -t download -n munin
(devuan, beowulf, arm64)
2. lxc-attach munin passwd

1. lxc-attach munin
2. apt update
3. apt autoremove
4. apt clean
5. dpkg-reconfigure tzdata

1. apt install munin
2. Change these lines in /etc/munin/munin.conf:

   graph_strategy cgi
   html_strategy cgi
   htmldir /var/www/munin

3. Add some nodes if you have any already:

   [Node1]
   address 1.2.3.4
   use_node_name yes

4. mkdir /var/www/munin
5. chown munin:munin /var/www/munin
6. service munin restart

1. apt install apache2 libapache2-mod-fcgid libcgi-fast-perl
2. vim /etc/apache2/sites-available/munin.conf

   <VirtualHost *:80>
       ServerName munin.example.org
       ServerAlias munin
    
       ServerAdmin  info@example.org
    
       DocumentRoot /var/www
    
       Alias /munin/static/ /etc/munin/static/
    
       <Location />
           AuthType Basic
           AuthName "Members Only"
           AuthUserFile /etc/apache2/.htpasswd
           require valid-user
       </Location>
    
       <Directory /etc/munin/static>
           Require all granted
       </Directory>
    
       ScriptAlias /munin-cgi/munin-cgi-graph /usr/lib/munin/cgi/munin-cgi-graph
       ScriptAlias /munin /usr/lib/munin/cgi/munin-cgi-html
       <Directory /usr/lib/munin/cgi>
           Require all granted
           <IfModule mod_fcgid.c>
               SetHandler fcgid-script
           </IfModule>
           <IfModule !mod_fcgid.c>
               SetHandler cgi-script
           </IfModule>
       </Directory>
    
       CustomLog /var/log/apache2/munin.example.org-access.log combined
       ErrorLog  /var/log/apache2/munin.example.org-error.log
   </VirtualHost>

3. htpasswd -c /etc/apache2/.htpasswd
4. a2ensite munin
5. a2dissite 000-default
6. a2enmod fcgid rewrite
7. service apache2 restart

To add nodes, all you need to do is this:

1. Install munin-node
   apt install munin-node
2. edit /etc/munin/munin and add a line allowing the master to connect, e.g.:
   

   allow ^10\.0\.3\.116$

3. Optionally, make it listen only on localhost:
   

   host 127.0.0.1

4. Restart the service.
   systemctl restart munin-node

On the master:

1. Add a new snippet to /etc/munin/munin.conf, e.g.:

   [Node1]
   address 1.2.3.4
   use_node_name yes

2. Restart the munin master service. service munin restart

On the nodes, enabled plugins are found in /etc/munin/plugins.

Available plugins are found in /usr/share/munin/plugins. Create symbolic links to the active configuration. Some of the plugins behave differently depending on how the links are named; the plugin files themselves should contain readable documentation for more info.

Plugins can be configured further via setting files in /etc/munin/plugin-conf.d. Here are some examples:

[diskstats]
  env.include_only sda,sdb,...

[df]
  env.include_re ^/$ ^/var/log$ ^/web$
  env.exclude_re ^/dev/shm$ ^/run/lock$ ^/sys/fs/cgroup$
 
[df_inode]
  env.include_re ^/$ ^/var/log$ ^/web$
  env.exclude_re ^/dev/shm$ ^/run/lock$ ^/sys/fs/cgroup$

A more secure way to connect a node to the master is via ssh. When connecting to a machine on the internet, ssh is a must have.

1. Create an SSH keypair on the host for user munin
   1. since the munin user can't login by default, create it as root and change ownership.
2. Move the .ssh folder to /var/lib/munin/ssh
3. Make sure it's owned by munin and not accessible to anyone else.
   

   # ls -la /var/lib/munin/ssh
   total 20
   drwx------  2 munin munin 4096 Mar  6 20:18 .
   drwxr-xr-x 13 munin munin 4096 Mar  8 14:15 ..
   -rw-------  1 munin munin  399 Mar  6 20:18 id_ed25519
   -rw-r--r--  1 munin munin   92 Mar  6 20:18 id_ed25519.pub
   -rw-r--r--  1 munin munin  666 Mar  6 20:21 known_hosts

4. Add the key to the node's .ssh/authorized_keys
5. configure ssh options in /etc/munin/munin.conf:
   

   ssh_options -o IdentityFile=/var/lib/munin/ssh/id_ed25519 -o UserKnownHostsFile=/var/lib/munin/ssh/known_hosts -o User=remoteuser -o PreferredAuthentications=publickey

6. A node section in /etc/munin/munin.conf might look like this:
   

   [node2.example.com]
       address ssh://node2.example.com/bin/nc 127.0.0.1 4949
       use_node_name yes

Though I would recommend the way outlined in the previous section, here's an example on how to automatically dig an SSH tunnel to a node system via an init.d script (since we have no systemd on Devuan).

1. Create a key pair on the munin master:
   ssh-keygen -t ed25519
2. Add the public key to the remote node's authorized_keys, e.g.:
   cat id_ed25519.pub » .ssh/authorized_keys
3. Create the script on the munin master:
   vi ssh_example
   

   #!/bin/sh
   ### BEGIN INIT INFO
   # Provides:          ssh_example
   # Required-Start:    $all
   # Required-Stop:
   # Default-Start:     2 3 4 5
   # Default-Stop:
   # Short-Description: your description here
   ### END INIT INFO
    
   PID_FILE="/var/run/ssh_example.pid"
    
   case "$1" in
   start)
       if [ -f $PID_FILE ]; then
           echo "SSH tunnel is already up apparently"
       else
           start-stop-daemon --start --exec /usr/bin/ssh --background --pidfile=$PID_FILE --make-pidfile -- -fNL 14949:localhost:4949 user@example.com
       fi
   ;;
    
   stop)
       start-stop-daemon --stop --pidfile=$PID_FILE
       rm $PID_FILE
   ;;
    
   restart)
       $0 stop
       $0 start
   ;;
    
   *)
       echo "Usage: $0 {start|stop|restart}"
       exit 1
   esac

4. chmod +x ssh_example
5. mv ssh_example /etc/init.d/
6. insserv /etc/init.d/ssh_example
7. update-rc.d ssh_example defaults
8. update-rc.d ssh_example enable
9. service ssh_example start

1. Make sure the node only listens on the loopback interface by removing this line from /etc/munin/munin-node.conf:
   

   host *

2. and adding either of these instead:
   

   host 127.0.0.1
   host ::1

3. it should only listen to localhost:
   

   allow ^127\.0\.0\.1$
   allow ^::1$

This one might be obvious. Add a new section to /etc/munin/munin.conf:

[example.com]
    address 127.0.0.1
    port 14949
    use_node_name yes

• LXC