I thought I’d pen this down right here as it took me a bit to really figure this out.

Problem: I have some Hosts I would like to monitor but I cannot access them directly (VPN also isn’t an option in this case), so I would like to monitor them using SSH, some directly, some behind a jumphost.

 

Usually the check_mk uses xinetd listening on Port 6556 only limited by allow_from in xinetd config and maybe iptables. This is fine in a closed, trusted environment but not really over public networks.

We could now either use VPN or tunnel the port through ssh port forwarding, but I found it more convenient just using ssh as a datasource program.

Preparing the nodes

We surely won’t use passwords for this, but rather a key with very limited capabilities.

So, first go to the monitoring site (make sure to do this as the monitoring user) and create a key pair:

OMD[site]:$ ssh-keygen -t ed25519
Generating public/private ed25519 key pair.
Enter file in which to save the key (/omd/sites/site/.ssh/id_ed25519): /omd/sites/site/.ssh/id_check_mk
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /omd/sites/site/.ssh/id_check_mk.
Your public key has been saved in /omd/sites/site/.ssh/id_check_mk.pub.
The key fingerprint is:
SHA256:uiD4Ozb7WRaDDDD7hzd/dWUUfQG+afGu88MGv6VkZM site@monitoringhost.example.com
The key's randomart image is:
+--[ED25519 256]--+
|. |
|o . . . . |
|.o . . . . . |
|oo. . . . .o |
|.o. . S. . +E |
| = . . o + ++ |
|= o o + .o +o+.|
| .+. = . . =o.+..|
| .+*o . o+Bo. |
+----[SHA256]-----+

Don’t use a passphrase. Now append the public key to the authorized_keys file on the monitored nodes. I am using ansible for this:

- name: add mod ssh-key
 authorized_key:
 user: root
 state: present
 key: "{{ lookup('file', '/root/.ssh/id_check_mk.pub') }}"
 key_options: 'command="/usr/bin/check_mk_agent"'

This results in:

command="/usr/bin/check_mk_agent" ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAXXXXXXT9PMJYIN4Mjdc9gsSAAAAAAxAZMhblN4+Mn18wh site@monitoringhost.example.com

Now you can ssh to that node from the monitoring host using the key, you should then get the output of the check_mk_agent.

Fine.

Case 1: Directly accessible node

In WATO, go to Host & Service Parameters => Datasource Programs => Individual program call instead of agent access.

We give a simple name and the command line using the <IP> macro.

ssh -tt -o StrictHostKeyChecking=no root@

Some hosts have issues spawning a tty, so we omit that with -tt. I also had some issues with the host keys which afford disabling StrictHostKeyChecking.

I am using this rule for all monitored hosts, therefor I don’t need any rules set up. If you only want to apply these rules to single hosts, just add their names below, or better create a rule as shown in the jumphost example below.

Case 2: Nodes behind jumphost

We use the same approach with a jumphost, just adding that host in between.

We extend our ssh line by -J jump@jumphostMake sure that you don’t use the root account on the jump host!

If you are using an older or different version of SSH which doesn’t support the -J switch, you need to do it using the old style -W version.

As I only want specific nodes to be monitored using the jumphost, I created another choice in the networking host tags, so all Hosts tagged with “No ping possible” are monitored using the jumphost.

Now another issue arises: Ping fails, which means that all services are being monitored properly while the host has the status “down”. So we need to change that, too with another rule, also tagged with the same tag above.

Go to Host & Service Parameters => Monitoring Configuration => Host Check Command and add a new rule.

Switch the command from PING to Use the status of the Check_MK Agent and chose the correct host tag.

Done.