SNMP is, and will be for a long time, one of the key protocols in monitoring. Widely used in hardware and appliance monitoring, and for sending events and alerts via TRAP or INFORM.
Now the problem of SNMPv1 or v2 is, that it has no real security. SNMPv3 offers that, but might cause you headaches, trying to understand, how it works.
With this post I want to explain, how snmptrapd can be used in a high availability setup, with the security of SNMPv3. I hope this gives you an inside and a quick start guide to try it out.

Security model

If you never worked with SNMPv3, just a quick introduction to authentication and security. There are no communities anymore, but a few other parameters are required:

  • securityName (username)
  • authProtocol (MD5 or SHA hashing algorithm)
  • authKey (secret to authenticate the peer)
  • privProtocol (AES or DES to encrypt the data)
  • privKey (secret to encrypt data)

Note: All keys are symmetric, which means both ends of the communication need to use the same keys (and protocol settings).
You can also disable authKey and/or privKey, but than why use SNMPv3? Check the manpage of snmptrapd for how to configure it in detail.

TRAP or INFORM?

With SNMPv3 a new notification type got introduced, called “INFORM”. The main differences between both types are:

  • INFORM is using a protocol to ensure delivery (Receiver sends an ack)
  • TRAP is working similar to v1/2, but its tricky with SNMPv3 security
  • INFORM has protection against message replay

The engineID

The tricky bit with SNMPv3 security in INFORMs come with the value of the engineID and engineBoots. Which is a generated value, that should be kept persistent for the authoritative node, which is:

  • TRAP sender – engineID of the sender has to be configured on receiver (every sender that might send traps)
  • INFORM receiver – engineID will be “learned” by the sender on first use (only user data for receiver)
  • walk/get target – discovered automatically and used for the session

From RFC 3414:

2.2 Replay Protection
   An authoritative SNMP engine is required to maintain the values of
   its snmpEngineID and snmpEngineBoots in non-volatile storage.

Basically, if you change the engineID or not increase the boot counter, you might not receive further INFORMS, until you reconfigured the sender. (Depends on the implementation, some just adapt and use the new values)

Persistence in net-snmp

Persistence is achieved with a locally written config file in /var/net-snmp/snmptrapd.conf or /var/lib/snmp/snmptrapd.conf, that is updated on every restart. This file will get read in addition to the normal config file, and can contain various (auto generated) users and the saved values.
Fair warning: When you start snmptrapd with the argument “-C” (capital C) the persistance file will not be loaded on start! It will get updated with a new engineID and engineBoots=1, but never been read.

Going high available

In the Icinga setup that I had to realize SNMPv3, we have a active/standby setup based on corosync and pacemaker. There is a DRBD cluster filesystem for various data, like RRD and Icinga logs, which can host such a file.
I made snmptrapd a cluster resource dependent on the cluster IP and the file system. There can’t be any load balancing or active/active with SNMPv3 security.
You can ensure snmptrapd is using a custom location for persistence, by putting the following into /etc/sysconfig/snmptrapd.options or /etc/default/snmptrapd

export SNMP_PERSISTENT_DIR=/data/icinga/drbd

Which will read and safe the file snmptrapd.conf to that location. Thats all you need to ensure a persistent engineID.

Example

Here is an example configuration derived from that setup.
/etc/snmp/snmptrapd.conf

createUser mrroboto SHA <THEAUTHKEY> AES <THESECRET>

/etc/sysconfig/snmptrapd.options

export SNMP_PERSISTENT_DIR=/data/icinga/drbd
# NOTE: Do NOT specify -C or persistent state WILL NOT LOAD
OPTIONS="-On -t -p /var/run/snmptrapd.pid"

/var/net-snmp/snmptrapd.conf (generated automatically)

#
# net-snmp (or ucd-snmp) persistent data file.
#
############################################################################
# STOP STOP STOP STOP STOP STOP STOP STOP STOP
#
# **** DO NOT EDIT THIS FILE ****
#
# STOP STOP STOP STOP STOP STOP STOP STOP STOP
############################################################################
#
# DO NOT STORE CONFIGURATION ENTRIES HERE.
# Please save normal configuration tokens for snmptrapd in SNMPCONFPATH/snmptrapd.conf.
# Only "createUser" tokens should be placed here by snmptrapd administrators.
# (Did I mention: do not edit this file?)
#
usmUser 1 3 0x80001f88804e432d76d2c1995700000000 "mrroboto" "mrroboto" NULL .1.3.6.1.6.3.10.1.1.3 0x5ac67b39801b206dbc2001ef5931139893090127 .1.3.6.1.6.3.10.1.2.4 0x5ac67b39801b206dbc2001ef59311398 ""
engineBoots 3
oldEngineID 0x80001f88804e432d76d2c1995700000000

Resources

Markus Frosch
Markus Frosch
Principal Consultant

Markus arbeitet bei NETWAYS als Principal Consultant und unterstützt Kunden bei der Implementierung von Nagios, Icinga und anderen Open Source Systems Management Tools. Neben seiner beruflichen Tätigkeit ist Markus aktiver Mitarbeiter im Debian Projekt.