Tuesday, June 16, 2009

SNMP v3 Configuration for non-SNMP-magi

Initial SNMP v3 Configuration

This works on Debian (and presumably Ubuntu). YMMV for other distros.

Start off with the following configuration in /etc/snmp/snmpd.conf
# create user 'initial' with auth pass 'setuppass'. This is used for the initial setup, then removed.
createUser initial MD5 setupauth DES setuppriv
# add 'initial' to group 'MyRWGroup' using the USM security model.
group MyRWGroup usm initial
# create a view 'viewall' that includes the entire OID tree.
view viewall included .1
# tie it all together. allow read and write access to view 'viewall' by users in group 'MyRWGroup'
access MyRWGroup "" usm authNoPriv exact viewall viewall none
Once this is done, restart snmpd and you should be able to test connectivity by walking the OID tree
snmpwalk -v3 -l authNoPriv -u initial -A setupauth localhost .
Assuming all is well so far, remove the 'createUser' line from the configuration file and restart snmpd. This prevents snmpd from recreating the 'initial' user every time is starts. You can now use the snmpusm command to create a new user 'cacti'
snmpusm -v3 -l authNoPriv -u initial -A setupauth localhost create cacti initial
Now use snmpvacm to create a group 'MyROGroup' and 'cacti'. Then create a new view 'allseeing' that can see the entire tree (this is actually the same as what is in the configuration file). Finally, create an access entry allowing 'MyROGroup' group members using USM (3) to connect via authNoPriv (2) using an exact context match (1). The 'allseeing' view is specified for reads, and 'none' is specified for writes and notifies.
snmpvacm -v3 -l authNoPriv -u initial -A setupauth localhost createSec2Group 3 cacti MyROGroup
snmpvacm -v3 -l authNoPriv -u initial -A setupauth localhost createView allseeing .1 80
snmpvacm -v3 -l authNoPriv -u initial -A setupauth localhost createAccess MyROGroup 3 2 1 allseeing none none
At this point you should be able to walk as cacti
snmpwalk -v3 -l authNoPriv -u cacti -A setupauth localhost .
You can also change cacti's Authentication password, but not as cacti. The reason for this is that 'cacti' currently only has read access to the tree. The initial user currently has the necessary write access to the tree. The Privacy password is also changed
snmpusm -v3 -l authNoPriv -u cacti -A setupauth localhost passwd -Ca setupauth authpass
Error in packet.
Reason: noAccess
snmpusm -v3 -l authNoPriv -u initial -A setupauth localhost passwd -Ca setupauth authpass cacti
SNMPv3 Key(s) successfully changed.
snmpusm -v3 -l authNoPriv -u initial -x DES -A setupauth localhost passwd -Cx setupauth privpass cacti
SNMPv3 Key(s) successfully changed.
Now we want to get rid of the 'initial' user. Before we can delete him we should create an administrative user that can make changes to the tree.
snmpusm -v3 -l authNoPriv -u initial -x DES -A setupauth localhost create admin initial
snmpusm -v3 -l authNoPriv -u initial -x DES -A setupauth localhost passwd setupauth secretpass admin
snmpvacm -v3 -l authNoPriv -u initial -A setupauth localhost createSec2Group 3 admin rwgrp
snmpvacm -v3 -l authNoPriv -u initial -A setupauth localhost createAccess rwgrp 3 2 1 allseeing allseeing none
snmpusm -v3 -l authNoPriv -u admin -a MD5 -n "" -A secretpass localhost delete initial
Some Notes

* When I tried creating access for rwgrp with 'priv' permissions I got timeouts (Timeout: No Response from localhost) from the daemon when trying to operate as admin. Not sure why, and the Internet was completely unhelpful in figuring out why.
* Once the above setup is complete, you can remove all of the initial configuration from the snmpd.conf file and restart the daemon. All the use information is stored in /var/lib/snmp/snmpd.conf, leaving a clean-looking /etc/snmp/snmpd.conf file.
* This is probably not the most secure implementation ever, you may want to do some more research and lock down the agent further.

Useful Resources
snmpvacm man page
snmpusm man page

Tuesday, June 9, 2009

Name that boot error!

I ran across an irritating boot error yesterday that basically sucked up my entire day resolving. The situation is this. I was testing a bare-metal recovery of a Debian Lenny server using bacula. I had restored a backup of the server onto new hardware and it was time to reboot the new machine and see what was broken. On boot, I received the following output (re-keyed from a camera-phone photo, may not be totally accurate).

Booting the kernel.
Loading, please wait...
mdadm: no devices found for /dev/md0
mdadm: no devices found for /dev/md1
[ ] device-mapper: table: 253:0: linear: dm-linear: Device lookup failed
device-mapper: release ioctl failed: Invalid argument
1 logical volume(s) in volume group "vg-root" now active
kinit: name_to_dev_t(/dev/sdb1) = sdb1(8,17)
kinit: trying to resume from /dev/sdb1
kinit: No resume image, doing normal boot...
resume: libgcrypt version: 1.4.1
mount: mounting /dev/mapper/vg--root-lv--root on /root failed: No such device
mount: mounting /dev on /root/dev failed: No such file or directory
mount: mounting /sys on /root/sys failed: No such file or directory
mount: mounting /proc on /root/proc failed: No such file or directory
Target filesystem doesn't have /sbin/init.
No init found. try passing init= bootarg.
It then dumped me into an initramfs prompt. I spent the day trying to get the broken system to boot, when the simple solution eluded me. The root problem here is that the kernel cannot assemble the md arrays. In my case, this was caused because my raid partitions were not labeled with raid superblocks. I managed to do this by using the --assemble option in mdadm when I built my arrays. The correct option is --create, which writes superblocks properly. Once I figured this out, I "created" the arrays from a Knoppix disk (Ubuntu works well too), rebooted, and voila, the system boots right up.

From what I have read, you can safely "create" an array over top of the existing array and not lose data. mdadm is careful to do the right thing. That said, don't take my word for it. I tried it on an array that was restored from backup. It worked for me, but if it had not I wasn't out more than 10 minutes of my own time.

Interestingly, I was able to get the system to boot from the broken initramfs prompt. Here is the basic procedure. I did this yesterday and I'm going from memory and notes scribbled on a scrap of paper, so YMMV here as well.
  1. Build md arrays using mdadm
  2. Using the lvm shell, do a pvscan (to redetect LVM structures properly)
  3. Using the lvm shell, activate the logical volumes (e.g. lvchange -ay /dev/mapper/vg--root-lv--root)
  4. Mount / on /root. Must be mounted READ-ONLY!
  5. Mount dev, proc, and sys under /root
  6. Copy the contents of /dev to /root/dev
  7. Type exit to escape the initramfs, this should dump you to a shell. You will get a kernel panic if everything isn't set up just right.
  8. Run /etc/init.d/rcS to startup the system.
  9. Run /etc/init.d/rc 2 to bring the system to a somewhat functioning state. I was actually able to get most everything working like this, but it is obviously not a state to leave a server in, since a reboot will bring you back to square one.