Tuesday, June 16, 2009

SNMP v3 Configuration for non-SNMP-magi

Initial SNMP v3 Configuration

This works on Debian (and presumably Ubuntu). YMMV for other distros.

Start off with the following configuration in /etc/snmp/snmpd.conf
# create user 'initial' with auth pass 'setuppass'. This is used for the initial setup, then removed.
createUser initial MD5 setupauth DES setuppriv
# add 'initial' to group 'MyRWGroup' using the USM security model.
group MyRWGroup usm initial
# create a view 'viewall' that includes the entire OID tree.
view viewall included .1
# tie it all together. allow read and write access to view 'viewall' by users in group 'MyRWGroup'
access MyRWGroup "" usm authNoPriv exact viewall viewall none
Once this is done, restart snmpd and you should be able to test connectivity by walking the OID tree
snmpwalk -v3 -l authNoPriv -u initial -A setupauth localhost .
Assuming all is well so far, remove the 'createUser' line from the configuration file and restart snmpd. This prevents snmpd from recreating the 'initial' user every time is starts. You can now use the snmpusm command to create a new user 'cacti'
snmpusm -v3 -l authNoPriv -u initial -A setupauth localhost create cacti initial
Now use snmpvacm to create a group 'MyROGroup' and 'cacti'. Then create a new view 'allseeing' that can see the entire tree (this is actually the same as what is in the configuration file). Finally, create an access entry allowing 'MyROGroup' group members using USM (3) to connect via authNoPriv (2) using an exact context match (1). The 'allseeing' view is specified for reads, and 'none' is specified for writes and notifies.
snmpvacm -v3 -l authNoPriv -u initial -A setupauth localhost createSec2Group 3 cacti MyROGroup
snmpvacm -v3 -l authNoPriv -u initial -A setupauth localhost createView allseeing .1 80
snmpvacm -v3 -l authNoPriv -u initial -A setupauth localhost createAccess MyROGroup 3 2 1 allseeing none none
At this point you should be able to walk as cacti
snmpwalk -v3 -l authNoPriv -u cacti -A setupauth localhost .
You can also change cacti's Authentication password, but not as cacti. The reason for this is that 'cacti' currently only has read access to the tree. The initial user currently has the necessary write access to the tree. The Privacy password is also changed
snmpusm -v3 -l authNoPriv -u cacti -A setupauth localhost passwd -Ca setupauth authpass
Error in packet.
Reason: noAccess
snmpusm -v3 -l authNoPriv -u initial -A setupauth localhost passwd -Ca setupauth authpass cacti
SNMPv3 Key(s) successfully changed.
snmpusm -v3 -l authNoPriv -u initial -x DES -A setupauth localhost passwd -Cx setupauth privpass cacti
SNMPv3 Key(s) successfully changed.
Now we want to get rid of the 'initial' user. Before we can delete him we should create an administrative user that can make changes to the tree.
snmpusm -v3 -l authNoPriv -u initial -x DES -A setupauth localhost create admin initial
snmpusm -v3 -l authNoPriv -u initial -x DES -A setupauth localhost passwd setupauth secretpass admin
snmpvacm -v3 -l authNoPriv -u initial -A setupauth localhost createSec2Group 3 admin rwgrp
snmpvacm -v3 -l authNoPriv -u initial -A setupauth localhost createAccess rwgrp 3 2 1 allseeing allseeing none
snmpusm -v3 -l authNoPriv -u admin -a MD5 -n "" -A secretpass localhost delete initial
Some Notes

* When I tried creating access for rwgrp with 'priv' permissions I got timeouts (Timeout: No Response from localhost) from the daemon when trying to operate as admin. Not sure why, and the Internet was completely unhelpful in figuring out why.
* Once the above setup is complete, you can remove all of the initial configuration from the snmpd.conf file and restart the daemon. All the use information is stored in /var/lib/snmp/snmpd.conf, leaving a clean-looking /etc/snmp/snmpd.conf file.
* This is probably not the most secure implementation ever, you may want to do some more research and lock down the agent further.

Useful Resources
snmpvacm man page
snmpusm man page

Tuesday, June 9, 2009

Name that boot error!

I ran across an irritating boot error yesterday that basically sucked up my entire day resolving. The situation is this. I was testing a bare-metal recovery of a Debian Lenny server using bacula. I had restored a backup of the server onto new hardware and it was time to reboot the new machine and see what was broken. On boot, I received the following output (re-keyed from a camera-phone photo, may not be totally accurate).

Booting the kernel.
Loading, please wait...
mdadm: no devices found for /dev/md0
mdadm: no devices found for /dev/md1
[ ] device-mapper: table: 253:0: linear: dm-linear: Device lookup failed
device-mapper: release ioctl failed: Invalid argument
1 logical volume(s) in volume group "vg-root" now active
kinit: name_to_dev_t(/dev/sdb1) = sdb1(8,17)
kinit: trying to resume from /dev/sdb1
kinit: No resume image, doing normal boot...
resume: libgcrypt version: 1.4.1
mount: mounting /dev/mapper/vg--root-lv--root on /root failed: No such device
mount: mounting /dev on /root/dev failed: No such file or directory
mount: mounting /sys on /root/sys failed: No such file or directory
mount: mounting /proc on /root/proc failed: No such file or directory
Target filesystem doesn't have /sbin/init.
No init found. try passing init= bootarg.
It then dumped me into an initramfs prompt. I spent the day trying to get the broken system to boot, when the simple solution eluded me. The root problem here is that the kernel cannot assemble the md arrays. In my case, this was caused because my raid partitions were not labeled with raid superblocks. I managed to do this by using the --assemble option in mdadm when I built my arrays. The correct option is --create, which writes superblocks properly. Once I figured this out, I "created" the arrays from a Knoppix disk (Ubuntu works well too), rebooted, and voila, the system boots right up.

From what I have read, you can safely "create" an array over top of the existing array and not lose data. mdadm is careful to do the right thing. That said, don't take my word for it. I tried it on an array that was restored from backup. It worked for me, but if it had not I wasn't out more than 10 minutes of my own time.

Interestingly, I was able to get the system to boot from the broken initramfs prompt. Here is the basic procedure. I did this yesterday and I'm going from memory and notes scribbled on a scrap of paper, so YMMV here as well.
  1. Build md arrays using mdadm
  2. Using the lvm shell, do a pvscan (to redetect LVM structures properly)
  3. Using the lvm shell, activate the logical volumes (e.g. lvchange -ay /dev/mapper/vg--root-lv--root)
  4. Mount / on /root. Must be mounted READ-ONLY!
  5. Mount dev, proc, and sys under /root
  6. Copy the contents of /dev to /root/dev
  7. Type exit to escape the initramfs, this should dump you to a shell. You will get a kernel panic if everything isn't set up just right.
  8. Run /etc/init.d/rcS to startup the system.
  9. Run /etc/init.d/rc 2 to bring the system to a somewhat functioning state. I was actually able to get most everything working like this, but it is obviously not a state to leave a server in, since a reboot will bring you back to square one.

Friday, May 29, 2009

LVM + XFS + VirtualBox - Resizing a guest partition

I've been doing a lot of work with virtualization lately and I am working on what I hope will become the great solution I've always wanted. The current configuration on the Host machine is a RAID0 array, with LVM logical volumes on top of that for each of my virtual machines. Each LV used for a Virtual Machine is formatted in XFS; from what I have read, it offers better performance with large files.

My Logical Volumes created for my VMs are created with a conservative sized disk. This allows plenty of free space on the LVM VG for future VMs, snapshots, scratch volumes, and expansion of existing LVs. That's right, it is possible to increase the size of all the disk components in this mess (Logical Volume, XFS Partition, Virtual Machine VDI, VM filesystem). For the most part this is a pretty painless process. The only real nasty part that involves downtime is resizing the VDI, which isn't really resizing at all. Rather, we will create a new larger VDI and copy the old disk over.

Here is the process:

1. Resize the logical volume on the host machine (if necessary).
lvextent -L+3G /dev/mapper/vg--vm-lv--ftp (extends ftp lv by 3GB)

2. Resize the filesystem on the host machine (if you resized the lv).
xfs_growfs /var/vm/ftp

3. Create a scratch lv on the host machine. This will be used to hold the resized clone, so make it large enough.
lvcreate -L 22G -n scratch vg-vm

4. Create a filesystem on scratch volume.
mkfs.xfs /dev/mapper/vg--vm-scratch

5. Create a mount point and mount the filesystem. Give vboxuser ownership.
mkdir /tmp/scratch
mount /dev/mapper/vg--vm-scratch /tmp/scratch
chown vboxuser /tmp/scratch

6. Change to vboxuser and create a new VDI of the desired size.
su vboxuser
VBoxManage createhd --filename /tmp/scratch/scratch.vdi --size 20480

7. Register scratch disk in virtualbox and verify
VBoxManage openmedium disk /tmp/scratch/scratch.vdi
VBoxManage list hdds

8. Register the gparted iso
VBoxManage openmedium dvd /root/gparted-live-0.4.5-2.iso

9. Shutdown the VM.
10. Add new VDI to VM and mount the gparted disk.
VBoxManage modifyvm "ftp" --hdb /tmp/scratch/scratch.vdi --dvd /root/gparted-live-0.4.5-2.iso

11. Start the VM, gparted should boot. Follow the default prompts to get to the desktop.
VBoxHeadless -startvm "ftp"

12. Open a prompt. Verify that the source (old) disk is hda and the destination disk (new) is hdb. Then use dd to copy hda to hdb. When finished, reread the partition table on the new disk.
sfdisk -l
dd if=/dev/hda of=/dev/hdb
hdparm -z /dev/hdb

13. Open gparted and use it to grow the disk partition you need to be bigger. Apply the changes, then shutdown the VM.
14. Remove the dvd and both drives. Then re-add the new VDI as hda. Startup the VM to verify that it boots cleanly.
VBoxManage modifyvm "ftp" --dvd none --hdb none --hda none
VBoxManage modifyvm "ftp" --hda /tmp/scratch/scratch.vdi
VBoxHeadless -startvm "ftp"

15. Set hda to none on VM. Close the two hard drive images.
VBoxManage modifyvm "ftp" --hda none
VBoxManage closemedium disk /tmp/
VBoxManage closemedium disk /var/vm/ftp/ftp.vdi

16. Move the scratch disk to the resized lv.
mv /tmp/scratch/scratch.vdi /var/vm/ftp/ftp.vdi

17. Re-open the new disk and add back into the VM
VBoxManage openmedium disk /var/vm/ftp/ftp.vdi
VBoxManage modifyvm "ftp" --hda /var/vm/ftp/ftp.vdi

18. Start VM
VBoxHeadless -startvm "ftp"

19. Unmount the scratch partition on the host machine and remove the lv to tidy up.
umount /tmp/scratch/
lvremove /dev/mapper/vg--vm-scratch

Friday, May 15, 2009

Using AES on the Cisco 837

While trying to setup an IPSec SA on a Cisco 837 router I ran across an annoying problem. The solution is not obvious (to me) and google wasn't very helpful, so I figure it is worth mentioning. When trying to configure a transform set using esp-aes I got the following message:

Cisco837(config)#crypto ipsec transform-set my-ts esp-aes 256
Transform ecessa-ts disabled because esp-aes is not supported by encryption hardware

Re-checking the 837 datasheet, it does support AES in software. Setting 128 or 192 for the cipher length also had the same result. I also tried setting up AES as an ISAKMP encryption parameter and also got an error:

Cisco837(config-isakmp)#enc aes 256
May 15 15:38:03.704: %CRYPTO-4-ENC_METHOD_NOT_SUPPORTED: Invalid encryption method for IKE policy 10

Google was oddly unhelpful, until I searched the comp.dcom.sys.cisco usenet group and found this thread. Apparently, to use AES in software on the 837, you need to disable the hardware 3des engine using the following global configuration command:

no crypto engine accelerator

After that, specifying AES works fine.

Sunday, May 10, 2009

The WS-C3524-XL isn't a good choice for CCNA practice anymore.

I am in the process of studying for the CCNA and I have a Cisco 3524 in my arsenal that I got for cheap before I really had an idea of what gear to buy. It is nice to have the switch around to complete my three switch lab configurations, but as I get deeper into my studying I am finding that there are some significant differences between the 3524-XL and newer Cisco switches that can make using this switch a bit of a headache. The problem is that this switch is EoL and Cisco stopped supporting this switch in IOS 12.0. Presumably these shortcomings apply to the 3548-XL and 2900-XL devices as well. For the sake of comparison, my other switches are 2950s and 3550s, which are much more capable/recent. here is a list of shortcomings that I have identified thus far; I will update the list as I find more.

  1. VLAN database - The newer IOS versions place VLAN and VTP configuration inside of configuration mode; namely the vlan and vtp global commands. At first glance, these commands appear to be missing from the 3524 interface, but really they are just placed elsewhere. These commands are located in the vlan database privileged-mode interface. Access this mode and you can configure VLANs and VTP to your hearts content. The command syntax is mostly the same too.
  2. switchport nonegotiate - This command does not exist on the 3524 IOS.
  3. switchport mode dynamic - This command does not exist on the 3524 IOS. Your choices are trunk or access. You also can not connect a 3524 trunk port to another switch (2950 in my case) configured for a dynamic mode. The other switch will freak.
  4. switchport voice - The 3524 does support this command, but it does not appear to work properly. I connected [linux] notebooks to voice ports on a connected 3524 and 2950 and configured 802.1q tagging interfaces. The tagged IP addresses could not communicate, but the untagged (access) IP addresses could. Repeating this with a 2950 and 3550 connected works just fine for tagged and untagged frames. I was able to approximate the behavior by setting the 3524 port to trunk mode, but this would probably not be acceptable in a prduction environment.
  5. show interface trunk - This very useful troubleshooting command does not exist on the 3524 IOS.

There are probably more I have forgotten. Bottom line, the 3524 and similar switches are okay for an extra switch if you can get them dirt cheap (<$50), but don't buy them with the expectation that you will be able to study everything on the ICND2/CCNA exam.

Friday, March 20, 2009

The Numbers

Ethernet
1518 Maximum Ethernet frame length in bytes
64 Minimum Ethernet frame length in bytes
14 Ethernet header length in bytes
46 Minimum Ethernet data length in bytes
1500 Maximum Ethernet data length in bytes
4 Ethernet trailer length in bytes
8 Ethernet Preamble length in bytes (inc. SoF Delimiter)
12 Interframe Gap in bytes

802.1Q
1522 Maximum Ethernet frame length w/ 802.1Q tagging
18 Ethernet header w/ 802.1Q tagging
4 802.1Q tag length in bytes

IP
20 Minimum IP header length in bytes

TCP
20 Minimum TCP header length in bytes

GRE
4 Minimum GRE header length in bytes
1476 GRE MTU in bytes, assuming 1500 Ethernet MTU
1436 GRE encapsulated TCP MSS in bytes

Sources:
http://en.wikipedia.org/wiki/Ethernet
http://sd.wareonearth.com/~phil/net/overhead/
http://www.ethermanage.com/ethernet/ethernet.html

Tuesday, March 3, 2009

ISAKMP: reserved not zero on payload 5!

I've been running into this issue at work, trying to connect an OpenSWAN peer to a Cisco PIX 501. I set both OpenSWAN the PIX to use aes and sha for ISAKMP encryption and hashing. When I establish the tunnel SA using the PIX as initiator, everything works fine. When initiating using OpenSWAN, the SA is not established. Checking the debug output on the PIX shows the PIX accepting the proposal from OpenSWAN, then failing later in the ISAKMP negotiation with the following error message.

ISAKMP: reserved not zero on payload 5!
ISAKMP: malformed payload

Googling around for the problem led to a lot of suggestions that this error is caused by an mismatched shared secret. This is certainly not the case for me, since the SA establishes just fine when set to use 3DES for Phase 1 encryption. I think the shared secret solution on the web is actually a mis-read of the Cisco documentation; they say, "This means that the ISAKMP keys do not match." The fact that 3DES works fine led us to discover that the ike parameter in my OpenSWAN configuration file was the problem. My config file looked like this:

conn Site50_common
auto=add
leftsubnet=192.168.20.0/24
leftid=20.20.20.2
right=50.50.50.2
rightsubnet=192.168.50.0/24
rightid=50.50.50.2
keyingtries=2
aggrmode=no
pfs=no
ikelifetime=8h
ike=aes-sha1-modp1536
auth=esp
esp=aes-sha1-96
authby=secret
dpddelay=30
dpdtimeout=60
dpdaction=restart

It appears that using ike=aes-sha1-modp1536 in OpenSWAN does not specify the 128-bit AES encryption expected by the PIX when the ISAKMP policy encryption is set to aes. I changed the OpenSWAN configuration file to specify aes128, and the tunnel now comes up immediately when the OpenSWAN client initiates the SA. Here is my updated OpenSWAn config file with the relevant line highlighted.

conn Site50_common
auto=add
leftsubnet=192.168.20.0/24
leftid=20.20.20.2
right=50.50.50.2
rightsubnet=192.168.50.0/24
rightid=50.50.50.2
keyingtries=2
aggrmode=no
pfs=no
ikelifetime=8h
ike=aes128-sha1-modp1536
auth=esp
esp=aes128-sha1-96
authby=secret
dpddelay=30
dpdtimeout=60
dpdaction=restart

And the [now] successful PIX debug output...

ISAKMP (0): processing SA payload. message ID = 0

ISAKMP (0): Checking ISAKMP transform 0 against priority 1 policy
ISAKMP: life type in seconds
ISAKMP: life duration (basic) of 28800
ISAKMP: encryption AES-CBC
ISAKMP: hash MD5
ISAKMP: auth pre-share
ISAKMP: default group 5
ISAKMP: keylength of 128
ISAKMP (0): atts are not acceptable. Next payload is 3
ISAKMP (0): Checking ISAKMP transform 1 against priority 1 policy
ISAKMP: life type in seconds
ISAKMP: life duration (basic) of 28800
ISAKMP: encryption AES-CBC
ISAKMP: hash SHA
ISAKMP: auth pre-share
ISAKMP: default group 5
ISAKMP: keylength of 128
ISAKMP (0): atts are acceptable. Next payload is 3
ISAKMP (0): processing vendor id payload

ISAKMP (0): processing vendor id payload

ISAKMP (0): remote peer supports dead peer detection

ISAKMP (0): processing vendor id payload

ISAKMP (0): processing vendor id payload

ISAKMP (0:0): vendor ID is NAT-T
ISAKMP (0): processing vendor id payload

ISAKMP (0): processing vendor id payload

ISAKMP (0:0): vendor ID is NAT-T
ISAKMP (0): processing vendor id payload

ISAKMP (0): SA is doing pre-shared key authentication using id type ID_IPV4_ADDR
return status is IKMP_NO_ERROR
crypto_isakmp_process_block:src:20.20.20.2, dest:50.50.50.2 spt:500 dpt:500
OAK_MM exchange
ISAKMP (0): processing KE payload. message ID = 0

ISAKMP (0): processing NONCE payload. message ID = 0

return status is IKMP_NO_ERROR
crypto_isakmp_process_block:src:20.20.20.2, dest:50.50.50.2 spt:500 dpt:500
OAK_MM exchange
ISAKMP (0): processing ID payload. message ID = 0
ISAKMP (0): processing HASH payload. message ID = 0
ISAKMP (0): SA has been authenticated

ISAKMP (0): ID payload
next-payload : 8
type : 1
protocol : 17
port : 500
length : 8
ISAKMP (0): Total payload length: 12
return status is IKMP_NO_ERROR
ISAKMP (0): sending INITIAL_CONTACT notify
ISAKMP (0): sending NOTIFY message 24578 protocol 1
VPN Peer: ISAKMP: Added new peer: ip:20.20.20.2/500 Total VPN Peers:1
VPN Peer: ISAKMP: Peer ip:20.20.20.2/500 Ref cnt incremented to:1 Total VPN Peers:1
crypto_isakmp_process_block:src:20.20.20.2, dest:50.50.50.2 spt:500 dpt:500
OAK_QM exchange
oakley_process_quick_mode:
OAK_QM_IDLE
ISAKMP (0): processing SA payload. message ID = 61453885

ISAKMP : Checking IPSec proposal 0

ISAKMP: transform 0, ESP_AES
ISAKMP: attributes in transform:
ISAKMP: encaps is 1
ISAKMP: SA life type in seconds
ISAKMP: SA life duration (basic) of 28800
ISAKMP: authenticator is HMAC-SHA
ISAKMP: key length is 256IPSEC(validate_proposal): transform proposal (prot 3, trans 12, hmac_alg 2) not supported

ISAKMP (0): atts not acceptable. Next payload is 3
ISAKMP: transform 1, ESP_AES
ISAKMP: attributes in transform:
ISAKMP: encaps is 1
ISAKMP: SA life type in seconds
ISAKMP: SA life duration (basic) of 28800
ISAKMP: authenticator is HMAC-SHA
ISAKMP: key length is 192IPSEC(validate_proposal): transform proposal (prot 3, trans 12, hmac_alg 2) not supported

ISAKMP (0): atts not acceptable. Next payload is 3
ISAKMP: transform 2, ESP_AES
ISAKMP: attributes in transform:
ISAKMP: encaps is 1
ISAKMP: SA life type in seconds
ISAKMP: SA life duration (basic) of 28800
ISAKMP: authenticator is HMAC-SHA
ISAKMP: key length is 128
ISAKMP (0): atts are acceptable.IPSEC(validate_proposal_request): proposal part #1,
(key eng. msg.) dest= 50.50.50.2, src= 20.20.20.2,
dest_proxy= 192.168.50.0/255.255.255.0/0/0 (type=4),
src_proxy= 192.168.20.0/255.255.255.0/0/0 (type=4),
protocol= ESP, transform= esp-aes esp-sha-hmac ,
lifedur= 0s and 0kb,
spi= 0x0(0), conn_id= 0, keysize= 128, flags= 0x4

ISAKMP (0): processing NONCE payload. message ID = 61453885

ISAKMP (0): processing ID payload. message ID = 61453885
ISAKMP (0): ID_IPV4_ADDR_SUBNET src 192.168.20.0/255.255.255.0 prot 0 port 0
ISAKMP (0): processing ID payload. message ID = 61453885
ISAKMP (0): ID_IPV4_ADDR_SUBNET dst 192.168.50.0/255.255.255.0 prot 0 port 0IPSEC(key_engine): got a queue event...
IPSEC(spi_response): getting spi 0x3bb5df50(1001774928) for SA
from 20.20.20.2 to 50.50.50.2 for prot 3

return status is IKMP_NO_ERROR
crypto_isakmp_process_block:src:20.20.20.2, dest:50.50.50.2 spt:500 dpt:500
OAK_QM exchange
oakley_process_quick_mode:
OAK_QM_AUTH_AWAIT
ISAKMP (0): Creating IPSec SAs
inbound SA from 20.20.20.2 to 50.50.50.2 (proxy 192.168.20.0 to 192.168.50.0)
has spi 1001774928 and conn_id 3 and flags 4
lifetime of 28800 seconds
outbound SA from 50.50.50.2 to 20.20.20.2 (proxy 192.168.50.0 to 192.168.20.0)
has spi 1662679932 and conn_id 4 and flags 4
lifetime of 28800 secondsIPSEC(key_engine): got a queue event...
IPSEC(initialize_sas): ,
(key eng. msg.) dest= 50.50.50.2, src= 20.20.20.2,
dest_proxy= 192.168.50.0/255.255.255.0/0/0 (type=4),
src_proxy= 192.168.20.0/255.255.255.0/0/0 (type=4),
protocol= ESP, transform= esp-aes esp-sha-hmac ,
lifedur= 28800s and 0kb,
spi= 0x3bb5df50(1001774928), conn_id= 3, keysize= 128, flags= 0x4
IPSEC(initialize_sas): ,
(key eng. msg.) src= 50.50.50.2, dest= 20.20.20.2,
src_proxy= 192.168.50.0/255.255.255.0/0/0 (type=4),
dest_proxy= 192.168.20.0/255.255.255.0/0/0 (type=4),
protocol= ESP, transform= esp-aes esp-sha-hmac ,
lifedur= 28800s and 0kb,
spi= 0x631a7b7c(1662679932), conn_id= 4, keysize= 128, flags= 0x4

VPN Peer: IPSEC: Peer ip:20.20.20.2/500 Ref cnt incremented to:2 Total VPN Peers:1
VPN Peer: IPSEC: Peer ip:20.20.20.2/500 Ref cnt incremented to:3 Total VPN Peers:1
return status is IKMP_NO_ERROR