I’ve been trying to learn linux networking and virtualisation using a donated server in a remote lab. The server didn’t have an IP-KVM attached but it did have a working IPMI connection. Not that I’d need it of course; I was experimenting with network settings whilst ssh’d into a server that was four and a half thousand miles away. What’s the worst that could happen?
Of course the inevitable happened and I haplessly disabled my eth0 interface. I was locked out of the server, but was happy to learn that the Intelligent Platform Management Interface (IPMI) was a powerful tool indeed.
Both the IPMI protocol and HP’s iLO allow you to connect to the Baseboard Management Controller (BMC) on high-end servers. The BMC is a micro-controller embedded on the server motherboard which allows remote management without relying upon the server OS. The two specific functions I would need were the ability to remotely reset the server, and a view of the server console.
The console view is provided by IPMI Serial-over-Lan (SoL) but requires IPMI v2.0, sometimes called ‘lanplus’. The BMC normally has an embedded web server which is fine for doing power resets, but requires a java-based applet to do SoL. It might work for you but I have a strong preference for the command line.

IPMI Power reset and SoL

I started by installing the ipmitool package on my local ubuntu machine.

 sudo apt-get install ipmitool

You can check the current power status of the server:

 ipmitool -I lanplus -H 192.168.99.1 -U user -P pass chassis power status

And you can reset the server using this brutal command (please check you’ve got the right server):

 ipmitool -I lanplus -H 192.168.99.1 -U user -P pass chassis power reset

Then quickly active the SoL console session and watch the server boot:

 ipmitool -I lanplus -H 192.168.99.1 -U user -P pass password sol activate

However I wasn’t off the hook just yet, I could monitor the server bootstrapping itself, choosing local disk boot instead of PXE, but just before the bootloader menu, the screen went dark. Noooooooo!

Grub and Linux Serial Re-direction

I knew that I would see the full boot cycle if I were standing in front to monitor, but couldn’t really get why my SoL session went dark. I had failed to get the distinction between VGA/Monitor output and serial console output, and that having one output might preclude the other.
It turned out that although the BIOS was configured to support serial output, both Grub and Ubuntu were not configured to direct their output to the serial port (I’m assuming this is the default). The IPMI SoL connection can only see what the serial port could see which – in this case – was nothing.
The Ubuntu Serial Console Howto Guide shows you how to update both Ubuntu and Grub to redirect to ttyS0. I won’t copy and paste those changes, but once you apply them you’re done. Full serial console visibility for all stages in the boot cycle, and the ability to correct my networking error.
Sometimes you can encounter a ‘payload already active’ error message when trying to activate the SoL session. This is the equivalent of a busy line on a Cisco console server. If it’s one of your own stale sessions you could just use ‘ps’ to search for the process and kill it. If the session belongs to another user you can use the command below to free up the session:

 ipmitool -I lanplus -H 192.168.99.1 -U user -P pass sol deactivate

That’s it. Now I can get back to blundering through linux networking without needing to pester my colleagues for remote-hands assistance.
[amazon_link asins=’B00979DMSM,B00HUZ6OMQ,B01AFNBC3K,B00DJUHGHI’ template=’ProductCarousel’ store=’seamlnetwo-20′ marketplace=’US’ link_id=’1e8ba47d-f3f3-11e7-8ce3-cf4c57effd3c’]