electron14 » February 4, 2009

Daily Archives: February 4, 2009

Linux Storage

Using a Ramdisk For High I/O Applications

Published by:

I recently provided this solution for a system with high iowait. It’s a monitoring system with highly transient data, yet not entirely temporary data. The server we happened to put it on had plenty of memory and CPU, but only a mirrored set of drives (a blade), and the application isn’t really important enough to eat expensive SAN space.  The solution was to utilize the memory in the system as a ramdisk.

It’s a simple procedure. You create the ramdisk, then make an LVM logical volume from it (which is why we don’t use tmpfs). Occasionally you’ll snapshot it and backup the contents, and also back it up during a controlled shutdown. On startup, you simply re-create the ramdisk and restore the backup.

This solution should work for a variety of applications, however, you have to pay attention to certain circumstances, such as applications that might keep changes in memory rather than flushing to disk regularly (a.k.a. writeback cache, in which case you’ll want to shut down the app before performing the backup), as well as the obvious chance of losing the data collected after the last backup in the event of an unforeseen interruption such as power loss.

First off, you have to have ramdisk support in your kernel.  If support is compiled into the kernel (i.e. Redhat based installations), you’ll need to add the option “ramdisk_size” to the kernel line in /boot/grub/grub.conf (or menu.lst), with the size specified in KB. For example, “ramdisk_size=10485760” would give you 10GB ramdisks. Afterward, simply reboot and you should have /dev/ram0 thru /dev/ram15. Yes, it creates 16 devices by default, but it doesn’t eat your memory unless you use it.

I prefer to have ramdisk support built as a module (such as opensuse does), because you can just reload the module in order to add/resize ramdisks. To do this, you have to know the filename of the module, usually rd.ko or brd.ko.

server:/lib # find . -name brd.ko
./modules/2.6.25.18-0.2-default/kernel/drivers/block/brd.ko
./modules/2.6.25.5-1.1-debug/kernel/drivers/block/brd.ko

Then load the module:

server:/lib # modprobe brd rd_size=1048576

server:/lib # fdisk -l /dev/ram0

Disk /dev/ram0: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000

Of course you can add this module to the appropriate module configuration file (depending on your distribution) for future reboots.

Now that we’ve covered how the ramdisks work, here’s the script I’ve included in /etc/rc.local (Red Hat) to create/restore the ramdisk LVM volume at boot.

#create pv :
pvcreate /dev/ram0 /dev/ram1 /dev/ram2

#create vg :
vgcreate vg1 /dev/ram0 /dev/ram1 /dev/ram2

#create lv :
lvcreate -L 18G -n ramlv vg1

#create fs:
mkfs.ext2 /dev/vg1/ramlv

#mount lv :
mount -o noatime /dev/vg1/ramlv /mnt/ramlv

#restore data
cd /mnt/ramlv && tar -zxf /opt/ramlv.tar

#start the service that relies on the high performance ramdisk

/etc/init.d/zenoss start

The backup is run at shutdown and on a cron. The main reason I’m using tar with gzip is that it facilitates super fast restores, as the data I have gets a 5:1 compression ratio. With the disk being the major bottleneck in this particular server, I get roughly a 4-5x speed boost when copying the data back to ram from the compressed archive compared to copying from a file level disk backup. YMMV, another option is to simply rsync the data to disk on a schedule. With tar, the backups work harder, but don’t really impact the performance of the ramdisk while they’re running. Here’s the script:

#!/bin/bash

#logging
echo
echo “#############################################”
date
echo “#############################################”

#create snapshot:
/usr/sbin/lvcreate -L 5G -s -n ramsnap /dev/vg1/ramlv

#mount snapshot:
/bin/mount -o noatime /dev/vg1/ramsnap /mnt/snapshot

#logging
df -h /mnt/snapshot
echo TIME

#backup the only directory in /mnt/snapshot (named ‘perf’):
mv -f /opt/ramlv.tar /opt/ramlv-old.tar
cd /mnt/snapshot && time /bin/tar zcf /opt/ramlv.tar perf

#logging
ls -la /opt/ramlv.tar
/usr/sbin/lvdisplay /dev/vg1/ramsnap
/usr/sbin/lvs

#remove snapshot:
cd && /bin/umount /mnt/snapshot &&  /usr/sbin/lvremove -f /dev/vg1/ramsnap

And there you have it. Insane disk performance at your fingertips. A few things, you’ll want to ensure that your snapshot volume is sizable enough to hold any changes that may occur while you’re performing the backup (which is why I’ve got the logging commands in there). You’ll also probably want to keep a few backups in case the system dies while one is being performed, hence the ‘mv -f’ command in the script. Other than that, have fun, and feel free to share your experiences and improvements.