AIX and USB memory sticks

Creating file systems with USB memory sticks

Tasks that involve text file data extraction, where performance is of the essence and the text files are to be placed on temporary storage, requires some thought. Typically, one could consider using a RAM disk (file system), a USB memory stick, or perhaps on a more permanent basis using SSD (solid state disk). RAM disk file systems are extremely fast, but at some point you need to move the files from the file system since you could lose everything if the file system is unmounted or the system is rebooted. Plus, you need to have enough memory (pinned) to create the RAM disk in the first place. For SSD, this would be the ideal solution, but these are not as cheap as the USB memory devices (when the requirement is temporary moveable storage). Another consideration is that you cannot place an SSD or RAM disk in your pocket and re-plug it into another machine like you can with a USB (memory) stick. These sticks are also known as:

• pen drives;
• memory sticks;
• USB sticks;
• flash drives; and
• thumb drives.

In this article, I will demonstrate how to create a USB memory stick file system (hereafter called USB stick) and compare the different speeds on text extraction using a 2MB CSV file. I am using AIX 7.1 with a 4GB Sandisk memory stick. These USB sticks can come in sizes up to 256GB.

Memory sticks are useful in my opinion because:

• They are ideal for fast data access.
• They can be used in data transfer where network performance is poor.
• They are great for scratch or temporary file systems.
• They are cheaper than other alternatives.

The only downside is that they do have a limited life in that the write cycles on these drives typically last around 8-10 years, before the memory cells wear out. But as they are cheap to buy, this is really not a concern as you would no doubt replace them within that time period anyway.

The AIX 6.1 documentation states that it supports the following branded USB sticks:

• Sandisk
• Lenovo
• Kingston

However, I have used other branded and non-branded USB sticks with no issues found.

USB sticks are ideal for creating scratch file systems where you want to transfer a large amount of data, via courier, for quick extraction to a remote destination sie cheaply because there is no good network link. As well, you could use them for data manipulation of large text files. USB sticks are great for fast reads but does not offer good performance when writing to them; you need to keep in mind what type of processing task you use it for.

Mounting a USB stick

Be sure to have the correct fileset(s) installed before inserting your USB stick. The following file-sets are required to access a USB stick:

# lslpp -L devices.usbif.08025002.rte
 Fileset Level State Type Description (Uninstaller)
devices.usbif.08025002.rte
7.1.0.15 A F USB Mass Storage Device
 Software 
# lslpp -L devices.common.IBM.usb.rte
  Fileset                      Level  State  Type  Description (Uninstaller)
  devices.common.IBM.usb.rte
                            7.1.0.15    A     F    USB System Software

Insert the USB stick and run cfgmgr to discover it:

# cfgmgr

Once discovered, you have two devices (the block and raw device):

ls -l *usbms0
cr--r--r--    1 root     system       44,  0 Aug 22 18:59 rusbms0
br--r--r--    1 root     system       44,  0 Aug 22 18:59 usbms0

It is also shown in the devices output:

# lsdev |grep usb
usb0       Available               USB System Software
usbhc0     Available 08-08         USB Host Controller (33103500)
usbhc1     Available 08-09         USB Host Controller (33103500)
usbhc2     Available 08-0a         USB Enhanced Host Controller (3310e000)
usbms0     Available 2.1           USB Mass Storage

For AIX 5.3, the first USB device is shown as flashdrive0.

At this point, I could treat the USB stick as a raw volume and write data to it. However, in this demonstration, I will mount the USB stick as a file system. I feel there is no need to create a log device for this USB mount, since the purpose of a scratch file system is that it is temporary and thus speed is of the essence when transferring data. To create the USB stick file system use the following command:

# mkfs -V jfs2 -o ea=v2 /dev/usbms0
mkfs: destroy /dev/usbms0 (yes)?
File system created successfully.
3927736 kilobytes total disk space.
Device /dev/usbms0:
  Standard empty filesystem
  Size:           7855472 512-byte (DEVBLKSIZE) blocks

The previous command initialized the USB stick. Select "Yes" to destroy, or rather initialize, the device. The file system type is of jfs2 type. I do not believe that a type of jfs2 really matters for this task, as the file system is not going to use the jfs2 log device. But as this is the norm when creating file systems, I created it as a jfs2. To ensure the file system is scalable, I specify that with the 'ea' option.

 In this example, the device to use is the USB stick inserted, which is usbms0.

For AIX 5.3, there is no NOLOG option in the mount command.

Next, mount the file system specifying that is it to be mounted without a log device; be sure to create the directory it is to be mounted on first:

# mkdir /usb_mnt
# mount -o log=NULL /dev/usbms0 /usb_mnt
# df -g |grep usb
/dev/usbms0        3.75      3.73    1%        4     1% /usb_mnt

If you wish to use a log device to mount the usb stick, use the inline log. This ensures that it is contained within the file system:

# mkfs -olog=INLINE,ea=v2 -Vjfs2 /dev/usbms0
mkfs: destroy /dev/usbms0 (yes)?
logform: Format inline log for  <y>?y
File system created successfully.
3912376 kilobytes total disk space.
Device /dev/usbms0:
  Standard empty filesystem
  Size:           7824752 512-byte (DEVBLKSIZE) blocks
# mkdir /usb_mnt
# mount -V jfs2 -o log=/dev/usbms0 /dev/usbms0 /usb_mnt

Unmounting a memory stick

Once you have finished using the a USB stick file system, unmount it, remove it, then delete the usbms0 device:

# umount /usb_mnt
# rmdev -dl usbms0

Data on the stick

In this demonstration, I am using a CSV text file called plaks.txt. It is an extract from a database, partial contents of the file is shown below:

alpha,uk01w,12001,jan,-2
bravo,ge01w,98801,jan,-3
charlie,se01w,98111,jan,0
delta,my01w,18811,jan,4
echo,sg01w,34131,janq,2

Earlier I suggested that reading from a USB stick is quick but writing to the USB stick is slower compared to internal disks. This can be somewhat verified by doing an example copy. First, I copied a file from the USB stick to an internal disk. Then, I copied the same file from the internal disk back to the USB stick using the timex command to show the timings of the copy commands. Before both copy operations are carried out, the file systems have been unmounted and re-mounted, so that the operation does not use the file system cache and thus does not blur the timings.

First, the file is copied from the USB stick onto an internal disk to the /holding directory, which resides on an internal disk:

# pwd
/usb_mnt
# timex cp plaks.txt /holding/
real 0.03

Next, I remove the file from the USB stick and unmount and remount the file systems.

Next, I copied the file back from /holding to the USB stick.

# cd /holding
# timex cp plaks.txt /usb_mnt/
real 0.06

The copy of the USB stick takes nearly half the time of the copy from an internal disk. I typically use the USB stick for text extraction. Using awk as an example on data extraction, I extract the pattern 'bravo' contained in the plaks.txt located on the USB stick:

# pwd
/usb_mnt
# timex awk '/bravo/' plaks.txt
real 5.50

Doing the same operation on an internal disk:

# pwd
/holding
# timex awk '/bravo/' plaks.txt
real 8.48

The text extraction is quicker on the stick; in fact, there is a difference of over 3 seconds. Here I am using a relatively small size text file. Though clearly the saving in time is greater when using larger sized exported file, which typically could be up to 8-10MB.

The above, in my opinion, justifies why I choose to use USB sticks for heavy test processing; however, as suggested earlier, if you need to write data you lose out big time. Here I create a 100MB file on the USB stick:

# pwd
/usb_mnt
# timex lmktemp myfile 100M
myfile
real 6.60

Next, I do it on an internal disk:

# pwd
/holding
# timex lmktemp myfile 100M
myfile
real 1.65

As you can see, there is a difference of over 5 seconds in the writing to a USB stick compared to an internal disk.

USB sticks are great for transferring data when there is no real alternative. I recall one incident, where one of our machines located in a remote region where the SAN disks were failing. These SAN disks contained work flow images. This machine did not have a sturdy network, and we wanted to get these images onto another remote machine for business users to be able to continue to work. The only method quickly available was to buy some 64GB USB sticks, mount them, and tar the images up on the USB sticks. The images were then extracted onto our other AIX box. This method worked with no data loss.

Mounting a previous initialized memory stick

If you are in possession of a memory stick that contains data what was previously mounted as a file system, to access it simply mount it. Be sure to create the mount point first. For example, assume the USB device is inserted in the second USB slot, then have the device discovered as usbms1:

# mkdir /usbstick
# mount -o log=NULL /dev/usbms0 /usbstick

Conclusion

Using USB sticks is a great when you want to create a fast read access scratch file system for processing or maybe a raw device to dump some files into for transport to another AIX system. USB sticks offer one solution where you have a poor network, and you need to get a large amount of data transferred.