Caution: Your VTL results may vary
By Beth Pariseau, News Writer
10 Mar 2006 | SearchStorage.com
Interest in virtual tape libraries (VTL) and disk-based backup in general exploded in 2005. But as with any new technology, when evaluation and acquisition give way to practical implementation - when, in a way, the "honeymoon period" ends -- users encounter new pitfalls they may not have expected.
One such case happened at the American Institute of Physics (AIP). According to James Wonder, director of online technology, AIP implemented a Sepaton Inc. S2100 VTL between its 13 terabyte StorageTek D178 SAN and Qualstar Corp. tape library last year. But when the new data-hungry disk system was attached to the back end of his approximately 40 Sun Microsystems Inc. servers running Solaris 8 and 10, however, he uncovered a new bottleneck in his backup system -- the servers themselves.
"The VTL pulled data out so fast, it increased CPU overhead on my production servers," Wonder said. "Like my Web server, for example, our production machines get hammered. We hadn't been able to roll the VTL out in big chunks like we wanted to, and it was affecting my customers."
Wonder's solution to this problem was to implement local replication by Kashya Inc. in which an image of each production server is mounted on a separate Sun backup server, which then sends the data to the VTL.
"That way, my CPU on a customer-facing machine doesn't get affected and my network load doesn't get affected writing to the VTL," he said. "We also use Kashya to clone to tape after writing data to the Sepaton VTL."
Greg Schulz, analyst and founder of the StorageIO Group notes that this user is probably moving his bottleneck around. "In general, whenever you move or remove a bottleneck, in this case the slow tape in the backup process, you invariably will shift or cause a bottleneck elsewhere, unless you have taken adequate precautions."
"We have seen VTL implementation expose 'new' bottlenecks," said Brad O'Neill, senior analyst with the Taneja Group. "The tape drive used to be the slowest part of the backup environment, but post-VTL, the customer discovers that their network or CPU or disk arrays are too slow to feed the VTL at maximum performance."
"When I was evaluating VTL solutions, I certainly did try to gauge what adverse impact, if any, was the host processor experiencing while trying to keep up with the enormous appetite for data of the virtual library," said Mark Stewart, backup and recovery storage administrator at Randolph Air Force Base in Texas and a user of FalconStor Software Inc.'s VirtualTape Library.
"On the vast majority of my hosts, there was no such impact," he added. But, he said, a collection of "antique" [Windows] NT4 systems have been difficult to back up with the VTL.
"I am forced to back up these boxes across the front-side LAN," he said, "And their connections are paltry, anemic 100 Mb fast-Ethernet NICs. The bottleneck there is the network connection, not an unduly high processor workload."
Newly networked storage using a VTL, as opposed to direct-attached tape, may also be an issue, Schulz said. "If you are moving data over a network, you may be encountering CPU cycles being consumed by TCP/IP processing as well."
"It cannot be overstated how vitally important it is to have from the very beginning a dedicated team of storage engineers familiar with every aspect of the entire backup environment evaluating this technology," Stewart said, "in order to be absolutely certain that there are no 'gotcha!' moments after you've already spent a large amount of money on a new piece of equipment."
The experts weigh in
"Even though VTLs can theoretically let you unplug your tape library and plug them in instead, that may not be the best way to go about implementing one," said Curtis Preston, senior analyst for GlassHouse Technologies Inc..
Other possible VTL pitfalls, according to Preston, include backup software licensing issues.
"Depending on your backup product, there may be pricing challenges with how the software is licensed for a VTL, as opposed to a tape library," he warned. Preston advises looking at pricing for VTLs by capacity rather than by tape drives or cartridges, which is how Veritas Backup Exec works.
"But other backup software vendors may still charge for licensing per tape drive or even tape cartridge. One thing lots of people like to do with VTLs is make lots of really small virtual tapes -- that's something they could wind up paying a whole lot more for until backup software pricing catches up."
Another thing to watch out for, Preston said, is that treating a VTL exactly like a tape library doesn't mean it won't work, "but if you acknowledge, at least when setting it up, that you're not actually writing to tape, there might be things you can do differently to get the best performance from the VTL."
For example, many storage administrators use a technique when writing to tape libraries called multiplexing, which is the splitting up of streams of data to push the most data through to tape.
"It's not something you need to do with a VTL, and if you continue doing it when sending data to disk rather than tape, you might defeat the purpose of a VTL, which is faster restores," Preston said.
In general, "look at your backup process and configurations, and see if the reason you set them up that way still exists."
Another note to Tivoli Storage Manager (TSM) users -- TSM doesn't multiplex, and most VTLs will support 32- or 64 virtual tape drives with some outliers at 96 or more. Meanwhile, it's common with TSM to be backing up hundreds of clients. Many TSM clients might require another disk pool in front of the VTL to prevent bottlenecks, according to Preston.
"Finally, if they weren't copying tapes before, and in my opinion most haven't been, and just have been sending originals off site," Preston added, "You can't eject tape from the VTL, so you need to figure out how to create a copy. It may be you have to buy a backup software module you didn't use before. It may come down to, 'I hope you're good at scripting.' "
"Another factor is what type of files, big or small, are being backed up -- as there is more performance overhead in opening and closing files -- so backing up lots of small files will have more of a pronounced impact," Schulz said
Added O'Neill, "Other factors include tweaking and tuning of the backup software. For example, the customer might need to adjust the block sizes via the backup software running on the backup server, which can make a huge difference, or turn off software compression."