The role of SCSI diagnostic test tools in the iSCSI manufacturer environment - Mike Jones, PTI

Lakewood, CO - April 9,2002    As iSCSI begins moving from designs to real world products, diagnostic tools are needed for a variety of purposes. This article will address a few real-world experiences gathered while working in this new environment.

What are the issues?

At the highest level, the question is "is this storage subsystem working?"  Does the computer system recognize the disk on the other end of the wire? Is the capacity of the disk readable? Can the inquiry data be shown? Can the disk write and read data reliably?

Once communication is established and verified, lower-level functions need to be confirmed. Can the write cache on the drive be turned on/off? Can new firmware be downloaded into the drive? Does the entire storage subsystem respond in a reliable way when an error occurs? These errors could be drive related (a drive failure), or system related (an illegal command sent from a software application).

Device / Firmware development

In the real world, disk drives do not always operate strictly according to standards. Will your storage system crash or misbehave if a drive has a peculiarity? For instance, in experimenting with an iSCSI ->Fibre Channel bridge/router this week I discovered a particular brand of disk drive that did not support the SCSI command that the iSCSI bridge was relying on when it did fibre channel device discovery. The drive failed this command, and the bridge decided that that rack of drives was not there. Invisible drives!

By using a controlled environment SCSI design tool (PTI’s SCSI toolbox32) we were able to quickly ascertain what the offending SCSI command was, duplicate that command and collect detailed information about how the drive was failing. We then took this information to the software engineers at the iSCSI bridge company, they made changes to their software, and voila – within 30 minutes our bridge could now use Hitachi fibre channel drives!

Functional and performance testing

The SCSI toolbox32 provides several "layers" of testing needed for iSCSI work. Its hot bus scanning allows discovering devices added to or removed from the iSCSI connection. Once a drive is discovered any SCSI command can be tested. In theory any legal or illegal SCSI command should be supported in the iSCSI environment. In today’s reality we are dealing with bridges accomplishing the protocol conversion between iSCSI and SCSI/FC. Any time there is protocol conversion there is a possibility for errors, and the SCSI toolbox32 helps identify those errors. Since it generates known good (or known bad) SCSI commands, the bridge conversion process can be completely tested and understood. SCSI compliance tests can be used to insure that all SCSI 2 and SCSI 3 commands are supported correctly.

Once command compliance is assured, testing can move into a performance phase. Writes and reads of varying blocks per transfer can be sent to one or more drives, from one or more source computers. Raw "best case" performance can be measured to one drive. "Real world" performance can be measured using multiple synchronized computers sending multiple data streams to one or more drives or volumes. Tests running 128 deep queued commands to multiple drives can easily generate enough data to completely swamp the iSCSI subsystem for "torture" type testing.

Surround your unknowns with knowns

In summary, testing an iSCSI HBA or an iSCSI->SCSI/FC bridge or router is easily accomplished with the following pieces:

1. A test tool that can generate known good SCSI traffic, and can eloquently deal with and report all data gathered during any error condition.

2. A known good SCSI or fibre channel disk drive.

In between these two "knowns" is placed the iSCSI HBA or iSCSI->SCSI/FC bridge or router. The theory is then "if something doesn’t work right, it’s the HBA or the bridge or the cables". As I mentioned above about the "invisible drives", this test setup can provide for very fast identification and correction of bugs.

One more example

In closing, another example came up when we used the SCSI toolbox32 to send an INQUIRY command that asked for 6 bytes of data to be returned (a perfectly legal thing to do). The iSCSI bridge received our iSCSI command, converted it to fibre channel, sent it to the drive, and got the data back. But then, instead of sending back the 6 bytes that we asked for, the bridge sent back 32 bytes of data. This made certain layers of the operating system device drivers very unhappy – trying to stuff 32 bytes of data into a 6 byte sack! The good news was that it was very easy to reproduce the error, the error information obtained was everything needed, and once again the firmware in the bridge was fixed in a very short time.