Queue depth is something that most storage admins tend to overlook. This is the reason for most performance issues. Let us look at what queue depth is, what it can do, how it impacts performance and how to make things better.
Queue depth is the number of commands that the hba can send / receive in a single chunk - per LUN. From a host-hba point (initiator), it is the number of commands that can be queued (or stored) and then sent to storage. From a storage point (target), it is the number of commands it can accept in one-shot, again, per lun. I keep stressing 'per lun' since this is the most important factor that determines what should the queue depth setting be on the host. On a storage target, you cannot change it as the target ports come pre-configured - usually with 2048 or 4096 (Fibre Channel). Most storage target ports use 4096 as the queue depth.
Let us examine how we can determine what should be the queue depth on a initiator. If you have a storage target port that supports 4096 queue depth, and there is a single host accessing that port and has 10 luns, the max queue depth setting is 4096/10 = 409 and since queue depth is set in factors of 2, i.e 2/4/8/16/32/64/128/256 and so on, we should be using 256 in this case. Most hbas have a default queue depth of 32 (0x20 hex) and by changing from 32 to 256, increases the response times for the storage-luns. One important thing to remember here is that you have now maxed out the storage target port queues (2560 out of 4096). You can add 6 more luns to the same host with 256 queue depth - anything above that and you will start seeing issues. You will see latency problems only if all the queues are full. But you have to plan for the worst case and assume the queues are full all the time (at least it will be during backups).
Performance degradation starts quickly once you reach max queue depth on target and it starts backing off the queues and it will hit hard on your response times. Also, you cannot add any further hosts and present any more luns from that target port. Ideally, if you have few hosts and luns, the default is good. If you have the luxury to dedicate ports for high performance applications, make sure the queue depth is configured appropriately so you get the best of performance.
In linux, you can set the queue depth through modules.conf (2.4 kernel) or modprobe.conf (2.6) and in Solaris from the Qlogic config file or Emulex config file. This is just on the host level. Your hba need to change that setting to match that of the host as well which can be done from hba bios or cli interface package that qlogic provides.
Queue depth is the number of commands that the hba can send / receive in a single chunk - per LUN. From a host-hba point (initiator), it is the number of commands that can be queued (or stored) and then sent to storage. From a storage point (target), it is the number of commands it can accept in one-shot, again, per lun. I keep stressing 'per lun' since this is the most important factor that determines what should the queue depth setting be on the host. On a storage target, you cannot change it as the target ports come pre-configured - usually with 2048 or 4096 (Fibre Channel). Most storage target ports use 4096 as the queue depth.
Let us examine how we can determine what should be the queue depth on a initiator. If you have a storage target port that supports 4096 queue depth, and there is a single host accessing that port and has 10 luns, the max queue depth setting is 4096/10 = 409 and since queue depth is set in factors of 2, i.e 2/4/8/16/32/64/128/256 and so on, we should be using 256 in this case. Most hbas have a default queue depth of 32 (0x20 hex) and by changing from 32 to 256, increases the response times for the storage-luns. One important thing to remember here is that you have now maxed out the storage target port queues (2560 out of 4096). You can add 6 more luns to the same host with 256 queue depth - anything above that and you will start seeing issues. You will see latency problems only if all the queues are full. But you have to plan for the worst case and assume the queues are full all the time (at least it will be during backups).
Performance degradation starts quickly once you reach max queue depth on target and it starts backing off the queues and it will hit hard on your response times. Also, you cannot add any further hosts and present any more luns from that target port. Ideally, if you have few hosts and luns, the default is good. If you have the luxury to dedicate ports for high performance applications, make sure the queue depth is configured appropriately so you get the best of performance.
In linux, you can set the queue depth through modules.conf (2.4 kernel) or modprobe.conf (2.6) and in Solaris from the Qlogic config file or Emulex config file. This is just on the host level. Your hba need to change that setting to match that of the host as well which can be done from hba bios or cli interface package that qlogic provides.
No comments:
Post a Comment