High Avg Disk Queue Length and Finding the Cause
Archived note. Originally published 12 September 2008 on iishacks.com. It describes disk performance diagnosis with Performance Monitor on Windows Server 2003 and 2008 as those systems behaved at that time. The original body text was not recovered from any surviving snapshot — only its title, date, categories and tags, plus the later 2012 note here that cited it for the rule of thumb of roughly two per physical disk — so what follows is a reconstruction of the subject, not a transcription. Windows Server 2003 is long past end of support. Verify against current vendor documentation before applying any of it to a live system.
What the counters actually count
Performance Monitor exposed disk activity under PhysicalDisk, reported per device as the operating system saw it, and LogicalDisk, reported per drive letter. Queue-length work belongs to PhysicalDisk: a queue forms at a device, not a partition.
PhysicalDisk\Current Disk Queue Length is an instantaneous sample: the number of requests outstanding at the moment the counter was read. It spikes constantly on a healthy system and is close to useless on its own, but it is the counter that shows whether requests really were piling up during a stall.
PhysicalDisk\Avg. Disk Queue Length is not a sample at all, and that is the point most often missed. It is a derived value: transfers per second multiplied by average seconds per transfer. A device serving two hundred transfers a second at ten milliseconds each reports an average queue length of two whether or not two requests were ever outstanding at the same instant. The counter is an arithmetic restatement of throughput and latency, dressed as a depth measurement — which explains everything else about how it misleads.
The per-spindle divisor
The rule of thumb of the era was to divide the observed average queue length by the number of physical disks behind the device and treat a sustained result of about two per disk as reasonable. A four-disk set at a queue length of eight was doing about as much as it could; the same set at forty was leaving requests waiting.
The word carrying the weight there is sustained. Backups, index rebuilds, virus sweeps and virtual machine starts all produce queues in the hundreds for a few seconds, and that is normal. A queue that is deep for an hour, or returns every ninety seconds with no scheduled task to explain it, is a bottleneck.
Why the divisor stopped being knowable
The divisor is what broke the counter. A hardware RAID controller presents an array as one device, and a SAN presents a LUN as one device. In both cases the operating system counts a queue against a single instance with no way to know whether one spindle or twenty sit behind it. The rule of thumb loses its denominator, and the reading becomes uninterpretable rather than imprecise.
Controller cache made it worse in both directions. A write-back cache absorbs a burst and acknowledges it immediately, so the host sees short queues while the array behind it is fully committed. When that cache filled, the queue at the host jumped from nothing to everything with no gradual approach to warn of it.
Latency is the signal to trust
Avg. Disk sec/Read and Avg. Disk sec/Write answer the question that actually matters: how long an individual request took from issue to completion. They are reported in seconds, so 0.010 is ten milliseconds. Latency survives every problem described above — it needs no spindle count, and it is measured at the boundary the application feels. The guidance of the period, for transactional workloads on spinning disks, put single-figure milliseconds in the comfortable range and anything sustained well above twenty where users notice. Those figures belong to their hardware generation and should not be carried forward to flash.
The usual causes
Not enough memory
The most common cause of a disk problem in this era was not a disk problem. A server short of RAM pages to disk, and that paging appears in disk counters as nothing more than load. Read Memory\Available MBytes and Memory\Pages/sec beside the disk counters, and note which volume holds the page file. Adding memory resolved more apparent storage bottlenecks than any storage change did.
A misaligned partition
Windows Server 2003 and earlier created the first partition at an offset of sixty-three sectors, a hangover from cylinder-head-sector geometry. That offset is not a multiple of a RAID stripe unit or of a 4 KB block, so one logical write could straddle two stripe units and force the controller into a read-modify-write cycle, roughly doubling the back-end operations for no visible reason at the host. Windows Server 2008 changed the default offset, which is why the problem disappeared on newly built systems and lingered for years on migrated ones. Correcting it meant recreating the partition and restoring the data.
A failing member in an array
A disk that has not failed outright but is retrying reads keeps an array online while adding tens or hundreds of milliseconds to any request that touches it. Nothing in Performance Monitor names the faulty member; that evidence lives in the controller logs and the drive's self-monitoring data — where to look once the counters show high latency at a low request rate.
Real-time antivirus scanning
A real-time scanner intercepts opens and closes through a filter driver. On a workstation that cost is unremarkable; on a file server, a mail store or a database volume it is multiplied by every operation the application performs. Diagnosis means measuring the same workload with the driver stopped during a maintenance window, not arguing about it from the counters.
A workable order of investigation
- Confirm the complaint against
Avg. Disk sec/ReadandAvg. Disk sec/Write, not a queue counter. - Read the request rate beside it. High latency at a high request rate is a capacity problem; high latency at a low request rate points at a fault.
- Check available memory and paging before touching storage.
- Establish the spindle count, RAID level and cache behind the device before applying any per-disk rule of thumb.
What is different now
Storage is now virtualised, tiered and usually flash-backed, and devices accept far deeper queues by design, so a depth that indicated saturation on a 2008-era array can be entirely healthy today. The authoritative view of a storage problem now comes from the storage platform's own telemetry, with host counters used only to establish whether the host is the one waiting. The reasoning transferred intact: measure the time a request took, not the depth of a queue that cannot be interpreted, and rule out memory before blaming a disk.
The 2012 note here on an SSD freezing under Windows 7 is a worked example of that sequence, and it is the post that cited this one. Further storage material sits under the hardware category, operating-system troubleshooting under Windows and Server, and the full index at post archives. For a supported system, start from the Windows Server troubleshooting documentation and the Windows Server documentation home, and check dates against the Microsoft product lifecycle.