Network Analysis Tip #31: QoS Queue Drops

Why is this important?

QoS is used in networks to prioritize important data over less important data when links become congested. You typically select time sensitive and business-critical applications to be given priority service over bulk data applications. There are many documents and best practice guides available on the ‘net to help you create good configurations.

Why use QoS? Why not just use really high bandwidth links? Those high bandwidth links certainly help, but they don’t eliminate queueing. Let’s say that you have a design with access-layer switches with fifty 100Mbps ports and the uplink to the distribution switches is 1Gbps. Ten file transfers or large web page loads will fill the uplink because TCP tries to use as much bandwidth as it can. Add some voice calls to the mix and you have a situation where the call gets garbled for a few seconds due to high jitter. As the applications continue to move large amounts of data, the voice calls will experience intermittent jitter problems. WAN links with less bandwidth exacerbate this problem. QoS allows you to prioritize the small voice packets when the interface queue begins to fill.

Once you have QoS configured, you should monitor its performance so that you know that it is performing the way it should. As the relative importance of applications changes, you may need to revisit the QoS configuration and only by observing its operation will you be able to make an informed decision on how to modify it. You may also use the information to decide if you need to add bandwidth to a particular path.

Looking for drops in QoS queues lets you know how much traffic is being dropped and in which queues. Expect to see drops in the scavenger and bulk data queues.

If you see drops in the high priority queues, it may be because the application is now using more bandwidth than you initially configured. Let’s say you designed QoS on a link to provide service to 5 concurrent voice calls and over time the number of concurrent calls grew to 10. You’ll see drops in the voice queue increase as the number of calls increases. Reallocating bandwidth may be the correct answer, or you may need a higher bandwidth link due to other critical business applications running on the same path. Having the data on queue drops allows you to make an informed decision instead of guessing that more bandwidth is the solution.

Manual Determination

Every vendor has different ways of examining the QoS queues. I’m going to use Cisco’s commands here. If you don’t have QoS configured, check to see if output drops are occurring:

Router# show interfaces serial 5/0

		 Serial5/0 is up, line protocol is up

		  ...
   
		  Input queue: 0/75/0 (size/max/drops); Total output drops: 1036

This is an interface that could potentially use QoS since it is oversubscribed and is dropping packets. Then, with QoS configured, check the queues for drops (the important output is in bold):

ts1#show policy-map interface fa1/0

		 FastEthernet1/0

		 Service-policy output: example

		 Class-map: class-default (match-any)
       
		 115404 packets, 9726599 bytes
       
		 5 minute offered rate 29000 bps, drop rate 2000 bps

		 Match: any
        
		 police:
         
		 15000 bps, 20000 limit, 20000 extended limit
        
		 conformed 4320 packets, 479920 bytes; action: transmit
         
		 exceeded 472 packets, 72276 bytes; action: drop
         
		 violated 3720 packets, 296768 bytes; action: drop
         
		 conformed 14000 bps, exceed 2000 bps violate 8000 bps

Automatic Determination

For interfaces without QoS configured, NetMRI collects interface discard counts and reports them in the Interface Congested issue as shown in Fig. 1.

Drilling down the the details on this interface, we find a traffic burst that resulted in congestion. (See Fig. 2)

But when QoS is implemented, there will be no discards, because the queueing mechanism is dropping the packets before they get to the interface. In this case, NetMRI monitors the Cisco CBQoS MIB to detect queues that are dropping packets, as shown below. This example is taken from a test configuration in our development lab. (See Fig. 3)

Some Cisco devices have hardware queueing support, and if you don’t define a queue, the hardware does CoS based queueing. The hardware won’t report queue drops, so the best thing to do is to implement a no-op policy and then the CBQoS MIB will be populated by the device.

Further Reference

Cisco QoS Output Scheduling on Catalyst 6500/6000 Series Switches Running CatOS System Software tells how packet drops are done in Cisco 6500 CatOS: http://www.cisco.com/en/US/products/hw/switches/ps700/products_tech_note09186a00801091a5.shtml

Cisco Understanding Packet Counters in show policy-map interface Output shows drop counters for each policy: http://www.cisco.com/en/US/tech/tk543/tk760/technologies_tech_note09186a0080108e2d.shtml