In part one of this series (see The Network Monitor Volume 3, Number 2
http://www.ccci.com/product/network_mon/tnm32/trace.htm) we examined how trace route discovers the paths selected
by routers between a source and destination device. With this knowledge,
a network administrator can obtain
an indication of how the network is
configured and utilized. Trace can also
be used as a troubleshooting tool to discover failed links.
In this issue, we will examine the use of trace in complex networks, its interaction with Cisco's fast switching and explore some options available in extended trace as well as trace characteristics in real networks. First, let's review a sample trace output and look at some possible variations in the output.
Output Variations
As a review, Output 1 shows a typical trace report from York to Rome for
the network illustrated in Figure 1.
Output 1
York#trace rome
Type escape sequence to abort.
Tracing the route to Rome (218.100.103.130)
1 Tacoma (133.3.32.2) 4 msec 4 msec 4 msec
2 Seattle (133.3.192.2) 8 msec 4 msec 8 msec
3 Tokyo (133.3.128.2) 20 msec 20 msec 20 msec
4 Rome (218.100.103.2) 16 msec * 24 msec
York#
The output displayed nn msec which means trace received a response in nn milliseconds. Other characters may be seen in the output. For example, "!N" represents ICMP network unreachable, "!H" represents ICMP host unreachable, "!P" represents ICMP protocol unreachable, and "!Q" represents source quench received. The asterisk "*" represents time-out, and "?" represents a source route failure. (Source route is discussed in the next section.)
Extended Trace Options
At the end of the last installment, extended trace command options were
presented. All but the last command option were discussed. The last command
option looked like:
Loose, Strict, Record, Timestamp, Verbose[none]:
This command option offers the opportunity to modify IP header option fields for which you may specify values. Loose, Strict, Record and Timestamp are defined in RFC 791 covering IP options. Verbose is a Cisco specific option. If you choose to modify an IP header field, the extended trace command will issue prompts for values. Trace will place the requested IP option values in each probe; however, there is no guarantee that all routers (or end-nodes) will process the options. Here is a description of the IP header options:
Loose Allows you to specify a partial list of nodes which must be traversed when going to the destination. Loose source routing says, "Trace can take any path to the destination, but it must include the specified set of routers." Specify the routers by entering the IP address for each router that you want to exercise. Note that the probe will travel the routers in the sequence specified. Therefore, if you enter a far end router address and then a near end router address, the probe will go out to the far end, then back to the near end, and then continue out to the destination. (You could make trace ping-pong between two routers this way!)
StrictYou specify a complete list of routers to reach the destination. Strict Source Routing says, "This is the exact and only path to take to reach the destination." You specify the routers by entering the IP address for each and every router along the path. The sequence that the routers are entered is crucial. The routers in the list must be sequentially adjacent, and they must exist.
RecordWith this option, you can exercise the IP record route option. You may specify the number of hops to leave room for, but the IP header will not support more than nine hops.
TimestampThe timestamp option reserves space in the IP header to record the time when the remote router received the trace probe. (The time is recorded according to the remote router's clock.) You may specify up to nine timestamp slots. However, if you concurrently use the record option, then no more than four hops will be recorded. (The IP option header space is shared by record route and timestamp.)
VerboseIf you select any of the aforementioned options, verbose mode is automatically selected and prints the contents of the IP option field values in any incoming packets. You can prevent verbose mode by selecting it again, toggling its current setting. The verbose output may be overwhelming on your terminal; therefore, use it with caution.
Trace and Fast Switching
Let's modify the network configuration slightly from that in Figure 1 where there was only one path between all of the routers. Now, assume that there are two equal cost paths available between Seattle and Tokyo. (See Figure 2) These could be wide area serial links, or local area network segments. But the key is that both links are of equal metrics for the routing protocol. Then, neither path is preferred over the other. Which path will the router select? One, or the other, or both? Routers may exercise load sharing across equal cost paths. When successive packets must cross the equal cost paths, the router will use one link, then the other in a round robin technique. In fact, if 4 equal cost paths were present, all 4 would be equally shared. Cisco IOS version 11.2 will even support up to six equal cost paths. RIP, IGRP and other routing protocols behave in this fashion. Output 2 therefore reflects normal router behavior. Clearly, traffic alternated between the two links. Examining Output 2, Seattle alternated output paths for TTL3: twice to Router A 133.3.128.2 (TTL3a,c) and once to Router B 133.3.64.2 (TTL3b). TTL4 probes also alternated as evidenced by the two Tokyo interface addresses. What about TTL5 probes? They too alternated the equal cost paths even though it is not obvious in the output.
Output 2
York#trace rome
Type escape sequence to abort.
Tracing the route to Rome (218.100.103.130)
1 Tacoma (133.3.32.2) 4 msec 4 msec 4 msec
2 Seattle (133.3.192.2) 8 msec 4 msec 8 msec
3 RouterA (133.3.128.2) 20 msec
RouterB (133.3.64.2) 20 msec
RouterA (133.3.128.2) 18 msec
4 Tokyo (10.0.2.2) 20 msec
Tokyo (10.0.1.2) 20 msec
Tokyo (10.0.2.2) 20 msec
5 Rome (218.100.103.2) 16 msec * 24 msec
York#
Be aware however, that you may see a display where only one link is utilized, even though two links are present that have equal metrics for the routing protocol. This may happen when the routers have Cisco's fast switching feature enabled.
It is beyond the scope of this article to detail fast switching, but a word
needs to be said about it here. As previously mentioned, a router normally
consults the routing table for each and every packet. But checking the
route table is a CPU intensive operation and slows down routing
performance. To improve performance, Cisco designed fast switching
architecture. When fast switching is enabled on Cisco routers
(the default setting), the router first examines a memory buffer, called a
route cache to see if it already sent traffic to the destination network.
This does not exercise the CPU like a route table lookup. If there is an
entry in the route cache, then the router can forward the packet without
checking the route table, significantly increasing router performance.
However, only one of the equal cost paths are entered into the route cache. With only one entry in cache, successive packets will use the same path as the first. Therefore, only one path will be utilized. Fast switching can be disabled by configuring interfaces with no ip route-cache.
Trace Characteristics
One characteristic of trace is that trace only reports the path of outgoing packets. It is possible that the ICMP responses may take a different path than the UDP outgoing messages. Obviously, if your network is linear with no alternate paths as in Figure 1, this is not true. But if you have a more complex network with loops, there is no guarantee that your trace output reflects the path for both directions. By exercising the loose source route option described earlier, you can create a trace from the source to a remote router, then back to the source thereby testing both directions.
As an example, we will initiate trace on Rome with Rome as the destination and use the extended trace option to specify Tacoma's IP address as a router which must be used. (See Figure 1) The extended trace input and corresponding output are:
Rome#trace
Protocol [ip]:
Target IP address: rome
Source address:
Numeric display [n]:
Timeout in seconds [3]:
Probe count [3]: 4
Minimum Time to Live [1]:
Maximum Time to Live [30]:
Port Number [33434]:
Loose, Strict, Record, Timestamp,Verbose[none]: loose
Source route: 133.3.192.1
Loose, Strict, Record, Timestamp,Verbose[LV]: v {Turn off verbose}
Loose, Strict, Record, Timestamp, Verbose[L]:
Type escape sequence to abort.
Tracing the route to Rome (218.100.103.2)
1 Tokyo (218.100.103.1) 4 msec 4 msec 4 msec 4 msec
2 Seattle (133.3.128.1) 4 msec 8 msec 8 msec 4 msec
3 Tacoma (133.3.192.1) 24 msec 24 msec 24 msec 24 msec
4 Seattle (133.3.192.2) 24 msec 20 msec 20 msec 20 msec
5 Tokyo (133.3.128.2) 24 msec 24 msec 24 msec 24 msec
6 Rome (218.100.103.2) 24 msec * 24 msec *
Rome#
Note in the extended trace configuration we specified a probe count of 4 to request four trace probes per TTL, and we disabled the verbose mode so that the monitor was not cluttered with the IP option information.
A second trace characteristic may be considered. This stems from trace
displaying only the address reached with the current TTL value. Changes may
occur in the network for routers already reported by trace at lower TTL
values before trace reaches the destination. Trace will not be aware of
these changes, leaving you to assume that the current network situation is
displayed on your screen, when in reality it is now different than what was
first reported. For example, assume that you have a multi-hop network with
loops. (See Figure 3)
Assume that trace reported on your terminal the first
two hops, A and B. After the second hop was reported, the link between A
and B failed. If dynamic routing is enabled, a new path will be used if one
exists. The dynamic routing protocol automatically selects this
alternate path. But trace has no way to know this and continues to issue
UDP datagrams with increased TTL values. Now the trace output blindly
reports router F as the third hop. Unless you know all the details of your
network, or at least have accurate and up to date network documentation,
you may erroneously assume that router F is directly connected to router B.
Another topology sensitive characteristic may be observed for the network
in Figure 4.
Trace (or any other packet for that matter) does not always
travel the most optimum path to a destination. Assume trace is initiated at
11.0.1.1 with 10.0.2.2 as the target. Note that 10.0.2.2 is a port on
router B. Trace obviously must pass through router A and out port 10.0.1.1.
But where does the trace probe go after this? Ideally, the probe would go
directly to 10.0.1.2 on router B since the target is an interface on B.
After all, it is obvious to you that A can get directly to the target
through B. But it is not so obvious to the routing protocol. Router A has
no local knowledge of network 10.0.2.0 and therefore does not know that
10.0.2.2 is directly attached to B. All router A knows is how to get to
network 10.0.2.0. A's job is simply to move the packet towards the target
network. From A's perspective, two paths exist to get to this network. If
fast switching is disabled, then trace probes will alternate between
10.0.1.2 and 10.0.1.3. If you had a network analyzer attached to network
10.0.2.0, you would observe every other trace probe since half would go
directly to B and the other half would go through C.
What if fast switching were enabled? Then depending upon which route is in the route cache, either all of the trace probes will go through router C, or all will go directly to router B. This could make troubleshooting interesting. If you had your analyzer attached to network 10.0.2.0, you may expect to see none of the trace probes thinking that they should go directly to B. Yet if the trace packets are traveling through router C, you might erroneously assume that something is wrong with the path to router B. Make sure that you check your routing tables on router A and the configuration of your route cache before you make any rash conclusions!
Another Trace characteristic may arise from improper implementations of IP. Some IP implementations of hosts and routers may create unusual trace responses. For example, when the trace UDP message finally reaches the destination, the destination should return an ICMP message reporting port unreachable. Unfortunately, not all destination devices return this message. Therefore, the trace output will display a string of asterisks until the maximum TTL is reached indicating no replies.
Another possibility results from the way some hosts handle ICMP TTL. Some devices incorrectly create the ICMP message by inserting the received TTL value in the response, rather than creating a new TTL value. This means that the responding device will insert a one in the TTL field, causing the message to go only one hop back! The source device will display an asterisk when the timeout value is reached. The source will continue to increment the TTL field until the maximum TTL value is reached. Hopefully the source will eventually increment to a value twice the distance to the target. Only when this occurs will the TTL value be sufficient to return the response back to the source.