Colgate University Uses NetMRI
"I have upgraded 400+ IOS devices while I have a cup of coffee with a simple script in NetMRI. Being able to focus on other duties and not worry about missing changes made is great...and being able to rapidly replace a device quickly is even better."
—Don Rhodes, Network and Systems Administrator
More CustomersConnect With Us
The Network Monitor, Volume 2 Number 3
Take the Next Step:
Related Information
Articles
- Frame Relay Design
- Proactive Network Management with NETSYS
- The Real Cost of Network Management
- President's Message
- Bob Lyons: His Experience Drives Excellence
Tech Tips
Frame Relay Design
By Peter J. Welcher, PhD, CCIE, Senior Consultant
Last issue we looked at how easy it is to configure Frame Relay. Now let's talk about design and scaling issues.
First, a disclaimer: there's more to be said on this subject than I can possibly cover in this article, so you should look at the Cisco Internetworking Design Guide on UniverCD. The Cisco Design course (CID), which I teach, also has sections on Frame Relay and ATM design.
Understanding Topology
Small Frame Relay networks of 10-20 sites are often built with a hub-and-spoke or star topology.
In such networks, Permanent Virtual Circuits (PVCs) are provisioned to a central site. This doesn't provide redundancy, but access lines are considered to be the component most likely to fail. And redundant access lines are costly, particularly if diversity is desired. Within the provider network, circuits are becoming fairly robust.
An alternative topology that does not seem to be getting much use is the double star, with PVCs to two hubs for redundancy or load balancing.
Creating Subinterfaces
With small networks, the main design issue is what approach to use in configuration. I recommend using subinterfaces, otherwise split horizon issues may become a problem. If using IP with the IGRP routing protocol, split horizon is disabled on Frame Relay interfaces in recent IOS releases. So IP routing with IGRP works correctly whether or not you use subinterfaces. Creating the subinterfaces is good discipline, leaves an obvious place to add other features, and allows for per-PVC access lists, bandwidth statements, descriptions, etc. And it ensures that the IP addressing scheme won't have to change at some future date, for example, to match subinterfaces added in support of features such as bridging, IPX, or AppleTalk. So, use subinterfaces.
With larger Frame Relay networks, the major design constraint is bandwidth. First, you have to ensure access lines have enough bandwidth to handle data being sent from other sites. Using the star topology, if 10 sites have a CIR (Committed Information Rate) of 28Kbps, the central hub site may have 280Kbps arriving on its access line. But if the sites can burst to 56Kbps, the central site might burst to 560Kbps. Usually we assume bursts won't come from all remote sites at the same time and provide the central site with an intermediate capacity. Depending on local rate structure, access lines may be either 56K or T1, with no point to anything in between. If so, we might go with a T1 from the Frame Relay service provider to the central site rate, figuring there will be traffic growth anyway. But this is not the only way bandwidth enters into the discussion.
The Effects Of Broadcast Duplication
Frame Relay is not a broadcast medium. But a router participating in dynamic routing needs to send updates to its peers, at the other end of the PVCs emanating from itself. There are really only two ways to do this. One way is with switch multicast, where the router sends on a special DLCI and the switch simulates a broadcast. The other way is what's currently used: the router duplicates the broadcast and sends it directly to each intended recipient.
One effect is on the CPU. There is some use of the CPU in duplicating the broadcast packet(s)—one copy per peer router. A 10-packet routing update going to 50 peer routers results in 500 outgoing packets. These packets might hit the outbound serial interface queue at one time. This can demonstrably tie up the queue (and interface) for a while, obstructing normal traffic.
Another effect is that the packets may take time to feed out the serial interface, the serialization delay in emitting bits on a slow link. Outgoing copies may also consume a fair amount of bandwidth. Various Cisco documents suggest the latter is an important sizing criterion. Total bandwidth consumed by routing updates and other overhead activity should be less than 20% of the link bandwidth. Otherwise, resources and time are wasted outputting these overhead packets.
Estimating Overhead Bandwidth
Some estimation is thus in order when designing a Frame Relay network of any size. How chatty is the routing protocol? How often do updates go out? How big are they? Are there hellos? Keepalives?
One way to estimate overhead bandwidth is with a spreadsheet. (See Figure 2.) Perhaps the greediest consumers of bandwith are Novell IPX SAP updates, which use 7 services per packet.
Suppose you add up all the numbers and too much bandwidth is being consumed by overhead activities. What are the alternatives? Well, adding more bandwidth is the easy answer. Another choice is to use less chatty protocols: IGRP, EIGRP, or OSPF instead of RIP; NLSP or EIGRP for IPX instead of Novell RIP/SAP. Static routing is the ultimate in low bandwidth, but high in administrative hassle. Cisco's snapshot routing (fairly new) is a possible alternative. When reviewing protocol and chattiness, don't make assumptions. The topology may alter things.
A less obvious choice is fewer peers. This introduces more routers, perhaps on a common LAN backbone. Or use a hierarchical structure, with access routers tied via X.25 or Frame Relay to a regional hub router, and Frame Relay tying these back to backbone routers at HQ. There is additional latency from passing packets through multiple routers, also from multiple serial/Frame Relay hops. But the resulting design is scalable and manageable. The Cisco UniverCD Design text discusses alternative topologies, as does the CID course.
A Frame Relay Design Example
Recently I saw a question on the Internet, asking whether a Cisco router could handle 500 or 1000 PVCs at the hub of a Frame Relay star. Interesting question! One reply was that the access line had better be a T3. If the remote sites have 56K access lines, T3 is in the right ballpark. However, some other questions come to mind: What is the overhead bandwidth? Assume 1 LAN outboard of each remote router. What if there are another 100 LAN segments somewhere? Static routing in the hub, default routes at the remote sites?
The CPU impact of pseudo-broadcast is a small amount per PVC. Multiply by roughly 1000 PVCs and you might have serious impact. Before attempting this, I'd want to talk to some Cisco engineers who'd tried something similar before. Maybe the CPU is ok but packet buffers or RAM is a concern?
But more important, is this good, modular, scalable design? It certainly has a single point of failure! If the remote sites go to 128K, how do you scale up the central site? How do you upgrade technology? How do you provide redundancy, another jumbo star? Clearly, a design needs to be analyzed in terms of how it will be used, both now and in the future.
Other Design Considerations
Redundancy is a design concern to Frame Relay users. Dial backup, either over modems or ISDN, is a common solution. ISDN has the advantage of using a single 25xx BRI port at a remote site, and MBRI on a 4000 or PRI on a 7000. It also saves playing around with modems, which is nobody's idea of fun. Dial backup is now available on a per-PVC basis with subinterfaces (IOS release 10.3).
Using older releases or the NBMA model (non-broadcast multi-access) may require floating static routes, that is, ones with a high administrative distance. Dial-on-demand (DDR) may also be used. Triggering ISDN dialing on a native BRI interface with an older IOS release may require this. The idea behind the use of a floating static route: it remains inactive as long as dynamic routing provides a route. When a PVC goes down, the dynamic route eventually goes away, and then the floating static route kicks in, sending traffic out the backup interface. This traffic can then be used with dial-on-demand routing (DDR) to trigger modem or ISDN dialing. A neat trick!
A final part of design is planning for network management. The Cisco routers support the Frame Relay DTE MIB (RFC 1315). This allows for a good amount of per-PVC information to be collected. Do allow bandwidth to cover your polling!
Summary
Good design is the hardest part of Frame Relay, but will ensure that your networks work as intended. The above suggests some of the issues to consider.
Dr. Peter J. Welcher, CCIE #1773 and CCCI senior consultant, teaches the Cisco ICRC, ACRC, ICWC, and CID courses. He also consults on network design and management.
Proactive Network Management with NETSYS
The Environment
Today's companies rely on networks to support almost every aspect of their business. From financial powerhouses to manufacturing firms to small consulting agencies, the corporate intranet has become a critical component.
The cost of downtime on corporate networks is measured in tens of thousands of dollars per hour. (See "LAN Downtime," Data Communications, March, 1990.) The causes can be categorized as device failure, link failure, or configuration error. Much of this downtime can be avoided by proper network design and implementation. Device and link failures can be handled through proper redundancy. Configuration errors can be avoided through proper design and implementation methodology.
NETSYS has created a set of tools that evaluates a Cisco router network on a system basis for the purposes of detecting failure modes, improving security, and pinpointing configuration errors. When used as part of the configuration control and evaluation process, the NETSYS tools can highlight configuration errors before they appear in your live network.
Management Styles
Before jumping into what these tools do, let's take a look at network management styles and when you would want to use the NETSYS tools. There are two levels of network management: reactive and proactive. In a reactive network management environment, the network management staff is typically driven by trouble calls. Some time is spent in planning, but the majority of the effort is in keeping the network running day to day. The network is typically viewed and managed on a per-device level.
In contrast, the proactive network management team has a smoothly-running network where changes are carefully planned and the network design is fault-tolerant enough to allow time for the team to react to device and link failures. Device configuration changes are carefully evaluated prior to implementation. Network health reports are generated on a regular basis so the team can plan for change. A proactively-managed network is typically viewed and managed on a system level.
Most of our network management teams fall somewhere between these two levels. Properly used, the NETSYS tools provide us the system-level view we need to analyze our networks, begin to eliminate the sources of trouble calls that lead to reactive network management, and allow us to evaluate proposed configuration changes.
Analysis Requirements
System level analysis starts by checking a large number of static configuration options that must match between connected devices in order for the network to function properly. This step includes ensuring the bandwidth statements match on both ends of a serial link, detecting duplicate node and network addresses, identifying duplicate token ring numbers, and verifying correct routing protocol parameters. Once the static configuration has been verified, a failure mode analysis should be performed to determine what happens when critical devices or links fail. From this analysis, we can add redundancy where it is needed so that when failures occur, we are not forced to react immediately. Note that we still need real-time network management to alert us to failures.
Using the connectivity analysis, we need to be able to evaluate network security as implemented in router access lists. Have we properly limited access where desired and provided access where needed? Are there any "back doors" in our network that we have not identified?
Another analysis requirement is to show the dynamics of the network under load. The best way to do this is to gather real data from the network and use it to analyze the operation of the network as a system. And if we can modify the data ourselves to simulate a change in the network, then we have the ability to do real "what-if" scenarios.
With a set of tools that provide this functionality, we can provide proactive network management. The NETSYS toolkit, in conjuction with CiscoWorks and a network management platform, provides the capabilities we need to achieve the above requirements through its analysis engines: Connectivity Baseliner and Performance Tools.
Connectivity Baseliner
Static analysis is performed by reading the router configurations, analyzing the resulting network, and generating a set of reports detailing any anomalies that it detects. (See Figure 3.) The analysis tests for over sixty different configuration problems, generating a report that identifies each problem and its severity level, from errors to warnings. (A full list of tests may be found on www.netsystech.com.) A topology map is also generated, providing visual indication of network connectivity. Besides checking your existing network, Connectivity Baseliner is also useful in "sanity checking" new router configurations that are being prepared for a network rollout or transition. Whether evaluating an existing network, or preparing new router configurations, Connectivity Baseliner prevents many problems that may turn into trouble calls.
Connectivity Solver
Once the baseline has been created, any number of connectivity requirements may be evaluated to verify network resiliency when devices or links fail. Are there critical points in our networks where a single failure will have a major impact on the business? Through the "what-if" capability of the tool, we can evaluate the addition of redundancy. The topology map produced by the Baseliner is used by the Connectivity Solver to display routing paths from source to destination and back. We can actually see asymmetric routing caused by incorrect bandwidth statements or by invalid routing metrics.
Security analysis of Cisco access lists is also checked by Connectivity Solver. We need to be able to verify that an access list is filtering out the packets we identified and passing the packets we intended. Since access lists tend to change over time, it is very useful to be able to verify that the required connectivity is maintained after router configuration changes.
Performance Tools
Static analysis, while providing very useful information, doesn't actually provide the full picture. NETSYS Performance Tools use RMON, Cisco IP Accounting, or user-generated data to perform a dynamic analysis of the network. Which links are likely to saturate under the given load? What path is the data taking? What happens if we move a server and its traffic to another network segment? These are the kinds of questions that we need to answer when designing for the future.
Another useful and unique feature of Performance Tools is the ability to change router models and simulate the resulting change in network throughput. We can also see the performance effect of implementing some of the router performance features, such as autonomous switching. To accomplish this, NETSYS uses Cisco router modeling information to drive its simulation.
Summary
The NETSYS tools help us understand the network layer connectivity issues in our intranets. They help us move from the concept of network design as an art to network design as a science, by allowing us to measure the quality of our designs.
The Real Cost of Network Management
At the outset, network management seems like an easy enough job. Purchase a workstation, select a base management platform, add device- specific applications, attend a network management class, and you're in business. The network engineering staff can operate this system. After all, it uses a graphical interface, so anyone should be able to operate it, right? Unfortunately, what you end up with is a collection of tools, not a solution.
So what does it really take to field a network management solution? First, you have to define your requirements, procure the components, and learn how to use them. Only then can you integrate the components into a system that will yield the results you wanted when you first defined the requirements. Let's take a look at how much the process might cost and how long it should take.
Requirements Analysis
Before assembling a set of requirements, you must have some understanding of network management —what protocols are used, along with their strengths and weaknesses; what capabilities are available in off-the-shelf tools; and what your business needs are in the way of reports, change control, problem solving, etc. The self-education component of this can take several months, but do not be tempted to shortcut the process. Mistakes made during this phase are very difficult and costly to correct. Many times, a consulting organization experienced with network management can help you sort through the tools available and make sure that your demands for functionality are realistic. By the end of this process, you will have invested between two and six months of effort.
Procurement
Purchasing the hardware and software are easy once you know what you want to purchase. For our example, let's say it takes a month of your time to purchase all the components you identified.
Don't forget the personnel part of the requirements analysis should be related to staffing. There should be two well-versed staff members for each critical component of the management system so that illness and vacation do not leave the company without an expert available. The knowledge required for network management is quite broad, so you should expect to pay dearly for people who can make a system work on an ongoing basis. Salaries from $75,000 to over $100,000 are not unreasonable.
Training
Training on the hardware and software you ordered is vital. Without it, you will typically spend six or more times as long learning to use the system as effectively as someone who attended formal training. At worst, an error in the application of a network management system may adversely impact or bring down a critical part of your network. Plan to spend at least a week of formal training on each major network management component (e.g. OpenView, CiscoWorks, etc.). If you have experienced personnel, then this may not be needed.
Integration
When the components arrive, you must then install them on the hardware platform and make sure that they work well together. This is where the experience of your staff will make a significant difference. Some programming may be required to provide the necessary interfaces between components or to produce custom reports required by your company's business needs. An estimate of one month to integrate each application into the overall system is about right.
Produce Solutions
Once all the components have been integrated, you may still have to build some additional functionality to produce the required network management solutions. Depending on the exact functionality required, the time may range anywhere from a few days to more than a month. For example, CiscoWorks has functions to aid you in storing router configuration files. However, the real task is to automate the process of periodically checking for differences between the router configurations on the network against those stored in the management station.
Total Cost
How much does the entire process cost in time and dollars? In total time, allow between 6 and 12 months to accomplish a majority of your initial goals. In cost, two network management staff can translate into $100,000 to $200,000. Add to that the hardware and software costs (workstation: $20,000; platform software: $15,000; CiscoWorks: $9,000; NetSys: $20,000; etc.). In total, the base costs for getting started can approximate as much as $500,000 for the first year. For this expenditure, you are often just starting to perform real management by the end of the year. This cost does not include opportunity costs or the cost of any significant network failure during the process of fielding network management.
Annual costs after this first year include the salaries and system maintenance fees. These can total to approximately $300,000 for each successive year, assuming that the system you have designed continues to operate smoothly with no major functionality enhancement or replacement.
In summary, functional network management is not cheap, nor is it easy to do. But it is something that is necessary if our corporate networks are going to continue to operate smoothly and adapt as company needs demand.
President's Message
Welcome!
Welcome to the sixth issue of The Network Monitor. Over the past year we have covered a wide variety of topics and have worked to improve the magazine's content and presentation. In this issue we are expanding the number of pages of technical content. The Cisco Router course catalog format is also modified, but in a way that allows us to retain the level of detail you need to properly select the courses important to you. Please let us know what you think.
In the last issue, we introduced a new section called "Tech Tips," which highlights short networking tips. If you have an idea for a Tech Tips, please e-mail it to us (training@ccci.com) and we will make sure you are properly identified as the source if the tip is published.
CCCI Update
We have just added a second classroom to our headquarters training facility and expanded our class offerings to include training in ATM and Cisco Access Server And Security. But our expansion is not just limited to Annapolis. We now have staff to serve you in Houston, Texas; Harrisburg, Pennsylvania; Richmond, Virginia; and Orlando, Florida.
Employment Opportunities
As we continue to grow, we are looking for qualified people interested in joining our world-class operation. Our focus is on quality that, in the networking arena, translates into very knowledgeable people. Part of our evaluation process tests your technical knowledge. If you are interested, please check out our web page for employment opportunities and a sample set of qualification questions.
Knowledgeware
The true "information age" is upon us—it is the use and distribution of information and knowledge that is the source of much of the competitive advantage used by companies today. CCCI's business is KnowledgewareSM—sharing our knowledge and expertise. Our primary task is to help you, our students and customers, become self- sufficient. Your self sufficiency comes through understanding the advantages of internetworking technology. We make a significant effort to ensure that our consultants and instructors bring the most complete and up-to-date information available to you. For example, the majority of our technical staff attends Cisco's annual Networkers conference. In addition, all of our instructors provide consulting services to a majority of our customers. They solve real-world network problems that you encounter day to day with your own networks. This combination of staying abreast of new technologies and hands-on problem solving improves our classes and results in increased productivity on consulting projects.
In summary, it is clear to us that in the information age, the companies that have the best control and use of information will be the most successful. CCCI has adopted a motto that reflects this vision: "Knowledge WorksSM"!
Bob Lyons: His Experience Drives Excellence
Bob Lyons knows computers almost from the start of commercial computing itself. Since his earliest experience as a night-shift programmer on an ancient IBM 1440 back in the early sixties, Bob has literally seen and done it all in the world of computer applications. And as a result of his nearly 30 years as a programmer, consultant, and trainer, Lyons has amassed a wealth of professional knowledge that he readily shares today as a Consultant-Senior Instructor for CCCI.
Been there, done that? In Bob's case, it's an understatement. "My first mainframes had about two megs of RAM," he laughs. "Our support equipment included card sorters and punchers, and we had to wire the plugboards ourselves on card reproducers and interpreters."
By the late sixties, Bob was in the Army, serving as a computer repair instructor and, in his own words, "really learning computers inside out." Following the Army came stints at Computer Science Corporation as a Senior Scientific Programmer creating communication system simulations for the Army, and later at Gray Drugstores, as that $1 billion company's Director of Inventory Systems. In 1981, he relocated to Texas, where an initial job as Director of Planning at Handy Dan Corporation, the nation's leading home center at the time, led to a decision to launch his solo consulting career.
"As a consultant, I worked a lot on early PC-based networks," Bob recalls. One of the high points of this period was Lyons' successful conversion of a $1 billion company, Hard Hanks, from mainframe to a 45-file server network. Though briefly lured back into the workplace as Manager of Network Systems for Multi-Net, Inc., Bob soon returned to consulting up to the point that he joined CCCI in early 1995.
At that time, Bob was conducting independent training for a competitor of CCCI. "It was lucrative," he says, "but the company had nowhere near the professional status of CCCI, which always hired top-notch technical people. So when CCCI made me an offer, I jumped at it!"
According to Bob, the opportunity to conduct training with Cisco Systems products was an additional motivator. "Cisco is far and away the industry leader in overall quality of the devices they sell and support," he affirms. "No router today has the breadth of functionality that Cisco routers do. There's simply no one who can rival them."
As a Senior Instructor, Bob currently crisscrosses the country conducting an average of 30 CCCI courses each year. Specific classes include Introduction to Cisco Router Configuration, Advanced Cisco Router Configuration, HP OpenView, and Introduction to Network Protocols. (Lyons originally developed the latter 3-day course as a custom program for Southwestern Bell prior to joining CCCI, and has since modified it for on-site training at such companies as Hewlett-Packard.)
In his courses, Bob is quick to dispel any misperceptions of high-tech mystery surrounding networks and their construction. "I don't think networking and routing is nearly as complicated as many people believe," he says, "so it's sometimes a challenge to get students to understand how simple overall they are." Lyons also firmly believes that common sense plays a major role in the learning process. "You don't have to be a heavyweight technically as a generalist, I'm certainly not. There's always a simpler way to solve a problem, especially in configuring routers. That's what I try to demonstrate in my CCCI courses."
Even with the high evaluations Bob has received from his students, he remains modest in his own self-assessment. "As a CCCI instructor, I try to relate to my students on a professional and a human level, " he notes. "If I can help someone do their job better, I feel good about it."
Debugging Frame Relay PVCs
Bringing up new Frame Relay (FR) circuits is always a challenge. Between FR switch settings, DLCI addresses, LMI configuration, and basic connectivity issues, there are many things that can go wrong. One way to help troubleshoot a new connection is to enable full FR debugging on the router interface. The debug output shows detailed information about the FR connection as it communicates with the FR switch, including the DLCI address configured on the FR switch. Some telco people have found the information that the Cisco router outputs to be so valuable that they have begun using the Cisco 750 router as a diagnostic tool. Each FR technician has one and connects it to the FR circuit, turns on full debug, and verifies that the link and PVC are correctly configured.
Router Password Security
The security of router networks has become very important these days. One very good method of improving the security of the routers themselves is to change the passwords periodically. This process used to be carried out by one or more staff members telnetting to each router and changing the passwords in each. However, with the Snap-In Manager component of CiscoWorks, this process can be easily automated. Performing the password changes on the network management system has the added benefit of keeping the router configurations on the network in sync with those stored in the management system.
Debugging X.25 Connections
A recent X.25 connectivity problem renewed the rule to double check all X.25 parameter settings. An X.25 connection was working fine with ping and inital telnet connection data, but consistently failed to transfer large amounts of data. After working on the connection problem for some time, we discovered that the X.25 switch's window size was not configured the same as on the Cisco router. Changing the window size on the router to match that of the switch solved the problem. The moral: don't count on ping or initial telnet success as an indication that the connection can carry a full load. Try the exended ping options to exercise the connection with a large number of packets of varying sizes.
Back to The Network Monitor Archive
Copyright © 1996 Chesapeake Computer Consultants, Inc. All rights reserved.