Netcordia Community

Capacity Planning - Do you know which ports are still being used?

mgowarty — Fri, 05 Mar 2010 20:09:00 GMT

Over the past several years, the capacity planning area for IT professionals has become much more challenging when dealing with managing active switch ports. For most organizations, the challenge is threefold:

The sheer number of ports requested has continued to grow year over year.
The transient nature of users keeps the infrastructure in continuous state of flux as users plug into an active for a period of time and then move on when done with that requirement.
The current economic state has greatly reduced expansion in regards to equipment, so the number of new ports hasn't grown as fast as the new demand for most organizations.

Keeping track of which ports are being used or were used but now are available isn't rocket science, it just falls to the bottom of the priority list for most IT staff. It's much easier to just add a new port you know is available instead of manually digging and sifting through each device to decipher where previously used ports are now available.

I recently published a new tech tip on the port saturation issue that covers how to better plan for capacity requirements. The good news is automation is available to help with this task as well as better managing your entire network infrastructure.

High-Level Network Design and Etherchannel Load Balancing

tslattery — Fri, 05 Mar 2010 13:56:00 GMT

After the experience with a wedged router interface (see prior post: Cisco Router Interface Wedged ), several of us at Netcraftsmen were discussing etherchannel load balancing and what to recommend to customers and how to check the setting in network equipment. James Ventre made a very good point for using src-dst-ip as the load balancing algorithm. Each pair of src-dst IP addresses will be sent on one of the etherchannels. His argument is that using IP addresses makes the load balancing operate...(read more)

Tracking Key Interface State

tslattery — Mon, 01 Mar 2010 18:58:00 GMT

The Baltimore-Washington area has been hit with several significant snow storms this winter and those storms have impacted the operations of a large metropolitan area network. Problems range from power outages at some sites to link problems between sites. The result is that a large number of interface up/down messages were recorded by syslog. Syslog is probably the first notification that a network team receives when an unexpected failure occurs, so watching it is a useful way to stay in touch with...(read more)

Default Passwords Unlock the Back Door for Security Attacks

mgowarty — Wed, 24 Feb 2010 12:57:00 GMT

A recent PCMag article by Larry Seltzer highlighted some of the security risks within routing infrastructures and how the uniquely named Chuck Norris botnet looks across network devices and finds default passwords as an entry point to spread harmful code or take control of devices. Even though this one has been named Chuck Norris, I’m much more a fan of Sydney Bristow who doesn't seem so 1980's.

It is interesting that organizations spend thousands or millions of dollars trying to build secure infrastructures by putting up the “biggest fence” to build a strong perimeter, but like any action/spy fan knows, all it takes is a little help from the inside or one small weakness and wham, the bad guys are in. While a strong foundation is important, it’s not the only step to ensure a secure and protected environment.

Although this article focused on the individual PC world, the problem can be just as huge for organizations dealing with their routing and switching network infrastructure. I’m scared to admit the amount of times I’ve seen organizations leave default passwords on their devices, or never change passwords when employees leave to join competitors or are terminated. That small back door is just enough for someone to get in and wreak havoc. Adding to the potential pain, once in, most IT teams would never see any modifications or changes unless an outage was caused. And the smarter bad guys know how to get the information they want without triggering alarms or build a sinister plot to cause as much damage as possible.

Organizations need to build and maintain a strong and secure infrastructure. With embedded intelligence and automation, users can set rules and policies notifying the IT team when passwords are weak or not changed periodically, and, if a bad guy happens to get in through the front or back door, the system logs and tracks all changes to the network infrastructure devices. So if an unplanned change is made, an issue is generated and the good guys can stop the bomb from going off and damaging the entire system.

New Tech Tip: Automating the Compliance Process

mgowarty — Tue, 23 Feb 2010 13:28:00 GMT

When dealing with both internal and external compliance requirements, IT staffs typically must shoulder the burden of proving the network infrastructure meets and maintains the best practices/gold standards (internal) or regulatory mandates (external). Traditionally, this process uses a great deal of manual time and effort to log in, capture the configs, review, compile and report. For a 150 device network, this effort could take as much as 24 hours to meet the needs of the auditor.

The newest Netcordia Tech Tip revolves around automating the compliance process and quickly highlights the primary time savings associated with automating the collection, analysis and reporting of data.

Be sure to check out the Tech Tips Library to learn how to take more control of your network today.

Unscheduled Router Change Takes Down WordPress.com and TechCrunch Blog Community

mgowarty — Fri, 19 Feb 2010 13:38:00 GMT

On February 18th, TechCrunch confirmed that its leading blog site was completely down for almost two hours because the site's host, WordPress.com, had been crippled by an unscheduled routing change to a core router.

TechCrunch is one of millions of blogs hosted on WordPress.com, and an estimated 10.2 million blogs went down because of the change, eliminating over 5.5 million pageviews.

Matt Mullenweg, WordPress founder, called it their worst outage in 4 years.

In the postmortem analysis, Mullunweg stated the initial diagnosis was, "an unscheduled change to a core router by one of our datacenter providers messed up our network in a way we haven't experienced before, and broke the site." He also added that even with the investments and mechanisms to prevent a total failure, this event tripped those up as well and the entire site crashed for 110 minutes.

Mullenweg concluded with the statement, "I hope it will be much longer than four years before we face a problem like this again."

The last statement caught my eye because of the word hope. When dealing with aspects such as a network infrastructure, hope should only be considered with things you cannot control such as an act of nature.

But an unscheduled change is something that can be addressed, monitored, and managed, so a major problem like this does not occur in the future. With the proper control, visibility, automation, and intelligence, the vast majority issues like this one can be completely eliminated. And for those that aren't eliminated, there is a huge difference if the problem is identified immediately and solved. In this example, think how much less severe the issue would be if it was caught and solved in a matter of minutes, not hours.

Organizations have invested huge sums in redundancy and back up plans to reduce the risk of catastrophic events, but often, they rely on manual processes and hope for the best when dealing with day to day management of the network infrastructure.

In situations like this, I think the old adage, "an ounce of prevention is worth more than a pound of cure," is never more relevant. Netcordia was founded on the premise of helping organizations reduce chaos and take control of network aspects such as configuration, change, and compliance.

While I also hope that outages like this never occur for organizations across the world, I think it's more important that IT organizations realize they don't need to rely on luck for managing their network infrastructure, and that with proper configuration and change management, they can reduce the risk of catastrophic events like this outage occurring again.

Spanning Layer 2 Between Data Centers

tslattery — Tue, 16 Feb 2010 15:36:00 GMT

After my posting about the dangers of spanning Layer 2 between data centers in my blog titled Spanning Tree Protocol and Failure Domains , Shivlu Jain ( www.mplsvpn.info ) asked about handling requests to span Layer 2 between data centers. One of his concerns was how to handle HA (High Availability) server configurations that require network connectivity within the same broadcast domain (subnet). Another question he had was about the deployment of FCIP (FiberChannel over IP), where the data center...(read more)

Saving Cisco Router Configuration

tslattery — Wed, 10 Feb 2010 17:52:00 GMT

As I write this, we're going through another "snow event", with a forecast of high winds and 10-20 inches of snow to top the 24-36 inches that fell a few days ago - over 67 inches so far this year, which has broken all snowfall records for Maryland. One of the network operations team sent out an email this morning asking if there was a way to save the configs on all the devices in the network as a safety step to avoid losing an unsaved config if or when power goes out at a site. I took...(read more)

Router Buffer Tuning

tslattery — Tue, 02 Feb 2010 20:08:00 GMT

Buffer tuning has long been an interesting topic for me. I recently found a blog post by Brough Turner, who wrote an interesting article about a potential misconfiguration of AT&T wireless routers that cause ping round trip times to be either < 200ms or around 8000ms (yes, 8 seconds!). http://blogs.broughturner.com/2009/10/is-att-wireless-data-congestion-selfinflicted.html There was quite a bit of controversy in the comments, which make for interesting reading. One of the things that I didn't...(read more)

Cisco Fan Failure

tslattery — Tue, 26 Jan 2010 21:12:00 GMT

Have you ever had a Cisco router or switch shutdown due to a fan failure? While looking through NetMRI 's daily list of analysis issues, I found a Fan Failure issue (it is named "Device Fan Problem" in NetMRI's Analysis page). It was really interesting to me because a fan failure produces a syslog message and it should have been caught by the NOC, who uses other tools to identify important syslog and trap messages. Of course, the problem with syslog and SNMP traps is that they typically...(read more)

The Impact of Virtualization on VoIP/IP Telephony

mgowarty — Tue, 26 Jan 2010 15:54:00 GMT

Eric Krapf, editor of No Jitter, leveraged Netcordia's 2010 Predictions of Top Causes of IT Headaches into a discussion on how virtualization can impact IP Telephony. I totally agree that voice will be one of the key applications where solid management was hard enough before virtualization, but with a new dynamic environment, the challenges grow exponentially. Besides having real-time requirements, VoIP is likely the highest-profile app any network manager will see.

In traditional best-effort networks, delay and jitter were probably only a concern if the measurements were in the minutes, not like how milliseconds is the primary measurement with VoIP. And if there is a problem with the phones, people will notice and they will complain.

Eric discussed several of the pretections and how they are hot buttons that will be addressed at VoiceCon. These include:

* The IT specialists responsible for voice/UC/real-time traffic must work much more closely with those who plan application and datacenter strategies.

* The underlying network that supports communications is dynamic, and its ongoing changes will affect the delivery of communications services--negatively so, if enterprises don't stay on top of this issue.

* The level of enterprise understanding, and vendor education, on issues relating to the cloud, virtualization, and SOA, is often inadequate, according to Tom Nolle. Tom's survey of enterprises suggests to me that there's high potential for the kind of trouble that Netcordia foresees.

At Netcordia, we see many of the new technologies (such as VoIP, virtualization/cloud, data center consolidation and new applications) all have a common criteria - even though the underlying network infrastructure is often overlooked, it is a critical component to success of new rollouts and ensuring service assurance from an organizational point of view. Without a good understanding of the network infrastructure, new deployments will likely fail and incur cost overruns.

Cisco Router Interface Wedged

tslattery — Thu, 21 Jan 2010 19:47:00 GMT

We recently had an interface wedge on a customer router, with some interesting repercussions. The network topology is shown below. Every day from about 8am until 6pm, a 250Mbps - 300Mbps traffic load starts between the Major Facility and the core network. NetMRI's interface utilization graphs show that this load has been occurring on a daily basis. The normal path for the load was from the facility, through 3550-01, to 7301-01, and on to the core network. In addition, the load was bi-directional...(read more)

Network World Lists 5 Must-Have Technology Requirements for 2010

mgowarty — Mon, 11 Jan 2010 22:01:00 GMT

As an avid follower of both Network World and Denise Dubie, I really enjoyed the a recent post on the "5 Must-Have IT Management Technologies for 2010." While the article includes service hot topic trends like virtualization/virtual systems and IT service assurance, I was excited to see IT Process Automation included on the list. Far too often, analyst and publication "new year predictions" focus on only the latest and gee whiz technologies that won't be implemented widely for years (I think I was reading how SIP and VoIP would eliminate 90% of landlines in 3 years - and that was in 1999). It's great to see how organizations will be able to successfully deploy the new technologies.

Denise's article highlights the importance of automation, especially as networks become more complex with virtualization and cloud computing. Jim Frey of EMA stated it perfectly, "People need automation for everything from provisioning virtual servers to auditing environments to ensuring consistent configurations. On the monitoring side, automation will be able to keep up with the pace of virtual environments and recognize when changes happen in ways a human operator simply could not. Automation will even be used to perform analytics and help find potential problems before they harm IT service delivery in these environments."

When dealing with new technologies, there is a huge difference between deploying the technology and deploying it successfully. IT process automation and managing network configuration, change and compliance will be a critical part in the success for organizations today, tomorrow and throughout the new decade.

Learn how Netcordia helps automate network change and configuration. You can also watch a recorded presentation on the Impact of Virtualization on Network Managers to see how managing your network infrastructure can greatly improve efficiency and reduce the risk of poor performance.

What is Bridge Assurance?

tslattery — Wed, 06 Jan 2010 15:27:00 GMT

While doing some other research recently, I ran across the Bridge Assurance feature in Cisco gear, which was new to me. Cisco has been working on ways to avoid STP melt-down and Bridge Assurance is another tool to help network administrators keep networks running. STP problems occur when forwarding loops are inadvertently created. Loops are caused by several problems: 1) Unidirectional link 2) Device malfunction 3) Configuration error 4) External system forwarding (hub or non-STP switch or dual-NIC...(read more)

Spanning Tree Protocol and Failure Domains

tslattery — Mon, 28 Dec 2009 16:11:00 GMT

Co-worker Pete Welcher recently helped a customer whose network experienced a spanning tree loop (i.e. a melt-down). Several things can be learned from thinking about the experience and how to avoid it. Lesson #1: Adequately plan the task. Rush jobs carry a higher risk of problems than well-planned tasks. In this case, the server operations team needed to bring up a new server for a project and decided to not wait for an access-layer switch to provide connectivity. Instead, a couple of ports on a...(read more)