Matt's Blog

Change and Configuration Management - Is it your strength or weakness

MWalsh — Thu, 24 Jun 2010 20:14:00 GMT

The recent SC magazine article, “Winds of Change: Change and Configuration Management,” drove home the necessity that organizations have strong change and configuration strategies in place – before the emergency hits.

It reminded me of a colleague’s comment more than a decade ago –“If it’s not a strength – then it’s a weakness.” In the world of IT and change and configuration management, this is absolutely true. How better to exemplify this than with UNICEF’s ability to avert risk and loss of crucial services – because they had a change and configuration strategy in place – they were prepared for disaster.

How many organizations have limited or no change and configuration management practice? Without one, how fast do you think you could get your location or region back up and running at full steam? What kind of havoc would the undocumented aspects and ad hoc management of your network over the years cause in terms of business service delays – and headaches?

While natural disasters get all of the headlines – it doesn’t take a huge event to cause chaos. It could be a local fire, a hardware failure or planned maintenance. If an organization doesn’t have solid documentation, visibility and a baseline of best practices, IT staff will have to react – and do whatever it takes to put a “functioning” solution in place. While this may get them back online – business services and potentially security are at risk – because standards, configurations and most recent improvements may be overlooked.

There are several best practices to make change and configuration management – especially for the network – a strength not a weakness:

- Automate NCCM – the time and effort to manually keep configurations and changes is huge – it’s virtually impossible to keep up to date

- Best practices – Develop, implement and track best practices and gold standards - throughout the device lifecycle

- Intelligence – embedded intelligence can help find subtle and the long term impact of changes – very often overlooked

- Documentation and tracking –enhancements can fall by the wayside without documentation and tracking

- Holistic view – NCCM touches many facets of the organization – restricting the scope to just one (security or compliance and not the entire network) limits benefits and potential

So if disaster occurs elsewhere, and you’re thinking about all of the things that could go wrong, remember that a strong NCCM strategy is a strong foundation for success. Let’s hope your organization never has to face a major catastrophe but if you do, make change and configuration management a strength, not a weakness.

Lack of Network Security Stings Dave & Buster’s, Highlights Value of PCI Standards

jhaines — Tue, 30 Mar 2010 16:37:00 GMT

If you’re ever been to a Dave & Buster’s or Jillian’s restaurant and entertainment complex, you’ve probably had a great time with lots of fun and games. The IT and security team recently found out that dealings with the Federal Trade Commission (FTC) when credit card information is compromised, is no fun at all.

Dave & Buster’s recently settled an FTC charge that it “failed to protect customers’ information when more than 130,000 credit and debit cards were compromised.” Dave & Buster’s failed to take sufficient measures to detect and prevent unauthorized access to the network and adequately restrict outside access to the network including access by its service providers. This was the FTC’s 27th case challenging faulty security practices for protecting sensitive consumer information.

As part of the settlement, Dave & Buster’s must establish and maintain a program designed to protect the security, confidentiality, and integrity of personal information collected from customers. Add to that a requirement for complete independent, professional audits, every other year for 10 years. Beyond taking the appropriate security measures, Dave & Buster’s now has to prove they’re keeping the infrastructure secure–that’s a lot of time and resources.

This recent announcement reminded me of the Payment Card Industry Data Security Standards (PCI DSS) and the network security standards that are a key part of the compliance mandate. Within the last few years, more and more companies have called on Netcordia to help maintain a safe and secure network infrastructure. With PCI DSS standards and rules embedded, organizations like the University of Houston have tapped NetMRI to provide verification of compliance with an automated report. The result–a successful audit without pulling staff off their game for weeks to compile necessary information.

From a network point of view, the good news is the availability of best practices and tools to help maintain a safe and secure network.

How Redundancy Hides Issues and Can Cause Bigger Problems Later

mgowarty — Wed, 24 Mar 2010 19:23:00 GMT

Every successful IT organization has built-in redundancy and disaster recovery plans, but often overlook how these can often mask problems for a period of time and then when something else occurs - BAM - bigger problem or complete outage.

Wikipedia just reported a recent global outage that had a huge impact. When its data centers servers in Europe overheated, the standard quick failover procedure re-routed traffic to their Florida cluster. However, shortly after the failover switch, it was determined the mechanism was now broken, causing Wikimedia sites to stop working globally. Even though the problem was found quickly and resolved, there are still performance delays hours after the event.

This outage just brought back the memories and horror stories I've heard from organizations where redundancy caused a false sense of security and compounded the problem when a problem arose. It reminded me of a story a friend told me the other day.

In this organization, they had an Access Layer 2950 switch that had dual fiber uplinks to two 6500 Catalyst switches and one of his techs disturbed the wrong fiber patch. Since it was redundant, there was no service impact. Months later, there was a problem on the other fiber patch and the issue was catastrophic because now both uplinks were out of commission. They were caught completely blindsided because they thought they were safe because of a strong redundancy or disaster recovery plan. Once the problem happened, they spent a lot of time trying to figure out how it occurred and how to avoid it in the future.

The story just reminded me what happens when we get over confident, we get complacent. Instead of looking for potential issues lurking on the network or the devices that could cause a problem hours, days, weeks or months later, we tend to assume everything will work fine or our failover plans will save us. As networks get more complex and more critical, I suspect these issues will continue until another major outage occurs or someone decides to get ahead of the curve and find the solution that looks for these potential issues before they cause a bigger problem later.

Capacity Planning - Do you know which ports are still being used?

mgowarty — Fri, 05 Mar 2010 20:09:00 GMT

Over the past several years, the capacity planning area for IT professionals has become much more challenging when dealing with managing active switch ports. For most organizations, the challenge is threefold:

The sheer number of ports requested has continued to grow year over year.
The transient nature of users keeps the infrastructure in continuous state of flux as users plug into an active for a period of time and then move on when done with that requirement.
The current economic state has greatly reduced expansion in regards to equipment, so the number of new ports hasn't grown as fast as the new demand for most organizations.

Keeping track of which ports are being used or were used but now are available isn't rocket science, it just falls to the bottom of the priority list for most IT staff. It's much easier to just add a new port you know is available instead of manually digging and sifting through each device to decipher where previously used ports are now available.

I recently published a new tech tip on the port saturation issue that covers how to better plan for capacity requirements. The good news is automation is available to help with this task as well as better managing your entire network infrastructure.

Default Passwords Unlock the Back Door for Security Attacks

mgowarty — Wed, 24 Feb 2010 12:57:00 GMT

A recent PCMag article by Larry Seltzer highlighted some of the security risks within routing infrastructures and how the uniquely named Chuck Norris botnet looks across network devices and finds default passwords as an entry point to spread harmful code or take control of devices. Even though this one has been named Chuck Norris, I’m much more a fan of Sydney Bristow who doesn't seem so 1980's.

It is interesting that organizations spend thousands or millions of dollars trying to build secure infrastructures by putting up the “biggest fence” to build a strong perimeter, but like any action/spy fan knows, all it takes is a little help from the inside or one small weakness and wham, the bad guys are in. While a strong foundation is important, it’s not the only step to ensure a secure and protected environment.

Although this article focused on the individual PC world, the problem can be just as huge for organizations dealing with their routing and switching network infrastructure. I’m scared to admit the amount of times I’ve seen organizations leave default passwords on their devices, or never change passwords when employees leave to join competitors or are terminated. That small back door is just enough for someone to get in and wreak havoc. Adding to the potential pain, once in, most IT teams would never see any modifications or changes unless an outage was caused. And the smarter bad guys know how to get the information they want without triggering alarms or build a sinister plot to cause as much damage as possible.

Organizations need to build and maintain a strong and secure infrastructure. With embedded intelligence and automation, users can set rules and policies notifying the IT team when passwords are weak or not changed periodically, and, if a bad guy happens to get in through the front or back door, the system logs and tracks all changes to the network infrastructure devices. So if an unplanned change is made, an issue is generated and the good guys can stop the bomb from going off and damaging the entire system.

New Tech Tip: Automating the Compliance Process

mgowarty — Tue, 23 Feb 2010 13:28:00 GMT

When dealing with both internal and external compliance requirements, IT staffs typically must shoulder the burden of proving the network infrastructure meets and maintains the best practices/gold standards (internal) or regulatory mandates (external). Traditionally, this process uses a great deal of manual time and effort to log in, capture the configs, review, compile and report. For a 150 device network, this effort could take as much as 24 hours to meet the needs of the auditor.

The newest Netcordia Tech Tip revolves around automating the compliance process and quickly highlights the primary time savings associated with automating the collection, analysis and reporting of data.

Be sure to check out the Tech Tips Library to learn how to take more control of your network today.

Unscheduled Router Change Takes Down WordPress.com and TechCrunch Blog Community

mgowarty — Fri, 19 Feb 2010 13:38:00 GMT

On February 18th, TechCrunch confirmed that its leading blog site was completely down for almost two hours because the site's host, WordPress.com, had been crippled by an unscheduled routing change to a core router.

TechCrunch is one of millions of blogs hosted on WordPress.com, and an estimated 10.2 million blogs went down because of the change, eliminating over 5.5 million pageviews.

Matt Mullenweg, WordPress founder, called it their worst outage in 4 years.

In the postmortem analysis, Mullunweg stated the initial diagnosis was, "an unscheduled change to a core router by one of our datacenter providers messed up our network in a way we haven't experienced before, and broke the site." He also added that even with the investments and mechanisms to prevent a total failure, this event tripped those up as well and the entire site crashed for 110 minutes.

Mullenweg concluded with the statement, "I hope it will be much longer than four years before we face a problem like this again."

The last statement caught my eye because of the word hope. When dealing with aspects such as a network infrastructure, hope should only be considered with things you cannot control such as an act of nature.

But an unscheduled change is something that can be addressed, monitored, and managed, so a major problem like this does not occur in the future. With the proper control, visibility, automation, and intelligence, the vast majority issues like this one can be completely eliminated. And for those that aren't eliminated, there is a huge difference if the problem is identified immediately and solved. In this example, think how much less severe the issue would be if it was caught and solved in a matter of minutes, not hours.

Organizations have invested huge sums in redundancy and back up plans to reduce the risk of catastrophic events, but often, they rely on manual processes and hope for the best when dealing with day to day management of the network infrastructure.

In situations like this, I think the old adage, "an ounce of prevention is worth more than a pound of cure," is never more relevant. Netcordia was founded on the premise of helping organizations reduce chaos and take control of network aspects such as configuration, change, and compliance.

While I also hope that outages like this never occur for organizations across the world, I think it's more important that IT organizations realize they don't need to rely on luck for managing their network infrastructure, and that with proper configuration and change management, they can reduce the risk of catastrophic events like this outage occurring again.

The Impact of Virtualization on VoIP/IP Telephony

mgowarty — Tue, 26 Jan 2010 15:54:00 GMT

Eric Krapf, editor of No Jitter, leveraged Netcordia's 2010 Predictions of Top Causes of IT Headaches into a discussion on how virtualization can impact IP Telephony. I totally agree that voice will be one of the key applications where solid management was hard enough before virtualization, but with a new dynamic environment, the challenges grow exponentially. Besides having real-time requirements, VoIP is likely the highest-profile app any network manager will see.

In traditional best-effort networks, delay and jitter were probably only a concern if the measurements were in the minutes, not like how milliseconds is the primary measurement with VoIP. And if there is a problem with the phones, people will notice and they will complain.

Eric discussed several of the pretections and how they are hot buttons that will be addressed at VoiceCon. These include:

* The IT specialists responsible for voice/UC/real-time traffic must work much more closely with those who plan application and datacenter strategies.

* The underlying network that supports communications is dynamic, and its ongoing changes will affect the delivery of communications services--negatively so, if enterprises don't stay on top of this issue.

* The level of enterprise understanding, and vendor education, on issues relating to the cloud, virtualization, and SOA, is often inadequate, according to Tom Nolle. Tom's survey of enterprises suggests to me that there's high potential for the kind of trouble that Netcordia foresees.

At Netcordia, we see many of the new technologies (such as VoIP, virtualization/cloud, data center consolidation and new applications) all have a common criteria - even though the underlying network infrastructure is often overlooked, it is a critical component to success of new rollouts and ensuring service assurance from an organizational point of view. Without a good understanding of the network infrastructure, new deployments will likely fail and incur cost overruns.

Network World Lists 5 Must-Have Technology Requirements for 2010

mgowarty — Mon, 11 Jan 2010 22:01:00 GMT

As an avid follower of both Network World and Denise Dubie, I really enjoyed the a recent post on the "5 Must-Have IT Management Technologies for 2010." While the article includes service hot topic trends like virtualization/virtual systems and IT service assurance, I was excited to see IT Process Automation included on the list. Far too often, analyst and publication "new year predictions" focus on only the latest and gee whiz technologies that won't be implemented widely for years (I think I was reading how SIP and VoIP would eliminate 90% of landlines in 3 years - and that was in 1999). It's great to see how organizations will be able to successfully deploy the new technologies.

Denise's article highlights the importance of automation, especially as networks become more complex with virtualization and cloud computing. Jim Frey of EMA stated it perfectly, "People need automation for everything from provisioning virtual servers to auditing environments to ensuring consistent configurations. On the monitoring side, automation will be able to keep up with the pace of virtual environments and recognize when changes happen in ways a human operator simply could not. Automation will even be used to perform analytics and help find potential problems before they harm IT service delivery in these environments."

When dealing with new technologies, there is a huge difference between deploying the technology and deploying it successfully. IT process automation and managing network configuration, change and compliance will be a critical part in the success for organizations today, tomorrow and throughout the new decade.

Learn how Netcordia helps automate network change and configuration. You can also watch a recorded presentation on the Impact of Virtualization on Network Managers to see how managing your network infrastructure can greatly improve efficiency and reduce the risk of poor performance.

White House CyberSecurity Officer Includes Network in Biggest Security Risks

mgowarty — Wed, 23 Dec 2009 17:24:00 GMT

The new White House CyberSecurity Officer has shared his top 10 predictions for 2010 and the network has now made it to the list. Bill Brenner takes it a step further, in yesterday’s article discussing the added risks to the network associated with 2009’s economic fallout – from disgruntled former employees compounded by the unprecedented number of layoffs highlights. There are two best practices associated with reducing this risk:

Eliminating a single log in access and proactively monitoring change. IT organizations tend to have a single login to reduce the complexity of maintaining individual user roles and rights. While IT staff often understands user-based access eliminates much of the unnecessary risk associated with a single admin account, they tend not to implement because it takes time. For example, if one person leaves, changing or eliminating their single access rights is easier. IT organizations are often guilty of giving full access or admin rights to network devices, and if there is a layoff or a person leaves, many times the passwords and access aren’t changed or if they are changed, the passwords usually just change a number at the end which can be easily guessed. This leads to additional risk associated with changing key configurations in devices across the network.
Insure against unwanted or unplanned changes. Many organizations assume a change management process or change window eliminates the risks of unplanned changes, but if the modification is meant to cause harm, it will never go through the process. If the attacker gets through the processes and security and actually makes a change, the organization needs to know exactly what changed on what device quickly and easily. Typically problems occur when the changes go unnoticed for hours, days or weeks, but if you are alerted to an unplanned changed and can see exactly what was modified, the IT team can resolve the issue much faster and eliminate the majority of risk.

We’ve got a whitepaper available on “Avoiding the Top 5 Network Management Mistakes” at http://www.netcordia.com/resources/whitepapers.asp. It covers these issues and other common mistakes that add unnecessary exposure for organizations.