My previous post talked about the basics of verifying a configuration prior to deployment as a way to minimize the number of incorrect configurations that are fielded. The second topic, which is closer to the original NANOG posting comments, is with respect to auditing existing configurations to make sure that they match your organization's policies.
Organizational policies originate from multiple sources:
- regulatory policies such as SOX, PCI, HIPAA, GLBP, etc;
- industry best practices such as placement of root bridge, MPLS design parameters, security, and ARP and CAM timers;
- organizational policies including address ranges for loopbacks, device naming, redundancy configuration, and additional security.
The figure below shows how policies are typically implemented. It generally starts with a text document of some sort that describes the policy and its intent. The next step is to turn the policies into device templates. These templates are for each device type and its intended role in the network. For example, a 6500 that's a core device will have a different configuration than a similar hardware platform that is deployed in a distribution or access layer role, resulting in multiple templates for the same/similar hardware. Once the templates are created, the per-device configurations are created, including specific IP addresses, VLAN definitions, routing neighbors (if needed), etc. These final configurations are installed on the hardware.
Configuration audit, or validation, has two major functions. The first is to verify that the fielded configurations match the templates that are derived from the policies. Validation will detect unauthorized changes or deviations from defined policies, proactively allowing the networking team to correct the configuration before a problem occurs (like failing a SOX or PCI audit, or having a security breech occur). Such validation processes also help organizations more easily pass SOX or PCI audits. It makes the auditor's job easier when you can point to your policies, templates, and process for verifying that the resulting configurations are correctly deployed across the network. It also helps significantly if you can point to specific exceptions to the policies. There are always exceptions and it it helpful to be able to easily identify them to the auditors and be able to explain how they do not impact the integrity of your network.
The second purpose of doing configuration validation has more to do with changes in policy. Let's take a simple case where you changed the location of your DNS server or your NTP server. All your routers and switches will need to be updated to reflect the new server locations. Many organizations take a snapshot of their device inventory (assuming that they have an accurate inventory) into a spreadsheet and assign someone to make the configuration changes. If other changes are happening in the network (device move/add/change/delete - MACD), then the spreadsheet list is soon out of date. The combination of a device list and MACD work means that there's the possibility of a device being missed in the shuffle. However, with a good template, it is easy to verify the configurations of all devices known to the system against that template. A good auto discovery is essential to having an up-to-date inventory so that no devices are missed. Any configurations that don't match the policy for whatever reason, will be identified. This now becomes the check-list of devices that need a configuration update.
Even better, is to have a system that when a simple policy is incorrect, it can automatically update the configuration. A good example of this is updating NTP or DNS server addresses.
Some network staff members are reluctant to use automated tools because they think it threatens their job. But I've never met anyone who likes the incredibly boring job of applying a simple configuration update across hundreds of devices. Using automated tools allows the network staff to tackle more interesting jobs. And with the right tools, you can make the network more reliable, allowing you to spend more time on interesting network upgrades.
-Terry