n the last post Handling NMS Performance Data, Part 2, I suggested writing raw performance data directly into the filesystem. A simple on-disk data structure can make access to the data fast and efficient. In the NMS database, each device has a device ID. Create sets of directories to hold data from sets of devices, organized by device ID, then interface name (see picture).
The interface directories would contain the data files. The data files would be organized according to date range. Each date file would contain data collected for a period of time, such as a week. For interfaces that require high polling frequencies, the date range could be selected based on file size and growth rate. Using interface names instead of the interface ifIndex values would allow easy export of the data to other systems.
The database would contain filenames that reference the stored data. The use of device ID makes it easy to access the data, but a bit more difficult for the administrator to use. I would create a file in the device directory where additional information about the device would be stored. Such data would include the device name, the information from the SNMP system group, the list of interfaces with both name and interface ifIndex value, and inventory information such as serial numbers. While this data replicates what is in the NMS database, it allows easy export of the data to other systems.
A data structure like this makes it easy to access to the data as necessary and optimizes the common case of writing the raw data into storage. Since the raw data is in the filesystem, it becomes easy to move it to a SAN for long-term storage. The NMS database can be effiicently updated to reflect the new storage location.
In the occasional case where an administrator requests a view of historical raw data, it is accessible within a reasonable amount of time. The benefit of using the filesystem is that the access doesn’t impact the operation of the core NMS functions and database access. In fact, the function of displaying the raw data could be offloaded to a separate CPU or server.
Designing and building a network management system may look easy. But it is much more than just collecting some data and building an interface that allows the network administrator to view it. It gets much more dificult as scaling issues are encountered, corner cases crop up, and as we try to do smart things with the collected data.
-Terry
_____________________________________________________________________________________________
Re-posted with Permission
NetCraftsmen would like to acknowledge Infoblox for their permission to re-post this article which originally appeared in the Applied Infrastructure blog under http://www.infoblox.com/en/communities/blogs.html
