Does Security Belong Near Endpoints?
I’ve been mulling over the SolarWinds presentation at Network Field Day 5. I should probably start by noting that SolarWinds gave the attendees a nifty messenger bag with Network Field Day 5 embroidered on it (thanks). I have somewhat of a love/irritation relationship with the SolarWinds products. So I was mulling over was the old principle of “it you can’t be positive, best to say nothing at all” and how to apply it. I’ve ended up deciding to list my perceptions, and invite comments as to whether readers agree or disagree. The hope here is that anything negative provides useful feedback to SolarWinds that might result in an improved product. Or in me learning that I’m wrong (wait, that never happens).
My starting point here is that my consulting customers love the SolarWinds Orion and related products.
Confession: I rarely get to install the SolarWinds product(s), do discovery, help with reports, troubleshoot, etc. Consulting customers can do it for themselves! That’s pretty remarkable, when you think about it!
I used to regularly deal with CiscoWorks over the years, up to around version 3.0, when my customer base pretty much gave up on CiscoWorks. Compared to the customer experience there, the SolarWinds customer experience is quite a contrast! I do miss the occasional network management consulting… It’s nice to be needed, but as a sensitive consultant I have to recognize what’s good for the customer. I’ve done SolarWinds work a couple of times at sites with staffing anorexia (too few people), but other than that, self-service seems to be the name of the game. That’s fine with me.
Some thoughts about SolarWinds follow…
First:, the SolarWinds products are easy to buy. Reasonably clear website, download, try, pricing / quote online, etc. And no sticker shock. The product family can be seductive, as in you get SolarWinds in the door and you soon find you’ve bought several modules and the collective price is accumulating. On the other hand, we’re not talking hundreds of thousands or millions of dollars here.
This aligns with a principle Terry Slattery and I agree on (I think): most network management products are vastly overpriced. And end up not doing all that much more for you, at least not without a large and ongoing consulting expense. I keep wondering if the price of net management products is like silicon chips: the product of projected sales volume times price is roughly constant. If you price it lower, your sales go up, and vice versa. If you price it high (and go to costly direct sales), your volume goes way down. I’ll also repeat what I’ve noted: consultants who make a living off network management consulting know and love the labor-intensive high priced products. That may just be what they consider a good solution and are familiar with. The cynical point of view is that the pricey software customization work represents multiple years of income and employment for them. If that provides value to the customer, great! I personally think most sites want solutions, not tool kits and lengthy projects.
Second: the SolarWinds products work. How many of us have spent hours futzing with products (CiscoWorks LMS, HP OpenView, others), spending up to a half day trying to figure out why the product could not manage a particular device when ping and SNMP were working, etc. Stubborn cases of the product exhibiting “I see it but I’m not totally happy with it” behavior. Amazing how often vendors write code that has silly dependencies on perfect information or doesn’t allow a manual over-ride for unexpected conditions.
Third, the SolarWinds products are easy to maintain. People seem to be able to keep SolarWinds products working satisfactorily. When there’s a problem, it’s fairly easy to fix, and the support tools are helpful (or so I have found a couple of times, and hear at other sites). It strikes me that a lot of network management vendors don’t understand reduced staffing levels. Who can afford a full-time network management person? It’s a side-job, and at most sites, it’s lucky to be a 10%-time job. It’s also low priority compared to new deploy and break/fix work. We’re there to run networks, not the tools.
The same principle applies to other technologies. Sometimes I think being on top of 6500 or Nexus is a full-time or perhaps half-time job, yet in the real world, that’s maybe 10% of what many people have to deal with on a day-to-day job. I really like the Cisco Nexus platforms. Having said that, it seems that some of my value as a consultant is keeping track of all the hardware and code dependencies and gotchas and helping people with them. Sometimes before-hand, sometimes unfortunately after they’ve run into a snag. Cisco, speed of new features and cost, versus consistency and user-side simplicity?
The Achilles heel of most network management (“NM”) products is the database. People treat high end NM products with kid gloves. Products like CiscoWorks (or SolarWinds) used to sometimes be on server(s) under a desk, not on UPS, etc. Corrupt the CiscoWorks LMS database and it would be slow or inoperable. Other products, SolarWinds? I don’t have the hands-on track record there. Using Microsoft SQL server does mean the server folks can help you out with backup etc.? Lesson learned over the years: do treat your NM servers gently: shut them down cleanly, don’t just cut off the power, etc. Or else don’t be surprised if the net management behaves weirdly.
If you do a backup immediately after install, and another after your first fairly complete device discovery, restoring to those checkpoints may save you a bunch of time!
I’d like to insert one other observation about costly NM products. What I keep running into is that a site has spent $1-2 M on a product (often from CA, picked by management). Everyone hates the tool. It doesn’t work that well. It’s hard to learn. You’re not allowed to try to “fix” it. And there’s a 1/2 to 1 to 2 full time person(s) fiddling with it onsite, often for years. The biggest problem is that the high cost means the tool only covers a fraction of the datacenter (and often not the campus / LAN / MAN). It also supports the “most important” applications, often the big iron — which might not even begin to address the biggest headaches for staff.
My personal feeling is that the priority MUST be managing every active port and device in the data center. And server/VM CPU/NIC utilization/disk/swap data.
I’ve seen too many “cluster events” where many senior / expensive people are tied up in meetings to solve an application performance or other problem. In several of them, the problem would have been detected as a simple cabling, or link or server capacity or error issue, with comprehensive monitoring. Fancy root cause analysis etc. make a lot of sense, at least if they can possibly be done cost-effectively, which I have my doubts about: too much human input / coding needed?. But they miss the point.
Fancy analysis is moot if we can SEE and FIX the basic stuff. A lot of the overlay and SDN talk seems to assume a perfect (or well-managed?) infrastructure. I rarely see that kind of correct operation today.
News flash for SDN’ers: running the shiny new stuff over a crumbling infrastructure is not going to work well. I sometimes have the feeling server and programming people aren’t aware of all the things network people experience and why we do things in certain ways. Not so much the stuff where we’re set in our ways, it’s more the notion that virtualization means you can ignore the physical. Recently I heard of a situation where the question was whether an application might be having problems on a 10 Gbps port due to hypervisor dvSwitch oversubscription or internal HP blade chassis switch oversubscription. Or maybe the network. NM tool visibility would help with things like that!
Note that the network now crosses over into the server or blade server chassis. Accelerated virtualization will increase that factor. The server folks may need our networking skills to understand this!
Heck, when we take e.g. a NetMRI into a new site for a “network assessment”, we often find several hundred to thousands of duplex and bad cable error problems. I think I’ve never seen a site with fewer than 200 duplex issues. Nobody’s been aware of them due to lack of tool visibility. And sometimes it’s interesting getting them fixed, as its not part of the operational / trouble ticket process. The usual process often seems to require or prioritize human complaints (re-active repairs) rather than pro-active repairs.
You can view the SolarWinds NFD5 presentation here. A summary follows:
Now that I have your attention …
Here are some of the minor things I’ve noted over the years:
Please comment where you agree or disagree. And help SolarWinds and me out: what features do you like or hate, what would you like to see the product do, etc.?
The vendors for NFD 5 paid for my travel expenses and perhaps small items, so I wish to disclose that in my blogs now. The vendors in question are: Cisco, Brocade, Juniper, Plexxi, Ruckus, and SolarWinds. I’d like to think that my blogs aren’t influenced by that. Yes, the time spent in presentations and discussion gets me and the other attendees looking at and thinking about the various vendors’ products, marketing spin, and their points of view. I intend to try to remain as objective as possible in my blogs. I’ll concede that cool technology gets my attention!
Does Security Belong Near Endpoints?
Replicating at Speed
Practice Safe BGP
Virgilio “Bong” has sixteen years of professional experience in IT industry from academe, technical and customer support, pre-sales, post sales, project management, training and enablement. He has worked in Cisco Technical Assistance Center (TAC) as a member of the WAN and LAN Switching team. Bong now works for Tech Data as the Field Solutions Architect with a focus on Cisco Security and holds a few Cisco certifications including Fire Jumper Elite.
John is our CTO and the practice lead for a talented team of consultants focused on designing and delivering scalable and secure infrastructure solutions to customers across multiple industry verticals and technologies. Previously he has held several positions including Executive Director/Chief Architect for Global Network Services at JPMorgan Chase. In that capacity, he led a team managing network architecture and services. Prior to his role at JPMorgan Chase, John was a Distinguished Engineer at Cisco working across a number of verticals including Higher Education, Finance, Retail, Government, and Health Care.
He is an expert in working with groups to identify business needs, and align technology strategies to enable business strategies, building in agility and scalability to allow for future changes. John is experienced in the architecture and design of highly available, secure, network infrastructure and data centers, and has worked on projects worldwide. He has worked in both the business and regulatory environments for the design and deployment of complex IT infrastructures.