Perception vs Reality How to Use Two Different Tools in Selecting an Enterprise Threat Prevention System: Gartner’s MQ & NSS Labs
Perception – Gartner & The Magic Quadrant
Most people are familiar with Gartner and their brilliant marketing term, The Magic Quadrant. That is where Gartner rates companies’ products on an X/Y axis. The X-axis is “completeness of vision,” and Y-axis is “ability to execute.” This rating is, of course, based on customers’ feedback, opinions and experiences of a number of things that a particular product does in the market place. In other words, it’s the perception of that product by its customers. Their entire methodology is explained here.
I do appreciate the information that Gartner provides about the end user perception. It is also important to know how it is derived because it is not based on testing in a lab environment. And understand, it is one data point when selecting an enterprise threat prevention system.
If you look back over the years of Gartner MQ results, you will see companies that come and go, but one that is consistent is Check Point Software Technologies. They are consistently in the upper right-hand quadrant — the place where everyone wants to be. They may move up, down, over, but they are always in the desired quadrant.
Cautions with Gartner & Check Point
There are not many engineers who would argue that Check Point does NOT provide the best security. This is backed up by the Gartner MQ staying power year over year. What I want to focus on are the cautions issued by Gartner for Check Point in their 2016 Magic Quadrant for Enterprise Network Firewalls. I’m going to summarize.
First caution. The subscription blades are included for free when the appliance is purchased, and the next year’s renewal is shocking to some.
Second caution. The sizing tool sizes appliances too tightly, perhaps leading to performance issues
Third caution. Gartner receives anecdotal report from clients about support issues, based on time to resolution and escalation.
Fourth caution. Check Point offers firewall integration with its Capsule and Sandblast product and Gartner reports that customers are slow to purchase this product
Fifth caution. Check Point underperforms in marketing and brand recognition
Non-Tech Issues
I will address number 4 and number 5 first, because they are not technical issues. As for integration with Sandblast and Capsule, if customers are slow to purchase these add-ons I’m not sure that this is a caution. At least they are able to be integrated without another management console.
And number 5, well, the fact that Check Point lacks recognition at marketing is well known. Check Point’s marketing is especially bad given the reliability and historic catch rate of their appliances. Everyone knows Palo Alto is the best marketed firewall on the market. The question is, “Do you want the best marketed firewall, or the best firewall?” (which I will address below in the reality section).
Other Concerns
Non-technical cautions out of the way, let’s look at Gartner’s other concerns –all of which I would argue are solved with a good partner.
The subscriptions are a shocker at renewal time.
This only occurs if you have a bad integrator. On the other hand, if you work with
- Someone who knows Check Point inside and out;
- A partner for your business, who shows you exactly what your year 1-3-5 costs will be;
- Someone who works on your behalf to get you the best pricing available;
then there is no sticker shock. In fact, the total investment will likely be less than the competition’s costs over time.
What happens if you work with a big-box reseller? Not mentioning any names here — one that is almost spelled like SHE or one that is similar to CDWG (but does not sell to government). Yes, you may be shocked at renewal time. Why? Because those reps are interested in making a quota this month or this quarter. They want to sell a widget today, not complicate the situation with the longer, drawn-out process that helps the customer understand the out-years.
Sizing Tool Sizes Appliances Too Tightly
Similar to the subscription shocker, if you had a partner who worked with you to size the appliance properly, this would not be an issue. I have only seen sizing too tightly occur when a competitive situation involves Palo (or other) vs. Check Point and where both of them are sizing their boxes down. Palo does it and so does Check Point. One of the reasons that Palo may perform better in these situations is they come out of the box configured with less security. However, when you configure it as tightly as it should be configured, Palo actually performs worse than the Check Point boxes. I have seen it multiple times in multiple different bake-offs. I can point to default configuration examples if you are interested.
Support
I will not argue that Check Point’s support could be improved. However, with the right partner support, these issues become non-issues. Once again, you are out of luck with a big-box reseller because, if you are not buying anything, they don’t have time to help. Even when you are buying something, by time all of the people needed for the conference call are finally on line, your support problem is likely to already have been handled.
Summary of Perception
The perception problem is a tough one, because, as they say, perception is reality. It is hard to change someone’s perception. However, if you are interested in the best security – Check Point is the obvious choice. If you are interested in the best security with the least amount of perceived problems — Check Point with a competent partner is absolutely the way to go.
Reality – NSS Labs
Above we discussed the perception measurement which is Gartner. Now we are going to talk about where the “rubber meets the road” non-partial lab testing. A company called NSS Labs https://www.nsslabs.com/ measures the performance of competing vendors and holds them to their claims in a laboratory setting.
NSS Labs describes their methodology in detail here. An excerpt of that document summarizes:
NSS Labs’ test reports are designed to address the challenges faced by enterprise security and IT professionals in selecting and managing security products. The scope of this particular methodology includes:
- Security effectiveness
- Performance
- Stability and reliability
- Total cost of ownership (TCO)
As NGFWs are deployed at critical choke points in the network, their stability and reliability is imperative. Therefore, regardless of any new deep inspection capabilities, the main requirement of any NGFW is that it must be as stable, as reliable, as fast, and as flexible as the firewall that it is replacing.
Reality of Two Top Vendors
Similar to Gartner and Check Point’s staying power year over year. Check Point has never performed in an NSS labs test lower than the best rating of “Recommended.”
In contrast, after an initial assessment by NSS labs, this blurb showed up in the EULA for Palo Alto. I do not know for sure why this is in Palo Alto’s EULA, but it definitely lead to speculation. None of which is quite good in my opinion. I will not deny, however, that they absolutely have the best marketing.
NSS Labs & Real-World Tests
In each of the NSS Labs’ testing for FW/NGFW/IPS/Unknown Malware, Check Point has outperformed. Since 2011 NSS Labs recommended Check Point nine times. Each FW, IPS and NGFW in which they were tested they received “Recommended.” In one test, they have also caught 100% of “unknown” malware, the first time that ever happened in NSS Lab’s testing.
In terms of recommendations NSS gives three grades: Recommended, Caution or Neutral. Only two vendors, Check Point and Cisco, have never received a rating of CAUTION. Out of 13 tests, Cisco was Recommended seven times and Neutral six times.
Over the history of the recommendations, Check Point is the only vendor to have been Recommended every year. You can see a complete picture at the NSS site for yourself, but these tests included competitors such as Fortinet, Cisco, Palo, Juniper and FireEye.
Block Rates
In terms of block rate for exploits year-over-year, Check Point is at 100% since 2012. And while there are some competitors that hit 100% some years, they are not consistent. Just like the Gartner MQ. It is on slide 3 of 5 at the following link: www.slideshare.net/zztop_2764/fortinet-nss-ngfw-2016-latency-catchrate.
In the past, after seeing the slide, some folks said, “But look at competitor X, they are at 100% some of those years.” My response: “Just some???” If security is important, then consistency and trends are what you should be considering.
Review, too, the completeness of the solution. Back once again to the Gartner perception. Can you have the same security in your datacenter, in the public and private cloud, and on the endpoint – all controlled by one pane of glass? You can with Check Point. The competitor that got 100% for one year — likely is not even close.
Summary of Reality
In summary, it is my opinion that, with the right partner, Check Point has no “Cautions,” or weaknesses. They provide the best security when tested side by side with their peers. And they have the leading vision for managing the complete picture with a single pane of glass.
I realize the haters and the lovers of other technologies are going to scream. I understand. However, if you want to have a bake-off, I would be happy to assist. Of course, you can just allow NSS Labs to do it for you.
If you disagree with any of these statements, or you would like to hear more about any single point, please contact me. I would be happy to discuss your thoughts on the subject.