Security Notes: Confessions of a Security Consultant

 

Gideon Samid • Gideon@AGSgo.com

 

 

 

“How many false positives will you tolerate to stop a single hacker from scoring a successful hit?”

 

This is a question I have learned to ask our clients. They squirm in their chair, coming up with all sorts of excuses not to answer. I continue: “If a hacker penetrates all the security guards I install around your data, you will hold me responsible, right? My reputation is on the line. But if a bona fide customer were turned down and walked away, first, you will not know about it, and second, my reputation will not suffer. So, like other security consultants, we tend to let you reject a whole bunch of good business risks, because we face the nightmare of a successful penetration.

 

“You, as the client, should understand this. Furthermore, you should insist that we come up with an official estimate as to the false positives that we drive away from your business. You should even test us (it can be done), and you should estimate the total loss. It’s not only the loss of the item you could have sold to that rejected customer, it is all the future merchandise that you won’t sell now to that same person. Your falsely rejected would-be customer is likely to blog about his experience, blast you on his Facebook page, and enter a negative comment on the many consumer outlet sites. In fact, the loss due to false positives is not only unattributed to the security consultant, it is unattributed to you, the shop manager. It fizzles its way in the accounting books with no one to blame. It expresses itself as less business.”

 

Having said all that, I quickly add: “If you close down the hot-water faucet in your bath, you chill the water—necessarily. Same here. If you loosen up the rejection algorithm, you increase the risk of a devastating penetration of your system. No two ways about it. You can’t ease up on the false positives without increasing the false negatives. There is always a give and take. The question is: Do you know where the ‘needle’ should be? And who makes that decision?”

 

The phenomenon of false positives creeps up in more and more places and instances throughout the digital-payments landscape. It applies to rejecting bona fide online customers. It applies to denying a customer her requested funds transfer. It locks up financial databases against legitimate queries. Increasingly, from logon to logoff, a user is being subjected to scrutiny where every move must be blessed with an “admit” decision by the selection algorithm, thereby making false positives more likely.

 

Most organizations are intimidated by the overwhelming complexity of security algorithms, which prepare a stew of hundreds of parameters, cook them in a mathematical pressure-cooker, and yield an admit/reject decision. Too many equate mathematical complexity with credibility. The reality is that mathematical complexity increases the potential for a smart selection, but it is like a faster car. It makes it more likely to arrive on time, but it’s also more likely to crash.

 

Here is a basic street reality: Some geeks in a basement somewhere come up with a mumbo-jumbo mathematical tool usually described with erudite terms like “discriminant analysis” and “clustering techniques,” and they proudly showcase how powerful it is. (They do so on a case of their choice, having the freedom to discard less favorable results). The new product is then installed by less geeky installers on a client’s customer inlet.

 

Now here is a dirty secret regarding these math wonders: No matter which algorithm you use, the process of drawing conclusions from data is not clean. It invariably involves out-of-the-blue, unfounded, arbitrary input parameters. Granted, it is rarely presented this way. The books refer to it as calibration. But a little bit of such obscure input will render a bunch of bona fide customers into rejects, and the same in the opposite direction. If you are not aware of the temptation for the security consultant to err self-protectively towards false positives, then you don’t measure, don’t assess, don’t realize, and don’t complain. It’s a silent loss.

 

So, first, clearly understand the nature of this confession. We in the security business will err towards false positives at any time because our nightmare scenario is a false negative. Second, apply known tools to measure the cost and damage due to false positives. And, third, you—and no one else—should decide on the balance point between false negatives and false positives.