Vendors are quick to include AI in their sales pitch, but Dr Celeste Fralick, Senior Principal Engineer and Chief Data Scientist at McAfee recommends technology buyers ask three questions to help see through the hype to determine what they are actually getting.
Fralick chairs McAfee’s Analytic Center of Excellence and is responsible for the cybersecurity company’s technical analytic strategy that integrates into McAfee consumer and enterprise products.
Prior to Intel’s divestiture of McAfee, she was Chief Data Scientist in Intel’s Internet of Things Group where she developed machine learning and deep learning analytics for over eight different markets.
And while broadly applying the term AI makes it easier to market and talk about, for Fralick who has 40 years expertise in the field, each type of model under the AI umbrella has different levels of complexity and intelligence behind them.
“As a data scientist and an engineer, I look at AI to be very specific mathematically. I look at deep learning to be very different mathematically from machine learning.”
Speaking with Which-50 between sessions at MPower in Las Vegas last week, Fralick said most technology vendors claiming to do AI are really doing machine learning “and maybe a little bit of deep learning.”
Fralick shared her top three questions to help technology buyers better understand what they are getting from a machine learning solution, as well as what makes for a good or a bad answer.
1. How often does your Machine Learning algorithm actually “learn”?
The first thing Fralick recommends asking is, how often does your machine learning algorithm actually “learn”?
Models change over time, meaning when new information comes in the model is updating itself to be more relevant or more accurate. The process is outlined in the image below.
A correct response will indicate, “The algorithm learns at the rate that was determined at the time of model development and is updated periodically and applied to new signals.”
Some models may be updated continuously, while other models, they are updated based on statistically significant changes to inputs to the model, Fralick said.
Buyers should be wary of vague answers like “routinely”. Good follow up questions include “How often?”, “What signals does your company use to update the model?” Or even the very basic “How do you know the model is working?”
2. How accurate is your Machine Learning model?
Nothing is ever 100 per cent accurate.
“Accuracy is the measure of error,” Fralick explained. “So if your ML model is not as accurate as it could be, it means it has more error and giving you incorrect decisions.”
Error is measured differently for different models, such as false positives, false negatives, true positives and true negatives. But there are other ways to calculate error which require taking a look at the math. Fralick provided an example of a correct answer:
“The specific ML model for this purpose has a 95 per cent accuracy. We also measure other forms of accuracy, such as Root Mean Square Error, Generalized R^2, and additional ROC metrics such as Recall, F1 Score, and Matthew’s Correlation Coefficient.”
“A follow-up question would be: So I have a 5 per cent chance of having a wrong answer. How does that break down into False Positives and False Negatives?”
3. Is your ML model predictive or diagnostic?
There are four types of outputs ML can determine when it is applied to data. Descriptive (what happened?), diagnostic (why did it happen?), predictive (what will happen?) and prescriptive (here are the recommendations for avoiding what will happen).
Fralick explained that each of these models builds on the one before it.
“You must have descriptive to do diagnostic, and you must have diagnostic to do predictive, and, lastly, you must have predictive to do prescriptive.
“Currently, in almost all industries (including security), ML is descriptive and/or diagnostic. It relates to what has happened in the past. A signal occurs, and the model is applied.
“With predictive ML, we are predicting what will happen in the future, for example ‘we can predict that you have a 95 per cent chance of being hacked tomorrow’.”
If a vendor says ML is predictive, Fralick recommends digging deeper by asking, How far in the future does it predict? How accurate is it (back to Question 2)?
If the vendor is predictive, ask if they are also prescriptive, for example ‘Do you provide recommendation on how to avoid this attack from happening in the future?’
The post How often does your algorithm learn? And other important questions to ask ML vendors appeared first on Which-50.
