Statistics, Machine Learning and Artificial Intelligence
Let’s start with a fact. Many companies that claim to be AI based actually don’t use any AI. They don’t even need AI! Does it really matter? Not much for the customers since they don’t pay unless the product has sufficiently high utility for them, irrespective of whether it has AI or not. But this AI-race has had a negative effect on many companies who often scramble to add some feature which they can use to justify their AI-tag. Unfortunately, this has become necessary to be able to charge a premium, especially in the B2B segment, and also to get VC money. Zerodha is a rare species!
What makes this situation even worse is that there is no clear definition of AI! We know what is Statistics and what is Machine Learning (ML), and have ways to differentiate between these two overlapping domains, but AI is another beast altogether.
So whats Statistics, and how is ML different? At the foundation, ML is also computing statistics, but there is an important difference between the two. The simplest form of statistics is computing quantities like mean, standard deviation, and quartiles to get a sense of the variation in the data. Next step is to estimate the probability distribution that can explain the data, which is a lot harder. Estimating the probability distributions is very important for building predictive models since not all data is Gaussian. Statistics transforms itself into Machine Learning when we start using our understanding of the data to build predictive models.
The simplest such predictive model is Linear Regression. In this algorithm, the objective is to use the given data and estimate the parameters of the linear regression model, which is essentially a straight line in some high dimensional space. This parameter estimation can be done using a simple least squares approach, which has a closed form analytical solution. As we go one step ahead, we come across Logistic Regression, which has a slightly more complicated model, and for estimating its parameters, we need to use gradient descent since close form analytical solutions are not available. Both Linear and Logistic Regression are still on the strongly overlapping region between Statistics and ML, and to get into purer ML territory, we need to reach Artificial Neural Networks (ANN). I say purer since no matter where you go in ML or AI, statistics will always be there somewhere like a shadow.
Artificial Neural Networks (ANN) are truly unique because unlike Linear and Logistic Regression, the functional form of the model is variable in an ANN model. In Linear and Logistic Regression, the underlying mathematical model or the function is fixed, and we are only trying to estimate the model parameters. But in ANN, the function keeps changing as the network connections and parameters change. In other words, as we train the ANN model, we are not only scanning the parameter space, but also the function space. ANN is basically trying to automatically figure out the most appropriate function for the data we have, which is a much more advanced technique as compared to conventional statistical analysis. So yes, ANN is surely in the purer ML category!
So then whats AI, and how is it different from ML? Even in ANN models, we have to conventionally specify the input parameters or features to be used for further predictive analysis. A purer AI model does away with this need, and figures out these input features on its own! This is also called Deep Learning (DL). So in an AI model, you just feed the data into the model, and almost magically, it extracts the relevant features and builds an accurate predictive model. Algorithms like Convolutional Neural Networks (CNN) for Computer Vision (CV) and Transformer based models for Natural Language Processing (NLP) fall in this purer AI category. Once again, we use the adjective purer since even the most advanced AI algorithm is basically using statistics in an intelligent way. However, these models are so far removed from conventional statistics that we need to give them a new name. Hence, AI!
So which companies are actually using AI in their products? This is a hard question to answer since companies don’t really reveal the inner workings of their algorithms. Also, nowadays many AI models are easily available as free Python packages and so it just takes a few lines of code to incorporate them into your algorithms. There was a time when we had to write everything from scratch, but now implementing AI and ML algorithms has become as easy as computing the mean of a random variable! So then, should these products still be called AI based? Perhaps yes, because they are using these models in their product in some way even if the hard work of developing the algorithms and code was done by someone else!
