What is Explainable AI?

Explainable AI (XAI) is a set of tools and frameworks that can be used to help you understand how your machine learning models make decisions. This shouldn’t be confused with showing a complete step-by-step deconstruction of an AI model, which can be close to impossible if you’re attempting to trace the millions of parameters used in deep learning algorithms. Rather, XAI aims to provide insights into how models work, so human experts are able to understand the logic that goes into making a decision.

When you apply XAI successfully, it offers three important benefits:

1. Increases trust in ML models
When decision-makers and other stakeholders have more visibility into how an ML model found its final output, they are more likely to trust AI-based systems. Explainable AI tools can be used to provide clear and understandable explanations of the reasoning that led to the model’s output. Say you are using a deep learning model to analyze medical images like X-rays, you can use explainable AI to produce saliency maps (i.e. heatmaps) that highlight the pixels that were used to get the diagnosis. For instance, an ML model that classifies a fracture would also highlight the pixels used to determine that the patient is suffering from a fracture.

2. Improves overall troubleshooting
Explainability in AI can also enable you to debug a model and troubleshoot how well a model is working. Let’s imagine your model is supposed to be able to identify animals in images. Over time, you notice that the model keeps classifying images of dogs playing in the snow as foxes. Explainable AI tools make it easier to figure out why this error keeps occurring. As you look into the explainable AI models that you’re using to show how a prediction is made, you discover that that ML model is using the background of an image to differentiate between dogs and foxes. The model has mistakenly learned that domestic backgrounds are dogs and snow in an image means the image contains a fox.

3. Busts biases and other potential AI potholes
XAI is also useful for identifying sources of bias. For example, you might have a model to identify when cars are making illegal left-hand turns. When you are asked to define what the violation is based on in an image, you find out that the model has picked up a bias from the training data. Instead of focusing on cars turning left illegally, it’s looking to see if there is a pothole. This influence could be caused by a skewed dataset that contained a large number of images taken on poorly maintained roads, or even real-bias, where a ticket might be more likely to be given out in an underfunded area of a city.

Where does explainability fit into the ML lifecycle?

Explainable AI should not be an afterthought that’s done at the end of your ML workflow. Instead, explainability should be integrated and applied every step of the way—from data collection, processing to model training, evaluation, and model serving.

There are a few ways you can work explainability into your ML lifecycle. This could mean using explainable AI to identify data set imbalances, ensure model behavior satisfies specific rules and fairness metrics, or show model behavior both locally and globally. For instance, if a model was trained using synthetic data, you need to ensure it behaves the same when it uses real data. Or, as discussed above with deep learning models for medical imaging, a common form of explainability is to create heatmaps to identify the pixels used for image classification.

Another tool you might use is sliced evaluations of machine learning model performance. You should avoid creating or reinforcing unfair bias. AI algorithms and datasets can often reflect or reinforce unfair biases. If you notice that a model is not performing well for a small minority of cases, it’s important for you to address any fairness concerns. Sliced evaluations will allow you to explore how different parts of a dataset might be affecting your results. In the case of imaging models, you might explore different images based on factors like poor lighting or over-exposure.

It’s also recommended to create model cards, which can help explain any potential limitations, any trade-offs you have to make for performance, and then, providing a way to test out what the model does.

Explainable AI methods

When we talk about explainable AI methods, it’s important to understand the difference between global and local methods.

• A global method is understanding the overall structure of how a model makes a decision.
• A local method is understanding how the model made decisions for a single instance.

For instance, a global method might be that you look at a table that includes all the features that were used, ranked by the overall importance they have for making a decision. Feature importance tables are commonly used to explain structured data models to help people understand how specific input variables impact the final output of a model.

But what about explaining how a model makes a decision for an individual prediction or a specific person? This is where local methods come into play.

Here are the most common explainable AI local methods:

• Local interpretable model-agnostic explanation (LIME)
• Kernel Shapley additive explanations (KernalSHAP)
• Integrated gradients (IG)
• Explainable explanations through AI (XRAI)

Both LIME and KernalShap break down an image into patches, which are randomly sampled from the prediction to create a number of perturbed (i.e. changed) images. The image will look like the original, but parts of the image have been zeroed out. Perturbed images are then fed to the trained model and asked to make a prediction.

Integrated gradients is a technique used to give importance value based on gradients of the final output. IG takes baseline images and compares them to the actual pixel value of the images that contain the information the corresponding model is designed to identify. The idea is that the value should improve in accuracy when the image contains what the model was trained to find. It helps determine how much a gradient changes from the baseline image to the point where it makes a prediction, providing an attribution mask that helps determine what the image is using to classify an image.

XRAI is a technique that combines all of the three methods mentioned above, combining patch identification with integrated gradients to show salient regions that have the most impact on a decision, rather than individual pixels. The larger regions in this approach tend to deliver better results.

Using Explainable AI in Google Cloud

Google launched Vertex Explainable AI to help data scientists not only improve their models but provide insights that make them more accessible for decision-makers.

With Vertex Explainable AI platform, you can:

1. Design interpretable and inclusive AI. Build AI systems from the ground up with Vertex Explainable AI tools designed to help detect and resolve bias, drift, and other gaps in data and models. With AI Explanations, data scientists can use AutoML Tables, Vertex Predictions, and Notebooks to explain how much a factor contributed to model predictions, helping to improve datasets and model architecture. The What-If Tool enables you to investigate model performance across a wide range of features, optimize strategies, and even manipulate individual datapoint values.
2. Deploy ML models with confidence by providing human-friendly explanations. When deploying a model on AutoML Tables or Vertex AI, you can reflect patterns found in your training data to get a prediction and a score in real-time about how different factors affected the final output.
3. Streamline model governance with performance monitoring and training. You can easily monitor predictions and provide ground truth labels for prediction inputs with the continuous evaluation feature. Vertex Data Labeling compares predictions with ground truth labels to incorporate feedback and optimize model performance.

AI continues to be an exciting frontier that will continue to shape and inspire the future of enterprises across all industries. But in order for AI to reach its full potential and gain wider adoption, it will require that all stakeholders, not just data scientists, understand how ML models work.