Given the impressive performance and widespread adoption of large language models (LLMs), there is a pressing need to explain how these systems work and what information they use for their predictions. This would not only allow us to better predict and control their behavior, thereby increasing their trustworthiness, but also help gain insight into the internal processes underlying linguistic behavior in LLMs. 

In this paper, I consider an important question in the philosophy of AI. Does the fact that we cannot know how an AI reaches its conclusions entail that we cannot reasonably trust it? I argue that it does not.

The relationship between trust and transparency, whether in AI or elsewhere, appears quite puzzling. It seems unreasonable to trust that which is opaque, but too much transparency renders trust superfluous, for trust requires some degree of uncertainty and vulnerability. 

The current call for explainable AI (XAI) is most often framed as an answer to the so-called black box problem of machine learning. Following this conceptualisation of the problem, the recent effectiveness of many machine learning (ML) based systems comes at the cost of intelligibility: the more accurate AI performs the less understandable it becomes to humans. Accordingly, XAI is seen as the endeavour to whiten the black box, so that society can profit from the latest AI success without endangering being alienated.