Sometimes, it is easy to forget that the technology we take for granted isn’t magical. The internet relies on a web of under-sea cables that make it possible for us to instantly see content from across the world and mobile phones require cell towers and data centers to stay “mobile”.
The same goes for generative AI models. While opening up Chat-GPT and asking it for a summary of a paper you can’t be bothered to read seems like a simple affair - a lot needs to happen before text appears on your screen. You may have heard that one Chat-GPT prompt equals a bottle of water.$^1$ You may have also heard that that statistic is not true.$^2$ Its hard to simplify something as complex as the generative AI supply chain - but that doesn’t mean it’s not worth trying.
This text will, hopefully, make it easier to understand how the infrastructure of generative AI works and why understanding it matters. First, we will look at how generative AI models like Chat-GPT work. Then, we will break down its underlying infrastructure - looking at computing power, information and data centers. Finally, we can explore why understanding the physical realities of modern technology matters.
When you ask Chat-GPT to summarize an article, the model doesn’t follow the same steps you would have to. It doesn’t have to actually read the article or draft a list of main points before it writes a summary.
Instead, it combines the data it was trained on and your prompt to calculate what is the statistically most likely string of words that could answer your question.$^3$
The data it was trained on doesn’t serve as a library of references that it can consult. Chat-GPT doesn’t look up an answer for you in its database - this is what sets it apart from search engines like Google. Instead, it uses data it has access to as a basis for calculating probability - in other words, for guessing.$^4$ And while you can make (un)educated guesses easily, ChatGPT needs a hell of a lot to happen before it gives you an answer that might be correct.
In order for a large language model (in this case, Chat GPT) to work, it first needs to be trained. Training the model starts with an algorithm - an equation with undefined coefficients. The model is then given data to determine what coefficient values fit the equation best. Once it has been exposed to enough data, the model gets better at making consistent guesses - it figures out what is statistically the most likely answer.$^5$ If the purpose of the model is to guess which picture shows a dog, it needs to be exposed to enough pictures of different breeds to “learn” what makes up a dog.
First calculations are bound to have inaccuracies and inconsistencies which need to be corrected. Following the previous example, the model might initially conclude that a dog has four legs - it then needs either direct input from the programmer, or more data, to learn that an amputee chihuahua is indeed still a dog. After enough corrections, the model “learns” to recognise patterns and understand context enough to generate a plausible answer. $^6$
In reality, the model doesn’t “learn” - it calculates. And it calculates A LOT.
Calculations need computing power. Power is provided by chips. In the same way that I can’t run a proper video game on my 100 euro laptop without the risk of it catching on fire, AI cannot be trained on just any computer chips.
It is mostly trained on graphics processing units (GPUs) which makes them a good starting point for looking into the materials and infrastructure needed for computing.$^7$
GPU’s used for AI are made of an array of materials, but a couple of them are worth singling out. The base for GPUs are silica gel and quartz, which are turned into silicon wafers. Alternatively, germanium might be used. The silicon then needs to be “doped” with an array of minerals to supercharge its conductivity.$^8$ Copper clad laminates, cobalt, tin, tantalum and tungsten are also necessary for the construction of chips.$^9$
Silicon is the second most common element found in the Earth’s crust, meaning supply isn’t really an issue.$^{10}$ However, it’s only useful for the production of electronic devices when it is in its purest form, which requires significant energy and resource-intensive mining and refining. These processes lead to deforestation, soil erosion and disruption of water supply, as well as degradation in air quality.$^{11}$