When Less Is More: A Technical Overview of LLMs and the Strength of Smaller Models

Hi guys, I'm Lugari Karasen. I'm a developer advocate at Red Hat AI, and today we're going to talk about a technical overview of large language models and the strengths of smaller models. So what we're going to touch on today and the next 10 minutes, we're going to look at data collection for the pre-training of these models, the tokenization of that data, what the internals of a neural network look like. And then inferencing these models once it's time to get them into production. Through that, we will then touch on the strengths of smaller models.

So first up, data collection for pre-training. If you've ever posted on the internet, you've probably helped contribute to training these models. Here we can see Common Crawl's statistical graph of what a web hierarchy looks like. Common Crawl is a web scraper that hosts a bunch of the internet's pages and the data that are on them. Every lab has some data set similar to what Common Crawl has captured to train these models on.

So once you pull from the website, people will typically filter the URLs that are going to predictably lead to bad results. They're going to pull the text from these websites, ignore things that are not text, filter for the type of language that you want. Maybe you want it to be 65% English, some other percent of a different language. At the end of the day, it's all going to be converted to mathematical representations. From there, you're going to remove duplicates and hopefully any personal identifiable information, such as social security numbers, passwords, etc. From there, we're going to tokenize that data.

So this is us converting human language into math. If we see an excerpt from Alice in Wonderland, we can convert it to binaries with UTF-8 encoding, grouping them into 8 bits. This process, combined with byte-pair encoding, results in a tokenized version of the human language, associating words with corresponding tokens.

During training, the model adjusts parameters to reflect patterns in the data it was trained on. This involves tweaking weights through an iterative process to reduce loss and make the model more representative of the dataset. The complexity lies in the billions of parameters models typically have, with examples like Chat2BT5 trained on trillions of parameters, creating an intricate vector embedding space.

Inference involves putting models into production to process user inputs quickly and generate new data. An inference engine, like VLLM, is vital for this process, requiring inputs such as a config, tokenizer, and safe tensors. VLLM, supported by Red Hat and available on HuggingFace, stands out for its user-friendly compatibility with various models.

So what is going on during training? For a mental model of this, I must preface and say every algorithm is going to be different and this is highly complex. But loosely what training is is finding the right parameters to make the model have outputs that are representative of the patterns in the data that you trained it on. So we are going to take the tokens that we created and the weights or parameters that correspond with them and tweak those weights and parameters through an iterative process. They begin random and then they slowly, via reducing loss, become more representative of the dataset.

A vector embedding space helps me visualize what this looks like. Again, this is a loose representation. This is so, so complex. In reality, a model can have billions of parameters, must almost have billions of parameters. It's rare you see any model with below one billion parameters. And Chat2BT5, for example, was trained on, I believe, three trillion parameters. But if we look at this and we have every dot represent a token, we can see that in this infinite dimensional space, these tokens or words are going to be placed near words that correspond with them. And queen might be equidistant from woman as king is equidistant from man.

So now inference. For the most part, we are not going to be training these models, but we will be responsible as developers for getting them into production and making sure that our users' inputs are processed quickly and we can generate new data from these models one token at a time. So we need an inference engine in order to put these into production. It's required. The required inputs for VLLM, which is an open source inference engine that Red Hat helps contribute to are a config, a tokenizer, and safe tensors. All of these can be found on HuggingFace, which is a repository for models. Some inference engines are compatible with the files on HuggingFace. Some are not. VLLM is, which definitely increases its user friendliness. So there are tradeoffs when you're deploying these LLMs. If you are optimizing for low latency and high accuracy, you're probably also going to get high cost. That high accuracy often comes from a very large model, which often means an expensive model.

So for example, if we boost accuracy, we might be adding parameters to our model, which is in turn leading to a decline in inference performance. So this is the first somewhat obvious argument for smaller models, which is that smaller models are faster to inference and they cost less to inference. Here is a chart that is showing how much a company would pay to run each of these different size models, 8 billion, 7 billion, 70 billion, 450 billion parameter models, as on-demand instances in the cloud, a.k.a. pinging an endpoint to run these models. Now you can see that the costs here are vastly different for a model whose accuracy might not be all that different.

Just to level set quickly, what is a small model? A small model is something that is easy to run in a local development environment on-prem, traditionally 8 billion parameters or less, whereas large models range from 70 billion to 1 trillion parameters requiring beefier hardware. To make models smaller but maintain accuracy, quantization is a viable option. It involves reducing the precision of numbers within the model, either in weights or activations used in generating new data during inference. Another method is distilling data from larger models to train smaller ones, teaching them to mimic the larger models. Additionally, setting some weights to zero, called sparsity, contributes to model size reduction.

A comparison between smaller and quantized models reveals differences in total parameters and precision. For instance, a native 16 floating point operations model compressed to a smaller size maintains 99% accuracy and runs twice as fast. Utilizing tools like LLM Compressor, a subset of the VLLM open-source project, allows for compressing models. Running local models offers benefits such as data privacy and control, especially for sensitive data. Smaller models enable experimentation without infrastructure constraints, serving as a sandbox for developing agents or rag pipelines efficiently. On-premises deployment of smaller models enhances privacy, control, and flexibility in tinkering with minimal infrastructure dependencies.

A smaller model is going to have fewer total parameters, but a quantized model will have the same amount of parameters with less precision. This interesting chart series illustrates a model at native 16 floating point operations compressed to maintain 99% accuracy and run twice as fast. Utilizing tools like LLM Compressor from the VLLM open-source project can aid in model compression. Running local models provides control, data privacy, and experimentation flexibility, especially beneficial for sensitive data.

Running local models offers the advantages of data privacy, control, and the ability to experiment without infrastructure constraints. The use of small language models as sandboxes for developing agents or rag pipelines can mitigate upfront infrastructure risks. Argument three for smaller models emphasizes the benefits of running models on-premises for enhanced privacy and control. Despite the contrast with tool-dependent workflows, smaller purpose-driven models can still play a significant role in agentic workflows for specific use cases.

To explore further, Hugging Face offers the opportunity to tinker with models and access pre-quantized models through the Red Hat AI Hugging Face repo. Benchmarking model performance with tools like Guide LLM can provide insights into system-level performance and compatibility with the VLLM inference engine. Evaluating model performance across different sizes such as smaller, quantized, and larger models can be done using LLM EvalHarness. For additional content on AI for sysadmins and developers, a newsletter is available, and for direct conversations, the speaker is reachable on LinkedIn.