How distillation technology is redrawing the AI landscape

 How distillation technology is redrawing the AI landscape

 

In a world that is witnessing a frantic acceleration in the race to develop artificial intelligence, a key question arises that occupies the minds of many, namely: how much does it cost to start a company specializing in artificial intelligence The answer is not a static number, but a dynamic reality that changes day by day, due to several key factors, most notably: low computing costs, the emergence of innovative technologies such as (distillation) Distillation, which has become the talk of the hour, which facilitated the construction of large language models with high quality and low costs.

However, distillation technology is not a new innovation, but it has gained exceptional importance in the artificial intelligence race, it has become a revolution in the development of language models, as this technology allows the creation of large high-quality language models at a lower cost, opening the door for startups and independent developers to compete in a market that was the monopoly of giant companies. But, does this development represent a golden opportunity for everyone, or does it carry with it new risks and challenges

 So in this article, we will dive into the details of the (distillation) technology, to find out how it is changing the rules of the game in the world of artificial intelligence, and what the implications of this development are, whether on startups, giant companies, or the future of artificial intelligence as a whole

Distillation technology.. An old concept with modern applications:

The essence of the distillation technology is to extract knowledge from a huge AI model to develop a smaller one with similar capabilities but at a lower cost.

In this process, a large – scale model, characterized by advanced capabilities in generating responses and identifying thought paths – similar to a teacher – is used to train a smaller, less complex model – similar to a student-to simulate the behavior of the larger model, as the smaller model learns how to generate similar responses and follow the same thought paths as the larger model. This approach reduces the need for huge computing resources, which makes the development of artificial intelligence more efficient and economical.

Examples of the power of distillation technology:

The Chinese artificial intelligence startup DeepSeek has caused a big stir in this field by developing models that compete with OpenAI models at a training cost of only about 5 million dollars, and this achievement has caused a panic in the stock market, and caused Nvidia to lose up to 600 billion dollars of its market value temporarily, due to fears of a decrease in demand for electronic chips, a decline that has not yet been achieved.

On the other hand, a team from the University of California – Berkeley succeeded in training two new models at a cost of less than 1000 dollars, according to research published last January, and in early February, researchers from Stanford and Washington universities and the Allen Institute for artificial intelligence succeeded in training a practical thinking model at a much lower cost, as stated in a research paper, and the distillation technology was the common factor in achieving these achievements.

The role of distillation technology in reducing costs:

Distillation technology is a vital tool for developers, as they use it together with fine-tuning technology, to improve the performance of models during the training phase, but at a much lower cost compared to traditional methods.

Developers use these two technologies in an integrated manner to provide forms with specific expertise and skills, allowing the creation of specialized and effective forms at an affordable cost. for example, a generic form such as the llama from meta can be used and distilled using another form to become an expert in US tax law.

The DeepSeek-R1 heuristic model can also be used to distill the Llama model to enhance its heuristic capabilities, allowing the model to generate more detailed answers and explain the process of accessing them step by step.

In an analysis published in January, SemiAnalysis highlighted the revolutionary potential of distillation technology in the field of artificial intelligence, as analysts confirmed that the most interesting part of the R1 model paper is the ability to convert small non-inferential models into inferential models by fine-tuning them using the outputs of an inferential model, meaning that distillation technology not only reduces the size of models, but can also improve its inferential capabilities.

In addition to the low cost, which is a big advantage in the field of artificial intelligence, DeepSeek has released distilled versions of other open source models, using the R1 heuristic model as a landmark model.

The full-size DeepSeek models, as well as the largest versions of Llama, are too bulky to be operated only in specialized devices, and here comes the role of distillation in solving this problem, as it allows the creation of smaller and more operable models in regular devices, including: mobile devices and edge devices.

Hence, the most important achievement of DeepSeek was the discovery that distilled models did not lose their quality as they were reduced in size, but, on the contrary, their performance improved. This represents a quantum leap in the field of distillation technology, proving that smaller and more efficient AI models can be created without sacrificing quality.

History of distillation technology and its development:

The roots of (distillation) technology go back to a research paper published in 2015, signed by prominent artificial intelligence pioneers at Google, Jeff Dean and Jeffrey Hinton, as well as Oriol Vinyals, vice president of (DeepMind) research at Google. However, it was recently revealed by Vinales that the paper was rejected from the then prestigious neurips conference, due to not appreciating its potential impact in the field.

But today, a decade later, distillation technology is at the center of debates about the future of artificial intelligence.

The main reason for the growing power of distillation technology at the moment lies in the abundance and quality of open source models that can be used as parametric models. Kate Saul, technical director of IBM's LLM Granite, confirmed on the mix of Experts podcast last January that DeepSeek's launch of a very high-capacity model, the most capable open source model to date, under the MIT license, has undermined the competitive barriers that large companies have maintained by keeping their models closed.

This means that the availability of powerful open source models allows developers to use them to distill smaller and more efficient models, reducing the gap between big companies, startups and independent developers.

Advantages and disadvantages of distillation technology:

Platforms such as Huggingface, a repository of large language models, are replete with distilled versions of popular open source models such as: Llama from Meta, and Qwen from Alibaba of China.

Statistics confirm this trend, out of 1.5 million models available on the (Huggingface) platform, up to 30 thousand models carry the word (distillation) in the name, which indicates their distilled nature. However, it should be noted that these distilled models have not yet managed to get into the list of the most popular models of the platform.

Kit soul likens the distillation process to shopping in cheap goods stores, as it offers the best value for the cost, but with limited selection and some disadvantages, the distillation technique may lead to the specialization of the model in one task at the expense of its performance in other tasks.

Apple researchers have tried to develop (a law for measuring distillation) distillation scaling law, which allows predicting the performance of a distilled artificial intelligence model, based on factors such as: the size of the model under construction, the size of the model (teacher), and the computing power used, and concluded that distillation may surpass traditional supervised learning in some cases, but provided that a high-quality model is used and larger than the model being trained, taking into account the existence of a maximum size of the large model to be used as a teacher for the smaller model.

However, distillation technology remains a valuable tool for reducing the gap between ideas and prototypes, reducing the barrier to entry into the field of artificial intelligence development. AI experts argue that this technology does not eliminate the need for large and expensive prototypes, but raises questions about the economic feasibility of companies investing in the construction of such huge models.

Strategies of large companies in the face of distillation technology:

Jensen Huang, CEO of Nvidia, said in a statement to CNBC after the announcement of the company's latest quarterly earnings, that the DeepSeek R1 model is used by almost every AI developer in the world today to distill new models, and this widespread deployment of distillation technology represents a huge opportunity to develop more efficient AI models at a lower cost.

However, this technology is facing increasing opposition due to the threat it poses to bulky, expensive and special models, such as those produced by leading companies such as: OpenAI and Anthropic.

Jasper Chang, one of the founders of the Hyperbolic platform, believes that foundation models will gradually turn into commodities, noting that there is a limit to the capabilities that can be achieved by previously trained models, and we are approaching this limit.

Chang suggests that the ideal solution for large companies in the field of large language models is to focus on building products that are widely popular instead of focusing only on developing models, and this opinion is in line with meta's decision to make its Llama models partially open source.

In addition, some experts have pointed out that there are more aggressive tactics that companies can take to meet the challenges posed by distillation technology, and these tactics include taking legal or technical measures to restrict the use of distillation technology.

For example: companies with heuristic models can delete or reduce the heuristic steps or reasoning traces that appear to the user, to prevent their use in the distillation process, it should be noted that OpenAI hides the full heuristic path in the O1 model, but later launched a smaller version called o3-mini that displays this information.

Over the coming months, we will see attempts by leading AI companies to disrupt distillation operations,”David Sachs, an adviser to President Trump on cryptocurrencies and artificial intelligence policies, told Fox News last January.

However, it can be difficult to contain the spread of distillation in a chaotic open-source AI environment.

This is confirmed by Kate Saul, saying:anyone can access the (Huggingface) platform, and find huge amounts of data generated using GPT models, and this data was coordinated and organized especially for training purposes, and often obtained without obtaining the necessary licenses. She stressed that this practice, which is an open secret, has been going on for a long time.

Constant revolution:

Distillation is not just a technical tool, but a revolution that redefines the cost and accessibility of artificial intelligence, and whether it continues to threaten large companies or pushes innovation to new heights, it is reshaping the future of the industry, as this technology will allow the creation of more efficient and personalized artificial intelligence models, and will open the way for more companies and developers to participate in this industry.