Red Hat on Open, Small Language Models for Responsible, Practical AI

As geopolitical events shape the landscape, they also influence technological advancements, notably in artificial intelligence (AI). The current AI market is evolving, impacting methodologies, development practices, and applications within enterprises. Expectations surrounding AI outcomes are being tempered by real-world challenges, leading to a mix of skepticism and enthusiasm for the technology, particularly as it continues to mature.

The closed-loop designs of prominent large language models (LLMs) are facing scrutiny, prompting a shift towards open-source alternatives such as Llama, DeepSeek, and Baidu’s Ernie X1. Open-source development introduces transparency and encourages contributions, aligning with the desire for “responsible AI.” This term reflects considerations of environmental impacts from large models, ethical usage, the datasets employed in training, and issues surrounding data sovereignty and language politics.

As a pioneer in economically sustainable open source development, Red Hat aims to apply this collaborative ethos to AI. In a recent conversation with Julio Guijarro, the Chief Technology Officer for EMEA at Red Hat, he discussed the company’s mission to harness the power of generative AI responsibly and transparently, thereby creating tangible enterprise value. Guijarro emphasized the need for ongoing education regarding AI’s complexities, noting that the technology often remains a ‘black box’ for many due to its intricate scientific foundations and development in largely inaccessible environments.

Moreover, challenges such as inadequate support for European and Middle-Eastern languages, data sovereignty, and trust issues loom large. Guijarro stated, “Data is an organisation’s most valuable asset, and businesses must recognize the risks of sharing sensitive information on public platforms with varying privacy practices.”

Red Hat’s Approach to AI

In response to increasing global demand for AI, Red Hat is committed to solutions that prioritize user benefits while addressing concerns that arise with conventional AI services. One innovative approach involves deploying small language models (SLMs) on local systems or hybrid clouds, utilizing non-specialist hardware and accessing internal business data. These SLMs are efficient alternatives to LLMs, delivering high performance for specific applications while consuming significantly fewer computational resources.

Businesses can leverage smaller cloud providers to manage computational tasks, but having the option to maintain critical data in-house, close to the application, is essential. “Data is continually evolving within an organization, and large language models may quickly become outdated since data generation occurs at the local level,” Guijarro explained.

Cost implications also deserve attention. Utilizing LLMs for customer service can incur substantial hidden expenses. Previously, data queries had predictable costs, but with LLMs operating on an iterative basis, increased use leads to rising costs. What used to be a single transaction can now escalate into numerous interactions, each incurring expenses. By running models in-house, organizations can manage costs based on their infrastructure rather than pay-per-query expenses.

Organizations need not fear a costly procurement process for GPUs. Red Hat is actively working on optimizing models openly, ensuring that businesses can harness the benefits of AI without financially overextending themselves.

Advancements in artificial intelligence are enabling it to operate effectively on more standard hardware. This capability arises from a shift in focus by businesses toward specialist models. These models do not necessitate the expansive, general-purpose data sets typically required, which can be costly to process for every inquiry. As Julio mentioned, “Many efforts currently involve examining large models and eliminating unnecessary components tailored to specific use cases. If we aspire to make AI more widespread, we must shift toward smaller language models. Additionally, we are enhancing vLLM, our inference engine project, ensuring users can efficiently interact with various models, whether locally, at the edge, or in the cloud.”

Staying Compact
Utilizing local data relevant to users allows for customized outputs tailored to specific needs. Julio highlighted initiatives in the Arab and Portuguese-speaking regions that would not be feasible with the mainstream, English-centered large language models (LLMs).

However, early adopters of LLMs have encountered several practical challenges. One significant concern is latency, particularly in environments where quick responses are essential, such as in customer service. Having streamlined resources that provide relevant, tailored results a short distance away is advantageous.

Another critical issue is trust, a fundamental aspect of responsible AI. Red Hat champions open platforms, tools, and models to foster transparency, understanding, and broad participation. Julio emphasized, “This is crucial for everyone. We are developing technologies to democratize AI, which extends beyond merely releasing a model; it involves equipping users with the necessary tools to replicate, refine, and deploy these models.”

Recently, Red Hat acquired Neural Magic to facilitate easier scaling of AI for enterprises, enhance inference performance, and expand options for deploying AI workloads through the vLLM initiative. In collaboration with IBM Research, Red Hat also launched InstructLab, aimed at aspiring AI developers who possess adequate business knowledge but may not be data scientists.

While discussions about the potential downturn of the AI market persist, they often converge on the financial realities that major LLM providers will soon encounter. Red Hat is optimistic about a future where AI takes a use case-specific and inherently open-source form, evolving into a technology that makes business sense and is widely accessible. As Matt Hicks, CEO of Red Hat, puts it, “The future of AI is open.”

Supporting Assets:
Tech Journey: Adopt and scale AI

Artificial Intelligence continues to evolve, impacting various sectors significantly. Recent developments highlight innovations in addressing challenges associated with AI systems.

On June 3, 2025, a standout initiative emerged from a MIT spinout focused on teaching artificial intelligence to recognize its limitations. This advancement aims to tackle common issues, such as “hallucinations,” where AI provides incorrect or nonsensical information.

In another significant collaboration, IBM and Roche announced their efforts to utilize AI for diabetes management. By forecasting blood sugar levels through data advancements, this partnership aims to enhance patient care in the healthcare sector.

However, not all developments have been seen positively. On May 30, 2025, a critique surfaced regarding DeepSeek’s latest AI model. It was labeled a setback for free speech, raising concerns about the implications of AI governance in communication.

These evolving narratives reflect the dual aspects of AI technology—its potential benefits in improving life quality and the ethical considerations surrounding its use.