Microsoft Introduces Phi-3 Family of Compact Language Models

Microsoft has launched the Phi-3 family of open small language models (SLMs), claiming them to be the most capable and cost-efficient in their class. A novel training method crafted by Microsoft researchers has enabled these Phi-3 models to surpass larger models in performance across language, coding, and mathematical evaluations.

According to Sonali Yadav, Principal Product Manager for Generative AI at Microsoft, the trend is shifting from a focus solely on large models to a diverse portfolio of models, allowing clients to select the most suitable option for their specific needs. The inaugural Phi-3 model, Phi-3-mini, featuring 3.8 billion parameters, is now available in the Azure AI Model Catalog, Hugging Face, Ollama, and as an NVIDIA NIM microservice. Notably, despite its smaller scale, Phi-3-mini outperforms competitors that are twice its size. Additional models in the Phi-3 lineup, including Phi-3-small with 7 billion parameters and Phi-3-medium with 14 billion parameters, are set to be released shortly.

“Some customers will only require smaller models, while others may need more complex ones or a combination of both,” stated Luis Vargas, Microsoft’s VP of AI. The primary advantage of SLMs lies in their compact size, enabling on-device deployment for low-latency AI experiences without the need for internet connectivity. This presents potential applications in various fields, including smart sensors, cameras, and agricultural equipment, while also enhancing privacy by keeping data local.

While large language models (LLMs) are exceptionally adept at complex reasoning across vast datasets—beneficial for tasks like drug discovery—SLMs present a practical alternative for straightforward query responses, summarization, and content generation. Victor Botev, CTO and Co-Founder of Iris.ai, remarked, “Instead of pursuing larger models, Microsoft is concentrating on tools powered by meticulously curated datasets and specialized training, improving both performance and reasoning capabilities without incurring the immense computational costs associated with models containing trillions of parameters.”

Innovative Training Techniques

The leap in quality for Microsoft’s SLMs was achieved through a groundbreaking data filtering and generation technique, inspired by bedtime stories. Sebastien Bubeck, Microsoft VP leading the SLM research, questioned why training models on purely raw web data should be the norm. This led to the creation of a ‘TinyStories’ dataset, crafted by utilizing a large model to generate millions of simple narratives based on combinations of words understandable by a four-year-old. Impressively, a model with merely 10 million parameters trained on TinyStories was capable of producing grammatically accurate stories.

Building on this initial success, high-quality web data was carefully selected for educational value to form the ‘CodeTextbook’ dataset. This dataset was produced through multiple rounds of prompting, generating, and filtering by both human input and large AI models. Bubeck emphasized, “Significant effort goes into curating this synthetic data. We are selective about what we retain.” This high-quality training data has drastically improved performance, making it easier for the model to comprehend and utilize the information.

Addressing AI Safety Concerns

Despite the meticulous process of data curation, Microsoft is committed to implementing additional safety measures with the release of the Phi-3 family, consistent with its comprehensive protocols for all generative AI models. A blog post from Microsoft detailed that “a multi-layered approach was employed to manage and mitigate risks” as part of the development of the Phi-3 models. This included the introduction of further training examples to solidify expected behaviors and evaluations to uncover potential vulnerabilities.