The training of AI models is being shaped not only by larger data sets, but also by smarter algorithms. Traditional training methods are no longer sufficient for language models to succeed in complex tasks such as mathematical reasoning and code generation. At this point, GRPO (Group Relative Policy Optimization), which stands out as a new approach in the education of large language models (LLM), is revolutionizing memory efficiency and cost effectiveness.
GRPO is particularly notable as a reinforcement learning technique used in open-source models such as DeepSeek-Math and DeepSeek-R1. So why is this method so important and how does it work?
GRPO (Group Relative Policy Optimization) is an optimization algorithm based on Reinforcement Learning (RL), used in the training of major language models. This method is specifically designed to improve the performance of models in areas where verifiable reward functions, such as mathematical problems and code writing, can be used.
Unlike traditional reinforcement learning methods, GRPO does not require a separate value function model. Instead, it calculates the advantage based on the average reward of multiple responses generated by the model for each question. This approach both significantly reduces memory usage and makes the training process more efficient.
The basic principle of operation of GRPO is quite simple, but effective. The model generates multiple responses for a question, these responses are scored by a reward model and determined which answers are better based on the group average. Responses that score above average are encouraged, while those below average are penalized. This process allows the model to produce better responses over time.
Reinforcement learning is a machine learning approach in which an agent learns through trial-and-error by interacting with the environment. In the case of large language models, the model receives a question (observation), produces a response (action), and receives a reward or punishment based on the quality of that response. The goal is to maximize the total reward over time.
In Supervised Fine-Tuning (SFT) methods, models are trained on pre-labeled data sets. But this approach has serious limitations. Collecting tagged data is both costly and time-consuming. There is also a risk of overfitting the model to the training data.
GRPO, on the other hand, is designed to overcome these limitations. It only works with a verification mechanism, without the need for labeled data. For example, tools such as a compiler, unit tests, or linter can be used to check whether the answer produced in a mathematical problem is correct. Thanks to this, the model can develop itself without human intervention.
The operating logic of the GRPO is based on a group-based evaluation system. The process proceeds through these steps:
First, the model generates multiple candidate responses for a given question. In the DeepSeek-Math model, this number is usually 64. Each response is evaluated by programmable reward functions. These functions can measure the accuracy of the response, format compatibility, or code quality.
Then the average reward of all responses produced is calculated. This average serves as a baseline. The reward of each response is compared with this average and thus the advantage value is determined. Responses that perform above average receive a positive advantage, while those that are below average receive a negative advantage.
Model parameters are updated based on these advantage values. While the probability of generating responses with high advantage is increased, the probability of those with low advantage is reduced. During this update, regulation mechanisms such as KL divergence (Kullback-Leibler divergence) are used so that the model does not stray too far from the old policy.
This process can be expressed mathematically as follows: The rewards of the answers generated for each question are summed up and the group average is calculated. The advantage function is found by subtracting this group average from the reward of each response. Policy update is performed using importance ratio and advantage values.
To understand the uniqueness of GRPO, it is necessary to compare it with other common methods of reinforcement learning.
Proximal Policy Optimization (PPO) is a widely used method in the industry. PPO is also used in OpenAI's human feedback reinforcement learning (RLHF) process. However, PPO needs a separate value function model. This model creates a significant memory and computational burden, as it is similar in size to the policy model. GRPO, on the other hand, eliminates this value function, reducing memory usage by half.
Direct Policy Optimization (DPO) is an alternative developed to reduce the complexity of PPO. It does not need a separate reward model and works through preference data pairs. However, DPO still requires a significant amount of human preference data. GRPO, on the other hand, works with automated verification mechanisms, which completely eliminates the cost of data collection.
The major advantage of GRPO is that it is superior to other methods in terms of both memory efficiency and data requirements. Furthermore, reducing variance using group averaging makes the training process more stable.
GRPO performs highly, especially in tasks that can be verified. Mathematical reasoning is at the top of these areas. The DeepSeek-MATH model achieved an accuracy rate of 88.2% in the GSM8K dataset and 51.7% in the MATH dataset when trained using GRPO. These results surpass much larger models such as Minerva, which has 540 billion parameters.
Code generation is another area where GRPO shows its effectiveness. It can be automatically checked whether the generated code is compiled, gives runtime errors, or passes unit tests. Therefore, GRPO is extremely useful in fine-tuning code generation models.
Tasks that require multistep logical inference also benefit from GRPO. The model generates intermediate steps to solve a problem, and the accuracy of each step can be evaluated. Thanks to this, the model learns not only the correctness of the final answer, but also the quality of the problem-solving process.
The most obvious advantage of GRPO is that it does not require labeled data. In traditional methods, human raters are required to give preference scores for each response of the model. This is both costly and difficult to scale. GRPO, on the other hand, eliminates this cost because it works with automated verification mechanisms.
Memory efficiency is another important advantage. Methods such as PPO train a separate value function model as well as the policy model. GRPO removes this extra model, reducing memory usage by about 50%. This is critical for researchers and companies with limited hardware resources.
Reducing the risk of overfitting is also among the benefits offered by GRPO. In supervised learning, models can over-adapt to training data. GRPO, on the other hand, encourages the model to explore new strategies through its active learning approach. Thanks to this, the model acquires more general capabilities.
In terms of cost-effectiveness, GRPO is also remarkable. High performance can be achieved with far fewer samples than traditional fine-tuning methods. For example, the GRPO training of the DeepSeek-Math model was conducted on only 144 thousand questions. This is far below the amount of data required for supervised learning.
The design of reward functions plays a critical role in the success of GRPO. Accuracy rewards assess whether the model's final response is correct. The reward functions used in DeepSeek-Math check both the mathematical accuracy and formal appropriateness of the answer.
Format rewards, on the other hand, check whether the response fits a particular structure. For example, in the DeepSeek-Math model, answers are expected to be presented within specific tags. This allows a clearer understanding of the model's thought process.
The temperature parameter is an important setting in the GRPO that controls the variety. Low temperature values cause the model to always choose the most likely response. This ensures consistency but limits diversity. High temperature values, on the other hand, bring more randomness and allow the discovery of different solutions. But in this case, the quality of individual estimates may decrease. Choosing the right temperature value is sometimes an art.
Reward hacking is a major problem that can be encountered in GRPO. Models can manipulate reward functions without achieving the real goal. For example, the model that is expected to produce tests for a code may write an empty test function that does not perform any checks. Restrictions should be added to reward functions to prevent such situations.
The DeepSeek-Math model has attracted attention with the results obtained using GRPO. The model increased from 82.9% to 88.2% in the GSM8K dataset and from 46.8% to 51.7% in the MATH dataset. These improvements demonstrate the effectiveness of GRPO in enhancing mathematical reasoning abilities.
The DeepSeek-R1 model also performed at a level that would rival OpenAI's o1 model using GRPO. This success proves that GRPO is not only an academic innovation, but also effective in real-world applications.
GRPO is ushering in a new era in the education of major language models. Without requiring labeled data, this method that works memory efficiently and offers cost effectiveness, is revolutionizing areas such as mathematical reasoning and code generation in particular. Achievements in models such as DeepSeek-Math and DeepSeek-R1 clearly reveal the potential of GRPO. By adopting this technology, businesses can train AI models more efficiently and effectively so that they can gain a competitive advantage.
Bibliography:
Especially in machine learning and natural language processing (NLP) projects, data is often represented as numerical vectors. At this point, traditional databases may be insufficient to manage vector-based data. This is where Vector Database (Vector DB) comes into play.
It is a three-step integration process used by companies to combine and synthesize raw data from many data sources into a data warehouse, data lake, data warehouse, relational database, or other application.
Domain Driven Design allows us to rethink the software development process by centralizing the business area. It focuses on a deep understanding of the business problem we are trying to solve before the technical details of the software and places this understanding at the heart of software design.
We work with leading companies in the field of Turkey by developing more than 200 successful projects with more than 120 leading companies in the sector.
Take your place among our successful business partners.
Fill out the form so that our solution consultants can reach you as quickly as possible.
By combining our 30+ years of experience and experience with the centers of excellence in our ecosystem, we prepare your organization to lead the digital transformation!
