Transcript (en)

Scaling is a primary driver for a lot of progress, maybe all of the major AI developments that we have witnessed in the last few years. And here we're going to discuss how we scale our model in different dimensions. So on the left hand side, the first figure you see here is kind of the standard scaling law. So on the x-axis you have the log of the number of training tokens, and on the y-axis you have the log loss. And as you scale the number of training tokens, you get a lower loss. But here the point is we're not going to just scale the number of training tokens, but we also want to improve the token efficiency, meaning that we want to move this curve to the left-hand side so that we can achieve a lower loss, a much lower loss, using the same number of training tokens. And this can be achieved by having better architectures and optimizers, as we'll discuss in our later slides. And the second scaling dimensions that we're very interested in is to scale the context length. As you can see in the second figure, if we increase the context length, then we can have a much higher accuracy in terms of predicting the token loss at a given position And this means that we can increase the capability of the model to achieve more complex tasks by increasing the context stance. So this is the second scaling dimensions that we're going to talk about. And the third scaling dimension is the number of agents. So we introduced this new learning paradigm of agents realms, where we don't just rely on a single agent. But we also orchestrate a swarm of agents that can accomplish the subtasks in parallel so that we can increase the task complexity. And we can translate all of this into the language of agents. So if you look at token efficiency, it's mostly about having a stronger prior so that you can have more efficiency when you do agent RL to search for a better solution. And when you think about long contacts, It's mostly about increasing the context length so that you can have a longer-running agent. It can probably run for days or even weeks or months to accomplish more tasks, more complex tasks. And for agent swarms, it's another dimension that added it. And at the end of the day, we're going to have a swarm of agents that each of them have a super long context. And each of them have a very strong prior for us to search in this entire agent RL system.