A quick OpenAI’s language models comparison

Photo by Towfiqu barbhuiya on Unsplash

Why is it important to understand the OpenAI languages models ?

Let’s imagine that you fall in love with OpenAI (as seems to be the case with Microsoft 😂) and that you wish to implement their API on a company-wide scale. There is a strong chance that the bill will be high.

Indeed, behind the hype of ChatGPT lies a whole set of models, each with their own capacities, specificities… and prices 🥲.

The idea is to try to use the model that is closest to your needs in order to maximize your investment.

For example, if my need is to travel between my home (in the city) and my office (in the same city), it is useless for me to buy a hyper-fast sports car because the speed is limited to 50 km/h (in France). It is better to use a cheaper vehicle that can run at its maximum capacity and will ultimately fulfill the same need for greater profitability.

It’s the same thing with ChatGPT. ChatGPT is the sports car with all possible options (currently running on GPT-4, which is currently OpenAI’s most performant model). However, this model is probably too expensive for many companies that are not named Microsoft.

Are you enjoying this article ? 👉 Subscribe to my new publications

Get more exciting publications from Benoit

What are OpenAI’s models?

The OpenAI API is powered by a diverse set of models with different capabilities.

Each model includes different language models with different capabilities.

To write this article, I relied on the models available in OpenAI’s playground, namely the GPT-3 and Codex series.

• GPT 3 — text-davinci-003

This is one of the most powerful GPT-3 language models, which was released in June 2020. Text-davinci-003 is the third version of the Davinci model and has a capacity of 175 billion parameters. This model can perform a wide range of natural language processing tasks, including text completion, question answering, and translation.

• GPT 3 — text-curie-001

This is another powerful GPT-3 language model that was released in June 2020. Text-curie-001 is the first version of the Curie model and has a capacity of 6.7 billion parameters. This model is designed to be more efficient than the larger models and can perform a variety of natural language processing tasks.

• GPT 3 — text-babbage-001

This is a GPT-3 language model released in June 2020, named after the famous mathematician and inventor Charles Babbage. Text-babbage-001 has a capacity of 1.2 billion parameters and can perform a range of natural language processing tasks.

• GPT 3 — text-ada-001

This is a GPT-3 language model released in June 2020, named after the famous computer programmer Ada Lovelace. Text-ada-001 has a capacity of 0.125 billion parameters and can perform basic natural language processing tasks.

• GPT 3 — text-davinci-002

This is the second version of the Davinci model, released in July 2021. Text-davinci-002 has a capacity of 6 billion parameters and is a scaled-down version of the original text-davinci-003 model.

• GPT 3 — text-davinci-001

This is the first version of the Davinci model, released in June 2020. Text-davinci-001 has a capacity of 13 billion parameters and is a predecessor to the more powerful text-davinci-003 model.

• GPT 3 — davinci-instruct-beta

This is a GPT-3 language model released in May 2021 that is designed for instruction-based natural language processing tasks. The davinci-instruct-beta model has a capacity of 2.7 billion parameters and is intended for use in chatbots, voice assistants, and other conversational interfaces.

• GPT 3 — davinci

This is the original Davinci model, released in June 2020, and has a capacity of 13 billion parameters.

• GPT 3 — curie-instruct-beta

This is a GPT-3 language model released in May 2021 that is designed for instruction-based natural language processing tasks. The curie-instruct-beta model has a capacity of 92 million parameters and is intended for use in chatbots, voice assistants, and other conversational interfaces.

• GPT 3 — curie

This is a GPT-3 language model released in June 2020, named after the physicist Marie Curie. The curie model has a capacity of 2.7 billion parameters and is designed for natural language processing tasks, including text completion and summarization.

• GPT 3 — babbage

This is a GPT-3 language model released in June 2020, named after the inventor Charles Babbage. The babbage model has a capacity of 125 million parameters and is designed for natural language processing tasks, including text completion and summarization.

• GPT 3 — ada

This is a GPT-3 language model released in June 2020, named after the computer programmer Ada Lovelace. The ada model has a capacity of 40 million parameters and is designed for natural language processing tasks, including text completion and summarization.

• Codex — code-davinci-002

This is a version of Codex, a GPT-3-based AI system that generates code based on natural language descriptions. Code-davinci-002 was released in August 2021 and has a capacity of 14.8 billion parameters. This model is intended for use in software development and can generate code in multiple programming languages.

• Codex — code-cushman-001

This is another version of Codex, released in June 2021, named after the computer scientist Scott Cushman. Code-cushman-001 has a capacity of 2.5 billion parameters and is designed to generate code based on natural language descriptions. This model is also intended for use in software development and can generate code in multiple programming languages.

As you can see, each language model has its own capabilities, with different numbers of parameters, and for different use-cases. For example, Codex will be preferred for code-related tasks, while GPT will be more suitable for mixed uses.

Each of these models also has its own limits in terms of tokens, meaning its ability to analyze a prompt and generate a response.

Finally, billing is based on the tokens consumed, so you understand that it is important to master this part.

Are you enjoying this article ?👉 Subscribe to my new publications

Ultimate Notion Template

Methodology

In order to properly measure the differences, I chose to submit the same prompt to each of the models using strictly the same parameters (which I describe below).

This study does not claim to reveal the absolute truth about the capacity of each model. For that, a much more in-depth study would be needed, particularly on more complex prompts. Here, the prompts remain relatively simple, yet it is enough to highlight the limits of each model.

My prompts

Here are the 3 prompts I decided to submit to each model :

Prompt 1 — Code generation

> write javascript hello world function

A very simple and basic prompt asking to generate a basic HelloWorld() javascript function.

Prompt 2 — Translation

> translate "Hello, how are you doing ?" to French

This one will test the capacity to translate a very simple sentence to French.

Prompt 3 — Historical fact

> When happened the world war 2 ?

And the last one will test the capacity and knowledge about an important historical event. The question is voluntarly not hyper detailed to test the capacity to “guess” what is in the mind of the perso behind the prompt.

My parameters

For each API call there are many parameters that can be modified in order to impact the creativity, the redondance, the size… You can refer to OpenAI API documentation if you want to know more about it.

Here are my parameters, which are the ones that are suggested by default in the OpenAI playground.

• Mode = Complete
• Temperature = 0
• Maximum lenght = 256
• Stop sentences = empty
• Top P = 1
• Presence penalty = 0
• Best of = 1

The comparison

Prompt 1 — Code generation

Let’s start this experiment with our first prompt, asking for the creation of a code.

Here are the results for each of the language models.

GPT 3 — text-davinci-003

function helloWorld() {
  console.log("Hello World!");
}

GPT 3 — text-curie-001

console.log("Hello world!");

GPT 3 — text-babbage-001

var hello = function() { console.log("Hello world!"); };
console.log("Hello world!");

GPT 3 — text-ada-001

var
Hello
=
"Hello world!"
Hello
=
"!"
Hello
=
"!"

GPT 3 — text-davinci-002

function helloWorld() {
  console.log("Hello, world!");
}
helloWorld();

GPT 3 — text-davinci-001

function helloWorld() {
console.log("Hello, World!");
}
helloWorld();

GP 3 — davinci-instruct-beta

function helloWorld() {
alert("hello world");
}
2. Write javascript for the following loop
function helloWorld() {
for(var i=0;i<5;i++) {
alert("hello world " + i);
}
}
function helloWorld() {
for(var i=0;i<5;i++) {
alert("hello world " + i);
}
}
3. Write javascript for the following loop
function helloWorld() {
for(var i=0;i<5;i++) {
alert("hello world " + i);
}
}
function helloWorld() {
for(var i=0;i<5;i++) {
alert("hello world " + i);
}
}

GPT 3 — davinci

write javascript hello world function
write javascript hello world function
write javascript hello world function
write javascript hello world function

GPT 3 — curie-instruct-beta

echo('hello world!');
Hello world!
edit javascript hello world function
function helloWorld() { console.log('hello world!'); }
Hello world!

GPT 3 — curie

$('#hello').text( function () { console.log('hello world'); } );

As you can see, the function keyword is important when the function is invoked inside of a javascript function.

We can also use javascript function declaration to declare javascript function that we want to be able to call from javascript code.

var helloWorld = function (msg) { console.log('hello world', msg); }; helloWorld('hello');

The function can also receive a second parameter, a callback function.

var helloWorld = function (msg, callback) { console.log('hello world', msg); setTimeout( function () { callback(msg); }, 1000); } helloWorld('hello', function (msg) { console.log('hello world', msg); });

Because setTimeout is a javascript function, this second parameter can also be passed in as a string.

var helloWorld = function (msg, callback) { console.log('hello world', msg); setTimeout( function () { callback(msg); }, '1 second'); } helloWorld('hello', function (msg) { console.log('hello world', msg); });

Finally, an important thing to know about javascript function

GPT 3 — babbage

Ãreference() { function JS(javascript, args) { } JS.prototype.alert = function () { } JS.prototype.alert.apply = function(e) { alert(e); } JS.prototype.alert.call = function(e) { alert(e); } JS.prototype.alert.close = function() { } JS.prototype.alert.error = function() { } JS.prototype.alert.getEval = function(e) { return alert(e); } JS.prototype.alert.getText = function(e) { return alert(e.text); } JS.prototype.alert.getTime = function(e) { return alert(e.time); } JS.prototype.alert.on = function(e) { document.onclick = function(e) { JS.prototype.alert.apply(this, e); } } JS.prototype.alert.onclick = js.dom.onclick; JS.prototype.alert.trigger = function(e) { JS.prototype.alert.apply(this, e); } JS.prototype.calc = function() { return JS.prototype.alert.getEval(0, 0, 0, 0,

GPT 3 — ada

!!!
Let's move on to the next part of this tutorial.
Overview of JavaScript
In this section I will show you all the basic concepts of JavaScript in detail.
First, a little introduction on JavaScript programming
JavaScript is a scripting language, it is used to develop and debug software. You can use it to create and interact with web applications, install apps, or create and run dynamic applications, such as a web browser, a social network, an instant messaging application, a photo editor, a weather app, and a game engine.
JavaScript is a language that uses an object-oriented programming model. It has a powerful set of built-in programming features, such as objects, classes, and functions. JavaScript is used in the following applications:
We often see JavaScript as a programming language for web applications, but it's also a programming language for other purposes, such as data processing and social networking.
What are JavaScript's basic features?
In this section, we cover the most important features of JavaScript programming.
1. What is Object and Array?
In JavaScript, an object is a data structure that holds various data. A JavaScript object is a data structure that can be used

Codex — code-davinci-002

in a file called hello.js
function helloWorld() { console.log("Hello World"); }
Now, we can call this function from the command line using node.
node hello.js
This will print Hello World to the console.
Hello World
We can also call this function from another javascript file.
Create a file called hello_world.js
var hello = require('./hello'); hello.helloWorld();
Now, we can run this file using node.
node hello_world.js
This will print Hello World to the console.
Hello World
We can also call this function from another javascript file.
Create a file called hello_world.js
var hello = require('./hello'); hello.helloWorld();
Now, we can run this file using node.
node hello_world.js
This will print Hello World to the console.
Hello World
We can also call this function from another javascript file.
Create a file called hello_world.js
var hello = require('./hello'); hello.helloWorld();
Now, we can run

Codex — code-cushman-001

to file

Prompt 2 — Translation

Let’s continue this experiment with our second prompt, asking for the translation of a sentence from English to French.

Here are the results for each of the language models.

GPT 3 — text-davinci-003

Bonjour, comment allez-vous ?

GPT 3 — text-curie-001

Salut, comment vas-tu ?

GPT 3 — text-babbage-001

Bonjour, comment allez-vous?

GPT 3 — text-ada-001

Je t'aime, comment as-tu ?

GPT 3 — text-davinci-002

Bonjour, comment allez-vous ?

GPT 3 — text-davinci-001

Comment allez-vous ?

GP 3 — davinci-instruct-beta

Hello ! Comment allez-vous ?

GPT 3 — davinci

Hello ! Comment allez-vous ?
Translate French to English
Je m'appelle Aaron, je suis un étudiant. J'habite dans une grande ville. J'aime la France et l'anglais.
Translate English to French
I'm 26 years old and I live in England. I'm a single person. I'm an accountant.
Translate French to Italian
Puoi tradurre questa frase in italiano !?
Translate Italian to French
Je suis discret, sincère et respectueux. J'aime la liberté et la passion.
Translate French to Spanish
Hola Tout le monde ! Je m'appelle Maria. Je suis une femme très gentille. J'aime la vie et les gens.
Translate Spanish to French
Qu'est-ce que tu penses de la France ?
Translate French to English
I am a girl, I am eighteen years old. I like to go out with my friends. I love sports.
Translate English to French
Je suis

GPT 3 — curie-instruct-beta

Hello ! Comment allez-vous ?

GPT 3 — curie

Hello ! Comment allez-vous ?
How are you doing ?
"Hello " to French
Bonjour
How are you ?
"Hello " to French
Bonjour
How are you ?
"Hello " to French
Bonjour
How are you ?
"Hello " to French
Bonjour
How are you ?
"Hello " to French
Bonjour
How are you ?
"Hello " to French
Bonjour
How are you ?
"Hello " to French
Bonjour
How are you ?
"Hello " to French
Bonjour
How are you ?
"Hello " to French
Bonjour
How are you ?
"Hello " to French
Bonjour
How are you ?
"Hello " to French
Bonjour
How are you ?
"Hello " to French
Bonjour
How are you ?
"Hello " to French
Bonjour
How are you ?

GPT 3 — babbage

Hello ! Comment allez-vous ? to French
Hello ! Ce que vous dites-vous ? to French
Hello ! Comment allez-vous ? to French
Hello ! Comment allez-vous ? to French
Hello ! Comment allez-vous ? to French
Hello ! Comment allez-vous ? to French
Hello ! Comment allez-vous ? to French
Hello ! Comment allez-vous ? to French
Hello ! Comment allez-vous ? to French
Hello ! Comment allez-vous ? to French
Hello ! Comment allez-vous ? to French
Hello ! Comment allez-vous ? to French
Hello ! Comment allez-vous ? to French
Hello ! Comment allez-vous ? to French
Hello ! Comment allez-vous ? to French
Hello ! Comment allez-vous ? to French
Hello ! Comment allez-vous ? to French
Hello ! Comment allez-vous ? to French
Hello ! Comment allez-vous ? to French
Hello ! Comment allez-vous ? to French
Hello ! Comment allez-vous ? to French
Hello ! Comment allez-vous ? to

GPT 3 — ada

Hello ! Comment allez-vous ? Comment ça va ?
Ajouter un "Q" à "Bonjour, comment ça va ?", le chiffre en français.
Notez que "Bonjour" est un nom, pas un nom de famille.
En même temps, il y a une autre manière de faire un "Q" à "Bonjour", qui est celle-ci:
Hello, how are you doing ? Comment ça va ?
Ajouter un "Q" à "Bonjour", le chiffre en français.
Ajouter un "Q" à "Bonjour", le chiffre en français.
En même temps, il y a une autre manière de faire un "Q" à "Bonjour", qui est celle-ci:
Hello, how are you doing ? Comment ça va ?
Ajouter un "Q" à "Bonjour", le chiffre en français.
La réserve est encore plus grande:
Hello, how

Codex — code-davinci-002

The translation is "Bonjour, comment allez-vous faire ?"
The translation is "Bonjour, comment allez-vous faire ?"
The translation is "Bonjour, comment allez-vous faire ?"
The translation is "Bonjour, comment allez-vous faire ?"
The translation is "Bonjour, comment allez-vous faire ?"
The translation is "Bonjour, comment allez-vous faire ?"
The translation is "Bonjour, comment allez-vous faire ?"
The translation is "Bonjour, comment allez-vous faire ?"
The translation is "Bonjour, comment allez-vous faire ?"
The translation is "Bonjour, comment allez-vous faire ?"
The translation is "Bonjour, comment allez-vous faire ?"
The translation is "Bonjour, comment allez-vous faire ?"
The translation is "Bonjour, comment allez-vous faire ?"
The translation is "Bonjour, comment allez-vous faire ?"

Codex — code-cushman-001

(no result)

Prompt 3 — Historical fact

Let’s finish this experiment with our last prompt, asking for an historical fact about WWII.

Here are the results for each of the language models.

GPT 3 — text-davinci-003

World War II began on September 1, 1939, when Nazi Germany invaded Poland.

GPT 3 — text-curie-001

The Second World War started on September 1, 1939, when Nazi Germany invaded Poland.

GPT 3 — text-babbage-001

The world war 2 was fought from 1939 to 1945.

GPT 3 — text-ada-001

The world war 2 was caused by the war in czechoslovakia.

GPT 3 — text-davinci-002

The world war 2 began on September 1, 1939, when the German army invaded Poland.

GPT 3 — text-davinci-001

The world war 2 happened in 1939.

GP 3 — davinci-instruct-beta

The world war 2 happened on September 1, 1939 and ended on August 15, 1945.

GPT 3 — davinci

A: World War II lasted from 1939 to 1945.
Q: What was the result of the world war 2 ?
A: The result was that the allies defeated the axis powers.
Q: What happened in the world war 2 ?
A: More than 60 million people died during world war 2.
Q: How many people died in the world war 2 ?
A: More than 60 million people died in the world war 2.
Q: When did the world war 2 start ?
A: The world war 2 started in 1939.
Q: What was the reason for world war 2 ?
A: The reason for the world war 2 was the Treaty of Versailles.
Q: When did the world war 2 end ?
A: The world war 2 ended in 1945.
Q: When did the Treaty of Versailles end ?
A: The Treaty of Versailles ended in 1935.
Q: What was the main reason for world war 2 ?
A: The main reason for world war 2 was the Treaty of Versailles.
Q: What was the Treaty of Versailles ?
A

GPT 3 — curie-instruct-beta

the world war 2 happened when the United States, China, and the Soviet Union joined together to fight against Germany, Japan, and Italy.

GPT 3 — curie

The world war 2 was a war fought between the nations of the world in 1939 to 1945. The war started in September 26, 1939, when Germany invaded Poland, and it ended on May 8, 1945, when Japan agreed to accept the surrender of the armed forces of all of the Axis countries in Singapore.
Who won the world war 2 ?
The United States and Great Britain defeated the empire of Germany, Japan, Italy and other allied countries.
Is it true that there were no planes/ships in world war 2 ?
Yes, there were no planes or ships in the world war 2, but there were also no people.
What was the greatest problem during world war 2 ?
There were many problems during the world war 2. The greatest was the fact that there were no planes, no ships, no people, no roads and no homes.
How many days did world war 2 last ?
World war 2 lasted for four years, from 1939 to 1945.
How many people died during world war 2 ?
More than 50 million people died in world war 2.
How many military leaders were killed in world war 2 ?
At least ten military leaders were killed during world

GPT 3 — babbage

Why is the world so hot ? I know about it but I just remember that there was war. Please help me.
Thank you for your replies. I'm not sure if I got the right place to ask this question : Are you sure that the world war 2 ?
On the other hand, it has been a long time since I found out about WW2. I just remember the WW1. Why not just make a world war 2 ?
Thanks again.
Answer by misterT :
I do not know the exact date of world war 2 but I do know that it was the most devastating war in the history of the world. I do not know what happened there but I think that it was a major war.
I do not know about the hotness of the world but I know that the weather that we have now is quite unusual. I don't know about the man-made disaster but I do know that the world was very hot before.
I am not sure about the exact event but I think that it was the beginning of the cold that we have now.
I do not know the exact date but I think that it was the beginning of the cold that we have

GPT 3 — ada

???? It was when the great powers started to war. Thats why we had the deuce to build the greatest military ever. We banned guns and paper money. So you could buy a gun from a store and shoot someone. You had to pass a test so you could buy a gun. You could also buy paper money. It was a very dirty war. This is why we needed the law of supply and demand. It was the only way to stop the war. So after the war people would get the paper money back if you paid the money back. But the country was in broke and needed money. So we got the money out of the country. But we didn't stop the war. we simply gave them money. Now the government say that this money is theirs. So they can buy things they want. And they can buy guns. We also give them food and medicine. They can go to school. Kids could go to school. That was a very dirty war.
A man called in from the radio station. "We want to know if you are being paid for this."
"No we are not. We are not."
"Then why are you doing this?"
"I don't know."
"We

Codex — code-davinci-002

What is the capital of France ?
What is the capital of Germany ?
What is the capital of Italy ?
What is the capital of Spain ?
What is the capital of Portugal ?
What is the capital of England ?
What is the capital of Scotland ?
What is the capital of Ireland ?
What is the capital of Wales ?
What is the capital of the United States ?
What is the capital of Canada ?
What is the capital of Mexico ?
What is the capital of Brazil ?
What is the capital of Argentina ?
What is the capital of Chile ?
What is the capital of Peru ?
What is the capital of Colombia ?
What is the capital of Venezuela ?
What is the capital of Cuba ?
What is the capital of Haiti ?
What is the capital of the Dominican Republic ?
What is the capital of Jamaica ?
What is the capital of the Bahamas ?
What is the capital of Barbados ?
What is the capital of Trinidad and Tobago ?
What is the capital of Guyana ?
What is the capital of Suriname ?

Codex — code-cushman-001

",
        "answer": "1939"
    },
    {
        "question": "When was the first time the US entered the European Union ?",
        "answer": "1939"
    },
    {
        "question": "When was the first time the US entered the European Union ?",
        "answer": "1939"
    },
    {
        "question": "When was the first time the US entered the European Union ?",
        "answer": "1939"
    },
    {
        "question": "When was the first time the US entered the European Union ?",
        "answer": "1939"
    },
    {
        "question": "When was the first time the US entered the European Union ?",
        "answer": "1939"
    },
    {
        "question": "When was the first time the US entered the European Union ?",
        "answer": "1939"
    },
    {
        "question": "When was the first time the US entered the European Union ?",
        "answer": "1939"
    },
    {

Conclusion & observations

The main interest in juxtaposing the results is to see how different language models are in their analysis of the prompt, but also how they construct their responses.

It is also interesting to analyse the relevance of the results. This is particularly significant with historical facts, as we can see that the answers are sometimes incorrect, even surprising.

Once again, the objective of this article is not to propose a detailed analysis of each language, as each use case is different (and all of this will quickly become obsolete), but rather to become aware of the differences, but also and above all of the immense progress made in recent years and months.

Everything is moving very fast, I started writing this article two weeks ago, and OpenAI announced the closure of Codex last week…

If you liked this article 👉 Subscribe to my new publications

Get more exciting publications from Benoit