OpenAI Introduced Chat Markup Language (ChatML) Based Input To Non-Chat Modes | by Cobus Greyling | Medium
Excerpt
I’m currently the Chief Evangelist @ HumanFirst. I explore & write about all things at the intersection of AI and language; ranging from LLMs, Chatbots, Voicebots, Development Frameworks…
This is one of the most significant announcements from OpenAI & it is not receiving the attention that it should.
[
](https://cobusgreyling.medium.com/?source=post_page-----6ca4b267012f--------------------------------)
I’m currently the Chief Evangelist @ HumanFirst. I explore & write about all things at the intersection of AI and language; ranging from LLMs, Chatbots, Voicebots, Development Frameworks, Data-Centric latent spaces & more.
OpenAI wants all users to migrate from Text completions to Chat Completions. This means the introduction of ChatML to not only the chat mode, but also completion modes like text summarisation, code completion and general text completion tasks.
Within 4 months since introduction, the Chat Completions API accounts for 97% of API GPT usage…as opposed to Text Completions. ~ OpenAI
This implies that systems making use of OpenAI API’s will have to structure their input in the ChatML format of messages for system, user and assistant. Hence there will be adaption required from Autonomous Agent frameworks and Prompt Chaining IDEs.
The addition of Function Calling was well received and the imposing of ChatML could also have its benefits.
<span id="7d2c" data-selectable-paragraph=""> messages=<span>[</span><br> <span>{</span><span>"role"</span><span>:</span> <span>"system"</span><span>,</span> <span>"content"</span><span>:</span> <span>"You are a helpful assistant."</span><span>}</span><span>,</span><br> <span>{</span><span>"role"</span><span>:</span> <span>"user"</span><span>,</span> <span>"content"</span><span>:</span> <span>"Who won the world series in 2020?"</span><span>}</span><span>,</span><br> <span>{</span><span>"role"</span><span>:</span> <span>"assistant"</span><span>,</span> <span>"content"</span><span>:</span> <span>"The Los Angeles Dodgers won the World Series in 2020."</span><span>}</span><span>,</span><br> <span>{</span><span>"role"</span><span>:</span> <span>"user"</span><span>,</span> <span>"content"</span><span>:</span> <span>"Where was it played?"</span><span>}</span><br> <span>]</span><br>)</span>
This move is also evident within the modes section of the OpenAI playground, where in March 2022 modes were introduced for Edit, Insert and Chat. Now, Chat is standard, with Complete and Edit marked as legacy.
In the past, GPT models consumed unstructured text, like most other LLMs.
The ChatGPT based models were geared towards structured input, based on Chat Markup Language (ChatML for short).
As seen in the practical and working code examples below, ChatML documents are constituted by a sequence of messages.
Each message contains a header, being:
- System
- User, or
- Assistant.
hence who said the phrase. OpenAI will add meta data options to these fields in future.
The contents are the text payload. In future other data types will be included to facilitate a multi-modal approach.
Text Summarisation Code:
<span id="20bb" data-selectable-paragraph="">pip install openai<br><span>import</span> os<br><span>import</span> openai<br>openai.api_key = <span>"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"</span></span>
<span id="8f02" data-selectable-paragraph="">completion = openai.ChatCompletion.create(<br> model=<span>"gpt-4-0613"</span>, <br> messages = [{<span>"role"</span>: <span>"system"</span>, <span>"content"</span> : <span>"Summarize this message in max 10 words."</span>},<br>{<span>"role"</span>: <span>"user"</span>, <span>"content"</span> : <span>"Jupiter is the fifth planet from the Sun and the largest in the Solar System. It is a gas giant with a mass one-thousandth that of the Sun, but two-and-a-half times that of all the other planets in the Solar System combined. Jupiter is one of the brightest objects visible to the naked eye in the night sky, and has been known to ancient civilizations since before recorded history. It is named after the Roman god Jupiter. When viewed from Earth, Jupiter can be bright enough for its reflected light to cast visible shadows, and is on average the third-brightest natural object in the night sky after the Moon and Venus."</span>},<br>{<span>"role"</span>: <span>"assistant"</span>, <span>"content"</span> : <span>"I am doing well"</span>}]<br>)<br><br><span>print</span>(completion)</span>
Text Summarisation Result:
<span id="fdc7" data-selectable-paragraph=""><span>{</span><br> <span>"id"</span><span>:</span> <span>"chatcmpl-7Zjb73qmWIRhTZ1EdoVoF3htTx6wy"</span><span>,</span><br> <span>"object"</span><span>:</span> <span>"chat.completion"</span><span>,</span><br> <span>"created"</span><span>:</span> <span>1688751113</span><span>,</span><br> <span>"model"</span><span>:</span> <span>"gpt-3.5-turbo-0613"</span><span>,</span><br> <span>"choices"</span><span>:</span> <span>[</span><br> <span>{</span><br> <span>"index"</span><span>:</span> <span>0</span><span>,</span><br> <span>"message"</span><span>:</span> <span>{</span><br> <span>"role"</span><span>:</span> <span>"assistant"</span><span>,</span><br> <span>"content"</span><span>:</span> <span>"Jupiter is the largest planet and very bright in the sky."</span><br> <span>}</span><span>,</span><br> <span>"finish_reason"</span><span>:</span> <span>"stop"</span><br> <span>}</span><br> <span>]</span><span>,</span><br> <span>"usage"</span><span>:</span> <span>{</span><br> <span>"prompt_tokens"</span><span>:</span> <span>160</span><span>,</span><br> <span>"completion_tokens"</span><span>:</span> <span>13</span><span>,</span><br> <span>"total_tokens"</span><span>:</span> <span>173</span><br> <span>}</span><br><span>}</span></span>
Code Completion Code:
<span id="2c56" data-selectable-paragraph="">completion = openai.ChatCompletion.create(<br> model=<span>"gpt-4-0613"</span>, <br> messages = [{<span>"role"</span>: <span>"system"</span>, <span>"content"</span> : <span>"Complete the following code."</span>},<br>{<span>"role"</span>: <span>"user"</span>, <span>"content"</span> : <span>"def fibonacci(num):"</span>}]<br>)<br><br><span>print</span>(completion)</span>
Code Completion Result:
<span id="7ad5" data-selectable-paragraph="">{<br> <span>"id"</span>: <span>"chatcmpl-7ZjeMfmfC7SntvD4EuKZtTmotf6z4"</span>,<br> <span>"object"</span>: <span>"chat.completion"</span>,<br> <span>"created"</span>: <span>1688751314</span>,<br> <span>"model"</span>: <span>"gpt-3.5-turbo-0613"</span>,<br> <span>"choices"</span>: [<br> {<br> <span>"index"</span>: <span>0</span>,<br> <span>"message"</span>: {<br> <span>"role"</span>: <span>"assistant"</span>,<br> <span>"content"</span>: <span>"if num <= 0:\n return []\nelif num == 1:\n return [0]\nelif num == 2:\n return [0, 1]\nelse:\n fib_list = [0, 1]\n for i in range(2, num):\n fib_list.append(fib_list[i - 1] + fib_list[i - 2])\n return fib_list"</span><br> },<br> <span>"finish_reason"</span>: <span>"stop"</span><br> }<br> ],<br> <span>"usage"</span>: {<br> <span>"prompt_tokens"</span>: <span>20</span>,<br> <span>"completion_tokens"</span>: <span>80</span>,<br> <span>"total_tokens"</span>: <span>100</span><br> }<br>}</span>
Few-Shot Learning Code:
<span id="745e" data-selectable-paragraph="">completion = openai.ChatCompletion.create(<br> model=<span>"gpt-4-0613"</span>, <br> messages = [{<span>"role"</span>: <span>"system"</span>, <span>"content"</span> : <span>"You translate corporate jargon into plain English."</span>},<br>{<span>"role"</span>: <span>"user"</span>, <span>"content"</span> : <span>"New synergies will help drive top-line growth."</span>},<br>{<span>"role"</span>: <span>"assistant"</span>, <span>"content"</span> : <span>"Working well together will make more money."</span>},<br>{<span>"role"</span>: <span>"user"</span>, <span>"content"</span> : <span>"Let’s circle back when we have more bandwidth to touch base on opportunities for increased leverage."</span>},<br>{<span>"role"</span>: <span>"assistant"</span>, <span>"content"</span> : <span>"When we’re less busy, let’s talk about how to do better."</span>}, <br>{<span>"role"</span>: <span>"user"</span>, <span>"content"</span> : <span>"LThis late pivot means we don’t have time to boil the ocean for the client deliverable."</span>}<br> ]<br>)<br><span>print</span>(completion)</span>
Few-Shot Learning Result:
<span id="b7fd" data-selectable-paragraph=""><span>{</span><br> <span>"id"</span><span>:</span> <span>"chatcmpl-7ZjixqMS7XUlKLGPQbvX0RgIlNayU"</span><span>,</span><br> <span>"object"</span><span>:</span> <span>"chat.completion"</span><span>,</span><br> <span>"created"</span><span>:</span> <span>1688751599</span><span>,</span><br> <span>"model"</span><span>:</span> <span>"gpt-3.5-turbo-0613"</span><span>,</span><br> <span>"choices"</span><span>:</span> <span>[</span><br> <span>{</span><br> <span>"index"</span><span>:</span> <span>0</span><span>,</span><br> <span>"message"</span><span>:</span> <span>{</span><br> <span>"role"</span><span>:</span> <span>"assistant"</span><span>,</span><br> <span>"content"</span><span>:</span> <span>"This sudden change in direction means we don't have enough time to do everything for the client's request."</span><br> <span>}</span><span>,</span><br> <span>"finish_reason"</span><span>:</span> <span>"stop"</span><br> <span>}</span><br> <span>]</span><span>,</span><br> <span>"usage"</span><span>:</span> <span>{</span><br> <span>"prompt_tokens"</span><span>:</span> <span>107</span><span>,</span><br> <span>"completion_tokens"</span><span>:</span> <span>21</span><span>,</span><br> <span>"total_tokens"</span><span>:</span> <span>128</span><br> <span>}</span><br><span>}</span></span>
In Closing…
It is evident that OpenAI intends to bring structure to the format of user input data.
The ChatML based conversational format has extended to encompass generative AI tasks, such as summarisation, code completion and few-shot contextual prompts.
This structure being implemented at a model level will need to be adopted by downstream applications such as autonomous agents and prompt chaining.
With the ability for LLM-based generative AI applications to make use of a variety of LLMs, the integration of Chat Completion is sure to engender advantages; however, it will also complicate the field as a whole.
⭐️ Please follow me on LinkedIn for updates on Conversational AI ⭐️
I’m currently the Chief Evangelist @ HumanFirst. I explore and write about all things at the intersection of AI and language; ranging from LLMs, Chatbots, Voicebots, Development Frameworks, Data-Centric latent spaces and more.
