How to work with ChatGPT and GPT-4 templates - Azure OpenAI Service (2023)

Working with the ChatGPT and GPT-4 templates

The following code snippet shows the most basic way to use the ChatGPT and GPT-4 templates with the Chat Completion API. If this is your first time using these templates programmatically, we recommend starting with our.

Currently, GPT-4 templates are available on request only.Existing Azure OpenAI customers canrequest access by filling out this form.

import osimport openaiopenai.api_type = "azure"openai.api_version = "2023-05-15" openai.api_base = os.getenv("OPENAI_API_BASE") # Endpoint value of your Azure resource OpenAI.openai.api_key = os. getenv("OPENAI_API_KEY ")response = openai.ChatCompletion.create( engine="gpt-35-turbo", # The deployment name you chose when deploying the ChatGPT or GPT-4 template. messages=[ {"role": "system", " content": "Assistant is a great language model trained by OpenAI."}, {"role": "user", "content": "Who were the founders of Microsoft?"} ])print( response)print(response[ 'choices'][0]['message']['content'])

Exit

{ "choices": [ { "finish_reason": "stop", "index": 0, "message": { "content": "The founders of Microsoft are Bill Gates and Paul Allen. They co-founded the company in 1975 .", "role": "assistant" } } ], "created": 1679014551, "id": "chatcmpl-6usfn2yyjkbmESe3G4jaQR6bsScO1", "model": "gpt-3.5-turbo-0301", "object": "chat .completion", "usage": { "completion_tokens": 86, "prompt_tokens": 37, "total_tokens": 123 }}

Each answer includes aend_reason. The possible values ​​forend_reasonthey are:

• to stop: The API returned the full output of the model.
• length: Model output incomplete due to max_tokens parameter or token limit.
• content_filter: Content omitted due to a flag from our content filters.
• null:API still in progress or incomplete.

consider setting upmax_tokensto a value slightly higher than normal, such as 300 or 500. This ensures that the template doesn't stop generating text before reaching the end of the message.

model version

Unlike previous GPT-3 and GPT-3.5 models, thegpt-35-turbomodel, as well as thegpt-4egpt-4-32kmodels will continue to be updated. When creating aImplantationof these templates, you will also need to specify a template version.

Currently, only the version0301is available for ChatGPT and0314for GPT-4 models. We will continue to make updated versions available in the future. You can find model discontinuation times on ourmodelspage.

Working with the Chat Completion API

OpenAI has trained the ChatGPT and GPT-4 models to accept input formatted as a conversation. The messages parameter takes an array of dictionaries with a conversation organized by function.

The format of a basic chat completion is as follows:

{"role": "system", "content": "Provide some context and/or instructions for the model"},{"role": "user", "content": "User messages go here"}

A conversation with an example reply followed by a question would look like this:

{"role": "system", "content": "Provide some context and/or instructions for the model."},{"role": "user", "content": "Sample question goes here."} , {"role": "assistant", "content": "Example answer goes here."},{"role": "user", "content": "First question/message for model to actually answer."}

system function

The system function, also known as the system message, is added at the beginning of the array. This message provides initial instructions for the template. You can provide a variety of information in the system role, including:

• A brief description of the wizard
• Assistant personality traits
• Instructions or rules you would like the assistant to follow
• Data or information needed for the model, such as relevant FAQ questions

You can customize the system role for your use case or just include basic instructions. The system function/message is optional, but it is recommended that you include at least a basic function for best results.

messages

After the system function, you can include a series of messages between thefrom the userit's atassistant.

{"role": "user", "content": "What is thermodynamics?"}

To trigger a template response, you must end with a message from the user indicating that it is the wizard's turn to respond. You could also include a series of example messages between the user and the assistant as a way to learn some shots.

Message prompt examples

The following section shows examples of different styles of prompts that you can use with the ChatGPT and GPT-4 templates. These examples are just a starting point, and you can experiment with different prompts to customize the behavior for your own use cases.

basic example

If you want the ChatGPT template to behave similarly tochat.openai.com, you can use a basic system message like "Assistant is a large language model trained by OpenAI".

{"role": "system", "content": "Assistant is a great language model trained by OpenAI."},{"role": "user", "content": "Who were the founders of Microsoft?" }

Example with instructions

For some scenarios, you might want to provide additional instructions to the model to define guardrails for what the model is capable of doing.

{"role": "system", "content": "Assistant is an intelligent chatbot designed to help users answer their tax-related questions. Instructions: - Answer tax-related questions only. - If unsure of a response, you can say "I don't know" or "I'm not sure" and recommend that users go to the IRS website for more information."},{"role": "user", "content": "When are my taxes due?"}

Using data for grounding

You can also include relevant data or information in the system message to give the template extra context for the conversation. If you only need to include a small amount of information, you can encode it in the system message. If you have a large amount of data that the model must be aware of, you can usemergersor a product likeAzure Cognitive Searchto retrieve the most relevant information at the time of the query.

{"role": "system", "content": "Assistant is an intelligent chatbot designed to help users answer technical questions about the Azure OpenAI service. Please only answer questions using the context below, and if unsure of a answer, you might say 'I don't know'. Context:- The Azure OpenAI service provides REST API access to OpenAI's powerful language models, including the GPT-3, Codex, and Embeddings model series.- The Azure OpenAI service offers users language-rich AI clients with OpenAI GPT-3, Codex, and DALL-E templates with the enterprise and security promise of Azure. Azure OpenAI co-develops the APIs with OpenAI, ensuring compatibility and a smooth transition from one to the next Other.- At Microsoft, we are committed to advancing AI driven by principles that put people first. Microsoft has made significant investments to help protect against abuse and unintentional harm, which includes requiring applicants to show well-defined use cases, incorporating Microsoft's principles for responsible use of AI."},{" role": "user", "content": "What is the Azure OpenAI service?"}

Few shots learning from chat completion

You can also give some sample photos for the template. The approach to learning a few shots has changed a bit because of the new prompt format. You can now include a series of messages between the user and the assistant at the prompt as some sample shots. These examples can be used to seed answers to common questions to prepare the model or teach model specific behaviors.

This is just one example of how you can use few shot learning with ChatGPT and GPT-4. You can experiment with different approaches to see what works best for your use case.

{"role": "system", "content": "Assistant is an intelligent chatbot designed to help users answer their tax-related questions. "},{"role": "user", "content": "When Do I need to file my taxes by?"},{"role": "assistant", "content": "In 2023, you will need to file your taxes by April 18. The date falls after the normal April 15 deadline because April 15 April falls on a Saturday in 2023. For more details, see https://www.irs.gov/filing/individuals/when-to-file."},{"role": "user", "content" : "How can I check the status of my tax refund?"},{"role": "assistant", "content": "You can check the status of your tax refund by visiting https://www.irs.irs. gov/refunds"}

Using chat completion for non-chat scenarios

The Chat Completion API is designed to work with multi-turn conversations, but it also works well for non-chat scenarios.

For example, for an entity extraction scenario, you could use the following prompt:

{"role": "system", "content": "You are a wizard designed to extract entities from text. Users will paste a string of text and you will respond with entities extracted from the text as a JSON object. Here is an example of the your output format:{ "name": "", "company": "", "phone_number": ""}"},{"role": "user", "content": "Hello. My name is Robert Smith. I'm calling from Contoso Insurance, Delaware. My colleague mentioned that you're interested in learning more about our comprehensive benefits policy. Would you please call me back at (555) 346-9322 when you get a chance so we can review benefits ?"}

Creating a basic conversation loop

The examples so far have shown the basic mechanics of interacting with the Chat Completion API. This example shows how to create a conversation loop that performs the following actions:

• Continuously takes input from the console and formats it appropriately as part of the array of messages as user role content.
• It issues responses that are printed to the console and formatted and added to the message array as assistant function content.

This means that every time a new question is asked, a transcript of the conversation so far is sent along with the last question. Since the template has no memory, you must submit an updated transcript with each new question or the template will lose context for previous questions and answers.

import osimport openaiopenai.api_type = "azure"openai.api_version = "2023-05-15" openai.api_base = os.getenv("OPENAI_API_BASE") # Azure OpenAI resource endpoint value .openai.api_key = os.getenv ("OPENAI_API_KEY ")conversation=[{"role": "system", "content": "You are a helpful assistant."}]while(True): user_input = input() conversation.append({"role": "user" , "content": user_input}) response = openai.ChatCompletion.create( engine="gpt-3.5-turbo", # The deployment name you chose when you deployed the ChatGPT or GPT-4 template. messages = conversation ) conversation.append ({"role": "assistant", "content": reply['choices'][0]['message']['content']}) print("\n" + reply['choices '][0 ]['message']['content'] + "\n")

When running the above code, you will get a blank console window. Type your first question into the window and press Enter. Once the answer is returned, you can repeat the process and continue asking questions.

managing conversations

The previous example will run until it reaches the model token limit. With every question asked and answer received, themessagesmatrix increases in size. The token limit forgpt-35-turbois 4096 tokens, while token limits forgpt-4egpt-4-32kare 8192 and 32768 respectively. These limits include the token count of the sent message array and the model response. The number of tokens in the message array combined with the value ofmax_tokensparameter must remain within these limits or you will get an error.

It is your responsibility to ensure that the request and completion are within the token limit. This means that for longer conversations you need to keep track of the token count and only send the model a prompt that falls within the limit.

The following code example shows a simple chat loop example with a technique for handling a 4096 token count using OpenAI's tiktoken library.

Or code requer tiktoken0.3.0. If you have an older version, runpip install tiktoken --upgrade.

import tiktokenimport openaiimport osopenai.api_type = "azure"openai.api_version = "2023-05-15" openai.api_base = os.getenv("OPENAI_API_BASE") # Valor do endpoint do recurso Azure OpenAI .openai.api_key = os.getenv(" OPENAI_API_KEY")system_message = {"role": "system", "content": "Você é um assistente útil."}max_response_tokens = 250token_limit= 4096conversation=[]conversation.append(system_message)def num_tokens_from_messages(messages, model="gpt -3.5-turbo-0301"): encoding = tiktoken.encoding_for_model(model) num_tokens = 0 for message in messages: num_tokens += 4 # toda mensagem segue{role/name}\n{content}\n for key, value in message.items(): num_tokens += len(encoding.encode(value)) if key == "name": # if there is a name, role is omitted num_tokens += -1 # role is always required and always 1 token num_tokens += 2 # each response is prepared withassistant return num_tokenswhile(True): user_input = input("") conversation.append({"role": "user", "content": user_input}) conv_history_tokens = num_tokens_from_messages(conversation) while (conv_history_tokens+max_response_tokens >= token_limit): del conversation[1] conv_history_tokens = num_tokens_from_messages(conversation) response = openai.ChatCompletion.create( engine="gpt-35-turbo", # The deployment name you chose when deploying the ChatGPT or GPT-4 template. messages = conversation , temperature=.7, max_tokens=max_response_tokens, ) conversation.append({"role": "assistant", "content": response['choices'][0]['message']['content'] }) print ("\n" + answer['choices'][0]['message']['content'] + "\n")

In this example, when the token count is reached, the oldest messages in the conversation transcript will be removed.dois used instead ofpop()for efficiency, and we start at index 1 to always preserve the system message and only remove user/assistant messages. Over time, this method of managing the conversation can cause the quality of the conversation to degrade, as the model will gradually lose context from earlier parts of the conversation.

An alternative approach is to limit the conversation duration to the maximum token size or a certain number of rounds. When the maximum token limit is reached and the model loses context if you allow the conversation to continue, you can ask the user that he needs to start a new conversation and clear the message array to start a new conversation with the token limit available.

The token counting portion of the code demonstrated earlier is a simplified version of one of theOpenAI cookbook examples.

Next steps

• Learn more about Azure OpenAI.
• Start with the ChatGPT template withthe ChatGPT quickstart.
• For more examples, check out theGitHub repository of Azure OpenAI samples

Working with the ChatGPT templates

The following code snippet shows the most basic way to use ChatGPT templates with ChatML. If this is your first time using these templates programmatically, we recommend starting with our.

import osimport openaiopenai.api_type = "azure"openai.api_base = "https://{your-resource-name}.openai.azure.com/"openai.api_version = "2023-05-15"openai.api_key = os. getenv("OPENAI_API_KEY")response = openai.Completion.create( engine="gpt-35-turbo", # The deployment name you chose when deploying the ChatGPT template prompt="<|im_start|>system\nAssistant is a great language model trained by OpenAI.\n<|im_end|>\n<|im_start|>user\nWho were the founders of Microsoft?\n<|im_end|>\n<|im_start|>assistant\n", temperature= 0, max_tokens=500, top_p=0.5, stop=["<|im_end|>"])print(response['choices'][0]['text'])

O<|im_end|>token indicates the end of a message. We recommend including<|im_end|>token as a stop string to ensure the template stops generating text when it reaches the end of the message. You can read more about special tokens atChat Markup Language (ChatML)section.

consider setting upmax_tokensto a value slightly higher than normal, such as 300 or 500. This ensures that the template doesn't stop generating text before reaching the end of the message.

model version

Unlike previous GPT-3 and GPT-3.5 models, thegpt-35-turbomodel, as well as thegpt-4egpt-4-32kmodels will continue to be updated. When creating aImplantationof these templates, you will also need to specify a template version.

Currently, only the version0301is available for ChatGPT. We will continue to make updated versions available in the future. You can find model discontinuation times on ourmodelspage.

Working with Chat Markup Language (ChatML)

OpenAI trained ChatGPT on special tokens that delineate the different parts of the prompt. The prompt begins with a system message that is used to prepare the model, followed by a series of messages between the user and the wizard.

The format of a basic ChatML prompt is as follows:

<|im_start|>system Provide some context and/or instructions for the model.<|im_end|> <|im_start|>user User message goes here<|im_end|> <|im_start|>wizard

system message

The system message is added to the beginning of the prompt between the<|im_start|>systeme<|im_end|>tokens. This message provides initial instructions for the template. You can provide a variety of information in the system message, including:

• A brief description of the wizard
• Assistant personality traits
• Instructions or rules you would like the assistant to follow
• Data or information needed for the model, such as relevant FAQ questions

You can customize the system message for your use case or just add a basic system message. The system message is optional, but it's recommended that you include at least a basic one for best results.

messages

After the system message, you can include a series of messages betweenfrom the userit's atassistant. Each message must begin with the<|in_start|>token followed by role (from the userorassistant) and finish with the<|im_end|>symbol.

<|im_start|>userWhat is thermodynamics?<|im_end|>

To trigger a template response, the prompt must end with<|im_start|>wizardtoken indicating that it is the assistant's turn to respond. You can also include messages between the user and the assistant in the prompt as a way to learn some pictures.

Prompt Examples

The following section shows examples of different styles of prompts that you can use with the ChatGPT and GPT-4 templates. These examples are just a starting point, and you can experiment with different prompts to customize the behavior for your own use cases.

basic example

If you want ChatGPT and GPT-4 templates to behave similarlychat.openai.com, you can use a basic system message like "Assistant is a large language model trained by OpenAI".

<|im_start|>systemAssistant is a great language model trained by OpenAI.<|im_end|><|im_start|>userWho were the founders of Microsoft?<|im_end|><|im_start|>assistant

Example with instructions

For some scenarios, you might want to provide additional instructions to the model to define guardrails for what the model is capable of doing.

<|im_start|>systemAssistant is an intelligent chatbot designed to help users answer their tax-related questions. Instructions:- Answer tax-related questions only. - If unsure of an answer, say "I don't know" or "I'm not sure" and recommend that users go to the IRS website for more information.<|im_end|><|im_start|> userWhen my taxes are due ?<|im_end|><|im_start|>assistant

Using data for grounding

You can also include relevant data or information in the system message to give the template extra context for the conversation. If you only need to include a small amount of information, you can encode it in the system message. If you have a large amount of data that the model must be aware of, you can usemergersor a product likeAzure Cognitive Searchto retrieve the most relevant information at the time of the query.

<|im_start|>systemAssistant is an intelligent chatbot designed to help users answer technical questions about the Azure OpenAI Service. Please only answer the questions using the context below, and if you are unsure of an answer, say "I don't know". Codex Template Series and Embeddings.- Azure OpenAI Service provides customers with language-rich AI with OpenAI GPT-3, Codex, and DALL-E templates with the enterprise and security promise of Azure. Azure OpenAI co-develops the APIs with OpenAI, ensuring compatibility and a smooth transition from one to the other. At Microsoft, we are committed to advancing AI driven by principles that put people first. Microsoft has made significant investments to help protect against abuse and unintentional harm, which includes requiring applicants to show well-defined use cases, incorporating Microsoft's principles for responsible use of AI<|im_end|><|im_start|> userWhat is Azure OpenAI Service?< |im_end|><|im_start|>wizard

Few shots learning with ChatML

You can also give some sample photos for the template. The approach to learning a few shots has changed a bit because of the new prompt format. You can now include a series of messages between the user and the assistant at the prompt as some sample shots. These examples can be used to seed answers to common questions to prepare the model or teach model specific behaviors.

This is just one example of how you can use few shot learning with ChatGPT. You can experiment with different approaches to see what works best for your use case.

<|im_start|>systemAssistant is an intelligent chatbot designed to help users answer their tax-related questions. <|im_end|><|im_start|>userWhen do I need to file my taxes by?<|im_end|><|im_start|>assistantIn 2023, you will need to file your taxes by April 18th. The date falls after the normal April 15 deadline because April 15 falls on a Saturday in 2023. For more details, see https://www.irs.gov/filing/individuals/when-to-file<|im_end |><| im_start|>userHow can I check the status of my tax refund?<|im_end|><|im_start|>assistantYou can check the status of your tax refund by visiting https://www.irs.gov/refunds<|im_end| >

Using chat markup language for non-chat scenarios

ChatML was designed to make it easy to manage multi-shift conversations, but it also works well in non-chat scenarios.

For example, for an entity extraction scenario, you could use the following prompt:

<|im_start|>systemYou is a wizard designed to extract entities from text. Users will paste a string of text and you will respond with entities extracted from the text as a JSON object. Here is an example of its output format:{ "name": "", "company": "", "phone_number": ""}<|im_end|><|im_start|>userHello. My name is Robert Smith. I'm calling from Contoso Insurance, Delaware. My colleague mentioned that you are interested in learning about our comprehensive benefits policy. Would you please call me back at (555) 346-9322 when you get a chance so we can talk about the benefits?<|im_end|><|im_start|>assistant

Preventing unsafe user input

It's important to add mitigations to your application to ensure secure use of Chat Markup Language.

We recommend that you prevent end users from being able to include special tokens in their entries, such as<|in_start|>e<|im_end|>. We also recommend that you include additional validation to ensure that the prompts you are sending to the template are well-formed and follow the Chat Markup Language format as described in this document.

You can also provide instructions in the system message to guide the model on how to respond to certain types of user input. For example, you can instruct the template to only reply to messages about a certain subject. You can also reinforce this behavior with some example shots.

managing conversations

The token limit forgpt-35-turbois 4096 chips. This limit includes the count of prompt and completion tokens. The number of tokens in the prompt combined with the value ofmax_tokensparameter must remain below 4096 or you will get an error.

It is your responsibility to ensure that the request and completion are within the token limit. This means that for longer conversations you need to keep track of the token count and only send the model a prompt that is within the token limit.

The following code example shows a simple example of how you can keep track of separate messages in the conversation.

import osimport openaiopenai.api_type = "azure"openai.api_base = "https://{your-resource-name}.openai.azure.com/" #This matches the endpoint value of your Azure resource OpenAIopenai.api_version = "2023-05 -15" openai.api_key = os.getenv("OPENAI_API_KEY")# defining a function to create the prompt from the system message and chat messagesdef create_prompt(system_message, messages): prompt = system_message for message in messages: prompt += f "\n<|im_start|>{message['sender']}\n{message['text']}\n<|im_end|>" prompt += "\n<|im_start |>assistant\n" return prompt# setting user input and system messageuser_input = "" system_message = f"<|im_start|>system\n{''}\n<|im_end|>"# creating a message list to track the conversationmessages = [{"sender": "user", "text": user_input}]response = openai.Completion.create( engine="gpt -35-turbo", # The deployment name you chose when you deployed the ChatGPT template. prompt=create_prompt(system_message, messages), temperature=0.5, max_tokens=250, top_p=0.9, frequency_penalty=0, Presence_penalty=0, stop=['<|im_end|>'])messages.append ({"sender": "assistant", "text": reply['choices'][0]['text']})print (answer['choices'][0]['text'])

Staying below the token limit

The simplest approach to getting under the token limit is to remove the oldest messages from the conversation when you hit the token limit.

You can choose to always include as many tokens as possible while staying under the limit, or you can always include a set number of past messages, assuming those messages stay under the limit. It's important to keep in mind that longer prompts take longer to generate a response and incur a higher cost than shorter prompts.

You can estimate the number of tokens in a string using the methodtiktokPython library as shown below.

import tiktoken cl100k_base = tiktoken.get_encoding("cl100k_base") enc = tiktoken.Encoding( name="gpt-35-turbo", pat_str=cl100k_base._pat_str, mergeable_ranks=cl100k_base._mergeable_ranks, special_tokens={ **cl100k_base._special_tokens, " <|im_start|>": 100264, "<|im_end|>": 100265 } ) tokens = enc.encode( "<|im_start|>usuário\nOlá<|im_end|><|im_start|>assistente", permitido_especial= {"<|im_start|>", "<|im_end|>"} ) assert len(tokens) == 7 assert tokens == [100264, 882, 198, 9906, 100265, 100264, 78191]

Next steps

• Learn more about Azure OpenAI.
• Start with the ChatGPT template withthe ChatGPT quickstart.
• For more examples, check out theGitHub repository of Azure OpenAI samples