Repository navigation
Project Types
The Instruction Driven Chat project type is designed to collect prompts, providing insights into how people interact with large language models (LLMs) like ChatGPT. Prompts are essentially the commands, questions, or instructions given to ChatGPT to achieve a specific outcome or satisfy an intention. In this project type, users are provided with detailed instructions, which include an intent, a domain, and a specified language format to follow when entering the prompt. Based on these instructions, the annotator (the user entering the prompt) types a prompt, which, when sent, will generate a response from the LLM. This interaction can be as simple as a single prompt-response pair, or it can evolve into a multi-turn conversation where the annotator continues to enter prompts in response to the model's answers.
Once the prompts are submitted, they are sent to specific reviewers assigned to the project. These reviewers have the authority to accept the prompt, reject it, or send it back to the annotator for modifications, accompanied by review notes. When a prompt is accepted, it is added to the data pool. However, if a prompt is sent back for revision, the annotator must revise and resubmit it for the same reviewer to reassess and potentially accept.
Below is what the intruction driven task project page, which lists out the tasks looks like:
Below is what the instruction driven chat task page, where the prompts are to be submitted look like:
The Model Interaction Evaluation project type is focused on evaluating the responses generated from the prompts created in the Instruction Driven Chat. In this project type, annotators assess the outputs produced by the LLM in response to every prompt. This evaluation is guided by a set of predesigned questions that help measure the quality and effectiveness of the responses.
By systematically evaluating these outputs, the collected data provides valuable insights into the performance and accuracy of the model's responses. This information is crucial for understanding how well the LLM meets the desired standards and expectations.
Below is what a model interaction evaluation task page interface looks like:
