Use these instructions to answer the questions according to the screenshot that will be
provided.
INSTRUCTIONS AND GUIDELINES
Your Goal
Create 3 challenging, single-turn (not co-dependant) Question and Answer pairs for
an image that contains at least 3 subplots (charts/graphs).
The questions can be formulated about a specific subplot or more generally
✅
across them.
✅
Questions must be difficult for the model (model must answer incorrectly).
You must provide the Correct Answer. It must be short answers, such as a
✅
single word or number, or identifying label.
Provide a step-by-step CoT explaining how a human would solve the
question.
Task Steps
1. Determine the number of subplots
2. Determine the chart type
3. Create a question-answer pair
4. Create a question based on the visual elements of the subplots
in the image.
✅ Provide the correct answer (a single word/number) that references specific information in
the figure, such as a legend name.
✅ Use the “Check model answer” button to confirm that the model answers incorrectly.
✅ If the model answers correctly, rewrite the question to make it harder, and repeat until it
gets one wrong.
⚠️ It is crucial to ensure that the model does not already provide the correct answer to
your prompt.
⚠️ Questions should require some level of complexity for chain of thought (CoT).
Question Dos and Don’ts
❌ Avoid ambiguous questions. Do not make assumptions on the terminology used in the
figures.
❌Don’t write long answers. There should be no sentences in the answer box.
❌ DO NOT use any LaTeX formatting in your question (or answer).
✅Answers should be short: one word/number or identifying label (e.g. FedAvg in (a))
✅Questions should be relevant to the domain, but more specifically, you need to ask questions
a user would realistically ask.
✅We need a good distribution of question types, so please try to vary not only across the tasks
you do, but in the 3 QA pairs you create for a figure.
5. Write the Chain of Thought (CoT)
Once you’ve created a sufficiently difficult question, you will need to write a detailed Chain of
Thought (CoT) — this means writing a step-by-step explanation that outlines the reasoning
process required to arrive at the correct answer.
Your prompt needs to have some level of complexity, even if the model has difficulty in
answering correctly. This means that we are really looking for CoTs with more than just
4 steps.
CoT Dos and Don'ts
❌DO NOT INSTRUCT THE MODEL ON WHAT TO DO. You are meant to be describing
observations/calculations (look at the Good Example provided). Avoid the following
terms/instructions in CoT steps:
Locate
Identify
Note that
Trace
Look at
Read
Navigate to
Check
Let’s examine
Analyze
This list is in no way exhaustive. If your CoT steps instruct the model on what to do, it doesn't
matter whether you've used the above words or not - you're still not meeting the formal
requirements
❌ DO NOT use any LaTeX formatting in your CoT steps.
✅The CoT needs to be grounded in actual data.
✅Please use the correct axis labels as much as is possible - don't use X-axis or Y-axis if they
are clearly labeled.💡The targeted user (which is someone who works or studies in the various
domains) would use the labels. We need to keep the language as natural as possible.
✅Identify location of subplot or subplots. Please use your first CoT step to identify the correct
subplot: The CMR chart is the top left subplot in the figure.
🌈The color of the plot lines should be clearly stated. Even if the plot line has a label, you need
to clearly identify the color: In the CMR chart, IDEAL-RAND is the gray line and US-BS-MQ is
the red line.
Step-by-Step Reasoning Guidelines for CoT
✅Each step must be clearly explained
a. Instead of simply stating, "The temperature in subplot A increases," provide a detailed
observation, such as "In subplot A, the temperature increases by 5°C between time points 1-3."
✅Logical sequence and coherence
a. Each step should build on previous observations, with no logical gaps or unsubstantiated
conclusions.
b. Logical gaps are often caused by too little detail; however, too much detail may make the
reasoning overly complex without adding value.
✅Detailed and reproducible explanations
a. The relevance of each step should be clear and universally agreed on
b. The reasoning should clearly connect to the final answer and the steps structured
so that others can follow and reproduce the process
