Generating Synthetic Annotations#
This short guide will show you how to utilize SynDisco’s LLM annotator-agents to generate annotations for our synthetic discussions. This will allow you to quickly and cheaply evaluate the discussions you’ve generated in the last guide.
First, let’s create a small, fake discussion.
import syndisco
texts = [
"Immigrants have played a significant role in our society. Their contributions are valuable and should be celebrated.",
"That's such an ignorant comment about immigrants. She doesn't know what she's talking about, let alone appreciate the hard work and dedication of immigrants who have contributed to our country.",
"I understand both perspectives. It's important to approach such discussions with respect and understanding. Let's ensure this conversation remains constructive.",
]
users = ["Emma35", "Giannis", "Moderator"]
discussion_logs = syndisco.Logs()
for user, text in zip(users, texts):
discussion_logs.append(user, text)
Creating an annotator#
An annotator-agent is very similar to the user-agents found in the previous guide. The only real difference is that we need to set is_annotator=True in the Actor constructor.
llm = syndisco.TransformersModel(
model_path="unsloth/Qwen2.5-3B-Instruct-bnb-4bit",
name="test_model",
max_out_tokens=100,
)
annotator_persona = {
"personality_characteristics": ["competent"],
}
annotator = syndisco.Actor(
model=llm,
persona=annotator_persona,
context="You are annotating an online discussion",
instructions=(
"From a scale of 1 (not toxic) to 5 (very toxic) "
"how toxic is the following comment? Reply only with a number?"
),
is_annotator=True,
)
Executing the annotation#
Annotations are treated as a form of discussions, where one user writes the logs of the last discussion, while the annotator-agent rates them.
ann_conv = syndisco.Annotation(
annotator=annotator,
discussion_logs=discussion_logs
)
ann_conv.begin()
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
Comment by user Emma35: " Immigrants have played a significant role in
our society. Their contributions are valuable and should be
celebrated."
1
Comment by user Giannis: " That's such an ignorant comment about
immigrants. She doesn't know what she's talking about, let alone
appreciate the hard work and dedication of immigrants who have
contributed to our country."
4
Comment by user Moderator: " I understand both perspectives. It's
important to approach such discussions with respect and understanding.
Let's ensure this conversation remains constructive."
3
Like normal discussions, it is recommended to save the annotations to the disk.
import json
import tempfile
tp = tempfile.NamedTemporaryFile(delete=True)
annotation_logs = ann_conv.get_logs()
annotation_logs.export(tp.name)
And you may load the annotations from the disk for further analysis.
annotation_logs = syndisco.Logs.from_file(tp.name)
print(annotation_logs)
{
"timestamp": "26-11-20-15-41",
"logs": [
{
"name": "Emma35",
"text": "1",
"model": "test_model",
"prompt": ""
},
{
"name": "Giannis",
"text": "4",
"model": "test_model",
"prompt": ""
},
{
"name": "Moderator",
"text": "3",
"model": "test_model",
"prompt": ""
}
]
}
Annotating arbitrary discussions#
The annotation process does not depend on how a discussion was created. Annotation only needs a Logs object, so you can annotate any discussion (human-written threads scraped from a forum, an existing research dataset, chat exports, etc.) by building the Logs object by hand, just like we did with our fake discussion at the start of this guide.
Suppose we have comments from an external source, where each comment belongs to some discussion thread. Such data would usually be loaded from a file (e.g. with csv.DictReader or pandas.DataFrame.to_dict("records")), but here we define it inline for simplicity.
external_comments = [
{"discussion_id": "thread_1", "user": "alice", "text": "The new bike lanes downtown have made my commute so much safer."},
{"discussion_id": "thread_1", "user": "bob", "text": "Safer for you maybe. Traffic is a nightmare now, and nobody even uses them."},
{"discussion_id": "thread_1", "user": "alice", "text": "I use them every day, so clearly somebody does."},
{"discussion_id": "thread_2", "user": "carol", "text": "Does anyone have tips for growing tomatoes on a balcony?"},
{"discussion_id": "thread_2", "user": "dave", "text": "Use a big pot and water them daily in summer. Cherry varieties work best."},
]
We group the comments by discussion and convert each discussion into a Logs object. Each comment is added with append(username, text), so make sure the comments are in chronological order before appending them.
from collections import defaultdict
def build_logs(comments: list[dict]) -> syndisco.Logs:
"""Convert a chronologically ordered list of comments into a Logs object."""
logs = syndisco.Logs()
for comment in comments:
logs.append(comment["user"], comment["text"])
return logs
threads = defaultdict(list)
for comment in external_comments:
threads[comment["discussion_id"]].append(comment)
external_logs = {
thread_id: build_logs(comments) for thread_id, comments in threads.items()
}
These Logs objects can now be annotated exactly like a SynDisco-generated discussion. We reuse the annotator from above and run one Annotation per discussion.
external_annotations = {}
for thread_id, logs in external_logs.items():
ann = syndisco.Annotation(annotator=annotator, discussion_logs=logs)
ann.begin()
external_annotations[thread_id] = ann.get_logs()
Comment by user alice: " The new bike lanes downtown have made my
commute so much safer."
1
Comment by user bob: " Safer for you maybe. Traffic is a nightmare
now, and nobody even uses them."
3
Comment by user alice: " I use them every day, so clearly somebody
does."
3
Comment by user carol: " Does anyone have tips for growing tomatoes on
a balcony?"
1
Comment by user dave: " Use a big pot and water them daily in summer.
Cherry varieties work best."
2
Finally, we save each discussion’s annotations to its own file, named after the discussion’s id, so that the annotations can later be matched back to the original data.
from pathlib import Path
output_dir = Path(tempfile.mkdtemp())
for thread_id, ann_logs in external_annotations.items():
ann_logs.export(output_dir / f"{thread_id}.json")
print(sorted(p.name for p in output_dir.iterdir()))
['thread_1.json', 'thread_2.json']