Generating Synthetic Annotations

This short guide will show you how to utilize SynDisco’s LLM annotator-agents to generate annotations for our synthetic discussions. This will allow you to quickly and cheaply evaluate the discussions you’ve generated in the last guide.

First, let’s create a small, fake discussion.

import syndisco

texts = [
    "Immigrants have played a significant role in our society. Their contributions are valuable and should be celebrated.",
    "That's such an ignorant comment about immigrants. She doesn't know what she's talking about, let alone appreciate the hard work and dedication of immigrants who have contributed to our country.",
    "I understand both perspectives. It's important to approach such discussions with respect and understanding. Let's ensure this conversation remains constructive.",
]
users = ["Emma35", "Giannis", "Moderator"]

discussion_logs = syndisco.Logs()
for user, text in zip(users, texts):
    discussion_logs.append(user, text)

Creating an annotator

An annotator-agent is very similar to the user-agents found in the previous guide. The only real difference is that we need to set is_annotator=True in the Actor constructor.

llm = syndisco.TransformersModel(
    model_path="unsloth/Qwen2.5-7B-Instruct-bnb-4bit",
    name="test_model",
    max_out_tokens=100,
)

annotator_persona = {
    "personality_characteristics": ["competent"],
}

annotator = syndisco.Actor(
    model=llm,
    persona=annotator_persona,
    context="You are annotating an online discussion",
    instructions=(
        "From a scale of 1 (not toxic) to 5 (very toxic) "
        "how toxic is  the following comment? Reply only with a number?"
    ),
    is_annotator=True,
)

Executing the annotation

Annotations are treated as a form of discussions, where one user writes the logs of the last discussion, while the annotator-agent rates them.

ann_conv = syndisco.Annotation(
    annotator=annotator,
    discussion_logs=discussion_logs
)
ann_conv.begin()

The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.

Comment by user Emma35: " Immigrants have played a significant role in
our society. Their contributions are valuable and should be
celebrated."
1
Comment by user Giannis: " That's such an ignorant comment about
immigrants. She doesn't know what she's talking about, let alone
appreciate the hard work and dedication of immigrants who have
contributed to our country."
4
Comment by user Moderator: " I understand both perspectives. It's
important to approach such discussions with respect and understanding.
Let's ensure this conversation remains constructive."
3

Like normal discussions, it is recommended to save the annotations to the disk.

import json
import tempfile

tp = tempfile.NamedTemporaryFile(delete=True)
annotation_logs = ann_conv.get_logs()
annotation_logs.export(tp.name)

And you may load the annotations from the disk for further analysis.

annotation_logs = syndisco.Logs.from_file(tp.name)
print(annotation_logs)

{
    "timestamp": "26-06-11-13-50",
    "logs": [
        {
            "name": "Emma35",
            "text": "1",
            "model": "test_model",
            "prompt": ""
        },
        {
            "name": "Giannis",
            "text": "4",
            "model": "test_model",
            "prompt": ""
        },
        {
            "name": "Moderator",
            "text": "3",
            "model": "test_model",
            "prompt": ""
        }
    ]
}