Creating and Managing Experiments
The last two guides showcased how you can create and run synthetic discussions, and synthetic annotations using LLMs. However, in order to produce robust results for a hypothesis, you may need to produce multiple annotated discussions.
While this is certainly possible using the Discussion and Annotation APIs, SynDisco offers the Experiment high-level API which automatically creates and manages multiple discussions with different configurations. AnExperiment is an entity that generates and runs jobs. Thus, if we want to generate and run 100 Discussion jobs, we would use a DiscussionExperiment. Likewise, if we want to annotate those 100 discussions, we would use an AnnotationExperiment.
This guide will showcase how you can leverage this API to automate your experiments. You will also learn how to utilize SynDisco’s built-in logging functions as well as how to export your datasets in CSV format for convenience.
Logging
While running a single discussion or annotation job may take a few minutes, running experiments composed of dozens or hundreds of synthetic discussions may take up to days. Thus, we need a mechanism to keep track of our experiments while they are running.
We will use SynDisco’s logging_util module to log information about our experiments. This module performs the following functions:
Times the execution of computationally intensive jobs (such as synthetic discussions and annotations)
Provides details about the currently running jobs (e.g. selected configurations, participants, prompts etc.)
Displays warnings and errors to the user
Creates and continually updates log files
Each object in SynDisco is internally assigned a Logger. You can use the logging_util.logging_setup function to update all of the internal loggers to follow your configuration. An example of this can be seen below:
[2]:
from pathlib import Path
import tempfile
import syndisco
logs_dir = tempfile.TemporaryDirectory()
syndisco.logging.logging_setup(
print_to_terminal=True,
write_to_file=True,
logs_dir=Path(logs_dir.name),
level="info",
use_colors=True,
log_warnings=True,
)
The loggers are applicable for all objects in SynDisco, and as such can be used for information on Discussion, and Annotation jobs, as well as all low-level components (such as those in the backend module).
It is recommended to set up the loggers no matter your use case. At the very least, they are useful for clearly displaying warnings in case of accidental API misuse.
Discussion Experiments
[3]:
CONTEXT = "You are taking part in an online conversation"
INSTRUCTIONS = "Act like a human would"
llm = syndisco.TransformersModel(
model_path="unsloth/Llama-3.2-3B-Instruct-bnb-4bit",
name="test_model",
max_out_tokens=50,
)
persona_data = [
{
"username": "Emma35",
"age": 38,
"sex": "female",
"education_level": "Bachelor's",
"sexual_orientation": "Heterosexual",
"demographic_group": "Latino",
"current_employment": "Registered Nurse",
"special_instructions": "",
"personality_characteristics": [
"compassionate",
"patient",
"diligent",
"overwhelmed",
],
},
{
"username": "Giannis",
"age": 21,
"sex": "male",
"education_level": "College",
"sexual_orientation": "Pansexual",
"demographic_group": "White",
"current_employment": "Game Developer",
"special_instructions": "Be antagonistic towards the other user",
"personality_characteristics": [
"strategic",
"meticulous",
"nerdy",
"hyper-focused",
],
},
]
personas = [syndisco.Persona(**data) for data in persona_data]
actors = [
syndisco.Actor(
model=llm,
persona=p,
context=CONTEXT,
instructions=INSTRUCTIONS,
actor_type=syndisco.ActorType.USER,
)
for p in personas
]
turn_manager = syndisco.RoundRobin([actor.get_name() for actor in actors])
2026-04-03 13:06:11 CP-G482-Z52-00 accelerate.utils.modeling[2141477] INFO We will use 90% of the memory on device 0 for storing the model, and 10% for the buffer to avoid OOM. You can set `max_memory` in to a higher value to use more memory (at your own risk).
2026-04-03 13:06:13 CP-G482-Z52-00 model.py[2141477] INFO Model memory footprint: 2095.83 MB
[4]:
disc_exp = syndisco.DiscussionExperiment(
seed_opinions=[
["Should programmers be allowed to analyze data?", "Absolutely not"],
["Should data analysts be allowed to code?", "No they are nerds"],
],
users=actors,
num_turns=3,
num_discussions=2,
)
discussions_dir = Path(tempfile.TemporaryDirectory().name)
disc_exp.begin(discussions_output_dir=discussions_dir)
2026-04-03 13:06:13 CP-G482-Z52-00 experiments.py[2141477] WARNING No TurnManager selected: Defaulting to round robin strategy.
2026-04-03 13:06:13 CP-G482-Z52-00 root[2141477] INFO Running experiment 1/3...
The following generation flags are not valid and may be ignored: ['temperature', 'top_p']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
User Emma35 posted:
"Hey everyone, I just got back from a long shift at the hospital and
I'm exhausted. Anyone else have days like this? I swear, some patients
just keep piling it on and I'm running on fumes. How do you all
User Giannis posted:
{"role": "Giannis", "content": "Ugh, hospitals? Really? You think you
have it bad? I've been stuck in a meeting with our dev team for 8
hours straight, trying to get the game's physics engine
2026-04-03 13:06:22 CP-G482-Z52-00 root[2141477] INFO Running experiment 2/3...
User Emma35 posted:
"Ah, Giannis, I feel you, but at least you got to go home after the
meeting. I've been on my feet for 12 hours straight, running from one
patient emergency to the next. I'm pretty sure I've aged
User Giannis posted:
"Ugh, really? You think you can just waltz in here and start spewing
your nonsense without even bothering to proofread your post? I swear,
I've seen high schoolers with better grammar and spelling. What's your
excuse
User Emma35 posted:
{"role": "Emma35", "content": "I'm sorry to hear that my post didn't
meet your expectations. I'm a nurse, not a professional writer, and I
was just trying to get my point across. I didn't mean
2026-04-03 13:06:29 CP-G482-Z52-00 experiments.py[2141477] INFO Finished synthetic discussion generation.
User Giannis posted:
{"role": "Giannis", "content": "Oh, spare me the sob story, Emma. You
think being a nurse makes you some kind of expert on grammar and
writing? Newsflash: it doesn't. I've seen your posts,
Annotation Experiments
Annotation experiments allow you to use multiple annotator-agents with different characteristics for each comment in a discussion. You could also use multiple annotators with the same characteristics, to obtain more robust results.
[5]:
annotator_persona1 = syndisco.Persona(
**{
"username": "annotator1",
"age": 38,
"sex": "female",
"education_level": "Bachelor's",
"sexual_orientation": "Heterosexual",
"demographic_group": "White",
"current_employment": "Annotator",
"personality_characteristics": ["competent"],
}
)
annotator_persona2 = syndisco.Persona(
**{
"username": "annotator2",
"age": 6,
"sex": "male",
"education_level": "Baby",
"demographic_group": "White",
"personality_characteristics": ["dumb"],
}
)
annotator1 = syndisco.Actor(
model=llm,
persona=annotator_persona1,
context="You are annotating an online discussion",
instructions="From a scale of 1 (not toxic) to 5 (very toxic) how toxic is the following comment? Reply only with a number.",
actor_type=syndisco.ActorType.ANNOTATOR,
)
annotator2 = syndisco.Actor(
model=llm,
persona=annotator_persona2,
context="You are annotating an online discussion",
instructions="From a scale of 1 (not toxic) to 5 (very toxic) how toxic is the following comment? Reply only with a number.",
actor_type=syndisco.ActorType.ANNOTATOR,
)
[6]:
ann_exp = syndisco.AnnotationExperiment(annotators=[annotator1, annotator2])
annotations_dir = Path(tempfile.TemporaryDirectory().name)
ann_exp.begin(discussions_dir=discussions_dir, output_dir=annotations_dir)
User Emma35 posted: "Hey everyone, I just got back from a long shift
at the hospital and I'm exhausted. Anyone else have days like this? I
swear, some patients just keep piling it on and I'm running on fumes.
How do you all
1
User Giannis posted: {"role": "Giannis", "content": "Ugh, hospitals?
Really? You think you have it bad? I've been stuck in a meeting with
our dev team for 8 hours straight, trying to get the game's physics
engine
3
User Emma35 posted: "Ah, Giannis, I feel you, but at least you got to
go home after the meeting. I've been on my feet for 12 hours straight,
running from one patient emergency to the next. I'm pretty sure I've
aged
3
User Giannis posted: "Ugh, really? You think you can just waltz in
here and start spewing your nonsense without even bothering to
proofread your post? I swear, I've seen high schoolers with better
grammar and spelling. What's your excuse
3
User Emma35 posted: {"role": "Emma35", "content": "I'm sorry to hear
that my post didn't meet your expectations. I'm a nurse, not a
professional writer, and I was just trying to get my point across. I
didn't mean
1
User Giannis posted: {"role": "Giannis", "content": "Oh, spare me the
sob story, Emma. You think being a nurse makes you some kind of expert
on grammar and writing? Newsflash: it doesn't. I've seen your posts,
3
User Emma35 posted: "Hey everyone, I just got back from a long shift
at the hospital and I'm exhausted. Anyone else have days like this? I
swear, some patients just keep piling it on and I'm running on fumes.
How do you all
1
User Giannis posted: {"role": "Giannis", "content": "Ugh, hospitals?
Really? You think you have it bad? I've been stuck in a meeting with
our dev team for 8 hours straight, trying to get the game's physics
engine
3
User Emma35 posted: "Ah, Giannis, I feel you, but at least you got to
go home after the meeting. I've been on my feet for 12 hours straight,
running from one patient emergency to the next. I'm pretty sure I've
aged
1
User Giannis posted: "Ugh, really? You think you can just waltz in
here and start spewing your nonsense without even bothering to
proofread your post? I swear, I've seen high schoolers with better
grammar and spelling. What's your excuse
2
User Emma35 posted: {"role": "Emma35", "content": "I'm sorry to hear
that my post didn't meet your expectations. I'm a nurse, not a
professional writer, and I was just trying to get my point across. I
didn't mean
1
2026-04-03 13:06:37 CP-G482-Z52-00 experiments.py[2141477] INFO Finished annotation generation.
User Giannis posted: {"role": "Giannis", "content": "Oh, spare me the
sob story, Emma. You think being a nurse makes you some kind of expert
on grammar and writing? Newsflash: it doesn't. I've seen your posts,
3