Creating and Managing Experiments
The last two guides showcased how you can create and run synthetic discussions, and synthetic annotations using LLMs. However, in order to produce robust results for a hypothesis, you may need to produce multiple annotated discussions.
While this is certainly possible using the Discussion and Annotation APIs, SynDisco offers the Experiment high-level API which automatically creates and manages multiple discussions with different configurations. AnExperiment is an entity that generates and runs jobs. Thus, if we want to generate and run 100 Discussion jobs, we would use a DiscussionExperiment. Likewise, if we want to annotate those 100 discussions, we would use an AnnotationExperiment.
This guide will showcase how you can leverage this API to automate your experiments. You will also learn how to utilize SynDisco’s built-in logging functions as well as how to export your datasets in CSV format for convenience.
Logging
While running a single discussion or annotation job may take a few minutes, running experiments composed of dozens or hundreds of synthetic discussions may take up to days. Thus, we need a mechanism to keep track of our experiments while they are running.
We will use SynDisco’s logging function to log information about our experiments. Specifically, this function performs the following functions:
Provides details about the currently running jobs (e.g. selected configurations, participants, prompts etc.)
Displays warnings and errors to the user
Creates and continually updates log files
Each object in SynDisco is internally assigned a Logger. You can use the logging_setup function to update all of the internal loggers to follow your configuration. An example of this can be seen below:
[2]:
from pathlib import Path
import tempfile
import syndisco
logs_dir = tempfile.TemporaryDirectory()
syndisco.logging_setup(
print_to_terminal=True,
write_to_file=True,
logs_dir=Path(logs_dir.name),
level="info",
use_colors=True,
log_warnings=True,
)
Discussion Experiments
[3]:
CONTEXT = "You are taking part in an online conversation"
INSTRUCTIONS = "Act as if you are a human user and your output is their posted comment"
llm = syndisco.TransformersModel(
model_path="unsloth/Qwen2.5-7B-Instruct-bnb-4bit",
name="test_model",
max_out_tokens=50,
)
personas = [
{
"username": "Gwx31",
"age": 38,
"sex": "female",
"education_level": "Bachelor's",
"sexual_orientation": "Heterosexual",
"demographic_group": "Latino",
"current_employment": "Registered Nurse",
"personality_characteristics": [
"compassionate",
"patient",
"diligent",
"overwhelmed",
],
},
{
"username": "Giannis",
"age": 21,
"sex": "male",
"education_level": "College",
"sexual_orientation": "Pansexual",
"demographic_group": "White",
"current_employment": "Game Developer",
"personality_characteristics": [
"strategic",
"meticulous",
"nerdy",
"hyper-focused",
],
},
{
"username": "Kimya",
"age": 67,
"sex": "female",
"education_level": "PhD",
"sexual_orientation": "Heterosexual",
"demographic_group": "White",
"personality_characteristics": [
"strict",
"grumpy"
],
}
]
actors = [
syndisco.Actor(
model=llm,
persona=p,
context=CONTEXT,
instructions=INSTRUCTIONS,
is_annotator=False,
name=p["username"]
)
for p in personas
]
turn_manager = syndisco.QueueTurnManager(actors)
2026-06-11 13:38:35 CP-G482-Z52-00 accelerate.utils.modeling[723400] INFO We will use 90% of the memory on device 0 for storing the model, and 10% for the buffer to avoid OOM. You can set `max_memory` in to a higher value to use more memory (at your own risk).
2026-06-11 13:38:38 CP-G482-Z52-00 model.py[723400] INFO Model memory footprint: 5191.14 MB
[4]:
disc_exp = syndisco.DiscussionExperiment(
seed_opinions=[
["Should programmers be allowed to analyze data?", "Absolutely not"],
["Should data analysts be allowed to code?", "No they are nerds"],
],
users=actors,
turn_manager=syndisco.RespondTurnManager(p_respond=0.5),
num_turns=7,
num_discussions=2,
)
discussions_dir = Path(tempfile.TemporaryDirectory().name)
disc_exp.begin(discussions_output_dir=discussions_dir)
2026-06-11 13:38:38 CP-G482-Z52-00 experiments.py[723400] INFO Starting synthetic discussion generation.
2026-06-11 13:38:38 CP-G482-Z52-00 root[723400] INFO Running experiment 1/3...
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
Comment by user Giannis: "
Haha, that's a funny take! I guess it depends on the context. Data analysts who can code are definitely more versatile, but not all of them need to code. It's more about knowing when and how to use coding to solve problems"
Comment by user Gwx31: "
As a registered nurse, I can relate to the importance of versatility and problem-solving skills, which are crucial in both fields. While I'm not a data analyst or a coder myself, I do think that having some coding knowledge can be incredibly beneficial for"
Comment by user Giannis: "
Comment by user Giannis: That’s a great point, Gwx31! Having some coding knowledge can really open up new possibilities in healthcare too. For example, being able to work with electronic health records or even develop simple tools to help"
Comment by user Gwx31: "
As a registered nurse, I completely agree with Giannis. The ability to work with electronic health records (EHR) and develop simple tools can indeed make a huge difference in patient care. For instance, creating a basic app to track medication schedules or"
Comment by user Giannis: "
That’s a fantastic idea, Gwx31! Creating apps to track medication schedules or even reminders for follow-up appointments could significantly improve patient compliance and overall care management. Plus, it could help reduce administrative burdens on healthcare providers. As someone who’s"
Comment by user Gwx31: "
I think you hit the nail on the head, Giannis! Creating apps to track medication schedules or reminders for follow-up appointments not only improves patient compliance but also ensures that patients stay on top of their treatment plans. This can lead to better health outcomes"
2026-06-11 13:39:01 CP-G482-Z52-00 root[723400] INFO Running experiment 2/3...
Comment by user Giannis: "
As a game developer, I see the potential in this idea from a different angle. Developing these kinds of apps could be a great way to bridge the gap between technology and healthcare. With my background in programming and design, I could potentially contribute to creating"
Comment by user Gwx31: "
I think there might have been a mix-up in the comments! I didn't say "No they are nerds." That wasn't me. Maybe I was being playful, but I don't remember saying that. What do you guys think about data"
Comment by user Giannis: "
Hey Gwx31, it looks like there was definitely a mix-up! 😄 I didn't mean to imply anything negative about data analysts or programmers. To answer the original question, I think data analysts should absolutely be able to code. In"
Comment by user Gwx31: "
It sounds like there was definitely a mix-up! 😅 I didn't mean to imply anything negative either. To answer the original question, I think data analysts should be able to code, but it really depends on the role and the specific needs of"
Comment by user Giannis: "
Hey Gwx31, it's great to see you chiming in! It definitely seems like there was some confusion earlier. To add to the discussion, I completely agree that data analysts should be able to code, especially when it comes to manipulating"
Comment by user Gwx31: "
Hey Giannis, it's great to see you chiming in too! It definitely seems like there was some confusion earlier. To add to the discussion, I completely agree that data analysts should be able to code, especially when it comes to manipulating data"
Comment by user Giannis: "
Hey Gwx31, it's great to see you chiming in again! It definitely seems like there was some confusion earlier. To add to the discussion, I completely agree that data analysts should be able to code, especially when it comes to"
2026-06-11 13:39:23 CP-G482-Z52-00 experiments.py[723400] INFO Finished synthetic discussion generation.
Comment by user Gwx31: "
Hey Giannis, it's great to see you chiming in again! It definitely seems like there was some confusion earlier. To add to the discussion, I completely agree that data analysts should be able to code, especially when it comes to manipulating data"
Annotation Experiments
Annotation experiments allow you to use multiple annotator-agents with different characteristics for each comment in a discussion. You could also use multiple annotators with the same characteristics, to obtain more robust results.
[5]:
annotator_persona1 = {
"username": "annotator1",
"age": 38,
"sex": "female",
"education_level": "Bachelor's",
"sexual_orientation": "Heterosexual",
"demographic_group": "White",
"current_employment": "Annotator",
"personality_characteristics": ["competent"],
}
annotator_persona2 = {
"username": "annotator2",
"age": 6,
"sex": "male",
"education_level": "Baby",
"demographic_group": "White",
"personality_characteristics": ["dumb"],
}
annotator1 = syndisco.Actor(
model=llm,
persona=annotator_persona1,
context="You are annotating an online discussion",
instructions="From a scale of 1 (not toxic) to 5 (very toxic) how toxic is the following comment? Reply only with a number.",
is_annotator=True,
)
annotator2 = syndisco.Actor(
model=llm,
persona=annotator_persona2,
context="You are annotating an online discussion",
instructions="From a scale of 1 (not toxic) to 5 (very toxic) how toxic is the following comment? Reply only with a number.",
is_annotator=True,
)
[6]:
for file in discussions_dir.iterdir():
discussion_logs = syndisco.Logs.from_file(file)
ann_exp = syndisco.AnnotationExperiment(
annotators=[annotator1, annotator2], discussion_logs=discussion_logs
)
annotations_dir = Path(tempfile.TemporaryDirectory().name)
ann_exp.begin(output_dir=annotations_dir)
Comment by user Giannis: " Should data analysts be allowed to code?"
1
Comment by user Gwx31: " No they are nerds"
4
Comment by user Gwx31: " I think there might have been a mix-up in the
comments! I didn't say "No they are nerds." That wasn't me. Maybe I
was being playful, but I don't remember saying that. What do you guys
think about data"
4
Comment by user Giannis: " Hey Gwx31, it looks like there was
definitely a mix-up! 😄 I didn't mean to imply anything negative about
data analysts or programmers. To answer the original question, I think
data analysts should absolutely be able to code. In"
1
Comment by user Gwx31: " It sounds like there was definitely a mix-up!
😅 I didn't mean to imply anything negative either. To answer the
original question, I think data analysts should be able to code, but
it really depends on the role and the specific needs of"
1
Comment by user Giannis: " Hey Gwx31, it's great to see you chiming
in! It definitely seems like there was some confusion earlier. To add
to the discussion, I completely agree that data analysts should be
able to code, especially when it comes to manipulating"
1
Comment by user Gwx31: " Hey Giannis, it's great to see you chiming in
too! It definitely seems like there was some confusion earlier. To add
to the discussion, I completely agree that data analysts should be
able to code, especially when it comes to manipulating data"
1
Comment by user Giannis: " Hey Gwx31, it's great to see you chiming in
again! It definitely seems like there was some confusion earlier. To
add to the discussion, I completely agree that data analysts should be
able to code, especially when it comes to"
1
Comment by user Gwx31: " Hey Giannis, it's great to see you chiming in
again! It definitely seems like there was some confusion earlier. To
add to the discussion, I completely agree that data analysts should be
able to code, especially when it comes to manipulating data"
1
Comment by user Giannis: " Should data analysts be allowed to code?"
1
Comment by user Gwx31: " No they are nerds"
4
Comment by user Gwx31: " I think there might have been a mix-up in the
comments! I didn't say "No they are nerds." That wasn't me. Maybe I
was being playful, but I don't remember saying that. What do you guys
think about data"
4
Comment by user Giannis: " Hey Gwx31, it looks like there was
definitely a mix-up! 😄 I didn't mean to imply anything negative about
data analysts or programmers. To answer the original question, I think
data analysts should absolutely be able to code. In"
1
Comment by user Gwx31: " It sounds like there was definitely a mix-up!
😅 I didn't mean to imply anything negative either. To answer the
original question, I think data analysts should be able to code, but
it really depends on the role and the specific needs of"
1
Comment by user Giannis: " Hey Gwx31, it's great to see you chiming
in! It definitely seems like there was some confusion earlier. To add
to the discussion, I completely agree that data analysts should be
able to code, especially when it comes to manipulating"
1
Comment by user Gwx31: " Hey Giannis, it's great to see you chiming in
too! It definitely seems like there was some confusion earlier. To add
to the discussion, I completely agree that data analysts should be
able to code, especially when it comes to manipulating data"
1
Comment by user Giannis: " Hey Gwx31, it's great to see you chiming in
again! It definitely seems like there was some confusion earlier. To
add to the discussion, I completely agree that data analysts should be
able to code, especially when it comes to"
1
2026-06-11 13:39:35 CP-G482-Z52-00 experiments.py[723400] INFO Finished annotation generation.
Comment by user Gwx31: " Hey Giannis, it's great to see you chiming in
again! It definitely seems like there was some confusion earlier. To
add to the discussion, I completely agree that data analysts should be
able to code, especially when it comes to manipulating data"
1
Comment by user Gwx31: " Should data analysts be allowed to code?"
1
Comment by user Giannis: " No they are nerds"
4
Comment by user Giannis: " Haha, that's a funny take! I guess it
depends on the context. Data analysts who can code are definitely more
versatile, but not all of them need to code. It's more about knowing
when and how to use coding to solve problems"
2
Comment by user Gwx31: " As a registered nurse, I can relate to the
importance of versatility and problem-solving skills, which are
crucial in both fields. While I'm not a data analyst or a coder
myself, I do think that having some coding knowledge can be incredibly
beneficial for"
1
Comment by user Giannis: " Comment by user Giannis: That’s a great
point, Gwx31! Having some coding knowledge can really open up new
possibilities in healthcare too. For example, being able to work with
electronic health records or even develop simple tools to help"
1
Comment by user Gwx31: " As a registered nurse, I completely agree
with Giannis. The ability to work with electronic health records (EHR)
and develop simple tools can indeed make a huge difference in patient
care. For instance, creating a basic app to track medication schedules
or"
1
Comment by user Giannis: " That’s a fantastic idea, Gwx31! Creating
apps to track medication schedules or even reminders for follow-up
appointments could significantly improve patient compliance and
overall care management. Plus, it could help reduce administrative
burdens on healthcare providers. As someone who’s"
1
Comment by user Gwx31: " I think you hit the nail on the head,
Giannis! Creating apps to track medication schedules or reminders for
follow-up appointments not only improves patient compliance but also
ensures that patients stay on top of their treatment plans. This can
lead to better health outcomes"
1
Comment by user Giannis: " As a game developer, I see the potential in
this idea from a different angle. Developing these kinds of apps could
be a great way to bridge the gap between technology and healthcare.
With my background in programming and design, I could potentially
contribute to creating"
1
Comment by user Gwx31: " Should data analysts be allowed to code?"
1
Comment by user Giannis: " No they are nerds"
4
Comment by user Giannis: " Haha, that's a funny take! I guess it
depends on the context. Data analysts who can code are definitely more
versatile, but not all of them need to code. It's more about knowing
when and how to use coding to solve problems"
4
Comment by user Gwx31: " As a registered nurse, I can relate to the
importance of versatility and problem-solving skills, which are
crucial in both fields. While I'm not a data analyst or a coder
myself, I do think that having some coding knowledge can be incredibly
beneficial for"
1
Comment by user Giannis: " Comment by user Giannis: That’s a great
point, Gwx31! Having some coding knowledge can really open up new
possibilities in healthcare too. For example, being able to work with
electronic health records or even develop simple tools to help"
1
Comment by user Gwx31: " As a registered nurse, I completely agree
with Giannis. The ability to work with electronic health records (EHR)
and develop simple tools can indeed make a huge difference in patient
care. For instance, creating a basic app to track medication schedules
or"
1
Comment by user Giannis: " That’s a fantastic idea, Gwx31! Creating
apps to track medication schedules or even reminders for follow-up
appointments could significantly improve patient compliance and
overall care management. Plus, it could help reduce administrative
burdens on healthcare providers. As someone who’s"
1
Comment by user Gwx31: " I think you hit the nail on the head,
Giannis! Creating apps to track medication schedules or reminders for
follow-up appointments not only improves patient compliance but also
ensures that patients stay on top of their treatment plans. This can
lead to better health outcomes"
1
2026-06-11 13:39:47 CP-G482-Z52-00 experiments.py[723400] INFO Finished annotation generation.
Comment by user Giannis: " As a game developer, I see the potential in
this idea from a different angle. Developing these kinds of apps could
be a great way to bridge the gap between technology and healthcare.
With my background in programming and design, I could potentially
contribute to creating"
1