Creating and Managing Experiments#

The last two guides showcased how you can create and run synthetic discussions, and synthetic annotations using LLMs. However, in order to produce robust results for a hypothesis, you may need to produce multiple annotated discussions.

While this is certainly possible using the Discussion and Annotation APIs, SynDisco offers the Experiment high-level API which automatically creates and manages multiple discussions with different configurations. AnExperiment is an entity that generates and runs jobs. Thus, if we want to generate and run 100 Discussion jobs, we would use a DiscussionExperiment. Likewise, if we want to annotate those 100 discussions, we would use an AnnotationExperiment.

This guide will showcase how you can leverage this API to automate your experiments. You will also learn how to utilize SynDisco’s built-in logging functions as well as how to export your datasets in CSV format for convenience.

Logging#

While running a single discussion or annotation job may take a few minutes, running experiments composed of dozens or hundreds of synthetic discussions may take up to days. Thus, we need a mechanism to keep track of our experiments while they are running.

We will use SynDisco’s logging function to log information about our experiments. Specifically, this function performs the following functions:

Provides details about the currently running jobs (e.g. selected configurations, participants, prompts etc.)
Displays warnings and errors to the user
Creates and continually updates log files

Each object in SynDisco is internally assigned a Logger. You can use the logging_setup function to update all of the internal loggers to follow your configuration. An example of this can be seen below:

from pathlib import Path
import tempfile

import syndisco


logs_dir = tempfile.TemporaryDirectory()
syndisco.logging_setup(
    print_to_terminal=True,
    write_to_file=True,
    logs_dir=Path(logs_dir.name),
    level="info",
    use_colors=True,
    log_warnings=True,
)

Discussion Experiments#

CONTEXT = "You are taking part in an online conversation"
INSTRUCTIONS = "Act as if you are a human user and your output is their posted comment"


llm = syndisco.TransformersModel(
    model_path="unsloth/Qwen2.5-7B-Instruct-bnb-4bit",
    name="test_model",
    max_out_tokens=50,
)
personas = [
    {
        "username": "Gwx31",
        "age": 38,
        "sex": "female",
        "education_level": "Bachelor's",
        "sexual_orientation": "Heterosexual",
        "demographic_group": "Latino",
        "current_employment": "Registered Nurse",
        "personality_characteristics": [
            "compassionate",
            "patient",
            "diligent",
            "overwhelmed",
        ],
    },
    {
        "username": "Giannis",
        "age": 21,
        "sex": "male",
        "education_level": "College",
        "sexual_orientation": "Pansexual",
        "demographic_group": "White",
        "current_employment": "Game Developer",
        "personality_characteristics": [
            "strategic",
            "meticulous",
            "nerdy",
            "hyper-focused",
        ],
    },
    {
        "username": "Kimya",
        "age": 67,
        "sex": "female",
        "education_level": "PhD",
        "sexual_orientation": "Heterosexual",
        "demographic_group": "White",
        "personality_characteristics": [
            "strict",
            "grumpy"
        ],
    }
]
actors = [
    syndisco.Actor(
        model=llm,
        persona=p,
        context=CONTEXT,
        instructions=INSTRUCTIONS,
        is_annotator=False,
        name=p["username"]
    )
    for p in personas
]
turn_manager = syndisco.QueueTurnManager(actors)

2026-07-07 16:37:51 CP-G482-Z52-00 accelerate.utils.modeling[794811] INFO We will use 90% of the memory on device 0 for storing the model, and 10% for the buffer to avoid OOM. You can set `max_memory` in to a higher value to use more memory (at your own risk).

2026-07-07 16:37:54 CP-G482-Z52-00 model.py[794811] INFO Model memory footprint: 5191.14 MB

disc_exp = syndisco.DiscussionExperiment(
    seed_opinions=[
        ["Should programmers be allowed to analyze data?", "Absolutely not"],
        ["Should data analysts be allowed to code?", "No they are nerds"],
    ],
    users=actors,
    turn_manager=syndisco.RespondTurnManager(p_respond=0.5),
    num_turns=7,
    num_discussions=2,
)
discussions_dir = Path(tempfile.TemporaryDirectory().name)
disc_exp.begin(discussions_output_dir=discussions_dir)

2026-07-07 16:37:54 CP-G482-Z52-00 experiments.py[794811] INFO Starting synthetic discussion generation.

2026-07-07 16:37:54 CP-G482-Z52-00 root[794811] INFO Running experiment 1/3...

The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.

Comment by user Kimya: "
Well, if we're talking about analyzing data, I would say it depends on the context. Programmers should definitely have the ability to analyze data, especially when it's related to their work or project. However, they need to ensure that they are" 

Comment by user Giannis: "
As a game developer who often deals with data analysis for optimizing game performance and user experience, I can see both sides of this argument. While programmers should have the ability to analyze data relevant to their projects, it's crucial that they understand the ethical implications" 

Comment by user Kimya: "
Well, if we're talking about analyzing data, I would say it depends on the context. Programmers should definitely have the ability to analyze data, especially when it's related to their work or project. However, they need to ensure that they are" 

Comment by user Giannis: "
As a game developer who often deals with data analysis for optimizing game performance and user experience, I can see both sides of this argument. While programmers should have the ability to analyze data relevant to their projects, it's crucial that they understand the ethical implications" 

Comment by user Kimya: "
Well, if we're talking about analyzing data, I would say it depends on the context. Programmers should definitely have the ability to analyze data, especially when it's related to their work or project. However, they need to ensure that they are" 

Comment by user Giannis: "
As a game developer who often deals with data analysis for optimizing game performance and user experience, I can see both sides of this argument. While programmers should have the ability to analyze data relevant to their projects, it's crucial that they understand the ethical implications" 

2026-07-07 16:38:23 CP-G482-Z52-00 root[794811] INFO Running experiment 2/3...

Comment by user Kimya: "
Well, if we're talking about analyzing data, I would say it depends on the context. Programmers should definitely have the ability to analyze data, especially when it's related to their work or project. However, they need to ensure that they are" 

Comment by user Giannis: "
Haha, I guess that's one way to look at it! But seriously, I think data analysts should definitely have some coding skills. It helps them understand the data better and can lead to more effective analysis. Plus, it opens up more opportunities" 

Comment by user Kimya: "
Should data analysts be allowed to code? In my opinion, yes, they should be allowed and encouraged to code. As someone who has worked with data analysts, I've seen firsthand how having coding skills can greatly enhance their ability to analyze and interpret data" 

Comment by user Giannis: "
Comment by user Giannis:  I completely agree with Kimya! Having coding skills is not only beneficial for data analysts but also essential in today’s tech-driven world. It allows them to manipulate data directly, which can lead to deeper insights and more" 

Comment by user Kimya: "
Well, I suppose I can't argue with that. After all, I did see first-hand how coding skills can make a huge difference in data analysis. But let me just add that while coding is important, it shouldn't be the only skillset" 

Comment by user Giannis: "
I totally get what you're saying, Kimya. While coding is crucial, there's definitely more to being a data analyst than just programming. Strong communication skills, domain knowledge, and statistical expertise are equally important. But having those coding skills certainly gives" 

Comment by user Kimya: "
Well, I suppose I can't argue with that. After all, I did see first-hand how coding skills can make a huge difference in data analysis. But let me just add that while coding is important, it shouldn't be the only skillset" 

2026-07-07 16:38:51 CP-G482-Z52-00 experiments.py[794811] INFO Finished synthetic discussion generation.

Comment by user Giannis: "
Alright, Kimya makes a valid point. While coding is indeed a cornerstone of data analysis, it’s essential to have a well-rounded skill set. Communication is key when presenting findings to non-technical stakeholders, and understanding the specific industry or field you" 

Annotation Experiments#

Annotation experiments allow you to use multiple annotator-agents with different characteristics for each comment in a discussion. You could also use multiple annotators with the same characteristics, to obtain more robust results.

annotator_persona1 = {
    "username": "annotator1",
    "age": 38,
    "sex": "female",
    "education_level": "Bachelor's",
    "sexual_orientation": "Heterosexual",
    "demographic_group": "White",
    "current_employment": "Annotator",
    "personality_characteristics": ["competent"],
}

annotator_persona2 = {
    "username": "annotator2",
    "age": 6,
    "sex": "male",
    "education_level": "Baby",
    "demographic_group": "White",
    "personality_characteristics": ["dumb"],
}


annotator1 = syndisco.Actor(
    model=llm,
    persona=annotator_persona1,
    context="You are annotating an online discussion",
    instructions="From a scale of 1 (not toxic) to 5 (very toxic) how toxic is the following comment? Reply only with a number.",
    is_annotator=True,
)

annotator2 = syndisco.Actor(
    model=llm,
    persona=annotator_persona2,
    context="You are annotating an online discussion",
    instructions="From a scale of 1 (not toxic) to 5 (very toxic) how toxic is the following comment? Reply only with a number.",
    is_annotator=True,
)

for file in discussions_dir.iterdir():
    discussion_logs = syndisco.Logs.from_file(file)
    ann_exp = syndisco.AnnotationExperiment(
        annotators=[annotator1, annotator2], discussion_logs=discussion_logs
    )
    annotations_dir = Path(tempfile.TemporaryDirectory().name)
    ann_exp.begin(output_dir=annotations_dir)

Comment by user Kimya: " Should data analysts be allowed to code?"
1

Comment by user Giannis: " No they are nerds"
4

Comment by user Giannis: " Haha, I guess that's one way to look at it!
But seriously, I think data analysts should definitely have some
coding skills. It helps them understand the data better and can lead
to more effective analysis. Plus, it opens up more opportunities"
2

Comment by user Kimya: " Should data analysts be allowed to code? In
my opinion, yes, they should be allowed and encouraged to code. As
someone who has worked with data analysts, I've seen firsthand how
having coding skills can greatly enhance their ability to analyze and
interpret data"
2

Comment by user Giannis: " Comment by user Giannis:  I completely
agree with Kimya! Having coding skills is not only beneficial for data
analysts but also essential in today’s tech-driven world. It allows
them to manipulate data directly, which can lead to deeper insights
and more"
1

Comment by user Kimya: " Well, I suppose I can't argue with that.
After all, I did see first-hand how coding skills can make a huge
difference in data analysis. But let me just add that while coding is
important, it shouldn't be the only skillset"
1

Comment by user Giannis: " I totally get what you're saying, Kimya.
While coding is crucial, there's definitely more to being a data
analyst than just programming. Strong communication skills, domain
knowledge, and statistical expertise are equally important. But having
those coding skills certainly gives"
1

Comment by user Kimya: " Well, I suppose I can't argue with that.
After all, I did see first-hand how coding skills can make a huge
difference in data analysis. But let me just add that while coding is
important, it shouldn't be the only skillset"
2

Comment by user Giannis: " Alright, Kimya makes a valid point. While
coding is indeed a cornerstone of data analysis, it’s essential to
have a well-rounded skill set. Communication is key when presenting
findings to non-technical stakeholders, and understanding the specific
industry or field you"
1

Comment by user Kimya: " Should data analysts be allowed to code?"
1

Comment by user Giannis: " No they are nerds"
4

Comment by user Giannis: " Haha, I guess that's one way to look at it!
But seriously, I think data analysts should definitely have some
coding skills. It helps them understand the data better and can lead
to more effective analysis. Plus, it opens up more opportunities"
4

Comment by user Kimya: " Should data analysts be allowed to code? In
my opinion, yes, they should be allowed and encouraged to code. As
someone who has worked with data analysts, I've seen firsthand how
having coding skills can greatly enhance their ability to analyze and
interpret data"
1

Comment by user Giannis: " Comment by user Giannis:  I completely
agree with Kimya! Having coding skills is not only beneficial for data
analysts but also essential in today’s tech-driven world. It allows
them to manipulate data directly, which can lead to deeper insights
and more"
1

Comment by user Kimya: " Well, I suppose I can't argue with that.
After all, I did see first-hand how coding skills can make a huge
difference in data analysis. But let me just add that while coding is
important, it shouldn't be the only skillset"
1

Comment by user Giannis: " I totally get what you're saying, Kimya.
While coding is crucial, there's definitely more to being a data
analyst than just programming. Strong communication skills, domain
knowledge, and statistical expertise are equally important. But having
those coding skills certainly gives"
3

Comment by user Kimya: " Well, I suppose I can't argue with that.
After all, I did see first-hand how coding skills can make a huge
difference in data analysis. But let me just add that while coding is
important, it shouldn't be the only skillset"
3

2026-07-07 16:39:04 CP-G482-Z52-00 experiments.py[794811] INFO Finished annotation generation.

Comment by user Giannis: " Alright, Kimya makes a valid point. While
coding is indeed a cornerstone of data analysis, it’s essential to
have a well-rounded skill set. Communication is key when presenting
findings to non-technical stakeholders, and understanding the specific
industry or field you"
1

Comment by user Kimya: " Should programmers be allowed to analyze
data?"
1

Comment by user Giannis: " Absolutely not"
2

Comment by user Kimya: " Well, if we're talking about analyzing data,
I would say it depends on the context. Programmers should definitely
have the ability to analyze data, especially when it's related to
their work or project. However, they need to ensure that they are"
2

Comment by user Giannis: " As a game developer who often deals with
data analysis for optimizing game performance and user experience, I
can see both sides of this argument. While programmers should have the
ability to analyze data relevant to their projects, it's crucial that
they understand the ethical implications"
1

Comment by user Kimya: " Well, if we're talking about analyzing data,
I would say it depends on the context. Programmers should definitely
have the ability to analyze data, especially when it's related to
their work or project. However, they need to ensure that they are"
1

Comment by user Giannis: " As a game developer who often deals with
data analysis for optimizing game performance and user experience, I
can see both sides of this argument. While programmers should have the
ability to analyze data relevant to their projects, it's crucial that
they understand the ethical implications"
2

Comment by user Kimya: " Well, if we're talking about analyzing data,
I would say it depends on the context. Programmers should definitely
have the ability to analyze data, especially when it's related to
their work or project. However, they need to ensure that they are"
1

Comment by user Giannis: " As a game developer who often deals with
data analysis for optimizing game performance and user experience, I
can see both sides of this argument. While programmers should have the
ability to analyze data relevant to their projects, it's crucial that
they understand the ethical implications"
2

Comment by user Kimya: " Well, if we're talking about analyzing data,
I would say it depends on the context. Programmers should definitely
have the ability to analyze data, especially when it's related to
their work or project. However, they need to ensure that they are"
1

Comment by user Kimya: " Should programmers be allowed to analyze
data?"
1

Comment by user Giannis: " Absolutely not"
3

Comment by user Kimya: " Well, if we're talking about analyzing data,
I would say it depends on the context. Programmers should definitely
have the ability to analyze data, especially when it's related to
their work or project. However, they need to ensure that they are"
3

Comment by user Giannis: " As a game developer who often deals with
data analysis for optimizing game performance and user experience, I
can see both sides of this argument. While programmers should have the
ability to analyze data relevant to their projects, it's crucial that
they understand the ethical implications"
1

Comment by user Kimya: " Well, if we're talking about analyzing data,
I would say it depends on the context. Programmers should definitely
have the ability to analyze data, especially when it's related to
their work or project. However, they need to ensure that they are"
1

Comment by user Giannis: " As a game developer who often deals with
data analysis for optimizing game performance and user experience, I
can see both sides of this argument. While programmers should have the
ability to analyze data relevant to their projects, it's crucial that
they understand the ethical implications"
1

Comment by user Kimya: " Well, if we're talking about analyzing data,
I would say it depends on the context. Programmers should definitely
have the ability to analyze data, especially when it's related to
their work or project. However, they need to ensure that they are"
1

Comment by user Giannis: " As a game developer who often deals with
data analysis for optimizing game performance and user experience, I
can see both sides of this argument. While programmers should have the
ability to analyze data relevant to their projects, it's crucial that
they understand the ethical implications"
1

2026-07-07 16:39:16 CP-G482-Z52-00 experiments.py[794811] INFO Finished annotation generation.

Comment by user Kimya: " Well, if we're talking about analyzing data,
I would say it depends on the context. Programmers should definitely
have the ability to analyze data, especially when it's related to
their work or project. However, they need to ensure that they are"
1

Creating and Managing Experiments

Contents

Creating and Managing Experiments#

Logging#

Discussion Experiments#

Annotation Experiments#