Apunim: Attributing polarization to sociodemographic groups

_images/logo.svg

Annotators disagree with each other all the time. The reasons may be:

  • Their understanding of the task and the guidelines

  • Socio-demographic factors

  • Ideology / personal opinions and experiences

  • Different expertise

These disagreements can either occur in the details (e.g., a difference between a 4-star and 5-star rating), or as fundamental disagreements (polarization).

Disregarding polarization in annotations is scientifically and ethically unsound. In tasks such as toxicity/hate speech detection it is outright counterproductive, since disregarding minority opinions makes systems biased and fundamentally misconfigured. However, it is often difficult to understand whether disagreement is caused by the (random) factors mentioned above, by mismatches in minority vs majority group opinions, or by ideology.

The Apunim (Aposteriori Unimodality) tool solves this problem. Using this python library, we can attribute polarization to each individual annotator characteristic.

Installation

This library is available in PyPi:

pip install apunim

For other installation options, consult the project’s Github repository.

Examples and API doc

Intended use and limitations

Intended uses

  • Auditing annotated datasets for group-based disagreement before training or evaluating models on them.

  • Comparing disagreement patterns between human annotators and LLM annotators.

  • Informing dataset design (e.g., how many annotators per item, and which groups need better coverage).

  • Research on annotation disagreement, with results reported together with dataset characteristics (number of items, annotators per item, group sizes).

Not intended for

  • Surveillance or demographic targeting of any population.

  • Adversarial use, such as crafting content to exploit gaps in moderation systems or annotator coverage.

  • Automated decision-making. Results should inform human review and dataset or policy improvements, not replace them.

Interpreting results

  • Apunim needs enough annotations per group on each item (at least three) and at least two such groups per item. Items or groups that do not meet this are excluded, which can leave small or minority groups under-represented in the result. Treat results for sparse groups with caution.

  • Please report apunim results together with the number of items and annotators per item they are based on.