[memo]Generating Plausible Distractors for Multiple-Choice Questions via Student Choice Prediction

• LLMにおける問題
• DPOを使って、生徒が実際に間違うようなdistractorを作成する手法

ソウル国立大学のグループYooseop Lee

Abstract

In education, plausible distractors is crucial for students

Prior studies on distractor generation have not paid attention on enhancing the difficulty.

Introduction

This study presents a model training pipeline for distractor generation.

Trained pairwise ranker to predict the most plausible distractors

Amon CS subjects, the ranker worked well.

Contribution

• Pairwise ranker that reasons through students’ misconceptions
• Student choice dataset with plausibility rankings among distractors
• Applied our method to MCQs in CS subjects

Related works

• Passage-based
• Cloze-style format

Method

• South Korea nationwide dataset
  • The dataset contains selection rate for sutdents
• Pairwise ranker
  • Input: Q, A, D_1, D_2 (D means a distractor)
  • Output: D_1 or D_2
  • Determine which question is difficult
• Student choice dataset
  • GPt-4o is used to generate three new distractors distinct from the human-authored ones
    • These distractors are scored by using pairwise ranker

Conclusion

• Proposed a pipeline for training a model to generate more plausible distractors for MCQs.
• Created chosen-rejected pairs of distractors for DPO.
• Performed effectively to human.

感想

Pairwise rankerなどを使って，うまく学生が間違いやすいdistractorを作ってあげようという研究