Towards Enhancing Coherence in Extractive Summarization: Dataset and Experiments with LLMs

This is a Plain English Papers summary of a research paper called Towards Enhancing Coherence in Extractive Summarization: Dataset and Experiments with LLMs. If you like these kinds of analysis, you should subscribe to the AImodels.fyi newsletter or follow me on Twitter.

Overview

This paper introduces a new dataset and experiments to enhance the coherence of extractive summarization using large language models (LLMs). Extractive summarization is the process of selecting and combining the most important sentences from a document to create a concise summary. The authors argue that current extractive summarization models often produce incoherent summaries, and they aim to address this issue.

Plain English Explanation

The researchers created a new dataset to train and evaluate extractive summarization models that produce more coherent summaries. Coherence refers to how well the sentences in a summary flow together and make sense as a whole.

The key innovations in this work are:

1. They developed a new dataset called LAMSUM that contains human-written summaries annotated for coherence. This allows them to train and evaluate models on coherence.

2. They conducted experiments using different LLM-based approaches to enhance the coherence of extractive summaries, including fine-tuning models on the LAMSUM dataset.

The goal is to create extractive summarization systems that generate summaries that are not just informative, but also flow logically and read naturally. This could make the summaries more useful for tasks like academic research, news reporting, and other applications where coherence is important.

Technical Explanation

The authors first constructed the LAMSUM dataset, which consists of article passages and corresponding human-written summaries. The summaries were annotated for coherence using crowdsourcing. This allows the researchers to train and evaluate models on their ability to produce coherent extractive summaries.

They then experimented with different LLM-based approaches to improve coherence in extractive summarization:

1. Fine-tuning large language models like BERT, RoBERTa, and GPT-2 on the LAMSUM dataset to directly optimize for coherence.

2. Using LLMs to rerank candidate extractive summaries based on coherence scores.

3. Incorporating coherence scoring as an additional objective when training extractive summarization models.

The results show that the LLM-based approaches can indeed enhance the coherence of extractive summaries compared to previous methods. The authors provide detailed analysis and ablation studies to understand the key factors driving these improvements.

Critical Analysis

The paper makes a valuable contribution by introducing the LAMSUM dataset and demonstrating effective ways to leverage LLMs to improve coherence in extractive summarization. However, the authors acknowledge some limitations:

• The LAMSUM dataset is relatively small compared to other summarization benchmarks, so the models may not generalize as well to larger-scale real-world applications.

• The coherence annotations in LAMSUM are subjective and may not fully capture all aspects of summary quality.

• The proposed methods still have room for improvement, as even the best-performing models do not achieve human-level coherence scores on the dataset.

It would also be interesting to see how these coherence-enhanced extractive summaries compare to abstractive summarization approaches in terms of both coherence and overall quality. Further research could explore using the LAMSUM dataset for training more advanced summarization models or combining extractive and abstractive techniques.

Conclusion

This paper presents an important step towards improving the coherence of extractive summarization using large language models. The new LAMSUM dataset and LLM-based methods offer a promising direction for developing more natural and useful text summarization systems. While there are still some limitations to address, this work demonstrates the value of focusing on summary coherence and provides a strong foundation for future research in this area.

If you enjoyed this summary, consider subscribing to the AImodels.fyi newsletter or following me on Twitter for more AI and machine learning content.