Domain-Adaptive Transformer Models for Enhanced Abstractive Text Summarization

Abstract

Abstractive text summarization contributes to natural language processing (NLP) by reducing and abstracting long documents. Even though there is great improvement in transformer based pre-trained models, their performance now varies from domain to domain and requires specific fine-tuning. The aim of the research is to test the performance of pretrained models for abstractive dialogue and legal document summarization. The goal is to contrast the performance of three fine-tuned state-of-the-art models—BART, Pegasus and Falconsai (T5-small) on SAMSum (dialogue summarization) and BillSum (Legal document summarization). Pegasus achieves the highest performance on the SAMSum dialogue dataset (ROUGE-1: 42.76), while BART leads on the BillSum legal dataset (ROUGE-1: 43.26). On the SAMSum dialogue dataset, Falconsai struck a reasonable compromise with the ROUGE-1 F1 score 39.4 while Pegasus committed to ROUGE-2 best signifying its strong performance in conversation context. The fine-tuned models improve the quality of summarization beyond previous benchmarks, while a length-tracking mechanism and an efficient beam-search make inference faster. The paper contributes systematic model selection recommendations focusing on Pegasus for dialogue, BART for legal and Falconsai where resources are limited. We discovered persistent problems such as redundancy, hallucination, and factual inconsistency, suggesting hybrid architectures and domain specific pre-training for future work.