Large Language Models are great at generating fluent summaries — but sometimes they confidently say things that aren’t fully true. This project explores a simple way to detect hallucinations in abstractive summaries generated by Large Language Models.
Instead of relying only on lexical metrics like ROUGE, the idea is to check factual consistency using question–answer matching.
I. How It Works ??
-
Generate summaries using a pretrained LLM
-
Automatically create factual questions from the original document
-
Answer those questions using:
a. the original document b. the generated summary
-
Compare the answers to measure QA consistency
Low consistency → likely hallucination or information omission.
II. Dataset
CNN / DailyMail (v3.0.0)
III. Models & Tools
Summarization: facebook/bart-large-cnn
Question Answering: deepset/roberta-base-squad2
Python, PyTorch, HuggingFace, NLTK, Matplotlib
IV. Results
-> Many summaries appear fluent but fail factual QA checks -> QA consistency reveals hallucinations missed by surface-level metrics -> Figure 1 shows that QA consistency scores are widely distributed, with many summaries exhibiting low factual consistency. -> Figure 2 indicates that a substantial portion of summaries are classified as hallucinated, despite appearing fluent. -> This confirms that QA-based evaluation captures factual errors missed by surface-level metrics.