NotebookLM (and other slide-generation tools) are useful, and as this Reddit post argues, quietly dangerous when they produce whole decks. The original poster shares an open-source tool built as a post-generation safety layer: a Citation-Check Skill that extracts claims from slides and verifies them against sources in a deterministic, two-pass workflow. Repo: https://github.com/serenakeyitan/Citation-Check-Skill. Reddit thread: https://reddit.com/r/notebooklm/comments/1qn1d6a/a_citation_checker_skill_because_ai_slides_made/.
What the skill does
- Extracts factual claims (numbers, statistics, explicit assertions) from slides, PDFs, tables, and charts (vision-enabled).
- Runs verification queries against either web search results or only the provided documents (two modes).
- Compares numbers precisely (e.g., 96.555% != 97%) and flags mismatches.
- Uses a two-pass approach: first extract claims, then verify each claim.
- Deterministic behavior: same input leads to the same output, which helps reproducibility and auditability.
Why this matters
Slide decks are compact summaries intended to persuade. A single hallucinated stat can change decisions. According to top comments on the Reddit thread, users have already caught NotebookLM hallucinating numbers in slides, so this addresses a real pain point.
The skill treats slide output as an artifact to be audited, not as an authoritative product. That is a useful mental model for teams using LLMs in business reporting, research briefings, or teaching materials.
Limits, assumptions, and things to watch for
- Source quality and coverage: web-mode verification inherits web-search pitfalls (paywalls, stale content, low-quality sources). Doc-only mode is safer when you control sources, but then the skill can only verify what you provided.
- OCR and vision errors: reading slides, charts, or low-resolution PDFs can introduce extraction errors. Verification is only as good as the extracted claim.
- False negatives and false positives: strict numeric matching (96.555% vs 97%) is good for precision, but may require user-configurable tolerance depending on context.
- Determinism helps audits but will not fix upstream biases or misinterpreted claims. Human review remains necessary for high-stakes use.
Practical deployment suggestions
- Run the Citation-Check as a mandatory post-generation step for any slide deck destined for decision-making presentations.
- Use doc-only mode for internal reports where you supply source materials; use web-mode for literature surveys but treat results as provisional.
- Tune numeric-tolerance settings and surface both the evidence snippet and the canonical URL for each flagged claim.
- Add a human sign-off step for any flagged claim before publishing externally.
My takeaways from this post
- AI slide generation needs deterministic, auditable verification by default, and this skill is a pragmatic first step.
- Vision plus two-pass verification addresses a common failure mode: mixed-source decks where provenance is lost.
- Practical integration examples (for Claude) and an open repo accelerate adoption. Expect forks and feature additions such as provenance ranking and confidence thresholds.
Final note
There is another side to the story: a verification layer reduces risk but does not eliminate the need for source curation and human oversight. Still, for workflows that produce many quick decks, this tool is high leverage: it turns “I hope the numbers are right” into “here is where each claim came from and whether it matches.”
Leave a Reply