Phase 1 memo: what full-text search can and can’t do for a course workbench

Question-set baseline, objective-support probe, concept-layer scan, and a recommendation for Phase 2
Published

September 26, 2026

Pilot: PCV (Pulmonary & Cardiovascular Systems), AY25-26, both campuses. Retrieval: DuckDB BM25 over slide/page text + speaker notes, objectives and event titles/descriptions, with optional MeSH expansion. No embeddings or LLMs (ADR 0004).

Artifacts: - questions.csv: 32 questions. - fts_baseline_runs.md: every question’s top results. - support_probe.csv: 735 objectives. - support_probe_judged.csv: 50 hand-judged objectives. - Scripts: scripts/search.py, scripts/phase1_baseline.py, scripts/phase1_support.py.

Note

The relevance judgments below were made by the agent (Claude) reading the retrieved text. They need a faculty spot-check before anything here is quoted to a course director.

Bottom line

  1. Keyword search is good enough to anchor a course workbench. Faculty ask about named things (ECG, tuberculosis, Frank-Starling, SGLT2 inhibitors), and the slides and speaker notes use those words. BM25 put the right session in the top 10 for 12 of 15 “where is X taught” questions.
  2. Embeddings are not needed for the Phase 2 workbench. They are a measured “later”. They would help two things we saw:
    • conceptual queries (“antibiotic stewardship”, “social determinants of cardiovascular health”);
    • vocabulary mismatch when matching objectives to slides.
    If we add them, the grain should be the slide (with deck title as context) plus the objective, using a local model, evaluated on this question set.
  3. Naive MeSH query expansion hurts. Use MeSH to tag objectives and slides, not to rewrite queries.
  4. Objective-to-material alignment is feasible as evidence, not as a score. Lexical overlap finds a plausible teaching slide for most objectives. The objectives it can’t support are mostly not retrieval failures (see below).
  5. Several of the most useful workbench questions aren’t search at all. Sessions without material, year-over-year change, AMC vs FCB divergence, and scheduled time are SQL over tables we already have.

1. Question-set baseline

32 questions (in questions.csv) from three audiences, all scoped to AY25-26: - faculty: “where is X taught?”, “what exists before I write a session?”; - course director: coverage, alignment, change, cross-course overlap; - leadership: time, campus differences.

Each question has the query a person would plausibly type and, where one exists, a list of the sessions a good answer must include.

Kind n Outcome
locate (“where is X taught”) 15 Gold session in top 10 for 12. The 3 partial misses are explained below.
trajectory / cross-course 8 Search finds the sessions. Ordering them in time is a display job, done by sorting on date, week and course sequence.
objectives / alignment 2 The objective channel works. Alignment needs the probe in §2.
structured (time, missing material, change, campus) 5 Not search problems. SQL answers them today (§4).
design (“before I write a session on…”) 2 Search plus course-sequence dates gives a usable “what already exists” list.

The recall figures are optimistic: the gold lists were written from session titles, which share words with the queries. The more telling evidence is in what went wrong.

Failure modes seen:

Mode Example What would fix it
Conceptual query, no distinctive words Q15 “social determinants of cardiovascular health” ranks sessions that just say “cardiovascular disease”. The relevant FCB atherosclerosis case (“Health inequities & atherosclerosis”) is on page 64 of a deck. Embeddings at the slide grain, or concept tags
Vocabulary drift “antiarrhythmic” vs the session title “Antidysrhythmics”. The slides say both, so BM25 coped. Usually fine; a small curated synonym list
Near-duplicate flooding Animation-build slides (pp. 13–16 of one deck, all the same text), AMC/FCB twins, with answers / no answers versions Collapse by file and near-duplicate family; show role tags
Objective-only hits “Autonomic Regulation” ranks for antiarrhythmic drugs because one objective mentions them, with no matching slide Show which channel fired (title / objective / material). The ranker already exposes this.
Title slides and quiz stems Q14 “smoking cessation” finds a make-up quiz and an RLC H&S session Role-aware display (assessment, answer_key)
Temporal questions Q32 “what have students learned about acid-base before PCV?”: RU’s acid-base block ranks first but comes after PCV (Jan vs Nov). FP’s Homeostasis session (Aug) is the real answer. Course sequence and dates in the result list. Not a retrieval problem.

MeSH expansion changed no gold recall, lowered it once (Q16), and added noise: - “Frank-Starling” pulled in Starlings, Mynahs, Oxpeckers (birds); - “pulmonary hypertension” pulled in High Blood Pressure; - “stenosis” pulled in Strictures; - generic entries like Therapy, Diseases, Acids crept in.

MeSH lacks common clinical shorthand (no “HFrEF”), though it has ECG/EKG, COPD and SGLT2. Conclusion: drop automatic expansion. A small curated abbreviation list is worth more.

2. Objective-support probe

For each of the 735 PCV AY25-26 objectives, we looked only at the pages of its own session’s files (AMC and FCB). We measured the share of the objective’s content words (stemmed; stopwords, Bloom verbs and generic words removed) that appear on the best page, in the best file, and anywhere in the session.

First finding: slides that restate objectives inflate everything. Many decks open with a learning-objectives slide, and the first run called 614 of 641 objectives “strongly supported”. For 444 of those, the best page was the objectives slide itself. We now exclude pages that list objectives, echo two or more objectives, or quote the objective being checked. That removes 781 of 11,250 pages.

Current distribution (share of content words found):

Grain ≥ 0.6 0.3–0.6 < 0.3 no material
best single page 358 230 53 94
best file 544 73 24 94
whole session 572 57 12 94

The 94 objectives with no material belong to 64 DOCS (clinical skills) events and 28 H&S events. Their materials aren’t in this library, probably in Canvas (NOTES).

Hand check of 50, stratified by best-page band:

Band Supported Partial Not supported Template objective Video-only Restatement slipped through
strong (18) 9 4 3 – – 2
partial (18) 5 10 2 1 – –
weak (14) – 2 7 4 1 –

What this says: - A high score doesn’t prove teaching. The strong band includes word-sense accidents (“biologic therapy” matching an objective about “biologic mechanisms”) and passing mentions (CF in a list of wheeze differentials matched against “formulate a management strategy for CF”). The page-level score is best used to pick which slides to show, not to certify coverage. - Partial is mostly fine. The teaching is there, but in images (mitral-stenosis diagrams, silhouette sign) or spread over several slides. File-level and session-level coverage are the more honest numbers. - Weak is where the review value is. Its causes: - Template objectives: 21 PCV objectives repeat weekly (“Utilize an advance organizer…”, “Apply a framework for identifying the individual, community, and structural factors…”, VISTA “new breakthroughs”). They describe the session format, not content, and should be shown separately. - Content in video or images: the Mtb lifecycle is taught in RWJF videos, and the prework doc only describes them. - Prework events bundle many topics. A pleura objective’s best page is a lipids handout, because the pleura prework isn’t in the text we have. These are real “can’t find it” cases, worth a course director’s look, whether the cause is a missing file, a linking gap or an image-only deck. - Two restatements slipped through. One was on a prework document page. The other quotes the objective with a typo (“Poiseiulle”). Fuzzy matching would catch most of these.

Side finding: explicit alignment tags. FCB’s AY25-26 “CONTENT_GCA” decks tag case and answer slides with objective numbers, e.g. “(LO1.1, P4)”. That’s 554 pages in PCV and about 1,400 across courses. These are author-made slide-to-objective links. They could serve as gold data for checking any alignment method, and as direct evidence in the workbench. They use deck-local numbering, so they’d need mapping to the school’s objectives.

3. Concept layer (scan)

Existing vocabularies give course-level views but can’t answer “where is ECG taught”: - system, chief concern, discipline and MEPO are tagged per objective; - 75% of objectives are MK1.1; - chief concern is set on only 118 of 276 PCV AY25-26 events.

A MeSH dictionary tagger, matching entry-term phrases from the anatomy, disease, drug, phenomena, psychiatry, procedure and health-care trees, tags 634 of 735 objectives (86%), about 2.3 descriptors each. For example: - “cholesterol biosynthesis … HMG-CoA reductase” → Cholesterol, Mevalonic Acid, Hydroxymethylglutaryl CoA Reductases; - “digoxin … heart failure” → Digoxin, Heart Failure.

The noise is predictable: - verbs that are also descriptors (Affect, Accounting from “account for”); - over-broad anatomy (Heart); - misses on education and social concepts (“validity” of diagnostic studies, “LGBTQ+ terminology”).

This is enough for topic trajectories and topic-time estimates once tuned with a stoplist and checked on a sample. No model is needed. Tagging slides the same way would let “time on heart failure” count cases, prework and review, not just the 4.5 h of sessions titled with it.

4. Structured questions (no search needed)

  • Sessions with objectives but no linked material: 19 PCV AMC events, all DOCS/H&S small groups, VISTA, and the week-6 case reviews.
  • Scheduled time on a topic by title: heart failure has 4.5 h of titled sessions, which undercounts (see §3).
  • Year-over-year change: ACS session files per year with near-duplicate families. A slide-level diff (added / removed / changed slides between years) needs no model.
  • AMC vs FCB: the Tuberculosis decks are in different near-duplicate families (AMC 45 slides, FCB 26). Both campuses share the objectives, but the materials are separate builds.

Recommendation for Phase 2 (single-course workbench, PCV)

Build on what is solid now. Label everything as evidence for review.

  1. Course map: weeks × sessions with method, scheduled/learner hours and objective counts. Links go to SharePoint.
  2. Objectives view:
    • Bloom level as written, MEPO and system for each objective.
    • For each objective, the top 3 teaching slides from its session. Skip restatement slides and collapse duplicates.
    • Status flags: no material in library, only restatements found, low overlap (review), template objective, mostly image/video content.
    • No coverage percentage headline.
  3. Course-scoped search:
    • BM25 with duplicate collapsing and role tags;
    • a small curated abbreviation list;
    • shows which channel matched;
    • results ordered by week so “first introduced / revisited” is visible.
  4. Change and campus panels: slide-level diff vs last year, and AMC vs FCB divergence per session.
  5. Data-quality panel: the course’s rows from the backlog’s DQ table (unscheduled objective titles, typo twins, missing methods).

Deferred until faculty have used 1–5: - Embeddings. A local model at the slide grain plus objectives, evaluated on this question set with the Q11/Q15-style conceptual queries added. - MeSH concept tagging of slides, for topic time and trajectories. - Parsing the FCB LO tags as alignment gold. - Any LLM judgment of “does this slide teach this objective” (needs approval).

Open questions carried forward: - which course director will review the workbench; - where DOCS/H&S small-group materials live; - whether the FCB “GCA” content format is used school-wide.