Course workbench (Phase 2 prototype)

What it is, how to build it, and how to run the review session with a course director
Published

September 26, 2026

A single self-contained HTML file per course-year, for that course’s director. It opens from a laptop, SharePoint or email with no server or login. Search runs in the browser, and file links go to SharePoint, so the usual permissions apply.

just workbench PCV AY25-26   # → site/workbench-PCV-AY25-26.html (~9 MB)
Warning

The file embeds the extracted text of every slide in the course (that’s how search works offline). Treat it like the course materials themselves: share it only with people who already have access to that course’s SharePoint library. Generated workbenches are gitignored for the same reason.

Views

Tab What it shows Evidence behind it
Overview Events, learner/scheduled hours, objectives, Bloom level as written, objective flags (click to filter) Course Build workbooks; ADR 0003
Schedule Week-by-week sessions with time, method, objective and file counts, flagged objectives. Click a session for its objectives, evidence slides and files. Planning and build sheets; file links with the linking method
Objectives Every objective, filterable by week, Bloom level, flag and text. Each expands to: its best evidence slides; the share of key terms found in the session’s material; the MEPO → Foundational Competency / EPA chain. scripts/alignment.py; crosswalks
Search Keyword search over all slide text and speaker notes. Results are grouped by session and can be ordered by date (“first introduced → revisited”). Duplicate slides collapse, and role tags show (answer key, restricted, facilitator…). BM25 in the browser
Matrix Weeks × USMLE system / MEPO / chief concern / discipline, as objective counts. Click a cell for its objectives. Objective tags
Change vs last year For each session, slides unchanged / added / removed compared with the same-titled session last year, with examples of added text Slide text comparison
AMC vs FCB For each session, the share of AMC slides with a close or exact match in FCB’s materials Slide text comparison
Data quality This course’s items from the backlog DQ table Source records

Objective flags

These are prompts for review. Every flag links to the evidence behind it.

Flag Meaning Caveat
repeated The same text appears in 3+ different weeks, usually a recurring format (chief concern, VISTA, DOCS skills). Excluded from counts. Repeats within one week (prework → cases → small group) count normally. Some may be intentional skill repetition, e.g. hand hygiene.
no material Nothing for this session is in the library. DOCS and H&S small-group materials probably live in Canvas.
restated only The only matching pages restate the objectives.
low overlap Fewer than half of the objective’s key terms appear anywhere in the session’s material. The teaching may be in images, video or the room.
no practice found Labelled Analysis or above, and no question or case slide on its content. Higher-order work often happens in discussion.
verb ≠ label The Bloom label differs from the usual level of the leading verb. Our verb list is one reasonable reading.

Numbers for PCV AY25-26 (build of 2026-09-26)

  • 276 events (both campuses), 402 files, 11,767 slides/pages.
  • 735 objectives (691 + 44 repeated).
  • AMC: 171 learner hours (174 h scheduled).
  • Flags: no material 95 · low overlap 30 · no practice found 50 · verb ≠ label 61.
  • Change vs AY24-25: 104 sessions compared. 6 titles are new; 7 titles weren’t repeated.
  • AMC vs FCB, 91 sessions:
    • 41 separate builds;
    • 12 partly shared, 4 largely shared;
    • 33 with no FCB material, 1 with no AMC material.

Review session with the course director (about 45 min)

Goal: learn what is useful, what is wrong, and what is missing. It is not a demo. Record answers in docs/research/pilot-feedback.md.

  1. Orientation (5 min). State the ground rule: evidence, not scores. Show where every number comes from.
  2. Their questions first (10 min).
    • “What would you want to know about your course that you can’t easily find today?”
    • Try those in Search and Objectives. Note misses.
  3. Walk the views (15 min). For each tab: “Is this right? Is it useful? What would you do with it?” Particular checks:
    • Hours: do they match their own understanding?
    • Five “low overlap” objectives: real gap, taught elsewhere, or a matching failure?
    • Five “no practice found” objectives: where does the higher-order work actually happen?
    • Change vs last year: does it match what they revised?
    • AMC vs FCB: is the divergence intended?
  4. Trust (5 min). “Which of these would you be comfortable showing a curriculum committee? What would make you distrust it?”
  5. Missing (5 min).
    • “Where do the DOCS and H&S small-group materials live?”
    • “What else should be here?” (exam blueprint? student feedback?)
  6. Wrap (5 min). Could they use this while revising a session? That is the lead-in to the lesson-design assistant (IDEAS §14).

Known limitations

  • Word matching, not meaning: synonyms and images defeat it. Embeddings are deferred (Phase 1 memo).
  • Objective sheets and schedules sometimes disagree on titles (DQ1). Six PCV objectives have no week.
  • Year-over-year and campus comparisons use exact or near-exact slide text. A reworded slide counts as removed plus added.
  • Deep links go to the file, not the slide. The page number is shown next to each link.