Phase 1b memo: can we trust the metadata?
These are sanity checks, not a study. The hand judgments were made by the agent (Claude) from extracted text and need a faculty spot-check.
Artifacts: - bloom_verb_mismatches.csv, bloom_material_signals.csv, bloom_judged.csv (36 PCV objectives); - link_check.csv (50 links); - script scripts/phase1b_bloom.py.
Bottom line
- Bloom labels are consistent with how objectives are worded. 87% agree with the leading verb. They are much weaker evidence of what the material asks students to do, especially at the top of the taxonomy. Show them in the workbench as “Bloom level as written”, with the objective’s verb alongside.
- Higher-order labels are the ones to question. In the PCV sample, only 2 of 18 objectives labelled Analysis, Evaluating or Creating had material visibly asking for that level. For lower levels it was 13 of 18.
- Template objectives distort the counts. Of the 107 AY25-26 objectives labelled Evaluating, 28% (30) are the same weekly Chief Concern objective, repeated across courses.
- File-to-event links are good where they come from real folder or event names: 38 of 40 correct for the date, folder and H&S-code methods. Week-level DOCS links are weak (5 of 10). Two fixable pipeline bugs account for most errors.
1. Bloom label vs the objective’s leading verb (all courses, AY25-26)
The leading verb of 94% of the 3,066 labelled objectives is in a standard verb list (after Anderson & Krathwohl 2001). Verbs that sit at two levels count as agreeing with either (“describe” can be Remembering or Understanding). 87% of labels agree with their verb. By course:
| EM | FP | MSI | PCV | RLC | RU |
|---|---|---|---|---|---|
| 0.88 | 0.69 | 0.94 | 0.91 | 0.90 | 0.87 |
The most common disagreements: - “Review …” labelled Remembering (45). This is arguably fine: our list treats review as Understanding. - “Correlate …” labelled Applying (21). - “Compare …” labelled Understanding (20). - “Identify …” labelled Applying (16). - “Predict …” labelled Evaluating (14).
FP’s lower agreement is worth a look by its course team. The full list is in bloom_verb_mismatches.csv (backlog DQ8).
2. Bloom label vs material
Aggregate signal (all courses). For each objective with material, we asked: does its session contain at least one question or clinical-vignette slide carrying at least half of the objective’s content words?
| Bloom level (as written) | n | has on-topic practice slide |
|---|---|---|
| Remembering | 463 | 63% |
| Understanding | 1,164 | 67% |
| Applying | 371 | 56% |
| Analysis | 341 | 68% |
| Evaluating | 79 | 49% |
| Creating | 24 | 46% |
If labels tracked what students do, practice would rise with level. It doesn’t: objectives labelled Evaluating or Creating are the least likely to have an on-topic question or case slide.
By method, labs (84%), small groups (82%) and TBL (86%) carry the most on-topic practice. Directed independent learning (prework) carries the least (42%).
Hand rating (PCV, 36 objectives, 6 per level, template objectives kept in as sampled). The rubric: - M: the material includes an activity at the labelled level or higher. For Remembering and Understanding, clear exposition counts. - L: only lower-level support is visible (exposition or recall questions). - A: no relevant material found in the text. - T: template objective describing session format, not content. - C: can’t tell from text.
| Level | M | L | A | T | C |
|---|---|---|---|---|---|
| Remembering | 4 | – | 2 | – | – |
| Understanding | 4 | – | 1 | – | 1 |
| Applying | 5 | 1 | – | – | – |
| Analysis | 1 | 4 | 1 | – | – |
| Evaluating | 1 | – | – | 5 | – |
| Creating | – | 5 | – | – | 1 |
Patterns: - Creating. “Formulate a differential for wide-/narrow-complex tachycardia” is supported by worked cases with answers, and “formulate a CF management strategy” by multiple-choice questions. Students see a differential or select an answer. The text doesn’t show them generating one. - Analysis. “Distinguish transudative vs exudative effusions” is supported by a Light’s-criteria slide, which is application of a rule, not analysis of pathophysiologic differences. - Evaluating. Five of the six sampled are the weekly template objective. The sixth (“clinically diagnose ACS”) is supported by a vignette MCQ. - Lower levels are mostly well supported. The misses are prework objectives whose material is video or isn’t in the library.
Caveat, and it matters: much higher-order work happens in the room: facilitated discussion, small-group case work, lab stations. Slides don’t capture it. “Not visible in the text” is not “not taught”. That’s why this is a review prompt for course directors, never a score.
3. File-to-event link accuracy (50 links, 10 per method)
| Method | Correct | Wrong | Unsure |
|---|---|---|---|
| folder name | 10 | 0 | 0 |
| dated folder (date + time) | 9 | 1 | 0 |
| H&S session code | 9 | 1 | 0 |
| file name | 7 | 2 | 1 |
| DOCS week | 5 | 5 | 0 |
The errors are mostly systematic: - Week ranges. DOCS folders named Wk1-5 are read as week 1 (all 5 DOCS-week errors). 173 files sit under week-range folders. Fix: parse ranges and link at course level, or to all weeks in the range. - Swapped time slots. When sessions swap times after folders are made, a date/time match picks the wrong event. 24 of 613 date links have titles that disagree with the folder name (e.g. Parturition ↔︎ Anatomy and Imaging of Pregnancy; Vaccines ↔︎ Immune System in Development). Fix: require title agreement for date matches, else fall through to the folder-title match. - H&S A/B sessions. “EM 9-1B” was linked to the 1A event. - File-name matches occasionally cross pillars, e.g. a DOCS ear-exam file linked to a DOCS ultrasound event.
4. Schedule anomalies
- MSI AY25-26 has one event dated 2024-04-09 (“Dermatology Cases and Review”), a year early (DQ3).
- The time-format issues, overnight assessment windows and repaired end times are covered in ADR 0003 (DQ2).
Implications for Phase 2
- Show Bloom levels as written, with the verb, and a per-course verb-mismatch count in the data-quality panel.
- Flag template objectives and exclude them from Bloom profiles by default (with a toggle).
- For objectives labelled Analysis and above, show whether an on-topic practice slide was found. Frame it as “no in-material activity found at this level; may happen in session”, for review.
- Fix the two linking bugs before the workbench goes to a course director, and show the link method on every file.