Backlog

Concrete work, newest priorities first. Vision lives in IDEAS.md (§ refs below); findings and open questions live in NOTES.md.

Guardrails: faculty- and leadership-facing only — nothing student-facing until faculty/leadership have used the tools. No embeddings or LLM passes until the Phase 1 research says what they’re for. Every number drills back to a source file/page/objective.

Decisions pending

• Pilot course. Proposed: PCV (largest: 276 events, ~735 LOs in AY25-26, builds through AY27-28). Real criterion: which course director will actually review it with us.
• First tool. Proposed: single-course workbench (IDEAS §12) before the lesson-design assistant (§14). It needs no generative model, uses data we trust, and builds the retrieval pieces §14 reuses.
• Docs site hosting. Public (no login) at https://curriculum-atlas.seandavis.net (fallback curriculum-atlas-docs.seandavi.workers.dev), a Cloudflare static-assets Worker (docs/wrangler.toml, just deploy). Workbenches stay local.
• Repo remote. If the repo itself ever leaves this machine: move generated data out of git history, scrub names in paths.
• Delivery format. Recommended (Phase 1c memo): one self-contained HTML file per course with client-side search; confirm after the pilot.
• LLM/vendor approval (NOTES open question 3) — gates hosted embeddings and any LLM step.

Done — Phase 0: foundations (branch phase0)

• Keep end time in events; define learner hours vs contact hours (parallel AMC/FCB sessions and small-group sections are double-counted by a plain sum). Done when per-course hours are reproducible and explained. → hours table, ADR 0003.
• Normalize Bloom spelling (Analysis/Analyzing, applying) and method spelling (small group). Label Bloom as “as written” everywhere.
• Reference frameworks → reference/ (source URL, version, retrieval date, license note per file; kept as markdown/CSV so agents can load them): school MEPOs (full text, not just codes — ask curriculum office), AAMC PCRS, AAMC Core EPAs, ACGME core competencies, USMLE Step 1 content outline, AAMC Curriculum Inventory method vocabularies, LCME standards touching curriculum management (Std 6, 8). Plus any MEPO → PCRS/EPA crosswalk the school has (NOTES open question 2). → reference/ + reference/crosswalks/ (SKOS, ADR 0006). Still missing: LCME standards (download blocked), school’s own crosswalk.
• DuckDB FTS indexes on pages (text + speaker notes), objectives, events (title + description), rebuilt by build_db.py. Done when a query returns ranked pages with course/event/year context.
• MeSH 2026 loaded locally (scripts/mesh.py, mesh_* tables) for synonym expansion and concept tagging.
• ADRs (docs/adr/) and a Quarto documentation site (docs/, just docs).

Done — Phase 1: research (FTS only) → docs/research/phase1.md

Goal: a short memo saying what the faculty tool should answer and what retrieval it needs. Output is a decision, not a system.

• Question set. Draft ~30 realistic course-director / faculty questions for the pilot course (§11 list as a seed); refine with a faculty contact.
• FTS baseline. Run the question set through FTS; judge the top-10 hits; categorize misses (synonyms/abbreviations like HFrEF, image-only slides, generic event titles, wrong grain). Done when there’s a hit/miss table with examples.
• Objective-support probe. For each pilot-course AY25-26 objective, search that event’s own pages. How often does lexical matching find plausible support? Hand-check a sample of 50. This is the experiment that decides whether embeddings are needed, and at what grain (page vs. objective vs. event).
• Concept-layer scan. Check whether the existing vocabularies (system, chief concern, discipline, MEPO) are enough for a course view, and what a finer concept layer would cost (UMLS/MeSH locally vs. LLM tagging).
• Memo → docs/research/phase1.md: questions worth answering, what FTS handles, where it fails, embeddings yes/no (grain, local vs. hosted), recommended scope for the Phase 2 tool.

Done — Phase 1b: validity checks → docs/research/validity.md

Goal: know how far the existing metadata can be trusted before a tool shows it to faculty. Sanity checks, not a study.

• Bloom label vs. objective wording. Label/leading-verb agreement is already ~80–90% for common verbs; list the disagreements (e.g. “describe” tagged Evaluating) as a data-quality table for the course.
• Bloom label vs. material. Do the linked slides/notes look like the labelled level? Cheap signals first: practice questions, “which of the following”, clinical vignettes, cases, TBL/case method vs. pure exposition. Then hand-rate a stratified sample (~60 objectives across levels) against their event’s material with a written rubric. Done when we can say, per Bloom level, how often the material plausibly supports it — with examples. An LLM rater can be compared to the hand ratings later if approved.
• Link accuracy. Hand-check ~50 file→event links by basis (date/folder/filename/hs_code/docs_week).
• Schedule anomalies. e.g. MSI AY25-26 dates starting 2024-04; events with no date/method.
• Findings → docs/research/validity.md, summarized in the Phase 1 memo.

Done — Phase 1c: what a course workbench looks like elsewhere → docs/research/workbench-landscape.md

Goal: borrow proven functionality and UI patterns; avoid rebuilding what the school’s curriculum-management system already does. Informed by the Phase 1 question set.

• Survey course-director-facing features in commercial systems (Elentra, one45, MedHub, Medtrics, others found) and open source (Ilios; others found): curriculum maps, objective linkage, coverage/gap reports, AAMC CI reporting, dashboards. Screenshots/docs where public.
• Identify which CMS produced the Course Build workbooks (“MEA Event ID”) and what it already offers our course directors.
• Note UI/UX patterns worth copying (drill-down, matrix views, evidence panels) and gaps our slide-level corpus could fill.
• → docs/research/workbench-landscape.md, with a recommended feature list and delivery format for Phase 2.

Later — beyond Phase 2 (from the Phase 1 memo)

• Local embeddings at slide grain + objectives; evaluate on the question set plus conceptual queries.
• MeSH concept tagging of objectives and slides (stoplist, sample check) → topic time, trajectories.
• Parse FCB “(LOx.y)” slide tags as author-made alignment evidence / gold data.
• Curated abbreviation list for search (HFrEF, GDMT, …); drop automatic MeSH expansion.

Now — Phase 2: single-course workbench (pilot course: PCV, pending confirmation) → docs/workbench.md

Scope confirmed by the Phase 1 memo, validity checks and landscape survey. Candidate contents, all drillable:

• Inventory: events, hours by method, objectives per event, material status.
• Coverage by system / chief concern / discipline / MEPO (and PCRS/EPA via crosswalk, if one exists); Bloom profile (as written, with Phase 1b caveats).
• Events with no linked material; Uploaded/Final with no file found.
• Objective-support signals (method from Phase 1), labelled “for review”.
• Year-over-year change (vs previous year, slide-level) and AMC vs. FCB divergence.
• Image-only slide share (where text-based analysis is blind): per-file counts shown in the session drawer; no course-level summary yet.
• Review session with the course director (guide in docs/workbench.md); record in docs/research/pilot-feedback.md.
• Faculty spot-check of the agent’s Phase 1/1b hand judgments before quoting them.
• DQ1 fix: fuzzy-match objective titles to scheduled events so all objectives get dates.

Data quality follow-ups

Issues found in the source data. Each is listed with where it lives, its size, and who can fix it: CO = curriculum office, CT = course team, us = pipeline fix. Keep adding as we go. Counts are across all courses and years unless a course is named.

#	Issue	Where	Size	Fix by	Status
DQ1	Objective event titles don’t match any scheduled event title (spelling/format drift, e.g. “PCV week 4 H&S large group” vs “PCV Week 4: H&S Large Group - AMC”), so those objectives aren’t tied to a date	LOs vs Planning workbooks	350 unscheduled titles; e.g. RU AY25-26 91 of 417 LOs, RLC AY24-25 109 of 480; AY23-24 EM/RLC have no planning sheet at all	us (fuzzy title match within course-year-week), CO (consistent titles)	open
DQ2	Time fields in mixed formats (1PM, 11:00am, 9:30AM), end before start (13:00–04:00), end dates in the time column for multi-day checkpoints	Planning / Granular Build	44 unparseable; a few repaired as missing PM (ADR 0003)	CO/CT (use time format), us (done: parser + repair)	mitigated
DQ3	MSI AY25-26 schedule starts 2024-04-09, a year early	MSI AY2526 Planning	≥1 event	CT	open
DQ4	Material-status vocabulary drift: Uploaded / uploaded / Uploaded-A; 2,455 blank	Granular Build	97 Uploaded-A, 51 lowercase	CO (what is “Uploaded-A”?), us (normalize)	open
DQ5	Duplicate events with typo titles, e.g. “Overview of Dysrytmias” (0 LOs) next to “Overview of Dysrhythmias” (12 LOs), same slot	PCV AY25-26 and others	118 same-slot pairs (some are genuine parallel sections)	us (flag twins), CT	open
DQ6	Scheduled events with no instructional method	Planning	29 dated events (+ undated)	CT	open
DQ7	Objectives with no Bloom level	LOs	27	CT	open
DQ8	Bloom label disagrees with the objective’s leading verb (e.g. “compare” tagged Understanding)	LOs	378 of 2,885 AY25-26 (13%); FP 31% (docs/research/bloom_verb_mismatches.csv)	CT	open
DQ9	Typos in controlled vocabularies: “Mysthenia Gravis”, “Physcial”; both “Differences of” and “Disorders of Sex Development”	Validation sheets, event titles	few	CO	open
DQ10	Some DOCS file names appear to contain individual (possibly student) names	Resource collection, DOCS	unknown	CO (FERPA review)	open
DQ11	Undated H&S events (e.g. “PCV Week 5: H&S Large Group - AMC”) that exist only in LO sheets	LOs	see DQ1	us	open
DQ12	Same objective text attached to more than one event in a course-year (large/small group pairs, prework + session)	LOs	1,015 texts	CT (intentional?), us (don’t double-count)	open
DQ14	Sessions swapped time slots after folders were created, so date/time links point at the wrong event	Resource folders vs schedule	24 of 613 date links (e.g. Parturition ↔︎ Anatomy and Imaging of Pregnancy)	us (require title agreement)	fixed: date links need title agreement (token-set ≥ 60), else fall through to folder match; 24 relinked
DQ15	Week-range folders (Wk1-5) parsed as a single week	DOCS folders	173 files	us	fixed: ranges give no week; 31 DOCS files now unlinked instead of wrongly linked
DQ16	Template objectives repeated weekly (“Evaluate the patient’s clinical narrative…”, “Utilize an advance organizer…”) inflate Applying/Evaluating counts	LOs	92 AY25-26 objectives in repeated texts; 30 of 107 Evaluating	CO/CT (intentional?), us (flag)	open
DQ17	H&S A/B session codes: “1B” files linked to the “1A” event	link.py	95 files	us	fixed: A/B code prefers “Week nA/nB” events
DQ13	~190 events marked Uploaded/Final in build sheets with no file found in this library	Granular Build vs library	~190	us (linking), CO (FCB drive location)	open

Blocked

• Assessment alignment (§8, output G) — needs item bank / blueprint tagged to objectives.
• Trajectories beyond one preclinical year (§4, §10) — needs other courses and clerkships.
• Ownership analysis (§8) — needs a decision on using instructor/role data.

Parked

• Lesson-design assistant (§14) — after workbench feedback and LLM approval.
• Skill collections and diagramming (§15–18), architecture (§19), leadership observatory beyond the existing report (§3).