Architecture

Pipeline, data model, and how the objective-evidence flags are computed

Context: where the atlas sits

The school’s curriculum-management system and LMS hold tags and schedules. The atlas adds a content-evidence layer built from the materials themselves (landscape memo).

flowchart LR
  subgraph School["School systems"]
    SP[("SharePoint library<br/>Plains Curriculum<br/>4,788 files · 74 GB")]
    CB["Course Build workbooks<br/>Planning · LOs · Granular Build"]
    CMS["Curriculum-management<br/>system (unknown)"]
    LMS["Canvas"]
  end
  subgraph Atlas["Curriculum atlas (this repo, local)"]
    PIPE["Pipeline<br/>scripts/*.py"]
    DB[("atlas.duckdb")]
    REF["reference/<br/>frameworks · crosswalks · MeSH"]
  end
  subgraph Out["Outputs"]
    WB["Course workbench<br/>(one HTML per course)"]
    RPT["Leadership report"]
    DOCS["This site<br/>+ llms.txt"]
  end
  SP -->|OneDrive sync| PIPE
  CB --> PIPE
  REF --> DB
  PIPE --> DB
  DB --> WB & RPT & DOCS
  WB -. links .-> SP
  CB -. exported to? .- CMS
  LMS -. small-group materials? .- SP
  style School fill:#f6f8fa,stroke:#c8ccd1
  style Atlas fill:#eef4ff,stroke:#8fb3e6
  style Out fill:#f3faf3,stroke:#9ccc9c

The atlas reads the synced SharePoint library and the Course Build workbooks. Everything it produces is read-only and links back to SharePoint.

Pipeline

Every step is re-runnable. Extraction skips unchanged files and never downloads cloud-only placeholders. just all rebuilds everything (see the justfile).

flowchart TB
  inv["inventory.py<br/>files.csv"] --> ext["extract.py<br/>MarkItDown / LiteParse<br/>→ pages.jsonl"]
  inv --> paths["paths.py<br/>course · year · stream · week · roles<br/>→ files_meta.csv"]
  cur["curriculum.py<br/>Course Build workbooks<br/>→ events · build · objectives · vocab"]
  paths --> link["link.py<br/>event catalog + file→event links<br/>(date · folder · H&S code · filename · DOCS week)"]
  cur --> link
  ext --> sim["similarity.py<br/>MinHash near-duplicate families"]
  ref["reference.py<br/>frameworks from PDFs"] --> db
  mesh["mesh.py<br/>NLM MeSH XML → Parquet"] --> db
  link --> db["build_db.py<br/>atlas.duckdb<br/>hours · normalization · FTS indexes"]
  sim --> db
  ext --> db
  db --> wb["workbench.py<br/>+ alignment.py"]
  db --> docs["docs/ (Quarto)"]
  db --> rep["report/ (Quarto)"]

Build pipeline. CSV/JSONL/Parquet intermediates in data/; the database is rebuilt from scratch each time (ADR 0002).

Data model

The core tables, with the keys that join them. Reference tables join through codes: MEPO codes on objectives, method names on events.

erDiagram
  EVENTS ||--o{ FILES : "event_key"
  EVENTS ||--o{ OBJECTIVES : "objectives_key = event_key"
  FILES ||--|{ PAGES : "rel_path"
  OBJECTIVES }o--o{ REF_MEPO : "mepo_codes"
  REF_MEPO ||--o{ CROSSWALK : "source_value"
  CROSSWALK }o--|| REF_FRAMEWORKS : "target_code"
  VOCAB ||--o{ CROSSWALK : "source_value"
  CROSSWALK }o--o{ MESH_DESCRIPTORS : "target_code"
  MESH_DESCRIPTORS ||--|{ MESH_TERMS : "descriptor_ui"
  EVENTS {
    string event_key PK
    string course
    string academic_year
    string campus
    string event_title
    int duration_min
    string method_primary
    string objectives_key
  }
  FILES {
    string rel_path PK
    string event_key FK
    string link_basis
    string roles
    string family_id
  }
  PAGES {
    int page_id PK
    string rel_path FK
    int page
    string text
    string notes
  }
  OBJECTIVES {
    int objective_id PK
    string event_key FK
    string description
    string blooms
    int blooms_level
    string mepo_codes
  }
  HOURS {
    string course
    string academic_year
    string campus
    float learner_hours
  }

Core tables in atlas.duckdb. FCB events share their AMC twin’s objectives through objectives_key.

How an objective gets its evidence and flags

scripts/alignment.py is shared by the Phase 1 probes and the workbench. It deliberately uses words only: no embeddings, no LLM (ADR 0004).

flowchart TB
  O["Objective text"] --> T["Content words<br/>stem · drop stopwords, Bloom verbs, generic words"]
  E["Session's files<br/>(AMC + FCB twin)"] --> P["Pages"]
  P --> R{"Restates objectives?<br/>objective-list slide · echoes ≥2 objectives<br/>· quotes this objective"}
  R -->|yes| X["ignored"]
  R -->|no| M["Overlap per page"]
  T --> M
  M --> TOP["Top 3 evidence slides<br/>one per file · no repeated text"]
  M --> COV["Share of key terms found<br/>anywhere in the session"]
  M --> PR["Question / case slides<br/>covering ≥ half the terms"]
  COV --> F1["low overlap (< 0.5)"]
  PR --> F2["no practice found<br/>(Analysis and above)"]
  O --> F3["verb ≠ label<br/>(leading verb vs Bloom)"]
  O --> F4["repeated<br/>(same text in ≥ 3 weeks)"]
  E --> F5["no material / restated only"]

Objective evidence. Restatement filtering matters: without it, 444 of 641 ‘best matches’ were the objectives slide itself.

Hours

flowchart LR
  A["Events with start/end"] --> B{"Exam, or > 6 h?"}
  B -->|yes| W["assessment window<br/>(excluded)"]
  B -->|no| C["per campus × day<br/>(campus-less events count for both)"]
  C --> S["scheduled hours = Σ durations"]
  C --> L["learner hours = length of the union<br/>of time intervals"]

Learner hours merge overlapping sessions per campus-day; scheduled hours simply sum (ADR 0003).