chapterwise — Cursor project rules

# chapterwise — Cursor project rules

Textbook PDF → ordered topics (chapters/sections/subsections) + per-topic explanation, Q&A, images, and TTS narration.

**Monorepo layout:** `backend/` is a standalone FastAPI app (`python backend/main.py`). `frontend/` is a standalone Vite React app (`npm run dev` from `frontend/`). Do not mix Python and JS across folders. **`frontend/src/config/api.js` is the only place the backend base URL/port should be hardcoded** (default `http://127.0.0.1:8766`).

---

## 1. What this app does (v1 goal)

1. User uploads a textbook PDF.
2. Backend extracts text + images per page (local PyMuPDF + pdfplumber, or optional **Kaggle** remote extract via tunnel).
3. **Regex heading scan** (default) builds an **ordered** hierarchical topic tree (chapter → section → subsection) in document order. Optional legacy path: Kaggle Llama / Azure via `CHAPTERWISE_TOPIC_BUILDER=llm`.
4. Each topic shows explanation, source page range, and formula/diagram images from those pages.
5. User asks questions grounded in that topic’s text only.
6. User presses Listen for OpenAI TTS narration.

**Different from magazine/Veda app:** topics are **hierarchical** and stay in **document order** (not layout/font driven). Images attach to topics by page range.

---

## 2. Build order (§8) — status

| Step | Scope | Status |
|------|--------|--------|
| **1** | Backend `/upload_pdf` + `pdf_extract.py` (per-page text + images) | ✅ Done |
| **2** | Backend `/build_topics` + `topic_builder.py` (ordered tree; **regex default**, optional LLM) | 🟡 Partial — **regex TOC works** (full Class 9 Science 1299 pp → 223 topics / 214 headings in **0.16s**); title/order still OCR-noisy; **full-book false `Chapter N` hits** (duplicate/out-of-order chapter nodes); LLM path kept behind `CHAPTERWISE_TOPIC_BUILDER=llm` |
| **3** | Frontend `UploadPanel` + `TopicTree` wired to upload + build_topics | ✅ Done — wired; **sidebar inner-scrolls** (`.app-shell` `height: 100vh; overflow: hidden`; `.topic-tree-list` `flex:1; min-height:0; overflow: auto`) |
| **4** | Backend `/topic/{id}` + Frontend `TopicView` (explanation + image gallery) | 🟡 Partial — **SSE-streamed** LLM explanation; Listen + **voice picker**; deferred image b64; gallery drops `isBackground`; markdown when complete; source card **collapsed by default**; enhance wired (step 7) |
| **5** | Backend `/ask` + Frontend `AskPanel` (grounded Q&A) | ✅ Done — `qa.py` + **SSE** `POST /ask` + FAB drawer; chat thread streams answers; clears on topic switch; no Q&A cache |
| **6** | Backend `/tts` + Listen button in `TopicView` | ✅ Done — `/tts` + Listen; default **`shimmer`**; UI picker (10 confirmed voices); persists across topic switches in the session |
| **7** | Backend `image_enhance.py` (informational → Lanczos, decorative → gpt-image-2) | 🟡 Partial — cheap classify + baked-text OCR; informational → Lanczos; decorative **Generate HD** uses **`images.edit`** (source bytes, `input_fidelity=high`); vision+`images.generate` only if edits fails. `overlapWordCount` still 0 on RapidOCR extract. |
| **8** | End-to-end polish; revisit `# TODO v2` items | ⬜ Not started |

**When continuing work:** follow this order. Do one step at a time unless the user explicitly asks for multiple steps. Get user test approval between major steps when they ask for that workflow.

---

## 3. Project structure

```
chapterwise/
  .cursorrules                 ← this file
  .gitignore
  backend/
    main.py                    ← FastAPI app
    azure_clients.py           ← Azure chat + TTS/image clients (separate keys) + /debug/azure
    text_provider.py           ← TEXT_PROVIDER routing (Azure chat / Kaggle Llama)
    pdf_extract.py             ← Per-page text + image extraction
    topic_builder.py           ← ordered topic tree (regex headings default; optional LLM)
    topic_detail.py            ← Topic payload (no LLM on GET) + SSE explanation stream + on-demand image b64
    qa.py                      ← grounded SSE /ask (step 5) ✅
    tts.py                     ← Azure TTS (step 6) ✅ default shimmer; TTS_VOICES allowlist
    image_enhance.py           ← classify + Lanczos + on-demand gpt-image-2 **edits** (step 7)
    security.py                ← PDF validation, CORS, size limits
    requirements.txt
    .env                       ← secrets (gitignored; copy from .env.example)
    .env.example
    active_doc.pdf             ← written on upload (gitignored)
    vendor/                    ← pip --target installs (gitignored)
    kaggle/
      chapterwise_gpu_server.py
      llm_server.py            ← Llama chat on Kaggle GPU
      chapterwise_kaggle.ipynb
      prepare_chapterwise_dataset.bat
      publish_chapterwise_dataset.py
      README.md
    kaggle_client.py           ← laptop → Kaggle HTTP client
  frontend/
    index.html
    package.json
    vite.config.js
    src/
      config/api.js            ← API base URL only here
      utils/apiClient.js       ← fetch + SSE consumeSse
      utils/mergeTopicPoll.js  ← keep streaming explanation across classify polls
      context/DocContext.jsx
      components/
        UploadPanel.jsx        ← step 3 ✅
        TopicTree.jsx          ← step 3 ✅
        TopicView.jsx          ← step 4 (AskPanel + gallery Generate HD)
        UploadLoadingScreen.jsx ← animated upload/build loading
        TopicViewSkeleton.jsx  ← shimmer skeleton while topic loads
        Skeleton.jsx           ← reusable shimmer block
        AskPanel.jsx           ← step 5 ✅ grounded Q&A
        MarkdownText.jsx       ← react-markdown for explanation + ask answers
      App.jsx
      main.jsx
      index.css
```

---

## 4. Implemented backend

### Endpoints (live)

| Method | Path | Purpose |
|--------|------|---------|
| GET | `/health` | Health check (+ Kaggle reachability, `text_provider`, LLM status) |
| POST | `/upload_pdf` | Save PDF, extract pages; optional `?page_start=&page_end=`; returns `timing` (receive/extract/total seconds) |
| POST | `/build_topics` | Build topic tree (default: heading regex; optional LLM). Returns `builder`, `timing.total_seconds` |
| GET | `/topic/{topic_id}` | Topic metadata, source, images, cached explanation if any. **Does not call the LLM.** `explanation_status`: ready \| pending \| skipped |
| GET | `/topic/{topic_id}/explanation/stream` | SSE explanation (`delta` / `done` / `error`). Caches full text only on `done` |
| POST | `/topic/{topic_id}/images/{image_id}/generate` | On-demand gpt-image-2 **edit** for **decorative + classify_status=ready** only; 400 if informational/pending |
| POST | `/ask` | SSE grounded answer `{ topic_id, question }` (`delta` / `done` / `error`) |
| POST | `/tts` | `{ text, voice? }` → MP3 (default `shimmer`; allowlist in `tts.py`) |
| GET | `/debug/text` | Smoke-test text provider (Kaggle Llama or Azure chat) |
| GET | `/debug/azure` | Smoke-test TTS and image Azure deployments |
| GET | `/debug/extraction` | Per-page extraction summary |
| GET | `/debug/extraction/{page}` | Full page text; `?include_image_data=true` for base64 |
| GET | `/debug/topics` | Full built topic tree + `topics_by_id` |

### State (in-memory v1)

- `_active_extraction` — full per-page text + images; cleared on new upload
- `_active_topics` — topic tree + flat `topics_by_id`; cleared on new upload
- `_explanation_cache` in `topic_detail.py` — per-topic LLM explanations; cleared on new upload
- `_IMAGE_URL_CACHE` in `topic_detail.py` — on-demand image data URLs keyed by `{pdf fingerprint}:{xref}`; cleared on new upload
- `_ENHANCE_CACHE` in `image_enhance.py` — SHA-256 of **source** bytes → Lanczos / generated / classify result (`classify_status` pending\|ready); cleared on new upload
- Resets on server restart (`# TODO v2`: disk cache)

### PDF extraction (`pdf_extract.py`)

- **PyMuPDF** (`fitz` / `pymupdf`): images via `get_images(full=True)` + `get_image_rects`; skip icons < 50px. `fitz` import shows deprecation warning — harmless for v1.
- **Text backends:** `CHAPTERWISE_TEXT_EXTRACTOR=pdfplumber` (default) or `rapidocr`. Laptop dev often uses **rapidocr** with `CHAPTERWISE_RAPIDOCR_FORCE=0` (digital pages use PyMuPDF `get_text`, not full OCR every page).
- **pdfplumber**: page text when extractor is pdfplumber; single open per page for text + overlap word counts
- **Parallel extraction:** `ProcessPoolExecutor` on Linux/Kaggle; `ThreadPoolExecutor` on Windows. `CHAPTERWISE_EXTRACT_WORKERS` (Kaggle default: CPU count).
- **Page-range extract:** `extract_document(pdf_path, page_start=, page_end=)` — only parses requested pages; returns `num_pages` (PDF total), `page_start`, `page_end`, `pages`.
- **Header/footer detection:** samples **36 evenly spaced pages** (not full PDF) via `PDFMetadataAnalyzer`; result **cached** by path + size + SHA256(head 64KB + tail 64KB). Fixes ~90s header scan on partial extracts (e.g. pages 1–10).
- **Debug timing:** `CHAPTERWISE_DEBUG_TIMING=1` logs per-phase seconds (pdf open, header/footer, rapidocr, images, worker pool).
- **Deferred image b64:** `CHAPTERWISE_DEFER_IMAGE_B64` defaults **on** (`main.py` `setdefault` to `1`, same as Kaggle). Extract stores xref + bbox only; `/topic` encodes via `encode_xref_data_url()`. JPEG2000/`jpx` converted to PNG so browsers can paint. Override with `CHAPTERWISE_DEFER_IMAGE_B64=0`.
- **Pixmap / SMask:** `encode_xref_data_url()` uses Pixmap (RGB + SMask as PNG alpha) when ext is unsafe, SMask present, bpc under 8, or CMYK/Separation/DeviceN. Raw `extract_image` PNG for 1-bit ICC Gray + SMask was **solid black** in browsers (artwork lived in the mask). Empty color+mask → fully transparent PNG.
- **`overlapWordCount`:** only attached on the **pdfplumber** text path. **RapidOCR extract path never computes it** (all images store `0`). Do **not** treat overlap=0 as evidence of decorative. Cheap classify uses aspect + gray/line-art only; baked-in labels need the image-pixel OCR gate.
- **RapidOCR init:** `_get_rapidocr_pdf()` double-checked under `_RAPIDOCR_LOCK` — one `RapidOCRPDF` per process, not one per first-wave thread.
- **Kaggle defaults** (in `chapterwise_gpu_server.py`): deferred image base64, skip overlap counts, process pool.
- `# TODO v2`: skip re-upload if PDF hash unchanged on Kaggle; restore `overlapWordCount` on RapidOCR extract path

### Kaggle remote extraction (`kaggle_client.py`, `kaggle/chapterwise_gpu_server.py`)

- **Dataset:** `eakeswar/chapterwise-models` on Kaggle — publish via `backend/kaggle/publish_chapterwise_dataset.bat`
- **Flow:** laptop saves PDF → chunked upload to Kaggle → async extract → poll status → fetch result JSON
- **`kaggle_client.py`:** byte uploads via `_post_with_retry()`; JSON chat via `_post_json_with_retry()` → prefer **async** `POST /chat` → poll `GET /chat/status` → `GET /chat/result` (falls back to sync `POST /chat/completions` if `/chat` is 404)
- **`EXTRACT_PROVIDER`:** `local` | `kaggle` | `auto` (fallback to local if Kaggle fails). **`local` recommended for laptop dev** with page-range testing.
- **Tunnel:** cloudflared `trycloudflare.com` URL changes each notebook session → update `KAGGLE_API_BASE_URL` in `.env`
- **Large PDF fixes (2026-07-24):**
  - Chunked upload: `POST /upload_chunk` (4 MB parts, `KAGGLE_UPLOAD_CHUNK_BYTES`)
  - Async extract: `POST /extract` → poll `GET /extract/status` → `GET /extract/result` (avoids Cloudflare 524)
  - Poll retries: `KAGGLE_POLL_TIMEOUT=90`, `KAGGLE_UPLOAD_RETRIES=3`
- **Kaggle server endpoints:** `/health`, `/chat`, `/chat/status`, `/chat/result`, `/chat/completions` (sync legacy), `/upload_pdf`, `/upload_chunk`, `/extract`, `/extract/status`, `/extract/result`, `/upload_and_extract` (small files only)
- **Auth:** `Authorization: Bearer <KAGGLE_API_SECRET>` required on chat/extract/upload. `/health` is unauthenticated. Laptop `.env` and notebook secret must match exactly (placeholder → 401).
- **Kaggle LLM (`llm_server.py`):** default `meta-llama/Llama-3.2-1B-Instruct` (4-bit on GPU). Requires `HF_TOKEN` on notebook + **gated-model access** on Hugging Face. Set `CHAPTERWISE_LLM_PRELOAD=1` to load at startup.
  - **Defaults (2026-08-10):** `CHAPTERWISE_LLM_MAX_INPUT_CHARS=12000`, `CHAPTERWISE_LLM_MAX_NEW_TOKENS=1024`, `CHAPTERWISE_LLM_ASSISTANT_CAP_CHARS=2000`.
  - **Truncation:** enforces total char budget across **all** messages (caps assistant, trims page chunk, not only last user). `torch.cuda.empty_cache()` after each generate.
  - **Generation:** `repetition_penalty=1.2`, `no_repeat_ngram_size=4`. `top_p` warning from transformers config is harmless.
  - **HF auth:** reads `HF_TOKEN` / `HUGGING_FACE_HUB_TOKEN`; calls `huggingface_hub.login(token=…)`; passes **`token=`** (not deprecated `use_auth_token=`) to `AutoTokenizer.from_pretrained` / `AutoModelForCausalLM.from_pretrained`.
  - **Notebook cell 2:** load secrets via `UserSecretsClient` — labels **`KAGGLE_API_SECRET`** and **`HF_TOKEN`** (Add-ons → Secrets). Print `present`/`length` only (never the token). Set LLM limits + optional `PYTORCH_ALLOC_CONF=expandable_segments:True` before `!python` server.
- **Note:** PDF parsing uses **CPU** (PyMuPDF/pdfplumber); **GPU** used for Llama chat via async `/chat` (or sync `/chat/completions`).
- `# TODO v2`: host extract on Azure Container Apps + Blob instead of free Cloudflare tunnel

---

### Topic building (`topic_builder.py`)

- **`CHAPTERWISE_TOPIC_BUILDER`:** `regex` (default) | `llm`
- **Regex path (Plan B, 2026-08-12):** Scan extracted page text for NCERT-style headings — `Chapter N` / OCR `hapterC`, `N.N Title`, `N.N.N TITLE`, `Activity N.N` (incl. OCR stutter `1.31.31.3` → `1.3`). **No Llama call.** Build ~0.01s on pages 1–8.
- **Body coverage (approach a):** Non-heading text is not dropped — each heading’s `page_start`/`page_end` spans until the next heading so `/topic` source slices include intervening body. First heading expands back to range start; last to range end.
- **Fallback (approach b):** If a page range has **zero** heading matches → one chapter node `Pages {start}–{end}` covering the whole range.
- **IDs:** `ch1`, `ch1_s1`, `ch1_s1_ss1`, … via `assign_topic_ids` / `flatten_topics` (same as before).
- **`/build_topics` response** includes `builder` (`regex`|`llm`) and optional `heading_count`.
- **Known regex limits (defer vs fix):** (1) `_SECTION_RE` title group can eat across column gaps (`Physical Nature of Matter of sa`); (2) headings glued mid-line truncate (`…glass1.2 Characteristics of Particles of`); (3) RapidOCR text is **not** column-aware reading order — e.g. page 6 emits `1.4.1 EFFECT` before `1.4 Can Matter…`, so tree order follows OCR stream.
- **LLM path (legacy):** `CHAPTERWISE_TOPIC_BUILDER=llm` — chunk by `CHAPTERWISE_TOPIC_CHUNK_PAGES`, Kaggle/Azure chat, JSON repair + retry. Large topic prompts (~12k chars) hang on Kaggle Llama 1B; tiny/medium chats (~2s) are fine.
- Optional body on `/build_topics`: `{ page_start, page_end, chunk_pages }` (`chunk_pages` only matters for `llm`). Range uses **extracted PDF page numbers**, not a 1-based remapped slice.

### Kaggle chat reliability (2026-08-12)

- Busy chat → **HTTP 409** (not `started:false`); each job has **`job_id`**
- Laptop polls/retries on 409; treats **502/503/524** as tunnel blips; sync `/chat/completions` fallback **only** on 404/405
- **Dead tunnel / DNS:** do **not** retry connection errors until `CHAPTERWISE_LLM_TIMEOUT` — raise after inner retries so `/topic` can return `explanation_status=failed` with source text
- Generate: `max_time` + **time StoppingCriteria** (~90s); job watchdog **`CHAPTERWISE_CHAT_JOB_MAX_SECONDS`** (default 120); **`POST /chat/reset`**
- Notebook cell 2 defaults: `MAX_NEW_TOKENS=512`, `MAX_TIME_SECONDS=90`, `CHAT_JOB_MAX_SECONDS=120`
- Republish `eakeswar/chapterwise-models` after server/llm_server changes; refresh tunnel URL each notebook session

---

## 5. Implemented frontend

- **UploadPanel:** PDF upload → `/upload_pdf` → `/build_topics`; optional “Test first pages only” passes `?page_start=1&page_end=N` to **upload** (partial extract) and build_topics; shows upload/extract + build timing after success
- **TopicTree:** Collapsible chapter → section → subsection; **never re-sort** client-side. Not virtualized. Default opens first chapter only. Sidebar list **inner-scrolls** (`.app-shell` `height: 100vh; overflow: hidden`; `.topic-tree-list` `overflow: auto`).
- **TopicView:** Fetches `GET /topic/{id}` (fast: no LLM). If `explanation_status=pending`, opens SSE `/explanation/stream`. Plain text + cursor while streaming; **Markdown** after `ready`. Incomplete stream keeps partial text + notice; **not** cached. Classify poll every 3s **merges** via `mergeTopicPoll` (does not wipe in-progress explanation). **Source text** card collapsed by default (`Source text (N chars)` + chevron); expands in place; resets collapsed on topic switch. **Listen** + **voice `<select>`** (default shimmer; `useState` survives topic switches). Gallery: informational/pending → no Generate HD; decorative + `classify_status=ready` → **Generate HD**. **Ask** = FAB + overlay drawer.
- **AskPanel:** `AskComposer`; **SSE** `POST /ask`; thread `turns[]` with streaming / incomplete / ready / failed; clears on `topicId` change; drawer closes on topic switch.
- **UploadLoadingScreen:** Animated main-panel loading during upload/build (shimmer progress bar + step labels + **live elapsed timer** + layout preview)
- **DocContext:** `topicTree`, `activeTopicId`, `topicCache`, `uploadTiming`, `buildTiming`, upload/build phases
- **App.jsx:** Sidebar + `TopicView` or `UploadLoadingScreen` in main panel
- Functional components, hooks only, **no `<form>` tags**, use `onClick` / file input handlers

### Run commands

```powershell
# Backend (port 8766)
python backend/main.py

# Frontend (port 5173)
cd frontend && npm run dev
```

```powershell
# Install deps (once)
python -m pip install --target backend/vendor -r backend/requirements.txt
cd frontend && npm install
```

---

## 6. Session updates & decisions (changelog)

- **Step 1:** Created backend with `/upload_pdf`, `pdf_extract.py`, `security.py`, debug extraction endpoints. Smoke-tested 90-page PDF.
- **Step 2:** Added `topic_builder.py`, `/build_topics`, `/debug/topics`. Azure OpenAI v1 API via standard `OpenAI` client + `base_url` + deployment name as `model=`.
- **Step 3:** Scaffolded Vite React frontend; `UploadPanel` + `TopicTree` + `DocContext` + `apiClient.js`.
- **PDF size limit:** Set to **500 MB** (`CHAPTERWISE_MAX_PDF_BYTES=524288000`).
- **`.env`:** Created `backend/.env` with Azure Foundry credentials (text, TTS, image deployments). Never commit `.env`.
- **Upload slowness (300 MB books):** Upload waits for **full local extraction** of every page before responding — not OpenAI. Discussed but not implemented: fast upload + background extraction, OpenAI/hybrid parsing for scanned PDFs.
- **Parallel extraction:** All CPU cores used locally; pdfplumber one open per page.
- **Kaggle remote extraction:** Bundle publish scripts, GPU notebook server, `kaggle_client.py`, `EXTRACT_PROVIDER` routing in `main.py`.
- **2026-07-24 — Kaggle large-PDF reliability:** Chunked upload (`/upload_chunk`), async extract with polling (`/extract/status`, `/extract/result`), poll retries. Published updated bundle to `eakeswar/chapterwise-models`. Tunnel URL must be refreshed in `.env` each notebook session.
- **2026-07-24 — Page-range extraction:** `extract_document()` + `/upload_pdf?page_start=&page_end=`; frontend “Test first pages only” now limits parse scope (full PDF still uploaded to Kaggle).
- **2026-07-24 — Azure topic build blocked:** API key + base URL for `edzhub-resource` work, but **`AZURE_TEXT_DEPLOYMENT=gpt-4o-mini` not deployed** → `DeploymentNotFound` on `/build_topics`. Need deployment created in Azure Foundry or new resource. API URLs are not browsable in Chrome (POST-only).
- **2026-07-27 — Azure deployment audit:** TTS (`gpt-4o-mini-tts`) and images (`gpt-image-2`) work on `https://edzhub-resource.services.ai.azure.com/openai/v1`. Chat (`gpt-4o-mini`) still `DeploymentNotFound` on `https://edzhub-resource.openai.azure.com/openai/v1`. Added `azure_clients.py`, `GET /debug/azure`, `POST /tts`, `tts.py`, basic `image_enhance.py`.
- **2026-07-27 — Step 4 shell:** `topic_detail.py`, `GET /topic/{topic_id}`, `TopicView.jsx` (source text + images, no LLM explanation yet), `topicCache`, shimmer skeletons, Listen button (TTS on source text).
- **2026-07-27 — Upload UX:** `UploadLoadingScreen.jsx` with animated progress bar + step labels during upload/build.
- **2026-07-27 — Kaggle tunnel refreshed:** `KAGGLE_API_BASE_URL=https://maiden-evaluate-ranked-short.trycloudflare.com` (ephemeral; update each notebook session). Verified reachable with 2 GPUs. Full PDF upload to Kaggle works; `/build_topics` still blocked by missing Azure chat deployment.
- **2026-07-28 — Azure re-verified:** `GET /debug/azure` → TTS + image OK; chat `gpt-4o-mini` still `DeploymentNotFound`. Use **deployment name** (not model name) for `AZURE_TEXT_DEPLOYMENT`. TTS smoke-tested with alternate voice (`shimmer`); samples in `backend/test_output/`.
- **2026-07-28 — Upload timing:** `/upload_pdf` returns `timing` (`receive_seconds`, `extract_seconds`, `total_seconds`, `file_mb`); `/build_topics` returns `timing.total_seconds`. Frontend live elapsed timer on `UploadLoadingScreen`; final times in `UploadPanel` success line. Backend logs include timing breakdown.
- **2026-07-28 — Kaggle tunnel refreshed:** `KAGGLE_API_BASE_URL=https://dial-controversial-lifetime-upgrade.trycloudflare.com` (ephemeral). Verified reachable (2 GPUs). Kaggle notebook pip warnings (`dopamine-rl`/`gym`, `gradio`/`pillow`) are harmless — extract server starts fine. Ensure `KAGGLE_API_SECRET` matches notebook (not placeholder). Laptop `EXTRACT_PROVIDER=kaggle`.
- **2026-07-30 — Kaggle Llama for text:** `llm_server.py` + `POST /chat/completions` on Kaggle GPU server (default `meta-llama/Llama-3.2-1B-Instruct`, 4-bit). Laptop `text_provider.py` + `TEXT_PROVIDER=kaggle|azure|auto`; `topic_builder.py` routes chat through it. `GET /debug/text` smoke test. Requires `HF_TOKEN` on Kaggle for gated Llama models.
- **2026-07-30 — Dataset published:** Republished `eakeswar/chapterwise-models` with `llm_server.py`. Laptop `TEXT_PROVIDER=kaggle`, `EXTRACT_PROVIDER=kaggle`.
- **2026-07-30 — Kaggle tunnel refreshed:** `KAGGLE_API_BASE_URL=https://lodge-salad-match-learned.trycloudflare.com` (ephemeral). Verified reachable (2 GPUs, Llama configured). Servers started for test then stopped. End-to-end upload → build topics pending user test after notebook + `HF_TOKEN` ready.
- **2026-08-03 — Step 4 LLM explanation:** `topic_detail.py` calls `text_provider.chat_completion` for grounded plain-language explanations; returns `explanation`, `explanation_status`, `explanation_error`. In-memory `_explanation_cache` cleared on upload. `TopicView` shows explanation; Listen prefers explanation over source text. CSS: `.topic-explanation`, `.topic-notice-error`.
- **2026-08-03 — Kaggle chat client fix:** `kaggle_client.py` — added `_post_json_with_retry()`; `kaggle_chat_completions()` was incorrectly passing `json=` to byte-upload `_post_with_retry()` → `name 'json' unexpected` / broken `/debug/text`.
- **2026-08-03 — Topic build fixes:** Removed dead `get_openai_client()` from `topic_builder.py`. Added JSON repair (trailing commas, unclosed brackets) + one automatic retry with “valid JSON only” follow-up for Llama 1B malformed output.
- **2026-08-03 — Kaggle ops (Class 9 Science textbook test, 301 MB / 1299 pages):** Full PDF still uploads to Kaggle even with “Test first pages only” (page range limits **extract** only). **`EXTRACT_PROVIDER=local`** recommended for laptop dev (skip 76-chunk Kaggle upload; ~3 min local extract for 15–20 pages). **`CHAPTERWISE_TOPIC_CHUNK_PAGES=8`** reduces Llama VRAM per chunk. **`KAGGLE_API_SECRET`** must match notebook cell 2 exactly (placeholder causes 401). **`HF_TOKEN`** + Hugging Face gated access required for Llama (“Expand to review and access” on model page; affiliation can be `Individual`). Set notebook `CHAPTERWISE_LLM_PRELOAD=1`, `CHAPTERWISE_LLM_4BIT=1`, lower `MAX_INPUT_CHARS` / `MAX_NEW_TOKENS` to avoid CUDA OOM on T4. Cloudflare tunnel dies mid-session (502 / error 1033) — refresh URL each notebook run. First successful `/debug/text` after preload: `"ok": true`, `"llm_loaded": true`. Sample born-digital NCERT-style PDF pages extract well with pdfplumber (no GPU OCR needed for v1).
- **2026-08-10 — Extract perf (header/footer):** `pdf_extract.py` header/footer scan was parsing all 1299 pages even for `page_start=1&page_end=10` (~93s of ~98s). Fixed: sample **36 evenly spaced pages** + cache by file fingerprint. Pages 1–10 local extract dropped to ~6–8s. Added `CHAPTERWISE_DEBUG_TIMING=1`. Laptop `CHAPTERWISE_TEXT_EXTRACTOR=rapidocr`, `CHAPTERWISE_RAPIDOCR_FORCE=0`.
- **2026-08-10 — Topic build OOM + tunnel 502:** Llama 1B on T4 OOM during `/chat/completions` when prompt + `max_new_tokens=4096` spiked attention (~13 GB). Pattern: first call often succeeds; **JSON retry** (bad output + full chunk) triggers CUDA OOM or Cloudflare **502 Bad gateway** (tunnel loses Kaggle mid-request). Fixes: `llm_server.py` — defaults `MAX_NEW_TOKENS=1024`, `MAX_INPUT_CHARS=12000`, truncate **all** messages to total budget, assistant cap 2000, `empty_cache()` after generate, `repetition_penalty=1.2`. `topic_builder.py` — retry assistant capped at `CHAPTERWISE_TOPIC_RETRY_CHARS=2000`. `main.py` — `/build_topics` uses `CHAPTERWISE_TOPIC_CHUNK_PAGES` from env (not hardcoded 18). Republished `eakeswar/chapterwise-models`. Topic prompt tightened (Activity rules, section limits, anti-leak example); sanity warnings for bad trees. Tree quality on Class 9 Science OCR still rough (fabricated subsections, parallel “chapters” for 1.1/1.2).
- **2026-08-11 — Kaggle secrets + HF auth:** Notebook cell 2 loads **`KAGGLE_API_SECRET`** and **`HF_TOKEN`** via `UserSecretsClient` (Add-ons → Secrets). Missing `HF_TOKEN` secret caused 401 gated-repo (not HF permissions). Missing/mismatched `KAGGLE_API_SECRET` caused laptop `401 Invalid or missing API token` on `/chat/completions` while `/health` still looked fine. `llm_server` prints `HF auth token present/len` before load; uses `token=` + `huggingface_hub.login`.
- **2026-08-11 — Async chat (tunnel 524):** Sync `/chat/completions` often exceeded Cloudflare free-tunnel ~100s → **524 A timeout occurred** even when Llama finished on Kaggle. Added Kaggle `POST /chat` + `GET /chat/status` + `GET /chat/result`; laptop `kaggle_client.kaggle_chat_completions()` polls like extract. Republished dataset. Prefer **Test first pages only = 8** and `CHAPTERWISE_TOPIC_CHUNK_PAGES=4` while validating. End-to-end build still needs user confirmation after attaching latest dataset version.
- **2026-08-12 — Kaggle chat reliability (Plan A):** Busy chat → 409 + `job_id`; client retries 409/502; no sync fallback except 404/405; generate `max_time` + StoppingCriteria; chat job watchdog + `/chat/reset`. Topic-sized Llama prompts (~12.5k chars) still hang on T4; tiny/medium chats OK (~2s).
- **2026-08-12 — Regex topic builder (Plan B):** Default `CHAPTERWISE_TOPIC_BUILDER=regex` — heading regex TOC, no Llama. Body text kept via page ranges (approach a); empty-range fallback chapter (approach b). Pages 1–8 Class 9 Science: ~21 topics in 0.01s. OCR title spill / two-column order issues documented; fix deferred pending user decision. LLM path: `CHAPTERWISE_TOPIC_BUILDER=llm`.
- **2026-08-13 — Regex TOC confirmed live:** `CHAPTERWISE_TOPIC_BUILDER` is often **unset** in `.env` (code default `regex`, not leftover `llm`). Pages 1–8 Class 9 Science: `POST /build_topics` → HTTP 200, `builder=regex`, `heading_count=20`, `topic_count=21` in ~0s. `GET /debug/topics` populated. OCR title spill still present (`Matter of sa`, truncated titles, `1.4.1 EFFECT` before `1.4`).
- **2026-08-13 — Dead Kaggle must not block `/topic`:** With `TEXT_PROVIDER=kaggle` and a dead tunnel, `GET /topic/{id}` hung until `CHAPTERWISE_LLM_TIMEOUT` (600s) because `_kaggle_start_chat_job` retried DNS/connection errors in a loop. **Fix:** after `_post_json_with_retry` exhausts retries, raise `Kaggle chat unreachable` immediately. `topic_detail` still returns HTTP 200 with `explanation_status=failed`, source text + images (~3s). UI already shows the failed-explanation notice.
- **2026-08-13 — Azure chat on a separate resource:** New Foundry resource `chapterwise-text` (user’s account). Chat uses **`AZURE_CHAT_API_KEY`** + **`AZURE_CHAT_BASE_URL`** (must include `/openai/v1`). TTS/images keep `AZURE_OPENAI_API_KEY` + services endpoint. Laptop `TEXT_PROVIDER=azure`, `AZURE_TEXT_DEPLOYMENT=gpt-4.1-mini`. Missing `/openai/v1` on the chat URL → 404 Resource not found.
- **2026-08-13 — Extract/TOC stay local:** Kaggle T4 GPUs run **Llama only**; PyMuPDF/RapidOCR + regex TOC are CPU. Do not move extract or regex tree to Kaggle to “use the 2 GPUs.” Optional later: Kaggle CPU extract + regex colocation to avoid pulling full extract JSON through the tunnel (`topic_builder.py` is not in the Kaggle dataset bundle yet).
- **2026-08-14 — Full-book local extract:** Class 9 Science 301 MB / 1299 pages, `EXTRACT_PROVIDER=local`, RapidOCR, 16 thread workers. First run (eager b64, RapidOCR init race): **286.91s** extract, peak private **~2.44 GB**. After defer-b64 + single RapidOCR init: **~209s** extract on a later full run. `images_extract_encode` timing still includes `get_images`/`get_image_rects` even when b64 is deferred.
- **2026-08-14 — Defer image b64 locally + RapidOCR init lock:** `main.py` `os.environ.setdefault("CHAPTERWISE_DEFER_IMAGE_B64", "1")` after `.env` load (`.env` can still set `0`). `_get_rapidocr_pdf()` constructs `RapidOCRPDF` once under `_RAPIDOCR_LOCK`.
- **2026-08-14 — On-demand image encode:** `GET /topic/{id}` hydrates missing `url`s via `encode_xref_data_url()` for that topic’s xrefs only; in-memory cache `{pdf fingerprint}:{xref}`. `jpx`/`jp2`/etc. converted to PNG (Separation/CMYK pixmaps go through RGB). Gallery **excludes `isBackground`** (page-2 Activity 1.3/1.4/1.5 duplicate galleries were only backgrounds). Images still attach by **page range only**; heading `start` char offsets are dropped after TOC build. Same-page sibling non-bg sharing still exists but is rare on pp.1–8 (e.g. one figure on p.3 shared by Activity 1.6/1.7/1.8). Position-based split **not built**.
- **2026-08-14 — Step 5 Q&A:** `qa.py` `answer_question()` + `POST /ask` `{ topic_id, question }`. Context = cached explanation (if any) + page-range source. System prompt: answer only from context, else say this section does not have enough information. HTTP 200 + `status=failed` on provider error (same fail-fast as `/topic`). `AskPanel` in `TopicView`. Smoke: Activity 1.1 → “Panch Tatva”; boiling point of water refused.
- **2026-08-14 — Full-book TOC + UI:** `/build_topics` on 1299 pp → **223 topics / 214 headings / 0.16s**. Regex false `Chapter N` matches → 37 chapter nodes, duplicate/out-of-order titles (e.g. several “Chapter 6”, last “Chapter 8” at pp.1290–1299 is likely answers/appendix). Start/mid/end `/topic` + `/ask` + Listen worked (Activity 1.1 p.1, Activity 8.4 p.650, ch37 pp.1290–1299). TTS default voice **`shimmer`**.
- **2026-08-14 — TTS voice:** `tts.py` `DEFAULT_VOICE = "shimmer"` (female). **Superseded 2026-08-20:** Listen now sends the picker voice (still defaults to shimmer).
- **2026-08-17 — Black gallery images (SMask):** Page-1-style 1-bit ICC Gray + **SMask** decoded as raw PNG looked solid black (artwork in the mask). `encode_xref_data_url()` now Pixmap-decodes unsafe/SMask/low-bpc/CMYK/Separation images to RGB PNG with mask as alpha.
- **2026-08-17 — Step 7 on-demand generate:** `/topic` Lanczos for informational+low quality; decorative stays original unless cache already has `kind=generated`. **`POST /topic/{id}/images/{image_id}/generate`** — 400 if not decorative (`InformationalImageError`); Azure/timeout → HTTP 200 `{status: failed}`. Size `1024x1024`. Gallery: Generate HD / Show original. Do **not** auto-run gpt-image-2 on topic open (pages with many decorative images, e.g. p174).
- **2026-08-17 — Generate prompt must follow the photo:** Vision caption describes **only** the attached pixels (no topic). gpt-image-2 prompt: redraw that caption; topic title is **style only**. Negatives: no people/vehicles/animals/tools not in the caption. Including chapter explanation as subject turned “wheat grains” (p684) into a farm scene when generated from **`ch20` (Chapter 12)** instead of `ch20_s1_ss1`.
- **2026-08-17 — Baked-in diagram labels:** Color labeled figures (p68 `p68_i0` XYLEM / `p68_i1` PHLOEM) classified decorative because overlap=0, aspect below 2.2, chroma above 25. Labels are **raster pixels**, not PDF text. **RapidOCR on the cropped image** (lazy, background, max side 512): at least one string (score ≥ 0.5, length ≥ 2) → force informational + Lanczos; photos like p684 wheat stay decorative. Eager full-book OCR is hours — **do not** run OCR on every extract image.
- **2026-08-17 — Classify must not block `/topic`:** Would-be-decorative tiles return `classify_status=pending`, `enhance_role=informational` (Generate HD **blocked** until OCR finishes — fail conservative). 2-worker pool; cache by source hash. `TopicView` polls `/topic` every 3s. Cheap informational (wide aspect / gray line-art) skips OCR.
- **2026-08-17 — Frontend polish:** `react-markdown` via `MarkdownText.jsx` for explanation + Ask answers (TTS still plain text). Topic tree viewport scroll (see §5). `/build_topics` page range uses **extracted PDF page numbers** so a 172–176 slice is not remapped to “Pages 1–5”.
- **2026-08-19 — Topic view polish (`4ba796f`):** Docked ask panel at bottom of topic view (scroll body + pinned footer); **section cards** + stronger `h2` hierarchy; primary **Listen** styling; removed `.topic-view { max-width: 920px }` so main panel spans full width; **ChatGPT-style composer** (`.ask-composer`, auto-grow textarea, send icon inside bordered box).
- **2026-08-19 — Ask layout experiments (reverted except C):** Tried three right-panel patterns on same stacked middle content — **(A)** top tabs (Explanation/Source/Images/Ask) + slim bottom audio bar; **(B)** persistent ~380px Ask column (+ drawer below 1200px); **(C)** **current:** full-width middle + floating **FAB** + overlay drawer (no fixed third column). Commits: `4ba796f` (dock baseline) → `0e6ed8d` (FAB drawer + `AskComposer` refactor + chat thread).
- **2026-08-19 — Floating Ask drawer (current UI, `0e6ed8d`):** `.ask-fab` fixed bottom-right; `.ask-drawer` overlays content (backdrop, ×, or FAB toggle to close); `AskPanel` `turns[]` thread with pending/ready/failed states; topic switch resets drawer open state + chat history.
- **2026-08-20 — Streamed explanation + Ask:** `GET /topic/{id}` no longer calls the LLM (returns `pending` unless cached). `GET /topic/{id}/explanation/stream` and `POST /ask` are **SSE** (`event: delta|done|error`). Azure `chat.completions.create(stream=True)`; Kaggle still one-shot (yields the full string). Cache explanation only after a complete `done`. Fail-soft: partial text + “may be incomplete”; truncated text is **not** cached. Frontend: raw text + blinking cursor while streaming; `MarkdownText` when ready. Classify-poll `GET /topic` mid-stream must use `mergeTopicPoll` (polls return `explanation: null` until cache is ready). Debug: `CHAPTERWISE_STREAM_FAIL_AFTER_CHARS`, `CHAPTERWISE_STREAM_CHUNK_DELAY_MS` (leave unset in normal `.env`).
- **2026-08-20 — TTS voice picker:** Confirmed `gpt-4o-mini-tts` voices (all 200 MP3): alloy, echo, fable, onyx, nova, **shimmer**, coral, ash, sage, verse. `tts.py` `TTS_VOICES` allowlist; UI select defaults to **shimmer**; `synthesizeSpeech(text, voice)` sends `voice` on `POST /tts`. Voice state lives in `TopicView` (not reset on topic switch).
- **2026-08-20 — Source text collapsed by default:** Header `Source text (N chars)` + `▸`/`▾`; click to expand the existing `<pre className="topic-source">`. `sourceOpen` resets on topic switch.
- **2026-08-20 — Generate HD uses `images.edit`:** Live probe: `client.images.edit` on edzhub `gpt-image-2` (`…/openai/v1/images/edits`) returns b64 PNG (`input_fidelity=high` accepted). `generate_decorative_image(..., source_bytes=)` edits the real photo with a quality-only prompt; vision caption + `images.generate` only if edits errors. Smoke: page-3 tribal-dress group and page-684 grain pile stayed subject-faithful (not generic dress / not a farm scene). Samples in `backend/test_output/hd_compare/` (gitignored). `chapterwise-text` is chat-only — images stay on edzhub.

---

## 7. Environment variables (`backend/.env`)

Azure OpenAI uses **Foundry v1 API** — `OpenAI` class with `base_url` + `api_key`; pass **deployment name** as `model=`. Do **not** use `AzureOpenAI` with `api_version`.

```env
# TTS + images (edzhub / services endpoint)
AZURE_OPENAI_API_KEY=...
AZURE_SERVICES_BASE_URL=https://edzhub-resource.services.ai.azure.com/openai/v1
AZURE_TTS_BASE_URL=https://edzhub-resource.services.ai.azure.com/openai/v1
AZURE_TTS_DEPLOYMENT=gpt-4o-mini-tts
AZURE_IMAGES_BASE_URL=https://edzhub-resource.services.ai.azure.com/openai/v1
AZURE_IMAGE_DEPLOYMENT=gpt-image-2

# Chat — separate resource (chapterwise-text); do not reuse TTS/image key
TEXT_PROVIDER=azure
AZURE_CHAT_API_KEY=...
AZURE_CHAT_BASE_URL=https://YOUR-RESOURCE.openai.azure.com/openai/v1
AZURE_TEXT_DEPLOYMENT=gpt-4.1-mini
CHAPTERWISE_TOPIC_BUILDER=regex
CHAPTERWISE_LLM_TIMEOUT=600

CHAPTERWISE_CORS_ORIGINS=http://localhost:5173,http://127.0.0.1:5173
CHAPTERWISE_TOPIC_CHUNK_PAGES=4
CHAPTERWISE_TOPIC_RETRY_CHARS=2000
CHAPTERWISE_MAX_PDF_BYTES=524288000
CHAPTERWISE_EXTRACT_WORKERS=16
CHAPTERWISE_TEXT_EXTRACTOR=rapidocr
CHAPTERWISE_RAPIDOCR_FORCE=0
CHAPTERWISE_DEBUG_TIMING=0
# Skip eager image base64 during extract (encode on /topic). Set 0 to restore old behavior.
CHAPTERWISE_DEFER_IMAGE_B64=1
# Stream debug only (unset in normal use): fail after N chars; delay ms per chunk
# CHAPTERWISE_STREAM_FAIL_AFTER_CHARS=80
# CHAPTERWISE_STREAM_CHUNK_DELAY_MS=50
BLUR_VARIANCE_THRESHOLD=100
SMALL_IMAGE_MIN_DIM=400

# Kaggle remote extract (optional)
KAGGLE_ENABLED=true
KAGGLE_API_BASE_URL=https://your-tunnel.trycloudflare.com
KAGGLE_API_SECRET=...
EXTRACT_PROVIDER=local
KAGGLE_EXTRACT_TIMEOUT=900
KAGGLE_CHUNK_TIMEOUT=120
KAGGLE_UPLOAD_CHUNK_BYTES=4194304
KAGGLE_CHUNK_THRESHOLD_BYTES=10485760
KAGGLE_UPLOAD_RETRIES=3
KAGGLE_POLL_INTERVAL=5
KAGGLE_POLL_TIMEOUT=90
KAGGLE_RESULT_TIMEOUT=300
KAGGLE_USERNAME=...
KAGGLE_API_TOKEN=...
KAGGLE_DATASET_SLUG=chapterwise-models

# Kaggle notebook only (not laptop .env unless testing locally)
# HF_TOKEN=hf_...
# CHAPTERWISE_LLM_MODEL=meta-llama/Llama-3.2-1B-Instruct
# CHAPTERWISE_LLM_PRELOAD=1
# CHAPTERWISE_LLM_4BIT=1
# CHAPTERWISE_LLM_MAX_INPUT_CHARS=12000
# CHAPTERWISE_LLM_MAX_NEW_TOKENS=1024
# CHAPTERWISE_LLM_ASSISTANT_CAP_CHARS=2000
# PYTORCH_ALLOC_CONF=expandable_segments:True
```

Frontend `.env` optional: `VITE_API_BASE_URL=http://127.0.0.1:8766`

---

## 8. Explicit non-goals for v1 (flag as `# TODO v2`)

- No **Azure** chat fallback when `TEXT_PROVIDER=kaggle` — if Kaggle text is selected, do not silently use the TTS/image key for chat. Current default is `TEXT_PROVIDER=azure` (separate `AZURE_CHAT_*` creds); TTS/images stay on the edzhub services endpoint.
- No karaoke word highlighting — play/pause audio per topic only
- No image **generation** for formula/diagram images — Lanczos upscale only for informational images; `gpt-image-2` only for small/blurry **decorative** images (step 7)
- No multi-user auth — single active document
- No vector DB — topic Q&A uses raw topic text in prompt context
- No disk caching — in-memory only
- No OpenAI for full-book PDF parsing (local extract first; hybrid vision fallback is v2)

---

## 9. Remaining implementation notes (steps 4–7)

### `/topic/{topic_id}` + `TopicView`

- ✅ Join topic node from `topics_by_id` with page-range slice of `_active_extraction`
- ✅ Source text + image gallery in `TopicView`
- ✅ LLM explanation via SSE stream (`pending` on GET until cached; `ready` | `skipped` | `failed` | `incomplete`)
- ✅ Listen button via `POST /tts` (narrates explanation, falls back to source text); voice picker default shimmer
- ✅ Deferred extract b64; `/topic` encodes on demand; gallery skips `isBackground`
- ✅ Images run through `image_enhance.py` on `/topic` (cheap classify + background baked-text OCR; Lanczos for informational+low quality)
- ✅ Source text card collapsed by default
- ✅ On-demand decorative HD: `POST /topic/{id}/images/{image_id}/generate` → `images.edit`
- ⬜ Optional: assign non-bg images to same-page sibling topics via heading offset vs image `y` (deferred — rare on Class 9 pp.1–8)
- `# TODO v2`: richer formula vs decorative classification; compute `overlapWordCount` on RapidOCR extract path (PyMuPDF words) if we want overlap as an informational shortcut again

### `/ask` + `AskPanel`

- ✅ `qa.py` + SSE `POST /ask` + `AskPanel` (explanation + source in context; refuse outside topic)
- ⬜ No Q&A cache (v1: every question is a fresh call)

### `/tts` + Listen

- ✅ `tts.py` + `POST /tts` on `AZURE_TTS_BASE_URL` (services endpoint)
- ✅ Listen button in `TopicView` (explanation text, fallback to source text)
- ✅ Default voice **`shimmer`**; UI picker + `TTS_VOICES` allowlist
- ⬜ Optional: `POST /tts` with `{ topic_id }` after explanation exists

### `image_enhance.py` (§4a)

- ✅ `generate_decorative_image()` via `gpt-image-2` **`images.edit`** (source PNG, `input_fidelity=high`, size `1024x1024`); `images.generate` fallback only
- ✅ Cheap classify: aspect ≥ 2.2 or gray/line-art (mean chroma under 25) → informational. **`overlapWordCount` is not a gate** (RapidOCR extract always stores 0).
- ✅ Baked-text OCR (RapidOCR on image pixels, downscaled max 512): if would-be decorative, run **in background**; ≥1 decent string → informational. Until `classify_status=ready`, treat as informational (no Generate HD).
- ✅ Quality: small (`SMALL_IMAGE_MIN_DIM=400`) or blurry (Laplacian under `BLUR_VARIANCE_THRESHOLD=100`)
- ✅ Route: informational + low quality → **Lanczos**; decorative → original until user **Generate HD** (never gpt-image-2 formulas/diagrams)
- ✅ Edit prompt: quality-only (sharpen/upscale; do not change people, clothes, patterns, poses). Vision+generate fallback still uses caption-primary + topic **title** as style only
- ✅ In-memory `_ENHANCE_CACHE` keyed by source SHA-256 (includes classify result; decorative originals are cached so OCR does not rerun)
- Do **not** auto-generate decorative images on `/topic` (too many per page; Azure cost + latency)

---

## 10. Coding preferences

- Python 3.11+, FastAPI, async where possible
- pip installs → `backend/vendor/` via `--target`
- Print logging OK for v1
- **Ask for approval before editing existing files** unless the user explicitly says to proceed/build/implement
- Never commit `backend/.env`, `backend/vendor/`, `backend/active_doc.pdf`
- Never hardcode API keys in source
- Keep changes scoped; match existing style
- Frontend: minimal CSS (no Tailwind in chapterwise v1), functional components only

---

## 11. Known issues & ops

- **Text/chat:** Default `TEXT_PROVIDER=azure` uses `AZURE_CHAT_API_KEY` + `AZURE_CHAT_BASE_URL` + `AZURE_TEXT_DEPLOYMENT=gpt-4.1-mini`. Verify with `GET /debug/text`. `TEXT_PROVIDER=kaggle` uses Kaggle async `/chat` (poll); `auto` falls back to Azure if Kaggle text fails. Dead Kaggle must fail fast (not spin until `CHAPTERWISE_LLM_TIMEOUT`) so `/topic` still returns source + images.
- **Azure TTS/images:** Still `AZURE_OPENAI_API_KEY` on `services.ai.azure.com`. Verify with `GET /debug/azure`. Default TTS voice is **`shimmer`**. Do not send chat requests with the TTS/image key.
- **Azure chat URL:** `AZURE_CHAT_BASE_URL` **must include `/openai/v1`**. Host-only URL → 404 Resource not found. Restart backend after `.env` edits.
- **Kaggle tunnel:** Free `trycloudflare.com` URLs are ephemeral — refresh `KAGGLE_API_BASE_URL` each notebook run. **`/health` is fast and unauthenticated**. Long sync `/chat/completions` can fail with **524** (timeout) or **502**; use async `/chat` polling on latest dataset. `KAGGLE_API_SECRET` must **exactly match** notebook UserSecrets (placeholder/`PASTE_…` → 401 on chat). Prefer **`EXTRACT_PROVIDER=local`** for fast dev; use `auto` or `kaggle` for full offloaded extract.
- **Kaggle Llama gated model:** Accept license on https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct (“Expand to review and access”); set `HF_TOKEN` via Kaggle Add-ons → Secrets. 401 with `present=False` → secret not loaded; 401 with `present=True` → HF access not granted for that token’s account.
- **Kaggle Llama runtime:** Use `CHAPTERWISE_LLM_PRELOAD=1`. CUDA OOM on T4 → `CHAPTERWISE_TOPIC_CHUNK_PAGES=4` or `8`, `CHAPTERWISE_LLM_MAX_INPUT_CHARS=12000`, `CHAPTERWISE_LLM_MAX_NEW_TOKENS=1024`, republish dataset after `llm_server.py` / `chapterwise_gpu_server.py` changes. Llama 1B may return malformed topic JSON → repair + retry in `topic_builder.py` (retry assistant capped at 2000 chars).
- **Large PDFs (e.g. 300 MB):** Browser sends full file to laptop every upload. “Test first pages only” limits **extract** page range, not upload size. Kaggle path uploads all 76×4 MB chunks before extract. Local **full-book** extract of Class 9 Science (1299 pp / 301 MB) ≈ **3.5 min** with defer-b64 + RapidOCR (was ~4.8 min with eager b64). Partial 15–20 pages ≈ 3–5 min was the old pdfplumber-era figure — RapidOCR + header-sample cache is much faster for small ranges (~6–8s for pp.1–10).
- **Full-book regex TOC:** 223 topics in 0.16s, but many false `Chapter N` headings (duplicate/out-of-order chapter nodes). Title spill / two-column OCR order still present on early pages.
- **`overlapWordCount` is unused on laptop RapidOCR extracts** (2732/2732 were 0 on Class 9 Science). Color diagrams with labels painted into the JPEG (p68 xylem/phloem) need baked-text OCR, not PDF overlap.
- **Generate HD (edits path, 2026-08-20):** Prefer Generate HD after classify is ready. Generations-only used to invent scenes (p684 wheat → farm; costume photos → generic dress). Current path sends source bytes to `images.edit`. Still avoid generating from a whole-chapter node when a tighter subsection exists.
- **Streaming + classify poll:** `GET /topic` during OCR poll returns `explanation: null` until the stream caches. Frontend must merge; do not `setDetail(pollPayload)` raw.
- Only one upload/build at a time (`_extraction_lock`); backend blocks `/health` during active job
- **Port 8766 in use:** Stop prior `python backend/main.py` (Ctrl+C) before restarting
- Restart backend after `.env` changes; restart clears in-memory extraction/topics/explanations
- Frontend dev server: port **5173** — must be running separately (`npm run dev`)
- After Kaggle code changes: republish dataset + re-run notebook cells 1–2; copy new tunnel URL to `.env`
- Kaggle notebook pip warnings (`dopamine-rl`/`gym`, `gradio`/`pillow`) are harmless

---

## 12. Next step when user says “continue”

**Next:** (1) Tighten regex TOC for full-book false `Chapter N` hits / title spill / two-column order. (2) Optional Ask UX tweaks (FAB drawer is current). (3) Optional: compute `overlapWordCount` on RapidOCR extract path. (4) Optional later: same-page non-bg image split by heading offset vs `y`. (5) `# TODO v2`: skip Kaggle upload when page range set; background extract; disk cache.

**Dev defaults (2026-08-20):** `EXTRACT_PROVIDER=local`, `CHAPTERWISE_TOPIC_BUILDER=regex`, `TEXT_PROVIDER=azure` (`gpt-4.1-mini` via `AZURE_CHAT_*`), `CHAPTERWISE_TEXT_EXTRACTOR=rapidocr`, `CHAPTERWISE_DEFER_IMAGE_B64=1`, TTS default **`shimmer`** (picker), decorative HD = **on-demand `images.edit`**, Ask UI = **FAB + overlay drawer**, explanations/Ask **SSE-streamed**. Kaggle optional.