osdb-1-claude-opusplan/docs/planning/specs/05-import-publications.md

Spec: 05 · Import Existing Publications

Taiga Story: #5 · 05-import-existing-publications Status: Ready Branch: feature/05-import-publications Depends on: 03-import-fruit-db (fruits must exist), 04-fruit-publications (publications tables)

---

1. Goal

A standalone Python script imports all publications from 03-data/osws.xml, their cover images, linked fruits (determined by filesystem scan), PDFs, and fruit images into the database.

---

2. Background / Scope / Context / Constraints

• Separate file from the fruit import script — do not merge.
• Publication identified by XML <obj> element where <osw> = 1.
• Fruit linking is NOT derived from obstsorten.xml tags (unreliable subset) but from filesystem scan of 03-data/osdb/{publicationId}/.
• Image existence is not guaranteed — skip silently if file not found.
• author field: skip if value is ".".
• Cover image path given by <img> tag content (relative to 03-data/).

File patterns in 03-data/osdb/{publicationId}/

Pattern	Import target
{fruitId}_{publicationId}_s0.jpg	publication_fruit_images
{fruitId}_{publicationId}.pdf	publication_descriptions

• Collect unique fruitId values from both patterns → union = linked fruit set.
• Look up each fruitId against fruits.osdb_number → skip if no match.
• Link matched fruits via publication_fruits.

---

3. Necessary Refactorings

• Script must run AFTER import_fruits.py (fruits table populated).
• Idempotent: use INSERT … ON CONFLICT (osdb_pub_id) DO UPDATE for publications; delete cascade clears linked data before re-import if re-run.

---

4. Related Data Model

Tables written (all created by story #4 migration):

• publications
• publication_fruits
• publication_descriptions
• publication_fruit_images

---

5. Affected Places / Related Components

scripts/
  import_fruits.py          ← existing (story #3)
  import_publications.py    ← new
  README_import.md          ← update with run order and env var requirements

Script: scripts/import_publications.py

Steps:

1. Parse 03-data/osws.xml; iterate <obj> where <osw> = 1
2. For each publication: a. Extract id, name, author (skip if ".") b. Resolve cover image: 03-data/{img_tag_value} — read bytes; skip if missing c. UPSERT into publications (conflict on osdb_pub_id) d. Scan 03-data/osdb/{id}/ for *_s0.jpg and *.pdf files e. Collect unique fruit IDs from filenames f. Resolve fruit IDs → fruits.id via osdb_number; skip unresolved g. Delete existing publication_fruits for this pub; re-insert linked fruits h. Delete existing publication_descriptions for this pub; import PDFs i. Delete existing publication_fruit_images for this pub; import images
3. Log: publications upserted, cover images loaded, fruits linked, PDFs imported, images imported, skips

DB from DATABASE_URL env var.

---

6. Out of Scope

• Modifying publication CRUD API
• GUI import trigger
• Fruit import (handled by script #3)

---

7. Verification

Check	Expected
Script runs to completion	Exit 0, logs summary counts
Re-run is idempotent	Same final row counts
Publication with osw != 1	Not imported
Author "."	Stored as NULL
Cover image file missing	Skipped, publication still imported
Fruit ID from filename not in fruits table	Skipped with log warning
GET /api/v1/publications after import	Returns populated list
Spot-check: known publication	Correct title, linked fruits visible

---

8. Open Questions

#	Question	Impact
1	Exact XML element structure in osws.xml? (tag names, nesting)	Parser field mapping
2	Can 03-data/osdb/{id}/ be missing entirely for some publications?	Directory scan guard
3	Re-import strategy: delete-then-insert vs upsert per file?	Idempotency
4	Encoding of XML file (UTF-8 or Latin-1)?	ElementTree encoding param