feat: Phase 8 TNCSC data migration scripts (members, students, opening balances)

Adds migrate_students.py (parent+child partner upsert, current-term
enrollment by school level code) and migrate_opening_balances.py (one
posted, balance-checked journal entry dated the last fiscal year-end).
All three Phase 8 scripts (incl. the existing migrate_members.py) verified
against a live Odoo 19 instance: fresh tncsc_migration_test DB with
tncsc_deployment installed, dry-run + real run + idempotency re-run each,
confirmed via RPC no duplicate records were created. See HANDOFF.md for
the full verification log and why a fresh DB was used instead of
communityos_dev.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

HANDOFF.md

@@ -17,7 +17,7 @@ file is the "where things actually stand" companion to it.
 | 5 | `community_interac` (Interac e-Transfer payment provider) | ✅ Done, committed |
 | 6 | `community_theme_base` + `community_portal` | ✅ Done, committed |
 | 7 | `tncsc_deployment` (client config layer) | ✅ Done, committed |
-| 8 | TNCSC data migration scripts | 🟡 Barely started, **not committed** — see below |
+| 8 | TNCSC data migration scripts | ✅ Written and verified live, **not committed** — see below |
 | 9 | Packaging for resale + production go-live | ⬜ Not started |
 
 Every phase 0–7 commit was verified against a **live** Odoo 19 + Postgres 16
@@ -58,32 +58,99 @@ docker compose up -d
 
 ## Phase 8 — exactly where it was left off
 
-Two files exist, committed as WIP (see the `wip:` commit), inert (not
-imported by anything, not installed, no manifest references them):
+All three migration scripts exist and have been verified against a **live**
+Odoo instance, but **the work is not committed yet** (see "To pick this back
+up" below for why).
 
 - `scripts/_migration_common.py` — shared JSON-RPC helpers (auth, execute_kw,
   CSV normalization, a `MigrationReport` class) for the migration scripts.
-  Finished, not yet tested against anything.
-- `scripts/migrate_members.py` — first pass at the member migration script
-  (idempotent upsert by email, dry-run flag, tier lookup by code, report
-  CSV). Written but **not tested at all** — no live run against Odoo yet.
+- `scripts/migrate_members.py` — idempotent upsert by email, dry-run flag,
+  tier lookup by code, report CSV.
+- `scripts/migrate_students.py` — idempotent upsert of parent partner (by
+  email, reusing a partner created by `migrate_members.py` if one exists)
+  + child partner + `community.school.student`, matched by (parent, student
+  name) since students don't have their own email in the source data.
+  Enrolls into the matching class in the current open term (matched by
+  `community.school.level` code) when one exists; otherwise still
+  creates/updates the student record and reports why no enrollment
+  happened.
+- `scripts/migrate_opening_balances.py` — posts one `account.move` in the
+  Miscellaneous Operations journal, dated the last fiscal year-end
+  (computed from `res.company.fiscalyear_last_day/month`, or `--date`
+  override). Refuses to post anything if the CSV doesn't balance
+  (`sum(debit) != sum(credit)`) or references an unknown account code —
+  checked before any Odoo write happens. Idempotent via a fixed `ref`
+  ("Opening Balances Import"): re-running finds the existing posted entry
+  and does nothing.
 
-Still to do for Phase 8 (see the plan's Phase 8 section):
-- `scripts/migrate_students.py` — not started.
-- `scripts/migrate_opening_balances.py` — not started.
-- Test all three against synthetic/sample CSV data on a live instance,
-  the same way `scripts/migrate_classroom.py` was tested (see that file
-  and Phase 3's commit for the pattern: dry-run first, then a real run,
-  then a second real run to prove idempotency).
-- Real TNCSC member/student data doesn't exist yet — these scripts were
-  intentionally designed against a *documented, assumed* CSV schema (see
-  each script's docstring) since the real export format isn't known yet.
-  Confirm the assumed column names match the real export before relying on
-  them, or adjust the scripts to match.
+### How each was tested
 
-To pick this back up: either continue `migrate_members.py`'s testing and
-write the other two scripts, or decide the assumed CSV schemas are wrong
-and redesign first once real export samples exist.
+Followed the same pattern as `scripts/migrate_classroom.py` (Phase 3):
+dry-run first, then a real run, then a second real run to prove
+idempotency — against a **fresh** database, not `communityos_dev`. See
+"Why a fresh DB, not communityos_dev" below for why.
+
+1. `docker compose up -d`, then created a throwaway DB with both product
+   modules and the client layer: `odoo -d tncsc_migration_test -i
+   community_membership,tncsc_deployment --stop-after-init` (installing
+   `tncsc_deployment` alone pulls in `community_school` and everything else
+   as dependencies). Restarted the container afterward.
+2. `migrate_members.py`: synthetic CSV at
+   `data/raw/members_export_sample.csv` (gitignored, left in place as a
+   fixture) with 4 valid rows + 2 deliberately-bad rows (missing email,
+   unknown tier code). Dry-run matched real-run output; second real run
+   reported `updated` not `created` for all 4; confirmed via RPC that
+   exactly one `res.partner` per email existed after both runs and
+   `membership_state` was `active`.
+3. `migrate_students.py`: needed a `community.school.term` (state=open),
+   `community.school.level` (code=`BEG`), and `community.school.class`
+   seeded first (the fresh DB has none) — also had to grant the `admin`
+   user the School Coordinator group, since a fresh install doesn't put
+   admin in it. Synthetic CSV at `data/raw/students_export_sample.csv`
+   with 3 valid rows (2 matching the open term's class, 1 with no
+   `level_code`) + 1 deliberately-bad row (missing parent). Confirmed via
+   RPC after 2 runs: exactly 3 `community.school.student` records, exactly
+   2 `community.school.enrollment` records (no duplicates), parent
+   partners correctly reused from step 2 instead of duplicated.
+4. `migrate_opening_balances.py`: synthetic CSV at
+   `data/raw/opening_balances_sample.csv` using real account codes from
+   the fresh DB's TNCSC chart of accounts (111100 Cash, 112110 Trade AR,
+   322000 Retained Earnings as the balancing line). Verified the
+   out-of-balance guard refuses to post (tested with a deliberately
+   mismatched debit/credit CSV). Real run posted one balanced, posted
+   `account.move`; second run detected the existing entry and did nothing.
+   Confirmed via RPC: exactly one move, three correctly-valued lines, AR
+   line correctly linked to the migrated Ravi Kumar partner.
+
+Real TNCSC member/student/trial-balance data doesn't exist yet — all three
+scripts were intentionally designed against a *documented, assumed* CSV
+schema (see each script's docstring) since the real export formats aren't
+known yet. Confirm the assumed column names match the real exports before
+relying on them, or adjust the scripts to match.
+
+### Why a fresh DB, not `communityos_dev`
+
+`communityos_dev` doesn't have `tncsc_deployment` installed (see the top of
+this file — it can't be, since it already has posted journal entries and
+Odoo refuses to change a company's currency once those exist). Testing the
+Phase 8 scripts there would exercise generic `community_membership`/
+`community_school` behavior, not the actual TNCSC-configured target
+(TNCSC's real chart of accounts, company currency, groups). Since Phase 8's
+entire purpose is rehearsing the real TNCSC data load, a fresh DB with
+`tncsc_deployment` installed is the more representative target — and it
+avoids adding more synthetic test data to the one long-lived dev DB used
+across every other phase. `tncsc_migration_test` was **not** dropped after
+this session (unlike the Phase 7 `tncsc_freshN` throwaways) — it's left in
+place in case the next session wants to keep testing against it; drop it
+before it's mistaken for anything resembling real data.
+
+### To pick this back up
+
+The scripts work and are proven idempotent, but haven't been committed —
+do that first (a `feat: Phase 8 migration scripts` commit, following the
+same "verified against a live container before committing" convention as
+every other phase), then either wait for real TNCSC export samples to
+validate the assumed CSV schemas against, or move on to Phase 9.
 
 ## Phase 9 — not started
 

scripts/migrate_opening_balances.py

@@ -0,0 +1,205 @@
+#!/usr/bin/env python3
+"""Post TNCSC's opening balances as a single journal entry dated the last
+fiscal year-end, in the company's Miscellaneous Operations journal.
+
+Input CSV columns (source system export - typically a trial balance dump;
+adjust the header names below to match the real export once available):
+
+    account_code,debit,credit,label,partner_email
+
+  - account_code: required, must match an existing account.account code for
+    the target company.
+  - debit / credit: each optional but exactly one of the two must be a
+    positive number per row (standard opening-balance convention: never
+    both, never neither).
+  - label: optional line label (defaults to "Opening Balance - <account
+    name>").
+  - partner_email: optional, links the line to a res.partner (useful for
+    AR/AP opening balances). Looked up by email; left blank if not found or
+    not given.
+
+The CSV must balance: total debit must equal total credit across all rows,
+or the script refuses to post anything (this is checked before any Odoo
+write happens).
+
+Behaviour:
+  - Idempotent: the entry is created with a fixed ref
+    ("Opening Balances Import") in the Miscellaneous Operations journal. If
+    a posted entry with that ref already exists in that journal dated the
+    same day, the script does nothing and reports it as already done -
+    re-running never creates a duplicate.
+  - The entry date is the most recently completed fiscal year-end before
+    today, computed from the company's fiscal year configuration, unless
+    --date overrides it.
+  - Draft entry is created first, then posted (action_post) - so a failure
+    partway through leaves nothing posted.
+
+Usage:
+    python scripts/migrate_opening_balances.py \\
+        --url https://your-odoo-host --db communityos_dev \\
+        --username admin --password admin \\
+        --csv data/raw/opening_balances.csv \\
+        [--date 2025-12-31] [--dry-run]
+"""
+import argparse
+import csv
+import sys
+from datetime import date
+from pathlib import Path
+
+from _migration_common import authenticate, execute_kw, normalize_email
+
+OPENING_BALANCE_REF = 'Opening Balances Import'
+
+
+def read_rows(csv_path):
+    with open(csv_path, newline='', encoding='utf-8') as handle:
+        yield from csv.DictReader(handle)
+
+
+def parse_amount(value):
+    value = (value or '').strip()
+    if not value:
+        return 0.0
+    return float(value.replace(',', ''))
+
+
+def last_fiscal_year_end(last_day, last_month, today):
+    candidate = date(today.year, last_month, last_day)
+    if candidate >= today:
+        candidate = date(today.year - 1, last_month, last_day)
+    return candidate.isoformat()
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    parser.add_argument('--url', required=True)
+    parser.add_argument('--db', required=True)
+    parser.add_argument('--username', required=True)
+    parser.add_argument('--password', required=True)
+    parser.add_argument('--csv', required=True, type=Path)
+    parser.add_argument('--date', help="Override the entry date (YYYY-MM-DD); default is the last fiscal year-end")
+    parser.add_argument('--dry-run', action='store_true')
+    args = parser.parse_args()
+
+    if not args.csv.exists():
+        print(f"CSV file not found: {args.csv}", file=sys.stderr)
+        return 1
+
+    uid = authenticate(args.url, args.db, args.username, args.password)
+
+    def rpc(model, method, call_args, kwargs=None):
+        return execute_kw(args.url, args.db, uid, args.password, model, method, call_args, kwargs)
+
+    company = rpc('res.users', 'read', [[uid], ['company_id']])[0]['company_id']
+    company_id = company[0]
+    company_info = rpc('res.company', 'read', [[company_id], ['fiscalyear_last_day', 'fiscalyear_last_month']])[0]
+
+    entry_date = args.date or last_fiscal_year_end(
+        company_info['fiscalyear_last_day'], int(company_info['fiscalyear_last_month']), date.today(),
+    )
+
+    account_by_code = {}
+    for account in rpc('account.account', 'search_read', [[('company_ids', 'in', [company_id])]], {'fields': ['code']}):
+        account_by_code[account['code']] = account['id']
+
+    lines = []
+    errors = []
+    total_debit = 0.0
+    total_credit = 0.0
+
+    for row_number, row in enumerate(read_rows(args.csv), start=2):
+        code = (row.get('account_code') or '').strip()
+        if not code:
+            errors.append(f"row {row_number}: missing account_code")
+            continue
+        account_id = account_by_code.get(code)
+        if not account_id:
+            errors.append(f"row {row_number}: unknown account_code '{code}'")
+            continue
+
+        try:
+            debit = parse_amount(row.get('debit'))
+            credit = parse_amount(row.get('credit'))
+        except ValueError:
+            errors.append(f"row {row_number}: debit/credit must be numeric")
+            continue
+
+        if bool(debit) == bool(credit):
+            errors.append(f"row {row_number}: exactly one of debit/credit must be positive (got debit={debit}, credit={credit})")
+            continue
+
+        partner_email = normalize_email(row.get('partner_email'))
+        partner_id = False
+        if partner_email:
+            partner_ids = rpc('res.partner', 'search', [[('email', '=', partner_email)]], {'limit': 1})
+            partner_id = partner_ids[0] if partner_ids else False
+
+        label = (row.get('label') or '').strip()
+        vals = {
+            'account_id': account_id,
+            'name': label or f"Opening Balance - {code}",
+            'debit': debit,
+            'credit': credit,
+        }
+        if partner_id:
+            vals['partner_id'] = partner_id
+        lines.append(vals)
+        total_debit += debit
+        total_credit += credit
+
+    if errors:
+        print("Refusing to post - CSV has errors:", file=sys.stderr)
+        for error in errors:
+            print(f"  {error}", file=sys.stderr)
+        return 1
+
+    if not lines:
+        print("No valid rows found - nothing to post.", file=sys.stderr)
+        return 1
+
+    if round(total_debit - total_credit, 2) != 0.0:
+        print(
+            f"Refusing to post - out of balance: total debit={total_debit:.2f}, total credit={total_credit:.2f}",
+            file=sys.stderr,
+        )
+        return 1
+
+    if args.dry_run:
+        print(f"DRY-RUN would post {len(lines)} lines dated {entry_date}, "
+              f"total debit={total_debit:.2f}, total credit={total_credit:.2f}")
+        return 0
+
+    journal_ids = rpc('account.journal', 'search', [[
+        ('type', '=', 'general'), ('company_id', '=', company_id),
+    ]], {'limit': 1})
+    if not journal_ids:
+        print("No general-type journal found for this company.", file=sys.stderr)
+        return 1
+    journal_id = journal_ids[0]
+
+    existing = rpc('account.move', 'search', [[
+        ('ref', '=', OPENING_BALANCE_REF), ('journal_id', '=', journal_id),
+        ('date', '=', entry_date), ('state', '=', 'posted'),
+    ]], {'limit': 1})
+    if existing:
+        print(f"Already done: posted entry {existing[0]} with ref '{OPENING_BALANCE_REF}' "
+              f"dated {entry_date} already exists. Nothing to do.")
+        return 0
+
+    move_id = rpc('account.move', 'create', [{
+        'move_type': 'entry',
+        'journal_id': journal_id,
+        'date': entry_date,
+        'ref': OPENING_BALANCE_REF,
+        'line_ids': [(0, 0, line) for line in lines],
+    }])
+    rpc('account.move', 'action_post', [[move_id]])
+
+    print(f"Done. Posted account.move id={move_id} dated {entry_date}, "
+          f"{len(lines)} lines, total debit={total_debit:.2f}, total credit={total_credit:.2f}")
+    return 0
+
+
+if __name__ == '__main__':
+    sys.exit(main())

scripts/migrate_students.py

@@ -0,0 +1,209 @@
+#!/usr/bin/env python3
+"""Migrate existing TNCSC school students (+ their parent/guardian) into
+Community OS School.
+
+Input CSV columns (source system export - adjust the header names below to
+match the real export once available, the column *names* are what matter,
+not their order):
+
+    student_full_name,student_date_of_birth,level_code,proficiency,grade_ref,
+    parent_full_name,parent_email,parent_phone,emergency_contact_name,
+    emergency_contact_phone,health_notes
+
+  - student_full_name: required.
+  - parent_full_name / parent_email: required. parent_email is the dedupe/
+    match key for the parent (case-insensitive) - if a partner with that
+    email already exists (e.g. created by migrate_members.py), it is reused
+    rather than duplicated.
+  - student_date_of_birth: optional, several common date formats accepted
+    and normalized to ISO (YYYY-MM-DD).
+  - level_code: optional, must match an existing community.school.level
+    code. Used to place the student into that level's class in the CURRENT
+    term (the community.school.term with state='open', most recent
+    start_date if more than one). If blank, unmatched, or no such class
+    exists in the current term, the student record is still created/updated
+    but no enrollment is made (reported as 'no_enrollment' with a reason).
+  - proficiency: optional, one of beginner/intermediate/advanced (defaults
+    to beginner if blank or unrecognized).
+  - parent_phone / emergency_contact_phone: optional, normalized to digits
+    (with a leading + kept if present).
+
+Behaviour:
+  - A student has no email of their own in most source exports, so there is
+    no natural unique key for them the way email works for members. Instead
+    a student is matched by (parent_partner_id, student name) - re-running
+    the same file updates the existing student rather than creating a
+    duplicate.
+  - Rows missing student_full_name/parent_full_name/parent_email, or that
+    duplicate a (parent_email, student_full_name) pair already seen earlier
+    in the SAME file, are marked 'failed' and skipped - they never reach
+    Odoo.
+  - Enrollment is only created once per (student, class) pair; re-running
+    does not create a second enrollment.
+
+Usage:
+    python scripts/migrate_students.py \\
+        --url https://your-odoo-host --db communityos_dev \\
+        --username admin --password admin \\
+        --csv data/raw/students_export.csv \\
+        --report data/raw/students_migration_report.csv \\
+        [--dry-run]
+"""
+import argparse
+import csv
+import sys
+from pathlib import Path
+
+from _migration_common import (
+    MigrationReport, authenticate, execute_kw, normalize_date, normalize_email, normalize_phone,
+)
+
+PROFICIENCIES = ('beginner', 'intermediate', 'advanced')
+
+
+def read_rows(csv_path):
+    with open(csv_path, newline='', encoding='utf-8') as handle:
+        yield from csv.DictReader(handle)
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    parser.add_argument('--url', required=True)
+    parser.add_argument('--db', required=True)
+    parser.add_argument('--username', required=True)
+    parser.add_argument('--password', required=True)
+    parser.add_argument('--csv', required=True, type=Path)
+    parser.add_argument('--report', required=True, type=Path)
+    parser.add_argument('--dry-run', action='store_true')
+    args = parser.parse_args()
+
+    if not args.csv.exists():
+        print(f"CSV file not found: {args.csv}", file=sys.stderr)
+        return 1
+
+    uid = authenticate(args.url, args.db, args.username, args.password)
+    report = MigrationReport()
+
+    def rpc(model, method, call_args, kwargs=None):
+        return execute_kw(args.url, args.db, uid, args.password, model, method, call_args, kwargs)
+
+    current_term = rpc(
+        'community.school.term', 'search_read',
+        [[('state', '=', 'open')]], {'fields': ['id'], 'order': 'start_date desc', 'limit': 1},
+    )
+    current_term_id = current_term[0]['id'] if current_term else None
+
+    level_by_code = {}
+    for level in rpc('community.school.level', 'search_read', [[]], {'fields': ['code']}):
+        level_by_code[level['code']] = level['id']
+
+    seen = set()
+
+    for row_number, row in enumerate(read_rows(args.csv), start=2):
+        student_name = (row.get('student_full_name') or '').strip()
+        parent_name = (row.get('parent_full_name') or '').strip()
+        parent_email = normalize_email(row.get('parent_email'))
+        key = f"{parent_email or 'row ' + str(row_number)} / {student_name or '?'}"
+
+        if not student_name or not parent_name or not parent_email:
+            report.add(key, 'failed', 'missing student_full_name, parent_full_name, or parent_email')
+            continue
+        dedupe_key = (parent_email, student_name.lower())
+        if dedupe_key in seen:
+            report.add(key, 'failed', 'duplicate (parent_email, student_full_name) in source file')
+            continue
+        seen.add(dedupe_key)
+
+        proficiency = (row.get('proficiency') or '').strip().lower()
+        if proficiency not in PROFICIENCIES:
+            proficiency = 'beginner'
+
+        level_code = (row.get('level_code') or '').strip()
+        level_id = level_by_code.get(level_code) if level_code else None
+
+        if args.dry_run:
+            action = 'would_create_or_update'
+            reason = ''
+            if level_code and not level_id:
+                reason = f"unknown level_code '{level_code}' (student still created/updated)"
+            elif not current_term_id:
+                reason = 'no open term - no enrollment' if level_code else ''
+            report.add(key, action, reason)
+            continue
+
+        parent_ids = rpc('res.partner', 'search', [[('email', '=', parent_email)]], {'limit': 1})
+        if parent_ids:
+            parent_id = parent_ids[0]
+            rpc('res.partner', 'write', [[parent_id], {
+                'name': parent_name,
+                'phone': normalize_phone(row.get('parent_phone')),
+            }])
+        else:
+            parent_id = rpc('res.partner', 'create', [{
+                'name': parent_name,
+                'email': parent_email,
+                'phone': normalize_phone(row.get('parent_phone')),
+            }])
+
+        existing_student = rpc(
+            'community.school.student', 'search_read',
+            [[('parent_partner_id', '=', parent_id)]], {'fields': ['id', 'partner_id']},
+        )
+        existing_student = [s for s in existing_student if s['partner_id'] and s['partner_id'][1] == student_name]
+
+        student_vals = {
+            'proficiency': proficiency,
+            'grade_ref': (row.get('grade_ref') or '').strip() or False,
+            'health_notes': (row.get('health_notes') or '').strip() or False,
+            'emergency_contact_name': (row.get('emergency_contact_name') or '').strip() or False,
+            'emergency_contact_phone': normalize_phone(row.get('emergency_contact_phone')),
+        }
+        dob = normalize_date(row.get('student_date_of_birth'))
+        if dob:
+            student_vals['date_of_birth'] = dob
+
+        if existing_student:
+            student_id = existing_student[0]['id']
+            student_partner_id = existing_student[0]['partner_id'][0]
+            rpc('res.partner', 'write', [[student_partner_id], {'name': student_name}])
+            rpc('community.school.student', 'write', [[student_id], student_vals])
+            action = 'updated'
+        else:
+            student_partner_id = rpc('res.partner', 'create', [{'name': student_name}])
+            student_vals.update({'partner_id': student_partner_id, 'parent_partner_id': parent_id})
+            student_id = rpc('community.school.student', 'create', [student_vals])
+            action = 'created'
+
+        reason = ''
+        if level_code and not level_id:
+            reason = f"unknown level_code '{level_code}' - no enrollment"
+        elif level_id and not current_term_id:
+            reason = 'no open term - no enrollment'
+        elif level_id and current_term_id:
+            class_ids = rpc('community.school.class', 'search', [[
+                ('level_id', '=', level_id), ('term_id', '=', current_term_id),
+            ]], {'limit': 1})
+            if not class_ids:
+                reason = f"no class for level '{level_code}' in current term - no enrollment"
+            else:
+                class_id = class_ids[0]
+                enrollment_ids = rpc('community.school.enrollment', 'search', [[
+                    ('student_id', '=', student_id), ('class_id', '=', class_id),
+                ]], {'limit': 1})
+                if not enrollment_ids:
+                    rpc('community.school.enrollment', 'create', [{
+                        'student_id': student_id, 'class_id': class_id,
+                    }])
+                    action += '+enrolled'
+
+        report.add(key, action, reason)
+
+    report.write(args.report)
+    summary = report.summary()
+    print(f"Done. {summary}")
+    print(f"Report written to {args.report}")
+    return 1 if summary.get('failed') else 0
+
+
+if __name__ == '__main__':
+    sys.exit(main())