From 9a24691c37d570178fa1a03d5036fd2991bd636d Mon Sep 17 00:00:00 2001 From: metatroncubeswdev Date: Wed, 19 Aug 2026 01:47:34 -0400 Subject: [PATCH] feat: Phase 8 TNCSC data migration scripts (members, students, opening balances) Adds migrate_students.py (parent+child partner upsert, current-term enrollment by school level code) and migrate_opening_balances.py (one posted, balance-checked journal entry dated the last fiscal year-end). All three Phase 8 scripts (incl. the existing migrate_members.py) verified against a live Odoo 19 instance: fresh tncsc_migration_test DB with tncsc_deployment installed, dry-run + real run + idempotency re-run each, confirmed via RPC no duplicate records were created. See HANDOFF.md for the full verification log and why a fresh DB was used instead of communityos_dev. Co-Authored-By: Claude Sonnet 5 --- HANDOFF.md | 111 ++++++++++++--- scripts/migrate_opening_balances.py | 205 +++++++++++++++++++++++++++ scripts/migrate_students.py | 209 ++++++++++++++++++++++++++++ 3 files changed, 503 insertions(+), 22 deletions(-) create mode 100644 scripts/migrate_opening_balances.py create mode 100644 scripts/migrate_students.py diff --git a/HANDOFF.md b/HANDOFF.md index 4cc8a56..06ad23e 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -17,7 +17,7 @@ file is the "where things actually stand" companion to it. | 5 | `community_interac` (Interac e-Transfer payment provider) | ✅ Done, committed | | 6 | `community_theme_base` + `community_portal` | ✅ Done, committed | | 7 | `tncsc_deployment` (client config layer) | ✅ Done, committed | -| 8 | TNCSC data migration scripts | 🟡 Barely started, **not committed** — see below | +| 8 | TNCSC data migration scripts | ✅ Written and verified live, **not committed** — see below | | 9 | Packaging for resale + production go-live | ⬜ Not started | Every phase 0–7 commit was verified against a **live** Odoo 19 + Postgres 16 @@ -58,32 +58,99 @@ docker compose up -d ## Phase 8 — exactly where it was left off -Two files exist, committed as WIP (see the `wip:` commit), inert (not -imported by anything, not installed, no manifest references them): +All three migration scripts exist and have been verified against a **live** +Odoo instance, but **the work is not committed yet** (see "To pick this back +up" below for why). - `scripts/_migration_common.py` — shared JSON-RPC helpers (auth, execute_kw, CSV normalization, a `MigrationReport` class) for the migration scripts. - Finished, not yet tested against anything. -- `scripts/migrate_members.py` — first pass at the member migration script - (idempotent upsert by email, dry-run flag, tier lookup by code, report - CSV). Written but **not tested at all** — no live run against Odoo yet. +- `scripts/migrate_members.py` — idempotent upsert by email, dry-run flag, + tier lookup by code, report CSV. +- `scripts/migrate_students.py` — idempotent upsert of parent partner (by + email, reusing a partner created by `migrate_members.py` if one exists) + + child partner + `community.school.student`, matched by (parent, student + name) since students don't have their own email in the source data. + Enrolls into the matching class in the current open term (matched by + `community.school.level` code) when one exists; otherwise still + creates/updates the student record and reports why no enrollment + happened. +- `scripts/migrate_opening_balances.py` — posts one `account.move` in the + Miscellaneous Operations journal, dated the last fiscal year-end + (computed from `res.company.fiscalyear_last_day/month`, or `--date` + override). Refuses to post anything if the CSV doesn't balance + (`sum(debit) != sum(credit)`) or references an unknown account code — + checked before any Odoo write happens. Idempotent via a fixed `ref` + ("Opening Balances Import"): re-running finds the existing posted entry + and does nothing. -Still to do for Phase 8 (see the plan's Phase 8 section): -- `scripts/migrate_students.py` — not started. -- `scripts/migrate_opening_balances.py` — not started. -- Test all three against synthetic/sample CSV data on a live instance, - the same way `scripts/migrate_classroom.py` was tested (see that file - and Phase 3's commit for the pattern: dry-run first, then a real run, - then a second real run to prove idempotency). -- Real TNCSC member/student data doesn't exist yet — these scripts were - intentionally designed against a *documented, assumed* CSV schema (see - each script's docstring) since the real export format isn't known yet. - Confirm the assumed column names match the real export before relying on - them, or adjust the scripts to match. +### How each was tested -To pick this back up: either continue `migrate_members.py`'s testing and -write the other two scripts, or decide the assumed CSV schemas are wrong -and redesign first once real export samples exist. +Followed the same pattern as `scripts/migrate_classroom.py` (Phase 3): +dry-run first, then a real run, then a second real run to prove +idempotency — against a **fresh** database, not `communityos_dev`. See +"Why a fresh DB, not communityos_dev" below for why. + +1. `docker compose up -d`, then created a throwaway DB with both product + modules and the client layer: `odoo -d tncsc_migration_test -i + community_membership,tncsc_deployment --stop-after-init` (installing + `tncsc_deployment` alone pulls in `community_school` and everything else + as dependencies). Restarted the container afterward. +2. `migrate_members.py`: synthetic CSV at + `data/raw/members_export_sample.csv` (gitignored, left in place as a + fixture) with 4 valid rows + 2 deliberately-bad rows (missing email, + unknown tier code). Dry-run matched real-run output; second real run + reported `updated` not `created` for all 4; confirmed via RPC that + exactly one `res.partner` per email existed after both runs and + `membership_state` was `active`. +3. `migrate_students.py`: needed a `community.school.term` (state=open), + `community.school.level` (code=`BEG`), and `community.school.class` + seeded first (the fresh DB has none) — also had to grant the `admin` + user the School Coordinator group, since a fresh install doesn't put + admin in it. Synthetic CSV at `data/raw/students_export_sample.csv` + with 3 valid rows (2 matching the open term's class, 1 with no + `level_code`) + 1 deliberately-bad row (missing parent). Confirmed via + RPC after 2 runs: exactly 3 `community.school.student` records, exactly + 2 `community.school.enrollment` records (no duplicates), parent + partners correctly reused from step 2 instead of duplicated. +4. `migrate_opening_balances.py`: synthetic CSV at + `data/raw/opening_balances_sample.csv` using real account codes from + the fresh DB's TNCSC chart of accounts (111100 Cash, 112110 Trade AR, + 322000 Retained Earnings as the balancing line). Verified the + out-of-balance guard refuses to post (tested with a deliberately + mismatched debit/credit CSV). Real run posted one balanced, posted + `account.move`; second run detected the existing entry and did nothing. + Confirmed via RPC: exactly one move, three correctly-valued lines, AR + line correctly linked to the migrated Ravi Kumar partner. + +Real TNCSC member/student/trial-balance data doesn't exist yet — all three +scripts were intentionally designed against a *documented, assumed* CSV +schema (see each script's docstring) since the real export formats aren't +known yet. Confirm the assumed column names match the real exports before +relying on them, or adjust the scripts to match. + +### Why a fresh DB, not `communityos_dev` + +`communityos_dev` doesn't have `tncsc_deployment` installed (see the top of +this file — it can't be, since it already has posted journal entries and +Odoo refuses to change a company's currency once those exist). Testing the +Phase 8 scripts there would exercise generic `community_membership`/ +`community_school` behavior, not the actual TNCSC-configured target +(TNCSC's real chart of accounts, company currency, groups). Since Phase 8's +entire purpose is rehearsing the real TNCSC data load, a fresh DB with +`tncsc_deployment` installed is the more representative target — and it +avoids adding more synthetic test data to the one long-lived dev DB used +across every other phase. `tncsc_migration_test` was **not** dropped after +this session (unlike the Phase 7 `tncsc_freshN` throwaways) — it's left in +place in case the next session wants to keep testing against it; drop it +before it's mistaken for anything resembling real data. + +### To pick this back up + +The scripts work and are proven idempotent, but haven't been committed — +do that first (a `feat: Phase 8 migration scripts` commit, following the +same "verified against a live container before committing" convention as +every other phase), then either wait for real TNCSC export samples to +validate the assumed CSV schemas against, or move on to Phase 9. ## Phase 9 — not started diff --git a/scripts/migrate_opening_balances.py b/scripts/migrate_opening_balances.py new file mode 100644 index 0000000..f22c9e7 --- /dev/null +++ b/scripts/migrate_opening_balances.py @@ -0,0 +1,205 @@ +#!/usr/bin/env python3 +"""Post TNCSC's opening balances as a single journal entry dated the last +fiscal year-end, in the company's Miscellaneous Operations journal. + +Input CSV columns (source system export - typically a trial balance dump; +adjust the header names below to match the real export once available): + + account_code,debit,credit,label,partner_email + + - account_code: required, must match an existing account.account code for + the target company. + - debit / credit: each optional but exactly one of the two must be a + positive number per row (standard opening-balance convention: never + both, never neither). + - label: optional line label (defaults to "Opening Balance - "). + - partner_email: optional, links the line to a res.partner (useful for + AR/AP opening balances). Looked up by email; left blank if not found or + not given. + +The CSV must balance: total debit must equal total credit across all rows, +or the script refuses to post anything (this is checked before any Odoo +write happens). + +Behaviour: + - Idempotent: the entry is created with a fixed ref + ("Opening Balances Import") in the Miscellaneous Operations journal. If + a posted entry with that ref already exists in that journal dated the + same day, the script does nothing and reports it as already done - + re-running never creates a duplicate. + - The entry date is the most recently completed fiscal year-end before + today, computed from the company's fiscal year configuration, unless + --date overrides it. + - Draft entry is created first, then posted (action_post) - so a failure + partway through leaves nothing posted. + +Usage: + python scripts/migrate_opening_balances.py \\ + --url https://your-odoo-host --db communityos_dev \\ + --username admin --password admin \\ + --csv data/raw/opening_balances.csv \\ + [--date 2025-12-31] [--dry-run] +""" +import argparse +import csv +import sys +from datetime import date +from pathlib import Path + +from _migration_common import authenticate, execute_kw, normalize_email + +OPENING_BALANCE_REF = 'Opening Balances Import' + + +def read_rows(csv_path): + with open(csv_path, newline='', encoding='utf-8') as handle: + yield from csv.DictReader(handle) + + +def parse_amount(value): + value = (value or '').strip() + if not value: + return 0.0 + return float(value.replace(',', '')) + + +def last_fiscal_year_end(last_day, last_month, today): + candidate = date(today.year, last_month, last_day) + if candidate >= today: + candidate = date(today.year - 1, last_month, last_day) + return candidate.isoformat() + + +def main(): + parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + parser.add_argument('--url', required=True) + parser.add_argument('--db', required=True) + parser.add_argument('--username', required=True) + parser.add_argument('--password', required=True) + parser.add_argument('--csv', required=True, type=Path) + parser.add_argument('--date', help="Override the entry date (YYYY-MM-DD); default is the last fiscal year-end") + parser.add_argument('--dry-run', action='store_true') + args = parser.parse_args() + + if not args.csv.exists(): + print(f"CSV file not found: {args.csv}", file=sys.stderr) + return 1 + + uid = authenticate(args.url, args.db, args.username, args.password) + + def rpc(model, method, call_args, kwargs=None): + return execute_kw(args.url, args.db, uid, args.password, model, method, call_args, kwargs) + + company = rpc('res.users', 'read', [[uid], ['company_id']])[0]['company_id'] + company_id = company[0] + company_info = rpc('res.company', 'read', [[company_id], ['fiscalyear_last_day', 'fiscalyear_last_month']])[0] + + entry_date = args.date or last_fiscal_year_end( + company_info['fiscalyear_last_day'], int(company_info['fiscalyear_last_month']), date.today(), + ) + + account_by_code = {} + for account in rpc('account.account', 'search_read', [[('company_ids', 'in', [company_id])]], {'fields': ['code']}): + account_by_code[account['code']] = account['id'] + + lines = [] + errors = [] + total_debit = 0.0 + total_credit = 0.0 + + for row_number, row in enumerate(read_rows(args.csv), start=2): + code = (row.get('account_code') or '').strip() + if not code: + errors.append(f"row {row_number}: missing account_code") + continue + account_id = account_by_code.get(code) + if not account_id: + errors.append(f"row {row_number}: unknown account_code '{code}'") + continue + + try: + debit = parse_amount(row.get('debit')) + credit = parse_amount(row.get('credit')) + except ValueError: + errors.append(f"row {row_number}: debit/credit must be numeric") + continue + + if bool(debit) == bool(credit): + errors.append(f"row {row_number}: exactly one of debit/credit must be positive (got debit={debit}, credit={credit})") + continue + + partner_email = normalize_email(row.get('partner_email')) + partner_id = False + if partner_email: + partner_ids = rpc('res.partner', 'search', [[('email', '=', partner_email)]], {'limit': 1}) + partner_id = partner_ids[0] if partner_ids else False + + label = (row.get('label') or '').strip() + vals = { + 'account_id': account_id, + 'name': label or f"Opening Balance - {code}", + 'debit': debit, + 'credit': credit, + } + if partner_id: + vals['partner_id'] = partner_id + lines.append(vals) + total_debit += debit + total_credit += credit + + if errors: + print("Refusing to post - CSV has errors:", file=sys.stderr) + for error in errors: + print(f" {error}", file=sys.stderr) + return 1 + + if not lines: + print("No valid rows found - nothing to post.", file=sys.stderr) + return 1 + + if round(total_debit - total_credit, 2) != 0.0: + print( + f"Refusing to post - out of balance: total debit={total_debit:.2f}, total credit={total_credit:.2f}", + file=sys.stderr, + ) + return 1 + + if args.dry_run: + print(f"DRY-RUN would post {len(lines)} lines dated {entry_date}, " + f"total debit={total_debit:.2f}, total credit={total_credit:.2f}") + return 0 + + journal_ids = rpc('account.journal', 'search', [[ + ('type', '=', 'general'), ('company_id', '=', company_id), + ]], {'limit': 1}) + if not journal_ids: + print("No general-type journal found for this company.", file=sys.stderr) + return 1 + journal_id = journal_ids[0] + + existing = rpc('account.move', 'search', [[ + ('ref', '=', OPENING_BALANCE_REF), ('journal_id', '=', journal_id), + ('date', '=', entry_date), ('state', '=', 'posted'), + ]], {'limit': 1}) + if existing: + print(f"Already done: posted entry {existing[0]} with ref '{OPENING_BALANCE_REF}' " + f"dated {entry_date} already exists. Nothing to do.") + return 0 + + move_id = rpc('account.move', 'create', [{ + 'move_type': 'entry', + 'journal_id': journal_id, + 'date': entry_date, + 'ref': OPENING_BALANCE_REF, + 'line_ids': [(0, 0, line) for line in lines], + }]) + rpc('account.move', 'action_post', [[move_id]]) + + print(f"Done. Posted account.move id={move_id} dated {entry_date}, " + f"{len(lines)} lines, total debit={total_debit:.2f}, total credit={total_credit:.2f}") + return 0 + + +if __name__ == '__main__': + sys.exit(main()) diff --git a/scripts/migrate_students.py b/scripts/migrate_students.py new file mode 100644 index 0000000..9b2bb3c --- /dev/null +++ b/scripts/migrate_students.py @@ -0,0 +1,209 @@ +#!/usr/bin/env python3 +"""Migrate existing TNCSC school students (+ their parent/guardian) into +Community OS School. + +Input CSV columns (source system export - adjust the header names below to +match the real export once available, the column *names* are what matter, +not their order): + + student_full_name,student_date_of_birth,level_code,proficiency,grade_ref, + parent_full_name,parent_email,parent_phone,emergency_contact_name, + emergency_contact_phone,health_notes + + - student_full_name: required. + - parent_full_name / parent_email: required. parent_email is the dedupe/ + match key for the parent (case-insensitive) - if a partner with that + email already exists (e.g. created by migrate_members.py), it is reused + rather than duplicated. + - student_date_of_birth: optional, several common date formats accepted + and normalized to ISO (YYYY-MM-DD). + - level_code: optional, must match an existing community.school.level + code. Used to place the student into that level's class in the CURRENT + term (the community.school.term with state='open', most recent + start_date if more than one). If blank, unmatched, or no such class + exists in the current term, the student record is still created/updated + but no enrollment is made (reported as 'no_enrollment' with a reason). + - proficiency: optional, one of beginner/intermediate/advanced (defaults + to beginner if blank or unrecognized). + - parent_phone / emergency_contact_phone: optional, normalized to digits + (with a leading + kept if present). + +Behaviour: + - A student has no email of their own in most source exports, so there is + no natural unique key for them the way email works for members. Instead + a student is matched by (parent_partner_id, student name) - re-running + the same file updates the existing student rather than creating a + duplicate. + - Rows missing student_full_name/parent_full_name/parent_email, or that + duplicate a (parent_email, student_full_name) pair already seen earlier + in the SAME file, are marked 'failed' and skipped - they never reach + Odoo. + - Enrollment is only created once per (student, class) pair; re-running + does not create a second enrollment. + +Usage: + python scripts/migrate_students.py \\ + --url https://your-odoo-host --db communityos_dev \\ + --username admin --password admin \\ + --csv data/raw/students_export.csv \\ + --report data/raw/students_migration_report.csv \\ + [--dry-run] +""" +import argparse +import csv +import sys +from pathlib import Path + +from _migration_common import ( + MigrationReport, authenticate, execute_kw, normalize_date, normalize_email, normalize_phone, +) + +PROFICIENCIES = ('beginner', 'intermediate', 'advanced') + + +def read_rows(csv_path): + with open(csv_path, newline='', encoding='utf-8') as handle: + yield from csv.DictReader(handle) + + +def main(): + parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + parser.add_argument('--url', required=True) + parser.add_argument('--db', required=True) + parser.add_argument('--username', required=True) + parser.add_argument('--password', required=True) + parser.add_argument('--csv', required=True, type=Path) + parser.add_argument('--report', required=True, type=Path) + parser.add_argument('--dry-run', action='store_true') + args = parser.parse_args() + + if not args.csv.exists(): + print(f"CSV file not found: {args.csv}", file=sys.stderr) + return 1 + + uid = authenticate(args.url, args.db, args.username, args.password) + report = MigrationReport() + + def rpc(model, method, call_args, kwargs=None): + return execute_kw(args.url, args.db, uid, args.password, model, method, call_args, kwargs) + + current_term = rpc( + 'community.school.term', 'search_read', + [[('state', '=', 'open')]], {'fields': ['id'], 'order': 'start_date desc', 'limit': 1}, + ) + current_term_id = current_term[0]['id'] if current_term else None + + level_by_code = {} + for level in rpc('community.school.level', 'search_read', [[]], {'fields': ['code']}): + level_by_code[level['code']] = level['id'] + + seen = set() + + for row_number, row in enumerate(read_rows(args.csv), start=2): + student_name = (row.get('student_full_name') or '').strip() + parent_name = (row.get('parent_full_name') or '').strip() + parent_email = normalize_email(row.get('parent_email')) + key = f"{parent_email or 'row ' + str(row_number)} / {student_name or '?'}" + + if not student_name or not parent_name or not parent_email: + report.add(key, 'failed', 'missing student_full_name, parent_full_name, or parent_email') + continue + dedupe_key = (parent_email, student_name.lower()) + if dedupe_key in seen: + report.add(key, 'failed', 'duplicate (parent_email, student_full_name) in source file') + continue + seen.add(dedupe_key) + + proficiency = (row.get('proficiency') or '').strip().lower() + if proficiency not in PROFICIENCIES: + proficiency = 'beginner' + + level_code = (row.get('level_code') or '').strip() + level_id = level_by_code.get(level_code) if level_code else None + + if args.dry_run: + action = 'would_create_or_update' + reason = '' + if level_code and not level_id: + reason = f"unknown level_code '{level_code}' (student still created/updated)" + elif not current_term_id: + reason = 'no open term - no enrollment' if level_code else '' + report.add(key, action, reason) + continue + + parent_ids = rpc('res.partner', 'search', [[('email', '=', parent_email)]], {'limit': 1}) + if parent_ids: + parent_id = parent_ids[0] + rpc('res.partner', 'write', [[parent_id], { + 'name': parent_name, + 'phone': normalize_phone(row.get('parent_phone')), + }]) + else: + parent_id = rpc('res.partner', 'create', [{ + 'name': parent_name, + 'email': parent_email, + 'phone': normalize_phone(row.get('parent_phone')), + }]) + + existing_student = rpc( + 'community.school.student', 'search_read', + [[('parent_partner_id', '=', parent_id)]], {'fields': ['id', 'partner_id']}, + ) + existing_student = [s for s in existing_student if s['partner_id'] and s['partner_id'][1] == student_name] + + student_vals = { + 'proficiency': proficiency, + 'grade_ref': (row.get('grade_ref') or '').strip() or False, + 'health_notes': (row.get('health_notes') or '').strip() or False, + 'emergency_contact_name': (row.get('emergency_contact_name') or '').strip() or False, + 'emergency_contact_phone': normalize_phone(row.get('emergency_contact_phone')), + } + dob = normalize_date(row.get('student_date_of_birth')) + if dob: + student_vals['date_of_birth'] = dob + + if existing_student: + student_id = existing_student[0]['id'] + student_partner_id = existing_student[0]['partner_id'][0] + rpc('res.partner', 'write', [[student_partner_id], {'name': student_name}]) + rpc('community.school.student', 'write', [[student_id], student_vals]) + action = 'updated' + else: + student_partner_id = rpc('res.partner', 'create', [{'name': student_name}]) + student_vals.update({'partner_id': student_partner_id, 'parent_partner_id': parent_id}) + student_id = rpc('community.school.student', 'create', [student_vals]) + action = 'created' + + reason = '' + if level_code and not level_id: + reason = f"unknown level_code '{level_code}' - no enrollment" + elif level_id and not current_term_id: + reason = 'no open term - no enrollment' + elif level_id and current_term_id: + class_ids = rpc('community.school.class', 'search', [[ + ('level_id', '=', level_id), ('term_id', '=', current_term_id), + ]], {'limit': 1}) + if not class_ids: + reason = f"no class for level '{level_code}' in current term - no enrollment" + else: + class_id = class_ids[0] + enrollment_ids = rpc('community.school.enrollment', 'search', [[ + ('student_id', '=', student_id), ('class_id', '=', class_id), + ]], {'limit': 1}) + if not enrollment_ids: + rpc('community.school.enrollment', 'create', [{ + 'student_id': student_id, 'class_id': class_id, + }]) + action += '+enrolled' + + report.add(key, action, reason) + + report.write(args.report) + summary = report.summary() + print(f"Done. {summary}") + print(f"Report written to {args.report}") + return 1 if summary.get('failed') else 0 + + +if __name__ == '__main__': + sys.exit(main())