Watch

No description

Python 99.9%
Shell 0.1%

Find a file

Repository files (latest commit first)
Filename	Latest commit message	Latest commit date
Kion a84b8fdd9d the generator was ALWAYS a pure function and nobody had ever run it that way -- seed in, room graph out, 10/10 first floors from a 16-bit seed alone, and the one function between that and a complete room record is named psz-godot asked two capability questions and said an honest "captures only" was a useful answer to the first, because it would tell them to build the weaker capture-comparison tool instead. It is not captures only. THE ALGORITHM WAS NEVER THE GAP. `29e04be` shipped `genreplay.generate(state, params)` and every caller in this repo fed it a state recovered BACKWARDS out of a savestate. A tool that needs a capture for the seed it is asked about is not seed-driven, so what was missing was not code -- it was a path from a CHOSEN seed to a field, and a score for it. `tools/fieldgen.py` is that path. `fieldgen.field(seed, params)` opens no savestate, no ROM and no emulator. THE INPUT IS A PAIR: (16-bit session seed, eleven quest generator parameters). The seed alone suffices only because `a583300` measured the seed-to-generation offset at exactly ZERO on a first floor, and the parameters come from the quest override record, so `--quest <id>` reads them out of the ROM and the caller need not know them either. SCORED IN THE DIRECTION THAT WOULD OVERTURN IT. All 14 fixtures were driven through the NEW arbitrary-seed path rather than the old recorded-state one -- a divergence between the two would have BEEN the finding, and there is none. 14/14 fixtures reproduce from their recorded generation-start state 10/10 FIRST floors reproduce from their 16-bit session seed ALONE 0/4 later floors do, which is the expected negative and the reason "seed -> field" is stated as a FIRST-FLOOR guarantee 0/1792 neighbouring 16-bit seeds reproduce anything 0/108 other parameter blocks reproduce anything A match requires the edge set, the transcribed cell order, the goal cell AND the transcription permutation, so nothing passes on the graph alone. TWO THINGS THAT WERE READ OUT OF CAPTURES ARE NOW DERIVED. FUN_020826EC transcribed gives the room SHAPE from the cell degree plus the START/GOAL flag bits -- 131/131 against the game's own transcribed record -- and the ROTATION for every shape that does not draw for one, 103/103. The 28 cross and straight rooms are excluded and counted rather than guessed at, because FUN_020826EC draws rand(4) and rand(2) for them. AND A MAPPING THE FIXTURES HAD SILENTLY DEPENDED ON IS NOW ASSERTED: the OUTPUT room index IS the internal PLACEMENT index, 14/14 fields. The transcriber writes to table + (cell+0x01) * 0x34 and FUN_02083590's shuffle of gen+0x60 reorders the DRAWS, not the room table -- a port that permutes its rooms by that shuffle puts every room in the wrong slot. WHAT A SEED DOES NOT GIVE, AND IT IS ONE FUNCTION. exits[].gate, keys, key_ids, layout, code, seed and the cross/straight rotations are all null, and all of them are blocked on FUN_02081F4C, the gate stage, which runs between placement and transcription and whose draw count depends on the graph: FUN_020B2698 (budget plus the room-list shuffle), FUN_020B2718 looping FUN_020B2740 until the budget is spent, FUN_020B2A54's weighted rare-room draw, FUN_020B2BF4's per-room rand(100) < params[8]. THE TRANSCRIBER IS NOT THE GAP -- tools/forward.py already replays its per-room draws and predicts rotation 10/10, key IDs 9/9 and the layout index 44/44 once anchored on a capture. Closing the gate stage turns every null into a value, and a sharp oracle is waiting: the predicted position of each per-room seed must land exactly, and the fixtures pin all 131 of them. A TENSION RECORDED RATHER THAN SMOOTHED. data/level_generation_topology.json says FUN_020B2740 PLACES ROOMS. The placement replay reproduces the full edge set 14/14 without it, which cannot hold if it adds cells -- so its "one or two rooms" is a branch ATTACHMENT count feeding the key gates (FUN_020B27EC ORs 1 or 2 into the doorway bitmask and FUN_020B2A14 propagates it), not a placement. Written as a new key in that file rather than rewriting the old entries, because it is a measurement about a function this pass read and did not execute. THE EXPORT, AND WHY OPTION TWO WAS RIGHT. docs/godot-parity-export.json carries 60 findings -- 47 current, 9 retracted, 2 superseded, 1 stale-output, 1 divergence -- across field generation, doorways and gates, object groups, object placement, waves and the treasure box. The argument for a curated export over a vendored nodes/ subset is this repo's own history: factory index 1 was "the invisible warp trigger" for two commits before `aef6d1b` found the PLAYER ENTRY POINT, o0c_trebox's "68/68 against 44" was read as a runtime spawn for four commits before `c7c18aa` counted the records, effect_id.rel column 1 was param_id until `07f83d1`, and the NOF0 window was one word early for the repo's entire history until `25ca38f`. ALL FOUR ARE IN THE EXPORT as worked examples, each naming what replaced it, because psz-godot may be acting on the old reading right now. So is 42931d0's known pending delta, as stale-output: object_placement_per_room.json and trap_placement_per_room.json still ship `group_sizes: null` for the 544 short files, and the export says to join against room_group_counts.json instead. So is section 12.1's treasure-box divergence, as `divergence`, so nobody reconciles it by accident. IT IS GENERATED AND GATED. tools/parityexport.py's curation table names a node, a checker, a topic and a status and holds NO prose about the game; every word the consumer reads -- assertion, oracle, the gate-written test status, the measured result, the rates pulled out of it, the addresses -- is read out of nodes/*.json at generation time, so a node edit changes the export. Supersession is checked for referential integrity both ways and an entry marked retracted with no superseded_by is refused. `the_curated_parity_export_is_in_sync_with_its_sources` rebuilds the file and fails if the shipped one differs, because a stale export does not look stale, it looks authoritative. The one deliberate exception is a RETRACTED claim, which by construction no longer lives in any node; those carry body_source "curation" and a frozen one-line claim so the two kinds are distinguished by machine rather than by eye. NOT DONE, plainly. - The gate stage is NOT replayed and this pass did not attempt it. It needs about fifteen more helpers out of the 0x020B2xxx cluster, and Ghidra has dropped arguments on at least two of them -- FUN_020B27EC's count and FUN_020B2978's unbounded draw -- so it is a pass of its own with disassembly-level care. - LATER FLOORS ARE NOT CHAINED, and it is the SAME blocker rather than a second one: the first floor's own total draw cost includes the gate stage, so no chain can be built before that is replayed. A two-sample lead is recorded in the node rather than acted on -- the two sessions with both floors captured (0x2748 s01a -> s01b at offset 223, 0xE7B3 s02a -> s02b at 205) leave a remainder of 129 and 130 draws after the first floor's LAST transcriber draw, taken as first_room_seed_at + run_span + 1. Nearly constant on two points, which is suggestive and is not a model: two points cannot separate a constant from something that varies with room count. - COVERAGE IS 9 STAGES: s01a s01b s02a s02b s03a s03b s04a s05a s06b, 14 fixtures. Stages 07 and 08 and every c/d/e area variant are ABSENT from the corpus. The generator code is stage-independent so a seed for one of those produces a field, but nothing has checked it against the real thing -- that is extrapolation, and fieldgen.py says so in its own output rather than only in a doc. - The two big placement data files were NOT regenerated against the authored group count. The export flags it rather than hiding it. - Nothing propagates a status change automatically. A retraction reaches the export only when someone curates it in the same pass that lands it; the gate check catches a stale BODY, not a finding nobody curated. SEVEN TEST RESULTS THE REPO WAS THROWING AWAY ARE NOW BANKED, and only seven. The export's `evidence` comes out of the node's recorded `result`, and ten findings would have shipped EMPTY because their tests carried `status: null` -- the gate ran them, and the convention of reverting the gate's date churn reverted the first recorded result along with it. So this keeps the gate-written status, result and verified for the tests that had none in the two nodes the export reads from: 5 in sys.room-object-density and 2 in fmt.room-object-set. Every other node the write run touched was reverted to HEAD, date churn and all. NOTED IN PASSING, NOT ACTED ON: the same run flipped three stale `failed` fossils to `passed` in sys.text-scratch and one in sys.slot-overlays. They are stale records, not live failures -- the gate computes its failure set from the run and not from the files, which is why the baseline is empty with them in the tree -- and they were reverted to keep this commit to one delta. A future write run should bank them. Files: tools/fieldgen.py (new), tools/parityexport.py (new), docs/godot-parity-export.json (new, generated), nodes/tool.godot-parity-export.json (new), three checkers appended to tools/checks.py, two tests, five addresses, two evidence rows and an open question on nodes/sys.field-generation-target.json, banked results on nodes/sys.room-object-density.json and nodes/fmt.room-object-set.json, docs/godot-field-parity.md sections 11.1 and 11.2, a cross-reference key added to data/level_generation_topology.json without deleting anything, and entries in docs/unattended-backlog.md and docs/godot-parity-requests.md. GATE STATE. - Method per docs/gate-counting.md: system python3, counted off the `failure set` block, never by grepping FAIL. - BEFORE, on a clean tree at `42931d0`: 586 passed, 0 failed, 5 skipped, failure set EMPTY -- exactly the recorded baseline. - WRITE RUN, on the finished tree: 588 passed, 1 failed -- and the one failure is `tool.godot-parity-export :: the_curated_parity_export_is_in_sync_with_its_sources` firing exactly as designed, because the gate had just rewritten statuses under the export. The export was regenerated and it went green. A staleness check that never fires is not a staleness check. - AFTER, --no-write on the committed tree: 589 passed, 0 failed, 5 skipped, failure set EMPTY. - Failure SET diff: EMPTY -> EMPTY. Nothing moved. The count rose 586 -> 589 because this commit adds three checks and all three pass. - The self-audit invariant is intact: emitted + emitted_volumes + declined == sum(group_sizes) in 1495 of 1495 rooms. The two big placement data files were NOT regenerated, so the pending delta `42931d0` recorded is still pending and the export now names it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>		2026-08-15 14:21:20 -05:00
data	the generator was ALWAYS a pure function and nobody had ever run it that way -- seed in, room graph out, 10/10 first floors from a 16-bit seed alone, and the one function between that and a complete room record is named	2026-08-15 14:21:20 -05:00
docs	the generator was ALWAYS a pure function and nobody had ever run it that way -- seed in, room graph out, 10/10 first floors from a 16-bit seed alone, and the one function between that and a complete room record is named	2026-08-15 14:21:20 -05:00
nodes	the generator was ALWAYS a pure function and nobody had ever run it that way -- seed in, room graph out, 10/10 first floors from a 16-bit seed alone, and the one function between that and a complete room record is named	2026-08-15 14:21:20 -05:00
ref	Seed the PSZ NDS reverse-engineering project	2026-07-27 02:30:52 -05:00
seeders	vtables: purity, not length, is what makes a run a vtable — database 13,934 -> 14,142	2026-07-30 09:21:05 -05:00
text	inventory: the equipped marker is not in the slot, and not an index in the header either	2026-07-30 13:43:29 -05:00
tools	the generator was ALWAYS a pure function and nobody had ever run it that way -- seed in, room graph out, 10/10 first floors from a 16-bit seed alone, and the one function between that and a complete room record is named	2026-08-15 14:21:20 -05:00
.gitignore	untrack `build` -- I committed a symlink that every checkout turned into a self-referential loop, destroying the real build directory	2026-08-02 16:45:16 -05:00
AGENTS.md	coverage: labelled hypothesis for every function + a queue to work it	2026-07-28 20:04:56 -05:00
CLAIMS.md	coverage: labelled hypothesis for every function + a queue to work it	2026-07-28 20:04:56 -05:00
README.md	build/ was destroyed and the README only got it half back -- both regeneration steps are now documented and checked	2026-08-02 17:41:32 -05:00
schema.md	Add knowledge-graph mechanism: schema + AGENTS + cheat-anchor seed	2026-07-27 20:46:27 -05:00

README.md

psz-re — Phantasy Star Zero (NDS) Reverse Engineering

Reverse engineering Phantasy Star Zero (Nintendo DS, 2008) to recover the original game's data tables and behaviour, so that psz-godot can be built from measured fact instead of guesswork.

Why this repo exists

psz-godot's bottleneck is not implementation speed. It is that a human has to sit down, play the original, and hand-author a spec for every unimplemented system — damage formulas, enemy AI, spawn tables, drop rates, photon arts. That human is the slowest part of the loop and has the least free time.

This repo removes the human from that loop. The original ROM is the ground truth. If we can read a number or a state machine out of the binary, nobody needs to play the game to find out what it should be.

Output of this repo is documents and JSON, not a playable build. Its deliverables land in psz-godot as specs, issues, and data files.

Explicit non-goal: matching decompilation

We are not chasing a byte-matching decomp.

For reference, sm64ds-decomp is the state of the art for NDS matching decomp: it required the proprietary mwccarm 1.2/sp2p3 compiler with exact flags, and after sustained effort sits at ~96.3% of functions matched (10,929 / 11,348). That is a multi-year commitment whose payoff is a rebuildable ROM.

We do not want a rebuildable ROM. We want to know what the numbers are. A readable, non-matching Ghidra-driven understanding gets ~95% of the value for psz-godot at a few percent of the cost. Anyone tempted to chase matching should open an issue and argue for it first.

Current state

The ROM has been unpacked and surveyed. Nothing is decompiled yet.

ROM: Phantasy Star 0 (USA).nds — PSZERO / C24E, 128 MB (md5 0ef2ab31b0068c0832bd6a5ef901dd0c). Kept on me-mini at /mnt/storage/psz-roms/, not committed here.

Component	Detail
ARM9	760,680 bytes, loads at `0x02000000`
ARM7	159,528 bytes, loads at `0x02380000`
ARM9 overlays	32
ARM7 overlays	0
NitroFS	6,676 files across 574 directories

Filesystem layout

apc/  camera/  chat/  common/  counter/  dwc/  ending/  enemy/  event/
item/ movie/   object/ player/  quest/    scene/ set/    sound/  sprite/
titleseq/

Largest: player/ 42M, sound/ 21M, movie/ 20M, scene/ 14M, enemy/ 14M.

Container formats

6,317 of 6,676 files are ZPR-compressed. ZPR is already solved — see DashGL/ArchiveUnpacker, which exposes a zpr() decompressor. Do not re-derive it.

ZPR is a 16-byte header wrapping an XOR-obscured PRS stream — the same Sega LZ77 compression used by Phantasy Star Online, so prs.ts in ArchiveUnpacker is shared with the PSO work.

0x00  char[4]  "ZPR\0"
0x04  u32      checksum / hash
0x08  u32      decompressed size    <- the one the decoder uses
0x0C  u32      (slightly smaller than 0x08; purpose unconfirmed)
0x10  ...      payload

Decode: skip the 16-byte header, XOR every payload byte with 0x95, then PRS decompress to the size at 0x08.

The XOR key is why raw ZPR payloads look like high-entropy noise dominated by 0x95/0x69/0x6a — those are runs of 0x00 in the underlying PRS stream. Anyone eyeballing a hexdump and concluding "this is entropy-coded" has been fooled by the XOR; it is ordinary LZ77 underneath.

Remaining magics: RCSN (NSCR, 88), ZARC (68), RLCN (NCLR, 54), MODS (18), SDAT (13). Extension histogram is dominated by .rel (3,388), .narc (758), .ncgr (477), .ncer (379), .nsbtx (324), .nanr (314).

Most non-.rel formats are stock Nintendo NITRO formats with existing public tooling. The .rel files are Sega-specific and are where the interesting gameplay data lives.

High-value targets already spotted

Path	Guess at contents
`item/wpnprm.rel`	Weapon parameter table
`item/uniprm.rel`	Unit (armour slot) parameter table
`item/genprm.rel`	General/global parameters
`item/mag_param.narc`	Mag growth + feeding tables
`item/weapon_id.narc`, `item/global_dat.narc`	Item ID ↔ data mapping
`enemy/group_*.rel`	Enemy group / spawn definitions
`enemy/<name>.narc`	Per-enemy model + animation + params (1,817 files)
`counter/shop.bin`	Shop inventory & pricing
`counter/item_conversion.bin`	Photon drop / conversion rates
`quest/*/quest_.rel`	Per-quest script data
`set/<stage>/<var>/*.rel`	Per-stage room/spawn set definitions

These names are inferred from filenames only. Verify before trusting.

Prior art — read before writing any extractor

This is the single most important section. A previous session started re-deriving ZPR from scratch when it was already solved in this author's own repos. Check these first, every time:

Resource	What it covers
DashGL/psz-asset-viewer	Existing PSZ asset export: sprites, NCER cells, palettes, models, merged pa00/pa01/pa02 animation packs. Substantial and working.
DashGL/ArchiveUnpacker	`zpr()` decompressor + other archive/compression formats
docs.dashgl.com PSZ formats	Written format docs: NDS header, NARC archive, ZPR
pedro-javierf/NTRGhidra	Ghidra loader for NDS ROMs — handles ARM9/ARM7/overlay mapping so you don't have to import raw binaries at manual base addresses
Starcube Labs: Reverse Engineering a DS Game	General NDS RE methodology walkthrough
AetiasHax/ds-decomp	NDS ROM splitting toolkit (code/asset separation, overlay handling)
tangosdev/sm64ds-decomp	Reference for how far NDS decomp can go, and what it costs
`ref/psz_cheats_usa.xml`	DeadSkullzJr cheat DB, USA build (`C24E`), 86 codes — known-good RAM addresses, see below

The cheat database is a shortcut, use it

ref/psz_cheats_usa.xml contains 86 working Action Replay codes for the exact USA build we're analysing. Every code is a verified RAM address — money, EXP, item slots, stats. Working backwards from a known address to the struct that contains it, and then to the code that writes it, is dramatically faster than searching a 760 KB ARM9 blind.

Codes for the EU (C24P) and JP (C24J) builds also exist in the source database if a cross-region diff is ever useful.

How an agent verifies its own work

Every task in this repo must have a machine-checkable success signal. If a task's only success signal is "kion looks at it and agrees", it does not belong here — it belongs in psz-godot's playtest backlog.

Acceptable oracles, strongest first:

Round-trip. Parse a file to a struct, re-serialise it, compare bytes to the original. Byte-identical means the layout is right. This is the gold standard for every table extractor and should be the default.
Cross-file consistency. A parsed table must have a plausible, uniform record size that evenly divides the decompressed length, with no leftover bytes, across all files of that type — not just the one you tested.
Known-value anchoring. A parsed value must agree with an independently known fact: a cheat-DB address, an in-game value from a wiki, or a string from the text bank.
Cross-region diff. USA vs EU vs JP builds should differ in text and pointers but agree on gameplay constants. Disagreement means a parse bug.

State confidence explicitly in every doc you write. CONFIRMED (round-tripped or anchored), LIKELY (consistent but unanchored), GUESS (filename-derived). Never let a GUESS reach psz-godot unlabelled.

Work breakdown

Roughly dependency-ordered. Each is intended to be one agent's task.

Phase 0 — foundation

Wire ArchiveUnpacker's zpr() into a batch decompressor; decompress all 6,317 ZPR files into build/decompressed/ preserving tree structure
Re-run the magic histogram on the decompressed output — this reveals the real inner formats, which the outer compression currently hides
Import ARM9 + 32 overlays into Ghidra via NTRGhidra; confirm overlay base addresses resolve and code disassembles cleanly
Audit psz-asset-viewer and write docs/prior-art-inventory.md: exactly which formats are already solved there, so nothing is rebuilt twice

Phase 1 — static tables (highest value, lowest risk)

item/wpnprm.rel → data/weapons.json (ATP range, ATA, required stats, class restrictions, element slots). Round-trip verified.
item/uniprm.rel → data/units.json
item/genprm.rel → data/global_params.json
item/mag_param.narc → data/mags.json (growth curves, feed tables)
counter/shop.bin → data/shop.json; counter/item_conversion.bin → data/photon_conversion.json
Enemy stat tables — locate first (may be in .rel or baked into ARM9/ overlay .rodata), then extract to data/enemies.json
EXP curve + level-up stat tables per class → data/level_curves.json
Drop tables → data/drops.json

Phase 2 — logic specs (needs Ghidra)

Damage formula: locate the melee damage-apply path, document the exact arithmetic including rounding and crit handling → specs/damage.md
Hit detection: how many frames is a swing's hitbox active, and is it a window or a single frame? (directly answers psz-godot #554)
Enemy AI state machine: enumerate states and transition conditions → specs/enemy-ai.md
Photon Art / Photon Blast trigger + effect logic
Technique (Force) casting: cost, cast time, power scaling
Quest script VM: opcode table and dispatch → specs/quest-vm.md
Spawn/set system: how set/<stage>/<variant>/*.rel drives room population

Phase 3 — delivery

For each completed spec, open a psz-godot issue with the extracted numbers, linking back to the doc here
Publish data/*.json in a form psz-godot can consume directly

Layout

rom/     original ROM             (gitignored — do not commit)
build/   unpacked ROM output      (gitignored — regenerate with tools/)
ref/     reference data           (large files gitignored)
tools/   extraction scripts
docs/    findings, format notes, prior-art inventory
specs/   behaviour specs destined for psz-godot
data/    extracted tables as JSON

Regenerating `build/`

python3 tools/nds_unpack.py "rom/Phantasy Star 0 (USA).nds" build/
python3 tools/blz.py build/

The first writes arm9.bin, arm7.bin, overlay_arm9/, banner.bin, the full fs/ tree with real filenames recovered from the FNT, and rom_meta.json describing the header, overlay table, and every file's offset/size/magic.

The second step is not optional. arm9.bin and the overlays ship BLZ-compressed; tools/blz.py decompresses them to arm9_dec.bin and overlay_arm9_dec/, and every static-analysis tool here reads those rather than the packed originals. Running only the first command leaves a tree that looks restored and a gate that cannot run — which is exactly what happened once, so tools/buildinputs.py now checks that this section documents a producer for every path the checkers require.

Legal

No copyrighted game data is committed to this repository — no ROM, no extracted assets, no decompressed archives. Everything here is original analysis, tooling, and documentation. Contributors must supply their own legally obtained copy of the ROM.