danvisimhadri commited on
Commit
c6ca84e
Β·
verified Β·
1 Parent(s): 0702a39

Add result renders and the agent handoff

Browse files
.gitattributes CHANGED
@@ -144,3 +144,14 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
144
  2025-10-28T22-19-52-sdfusion_model_img2shape-building-LR1e-5/images/test_step03000_gen_.png filter=lfs diff=lfs merge=lfs -text
145
  2025-10-28T22-19-52-sdfusion_model_img2shape-building-LR1e-5/images/test_step02000_gen_.png filter=lfs diff=lfs merge=lfs -text
146
  2025-10-28T22-19-52-sdfusion_model_img2shape-building-LR1e-5/images/test_step12000_gen_.png filter=lfs diff=lfs merge=lfs -text
 
 
 
 
 
 
 
 
 
 
 
 
144
  2025-10-28T22-19-52-sdfusion_model_img2shape-building-LR1e-5/images/test_step03000_gen_.png filter=lfs diff=lfs merge=lfs -text
145
  2025-10-28T22-19-52-sdfusion_model_img2shape-building-LR1e-5/images/test_step02000_gen_.png filter=lfs diff=lfs merge=lfs -text
146
  2025-10-28T22-19-52-sdfusion_model_img2shape-building-LR1e-5/images/test_step12000_gen_.png filter=lfs diff=lfs merge=lfs -text
147
+ images/band-fix-220000-comparison.jpg filter=lfs diff=lfs merge=lfs -text
148
+ images/band-fix-230000-comparison.jpg filter=lfs diff=lfs merge=lfs -text
149
+ images/band-fix-240000-comparison.jpg filter=lfs diff=lfs merge=lfs -text
150
+ images/band-fix-hollowing-montage.png filter=lfs diff=lfs merge=lfs -text
151
+ images/convergence-run-montage.png filter=lfs diff=lfs merge=lfs -text
152
+ images/decoder-tolerance-montage.png filter=lfs diff=lfs merge=lfs -text
153
+ images/deployed-vs-dora.png filter=lfs diff=lfs merge=lfs -text
154
+ images/final-41epoch-montage.png filter=lfs diff=lfs merge=lfs -text
155
+ images/harness-baseline-montage.png filter=lfs diff=lfs merge=lfs -text
156
+ images/no-op-montage.png filter=lfs diff=lfs merge=lfs -text
157
+ images/surface-loss-before-after.png filter=lfs diff=lfs merge=lfs -text
AGENT-HANDOFF.md ADDED
@@ -0,0 +1,138 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Agent handoff β€” continuing this work on the AMD workstation
2
+
3
+ Companion to [`README.md`](README.md) (the public model card, which holds the results, method,
4
+ literature, and traps). **Read that first.** This file is the operational half: environment, layout,
5
+ and what to actually do next.
6
+
7
+ Written 2026-08-04, at the migration off the Gilbreth cluster.
8
+
9
+ ---
10
+
11
+ ## 1. The machine changed, and it is AMD
12
+
13
+ | | |
14
+ |---|---|
15
+ | CPU | AMD Ryzen AI Max+ 395 (Strix Halo), 16C/32T |
16
+ | GPU | **Radeon 8060S iGPU β€” gfx1151, RDNA 3.5** |
17
+ | memory | 128 GB unified (61 GB system + 64 GB VRAM carve-out), 8 GB swap |
18
+ | disk | 1.9 TB NVMe, ~1.4 TB free |
19
+ | OS | Ubuntu 26.04 |
20
+
21
+ Everything before this ran on an **A100 80GB with CUDA 12.6**. The port was assessed by reading
22
+ imports, not by assumption:
23
+
24
+ **βœ… What is fine**
25
+ - `scripts/train_vecset.py` imports only numpy / torch / h5py + project modules. **Training does not
26
+ need pytorch3d.**
27
+ - **No custom CUDA kernels anywhere in project code** β€” no `CUDAExtension`, no `cpp_extension.load`.
28
+ - No flash-attn, xformers, deepspeed, apex, or bitsandbytes. The usual ROCm blockers are all absent.
29
+ - πŸ”‘ **The ~40 hard-coded `.cuda()` / `device='cuda'` call sites are NOT a problem.** ROCm PyTorch keeps
30
+ the `torch.cuda` namespace and maps it onto HIP. **Do not "fix" them** β€” rewriting them to
31
+ `device_type` strings is churn that buys nothing and risks breaking the CUDA path.
32
+
33
+ **⚠️ The one real blocker β€” pytorch3d**
34
+ - No ROCm wheels exist. Needed by `utils/util_3d.py` and `scripts/foundations/eval_massing_arms.py`
35
+ (**the harness**) for `MeshRasterizer` / `MeshRenderer`.
36
+ - Build it **CPU-only** (`FORCE_CUDA=0`). At n=48 buildings the rasterisation cost is tolerable.
37
+ - If the CPU build also fights you, the rendering is separable from the metrics β€” `fp_iou`, `missing`,
38
+ `extra`, `vol_iou` are voxel operations and do not need pytorch3d. Only the montages do.
39
+
40
+ **⚠️ Do NOT `pip install -r requirements-frozen.txt` verbatim.** It pins 14 `nvidia-*-cu12` packages,
41
+ `torch==2.8.0+cu126`, `torchvision==0.23.0+cu126`, and `triton==3.4.0`. Install ROCm torch wheels first,
42
+ then the non-torch remainder. Note ROCm's official Ubuntu support may lag 26.04; gfx1151 support is
43
+ recent (ROCm 6.4+).
44
+
45
+ **⚠️ Expect materially slower training.** The denoiser is only 49M params and fits the 64 GB carve-out
46
+ with enormous headroom, but an integrated GPU is not an A100. Budget accordingly before planning a
47
+ 240k-step run β€” the runs behind this work were ~10–11 GPU-hours *each* on A100.
48
+
49
+ ---
50
+
51
+ ## 2. What to bring over
52
+
53
+ **Do not copy the cluster tree.** It is 1.4 TB: `data/` 493 GB, `logs_building/` 406 GB, `legacy/`
54
+ 369 GB. Roughly 700 GB of that is superseded dense-grid snapshots β€” **delete, don't copy.**
55
+
56
+ | item | how |
57
+ |---|---|
58
+ | code, docs, results, montages | `git clone`, branch `massing-solid-gate-retrain` |
59
+ | the corpus | **regenerate** β€” 25 MB in git rebuilds 35 GB SDF + 17.4 GB latents (`REPRODUCING.md` Β§4) |
60
+ | model weights | this folder β€” **945 MB** for the five vecset checkpoints |
61
+ | `stage3a_lod2_deployed.pth` | 7.2 GB, **optional** β€” superseded baseline, comparison arm only |
62
+ | Dora-VAE (`dora_vae_1_1.ckpt`) | 2.1 GB, re-download from Hugging Face |
63
+
64
+ Regenerated `data/` lands near ~55 GB, not 493 β€” `REPRODUCING.md` Β§4d notes `nrw.h5` and `plateau.h5`
65
+ are intermediate staging nothing current uses.
66
+
67
+ Verify after transfer: `sha256sum -c SHA256SUMS`.
68
+
69
+ ---
70
+
71
+ ## 3. Where things are
72
+
73
+ | what | path |
74
+ |---|---|
75
+ | **THE harness** β€” 48 pinned ids, all arms, one pass | `scripts/foundations/eval_massing_arms.py` |
76
+ | cheap tracker (has a no-op detector; **not authoritative**) | `scripts/foundations/probe_vecset_checkpoint.py` |
77
+ | training | `scripts/train_vecset.py` (`--resume`, `--surf_weight`, `--surf_t_center`, `--archive_every`) |
78
+ | the model | `models/networks/vecset_denoiser.py` |
79
+ | the gradient path | `DoraCodec(differentiable=True).freeze()` β€” off by default |
80
+ | corpus loading | `load_surfaces` β€” ⚠️ **never read the h5 directly**, it is inward-wound |
81
+ | frame conversion | `scene.surface_sampling.to_array_frame`, guarded by `verify_frame` |
82
+ | results artifacts | `execution/artifacts/massing_arms_eval_*.json` |
83
+ | reasoning trail | `docs/wayfinding/vecset-convergence/` |
84
+
85
+ `--ids_from` replays a pinned id set so runs stay comparable. **Use it.** Every number in the model
86
+ card is on the same 48 ids; a number from a different sample is not comparable to any of them.
87
+
88
+ ---
89
+
90
+ ## 4. What to do next
91
+
92
+ **The open question: what separates the 29 solid buildings from the 19 hollow ones?**
93
+
94
+ This is the first *specific* investigation this project has had β€” every prior step was a blind lever
95
+ pull. If the split correlates with something legible, it is a targeted fix rather than a sweep.
96
+
97
+ - ❌ **Building size is ruled out** *(tested 2026-08-04)*. Median GT volume 50,515 (solid) vs 43,554
98
+ (hollow), Mann-Whitney **p=0.246**, point-biserial r=0.186 (p=0.204). Not significant at n=48.
99
+ - οΏ½οΏ½ **Untested:** footprint complexity (vertex count, concavity, aspect ratio), source corpus
100
+ (3DBAG / NRW / PLATEAU), height, roof-form class.
101
+
102
+ Secondary, cheaper: **lower `--surf_weight` at the same band** β€” the collapses may simply be the term
103
+ overshooting on harder cases.
104
+
105
+ **Then, in priority order:**
106
+
107
+ 1. **Criterion 2 β€” footprint fidelity 0.962 β†’ 1.000.** This is the *hard, non-negotiable* gate and it
108
+ was under-weighted for a whole cycle in favour of 3D IoU, which the specification marks
109
+ diagnostic-only. Weight it first.
110
+ 2. **Add a collapse rate to the harness.** It reports medians, which is correct for unimodal noise and
111
+ actively misleading here. The bimodality was caught by eye, from a render β€” not by the metrics.
112
+ 3. **#79 β€” SNE (sharp-normal-error).** The only proposed instrument that might separate crisp from
113
+ melted, and still absent from the harness. ⚠️ `guard_roughness` is **not cross-arm comparable** β€”
114
+ do not use it to compare models.
115
+ 4. **#82 β€” footprint-only height inference.** Until this lands, the claim is *"footprint + height β†’
116
+ mass"*, not *"footprint alone"*. **Keep that exact wherever the work is written up.**
117
+
118
+ ---
119
+
120
+ ## 5. Read this before trusting any number
121
+
122
+ The full list is in the model card's **Measurement traps** section. The three that will actually cost
123
+ you time:
124
+
125
+ 1. **Never extrapolate the training curve.** It went 0.719 β†’ 0.657 β†’ 0.532 β†’ **0.840**. Two separate
126
+ runs were nearly killed during multi-checkpoint collapses that recovered. A 30,000-step window of
127
+ garbage output is not evidence of a dead run *in this model*.
128
+ 2. **Always report `vs input` beside any quality number.** The generator scores well by *declining to
129
+ act* β€” at s=0.45 it returned its input at 99.9% and inherited its score. A model that does nothing
130
+ looks excellent on every aggregate metric here.
131
+ 3. **Render before you conclude.** The aggregate went 0.195 β†’ 0.200 ("better") while a building went
132
+ from a box to a shredded cage. Both bimodality and the cage failure were caught by eye, never by a
133
+ scalar. This project is judged visually and the metrics have repeatedly failed to see what mattered.
134
+
135
+ ⚠️ **One historical correction to be aware of:** a "beat 0.840 3D IoU" bar appears throughout older
136
+ documents. It is **retired**. The criteria are: (1) human visual judgement β€” primary, (2) footprint
137
+ match β€” hard gate, (3) 3D IoU β€” diagnostic only. A retired criterion was re-imposed and dominated the
138
+ framing of two tickets before this was caught.
images/band-fix-220000-comparison.jpg ADDED

Git LFS Details

  • SHA256: 91b318a05d20d24d7664c172edf64c64389298520395efde6032ebf4e45d40f6
  • Pointer size: 131 Bytes
  • Size of remote file: 156 kB
images/band-fix-230000-comparison.jpg ADDED

Git LFS Details

  • SHA256: a65eba57737c260685876827798682a18551bd8213767afcf0ed2623b770fcda
  • Pointer size: 131 Bytes
  • Size of remote file: 135 kB
images/band-fix-240000-comparison.jpg ADDED

Git LFS Details

  • SHA256: 373ec4a393951adb8e0bfedbfa474624e48703f21114700d97f0824ee16eb491
  • Pointer size: 131 Bytes
  • Size of remote file: 142 kB
images/band-fix-hollowing-montage.png ADDED

Git LFS Details

  • SHA256: 8c3c74f60892c145d0622d8b1e950a06ddac6aa214fdc57a36baaffafaac19a4
  • Pointer size: 131 Bytes
  • Size of remote file: 302 kB
images/convergence-run-montage.png ADDED

Git LFS Details

  • SHA256: 999922c1fbbb8be3366edf00e58bf8121df022581504b7aede9324151bc369f8
  • Pointer size: 131 Bytes
  • Size of remote file: 380 kB
images/decoder-tolerance-montage.png ADDED

Git LFS Details

  • SHA256: 90e5efaf9009a1bfbff983a4b13e8a4c0b8206f16eaa636480d991550c48a901
  • Pointer size: 131 Bytes
  • Size of remote file: 923 kB
images/deployed-vs-dora.png ADDED

Git LFS Details

  • SHA256: f94f441635f8c253c2a9afcd86790b417d1b63d1b979aba3956d25616221f054
  • Pointer size: 131 Bytes
  • Size of remote file: 230 kB
images/final-41epoch-montage.png ADDED

Git LFS Details

  • SHA256: e8609b8fe6fcd06999215ef8b6dd4f1318870415c472fa34cababd03142596bb
  • Pointer size: 131 Bytes
  • Size of remote file: 347 kB
images/harness-baseline-montage.png ADDED

Git LFS Details

  • SHA256: 0b15476ea72a28505536a50c769cea16fee3f4f1592e2b91496fbd092ba9ce4a
  • Pointer size: 131 Bytes
  • Size of remote file: 523 kB
images/no-op-montage.png ADDED

Git LFS Details

  • SHA256: f09b5beb664cf36941633a7586e5134f179d97351ec458ce663eb6f6bd9210ef
  • Pointer size: 131 Bytes
  • Size of remote file: 229 kB
images/surface-loss-before-after.png ADDED

Git LFS Details

  • SHA256: 55ea1d65c911df0f2686ce3620af599bec4d64d314acc5622a72f64ab23b5323
  • Pointer size: 131 Bytes
  • Size of remote file: 129 kB