Skip to content

Friction log — everything we hit, in order

The chronological record: each entry is one thing that actually broke (or silently lied), when, what it looked like, and what fixed it. The condensed symptom→fix reference is FRICTION_POINTS.md; deep per-topic detail lives in NOTES.md. Commits are receipts.

Convention: [where it bit] symptom → root cause → fix.

2026-06-11 — scaffold day (local)

  1. [local env] Fresh workspace defaulted to Python 3.14 → no pyarrow wheels exist for it; nothing imported. → Pin the dev env to 3.13 (uv venv --python 3.13). (f3a2386)
  2. [tests] Schema round-trip "mismatch" that wasn't: parquet renames list fields item→element on write; float32 round-trips are approximate. → Compare with Schema.equals(check_metadata=False); approximate float asserts. (f3a2386)
  3. [ingest] Daft's glob returns file:// URIs for local paths; h5py can't open URIs. → Strip the scheme before opening (_local_path). (f3a2386)

2026-06-12 — first Modal images (build + first GPU run)

  1. [image build] One broken image blocked everything: Modal builds every image an app references, so the unverified VLA-JEPA image failed the OpenVLA smoke too. → Split apps / don't reference unbuilt images. (356b599)
  2. [image build] evdev C extension: linux/input.h missing (pulled via robosuite→pynput). → apt linux-libc-dev. (356b599)
  3. [image build] error: command 'clang' failed: No such file or directory — CUDA runtime base has no compiler, and Modal's add_python interpreter is clang-built. → apt build-essential clang. (356b599)
  4. [runtime] Editable-installed LIBERO (pip install -e) reported success but import libero failed inside the Modal function. → PYTHONPATH=/opt/LIBERO. (Obsoleted later by the hf-libero wheel.) (356b599)
  5. [runtime] EOFError: EOF when reading a line on import libero — LIBERO prompts input() for a dataset path on first import; containers have no stdin. → Pre-write its config at image build. (356b599)
  6. [runtime] Failed to initialize NumPy: _ARRAY_API not found — daft pulled pyarrow 24 which pulled numpy 2.2.6; torch 2.2 is compiled against numpy 1.x. → Pin numpy==1.26.4. (356b599)
  7. [build hygiene] opencv 4.13 declares numpy>=2, warning against the pinned 1.26.4. → Pin opencv-python==4.9.0.80 (last numpy-1-clean line). (356b599)
  8. [first GPU run] ModuleNotFoundError: matplotlib at env construction — pip --no-deps on LIBERO dropped runtime deps that only libero.libero.envs imports; the import-only CPU smoke never touched them. → Add matplotlib, einops to the sim pins. Lesson: import smokes under-test; only real env construction catches these. (356b599)
  9. [first GPU run] AssertionError: The unnorm_key you chose is not in the set of available dataset statistics ... dict_keys(['libero_spatial']) — the LIBERO fine-tunes key their norm stats by suite name, not the <suite>_no_noops dataset name the docs suggest. → Fix the key table + fall back to the sole available key. (98c4e67)

2026-07-02 — the in-process flip (build + first real rollouts)

  1. [image build] egl_probe: "CMake must be installed" — hf-libero depends on a CMake-built C extension; CUDA base has no cmake. → apt cmake. (269ec16)
  2. [runtime] The input() prompt again — it survived into the packaged hf-libero wheel. → Bake the config at build: printf 'n\n' | python -c 'import libero.libero'. (269ec16)
  3. [first in-process rollout] ValueError: At least one stride in the given numpy array is negative — the runner's 180° de-rotation (img[::-1, ::-1]) is a reversed view; PIL-based policies copy implicitly, torch.from_numpy refuses. → np.ascontiguousarray in the adapter. (9ea1c92)

2026-07-02 — the sweeps (scale reveals everything)

  1. [sweep, ~episode 4] ValueError: executing action in terminated episode, then every subsequent episode dead — set_init_state restores sim state but does NOT clear robosuite's internal step counter, which accumulates across episodes toward the horizon (1000) when the env is cached. Invisible in short runs (our 2-episode verification = ~330 cumulative steps). → env.reset() before set_init_state, every episode. openpi's loop does this; now we know why. (7410577)
  2. [sweep results] OpenVLA 0/7, every episode at the 250-step cap — no error, just a plausible-looking robot that never succeeds. One parquet query (min/max gripper_action = [0.000, 0.996]) → the RLDS gripper convention fed raw into LIBERO: the hand could never open, polarity inverted on top. → normalize → binarize → invert (OpenVLA's own eval utils), plus the center-crop its eval applies for aug-trained checkpoints. After: 10/10 on task 0; final sweep 84/100 vs the published 84.7%. (df6c8ee)
  3. [ops] Both detached sweeps died mid-run with "Received a cancellation signal" — modal run --detach survives network drops but a client teardown propagates a cancel. → modal deploy + Function.spawn() (nothing local to kill) + resumable sweeps that skip episodes whose part file already exists (filenames are deterministic episode ids). (b457249)
  4. [analysis] A 500-step episode in a 250-cap suite — episode_id names the episode spec, so both policies produce libero_spatial/5/1/7, and a groupby("episode_id") chimera'd two policies' trajectories into one phantom (113 phantom grasp cycles). Caught only because 500 > 250. → Group by (policy_type, episode_id); schema docs updated; policy-qualified id parked as a breaking change. (b137672)
  5. [observation] After the env.reset() fix, episode libero_spatial/0/1/7 (VLA-JEPA) flipped from fail-at-cap to success — cached-env contamination degrades outcomes well before it hard-crashes. (n=1; GPU nondeterminism not fully excluded.) (noted in NOTES)

2026-07-03 — the Python-3.12 unification

  1. [image rebuild] TypeError: mj_fullM(): incompatible function arguments at env stepping — NOT the interpreter bump: mujoco was unpinned, the June image happened to resolve 3.9.0 (verified), and today's rebuild drifted to a newer mujoco whose mj_fullM binding signature robosuite 1.4.x cannot call. Any rebuild on any Python would have hit this. → Pin mujoco==3.9.0 (the sweep-verified version). The SAME rebuild also drifted scipy to 1.18 (requires numpy>=2, vs our 1.26.4 pin) → pin scipy==1.15.3 too. Lesson: an unpinned transitive dep makes "the same image" a function of the build date — pin the whole verified resolve set around anything numpy-adjacent.

2026-07-06 — ship day: the universal lock

  1. [CI / uv] uv run pytest died resolving the project — uv's universal lock resolves EVERY extra over the FULL requires-python range and all platforms, so one git-dep extra bricks uv lock/uv run for users who never asked for it. Peeling the onion: lerobot's py>=3.12 floor vs our >=3.10 (marker-fixable) → lerobot's numpy>=2 vs core numpy<2 (moved the <2 pin into the openvla extra, where its reason — torch 2.2 — lives) → "no version of torch==2.2.0", which was NOT tags, NOT a yank, NOT platforms (cp312 wheels exist; a minimal repro holding two torch versions across [tool.uv] conflicts splits resolves fine from PyPI): the lerobot GIT dep's own marker-conditional torch pins poison the sibling split in every conflicts shape we tried (uv 0.11.21–0.11.26). → Fix: the vla_jepa extra becomes a documented pointer (the Modal image was always the canonical install), [tool.uv] environments caps the lock at the verified <3.13 (torch 2.2.0 ships no cp313 wheel), .python-version pins the canonical 3.12, and dev deps become a PEP 735 group so fresh-clone uv run pytest just works. CI runs .venv/bin/pytest directly (its env is hand-composed: nightly daft + CPU torch). Lesson: a universal resolver makes one exotic dependency everyone's problem — scope the lock to exactly what you verify.

Standing lessons

  • Loud failures are the cheap ones. Everything in the build section cost minutes. Everything in the "silent" sections cost a sweep — or would have cost a wrong conclusion.
  • Import-only smoke tests under-test. Env construction, GPU inference, and multi-episode sweeps each surfaced a class of bug the previous layer could not see.
  • Record enough to interrogate. Every silent bug in this log was diagnosed from the per-step parquet in one or two queries. That is the whole thesis of the repo.