[AI Generated] Add `extract-asset-metadata` binary target.

sessamekesh commented

2026-10-12 21:19:17 +00:00

Owner

The purpose of this library up to this point has been exclusively game runtime asset loading, with a couple binaries for offline transcoding + bundling.

I'd like to add a bit more robustness to allow more asset pipeline integration use cases though, i.e. frontends for building igasset and igpack files.

This PR adds a new staged-asset-io library that expands the existing Filesystem facade between tooling (igasset-gen and igpack-bundle) and the filesystem. This is a precursor to adding a new extract-asset-metadata binary which will be used for identifying a raw input asset (image file, 3D model file, etc.) and extracting information about how it can be used with other offline tools.

There should be (and to an extent, is) some extra security concerns here - if the frontend accepts untrusted resources, malicious input should be considered. To an extent that case is considered (most notably the staged-asset-io mapping that allows users to sandbox the filesystem at the application level), but any application that accepts untrusted data SHOULD VERY MUCH still enforce their own sandboxing rules (i.e. run this process in an isolated environment like Docker).

AI usage disclosure

This PR was overwhelmingly, almost exclusively, authored with generative AI. Two caveats I want to add here:

No AI generated code from this PR is in the igasset library, which is the core runtime library.
This PR requires full human review and will have human-authored fixes.

I don't put those caveats there because I think AI is fundamentally worse at authoring such non-novel code as this, I put it there to highlight that I don't like having AI ownership and accountability. I, @sessamekesh, stubbornly take responsibility for code authored to this code base and under my name. If something is wrong here, GLM 5.3 is literally incapable of caring.

The purpose of this library up to this point has been exclusively game runtime asset loading, with a couple binaries for offline transcoding + bundling. I'd like to add a bit more robustness to allow more asset pipeline integration use cases though, i.e. frontends for building `igasset` and `igpack` files. This PR adds a new `staged-asset-io` library that expands the existing `Filesystem` facade between tooling (`igasset-gen` and `igpack-bundle`) and the filesystem. This is a precursor to adding a new `extract-asset-metadata` binary which will be used for identifying a raw input asset (image file, 3D model file, etc.) and extracting information about how it can be used with other offline tools. There **should be** (and to an extent, is) some extra security concerns here - if the frontend accepts untrusted resources, malicious input should be considered. To an extent that case is considered (most notably the `staged-asset-io` mapping that allows users to sandbox the filesystem at the application level), but **any application that accepts untrusted data SHOULD VERY MUCH still enforce their own sandboxing rules** (i.e. run this process in an isolated environment like Docker). ## AI usage disclosure This PR was overwhelmingly, almost exclusively, authored with generative AI. Two caveats I want to add here: 1. No AI generated code from this PR is in the `igasset` library, which is the core runtime library. 2. This PR requires full human review and will have human-authored fixes. I don't put those caveats there because I think AI is fundamentally worse at authoring such non-novel code as this, I put it there to highlight that I don't like having AI ownership and accountability. I, @sessamekesh, stubbornly take responsibility for code authored to this code base and under my name. If something is wrong here, GLM 5.3 is literally incapable of caring.

sessamekesh added 5 commits

2026-10-12 21:19:18 +00:00

Add staged-asset-io facade + manifest; sync I/O swap; narrow Assimp importers to FBX/glTF/OBJ at v6.0.5 7f6aa4387c

Phase 0 - synchronous I/O: toolutils::Filesystem is now a sync facade (read_bin/write_bin with no task list); igasset-gen drops its io thread pool and keeps the exec pool for CPU work; processors collapsed to single exec tasks (basisu per-output compression is now genuinely parallel via PromiseCombiner); igpack-bundle no longer links igasync.

Phase 1 - staged-asset-io: two-policy filesystem facade (direct = historical operator/ semantics; staged = deny-by-default manifest capability list), logical-name normalization, thread-safe access journal, Assimp IOSystem adapters (staged + recording) and a shared Assimp import entry point; schema/staged-asset-manifest.fbs; Assimp bumped v6.0.4 -> v6.0.5 with importers narrowed to FBX/glTF/OBJ; igasset-gen now uses stagedio::load_scene. Includes a 67-check selftest.

Also adds STAGED-ASSET-PIPELINE.md - self-contained design + status handoff document for the staged asset pipeline initiative (Phases 2-3 planned: extract-asset-metadata tool + igasset-gen staged rewiring).

Scaffold extract-asset-metadata (CLI + renderers + manifest extra entries) 9235389bc3

Restore buildability after the crashed Phase-2 start: implement
StagedAssetManifest::with_extra_entry (validated runtime mapping for
inputs missing from the orchestrator manifest), land the extract-asset-
metadata scaffold (CLI contract, size cap, chunked SHA-256, detection
orchestration, action advice, dependency finalization, text + JSON
renderers), and stub the detector bodies pending Phase-2 implementation.
Selftest: 77 checks ALL PASS; igasset-gen e2e 23/23; igpack-bundle e2e
23/23.

Verify clang + Ninja parity; ignore local build dirs 198a930f8d

Match the CI compiler config exactly (clang 21.1.8 + Ninja + scan-for-
modules off, igasync tests/examples off) in a fresh build-clang/ tree:
all dependencies rebuilt from scratch with zero errors and zero parity
fixes. Validated against the clang binaries: staged-asset-io-selftest
77/77, igasset-gen e2e 23/23, igpack-bundle e2e 23/23, SHA-256 matches
GCC + sha256sum. build*/ gitignored.

Implement extract-asset-metadata detectors, dependency extraction, --deep f34d732bff

All light-scan detection (image magics + header-only stb inspection with
PNG chunk / JPEG SOF / GIF frame-walk / PNM detail, binary FBX + GLB v1/v2
magics, OBJ/MTL/glTF/FBX-ASCII text heuristics, WGSL entry points, Assimp
content sniff), dependency finalization (normalize + truncate + status +
multi-source dedupe), and full --deep imports (staged manifest synthesis +
base_dir-aware IOSystem; direct recording IOSystem; mesh/material/
animation inventory + selection groups).

Fixes along the way: AssimpStagedIOSystem now resolves requests relative
to the model's logical parent dir before the qualified form (OBJ opens
raw mtllib refs; glTF self-prefixes - single-step normalize broke nested
logical paths; selftest 77->85); load_scene surfaces Assimp's error to
--deep callers; direct-mode probes use the non-throwing exists() (paths
over NAME_MAX from untrusted refs terminated the process); journal-derived
dependencies are canonical names and are never re-normalized (double
prefix bug); GLB v1 20-byte header support.

Validated on GCC + clang: selftest 85/85, igasset-gen e2e 23/23,
igpack-bundle e2e 23/23, functional smokes across every kind and exit
code.

Add extract-asset-metadata fixtures, e2e suite, and CI coverage

igasset-gen Integration Tests / build-and-test (pull_request_target) Successful in 9m53s

Details

Build WASM Library / wasm-build (pull_request_target) Successful in 11m27s

Details

8cefaaea6c

Hand-authored fixtures (triangle.obj+mtl with a stable deep mesh name,
multi-file scene.gltf+bin, generated tex.png, entry-points.wgsl) plus
box.fbx copied from assimp's vendored test suite (provenance in
test_assets/README.md; no Blender locally, assimp exporters disabled).
31-test pytest e2e suite per the plan: direct detection, JSON round-trip,
staged mode over nested logical names (two-step resolution coverage),
--deep inventories + journal-refined statuses + exit 3, and the full
exit-code contract. CI workflow now builds the new tool + the
staged-asset-io selftest and runs the new suite.

Fix found by the suite: Importer::GetImporterIndex compares its argument
verbatim against importer extension lists - deep reports silently lacked
importer_name when given a full path; now passed the bare extension.

Validated on GCC + clang: selftest 85/85, extract e2e 31/31, igasset-gen
e2e 23/23, igpack-bundle e2e 23/23.

sessamekesh changed title from ~~WIP: [AI Generated] Add extract-asset-metadata binary target.~~ to [AI Generated] Add extract-asset-metadata binary target.

2026-10-12 21:40:50 +00:00

sessamekesh commented

2026-10-12 21:41:27 +00:00

Author

Owner

Phases 1+2 LGTM

sessamekesh added 1 commit

2026-10-12 21:41:43 +00:00

Change a couple comments

Build WASM Library / wasm-build (pull_request_target) Successful in 1m41s

Details

igasset-gen Integration Tests / build-and-test (pull_request_target) Successful in 2m23s