NC versions of PreImages, PreImagesSet, PreImagesElm and PreImagesRepresentative - version 2

doc/ref/mloop.xml

@@ -1,4 +1,5 @@
-<!-- %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -->
+<!-- %%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -->
 <!-- %% -->
 <!-- %A  mloop.xml                  GAP documentation             Martin Schönert -->
 <!-- %% -->
@@ -477,6 +478,7 @@ indicate that you are in a break loop.
 <Log><![CDATA[
 gap> 1/0;
 Error, Rational operations: <divisor> must not be zero
+Stack trace:
 not in any function at *stdin*:2
 you can enter 'quit;' to quit to outer loop
 ]]></Log>

etc/emscripten/README.md

@@ -2,7 +2,10 @@
 
 Build GAP as a WebAssembly module and serve it as a self-contained website.
 The terminal interface uses [xterm-pty](https://github.com/mame/xterm-pty),
-so the resulting page behaves like a normal GAP REPL.
+so the resulting page behaves like a normal GAP REPL. The page is styled
+to match <https://www.gap-system.org> (Ubuntu fonts, GAP logo, crimson
+accents, dark-mode support) and adds a side panel for moving files in and
+out of the browser session, an examples menu, and a restart button.
 
 ## Quick start
 
@@ -50,6 +53,81 @@ TTY device so `tcsetattr()` reaches the line discipline. Both are sensitive
 to the toolchain version (a newer emsdk leaves `tcsetattr` unhonoured, so
 typed input is echoed twice), so re-test the REPL when changing emsdk.
 
+## The `io` package
+
+The wasm build has no `dlopen`, so a package kernel extension cannot be
+loaded the usual way. `build.sh` works around this for `io` by compiling
+its single C file into the GAP kernel and registering it in the
+static-module table (`src/compstat.c`), so `LoadKernelExtension("io")`
+finds it without `dlopen`. This is the GASMAN-safe approach: `io` ends up
+in the one wasm module, so ASYNCIFY's link-time pass instruments it
+uniformly and `emscripten_scan_registers()` sees its frames (unlike a
+dynamically loaded side module). The mechanism is detailed in the
+comments in `build.sh` around the `IO_OBJ` block, with the registration
+helper in `register_static_module.py`.
+
+`io` is the kernel dependency of several otherwise pure-GAP packages
+(simpcomp, fr, rcwa, unitlib, …), so linking it in lets them load. Its
+file/directory/time functions work; its socket/fork/select functions
+compile but fail at runtime, since wasm has no such facilities.
+
+## The page: file transfer, examples, restart
+
+GAP runs with its working directory set to `/home/web_user` in the
+virtual filesystem (the GAP tree itself sits at `/`, passed via `-l /`).
+That directory starts empty, so everything in it is the user's: uploads
+land there, and anything GAP creates there (`PrintTo`, `LogTo`, …) shows
+up in the page's Files panel for download.
+
+The design is shaped by one constraint. Once GAP's `main()` starts, the
+worker thread never returns to its event loop: every terminal operation
+blocks in `Atomics.wait` inside xterm-pty's `TtyClient` until the main
+thread responds. So `postMessage` *to* the worker is never delivered
+after startup, while `postMessage` *from* the worker (the tty request
+channel itself) always works. Hence:
+
+- **Uploads** go through a `SharedArrayBuffer` mailbox (chunked, see the
+  protocol comment in `gap-worker.js`; the sending side is in
+  `gap-ui.js`). The worker drains it at the start of every terminal
+  operation — the moments it is provably awake. An upload made while GAP
+  sits idle at its prompt therefore completes on the next terminal
+  activity; in particular, the Files panel's "insert `Read("file");`"
+  button pastes keystrokes whose processing drains the mailbox before
+  GAP ever sees the Enter, so the file is always in place in time.
+- **Downloads** are pushed: when GAP asks for input (output has settled,
+  files are complete), the worker rescans `/home/web_user` and posts new
+  or changed files to the page, which caches the bytes. A download click
+  is served from that cache — it never has to ask the possibly-blocked
+  worker. This is also why a freshly written file only appears in the
+  panel once GAP is back at a prompt (or asks for input).
+- Inserted commands (file hints and the examples menu) are pasted
+  *without* a newline; the user always presses Enter, so nothing is
+  auto-executed.
+
+### Cache invalidation across redeployments
+
+Library files are cached in the browser's IndexedDB and reused on later
+visits without revalidation. To stop a redeployment from pairing a new
+kernel with stale cached library files, `assemble-website.sh` writes a
+fresh `build-id` file into the site; `gap-fs.js` compares it (fetched
+with `cache: no-cache`) against the copy stored in the cache and
+discards the cache when it differs. The page footer also has a "Reset
+cached data" link that drops the cache and any registered service
+worker and reloads — the escape hatch for any stuck client state.
+
+Known limitations, all consequences of the same architecture:
+
+- Ctrl-C cannot interrupt a running computation: there are no signals,
+  the main thread cannot reach the worker's wasm heap, and GAP cannot
+  poll JavaScript state while computing. The Restart button (which
+  terminates and respawns the worker) is the only escape from a runaway
+  computation. The Files panel keeps its cached entries across a
+  restart, still downloadable; inserting one re-uploads it first.
+- User files do not persist across reloads or restarts (syncing them to
+  IndexedDB would need the worker's event loop, which is starved).
+- The terminal is sized to the window before GAP starts; GAP does not
+  notice later resizes (no SIGWINCH), though xterm itself reflows.
+
 ## Hosting
 
 The xterm-pty terminal uses `SharedArrayBuffer`, which browsers only allow
@@ -71,12 +149,13 @@ re-fetches resources through a service worker that adds the headers).
 | ---- | ---- |
 | `build-in-docker.sh` | One-stop entry point. Builds the image, runs `build.sh` inside, then `assemble-website.sh`. |
 | `Dockerfile` | Pinned `emscripten/emsdk:3.1.23` with autotools, python3, bison/byacc/m4, and a baked-in copy of the GAP package distribution tarball at `/opt/gap-packages.tar.gz`. |
-| `build.sh` | Configures and builds GMP, zlib, and GAP itself for wasm. |
+| `build.sh` | Configures and builds GMP, zlib, and GAP itself for wasm. Also statically links the `io` package's kernel module into the kernel (see below). |
 | `assemble-website.sh` | Copies the build outputs and data directories (`pkg`, `lib`, `grp`, …) into `web-example/`. |
 | `generate_gap_fs_json.py` | Reads file paths on stdin, writes `gap-fs.json` (the manifest of every file in the virtual FS). |
 | `startup_manifest.json` | List of files to fetch eagerly at startup, captured from a real GAP run. Anything not in this list is fetched lazily on first read. See "Updating the startup manifest" below for how to refresh it. |
 | `serve.py` | Local server that adds the COOP/COEP headers. |
-| `web-template/` | Static UI: `index.html`, the worker scripts, the FS init shim, and the COOP/COEP service worker for hosts where you can't set headers. |
+| `web-template/` | Static UI: `index.html`, the page logic (`gap-ui.js`), the worker scripts, the FS init shim, and the COOP/COEP service worker for hosts where you can't set headers. |
+| `web-template/vendor/` | Pinned local copies of xterm, xterm-pty, the fit addon, the GAP logo and the Ubuntu fonts, so the deployed site has no CDN dependency. See `web-template/vendor/README.md` for versions and how to update them. |
 
 ## Updating the startup manifest
 
@@ -111,8 +190,8 @@ To regenerate it after such changes:
    `window.fetchedUrls` is an array of every unique URL the worker
    requested. Chrome/Firefox provide a `copy()` console helper:
    `copy(JSON.stringify(fetchedUrls))` puts the JSON on your clipboard.
-5. Strip non-GAP-FS entries (`gap.js`, `gap.wasm`, `gap-fs.json`, the
-   xterm CDN URLs) and write the result to
+5. Strip non-GAP-FS entries (`gap.js`, `gap.wasm`, `gap-fs.json`, and any
+   `vendor/` assets) and write the result to
    `etc/emscripten/startup_manifest.json` (so it's checked in and gets
    picked up by the next `assemble-website.sh`). A `jq` filter that
    keeps just GAP filesystem paths:

etc/emscripten/assemble-website.sh

@@ -26,7 +26,7 @@ done
 rm -rf "$OUT_DIR"
 mkdir -p "$OUT_DIR"
 
-cp "$SCRIPT_DIR"/web-template/* "$OUT_DIR"/
+cp -R "$SCRIPT_DIR"/web-template/* "$OUT_DIR"/
 cp "$SCRIPT_DIR"/startup_manifest.json "$OUT_DIR"/
 cp gap.js gap.wasm gap-fs.json "$OUT_DIR"/
 # Emscripten only emits a separate gap.worker.js for pthread builds; this
@@ -41,20 +41,27 @@ cp LICENSE COPYRIGHT "$OUT_DIR"/
 # loaded (if listed in startup_manifest.json) or lazily fetched on first
 # read by Emscripten's createLazyFile.
 #
-# -L dereferences symlinks so the output tree is self-contained (a setup
-# where pkg/X is a symlink into a separate checkout still works). Some
-# packages ship dangling symlinks as build artefacts (e.g. pkg/vole's
-# rust/target/*.dSYM); cp prints those and exits non-zero but still copies
-# everything else, so we don't let that abort the run. The assertion below
-# catches a genuinely incomplete copy.
+# tar -h dereferences symlinks so the output tree is self-contained (a
+# setup where pkg/X is a symlink into a separate checkout still works);
+# --exclude .git keeps such checkouts' git internals out of the shipped
+# site. Some packages ship dangling symlinks as build artefacts (e.g.
+# pkg/vole's rust/target/*.dSYM); tar reports those and exits non-zero
+# but still copies everything else, so we don't let that abort the run.
+# The assertion below catches a genuinely incomplete copy.
 for d in pkg lib grp tst doc hpcgap dev benchmark; do
-    cp -RL "$d" "$OUT_DIR"/ 2>/dev/null || true
+    tar -c -h --exclude '.git' -f - "$d" 2>/dev/null | tar -x -C "$OUT_DIR" -f - || true
 done
 
 # pkg/log holds package test logs (.log/.err/.out) that are never read at
 # runtime; keep them out of the shipped site.
 rm -rf "$OUT_DIR/pkg/log"
 
+# A fresh id per assembled site. gap-fs.js compares it against the copy
+# stored in the browser's IndexedDB cache and discards the cache when it
+# differs, so a redeployment can never pair a new kernel with stale
+# cached library files.
+echo "$(date -u +%Y%m%dT%H%M%SZ)-$$" > "$OUT_DIR/build-id"
+
 # The library bootstrap must be present, or GAP 404s during startup.
 if [[ ! -f "$OUT_DIR/lib/init.g" ]]; then
     echo "Error: lib/init.g missing from $OUT_DIR after copy." >&2

etc/emscripten/build.sh

@@ -96,9 +96,17 @@ if [[ ! -f GNUmakefile ]] || ! grep -q '/emcc' GNUmakefile; then
     # emcc would be regenerated by a non-executable JS shim; stale .o
     # files have the wrong architecture. Configure regenerates build/.
     rm -rf build ffgen
+    # PTHREAD_CFLAGS/PTHREAD_LIBS: configure unconditionally adopts
+    # -pthread (for the sake of native kernel extensions, see configure.ac),
+    # but under emcc that flag enables full USE_PTHREADS mode — shared
+    # wasm memory, a helper worker, and a slow path when combined with
+    # ALLOW_MEMORY_GROWTH. GAP itself is single-threaded here, and
+    # emscripten's libc provides single-threaded pthread API stubs, so
+    # pre-seeding the variables makes AX_PTHREAD settle on "no flags".
     emconfigure ./configure ABI=32 \
         --with-gmp="$AUX_PREFIX" \
         --with-zlib="$AUX_PREFIX" \
+        PTHREAD_CFLAGS=" " PTHREAD_LIBS=" " \
         LDFLAGS="-s ASYNCIFY=1 -O2"
 fi
 
@@ -125,13 +133,76 @@ fi
 # Copy host-built generated sources into place
 cp native-build/build/c_*.c native-build/build/ffdata.* src/
 
+# Statically link the 'io' package's kernel module into the GAP kernel.
+#
+# The wasm build has no dlopen, so a package kernel extension cannot be
+# loaded the usual way. But GAP's loader already supports STATIC modules:
+# LoadKernelExtension("io") (in io's init.g) checks SHOW_STAT() -- the
+# list of names in CompInitFuncs[] (src/compstat.c) -- and loads a match
+# without dlopen (see lib/files.gd). So we compile io's single C file into
+# the kernel and add it to that table; no change to GAP's loader or io's
+# GAP code is needed. This is the safe approach for GASMAN: io ends up in
+# the one wasm module, so ASYNCIFY's link-time pass instruments it
+# uniformly with the rest of the kernel and emscripten_scan_registers
+# sees its frames (unlike a dlopen'd side module).
+#
+# io is the kernel dependency of several otherwise pure-GAP packages
+# (simpcomp, fr, rcwa, unitlib, ...). Its socket/fork/select-based
+# functions compile (emscripten declares the symbols) but fail at runtime
+# under wasm; its file/directory/time functions work.
+IO_OBJ=""
+if [[ -d pkg/io ]]; then
+    IO_PKG="$PWD/pkg/io"
+    # io's configure (autoconf feature probe) writes gen/pkgconfig.h, which
+    # io.c needs. FIND_GAP may fail without a fully built GAP in $BASEDIR
+    # and exit non-zero, but pkgconfig.h is still generated by then, so we
+    # tolerate the exit code and assert on the file instead.
+    (
+        cd "$IO_PKG"
+        [[ -f configure ]] || ./autogen.sh
+        emconfigure ./configure --with-gaproot="$BASEDIR" || true
+    )
+    if [[ ! -f "$IO_PKG/gen/pkgconfig.h" ]]; then
+        echo "Error: io's pkgconfig.h was not generated; cannot link io." >&2
+        exit 1
+    fi
+    # io.c (via gap_all.h) includes generated kernel headers. The normal
+    # build generates these before compiling any object; since we compile
+    # io.o ahead of that, generate them first.
+    emmake make build/version.h build/config.h
+    # Compile io.c, renaming its generic Init__Dynamic to the unique
+    # Init__io that the static-module table references. ASYNCIFY is a
+    # link-time pass, so it is not (and must not be) given here.
+    emcc -m32 -c "$IO_PKG/src/io.c" -o "$AUX_BUILD/io.o" \
+        -DInit__Dynamic=Init__io \
+        -Isrc -Ibuild -Isrc/extra \
+        -I"$IO_PKG/gen" -I"$IO_PKG/src" \
+        -I"$AUX_PREFIX/include" \
+        -fPIC -fno-strict-aliasing -O2
+    python3 etc/emscripten/register_static_module.py io src/compstat.c
+    IO_OBJ="$AUX_BUILD/io.o"
+fi
+
 # Build the file list that will be served. -L follows symlinks so that
 # users' local development setups (e.g. replacing pkg/foo with a symlink
 # to a git checkout under git/foo) are picked up; -type f then drops any
 # symlinks themselves.
 find -L pkg lib grp tst doc hpcgap dev benchmark -type f ! -path 'pkg/log/*' \
+        ! -path '*/.git/*' \
     | python3 etc/emscripten/generate_gap_fs_json.py
 
+# Emscripten only emits gap.worker.js for pthread builds; remove any
+# leftover from a previous configuration so assemble-website.sh can't
+# ship a stale one.
+rm -f gap.worker.js
+
+# Memory: start small and grow on demand (sbrk stays contiguous, which
+# GASMAN's workspace extension requires, since wasm memory grows in
+# place). A 2GB up-front allocation was refused outright on iOS.
+# $IO_OBJ (if set) is the statically-linked io module object; it goes on
+# the link line, where the Init__io reference from src/compstat.c pulls it
+# in. LDFLAGS flows into GAP_LDFLAGS (see Makefile.rules), which is part of
+# the final link command.
 emmake make -j"$JOBS" \
-    LDFLAGS="-lidbfs.js -s ASYNCIFY=1 -sTOTAL_STACK=32mb -sASYNCIFY_STACK_SIZE=32000000 -sINITIAL_MEMORY=2048mb -O2" \
+    LDFLAGS="-lidbfs.js -s ASYNCIFY=1 -sTOTAL_STACK=32mb -sASYNCIFY_STACK_SIZE=32000000 -sINITIAL_MEMORY=256mb -sALLOW_MEMORY_GROWTH=1 -sMAXIMUM_MEMORY=2048mb $IO_OBJ -O2" \
     EXEEXT=".html"

etc/emscripten/register_static_module.py

@@ -0,0 +1,36 @@
+#!/usr/bin/env python3
+"""Register a package kernel module as a static module in src/compstat.c.
+
+GAP's static-module table (CompInitFuncs[] in src/compstat.c) is the list
+of init functions that SHOW_STAT() reports; LoadKernelExtension finds a
+module there and loads it without dlopen (which the wasm build lacks).
+This adds an entry for a module whose init function has been renamed to
+Init__<name> (via -DInit__Dynamic=Init__<name> when compiling it).
+
+Idempotent: running twice for the same module is a no-op. Usage:
+    register_static_module.py <module-name> [path/to/compstat.c]
+"""
+import re
+import sys
+
+name = sys.argv[1]
+path = sys.argv[2] if len(sys.argv) > 2 else "src/compstat.c"
+init = "Init__" + name
+
+src = open(path).read()
+if init in src:
+    sys.exit(0)  # already registered
+
+# Add the extern declaration after the last existing one.
+externs = list(re.finditer(r"extern StructInitInfo \* Init__\w+\(void\);\n", src))
+if not externs:
+    sys.exit("register_static_module: no extern declarations found in " + path)
+at = externs[-1].end()
+src = src[:at] + "extern StructInitInfo * %s(void);\n" % init + src[at:]
+
+# Add the entry before the 0 terminator of CompInitFuncs[].
+src, n = re.subn(r"\n(\s*)0(,?\n\};)", r"\n\1%s,\n\g<1>0\2" % init, src, count=1)
+if n != 1:
+    sys.exit("register_static_module: could not find CompInitFuncs terminator in " + path)
+
+open(path, "w").write(src)

etc/emscripten/serve.py

@@ -18,6 +18,9 @@ class CrossOriginIsolatedHandler(http.server.SimpleHTTPRequestHandler):
     def end_headers(self):
         self.send_header("Cross-Origin-Opener-Policy", "same-origin")
         self.send_header("Cross-Origin-Embedder-Policy", "require-corp")
+        # Without this, browsers cache the UI files heuristically and
+        # edits to web-example/ don't show up on reload.
+        self.send_header("Cache-Control", "no-cache")
         super().end_headers()