IGSN samples: scaling the full DataCite PhysicalObject corpus (13.4M points) on the map

[nuest]

Context

Spun out of #187 (add IGSN physical samples to the map). Before implementing, we probed
the live DataCite API (2026-07-01) to size the IGSN/PhysicalObject corpus and understand
its metadata quality. The numbers show the full corpus cannot be ingested as individual
map markers and needs a dedicated scaling strategy. This issue records the research and
the candidate approaches so the initial implementation can stay scoped to a bounded subset.

Corpus size (DataCite types.resourceTypeGeneral:PhysicalObject)

Metric	Count
Total PhysicalObject records	16.1 M
…with geoLocations	15.0 M (93%)
…with geoLocationPoint	13.4 M
…with geoLocationBox / geoLocationPolygon	3 / 0
geoLocations, place-name only (no point)	1.6 M

Geometry is essentially all points (geoLocations[].geoLocationPoint.{pointLatitude, pointLongitude}); boxes/polygons are negligible. Temporal coverage via dates[]
(dateType:"Collected").

By publisher

Publisher	PhysicalObject records	Coords in DataCite?	Related identifiers
Geoscience Australia	7.35 M	(mostly points)	not assessed
SESAR	5.83 M	✅ geoLocationPoint (~93%)	✅ real values (13 k with DOI links)
GFZ Data Services	38 k	❌ none (geoLocations: [] for all)	❌ target value null

GFZ caveat: GFZ sample coordinates are not in DataCite — they are only
server-rendered in the GFZ IGSN HTML landing page
(dataservices.gfz-potsdam.de/igsn/esg/index.php?igsn=<IGSN>). A 40-page sample showed
~65% with real coords, ~35% "N/A". Harvesting GFZ therefore requires HTML scraping.

Sample ↔ work relationship metadata

• DataCite relatedIdentifiers is the source of truth: 57,610 PhysicalObjects have
  IsCitedBy/IsReferencedBy → DOI (SESAR ~13 k). Targets include papers and datasets.
• Scholix / Scholexplorer: returned 0 links even for samples DataCite marks as cited —
  poorly populated for IGSN.
• OpenAlex: indexes some samples as type other but with 0 referenced/related works and
  no geo.
• OpenAIRE Graph API: knows samples but relationship extraction is unclear and the
  endpoint was flaky (503s); redundant with DataCite.

Candidate approaches for the full corpus

1. Server-side aggregation / clustering (primary). Store samples but serve the map
   aggregated density (server-side clustering, heatmap, or vector tiles) instead of
   per-sample markers; drill down to individual points only at high zoom. Most infra work
   (new storage + tile/aggregation endpoint) but the only approach that scales to millions.
2. Live bbox-proxy overlay (no DB storage). Query DataCite live for the current viewport
   above a zoom threshold; show points on demand; store nothing. Keeps DB/map light, always
   fresh, but couples the map to DataCite availability.
3. Bounded harvested subset. Ingest only a filtered slice (one provider, a study region,
   a date range, or only samples that link to a publication) so counts stay in the thousands
   and the existing map/API/feeds work unchanged. (This is what Add samples from IGSN to the map #187's initial implementation
   uses.)
4. Only samples linked to existing works. Harvest just the samples referenced by
   publications already in OPTIMAP (via DataCite relatedIdentifiers). Tiny counts; ties
   into work↔work relationship linking.

Suggested next steps

• Ship Add samples from IGSN to the map #187 with a bounded subset (approach 3/4).
• Prototype approach 1 (server-side clustering/vector tiles) as the path to the full corpus.
• Re-assess Geoscience Australia and SESAR metadata quality (coords + related identifiers)
  before any large ingest.