query-based architecture with pyLodStorage endpoints and declarative forms - enabling federated queries

[WolfgangFahl]

Motivation

The current architecture is a dead end for the UI ambitions of the migration:

• eager graph loading: crm_web.py loads ALL records of all 8 topics into an in-memory MogwaiGraph at startup (~30 s on noah, memo/content fields included) - slow, memory-hungry and awkward: the graph holds raw German records so even the schema key_field settings do not match
• rudimentary UI: generic mogwai NodeTableView/NodeView - no forms, no search, no per-entity layouts, far from the generated Formset UI the legacy smartGENERATOR Java client had
• no federation: smartCRM data lives in more places than the MariaDB - the BITPlan SMW wikis link CRM records via the {{CRM}} template, and issue Look for Formset/Form/Group/Field description format #4 shows the Wikidata/SPARQL direction; crm/resources/queries.yaml already mixes an SQL query with an SMW ask query, but nothing executes them

Proposed architecture

1. Query-based data layer (replaces eager loading): named parameterized queries in crm/resources/queries.yaml executed on demand via pyLodStorage Query/Endpoint abstractions over multiple backend types:
   • SQL: the shared smartcrm MariaDB (production on r, local family-db copy)
   • SMW ask: the BITPlan wikis (wiki/media/cr) for {{CRM}}-linked pages
   • SPARQL: Wikidata / QLever endpoints for enrichment (see the PythonWebFrameworks example in Look for Formset/Form/Group/Field description format #4)
     All results are LoD (list of dicts) - one uniform shape for UI, REST and export.
2. Declarative UI layer (the Look for Formset/Form/Group/Field description format #4 counterpart): Formset/Form/Group/Field descriptions in YAML drive per-entity forms and AG-Grid list views (ngwidgets) with server-side pagination/filtering/sorting. First cut generated from the smartCRM UML model (smartCRM.mdl/.wsv) that already drove the generated legacy UI - the anti-corruption layer maps to English labels.
3. Federation by entity resolution: smartCRM IDs (PersonNummer etc.) as join keys across sources - a Person page then combines the CRM record, wiki backlinks (Link targetPage/SmartCRM Id properties) and SPARQL enrichment, scholia-style.
4. REST: named queries exposed as API endpoints (extends the existing /api/{plural} read API), giving the legacy find-by/search endpoints a more general replacement.

Acceptance criteria

• startup no longer loads all records - list views paginate via queries
• queries.yaml queries (SQL + ask) are executable through one QueryManager, endpoint type transparent
• at least one SPARQL federation example works end to end (e.g. Organization enriched via Wikidata)
• Formset/Form/Group/Field YAML format defined and rendered for at least Person and Organization (closes the research part of Look for Formset/Form/Group/Field description format #4)
• UI supports search/filter per entity without full data load

Relations

• builds on Look for Formset/Form/Group/Field description format #4 (Formset/Form/Group/Field description format)
• continues migrate from smartCRM SQL #3 (entity coverage, closed with v0.0.1)
• plan: https://media.bitplan.com/index.php/SmartCRM/Migration2026

[WolfgangFahl]

✍️🤖 Agent/Martin: finding harvested from a recorded metamodel review on 2026-07-08 (Reel Driven Development).

Situation: the review compares a four-corner metamodel - RDF/OWL, property graphs with Cypher/GQL, object orientation with JSON-LD, XML - against real data, and needs a validation case that is neither invented nor toy sized.

Action: this repository was opened during the walk, its README with the legacy and the target model as PlantUML diagrams was shown, and this ticket was named as the case to work on together.

Expected result: the niceSmartCRM legacy model maps onto the metamodel and back, so that the export paths - RDF, JSON-LD, Cypher - are produced from one representation. If the mapping needs expressiveness the metamodel does not offer, that is the finding worth having.

Acceptance criterion, and thereby the example: the legacy model is transformed into the metamodel form and exported to at least two of the target representations, with the round trip compared against the existing model.