Repository navigation
Data API: no way to detect edited submissions #7259
Description
Description
The submission JSON served by GET /api/v2/assets/{uid}/data/ carries no last-modified timestamp. API consumers replicating data can't detect edits incrementally: we see no other choice but to re-fetch all submissions on every sync and hash the payloads.
Ask: add a _date_modified key to the served document, updated on every path that changes it (Enketo edits, validation-status changes, attachment changes), and accept it in the query parameter so consumers can sweep ?query={"_date_modified": {"$gt": "..."}}.
Steps to Reproduce
- Submit a record and fetch it via the data API. Note
_submission_timeand the full JSON. - Edit the record in Enketo (or change its validation status, or remove an attachment).
- Fetch again. The content changed, but no timestamp reflects it:
_submission_timeis unchanged and no_date_modifiedkey exists.
Expected behavior
A machine-readable timestamp that moves whenever the served document changes, usable for incremental sync.
Actual behavior
_submission_time is Instance.date_created and never moves. The Mongo document built by ParsedInstance.to_dict_for_mongo() (kobo/apps/openrosa/apps/viewer/models/parsed_instance.py) serializes no modified time at all.
The Postgres model already tracks Instance.date_modified via AbstractTimeStampedModel. It's just never written into the document the API serves.
Two wrinkles for an implementation to handle:
- The bulk validation-status path updates instances via a queryset
.update()(kobo/apps/openrosa/apps/logger/utils/instance.py), which bypassessave()and won't bumpdate_modifiedtoday. - Attachment deletion updates Mongo directly via
ParsedInstance.bulk_update_attachments().
Additional details
We replicate ~35 forms / ~230k submissions nightly into a SQL warehouse. Post-submission edits are routine (enumerator corrections, validation workflow), so we fetch everything every run and hash each payload to find the ~0.1% that changed. It works, but it's a full re-download per sync to catch a tenth of a percent of changes.
Existing signals don't cover this. _uuid changes on Enketo edits (previous value in meta/deprecatedID), but noticing that still requires fetching every record, and it doesn't move on validation-status or attachment-only changes. The project-history logs API (/api/v2/assets/{uid}/history/) records submission modifications but requires manage-project permission and has had tier-limited lookback since #7137.
Related: #2781 (webhook on submission update), and two community threads where staff confirmed in 2021 that _submission_time doesn't update on edit:
Activity
bhonermann commentedon Aug 25, 2026
Just coming here to comment that this is desperately needed for our project as well. I see the pull request has been issued. Would love to see this approved.
humanifest-bot commentedon Sep 9, 2026
Humanifest would like to support the existing PR #7260 with a focused regression test and sync-documentation contribution, if useful to its author and maintainers.
One boundary case to exercise: at head
7968471, the original_date_modifiedformatter maps12:00:00.100000and12:00:00.900000to the same12:00:00string. A consumer using that returned value as a strict$gtcursor can therefore miss the second update. The new tests check that the model datetime advances and Mongo matches its formatted value, but do not query across this boundary.We prepared a source-pinned reproduction and coverage note. The probe ran the original formatter expression with synthetic timestamps; it did not run the Kobo stack or reproduce production data loss. The documented legacy/backfill limitation is separate.
Would a frozen-time API test plus clarification of the intended cursor/overlap contract be welcome within #7260? We would coordinate additions with the existing author. This is an AI-assisted investigation by OpenAI Codex under Humanifest's direction; no independent human review is claimed.
RuneO commentedon Sep 9, 2026
Thanks for taking a careful look at it @humanifest-bot.
You're right about the resolution.
_date_modifieduses the sameMONGO_STRFTIMEformat as_submission_time(%Y-%m-%dT%H:%M:%S), so it has one-second granularity, and two writes inside the same second produce the same string.I kept the format identical to
_submission_timeon purpose, so consumers can treat both fields the same way, but that does mean the same caveat applies to both.The intended contract is that
_date_modifiedis a change indicator, not an exactly-once cursor.A consumer should query with
$gtefrom the last value it saw (or back the cursor off by a small overlap window) and de-duplicate on_id. That also covers the fact that the PostgreSQL and Mongo writes are not transactional, so a change can become visible in Mongo slightly after its timestamp.This is how our own sync uses it, and the small overlap costs almost nothing compared to re-fetching everything.
A frozen-time test around the same-second boundary and a short note on the
$gte+ overlap recommendation would both be welcome.Since the PR is on my fork, the easiest route is a small PR against
RuneO/kpi, branchadd-submission-date-modified, and I'll fold it in.Please keep it to those two things so the diff stays easy for the maintainers to review.
I'll leave it to the maintainers whether they want the doc note in this PR or handled separately.
RuneO commentedon Sep 9, 2026
NB @humanifest-bot
A note on documentation, since it came up above.
I have deliberately not added documentation for
_date_modifiedin this PR. The data API docs inkpi/docs/api/v2/data/list.mdare fairly thin today (thequeryparameter is covered by two curl examples and a link to the old Formhub wiki), so a single new paragraph would not really give consumers what they need for incremental sync.Doing it properly means describing the cursor contract, the one-second resolution, the
$gte+ overlap recommendation, and the fallback behaviour for documents written before this change.I am happy to put that time in, but I would like to first hear from the Kobo-team whether this feature is something they want to take in, and in what shape. Once there is a signal on that, I will follow up with a proper documentation pass, either in this PR or in a separate one, whichever the maintainers prefer.
humanifest-bot commentedon Oct 7, 2026
@RuneO Thanks for clarifying the documentation scope. We will leave the full documentation pass with you and the Kobo team until they decide the feature shape; we will not add a standalone docs paragraph to #7260.
For the test you invited, the existing
test_list_submissions_filter_by_date_modifiedcovers$gt/$ltacross different days. The focused addition would make two changes to one submission within a second and assert that strict$gtmisses the later state while$gtereturns the_idfor consumer deduplication. That would capture the cursor boundary in a regression test without changing timestamp precision. We have not run the Kobo stack or submitted code. Any test PR would disclose AI assistance.