Skip to content

Fix postgis_db paths after move and add Socrata sync probe - #762

Merged
gregpawin merged 1 commit into
hackforla:mainfrom
gregpawin:feat/postgis-paths-and-socrata-sync
Sep 22, 2026
Merged

gregpawin merged 1 commit into
hackforla:mainfrom
gregpawin:feat/postgis-paths-and-socrata-sync

Conversation

@gregpawin

Copy link
Copy Markdown
Member

Summary

  • Update postgis_db docs and script comments for the projects/data-science/ layout after Move data-science/ to projects/data-science/ #759.
  • Fix postgis_db/dumps/* gitignore so probe/dump artifacts are ignored while keeping .gitkeep.
  • Add a dry-run/apply Socrata → PostGIS sync probe (scripts/test_socrata_sync.py) ported from the beta_pipeline SQLite sync idea.
  • Repair broken Jinja conditionals in the citation explorer template (index.html).

Test plan

  • Confirm README / VPS paths match projects/data-science/postgis_db
  • Confirm git check-ignore matches files under postgis_db/dumps/ (except .gitkeep)
  • Dry-run: scripts/test_socrata_sync.cmd (or .py) against local PostGIS with SOCRATA_APP_TOKEN
  • Load explorer UI and verify region select / single vs compare mode still work
  • Optional: run pytest tests/test_service_integration.py -m integration with PostGIS up

Made with Cursor

…ucture

- Updated paths in `README.md` to reflect the move from `data-science/` to `projects/data-science/`.
- Adjusted commands in `VPS.md` for consistency with the new directory structure.
- Enhanced integration test instructions in `test_service_integration.py` to align with the updated paths.
- Modified `app.py` in the web sheet to reference the new project structure.
- Updated `smoke_test.cmd` permissions to ensure executable status.

This commit ensures all documentation and scripts are consistent with the recent refactor, improving clarity for users and developers.
@github-actions

Copy link
Copy Markdown

@gregpawin, this Pull Request is not linked to a valid issue. Please provide a valid linked issue in "Related Issues" above, using the format of "Resolves #" + issue number.

@gregpawin
gregpawin merged commit 5a9ecc6 into hackforla:main Sep 22, 2026
2 checks passed
@gregpawin

Copy link
Copy Markdown
Member Author

Follow-up notes from macOS verification (Nov 2026)

Re-ran the local stack and Socrata sync probe from this PR on macOS after syncing a work-PC checkout.

Local stack

  • scripts/*.sh were mode 100644 after checkout, so ./scripts/start.sh failed with Permission denied. Invoking via bash scripts/start.sh still worked. Opened a small follow-up: Connect main website to PostGIS citations API #769
  • A .env copied from Windows still had CITATIONS_DATA_DIR pointed at a C:/... path and CRLF line endings. Updating that to a macOS path (and stripping \r when sourcing) unblocked preflight.
  • Verification sample with CITATIONS_LOAD_LIMIT=200000 on alternate ports (COMPOSE_PROJECT_NAME=lucky-parking-local-check) came up healthy.

Socrata sync probe (scripts/test_socrata_sync.py)

  • Needs a valid Socrata App Token in SOCRATA_APP_TOKEN (not the secret token / API key). An invalid token returns HTTP 403 Invalid app_token specified.
  • Dry-run against the sample DB succeeded once the token was fixed.
  • Gap-fill apply recommended in the README worked end-to-end:
--since-db-max --order "issue_date DESC" --no-stop-on-match --apply
  • Started from DB max 2025-09-24, caught up with api_exhausted
  • Inserted ~2.05M new contract rows
  • DB max advanced to 2026-11-01
  • Runtime ~66 minutes for that gap

Caveat

  • Gap-fill from a 200k sample only fills forward from that sample’s max date; it does not reconstruct a complete historical load. Full history still needs a full CSV import without CITATIONS_LOAD_LIMIT.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants