Skip to content

Add the crawl job API and Tor diagnostics - #82

Merged
KingAkeem merged 2 commits into
mainfrom
agent/job-api-torbot-integration
Jul 17, 2026
Merged

Add the crawl job API and Tor diagnostics#82
KingAkeem merged 2 commits into
mainfrom
agent/job-api-torbot-integration

Conversation

@KingAkeem

@KingAkeem KingAkeem commented Jul 17, 2026

Copy link
Copy Markdown
Member

What changed

  • adds a versioned JSON crawl report and cancellable REST job API
  • restores health/readiness endpoints and adds Tor configuration diagnostics
  • returns parent relationships, metadata, contacts, skip reasons, and partial errors
  • adds per-host limits, content guardrails, randomized user agents, and structured logging
  • hardens the container and CI workflow
  • refreshes the README, API reference, and changelog

Why

TorBot App previously had no stable backend integration surface. The concurrent crawler also removed the older REST service and only logged results, which made it difficult for desktop and automation clients to consume crawl state safely.

Impact

Clients can now start, poll, and cancel jobs through /api/v1/jobs. They can also inspect SOCKS reachability and local torrc details through /api/v1/tor/status. CLI users retain list/tree output and gain a stable -f json format.

Validation

  • go vet ./...
  • go test -race -cover ./...
  • go build -trimpath ./cmd/main
  • live direct crawl of https://example.com
  • Electron adapter end-to-end job and Tor-status smoke test

Related issues

Closes #44
Closes #72
Closes #74
Closes #76
Addresses part of #77

Expose stable JSON crawl reports and cancellable REST jobs for desktop clients. Add Tor configuration discovery, crawler safety controls, structured logs, tests, and refreshed operational documentation.
Split link extraction, response guards, HTML analysis, and child scheduling into focused helpers without changing the crawl report contract.
@KingAkeem
KingAkeem marked this pull request as ready for review July 17, 2026 22:29
@KingAkeem
KingAkeem merged commit 0965949 into main Jul 17, 2026
5 checks passed
@KingAkeem
KingAkeem deleted the agent/job-api-torbot-integration branch July 17, 2026 22:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant