Move ingestion and scanner work to durable jobs/workflows

[ishaanxgupta]

We performs ingestion/scanning through request-time waits, asyncio.create_task, local cancellation flags, and singleton process pipelines in memory.py and scanner.py.

Acceptance criteria:

• Introduce durable job records with status, retry count, timeout, and error state.
• Add queue/workflow worker for ingest, batch ingest, scraping, and scanner jobs.
• Make jobs idempotent by content hash/repo commit/user/org.
• Add dead-letter handling and status APIs.

Bounty 30$ API Credits

[ishaanxgupta]

Phase 1 - Task Queue: Introduce a task queue (Celery + Redis). The /v1/memory/ingest endpoint should: Validate and enqueue the job, returning a job_id immediately, Workers process the LangGraph pipeline asynchronously, Add a GET /v1/memory/ingest/{job_id}/status endpoint
Phase 2 - Step Persistence: Store intermediate state between LangGraph nodes (classification result, extraction results, judge decisions) in Redis. If a step fails, resume from the last checkpoint.
Phase 3 - Retry + Timeout: Wrap each agent call with asyncio.wait_for(timeout=...) and add exponential backoff retries. We already has some timeout logic from a previous conversation but it needs to be systematized.

[dttxorg]

I can take this on. I will start with a focused Phase 1 implementation so the maintainers can review it in a bounded PR:

• add durable job records for ingest / batch ingest / scrape / scanner work with status, retry_count, timeout, and error fields
• enqueue request-time work and return a job_id immediately
• add a worker path for processing jobs asynchronously, keeping the existing pipeline behavior intact
• add status APIs for checking job progress and terminal errors
• include tests for enqueue, status transitions, retry/error handling, and idempotency keys where the current code allows it

I will keep the first PR scoped to the task-queue/status foundation and avoid overreaching into the full checkpoint/resume design until that direction is reviewed.

[massy-o]

Submitted a focused Phase 1 PR for the durable queue/status foundation: #180

It adds durable Mongo-backed job records with idempotency, retry/timeout/dead-letter handling, /v1/jobs/{job_id} status, async memory ingest/batch-ingest jobs, and routes scanner start/resume through the same queue while preserving existing scanner status records.

Validation is included in the PR body.

[laolaoqi]

Bid: Move ingestion/scanner to durable jobs/workflows

What I'll deliver:

• Durable job records with status, retry count, timeout, and error state (SQLite/Postgres)
• Queue/worker for: ingest, batch ingest, scraping, scanner jobs
• Idempotency via content hash / repo commit / user / org dedup
• Dead-letter queue handling and status query APIs
• Clean separation from current request-time �syncio.create_task patterns

Timeline: 5-7 days from assignment

Why me: Built production job queue systems with Celery + Redis, plus durable workflow engines. Deep experience with async Python, task idempotency patterns, and dead-letter handling.

Payment address: 0x6c21794E27C34E93cd0049918d04Da58FC72595c