Skip to main content

SCOPE Research Data Collector

Documentation source: collector/README.md at revision d127243ba072cb8572df536d40a31be3935e0106.

The collector is an independent Python library and command-line tool for explicitly scoped, source-native scholarly metadata from OpenAlex, DBLP, ORCID, and arXiv. It does not import, install, build, or run the SCOPE website. It collects no paper full text, performs no cross-source identity matching, and produces no researcher rankings or analytical scores.

Prerequisites​

  • Python 3.12, 3.13, or 3.14
  • uv
  • Git when reviewing a prepared data/ release
  • Network access only for live collection
  • OPENALEX_API_KEY for OpenAlex
  • ORCID_CLIENT_ID and ORCID_CLIENT_SECRET for ORCID Public API v3

Credentials belong only in the process environment. Never put them in request JSON, examples, logs, .env files under this repository, candidates, or Git history.

Install the locked environment​

From collector/:

uv sync --locked --all-groups
uv run --locked scope-collect --version
uv run --locked scope-collect --help

The lockfile is cross-platform and committed. No Node or website dependency is installed.

Command workflow​

The public workflow has separate download, offline processing, and reviewed promotion boundaries. Validate a saved request offline, then download exact source responses:

uv run --locked scope-collect validate-request examples/requests/openalex-work.json
uv run --locked scope-collect download examples/requests/openalex-work.json

The download summary supplies <download-id>. Resume only a reported incomplete download and verify a completed bundle without provider access:

uv run --locked scope-collect resume-download <download-id>
uv run --locked scope-collect validate-download <download-id>

Initialize processing and invoke every stage independently in the declared recipe order. These commands are offline and write only beneath .download/processing/:

uv run --locked scope-collect init-processing <download-id> \
  --recipe pipelines/openalex-work-api/recipe-1.0.json
uv run --locked scope-collect process-stage <processing-id> decode-source-pages
uv run --locked scope-collect process-stage <processing-id> deduplicate-records
uv run --locked scope-collect process-stage <processing-id> canonicalize-and-shard
uv run --locked scope-collect process-stage <processing-id> assess-and-package
uv run --locked scope-collect validate-candidate <processing-id>

process <download-id> --recipe ... is the offline convenience form. It still creates only a temporary candidate. After a maintainer creates and reviews a decision bound to the candidate's current quality hash, promotion is the only command that may create permanent data:

uv run --locked scope-collect promote <processing-id> \
  --decision /path/to/release-decision.json

Promotion copies allowlisted files only. It never includes raw pages or stage intermediates, runs Git, or overwrites an existing release ID.

The repository exposes three contributor-facing catalogs: downloaders/ for acquisition-only provider entries, processors/ for independently runnable offline stages, and pipelines/ for dataset-specific end-to-end scripts and their colocated versioned recipes. Inspect them offline:

uv run --locked scope-collect list-components
uv run --locked scope-collect describe-component pipeline dblp-publication-search

Each definition and registry entry is digest-pinned. The thin scripts delegate to the tested CLI and contain no provider or transformation implementation.

Download an arXiv subject​

Use the category code shown by arXiv, such as cs.AI for Artificial Intelligence. This command uses arXiv's selective OAI-PMH category set, follows resumption tokens sequentially, and downloads exact public metadata responses only:

uv run --locked scope-collect download-arxiv-category cs.AI

Omitting --limit means exhaust the complete category set; it does not mean “take the first 30,000 search results.” For a quick sample before a potentially long full harvest:

uv run --locked scope-collect download-arxiv-category cs.AI --limit 100

The returned download ID identifies exact request evidence, durable OAI pages, checkpoint, and input-bundle hash beneath .download/downloads/. To construct records, use pipelines/arxiv-preprint-oai/recipe-1.0.json in the separate processing sequence. Its outputs preserve the arXiv identifier, title, abstract, categories, dates, DOI/journal reference when supplied, license, and ordered author assertions. The collector does not infer unique people or merge authors with ORCID.

The equivalent end-to-end entry is pipelines/arxiv-preprint-oai/run.py; without an explicit reviewed decision it stops with a temporary candidate awaiting review.

arXiv permits only one request at a time and requires at least three seconds between requests, so a large category can take substantial time. If a transient failure leaves a download incomplete, resume its ID with scope-collect resume-download <download-id>. The optional --from-date YYYY-MM-DD selects OAI records modified since that datestamp; it is not a paper submission-date filter.

Only descriptive metadata is downloaded. PDFs and source archives remain excluded. A large candidate may correctly be blocked by repository policy; that does not invalidate its local download or authorize committing it.

Output locations​

  • .download/downloads/<download-id>/: ignored request, exact raw pages, checkpoint, state, and immutable input-bundle seal
  • .download/processing/<processing-id>/: ignored stage results, deterministic record shards, lineage, quality evidence, and temporary candidate
  • .download/pipelines/<pipeline-run-id>/: ignored end-to-end component state and recovery evidence
  • data/<source>/<release-id>/: only artifacts approved by a separate reviewed promotion

.download/ is temporary and may be deleted deliberately after dependent work is complete. It is ignored by Git, and CI also rejects any tracked or staged path beneath it. .scope-collector/ remains ignored only for migration of older runs; new commands never write there.

The CLI writes exactly one compact JSON summary to stdout. Progress and diagnostics go to stderr. Source text and response bodies are never printed.

Completion and Failure evidence​

sealed means the declared download boundary was reached or exhausted and every raw artifact verified; it does not authorize publication. A successful zero-result bundle is explicit. incomplete retains a safe checkpoint and exits 3. Invalid input/configuration exits 2, non-retriable acquisition exits 4, material processing/candidate findings exit 5, unauthorized or withheld promotion exits 6, and a handled interrupt exits 130. Inspect download-state.json, input-bundle.json, processing-run.json, stage result.json, lineage-manifest.json, and quality-report.json instead of inferring success from file presence.

Reproduction, refresh, cleanup, and legacy state​

The input-bundle digest excludes run timestamps and IDs. Processing identical request/raw bytes with the same recipe and parameters therefore produces the same payload digest. It is an exact reproduction only when the referenced release has the same verified bundle, recipe, parameters, serialization profile, and payload; a new mutable-source observation is a refresh.

Clean one exact temporary resource only after review:

uv run --locked scope-collect cleanup processing <processing-id> \
  --confirm <processing-id> --discard-candidate --reason 'review complete'
uv run --locked scope-collect cleanup download <download-id> --confirm <download-id>

Cleanup refuses active locks, dependency-bearing downloads, mismatched confirmation, unsafe paths, and unreviewed candidates. Inventory and copy legacy state without modifying the original:

uv run --locked scope-collect legacy-status ../.scope-collector
uv run --locked scope-collect migrate-legacy /absolute/path/to/legacy-run

After byte/hash verification and successful use of the migrated state, deleting the old ignored directory remains an explicit manual decision.

During the compatibility window, collect, collect-arxiv-category, resume, finalize, and release are deprecated aliases for the corresponding separated commands. They use the same fixed .download/ and reviewed-promotion semantics. validate-snapshot remains a read-only tool for existing feature-002 candidates; new candidates use validate-candidate. No alias restores automatic category finalization or accepts an arbitrary runtime/permanent root.

Source examples​

All requests are bounded and preserve provider fields rather than implying equivalence.

OpenAlex​

openalex-author.json selects one author and openalex-work.json selects one work. Both require OPENALEX_API_KEY; outputs preserve OpenAlex identifiers, additive fields, and documented algorithmic-disambiguation and changing-affiliation limitations. The bounded metadata license is recorded as CC0.

DBLP​

dblp-person.json uses a stable PID, dblp-person-search.json performs bounded author discovery, and dblp-publication.json uses a five-record publication search. No credential is required. Requests are sequential and at least two seconds apart. Run the matching scripts under pipelines/dblp-person-pid/, pipelines/dblp-person-search/, or pipelines/dblp-publication-search/. DBLP is computer-science-centered, omits abstracts, and its metadata is recorded as CC0 1.0.

ORCID​

orcid-record.json requests public fields and orcid-work.json requests public work assertions for one ORCID iD. They require registered Public API client credentials; the two-legged /read-public token remains in memory. Limited/private items and linked content are excluded. API-derived data is terms-governed and defaults to metadata-only.

arXiv​

arxiv-preprint.json requests one preprint. arxiv-category-cs-ai.json declares the same complete OAI-PMH cs.AI boundary used by the category-specific workflow. No credential is currently required. arXiv access is single-connection and at least three seconds apart. Ordered author strings remain publication assertions and never become researcher entities; PDFs and source archives are excluded.

See examples/README.md for scope, expected results, and caveats for every request.

Offline verification​

Normal CI never contacts providers:

uv run --locked pytest -m "not live" -q
uv run --locked pytest tests/integration/test_adapter_independence.py -q
uv run --locked pytest tests/recovery -q

Fixtures cover normal and zero results, paging, malformed responses, duplicates, throttling, recovery, determinism, and release refusal. Optional live smoke checks require an explicit flag:

SCOPE_LIVE_TESTS=1 uv run --locked pytest tests/integration/live -q

Library boundary​

The stable Python surface mirrors the operational workflow:

from pathlib import Path

from scope_collector import run_pipeline

pipeline_directory, pipeline = run_pipeline(
    "dblp-publication-search",
    request_file=Path("examples/requests/dblp-publication.json"),
)
assert pipeline["status"] == "awaiting-review"

The public exports cover catalog discovery, pipeline run/resume, request validation, download/resume/verification, offline processing, candidate validation, reviewed promotion, reproduction classification, compatibility facades, the adapter protocol, and result dictionaries. Provider internals are not a stable API.

Extending or changing an adapter​

Each module in src/scope_collector/sources/ owns only its interface mapping, credentials, media type, field-preserving offline decoder, acquisition inspection, continuation, pacing, governance references, and limitations. Shared HTTP safety, durability, identity, processing, quality, and promotion behavior stays under core/ and processing/.

To change one source:

  1. Update only that adapter's capability and source-contract evidence.
  2. Add sanitized fixtures for normal, empty, multi-page, additive/missing, malformed, oversized, duplicate/conflict, transient, rate, interrupted, capped, and expired-continuation cases.
  3. Run its unit tests and test_adapter_independence.py with the other transports unavailable.
  4. Run download recovery, offline stage, deterministic payload, CLI, and promotion-gate tests.
  5. Treat changed selector meaning, record identity, serialization, counts, exits, or release rules as a major contract change; additive payload fields are compatible.

Never import another adapter from a provider module. Register a new source lazily in sources/base.py, keep its credentials out of requests and evidence, and fail closed when current terms are unknown or more restrictive.