Sync

This page is the authoritative reference for how the snapshot changes over time: when releases happen, what a release means, and how to keep a local copy — including deletions and merges — in sync with OpenAlex.

Release cadence

The authoritative history of what shipped when is RELEASE_NOTES.txt, at the root of the public bucket. Each entry summarizes the data changes in that release — new sources, quality fixes, schema additions. Check it (and the manifest date) rather than guessing from file timestamps.

Note: There is no announcement feed for releases yet — watch RELEASE_NOTES.txt or the combined manifest’s date field. If a scheduled release seems overdue, that’s usually release engineering rather than a policy change.

What a release is: how partitions work

Records are partitioned by updated_date, and each partition holds only the records that last changed on that date. A release doesn’t add a new dated copy of the whole database — it updates the one copy in place, moving changed records into newer partitions.

Imagine launching OpenAlex with 1,000 Authors, all created on 2024-01-01:

/data/jsonl/authors/
├── manifest.json
└── updated_date=2024-01-01 [1000 Authors]

If 50 of those Authors are updated on 2024-01-15, they move out of the old partition and into the new one; 50 brand-new Authors would land in the same new partition:

/data/jsonl/authors/
├── manifest.json
├── updated_date=2024-01-01 [950 Authors]
└── updated_date=2024-01-15 [100 Authors]

Consequences:

• The bucket is always the complete current database — the union of all partitions, exactly once per record. No partition is a “base file”; no folder is cumulative.
• To update a copy you made on date X, download only partitions with updated_date > X and upsert them by id. You never need to re-download a partition you already have.
• Old partitions shrink as their records move forward. If you sync files rather than upserting records, use --delete so vacated files disappear (see below).

Keeping in sync with OpenAlex

There are four ways to keep a copy of OpenAlex current. They differ in freshness, in how much infrastructure they demand, and in how much of the database they suit — pick by what you’re mirroring:

The two snapshot methods move the whole database in bulk; the API-filter method needs nothing but HTTP requests, which makes it the right fit when you only care about a slice of OpenAlex.

The public snapshot (free)

Re-sync from each quarterly release using the manifest (per entity or combined):

1. Download the manifest — aws s3 cp s3://openalex/data/jsonl/works/manifest.json . --no-sign-request
2. Find new partitions — any updated_date in the manifest’s file list that you haven’t ingested.
3. Download those partitions.
4. Verify consistency — re-download the manifest; if unchanged since step 1, no records moved mid-download.
5. Upsert by id into your store.

If you mirror files with aws s3 sync, always pass --delete — otherwise files vacated by records moving to newer partitions linger and you get duplicates. Full commands in the download recipe.

The daily snapshot (paid plans)

Subscribers get the complete database, rebuilt and published every day — each day’s copy in its own dated folder in the staging bucket, partitioned by updated_date just like the public snapshot (access details).

Because every daily copy is both complete and partitioned by change date, it covers every sync rhythm:

• Incremental sync, on your schedule. Grab today’s copy and download only the partitions newer than your last sync — whether that was yesterday, the 15th of last month, or whatever day you fancy. There’s no fixed release calendar to wait on.
• Full rebuild, as often as you like. Rebuild your entire database from scratch every day if you want — each dated folder is the whole thing.
• Deletions handled. Each day’s copy is the complete current corpus, so records that were deleted or merged away are simply absent — reconcile against it and they fall out of your mirror. For works, each copy also names its deletions explicitly in deleted_ids.csv (see below).

Premium API filters (paid plans)

Paid plans unlock two filters on every entity type that make the API itself a sync mechanism:

• from_created_date — records created since a date
• from_updated_date — records updated since a date

https://api.openalex.org/works?filter=from_updated_date:2026-07-30

Poll with these and upsert the results by id — no snapshot downloads, no bulk infrastructure, just HTTP requests. It’s also the freshest view of the data, since the API updates continuously rather than on a release schedule.

The real superpower is that they combine with any other filter, which makes this the method for keeping a subset of OpenAlex synced. Most mirrors don’t need the whole database: an institution tracking its own research output, say, only needs its own works, and standing up snapshot infrastructure for that is wild overkill. One polling loop covers it:

https://api.openalex.org/works?filter=institutions.id:I27837315,from_updated_date:2026-07-30

Trade-offs: paging through the API is slower per record than bulk files, so this method fits modest change volumes — for full-database mirrors, use the daily snapshot. And the filters only surface records that exist: deleted or merged-away records just stop appearing (and their IDs 404), so reconcile your ID set against a snapshot periodically if deletions matter to you. Filter mechanics: Sync filters in the API reference.

The Unpaywall Data Feed (paid plans)

Unpaywall subscribers can receive daily change updates in the legacy Unpaywall record format — works with Crossref DOIs only. It delivers changes to the same underlying OpenAlex data, and exists for the ecosystem of integrations that already speak the Unpaywall schema; for anything new, use one of the OpenAlex-native methods above. Details: the Unpaywall Data Feed.

Free data, paid services

All OpenAlex metadata is CC0 and free — anyone can download the whole database, forever, at no cost. What’s paid is freshness as a service: rebuilding and publishing the full database every day costs us real money every day, so we pass that cost on to the users who need daily updates. That’s not an accident of pricing — it’s the sustainability model encouraged by POSI, the Principles of Open Scholarly Infrastructure: keep the data open, charge for the services that fund it. Plans and pricing: Pricing.

Deletions and merged entities

Records don’t just get created and updated — they also disappear: works get merged when they’re found to be duplicates, author profiles get merged or deleted, and bogus records get removed.

How that shows up today:

• In the API: a deleted or merged-away ID returns 404. There is no redirect to the surviving record. (Special case: works of removed author profiles point to the null author A9999999999, and A5317838346 marks deleted authors.)
• In the snapshot: the record is gone from the current release — it doesn’t appear in any partition, and the vacated file space disappears from the manifest. This is true of every daily copy too, so reconciling against a snapshot picks up deletions.
• In deleted_ids.csv (works): every snapshot now names its deleted works explicitly — see below.

The works deletion log: deleted_ids.csv

Works only, for now. Works are the first entity type with an explicit deletion log; every other entity type still relies on the reconcile methods below to detect deletions.

Each snapshot’s works directory carries the log at {format}/works/deleted_ids.csv (same file in both format trees, next to manifest.json). It’s a plain CSV with a header and two columns:

work_id,deleted_date
https://openalex.org/W4245566371,2026-08-14

• work_id — the deleted work’s ID, in the same URL form as the works data files.
• deleted_date — the date the work disappeared from the corpus. (For works deleted before the log existed and backfilled into it, this is the date the deletion was detected, not the original deletion date.)

The file is cumulative — each release carries the full history, so you don’t need to collect it daily. Apply it as: remove these IDs from your copy of works. That makes mirror deletions a direct lookup instead of a full ID-set diff (the reconcile methods below still work and remain the belt-and-braces option).

Deletion is final. A deleted ID stays deleted, and work IDs are never reused — a deleted ID will never point to a different work.

Available in every daily snapshot since 2026-08-15; the free public bucket picks it up with the next quarterly release.

Tracing merges: location IDs move to the surviving work

When two works are found to be duplicates and merged, the surviving work absorbs the other’s locations. Each location carries an internal id — a stable handle for the underlying harvested record, like doi:10.7717/peerj.4375, pmid:29456894, or pmh:oai:europepmc.org:4724910 — and that handle survives the merge. So between releases you can see a location id that used to sit on work A now sitting on work B: that’s the merge trail.

Combining the two signals gives you the full picture of records leaving the corpus:

Picking up deletions in a mirror

Reconcile against a full release periodically (daily-snapshot subscribers can do this any day):

• Works: delete every ID in deleted_ids.csv from your copy — cheap and exact.
• File mirror: aws s3 sync ... --delete, then rebuild — the synced tree is the current corpus.
• Database mirror (all entities): diff your ID set against the release’s ID set (stream IDs from the part files, or compare per-partition record_counts in the manifest) and delete local records that no longer exist upstream.

Note: The pre-Walden snapshot published a merged_ids/ directory mapping merged IDs to their survivors. That mechanism ended with the 2025 Walden cutover (historical files preserved under legacy-data/, not updated). deleted_ids.csv is its successor for works deletions, with location IDs providing the merge trail; an explicit survivor mapping and coverage of other entity types are known asks — if they matter to your pipeline, tell us about your use case.

Point-in-time and reproducibility

The public bucket holds only the current release — once a release lands, the previous state is gone, so cite the combined manifest’s date (or the RELEASE_NOTES.txt entry) in anything you need to reproduce, and archive your own copy if you need to re-run against it. Paid plans get closer to point-in-time: the staging bucket keeps each day’s full snapshot in its own dated folder.

Related pages

• Snapshot — formats, layout, manifests, and API parity
• Access & authentication — buckets and credentials
• Download the snapshot — the hands-on recipe