The MongoDB oplog (local.oplog.rs) is the durable, ordered log of every write a replica-set primary accepts; it is the substrate change streams are built on, and tailing it directly with a tailable cursor predates change streams by years. Most CDC use cases should reach for change streams — they handle resume tokens, sharded ordering, and cluster failover correctly. But there are scenarios — pre-3.6 driver compatibility, custom filtering on internal oplog fields, debugging — where direct oplog tailing is the right tool. This article covers the entry format, the cluster-time bookkeeping, and the trade-offs against change streams.
How to verify
Inspect the oplog directly to see the entries change streams expose at a higher level:
mongosh --host primary.example.internal:27017
use local
db.oplog.rs.find().sort({ $natural: -1 }).limit(3).pretty()
db.oplog.rs.stats().maxSize
rs.printReplicationInfo()
Each entry has an op (i insert, u update, d delete, c command, n no-op), a namespace ns, an operation timestamp ts, and the payload — o for inserts and commands, o2 + o for updates (criteria + modifier).
What’s happening
The oplog is a capped collection in the local database. Its size is fixed at startup (default 5% of free disk, capped at 50 GiB) and entries are evicted FIFO once the cap is reached. Every write the primary applies produces an oplog entry; secondaries tail the oplog from the primary (or another secondary in chained replication) and apply each entry to their local state.
A tailable cursor in MongoDB stays open across the end of a capped collection — when new documents arrive, the cursor delivers them. Combining tailable: true with awaitData: true and a max await time gives a long-polling consumer: the server blocks for up to maxAwaitTimeMS waiting for new entries, then returns.
The oplog entries carry a ts field of type Timestamp (BSON type, two 32-bit ints — seconds since epoch and an increment to disambiguate entries within the same second). To resume a tail, you persist the last applied ts and ask for entries with ts > lastTs. Change streams wrap this in resume tokens that also handle failover; direct tailing requires you to handle primary-change yourself by reconnecting to the new primary and continuing from the same ts.
The procedure
-
Identify the resume point. On first start, find the most recent oplog entry to begin tailing forward from.
const lastTs = db.oplog.rs.find().sort({ $natural: -1 }).limit(1).next().ts -
Open the tailable cursor with the right options.
const cursor = db.oplog.rs.find( { ts: { $gt: lastTs }, ns: /^shop\./ }, { noCursorTimeout: true } ).addOption(DBQuery.Option.tailable).addOption(DBQuery.Option.awaitData) -
Loop, processing entries, persisting
tsafter each successful step.while (cursor.hasNext()) { const entry = cursor.next() try { process(entry) db.checkpoints.updateOne({ _id: "oplog-consumer" }, { $set: { ts: entry.ts } }, { upsert: true }) } catch (e) { backoff(e); break; } } -
Handle the four operation types differently. Each has a different shape.
i(insert):ois the full new document.u(update):o2is the criteria ({ _id: ... }),ois the modifier ($set,$inc, etc.).d(delete):ois the criteria.c(command):odescribes the command (drop,create,dropDatabase).
-
On primary failover, reconnect and resume from saved
ts. Use a SRV connection string withreplicaSet=…so the driver finds the new primary.// After ConnectionFailure / NodeNotInPrimaryError, reopen against the new primary -
Filter at the cursor level when possible. The oplog is shared by every collection — without a
nsfilter your consumer sees everything.
Operational notes
- Use change streams unless you have a documented reason not to. They handle resume, failover, and sharded ordering correctly. Direct tailing exists because it predates change streams; new code should default to the higher-level API.
- Sharded clusters do not have a single oplog. Each shard’s primary has its own. To tail a sharded cluster directly, you tail every shard and merge by cluster time — much of what change streams do for you.
- Oplog format changes between major versions. A tailing consumer that parses entries directly may break on upgrade. Test against the target version before rolling out.
- No-op entries (
op: "n") exist for heartbeat-like purposes. They keep cluster time advancing on quiet primaries. A naive consumer that does not skipop: "n"does no harm but does no useful work either. - Index hints help on busy oplogs.
db.oplog.rs.find().hint({ $natural: 1 })is the natural-order scan, the only access pattern that does not require building an index on a capped collection. - The
localdatabase is not replicated. Tailing must be against an individual node, not viamongos.
Stack Harbor builds CDC pipelines with change streams as the default and falls back to direct oplog tailing only for the narrow cases where the higher-level API does not fit. We instrument oplog-window-vs-consumer-lag, plan for window expansion under upstream outages, and document the resume semantics so a new operator can take over an incident at 3 AM. See /en/knowledge-base/mongodb-change-streams-realtime/ for the higher-level alternative and /en/services/managed-operations/ for the engagement model.