Firestore Fieldvalue.increment and duplicate writes

[astanb]

Over the last couple of days I have noticed some concerning behaviour with Fieldvalue.increment.

Every so often set/update operations get duplicated, and if that operation has an increment in it then it messes up my data. I am 100% certain that I am not calling the operation twice, I have tested extensively to confirm this.

From inspecting the firebase logs I can see when it happens that duplicate writes are happening within single digit milliseconds of each other. It happens randomly so it is very hard to reproduce.

This is happening in offline mode as well as online, both on the emulators and using remote firebase services. I am the only one running the client so it is not a case of multiple people incrementing.

I am not writing to the documents more than once a second.

Am I misunderstanding something about how firestore works internally? What happens under the hood that would cause a write to be duplicated?

I'm at my wits end here as this is affecting users in production and it's looking like I am going to have to refactor to remove any usage of increment and do it manually, which is going to end up costing more.

I will create a reproducible demo if needed, but first I wanted to ask whether I am missing something crucial in what I expect firestore to be able to do.

[mikehardy]

This has the feel of an issue report but you opened a discussion and avoided populating all the useful information I'd need to offer help unfortunately

Make sure you are on up to date versions, first. If you're not, do that. I'll assume you've done that now.

I do not believe you are missing anything, but before going to the trouble of a reproduction here I'd check upstream
You have not included what platform you are using, so I advise scanning issues / pulls on both repos:
https://github.com/firebase/firebase-ios-sdk
https://github.com/firebase/firebase-android-sdk/

and trying to reproduce there with their quickstart, as I do not think it's possible for the code in this module to be double-writing
https://github.com/firebase/quickstart-android/tree/master/firestore
https://github.com/firebase/quickstart-ios/tree/master/firestore

[wfern]

Getting later here, but I know this bug very well. It's not from this library, it's from Firestore itself. You can reproduce this bug in every SDK available. Here is a simulator using the js-sdk: https://codesandbox.io/s/dreamy-shadow-isgwl . There is a big discussion on slack too, between me and some other guys, but it is too old to access: https://firebase-community.slack.com/archives/C7CQWAH7F/p1608061793176100.

In January 2021 I open up a support ticket with the Firebase support team and they successfully reproduce the "bug" but they didn't have a foresight to solve the problem. That was the final response:

We appreciate your patience with this issue. Unfortunately, at this point the backend only supports idempotency in a transaction. This is a known issue and we will consider improving it for future.

However, for the cases which includes uniquely identifiable fields, we would recommend a workaround, where you could create an array and leverage the array union to add. Array remove can be used to remove it when the device is back online.

In my case, the application don't multiple devices writing on the same "field" so I made a workaround that was increment locally and update the field.

[mpmazim]

I have the same problem with firestore increment, it does double the value,

if i increment with vakue 100 it increments with 200.

i am using angular fire,

did you figure out the the problem?

[mpmazim]

Wow! That's quite the history, thank you very much for sharing that. Is it possible to show just a little bit of code for your solution, or stated differently at least the structure of the firestore objects that you update / set ?

did you found the issue?
it does happen to me

[mpmazim]

trust me, its a normal code, it doesn't double all the time, only for certain users. I even set firestore rules, still happend, i am 100% sure its firestore.

[mikehardy]

@mpmazim trust is not a thing in software development and code is essential for troubleshooting https://stackoverflow.com/help/how-to-ask / https://stackoverflow.com/help/minimal-reproducible-example - no offense, that's just how it goes - software is too surprising (thus the need for code) and environment dependent (thus the need to include all the info when asking) for troubleshooting to be efficient as a trust-based thing

If you have a reproduction with current versions and - as the original discussion starter stated - it is reproducible as a general firestore issue in all the SDKs, then pursuing it upstream in firebase-ios-sdk or firebase-android-sdk will be the best course of action

[mpmazim]

Hey mike, Take a look at my code
// ====this is security rules

match /holds/{holds}/{document=**}{

allow write , read: if

(request.resource.data.balance == ((resource == null?0: resource.data.balance) + float(request.resource.data._y)))
&&
(resource == null || !('_x' in resource.data.keys()) ||
resource.data._x != request.resource.data._x)

}

// =============
** I am using angularfire**

makeCashEntry(batch, data) {
data.amnt = (Math.round(data.amnt * 100) / 100)

const myRandomNum=Math.floor((Math.random() * 10000000) + 1)


var refBlUp = this.db.firestore.collection(this.CASH).doc("balance")


  batch.set(refBlUp, { balance: firestore.firestore.FieldValue.increment(-data.amnt) ,_x:myRandomNum ,_y:''+-data.amnt}, { merge: true });

}

[astanb]

I came to a similar conclusion and workaround, looks like Fieldvalue.increment is not really usable unless you are certain that you aren't writing to the same document in quick succession. Using the test above I could only recreate the issue when writes were performed < approx 300ms apart.

The Firestore Best practices do state that documents shouldn't be written to more than once per second.

I also couldn't use transactions because my app needs to work in offline mode (the main reason we use Firestore at all)

[mikehardy]

Could I beg 🙏 one of you to check the warning to the docs for that field that I am proposing?

I took the information from this discussion
https://github.com/invertase/react-native-firebase/pull/6208/files

This seems like something that could mess up accounting (based on your examples above) and I would hate for other people to suffer through discovering this

[astanb]

I'm not sure how easy it is to clarify what to be careful of when using increment. As @wfern states (and I did with my initial issue) it doesn't seem to have a correlation with the frequency of writes. My later test using the codesandbox did seem to indicate write frequency being the cause though, but I don't think it's as cut and dry as that. I'd probably go so far as to warn that any counters that need to be totally accurate should avoid using increment altogether.

[mikehardy]

Great feedback @alexstanbury thank you for checking it. I just altered the wording based on your feedback to be both more serious and more vague at the same time 😆 - what do you think?

https://github.com/invertase/react-native-firebase/pull/6208/files

Feel free to review the PR and propose actual different words if needed, the goal is a good warning to save developer's heartache, and I'm happy for the help

[mpmazim]

Hey mike, it means we should never use increment(). if it increments with the wrong value rather than fail, makes firestore unrelyable as other google products are.

[mikehardy]

You can use increment all you want @mpmazim but you should not do it in rapid succession, it is a documented limitation https://firebase.google.com/docs/firestore/best-practices#updates_to_a_single_document

makes firestore unrelyable as other google products are

I don't personally care as I'm not a google employee but if asking for help and/or advice in a forum supported (even indirectly) by Google, insults aren't the best way to go - dampens the mood in human to human interaction, right?

[frederichoule]

Is there any open issue with Google about this? I have the same problem on a big scale system right now, it's driving me crazy. Switching from FieldValue.increment() to get+update will drive up the costs.

[mikehardy]

@frederichoule no, this is a known limitation in firebase as linked above, if I understand your issue correctly. There are several workarounds discussed as general things - distributed counters etc https://firebase.google.com/docs/firestore/solutions/counters

[frederichoule]

Good to know. Thanks!

[mpmazim]

so, using incerement() doesnt gurantee anything, 1 write per second? seriously? , how deployed app can make sure thta?

[mpmazim]

i am pissed, i am on a big project based on increment(), the documentation didnt warn that limitation strong enough.

[astanb]

Yeah, it's not ideal, counters in Firestore have been a constant source of frustration for me.

However, there are workarounds, such as sharding as linked by Mike above, and using transactions.

Another method that may work for you is using the quite recently introduced COUNT functionality, which didn't exist when this issue was opened. Depending on your use case you may be able to use this in a cloud function to keep your counters up to date, instead of using increment.

Although this issue was opened in the react-native-firebase repo, it's certainly not a fault of this project, it is a problem with the underlying way Firestore works, so it may be a good idea to kick up a stink with them if you want anything to be done about it (good luck with that!).

[larssn]

It kinda pisses me off that Google/Firebase are hiding all these limitations from us, and I have to find it in a discussion thread on GitHub; and we don't even use React!

I WANT to know these things before working on my architecture. Now we need to rework a lot of code, that uses FieldValue.increment, because of this.

EDIT:
I should clarify that we're NOT updating the document is quick succession, like the OP mentioned he wasn't either. I can see from our logs, that one FieldValue.increment operation caused the document to get updated twice, with several seconds between each update (we log trigger events in BigQuery). The result was that the increment operation had doubled the value in question.

EDIT2: They've since removed the 1 write per second recommendation btw: https://stackoverflow.com/a/74455395/536485

[jgavazzisp]

Hey folks, one somewhat low cost mitigation would be to have an integrityCount using FieldValue.increment running inside a batch together with whatever operation you are running it in, if the FieldValue.increment runs twice inside the batch it is most likely due to batch being retried due to failure by the client to acknoledge the write from the server when they were merging the data. In our situation we noticed that that all FieldValue.increment would duplicate together inside the batch, so we have a cloud function which checks this integrityCount and ammend the data when duplication is detected.

[astanb]

Finding it a bit tricky to understand what you mean, could you give a code example of how you'd use this approach?

[jgavazzisp]

Hi there, here is an quick idea of what I mean. In our case all the FieldValue.increment inside a batch would duplicate together if one of them duplicated.

// Client-Side Batch Operation with FieldValue.increment

function runBatchOperation(docRef) {
  const batch = firestore.batch();

  // Increment integrityCount and perform other data operations
  batch.update(docRef, {
    integrityCount: FieldValue.increment(1), // Tracks integrity for detecting duplication
    someField: FieldValue.increment(10) // Example field that could get duplicated
  });

  // Commit the batch
  batch.commit();
}

// Cloud Function or Server-Side Check for Duplicates

function cloudFunctionCheckForDuplication(docRef) {
  firestore.get(docRef)
    .then((doc) => {
      const integrityCount = doc.data().integrityCount;
      const expectedCount = calculateExpectedCount(); // Logic to calculate expected count
      if (integrityCount > expectedCount) {
        // Duplication detected, amend the data
        amendDuplicatedData(docRef);
      }
    })
    .catch((error) => {
      console.log("Error fetching document:", error);
    });
}

// Function to amend duplicated data
function amendDuplicatedData(docRef) {
  firestore.update(docRef, {
    // Logic to correct the duplicated data
    someField: FieldValue.increment(-10) // Reverse duplicated increment
  });
}

// Example logic to calculate expected integrity count (custom logic)
function calculateExpectedCount() {
  // Replace this with the actual logic to determine the correct count
  return 1; // For example purposes, we expect 1 increment
}

[jgavazzisp]

Hey guys I found a better solution for it:

If the client never receives the server's ack for a commit (flaky network, app backgrounded), the Firestore SDK can resend the whole batch. Plain set/update writes are idempotent, so a resend does no harm. But FieldValue.increment(x) gets applied twice, so balances and counters drift.

Fix: add an idempotency marker to the same batch and let security rules refuse a second copy.

1. For every batch that increments anything, generate a fresh random saveId on the client. Add one extra write to the batch:

batch.set(doc(db, `parents/${parentId}/saveMarkers/${saveId}`), {
  createdAt: serverTimestamp(),
  expireAt: Timestamp.fromMillis(Date.now() + 30 * 24 * 3600 * 1000),
});

Do this in one shared place every batch passes through, not screen by screen.

2. Security rules: the marker can be created once, never updated or deleted by clients:

match /parents/{parentId}/saveMarkers/{saveId} {
  allow create: if canWrite(parentId);
  allow update, delete: if false;
}

On a resend (automatic retry from the SDK), the marker already exists, so the set counts as an update and is denied. Batches are atomic, so the whole resent batch is rejected, increments included. The user sees nothing.

3. Add a TTL policy on saveMarkers.expireAt so old markers clean themselves up (resends arrive within minutes or hours). There's no need to index the collection.

4. Old clients that don't send markers keep working; just never make the marker mandatory while they're still in use.

Testing: commit a batch, then commit an identical copy (same saveId, same amounts) against the emulator with the real rules on. The first commit succeeds, the second fails with PERMISSION_DENIED, and the counters equal the values after the first commit.