Investigate slowdown seen in test of offline entities

[matthew-white]

The following test in Backend takes a long time. Even at 16 seconds, the test sometimes times out (#2027).

https://github.com/getodk/central-backend/blob/e01a718c52a0840ae5f34a075893bf971ae054c4/test/integration/api/offline-entities.js#L1930

describe('locking an entity while processing a related submission @slow', function() {
  this.timeout(16_000);

  // https://github.com/getodk/central/issues/705
  it('should concurrently process an offline create + update', testServiceFullTrx(async (service, container) => {

The test was introduced in getodk/central-backend#1202 as part of #705. Also related: getodk/central-backend#1195.

@alxndrsn thinks this might be a a real issue, not just a slow test. From getodk/central-backend#1860 (comment):

I'm suspicious that this is a postgres issue rather than a timeout that can safely be increased…

Notice that the event processing gets very slow just before failure: …

Perhaps getting progressively slower, or something has caused a slowdown which then causes test timeout.

This issue is to investigate this slowdown. When does it occur / what causes it? Does it indicate a real issue in production?

[matthew-white]

It looks like I was the one to work on #705 and write this test, which I didn't remember. The test is unusual in that it runs the same race condition 50 times, as the underlying issue was thought to be intermittent. The test is the only one to use exhaustParallel().

[matthew-white]

Discussion from team meeting today: Change the timeout from 16 seconds to 120 seconds. If it's timing out even at that point, then it's more important to look into.