Commit graph

63 commits

Author SHA1 Message Date
Jakob Ackermann
842f6c289f [document-updater] make setDoc aware of tracked deletes in history-ot (#26126)
GitOrigin-RevId: efa1a94f2f435058b553f639e43832454c58591d
2025-06-06 08:05:23 +00:00
Jakob Ackermann
1e6b13f9d5 [history-ot] rename remaining history-v1-ot references to history-ot (#25428)
* [history-ot] rename remaining history-v1-ot references to history-ot

* [web] rename History-v1 OT -> History OT in admin panel

* [web] rename OT Migration -> History OT Migration in admin panel

GitOrigin-RevId: 103ce816d5320d6379d51009cdc08b8a71aa48e6
2025-05-20 08:06:46 +00:00
Jakob Ackermann
e8b5ee2ff9 [history-ot] initial implementation of using doc-level history-ot (#25054)
* [history-v1-ot] initial implementation of using doc-level history-v1-ot

* [web] fix advancing of the otMigrationStage

Use 'nextStage' for the user provided, desired stage when advancing.

Co-authored-by: Brian Gough <brian.gough@overleaf.com>

* [document-updater] document size check in editor-core

* [history-ot] rename history-v1-ot to history-ot and add types

* [history-ot] apply review feedback

- remove extra !!
- merge variable assignment when processing diff-match-match output
- add helper function for getting docstore lines view of StringFileData

Co-authored-by: Alf Eaton <alf.eaton@overleaf.com>

* Revert "[document-updater] add safe rollback point for history-ot (#25283)"

This reverts commit d7230dd14a379a27d2c6ab03a006463a18979d06

Signed-off-by: Jakob Ackermann <jakob.ackermann@overleaf.com>

---------

Signed-off-by: Jakob Ackermann <jakob.ackermann@overleaf.com>
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
Co-authored-by: Alf Eaton <alf.eaton@overleaf.com>
GitOrigin-RevId: 89c497782adb0427635d50d02263d6f535b12481
2025-05-08 08:05:44 +00:00
Jakob Ackermann
a5e2708eae [document-updater] add safe rollback point for history-ot (#25283)
GitOrigin-RevId: d7230dd14a379a27d2c6ab03a006463a18979d06
2025-05-06 08:05:28 +00:00
Jakob Ackermann
7f086b21c8 [document-updater] modernize ApplyingUpdatesToADocTests (#25038)
- use beforeEach to ensure tests do not interfere with each other
  Notably, the 'when the ops come in a single linear order' test suite
  had state-changing tests that were dependent on the correct order.
  Also, the assigment of 'this.firstOpTimestamp' was in a test.
- consolidate populating project and doc ids
  The doc reference in this.update was undefined.
- fix doc reference in updates
  There were two misuses of 'doc_id' instead of 'doc'.
- Move mocking of MockWebApi.getDocument to the top and use
  sinon.resetHistory() or sinon.restore() for controlling the stub.
- Add another test for simple transforming of updates
  See 'when another client is sending a concurrent update'.

GitOrigin-RevId: 61ca8a1b0172920ad6ab1b604a9b9259cebddaad
2025-04-24 08:05:10 +00:00
Jakob Ackermann
ff78f687d8 [document-updater] add endpoint for project wide last updated timestamp (#24352)
* [document-updater] fix acceptance test for sending two updates

The Object.create() invocation yields an empty object. The following v
assignment works as expected. The effective update is { v: 43 }.
Processing that fails as no operations were included.

* [document-updater] add endpoint project wide last updated timestamp

* [document-updater] apply review feedback

Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>

---------

Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
GitOrigin-RevId: 81397537bfd85c2077f19669860b1391c15b34a3
2025-03-24 10:46:38 +00:00
Jakob Ackermann
0a7b2004d2 [misc] silence logger when running tests (#22243)
* [misc] silence logger when running tests

* [misc] re-enable logging in some tests for scripts

* [misc] make it easy to turn on verbose logging for tests

```
LOG_LEVEL=debug make test_unit
LOG_LEVEL=debug make test_acceptance
```
GitOrigin-RevId: 219bc6d1f9cbdb89ddd7d94742920913ddde4514
2025-02-10 09:06:02 +00:00
Antoine Clausse
7f48c67512 Add prefer-node-protocol ESLint rule (#21532)
* Add `unicorn/prefer-node-protocol`

* Fix `unicorn/prefer-node-protocol` ESLint errors

* Run `npm run format:fix`

* Add sandboxed-module sourceTransformers in mocha setups

Fix `no such file or directory, open 'node:fs'` in `sandboxed-module`

* Remove `node:` in the SandboxedModule requires

* Fix new linting errors with `node:`

GitOrigin-RevId: 68f6e31e2191fcff4cb8058dd0a6914c14f59926
2024-11-11 09:04:51 +00:00
Jakob Ackermann
a98fefd24b Merge pull request #20129 from overleaf/jpa-handle-partial-deletion
[misc] improve handling of document deletion

GitOrigin-RevId: bd6b225b91ab38365e9ff272c50ece995e767bf2
2024-08-27 08:04:53 +00:00
Eric Mc Sween
22257cf037 Merge pull request #18583 from overleaf/em-multi-hash-key
Ensure that MULTI commands always go to a single node

GitOrigin-RevId: 223d29986766577df89c82983126dabca5d16eed
2024-05-30 08:05:17 +00:00
Jakob Ackermann
a540754f6e Merge pull request #18116 from overleaf/jpa-bulk-replace-localhost
[misc] bulk replace localhost with 127.0.0.1

GitOrigin-RevId: d238f3635302e8ff5500d611108c4d1bef216726
2024-04-26 08:04:39 +00:00
Domagoj Kriskovic
c4437c69bc Send operations to project-history when accepting tracked changes (#17599)
* added getHistoryOpForAcceptedChange in RangesManager

* rename adjustHistoryUpdatesMetadata to be treated as public

* handle retain op in UpdateTranslator and updateCompressor

* send op to project-history in acceptChanges

* use promises.queueOps

* use ranges in getHistoryOpForAcceptedChange

* using rangesWithChangeRemoved

* acceptChanges acceptance test

* using change.op.hpos

* Revert "using change.op.hpos"

This reverts commit f53333b5099c840ab8fb8bb08df198ad6cfa2d84.

* use getHistoryOpForAcceptedChanges

* fix historyDocLength

* Revert "rename adjustHistoryUpdatesMetadata to be treated as public"

This reverts commit 2ba9443fd040a5c953828584285887c00dc40ea6.

* fix typescript issues

* sort changes before creating history updates

* fix tests

* sinon spy RangesManager.getHistoryUpdatesForAcceptedChanges

* added unit tests

* sort deletes before inserts

* use getDocLength function

* fix docLength calculation

* fix typo

* allow all retains

* fix lint error

* refactor RangesTests

* fix ts error

* fix history_doc_length calculation in RangesManager

* remove retain tracking check from UpdateCompressor

* use makeRanges() properly in tests

* refactor acceptance tests

GitOrigin-RevId: ab12ec53c5f52c20d44827c6037335e048f2edb0
2024-04-17 08:04:17 +00:00
Eric Mc Sween
377cd82f20 Merge pull request #16928 from overleaf/em-hpos-hlen
Add history metadata to updates sent to project-history

GitOrigin-RevId: 915beaa01f2bbe48869a40b229397be8e0401852
2024-02-14 09:05:02 +00:00
Jakob Ackermann
216e3b2091 Merge pull request #16659 from overleaf/jpa-loose-progress-assertion
[document-updater] loose assertion on progress with low LIMIT

GitOrigin-RevId: f5046e0ed46b889e5a49a69b9752961f847d361a
2024-01-24 09:04:41 +00:00
Jakob Ackermann
a7c7f2e69b Merge pull request #16589 from overleaf/jpa-script-redis-mongo-mismatch
[document-updater] add script for checking for redis vs mongo de-syncs

GitOrigin-RevId: b51b61efd750f37180c09af5ccf5681712719ee2
2024-01-24 09:03:59 +00:00
Christopher Hoskin
f5089f596c Merge pull request #15477 from overleaf/csh-td-issue-11625-mongo-ug-5-document-updater
Upgrade mongodb module for document updater from 4.11.0 to 6.2.0

GitOrigin-RevId: 4154763ded77e04cd3f4002824fee4af8b7f8e8a
2023-12-15 09:03:38 +00:00
Jakob Ackermann
06b93aac50 Merge pull request #14377 from overleaf/tw-docupdater-flush
add check for unflushedTime

GitOrigin-RevId: e09d63b4de09e30ceb82792526224ad8b2415119
2023-09-07 08:04:57 +00:00
Eric Mc Sween
1aff1b64ab Merge pull request #14662 from overleaf/em-revert-docupdater-history-ranges-support
Revert document-updater changes from #14567

GitOrigin-RevId: 72af1d24c94f7c46f2557700e6425f278356cffd
2023-09-06 08:06:46 +00:00
Eric Mc Sween
3fa1245860 Merge pull request #14567 from overleaf/em-history-ranges-flag
Add historyRangesSupport flag to projects

GitOrigin-RevId: 1e3f24a7c6f209bbd34eaaf4caee56dc7061b3da
2023-09-06 08:06:19 +00:00
Miguel Serrano
2ccc1aeea5 Merge pull request #13264 from overleaf/msm-docupdater-cleanup
[docupdater] Cleanup track-changes code

GitOrigin-RevId: 906a4f57a9e7348f47af67cff44a32ae1bb2debb
2023-07-17 10:24:47 +00:00
Eric Mc Sween
3831416c2f Merge pull request #12206 from overleaf/em-camel-case-docupdater
Camel case variables in document-updater

GitOrigin-RevId: 76ad0921cc059878f21639547fad1bff1913bc8b
2023-03-22 09:04:37 +00:00
Eric Mc Sween
f76ab26822 Merge pull request #10998 from overleaf/em-upgrade-mongo-document-updater
Upgrade Mongo driver in document updater

GitOrigin-RevId: e3c4d86bbcf86dcf16fdfdbfab2aaf0d8eb1037c
2023-01-11 09:05:28 +00:00
Lucie Germain
e7203ed0f8 Merge pull request #10232 from overleaf/jpa-lg-silent-no-op-external-update
[document-updater] skip emitting no-op external updates

GitOrigin-RevId: 3fc29740fb8c2bc8b1ba21428a0e8d0d34b6eaf8
2022-11-04 09:03:54 +00:00
Eric Mc Sween
59cdcccc26 Merge pull request #9647 from overleaf/bg-dropbox-to-overleaf-existing-doc
handle updates to existing doc in dropbox to overleaf metadata

GitOrigin-RevId: e82955a4a76e62fb649263a94103fdb7f322de85
2022-09-22 08:04:11 +00:00
Brian Gough
587cd33d82 Merge pull request #5860 from overleaf/bg-fix-docupdater-delete-doc
fix docupdater delete operation

GitOrigin-RevId: daab03b7659688d53c54ac80756394796d3c6348
2021-11-23 09:02:35 +00:00
Jakob Ackermann
7f9fd00bda Merge pull request #5367 from overleaf/jpa-node-handle-callback-err
[misc] fix eslint violations for node/handle-callback-err

GitOrigin-RevId: 83a4900e8861010df1917bff49382bd9c93375bd
2021-10-28 08:03:26 +00:00
Brian Gough
5d6a833216 Merge pull request #5338 from overleaf/bg-update-docupdater-logger
[docupdater] upgrade from logger-sharelatex to @overleaf/logger

GitOrigin-RevId: 26294a1e61d25b907d594a586f9b4714bc36ae4f
2021-10-07 08:04:03 +00:00
Brian Gough
865a973426 Merge pull request #5132 from overleaf/bg-fix-size-check
[docupdater] use more accurate doc size check

GitOrigin-RevId: f66d68a7f7fdc127cc31539abdcab65549823d02
2021-09-21 08:02:55 +00:00
Brian Gough
5e2d4d2169 add acceptance tests for peek 2021-08-02 14:13:38 +01:00
Jakob Ackermann
29cea1743b [misc] DeletingADocumentTests: fix behavior checks
The preloading setup does an initial getDocument request to web-api.
2021-07-13 13:12:34 +01:00
Jakob Ackermann
bf8224e2a0 [misc] ApplyingUpdatesToADocTests: adhere to stricter chai assertions 2021-07-13 12:51:06 +01:00
Jakob Ackermann
1184f8f5b6 [misc] fix chai assertions .equal.false -> .equals(false) 2021-07-13 12:35:32 +01:00
Jakob Ackermann
c532376e21 [misc] run format_fix and lint:fix 2021-07-13 12:04:42 +01:00
Jakob Ackermann
519dd274bf [misc] switch from settings-sharelatex to @overleaf/settings 2021-07-12 17:47:15 +01:00
Jakob Ackermann
309ad818f6 [SizeCheckTests] add acceptance tests for size limit checks 2021-05-06 18:21:19 +01:00
Eric Mc Sween
4d70bd664f Reintroduce Node 12 and metrics upgrades
These changes were previously merged, not deployed, and reverted. This
reverts the revert.

This reverts commit a6b8c6c658b33b6eee78b8b99e43308f32211ae2, reversing
changes made to 93c98921372eed4244d22fce800716cb27eca299.
2021-04-01 15:51:00 -04:00
Eric Mc Sween
4dd1b26b2e Revert "Merge pull request #161 from overleaf/em-upgrade-node-12"
This reverts commit d44102751b9436ad89c5b3b05e7abdff51fcc78a, reversing
changes made to 6c2f5b8d053b75c677da2b7ddd04f998d2be6fff.
2021-03-31 12:07:11 -04:00
Eric Mc Sween
dc5bfea181 Global test setup
Configure chai and SandboxedModule globally with options used in many
tests.

That required upgrading mocha and SandboxedModule.
2021-03-16 16:48:45 -04:00
Thomas
23738540ed Fix API request errors which could contain API hostname or address (#160)
Wrap errors produced by failing requests to web API, and remove the url/hostname from thrown error messages. (But keep the URL path for info.)
2021-02-24 15:09:19 +01:00
Henry Oswald
0cdeffae6c fix off by 1 error in Doc updater client helper file 2021-02-09 10:50:37 +00:00
Henry Oswald
11c8cfc939 shard the pending-updates-list queue 2021-02-02 16:38:25 +00:00
Jakob Ackermann
08ed5f6c9b [misc] bump @overleaf/redis-wrapper to version 2.0.0 2020-11-11 16:20:32 +00:00
Jakob Ackermann
1d57706d44 [misc] migrate acceptance tests to the native mongo driver, drop mongojs 2020-08-28 17:06:25 +01:00
Eric Mc Sween
1d1f204021 Remove backwards-compat project update API
The project update endpoint accepted updates both in two array params:
docUpdates and fileUpdates, and in a single array: updates. This commit
removes the docUpdates/fileUpdates params now that web uses the updates
param.
2020-05-20 16:26:22 -04:00
Eric Mc Sween
c018fee72c Decaf cleanup: camel case variables 2020-05-20 16:12:27 -04:00
Eric Mc Sween
abb7e8fa20 Decaf cleanup: error handling 2020-05-20 16:08:03 -04:00
Eric Mc Sween
05a2cf829c Decaf cleanup: simplify loops 2020-05-20 16:06:42 -04:00
Eric Mc Sween
e9df9714e5 Decaf cleanup: unnecessary returns 2020-05-20 15:54:36 -04:00
Eric Mc Sween
8bbfd25d47 Decaf cleanup: simplify null checks 2020-05-20 15:53:06 -04:00
Eric Mc Sween
3830b8029a Decaf cleanup: camel case variables 2020-05-20 15:35:12 -04:00