Commit graph

800 commits

Author SHA1 Message Date
Jakob Ackermann
8c39add865 [clsi-cache] meter ingress and egress bandwidth (#27143)
* [mics] fix "app" label in clsi-cache metrics in dev-env

* [clsi-cache] validate filePath when processing file

* [clsi-cache] meter ingress and egress bandwidth

Files are downloaded directly from nginx, hence we cannot meter egress
in clsi-cache easily.

GitOrigin-RevId: 24de8c41728f0e9c984113c1470dec6153e75f20
2025-07-16 08:05:59 +00:00
Andrew Rumble
1a1f283245 Update node to 22.17.0
GitOrigin-RevId: 7a15f7a0e95e3331c589955aca1fbc5dfb13a568
2025-07-09 08:06:01 +00:00
Andrew Rumble
b42b0a8d3e Make file related BlobStore type more specific
GitOrigin-RevId: d5e70b13cbdfe1faf133eb4c040b0bbf6fa47dea
2025-06-25 08:06:23 +00:00
Andrew Rumble
4157f8ca00 Add a list directory method to the S3 persistor
GitOrigin-RevId: 6ecff3eb457dc2168ca49ff9409bb09fa932781c
2025-06-25 08:06:05 +00:00
Jakob Ackermann
5b39c76aa8 Merge pull request #26506 from overleaf/em-lazy-string-file-data-store
Fix lazy file data truncation on store()

GitOrigin-RevId: 2316a096e6a365178afbded58351359893a36312
2025-06-19 08:05:13 +00:00
Eric Mc Sween
6f461564d5 Merge pull request #26367 from overleaf/em-history-ot-undo
Fixes to TextOperation invert algorithm

GitOrigin-RevId: dd655660f6ecad7b6e9b2d4435dc9a5364d0fde2
2025-06-17 08:06:11 +00:00
Jakob Ackermann
ab0199f238 [misc] migrate remaining references to our GCR repositories to AR (#26370)
* [misc] migrate remaining references to our GCR repositories to AR

* [server-ce] fix retagging of texlive images

GitOrigin-RevId: 81f955ad4c4486ad42b29cbd6bcc9d5ef4b1a432
2025-06-13 08:06:18 +00:00
Eric Mc Sween
101c994fec Merge pull request #26279 from overleaf/em-compose-tracking-props
Merge tracked inserts and deletes during composition

GitOrigin-RevId: f8cfcf79aef7cb3e7acaecf7c3baa69d71a4efa9
2025-06-12 08:05:21 +00:00
Liangjun Song
2f87db9c0d Merge pull request #24790 from overleaf/ls-use-script-runner
Update some scripts to use Script Runner

GitOrigin-RevId: aaa11f94dcfd328c158bb02d1b9fb2adfb1bb146
2025-05-23 08:05:23 +00:00
Antoine Clausse
436dcc977f Update Node to 22.15.1 (#25785)
GitOrigin-RevId: 52428d2d7e67c3135a1604fa487dd142aa08bf15
2025-05-22 08:07:42 +00:00
Alf Eaton
827fb19df7 Define Change.toRaw result as RawChange (#25676)
GitOrigin-RevId: 0fa23806e9d7ee015b5a2a542d52382499edf9ab
2025-05-22 08:07:07 +00:00
Antoine Clausse
67ab5a749a Update Node to 22.15.0 (#24699)
* Run `bin/update_node 20.18.2 22.15.0`

* Remove expects on `fetchMock.callHistory.done()` to fix tests: are they necessary?

* Set node version to `22.x` in linked-url-proxy

* Increase test timeout to 30s in `github-sync`, Add waiting steps

* Define `navigator.onLine` in tests setup

GitOrigin-RevId: 75eb556e9f51b665e57497a0879b6915d14069ce
2025-05-16 08:06:26 +00:00
Eric Mc Sween
2dd054d602 Merge pull request #25606 from overleaf/em-o-error-types
OError: handle unknown error types
GitOrigin-RevId: a52dd92a4f95cd738556612599805f41f24de69f
2025-05-16 08:06:22 +00:00
Antoine Clausse
fee5ea8411 Remove returns of promises within functions with callbacks (address DeprecationWarning) (#25603)
* [document-updater] Don't return promises within functions with callbacks

Remove the errors:
DeprecationWarning: Calling promisify on a function that returns a Promise is likely a mistake
https://cloudlogging.app.goo.gl/YHDhoarvLEw2w9rXA

* Remove some more unnecessary returns in functions with callbacks, for consistency

* Add `sendCanaryAppliedOp` to excluded methods for promisification

GitOrigin-RevId: fa6d3e47c4e6561dc29d4c15e57c3289fc1f3dfa
2025-05-16 08:05:33 +00:00
Jakob Ackermann
e8b5ee2ff9 [history-ot] initial implementation of using doc-level history-ot (#25054)
* [history-v1-ot] initial implementation of using doc-level history-v1-ot

* [web] fix advancing of the otMigrationStage

Use 'nextStage' for the user provided, desired stage when advancing.

Co-authored-by: Brian Gough <brian.gough@overleaf.com>

* [document-updater] document size check in editor-core

* [history-ot] rename history-v1-ot to history-ot and add types

* [history-ot] apply review feedback

- remove extra !!
- merge variable assignment when processing diff-match-match output
- add helper function for getting docstore lines view of StringFileData

Co-authored-by: Alf Eaton <alf.eaton@overleaf.com>

* Revert "[document-updater] add safe rollback point for history-ot (#25283)"

This reverts commit d7230dd14a379a27d2c6ab03a006463a18979d06

Signed-off-by: Jakob Ackermann <jakob.ackermann@overleaf.com>

---------

Signed-off-by: Jakob Ackermann <jakob.ackermann@overleaf.com>
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
Co-authored-by: Alf Eaton <alf.eaton@overleaf.com>
GitOrigin-RevId: 89c497782adb0427635d50d02263d6f535b12481
2025-05-08 08:05:44 +00:00
Antoine Clausse
81941ff335 Update some dependencies so they're compatible with Node 22 (#25317)
* `"@google-cloud/profiler": "^6.0.3"`

* `bin/npm update pprof`

* `bin/npm update nan`

* `bin/npm update @google-cloud/profiler`

* Ignore false positive of `@typescript-eslint/return-await`

> Returning an awaited value that is not a promise is not allowed
Though the function was promisified

GitOrigin-RevId: 24dbe3e8df2b55c0b9583ac79a61e0956ac3fac0
2025-05-07 08:06:57 +00:00
Antoine Clausse
9a2847dbee [web] Add startup metrics (#25277)
* [web] refactor startup sequence

The primary objective here is to call loadGlobalBlobs() only once.
But to get there, we need to reorder things and add extra try/catch
sections to ensure we are not letting the global uncaughtException
handler catch startup errors.

Co-authored-by: Antoine Clausse <antoine.clausse@overleaf.com>

* [web] add metrics for startup steps

Co-authored-by: Antoine Clausse <antoine.clausse@overleaf.com>

---------

Co-authored-by: Jakob Ackermann <jakob.ackermann@overleaf.com>
GitOrigin-RevId: c73edea02516e919d55b896588dcd1862835fedf
2025-05-07 08:06:52 +00:00
Jakob Ackermann
07b2255426 [misc] cleanup .dockerignore and .gitignore files (#25312)
- Remove settings ignore, they are inconsistent and break local prettier
- Remove .dockerignore files, only root ignore file is used
- Move .idea/.run/*.swp/coverage to root
- Remove .npmrc entries, we are no longer writing the rc file
- Remove node_modules/.DS_Store, is contained in root
- Remove cruft

GitOrigin-RevId: 3025fd5acaef343312f55149466c638e759a6e1f
2025-05-07 08:05:24 +00:00
Jakob Ackermann
478e264817 [fetch-utils] fix leak of abort event handlers in AbortSignal (#25172)
GitOrigin-RevId: 992496010eb1cbe571b2e87fab8e7227b0d64538
2025-04-30 08:06:05 +00:00
Brian Gough
fe8d6392d5 Merge pull request #24793 from overleaf/bg-update-build-scripts-to-include-history-redis
update build scripts to include HISTORY_REDIS_HOST

GitOrigin-RevId: da0f317c80401067c0f4aa772196cb2f24849b8e
2025-04-11 08:06:03 +00:00
Brian Gough
9f527f10e1 Merge pull request #24104 from overleaf/bg-batched-update-concurrency-warning
prevent concurrent execution of batchedUpdate

GitOrigin-RevId: 90853ccd83943d8cd7b01fd11f152512d806e9a7
2025-04-04 08:05:00 +00:00
Andrew Rumble
d2d556ddf6 Allow cached persistor to get size of a blob
GitOrigin-RevId: 7e2c14381cf717c21325917ef4f78acb340a50d9
2025-03-24 10:48:35 +00:00
Jakob Ackermann
70601db76f [fetch-utils] export fetchRedirectWithResponse (#24372)
GitOrigin-RevId: 7cd30479fbe70fce68f2d6ef573187fc4f412749
2025-03-24 10:46:19 +00:00
Jakob Ackermann
e99cd74cca [fetch-utils] add fetchRedirectWithResponse (#24341)
GitOrigin-RevId: 9b1e9b02399beea87dbae0f87d1d7d897d145178
2025-03-18 09:05:44 +00:00
Andrew Rumble
d04bfbcdea Add promiseMapSettledWithLimit utility
GitOrigin-RevId: e34102de17f28e43deb383d630088c6e753e2ec1
2025-03-12 09:05:22 +00:00
Eric Mc Sween
290bdf4361 Merge pull request #23993 from overleaf/em-remove-fixed-remove-change-flag
Remove fixedRemoveChange flag from editor updates

GitOrigin-RevId: bf74e1137560184c4b024a3b5c6ede5a841d3559
2025-03-04 09:04:48 +00:00
Andrew Rumble
767591973c Merge pull request #23905 from overleaf/ar-mocha-11
[all] Bump mocha everywhere

GitOrigin-RevId: bf5587047f9ceb88872cbd356c2029717bb8ca9d
2025-02-28 09:05:40 +00:00
Miguel Serrano
2e72aa8a47 Merge pull request #23652 from overleaf/msm-update-path-to-regexp
[monorepo] Update express to 4.1.2

GitOrigin-RevId: b3db75257e1edc1a25a1a657def352ecca926920
2025-02-21 09:05:26 +00:00
Eric Mc Sween
879aa33cbe Merge pull request #23519 from overleaf/em-remove-ordered-rejections-flag
Remove orderedRejections flag on ops

GitOrigin-RevId: df464023e7e6c3b5899e6a424f423a374fd33df6
2025-02-13 09:05:53 +00:00
Eric Mc Sween
510ad0ce9a Merge pull request #23509 from overleaf/em-fix-remove-change
Fix RangesTracker._removeChange() when multiple changes have the same id

GitOrigin-RevId: 2dafcf275b283da320bca70b460aba3051ca9575
2025-02-12 09:05:29 +00:00
Jakob Ackermann
0a7b2004d2 [misc] silence logger when running tests (#22243)
* [misc] silence logger when running tests

* [misc] re-enable logging in some tests for scripts

* [misc] make it easy to turn on verbose logging for tests

```
LOG_LEVEL=debug make test_unit
LOG_LEVEL=debug make test_acceptance
```
GitOrigin-RevId: 219bc6d1f9cbdb89ddd7d94742920913ddde4514
2025-02-10 09:06:02 +00:00
Eric Mc Sween
ce4c8a4e47 Merge pull request #23398 from overleaf/em-log-doc-hash-mismatches
Validate content hashes in history (log only)

GitOrigin-RevId: ed772fc4e4d0aa9e980f9693a759647bd937e13a
2025-02-07 09:05:59 +00:00
Jakob Ackermann
26321c5dba [history-v1] block deletion of bucket prefix that is not a valid project prefix (#22889)
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
GitOrigin-RevId: a3aff76a2e299b2e2ed030e34da0631fce51ad4b
2025-02-07 09:05:01 +00:00
Jakob Ackermann
6b8a50eb92 [project-history] trim logging (#23408)
* [project-history] trim logging when projectVersion is out of order

* [editor-core] record reason rather than full pathname when not clean

* [editor-core] remove pathname from error messages

* [project-history] avoid logging updates when historyId is inconsistent

* [project-history] avoid logging failed history-v1 requests repeatedly

* [project-history] log once when doc version is out of order in chunk

GitOrigin-RevId: 5ac431ac87b57c00fac9d5ab486ca305b1775372
2025-02-06 09:04:35 +00:00
Eric Mc Sween
39842c9857 Merge pull request #23250 from overleaf/em-project-history-doc-hash
Propagate the doc hash from project-history to history

GitOrigin-RevId: 341df52df41f7a5f8dbebbce53a47f9d5e1f8175
2025-02-04 09:05:03 +00:00
Jakob Ackermann
ba60f885a4 [project-history] fetch most recent chunk once when processing updates (#23261)
GitOrigin-RevId: 08ddd1f953b0dbae0541a60b45fec0f88e0a1d06
2025-01-31 09:05:14 +00:00
Antoine Clausse
272108a213 Run bin/update_node 20.18.0 20.18.2 (#23074)
GitOrigin-RevId: 3f1ab14c4c36ea54408e0ce2c404a323bfe8d9a6
2025-01-24 09:06:05 +00:00
Andrew Rumble
47db5c1236 Merge pull request #23039 from overleaf/ar-bump-mongo-6.12
[many] Bump mongodb to 6.12.0 universally

GitOrigin-RevId: 6df934518e95217b6157ca5a41cf8acf4beb7689
2025-01-23 09:06:13 +00:00
Eric Mc Sween
e9c1c0f9c8 Merge pull request #22650 from overleaf/em-tracked-deletes-at-same-position
Handle multiple tracked deletes at same position

GitOrigin-RevId: 3cbf1c418bcd50cf08e1b90ce6ba3bc480236079
2025-01-14 09:05:34 +00:00
Eric Mc Sween
a323a5c915 Merge pull request #22527 from overleaf/revert-22471-em-tracked-deletes-at-same-position
Revert "Improve handling of tracked delete rejections"

GitOrigin-RevId: 444a5a73aa2b93162036dfc463cd76d9c463aadb
2024-12-16 09:05:10 +00:00
Eric Mc Sween
6737cfa38d Merge pull request #22471 from overleaf/em-tracked-deletes-at-same-position
Improve handling of tracked delete rejections

GitOrigin-RevId: 35857d0a3f739c0531223737b2b649c9e8033157
2024-12-16 09:04:58 +00:00
Rebeka Dekany
7493936bad Merge pull request #22310 from overleaf/ar-bump-mongoose-8.8.3
[web] Upgrade mongoose to 8.8.3

GitOrigin-RevId: 5540f7384b74daafa01522c23633587fc56a7faa
2024-12-06 09:05:25 +00:00
Antoine Clausse
a6ec075f78 Increase checkSummaryValues ranges to fix flakiness (#22227)
Co-authored-by: Rebeka <rebeka.dekany@overleaf.com>
GitOrigin-RevId: d6da0b1e66cf5cb4ef4dbed91ae468ffe5807e37
2024-12-02 09:06:22 +00:00
Jakob Ackermann
3d7254b419 Merge pull request #22153 from overleaf/jpa-backup-verifier-minimal
[history-v1] backup-verifier-app: initial revision

GitOrigin-RevId: 922c9f94cb7ca7c129e38fd6961d42bdff819cd8
2024-11-27 09:04:55 +00:00
Brian Gough
6672372828 remove sentry from backend services (#20752)
* remove sentry from backend services - no longer required

* Remove Sentry integration from logging manager

* Remove Sentry from clsi default settings

* Remove `initializeErrorReporting` in libraries/logger

* Remove `@sentry/node` from `libraries/logger`

---------

Co-authored-by: Antoine Clausse <antoine.clausse@overleaf.com>
GitOrigin-RevId: 8149a885f5258804b93ae39cde7b7333e992532a
2024-11-27 09:04:50 +00:00
Jakob Ackermann
0bd8729cc1 Merge pull request #22115 from overleaf/jpa-skip-get
[history-v1] back_fill_file_hash: optimize obtaining the DEK

GitOrigin-RevId: 6fc5218737551ae4e6152414c32560f545767091
2024-11-26 09:04:50 +00:00
Jakob Ackermann
ed27af11f8 Merge pull request #22107 from overleaf/jpa-error-handling
[history-v1] back_fill_file_hash: error handling

GitOrigin-RevId: 2dbf52d015206cf42ca434b1f16221d6e0f2cc2b
2024-11-26 09:04:45 +00:00
Jakob Ackermann
69021ca4b0 Merge pull request #22061 from overleaf/jpa-reduce-idle-time
[history-v1] back_fill_file_hash: reduce idle time between batches

GitOrigin-RevId: 9b24bb882b158f33915d2e0ab2f82273eac09739
2024-11-22 09:05:55 +00:00
Jakob Ackermann
4f25b0de91 Merge pull request #22028 from overleaf/jpa-logging
[misc] logging tweaks

GitOrigin-RevId: 7757b81b6c684955374038376035dc2bd272c4e8
2024-11-21 09:05:41 +00:00
Jakob Ackermann
8e74d3c58c Merge pull request #21947 from overleaf/bg-jpa-back-fill-script-tweaks
[history-v1] back_fill_file_hash: performance tweaks

GitOrigin-RevId: c3d0c7906707fc902addcde64eaf41c24ceeece7
2024-11-20 09:04:51 +00:00