Commit graph

717 commits

Author SHA1 Message Date
Andrew Rumble
1a1f283245 Update node to 22.17.0
GitOrigin-RevId: 7a15f7a0e95e3331c589955aca1fbc5dfb13a568
2025-07-09 08:06:01 +00:00
Christopher Hoskin
fe64856be7 Merge pull request #26021 from overleaf/csh-issue-25976-dev-env-ci
Upgrade to Redis 7.4 in dev and CI

GitOrigin-RevId: 068e54899bf50a247fedd0243d66f1545bc7cf01
2025-06-02 08:05:21 +00:00
Antoine Clausse
436dcc977f Update Node to 22.15.1 (#25785)
GitOrigin-RevId: 52428d2d7e67c3135a1604fa487dd142aa08bf15
2025-05-22 08:07:42 +00:00
Antoine Clausse
b667cef262 Revert "Update defaultHighWaterMark to 64KiB (Node 22's default) (#25522)" (#25789)
This reverts commit 19d731abf683066654027de3a4f9ac0b8916f22c.

GitOrigin-RevId: eb7c45ab45e02054601b607a4bfeb432424a1837
2025-05-22 08:07:38 +00:00
Jakob Ackermann
1e6b13f9d5 [history-ot] rename remaining history-v1-ot references to history-ot (#25428)
* [history-ot] rename remaining history-v1-ot references to history-ot

* [web] rename History-v1 OT -> History OT in admin panel

* [web] rename OT Migration -> History OT Migration in admin panel

GitOrigin-RevId: 103ce816d5320d6379d51009cdc08b8a71aa48e6
2025-05-20 08:06:46 +00:00
Jakob Ackermann
18cd52cfa1 [real-time] avoid shutting down all pods simultaneously (#25627)
GitOrigin-RevId: e416e06588b915548c83d70433f411c9f303ad87
2025-05-19 08:05:25 +00:00
Jakob Ackermann
0e13796882 [real-time] bail out early upon hitting a redis error (#25614)
GitOrigin-RevId: 2563094d1ec8017450fdfdb2b0e77d74bbc825d1
2025-05-19 08:05:21 +00:00
Andrew Rumble
daa52d62fa Allow an empty origin request in real-time
This will only happen with a same-origin request (or if someone has
tampered with the request - in which case they could set anything).

Co-authored-by: Tim Down <158919+timdown@users.noreply.github.com>
GitOrigin-RevId: 9dfe49f974a476bfe215768d3984dd60a381d37a
2025-05-19 08:05:16 +00:00
Antoine Clausse
67ab5a749a Update Node to 22.15.0 (#24699)
* Run `bin/update_node 20.18.2 22.15.0`

* Remove expects on `fetchMock.callHistory.done()` to fix tests: are they necessary?

* Set node version to `22.x` in linked-url-proxy

* Increase test timeout to 30s in `github-sync`, Add waiting steps

* Define `navigator.onLine` in tests setup

GitOrigin-RevId: 75eb556e9f51b665e57497a0879b6915d14069ce
2025-05-16 08:06:26 +00:00
Antoine Clausse
aa002369cb Update defaultHighWaterMark to 64KiB (Node 22's default) (#25522)
* Set defaultHighWaterMark to 16KiB

This is already the default in Node 20

* Set defaultHighWaterMark to 64KiB

Per https://github.com/overleaf/internal/pull/25522#issuecomment-2872035192

GitOrigin-RevId: 19d731abf683066654027de3a4f9ac0b8916f22c
2025-05-14 08:05:16 +00:00
Jakob Ackermann
e8b5ee2ff9 [history-ot] initial implementation of using doc-level history-ot (#25054)
* [history-v1-ot] initial implementation of using doc-level history-v1-ot

* [web] fix advancing of the otMigrationStage

Use 'nextStage' for the user provided, desired stage when advancing.

Co-authored-by: Brian Gough <brian.gough@overleaf.com>

* [document-updater] document size check in editor-core

* [history-ot] rename history-v1-ot to history-ot and add types

* [history-ot] apply review feedback

- remove extra !!
- merge variable assignment when processing diff-match-match output
- add helper function for getting docstore lines view of StringFileData

Co-authored-by: Alf Eaton <alf.eaton@overleaf.com>

* Revert "[document-updater] add safe rollback point for history-ot (#25283)"

This reverts commit d7230dd14a379a27d2c6ab03a006463a18979d06

Signed-off-by: Jakob Ackermann <jakob.ackermann@overleaf.com>

---------

Signed-off-by: Jakob Ackermann <jakob.ackermann@overleaf.com>
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
Co-authored-by: Alf Eaton <alf.eaton@overleaf.com>
GitOrigin-RevId: 89c497782adb0427635d50d02263d6f535b12481
2025-05-08 08:05:44 +00:00
Jakob Ackermann
07b2255426 [misc] cleanup .dockerignore and .gitignore files (#25312)
- Remove settings ignore, they are inconsistent and break local prettier
- Remove .dockerignore files, only root ignore file is used
- Move .idea/.run/*.swp/coverage to root
- Remove .npmrc entries, we are no longer writing the rc file
- Remove node_modules/.DS_Store, is contained in root
- Remove cruft

GitOrigin-RevId: 3025fd5acaef343312f55149466c638e759a6e1f
2025-05-07 08:05:24 +00:00
Andrew Rumble
adb9723d62 Log a warning when a user is rejected from accessing real-time by CORS
GitOrigin-RevId: 04a7ffbc24654c876688db446164bf36a162828f
2025-04-18 08:04:47 +00:00
Jakob Ackermann
f0edc7ba00 [web] update the projects lastUpdated timestamp when changing file-tree (#24867)
* [misc] freeze time before any other unit test setup steps

Freezing it after other work (notably sandboxed-module imports) will
result in flaky tests.

* [web] update the projects lastUpdated timestamp when changing file-tree

GitOrigin-RevId: b82b2ff74dc31886f3c4bd300375117eead6e0cd
2025-04-16 08:05:14 +00:00
Tim Down
773cbc92eb Merge pull request #24592 from overleaf/td-socket-io-origins
Use updated socket.io with CORS origin check in real-time

GitOrigin-RevId: a1a874ba298e145aa4c9dd55ef723d79843f8145
2025-04-15 08:05:25 +00:00
Brian Gough
fe8d6392d5 Merge pull request #24793 from overleaf/bg-update-build-scripts-to-include-history-redis
update build scripts to include HISTORY_REDIS_HOST

GitOrigin-RevId: da0f317c80401067c0f4aa772196cb2f24849b8e
2025-04-11 08:06:03 +00:00
Jakob Ackermann
3f10b29869 [real-time] backwards compatibility fix for redis 6.2 (#24734)
GitOrigin-RevId: ffc51a8280e0d0708e7dcb2638cabed2b7adfbf5
2025-04-09 08:05:37 +00:00
Jakob Ackermann
ca111771c2 [real-time] rename metric label to align with common name (#24732)
GitOrigin-RevId: fd161f2345f0c30aa82395dbca673025698f13fe
2025-04-09 08:05:33 +00:00
Jakob Ackermann
dadf6f0ed4 [real-time] record metrics on number of connected clients per project (#24727)
Note: "connected" here means across all real-time pods.

- editing_session_mode, counter
  - mode=connect - a client connected
  - mode=disconnect - a client disconnect
  - mode=update - continuous editing
  - status=empty - all clients disconnected
  - status=single - a single client is connected
  - status=multi - multiple clients are connected

- project_not_empty_since histogram with buckets [0,1h,2h,1d,2d,1w,30d]
  - status=empty/single/multi as described above

GitOrigin-RevId: 1cc42e72bbb5aae754399bdbc3f8771642f35c22
2025-04-09 08:05:29 +00:00
Brian Gough
a68e96400b Merge pull request #24670 from overleaf/bg-remove-logging-of-update-errors
remove update parameter in applyOtUpdate error handling

GitOrigin-RevId: 46fa9d669327850f956154b20676317a7b13eb78
2025-04-08 08:06:46 +00:00
Jakob Ackermann
8d569815e6 [real-time] only log userId from session (#24281)
While we are at it, log projectId and transport.

GitOrigin-RevId: ac10220247947e0050f1374c81091c50c8cc0e85
2025-03-13 09:06:44 +00:00
Jakob Ackermann
6916e22b09 [real-time] log transport and clientId in both directions of serverPing (#24084)
GitOrigin-RevId: dabd8db6949498ef1a6252864cef94cd4329e3c0
2025-03-07 09:04:40 +00:00
Jakob Ackermann
dddb091d77 [real-time] retire legacy transports on the server-side as well (#23983)
GitOrigin-RevId: 27dad39e889a5df46943c4f393277be16ab8ffdd
2025-03-03 09:05:30 +00:00
Andrew Rumble
767591973c Merge pull request #23905 from overleaf/ar-mocha-11
[all] Bump mocha everywhere

GitOrigin-RevId: bf5587047f9ceb88872cbd356c2029717bb8ca9d
2025-02-28 09:05:40 +00:00
Antoine Clausse
fe03d8d7fd Add shellcheck to web, server-pro, server-ce (#23823)
* Update shellcheck files to include non ".sh" files having the sh shebang

* Add shellcheck to root Makefile

* `make shellcheck_fix`

* Add shellcheck to server-ce and server-pro

* Exclude SC1091 (Not following) in SP/CE

* Fix errors in SP

* Fix errors in CE

* Update root shellcheck to ignore all failing rules. We can later remove rules one by one.

* Add shellcheck to web

* Add shellcheck step for server-ce and server-pro in cloudbuild.yaml

* Revert "Add shellcheck to root Makefile"

This reverts commit e0fa56f2

* Revert "`make shellcheck_fix`"

This reverts commit eb179245c109a9e742a7fdeeb75a4bdd03963587.

* `make shellcheck_fix` in server-ce

* Fix: Use $(...) notation instead of legacy backticked

```
In init_scripts/100_set_docker_host_ipaddress.sh line 5:
echo "`route -n | awk '/UG[ \t]/{print $2}'` dockerhost" >> /etc/hosts
      ^-- SC2006 (style): Use $(...) notation instead of legacy backticks `...`.

Did you mean:
echo "$(route -n | awk '/UG[ \t]/{print $2}') dockerhost" >> /etc/hosts

For more information:
  https://www.shellcheck.net/wiki/SC2006 -- Use $(...) notation instead of le...
```

* `make shellcheck_fix` in web

* Manual corrections on autofix

* Update SHELLCHECK_FILES to Perl regex so it matches at start of file

GitOrigin-RevId: 815d53f319a3792efa22703beb75570de5977450
2025-02-26 09:05:02 +00:00
Jakob Ackermann
8138c76c1d [web] restrict external heartbeat/server ping to websocket transport (#23830)
Co-authored-by: Alf Eaton <alf.eaton@overleaf.com>
GitOrigin-RevId: c03a78e2c05cd380b705bd14652060980682cb39
2025-02-25 09:05:02 +00:00
Jakob Ackermann
c3f94358eb [real-time] ignore error from duplicate send action on socket.io socket (#23787)
GitOrigin-RevId: 992a39a5cff04bcc0814ac6a620ad350d6e1da14
2025-02-21 09:05:52 +00:00
Miguel Serrano
2e72aa8a47 Merge pull request #23652 from overleaf/msm-update-path-to-regexp
[monorepo] Update express to 4.1.2

GitOrigin-RevId: b3db75257e1edc1a25a1a657def352ecca926920
2025-02-21 09:05:26 +00:00
Jakob Ackermann
0a7b2004d2 [misc] silence logger when running tests (#22243)
* [misc] silence logger when running tests

* [misc] re-enable logging in some tests for scripts

* [misc] make it easy to turn on verbose logging for tests

```
LOG_LEVEL=debug make test_unit
LOG_LEVEL=debug make test_acceptance
```
GitOrigin-RevId: 219bc6d1f9cbdb89ddd7d94742920913ddde4514
2025-02-10 09:06:02 +00:00
Brian Gough
01ed0c10a0 Merge pull request #23317 from overleaf/bg-improve-real-time-ping-logs
Add connection details to real-time ping logs

GitOrigin-RevId: f993d71deb1a512295af6c7146a19b730742e33f
2025-02-04 09:04:48 +00:00
Jakob Ackermann
5404dcb93d [misc] update build scripts (#23316)
* [misc] update build scripts

* [misc] make shellcheck happy

GitOrigin-RevId: 343655fda54aa7e835dd666dcd4218d61c58fd09
2025-02-04 09:04:37 +00:00
Brian Gough
a6bfbad5bd Merge pull request #23246 from overleaf/bg-real-time-server-ping
add real-time ping-pong messages for debugging lost connections

GitOrigin-RevId: 28feb8c8c7c660a5a23ca394acd0f69ac828c5a3
2025-02-03 09:04:39 +00:00
Antoine Clausse
272108a213 Run bin/update_node 20.18.0 20.18.2 (#23074)
GitOrigin-RevId: 3f1ab14c4c36ea54408e0ce2c404a323bfe8d9a6
2025-01-24 09:06:05 +00:00
Antoine Clausse
d4a10c7b41 [web] Socket diagnostics updates (#22951)
* Increase threshold for "latency in red color"

* Fix online status in Chrome and Safari

* Add "Auto ping" checkbox

* Put `/socket-diagnostics` behind `AuthenticationController.requireLogin`

* Set logs to `logger.info` when debugging

* Add `publicId` and `clientId` to logs

* Fix disconnect logs when debugging

* Refresh UI every second. Display red "Ping Count" if unanswered for 3s

* Update services/web/frontend/js/features/socket-diagnostics/components/socket-diagnostics.tsx

Co-authored-by: Jakob Ackermann <jakob.ackermann@overleaf.com>

* Update services/web/frontend/js/features/socket-diagnostics/components/socket-diagnostics.tsx

Co-authored-by: Jakob Ackermann <jakob.ackermann@overleaf.com>

* `npm run format:fix`

---------

Co-authored-by: Jakob Ackermann <jakob.ackermann@overleaf.com>
GitOrigin-RevId: 9faf2abdac51fa4b87c67d8fe89c4125d01d826f
2025-01-21 09:05:01 +00:00
Antoine Clausse
7bef003c56 [real-time, web] Create a UI to test socket connection (#22907)
* Create a UI to test socket connection

* Add Clock Delta to the measurements

* Add colors to DiagnosticItem

* Update icon

* Add more info to the diagnostics screen

* Add logs in backend on debug messages, disconnections and connection

* Add last received ping info

* Reorder DiagnosticItems

* Remove "warning" text color (too light)

* Replace Phosphor icons by Material Icons

GitOrigin-RevId: 6a015b4928cd19849ff287cf254f671840ed44af
2025-01-17 09:05:49 +00:00
Domagoj Kriskovic
5a447aa349 [real-time] ensure tc is enabled when reviewer role is used (#22245)
* Support for adding reviewer role

* show reviewer in track changes user list

* added "review" in assertClientCanViewProject

* test if reviewer can read project

* added collaboratorsGetter tests

* eit toggle-track-changes when track changes changes

* [real-time] ensure tc is enabled when reviewer role is used

* use assertClientCanReviewProjectAndDoc, refactor _assertClientCanApplyUpdate

GitOrigin-RevId: 158bd1ff0d4b4977da950134f8ad8b3740855290
2024-12-10 09:05:18 +00:00
Domagoj Kriskovic
13e3a941f4 [real-time] make sure reviewer has permissions to read a project (#22317)
* added "review" in assertClientCanViewProject

* test if reviewer can read project

GitOrigin-RevId: 92a30f8b456bec9f0fc81a1361e454eecff838bd
2024-12-10 09:05:14 +00:00
Domagoj Kriskovic
511fe60a9c Revert "Support for adding reviewer role (#22137)" (#22293)
This reverts commit 54064a7f961fe06f188ab449cd469cdaaf01b20a.

GitOrigin-RevId: 1dabc635756aff465d2fe9aa0d18121b081df3b0
2024-12-04 09:05:12 +00:00
Domagoj Kriskovic
f676eca2b8 Support for adding reviewer role (#22137)
* Support for adding reviewer role

* show reviewer in track changes user list

* added "review" in assertClientCanViewProject

* test if reviewer can read project

* added collaboratorsGetter tests

* eit toggle-track-changes when track changes changes

* Support for changing privilege to reviewers for invited users (#22159)

* Add reviewer in change privilege level handler

* added reviewer translation

* added acceptance tests

* fix tests

* Set track changes state permissions for reviewer role (#22167)

* Add reviewer in change privilege level handler

* added reviewer translation

* added acceptance tests

* fix tests

* Set track changes state permissions for reviewer role

* added authorization helper tests

* added ensureUserCanReviewProjectContent middleware

* allow changing track changes only with write permissions

* removed canUserReviewProjectContent

* List projects where user is added as a reviewer (#22249)

* List projects where user is added as reviewer

* list projects in /user/projects

* fix tests

GitOrigin-RevId: 54064a7f961fe06f188ab449cd469cdaaf01b20a
2024-12-04 09:05:00 +00:00
Brian Gough
6672372828 remove sentry from backend services (#20752)
* remove sentry from backend services - no longer required

* Remove Sentry integration from logging manager

* Remove Sentry from clsi default settings

* Remove `initializeErrorReporting` in libraries/logger

* Remove `@sentry/node` from `libraries/logger`

---------

Co-authored-by: Antoine Clausse <antoine.clausse@overleaf.com>
GitOrigin-RevId: 8149a885f5258804b93ae39cde7b7333e992532a
2024-11-27 09:04:50 +00:00
Andrew Rumble
11213d85fe Merge pull request #21883 from overleaf/ar-disable-keep-alive-remaining-services
Disable keepAlive in all services

GitOrigin-RevId: 9a3bac37e3fb09ee64b05cfda300dfe8d8672aad
2024-11-15 09:05:52 +00:00
Andrew Rumble
73b17dd694 Merge pull request #21627 from overleaf/ar-node-20
[all] node 20 upgrade v2

GitOrigin-RevId: aed13a800894717a796062e3dd4050e045040258
2024-11-15 09:04:32 +00:00
Jakob Ackermann
805c4f8321 Merge pull request #21764 from overleaf/jpa-esm-build-scripts
[misc] prepare linting for ESM files in services

GitOrigin-RevId: 468983ebff40617158bac28514381a6aaf725893
2024-11-12 09:05:50 +00:00
Antoine Clausse
7f48c67512 Add prefer-node-protocol ESLint rule (#21532)
* Add `unicorn/prefer-node-protocol`

* Fix `unicorn/prefer-node-protocol` ESLint errors

* Run `npm run format:fix`

* Add sandboxed-module sourceTransformers in mocha setups

Fix `no such file or directory, open 'node:fs'` in `sandboxed-module`

* Remove `node:` in the SandboxedModule requires

* Fix new linting errors with `node:`

GitOrigin-RevId: 68f6e31e2191fcff4cb8058dd0a6914c14f59926
2024-11-11 09:04:51 +00:00
Andrew Rumble
57d12dfb17 Update to node 18.20.4
GitOrigin-RevId: 4b613ea3973cb7944c3d414ea91bcf367dbf50a6
2024-11-07 09:04:31 +00:00
Andrew Rumble
79d80cc266 Merge pull request #21462 from overleaf/revert-18828-ac-node-20
Revert "Upgrade Node from `18.20.2` to `20.17.0`"

GitOrigin-RevId: cc12c33b4588caa2959817a113547d20d437d6f4
2024-10-30 09:05:07 +00:00
Andrew Rumble
a480df8a89 Merge pull request #18828 from overleaf/ac-node-20
Upgrade Node from `18.20.2` to `20.17.0`

GitOrigin-RevId: 0ba9b0921fc246a97d3da0aad7d48635c0e24ae8
2024-10-30 09:04:38 +00:00
Liangjun Song
cfd0b39b96 Merge pull request #20557 from overleaf/ls-bump-express
update express and other dependencies

GitOrigin-RevId: 22107eab0fa95d534ab07360548ae2b2123776db
2024-09-27 08:04:18 +00:00
Jakob Ackermann
e3feafb7b0 Merge pull request #20332 from overleaf/jpa-small-runner
[misc] CI improvements

GitOrigin-RevId: 819d4d357fcb130f133337668644eedcf5f4a688
2024-09-26 08:04:38 +00:00
Alf Eaton
2976fd0b44 Fix Dockerfile FromAsCasing warnings (#20388)
GitOrigin-RevId: 6d6be3ba17eec695907dde25da2dbe98818fae10
2024-09-23 08:04:52 +00:00