Commit graph

251 commits

Author SHA1 Message Date
yu-i-i
3461c105f4 Fix: always include repository prefix in sandboxed compiles image name 2025-07-22 19:22:12 +02:00
yu-i-i
3b3fc01308 Allow selecting a TeX Live image for a project 2025-07-22 19:22:06 +02:00
Jakob Ackermann
8c39add865 [clsi-cache] meter ingress and egress bandwidth (#27143)
* [mics] fix "app" label in clsi-cache metrics in dev-env

* [clsi-cache] validate filePath when processing file

* [clsi-cache] meter ingress and egress bandwidth

Files are downloaded directly from nginx, hence we cannot meter egress
in clsi-cache easily.

GitOrigin-RevId: 24de8c41728f0e9c984113c1470dec6153e75f20
2025-07-16 08:05:59 +00:00
Brian Gough
2f2862ecd7 Merge pull request #26637 from overleaf/bg-clsi-fix-process-group-for-local-compiles
fix "stop compile" option for local command runner in CE/SP

GitOrigin-RevId: 7986b505362aaf33ac6e161b3b54458baba1e2e6
2025-07-11 08:07:11 +00:00
Jakob Ackermann
26a7a7d7b8 [clsi] mark VM as unhealthy when detecting of-of-disk condition (#25721)
* [clsi] shed load when detecting out-of-disk condition

* [clsi] mark VM as unhealthy when detecting of-of-disk condition

GitOrigin-RevId: 25cda6785c0d973f50ec6206bee389804f35917e
2025-05-21 08:05:34 +00:00
Jakob Ackermann
ec1bd69605 [clsi-cache] remove non sharded instances (#25645)
* Revert "[clsi-cache] only use sharding from updated project editor tabs (#25326)"

This reverts commit 1754276bed3186c0536055c983e32476cc90d416.

* [clsi-cache] remove non sharded instances

GitOrigin-RevId: aa3ac46140dfc1722a3350cf7071e5b11af61199
2025-05-16 08:05:02 +00:00
Jakob Ackermann
fba8f776a1 [web] avoid trying to fetch synctex.gz from clsi-cache in free projects (#25445)
* [web] avoid trying to fetch synctex.gz from clsi-cache in free projects

* [clsi] parse boolean query parameter

GitOrigin-RevId: 99c98aac8147a626b704e9a888b7fc660cc5ab17
2025-05-12 08:05:24 +00:00
Jakob Ackermann
d489e35782 [web] emit event when synctex mapping was downloaded from clsi-cache (#25424)
* [clsi] tell frontend when synctex mapping was downloaded from clsi-cache

* [web] emit event when synctex mapping was downloaded from clsi-cache

GitOrigin-RevId: 1f6b7e0faaa7dd76449aad566802da971a4cf9ed
2025-05-09 08:06:00 +00:00
Jakob Ackermann
6881ba956a [clsi-cache] only use sharding from updated project editor tabs (#25326)
GitOrigin-RevId: 1754276bed3186c0536055c983e32476cc90d416
2025-05-07 08:06:39 +00:00
Jakob Ackermann
5ce1685b5b [clsi-cache] shard each zone into three instances (#25301)
* [clsi-cache] shard per zone into three instances

Keep the old instance as read fallback. We can remove it in 4 days.

Disk size: 2Ti gives us the maximum write throughput of 240MiB/s on a
N2D instance with fewer than 8 vCPUs.

* [clsi] fix format

* [k8s] clsi-cache: bring back storage-classes

* [k8s] clsi-cache: fix reference to zonal storage-classes

* [k8s] clsi-cache: add logging configs

* [clsi] improve sharding

Co-authored-by: Brian Gough <brian.gough@overleaf.com>

* [clsi] fix sharding

Index needs to be positive.

* [clsi] fix sharding

The random part is static per machine/process.

* [clsi] restrict clsi-cache to user projects

Co-authored-by: Brian Gough <brian.gough@overleaf.com>

* [k8s] clsi-cache: align CLSI_CACHE_NGINX_HOST with service LB

---------

Co-authored-by: Brian Gough <brian.gough@overleaf.com>
GitOrigin-RevId: 1efb1b3245c8194c305420b25e774ea735251fb3
2025-05-07 08:06:16 +00:00
Antoine Clausse
fa62529d82 [clsi] Replace diskusage by fs (#24789)
* Replace `diskusage` by `fs` in clsi

* Replace `diskusage` by `fs` in clsi-cache

* Update disk space calculations to include block size in bytes

Co-authored-by: Jakob Ackermann <jakob.ackermann@overleaf.com>

* Add warning comments about Docker-for-Mac fs stats being off by a factor

---------

Co-authored-by: Jakob Ackermann <jakob.ackermann@overleaf.com>
GitOrigin-RevId: 02ea07e531b89bb3d10ddfe780348b19cbddad1f
2025-04-17 08:06:16 +00:00
Jakob Ackermann
39110d9da9 [clsi-cache] check compiler settings before using compile from cache (#24845)
* [web] provide an actual rootFolder from EditorProviders in tests

- Fixup SocketIOMock and ShareJS mocks to provide the complete interface
- Extend SocketIOMock interface to count event listeners
- Fixup test that did not expect to find a working rootDoc

* [web] expose imageName from ProjectContext

* [clsi-cache] check compiler settings before using compile from cache

* [web] avoid fetching initial compile from clsi-cache in PDF detach tab

GitOrigin-RevId: e3c754a7ceca55f03a317e1bc8ae45ed12cc2f02
2025-04-16 08:05:35 +00:00
Jakob Ackermann
4a17a1e713 [web] gracefully access compile stats for event (#24818)
* [web] gracefully access compile stats for event

* [clsi] always emit stats and timings

GitOrigin-RevId: 959e5fe1508245ffecfab1219fd86e53b210fca1
2025-04-14 08:04:51 +00:00
Jakob Ackermann
d99ba08d01 [clsi] run SyncTeX in specific output dir rather than compile dir (#24690)
* [clsi] drop support for docker-in-docker

* [clsi] run SyncTeX in specific output dir rather than compile dir

* [clsi] store output.synctex.gz outside of tar-ball in clsi-cache

* [clsi] add documentation for rewriting of docker bind-mounts

* [server-pro] update env vars for sandboxed compiles in sample config

GitOrigin-RevId: 8debd7102ac612544961f237aa4ff1c530aa3da3
2025-04-10 08:05:26 +00:00
Jakob Ackermann
b538d56591 [clsi-cache] backend (#24388)
* [clsi-cache] initial revision of the clsi-cache service

* [clsi] send output files to clsi-cache and import from clsi-cache

* [web] pass editorId to clsi

* [web] clear clsi-cache when clearing clsi cache

* [web] add split-tests for controlling clsi-cache rollout

* [web] populate clsi-cache when cloning/creating project from template

* [clsi-cache] produce less noise when populating cache hits 404

* [clsi-cache] push docker image to AR

* [clsi-cache] push docker image to AR

* [clsi-cache] allow compileGroup in job payload

* [clsi-cache] set X-Zone header from latest endpoint

* [clsi-cache] use method POST for /enqueue endpoint

* [web] populate clsi-cache in zone b with template data

* [clsi-cache] limit number of editors per project/user folder to 10

* [web] clone: populate the clsi-cache unless the TeXLive release changed

* [clsi-cache] keep user folder when clearing cache as anonymous user

* [clsi] download old output.tar.gz when synctex finds empty compile dir

* [web] fix lint

* [clsi-cache] multi-zonal lookup of single build output

* [clsi-cache] add more validation and limits

Co-authored-by: Brian Gough <brian.gough@overleaf.com>

* [clsi] do not include clsi-cache tar-ball in output.zip

* [clsi-cache] fix reference after remaining constant

Co-authored-by: Alf Eaton <alf.eaton@overleaf.com>

* [web] consolidate validation of filename into ClsiCacheHandler

* [clsi-cache] extend metrics and event tracking

- break down most of the clsi metrics by label
  - compile=initial - new compile dir without previous output files
  - compile=recompile - recompile in existing compile dir
  - compile=from-cache - compile using previous clsi-cache
- extend segmentation on compile-result-backend event
  - isInitialCompile=true - found new compile dir at start of request
  - restoredClsiCache=true - restored compile dir from clsi-cache

* [clsi] rename metrics labels for download of clsi-cache

This is in preparation for synctex changes.

* [clsi] use constant for limit of entries in output.tar.gz

Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>

* [clsi-cache] fix cloning of project cache

---------

Co-authored-by: Brian Gough <brian.gough@overleaf.com>
Co-authored-by: Alf Eaton <alf.eaton@overleaf.com>
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
GitOrigin-RevId: 4901a65497af13be1549af7f38ceee3188fcf881
2025-04-10 08:05:17 +00:00
Jakob Ackermann
54f5c3115c [web] convert file-list to typescript (#24354)
* [web] convert file-list to typescript

* [web] add type annotation for clsiServerId without providing default

Co-authored-by: David Powell <david.powell@overleaf.com>

---------

Co-authored-by: David Powell <david.powell@overleaf.com>
GitOrigin-RevId: 5ecb79c2540a3e46e296c6bf7f8573fb65febc3f
2025-03-24 10:46:48 +00:00
Jakob Ackermann
1117ea1b3e [clsi] add helper function for detecting extraneous files (#24376)
* [clsi] add helper function for detecting extraneous files

* [clsi] output.pdfxref and output.synctex.gz are extraneous as well

* [clsi] minor code cleanup

Co-authored-by: Mathias Jakobsen <mathias.jakobsen@overleaf.com>

---------

Co-authored-by: Mathias Jakobsen <mathias.jakobsen@overleaf.com>
GitOrigin-RevId: 0d6c96c99b6be3adaee1558b1f958da885e6307a
2025-03-24 10:46:43 +00:00
Jakob Ackermann
13bf214a3c [web] generate clsi buildId ahead of fetching project content (#24337)
* [web] generate clsi buildId ahead of fetching project content

The buildIds timestamp component will be used for cache invalidation.

* [clsi] strict validation for buildId

* [clsi] validate buildId parameter

GitOrigin-RevId: 88e8b2d48e78fa137b6dca7f2e6b93bbcf88a777
2025-03-24 10:46:02 +00:00
Jakob Ackermann
5cdaa424ee [clsi] increase CACHE_AGE for output files to 90 minutes (#24322)
GitOrigin-RevId: d6208281be8d9a97278f7c60e203de68eb3f7501
2025-03-17 09:04:53 +00:00
Jakob Ackermann
0c3a62297a [clsi] collect disk usage more frequently (#24314)
GitOrigin-RevId: 0c5b3ebeadec6d2ac70b7adc77935aa0cea92ffc
2025-03-17 09:04:49 +00:00
Jakob Ackermann
f7e716c826 [clsi] add metric for disk usage (#24303)
GitOrigin-RevId: e21b867a2fdaf54e9ec5b29b0f80b29349eb901c
2025-03-14 09:05:23 +00:00
Jakob Ackermann
d8b0ab9436 [clsi] avoid downloads from compile directory (#23975)
* [clsi] make error copying output files a fatal compile error

Co-authored-by: Rebeka <rebeka.dekany@overleaf.com>

* [clsi] remove unused endpoints for downloading files from compileDir

Co-authored-by: Rebeka <rebeka.dekany@overleaf.com>

* [clsi] avoid useless clone of output files

* [clsi] add test for output files when compile did not produce a PDF

---------

Co-authored-by: Rebeka <rebeka.dekany@overleaf.com>
GitOrigin-RevId: cb998b99b4d96cb48ddd70987958f614ad3b40fc
2025-03-03 09:05:04 +00:00
Tim Down
4e353956db Merge pull request #17378 from overleaf/td-stream-buffer-consolidation
Stream and buffer consolidation

GitOrigin-RevId: 284f411e6403e190d2dda3d9ebf806b1935b3949
2025-02-25 09:05:26 +00:00
Jakob Ackermann
d19c5e236f Merge pull request #22208 from overleaf/jpa-clsi-hash
[misc] clsi: read files from history-v1 with fallback to filestore

GitOrigin-RevId: c54bb128780198c14e7a63818f39fad62ce65d4e
2024-11-29 09:05:39 +00:00
Antoine Clausse
7f48c67512 Add prefer-node-protocol ESLint rule (#21532)
* Add `unicorn/prefer-node-protocol`

* Fix `unicorn/prefer-node-protocol` ESLint errors

* Run `npm run format:fix`

* Add sandboxed-module sourceTransformers in mocha setups

Fix `no such file or directory, open 'node:fs'` in `sandboxed-module`

* Remove `node:` in the SandboxedModule requires

* Fix new linting errors with `node:`

GitOrigin-RevId: 68f6e31e2191fcff4cb8058dd0a6914c14f59926
2024-11-11 09:04:51 +00:00
andrew rumble
bb12dd33ee Capture more logging information for output.zip errors
GitOrigin-RevId: a2f15d05100dfdba06778a1a795c8eabefd2666d
2024-09-24 08:05:33 +00:00
andrew rumble
d4911ea246 Stop waiting for finalize
Finalize promise will only resolve when the archive is closed. We want
to stream as soon as we have the data so this does not suit us. We want
to log errors that are thrown due to finalize, these should be
propogated by archiver to the response already.

GitOrigin-RevId: 4f9d727a00ead1be3caee62e1e0adba069a545c7
2024-09-24 08:05:28 +00:00
andrew rumble
d4824b9c5f Get full path only when adding file to archive
This allows us to:
* Get files that are in subdirectories (not known to happen but might as
well support)
* Removes need to sanitise missing_files output

GitOrigin-RevId: 4b27b7f1b506578a6d713ea23979aa9370dfdcb0
2024-09-24 08:05:24 +00:00
andrew rumble
60a75d12b0 Rename files to reflect refactoring elsewhere
GitOrigin-RevId: 335f876913b7868cf69dcd55dd156261edd98945
2024-09-24 08:05:19 +00:00
andrew rumble
487d9125a2 Improve stream error safety
GitOrigin-RevId: de4c512a62d304b3eeb2a1521aac173fa93d8411
2024-09-24 08:05:15 +00:00
andrew rumble
1409e32010 Move logging into ArchiveManager
GitOrigin-RevId: 71a28a0215c5f1a6975c9e2fcdcd476513df1cbc
2024-09-24 08:05:10 +00:00
andrew rumble
c387e60a28 Remove unnecessary symlink check
GitOrigin-RevId: 08d7295403a258818276b9fbd7666a20fbc2e00f
2024-09-24 08:05:06 +00:00
andrew rumble
3fa35ef03a Share ignore logic between web and clsi
GitOrigin-RevId: c4ae306413efb71989d2db410c3859cb69b3d5de
2024-09-24 08:05:01 +00:00
andrew rumble
d627a0bee9 Remove files param from output.zip endpoint
GitOrigin-RevId: 300cd870578237ad8e61be4e9e41991f884aa754
2024-09-24 08:04:57 +00:00
andrew rumble
c764566148 Allow all files to be in zip (in same directory)
GitOrigin-RevId: 14645a0c3db88faf00e2718b9574b5892ac3efcb
2024-09-24 08:04:53 +00:00
Liangjun Song
2133dde8bf remove dry run (#19820)
GitOrigin-RevId: b92e08da6654cdd37314f7c52a6946cc7ec8983a
2024-08-08 08:04:17 +00:00
Liangjun Song
5d472e9b38 limit the number of concurrent compile requests in clsi (#19717)
GitOrigin-RevId: 17909a4dd0717ea4a75288f734ddef19c7d6592e
2024-08-06 08:04:59 +00:00
Liangjun Song
e2d87f4821 Revert "Revert "log clsi concurrency (#19651)" (#19655)" (#19656)
This reverts commit 3719f468a6336ff4bf5126f3e5d79dcde172b7ea.

GitOrigin-RevId: 6f4809f3c607c62892fc1e110eeaa2a1e7a124a4
2024-07-31 08:04:28 +00:00
Liangjun Song
39112dafb7 Revert "log clsi concurrency (#19651)" (#19655)
This reverts commit 504f6f9cc3d4b7386dbcde86f76c1cf3a13e944f.

GitOrigin-RevId: 3719f468a6336ff4bf5126f3e5d79dcde172b7ea
2024-07-30 08:05:52 +00:00
Liangjun Song
01b7896717 log clsi concurrency (#19651)
GitOrigin-RevId: 504f6f9cc3d4b7386dbcde86f76c1cf3a13e944f
2024-07-30 08:05:26 +00:00
Jakob Ackermann
9f68bc5660 Merge pull request #19296 from overleaf/jpa-issue-19290-3
[clsi] atomic writing of LaTeXMk output

GitOrigin-RevId: d81c497370587b98fc7ad282035cd59b0ae09ec8
2024-07-15 09:01:04 +00:00
Jakob Ackermann
51a24601ec Merge pull request #19293 from overleaf/jpa-issue-19290-2
[clsi] fix parsing of the requested file in symlink validation

GitOrigin-RevId: 86cfe8d62bb99ed6844faee0ff4af507e571e04d
2024-07-15 09:00:59 +00:00
Andrew Rumble
80ede301fa Merge pull request #18474 from overleaf/ar-return-build-id-from-clsi-after-compile
[clsi] Return buildId after compiles

GitOrigin-RevId: 872048f4fea8f5a00b817e29bd26a444d179a45f
2024-05-27 10:24:06 +00:00
Andrew Rumble
71187a51ba Merge pull request #18289 from overleaf/ac-ar-eslint-return-await
Add ESLint rule @typescript-eslint/return-await to backend services

GitOrigin-RevId: 75e3e32597827fcc852e69d479515fc72e8f45e4
2024-05-27 10:22:49 +00:00
Christopher Hoskin
3342d672c2 Merge pull request #18397 from overleaf/em-revert-download-all-link
Revert "Merge pull request #18190 from overleaf/ar-add-download-all-l…

GitOrigin-RevId: 681eb2734636d76558e682dc85083bfcaa6b7d2d
2024-05-17 08:05:10 +00:00
Brian Gough
c1a4e4a873 Merge pull request #16775 from overleaf/bg-clsi-timeout-use-fetchutils-connect-timeout
Update UrlFetcher to use custom agent with connect timeout

GitOrigin-RevId: c92dc27a0e682b0b79859b3521bf64cbf8485345
2024-05-16 08:05:18 +00:00
Andrew Rumble
105d67bd04 Merge pull request #18190 from overleaf/ar-add-download-all-link
[web] Add download all link for output files

GitOrigin-RevId: a275f17b6acf3130250d91462ba01588b93b62f5
2024-05-16 08:05:04 +00:00
Andrew Rumble
92f62f91c1 Merge pull request #18148 from overleaf/ar-add-output-zip-endpoint-to-clsi
[clsi] Add endpoints to get zip of output files

GitOrigin-RevId: a1a935e8170ab5a8d40baa6d96f8e42fe22c2e8c
2024-05-02 08:03:44 +00:00
Jakob Ackermann
9351c03c3b Merge pull request #17874 from overleaf/jpa-ol-project-id-in-env
[clsi] expose Overleaf project id to LaTeX compilation via env var

GitOrigin-RevId: cf8be6f8d83351a587d9ac7940f61a046a14035e
2024-04-12 08:04:56 +00:00
Alf Eaton
6cc2db3cdd Merge pull request #17525 from overleaf/ae-upgrade-prettier
Upgrade Prettier to v3

GitOrigin-RevId: 6f1338f196408f3edb4892d5220ad3665ff1a5bc
2024-03-26 09:04:05 +00:00