* [clsi] tell frontend when synctex mapping was downloaded from clsi-cache
* [web] emit event when synctex mapping was downloaded from clsi-cache
GitOrigin-RevId: 1f6b7e0faaa7dd76449aad566802da971a4cf9ed
* [clsi-cache] shard per zone into three instances
Keep the old instance as read fallback. We can remove it in 4 days.
Disk size: 2Ti gives us the maximum write throughput of 240MiB/s on a
N2D instance with fewer than 8 vCPUs.
* [clsi] fix format
* [k8s] clsi-cache: bring back storage-classes
* [k8s] clsi-cache: fix reference to zonal storage-classes
* [k8s] clsi-cache: add logging configs
* [clsi] improve sharding
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
* [clsi] fix sharding
Index needs to be positive.
* [clsi] fix sharding
The random part is static per machine/process.
* [clsi] restrict clsi-cache to user projects
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
* [k8s] clsi-cache: align CLSI_CACHE_NGINX_HOST with service LB
---------
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
GitOrigin-RevId: 1efb1b3245c8194c305420b25e774ea735251fb3
* Replace `diskusage` by `fs` in clsi
* Replace `diskusage` by `fs` in clsi-cache
* Update disk space calculations to include block size in bytes
Co-authored-by: Jakob Ackermann <jakob.ackermann@overleaf.com>
* Add warning comments about Docker-for-Mac fs stats being off by a factor
---------
Co-authored-by: Jakob Ackermann <jakob.ackermann@overleaf.com>
GitOrigin-RevId: 02ea07e531b89bb3d10ddfe780348b19cbddad1f
* [clsi] drop support for docker-in-docker
* [clsi] run SyncTeX in specific output dir rather than compile dir
* [clsi] store output.synctex.gz outside of tar-ball in clsi-cache
* [clsi] add documentation for rewriting of docker bind-mounts
* [server-pro] update env vars for sandboxed compiles in sample config
GitOrigin-RevId: 8debd7102ac612544961f237aa4ff1c530aa3da3
* [clsi-cache] initial revision of the clsi-cache service
* [clsi] send output files to clsi-cache and import from clsi-cache
* [web] pass editorId to clsi
* [web] clear clsi-cache when clearing clsi cache
* [web] add split-tests for controlling clsi-cache rollout
* [web] populate clsi-cache when cloning/creating project from template
* [clsi-cache] produce less noise when populating cache hits 404
* [clsi-cache] push docker image to AR
* [clsi-cache] push docker image to AR
* [clsi-cache] allow compileGroup in job payload
* [clsi-cache] set X-Zone header from latest endpoint
* [clsi-cache] use method POST for /enqueue endpoint
* [web] populate clsi-cache in zone b with template data
* [clsi-cache] limit number of editors per project/user folder to 10
* [web] clone: populate the clsi-cache unless the TeXLive release changed
* [clsi-cache] keep user folder when clearing cache as anonymous user
* [clsi] download old output.tar.gz when synctex finds empty compile dir
* [web] fix lint
* [clsi-cache] multi-zonal lookup of single build output
* [clsi-cache] add more validation and limits
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
* [clsi] do not include clsi-cache tar-ball in output.zip
* [clsi-cache] fix reference after remaining constant
Co-authored-by: Alf Eaton <alf.eaton@overleaf.com>
* [web] consolidate validation of filename into ClsiCacheHandler
* [clsi-cache] extend metrics and event tracking
- break down most of the clsi metrics by label
- compile=initial - new compile dir without previous output files
- compile=recompile - recompile in existing compile dir
- compile=from-cache - compile using previous clsi-cache
- extend segmentation on compile-result-backend event
- isInitialCompile=true - found new compile dir at start of request
- restoredClsiCache=true - restored compile dir from clsi-cache
* [clsi] rename metrics labels for download of clsi-cache
This is in preparation for synctex changes.
* [clsi] use constant for limit of entries in output.tar.gz
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
* [clsi-cache] fix cloning of project cache
---------
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
Co-authored-by: Alf Eaton <alf.eaton@overleaf.com>
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
GitOrigin-RevId: 4901a65497af13be1549af7f38ceee3188fcf881
* [clsi] add helper function for detecting extraneous files
* [clsi] output.pdfxref and output.synctex.gz are extraneous as well
* [clsi] minor code cleanup
Co-authored-by: Mathias Jakobsen <mathias.jakobsen@overleaf.com>
---------
Co-authored-by: Mathias Jakobsen <mathias.jakobsen@overleaf.com>
GitOrigin-RevId: 0d6c96c99b6be3adaee1558b1f958da885e6307a
* [web] generate clsi buildId ahead of fetching project content
The buildIds timestamp component will be used for cache invalidation.
* [clsi] strict validation for buildId
* [clsi] validate buildId parameter
GitOrigin-RevId: 88e8b2d48e78fa137b6dca7f2e6b93bbcf88a777
* [clsi] make error copying output files a fatal compile error
Co-authored-by: Rebeka <rebeka.dekany@overleaf.com>
* [clsi] remove unused endpoints for downloading files from compileDir
Co-authored-by: Rebeka <rebeka.dekany@overleaf.com>
* [clsi] avoid useless clone of output files
* [clsi] add test for output files when compile did not produce a PDF
---------
Co-authored-by: Rebeka <rebeka.dekany@overleaf.com>
GitOrigin-RevId: cb998b99b4d96cb48ddd70987958f614ad3b40fc
* [misc] silence logger when running tests
* [misc] re-enable logging in some tests for scripts
* [misc] make it easy to turn on verbose logging for tests
```
LOG_LEVEL=debug make test_unit
LOG_LEVEL=debug make test_acceptance
```
GitOrigin-RevId: 219bc6d1f9cbdb89ddd7d94742920913ddde4514
* Add `unicorn/prefer-node-protocol`
* Fix `unicorn/prefer-node-protocol` ESLint errors
* Run `npm run format:fix`
* Add sandboxed-module sourceTransformers in mocha setups
Fix `no such file or directory, open 'node:fs'` in `sandboxed-module`
* Remove `node:` in the SandboxedModule requires
* Fix new linting errors with `node:`
GitOrigin-RevId: 68f6e31e2191fcff4cb8058dd0a6914c14f59926
Finalize promise will only resolve when the archive is closed. We want
to stream as soon as we have the data so this does not suit us. We want
to log errors that are thrown due to finalize, these should be
propogated by archiver to the response already.
GitOrigin-RevId: 4f9d727a00ead1be3caee62e1e0adba069a545c7
