Commit graph

204 commits

Author SHA1 Message Date
Jakob Ackermann
ce0d5fd383 Merge pull request #22177 from overleaf/jpa-file-view-hash-1
[web] migrate file-view to download from history-v1 (via web) 1/2

GitOrigin-RevId: b787e90c57af5e2704b06ba63502aa6fc09ea1df
2024-11-28 09:06:33 +00:00
Brian Gough
be90a3b2bb Merge pull request #22170 from overleaf/bg-history-v1-copy-blob
add copyBlob support to history-v1

GitOrigin-RevId: 797ea66c37ca938fc906c4dff7bb1c8bf14c031e
2024-11-28 09:05:30 +00:00
Jakob Ackermann
3d7254b419 Merge pull request #22153 from overleaf/jpa-backup-verifier-minimal
[history-v1] backup-verifier-app: initial revision

GitOrigin-RevId: 922c9f94cb7ca7c129e38fd6961d42bdff819cd8
2024-11-27 09:04:55 +00:00
Jakob Ackermann
40603e0561 Merge pull request #22134 from overleaf/jpa-broken-file-trees
[history-v1] back_fill_file_hash: gracefully handle broken file-trees

GitOrigin-RevId: 463c785e98581364b107f3262951e7fa0fb88a0f
2024-11-26 09:05:51 +00:00
Jakob Ackermann
b7d37b434a Merge pull request #22123 from overleaf/jpa-process-hashed
[history-v1] back_fill_file_hash: optionally process hashed files

GitOrigin-RevId: 53ebaa7b03166a6b1aacc0f985bb295c9ec04804
2024-11-26 09:05:47 +00:00
Jakob Ackermann
0bd8729cc1 Merge pull request #22115 from overleaf/jpa-skip-get
[history-v1] back_fill_file_hash: optimize obtaining the DEK

GitOrigin-RevId: 6fc5218737551ae4e6152414c32560f545767091
2024-11-26 09:04:50 +00:00
Jakob Ackermann
ed27af11f8 Merge pull request #22107 from overleaf/jpa-error-handling
[history-v1] back_fill_file_hash: error handling

GitOrigin-RevId: 2dbf52d015206cf42ca434b1f16221d6e0f2cc2b
2024-11-26 09:04:45 +00:00
Jakob Ackermann
1a6bd670d4 Merge pull request #22104 from overleaf/jpa-graceful-shutdown-abort-retries
[history-v1] back_fill_file_hash: abort retries during graceful shutdown

GitOrigin-RevId: 0d52dde32e4a7bf2293f341644b7416e669b2bdc
2024-11-25 09:05:31 +00:00
Jakob Ackermann
838ae23b52 Merge pull request #22091 from overleaf/jpa-bad-file-trees
[history-v1] back_fill_file_hash: gracefully handle bad file-trees

GitOrigin-RevId: 0419c06b1ccf827f4c6c5170978a38659435f26f
2024-11-25 09:05:02 +00:00
Jakob Ackermann
c2b876372b Merge pull request #22070 from overleaf/jpa-args
[history-v1] refactor env vars to args for GNU-parallel usage

GitOrigin-RevId: 1ebfa3dfc25d36f2f86c22fa22e4864d55b511b2
2024-11-22 09:06:11 +00:00
Jakob Ackermann
69021ca4b0 Merge pull request #22061 from overleaf/jpa-reduce-idle-time
[history-v1] back_fill_file_hash: reduce idle time between batches

GitOrigin-RevId: 9b24bb882b158f33915d2e0ab2f82273eac09739
2024-11-22 09:05:55 +00:00
Jakob Ackermann
4f25b0de91 Merge pull request #22028 from overleaf/jpa-logging
[misc] logging tweaks

GitOrigin-RevId: 7757b81b6c684955374038376035dc2bd272c4e8
2024-11-21 09:05:41 +00:00
Jakob Ackermann
44e0428496 Merge pull request #22019 from overleaf/jpa-cleanup
[history-v1] back_fill_file_hash: cleanup BUFFER_DIR when done

GitOrigin-RevId: 6413292bbe8cca470919b740d7e2b45fc5e7d824
2024-11-21 09:05:05 +00:00
Jakob Ackermann
2e630e50dc Merge pull request #22017 from overleaf/jpa-process-all-projects
[history-v1] back_fill_file_hash: process all projects

GitOrigin-RevId: 41c3cd59022bbac09552684ef2a99c58f2235ac3
2024-11-21 09:04:50 +00:00
Jakob Ackermann
6de1817ef5 Merge pull request #22013 from overleaf/jpa-flush-mongo-queues-early
[history-v1] back_fill_file_hash: flush mongo write queues early

GitOrigin-RevId: 9b6c6ff9861945e69e42d15dc19f4c03c39193e9
2024-11-21 09:04:42 +00:00
Jakob Ackermann
73aea01f37 Merge pull request #21996 from overleaf/jpa-stream-pg-result
[history-v1] postgres: getProjectBlobsBatch: stream records

GitOrigin-RevId: 94ed6dfc4a263fd9369cd380e6cc25c7bbf6decc
2024-11-21 09:04:38 +00:00
Jakob Ackermann
0253130c36 Merge pull request #21972 from overleaf/jpa-get-project-blobs-batch
[history-v1] implement getProjectBlobsBatch

GitOrigin-RevId: f03dcc690ef63f72400ccf001c6e497bd4fbe790
2024-11-20 09:05:34 +00:00
Jakob Ackermann
4964d6414b Merge pull request #21982 from overleaf/jpa-cleanup
[history-v1] back_fill_file_hash: cleanup fs and graceful shutdown

GitOrigin-RevId: 362669ff988ad71fda713bf4896b1abcb36caf35
2024-11-20 09:05:30 +00:00
Jakob Ackermann
24f2388aa2 Merge pull request #21948 from overleaf/bg-jpa-back-fill-project-blobs
[history-v1] back_fill_file_hash: process blobs

GitOrigin-RevId: e54d0f8ab537ce43a12f9c972ba2ee82836073c8
2024-11-20 09:05:04 +00:00
Jakob Ackermann
fb36fff63d Merge pull request #21931 from overleaf/bg-get-all-blobs-for-project
add getProjectBlobs method to retrieve metadata for all blobs in a project

GitOrigin-RevId: 38f504a4fb56cd8ef8beaff1d8917ead26e85f5a
2024-11-20 09:04:56 +00:00
Jakob Ackermann
8e74d3c58c Merge pull request #21947 from overleaf/bg-jpa-back-fill-script-tweaks
[history-v1] back_fill_file_hash: performance tweaks

GitOrigin-RevId: c3d0c7906707fc902addcde64eaf41c24ceeece7
2024-11-20 09:04:51 +00:00
Jakob Ackermann
27076c50cc Merge pull request #21670 from overleaf/jpa-mongo-backend-types
[history-v1] add types to mongo BlobStore backend

GitOrigin-RevId: 7d91074eaa781904f7f3b56390aacee1800a7f67
2024-11-19 09:05:23 +00:00
Jakob Ackermann
ca0a46b5bb Merge pull request #21928 from overleaf/jpa-handle-already-hard-deleted
[history-v1] backup-deletion-app: use deletedProjectOverleafHistoryId

GitOrigin-RevId: 169ba0fba71c42b0415e5fa40424547b054dd5b0
2024-11-18 09:06:13 +00:00
Jakob Ackermann
dd15ade2b9 Merge pull request #21880 from overleaf/jpa-history-store
[history-v1] make HistoryStore generic and add backupHistoryStore

GitOrigin-RevId: 65d275de182dbcf5d4b6bf3c610b71b58db68e70
2024-11-18 09:05:48 +00:00
Jakob Ackermann
9cc6f2a9d5 Merge pull request #21829 from overleaf/jpa-backup-deletion
[history-v1] backup-deletion-app: initial revision

GitOrigin-RevId: 9e05c0e16595cb8d180861ef503c7b4522110d39
2024-11-18 09:05:44 +00:00
Jakob Ackermann
97b39ef98f Merge pull request #21681 from overleaf/jpa-back-fill-file-hash
[history-v1] add script for back filling hash into project file-tree

GitOrigin-RevId: 70ab8c72f3eea1062708e4471b4ad1c60317ad26
2024-11-18 09:05:39 +00:00
Antoine Clausse
7f48c67512 Add prefer-node-protocol ESLint rule (#21532)
* Add `unicorn/prefer-node-protocol`

* Fix `unicorn/prefer-node-protocol` ESLint errors

* Run `npm run format:fix`

* Add sandboxed-module sourceTransformers in mocha setups

Fix `no such file or directory, open 'node:fs'` in `sandboxed-module`

* Remove `node:` in the SandboxedModule requires

* Fix new linting errors with `node:`

GitOrigin-RevId: 68f6e31e2191fcff4cb8058dd0a6914c14f59926
2024-11-11 09:04:51 +00:00
Jakob Ackermann
a3d8caf87b Merge pull request #21664 from overleaf/jpa-blob-hash
[overleaf-editor-core] stricter types for Blob interface

GitOrigin-RevId: 8595fce0d5c98074d2313be5a5634e80f92c68b5
2024-11-08 09:07:05 +00:00
Brian Gough
37806a9505 Merge pull request #20695 from overleaf/bg-issue15972
Await knex.transaction for deleteProjectChunks in history-v1

GitOrigin-RevId: fcf342c5d1e0e9beed35c75d0317f8cba5976e24
2024-10-01 08:04:52 +00:00
Eric Mc Sween
250acbf33c Merge pull request #20520 from overleaf/em-ts-import
Use new import JSDoc syntax for Typescript annotations

GitOrigin-RevId: 782456d637fc8f2de6163b5d70fabf06c1d74964
2024-09-23 08:04:47 +00:00
Jakob Ackermann
3836323724 Merge pull request #19817 from overleaf/jpa-types
[overleaf-editor-core] stronger type checking via web

GitOrigin-RevId: 427019f40e2905f2e0ec11dc09f5fccdbb1f905b
2024-08-08 08:05:01 +00:00
Andrew Rumble
71187a51ba Merge pull request #18289 from overleaf/ac-ar-eslint-return-await
Add ESLint rule @typescript-eslint/return-await to backend services

GitOrigin-RevId: 75e3e32597827fcc852e69d479515fc72e8f45e4
2024-05-27 10:22:49 +00:00
Eric Mc Sween
e87df4a791 Merge pull request #18167 from overleaf/em-simpler-typescript-backend
Simpler Typescript setup in the backend

GitOrigin-RevId: 3a90f26a5ac8029d14203faa98008375ce933e7d
2024-05-07 08:04:00 +00:00
Eric Mc Sween
b8195f537d Merge pull request #18065 from overleaf/em-doc-versions-recovery-fix
Doc versions recovery for Server Pro 5.0.3

GitOrigin-RevId: 226b09383f9499c7eb1c20bf9e8e8e549bdbccb0
2024-04-25 08:04:50 +00:00
Eric Mc Sween
a2d5e030c6 Merge pull request #18041 from overleaf/em-jpa-recovery-script
[history-v1] add doc version recovery script

GitOrigin-RevId: 3f240f313465ce5fa9c53f72a992807f9396ebb4
2024-04-23 08:05:40 +00:00
Alf Eaton
6cc2db3cdd Merge pull request #17525 from overleaf/ae-upgrade-prettier
Upgrade Prettier to v3

GitOrigin-RevId: 6f1338f196408f3edb4892d5220ad3665ff1a5bc
2024-03-26 09:04:05 +00:00
Mathias Jakobsen
ce19b5f568 Merge pull request #17359 from overleaf/mj-download-filtering
[overleaf-editor-core+history-v1] Filter tracked deletes when creating zip files

GitOrigin-RevId: 1c19d3cb849150d33e750772399ea81c280fdd57
2024-03-13 09:03:23 +00:00
Mathias Jakobsen
058b40ae70 Merge pull request #17331 from overleaf/mj-raw-json-blobs
[history-v1] Don't gzip range blobs

GitOrigin-RevId: 077630b10562d43a5a1cb00a478276a765acb902
2024-03-07 16:30:39 +00:00
Mathias Jakobsen
b80d0e8634 Merge pull request #17047 from overleaf/mj-ranges-in-blobstore
[overleaf-editor-core] Store tracked changes and comments in blob-store

GitOrigin-RevId: 05cc69044bf26808b64b2687d81f2c40ef4cc972
2024-02-27 09:03:10 +00:00
Jakob Ackermann
1dc2bd3a24 Merge pull request #15445 from overleaf/jpa-history-v1-gzip-in-place
[history-v1] gzip chunk string in-place before passing to object layer

GitOrigin-RevId: db5ded0b0db32757ed35c31c77d190bc121711a7
2023-10-26 08:03:41 +00:00
Christopher Hoskin
0ae174a47b Merge pull request #15382 from overleaf/csh-issue-11625-mongo-ug-5-history-v1
Upgrade mongodb module for history-v1 from 4.11.0 to 6.2.0

GitOrigin-RevId: 2f0a8c07f2797ac23d202858613715b177b41e15
2023-10-26 08:03:38 +00:00
Jakob Ackermann
c03f2807bf Merge pull request #15410 from overleaf/jpa-history-v1-verbose-logging
[history-v1] add verbose logging to I/O operations via object-persistor

GitOrigin-RevId: 4fe7a77f289d5d22a63d6fd0665e5cffe5843b65
2023-10-25 08:04:54 +00:00
Christopher Hoskin
5855ba7b49 Merge pull request #15241 from overleaf/em-gcp-opentelemetry
Replace the GCP trace agent with OpenTelemetry

GitOrigin-RevId: 591410029349e4ef5a39a5b79868e57c7bd089b6
2023-10-19 08:02:52 +00:00
John Lees-Miller
2d9db134a2 Merge pull request #14544 from overleaf/jlm-add-history-restore-script
Add project recovery script

GitOrigin-RevId: b98fb2988245d2b67f8fc0711742294e895b1f07
2023-08-31 08:04:31 +00:00
Christopher Hoskin
cbaf1c3556 Merge pull request #13256 from overleaf/csh-issue-13004-max-retries
Increase the maximum retries for the delete-old-history-chunks cron job

GitOrigin-RevId: 8054ac76bbae24d0a3ba032c50c9647c044aef27
2023-06-02 08:07:17 +00:00
Eric Mc Sween
12e7471213 Merge pull request #12916 from overleaf/bg-move-stream-buffer-code-to-library
move stream-related code to separate  `@overleaf/stream-utils` library

GitOrigin-RevId: a79a873109b927b4fc0ae36f47d5c67e0df58041
2023-06-02 08:05:57 +00:00
Brian Gough
333b54c237 Merge pull request #12904 from overleaf/bg-history-v1-streams-for-node-18
use pipeline for stream operations in history-v1

GitOrigin-RevId: 301a78c2c264d4951ab23054067d6be381778fcf
2023-05-10 08:04:51 +00:00
Eric Mc Sween
e323a22666 Merge pull request #12932 from overleaf/em-history-chunks-project-deletion
Use index when deleting project history chunks

GitOrigin-RevId: 804b8fe8da8bab04a4c0e9fe0008c5554b51817e
2023-05-05 08:05:02 +00:00
Eric Mc Sween
cff54eae78 Merge pull request #12575 from overleaf/em-fix-chunk-replace-transaction
Fix history chunk replacement transaction

GitOrigin-RevId: fc0fa8f4a55ae5f402ba87db5e4ac2856fe6627b
2023-04-19 08:03:10 +00:00
Eric Mc Sween
e41d3cf5db Merge pull request #11890 from overleaf/em-fix-deleted-chunks-indexes
Add a partial index for pending chunks

GitOrigin-RevId: ec0705d1de7ffacb2cb88a8e5e1ff9f05c5acf88
2023-02-23 09:05:50 +00:00