davrot/overleaf-cep
1
0
Fork 0
mirror of https://github.com/yu-i-i/overleaf-cep.git synced 2025-07-23 14:00:08 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions
25981 commits 9 branches 11 tags 216 MiB
Commit graph

204 commits

Author SHA1 Message Date
Andrew Rumble
282f5f92ff Ensure global blobs are loaded before verifying 
GitOrigin-RevId: 3f78c44e6b4d98fd6c469b0f4fbacc033b0a0868
 2025-03-12 09:05:39 +00:00
Andrew Rumble
afedce1b0e Add script for verifying sampled sets of projects 
GitOrigin-RevId: 23a599ea269a528201c67cda66bd2d00c0d376e4
 2025-03-12 09:05:35 +00:00
Andrew Rumble
061d67ee4b Emit more specific errors from backupVerifier 
GitOrigin-RevId: 99475608f096be3e35fbaaf1825b99d145ea86f3
 2025-03-12 09:05:31 +00:00
Andrew Rumble
36056e75d7 Improve chunk loading in backupVerifier 
Brings the process closer to history_store.

We can't use the backup history_store because the keys are generated
differently for chunks than the standard history_store way of doing it.

GitOrigin-RevId: 07adfc0531f6ec0f38bb70ea0fe8ae0d41f508cc
 2025-03-12 09:05:26 +00:00
Brian Gough
efd53e567c Merge pull request #24237 from overleaf/bg-fix-backup-worker 
fixes for backup worker

GitOrigin-RevId: 79c99e5b2d99d1b0298f69d968e0e2cb89281dd6
 2025-03-12 09:05:04 +00:00
Brian Gough
c233243948 Merge pull request #24200 from overleaf/bg-backup-queue-pending-jobs 
fix backup worker and backup scheduler to handle pending projects

GitOrigin-RevId: a97e011615666b3ae2b8fafe26a96d41b3609edd
 2025-03-11 09:06:05 +00:00
Andrew Rumble
441c7a89a7 Merge pull request #24204 from overleaf/ar-jpa-add-chunk-verification 
[history-v1] add chunk verification

GitOrigin-RevId: 7208ad20872386813bb1c6946283afddb5e8b1cf
 2025-03-11 09:05:57 +00:00
Brian Gough
9ff3a45690 Merge pull request #24142 from overleaf/bg-backup-fix-blob-error-handling 
fix bug which caused errors to be ignored when backing up blobs

GitOrigin-RevId: 64674a484a6e22493cba8f8e47d4671072dc2e9a
 2025-03-07 09:06:26 +00:00
Brian Gough
281168fd52 Merge pull request #24080 from overleaf/bg-serialise-backup-initialisation 
fix backup initialization to avoid incorrect use of batchedUpdate

GitOrigin-RevId: 6984f3510c6b03b3dfda35efea8173f848e58eff
 2025-03-07 09:06:22 +00:00
Brian Gough
154de1a0f9 Merge pull request #24102 from overleaf/bg-backup-stats-script 
add backup sampling script

GitOrigin-RevId: cfd546d421cdcfb9f8ad2111703a437a7272bd54
 2025-03-07 09:06:17 +00:00
Brian Gough
893294e6b8 Merge pull request #24069 from overleaf/bg-backup-errors 
more tweaks for backup errors

GitOrigin-RevId: 0f7c7bb5004923c3c22c6e3471bb7152cc3e05e2
 2025-03-05 09:05:50 +00:00
Brian Gough
34b397073f Merge pull request #24053 from overleaf/bg-backup-queue-tweaks-II 
additional backup queue tweaks

GitOrigin-RevId: b54e0c5f4a23bb9962c5062bd1b80f9f124c3c87
 2025-03-05 09:05:15 +00:00
Brian Gough
c815106d62 Merge pull request #24037 from overleaf/bg-fix-backup-worker-errors 
backup worker tweaks

GitOrigin-RevId: 7901bb0257c7c7320afd08ef965f124347a1de30
 2025-03-04 09:05:12 +00:00
Brian Gough
1dbf5dca10 Merge pull request #24019 from overleaf/bg-backup-add-error-logging 
add logging of backup errors in batch mode

GitOrigin-RevId: 96ae7a2354f82451b3bbe8c5459c4d4cf47e5459
 2025-03-04 09:04:44 +00:00
Brian Gough
4720e8f2c5 Merge pull request #23989 from overleaf/bg-backup-worker-reduce-logging 
reduce log level from info to debug for backup logging

GitOrigin-RevId: 10f541b0e0068aafd50710226fb8425527fee7c5
 2025-03-03 09:05:22 +00:00
Brian Gough
8d0fa5be65 Merge pull request #23977 from overleaf/bg-backup-worker-load-global-blobs 
ensure global blobs are loaded in backup worker

GitOrigin-RevId: b80791ef38e7fa8db364bdd3a9c461757778409d
 2025-03-03 09:05:08 +00:00
Brian Gough
b37b409994 Merge pull request #23946 from overleaf/bg-backup-queue-implementation 
additional backup queue implementation

GitOrigin-RevId: 03754e57a6c6798a07dcca6a5248dec61b3cdc7a
 2025-02-28 09:05:50 +00:00
Brian Gough
763e5ba82c Merge pull request #23920 from overleaf/bg-backup-queue 
initial history backup queue worker app

GitOrigin-RevId: e9647a2ec3caeb1fff843cead12164ba89fdb1f8
 2025-02-28 09:04:46 +00:00
Brian Gough
6e55a9fa81 Merge pull request #23837 from overleaf/bg-redis-redact-config 
sanitise password in redis config output

GitOrigin-RevId: 45d8bd13f01c3c4f138549c8a9f47dc0db29d0b0
 2025-02-25 09:05:22 +00:00
Brian Gough
0acc067205 Merge pull request #23836 from overleaf/bg-backup-extend-debugging-script 
add support for fetching chunks in backup debugging script

GitOrigin-RevId: 641298088d62148693525259748dcddf4d67d14b
 2025-02-25 09:05:18 +00:00
Brian Gough
9d8f9e5358 Merge pull request #23835 from overleaf/bg-backup-show-errors 
add error cause logging for missing chunks and blobs in backup comparison

GitOrigin-RevId: 9a1bf84b64621add71fb1d1e0cb013598b226bc8
 2025-02-25 09:05:14 +00:00
Brian Gough
dd8a683e9e Merge pull request #23834 from overleaf/bg-backup-graceful-shutdown 
add graceful shutdown handling to backup script

GitOrigin-RevId: f0fb9a3867b26c7800edefab126fd05fbabdae02
 2025-02-25 09:05:10 +00:00
Andrew Rumble
b0ed2efdf9 Merge pull request #23812 from overleaf/ar-allow-debugging-for-remove-backup-blobs-script-errors 
[history-v1] Minor improvements to AWS cleanup script

GitOrigin-RevId: bd01a72169b9cbc018612173294cfbf8649a9fbd
 2025-02-24 09:06:15 +00:00
Andrew Rumble
a0d79d5871 Merge pull request #23742 from overleaf/ar-script-for-removing-blobs-in-wrong-place 
[history-v1] Script for deleting blobs backed up to wrong location

GitOrigin-RevId: 6abe1dba5164f8fa7d41c4ee3e4a07764e73b7e1
 2025-02-24 09:05:53 +00:00
Brian Gough
981e91f012 Merge pull request #21763 from overleaf/bg-backup-script 
initial script for running backups

GitOrigin-RevId: d22c373de30738d8080d40dce10790f0bdcb9f51
 2025-02-24 09:04:32 +00:00
Brian Gough
32983da29c Merge pull request #23700 from overleaf/bg-history-redis 
add redis to history-v1

GitOrigin-RevId: def900c4b560ea0a297b11b6c0ecc8cc8caa5b49
 2025-02-21 09:04:06 +00:00
Brian Gough
d2738fda73 Merge pull request #23565 from overleaf/bg-fix-history-metadata-in-projects-collection 
fix history metadata in projects collection

GitOrigin-RevId: 18c821ef5966a8470b24dfa60313b09bdda9707d
 2025-02-14 09:03:33 +00:00
Jakob Ackermann
64e8d2b8b3 [history-v1] fix backup deletion for postgres projects (#23466) 
* [history-v1] fix backup deletion for postgres projects

* [history-v1] convert historyId to string on assignment

Co-authored-by: Brian Gough <brian.gough@overleaf.com>

---------

Co-authored-by: Brian Gough <brian.gough@overleaf.com>
GitOrigin-RevId: 5e1033972745a9b72606638f56ebf2147406cc39
 2025-02-10 09:05:06 +00:00
Eric Mc Sween
9893f18ca5 Merge pull request #23447 from overleaf/em-reduce-validation-error-logs 
Only log validation errors once per flush

GitOrigin-RevId: ee3f656c4c7c09fd7f3ff2462278c9aef81b9bb5
 2025-02-07 09:06:10 +00:00
Eric Mc Sween
3c1f20a6d1 Merge pull request #23433 from overleaf/em-do-not-store-content-hashes 
Do not store content hashes in chunks

GitOrigin-RevId: 65a255b92f9c4e216ad5a1fb5fb3fa5a6b9158c4
 2025-02-07 09:06:03 +00:00
Eric Mc Sween
ce4c8a4e47 Merge pull request #23398 from overleaf/em-log-doc-hash-mismatches 
Validate content hashes in history (log only)

GitOrigin-RevId: ed772fc4e4d0aa9e980f9693a759647bd937e13a
 2025-02-07 09:05:59 +00:00
Eric Mc Sween
e145667a81 Merge pull request #23282 from overleaf/em-async-await-persist-changes 
Convert the history changes import code to async/await

GitOrigin-RevId: 6421fcaaf3bac69a3404754f935b4902979b4689
 2025-02-07 09:05:44 +00:00
Jakob Ackermann
26321c5dba [history-v1] block deletion of bucket prefix that is not a valid project prefix (#22889) 
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
GitOrigin-RevId: a3aff76a2e299b2e2ed030e34da0631fce51ad4b
 2025-02-07 09:05:01 +00:00
Brian Gough
b2ff14f669 Merge pull request #23375 from overleaf/bg-add-id-checks-for-chunks 
guard against non-postgres projectIds

GitOrigin-RevId: eab2024e4e893591f4b1c6a507b26d935273ae5f
 2025-02-05 09:06:54 +00:00
Brian Gough
098d91f0bb Merge pull request #23345 from overleaf/bg-write-latest-history-version-to-project 
update project entry with history metadata on chunk creation

GitOrigin-RevId: dd19898f3d16e2e3360ff1bcccbf79f7dd27addb
 2025-02-05 09:05:28 +00:00
Jakob Ackermann
3a4c5a0d0f [history-v1] add readOnly lookup for raw chunks (#23318) 
* [history-v1] add readOnly lookup for raw chunks

Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>

* [history-v1] reduce min poolsize for readOnly pool to 0

Co-authored-by: Brian Gough <brian.gough@overleaf.com>

---------

Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
GitOrigin-RevId: a711c4ee4f3ea3775bd090e620d1ef52689fa1f4
 2025-02-04 09:04:52 +00:00
Brian Gough
8831fb9a18 Merge pull request #23169 from overleaf/bg-export-global-blobs 
Add script to export global blobs to CSV

GitOrigin-RevId: 86c885981b150aae4541770a094c822e1fb364b9
 2025-02-03 09:05:53 +00:00
Jakob Ackermann
c6c623da78 [project-history] script for fixing-up files/metadata with bulk resync (#23184) 
* [history-v1] add cheap endpoint for checking time of last history write

The /raw endpoint skips the GCS lookup for the chunk.

* [project-history] script for fixing-up files/metadata with bulk resync

* [project-history] upgrade structure only resync when full sync is needed

* [project-history] start resync and process resync updates under lock

* [project-history] stop retrying during graceful shutdown

GitOrigin-RevId: 73184d5786e1d40f5b7e21f387fc37cf43f0ac2d
 2025-02-03 09:05:43 +00:00
Brian Gough
d6e89c7338 Merge pull request #23058 from overleaf/bg-export-backed-up-blobs 
export backed up blobs to csv file

GitOrigin-RevId: 5a176055dc85c56ced64cbf13aa705b56071cdae
 2025-01-24 09:05:31 +00:00
Jakob Ackermann
9c377b74c1 [history-v1] add flag for back-filling hashes for projectIds from file (#23005) 
* [history-v1] delay process exit to give logging time to flush

* [history-v1] add flag for back-filling hashes for projectIds from file

GitOrigin-RevId: 887a1e1c72d6f5a13bfc8d0e54023afbf5bc671c
 2025-01-22 09:05:21 +00:00
Jakob Ackermann
6884338a34 [history-v1] fix remaining files with missing hashes (#22985) 
GitOrigin-RevId: df382732954538f9db177180aee0a44c5d5679ca
 2025-01-22 09:05:16 +00:00
Jakob Ackermann
ab4d1e0986 Merge pull request #22870 from overleaf/jpa-back-fill-fix-up 
[history-v1] add script for fixing up back-fill errors

GitOrigin-RevId: 118992a32c1f6da4289cd35399ddd07a741da4ee
 2025-01-20 09:05:14 +00:00
Jakob Ackermann
70fd6cacbc Merge pull request #22711 from overleaf/jpa-gzip 
[history-v1] compress blobs before sending them to AWS

GitOrigin-RevId: 1ca1dda6f36738fbabbf00fdab62b86230b9e4f9
 2025-01-08 09:04:57 +00:00
Andrew Rumble
2262d03a21 Merge pull request #22648 from overleaf/ar-store-backup-blobs-by-history-id 
Use historyId when constructing path for backup

GitOrigin-RevId: 954576b509d5e78511b5008fb7d74e0bc5fa45fd
 2024-12-24 09:04:52 +00:00
Andrew Rumble
31e8a908ee Merge pull request #22334 from overleaf/ar-guard-against-integer-like-strings-when-working-with-postgres 
[history-v1] Guard against non-postgres projectIds

GitOrigin-RevId: 5bf75c67424297f52f2abd9d0f0f14a0f79f8921
 2024-12-13 09:04:59 +00:00
Andrew Rumble
a92a37bc3c Merge pull request #22466 from overleaf/ar-backup-files-when-inserting 
[history-v1] backup files when inserting

GitOrigin-RevId: e636bce178604978c6d41c083bf671795d20b5a1
 2024-12-13 09:04:54 +00:00
Andrew Rumble
52254b5695 Merge pull request #22459 from overleaf/revert-22392-ar-backup-files-when-inserting 
Revert "[history-v1] backup files when inserting"

GitOrigin-RevId: f21d49dbc8909ab93bdde78c321672124bb13697
 2024-12-12 09:05:27 +00:00
Andrew Rumble
6404e3047d Merge pull request #22392 from overleaf/ar-backup-files-when-inserting 
[history-v1] backup files when inserting

GitOrigin-RevId: 1649b2828899d67ee37c0ac331917c6d5424c803
 2024-12-12 09:05:11 +00:00
Brian Gough
104ae341b1 Merge pull request #22327 from overleaf/bg-fix-copy-blob 
fix bug that prevents copying blobs between different backends in history-v1

GitOrigin-RevId: 41140ad42d0d7c1beda83e588649127c22603dec
 2024-12-05 09:05:22 +00:00
Jakob Ackermann
d19c5e236f Merge pull request #22208 from overleaf/jpa-clsi-hash 
[misc] clsi: read files from history-v1 with fallback to filestore

GitOrigin-RevId: c54bb128780198c14e7a63818f39fad62ce65d4e
 2024-11-29 09:05:39 +00:00