davrot/overleaf-cep
1
0
Fork 0
mirror of https://github.com/yu-i-i/overleaf-cep.git synced 2025-07-23 05:00:07 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions
25981 commits 9 branches 11 tags 216 MiB
Commit graph

327 commits

Author SHA1 Message Date
Brian Gough
4ba0e97b95 Merge pull request #24749 from overleaf/bg-history-redis-buffer 
add history redis to history-v1

GitOrigin-RevId: 70dc1aee809ad17902c93c020f3804c0f1429238
 2025-04-10 08:05:35 +00:00
Brian Gough
d85dbe429d Merge pull request #24745 from overleaf/bg-history-use-consistent-import-for-chunk-store 
use consistent import for chunk_store

GitOrigin-RevId: 427b148c53c9d0913b2cdfdc634273a1d8ece060
 2025-04-10 08:05:31 +00:00
Brian Gough
040f70471c Merge pull request #24636 from overleaf/bg-history-backup-fix-broken-projects 
add --fix option to backup script

GitOrigin-RevId: 568c9158669bb1cede0f0dd75e7507b10e8ff5a2
 2025-04-08 08:05:59 +00:00
Brian Gough
f08532dfb0 Merge pull request #24637 from overleaf/bg-history-backup-uninitialised-projects 
backup uninitialised projects

GitOrigin-RevId: 9310ef9f803decffbd674024a1ffd33d1960a2c4
 2025-04-08 08:05:54 +00:00
Andrew Rumble
93793fe723 Increase default value 
GitOrigin-RevId: 6674716042415aaa6b73b4efcf46443c3faab0cc
 2025-03-27 14:17:15 +00:00
Andrew Rumble
46f3b595a9 Configure backup RPO for backup-verifier 
GitOrigin-RevId: 376c2f18a1e9bef4e77f0ce5999ec7ce5341d481
 2025-03-27 14:17:11 +00:00
Andrew Rumble
99ab41fd76 Allow scaling in getEndDateForRPO 
RPO can now be scaled to allow a little extra grace in certain
circumstances.

Co-authored-by: Brian Gough <brian.gough@overleaf.com>
GitOrigin-RevId: fa60a9ffe966977e396b5645919ddd1451fb1b7a
 2025-03-27 14:16:40 +00:00
Andrew Rumble
c921c8f586 Monitor backup status in health check 
GitOrigin-RevId: 1a90b2fcf85bfa5ec1e5aabcfe4183d406da95e6
 2025-03-27 14:16:36 +00:00
Andrew Rumble
4707842642 Add functions for monitoring backup status of db records 
GitOrigin-RevId: f0d3564dfd3b38ef9ee451de866c0c05acd0cb20
 2025-03-27 14:16:32 +00:00
Andrew Rumble
dabf610764 Extract getEndDateForRPO method to utils 
This will allow sharing with other functionality.

GitOrigin-RevId: a6e11447180511cc3741ca03f4996ef7ceb45ea5
 2025-03-27 14:16:28 +00:00
Eric Mc Sween
0e9c310d1d Merge pull request #24390 from overleaf/em-enforce-content-hash-validation 
Enforce content hash validation in history

GitOrigin-RevId: 90de21ea86ddc6548001059c41139a2af5b27060
 2025-03-24 10:50:01 +00:00
Andrew Rumble
800b151024 Ensure metrics exist before collecting them 
GitOrigin-RevId: 811301ddebb794341dccc0f27954c66dc5c44aa4
 2025-03-24 10:48:58 +00:00
Andrew Rumble
47d8e59938 Add code to shutdown message for debugging 
GitOrigin-RevId: 1efbda8c82ca1bc160ec4334388fb14259be563c
 2025-03-24 10:48:53 +00:00
Andrew Rumble
56f8993bd7 Remove low information log message 
GitOrigin-RevId: 8e961d7dea04debe0f459db3b461673618ab7597
 2025-03-24 10:48:48 +00:00
Andrew Rumble
432a92173a Perform verification smoothly over time window 
GitOrigin-RevId: b00c18a61a5473f3b7353fd5ab1e55f191722ecd
 2025-03-24 10:48:44 +00:00
Andrew Rumble
2779691cd9 Add a script for verifying a set of blobs have been backed up 
GitOrigin-RevId: 8f7af647ce0ac86ee9800cdf3d386fb78debed7a
 2025-03-24 10:48:39 +00:00
Jakob Ackermann
a530cca2c5 [history-v1] increase timeout for preparing test environment (#24345) 
GitOrigin-RevId: a11a16f07da61603218ff14bef32172d4f32c469
 2025-03-18 09:05:53 +00:00
Brian Gough
0292bc418d Merge pull request #24335 from overleaf/bg-warn-old-pending-changes 
add warning for backups outside rpo threshold

GitOrigin-RevId: a8421529ae64693d860b0325961b010a132426da
 2025-03-18 09:05:49 +00:00
Brian Gough
3f7c88108c Merge pull request #24275 from overleaf/bg-fix-pending-change-timestamp 
fix pending change timestamp

GitOrigin-RevId: 9297a4b57ea718e6a2e1ca62388919c62911af6c
 2025-03-18 09:05:08 +00:00
Andrew Rumble
f134746c9c Extend script to allow multiple blobs to be processed from csv 
GitOrigin-RevId: ad47eb754436ddc7f56b27ceda627268c3a030a1
 2025-03-18 09:05:03 +00:00
Andrew Rumble
b5d6484991 Add a script to backup a single blob from a project 
GitOrigin-RevId: 464e6d69093b87891497e07d1627cd20e2285380
 2025-03-18 09:04:59 +00:00
Andrew Rumble
78481e010e Add verification looper and handle shutdown signals 
Shutdown signals become more relevant now that we are looping as we want
to gracefully stop processing records rather than continue looping.

GitOrigin-RevId: dbb499388c86d552d77954988f8fc27d140da3f1
 2025-03-18 09:04:54 +00:00
Andrew Rumble
9d72eeeeac Add new strategy to verify_sampled_projects 
GitOrigin-RevId: d967da41250bb5945d5b8668b212d4a61b4f9d69
 2025-03-18 09:04:50 +00:00
Andrew Rumble
f85fdd3a97 Refactor project sampler and add new sampler type 
GitOrigin-RevId: 984aa35cef1165e1c8342073b9211a387bd6089e
 2025-03-18 09:04:46 +00:00
Jakob Ackermann
f6bd485863 [misc] align initializing of mongodb replica set (#24287) 
* [misc] align initializing of mongodb replica set

* [misc] fix volumes for mono container in dev-env

* Remove duplicate volumes key

---------

Co-authored-by: Alf Eaton <alf.eaton@overleaf.com>
GitOrigin-RevId: 0119c42ee8b13a0fca9373e40a27fc5b2a52d73b
 2025-03-17 09:04:58 +00:00
Brian Gough
2d48c86e61 Merge pull request #24269 from overleaf/bg-backup-replication-smoothing 
extend backup scheduler to delay jobs for load levelling

GitOrigin-RevId: be62c458d3be464bf80ca590fcc107c2760aa061
 2025-03-13 09:06:28 +00:00
Brian Gough
a178c0f400 Merge pull request #24268 from overleaf/bg-backup-replication-lag-metrics 
add backup replication metrics

GitOrigin-RevId: a734435e3c6ce56350b2286bd218a5e2324d93a9
 2025-03-13 09:06:24 +00:00
Brian Gough
cf105cf01d Merge pull request #24267 from overleaf/bg-fix-init-option 
update backup script to use index when finding uninitialised projects

GitOrigin-RevId: 04ce0654b3d42de1c1a9bb542482c2dd53540628
 2025-03-13 09:06:20 +00:00
Brian Gough
3b93efdf5c Merge pull request #24256 from overleaf/bg-backup-fix-sample 
fix backup sample script

GitOrigin-RevId: 4bf4c15324d9b3b7ebd809f7b212e52d75ab216d
 2025-03-13 09:06:16 +00:00
Andrew Rumble
768180c456 Handle errors during startup 
GitOrigin-RevId: ba8e5964e78439b0ea3650191f5ea9c9e22f2b50
 2025-03-13 09:04:55 +00:00
Andrew Rumble
dad6f97cce Load global blobs when starting backup-verifier 
GitOrigin-RevId: 31aa2a3639f8d9a69b5b98f27aff17b98bd14694
 2025-03-13 09:04:51 +00:00
Andrew Rumble
273ae4aecd Split healthCheck out into separate module 
GitOrigin-RevId: 847d00b696fe6d82f4bd5fea8f9130437c68e7b2
 2025-03-13 09:04:47 +00:00
Andrew Rumble
b5f8bfa28e Switch health check to use projects instead of blobs 
Co-authored-by: Jakob Ackermann <jakob.ackermann@overleaf.com>
GitOrigin-RevId: db1a1c8ce5968e558b0754e5e0da50af89fd80db
 2025-03-13 09:04:43 +00:00
Andrew Rumble
19eefebe95 Revert "Switch health check to use projects instead of blobs" 
This reverts commit c318b70397ed5e2fcbb07fa019412b56844260ef.

GitOrigin-RevId: 087ae9d21be83bf3dae47c4e6d27eb4e74f387df
 2025-03-12 09:06:34 +00:00
Andrew Rumble
087a9daf34 Revert "Split healthCheck out into separate module" 
This reverts commit 96061812977d5c854e494cd44163b16a96722b17.

GitOrigin-RevId: f30a185b65a4f1346ed13fa0c6e9ea0852d44335
 2025-03-12 09:06:30 +00:00
Andrew Rumble
a7be1f3430 Split healthCheck out into separate module 
GitOrigin-RevId: 96061812977d5c854e494cd44163b16a96722b17
 2025-03-12 09:06:22 +00:00
Andrew Rumble
c373db4f86 Switch health check to use projects instead of blobs 
Co-authored-by: Jakob Ackermann <jakob.ackermann@overleaf.com>
GitOrigin-RevId: c318b70397ed5e2fcbb07fa019412b56844260ef
 2025-03-12 09:06:18 +00:00
Andrew Rumble
0b387c5116 Simplify error identification 
GitOrigin-RevId: 6dae1ed99269deb26e15ec921ba173f85f5e6fc9
 2025-03-12 09:05:48 +00:00
Andrew Rumble
149b590413 Fix script name in usage prompt 
GitOrigin-RevId: ab4ed7479c96d3d1db82b23ef3c883706c61995e
 2025-03-12 09:05:44 +00:00
Andrew Rumble
282f5f92ff Ensure global blobs are loaded before verifying 
GitOrigin-RevId: 3f78c44e6b4d98fd6c469b0f4fbacc033b0a0868
 2025-03-12 09:05:39 +00:00
Andrew Rumble
afedce1b0e Add script for verifying sampled sets of projects 
GitOrigin-RevId: 23a599ea269a528201c67cda66bd2d00c0d376e4
 2025-03-12 09:05:35 +00:00
Andrew Rumble
061d67ee4b Emit more specific errors from backupVerifier 
GitOrigin-RevId: 99475608f096be3e35fbaaf1825b99d145ea86f3
 2025-03-12 09:05:31 +00:00
Andrew Rumble
36056e75d7 Improve chunk loading in backupVerifier 
Brings the process closer to history_store.

We can't use the backup history_store because the keys are generated
differently for chunks than the standard history_store way of doing it.

GitOrigin-RevId: 07adfc0531f6ec0f38bb70ea0fe8ae0d41f508cc
 2025-03-12 09:05:26 +00:00
Brian Gough
efd53e567c Merge pull request #24237 from overleaf/bg-fix-backup-worker 
fixes for backup worker

GitOrigin-RevId: 79c99e5b2d99d1b0298f69d968e0e2cb89281dd6
 2025-03-12 09:05:04 +00:00
Brian Gough
c233243948 Merge pull request #24200 from overleaf/bg-backup-queue-pending-jobs 
fix backup worker and backup scheduler to handle pending projects

GitOrigin-RevId: a97e011615666b3ae2b8fafe26a96d41b3609edd
 2025-03-11 09:06:05 +00:00
Andrew Rumble
441c7a89a7 Merge pull request #24204 from overleaf/ar-jpa-add-chunk-verification 
[history-v1] add chunk verification

GitOrigin-RevId: 7208ad20872386813bb1c6946283afddb5e8b1cf
 2025-03-11 09:05:57 +00:00
Brian Gough
9ff3a45690 Merge pull request #24142 from overleaf/bg-backup-fix-blob-error-handling 
fix bug which caused errors to be ignored when backing up blobs

GitOrigin-RevId: 64674a484a6e22493cba8f8e47d4671072dc2e9a
 2025-03-07 09:06:26 +00:00
Brian Gough
281168fd52 Merge pull request #24080 from overleaf/bg-serialise-backup-initialisation 
fix backup initialization to avoid incorrect use of batchedUpdate

GitOrigin-RevId: 6984f3510c6b03b3dfda35efea8173f848e58eff
 2025-03-07 09:06:22 +00:00
Brian Gough
154de1a0f9 Merge pull request #24102 from overleaf/bg-backup-stats-script 
add backup sampling script

GitOrigin-RevId: cfd546d421cdcfb9f8ad2111703a437a7272bd54
 2025-03-07 09:06:17 +00:00
Brian Gough
893294e6b8 Merge pull request #24069 from overleaf/bg-backup-errors 
more tweaks for backup errors

GitOrigin-RevId: 0f7c7bb5004923c3c22c6e3471bb7152cc3e05e2
 2025-03-05 09:05:50 +00:00