davrot/overleaf-cep
1
0
Fork 0
mirror of https://github.com/yu-i-i/overleaf-cep.git synced 2025-07-25 02:00:07 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions
25942 commits 9 branches 12 tags 216 MiB
Commit graph

101 commits

Author SHA1 Message Date
Jakob Ackermann
81f0807fc6 [web] prepare filestore migration for Server Pro/CE (#27230) 
* [web] prepare filestore migration for Server Pro/CE

* [history-v1] remove unused USER_FILES_BUCKET_NAME env var from script

* [server-ce] tests: write default docker-compose.override.yml on startup

* [server-ce] tests: extend access logging of host-admin for response

* [server-ce] tests: test text and binary file upload

* [server-ce] tests: add tests for filestore migration

* [web] simplify feature gate for filestore/project-history-blobs logic

Co-authored-by: Brian Gough <brian.gough@overleaf.com>

* [server-ce] test: fix flaky test helper

---------

Co-authored-by: Brian Gough <brian.gough@overleaf.com>
GitOrigin-RevId: f89bdab2749e2b7a49d609e2eac6bf621c727966
 2025-07-22 08:06:58 +00:00
Jakob Ackermann
bf43d4f709 [history-v1] make back_fill_file_hash_fix_up compatible with Server Pro (#27280) 
* [history-v1] move MockFilestore into shared place

Co-authored-by: Brian Gough <brian.gough@overleaf.com>

* [history-v1] make back_fill_file_hash_fix_up compatible with Server Pro

---------

Co-authored-by: Brian Gough <brian.gough@overleaf.com>
GitOrigin-RevId: 70ea57e1503031d9f14dcd60c4c110e746450587
 2025-07-22 08:06:41 +00:00
Brian Gough
5208ad39ec Merge pull request #27173 from overleaf/bg-filestore-migration-remove-backups 
remove backup code from back_fill_file_hash script and tests

GitOrigin-RevId: 364eefc47670e6e7f2314de810ea259b609ff976
 2025-07-18 08:06:14 +00:00
Brian Gough
5669a7d1c8 Merge pull request #27147 from overleaf/bg-filestore-migration-for-server-pro-II 
add support for fetching files via http from filestore in back_fill_file_hash script and tests

GitOrigin-RevId: 8dea6383ed6fe9ee6786a5695e2deee93b1cdd84
 2025-07-17 08:05:30 +00:00
Jakob Ackermann
cf472f54d0 [history-v1] use POST requests for expiring redis buffer from cron (#26568) 
* [history-v1] use POST requests for expiring redis buffer from cron

(cherry picked from commit 15780ac54e36b96e1aed9fd9eb6dfe9d4fbf842f)

* [history-v1] remove double claim of expire job

GitOrigin-RevId: 8b2eab07006a5819a47eed3f646b2a4d75f86e5b
 2025-07-08 08:05:47 +00:00
Andrew Rumble
745043ca92 Add more detail for the modes 
GitOrigin-RevId: 44b63b76b1edb8bfb78049c9c8560934e340ef96
 2025-06-25 08:06:41 +00:00
Andrew Rumble
f2b0a982ac Use BackupBlobStore to get chunk content 
GitOrigin-RevId: 91de53101ea289b43bdb08352aecc09ae89d0f74
 2025-06-25 08:06:32 +00:00
Andrew Rumble
797f29d40a Handle errors when processing command line arguments 
GitOrigin-RevId: 7c1f011e705d6e4b6fbf0afee3dc3aab749f0de4
 2025-06-25 08:06:27 +00:00
Andrew Rumble
392037efd6 Improve types and export functions we want to use 
GitOrigin-RevId: 25fd93214f88c425bee23d43de113cf754f89524
 2025-06-25 08:06:18 +00:00
Andrew Rumble
3600aa4b75 Add script for restoring projects from backup 
GitOrigin-RevId: b639d74989afa17638bbcba0f8ee34c978fcd5d9
 2025-06-25 08:06:14 +00:00
Eric Mc Sween
1042092144 Merge pull request #26547 from overleaf/em-revert-expiry-post 
Revert "use POST requests for expiring redis buffer from cron"

GitOrigin-RevId: 95e9fd1be7b73699d6fac24035437d467c273d0a
 2025-06-19 08:06:58 +00:00
Eric Mc Sween
150dfd6cba Merge pull request #26539 from overleaf/jpa-post 
[history-v1] use POST requests for expiring redis buffer from cron

GitOrigin-RevId: 51c9a25b998e581ed20c0e113bd4989537a1e6ef
 2025-06-19 08:06:50 +00:00
Eric Mc Sween
7bdc4291fc Merge pull request #26533 from overleaf/jpa-queue 
[history-v1] use POST requests for flushing history queues

GitOrigin-RevId: ae87a046a7844b25cc123d08ac6c8d1624608394
 2025-06-19 08:06:16 +00:00
Eric Mc Sween
f68bf5a69f Merge pull request #26477 from overleaf/em-redis-buffer-resync 
Handle invalid content hash when persisting changes

GitOrigin-RevId: 5259190396c8c261cad1abcd5de66314c1e871fb
 2025-06-18 08:06:17 +00:00
Eric Mc Sween
9cb4ef4d7d Merge pull request #26353 from overleaf/bg-history-redis-extend-persist-worker 
extend persist worker to make parallel requests

GitOrigin-RevId: 8def7d5a8b5c9fcbe5fe45ac8f3ace503d31877a
 2025-06-17 08:05:55 +00:00
Brian Gough
2f44a4eb5a Merge pull request #26351 from overleaf/bg-history-redis-convert-persist-worker-to-esm 
history redis convert persist worker to esm

GitOrigin-RevId: edcbac6e3f1d3dde3fa8239378995f3ff3afcfdd
 2025-06-13 08:06:32 +00:00
Brian Gough
fdd0d95554 Merge pull request #26293 from overleaf/bg-history-redis-fix-persist-worker 
add missing load global blobs from persist worker

GitOrigin-RevId: ae9393f2353fb4d5afe349aa7d0a26bab80c7f53
 2025-06-11 08:06:59 +00:00
Brian Gough
c23e84eb37 Merge pull request #26273 from overleaf/bg-history-redis-add-persist-worker-to-cron 
modify existing run-chunk-lifecycle cron job to persist and expire redis queues

GitOrigin-RevId: afb94b3e2fba7368cfec11997dfd5b2bbd6321a9
 2025-06-11 08:06:23 +00:00
Brian Gough
2eb695f4c3 Merge pull request #26122 from overleaf/bg-history-redis-make-persist-buffer-consistent 
make persistBuffer export consistent with other methods

GitOrigin-RevId: 24536e521e1d20ef63cc74bd9ba40e095025d512
 2025-06-09 08:05:39 +00:00
Brian Gough
d280f40885 Merge pull request #26116 from overleaf/bg-history-redis-show-buffer 
add script to display redis buffer for a given history ID

GitOrigin-RevId: 71c2e79480c0873d30801ed3c13aa9a7fc7873f6
 2025-06-09 08:05:35 +00:00
Brian Gough
393cee7af5 Merge pull request #25993 from overleaf/bg-history-refactor-persist-buffer-limits 
refactor persist buffer to add limits

GitOrigin-RevId: 4a40a7a8812acf5bb7f98bfd7b94d81ebe19fc57
 2025-06-04 08:06:19 +00:00
Brian Gough
50df3862e9 Merge pull request #25954 from overleaf/bg-history-expire-worker-fix 
fix expire_redis_chunks to only clear job on error

GitOrigin-RevId: f7ec435edda95958b453fba501686dcfd84426f7
 2025-06-04 08:06:14 +00:00
Brian Gough
a80203f748 Merge pull request #25909 from overleaf/bg-history-persist-worker 
add history persist worker

GitOrigin-RevId: b9e31e7bdd84570efc0b87b9f5e90b4078551a8c
 2025-06-04 08:06:10 +00:00
Jakob Ackermann
11b94593c2 [web] remove deledFiles collection (#25750) 
* [history-v1] remove processing of deleted files when back-filling hashes

* [web] remove deledFiles collection

GitOrigin-RevId: 7c080e564f7d7acb33ebe7ebe012f415a847d0df
 2025-05-23 08:05:01 +00:00
Brian Gough
0d70223a48 Merge pull request #25482 from overleaf/bg-fix-bull-exporter-errors 
retain completed and failed jobs for backup queue

GitOrigin-RevId: 7831ce2565dc493e3ce7f55001207daea2140575
 2025-05-13 08:05:40 +00:00
Brian Gough
9e07549ecb Merge pull request #25449 from overleaf/bg-histoy-redis-refactor-expire-worker 
refactor the expire worker to make it easier to extend

GitOrigin-RevId: 7b277b243ed51ab3b46316d98b7157af95a9e42b
 2025-05-12 08:05:19 +00:00
Eric Mc Sween
391fca9e83 Merge pull request #25361 from overleaf/em-load-latest-raw 
Rename loadLatestRaw() to getLatestChunkMetadata()

GitOrigin-RevId: e089dcfa57cbbc43df8666b51eca0d81d595a5a7
 2025-05-09 08:06:14 +00:00
Eric Mc Sween
5717ea7f5c Merge pull request #25306 from overleaf/em-redis-buffer-read-operations 
Add changes from Redis when reading chunks from the chunk store

GitOrigin-RevId: c0ebf0669b91eb2efc5d1091d025e81efdff9fe4
 2025-05-09 08:06:09 +00:00
Brian Gough
f9b36cd5be Merge pull request #25241 from overleaf/bg-remove-existing-chunk-buffer 
remove existing chunk redis backend and chunk buffer

GitOrigin-RevId: 28fb02d1802312de6892e2fb7dd59191e3fc8914
 2025-05-08 08:05:09 +00:00
Brian Gough
850da34778 Merge pull request #25086 from overleaf/bg-history-buffer-use-expire-time 
add expire time to redis buffer in history-v1

GitOrigin-RevId: 3d74957c341e62e536dc60869a7ca71ac173e380
 2025-04-29 08:05:14 +00:00
Brian Gough
626416ed02 Merge pull request #24945 from overleaf/bg-redis-buffer-stats-script 
add script for listing buffer stats from redis

GitOrigin-RevId: 7836563d51a5c6ded264d0e709d5cfcda70596e3
 2025-04-24 08:05:49 +00:00
Brian Gough
d6c2188f2d Merge pull request #24903 from overleaf/bg-fix-backup-scheduler 
remove history.id check from processPendingProjects function

GitOrigin-RevId: 017ab7c9bf693ed4077d4619574154166af04fe3
 2025-04-16 08:05:06 +00:00
Brian Gough
3850e97446 Merge pull request #24857 from overleaf/bg-backup-scheduler-fix-for-broken-projects 
handle broken projects in history backup

GitOrigin-RevId: a295d9d7a29715807a8172dae46e0fe3c14ecf12
 2025-04-15 08:06:23 +00:00
Brian Gough
835e14b8b2 Merge pull request #24768 from overleaf/bg-history-redis-buffer 
test redis caching when loading latest chunk in history-v1

GitOrigin-RevId: f0ee09e5e9e1d7605e228913cb8539be4134e1f7
 2025-04-15 08:05:03 +00:00
Brian Gough
4ba0e97b95 Merge pull request #24749 from overleaf/bg-history-redis-buffer 
add history redis to history-v1

GitOrigin-RevId: 70dc1aee809ad17902c93c020f3804c0f1429238
 2025-04-10 08:05:35 +00:00
Brian Gough
040f70471c Merge pull request #24636 from overleaf/bg-history-backup-fix-broken-projects 
add --fix option to backup script

GitOrigin-RevId: 568c9158669bb1cede0f0dd75e7507b10e8ff5a2
 2025-04-08 08:05:59 +00:00
Brian Gough
f08532dfb0 Merge pull request #24637 from overleaf/bg-history-backup-uninitialised-projects 
backup uninitialised projects

GitOrigin-RevId: 9310ef9f803decffbd674024a1ffd33d1960a2c4
 2025-04-08 08:05:54 +00:00
Andrew Rumble
2779691cd9 Add a script for verifying a set of blobs have been backed up 
GitOrigin-RevId: 8f7af647ce0ac86ee9800cdf3d386fb78debed7a
 2025-03-24 10:48:39 +00:00
Brian Gough
0292bc418d Merge pull request #24335 from overleaf/bg-warn-old-pending-changes 
add warning for backups outside rpo threshold

GitOrigin-RevId: a8421529ae64693d860b0325961b010a132426da
 2025-03-18 09:05:49 +00:00
Andrew Rumble
f134746c9c Extend script to allow multiple blobs to be processed from csv 
GitOrigin-RevId: ad47eb754436ddc7f56b27ceda627268c3a030a1
 2025-03-18 09:05:03 +00:00
Andrew Rumble
b5d6484991 Add a script to backup a single blob from a project 
GitOrigin-RevId: 464e6d69093b87891497e07d1627cd20e2285380
 2025-03-18 09:04:59 +00:00
Andrew Rumble
78481e010e Add verification looper and handle shutdown signals 
Shutdown signals become more relevant now that we are looping as we want
to gracefully stop processing records rather than continue looping.

GitOrigin-RevId: dbb499388c86d552d77954988f8fc27d140da3f1
 2025-03-18 09:04:54 +00:00
Andrew Rumble
9d72eeeeac Add new strategy to verify_sampled_projects 
GitOrigin-RevId: d967da41250bb5945d5b8668b212d4a61b4f9d69
 2025-03-18 09:04:50 +00:00
Brian Gough
2d48c86e61 Merge pull request #24269 from overleaf/bg-backup-replication-smoothing 
extend backup scheduler to delay jobs for load levelling

GitOrigin-RevId: be62c458d3be464bf80ca590fcc107c2760aa061
 2025-03-13 09:06:28 +00:00
Brian Gough
a178c0f400 Merge pull request #24268 from overleaf/bg-backup-replication-lag-metrics 
add backup replication metrics

GitOrigin-RevId: a734435e3c6ce56350b2286bd218a5e2324d93a9
 2025-03-13 09:06:24 +00:00
Brian Gough
cf105cf01d Merge pull request #24267 from overleaf/bg-fix-init-option 
update backup script to use index when finding uninitialised projects

GitOrigin-RevId: 04ce0654b3d42de1c1a9bb542482c2dd53540628
 2025-03-13 09:06:20 +00:00
Brian Gough
3b93efdf5c Merge pull request #24256 from overleaf/bg-backup-fix-sample 
fix backup sample script

GitOrigin-RevId: 4bf4c15324d9b3b7ebd809f7b212e52d75ab216d
 2025-03-13 09:06:16 +00:00
Andrew Rumble
149b590413 Fix script name in usage prompt 
GitOrigin-RevId: ab4ed7479c96d3d1db82b23ef3c883706c61995e
 2025-03-12 09:05:44 +00:00
Andrew Rumble
282f5f92ff Ensure global blobs are loaded before verifying 
GitOrigin-RevId: 3f78c44e6b4d98fd6c469b0f4fbacc033b0a0868
 2025-03-12 09:05:39 +00:00
Andrew Rumble
afedce1b0e Add script for verifying sampled sets of projects 
GitOrigin-RevId: 23a599ea269a528201c67cda66bd2d00c0d376e4
 2025-03-12 09:05:35 +00:00