Commit graph

127 commits

Author SHA1 Message Date
Eric Mc Sween
1833bd3d00 Merge pull request #27126 from overleaf/em-fix-persist-changes
Fix chunk creation over a one change chunk

GitOrigin-RevId: aecae334849522975b83c77224ee27db64de4ed8
2025-07-16 08:05:40 +00:00
Brian Gough
8fab1b54a3 Merge pull request #27025 from overleaf/bg-delete-redis-buffer-when-project-deleted
delete redis buffer when project deleted

GitOrigin-RevId: eef7b6fdeb04cb556ae47794379d83e659f89b2e
2025-07-11 08:06:08 +00:00
Jakob Ackermann
d7a0cbefea [history-v1] fix typo in metrics name in redis chunk store (#26865)
GitOrigin-RevId: 5b37e3b22a3250fcfbd7411e635cc11ed38dcb27
2025-07-08 08:05:52 +00:00
Eric Mc Sween
735cc2272f Merge pull request #26505 from overleaf/em-persist-buffer-batch
Persist Redis buffer in batches

GitOrigin-RevId: 1de22807dae3554b3274ec103783b0868b1439d9
2025-06-30 08:05:25 +00:00
Andrew Rumble
903277c222 Handle rangeHashes
GitOrigin-RevId: 9b9a67d8e45f17e948d003aed33cef6128a25744
2025-06-25 08:06:50 +00:00
Andrew Rumble
34b674aa6f Handle errors when converting lazyfiles to eagerfiles
GitOrigin-RevId: 6ada48e2f2b7ffee9c8560856436dc5e2076099d
2025-06-25 08:06:45 +00:00
Andrew Rumble
6ed488cc65 Change raw mode to just fetch all raw content from backup
GitOrigin-RevId: 397060d63c824dbe688d187312dd78d1f3e4bf5d
2025-06-25 08:06:36 +00:00
Andrew Rumble
f2b0a982ac Use BackupBlobStore to get chunk content
GitOrigin-RevId: 91de53101ea289b43bdb08352aecc09ae89d0f74
2025-06-25 08:06:32 +00:00
Andrew Rumble
392037efd6 Improve types and export functions we want to use
GitOrigin-RevId: 25fd93214f88c425bee23d43de113cf754f89524
2025-06-25 08:06:18 +00:00
Andrew Rumble
3600aa4b75 Add script for restoring projects from backup
GitOrigin-RevId: b639d74989afa17638bbcba0f8ee34c978fcd5d9
2025-06-25 08:06:14 +00:00
Andrew Rumble
25a911d4cb Export loadChunk from verifier to allow code reuse
GitOrigin-RevId: fc5f2de50583ab2469e8ab3c86776c55501ba2e2
2025-06-25 08:06:09 +00:00
Eric Mc Sween
fd9fd9f0e7 Merge pull request #26545 from overleaf/jpa-fix-resync
[history-v1] use mongo projectId for initiating resync

GitOrigin-RevId: f93f2358695782fb222d23ba3720d98724b9a291
2025-06-19 08:06:45 +00:00
Eric Mc Sween
f68bf5a69f Merge pull request #26477 from overleaf/em-redis-buffer-resync
Handle invalid content hash when persisting changes

GitOrigin-RevId: 5259190396c8c261cad1abcd5de66314c1e871fb
2025-06-18 08:06:17 +00:00
Brian Gough
d189c91c59 Merge pull request #26378 from overleaf/bg-history-redis-fix-misc
add type check to scan.js

GitOrigin-RevId: a306ec841b8801a2a5eab8c35f246180206f7231
2025-06-13 08:06:27 +00:00
Brian Gough
eed6a982f7 Merge pull request #26377 from overleaf/bg-history-redis-remove-unwanted-parameters
remove unwanted parameters in queueChangesFake and queueChangesFakeOnlyIfExists functions

GitOrigin-RevId: 5946cd1f81db7076eb545b5a0aca28b81fa19be4
2025-06-13 08:06:22 +00:00
Brian Gough
2e4b57bf81 Merge pull request #26357 from overleaf/bg-history-redis-fix-history-buffer-level
remove incorrect parameter from queueChanges call in queueChangesFake function

GitOrigin-RevId: 4567be5eb5634c0c419f7f8758f1a6f27586a9af
2025-06-12 08:05:58 +00:00
Brian Gough
8a90173aa7 Merge pull request #26271 from overleaf/bg-history-redis-deployment-refactor
introduce history-v1 buffering levels

GitOrigin-RevId: 7709935a5ceb19ef6c5723ded647217b7399759a
2025-06-12 08:05:39 +00:00
Brian Gough
b290e93441 Merge pull request #26270 from overleaf/bg-history-redis-commit-change-manager
replace redis logic in persistChanges with new commitChanges method

GitOrigin-RevId: e06f9477b9d5548fa92ef87fb6e1f4f672001a35
2025-06-11 08:07:45 +00:00
Brian Gough
b946c2abff Merge pull request #26304 from overleaf/bg-history-redis-clear-persist-time-on-persist
add persist time handling to setPersistedVersion method

GitOrigin-RevId: 5e115b49116ee4604e3e478c206c7e9cf147cbc8
2025-06-11 08:07:32 +00:00
Brian Gough
2a833aa23a Merge pull request #26250 from overleaf/bg-history-redis-add-return-value-to-persistBuffer
provide return value from persistBuffer

GitOrigin-RevId: ba52ff42b91ffe9adc23ab0461fa836540735563
2025-06-11 08:06:54 +00:00
Brian Gough
fec6dde00f Merge pull request #26203 from overleaf/bg-history-redis-fix-loadAtVersion
Extend loadAtVersion to handle nonpersisted versions

GitOrigin-RevId: 22060605ea7bb89a8d4d61bafab8f63b94d59067
2025-06-11 08:06:50 +00:00
Brian Gough
c81cc4055e Merge pull request #26220 from overleaf/bg-history-redis-fix-loadAtVersion-startVersion
correct startVersion calculation in loadAtVersion

GitOrigin-RevId: b81c30dcab90b137169a4bddef3c22f44a957f68
2025-06-11 08:06:45 +00:00
Brian Gough
2d0706591b Merge pull request #26219 from overleaf/bg-history-redis-fix-loadAtTimestamp
correct startVersion calculation in loadAtTimestamp

GitOrigin-RevId: ad46aae47c0769943e787199d68e895cf139bb56
2025-06-11 08:06:41 +00:00
Brian Gough
f904933d68 Merge pull request #26180 from overleaf/bg-history-redis-add-queueChanges
add queueChanges method to history-v1

GitOrigin-RevId: fb6da79bd5ca40e7cbdcb077ad3a036cc5509ced
2025-06-11 08:06:36 +00:00
Brian Gough
2eb695f4c3 Merge pull request #26122 from overleaf/bg-history-redis-make-persist-buffer-consistent
make persistBuffer export consistent with other methods

GitOrigin-RevId: 24536e521e1d20ef63cc74bd9ba40e095025d512
2025-06-09 08:05:39 +00:00
Brian Gough
54c0eb7fdc Merge pull request #25958 from overleaf/bg-history-redis-check-persisted-version-on-update
prevent setPersistedVersion from setting an out of bounds version

GitOrigin-RevId: 9561b7b96399bed901db5c2ac20a0cdbf4c67395
2025-06-04 08:07:19 +00:00
Brian Gough
393cee7af5 Merge pull request #25993 from overleaf/bg-history-refactor-persist-buffer-limits
refactor persist buffer to add limits

GitOrigin-RevId: 4a40a7a8812acf5bb7f98bfd7b94d81ebe19fc57
2025-06-04 08:06:19 +00:00
Brian Gough
cb350ecc65 Merge pull request #25907 from overleaf/bg-history-redis-persist-buffer
add a `persistBuffer` method to history-v1

GitOrigin-RevId: 71a34e48e9ebe378e2f765f3216023e505a58a5d
2025-06-04 08:06:06 +00:00
Brian Gough
3296fc15da Merge pull request #25905 from overleaf/bg-history-redis-fix-import-path
fix import path for Job errors in history-v1

GitOrigin-RevId: f5f88bd34e713cd2ed78185ed4ce917e10d09caf
2025-05-29 08:06:06 +00:00
Brian Gough
b525a80d28 Merge pull request #25470 from overleaf/bg-history-redis-downgrade-job-related-errors
downgrade expected job errors in scanAndProcessDueItems

GitOrigin-RevId: 0a2689699bfc6512c5017c7f5e51ac4f80c409fe
2025-05-29 08:06:01 +00:00
Eric Mc Sween
c18b3f95b2 Merge pull request #25492 from overleaf/em-paginate-changes-1
Paginate history changes endpoint

GitOrigin-RevId: 2b48044d64244404efcd2e090b682c1f571a5567
2025-05-22 08:08:09 +00:00
Miguel Serrano
85533a36e9 [history-v1] Disable backups on CE/SP (#25591)
Disables backup when `backupStore` is not present,
as it's the case for CE/SP

GitOrigin-RevId: a920f041c639e599084fa97d2ef2643a01da70e3
2025-05-19 12:26:30 +00:00
Eric Mc Sween
732b1d146e Merge pull request #25456 from overleaf/em-concurrency-handling
Add consistency constraints to the chunk store and Redis buffer

GitOrigin-RevId: 6f983ff207a13d204645e343290c94443dc537b0
2025-05-13 08:07:36 +00:00
Brian Gough
9762cf95e3 Merge pull request #25463 from overleaf/bg-history-redis-fix-logging-in-expire-worker
reduce expire_redis_chunks log verbosity in production

GitOrigin-RevId: afcf6edc7154d49714bc60c276c129d70eaa49c7
2025-05-12 08:05:44 +00:00
Brian Gough
9e07549ecb Merge pull request #25449 from overleaf/bg-histoy-redis-refactor-expire-worker
refactor the expire worker to make it easier to extend

GitOrigin-RevId: 7b277b243ed51ab3b46316d98b7157af95a9e42b
2025-05-12 08:05:19 +00:00
Eric Mc Sween
391fca9e83 Merge pull request #25361 from overleaf/em-load-latest-raw
Rename loadLatestRaw() to getLatestChunkMetadata()

GitOrigin-RevId: e089dcfa57cbbc43df8666b51eca0d81d595a5a7
2025-05-09 08:06:14 +00:00
Eric Mc Sween
5717ea7f5c Merge pull request #25306 from overleaf/em-redis-buffer-read-operations
Add changes from Redis when reading chunks from the chunk store

GitOrigin-RevId: c0ebf0669b91eb2efc5d1091d025e81efdff9fe4
2025-05-09 08:06:09 +00:00
Brian Gough
dc73a18ca4 Merge pull request #25432 from overleaf/em-redis-buffer-strict-apply
Use strict validation for the Redis buffer

GitOrigin-RevId: 43e73af5deabbf3de9f5eed14f062acc5fa35e36
2025-05-09 08:05:50 +00:00
Brian Gough
9cf284aefa Merge pull request #25414 from overleaf/bg-history-only-log-content-mismatch-once-per-request
log warning for content hash mismatch only on first occurrence

GitOrigin-RevId: ff09f8c262461488bd564ea0644d414bb32ff17e
2025-05-09 08:05:46 +00:00
Brian Gough
6eada92966 Merge pull request #25406 from overleaf/em-content-hash-validation-resync
Resync project when content hash validation fails

GitOrigin-RevId: ea9b5a78f89c55276fd67835bc262717bc778e92
2025-05-09 08:05:12 +00:00
Brian Gough
ad94c29659 Merge pull request #25391 from overleaf/em-queue-changes-verification-rollout-stage-2
queue changes verification rollout stage 2

GitOrigin-RevId: c79a5a252c6fc8caf6fd164a31e6e360b6fc3e73
2025-05-08 08:06:27 +00:00
Brian Gough
ec91c120b1 Merge pull request #25284 from overleaf/em-queue-changes-verification
Exercise the Redis buffer when persisting changes

GitOrigin-RevId: a649b9808b6472e7c5dd9c8bfa6e3c98fb6ef4d4
2025-05-08 08:06:22 +00:00
Brian Gough
f9b36cd5be Merge pull request #25241 from overleaf/bg-remove-existing-chunk-buffer
remove existing chunk redis backend and chunk buffer

GitOrigin-RevId: 28fb02d1802312de6892e2fb7dd59191e3fc8914
2025-05-08 08:05:09 +00:00
Brian Gough
4464320757 Merge pull request #25224 from overleaf/em-disable-chunk-buffer
Bypass chunk buffer when loading the latest chunk

GitOrigin-RevId: 98a15b496b0d52802f9b61cefb60a7b8df653fb2
2025-05-02 08:05:17 +00:00
Brian Gough
19a804d5bf Merge pull request #25147 from overleaf/bg-history-buffer-use-persist-time
use persist time in history buffer

GitOrigin-RevId: 881c42f86c6cd3cc2ea8373af4371ccc1a89e9ed
2025-05-01 08:05:21 +00:00
Brian Gough
850da34778 Merge pull request #25086 from overleaf/bg-history-buffer-use-expire-time
add expire time to redis buffer in history-v1

GitOrigin-RevId: 3d74957c341e62e536dc60869a7ca71ac173e380
2025-04-29 08:05:14 +00:00
Brian Gough
612981bedb Merge pull request #24993 from overleaf/bg-history-buffer-use-cache-in-persist-changes
use chunkBuffer in persistChanges

GitOrigin-RevId: dd4cdf39ba53c3becf306119fed7eacfe67de15d
2025-04-25 08:05:12 +00:00
Brian Gough
a7466a7291 Merge pull request #24966 from overleaf/bg-history-buffer-optimised-get
add getCurrentChunkIfValid function

GitOrigin-RevId: e947a99ac928b58048a87cea0be1da34fcf3a9f8
2025-04-25 08:05:04 +00:00
Brian Gough
626416ed02 Merge pull request #24945 from overleaf/bg-redis-buffer-stats-script
add script for listing buffer stats from redis

GitOrigin-RevId: 7836563d51a5c6ded264d0e709d5cfcda70596e3
2025-04-24 08:05:49 +00:00
Eric Mc Sween
315bde6f1b Merge pull request #24968 from overleaf/em-chunks-concurrency-postgres
Handle concurrency during chunk extension in the Postgres backend

GitOrigin-RevId: fd706b73deacf141cbd478d3ed47f298e6c6db72
2025-04-18 08:05:37 +00:00