Brian Gough
34b397073f
Merge pull request #24053 from overleaf/bg-backup-queue-tweaks-II
...
additional backup queue tweaks
GitOrigin-RevId: b54e0c5f4a23bb9962c5062bd1b80f9f124c3c87
2025-03-05 09:05:15 +00:00
Brian Gough
c815106d62
Merge pull request #24037 from overleaf/bg-fix-backup-worker-errors
...
backup worker tweaks
GitOrigin-RevId: 7901bb0257c7c7320afd08ef965f124347a1de30
2025-03-04 09:05:12 +00:00
Brian Gough
1dbf5dca10
Merge pull request #24019 from overleaf/bg-backup-add-error-logging
...
add logging of backup errors in batch mode
GitOrigin-RevId: 96ae7a2354f82451b3bbe8c5459c4d4cf47e5459
2025-03-04 09:04:44 +00:00
Brian Gough
4720e8f2c5
Merge pull request #23989 from overleaf/bg-backup-worker-reduce-logging
...
reduce log level from info to debug for backup logging
GitOrigin-RevId: 10f541b0e0068aafd50710226fb8425527fee7c5
2025-03-03 09:05:22 +00:00
Brian Gough
8d0fa5be65
Merge pull request #23977 from overleaf/bg-backup-worker-load-global-blobs
...
ensure global blobs are loaded in backup worker
GitOrigin-RevId: b80791ef38e7fa8db364bdd3a9c461757778409d
2025-03-03 09:05:08 +00:00
Brian Gough
a8030f9e32
Merge pull request #23970 from overleaf/bg-change-history-worker-to-use-queue-redis
...
fix backup worker to use queues redis
GitOrigin-RevId: 4f278c12ea00ec98d9fa0a1430e1dd8bc8b40409
2025-03-03 09:04:40 +00:00
Brian Gough
b37b409994
Merge pull request #23946 from overleaf/bg-backup-queue-implementation
...
additional backup queue implementation
GitOrigin-RevId: 03754e57a6c6798a07dcca6a5248dec61b3cdc7a
2025-02-28 09:05:50 +00:00
Andrew Rumble
767591973c
Merge pull request #23905 from overleaf/ar-mocha-11
...
[all] Bump mocha everywhere
GitOrigin-RevId: bf5587047f9ceb88872cbd356c2029717bb8ca9d
2025-02-28 09:05:40 +00:00
Brian Gough
763e5ba82c
Merge pull request #23920 from overleaf/bg-backup-queue
...
initial history backup queue worker app
GitOrigin-RevId: e9647a2ec3caeb1fff843cead12164ba89fdb1f8
2025-02-28 09:04:46 +00:00
Antoine Clausse
fe03d8d7fd
Add shellcheck to web, server-pro, server-ce ( #23823 )
...
* Update shellcheck files to include non ".sh" files having the sh shebang
* Add shellcheck to root Makefile
* `make shellcheck_fix`
* Add shellcheck to server-ce and server-pro
* Exclude SC1091 (Not following) in SP/CE
* Fix errors in SP
* Fix errors in CE
* Update root shellcheck to ignore all failing rules. We can later remove rules one by one.
* Add shellcheck to web
* Add shellcheck step for server-ce and server-pro in cloudbuild.yaml
* Revert "Add shellcheck to root Makefile"
This reverts commit e0fa56f2
* Revert "`make shellcheck_fix`"
This reverts commit eb179245c109a9e742a7fdeeb75a4bdd03963587.
* `make shellcheck_fix` in server-ce
* Fix: Use $(...) notation instead of legacy backticked
```
In init_scripts/100_set_docker_host_ipaddress.sh line 5:
echo "`route -n | awk '/UG[ \t]/{print $2}'` dockerhost" >> /etc/hosts
^-- SC2006 (style): Use $(...) notation instead of legacy backticks `...`.
Did you mean:
echo "$(route -n | awk '/UG[ \t]/{print $2}') dockerhost" >> /etc/hosts
For more information:
https://www.shellcheck.net/wiki/SC2006 -- Use $(...) notation instead of le...
```
* `make shellcheck_fix` in web
* Manual corrections on autofix
* Update SHELLCHECK_FILES to Perl regex so it matches at start of file
GitOrigin-RevId: 815d53f319a3792efa22703beb75570de5977450
2025-02-26 09:05:02 +00:00
Brian Gough
6e55a9fa81
Merge pull request #23837 from overleaf/bg-redis-redact-config
...
sanitise password in redis config output
GitOrigin-RevId: 45d8bd13f01c3c4f138549c8a9f47dc0db29d0b0
2025-02-25 09:05:22 +00:00
Brian Gough
0acc067205
Merge pull request #23836 from overleaf/bg-backup-extend-debugging-script
...
add support for fetching chunks in backup debugging script
GitOrigin-RevId: 641298088d62148693525259748dcddf4d67d14b
2025-02-25 09:05:18 +00:00
Brian Gough
9d8f9e5358
Merge pull request #23835 from overleaf/bg-backup-show-errors
...
add error cause logging for missing chunks and blobs in backup comparison
GitOrigin-RevId: 9a1bf84b64621add71fb1d1e0cb013598b226bc8
2025-02-25 09:05:14 +00:00
Brian Gough
dd8a683e9e
Merge pull request #23834 from overleaf/bg-backup-graceful-shutdown
...
add graceful shutdown handling to backup script
GitOrigin-RevId: f0fb9a3867b26c7800edefab126fd05fbabdae02
2025-02-25 09:05:10 +00:00
Andrew Rumble
b0ed2efdf9
Merge pull request #23812 from overleaf/ar-allow-debugging-for-remove-backup-blobs-script-errors
...
[history-v1] Minor improvements to AWS cleanup script
GitOrigin-RevId: bd01a72169b9cbc018612173294cfbf8649a9fbd
2025-02-24 09:06:15 +00:00
Andrew Rumble
a0d79d5871
Merge pull request #23742 from overleaf/ar-script-for-removing-blobs-in-wrong-place
...
[history-v1] Script for deleting blobs backed up to wrong location
GitOrigin-RevId: 6abe1dba5164f8fa7d41c4ee3e4a07764e73b7e1
2025-02-24 09:05:53 +00:00
Brian Gough
981e91f012
Merge pull request #21763 from overleaf/bg-backup-script
...
initial script for running backups
GitOrigin-RevId: d22c373de30738d8080d40dce10790f0bdcb9f51
2025-02-24 09:04:32 +00:00
Miguel Serrano
2e72aa8a47
Merge pull request #23652 from overleaf/msm-update-path-to-regexp
...
[monorepo] Update express to 4.1.2
GitOrigin-RevId: b3db75257e1edc1a25a1a657def352ecca926920
2025-02-21 09:05:26 +00:00
Brian Gough
32983da29c
Merge pull request #23700 from overleaf/bg-history-redis
...
add redis to history-v1
GitOrigin-RevId: def900c4b560ea0a297b11b6c0ecc8cc8caa5b49
2025-02-21 09:04:06 +00:00
Brian Gough
d2738fda73
Merge pull request #23565 from overleaf/bg-fix-history-metadata-in-projects-collection
...
fix history metadata in projects collection
GitOrigin-RevId: 18c821ef5966a8470b24dfa60313b09bdda9707d
2025-02-14 09:03:33 +00:00
Jakob Ackermann
0a7b2004d2
[misc] silence logger when running tests ( #22243 )
...
* [misc] silence logger when running tests
* [misc] re-enable logging in some tests for scripts
* [misc] make it easy to turn on verbose logging for tests
```
LOG_LEVEL=debug make test_unit
LOG_LEVEL=debug make test_acceptance
```
GitOrigin-RevId: 219bc6d1f9cbdb89ddd7d94742920913ddde4514
2025-02-10 09:06:02 +00:00
Jakob Ackermann
64e8d2b8b3
[history-v1] fix backup deletion for postgres projects ( #23466 )
...
* [history-v1] fix backup deletion for postgres projects
* [history-v1] convert historyId to string on assignment
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
---------
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
GitOrigin-RevId: 5e1033972745a9b72606638f56ebf2147406cc39
2025-02-10 09:05:06 +00:00
Eric Mc Sween
9893f18ca5
Merge pull request #23447 from overleaf/em-reduce-validation-error-logs
...
Only log validation errors once per flush
GitOrigin-RevId: ee3f656c4c7c09fd7f3ff2462278c9aef81b9bb5
2025-02-07 09:06:10 +00:00
Eric Mc Sween
3c1f20a6d1
Merge pull request #23433 from overleaf/em-do-not-store-content-hashes
...
Do not store content hashes in chunks
GitOrigin-RevId: 65a255b92f9c4e216ad5a1fb5fb3fa5a6b9158c4
2025-02-07 09:06:03 +00:00
Eric Mc Sween
ce4c8a4e47
Merge pull request #23398 from overleaf/em-log-doc-hash-mismatches
...
Validate content hashes in history (log only)
GitOrigin-RevId: ed772fc4e4d0aa9e980f9693a759647bd937e13a
2025-02-07 09:05:59 +00:00
Eric Mc Sween
e145667a81
Merge pull request #23282 from overleaf/em-async-await-persist-changes
...
Convert the history changes import code to async/await
GitOrigin-RevId: 6421fcaaf3bac69a3404754f935b4902979b4689
2025-02-07 09:05:44 +00:00
Jakob Ackermann
668e9428f8
[history-v1] speed up tests for back-fill script ( #23440 )
...
* [history-v1] speed up tests for back-fill script
* [history-v1] move "should process nothing on re-run" test to the end
GitOrigin-RevId: d5180cfaefca3c46c440230eb3566539de1582bd
2025-02-07 09:05:16 +00:00
Jakob Ackermann
26321c5dba
[history-v1] block deletion of bucket prefix that is not a valid project prefix ( #22889 )
...
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
GitOrigin-RevId: a3aff76a2e299b2e2ed030e34da0631fce51ad4b
2025-02-07 09:05:01 +00:00
Brian Gough
25d9ef33c5
Merge pull request #23420 from overleaf/bg-add-plimit-to-history
...
add plimit to history-v1
GitOrigin-RevId: bae1b12b2031100144afc797094dba95524a8a3e
2025-02-06 09:04:48 +00:00
Jakob Ackermann
81aab1e159
[misc] fix logger.error(err) and logger.warn(err) calls ( #23369 )
...
* [misc] fix logger.error(err) calls
The signature is "logger.error({ err }, 'MESSAGE')".
* [project-history] remove duplicate logger.err calls in health check
The call-site is already logging any errors. Also, the logger.err call
signature was not quite right.
* [web] log userId when removeDropbox/removeGithub hook fails
* [misc] fix logger.warn(err) calls
The signature is "logger.warn({ err }, 'MESSAGE')".
* [misc] fix logger.error(OError.tag(err)) calls
* [web] make eslint happy
GitOrigin-RevId: 7f528113a3f7e9f6293b7d2d45adc079380325bb
2025-02-06 09:04:31 +00:00
Brian Gough
b2ff14f669
Merge pull request #23375 from overleaf/bg-add-id-checks-for-chunks
...
guard against non-postgres projectIds
GitOrigin-RevId: eab2024e4e893591f4b1c6a507b26d935273ae5f
2025-02-05 09:06:54 +00:00
Brian Gough
098d91f0bb
Merge pull request #23345 from overleaf/bg-write-latest-history-version-to-project
...
update project entry with history metadata on chunk creation
GitOrigin-RevId: dd19898f3d16e2e3360ff1bcccbf79f7dd27addb
2025-02-05 09:05:28 +00:00
Jakob Ackermann
3a4c5a0d0f
[history-v1] add readOnly lookup for raw chunks ( #23318 )
...
* [history-v1] add readOnly lookup for raw chunks
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
* [history-v1] reduce min poolsize for readOnly pool to 0
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
---------
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
GitOrigin-RevId: a711c4ee4f3ea3775bd090e620d1ef52689fa1f4
2025-02-04 09:04:52 +00:00
Jakob Ackermann
5404dcb93d
[misc] update build scripts ( #23316 )
...
* [misc] update build scripts
* [misc] make shellcheck happy
GitOrigin-RevId: 343655fda54aa7e835dd666dcd4218d61c58fd09
2025-02-04 09:04:37 +00:00
Brian Gough
8831fb9a18
Merge pull request #23169 from overleaf/bg-export-global-blobs
...
Add script to export global blobs to CSV
GitOrigin-RevId: 86c885981b150aae4541770a094c822e1fb364b9
2025-02-03 09:05:53 +00:00
Jakob Ackermann
c6c623da78
[project-history] script for fixing-up files/metadata with bulk resync ( #23184 )
...
* [history-v1] add cheap endpoint for checking time of last history write
The /raw endpoint skips the GCS lookup for the chunk.
* [project-history] script for fixing-up files/metadata with bulk resync
* [project-history] upgrade structure only resync when full sync is needed
* [project-history] start resync and process resync updates under lock
* [project-history] stop retrying during graceful shutdown
GitOrigin-RevId: 73184d5786e1d40f5b7e21f387fc37cf43f0ac2d
2025-02-03 09:05:43 +00:00
Antoine Clausse
272108a213
Run bin/update_node 20.18.0 20.18.2
( #23074 )
...
GitOrigin-RevId: 3f1ab14c4c36ea54408e0ce2c404a323bfe8d9a6
2025-01-24 09:06:05 +00:00
Brian Gough
d6e89c7338
Merge pull request #23058 from overleaf/bg-export-backed-up-blobs
...
export backed up blobs to csv file
GitOrigin-RevId: 5a176055dc85c56ced64cbf13aa705b56071cdae
2025-01-24 09:05:31 +00:00
Andrew Rumble
47db5c1236
Merge pull request #23039 from overleaf/ar-bump-mongo-6.12
...
[many] Bump mongodb to 6.12.0 universally
GitOrigin-RevId: 6df934518e95217b6157ca5a41cf8acf4beb7689
2025-01-23 09:06:13 +00:00
Jakob Ackermann
9c377b74c1
[history-v1] add flag for back-filling hashes for projectIds from file ( #23005 )
...
* [history-v1] delay process exit to give logging time to flush
* [history-v1] add flag for back-filling hashes for projectIds from file
GitOrigin-RevId: 887a1e1c72d6f5a13bfc8d0e54023afbf5bc671c
2025-01-22 09:05:21 +00:00
Jakob Ackermann
6884338a34
[history-v1] fix remaining files with missing hashes ( #22985 )
...
GitOrigin-RevId: df382732954538f9db177180aee0a44c5d5679ca
2025-01-22 09:05:16 +00:00
Jakob Ackermann
ab4d1e0986
Merge pull request #22870 from overleaf/jpa-back-fill-fix-up
...
[history-v1] add script for fixing up back-fill errors
GitOrigin-RevId: 118992a32c1f6da4289cd35399ddd07a741da4ee
2025-01-20 09:05:14 +00:00
Jakob Ackermann
98e0ed6a06
Merge pull request #22894 from overleaf/jpa-large-chunk
...
[history-v1] add test case for large text file
GitOrigin-RevId: 5d7cc37d74c67e1927785ea9c544d897491e6abd
2025-01-16 09:06:02 +00:00
Jakob Ackermann
70fd6cacbc
Merge pull request #22711 from overleaf/jpa-gzip
...
[history-v1] compress blobs before sending them to AWS
GitOrigin-RevId: 1ca1dda6f36738fbabbf00fdab62b86230b9e4f9
2025-01-08 09:04:57 +00:00
Jakob Ackermann
8171ffafc5
Merge pull request #22710 from overleaf/jpa-fix-tests
...
[history-v1] fix tests
GitOrigin-RevId: dd16d078a4bfea108e9035c0a83f4fd5c3dd715b
2025-01-08 09:04:52 +00:00
Andrew Rumble
2262d03a21
Merge pull request #22648 from overleaf/ar-store-backup-blobs-by-history-id
...
Use historyId when constructing path for backup
GitOrigin-RevId: 954576b509d5e78511b5008fb7d74e0bc5fa45fd
2024-12-24 09:04:52 +00:00
Andrew Rumble
31e8a908ee
Merge pull request #22334 from overleaf/ar-guard-against-integer-like-strings-when-working-with-postgres
...
[history-v1] Guard against non-postgres projectIds
GitOrigin-RevId: 5bf75c67424297f52f2abd9d0f0f14a0f79f8921
2024-12-13 09:04:59 +00:00
Andrew Rumble
a92a37bc3c
Merge pull request #22466 from overleaf/ar-backup-files-when-inserting
...
[history-v1] backup files when inserting
GitOrigin-RevId: e636bce178604978c6d41c083bf671795d20b5a1
2024-12-13 09:04:54 +00:00
Andrew Rumble
52254b5695
Merge pull request #22459 from overleaf/revert-22392-ar-backup-files-when-inserting
...
Revert "[history-v1] backup files when inserting"
GitOrigin-RevId: f21d49dbc8909ab93bdde78c321672124bb13697
2024-12-12 09:05:27 +00:00
Andrew Rumble
6404e3047d
Merge pull request #22392 from overleaf/ar-backup-files-when-inserting
...
[history-v1] backup files when inserting
GitOrigin-RevId: 1649b2828899d67ee37c0ac331917c6d5424c803
2024-12-12 09:05:11 +00:00