+
- Figure 1: A screenshot of a project being edited in Overleaf Community Edition. + Figure 1: A screenshot of a project being edited in Overleaf Extended Community Edition.
## Community Edition -[Overleaf](https://www.overleaf.com) is an open-source online real-time collaborative LaTeX editor. We run a hosted version at [www.overleaf.com](https://www.overleaf.com), but you can also run your own local version, and contribute to the development of Overleaf. +[Overleaf](https://www.overleaf.com) is an open-source online real-time collaborative LaTeX editor. Overleaf runs a hosted version at [www.overleaf.com](https://www.overleaf.com), but you can also run your own local version, and contribute to the development of Overleaf. + +## Extended Community Edition + +The present "extended" version of Overleaf CE includes: + +- Template Gallery +- Sandboxed Compiles with TeX Live image selection +- LDAP authentication +- SAML authentication +- OpenID Connect authentication +- Real-time track changes and comments +- Autocomplete of reference keys +- Symbol Palette +- "From External URL" feature > [!CAUTION] -> Overleaf Community Edition is intended for use in environments where **all** users are trusted. Community Edition is **not** appropriate for scenarios where isolation of users is required due to Sandbox Compiles not being available. When not using Sandboxed Compiles, users have full read and write access to the `sharelatex` container resources (filesystem, network, environment variables) when running LaTeX compiles. +> Overleaf Community Edition is intended for use in environments where **all** users are trusted. Community Edition is **not** appropriate for scenarios where isolation of users is required due to Sandbox Compiles not being available. When not using Sandboxed Compiles, users have full read and write access to the `sharelatex` container resources (filesystem, network, environment variables) when running LaTeX compiles. +Therefore, in any environment where not all users can be fully trusted, it is strongly recommended to enable the Sandboxed Compiles feature available in the Extended Community Edition. -For more information on Sandbox Compiles check out our [documentation](https://docs.overleaf.com/on-premises/configuration/overleaf-toolkit/server-pro-only-configuration/sandboxed-compiles). +For more information on Sandbox Compiles check out Overleaf [documentation](https://docs.overleaf.com/on-premises/configuration/overleaf-toolkit/server-pro-only-configuration/sandboxed-compiles). ## Enterprise -If you want help installing and maintaining Overleaf in your lab or workplace, we offer an officially supported version called [Overleaf Server Pro](https://www.overleaf.com/for/enterprises). It also includes more features for security (SSO with LDAP or SAML), administration and collaboration (e.g. tracked changes). [Find out more!](https://www.overleaf.com/for/enterprises) - -## Keeping up to date - -Sign up to the [mailing list](https://mailchi.mp/overleaf.com/community-edition-and-server-pro) to get updates on Overleaf releases and development. +If you want help installing and maintaining Overleaf in your lab or workplace, Overleaf offers an officially supported version called [Overleaf Server Pro](https://www.overleaf.com/for/enterprises). ## Installation -We have detailed installation instructions in the [Overleaf Toolkit](https://github.com/overleaf/toolkit/). - -## Upgrading - -If you are upgrading from a previous version of Overleaf, please see the [Release Notes section on the Wiki](https://github.com/overleaf/overleaf/wiki#release-notes) for all of the versions between your current version and the version you are upgrading to. +Detailed installation instructions can be found in the [Overleaf Toolkit](https://github.com/overleaf/toolkit/). +Configuration details and release history for the Extended Community Edition can be found on the [Extended CE Wiki Page](https://github.com/yu-i-i/overleaf-cep/wiki). ## Overleaf Docker Image This repo contains two dockerfiles, [`Dockerfile-base`](server-ce/Dockerfile-base), which builds the -`sharelatex/sharelatex-base` image, and [`Dockerfile`](server-ce/Dockerfile) which builds the -`sharelatex/sharelatex` (or "community") image. +`sharelatex/sharelatex-base:ext-ce` image, and [`Dockerfile`](server-ce/Dockerfile) which builds the +`sharelatex/sharelatex:ext-ce` image. The Base image generally contains the basic dependencies like `wget`, plus `texlive`. -We split this out because it's a pretty heavy set of +This is split out because it's a pretty heavy set of dependencies, and it's nice to not have to rebuild all of that every time. The `sharelatex/sharelatex` image extends the base image and adds the actual Overleaf code @@ -59,20 +67,16 @@ and services. Use `make build-base` and `make build-community` from `server-ce/` to build these images. -We use the [Phusion base-image](https://github.com/phusion/baseimage-docker) -(which is extended by our `base` image) to provide us with a VM-like container +The [Phusion base-image](https://github.com/phusion/baseimage-docker) +(which is extended by the `base` image) provides a VM-like container in which to run the Overleaf services. Baseimage uses the `runit` service -manager to manage services, and we add our init-scripts from the `server-ce/runit` -folder. - - -## Contributing - -Please see the [CONTRIBUTING](CONTRIBUTING.md) file for information on contributing to the development of Overleaf. +manager to manage services, and init scripts from the `server-ce/runit` +folder are added. ## Authors -[The Overleaf Team](https://www.overleaf.com/about) +[The Overleaf Team](https://www.overleaf.com/about) +[yu-i-i](https://github.com/yu-i-i/overleaf-cep) — Extensions for CE unless otherwise noted ## License diff --git a/develop/README.md b/develop/README.md index 8d45383c23..8e3f89862c 100644 --- a/develop/README.md +++ b/develop/README.md @@ -77,6 +77,7 @@ each service: | `filestore` | 9235 | | `notifications` | 9236 | | `real-time` | 9237 | +| `references` | 9238 | | `history-v1` | 9239 | | `project-history` | 9240 | diff --git a/develop/dev.env b/develop/dev.env index 6ebbbb1ffd..b003e4e0eb 100644 --- a/develop/dev.env +++ b/develop/dev.env @@ -15,6 +15,7 @@ PROJECT_HISTORY_HOST=project-history QUEUES_REDIS_HOST=redis REALTIME_HOST=real-time REDIS_HOST=redis +REFERENCES_HOST=references SESSION_SECRET=foo V1_HISTORY_HOST=history-v1 WEBPACK_HOST=webpack diff --git a/develop/docker-compose.dev.yml b/develop/docker-compose.dev.yml index 3d2fca7e0b..65e15ef07b 100644 --- a/develop/docker-compose.dev.yml +++ b/develop/docker-compose.dev.yml @@ -112,6 +112,17 @@ services: - ../services/real-time/app.js:/overleaf/services/real-time/app.js - ../services/real-time/config:/overleaf/services/real-time/config + references: + command: ["node", "--watch", "app.js"] + environment: + - NODE_OPTIONS=--inspect=0.0.0.0:9229 + ports: + - "127.0.0.1:9238:9229" + volumes: + - ../services/references/app:/overleaf/services/references/app + - ../services/references/config:/overleaf/services/references/config + - ../services/references/app.js:/overleaf/services/references/app.js + web: command: ["node", "--watch", "app.mjs", "--watch-locales"] environment: diff --git a/develop/docker-compose.yml b/develop/docker-compose.yml index 7161e0686a..e5b84c38b3 100644 --- a/develop/docker-compose.yml +++ b/develop/docker-compose.yml @@ -123,7 +123,7 @@ services: dockerfile: services/real-time/Dockerfile env_file: - dev.env - + redis: image: redis:5 ports: @@ -131,6 +131,13 @@ services: volumes: - redis-data:/data + references: + build: + context: .. + dockerfile: services/references/Dockerfile + env_file: + - dev.env + web: build: context: .. @@ -140,7 +147,7 @@ services: - dev.env environment: - APP_NAME=Overleaf Community Edition - - ENABLED_LINKED_FILE_TYPES=project_file,project_output_file + - ENABLED_LINKED_FILE_TYPES=project_file,project_output_file,url - EMAIL_CONFIRMATION_DISABLED=true - NODE_ENV=development - OVERLEAF_ALLOW_PUBLIC_ACCESS=true @@ -161,6 +168,7 @@ services: - notifications - project-history - real-time + - references webpack: build: diff --git a/doc/screenshot.png b/doc/screenshot.png index 1c1f339630..92633192a5 100644 Binary files a/doc/screenshot.png and b/doc/screenshot.png differ diff --git a/docker-compose.yml b/docker-compose.yml index e257716789..962adfb5d8 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -32,7 +32,7 @@ services: OVERLEAF_REDIS_HOST: redis REDIS_HOST: redis - ENABLED_LINKED_FILE_TYPES: 'project_file,project_output_file' + ENABLED_LINKED_FILE_TYPES: 'project_file,project_output_file,url' # Enables Thumbnail generation using ImageMagick ENABLE_CONVERSIONS: 'true' diff --git a/package-lock.json b/package-lock.json index 2b3a5868a2..d9d8285618 100644 --- a/package-lock.json +++ b/package-lock.json @@ -35581,6 +35581,7 @@ "resolved": "https://registry.npmjs.org/request/-/request-2.88.2.tgz", "integrity": "sha512-MsvtOrfG9ZcrOwAW+Qi+F6HbD0CWXEh9ou77uOb7FM2WPhwT7smM833PzanhJLsgXjN89Ir6V2PczXNnMpwKhw==", "deprecated": "request has been deprecated, see https://github.com/request/request/issues/3142", + "license": "Apache-2.0", "dependencies": { "aws-sign2": "~0.7.0", "aws4": "^1.8.0", @@ -35638,15 +35639,15 @@ } }, "node_modules/request/node_modules/tough-cookie": { - "version": "2.5.0", - "resolved": "https://registry.npmjs.org/tough-cookie/-/tough-cookie-2.5.0.tgz", - "integrity": "sha512-nlLsUzgm1kfLXSXfRZMc1KLAugd4hqJHDTvc2hDIwS3mZAfMEuMbc03SujMF+GEcpaX/qboeycw6iO8JwVv2+g==", + "version": "5.1.2", + "resolved": "https://registry.npmjs.org/tough-cookie/-/tough-cookie-5.1.2.tgz", + "integrity": "sha512-FVDYdxtnj0G6Qm/DhNPSb8Ju59ULcup3tuJxkFb5K8Bv2pUXILbf0xZWU8PX8Ov19OXljbUyveOFwRMwkXzO+A==", + "license": "BSD-3-Clause", "dependencies": { - "psl": "^1.1.28", - "punycode": "^2.1.1" + "tldts": "^6.1.32" }, "engines": { - "node": ">=0.8" + "node": ">=16" } }, "node_modules/requestretry": { @@ -39612,6 +39613,24 @@ "tlds": "bin.js" } }, + "node_modules/tldts": { + "version": "6.1.86", + "resolved": "https://registry.npmjs.org/tldts/-/tldts-6.1.86.tgz", + "integrity": "sha512-WMi/OQ2axVTf/ykqCQgXiIct+mSQDFdH2fkwhPwgEwvJ1kSzZRiinb0zF2Xb8u4+OqPChmyI6MEu4EezNJz+FQ==", + "license": "MIT", + "dependencies": { + "tldts-core": "^6.1.86" + }, + "bin": { + "tldts": "bin/cli.js" + } + }, + "node_modules/tldts-core": { + "version": "6.1.86", + "resolved": "https://registry.npmjs.org/tldts-core/-/tldts-core-6.1.86.tgz", + "integrity": "sha512-Je6p7pkk+KMzMv2XXKmAE3McmolOQFdxkKw0R8EYNr7sELW46JqnNeTX8ybPiQgvg1ymCoF8LXs5fzFaZvJPTA==", + "license": "MIT" + }, "node_modules/tmp": { "version": "0.2.3", "resolved": "https://registry.npmjs.org/tmp/-/tmp-0.2.3.tgz", diff --git a/package.json b/package.json index 388b750c3d..44fffc4664 100644 --- a/package.json +++ b/package.json @@ -33,6 +33,9 @@ "path-to-regexp": "3.3.0", "body-parser": "1.20.3", "multer": "2.0.1" + }, + "request@2.88.2": { + "tough-cookie": "5.1.2" } }, "scripts": { diff --git a/patches/@node-saml+node-saml+4.0.5.patch b/patches/@node-saml+node-saml+4.0.5.patch new file mode 100644 index 0000000000..81fd700b31 --- /dev/null +++ b/patches/@node-saml+node-saml+4.0.5.patch @@ -0,0 +1,23 @@ +diff --git a/node_modules/@node-saml/node-saml/lib/saml.js b/node_modules/@node-saml/node-saml/lib/saml.js +index fba15b9..a5778cb 100644 +--- a/node_modules/@node-saml/node-saml/lib/saml.js ++++ b/node_modules/@node-saml/node-saml/lib/saml.js +@@ -336,7 +336,8 @@ class SAML { + const requestOrResponse = request || response; + (0, utility_1.assertRequired)(requestOrResponse, "either request or response is required"); + let buffer; +- if (this.options.skipRequestCompression) { ++ // logout requestOrResponse must be compressed anyway ++ if (this.options.skipRequestCompression && operation !== "logout") { + buffer = Buffer.from(requestOrResponse, "utf8"); + } + else { +@@ -495,7 +496,7 @@ class SAML { + try { + xml = Buffer.from(container.SAMLResponse, "base64").toString("utf8"); + doc = await (0, xml_1.parseDomFromString)(xml); +- const inResponseToNodes = xml_1.xpath.selectAttributes(doc, "/*[local-name()='Response']/@InResponseTo"); ++ const inResponseToNodes = xml_1.xpath.selectAttributes(doc, "/*[local-name()='Response' or local-name()='LogoutResponse']/@InResponseTo"); + if (inResponseToNodes) { + inResponseTo = inResponseToNodes.length ? inResponseToNodes[0].nodeValue : null; + await this.validateInResponseTo(inResponseTo); diff --git a/patches/ldapauth-fork+4.3.3.patch b/patches/ldapauth-fork+4.3.3.patch new file mode 100644 index 0000000000..4d31210c9d --- /dev/null +++ b/patches/ldapauth-fork+4.3.3.patch @@ -0,0 +1,64 @@ +diff --git a/node_modules/ldapauth-fork/lib/ldapauth.js b/node_modules/ldapauth-fork/lib/ldapauth.js +index 85ecf36a8b..a7d07e0f78 100644 +--- a/node_modules/ldapauth-fork/lib/ldapauth.js ++++ b/node_modules/ldapauth-fork/lib/ldapauth.js +@@ -69,6 +69,7 @@ function LdapAuth(opts) { + this.opts.bindProperty || (this.opts.bindProperty = 'dn'); + this.opts.groupSearchScope || (this.opts.groupSearchScope = 'sub'); + this.opts.groupDnProperty || (this.opts.groupDnProperty = 'dn'); ++ this.opts.tlsStarted = false; + + EventEmitter.call(this); + +@@ -108,21 +109,7 @@ function LdapAuth(opts) { + this._userClient.on('error', this._handleError.bind(this)); + + var self = this; +- if (this.opts.starttls) { +- // When starttls is enabled, this callback supplants the 'connect' callback +- this._adminClient.starttls(this.opts.tlsOptions, this._adminClient.controls, function(err) { +- if (err) { +- self._handleError(err); +- } else { +- self._onConnectAdmin(); +- } +- }); +- this._userClient.starttls(this.opts.tlsOptions, this._userClient.controls, function(err) { +- if (err) { +- self._handleError(err); +- } +- }); +- } else if (opts.reconnect) { ++ if (opts.reconnect && !this.opts.starttls) { + this.once('_installReconnectListener', function() { + self.log && self.log.trace('install reconnect listener'); + self._adminClient.on('connect', function() { +@@ -384,6 +371,28 @@ LdapAuth.prototype._findGroups = function(user, callback) { + */ + LdapAuth.prototype.authenticate = function(username, password, callback) { + var self = this; ++ if (this.opts.starttls && !this.opts.tlsStarted) { ++ // When starttls is enabled, this callback supplants the 'connect' callback ++ this._adminClient.starttls(this.opts.tlsOptions, this._adminClient.controls, function (err) { ++ if (err) { ++ self._handleError(err); ++ } else { ++ self._onConnectAdmin(function(){self._handleAuthenticate(username, password, callback);}); ++ } ++ }); ++ this._userClient.starttls(this.opts.tlsOptions, this._userClient.controls, function (err) { ++ if (err) { ++ self._handleError(err); ++ } ++ }); ++ } else { ++ self._handleAuthenticate(username, password, callback); ++ } ++}; ++ ++LdapAuth.prototype._handleAuthenticate = function (username, password, callback) { ++ this.opts.tlsStarted = true; ++ var self = this; + + if (typeof password === 'undefined' || password === null || password === '') { + return callback(new Error('no password given')); diff --git a/server-ce/Makefile b/server-ce/Makefile index 853a99d05e..8e2fe7d14f 100644 --- a/server-ce/Makefile +++ b/server-ce/Makefile @@ -24,6 +24,7 @@ build-base: --cache-from $(OVERLEAF_BASE_BRANCH) \ --tag $(OVERLEAF_BASE_TAG) \ --tag $(OVERLEAF_BASE_BRANCH) \ + --network=host \ $(MONOREPO_ROOT) @@ -39,6 +40,7 @@ build-community: --file Dockerfile \ --tag $(OVERLEAF_TAG) \ --tag $(OVERLEAF_BRANCH) \ + --network=host \ $(MONOREPO_ROOT) SHELLCHECK_OPTS = \ diff --git a/server-ce/config/env.sh b/server-ce/config/env.sh index b12ca242d3..81cebe4caa 100644 --- a/server-ce/config/env.sh +++ b/server-ce/config/env.sh @@ -9,5 +9,6 @@ export HISTORY_V1_HOST=127.0.0.1 export NOTIFICATIONS_HOST=127.0.0.1 export PROJECT_HISTORY_HOST=127.0.0.1 export REALTIME_HOST=127.0.0.1 +export REFERENCES_HOST=127.0.0.1 export WEB_HOST=127.0.0.1 export WEB_API_HOST=127.0.0.1 diff --git a/server-ce/runit/references-overleaf/run b/server-ce/runit/references-overleaf/run new file mode 100755 index 0000000000..875023df9f --- /dev/null +++ b/server-ce/runit/references-overleaf/run @@ -0,0 +1,12 @@ +#!/bin/bash + +NODE_PARAMS="" +if [ "$DEBUG_NODE" == "true" ]; then + echo "running debug - references" + NODE_PARAMS="--inspect=0.0.0.0:30560" +fi + +source /etc/overleaf/env.sh +export LISTEN_ADDRESS=127.0.0.1 + +exec /sbin/setuser www-data /usr/bin/node $NODE_PARAMS /overleaf/services/references/app.js >> /var/log/overleaf/references.log 2>&1 diff --git a/server-ce/services.js b/server-ce/services.js index d0b0a9c076..e0282f3bad 100644 --- a/server-ce/services.js +++ b/server-ce/services.js @@ -29,6 +29,9 @@ module.exports = [ { name: 'project-history', }, + { + name: 'references', + }, { name: 'history-v1', }, diff --git a/server-ce/test/Makefile b/server-ce/test/Makefile index 6c56b7e8fe..fb7c980293 100644 --- a/server-ce/test/Makefile +++ b/server-ce/test/Makefile @@ -21,9 +21,11 @@ test-e2e-native: test-e2e: docker compose build host-admin + docker compose up -d host-admin docker compose up --no-log-prefix --exit-code-from=e2e e2e test-e2e-open: + docker compose up -d host-admin docker compose up --no-log-prefix --exit-code-from=e2e-open e2e-open clean: diff --git a/server-ce/test/docker-compose.yml b/server-ce/test/docker-compose.yml index 029b73fc62..d16c5e2b71 100644 --- a/server-ce/test/docker-compose.yml +++ b/server-ce/test/docker-compose.yml @@ -20,7 +20,7 @@ services: OVERLEAF_EMAIL_SMTP_HOST: 'mailtrap' OVERLEAF_EMAIL_SMTP_PORT: '25' OVERLEAF_EMAIL_SMTP_IGNORE_TLS: 'true' - ENABLED_LINKED_FILE_TYPES: 'project_file,project_output_file' + ENABLED_LINKED_FILE_TYPES: 'project_file,project_output_file,url' ENABLE_CONVERSIONS: 'true' EMAIL_CONFIRMATION_DISABLED: 'true' healthcheck: @@ -35,7 +35,7 @@ services: MAILTRAP_PASSWORD: 'password-for-mailtrap' mongo: - image: mongo:6.0 + image: mongo:8.0.11 command: '--replSet overleaf' volumes: - ../bin/shared/mongodb-init-replica-set.js:/docker-entrypoint-initdb.d/mongodb-init-replica-set.js diff --git a/server-ce/test/editor.spec.ts b/server-ce/test/editor.spec.ts index d0060518de..3e57b94f8f 100644 --- a/server-ce/test/editor.spec.ts +++ b/server-ce/test/editor.spec.ts @@ -2,6 +2,7 @@ import { createNewFile, createProject, openProjectById, + testNewFileUpload, } from './helpers/project' import { isExcludedBySharding, startWith } from './helpers/config' import { ensureUserExists, login } from './helpers/login' @@ -119,24 +120,7 @@ describe('editor', () => { cy.get('button').contains('New file').click({ force: true }) }) - it('can upload file', () => { - const name = `${uuid()}.txt` - const content = `Test File Content ${name}` - cy.get('button').contains('Upload').click({ force: true }) - cy.get('input[type=file]') - .first() - .selectFile( - { - contents: Cypress.Buffer.from(content), - fileName: name, - lastModified: Date.now(), - }, - { force: true } - ) - // force: The file-tree pane is too narrow to display the full name. - cy.findByTestId('file-tree').findByText(name).click({ force: true }) - cy.findByText(content) - }) + testNewFileUpload() it('should not display import from URL', () => { cy.findByText('From external URL').should('not.exist') diff --git a/server-ce/test/filestore-migration.spec.ts b/server-ce/test/filestore-migration.spec.ts new file mode 100644 index 0000000000..25875ad374 --- /dev/null +++ b/server-ce/test/filestore-migration.spec.ts @@ -0,0 +1,104 @@ +import { ensureUserExists, login } from './helpers/login' +import { + createProject, + openProjectById, + prepareFileUploadTest, +} from './helpers/project' +import { isExcludedBySharding, startWith } from './helpers/config' +import { prepareWaitForNextCompileSlot } from './helpers/compile' +import { beforeWithReRunOnTestRetry } from './helpers/beforeWithReRunOnTestRetry' +import { v4 as uuid } from 'uuid' +import { purgeFilestoreData, runScript } from './helpers/hostAdminClient' + +describe('filestore migration', function () { + if (isExcludedBySharding('CE_CUSTOM_3')) return + startWith({ withDataDir: true, resetData: true, vars: {} }) + ensureUserExists({ email: 'user@example.com' }) + + let projectName: string + let projectId: string + let waitForCompileRateLimitCoolOff: (fn: () => void) => void + const previousBinaryFiles: (() => void)[] = [] + beforeWithReRunOnTestRetry(function () { + projectName = `project-${uuid()}` + login('user@example.com') + createProject(projectName, { type: 'Example project' }).then( + id => (projectId = id) + ) + let queueReset + ;({ waitForCompileRateLimitCoolOff, queueReset } = + prepareWaitForNextCompileSlot()) + queueReset() + previousBinaryFiles.push(prepareFileUploadTest(true)) + }) + + beforeEach(() => { + login('user@example.com') + waitForCompileRateLimitCoolOff(() => { + openProjectById(projectId) + }) + }) + + function checkFilesAreAccessible() { + it('can upload new binary file and read previous uploads', function () { + previousBinaryFiles.push(prepareFileUploadTest(true)) + for (const check of previousBinaryFiles) { + check() + } + }) + + it('renders frog jpg', () => { + cy.findByTestId('file-tree').findByText('frog.jpg').click() + cy.get('[alt="frog.jpg"]') + .should('be.visible') + .and('have.prop', 'naturalWidth') + .should('be.greaterThan', 0) + }) + } + + describe('OVERLEAF_FILESTORE_MIGRATION_LEVEL not set', function () { + startWith({ withDataDir: true, vars: {} }) + checkFilesAreAccessible() + }) + + describe('OVERLEAF_FILESTORE_MIGRATION_LEVEL=0', function () { + startWith({ + withDataDir: true, + vars: { OVERLEAF_FILESTORE_MIGRATION_LEVEL: '0' }, + }) + checkFilesAreAccessible() + + describe('OVERLEAF_FILESTORE_MIGRATION_LEVEL=1', function () { + startWith({ + withDataDir: true, + vars: { OVERLEAF_FILESTORE_MIGRATION_LEVEL: '1' }, + }) + checkFilesAreAccessible() + + describe('OVERLEAF_FILESTORE_MIGRATION_LEVEL=2', function () { + startWith({ + withDataDir: true, + vars: { OVERLEAF_FILESTORE_MIGRATION_LEVEL: '1' }, + }) + before(async function () { + await runScript({ + cwd: 'services/history-v1', + script: 'storage/scripts/back_fill_file_hash.mjs', + }) + }) + startWith({ + withDataDir: true, + vars: { OVERLEAF_FILESTORE_MIGRATION_LEVEL: '2' }, + }) + checkFilesAreAccessible() + + describe('purge filestore data', function () { + before(async function () { + await purgeFilestoreData() + }) + checkFilesAreAccessible() + }) + }) + }) + }) +}) diff --git a/server-ce/test/helpers/config.ts b/server-ce/test/helpers/config.ts index 030e70ceb5..78e81be1f7 100644 --- a/server-ce/test/helpers/config.ts +++ b/server-ce/test/helpers/config.ts @@ -9,6 +9,7 @@ export function isExcludedBySharding( | 'CE_DEFAULT' | 'CE_CUSTOM_1' | 'CE_CUSTOM_2' + | 'CE_CUSTOM_3' | 'PRO_DEFAULT_1' | 'PRO_DEFAULT_2' | 'PRO_CUSTOM_1' diff --git a/server-ce/test/helpers/hostAdminClient.ts b/server-ce/test/helpers/hostAdminClient.ts index cafeaa2db6..dadfe2b059 100644 --- a/server-ce/test/helpers/hostAdminClient.ts +++ b/server-ce/test/helpers/hostAdminClient.ts @@ -85,6 +85,12 @@ export async function getRedisKeys() { return stdout.split('\n') } +export async function purgeFilestoreData() { + await fetchJSON(`${hostAdminURL}/data/user_files`, { + method: 'DELETE', + }) +} + async function sleep(ms: number) { return new Promise(resolve => { setTimeout(resolve, ms) diff --git a/server-ce/test/helpers/project.ts b/server-ce/test/helpers/project.ts index abcce3f9b2..4b3197afed 100644 --- a/server-ce/test/helpers/project.ts +++ b/server-ce/test/helpers/project.ts @@ -216,3 +216,43 @@ export function createNewFile() { return fileName } + +export function prepareFileUploadTest(binary = false) { + const name = `${uuid()}.txt` + const content = `Test File Content ${name}${binary ? ' \x00' : ''}` + cy.get('button').contains('Upload').click({ force: true }) + cy.get('input[type=file]') + .first() + .selectFile( + { + contents: Cypress.Buffer.from(content), + fileName: name, + lastModified: Date.now(), + }, + { force: true } + ) + + // wait for the upload to finish + cy.findByRole('treeitem', { name }) + + return function check() { + cy.findByRole('treeitem', { name }).click() + if (binary) { + cy.findByText(content).should('not.have.class', 'cm-line') + } else { + cy.findByText(content).should('have.class', 'cm-line') + } + } +} + +export function testNewFileUpload() { + it('can upload text file', () => { + const check = prepareFileUploadTest(false) + check() + }) + + it('can upload binary file', () => { + const check = prepareFileUploadTest(true) + check() + }) +} diff --git a/server-ce/test/host-admin.js b/server-ce/test/host-admin.js index f73209d58f..b3dcd72b1f 100644 --- a/server-ce/test/host-admin.js +++ b/server-ce/test/host-admin.js @@ -29,6 +29,17 @@ const IMAGES = { PRO: process.env.IMAGE_TAG_PRO.replace(/:.+/, ''), } +function defaultDockerComposeOverride() { + return { + services: { + sharelatex: { + environment: {}, + }, + 'git-bridge': {}, + }, + } +} + let previousConfig = '' function readDockerComposeOverride() { @@ -38,14 +49,7 @@ function readDockerComposeOverride() { if (error.code !== 'ENOENT') { throw error } - return { - services: { - sharelatex: { - environment: {}, - }, - 'git-bridge': {}, - }, - } + return defaultDockerComposeOverride } } @@ -77,12 +81,21 @@ app.use(bodyParser.json()) app.use((req, res, next) => { // Basic access logs console.log(req.method, req.url, req.body) + const json = res.json + res.json = body => { + console.log(req.method, req.url, req.body, '->', body) + json.call(res, body) + } + next() +}) +app.use((req, res, next) => { // Add CORS headers const accessControlAllowOrigin = process.env.ACCESS_CONTROL_ALLOW_ORIGIN || 'http://sharelatex' res.setHeader('Access-Control-Allow-Origin', accessControlAllowOrigin) res.setHeader('Access-Control-Allow-Headers', 'Content-Type') res.setHeader('Access-Control-Max-Age', '3600') + res.setHeader('Access-Control-Allow-Methods', 'DELETE, GET, HEAD, POST, PUT') next() }) @@ -133,6 +146,7 @@ const allowedVars = Joi.object( 'V1_HISTORY_URL', 'SANDBOXED_COMPILES', 'ALL_TEX_LIVE_DOCKER_IMAGE_NAMES', + 'OVERLEAF_FILESTORE_MIGRATION_LEVEL', 'OVERLEAF_TEMPLATES_USER_ID', 'OVERLEAF_NEW_PROJECT_TEMPLATE_LINKS', 'OVERLEAF_ALLOW_PUBLIC_ACCESS', @@ -319,8 +333,19 @@ app.get('/redis/keys', (req, res) => { ) }) +app.delete('/data/user_files', (req, res) => { + runDockerCompose( + 'exec', + ['sharelatex', 'rm', '-rf', '/var/lib/overleaf/data/user_files'], + (error, stdout, stderr) => { + res.json({ error, stdout, stderr }) + } + ) +}) + app.use(handleValidationErrors()) purgeDataDir() +writeDockerComposeOverride(defaultDockerComposeOverride()) app.listen(80) diff --git a/services/chat/docker-compose.ci.yml b/services/chat/docker-compose.ci.yml index 24b57ab084..ca3303a079 100644 --- a/services/chat/docker-compose.ci.yml +++ b/services/chat/docker-compose.ci.yml @@ -42,7 +42,7 @@ services: command: tar -czf /tmp/build/build.tar.gz --exclude=build.tar.gz --exclude-vcs . user: root mongo: - image: mongo:7.0.20 + image: mongo:8.0.11 command: --replSet overleaf volumes: - ../../bin/shared/mongodb-init-replica-set.js:/docker-entrypoint-initdb.d/mongodb-init-replica-set.js diff --git a/services/chat/docker-compose.yml b/services/chat/docker-compose.yml index ddc5f9e698..e7b8ce7385 100644 --- a/services/chat/docker-compose.yml +++ b/services/chat/docker-compose.yml @@ -44,7 +44,7 @@ services: command: npm run --silent test:acceptance mongo: - image: mongo:7.0.20 + image: mongo:8.0.11 command: --replSet overleaf volumes: - ../../bin/shared/mongodb-init-replica-set.js:/docker-entrypoint-initdb.d/mongodb-init-replica-set.js diff --git a/services/clsi/app/js/CompileController.js b/services/clsi/app/js/CompileController.js index 7329c14342..b3343ee233 100644 --- a/services/clsi/app/js/CompileController.js +++ b/services/clsi/app/js/CompileController.js @@ -129,7 +129,7 @@ function compile(req, res, next) { compiler: request.compiler, draft: request.draft, imageName: request.imageName - ? Path.basename(request.imageName) + ? request.imageName : undefined, rootResourcePath: request.rootResourcePath, stopOnFirstError: request.stopOnFirstError, diff --git a/services/clsi/app/js/DockerRunner.js b/services/clsi/app/js/DockerRunner.js index def02eaf5b..97053c1875 100644 --- a/services/clsi/app/js/DockerRunner.js +++ b/services/clsi/app/js/DockerRunner.js @@ -232,8 +232,8 @@ const DockerRunner = { } } // set the path based on the image year - const match = image.match(/:([0-9]+)\.[0-9]+/) - const year = match ? match[1] : '2014' + const match = image.match(/:([0-9]+)\.[0-9]+|:TL([0-9]+)/) + const year = match ? match[1] || match[2] : '2014' env.PATH = `/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/texlive/${year}/bin/x86_64-linux/` const options = { Cmd: command, diff --git a/services/clsi/config/settings.defaults.js b/services/clsi/config/settings.defaults.js index 1d82258a8e..bd5614eb98 100644 --- a/services/clsi/config/settings.defaults.js +++ b/services/clsi/config/settings.defaults.js @@ -107,7 +107,7 @@ if ((process.env.DOCKER_RUNNER || process.env.SANDBOXED_COMPILES) === 'true') { CLSI: 1, }, socketPath: '/var/run/docker.sock', - user: process.env.TEXLIVE_IMAGE_USER || 'tex', + user: process.env.TEXLIVE_IMAGE_USER || 'www-data', }, optimiseInDocker: true, expireProjectAfterIdleMs: 24 * 60 * 60 * 1000, diff --git a/services/clsi/seccomp/clsi-profile.json b/services/clsi/seccomp/clsi-profile.json index 084354b15c..ad95130f76 100644 --- a/services/clsi/seccomp/clsi-profile.json +++ b/services/clsi/seccomp/clsi-profile.json @@ -829,13 +829,19 @@ "args": [] }, { - "name": "gettimeofday", - "action": "SCMP_ACT_ALLOW", - "args": [] - }, { - "name": "epoll_pwait", - "action": "SCMP_ACT_ALLOW", - "args": [] + "name": "gettimeofday", + "action": "SCMP_ACT_ALLOW", + "args": [] + }, + { + "name": "epoll_pwait", + "action": "SCMP_ACT_ALLOW", + "args": [] + }, + { + "name": "poll", + "action": "SCMP_ACT_ALLOW", + "args": [] } ] -} \ No newline at end of file +} diff --git a/services/contacts/docker-compose.ci.yml b/services/contacts/docker-compose.ci.yml index 24b57ab084..ca3303a079 100644 --- a/services/contacts/docker-compose.ci.yml +++ b/services/contacts/docker-compose.ci.yml @@ -42,7 +42,7 @@ services: command: tar -czf /tmp/build/build.tar.gz --exclude=build.tar.gz --exclude-vcs . user: root mongo: - image: mongo:7.0.20 + image: mongo:8.0.11 command: --replSet overleaf volumes: - ../../bin/shared/mongodb-init-replica-set.js:/docker-entrypoint-initdb.d/mongodb-init-replica-set.js diff --git a/services/contacts/docker-compose.yml b/services/contacts/docker-compose.yml index 6c77ef5e31..474ea224f8 100644 --- a/services/contacts/docker-compose.yml +++ b/services/contacts/docker-compose.yml @@ -44,7 +44,7 @@ services: command: npm run --silent test:acceptance mongo: - image: mongo:7.0.20 + image: mongo:8.0.11 command: --replSet overleaf volumes: - ../../bin/shared/mongodb-init-replica-set.js:/docker-entrypoint-initdb.d/mongodb-init-replica-set.js diff --git a/services/docstore/docker-compose.ci.yml b/services/docstore/docker-compose.ci.yml index 40decc4aea..cdb4783c5a 100644 --- a/services/docstore/docker-compose.ci.yml +++ b/services/docstore/docker-compose.ci.yml @@ -47,7 +47,7 @@ services: command: tar -czf /tmp/build/build.tar.gz --exclude=build.tar.gz --exclude-vcs . user: root mongo: - image: mongo:7.0.20 + image: mongo:8.0.11 command: --replSet overleaf volumes: - ../../bin/shared/mongodb-init-replica-set.js:/docker-entrypoint-initdb.d/mongodb-init-replica-set.js diff --git a/services/docstore/docker-compose.yml b/services/docstore/docker-compose.yml index 8c11eb5a91..a9099c7e7b 100644 --- a/services/docstore/docker-compose.yml +++ b/services/docstore/docker-compose.yml @@ -49,7 +49,7 @@ services: command: npm run --silent test:acceptance mongo: - image: mongo:7.0.20 + image: mongo:8.0.11 command: --replSet overleaf volumes: - ../../bin/shared/mongodb-init-replica-set.js:/docker-entrypoint-initdb.d/mongodb-init-replica-set.js diff --git a/services/document-updater/docker-compose.ci.yml b/services/document-updater/docker-compose.ci.yml index ca15f35fef..c6ec24a84b 100644 --- a/services/document-updater/docker-compose.ci.yml +++ b/services/document-updater/docker-compose.ci.yml @@ -55,7 +55,7 @@ services: retries: 20 mongo: - image: mongo:7.0.20 + image: mongo:8.0.11 command: --replSet overleaf volumes: - ../../bin/shared/mongodb-init-replica-set.js:/docker-entrypoint-initdb.d/mongodb-init-replica-set.js diff --git a/services/document-updater/docker-compose.yml b/services/document-updater/docker-compose.yml index cf7c9a2eb6..c1b23c11c5 100644 --- a/services/document-updater/docker-compose.yml +++ b/services/document-updater/docker-compose.yml @@ -57,7 +57,7 @@ services: retries: 20 mongo: - image: mongo:7.0.20 + image: mongo:8.0.11 command: --replSet overleaf volumes: - ../../bin/shared/mongodb-init-replica-set.js:/docker-entrypoint-initdb.d/mongodb-init-replica-set.js diff --git a/services/filestore/app.js b/services/filestore/app.js index 24741e079c..178e8c7ff0 100644 --- a/services/filestore/app.js +++ b/services/filestore/app.js @@ -111,6 +111,11 @@ if (settings.filestore.stores.template_files) { keyBuilder.templateFileKeyMiddleware, fileController.insertFile ) + app.delete( + '/template/:template_id/v/:version/:format', + keyBuilder.templateFileKeyMiddleware, + fileController.deleteFile + ) } app.get( diff --git a/services/filestore/app/js/FileConverter.js b/services/filestore/app/js/FileConverter.js index ac3dccec1f..bfc34314e9 100644 --- a/services/filestore/app/js/FileConverter.js +++ b/services/filestore/app/js/FileConverter.js @@ -5,7 +5,7 @@ const { callbackify } = require('node:util') const safeExec = require('./SafeExec').promises const { ConversionError } = require('./Errors') -const APPROVED_FORMATS = ['png'] +const APPROVED_FORMATS = ['png', 'jpg'] const FOURTY_SECONDS = 40 * 1000 const KILL_SIGNAL = 'SIGTERM' @@ -34,16 +34,14 @@ async function convert(sourcePath, requestedFormat) { } async function thumbnail(sourcePath) { - const width = '260x' - return await convert(sourcePath, 'png', [ + const width = '548x' + return await _convert(sourcePath, 'jpg', [ 'convert', '-flatten', '-background', 'white', '-density', '300', - '-define', - `pdf:fit-page=${width}`, `${sourcePath}[0]`, '-resize', width, @@ -51,16 +49,14 @@ async function thumbnail(sourcePath) { } async function preview(sourcePath) { - const width = '548x' - return await convert(sourcePath, 'png', [ + const width = '794x' + return await _convert(sourcePath, 'jpg', [ 'convert', '-flatten', '-background', 'white', '-density', '300', - '-define', - `pdf:fit-page=${width}`, `${sourcePath}[0]`, '-resize', width, diff --git a/services/filestore/app/js/FileHandler.js b/services/filestore/app/js/FileHandler.js index 2ed28bd435..0c092c85cd 100644 --- a/services/filestore/app/js/FileHandler.js +++ b/services/filestore/app/js/FileHandler.js @@ -150,7 +150,9 @@ async function _getConvertedFileAndCache(bucket, key, convertedKey, opts) { let convertedFsPath try { convertedFsPath = await _convertFile(bucket, key, opts) - await ImageOptimiser.promises.compressPng(convertedFsPath) + if (convertedFsPath.toLowerCase().endsWith(".png")) { + await ImageOptimiser.promises.compressPng(convertedFsPath) + } await PersistorManager.sendFile(bucket, convertedKey, convertedFsPath) } catch (err) { LocalFileWriter.deleteFile(convertedFsPath, () => {}) diff --git a/services/history-v1/docker-compose.ci.yml b/services/history-v1/docker-compose.ci.yml index da664d6b30..cf6ec3357d 100644 --- a/services/history-v1/docker-compose.ci.yml +++ b/services/history-v1/docker-compose.ci.yml @@ -75,7 +75,7 @@ services: retries: 20 mongo: - image: mongo:7.0.20 + image: mongo:8.0.11 command: --replSet overleaf volumes: - ../../bin/shared/mongodb-init-replica-set.js:/docker-entrypoint-initdb.d/mongodb-init-replica-set.js diff --git a/services/history-v1/docker-compose.yml b/services/history-v1/docker-compose.yml index 22b739abf9..3a33882d28 100644 --- a/services/history-v1/docker-compose.yml +++ b/services/history-v1/docker-compose.yml @@ -83,7 +83,7 @@ services: retries: 20 mongo: - image: mongo:7.0.20 + image: mongo:8.0.11 command: --replSet overleaf volumes: - ../../bin/shared/mongodb-init-replica-set.js:/docker-entrypoint-initdb.d/mongodb-init-replica-set.js diff --git a/services/history-v1/storage/scripts/back_fill_file_hash.mjs b/services/history-v1/storage/scripts/back_fill_file_hash.mjs index 0ccadaf5a9..2e12328e5c 100644 --- a/services/history-v1/storage/scripts/back_fill_file_hash.mjs +++ b/services/history-v1/storage/scripts/back_fill_file_hash.mjs @@ -150,10 +150,6 @@ const CONCURRENT_BATCHES = parseInt(process.env.CONCURRENT_BATCHES || '2', 10) const RETRIES = parseInt(process.env.RETRIES || '10', 10) const RETRY_DELAY_MS = parseInt(process.env.RETRY_DELAY_MS || '100', 10) -const USER_FILES_BUCKET_NAME = process.env.USER_FILES_BUCKET_NAME || '' -if (!USER_FILES_BUCKET_NAME) { - throw new Error('env var USER_FILES_BUCKET_NAME is missing') -} const RETRY_FILESTORE_404 = process.env.RETRY_FILESTORE_404 === 'true' const BUFFER_DIR = fs.mkdtempSync( process.env.BUFFER_DIR_PREFIX || '/tmp/back_fill_file_hash-' diff --git a/services/history-v1/storage/scripts/back_fill_file_hash_fix_up.mjs b/services/history-v1/storage/scripts/back_fill_file_hash_fix_up.mjs index 7bab794692..2525ee1d6e 100644 --- a/services/history-v1/storage/scripts/back_fill_file_hash_fix_up.mjs +++ b/services/history-v1/storage/scripts/back_fill_file_hash_fix_up.mjs @@ -9,15 +9,12 @@ import { Blob } from 'overleaf-editor-core' import { BlobStore, getStringLengthOfFile, - GLOBAL_BLOBS, makeBlobForFile, } from '../lib/blob_store/index.js' import { db } from '../lib/mongodb.js' import commandLineArgs from 'command-line-args' import readline from 'node:readline' -import { _blobIsBackedUp, backupBlob } from '../lib/backupBlob.mjs' import { NotFoundError } from '@overleaf/object-persistor/src/Errors.js' -import filestorePersistor from '../lib/persistor.js' import { setTimeout } from 'node:timers/promises' // Silence warning. @@ -52,12 +49,11 @@ ObjectId.cacheHexString = true */ /** - * @return {{FIX_NOT_FOUND: boolean, FIX_HASH_MISMATCH: boolean, FIX_DELETE_PERMISSION: boolean, FIX_MISSING_HASH: boolean, LOGS: string}} + * @return {{FIX_NOT_FOUND: boolean, FIX_HASH_MISMATCH: boolean, FIX_MISSING_HASH: boolean, LOGS: string}} */ function parseArgs() { const args = commandLineArgs([ { name: 'fixNotFound', type: String, defaultValue: 'true' }, - { name: 'fixDeletePermission', type: String, defaultValue: 'true' }, { name: 'fixHashMismatch', type: String, defaultValue: 'true' }, { name: 'fixMissingHash', type: String, defaultValue: 'true' }, { name: 'logs', type: String, defaultValue: '' }, @@ -74,20 +70,13 @@ function parseArgs() { } return { FIX_HASH_MISMATCH: boolVal('fixNotFound'), - FIX_DELETE_PERMISSION: boolVal('fixDeletePermission'), FIX_NOT_FOUND: boolVal('fixHashMismatch'), FIX_MISSING_HASH: boolVal('fixMissingHash'), LOGS: args.logs, } } -const { - FIX_HASH_MISMATCH, - FIX_DELETE_PERMISSION, - FIX_NOT_FOUND, - FIX_MISSING_HASH, - LOGS, -} = parseArgs() +const { FIX_HASH_MISMATCH, FIX_NOT_FOUND, FIX_MISSING_HASH, LOGS } = parseArgs() if (!LOGS) { throw new Error('--logs parameter missing') } @@ -105,6 +94,37 @@ const STREAM_HIGH_WATER_MARK = parseInt( ) const SLEEP_BEFORE_EXIT = parseInt(process.env.SLEEP_BEFORE_EXIT || '1000', 10) +// Filestore endpoint location +const FILESTORE_HOST = process.env.FILESTORE_HOST || '127.0.0.1' +const FILESTORE_PORT = process.env.FILESTORE_PORT || '3009' + +async function fetchFromFilestore(projectId, fileId) { + const url = `http://${FILESTORE_HOST}:${FILESTORE_PORT}/project/${projectId}/file/${fileId}` + const response = await fetch(url) + if (!response.ok) { + if (response.status === 404) { + throw new NotFoundError('file not found in filestore', { + status: response.status, + }) + } + const body = await response.text() + throw new OError('fetchFromFilestore failed', { + projectId, + fileId, + status: response.status, + body, + }) + } + if (!response.body) { + throw new OError('fetchFromFilestore response has no body', { + projectId, + fileId, + status: response.status, + }) + } + return response.body +} + /** @type {ProjectsCollection} */ const projectsCollection = db.collection('projects') /** @type {DeletedProjectsCollection} */ @@ -302,19 +322,16 @@ async function setHashInMongo(projectId, fileId, hash) { * @return {Promise
{t('latex_templates_for_journal_articles')} +
+{description}
+![{link.name}]({`/img/website-redesign/gallery/${link.url.split("/").pop()}.svg`})
+ {link.description}
+{t('no_templates_found')}
+{t('about_to_delete_template')}
+