Compare commits

..

9 commits

Author SHA1 Message Date
yu-i-i
f300441211 Update README.md 2024-12-12 18:49:19 +01:00
yu-i-i
e76ca07da2 Enable Symbol Palette 2024-12-12 18:49:19 +01:00
yu-i-i
3e7d1d3160 Allow selecting a TeX Live image for a project 2024-12-12 18:49:19 +01:00
Sam Van den Vonder
12690a9b06 Enable Sandboxed Compiles feature 2024-12-12 18:49:04 +01:00
yu-i-i
5d68b00a7c Enable autocomplete of reference keys feature 2024-12-12 18:27:59 +01:00
yu-i-i
409293cb15 Enable track changes and comments feature 2024-12-12 18:25:29 +01:00
yu-i-i
b44addd5a1 Enable LDAP and SAML authentication support 2024-12-12 18:25:21 +01:00
yu-i-i
4345ff5a22 Redirect non-existing links to Overleaf page 2024-12-12 18:20:38 +01:00
Miguel Serrano
f0022392ac Merge pull request #21327 from overleaf/msm-optional-subnet-rate-limiter
[web] Add option to disable subnet rate limiting (+CE/SP Hotfix `5.2.1`)

GitOrigin-RevId: 78d60c9638cede729dd93c3c2421f55b34c0dbfe
2024-11-29 15:11:35 +01:00
3711 changed files with 257398 additions and 188012 deletions

View file

@ -1,19 +1,10 @@
---
name: Bug report
about: Report a bug
title: ''
labels: type:bug
assignees: ''
---
<!--
Note: If you are using www.overleaf.com and have a problem,
Note: If you are using www.overleaf.com and have a problem,
or if you would like to request a new feature please contact
the support team at support@overleaf.com
This form should only be used to report bugs in the
This form should only be used to report bugs in the
Community Edition release of Overleaf.
-->

View file

@ -14,8 +14,6 @@ When submitting an issue please describe the issue as clearly as possible, inclu
reproduce the bug, which situations it appears in, what you expected to happen, and what actually happens.
If you can include a screenshot for front end issues that is very helpful.
**Note**: If you are using [www.overleaf.com](www.overleaf.com) and have a problem, or if you would like to request a new feature, please contact the Support team at support@overleaf.com. Raise an issue here only to report bugs in the Community Edition release of Overleaf.
Pull Requests
-------------
@ -36,7 +34,7 @@ Please see [our security policy](https://github.com/overleaf/overleaf/security/p
Contributor License Agreement
-----------------------------
Before we can accept any contributions of code, we need you to agree to our
Before we can accept any contributions of code, we need you to agree to our
[Contributor License Agreement](https://docs.google.com/forms/d/e/1FAIpQLSef79XH3mb7yIiMzZw-yALEegS-wyFetvjTiNBfZvf_IHD2KA/viewform?usp=sf_link).
This is to ensure that you own the copyright of your contribution, and that you
agree to give us a license to use it in both the open source version, and the version

676
README.md
View file

@ -27,21 +27,12 @@
The present "extended" version of Overleaf CE includes:
- Template Gallery
- Sandboxed Compiles with TeX Live image selection
- LDAP authentication
- SAML authentication
- OpenID Connect authentication
- Real-time track changes and comments
- Autocomplete of reference keys
- Symbol Palette
- "From External URL" feature
> [!CAUTION]
> Overleaf Community Edition is intended for use in environments where **all** users are trusted. Community Edition is **not** appropriate for scenarios where isolation of users is required due to Sandbox Compiles not being available. When not using Sandboxed Compiles, users have full read and write access to the `sharelatex` container resources (filesystem, network, environment variables) when running LaTeX compiles.
Therefore, in any environment where not all users can be fully trusted, it is strongly recommended to enable the Sandboxed Compiles feature available in the Extended Community Edition.
For more information on Sandbox Compiles check out Overleaf [documentation](https://docs.overleaf.com/on-premises/configuration/overleaf-toolkit/server-pro-only-configuration/sandboxed-compiles).
## Enterprise
@ -50,16 +41,658 @@ If you want help installing and maintaining Overleaf in your lab or workplace, O
## Installation
Detailed installation instructions can be found in the [Overleaf Toolkit](https://github.com/overleaf/toolkit/).
Configuration details and release history for the Extended Community Edition can be found on the [Extended CE Wiki Page](https://github.com/yu-i-i/overleaf-cep/wiki).
To run a custom image, add a file named docker-compose.override.yml with the following or similar content into the `overleaf-toolkit/config directory`:
```yml
---
version: '2.2'
services:
sharelatex:
image: sharelatex/sharelatex:ext-ce
volumes:
- ../config/certs:/overleaf/certs
```
Here, the attached volume provides convenient access for the container to the certificates needed for SAML or LDAP authentication.
If you want to build a Docker image of the extended CE based on the upstream v5.2.1 codebase, you can check out the corresponding tag by running:
```
git checkout v5.2.1-ext
```
Alternatively, you can download a prebuilt image from Docker Hub:
```
docker pull overleafcep/sharelatex:5.2.1-ext
```
Make sure to update the image name in overleaf-toolkit/config/docker-compose.override.yml accordingly.
## Sandboxed Compiles
To enable sandboxed compiles (also known as "Sibling containers"), set the following configuration options in `overleaf-toolkit/config/overleaf.rc`:
```
SERVER_PRO=true
SIBLING_CONTAINERS_ENABLED=true
```
The following environment variables are used to specify which TeX Live images to use for sandboxed compiles:
- `ALL_TEX_LIVE_DOCKER_IMAGES` **(required)**
* A comma-separated list of TeX Live images to use. These images will be downloaded or updated.
To skip downloading the images, set `SIBLING_CONTAINERS_PULL=false` in `config/overleaf.rc`.
- `ALL_TEX_LIVE_DOCKER_IMAGE_NAMES`
* A comma-separated list of friendly names for the images. If omitted, the version name will be used (e.g., `latest-full`).
- `TEX_LIVE_DOCKER_IMAGE` **(required)**
* The default TeX Live image that will be used for compiling new projects. The environment variable `ALL_TEX_LIVE_DOCKER_IMAGES` must include this image.
Users can select the image for their project in the project menu.
Here is an example where the default TeX Live image is `latest-full` from Docker Hub, but the `TL2023-historic` image can be used for older projects:
```
ALL_TEX_LIVE_DOCKER_IMAGES=texlive/texlive:latest-full, texlive/texlive:TL2023-historic
ALL_TEX_LIVE_DOCKER_IMAGE_NAMES=TeXLive 2024, TeXLive 2023
TEX_LIVE_DOCKER_IMAGE=texlive/texlive:latest-full
```
For additional details refer to
[Server Pro: Sandboxed Compiles](https://github.com/overleaf/overleaf/wiki/Server-Pro:-Sandboxed-Compiles) and
[Toolkit: Sandboxed Compiles](https://github.com/overleaf/toolkit/blob/master/doc/sandboxed-compiles.md).
<details>
<summary><h4>Sample variables.env file</h4></summary>
```
OVERLEAF_APP_NAME="Our Overleaf Instance"
ENABLED_LINKED_FILE_TYPES=project_file,project_output_file
# Enables Thumbnail generation using ImageMagick
ENABLE_CONVERSIONS=true
# Disables email confirmation requirement
EMAIL_CONFIRMATION_DISABLED=true
## Nginx
# NGINX_WORKER_PROCESSES=4
# NGINX_WORKER_CONNECTIONS=768
## Set for TLS via nginx-proxy
# OVERLEAF_BEHIND_PROXY=true
# OVERLEAF_SECURE_COOKIE=true
OVERLEAF_SITE_URL=http://my-overleaf-instance.com
OVERLEAF_NAV_TITLE=Our Overleaf Instance
# OVERLEAF_HEADER_IMAGE_URL=http://somewhere.com/mylogo.png
OVERLEAF_ADMIN_EMAIL=support@example.com
OVERLEAF_LEFT_FOOTER=[{"text": "Contact your support team", "url": "mailto:support@example.com"}]
OVERLEAF_RIGHT_FOOTER=[{"text":"Hello, I am on the Right", "url":"https://github.com/yu-i-i/overleaf-cep"}]
OVERLEAF_EMAIL_FROM_ADDRESS=team@example.com
OVERLEAF_EMAIL_SMTP_HOST=smtp.example.com
OVERLEAF_EMAIL_SMTP_PORT=587
OVERLEAF_EMAIL_SMTP_SECURE=false
# OVERLEAF_EMAIL_SMTP_USER=
# OVERLEAF_EMAIL_SMTP_PASS=
# OVERLEAF_EMAIL_SMTP_NAME=
OVERLEAF_EMAIL_SMTP_LOGGER=false
OVERLEAF_EMAIL_SMTP_TLS_REJECT_UNAUTH=true
OVERLEAF_EMAIL_SMTP_IGNORE_TLS=false
OVERLEAF_CUSTOM_EMAIL_FOOTER=This system is run by department x
OVERLEAF_PROXY_LEARN=true
NAV_HIDE_POWERED_BY=true
########################
## Sandboxed Compiles ##
########################
ALL_TEX_LIVE_DOCKER_IMAGES=texlive/texlive:latest-full, texlive/texlive:TL2023-historic
ALL_TEX_LIVE_DOCKER_IMAGE_NAMES=TeXLive 2024, TeXLive 2023
TEX_LIVE_DOCKER_IMAGE=texlive/texlive:latest-full
```
</details>
## Authentication Methods
The following authentication methods are supported: Local authentication, LDAP authentication, and SAML authentication. Local authentication is always active.
To enable LDAP or SAML authentication, the environment variable `EXTERNAL_AUTH` must be set to `ldap` or `saml`, respectively.
<details>
<summary><h3>Local Authentication</h3></summary>
Password of local users stored in the MongoDB database. An admin user can create a new local user. For details, visit the
[wiki of Overleaf project](https://github.com/overleaf/overleaf/wiki/Creating-and-managing-users).
It is possible to enforce password restrictions on local users:
* `OVERLEAF_PASSWORD_VALIDATION_MIN_LENGTH`: The minimum length required
* `OVERLEAF_PASSWORD_VALIDATION_MAX_LENGTH`: The maximum length allowed
* `OVERLEAF_PASSWORD_VALIDATION_PATTERN`: is used to validate password strength
- `abc123` password requires 3 letters and 3 numbers and be at least 6 characters long
- `aA` password requires lower and uppercase letters and be at least 2 characters long
- `ab$3` it must contain letters, digits and symbols and be at least 4 characters long
- There are 4 groups of characters: letters, UPPERcase letters, digits, symbols. Anything that is neither a letter nor a digit is considered to be a symbol.
</details>
<details>
<summary><h3>LDAP Authentication</h3></summary>
Internally, Overleaf LDAP uses the [passport-ldapauth](https://github.com/vesse/passport-ldapauth) library. Most of these configuration options are passed through to the `server` config object which is used to configure `passport-ldapauth`. If you are having issues configuring LDAP, it is worth reading the README for `passport-ldapauth` to understand the configuration it expects.
#### Environment Variables
- `OVERLEAF_LDAP_URL` **(required)**
* URL of the LDAP server.
- Example: `ldaps://ldap.example.com:636` (LDAP over SSL)
- Example: `ldap://ldap.example.com:389` (unencrypted or STARTTLS, if configured).
- `OVERLEAF_LDAP_EMAIL_ATT`
* The email attribute returned by the LDAP server, default `mail`. Each LDAP user must have at least one email address.
If multiple addresses are provided, only the first one will be used.
- `OVERLEAF_LDAP_FIRST_NAME_ATT`
* The property name holding the first name of the user which is used in the application, usually `givenName`.
- `OVERLEAF_LDAP_LAST_NAME_ATT`
* The property name holding the family name of the user which is used in the application, usually `sn`.
- `OVERLEAF_LDAP_NAME_ATT`
* The property name holding the full name of the user, usually `cn`. If either of the two previous variables is not defined,
the first and/or last name of the user is extracted from this variable. Otherwise, it is not used.
- `OVERLEAF_LDAP_PLACEHOLDER`
* The placeholder for the login form, defaults to `Username`.
- `OVERLEAF_LDAP_UPDATE_USER_DETAILS_ON_LOGIN`
* If set to `true`, updates the LDAP user `first_name` and `last_name` field on login, and turn off the user details form on the `/user/settings`
page for LDAP users. Otherwise, details will be fetched only on first login.
- `OVERLEAF_LDAP_BIND_DN`
* The distinguished name of the LDAP user that should be used for the LDAP connection
(this user should be able to search/list accounts on the LDAP server),
e.g., `cn=ldap_reader,dc=example,dc=com`. If not defined, anonymous binding is used.
- `OVERLEAF_LDAP_BIND_CREDENTIALS`
* Password for `OVERLEAF_LDAP_BIND_DN`.
- `OVERLEAF_LDAP_BIND_PROPERTY`
* Property of the user to bind against the client, defaults to `dn`.
- `OVERLEAF_LDAP_SEARCH_BASE` **(required)**
* The base DN from which to search for users. E.g., `ou=people,dc=example,dc=com`.
- `OVERLEAF_LDAP_SEARCH_FILTER`
* LDAP search filter with which to find a user. Use the literal '{{username}}' to have the given username be interpolated in for the LDAP search.
- Example: `(|(uid={{username}})(mail={{username}}))` (user can login with email or with login name).
- Example: `(sAMAccountName={{username}})` (Active Directory).
- `OVERLEAF_LDAP_SEARCH_SCOPE`
* The scope of the search can be `base`, `one`, or `sub` (default).
- `OVERLEAF_LDAP_SEARCH_ATTRIBUTES`
* JSON array of attributes to fetch from the LDAP server, e.g., `["uid", "mail", "givenName", "sn"]`.
By default, all attributes are fetched.
- `OVERLEAF_LDAP_STARTTLS`
* If `true`, LDAP over TLS is used.
- `OVERLEAF_LDAP_TLS_OPTS_CA_PATH`
* Path to the file containing the CA certificate used to verify the LDAP server's SSL/TLS certificate. If there are multiple certificates, then
it can be a JSON array of paths to the certificates. The files must be accessible to the docker container.
- Example (one certificate): `/overleaf/certs/ldap_ca_cert.pem`
- Example (multiple certificates): `["/overleaf/certs/ldap_ca_cert1.pem", "/overleaf/certs/ldap_ca_cert2.pem"]`
- `OVERLEAF_LDAP_TLS_OPTS_REJECT_UNAUTH`
* If `true`, the server certificate is verified against the list of supplied CAs.
- `OVERLEAF_LDAP_CACHE`
* If `true`, then up to 100 credentials at a time will be cached for 5 minutes.
- `OVERLEAF_LDAP_TIMEOUT`
* How long the client should let operations live for before timing out, ms (Default: Infinity).
- `OVERLEAF_LDAP_CONNECT_TIMEOUT`
* How long the client should wait before timing out on TCP connections, ms (Default: OS default).
- `OVERLEAF_LDAP_IS_ADMIN_ATT` and `OVERLEAF_LDAP_IS_ADMIN_ATT_VALUE`
* When both environment variables are set, the login process updates `user.isAdmin = true` if the LDAP profile contains the attribute specified by
`OVERLEAF_LDAP_IS_ADMIN_ATT` and its value either matches `OVERLEAF_LDAP_IS_ADMIN_ATT_VALUE` or is an array containing `OVERLEAF_LDAP_IS_ADMIN_ATT_VALUE`,
otherwise `user.isAdmin` is set to `false`. If either of these variables is not set, then the admin status is only set to `true` during admin user
creation in Launchpad.
The following five variables are used to configure how user contacts are retrieved from the LDAP server.
- `OVERLEAF_LDAP_CONTACTS_FILTER`
* The filter used to search for users in the LDAP server to be loaded into contacts. The placeholder '{{userProperty}}' within the filter is replaced with the value of
the property specified by `OVERLEAF_LDAP_CONTACTS_PROPERTY` from the LDAP user initiating the search. If not defined, no users are retrieved from the LDAP server into contacts.
- `OVERLEAF_LDAP_CONTACTS_SEARCH_BASE`
* Specifies the base DN from which to start searching for the contacts. Defaults to `OVERLEAF_LDAP_SEARCH_BASE`.
- `OVERLEAF_LDAP_CONTACTS_SEARCH_SCOPE`
* The scope of the search can be `base`, `one`, or `sub` (default).
- `OVERLEAF_LDAP_CONTACTS_PROPERTY`
* Specifies the property of the user object that will replace the '{{userProperty}}' placeholder in the `OVERLEAF_LDAP_CONTACTS_FILTER`.
- `OVERLEAF_LDAP_CONTACTS_NON_LDAP_VALUE`
* Specifies the value of the `OVERLEAF_LDAP_CONTACTS_PROPERTY` if the search is initiated by a non-LDAP user. If this variable is not defined, the resulting filter
will match nothing. The value `*` can be used as a wildcard.
<details>
<summary><h5>Example</h5></summary>
OVERLEAF_LDAP_CONTACTS_FILTER=(gidNumber={{userProperty}})
OVERLEAF_LDAP_CONTACTS_PROPERTY=gidNumber
OVERLEAF_LDAP_CONTACTS_NON_LDAP_VALUE=1000
The above example results in loading into the contacts of the current LDAP user all LDAP users who have the same UNIX `gid`. Non-LDAP users will have all LDAP users with UNIX `gid=1000` in their contacts.
</details>
<details>
<summary><h4>Sample variables.env file</h4></summary>
```
OVERLEAF_APP_NAME="Our Overleaf Instance"
ENABLED_LINKED_FILE_TYPES=project_file,project_output_file
# Enables Thumbnail generation using ImageMagick
ENABLE_CONVERSIONS=true
# Disables email confirmation requirement
EMAIL_CONFIRMATION_DISABLED=true
## Nginx
# NGINX_WORKER_PROCESSES=4
# NGINX_WORKER_CONNECTIONS=768
## Set for TLS via nginx-proxy
# OVERLEAF_BEHIND_PROXY=true
# OVERLEAF_SECURE_COOKIE=true
OVERLEAF_SITE_URL=http://my-overleaf-instance.com
OVERLEAF_NAV_TITLE=Our Overleaf Instance
# OVERLEAF_HEADER_IMAGE_URL=http://somewhere.com/mylogo.png
OVERLEAF_ADMIN_EMAIL=support@example.com
OVERLEAF_LEFT_FOOTER=[{"text": "Contact your support team", "url": "mailto:support@example.com"}]
OVERLEAF_RIGHT_FOOTER=[{"text":"Hello, I am on the Right", "url":"https://github.com/yu-i-i/overleaf-cep"}]
OVERLEAF_EMAIL_FROM_ADDRESS=team@example.com
OVERLEAF_EMAIL_SMTP_HOST=smtp.example.com
OVERLEAF_EMAIL_SMTP_PORT=587
OVERLEAF_EMAIL_SMTP_SECURE=false
# OVERLEAF_EMAIL_SMTP_USER=
# OVERLEAF_EMAIL_SMTP_PASS=
# OVERLEAF_EMAIL_SMTP_NAME=
OVERLEAF_EMAIL_SMTP_LOGGER=false
OVERLEAF_EMAIL_SMTP_TLS_REJECT_UNAUTH=true
OVERLEAF_EMAIL_SMTP_IGNORE_TLS=false
OVERLEAF_CUSTOM_EMAIL_FOOTER=This system is run by department x
OVERLEAF_PROXY_LEARN=true
NAV_HIDE_POWERED_BY=true
#################
## LDAP for CE ##
#################
EXTERNAL_AUTH=ldap
OVERLEAF_LDAP_URL=ldap://ldap.example.com:389
OVERLEAF_LDAP_STARTTLS=true
OVERLEAF_LDAP_TLS_OPTS_CA_PATH=/overleaf/certs/ldap_ca_cert.pem
OVERLEAF_LDAP_SEARCH_BASE=ou=people,dc=example,dc=com
OVERLEAF_LDAP_SEARCH_FILTER=(|(uid={{username}})(mail={{username}}))
OVERLEAF_LDAP_BIND_DN=cn=ldap_reader,dc=example,dc=com
OVERLEAF_LDAP_BIND_CREDENTIALS=GoodNewsEveryone
OVERLEAF_LDAP_EMAIL_ATT=mail
OVERLEAF_LDAP_FIRST_NAME_ATT=givenName
OVERLEAF_LDAP_LAST_NAME_ATT=sn
# OVERLEAF_LDAP_NAME_ATT=cn
OVERLEAF_LDAP_SEARCH_ATTRIBUTES=["uid", "sn", "givenName", "mail"]
OVERLEAF_LDAP_UPDATE_USER_DETAILS_ON_LOGIN=true
OVERLEAF_LDAP_PLACEHOLDER='Username or email address'
OVERLEAF_LDAP_IS_ADMIN_ATT=mail
OVERLEAF_LDAP_IS_ADMIN_ATT_VALUE=admin@example.com
OVERLEAF_LDAP_CONTACTS_FILTER=(gidNumber={{userProperty}})
OVERLEAF_LDAP_CONTACTS_PROPERTY=gidNumber
OVERLEAF_LDAP_CONTACTS_NON_LDAP_VALUE='*'
```
</details>
<details>
<summary><i>Deprecated variables</i></summary>
**These variables will be removed soon**, use `OVERLEAF_LDAP_IS_ADMIN_ATT` and `OVERLEAF_LDAP_IS_ADMIN_ATT_VALUE` instead.
The following variables are used to determine if the user has admin rights.
Please note: the user gains admin status if the search result is not empty, not when the user is explicitly included in the search results.
- `OVERLEAF_LDAP_ADMIN_SEARCH_BASE`
* Specifies the base DN from which to start searching for the admin group. If this variable is defined,
`OVERLEAF_LDAP_ADMIN_SEARCH_FILTER` must also be defined for the search to function properly.
- `OVERLEAF_LDAP_ADMIN_SEARCH_FILTER`
* Defines the LDAP search filter used to identify the admin group. The placeholder `{{dn}}` within the filter
is replaced with the value of the property specified by `OVERLEAF_LDAP_ADMIN_DN_PROPERTY`. The placeholder `{{username}}` is also supported.
- `OVERLEAF_LDAP_ADMIN_DN_PROPERTY`
* Specifies the property of the user object that will replace the '{{dn}}' placeholder
in the `OVERLEAF_LDAP_ADMIN_SEARCH_FILTER`, defaults to `dn`.
- `OVERLEAF_LDAP_ADMIN_SEARCH_SCOPE`
* The scope of the LDAP search can be `base`, `one`, or `sub` (default)
<details>
<summary><h5>Example</h5></summary>
In the following example admins are members of a group `admins`, the objectClass of the entry `admins` is `groupOfNames`:
OVERLEAF_LDAP_ADMIN_SEARCH_BASE='cn=admins,ou=group,dc=example,dc=com'
OVERLEAF_LDAP_ADMIN_SEARCH_FILTER='(member={{dn}})'
In the following example admins are members of a group 'admins', the objectClass of the entry `admins` is `posixGroup`:
OVERLEAF_LDAP_ADMIN_SEARCH_BASE='cn=admins,ou=group,dc=example,dc=com'
OVERLEAF_LDAP_ADMIN_SEARCH_FILTER='(memberUid={{username}})'
In the following example admins are users with UNIX gid=1234:
OVERLEAF_LDAP_ADMIN_SEARCH_BASE='ou=people,dc=example,dc=com'
OVERLEAF_LDAP_ADMIN_SEARCH_FILTER='(&(gidNumber=1234)(uid={{username}}))'
In the following example admin is the user with `uid=someuser`:
OVERLEAF_LDAP_ADMIN_SEARCH_BASE='ou=people,dc=example,dc=com'
OVERLEAF_LDAP_ADMIN_SEARCH_FILTER='(&(uid=someuser)(uid={{username}}))'
The filter
OVERLEAF_LDAP_ADMIN_SEARCH_FILTER='(uid=someuser)'
where `someuser` is the uid of an existing user, will always produce a non-empty search result.
As a result, **every user will be granted admin rights**, not just `someuser`, as one might expect.
</details>
</details>
</details>
<details>
<summary><h3>SAML Authentication</h3></summary>
Internally, Overleaf SAML module uses the [passport-saml](https://github.com/node-saml/passport-saml) library, most of the following
configuration options are passed through to `passport-saml`. If you are having issues configuring SAML, it is worth reading the README
for `passport-saml` to get a feel for the configuration it expects.
#### Environment Variables
- `OVERLEAF_SAML_IDENTITY_SERVICE_NAME`
* Display name for the identity service, used on the login page (default: `Login with SAML IdP`).
- `OVERLEAF_SAML_EMAIL_FIELD`
* Name of the Email field in user profile, default to 'nameID'.
- `OVERLEAF_SAML_FIRST_NAME_FIELD`
* Name of the firstName field in user profile, default to 'givenName'.
- `OVERLEAF_SAML_LAST_NAME_FIELD`
* Name of the lastName field in user profile, default to 'lastName'
- `OVERLEAF_SAML_UPDATE_USER_DETAILS_ON_LOGIN`
* If set to `true`, updates the user `first_name` and `last_name` field on login,
and turn off the user details form on `/user/settings` page.
- `OVERLEAF_SAML_ENTRYPOINT` **(required)**
* Entrypoint URL for the SAML identity service.
- Example: `https://idp.example.com/simplesaml/saml2/idp/SSOService.php`
- Azure Example: `https://login.microsoftonline.com/8b26b46a-6dd3-45c7-a104-f883f4db1f6b/saml2`
- `OVERLEAF_SAML_CALLBACK_URL` **(required)**
* Callback URL for Overleaf service. Should be the full URL of the `/saml/login/callback` path.
- Example: `https://my-overleaf-instance.com/saml/login/callback`
- `OVERLEAF_SAML_ISSUER` **(required)**
* The Issuer name.
- `OVERLEAF_SAML_AUDIENCE`
* Expected saml response Audience, defaults to value of `OVERLEAF_SAML_ISSUER`.
- `OVERLEAF_SAML_IDP_CERT` **(required)**
* Path to a file containing the Identity Provider's public certificate, used to validate the signatures of incoming SAML responses. If the Identity Provider has multiple valid signing certificates, then
it can be a JSON array of paths to the certificates.
- Example (one certificate): `/overleaf/certs/idp_cert.pem`
- Example (multiple certificates): `["/overleaf/certs/idp_cert.pem", "/overleaf/certs/idp_cert_old.pem"]`
- `OVERLEAF_SAML_PUBLIC_CERT`
* Path to a file containing public signing certificate used to embed in auth requests in order for the IdP to validate the signatures of the incoming SAML Request. It's required when setting up the [metadata endpoint](#metadata-for-the-identity-provider)
when the strategy is configured with a `OVERLEAF_SAML_PRIVATE_KEY`. A JSON array of paths to certificates can be provided to support certificate rotation. When supplying an array of certificates, the first entry in the array should match the
current `OVERLEAF_SAML_PRIVATE_KEY`. Additional entries in the array can be used to publish upcoming certificates to IdPs before changing the `OVERLEAF_SAML_PRIVATE_KEY`.
- `OVERLEAF_SAML_PRIVATE_KEY`
* Path to a file containing a PEM-formatted private key matching the `OVERLEAF_SAML_PUBLIC_CERT` used to sign auth requests sent by passport-saml.
- `OVERLEAF_SAML_DECRYPTION_CERT`
* Path to a file containing public certificate, used for the [metadata endpoint](#metadata-for-the-identity-provider).
- `OVERLEAF_SAML_DECRYPTION_PVK`
* Path to a file containing private key matching the `OVERLEAF_SAML_DECRYPTION_CERT` that will be used to attempt to decrypt any encrypted assertions that are received.
- `OVERLEAF_SAML_SIGNATURE_ALGORITHM`
* Optionally set the signature algorithm for signing requests,
valid values are 'sha1' (default), 'sha256' (prefered), 'sha512' (most secure, check if your IdP supports it).
- `OVERLEAF_SAML_ADDITIONAL_PARAMS`
* JSON dictionary of additional query params to add to all requests.
- `OVERLEAF_SAML_ADDITIONAL_AUTHORIZE_PARAMS`
* JSON dictionary of additional query params to add to 'authorize' requests.
- Example: `{"some_key": "some_value"}`
- `OVERLEAF_SAML_IDENTIFIER_FORMAT`
* Name identifier format to request from identity provider (default: `urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress`).
- `OVERLEAF_SAML_ACCEPTED_CLOCK_SKEW_MS`
* Time in milliseconds of skew that is acceptable between client and server when checking OnBefore and NotOnOrAfter assertion
condition validity timestamps. Setting to -1 will disable checking these conditions entirely. Default is 0.
- `OVERLEAF_SAML_ATTRIBUTE_CONSUMING_SERVICE_INDEX`
* `AttributeConsumingServiceIndex` attribute to add to AuthnRequest to instruct the IdP which attribute set to attach
to the response ([link](http://blog.aniljohn.com/2014/01/data-minimization-front-channel-saml-attribute-requests.html)).
- `OVERLEAF_SAML_AUTHN_CONTEXT`
* JSON array of name identifier format values to request auth context. Default: `["urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport"]`.
- `OVERLEAF_SAML_FORCE_AUTHN`
* If `true`, the initial SAML request from the service provider specifies that the IdP should force re-authentication of the user,
even if they possess a valid session.
- `OVERLEAF_SAML_DISABLE_REQUESTED_AUTHN_CONTEXT`
* If `true`, do not request a specific auth context. For example, you can this this to `true` to allow additional contexts such as password-less logins (`urn:oasis:names:tc:SAML:2.0:ac:classes:X509`). Support for additional contexts is dependant on your IdP.
- `OVERLEAF_SAML_AUTHN_REQUEST_BINDING`
* If set to `HTTP-POST`, will request authentication from IdP via HTTP POST binding, otherwise defaults to HTTP-Redirect.
- `OVERLEAF_SAML_VALIDATE_IN_RESPONSE_TO`
* If `always`, then InResponseTo will be validated from incoming SAML responses.
* If `never`, then InResponseTo won't be validated (default).
* If `ifPresent`, then InResponseTo will only be validated if present in the incoming SAML response.
- `OVERLEAF_SAML_REQUEST_ID_EXPIRATION_PERIOD_MS`
* Defines the expiration time when a Request ID generated for a SAML request will not be valid if seen
in a SAML response in the `InResponseTo` field. Default: 28800000 (8 hours).
- `OVERLEAF_SAML_LOGOUT_URL`
* base address to call with logout requests (default: `entryPoint`).
- Example: `https://idp.example.com/simplesaml/saml2/idp/SingleLogoutService.php`
- `OVERLEAF_SAML_LOGOUT_CALLBACK_URL`
* Callback URL for IdP initiated logout. Should be the full URL of the `/saml/logot/callback` path.
With this value the `Location` attribute in the `SingleLogoutService` elements in the generated service provider metadata is populated with this value.
- Example: `https://my-overleaf-instance.com/saml/logout/callback`
- `OVERLEAF_SAML_ADDITIONAL_LOGOUT_PARAMS`
* JSON dictionary of additional query params to add to 'logout' requests.
- `OVERLEAF_SAML_IS_ADMIN_FIELD` and `OVERLEAF_SAML_IS_ADMIN_FIELD_VALUE`
* When both environment variables are set, the login process updates `user.isAdmin = true` if the profile returned by the SAML IdP contains the attribute specified by
`OVERLEAF_SAML_IS_ADMIN_FIELD` and its value either matches `OVERLEAF_SAML_IS_ADMIN_FIELD_VALUE` or is an array containing `OVERLEAF_SAML_IS_ADMIN_FIELD_VALUE`,
otherwise `user.isAdmin` is set to `false`. If either of these variables is not set, then the admin status is only set to `true` during admin user.
creation in Launchpad.
#### Metadata for the Identity Provider
The current version of Overleaf CE includes and endpoint to retrieve Service Provider Metadata: `http://my-overleaf-instance.com/saml/meta`
The Identity Provider will need to be configured to recognize the Overleaf server as a "Service Provider". Consult the documentation for your SAML server for instructions on how to do this.
Below is an example of appropriate Service Provider metadata:
<details>
<summary><h5>ol-meta.xml</h5></summary>
```
<?xml version="1.0"?>
<EntityDescriptor xmlns="urn:oasis:names:tc:SAML:2.0:metadata"
xmlns:ds="http://www.w3.org/2000/09/xmldsig#"
entityID="MyOverleaf"
ID="_b508c83b7dda452f5b269383fb391107116f8f57">
<SPSSODescriptor protocolSupportEnumeration="urn:oasis:names:tc:SAML:2.0:protocol" AuthnRequestsSigned="true" WantAssertionsSigned="true">
<KeyDescriptor use="signing">
<ds:KeyInfo>
<ds:X509Data>
<ds:X509Certificate>MII...
[skipped]
</ds:X509Certificate>
</ds:X509Data>
</ds:KeyInfo>
</KeyDescriptor>
<KeyDescriptor use="encryption">
<ds:KeyInfo>
<ds:X509Data>
<ds:X509Certificate>MII...
[skipped]
</ds:X509Certificate>
</ds:X509Data>
</ds:KeyInfo>
<EncryptionMethod Algorithm="http://www.w3.org/2009/xmlenc11#aes256-gcm"/>
<EncryptionMethod Algorithm="http://www.w3.org/2009/xmlenc11#aes128-gcm"/>
<EncryptionMethod Algorithm="http://www.w3.org/2001/04/xmlenc#aes256-cbc"/>
<EncryptionMethod Algorithm="http://www.w3.org/2001/04/xmlenc#aes128-cbc"/>
</KeyDescriptor>
<SingleLogoutService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST"
Location="https://my-overleaf-instance.com/saml/logout/callback"/>
<NameIDFormat>urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress</NameIDFormat>
<AssertionConsumerService index="1"
isDefault="true"
Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST"
Location="https://my-overleaf-instance.com/saml/login/callback"/>
</SPSSODescriptor>
</EntityDescriptor>
```
</details>
Note the certificates, `AssertionConsumerService.Location`, `SingleLogoutService.Location` and `EntityDescriptor.entityID`
and set as appropriate in your IdP configuration, or send the metadata file to the IdP admin.
<details>
<summary><h4>Sample variables.env file</h4></summary>
```
OVERLEAF_APP_NAME="Our Overleaf Instance"
ENABLED_LINKED_FILE_TYPES=project_file,project_output_file
# Enables Thumbnail generation using ImageMagick
ENABLE_CONVERSIONS=true
# Disables email confirmation requirement
EMAIL_CONFIRMATION_DISABLED=true
## Nginx
# NGINX_WORKER_PROCESSES=4
# NGINX_WORKER_CONNECTIONS=768
## Set for TLS via nginx-proxy
# OVERLEAF_BEHIND_PROXY=true
# OVERLEAF_SECURE_COOKIE=true
OVERLEAF_SITE_URL=http://my-overleaf-instance.com
OVERLEAF_NAV_TITLE=Our Overleaf Instance
# OVERLEAF_HEADER_IMAGE_URL=http://somewhere.com/mylogo.png
OVERLEAF_ADMIN_EMAIL=support@example.com
OVERLEAF_LEFT_FOOTER=[{"text": "Contact your support team", "url": "mailto:support@example.com"}]
OVERLEAF_RIGHT_FOOTER=[{"text":"Hello, I am on the Right", "url":"https://github.com/yu-i-i/overleaf-cep"}]
OVERLEAF_EMAIL_FROM_ADDRESS=team@example.com
OVERLEAF_EMAIL_SMTP_HOST=smtp.example.com
OVERLEAF_EMAIL_SMTP_PORT=587
OVERLEAF_EMAIL_SMTP_SECURE=false
# OVERLEAF_EMAIL_SMTP_USER=
# OVERLEAF_EMAIL_SMTP_PASS=
# OVERLEAF_EMAIL_SMTP_NAME=
OVERLEAF_EMAIL_SMTP_LOGGER=false
OVERLEAF_EMAIL_SMTP_TLS_REJECT_UNAUTH=true
OVERLEAF_EMAIL_SMTP_IGNORE_TLS=false
OVERLEAF_CUSTOM_EMAIL_FOOTER=This system is run by department x
OVERLEAF_PROXY_LEARN=true
NAV_HIDE_POWERED_BY=true
#################
## SAML for CE ##
#################
EXTERNAL_AUTH=saml
OVERLEAF_SAML_IDENTITY_SERVICE_NAME='Login with My IdP'
OVERLEAF_SAML_EMAIL_FIELD=mail
OVERLEAF_SAML_FIRST_NAME_FIELD=givenName
OVERLEAF_SAML_LAST_NAME_FIELD=sn
OVERLEAF_SAML_ENTRYPOINT=https://idp.example.com/simplesamlphp/saml2/idp/SSOService.php
OVERLEAF_SAML_CALLBACK_URL=https://my-overleaf-instance.com/saml/login/callback
OVERLEAF_SAML_LOGOUT_URL=https://idp.example.com/simplesamlphp/saml2/idp/SingleLogoutService.php
OVERLEAF_SAML_LOGOUT_CALLBACK_URL=https://my-overleaf-instance.com/saml/logout/callback
OVERLEAF_SAML_ISSUER=MyOverleaf
OVERLEAF_SAML_IDP_CERT=/overleaf/certs/idp_cert.pem
OVERLEAF_SAML_PUBLIC_CERT=/overleaf/certs/myol_cert.pem
OVERLEAF_SAML_PRIVATE_KEY=/overleaf/certs/myol_key.pem
OVERLEAF_SAML_DECRYPTION_CERT=/overleaf/certs/myol_decr_cert.pem
OVERLEAF_SAML_DECRYPTION_PVK=/overleaf/certs/myol_decr_key.pem
OVERLEAF_SAML_IS_ADMIN_FIELD=mail
OVERLEAF_SAML_IS_ADMIN_FIELD_VALUE=overleaf.admin@example.com
```
</details>
</details>
## Overleaf Docker Image
This repo contains two dockerfiles, [`Dockerfile-base`](server-ce/Dockerfile-base), which builds the
`sharelatex/sharelatex-base:ext-ce` image, and [`Dockerfile`](server-ce/Dockerfile) which builds the
`sharelatex/sharelatex:ext-ce` image.
`sharelatex/sharelatex-base` image, and [`Dockerfile`](server-ce/Dockerfile) which builds the
`sharelatex/sharelatex` (or "community") image.
The Base image generally contains the basic dependencies like `wget`, plus `texlive`.
This is split out because it's a pretty heavy set of
The Base image generally contains the basic dependencies like `wget` and
`aspell`, plus `texlive`. We split this out because it's a pretty heavy set of
dependencies, and it's nice to not have to rebuild all of that every time.
The `sharelatex/sharelatex` image extends the base image and adds the actual Overleaf code
@ -67,19 +700,20 @@ and services.
Use `make build-base` and `make build-community` from `server-ce/` to build these images.
The [Phusion base-image](https://github.com/phusion/baseimage-docker)
(which is extended by the `base` image) provides a VM-like container
We use the [Phusion base-image](https://github.com/phusion/baseimage-docker)
(which is extended by our `base` image) to provide us with a VM-like container
in which to run the Overleaf services. Baseimage uses the `runit` service
manager to manage services, and init scripts from the `server-ce/runit`
folder are added.
manager to manage services, and we add our init-scripts from the `server-ce/runit`
folder.
## Authors
[The Overleaf Team](https://www.overleaf.com/about)
[yu-i-i](https://github.com/yu-i-i/overleaf-cep) — Extensions for CE unless otherwise noted
[The Overleaf Team](https://www.overleaf.com/about)
<br>
Extensions for CE by: [yu-i-i](https://github.com/yu-i-i/overleaf-cep)
## License
The code in this repository is released under the GNU AFFERO GENERAL PUBLIC LICENSE, version 3. A copy can be found in the [`LICENSE`](LICENSE) file.
Copyright (c) Overleaf, 2014-2025.
Copyright (c) Overleaf, 2014-2024.

View file

@ -1,3 +0,0 @@
/* eslint-disable no-undef */
rs.initiate({ _id: 'overleaf', members: [{ _id: 0, host: 'mongo:27017' }] })

View file

@ -11,6 +11,12 @@ bin/build
> [!NOTE]
> If Docker is running out of RAM while building the services in parallel, create a `.env` file in this directory containing `COMPOSE_PARALLEL_LIMIT=1`.
Next, initialize the database:
```shell
bin/init
```
Then start the services:
```shell
@ -42,7 +48,7 @@ To do this, use the included `bin/dev` script:
bin/dev
```
This will start all services using `node --watch`, which will automatically monitor the code and restart the services as necessary.
This will start all services using `nodemon`, which will automatically monitor the code and restart the services as necessary.
To improve performance, you can start only a subset of the services in development mode by providing a space-separated list to the `bin/dev` script:
@ -77,7 +83,6 @@ each service:
| `filestore` | 9235 |
| `notifications` | 9236 |
| `real-time` | 9237 |
| `references` | 9238 |
| `history-v1` | 9239 |
| `project-history` | 9240 |

6
develop/bin/init Executable file
View file

@ -0,0 +1,6 @@
#!/usr/bin/env bash
docker compose up --detach mongo
curl --max-time 10 --retry 5 --retry-delay 5 --retry-all-errors --silent --output /dev/null localhost:27017
docker compose exec mongo mongosh --eval "rs.initiate({ _id: 'overleaf', members: [{ _id: 0, host: 'mongo:27017' }] })"
docker compose down mongo

View file

@ -6,18 +6,15 @@ DOCUMENT_UPDATER_HOST=document-updater
FILESTORE_HOST=filestore
GRACEFUL_SHUTDOWN_DELAY_SECONDS=0
HISTORY_V1_HOST=history-v1
HISTORY_REDIS_HOST=redis
LISTEN_ADDRESS=0.0.0.0
MONGO_HOST=mongo
MONGO_URL=mongodb://mongo/sharelatex?directConnection=true
NOTIFICATIONS_HOST=notifications
PROJECT_HISTORY_HOST=project-history
QUEUES_REDIS_HOST=redis
REALTIME_HOST=real-time
REDIS_HOST=redis
REFERENCES_HOST=references
SESSION_SECRET=foo
V1_HISTORY_HOST=history-v1
SPELLING_HOST=spelling
WEBPACK_HOST=webpack
WEB_API_PASSWORD=overleaf
WEB_API_USER=overleaf

View file

@ -117,14 +117,14 @@ services:
environment:
- NODE_OPTIONS=--inspect=0.0.0.0:9229
ports:
- "127.0.0.1:9238:9229"
- "127.0.0.1:9236:9229"
volumes:
- ../services/references/app:/overleaf/services/references/app
- ../services/references/config:/overleaf/services/references/config
- ../services/references/app.js:/overleaf/services/references/app.js
web:
command: ["node", "--watch", "app.mjs", "--watch-locales"]
command: ["node", "--watch", "app.js", "--watch-locales"]
environment:
- NODE_OPTIONS=--inspect=0.0.0.0:9229
ports:

View file

@ -1,5 +1,6 @@
volumes:
clsi-cache:
clsi-output:
filestore-public-files:
filestore-template-files:
filestore-uploads:
@ -7,6 +8,7 @@ volumes:
mongo-data:
redis-data:
sharelatex-data:
spelling-cache:
web-data:
history-v1-buckets:
@ -25,16 +27,15 @@ services:
env_file:
- dev.env
environment:
- DOCKER_RUNNER=true
- TEXLIVE_IMAGE=texlive-full # docker build texlive -t texlive-full
- SANDBOXED_COMPILES=true
- SANDBOXED_COMPILES_HOST_DIR_COMPILES=${PWD}/compiles
- SANDBOXED_COMPILES_HOST_DIR_OUTPUT=${PWD}/output
- COMPILES_HOST_DIR=${PWD}/compiles
user: root
volumes:
- ${PWD}/compiles:/overleaf/services/clsi/compiles
- ${PWD}/output:/overleaf/services/clsi/output
- ${DOCKER_SOCKET_PATH:-/var/run/docker.sock}:/var/run/docker.sock
- clsi-cache:/overleaf/services/clsi/cache
- clsi-output:/overleaf/services/clsi/output
contacts:
build:
@ -88,20 +89,12 @@ services:
- history-v1-buckets:/buckets
mongo:
image: mongo:6.0
image: mongo:5
command: --replSet overleaf
ports:
- "127.0.0.1:27017:27017" # for debugging
volumes:
- mongo-data:/data/db
- ../bin/shared/mongodb-init-replica-set.js:/docker-entrypoint-initdb.d/mongodb-init-replica-set.js
environment:
MONGO_INITDB_DATABASE: sharelatex
extra_hosts:
# Required when using the automatic database setup for initializing the
# replica set. This override is not needed when running the setup after
# starting up mongo.
- mongo:127.0.0.1
notifications:
build:
@ -123,7 +116,7 @@ services:
dockerfile: services/real-time/Dockerfile
env_file:
- dev.env
redis:
image: redis:5
ports:
@ -138,6 +131,15 @@ services:
env_file:
- dev.env
spelling:
build:
context: ..
dockerfile: services/spelling/Dockerfile
env_file:
- dev.env
volumes:
- spelling-cache:/overleaf/services/spelling/cache
web:
build:
context: ..
@ -147,11 +149,11 @@ services:
- dev.env
environment:
- APP_NAME=Overleaf Community Edition
- ENABLED_LINKED_FILE_TYPES=project_file,project_output_file,url
- ENABLED_LINKED_FILE_TYPES=project_file,project_output_file
- EMAIL_CONFIRMATION_DISABLED=true
- NODE_ENV=development
- OVERLEAF_ALLOW_PUBLIC_ACCESS=true
command: ["node", "app.mjs"]
command: ["node", "app.js"]
volumes:
- sharelatex-data:/var/lib/overleaf
- web-data:/overleaf/services/web/data
@ -169,12 +171,13 @@ services:
- project-history
- real-time
- references
- spelling
webpack:
build:
context: ..
dockerfile: services/web/Dockerfile
target: webpack
target: dev
command: ["npx", "webpack", "serve", "--config", "webpack.config.dev-env.js"]
ports:
- "127.0.0.1:80:3808"

Binary file not shown.

Before

Width:  |  Height:  |  Size: 13 KiB

After

Width:  |  Height:  |  Size: 71 KiB

Before After
Before After

View file

@ -32,7 +32,7 @@ services:
OVERLEAF_REDIS_HOST: redis
REDIS_HOST: redis
ENABLED_LINKED_FILE_TYPES: 'project_file,project_output_file,url'
ENABLED_LINKED_FILE_TYPES: 'project_file,project_output_file'
# Enables Thumbnail generation using ImageMagick
ENABLE_CONVERSIONS: 'true'
@ -40,6 +40,10 @@ services:
# Disables email confirmation requirement
EMAIL_CONFIRMATION_DISABLED: 'true'
# temporary fix for LuaLaTex compiles
# see https://github.com/overleaf/overleaf/issues/695
TEXMFVAR: /var/lib/overleaf/tmp/texmf-var
## Set for SSL via nginx-proxy
#VIRTUAL_HOST: 103.112.212.22
@ -73,19 +77,11 @@ services:
## Server Pro ##
################
## The Community Edition is intended for use in environments where all users are trusted and is not appropriate for
## scenarios where isolation of users is required. Sandboxed Compiles are not available in the Community Edition,
## so the following environment variables must be commented out to avoid compile issues.
##
## Sandboxed Compiles: https://docs.overleaf.com/on-premises/configuration/overleaf-toolkit/server-pro-only-configuration/sandboxed-compiles
## Sandboxed Compiles: https://github.com/overleaf/overleaf/wiki/Server-Pro:-Sandboxed-Compiles
SANDBOXED_COMPILES: 'true'
### Bind-mount source for /var/lib/overleaf/data/compiles inside the container.
SANDBOXED_COMPILES_HOST_DIR_COMPILES: '/home/user/sharelatex_data/data/compiles'
### Bind-mount source for /var/lib/overleaf/data/output inside the container.
SANDBOXED_COMPILES_HOST_DIR_OUTPUT: '/home/user/sharelatex_data/data/output'
### Backwards compatibility (before Server Pro 5.5)
DOCKER_RUNNER: 'true'
SANDBOXED_COMPILES_SIBLING_CONTAINERS: 'true'
### Bind-mount source for /var/lib/overleaf/data/compiles inside the container.
SANDBOXED_COMPILES_HOST_DIR: '/home/user/sharelatex_data/data/compiles'
## Works with test LDAP server shown at bottom of docker compose
# OVERLEAF_LDAP_URL: 'ldap://ldap:389'
@ -106,12 +102,12 @@ services:
mongo:
restart: always
image: mongo:6.0
image: mongo:5.0
container_name: mongo
command: '--replSet overleaf'
volumes:
- ~/mongo_data:/data/db
- ./bin/shared/mongodb-init-replica-set.js:/docker-entrypoint-initdb.d/mongodb-init-replica-set.js
- ./mongodb-init-replica-set.js:/docker-entrypoint-initdb.d/mongodb-init-replica-set.js
environment:
MONGO_INITDB_DATABASE: sharelatex
extra_hosts:
@ -119,7 +115,7 @@ services:
# This override is not needed when running the setup after starting up mongo.
- mongo:127.0.0.1
healthcheck:
test: echo 'db.stats().ok' | mongosh localhost:27017/test --quiet
test: echo 'db.stats().ok' | mongo localhost:27017/test --quiet
interval: 10s
timeout: 10s
retries: 5

View file

@ -0,0 +1 @@
node_modules/

View file

@ -0,0 +1,46 @@
compileFolder
Compiled source #
###################
*.com
*.class
*.dll
*.exe
*.o
*.so
# Packages #
############
# it's better to unpack these files and commit the raw source
# git has its own built in compression methods
*.7z
*.dmg
*.gz
*.iso
*.jar
*.rar
*.tar
*.zip
# Logs and databases #
######################
*.log
*.sql
*.sqlite
# OS generated files #
######################
.DS_Store?
ehthumbs.db
Icon?
Thumbs.db
/node_modules/*
data/*/*
**.swp
/log.json
hash_folder
.npmrc

View file

@ -1 +1 @@
22.17.0
18.20.2

View file

@ -1,10 +1,10 @@
access-token-encryptor
--dependencies=None
--docker-repos=us-east1-docker.pkg.dev/overleaf-ops/ol-docker
--docker-repos=gcr.io/overleaf-ops
--env-add=
--env-pass-through=
--esmock-loader=False
--is-library=True
--node-version=22.17.0
--node-version=18.20.2
--public-repo=False
--script-version=4.7.0
--script-version=4.5.0

View file

@ -1,5 +1,5 @@
const { promisify } = require('node:util')
const crypto = require('node:crypto')
const { promisify } = require('util')
const crypto = require('crypto')
const ALGORITHM = 'aes-256-ctr'

View file

@ -21,7 +21,7 @@
"devDependencies": {
"chai": "^4.3.6",
"chai-as-promised": "^7.1.1",
"mocha": "^11.1.0",
"mocha": "^10.2.0",
"sandboxed-module": "^2.0.4",
"typescript": "^5.0.4"
}

View file

@ -1,13 +1,4 @@
const chai = require('chai')
const chaiAsPromised = require('chai-as-promised')
const SandboxedModule = require('sandboxed-module')
chai.use(chaiAsPromised)
SandboxedModule.configure({
sourceTransformers: {
removeNodePrefix: function (source) {
return source.replace(/require\(['"]node:/g, "require('")
},
},
})

View file

@ -0,0 +1 @@
node_modules/

3
libraries/fetch-utils/.gitignore vendored Normal file
View file

@ -0,0 +1,3 @@
# managed by monorepo$ bin/update_build_scripts
.npmrc

View file

@ -1 +1 @@
22.17.0
18.20.2

View file

@ -1,10 +1,10 @@
fetch-utils
--dependencies=None
--docker-repos=us-east1-docker.pkg.dev/overleaf-ops/ol-docker
--docker-repos=gcr.io/overleaf-ops
--env-add=
--env-pass-through=
--esmock-loader=False
--is-library=True
--node-version=22.17.0
--node-version=18.20.2
--public-repo=False
--script-version=4.7.0
--script-version=4.5.0

View file

@ -1,9 +1,9 @@
const _ = require('lodash')
const { Readable } = require('node:stream')
const { Readable } = require('stream')
const OError = require('@overleaf/o-error')
const fetch = require('node-fetch')
const http = require('node:http')
const https = require('node:https')
const http = require('http')
const https = require('https')
/**
* @import { Response } from 'node-fetch'
@ -23,11 +23,11 @@ async function fetchJson(url, opts = {}) {
}
async function fetchJsonWithResponse(url, opts = {}) {
const { fetchOpts, detachSignal } = parseOpts(opts)
const { fetchOpts } = parseOpts(opts)
fetchOpts.headers = fetchOpts.headers ?? {}
fetchOpts.headers.Accept = fetchOpts.headers.Accept ?? 'application/json'
const response = await performRequest(url, fetchOpts, detachSignal)
const response = await performRequest(url, fetchOpts)
if (!response.ok) {
const body = await maybeGetResponseBody(response)
throw new RequestFailedError(url, opts, response, body)
@ -53,8 +53,8 @@ async function fetchStream(url, opts = {}) {
}
async function fetchStreamWithResponse(url, opts = {}) {
const { fetchOpts, abortController, detachSignal } = parseOpts(opts)
const response = await performRequest(url, fetchOpts, detachSignal)
const { fetchOpts, abortController } = parseOpts(opts)
const response = await performRequest(url, fetchOpts)
if (!response.ok) {
const body = await maybeGetResponseBody(response)
@ -76,8 +76,8 @@ async function fetchStreamWithResponse(url, opts = {}) {
* @throws {RequestFailedError} if the response has a failure status code
*/
async function fetchNothing(url, opts = {}) {
const { fetchOpts, detachSignal } = parseOpts(opts)
const response = await performRequest(url, fetchOpts, detachSignal)
const { fetchOpts } = parseOpts(opts)
const response = await performRequest(url, fetchOpts)
if (!response.ok) {
const body = await maybeGetResponseBody(response)
throw new RequestFailedError(url, opts, response, body)
@ -95,22 +95,9 @@ async function fetchNothing(url, opts = {}) {
* @throws {RequestFailedError} if the response has a non redirect status code or missing Location header
*/
async function fetchRedirect(url, opts = {}) {
const { location } = await fetchRedirectWithResponse(url, opts)
return location
}
/**
* Make a request and extract the redirect from the response.
*
* @param {string | URL} url - request URL
* @param {object} opts - fetch options
* @return {Promise<{location: string, response: Response}>}
* @throws {RequestFailedError} if the response has a non redirect status code or missing Location header
*/
async function fetchRedirectWithResponse(url, opts = {}) {
const { fetchOpts, detachSignal } = parseOpts(opts)
const { fetchOpts } = parseOpts(opts)
fetchOpts.redirect = 'manual'
const response = await performRequest(url, fetchOpts, detachSignal)
const response = await performRequest(url, fetchOpts)
if (response.status < 300 || response.status >= 400) {
const body = await maybeGetResponseBody(response)
throw new RequestFailedError(url, opts, response, body)
@ -125,7 +112,7 @@ async function fetchRedirectWithResponse(url, opts = {}) {
)
}
await discardResponseBody(response)
return { location, response }
return location
}
/**
@ -142,8 +129,8 @@ async function fetchString(url, opts = {}) {
}
async function fetchStringWithResponse(url, opts = {}) {
const { fetchOpts, detachSignal } = parseOpts(opts)
const response = await performRequest(url, fetchOpts, detachSignal)
const { fetchOpts } = parseOpts(opts)
const response = await performRequest(url, fetchOpts)
if (!response.ok) {
const body = await maybeGetResponseBody(response)
throw new RequestFailedError(url, opts, response, body)
@ -178,14 +165,13 @@ function parseOpts(opts) {
const abortController = new AbortController()
fetchOpts.signal = abortController.signal
let detachSignal = () => {}
if (opts.signal) {
detachSignal = abortOnSignal(abortController, opts.signal)
abortOnSignal(abortController, opts.signal)
}
if (opts.body instanceof Readable) {
abortOnDestroyedRequest(abortController, fetchOpts.body)
}
return { fetchOpts, abortController, detachSignal }
return { fetchOpts, abortController }
}
function setupJsonBody(fetchOpts, json) {
@ -209,9 +195,6 @@ function abortOnSignal(abortController, signal) {
abortController.abort(signal.reason)
}
signal.addEventListener('abort', listener)
return () => {
signal.removeEventListener('abort', listener)
}
}
function abortOnDestroyedRequest(abortController, stream) {
@ -230,12 +213,11 @@ function abortOnDestroyedResponse(abortController, response) {
})
}
async function performRequest(url, fetchOpts, detachSignal) {
async function performRequest(url, fetchOpts) {
let response
try {
response = await fetch(url, fetchOpts)
} catch (err) {
detachSignal()
if (fetchOpts.body instanceof Readable) {
fetchOpts.body.destroy()
}
@ -244,7 +226,6 @@ async function performRequest(url, fetchOpts, detachSignal) {
method: fetchOpts.method ?? 'GET',
})
}
response.body.on('close', detachSignal)
if (fetchOpts.body instanceof Readable) {
response.body.on('close', () => {
if (!fetchOpts.body.readableEnded) {
@ -316,7 +297,6 @@ module.exports = {
fetchStreamWithResponse,
fetchNothing,
fetchRedirect,
fetchRedirectWithResponse,
fetchString,
fetchStringWithResponse,
RequestFailedError,

View file

@ -20,8 +20,8 @@
"body-parser": "^1.20.3",
"chai": "^4.3.6",
"chai-as-promised": "^7.1.1",
"express": "^4.21.2",
"mocha": "^11.1.0",
"express": "^4.21.0",
"mocha": "^10.2.0",
"typescript": "^5.0.4"
},
"dependencies": {

View file

@ -1,10 +1,7 @@
const { expect } = require('chai')
const fs = require('node:fs')
const events = require('node:events')
const { FetchError, AbortError } = require('node-fetch')
const { Readable } = require('node:stream')
const { pipeline } = require('node:stream/promises')
const { once } = require('node:events')
const { Readable } = require('stream')
const { once } = require('events')
const { TestServer } = require('./helpers/TestServer')
const selfsigned = require('selfsigned')
const {
@ -27,17 +24,13 @@ const pems = selfsigned.generate(attrs, { days: 365 })
const PRIVATE_KEY = pems.private
const PUBLIC_CERT = pems.cert
const dns = require('node:dns')
const dns = require('dns')
const _originalLookup = dns.lookup
// Custom DNS resolver function
dns.lookup = (hostname, options, callback) => {
if (hostname === 'example.com') {
// If the hostname is our test case, return the ip address for the test server
if (options?.all) {
callback(null, [{ address: '127.0.0.1', family: 4 }])
} else {
callback(null, '127.0.0.1', 4)
}
callback(null, '127.0.0.1', 4)
} else {
// Otherwise, use the default lookup
_originalLookup(hostname, options, callback)
@ -206,31 +199,6 @@ describe('fetch-utils', function () {
).to.be.rejectedWith(AbortError)
expect(stream.destroyed).to.be.true
})
it('detaches from signal on success', async function () {
const signal = AbortSignal.timeout(10_000)
for (let i = 0; i < 20; i++) {
const s = await fetchStream(this.url('/hello'), { signal })
expect(events.getEventListeners(signal, 'abort')).to.have.length(1)
await pipeline(s, fs.createWriteStream('/dev/null'))
expect(events.getEventListeners(signal, 'abort')).to.have.length(0)
}
})
it('detaches from signal on error', async function () {
const signal = AbortSignal.timeout(10_000)
for (let i = 0; i < 20; i++) {
try {
await fetchStream(this.url('/500'), { signal })
} catch (err) {
if (err instanceof RequestFailedError && err.response.status === 500)
continue
throw err
} finally {
expect(events.getEventListeners(signal, 'abort')).to.have.length(0)
}
}
})
})
describe('fetchNothing', function () {
@ -419,16 +387,9 @@ async function* infiniteIterator() {
async function abortOnceReceived(func, server) {
const controller = new AbortController()
const promise = func(controller.signal)
expect(events.getEventListeners(controller.signal, 'abort')).to.have.length(1)
await once(server.events, 'request-received')
controller.abort()
try {
return await promise
} finally {
expect(events.getEventListeners(controller.signal, 'abort')).to.have.length(
0
)
}
return await promise
}
async function expectRequestAborted(req) {

View file

@ -1,9 +1,9 @@
const express = require('express')
const bodyParser = require('body-parser')
const { EventEmitter } = require('node:events')
const http = require('node:http')
const https = require('node:https')
const { promisify } = require('node:util')
const { EventEmitter } = require('events')
const http = require('http')
const https = require('https')
const { promisify } = require('util')
class TestServer {
constructor() {

View file

@ -0,0 +1 @@
node_modules/

3
libraries/logger/.gitignore vendored Normal file
View file

@ -0,0 +1,3 @@
node_modules
.npmrc

View file

@ -1 +1 @@
22.17.0
18.20.2

View file

@ -1,10 +1,10 @@
logger
--dependencies=None
--docker-repos=us-east1-docker.pkg.dev/overleaf-ops/ol-docker
--docker-repos=gcr.io/overleaf-ops
--env-add=
--env-pass-through=
--esmock-loader=False
--is-library=True
--node-version=22.17.0
--node-version=18.20.2
--public-repo=False
--script-version=4.7.0
--script-version=4.5.0

View file

@ -1,5 +1,5 @@
const { fetchString } = require('@overleaf/fetch-utils')
const fs = require('node:fs')
const fs = require('fs')
class LogLevelChecker {
constructor(logger, defaultLevel) {

View file

@ -1,6 +1,7 @@
const Stream = require('node:stream')
const Stream = require('stream')
const bunyan = require('bunyan')
const GCPManager = require('./gcp-manager')
const SentryManager = require('./sentry-manager')
const Serializers = require('./serializers')
const {
FileLogLevelChecker,
@ -14,10 +15,8 @@ const LoggingManager = {
initialize(name) {
this.isProduction =
(process.env.NODE_ENV || '').toLowerCase() === 'production'
const isTest = (process.env.NODE_ENV || '').toLowerCase() === 'test'
this.defaultLevel =
process.env.LOG_LEVEL ||
(this.isProduction ? 'info' : isTest ? 'fatal' : 'debug')
process.env.LOG_LEVEL || (this.isProduction ? 'info' : 'debug')
this.loggerName = name
this.logger = bunyan.createLogger({
name,
@ -34,6 +33,10 @@ const LoggingManager = {
return this
},
initializeErrorReporting(dsn, options) {
this.sentryManager = new SentryManager()
},
/**
* @param {Record<string, any>|string} attributes - Attributes to log (nice serialization for err, req, res)
* @param {string} [message] - Optional message
@ -65,6 +68,9 @@ const LoggingManager = {
})
}
this.logger.error(attributes, message, ...Array.from(args))
if (this.sentryManager) {
this.sentryManager.captureExceptionRateLimited(attributes, message)
}
},
/**
@ -92,6 +98,9 @@ const LoggingManager = {
*/
fatal(attributes, message) {
this.logger.fatal(attributes, message)
if (this.sentryManager) {
this.sentryManager.captureException(attributes, message, 'fatal')
}
},
_getOutputStreamConfig() {

View file

@ -23,11 +23,12 @@
"@google-cloud/logging-bunyan": "^5.1.0",
"@overleaf/fetch-utils": "*",
"@overleaf/o-error": "*",
"@sentry/node": "^6.13.2",
"bunyan": "^1.8.14"
},
"devDependencies": {
"chai": "^4.3.6",
"mocha": "^11.1.0",
"mocha": "^10.2.0",
"sandboxed-module": "^2.0.4",
"sinon": "^9.2.4",
"sinon-chai": "^3.7.0",

View file

@ -0,0 +1,106 @@
const Serializers = require('./serializers')
const RATE_LIMIT_MAX_ERRORS = 5
const RATE_LIMIT_INTERVAL_MS = 60000
class SentryManager {
constructor(dsn, options) {
this.Sentry = require('@sentry/node')
this.Sentry.init({ dsn, ...options })
// for rate limiting on sentry reporting
this.lastErrorTimeStamp = 0
this.lastErrorCount = 0
}
captureExceptionRateLimited(attributes, message) {
const now = Date.now()
// have we recently reported an error?
const recentSentryReport =
now - this.lastErrorTimeStamp < RATE_LIMIT_INTERVAL_MS
// if so, increment the error count
if (recentSentryReport) {
this.lastErrorCount++
} else {
this.lastErrorCount = 0
this.lastErrorTimeStamp = now
}
// only report 5 errors every minute to avoid overload
if (this.lastErrorCount < RATE_LIMIT_MAX_ERRORS) {
// add a note if the rate limit has been hit
const note =
this.lastErrorCount + 1 === RATE_LIMIT_MAX_ERRORS
? '(rate limited)'
: ''
// report the exception
this.captureException(attributes, message, `error${note}`)
}
}
captureException(attributes, message, level) {
// handle case of logger.error "message"
if (typeof attributes === 'string') {
attributes = { err: new Error(attributes) }
}
// extract any error object
let error = Serializers.err(attributes.err || attributes.error)
// avoid reporting errors twice
for (const key in attributes) {
const value = attributes[key]
if (value instanceof Error && value.reportedToSentry) {
return
}
}
// include our log message in the error report
if (error == null) {
if (typeof message === 'string') {
error = { message }
}
} else if (message != null) {
attributes.description = message
}
// report the error
if (error != null) {
// capture attributes and use *_id objects as tags
const tags = {}
const extra = {}
for (const key in attributes) {
let value = attributes[key]
if (Serializers[key]) {
value = Serializers[key](value)
}
if (key.match(/_id/) && typeof value === 'string') {
tags[key] = value
}
extra[key] = value
}
// OError integration
extra.info = error.info
delete error.info
// Sentry wants to receive an Error instance.
const errInstance = new Error(error.message)
Object.assign(errInstance, error)
try {
// send the error to sentry
this.Sentry.captureException(errInstance, { tags, extra, level })
// put a flag on the errors to avoid reporting them multiple times
for (const key in attributes) {
const value = attributes[key]
if (value instanceof Error) {
value.reportedToSentry = true
}
}
} catch (err) {
// ignore Sentry errors
}
}
}
}
module.exports = SentryManager

View file

@ -8,9 +8,4 @@ chai.use(sinonChai)
SandboxedModule.configure({
globals: { Buffer, JSON, console, process },
sourceTransformers: {
removeNodePrefix: function (source) {
return source.replace(/require\(['"]node:/g, "require('")
},
},
})

View file

@ -1,5 +1,5 @@
const Path = require('node:path')
const { promisify } = require('node:util')
const Path = require('path')
const { promisify } = require('util')
const SandboxedModule = require('sandboxed-module')
const sinon = require('sinon')
const { expect } = require('chai')

View file

@ -1,7 +1,7 @@
const SandboxedModule = require('sandboxed-module')
const bunyan = require('bunyan')
const { expect } = require('chai')
const path = require('node:path')
const path = require('path')
const sinon = require('sinon')
const MODULE_PATH = path.join(__dirname, '../../logging-manager.js')
@ -43,14 +43,20 @@ describe('LoggingManager', function () {
.stub()
.returns(this.GCEMetadataLogLevelChecker),
}
this.SentryManager = {
captureException: sinon.stub(),
captureExceptionRateLimited: sinon.stub(),
}
this.LoggingManager = SandboxedModule.require(MODULE_PATH, {
requires: {
bunyan: this.Bunyan,
'./log-level-checker': this.LogLevelChecker,
'./sentry-manager': sinon.stub().returns(this.SentryManager),
},
})
this.loggerName = 'test'
this.logger = this.LoggingManager.initialize(this.loggerName)
this.logger.initializeErrorReporting('test_dsn')
})
afterEach(function () {
@ -154,6 +160,13 @@ describe('LoggingManager', function () {
})
})
describe('logger.error', function () {
it('should report errors to Sentry', function () {
this.logger.error({ foo: 'bar' }, 'message')
expect(this.SentryManager.captureExceptionRateLimited).to.have.been.called
})
})
describe('ringbuffer', function () {
beforeEach(function () {
this.logBufferMock = [

View file

@ -0,0 +1,247 @@
const Path = require('path')
const SandboxedModule = require('sandboxed-module')
const { expect } = require('chai')
const sinon = require('sinon')
const MODULE_PATH = Path.join(__dirname, '../../sentry-manager.js')
describe('SentryManager', function () {
beforeEach(function () {
this.clock = sinon.useFakeTimers(Date.now())
this.Sentry = {
init: sinon.stub(),
captureException: sinon.stub(),
}
this.SentryManager = SandboxedModule.require(MODULE_PATH, {
requires: {
'@sentry/node': this.Sentry,
},
})
this.sentryManager = new this.SentryManager('test_dsn')
})
afterEach(function () {
this.clock.restore()
})
describe('captureExceptionRateLimited', function () {
it('should report a single error to sentry', function () {
this.sentryManager.captureExceptionRateLimited({ foo: 'bar' }, 'message')
expect(this.Sentry.captureException).to.have.been.calledOnce
})
it('should report the same error to sentry only once', function () {
const error1 = new Error('this is the error')
this.sentryManager.captureExceptionRateLimited(
{ foo: error1 },
'first message'
)
this.sentryManager.captureExceptionRateLimited(
{ bar: error1 },
'second message'
)
expect(this.Sentry.captureException).to.have.been.calledOnce
})
it('should report two different errors to sentry individually', function () {
const error1 = new Error('this is the error')
const error2 = new Error('this is the error')
this.sentryManager.captureExceptionRateLimited(
{ foo: error1 },
'first message'
)
this.sentryManager.captureExceptionRateLimited(
{ bar: error2 },
'second message'
)
expect(this.Sentry.captureException).to.have.been.calledTwice
})
it('for multiple errors should only report a maximum of 5 errors to sentry', function () {
for (let i = 0; i < 10; i++) {
this.sentryManager.captureExceptionRateLimited(
{ foo: 'bar' },
'message'
)
}
expect(this.Sentry.captureException).to.have.callCount(5)
})
it('for multiple errors with a minute delay should report 10 errors to sentry', function () {
for (let i = 0; i < 10; i++) {
this.sentryManager.captureExceptionRateLimited(
{ foo: 'bar' },
'message'
)
}
expect(this.Sentry.captureException).to.have.callCount(5)
// allow a minute to pass
this.clock.tick(61 * 1000)
for (let i = 0; i < 10; i++) {
this.sentryManager.captureExceptionRateLimited(
{ foo: 'bar' },
'message'
)
}
expect(this.Sentry.captureException).to.have.callCount(10)
})
})
describe('captureException', function () {
it('should remove the path from fs errors', function () {
const fsError = new Error(
"Error: ENOENT: no such file or directory, stat '/tmp/3279b8d0-da10-11e8-8255-efd98985942b'"
)
fsError.path = '/tmp/3279b8d0-da10-11e8-8255-efd98985942b'
this.sentryManager.captureException({ err: fsError }, 'message', 'error')
expect(this.Sentry.captureException).to.have.been.calledWith(
sinon.match.has(
'message',
'Error: ENOENT: no such file or directory, stat'
)
)
})
it('should sanitize error', function () {
const err = {
name: 'CustomError',
message: 'hello',
_oErrorTags: [{ stack: 'here:1', info: { one: 1 } }],
stack: 'here:0',
info: { key: 'value' },
code: 42,
signal: 9,
path: '/foo',
}
this.sentryManager.captureException({ err }, 'message', 'error')
const expectedErr = {
name: 'CustomError',
message: 'hello',
stack: 'here:0\nhere:1',
code: 42,
signal: 9,
path: '/foo',
}
expect(this.Sentry.captureException).to.have.been.calledWith(
sinon.match(expectedErr),
sinon.match({
tags: sinon.match.any,
level: sinon.match.any,
extra: {
description: 'message',
info: sinon.match({
one: 1,
key: 'value',
}),
},
})
)
// Chai is very picky with comparing Error instances. Go the long way of comparing all the fields manually.
const gotErr = this.Sentry.captureException.args[0][0]
for (const [key, wanted] of Object.entries(expectedErr)) {
expect(gotErr).to.have.property(key, wanted)
}
})
it('should sanitize request', function () {
const req = {
ip: '1.2.3.4',
method: 'GET',
url: '/foo',
headers: {
referer: 'abc',
'content-length': 1337,
'user-agent': 'curl',
authorization: '42',
},
}
this.sentryManager.captureException({ req }, 'message', 'error')
const expectedReq = {
remoteAddress: '1.2.3.4',
method: 'GET',
url: '/foo',
headers: {
referer: 'abc',
'content-length': 1337,
'user-agent': 'curl',
},
}
expect(this.Sentry.captureException).to.have.been.calledWith(
sinon.match({
message: 'message',
}),
sinon.match({
tags: sinon.match.any,
level: sinon.match.any,
extra: {
info: sinon.match.any,
req: expectedReq,
},
})
)
expect(this.Sentry.captureException.args[0][1].extra.req).to.deep.equal(
expectedReq
)
})
it('should sanitize response', function () {
const res = {
statusCode: 417,
body: Buffer.from('foo'),
getHeader(key) {
expect(key).to.be.oneOf(['content-length'])
if (key === 'content-length') return 1337
},
}
this.sentryManager.captureException({ res }, 'message', 'error')
const expectedRes = {
statusCode: 417,
headers: {
'content-length': 1337,
},
}
expect(this.Sentry.captureException).to.have.been.calledWith(
sinon.match({
message: 'message',
}),
sinon.match({
tags: sinon.match.any,
level: sinon.match.any,
extra: {
info: sinon.match.any,
res: expectedRes,
},
})
)
expect(this.Sentry.captureException.args[0][1].extra.res).to.deep.equal(
expectedRes
)
})
describe('reportedToSentry', function () {
it('should mark the error as reported to sentry', function () {
const err = new Error()
this.sentryManager.captureException({ err }, 'message')
expect(this.Sentry.captureException).to.have.been.called
expect(err.reportedToSentry).to.equal(true)
})
it('should mark two errors as reported to sentry', function () {
const err1 = new Error()
const err2 = new Error()
this.sentryManager.captureException({ err: err1, err2 }, 'message')
expect(this.Sentry.captureException).to.have.been.called
expect(err1.reportedToSentry).to.equal(true)
expect(err2.reportedToSentry).to.equal(true)
})
it('should not mark arbitrary objects as reported to sentry', function () {
const err = new Error()
const ctx = { foo: 'bar' }
this.sentryManager.captureException({ err, ctx }, 'message')
expect(this.Sentry.captureException).to.have.been.called
expect(ctx.reportedToSentry).not.to.exist
})
})
})
})

View file

@ -0,0 +1 @@
node_modules/

3
libraries/metrics/.gitignore vendored Normal file
View file

@ -0,0 +1,3 @@
node_modules
.npmrc

View file

@ -1 +1 @@
22.17.0
18.20.2

View file

@ -1,10 +1,10 @@
metrics
--dependencies=None
--docker-repos=us-east1-docker.pkg.dev/overleaf-ops/ol-docker
--docker-repos=gcr.io/overleaf-ops
--env-add=
--env-pass-through=
--esmock-loader=False
--is-library=True
--node-version=22.17.0
--node-version=18.20.2
--public-repo=False
--script-version=4.7.0
--script-version=4.5.0

View file

@ -5,8 +5,6 @@
* before any other module to support code instrumentation.
*/
const metricsModuleImportStartTime = performance.now()
const APP_NAME = process.env.METRICS_APP_NAME || 'unknown'
const BUILD_VERSION = process.env.BUILD_VERSION
const ENABLE_PROFILE_AGENT = process.env.ENABLE_PROFILE_AGENT === 'true'
@ -90,7 +88,7 @@ function initializeProfileAgent() {
}
function initializePrometheus() {
const os = require('node:os')
const os = require('os')
const promClient = require('prom-client')
promClient.register.setDefaultLabels({ app: APP_NAME, host: os.hostname() })
promClient.collectDefaultMetrics({ timeout: 5000, prefix: '' })
@ -105,5 +103,3 @@ function recordProcessStart() {
const metrics = require('.')
metrics.inc('process_startup')
}
module.exports = { metricsModuleImportStartTime }

View file

@ -6,8 +6,8 @@
* logged along with the corresponding information from /proc/net/tcp.
*/
const fs = require('node:fs')
const diagnosticsChannel = require('node:diagnostics_channel')
const fs = require('fs')
const diagnosticsChannel = require('diagnostics_channel')
const SOCKET_MONITOR_INTERVAL = 60 * 1000
// set the threshold for logging leaked sockets in minutes, defaults to 15

View file

@ -11,13 +11,13 @@ const seconds = 1000
// In Node 0.10 the default is 5, which means only 5 open connections at one.
// Node 0.12 has a default of Infinity. Make sure we have no limit set,
// regardless of Node version.
require('node:http').globalAgent.maxSockets = Infinity
require('node:https').globalAgent.maxSockets = Infinity
require('http').globalAgent.maxSockets = Infinity
require('https').globalAgent.maxSockets = Infinity
const SOCKETS_HTTP = require('node:http').globalAgent.sockets
const SOCKETS_HTTPS = require('node:https').globalAgent.sockets
const FREE_SOCKETS_HTTP = require('node:http').globalAgent.freeSockets
const FREE_SOCKETS_HTTPS = require('node:https').globalAgent.freeSockets
const SOCKETS_HTTP = require('http').globalAgent.sockets
const SOCKETS_HTTPS = require('https').globalAgent.sockets
const FREE_SOCKETS_HTTP = require('http').globalAgent.freeSockets
const FREE_SOCKETS_HTTPS = require('https').globalAgent.freeSockets
// keep track of set gauges and reset them in the next collection cycle
const SEEN_HOSTS_HTTP = new Set()

View file

@ -9,7 +9,7 @@
"main": "index.js",
"dependencies": {
"@google-cloud/opentelemetry-cloud-trace-exporter": "^2.1.0",
"@google-cloud/profiler": "^6.0.3",
"@google-cloud/profiler": "^6.0.0",
"@opentelemetry/api": "^1.4.1",
"@opentelemetry/auto-instrumentations-node": "^0.39.1",
"@opentelemetry/exporter-trace-otlp-http": "^0.41.2",
@ -23,7 +23,7 @@
"devDependencies": {
"bunyan": "^1.0.0",
"chai": "^4.3.6",
"mocha": "^11.1.0",
"mocha": "^10.2.0",
"sandboxed-module": "^2.0.4",
"sinon": "^9.2.4",
"typescript": "^5.0.4"

View file

@ -1,6 +1,6 @@
const { promisify } = require('node:util')
const os = require('node:os')
const http = require('node:http')
const { promisify } = require('util')
const os = require('os')
const http = require('http')
const { expect } = require('chai')
const Metrics = require('../..')
@ -316,7 +316,7 @@ async function checkSummaryValues(key, values) {
for (const quantile of Object.keys(values)) {
expect(found[quantile]).to.be.within(
values[quantile] - 5,
values[quantile] + 15,
values[quantile] + 5,
`quantile: ${quantile}`
)
}

View file

@ -5,7 +5,7 @@
*/
const chai = require('chai')
const { expect } = chai
const path = require('node:path')
const path = require('path')
const modulePath = path.join(__dirname, '../../../event_loop.js')
const SandboxedModule = require('sandboxed-module')
const sinon = require('sinon')

View file

@ -1,4 +1,4 @@
const Path = require('node:path')
const Path = require('path')
const SandboxedModule = require('sandboxed-module')
const sinon = require('sinon')

View file

@ -1 +0,0 @@
22.17.0

View file

@ -1,321 +0,0 @@
// @ts-check
/* eslint-disable no-console */
const { ObjectId, ReadPreference } = require('mongodb')
const READ_PREFERENCE_SECONDARY =
process.env.MONGO_HAS_SECONDARIES === 'true'
? ReadPreference.secondary.mode
: ReadPreference.secondaryPreferred.mode
const ONE_MONTH_IN_MS = 1000 * 60 * 60 * 24 * 31
let ID_EDGE_PAST
const ID_EDGE_FUTURE = objectIdFromMs(Date.now() + 1000)
let BATCH_DESCENDING
let BATCH_SIZE
let VERBOSE_LOGGING
let BATCH_RANGE_START
let BATCH_RANGE_END
let BATCH_MAX_TIME_SPAN_IN_MS
let BATCHED_UPDATE_RUNNING = false
/**
* @typedef {import("mongodb").Collection} Collection
* @typedef {import("mongodb-legacy").Collection} LegacyCollection
* @typedef {import("mongodb").Document} Document
* @typedef {import("mongodb").FindOptions} FindOptions
* @typedef {import("mongodb").UpdateFilter<Document>} UpdateDocument
*/
/**
* @typedef {Object} BatchedUpdateOptions
* @property {string} [BATCH_DESCENDING]
* @property {string} [BATCH_LAST_ID]
* @property {string} [BATCH_MAX_TIME_SPAN_IN_MS]
* @property {string} [BATCH_RANGE_END]
* @property {string} [BATCH_RANGE_START]
* @property {string} [BATCH_SIZE]
* @property {string} [VERBOSE_LOGGING]
* @property {(progress: string) => Promise<void>} [trackProgress]
*/
/**
* @param {BatchedUpdateOptions} options
*/
function refreshGlobalOptionsForBatchedUpdate(options = {}) {
options = Object.assign({}, options, process.env)
BATCH_DESCENDING = options.BATCH_DESCENDING === 'true'
BATCH_SIZE = parseInt(options.BATCH_SIZE || '1000', 10) || 1000
VERBOSE_LOGGING = options.VERBOSE_LOGGING === 'true'
if (options.BATCH_LAST_ID) {
BATCH_RANGE_START = objectIdFromInput(options.BATCH_LAST_ID)
} else if (options.BATCH_RANGE_START) {
BATCH_RANGE_START = objectIdFromInput(options.BATCH_RANGE_START)
} else {
if (BATCH_DESCENDING) {
BATCH_RANGE_START = ID_EDGE_FUTURE
} else {
BATCH_RANGE_START = ID_EDGE_PAST
}
}
BATCH_MAX_TIME_SPAN_IN_MS = parseInt(
options.BATCH_MAX_TIME_SPAN_IN_MS || ONE_MONTH_IN_MS.toString(),
10
)
if (options.BATCH_RANGE_END) {
BATCH_RANGE_END = objectIdFromInput(options.BATCH_RANGE_END)
} else {
if (BATCH_DESCENDING) {
BATCH_RANGE_END = ID_EDGE_PAST
} else {
BATCH_RANGE_END = ID_EDGE_FUTURE
}
}
}
/**
* @param {Collection | LegacyCollection} collection
* @param {Document} query
* @param {ObjectId} start
* @param {ObjectId} end
* @param {Document} projection
* @param {FindOptions} findOptions
* @return {Promise<Array<Document>>}
*/
async function getNextBatch(
collection,
query,
start,
end,
projection,
findOptions
) {
if (BATCH_DESCENDING) {
query._id = {
$gt: end,
$lte: start,
}
} else {
query._id = {
$gt: start,
$lte: end,
}
}
return await collection
.find(query, findOptions)
.project(projection)
.sort({ _id: BATCH_DESCENDING ? -1 : 1 })
.limit(BATCH_SIZE)
.toArray()
}
/**
* @param {Collection | LegacyCollection} collection
* @param {Array<Document>} nextBatch
* @param {UpdateDocument} update
* @return {Promise<void>}
*/
async function performUpdate(collection, nextBatch, update) {
await collection.updateMany(
{ _id: { $in: nextBatch.map(entry => entry._id) } },
update
)
}
/**
* @param {string} input
* @return {ObjectId}
*/
function objectIdFromInput(input) {
if (input.includes('T')) {
const t = new Date(input).getTime()
if (Number.isNaN(t)) throw new Error(`${input} is not a valid date`)
return objectIdFromMs(t)
} else {
return new ObjectId(input)
}
}
/**
* @param {ObjectId} objectId
* @return {string}
*/
function renderObjectId(objectId) {
return `${objectId} (${objectId.getTimestamp().toISOString()})`
}
/**
* @param {number} ms
* @return {ObjectId}
*/
function objectIdFromMs(ms) {
return ObjectId.createFromTime(ms / 1000)
}
/**
* @param {ObjectId} id
* @return {number}
*/
function getMsFromObjectId(id) {
return id.getTimestamp().getTime()
}
/**
* @param {ObjectId} start
* @return {ObjectId}
*/
function getNextEnd(start) {
let end
if (BATCH_DESCENDING) {
end = objectIdFromMs(getMsFromObjectId(start) - BATCH_MAX_TIME_SPAN_IN_MS)
if (getMsFromObjectId(end) <= getMsFromObjectId(BATCH_RANGE_END)) {
end = BATCH_RANGE_END
}
} else {
end = objectIdFromMs(getMsFromObjectId(start) + BATCH_MAX_TIME_SPAN_IN_MS)
if (getMsFromObjectId(end) >= getMsFromObjectId(BATCH_RANGE_END)) {
end = BATCH_RANGE_END
}
}
return end
}
/**
* @param {Collection | LegacyCollection} collection
* @return {Promise<ObjectId|null>}
*/
async function getIdEdgePast(collection) {
const [first] = await collection
.find({})
.project({ _id: 1 })
.sort({ _id: 1 })
.limit(1)
.toArray()
if (!first) return null
// Go one second further into the past in order to include the first entry via
// first._id > ID_EDGE_PAST
return objectIdFromMs(Math.max(0, getMsFromObjectId(first._id) - 1000))
}
/**
* @param {Collection | LegacyCollection} collection
* @param {Document} query
* @param {UpdateDocument | ((batch: Array<Document>) => Promise<void>)} update
* @param {Document} [projection]
* @param {FindOptions} [findOptions]
* @param {BatchedUpdateOptions} [batchedUpdateOptions]
*/
async function batchedUpdate(
collection,
query,
update,
projection,
findOptions,
batchedUpdateOptions = {}
) {
// only a single batchedUpdate can run at a time due to global variables
if (BATCHED_UPDATE_RUNNING) {
throw new Error('batchedUpdate is already running')
}
try {
BATCHED_UPDATE_RUNNING = true
ID_EDGE_PAST = await getIdEdgePast(collection)
if (!ID_EDGE_PAST) {
console.warn(
`The collection ${collection.collectionName} appears to be empty.`
)
return 0
}
refreshGlobalOptionsForBatchedUpdate(batchedUpdateOptions)
const { trackProgress = async progress => console.warn(progress) } =
batchedUpdateOptions
findOptions = findOptions || {}
findOptions.readPreference = READ_PREFERENCE_SECONDARY
projection = projection || { _id: 1 }
let nextBatch
let updated = 0
let start = BATCH_RANGE_START
while (start !== BATCH_RANGE_END) {
let end = getNextEnd(start)
nextBatch = await getNextBatch(
collection,
query,
start,
end,
projection,
findOptions
)
if (nextBatch.length > 0) {
end = nextBatch[nextBatch.length - 1]._id
updated += nextBatch.length
if (VERBOSE_LOGGING) {
console.log(
`Running update on batch with ids ${JSON.stringify(
nextBatch.map(entry => entry._id)
)}`
)
}
await trackProgress(
`Running update on batch ending ${renderObjectId(end)}`
)
if (typeof update === 'function') {
await update(nextBatch)
} else {
await performUpdate(collection, nextBatch, update)
}
}
await trackProgress(`Completed batch ending ${renderObjectId(end)}`)
start = end
}
return updated
} finally {
BATCHED_UPDATE_RUNNING = false
}
}
/**
* @param {Collection | LegacyCollection} collection
* @param {Document} query
* @param {UpdateDocument | ((batch: Array<Object>) => Promise<void>)} update
* @param {Document} [projection]
* @param {FindOptions} [findOptions]
* @param {BatchedUpdateOptions} [batchedUpdateOptions]
*/
function batchedUpdateWithResultHandling(
collection,
query,
update,
projection,
findOptions,
batchedUpdateOptions
) {
batchedUpdate(
collection,
query,
update,
projection,
findOptions,
batchedUpdateOptions
)
.then(processed => {
console.error({ processed })
process.exit(0)
})
.catch(error => {
console.error({ error })
process.exit(1)
})
}
module.exports = {
READ_PREFERENCE_SECONDARY,
objectIdFromInput,
renderObjectId,
batchedUpdate,
batchedUpdateWithResultHandling,
}

View file

@ -1,10 +0,0 @@
mongo-utils
--dependencies=None
--docker-repos=us-east1-docker.pkg.dev/overleaf-ops/ol-docker
--env-add=
--env-pass-through=
--esmock-loader=False
--is-library=True
--node-version=22.17.0
--public-repo=False
--script-version=4.7.0

View file

@ -1,30 +0,0 @@
{
"name": "@overleaf/mongo-utils",
"version": "0.0.1",
"description": "utilities to help working with mongo",
"main": "index.js",
"scripts": {
"test": "npm run lint && npm run format && npm run types:check && npm run test:unit",
"test:unit": "mocha --exit test/**/*.{js,cjs}",
"lint": "eslint --ext .js --ext .cjs --ext .ts --max-warnings 0 --format unix .",
"lint:fix": "eslint --fix --ext .js --ext .cjs --ext .ts .",
"format": "prettier --list-different $PWD/'**/*.{js,cjs,ts}'",
"format:fix": "prettier --write $PWD/'**/*.{js,cjs,ts}'",
"test:ci": "npm run test:unit",
"types:check": "tsc --noEmit"
},
"author": "Overleaf (https://www.overleaf.com)",
"license": "AGPL-3.0-only",
"dependencies": {
"mongodb": "6.12.0",
"mongodb-legacy": "6.1.3"
},
"devDependencies": {
"chai": "^4.3.6",
"mocha": "^11.1.0",
"sandboxed-module": "^2.0.4",
"sinon": "^9.2.4",
"sinon-chai": "^3.7.0",
"typescript": "^5.0.4"
}
}

View file

@ -1,11 +0,0 @@
const chai = require('chai')
const sinonChai = require('sinon-chai')
const SandboxedModule = require('sandboxed-module')
// Chai configuration
chai.should()
chai.use(sinonChai)
SandboxedModule.configure({
globals: { Buffer, JSON, console, process },
})

View file

@ -1,7 +0,0 @@
{
"extends": "../../tsconfig.backend.json",
"include": [
"**/*.js",
"**/*.cjs"
]
}

View file

@ -0,0 +1 @@
node_modules/

5
libraries/o-error/.gitignore vendored Normal file
View file

@ -0,0 +1,5 @@
.nyc_output
coverage
node_modules/
.npmrc

View file

@ -1 +1 @@
22.17.0
18.20.2

View file

@ -1,10 +1,10 @@
o-error
--dependencies=None
--docker-repos=us-east1-docker.pkg.dev/overleaf-ops/ol-docker
--docker-repos=gcr.io/overleaf-ops
--env-add=
--env-pass-through=
--esmock-loader=False
--is-library=True
--node-version=22.17.0
--node-version=18.20.2
--public-repo=False
--script-version=4.7.0
--script-version=4.5.0

View file

@ -70,7 +70,7 @@ sayHi3(43, (err, result) => {
}
})
const promisify = require('node:util').promisify
const promisify = require('util').promisify
demoDatabase.findUserAsync = promisify(demoDatabase.findUser)
async function sayHi4NoHandling(userId) {

View file

@ -1,34 +1,20 @@
// @ts-check
/**
* Light-weight helpers for handling JavaScript Errors in node.js and the
* browser.
*/
class OError extends Error {
/**
* The error that is the underlying cause of this error
*
* @type {unknown}
*/
cause
/**
* List of errors encountered as the callback chain is unwound
*
* @type {TaggedError[] | undefined}
*/
_oErrorTags
/**
* @param {string} message as for built-in Error
* @param {Object} [info] extra data to attach to the error
* @param {unknown} [cause] the internal error that caused this error
* @param {Error} [cause] the internal error that caused this error
*/
constructor(message, info, cause) {
super(message)
this.name = this.constructor.name
if (info) this.info = info
if (cause) this.cause = cause
/** @private @type {Array<TaggedError> | undefined} */
this._oErrorTags // eslint-disable-line
}
/**
@ -45,7 +31,7 @@ class OError extends Error {
/**
* Wrap the given error, which caused this error.
*
* @param {unknown} cause the internal error that caused this error
* @param {Error} cause the internal error that caused this error
* @return {this}
*/
withCause(cause) {
@ -79,16 +65,13 @@ class OError extends Error {
* }
* }
*
* @template {unknown} E
* @param {E} error the error to tag
* @param {Error} error the error to tag
* @param {string} [message] message with which to tag `error`
* @param {Object} [info] extra data with wich to tag `error`
* @return {E} the modified `error` argument
* @return {Error} the modified `error` argument
*/
static tag(error, message, info) {
const oError = /** @type {{ _oErrorTags: TaggedError[] | undefined }} */ (
error
)
const oError = /** @type{OError} */ (error)
if (!oError._oErrorTags) oError._oErrorTags = []
@ -119,7 +102,7 @@ class OError extends Error {
*
* If an info property is repeated, the last one wins.
*
* @param {unknown} error any error (may or may not be an `OError`)
* @param {Error | null | undefined} error any error (may or may not be an `OError`)
* @return {Object}
*/
static getFullInfo(error) {
@ -146,7 +129,7 @@ class OError extends Error {
* Return the `stack` property from `error`, including the `stack`s for any
* tagged errors added with `OError.tag` and for any `cause`s.
*
* @param {unknown} error any error (may or may not be an `OError`)
* @param {Error | null | undefined} error any error (may or may not be an `OError`)
* @return {string}
*/
static getFullStack(error) {
@ -160,7 +143,7 @@ class OError extends Error {
stack += `\n${oError._oErrorTags.map(tag => tag.stack).join('\n')}`
}
const causeStack = OError.getFullStack(oError.cause)
const causeStack = oError.cause && OError.getFullStack(oError.cause)
if (causeStack) {
stack += '\ncaused by:\n' + indent(causeStack)
}

View file

@ -34,7 +34,7 @@
"@types/chai": "^4.3.0",
"@types/node": "^18.17.4",
"chai": "^4.3.6",
"mocha": "^11.1.0",
"mocha": "^10.2.0",
"typescript": "^5.0.4"
}
}

View file

@ -1,5 +1,5 @@
const { expect } = require('chai')
const { promisify } = require('node:util')
const { promisify } = require('util')
const OError = require('..')
@ -268,11 +268,6 @@ describe('utils', function () {
expect(OError.getFullInfo(null)).to.deep.equal({})
})
it('works when given a string', function () {
const err = 'not an error instance'
expect(OError.getFullInfo(err)).to.deep.equal({})
})
it('works on a normal error', function () {
const err = new Error('foo')
expect(OError.getFullInfo(err)).to.deep.equal({})

View file

@ -35,14 +35,6 @@ describe('OError', function () {
expect(err2.cause.message).to.equal('cause 2')
})
it('accepts non-Error causes', function () {
const err1 = new OError('foo', {}, 'not-an-error')
expect(err1.cause).to.equal('not-an-error')
const err2 = new OError('foo').withCause('not-an-error')
expect(err2.cause).to.equal('not-an-error')
})
it('handles a custom error type with a cause', function () {
function doSomethingBadInternally() {
throw new Error('internal error')

View file

@ -23,7 +23,7 @@ exports.expectError = function OErrorExpectError(e, expected) {
).to.be.true
expect(
require('node:util').types.isNativeError(e),
require('util').types.isNativeError(e),
'error should be recognised by util.types.isNativeError'
).to.be.true

View file

@ -0,0 +1 @@
node_modules/

4
libraries/object-persistor/.gitignore vendored Normal file
View file

@ -0,0 +1,4 @@
/node_modules
*.swp
.npmrc

View file

@ -1 +1 @@
22.17.0
18.20.2

View file

@ -1,10 +1,10 @@
object-persistor
--dependencies=None
--docker-repos=us-east1-docker.pkg.dev/overleaf-ops/ol-docker
--docker-repos=gcr.io/overleaf-ops
--env-add=
--env-pass-through=
--esmock-loader=False
--is-library=True
--node-version=22.17.0
--node-version=18.20.2
--public-repo=False
--script-version=4.7.0
--script-version=4.5.0

View file

@ -24,8 +24,7 @@
"@overleaf/logger": "*",
"@overleaf/metrics": "*",
"@overleaf/o-error": "*",
"@overleaf/stream-utils": "*",
"aws-sdk": "^2.1691.0",
"aws-sdk": "^2.718.0",
"fast-crc32c": "overleaf/node-fast-crc32c#aae6b2a4c7a7a159395df9cc6c38dfde702d6f51",
"glob": "^7.1.6",
"range-parser": "^1.2.1",
@ -34,9 +33,9 @@
"devDependencies": {
"chai": "^4.3.6",
"chai-as-promised": "^7.1.1",
"mocha": "^11.1.0",
"mocha": "^10.2.0",
"mock-fs": "^5.2.0",
"mongodb": "6.12.0",
"mongodb": "^6.1.0",
"sandboxed-module": "^2.0.4",
"sinon": "^9.2.4",
"sinon-chai": "^3.7.0",

View file

@ -1,12 +1,6 @@
const { NotImplementedError } = require('./Errors')
module.exports = class AbstractPersistor {
/**
* @param location
* @param target
* @param {string} source
* @return {Promise<void>}
*/
async sendFile(location, target, source) {
throw new NotImplementedError('method not implemented in persistor', {
method: 'sendFile',
@ -16,13 +10,6 @@ module.exports = class AbstractPersistor {
})
}
/**
* @param location
* @param target
* @param {NodeJS.ReadableStream} sourceStream
* @param {Object} opts
* @return {Promise<void>}
*/
async sendStream(location, target, sourceStream, opts = {}) {
throw new NotImplementedError('method not implemented in persistor', {
method: 'sendStream',
@ -35,12 +22,12 @@ module.exports = class AbstractPersistor {
/**
* @param location
* @param name
* @param {Object} [opts]
* @param {Number} [opts.start]
* @param {Number} [opts.end]
* @return {Promise<NodeJS.ReadableStream>}
* @param {Object} opts
* @param {Number} opts.start
* @param {Number} opts.end
* @return {Promise<Readable>}
*/
async getObjectStream(location, name, opts = {}) {
async getObjectStream(location, name, opts) {
throw new NotImplementedError('method not implemented in persistor', {
method: 'getObjectStream',
location,
@ -49,11 +36,6 @@ module.exports = class AbstractPersistor {
})
}
/**
* @param {string} location
* @param {string} name
* @return {Promise<string>}
*/
async getRedirectUrl(location, name) {
throw new NotImplementedError('method not implemented in persistor', {
method: 'getRedirectUrl',
@ -62,13 +44,7 @@ module.exports = class AbstractPersistor {
})
}
/**
* @param {string} location
* @param {string} name
* @param {Object} opts
* @return {Promise<number>}
*/
async getObjectSize(location, name, opts) {
async getObjectSize(location, name) {
throw new NotImplementedError('method not implemented in persistor', {
method: 'getObjectSize',
location,
@ -76,13 +52,7 @@ module.exports = class AbstractPersistor {
})
}
/**
* @param {string} location
* @param {string} name
* @param {Object} opts
* @return {Promise<string>}
*/
async getObjectMd5Hash(location, name, opts) {
async getObjectMd5Hash(location, name) {
throw new NotImplementedError('method not implemented in persistor', {
method: 'getObjectMd5Hash',
location,
@ -90,14 +60,7 @@ module.exports = class AbstractPersistor {
})
}
/**
* @param {string} location
* @param {string} fromName
* @param {string} toName
* @param {Object} opts
* @return {Promise<void>}
*/
async copyObject(location, fromName, toName, opts) {
async copyObject(location, fromName, toName) {
throw new NotImplementedError('method not implemented in persistor', {
method: 'copyObject',
location,
@ -106,11 +69,6 @@ module.exports = class AbstractPersistor {
})
}
/**
* @param {string} location
* @param {string} name
* @return {Promise<void>}
*/
async deleteObject(location, name) {
throw new NotImplementedError('method not implemented in persistor', {
method: 'deleteObject',
@ -119,13 +77,7 @@ module.exports = class AbstractPersistor {
})
}
/**
* @param {string} location
* @param {string} name
* @param {string} [continuationToken]
* @return {Promise<void>}
*/
async deleteDirectory(location, name, continuationToken) {
async deleteDirectory(location, name) {
throw new NotImplementedError('method not implemented in persistor', {
method: 'deleteDirectory',
location,
@ -133,13 +85,7 @@ module.exports = class AbstractPersistor {
})
}
/**
* @param {string} location
* @param {string} name
* @param {Object} opts
* @return {Promise<boolean>}
*/
async checkIfObjectExists(location, name, opts) {
async checkIfObjectExists(location, name) {
throw new NotImplementedError('method not implemented in persistor', {
method: 'checkIfObjectExists',
location,
@ -147,13 +93,7 @@ module.exports = class AbstractPersistor {
})
}
/**
* @param {string} location
* @param {string} name
* @param {string} [continuationToken]
* @return {Promise<number>}
*/
async directorySize(location, name, continuationToken) {
async directorySize(location, name) {
throw new NotImplementedError('method not implemented in persistor', {
method: 'directorySize',
location,

View file

@ -5,8 +5,6 @@ class WriteError extends OError {}
class ReadError extends OError {}
class SettingsError extends OError {}
class NotImplementedError extends OError {}
class AlreadyWrittenError extends OError {}
class NoKEKMatchedError extends OError {}
module.exports = {
NotFoundError,
@ -14,6 +12,4 @@ module.exports = {
ReadError,
SettingsError,
NotImplementedError,
AlreadyWrittenError,
NoKEKMatchedError,
}

View file

@ -1,26 +1,20 @@
const crypto = require('node:crypto')
const fs = require('node:fs')
const fsPromises = require('node:fs/promises')
const crypto = require('crypto')
const fs = require('fs')
const fsPromises = require('fs/promises')
const globCallbacks = require('glob')
const Path = require('node:path')
const { PassThrough } = require('node:stream')
const { pipeline } = require('node:stream/promises')
const { promisify } = require('node:util')
const Path = require('path')
const { PassThrough } = require('stream')
const { pipeline } = require('stream/promises')
const { promisify } = require('util')
const AbstractPersistor = require('./AbstractPersistor')
const { ReadError, WriteError, NotImplementedError } = require('./Errors')
const { ReadError, WriteError } = require('./Errors')
const PersistorHelper = require('./PersistorHelper')
const glob = promisify(globCallbacks)
module.exports = class FSPersistor extends AbstractPersistor {
constructor(settings = {}) {
if (settings.storageClass) {
throw new NotImplementedError(
'FS backend does not support storage classes'
)
}
super()
this.useSubdirectories = Boolean(settings.useSubdirectories)
}
@ -42,14 +36,6 @@ module.exports = class FSPersistor extends AbstractPersistor {
}
async sendStream(location, target, sourceStream, opts = {}) {
if (opts.ifNoneMatch === '*') {
// The standard library only has fs.rename(), which does not support exclusive flags.
// Refuse to act on this write operation.
throw new NotImplementedError(
'Overwrite protection required by caller, but it is not available is FS backend. Configure GCS or S3 backend instead, get in touch with support for further information.'
)
}
const targetPath = this._getFsPath(location, target)
try {
@ -69,7 +55,7 @@ module.exports = class FSPersistor extends AbstractPersistor {
throw PersistorHelper.wrapError(
err,
'failed to write stream',
{ location, target, ifNoneMatch: opts.ifNoneMatch },
{ location, target },
WriteError
)
}
@ -77,11 +63,6 @@ module.exports = class FSPersistor extends AbstractPersistor {
// opts may be {start: Number, end: Number}
async getObjectStream(location, name, opts = {}) {
if (opts.autoGunzip) {
throw new NotImplementedError(
'opts.autoGunzip is not supported by FS backend. Configure GCS or S3 backend instead, get in touch with support for further information.'
)
}
const observer = new PersistorHelper.ObserverStream({
metric: 'fs.ingress', // ingress to us from disk
bucket: location,
@ -305,10 +286,8 @@ module.exports = class FSPersistor extends AbstractPersistor {
async _listDirectory(path) {
if (this.useSubdirectories) {
// eslint-disable-next-line @typescript-eslint/return-await
return await glob(Path.join(path, '**'))
} else {
// eslint-disable-next-line @typescript-eslint/return-await
return await glob(`${path}_*`)
}
}

View file

@ -1,28 +1,17 @@
const fs = require('node:fs')
const { pipeline } = require('node:stream/promises')
const { PassThrough } = require('node:stream')
const fs = require('fs')
const { pipeline } = require('stream/promises')
const { PassThrough } = require('stream')
const { Storage, IdempotencyStrategy } = require('@google-cloud/storage')
const {
WriteError,
ReadError,
NotFoundError,
NotImplementedError,
} = require('./Errors')
const { WriteError, ReadError, NotFoundError } = require('./Errors')
const asyncPool = require('tiny-async-pool')
const AbstractPersistor = require('./AbstractPersistor')
const PersistorHelper = require('./PersistorHelper')
const Logger = require('@overleaf/logger')
const zlib = require('node:zlib')
module.exports = class GcsPersistor extends AbstractPersistor {
constructor(settings) {
if (settings.storageClass) {
throw new NotImplementedError(
'Use default bucket class for GCS instead of settings.storageClass'
)
}
super()
this.settings = settings
// endpoint settings will be null by default except for tests
@ -89,14 +78,10 @@ module.exports = class GcsPersistor extends AbstractPersistor {
writeOptions.metadata = writeOptions.metadata || {}
writeOptions.metadata.contentEncoding = opts.contentEncoding
}
const fileOptions = {}
if (opts.ifNoneMatch === '*') {
fileOptions.generation = 0
}
const uploadStream = this.storage
.bucket(bucketName)
.file(key, fileOptions)
.file(key)
.createWriteStream(writeOptions)
await pipeline(readStream, observer, uploadStream)
@ -112,7 +97,7 @@ module.exports = class GcsPersistor extends AbstractPersistor {
throw PersistorHelper.wrapError(
err,
'upload to GCS failed',
{ bucketName, key, ifNoneMatch: opts.ifNoneMatch },
{ bucketName, key },
WriteError
)
}
@ -128,14 +113,12 @@ module.exports = class GcsPersistor extends AbstractPersistor {
.file(key)
.createReadStream({ decompress: false, ...opts })
let contentEncoding
try {
await new Promise((resolve, reject) => {
stream.on('response', res => {
switch (res.statusCode) {
case 200: // full response
case 206: // partial response
contentEncoding = res.headers['content-encoding']
return resolve()
case 404:
return reject(new NotFoundError())
@ -156,11 +139,7 @@ module.exports = class GcsPersistor extends AbstractPersistor {
}
// Return a PassThrough stream with a minimal interface. It will buffer until the caller starts reading. It will emit errors from the source stream (Stream.pipeline passes errors along).
const pass = new PassThrough()
const transformer = []
if (contentEncoding === 'gzip' && opts.autoGunzip) {
transformer.push(zlib.createGunzip())
}
pipeline(stream, observer, ...transformer, pass).catch(() => {})
pipeline(stream, observer, pass).catch(() => {})
return pass
}
@ -197,7 +176,7 @@ module.exports = class GcsPersistor extends AbstractPersistor {
.bucket(bucketName)
.file(key)
.getMetadata()
return parseInt(metadata.size, 10)
return metadata.size
} catch (err) {
throw PersistorHelper.wrapError(
err,
@ -306,10 +285,7 @@ module.exports = class GcsPersistor extends AbstractPersistor {
)
}
return files.reduce(
(acc, file) => parseInt(file.metadata.size, 10) + acc,
0
)
return files.reduce((acc, file) => Number(file.metadata.size) + acc, 0)
}
async checkIfObjectExists(bucketName, key) {

View file

@ -1,8 +1,8 @@
const AbstractPersistor = require('./AbstractPersistor')
const Logger = require('@overleaf/logger')
const Metrics = require('@overleaf/metrics')
const Stream = require('node:stream')
const { pipeline } = require('node:stream/promises')
const Stream = require('stream')
const { pipeline } = require('stream/promises')
const { NotFoundError, WriteError } = require('./Errors')
// Persistor that wraps two other persistors. Talks to the 'primary' by default,

View file

@ -1,483 +0,0 @@
// @ts-check
const Crypto = require('node:crypto')
const Stream = require('node:stream')
const fs = require('node:fs')
const { promisify } = require('node:util')
const { WritableBuffer } = require('@overleaf/stream-utils')
const { S3Persistor, SSECOptions } = require('./S3Persistor.js')
const {
AlreadyWrittenError,
NoKEKMatchedError,
NotFoundError,
NotImplementedError,
ReadError,
} = require('./Errors')
const logger = require('@overleaf/logger')
const Path = require('node:path')
const generateKey = promisify(Crypto.generateKey)
const hkdf = promisify(Crypto.hkdf)
const AES256_KEY_LENGTH = 32
/**
* @typedef {import('aws-sdk').AWSError} AWSError
*/
/**
* @typedef {Object} Settings
* @property {boolean} automaticallyRotateDEKEncryption
* @property {string} dataEncryptionKeyBucketName
* @property {boolean} ignoreErrorsFromDEKReEncryption
* @property {(bucketName: string, path: string) => string} pathToProjectFolder
* @property {() => Promise<Array<RootKeyEncryptionKey>>} getRootKeyEncryptionKeys
*/
/**
* @typedef {import('./types').ListDirectoryResult} ListDirectoryResult
*/
/**
* Helper function to make TS happy when accessing error properties
* AWSError is not an actual class, so we cannot use instanceof.
* @param {any} err
* @return {err is AWSError}
*/
function isAWSError(err) {
return !!err
}
/**
* @param {any} err
* @return {boolean}
*/
function isForbiddenError(err) {
if (!err || !(err instanceof ReadError || err instanceof NotFoundError)) {
return false
}
const cause = err.cause
if (!isAWSError(cause)) return false
return cause.statusCode === 403
}
class RootKeyEncryptionKey {
/** @type {Buffer} */
#keyEncryptionKey
/** @type {Buffer} */
#salt
/**
* @param {Buffer} keyEncryptionKey
* @param {Buffer} salt
*/
constructor(keyEncryptionKey, salt) {
if (keyEncryptionKey.byteLength !== AES256_KEY_LENGTH) {
throw new Error(`kek is not ${AES256_KEY_LENGTH} bytes long`)
}
this.#keyEncryptionKey = keyEncryptionKey
this.#salt = salt
}
/**
* @param {string} prefix
* @return {Promise<SSECOptions>}
*/
async forProject(prefix) {
return new SSECOptions(
Buffer.from(
await hkdf(
'sha256',
this.#keyEncryptionKey,
this.#salt,
prefix,
AES256_KEY_LENGTH
)
)
)
}
}
class PerProjectEncryptedS3Persistor extends S3Persistor {
/** @type {Settings} */
#settings
/** @type {Promise<Array<RootKeyEncryptionKey>>} */
#availableKeyEncryptionKeysPromise
/**
* @param {Settings} settings
*/
constructor(settings) {
if (!settings.dataEncryptionKeyBucketName) {
throw new Error('settings.dataEncryptionKeyBucketName is missing')
}
super(settings)
this.#settings = settings
this.#availableKeyEncryptionKeysPromise = settings
.getRootKeyEncryptionKeys()
.then(rootKEKs => {
if (rootKEKs.length === 0) throw new Error('no root kek provided')
return rootKEKs
})
}
async ensureKeyEncryptionKeysLoaded() {
await this.#availableKeyEncryptionKeysPromise
}
/**
* @param {string} bucketName
* @param {string} path
* @return {{dekPath: string, projectFolder: string}}
*/
#buildProjectPaths(bucketName, path) {
const projectFolder = this.#settings.pathToProjectFolder(bucketName, path)
const dekPath = Path.join(projectFolder, 'dek')
return { projectFolder, dekPath }
}
/**
* @param {string} projectFolder
* @return {Promise<SSECOptions>}
*/
async #getCurrentKeyEncryptionKey(projectFolder) {
const [currentRootKEK] = await this.#availableKeyEncryptionKeysPromise
return await currentRootKEK.forProject(projectFolder)
}
/**
* @param {string} bucketName
* @param {string} path
*/
async getDataEncryptionKeySize(bucketName, path) {
const { projectFolder, dekPath } = this.#buildProjectPaths(bucketName, path)
for (const rootKEK of await this.#availableKeyEncryptionKeysPromise) {
const ssecOptions = await rootKEK.forProject(projectFolder)
try {
return await super.getObjectSize(
this.#settings.dataEncryptionKeyBucketName,
dekPath,
{ ssecOptions }
)
} catch (err) {
if (isForbiddenError(err)) continue
throw err
}
}
throw new NoKEKMatchedError('no kek matched')
}
/**
* @param {string} bucketName
* @param {string} path
* @return {Promise<CachedPerProjectEncryptedS3Persistor>}
*/
async forProject(bucketName, path) {
return new CachedPerProjectEncryptedS3Persistor(
this,
await this.#getDataEncryptionKeyOptions(bucketName, path)
)
}
/**
* @param {string} bucketName
* @param {string} path
* @return {Promise<CachedPerProjectEncryptedS3Persistor>}
*/
async forProjectRO(bucketName, path) {
return new CachedPerProjectEncryptedS3Persistor(
this,
await this.#getExistingDataEncryptionKeyOptions(bucketName, path)
)
}
/**
* @param {string} bucketName
* @param {string} path
* @return {Promise<CachedPerProjectEncryptedS3Persistor>}
*/
async generateDataEncryptionKey(bucketName, path) {
return new CachedPerProjectEncryptedS3Persistor(
this,
await this.#generateDataEncryptionKeyOptions(bucketName, path)
)
}
/**
* @param {string} bucketName
* @param {string} path
* @return {Promise<SSECOptions>}
*/
async #generateDataEncryptionKeyOptions(bucketName, path) {
const dataEncryptionKey = (
await generateKey('aes', { length: 256 })
).export()
const { projectFolder, dekPath } = this.#buildProjectPaths(bucketName, path)
await super.sendStream(
this.#settings.dataEncryptionKeyBucketName,
dekPath,
Stream.Readable.from([dataEncryptionKey]),
{
// Do not overwrite any objects if already created
ifNoneMatch: '*',
ssecOptions: await this.#getCurrentKeyEncryptionKey(projectFolder),
contentLength: 32,
}
)
return new SSECOptions(dataEncryptionKey)
}
/**
* @param {string} bucketName
* @param {string} path
* @return {Promise<SSECOptions>}
*/
async #getExistingDataEncryptionKeyOptions(bucketName, path) {
const { projectFolder, dekPath } = this.#buildProjectPaths(bucketName, path)
let res
let kekIndex = 0
for (const rootKEK of await this.#availableKeyEncryptionKeysPromise) {
const ssecOptions = await rootKEK.forProject(projectFolder)
try {
res = await super.getObjectStream(
this.#settings.dataEncryptionKeyBucketName,
dekPath,
{ ssecOptions }
)
break
} catch (err) {
if (isForbiddenError(err)) {
kekIndex++
continue
}
throw err
}
}
if (!res) throw new NoKEKMatchedError('no kek matched')
const buf = new WritableBuffer()
await Stream.promises.pipeline(res, buf)
if (kekIndex !== 0 && this.#settings.automaticallyRotateDEKEncryption) {
const ssecOptions = await this.#getCurrentKeyEncryptionKey(projectFolder)
try {
await super.sendStream(
this.#settings.dataEncryptionKeyBucketName,
dekPath,
Stream.Readable.from([buf.getContents()]),
{ ssecOptions }
)
} catch (err) {
if (this.#settings.ignoreErrorsFromDEKReEncryption) {
logger.warn({ err, dekPath }, 'failed to persist re-encrypted DEK')
} else {
throw err
}
}
}
return new SSECOptions(buf.getContents())
}
/**
* @param {string} bucketName
* @param {string} path
* @return {Promise<SSECOptions>}
*/
async #getDataEncryptionKeyOptions(bucketName, path) {
try {
return await this.#getExistingDataEncryptionKeyOptions(bucketName, path)
} catch (err) {
if (err instanceof NotFoundError) {
try {
return await this.#generateDataEncryptionKeyOptions(bucketName, path)
} catch (err2) {
if (err2 instanceof AlreadyWrittenError) {
// Concurrent initial write
return await this.#getExistingDataEncryptionKeyOptions(
bucketName,
path
)
}
throw err2
}
}
throw err
}
}
async sendStream(bucketName, path, sourceStream, opts = {}) {
const ssecOptions =
opts.ssecOptions ||
(await this.#getDataEncryptionKeyOptions(bucketName, path))
return await super.sendStream(bucketName, path, sourceStream, {
...opts,
ssecOptions,
})
}
async getObjectStream(bucketName, path, opts = {}) {
const ssecOptions =
opts.ssecOptions ||
(await this.#getExistingDataEncryptionKeyOptions(bucketName, path))
return await super.getObjectStream(bucketName, path, {
...opts,
ssecOptions,
})
}
async getObjectSize(bucketName, path, opts = {}) {
const ssecOptions =
opts.ssecOptions ||
(await this.#getExistingDataEncryptionKeyOptions(bucketName, path))
return await super.getObjectSize(bucketName, path, { ...opts, ssecOptions })
}
async getObjectStorageClass(bucketName, path, opts = {}) {
const ssecOptions =
opts.ssecOptions ||
(await this.#getExistingDataEncryptionKeyOptions(bucketName, path))
return await super.getObjectStorageClass(bucketName, path, {
...opts,
ssecOptions,
})
}
async directorySize(bucketName, path, continuationToken) {
// Note: Listing a bucket does not require SSE-C credentials.
return await super.directorySize(bucketName, path, continuationToken)
}
async deleteDirectory(bucketName, path, continuationToken) {
// Let [Settings.pathToProjectFolder] validate the project path before deleting things.
const { projectFolder, dekPath } = this.#buildProjectPaths(bucketName, path)
// Note: Listing/Deleting a prefix does not require SSE-C credentials.
await super.deleteDirectory(bucketName, path, continuationToken)
if (projectFolder === path) {
await super.deleteObject(
this.#settings.dataEncryptionKeyBucketName,
dekPath
)
}
}
async getObjectMd5Hash(bucketName, path, opts = {}) {
// The ETag in object metadata is not the MD5 content hash, skip the HEAD request.
opts = { ...opts, etagIsNotMD5: true }
return await super.getObjectMd5Hash(bucketName, path, opts)
}
async copyObject(bucketName, sourcePath, destinationPath, opts = {}) {
const ssecOptions =
opts.ssecOptions ||
(await this.#getDataEncryptionKeyOptions(bucketName, destinationPath))
const ssecSrcOptions =
opts.ssecSrcOptions ||
(await this.#getExistingDataEncryptionKeyOptions(bucketName, sourcePath))
return await super.copyObject(bucketName, sourcePath, destinationPath, {
...opts,
ssecOptions,
ssecSrcOptions,
})
}
/**
* @param {string} bucketName
* @param {string} path
* @return {Promise<string>}
*/
async getRedirectUrl(bucketName, path) {
throw new NotImplementedError('signed links are not supported with SSE-C')
}
}
/**
* Helper class for batch updates to avoid repeated fetching of the project path.
*
* A general "cache" for project keys is another alternative. For now, use a helper class.
*/
class CachedPerProjectEncryptedS3Persistor {
/** @type SSECOptions */
#projectKeyOptions
/** @type PerProjectEncryptedS3Persistor */
#parent
/**
* @param {PerProjectEncryptedS3Persistor} parent
* @param {SSECOptions} projectKeyOptions
*/
constructor(parent, projectKeyOptions) {
this.#parent = parent
this.#projectKeyOptions = projectKeyOptions
}
/**
* @param {string} bucketName
* @param {string} path
* @param {string} fsPath
*/
async sendFile(bucketName, path, fsPath) {
return await this.sendStream(bucketName, path, fs.createReadStream(fsPath))
}
/**
*
* @param {string} bucketName
* @param {string} path
* @return {Promise<number>}
*/
async getObjectSize(bucketName, path) {
return await this.#parent.getObjectSize(bucketName, path)
}
/**
*
* @param {string} bucketName
* @param {string} path
* @return {Promise<ListDirectoryResult>}
*/
async listDirectory(bucketName, path) {
return await this.#parent.listDirectory(bucketName, path)
}
/**
* @param {string} bucketName
* @param {string} path
* @param {NodeJS.ReadableStream} sourceStream
* @param {Object} opts
* @param {string} [opts.contentType]
* @param {string} [opts.contentEncoding]
* @param {number} [opts.contentLength]
* @param {'*'} [opts.ifNoneMatch]
* @param {SSECOptions} [opts.ssecOptions]
* @param {string} [opts.sourceMd5]
* @return {Promise<void>}
*/
async sendStream(bucketName, path, sourceStream, opts = {}) {
return await this.#parent.sendStream(bucketName, path, sourceStream, {
...opts,
ssecOptions: this.#projectKeyOptions,
})
}
/**
* @param {string} bucketName
* @param {string} path
* @param {Object} opts
* @param {number} [opts.start]
* @param {number} [opts.end]
* @param {boolean} [opts.autoGunzip]
* @param {SSECOptions} [opts.ssecOptions]
* @return {Promise<NodeJS.ReadableStream>}
*/
async getObjectStream(bucketName, path, opts = {}) {
return await this.#parent.getObjectStream(bucketName, path, {
...opts,
ssecOptions: this.#projectKeyOptions,
})
}
}
module.exports = {
PerProjectEncryptedS3Persistor,
CachedPerProjectEncryptedS3Persistor,
RootKeyEncryptionKey,
}

View file

@ -1,20 +1,15 @@
const Logger = require('@overleaf/logger')
const { SettingsError } = require('./Errors')
const GcsPersistor = require('./GcsPersistor')
const { S3Persistor } = require('./S3Persistor')
const S3Persistor = require('./S3Persistor')
const FSPersistor = require('./FSPersistor')
const MigrationPersistor = require('./MigrationPersistor')
const {
PerProjectEncryptedS3Persistor,
} = require('./PerProjectEncryptedS3Persistor')
function getPersistor(backend, settings) {
switch (backend) {
case 'aws-sdk':
case 's3':
return new S3Persistor(settings.s3)
case 's3SSEC':
return new PerProjectEncryptedS3Persistor(settings.s3SSEC)
case 'fs':
return new FSPersistor({
useSubdirectories: settings.useSubdirectories,

View file

@ -1,9 +1,9 @@
const Crypto = require('node:crypto')
const Stream = require('node:stream')
const { pipeline } = require('node:stream/promises')
const Crypto = require('crypto')
const Stream = require('stream')
const { pipeline } = require('stream/promises')
const Logger = require('@overleaf/logger')
const Metrics = require('@overleaf/metrics')
const { WriteError, NotFoundError, AlreadyWrittenError } = require('./Errors')
const { WriteError, NotFoundError } = require('./Errors')
const _128KiB = 128 * 1024
const TIMING_BUCKETS = [
@ -26,14 +26,12 @@ const SIZE_BUCKETS = [
*/
class ObserverStream extends Stream.Transform {
/**
* @param {Object} opts
* @param {string} opts.metric prefix for metrics
* @param {string} opts.bucket name of source/target bucket
* @param {string} [opts.hash] optional hash algorithm, e.g. 'md5'
* @param {string} metric prefix for metrics
* @param {string} bucket name of source/target bucket
* @param {string} hash optional hash algorithm, e.g. 'md5'
*/
constructor(opts) {
constructor({ metric, bucket, hash = '' }) {
super({ autoDestroy: true })
const { metric, bucket, hash = '' } = opts
this.bytes = 0
this.start = performance.now()
@ -140,10 +138,6 @@ async function verifyMd5(persistor, bucket, key, sourceMd5, destMd5 = null) {
}
function wrapError(error, message, params, ErrorType) {
params = {
...params,
cause: error,
}
if (
error instanceof NotFoundError ||
['NoSuchKey', 'NotFound', 404, 'AccessDenied', 'ENOENT'].includes(
@ -152,13 +146,6 @@ function wrapError(error, message, params, ErrorType) {
(error.response && error.response.statusCode === 404)
) {
return new NotFoundError('no such file', params, error)
} else if (
params.ifNoneMatch === '*' &&
(error.code === 'PreconditionFailed' ||
error.response?.statusCode === 412 ||
error instanceof AlreadyWrittenError)
) {
return new AlreadyWrittenError(message, params, error)
} else {
return new ErrorType(message, params, error)
}

View file

@ -1,6 +1,5 @@
// @ts-check
const http = require('node:http')
const https = require('node:https')
const http = require('http')
const https = require('https')
if (http.globalAgent.maxSockets < 300) {
http.globalAgent.maxSockets = 300
}
@ -8,104 +7,27 @@ if (https.globalAgent.maxSockets < 300) {
https.globalAgent.maxSockets = 300
}
const Crypto = require('node:crypto')
const Metrics = require('@overleaf/metrics')
const AbstractPersistor = require('./AbstractPersistor')
const PersistorHelper = require('./PersistorHelper')
const { pipeline, PassThrough } = require('node:stream')
const fs = require('node:fs')
const { pipeline, PassThrough } = require('stream')
const fs = require('fs')
const S3 = require('aws-sdk/clients/s3')
const { URL } = require('node:url')
const { URL } = require('url')
const { WriteError, ReadError, NotFoundError } = require('./Errors')
const zlib = require('node:zlib')
/**
* @typedef {import('aws-sdk/clients/s3').ListObjectsV2Output} ListObjectsV2Output
*/
/**
* @typedef {import('aws-sdk/clients/s3').Object} S3Object
*/
/**
* @typedef {import('./types').ListDirectoryResult} ListDirectoryResult
*/
/**
* Wrapper with private fields to avoid revealing them on console, JSON.stringify or similar.
*/
class SSECOptions {
#keyAsBuffer
#keyMD5
/**
* @param {Buffer} keyAsBuffer
*/
constructor(keyAsBuffer) {
this.#keyAsBuffer = keyAsBuffer
this.#keyMD5 = Crypto.createHash('md5').update(keyAsBuffer).digest('base64')
}
getPutOptions() {
return {
SSECustomerKey: this.#keyAsBuffer,
SSECustomerKeyMD5: this.#keyMD5,
SSECustomerAlgorithm: 'AES256',
}
}
getGetOptions() {
return {
SSECustomerKey: this.#keyAsBuffer,
SSECustomerKeyMD5: this.#keyMD5,
SSECustomerAlgorithm: 'AES256',
}
}
getCopyOptions() {
return {
CopySourceSSECustomerKey: this.#keyAsBuffer,
CopySourceSSECustomerKeyMD5: this.#keyMD5,
CopySourceSSECustomerAlgorithm: 'AES256',
}
}
}
class S3Persistor extends AbstractPersistor {
/** @type {Map<string, S3>} */
#clients = new Map()
module.exports = class S3Persistor extends AbstractPersistor {
constructor(settings = {}) {
super()
settings.storageClass = settings.storageClass || {}
this.settings = settings
}
/**
* @param {string} bucketName
* @param {string} key
* @param {string} fsPath
* @return {Promise<void>}
*/
async sendFile(bucketName, key, fsPath) {
await this.sendStream(bucketName, key, fs.createReadStream(fsPath))
return await this.sendStream(bucketName, key, fs.createReadStream(fsPath))
}
/**
* @param {string} bucketName
* @param {string} key
* @param {NodeJS.ReadableStream} readStream
* @param {Object} opts
* @param {string} [opts.contentType]
* @param {string} [opts.contentEncoding]
* @param {number} [opts.contentLength]
* @param {'*'} [opts.ifNoneMatch]
* @param {SSECOptions} [opts.ssecOptions]
* @param {string} [opts.sourceMd5]
* @return {Promise<void>}
*/
async sendStream(bucketName, key, readStream, opts = {}) {
try {
const observeOptions = {
@ -117,71 +39,42 @@ class S3Persistor extends AbstractPersistor {
// observer will catch errors, clean up and log a warning
pipeline(readStream, observer, () => {})
/** @type {S3.PutObjectRequest} */
// if we have an md5 hash, pass this to S3 to verify the upload
const uploadOptions = {
Bucket: bucketName,
Key: key,
Body: observer,
}
if (this.settings.storageClass[bucketName]) {
uploadOptions.StorageClass = this.settings.storageClass[bucketName]
}
if (opts.contentType) {
uploadOptions.ContentType = opts.contentType
}
if (opts.contentEncoding) {
uploadOptions.ContentEncoding = opts.contentEncoding
}
if (opts.contentLength) {
uploadOptions.ContentLength = opts.contentLength
}
if (opts.ifNoneMatch === '*') {
uploadOptions.IfNoneMatch = '*'
}
if (opts.ssecOptions) {
Object.assign(uploadOptions, opts.ssecOptions.getPutOptions())
}
// if we have an md5 hash, pass this to S3 to verify the upload - otherwise
// we rely on the S3 client's checksum calculation to validate the upload
let computeChecksums = false
const clientOptions = {}
if (opts.sourceMd5) {
uploadOptions.ContentMD5 = PersistorHelper.hexToBase64(opts.sourceMd5)
} else {
computeChecksums = true
clientOptions.computeChecksums = true
}
if (this.settings.disableMultiPartUpload) {
await this._getClientForBucket(bucketName, computeChecksums)
.putObject(uploadOptions)
.promise()
} else {
await this._getClientForBucket(bucketName, computeChecksums)
.upload(uploadOptions, { partSize: this.settings.partSize })
.promise()
}
await this._getClientForBucket(bucketName, clientOptions)
.upload(uploadOptions, { partSize: this.settings.partSize })
.promise()
} catch (err) {
throw PersistorHelper.wrapError(
err,
'upload to S3 failed',
{ bucketName, key, ifNoneMatch: opts.ifNoneMatch },
{ bucketName, key },
WriteError
)
}
}
/**
* @param {string} bucketName
* @param {string} key
* @param {Object} [opts]
* @param {number} [opts.start]
* @param {number} [opts.end]
* @param {boolean} [opts.autoGunzip]
* @param {SSECOptions} [opts.ssecOptions]
* @return {Promise<NodeJS.ReadableStream>}
*/
async getObjectStream(bucketName, key, opts) {
opts = opts || {}
@ -192,9 +85,6 @@ class S3Persistor extends AbstractPersistor {
if (opts.start != null && opts.end != null) {
params.Range = `bytes=${opts.start}-${opts.end}`
}
if (opts.ssecOptions) {
Object.assign(params, opts.ssecOptions.getGetOptions())
}
const observer = new PersistorHelper.ObserverStream({
metric: 's3.ingress', // ingress from S3 to us
bucket: bucketName,
@ -203,21 +93,18 @@ class S3Persistor extends AbstractPersistor {
const req = this._getClientForBucket(bucketName).getObject(params)
const stream = req.createReadStream()
let contentEncoding
try {
await new Promise((resolve, reject) => {
req.on('httpHeaders', (statusCode, headers) => {
req.on('httpHeaders', statusCode => {
switch (statusCode) {
case 200: // full response
case 206: // partial response
contentEncoding = headers['content-encoding']
return resolve(undefined)
case 403: // AccessDenied
return // handled by stream.on('error') handler below
return resolve()
case 403: // AccessDenied is handled the same as NoSuchKey
case 404: // NoSuchKey
return reject(new NotFoundError('not found'))
return reject(new NotFoundError())
default:
// handled by stream.on('error') handler below
return reject(new Error('non success status: ' + statusCode))
}
})
// The AWS SDK is forwarding any errors from the request to the stream.
@ -235,32 +122,23 @@ class S3Persistor extends AbstractPersistor {
}
// Return a PassThrough stream with a minimal interface. It will buffer until the caller starts reading. It will emit errors from the source stream (Stream.pipeline passes errors along).
const pass = new PassThrough()
const transformer = []
if (contentEncoding === 'gzip' && opts.autoGunzip) {
transformer.push(zlib.createGunzip())
}
pipeline(stream, observer, ...transformer, pass, err => {
pipeline(stream, observer, pass, err => {
if (err) req.abort()
})
return pass
}
/**
* @param {string} bucketName
* @param {string} key
* @return {Promise<string>}
*/
async getRedirectUrl(bucketName, key) {
const expiresSeconds = Math.round(this.settings.signedUrlExpiryInMs / 1000)
try {
return await this._getClientForBucket(bucketName).getSignedUrlPromise(
'getObject',
{
Bucket: bucketName,
Key: key,
Expires: expiresSeconds,
}
)
const url = await this._getClientForBucket(
bucketName
).getSignedUrlPromise('getObject', {
Bucket: bucketName,
Key: key,
Expires: expiresSeconds,
})
return url
} catch (err) {
throw PersistorHelper.wrapError(
err,
@ -271,20 +149,28 @@ class S3Persistor extends AbstractPersistor {
}
}
/**
* @param {string} bucketName
* @param {string} key
* @param {string} [continuationToken]
* @return {Promise<void>}
*/
async deleteDirectory(bucketName, key, continuationToken) {
const { contents, response } = await this.listDirectory(
bucketName,
key,
continuationToken
)
const objects = contents.map(item => ({ Key: item.Key || '' }))
if (objects?.length) {
let response
const options = { Bucket: bucketName, Prefix: key }
if (continuationToken) {
options.ContinuationToken = continuationToken
}
try {
response = await this._getClientForBucket(bucketName)
.listObjectsV2(options)
.promise()
} catch (err) {
throw PersistorHelper.wrapError(
err,
'failed to list objects in S3',
{ bucketName, key },
ReadError
)
}
const objects = response.Contents.map(item => ({ Key: item.Key }))
if (objects.length) {
try {
await this._getClientForBucket(bucketName)
.deleteObjects({
@ -314,52 +200,12 @@ class S3Persistor extends AbstractPersistor {
}
}
/**
*
* @param {string} bucketName
* @param {string} key
* @param {string} [continuationToken]
* @return {Promise<ListDirectoryResult>}
*/
async listDirectory(bucketName, key, continuationToken) {
let response
const options = { Bucket: bucketName, Prefix: key }
if (continuationToken) {
options.ContinuationToken = continuationToken
}
async getObjectSize(bucketName, key) {
try {
response = await this._getClientForBucket(bucketName)
.listObjectsV2(options)
.promise()
} catch (err) {
throw PersistorHelper.wrapError(
err,
'failed to list objects in S3',
{ bucketName, key },
ReadError
)
}
return { contents: response.Contents ?? [], response }
}
/**
* @param {string} bucketName
* @param {string} key
* @param {Object} opts
* @param {SSECOptions} [opts.ssecOptions]
* @return {Promise<S3.HeadObjectOutput>}
*/
async #headObject(bucketName, key, opts = {}) {
const params = { Bucket: bucketName, Key: key }
if (opts.ssecOptions) {
Object.assign(params, opts.ssecOptions.getGetOptions())
}
try {
return await this._getClientForBucket(bucketName)
.headObject(params)
const response = await this._getClientForBucket(bucketName)
.headObject({ Bucket: bucketName, Key: key })
.promise()
return response.ContentLength
} catch (err) {
throw PersistorHelper.wrapError(
err,
@ -370,51 +216,19 @@ class S3Persistor extends AbstractPersistor {
}
}
/**
* @param {string} bucketName
* @param {string} key
* @param {Object} opts
* @param {SSECOptions} [opts.ssecOptions]
* @return {Promise<number>}
*/
async getObjectSize(bucketName, key, opts = {}) {
const response = await this.#headObject(bucketName, key, opts)
return response.ContentLength || 0
}
/**
* @param {string} bucketName
* @param {string} key
* @param {Object} opts
* @param {SSECOptions} [opts.ssecOptions]
* @return {Promise<string | undefined>}
*/
async getObjectStorageClass(bucketName, key, opts = {}) {
const response = await this.#headObject(bucketName, key, opts)
return response.StorageClass
}
/**
* @param {string} bucketName
* @param {string} key
* @param {Object} opts
* @param {SSECOptions} [opts.ssecOptions]
* @param {boolean} [opts.etagIsNotMD5]
* @return {Promise<string>}
*/
async getObjectMd5Hash(bucketName, key, opts = {}) {
async getObjectMd5Hash(bucketName, key) {
try {
if (!opts.etagIsNotMD5) {
const response = await this.#headObject(bucketName, key, opts)
const md5 = S3Persistor._md5FromResponse(response)
if (md5) {
return md5
}
const response = await this._getClientForBucket(bucketName)
.headObject({ Bucket: bucketName, Key: key })
.promise()
const md5 = S3Persistor._md5FromResponse(response)
if (md5) {
return md5
}
// etag is not in md5 format
Metrics.inc('s3.md5Download')
return await PersistorHelper.calculateStreamMd5(
await this.getObjectStream(bucketName, key, opts)
await this.getObjectStream(bucketName, key)
)
} catch (err) {
throw PersistorHelper.wrapError(
@ -426,11 +240,6 @@ class S3Persistor extends AbstractPersistor {
}
}
/**
* @param {string} bucketName
* @param {string} key
* @return {Promise<void>}
*/
async deleteObject(bucketName, key) {
try {
await this._getClientForBucket(bucketName)
@ -447,27 +256,12 @@ class S3Persistor extends AbstractPersistor {
}
}
/**
* @param {string} bucketName
* @param {string} sourceKey
* @param {string} destKey
* @param {Object} opts
* @param {SSECOptions} [opts.ssecSrcOptions]
* @param {SSECOptions} [opts.ssecOptions]
* @return {Promise<void>}
*/
async copyObject(bucketName, sourceKey, destKey, opts = {}) {
async copyObject(bucketName, sourceKey, destKey) {
const params = {
Bucket: bucketName,
Key: destKey,
CopySource: `${bucketName}/${sourceKey}`,
}
if (opts.ssecSrcOptions) {
Object.assign(params, opts.ssecSrcOptions.getCopyOptions())
}
if (opts.ssecOptions) {
Object.assign(params, opts.ssecOptions.getPutOptions())
}
try {
await this._getClientForBucket(bucketName).copyObject(params).promise()
} catch (err) {
@ -480,16 +274,9 @@ class S3Persistor extends AbstractPersistor {
}
}
/**
* @param {string} bucketName
* @param {string} key
* @param {Object} opts
* @param {SSECOptions} [opts.ssecOptions]
* @return {Promise<boolean>}
*/
async checkIfObjectExists(bucketName, key, opts) {
async checkIfObjectExists(bucketName, key) {
try {
await this.getObjectSize(bucketName, key, opts)
await this.getObjectSize(bucketName, key)
return true
} catch (err) {
if (err instanceof NotFoundError) {
@ -504,12 +291,6 @@ class S3Persistor extends AbstractPersistor {
}
}
/**
* @param {string} bucketName
* @param {string} key
* @param {string} [continuationToken]
* @return {Promise<number>}
*/
async directorySize(bucketName, key, continuationToken) {
try {
const options = {
@ -523,8 +304,7 @@ class S3Persistor extends AbstractPersistor {
.listObjectsV2(options)
.promise()
const size =
response.Contents?.reduce((acc, item) => (item.Size || 0) + acc, 0) || 0
const size = response.Contents.reduce((acc, item) => item.Size + acc, 0)
if (response.IsTruncated) {
return (
size +
@ -546,38 +326,15 @@ class S3Persistor extends AbstractPersistor {
}
}
/**
* @param {string} bucket
* @param {boolean} computeChecksums
* @return {S3}
* @private
*/
_getClientForBucket(bucket, computeChecksums = false) {
/** @type {S3.Types.ClientConfiguration} */
const clientOptions = {}
const cacheKey = `${bucket}:${computeChecksums}`
if (computeChecksums) {
clientOptions.computeChecksums = true
}
let client = this.#clients.get(cacheKey)
if (!client) {
client = new S3(
this._buildClientOptions(
this.settings.bucketCreds?.[bucket],
clientOptions
)
_getClientForBucket(bucket, clientOptions) {
return new S3(
this._buildClientOptions(
this.settings.bucketCreds?.[bucket],
clientOptions
)
this.#clients.set(cacheKey, client)
}
return client
)
}
/**
* @param {Object} bucketCredentials
* @param {S3.Types.ClientConfiguration} clientOptions
* @return {S3.Types.ClientConfiguration}
* @private
*/
_buildClientOptions(bucketCredentials, clientOptions) {
const options = clientOptions || {}
@ -599,7 +356,7 @@ class S3Persistor extends AbstractPersistor {
if (this.settings.endpoint) {
const endpoint = new URL(this.settings.endpoint)
options.endpoint = this.settings.endpoint
options.sslEnabled = endpoint.protocol === 'https:'
options.sslEnabled = endpoint.protocol === 'https'
}
// path-style access is only used for acceptance tests
@ -613,22 +370,9 @@ class S3Persistor extends AbstractPersistor {
}
}
if (options.sslEnabled && this.settings.ca && !options.httpOptions?.agent) {
options.httpOptions = options.httpOptions || {}
options.httpOptions.agent = new https.Agent({
rejectUnauthorized: true,
ca: this.settings.ca,
})
}
return options
}
/**
* @param {S3.HeadObjectOutput} response
* @return {string|null}
* @private
*/
static _md5FromResponse(response) {
const md5 = (response.ETag || '').replace(/[ "]/g, '')
if (!md5.match(/^[a-f0-9]{32}$/)) {
@ -638,8 +382,3 @@ class S3Persistor extends AbstractPersistor {
return md5
}
}
module.exports = {
S3Persistor,
SSECOptions,
}

View file

@ -1,6 +0,0 @@
import type { ListObjectsV2Output, Object } from 'aws-sdk/clients/s3'
export type ListDirectoryResult = {
contents: Array<Object>
response: ListObjectsV2Output
}

View file

@ -25,9 +25,4 @@ SandboxedModule.configure({
},
},
globals: { Buffer, Math, console, process, URL },
sourceTransformers: {
removeNodePrefix: function (source) {
return source.replace(/require\(['"]node:/g, "require('")
},
},
})

View file

@ -1,10 +1,10 @@
const crypto = require('node:crypto')
const crypto = require('crypto')
const { expect } = require('chai')
const mockFs = require('mock-fs')
const fs = require('node:fs')
const fsPromises = require('node:fs/promises')
const Path = require('node:path')
const StreamPromises = require('node:stream/promises')
const fs = require('fs')
const fsPromises = require('fs/promises')
const Path = require('path')
const StreamPromises = require('stream/promises')
const SandboxedModule = require('sandboxed-module')
const Errors = require('../../src/Errors')

View file

@ -1,4 +1,4 @@
const { EventEmitter } = require('node:events')
const { EventEmitter } = require('events')
const sinon = require('sinon')
const chai = require('chai')
const { expect } = chai
@ -45,11 +45,11 @@ describe('GcsPersistorTests', function () {
files = [
{
metadata: { size: '11', md5Hash: '/////wAAAAD/////AAAAAA==' },
metadata: { size: 11, md5Hash: '/////wAAAAD/////AAAAAA==' },
delete: sinon.stub(),
},
{
metadata: { size: '22', md5Hash: '/////wAAAAD/////AAAAAA==' },
metadata: { size: 22, md5Hash: '/////wAAAAD/////AAAAAA==' },
delete: sinon.stub(),
},
]
@ -63,7 +63,7 @@ describe('GcsPersistorTests', function () {
read() {
if (this.err) return this.emit('error', this.err)
this.emit('response', { statusCode: this.statusCode, headers: {} })
this.emit('response', { statusCode: this.statusCode })
}
}
@ -302,7 +302,7 @@ describe('GcsPersistorTests', function () {
})
it('should return the object size', function () {
expect(size).to.equal(11)
expect(size).to.equal(files[0].metadata.size)
})
it('should pass the bucket and key to GCS', function () {

View file

@ -1,7 +1,7 @@
const chai = require('chai')
const { expect } = chai
const SandboxedModule = require('sandboxed-module')
const StreamPromises = require('node:stream/promises')
const StreamPromises = require('stream/promises')
const MODULE_PATH = '../../src/PersistorFactory.js'
@ -32,7 +32,7 @@ describe('PersistorManager', function () {
Settings = {}
const requires = {
'./GcsPersistor': GcsPersistor,
'./S3Persistor': { S3Persistor },
'./S3Persistor': S3Persistor,
'./FSPersistor': FSPersistor,
'@overleaf/logger': {
info() {},

View file

@ -3,7 +3,7 @@ const chai = require('chai')
const { expect } = chai
const SandboxedModule = require('sandboxed-module')
const Errors = require('../../src/Errors')
const { EventEmitter } = require('node:events')
const { EventEmitter } = require('events')
const MODULE_PATH = '../../src/S3Persistor.js'
@ -91,22 +91,8 @@ describe('S3PersistorTests', function () {
createReadStream() {
setTimeout(() => {
if (this.notFoundSSEC) {
// special case for AWS S3: 404 NoSuchKey wrapped in a 400. A single request received a single response, and multiple httpHeaders events are triggered. Don't ask.
this.emit('httpHeaders', 400, {})
this.emit('httpHeaders', 404, {})
ReadStream.emit('error', S3NotFoundError)
return
}
if (this.err) return ReadStream.emit('error', this.err)
this.emit('httpHeaders', this.statusCode, {})
if (this.statusCode === 403) {
ReadStream.emit('error', S3AccessDeniedError)
}
if (this.statusCode === 404) {
ReadStream.emit('error', S3NotFoundError)
}
this.emit('httpHeaders', this.statusCode)
})
return ReadStream
}
@ -147,7 +133,7 @@ describe('S3PersistorTests', function () {
deleteObjects: sinon.stub().returns(EmptyPromise),
getSignedUrlPromise: sinon.stub().resolves(redirectUrl),
}
S3 = sinon.stub().callsFake(() => Object.assign({}, S3Client))
S3 = sinon.stub().returns(S3Client)
Hash = {
end: sinon.stub(),
@ -173,7 +159,7 @@ describe('S3PersistorTests', function () {
crypto,
},
globals: { console, Buffer },
}).S3Persistor)(settings)
}))(settings)
})
describe('getObjectStream', function () {
@ -352,34 +338,6 @@ describe('S3PersistorTests', function () {
})
})
describe("when the file doesn't exist -- SSEC", function () {
let error, stream
beforeEach(async function () {
S3GetObjectRequest.notFoundSSEC = 404
try {
stream = await S3Persistor.getObjectStream(bucket, key)
} catch (err) {
error = err
}
})
it('does not return a stream', function () {
expect(stream).not.to.exist
})
it('throws a NotFoundError', function () {
expect(error).to.be.an.instanceOf(Errors.NotFoundError)
})
it('wraps the error', function () {
expect(error.cause).to.exist
})
it('stores the bucket and key in the error', function () {
expect(error.info).to.include({ bucketName: bucket, key })
})
})
describe('when access to the file is denied', function () {
let error, stream
@ -401,7 +359,7 @@ describe('S3PersistorTests', function () {
})
it('wraps the error', function () {
expect(error.cause).to.equal(S3AccessDeniedError)
expect(error.cause).to.exist
})
it('stores the bucket and key in the error', function () {
@ -1027,22 +985,4 @@ describe('S3PersistorTests', function () {
})
})
})
describe('_getClientForBucket', function () {
it('should return same instance for same bucket', function () {
const a = S3Persistor._getClientForBucket('foo')
const b = S3Persistor._getClientForBucket('foo')
expect(a).to.equal(b)
})
it('should return different instance for different bucket', function () {
const a = S3Persistor._getClientForBucket('foo')
const b = S3Persistor._getClientForBucket('bar')
expect(a).to.not.equal(b)
})
it('should return different instance for same bucket different computeChecksums', function () {
const a = S3Persistor._getClientForBucket('foo', false)
const b = S3Persistor._getClientForBucket('foo', true)
expect(a).to.not.equal(b)
})
})
})

View file

@ -0,0 +1 @@
node_modules/

View file

@ -0,0 +1,5 @@
/coverage
/node_modules
# managed by monorepo$ bin/update_build_scripts
.npmrc

View file

@ -1 +1 @@
22.17.0
18.20.2

View file

@ -1,10 +1,10 @@
overleaf-editor-core
--dependencies=None
--docker-repos=us-east1-docker.pkg.dev/overleaf-ops/ol-docker
--docker-repos=gcr.io/overleaf-ops
--env-add=
--env-pass-through=
--esmock-loader=False
--is-library=True
--node-version=22.17.0
--node-version=18.20.2
--public-repo=False
--script-version=4.7.0
--script-version=4.5.0

View file

@ -18,7 +18,6 @@ const MoveFileOperation = require('./lib/operation/move_file_operation')
const SetCommentStateOperation = require('./lib/operation/set_comment_state_operation')
const EditFileOperation = require('./lib/operation/edit_file_operation')
const EditNoOperation = require('./lib/operation/edit_no_operation')
const EditOperationTransformer = require('./lib/operation/edit_operation_transformer')
const SetFileMetadataOperation = require('./lib/operation/set_file_metadata_operation')
const NoOperation = require('./lib/operation/no_operation')
const Operation = require('./lib/operation')
@ -44,8 +43,6 @@ const TrackingProps = require('./lib/file_data/tracking_props')
const Range = require('./lib/range')
const CommentList = require('./lib/file_data/comment_list')
const LazyStringFileData = require('./lib/file_data/lazy_string_file_data')
const StringFileData = require('./lib/file_data/string_file_data')
const EditOperationBuilder = require('./lib/operation/edit_operation_builder')
exports.AddCommentOperation = AddCommentOperation
exports.Author = Author
@ -61,7 +58,6 @@ exports.DeleteCommentOperation = DeleteCommentOperation
exports.File = File
exports.FileMap = FileMap
exports.LazyStringFileData = LazyStringFileData
exports.StringFileData = StringFileData
exports.History = History
exports.Label = Label
exports.AddFileOperation = AddFileOperation
@ -69,8 +65,6 @@ exports.MoveFileOperation = MoveFileOperation
exports.SetCommentStateOperation = SetCommentStateOperation
exports.EditFileOperation = EditFileOperation
exports.EditNoOperation = EditNoOperation
exports.EditOperationBuilder = EditOperationBuilder
exports.EditOperationTransformer = EditOperationTransformer
exports.SetFileMetadataOperation = SetFileMetadataOperation
exports.NoOperation = NoOperation
exports.Operation = Operation

View file

@ -40,11 +40,6 @@ class Blob {
static NotFoundError = NotFoundError
/**
* @param {string} hash
* @param {number} byteLength
* @param {number} [stringLength]
*/
constructor(hash, byteLength, stringLength) {
this.setHash(hash)
this.setByteLength(byteLength)
@ -68,14 +63,14 @@ class Blob {
/**
* Hex hash.
* @return {String}
* @return {?String}
*/
getHash() {
return this.hash
}
setHash(hash) {
assert.match(hash, Blob.HEX_HASH_RX, 'bad hash')
assert.maybe.match(hash, Blob.HEX_HASH_RX, 'bad hash')
this.hash = hash
}
@ -88,13 +83,13 @@ class Blob {
}
setByteLength(byteLength) {
assert.integer(byteLength, 'bad byteLength')
assert.maybe.integer(byteLength, 'bad byteLength')
this.byteLength = byteLength
}
/**
* Utf-8 length of the blob content, if it appears to be valid UTF-8.
* @return {number|undefined}
* @return {?number}
*/
getStringLength() {
return this.stringLength

View file

@ -13,7 +13,7 @@ const V2DocVersions = require('./v2_doc_versions')
/**
* @import Author from "./author"
* @import { BlobStore, RawChange, ReadonlyBlobStore } from "./types"
* @import { BlobStore } from "./types"
*/
/**
@ -54,7 +54,7 @@ class Change {
/**
* For serialization.
*
* @return {RawChange}
* @return {Object}
*/
toRaw() {
function toRaw(object) {
@ -100,9 +100,6 @@ class Change {
)
}
/**
* @return {Operation[]}
*/
getOperations() {
return this.operations
}
@ -219,7 +216,7 @@ class Change {
* If this Change contains any File objects, load them.
*
* @param {string} kind see {File#load}
* @param {ReadonlyBlobStore} blobStore
* @param {BlobStore} blobStore
* @return {Promise<void>}
*/
async loadFiles(kind, blobStore) {
@ -251,24 +248,6 @@ class Change {
* @param {boolean} [opts.strict] - Do not ignore recoverable errors
*/
applyTo(snapshot, opts = {}) {
// eslint-disable-next-line no-unused-vars
for (const operation of this.iterativelyApplyTo(snapshot, opts)) {
// Nothing to do: we're just consuming the iterator for the side effects
}
}
/**
* Generator that applies this change to a snapshot and yields each
* operation after it has been applied.
*
* Recoverable errors (caused by historical bad data) are ignored unless
* opts.strict is true
*
* @param {Snapshot} snapshot modified in place
* @param {object} opts
* @param {boolean} [opts.strict] - Do not ignore recoverable errors
*/
*iterativelyApplyTo(snapshot, opts = {}) {
assert.object(snapshot, 'bad snapshot')
for (const operation of this.operations) {
@ -282,7 +261,6 @@ class Change {
throw err
}
}
yield operation
}
// update project version if present in change

View file

@ -10,7 +10,7 @@ const Change = require('./change')
class ChangeNote {
/**
* @param {number} baseVersion the new base version for the change
* @param {Change} [change]
* @param {?Change} change
*/
constructor(baseVersion, change) {
assert.integer(baseVersion, 'bad baseVersion')

View file

@ -95,7 +95,7 @@ class File {
/**
* @param {number} byteLength
* @param {number} [stringLength]
* @param {number?} stringLength
* @param {Object} [metadata]
* @return {File}
*/

View file

@ -1,7 +1,7 @@
// @ts-check
/**
* @import { ClearTrackingPropsRawData, TrackingDirective } from '../types'
* @import { ClearTrackingPropsRawData } from '../types'
*/
class ClearTrackingProps {
@ -11,27 +11,12 @@ class ClearTrackingProps {
/**
* @param {any} other
* @returns {other is ClearTrackingProps}
* @returns {boolean}
*/
equals(other) {
return other instanceof ClearTrackingProps
}
/**
* @param {TrackingDirective} other
* @returns {other is ClearTrackingProps}
*/
canMergeWith(other) {
return other instanceof ClearTrackingProps
}
/**
* @param {TrackingDirective} other
*/
mergeWith(other) {
return this
}
/**
* @returns {ClearTrackingPropsRawData}
*/

View file

@ -47,7 +47,7 @@ class FileData {
/** @see File.createHollow
* @param {number} byteLength
* @param {number} [stringLength]
* @param {number|null} stringLength
*/
static createHollow(byteLength, stringLength) {
if (stringLength == null) {
@ -63,14 +63,20 @@ class FileData {
*/
static createLazyFromBlobs(blob, rangesBlob) {
assert.instance(blob, Blob, 'FileData: bad blob')
const stringLength = blob.getStringLength()
if (stringLength == null) {
return new BinaryFileData(blob.getHash(), blob.getByteLength())
if (blob.getStringLength() == null) {
return new BinaryFileData(
// TODO(das7pad): see call-sites
// @ts-ignore
blob.getHash(),
blob.getByteLength()
)
}
return new LazyStringFileData(
// TODO(das7pad): see call-sites
// @ts-ignore
blob.getHash(),
rangesBlob?.getHash(),
stringLength
blob.getStringLength()
)
}

View file

@ -11,7 +11,7 @@ const EditOperation = require('../operation/edit_operation')
const EditOperationBuilder = require('../operation/edit_operation_builder')
/**
* @import { BlobStore, ReadonlyBlobStore, RangesBlob, RawHashFileData, RawLazyStringFileData } from '../types'
* @import { BlobStore, ReadonlyBlobStore, RangesBlob, RawFileData, RawLazyStringFileData } from '../types'
*/
class LazyStringFileData extends FileData {
@ -159,11 +159,11 @@ class LazyStringFileData extends FileData {
/** @inheritdoc
* @param {BlobStore} blobStore
* @return {Promise<RawHashFileData>}
* @return {Promise<RawFileData>}
*/
async store(blobStore) {
if (this.operations.length === 0) {
/** @type RawHashFileData */
/** @type RawFileData */
const raw = { hash: this.hash }
if (this.rangesHash) {
raw.rangesHash = this.rangesHash
@ -171,11 +171,9 @@ class LazyStringFileData extends FileData {
return raw
}
const eager = await this.toEager(blobStore)
const raw = await eager.store(blobStore)
this.hash = raw.hash
this.rangesHash = raw.rangesHash
this.operations.length = 0
return raw
/** @type RawFileData */
return await eager.store(blobStore)
}
}

View file

@ -8,7 +8,7 @@ const CommentList = require('./comment_list')
const TrackedChangeList = require('./tracked_change_list')
/**
* @import { StringFileRawData, RawHashFileData, BlobStore, CommentRawData } from "../types"
* @import { StringFileRawData, RawFileData, BlobStore, CommentRawData } from "../types"
* @import { TrackedChangeRawData, RangesBlob } from "../types"
* @import EditOperation from "../operation/edit_operation"
*/
@ -88,14 +88,6 @@ class StringFileData extends FileData {
return content
}
/**
* Return docstore view of a doc: each line separated
* @return {string[]}
*/
getLines() {
return this.getContent({ filterTrackedDeletes: true }).split('\n')
}
/** @inheritdoc */
getByteLength() {
return Buffer.byteLength(this.content)
@ -139,7 +131,7 @@ class StringFileData extends FileData {
/**
* @inheritdoc
* @param {BlobStore} blobStore
* @return {Promise<RawHashFileData>}
* @return {Promise<RawFileData>}
*/
async store(blobStore) {
const blob = await blobStore.putString(this.content)
@ -150,8 +142,12 @@ class StringFileData extends FileData {
trackedChanges: this.trackedChanges.toRaw(),
}
const rangesBlob = await blobStore.putObject(ranges)
// TODO(das7pad): Provide interface that guarantees hash exists?
// @ts-ignore
return { hash: blob.getHash(), rangesHash: rangesBlob.getHash() }
}
// TODO(das7pad): Provide interface that guarantees hash exists?
// @ts-ignore
return { hash: blob.getHash() }
}
}

View file

@ -84,21 +84,6 @@ class TrackedChange {
)
)
}
/**
* Return an equivalent tracked change whose extent is limited to the given
* range
*
* @param {Range} range
* @returns {TrackedChange | null} - the result or null if the intersection is empty
*/
intersectRange(range) {
const intersection = this.range.intersect(range)
if (intersection == null) {
return null
}
return new TrackedChange(intersection, this.tracking)
}
}
module.exports = TrackedChange

Some files were not shown because too many files have changed in this diff Show more