bug fix ssh key display (again)

.github/ISSUE_TEMPLATE.md

@@ -0,0 +1,44 @@
+<!--
+
+Note: If you are using www.overleaf.com and have a problem, 
+      or if you would like to request a new feature please contact
+      the support team at support@overleaf.com
+      
+      This form should only be used to report bugs in the 
+      Community Edition release of Overleaf.
+
+-->
+
+
+
+<!-- BUG REPORT TEMPLATE -->
+
+## Steps to Reproduce
+<!-- Describe the steps leading up to when / where you found the bug. -->
+<!-- Screenshots may be helpful here. -->
+
+1.
+2.
+3.
+
+## Expected Behaviour
+<!-- What should have happened when you completed the steps above? -->
+
+## Observed Behaviour
+<!-- What actually happened when you completed the steps above? -->
+<!-- Screenshots may be helpful here. -->
+
+## Context
+<!-- How has this issue affected you? What were you trying to accomplish? -->
+
+## Technical Info
+<!-- Provide any technical details that may be applicable (or N/A if not applicable). -->
+
+* URL:
+* Browser Name and version:
+* Operating System and version (desktop or mobile):
+* Signed in as:
+* Project and/or file:
+
+## Analysis
+<!--- Optionally, document investigation of / suggest a fix for the bug, e.g. 'comes from this line / commit' -->

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,11 @@
+## Description
+<!-- Goal of the pull request -->
+
+
+## Related issues / Pull Requests
+<!-- Fixes #xyz, Contributes to #xyz, Related to #xyz-->
+
+
+## Contributor Agreement
+
+- [ ] I confirm I have signed the [Contributor License Agreement](https://github.com/overleaf/overleaf/blob/master/CONTRIBUTING.md#contributor-license-agreement)

.gitignore

@@ -0,0 +1,6 @@
+# docker image build
+.dockerignore
+
+# user defined files
+.env
+docker-compose.override.yml

CONTRIBUTING.md

@@ -0,0 +1,43 @@
+Contributing to Overleaf
+==========================
+
+Thank you for reading this! If you'd like to report a bug or join in the development
+of Overleaf, then here are some notes on how to do that.
+
+Reporting bugs and opening issues
+---------------------------------
+
+If you'd like to report a bug or open an issue, please **[check if there is an existing issue](https://github.com/overleaf/overleaf/issues).**
+If there is then please add any more information that you have, or give it a 👍.
+
+When submitting an issue please describe the issue as clearly as possible, including how to
+reproduce the bug, which situations it appears in, what you expected to happen, and what actually happens.
+If you can include a screenshot for front end issues that is very helpful.
+
+**Note**: If you are using [www.overleaf.com](www.overleaf.com) and have a problem, or if you would like to request a new feature, please contact the Support team at support@overleaf.com. Raise an issue here only to report bugs in the Community Edition release of Overleaf.
+
+Pull Requests
+-------------
+
+See [our wiki](https://github.com/overleaf/overleaf/wiki)
+for how to manage the Overleaf development environment and for our developer guidelines.
+
+We love pull requests, so be bold with them! Don't be afraid of going ahead
+and changing something, or adding a new feature. We're very happy to work with you
+to get your changes merged into Overleaf.
+
+If you're looking for something to work on, have a look at the [open issues](https://github.com/overleaf/overleaf/issues).
+
+Security
+--------
+
+Please see [our security policy](https://github.com/overleaf/overleaf/security/policy) if you would like to report a potential security vulnerability.
+
+Contributor License Agreement
+-----------------------------
+
+Before we can accept any contributions of code, we need you to agree to our
+[Contributor License Agreement](https://docs.google.com/forms/d/e/1FAIpQLSef79XH3mb7yIiMzZw-yALEegS-wyFetvjTiNBfZvf_IHD2KA/viewform?usp=sf_link).
+This is to ensure that you own the copyright of your contribution, and that you
+agree to give us a license to use it in both the open source version, and the version
+of Overleaf running at www.overleaf.com, which may have additional changes.

LICENSE

@@ -0,0 +1,661 @@
+                    GNU AFFERO GENERAL PUBLIC LICENSE
+                       Version 3, 19 November 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+                            Preamble
+
+  The GNU Affero General Public License is a free, copyleft license for
+software and other kinds of works, specifically designed to ensure
+cooperation with the community in the case of network server software.
+
+  The licenses for most software and other practical works are designed
+to take away your freedom to share and change the works.  By contrast,
+our General Public Licenses are intended to guarantee your freedom to
+share and change all versions of a program--to make sure it remains free
+software for all its users.
+
+  When we speak of free software, we are referring to freedom, not
+price.  Our General Public Licenses are designed to make sure that you
+have the freedom to distribute copies of free software (and charge for
+them if you wish), that you receive source code or can get it if you
+want it, that you can change the software or use pieces of it in new
+free programs, and that you know you can do these things.
+
+  Developers that use our General Public Licenses protect your rights
+with two steps: (1) assert copyright on the software, and (2) offer
+you this License which gives you legal permission to copy, distribute
+and/or modify the software.
+
+  A secondary benefit of defending all users' freedom is that
+improvements made in alternate versions of the program, if they
+receive widespread use, become available for other developers to
+incorporate.  Many developers of free software are heartened and
+encouraged by the resulting cooperation.  However, in the case of
+software used on network servers, this result may fail to come about.
+The GNU General Public License permits making a modified version and
+letting the public access it on a server without ever releasing its
+source code to the public.
+
+  The GNU Affero General Public License is designed specifically to
+ensure that, in such cases, the modified source code becomes available
+to the community.  It requires the operator of a network server to
+provide the source code of the modified version running there to the
+users of that server.  Therefore, public use of a modified version, on
+a publicly accessible server, gives the public access to the source
+code of the modified version.
+
+  An older license, called the Affero General Public License and
+published by Affero, was designed to accomplish similar goals.  This is
+a different license, not a version of the Affero GPL, but Affero has
+released a new version of the Affero GPL which permits relicensing under
+this license.
+
+  The precise terms and conditions for copying, distribution and
+modification follow.
+
+                       TERMS AND CONDITIONS
+
+  0. Definitions.
+
+  "This License" refers to version 3 of the GNU Affero General Public License.
+
+  "Copyright" also means copyright-like laws that apply to other kinds of
+works, such as semiconductor masks.
+
+  "The Program" refers to any copyrightable work licensed under this
+License.  Each licensee is addressed as "you".  "Licensees" and
+"recipients" may be individuals or organizations.
+
+  To "modify" a work means to copy from or adapt all or part of the work
+in a fashion requiring copyright permission, other than the making of an
+exact copy.  The resulting work is called a "modified version" of the
+earlier work or a work "based on" the earlier work.
+
+  A "covered work" means either the unmodified Program or a work based
+on the Program.
+
+  To "propagate" a work means to do anything with it that, without
+permission, would make you directly or secondarily liable for
+infringement under applicable copyright law, except executing it on a
+computer or modifying a private copy.  Propagation includes copying,
+distribution (with or without modification), making available to the
+public, and in some countries other activities as well.
+
+  To "convey" a work means any kind of propagation that enables other
+parties to make or receive copies.  Mere interaction with a user through
+a computer network, with no transfer of a copy, is not conveying.
+
+  An interactive user interface displays "Appropriate Legal Notices"
+to the extent that it includes a convenient and prominently visible
+feature that (1) displays an appropriate copyright notice, and (2)
+tells the user that there is no warranty for the work (except to the
+extent that warranties are provided), that licensees may convey the
+work under this License, and how to view a copy of this License.  If
+the interface presents a list of user commands or options, such as a
+menu, a prominent item in the list meets this criterion.
+
+  1. Source Code.
+
+  The "source code" for a work means the preferred form of the work
+for making modifications to it.  "Object code" means any non-source
+form of a work.
+
+  A "Standard Interface" means an interface that either is an official
+standard defined by a recognized standards body, or, in the case of
+interfaces specified for a particular programming language, one that
+is widely used among developers working in that language.
+
+  The "System Libraries" of an executable work include anything, other
+than the work as a whole, that (a) is included in the normal form of
+packaging a Major Component, but which is not part of that Major
+Component, and (b) serves only to enable use of the work with that
+Major Component, or to implement a Standard Interface for which an
+implementation is available to the public in source code form.  A
+"Major Component", in this context, means a major essential component
+(kernel, window system, and so on) of the specific operating system
+(if any) on which the executable work runs, or a compiler used to
+produce the work, or an object code interpreter used to run it.
+
+  The "Corresponding Source" for a work in object code form means all
+the source code needed to generate, install, and (for an executable
+work) run the object code and to modify the work, including scripts to
+control those activities.  However, it does not include the work's
+System Libraries, or general-purpose tools or generally available free
+programs which are used unmodified in performing those activities but
+which are not part of the work.  For example, Corresponding Source
+includes interface definition files associated with source files for
+the work, and the source code for shared libraries and dynamically
+linked subprograms that the work is specifically designed to require,
+such as by intimate data communication or control flow between those
+subprograms and other parts of the work.
+
+  The Corresponding Source need not include anything that users
+can regenerate automatically from other parts of the Corresponding
+Source.
+
+  The Corresponding Source for a work in source code form is that
+same work.
+
+  2. Basic Permissions.
+
+  All rights granted under this License are granted for the term of
+copyright on the Program, and are irrevocable provided the stated
+conditions are met.  This License explicitly affirms your unlimited
+permission to run the unmodified Program.  The output from running a
+covered work is covered by this License only if the output, given its
+content, constitutes a covered work.  This License acknowledges your
+rights of fair use or other equivalent, as provided by copyright law.
+
+  You may make, run and propagate covered works that you do not
+convey, without conditions so long as your license otherwise remains
+in force.  You may convey covered works to others for the sole purpose
+of having them make modifications exclusively for you, or provide you
+with facilities for running those works, provided that you comply with
+the terms of this License in conveying all material for which you do
+not control copyright.  Those thus making or running the covered works
+for you must do so exclusively on your behalf, under your direction
+and control, on terms that prohibit them from making any copies of
+your copyrighted material outside their relationship with you.
+
+  Conveying under any other circumstances is permitted solely under
+the conditions stated below.  Sublicensing is not allowed; section 10
+makes it unnecessary.
+
+  3. Protecting Users' Legal Rights From Anti-Circumvention Law.
+
+  No covered work shall be deemed part of an effective technological
+measure under any applicable law fulfilling obligations under article
+11 of the WIPO copyright treaty adopted on 20 December 1996, or
+similar laws prohibiting or restricting circumvention of such
+measures.
+
+  When you convey a covered work, you waive any legal power to forbid
+circumvention of technological measures to the extent such circumvention
+is effected by exercising rights under this License with respect to
+the covered work, and you disclaim any intention to limit operation or
+modification of the work as a means of enforcing, against the work's
+users, your or third parties' legal rights to forbid circumvention of
+technological measures.
+
+  4. Conveying Verbatim Copies.
+
+  You may convey verbatim copies of the Program's source code as you
+receive it, in any medium, provided that you conspicuously and
+appropriately publish on each copy an appropriate copyright notice;
+keep intact all notices stating that this License and any
+non-permissive terms added in accord with section 7 apply to the code;
+keep intact all notices of the absence of any warranty; and give all
+recipients a copy of this License along with the Program.
+
+  You may charge any price or no price for each copy that you convey,
+and you may offer support or warranty protection for a fee.
+
+  5. Conveying Modified Source Versions.
+
+  You may convey a work based on the Program, or the modifications to
+produce it from the Program, in the form of source code under the
+terms of section 4, provided that you also meet all of these conditions:
+
+    a) The work must carry prominent notices stating that you modified
+    it, and giving a relevant date.
+
+    b) The work must carry prominent notices stating that it is
+    released under this License and any conditions added under section
+    7.  This requirement modifies the requirement in section 4 to
+    "keep intact all notices".
+
+    c) You must license the entire work, as a whole, under this
+    License to anyone who comes into possession of a copy.  This
+    License will therefore apply, along with any applicable section 7
+    additional terms, to the whole of the work, and all its parts,
+    regardless of how they are packaged.  This License gives no
+    permission to license the work in any other way, but it does not
+    invalidate such permission if you have separately received it.
+
+    d) If the work has interactive user interfaces, each must display
+    Appropriate Legal Notices; however, if the Program has interactive
+    interfaces that do not display Appropriate Legal Notices, your
+    work need not make them do so.
+
+  A compilation of a covered work with other separate and independent
+works, which are not by their nature extensions of the covered work,
+and which are not combined with it such as to form a larger program,
+in or on a volume of a storage or distribution medium, is called an
+"aggregate" if the compilation and its resulting copyright are not
+used to limit the access or legal rights of the compilation's users
+beyond what the individual works permit.  Inclusion of a covered work
+in an aggregate does not cause this License to apply to the other
+parts of the aggregate.
+
+  6. Conveying Non-Source Forms.
+
+  You may convey a covered work in object code form under the terms
+of sections 4 and 5, provided that you also convey the
+machine-readable Corresponding Source under the terms of this License,
+in one of these ways:
+
+    a) Convey the object code in, or embodied in, a physical product
+    (including a physical distribution medium), accompanied by the
+    Corresponding Source fixed on a durable physical medium
+    customarily used for software interchange.
+
+    b) Convey the object code in, or embodied in, a physical product
+    (including a physical distribution medium), accompanied by a
+    written offer, valid for at least three years and valid for as
+    long as you offer spare parts or customer support for that product
+    model, to give anyone who possesses the object code either (1) a
+    copy of the Corresponding Source for all the software in the
+    product that is covered by this License, on a durable physical
+    medium customarily used for software interchange, for a price no
+    more than your reasonable cost of physically performing this
+    conveying of source, or (2) access to copy the
+    Corresponding Source from a network server at no charge.
+
+    c) Convey individual copies of the object code with a copy of the
+    written offer to provide the Corresponding Source.  This
+    alternative is allowed only occasionally and noncommercially, and
+    only if you received the object code with such an offer, in accord
+    with subsection 6b.
+
+    d) Convey the object code by offering access from a designated
+    place (gratis or for a charge), and offer equivalent access to the
+    Corresponding Source in the same way through the same place at no
+    further charge.  You need not require recipients to copy the
+    Corresponding Source along with the object code.  If the place to
+    copy the object code is a network server, the Corresponding Source
+    may be on a different server (operated by you or a third party)
+    that supports equivalent copying facilities, provided you maintain
+    clear directions next to the object code saying where to find the
+    Corresponding Source.  Regardless of what server hosts the
+    Corresponding Source, you remain obligated to ensure that it is
+    available for as long as needed to satisfy these requirements.
+
+    e) Convey the object code using peer-to-peer transmission, provided
+    you inform other peers where the object code and Corresponding
+    Source of the work are being offered to the general public at no
+    charge under subsection 6d.
+
+  A separable portion of the object code, whose source code is excluded
+from the Corresponding Source as a System Library, need not be
+included in conveying the object code work.
+
+  A "User Product" is either (1) a "consumer product", which means any
+tangible personal property which is normally used for personal, family,
+or household purposes, or (2) anything designed or sold for incorporation
+into a dwelling.  In determining whether a product is a consumer product,
+doubtful cases shall be resolved in favor of coverage.  For a particular
+product received by a particular user, "normally used" refers to a
+typical or common use of that class of product, regardless of the status
+of the particular user or of the way in which the particular user
+actually uses, or expects or is expected to use, the product.  A product
+is a consumer product regardless of whether the product has substantial
+commercial, industrial or non-consumer uses, unless such uses represent
+the only significant mode of use of the product.
+
+  "Installation Information" for a User Product means any methods,
+procedures, authorization keys, or other information required to install
+and execute modified versions of a covered work in that User Product from
+a modified version of its Corresponding Source.  The information must
+suffice to ensure that the continued functioning of the modified object
+code is in no case prevented or interfered with solely because
+modification has been made.
+
+  If you convey an object code work under this section in, or with, or
+specifically for use in, a User Product, and the conveying occurs as
+part of a transaction in which the right of possession and use of the
+User Product is transferred to the recipient in perpetuity or for a
+fixed term (regardless of how the transaction is characterized), the
+Corresponding Source conveyed under this section must be accompanied
+by the Installation Information.  But this requirement does not apply
+if neither you nor any third party retains the ability to install
+modified object code on the User Product (for example, the work has
+been installed in ROM).
+
+  The requirement to provide Installation Information does not include a
+requirement to continue to provide support service, warranty, or updates
+for a work that has been modified or installed by the recipient, or for
+the User Product in which it has been modified or installed.  Access to a
+network may be denied when the modification itself materially and
+adversely affects the operation of the network or violates the rules and
+protocols for communication across the network.
+
+  Corresponding Source conveyed, and Installation Information provided,
+in accord with this section must be in a format that is publicly
+documented (and with an implementation available to the public in
+source code form), and must require no special password or key for
+unpacking, reading or copying.
+
+  7. Additional Terms.
+
+  "Additional permissions" are terms that supplement the terms of this
+License by making exceptions from one or more of its conditions.
+Additional permissions that are applicable to the entire Program shall
+be treated as though they were included in this License, to the extent
+that they are valid under applicable law.  If additional permissions
+apply only to part of the Program, that part may be used separately
+under those permissions, but the entire Program remains governed by
+this License without regard to the additional permissions.
+
+  When you convey a copy of a covered work, you may at your option
+remove any additional permissions from that copy, or from any part of
+it.  (Additional permissions may be written to require their own
+removal in certain cases when you modify the work.)  You may place
+additional permissions on material, added by you to a covered work,
+for which you have or can give appropriate copyright permission.
+
+  Notwithstanding any other provision of this License, for material you
+add to a covered work, you may (if authorized by the copyright holders of
+that material) supplement the terms of this License with terms:
+
+    a) Disclaiming warranty or limiting liability differently from the
+    terms of sections 15 and 16 of this License; or
+
+    b) Requiring preservation of specified reasonable legal notices or
+    author attributions in that material or in the Appropriate Legal
+    Notices displayed by works containing it; or
+
+    c) Prohibiting misrepresentation of the origin of that material, or
+    requiring that modified versions of such material be marked in
+    reasonable ways as different from the original version; or
+
+    d) Limiting the use for publicity purposes of names of licensors or
+    authors of the material; or
+
+    e) Declining to grant rights under trademark law for use of some
+    trade names, trademarks, or service marks; or
+
+    f) Requiring indemnification of licensors and authors of that
+    material by anyone who conveys the material (or modified versions of
+    it) with contractual assumptions of liability to the recipient, for
+    any liability that these contractual assumptions directly impose on
+    those licensors and authors.
+
+  All other non-permissive additional terms are considered "further
+restrictions" within the meaning of section 10.  If the Program as you
+received it, or any part of it, contains a notice stating that it is
+governed by this License along with a term that is a further
+restriction, you may remove that term.  If a license document contains
+a further restriction but permits relicensing or conveying under this
+License, you may add to a covered work material governed by the terms
+of that license document, provided that the further restriction does
+not survive such relicensing or conveying.
+
+  If you add terms to a covered work in accord with this section, you
+must place, in the relevant source files, a statement of the
+additional terms that apply to those files, or a notice indicating
+where to find the applicable terms.
+
+  Additional terms, permissive or non-permissive, may be stated in the
+form of a separately written license, or stated as exceptions;
+the above requirements apply either way.
+
+  8. Termination.
+
+  You may not propagate or modify a covered work except as expressly
+provided under this License.  Any attempt otherwise to propagate or
+modify it is void, and will automatically terminate your rights under
+this License (including any patent licenses granted under the third
+paragraph of section 11).
+
+  However, if you cease all violation of this License, then your
+license from a particular copyright holder is reinstated (a)
+provisionally, unless and until the copyright holder explicitly and
+finally terminates your license, and (b) permanently, if the copyright
+holder fails to notify you of the violation by some reasonable means
+prior to 60 days after the cessation.
+
+  Moreover, your license from a particular copyright holder is
+reinstated permanently if the copyright holder notifies you of the
+violation by some reasonable means, this is the first time you have
+received notice of violation of this License (for any work) from that
+copyright holder, and you cure the violation prior to 30 days after
+your receipt of the notice.
+
+  Termination of your rights under this section does not terminate the
+licenses of parties who have received copies or rights from you under
+this License.  If your rights have been terminated and not permanently
+reinstated, you do not qualify to receive new licenses for the same
+material under section 10.
+
+  9. Acceptance Not Required for Having Copies.
+
+  You are not required to accept this License in order to receive or
+run a copy of the Program.  Ancillary propagation of a covered work
+occurring solely as a consequence of using peer-to-peer transmission
+to receive a copy likewise does not require acceptance.  However,
+nothing other than this License grants you permission to propagate or
+modify any covered work.  These actions infringe copyright if you do
+not accept this License.  Therefore, by modifying or propagating a
+covered work, you indicate your acceptance of this License to do so.
+
+  10. Automatic Licensing of Downstream Recipients.
+
+  Each time you convey a covered work, the recipient automatically
+receives a license from the original licensors, to run, modify and
+propagate that work, subject to this License.  You are not responsible
+for enforcing compliance by third parties with this License.
+
+  An "entity transaction" is a transaction transferring control of an
+organization, or substantially all assets of one, or subdividing an
+organization, or merging organizations.  If propagation of a covered
+work results from an entity transaction, each party to that
+transaction who receives a copy of the work also receives whatever
+licenses to the work the party's predecessor in interest had or could
+give under the previous paragraph, plus a right to possession of the
+Corresponding Source of the work from the predecessor in interest, if
+the predecessor has it or can get it with reasonable efforts.
+
+  You may not impose any further restrictions on the exercise of the
+rights granted or affirmed under this License.  For example, you may
+not impose a license fee, royalty, or other charge for exercise of
+rights granted under this License, and you may not initiate litigation
+(including a cross-claim or counterclaim in a lawsuit) alleging that
+any patent claim is infringed by making, using, selling, offering for
+sale, or importing the Program or any portion of it.
+
+  11. Patents.
+
+  A "contributor" is a copyright holder who authorizes use under this
+License of the Program or a work on which the Program is based.  The
+work thus licensed is called the contributor's "contributor version".
+
+  A contributor's "essential patent claims" are all patent claims
+owned or controlled by the contributor, whether already acquired or
+hereafter acquired, that would be infringed by some manner, permitted
+by this License, of making, using, or selling its contributor version,
+but do not include claims that would be infringed only as a
+consequence of further modification of the contributor version.  For
+purposes of this definition, "control" includes the right to grant
+patent sublicenses in a manner consistent with the requirements of
+this License.
+
+  Each contributor grants you a non-exclusive, worldwide, royalty-free
+patent license under the contributor's essential patent claims, to
+make, use, sell, offer for sale, import and otherwise run, modify and
+propagate the contents of its contributor version.
+
+  In the following three paragraphs, a "patent license" is any express
+agreement or commitment, however denominated, not to enforce a patent
+(such as an express permission to practice a patent or covenant not to
+sue for patent infringement).  To "grant" such a patent license to a
+party means to make such an agreement or commitment not to enforce a
+patent against the party.
+
+  If you convey a covered work, knowingly relying on a patent license,
+and the Corresponding Source of the work is not available for anyone
+to copy, free of charge and under the terms of this License, through a
+publicly available network server or other readily accessible means,
+then you must either (1) cause the Corresponding Source to be so
+available, or (2) arrange to deprive yourself of the benefit of the
+patent license for this particular work, or (3) arrange, in a manner
+consistent with the requirements of this License, to extend the patent
+license to downstream recipients.  "Knowingly relying" means you have
+actual knowledge that, but for the patent license, your conveying the
+covered work in a country, or your recipient's use of the covered work
+in a country, would infringe one or more identifiable patents in that
+country that you have reason to believe are valid.
+
+  If, pursuant to or in connection with a single transaction or
+arrangement, you convey, or propagate by procuring conveyance of, a
+covered work, and grant a patent license to some of the parties
+receiving the covered work authorizing them to use, propagate, modify
+or convey a specific copy of the covered work, then the patent license
+you grant is automatically extended to all recipients of the covered
+work and works based on it.
+
+  A patent license is "discriminatory" if it does not include within
+the scope of its coverage, prohibits the exercise of, or is
+conditioned on the non-exercise of one or more of the rights that are
+specifically granted under this License.  You may not convey a covered
+work if you are a party to an arrangement with a third party that is
+in the business of distributing software, under which you make payment
+to the third party based on the extent of your activity of conveying
+the work, and under which the third party grants, to any of the
+parties who would receive the covered work from you, a discriminatory
+patent license (a) in connection with copies of the covered work
+conveyed by you (or copies made from those copies), or (b) primarily
+for and in connection with specific products or compilations that
+contain the covered work, unless you entered into that arrangement,
+or that patent license was granted, prior to 28 March 2007.
+
+  Nothing in this License shall be construed as excluding or limiting
+any implied license or other defenses to infringement that may
+otherwise be available to you under applicable patent law.
+
+  12. No Surrender of Others' Freedom.
+
+  If conditions are imposed on you (whether by court order, agreement or
+otherwise) that contradict the conditions of this License, they do not
+excuse you from the conditions of this License.  If you cannot convey a
+covered work so as to satisfy simultaneously your obligations under this
+License and any other pertinent obligations, then as a consequence you may
+not convey it at all.  For example, if you agree to terms that obligate you
+to collect a royalty for further conveying from those to whom you convey
+the Program, the only way you could satisfy both those terms and this
+License would be to refrain entirely from conveying the Program.
+
+  13. Remote Network Interaction; Use with the GNU General Public License.
+
+  Notwithstanding any other provision of this License, if you modify the
+Program, your modified version must prominently offer all users
+interacting with it remotely through a computer network (if your version
+supports such interaction) an opportunity to receive the Corresponding
+Source of your version by providing access to the Corresponding Source
+from a network server at no charge, through some standard or customary
+means of facilitating copying of software.  This Corresponding Source
+shall include the Corresponding Source for any work covered by version 3
+of the GNU General Public License that is incorporated pursuant to the
+following paragraph.
+
+  Notwithstanding any other provision of this License, you have
+permission to link or combine any covered work with a work licensed
+under version 3 of the GNU General Public License into a single
+combined work, and to convey the resulting work.  The terms of this
+License will continue to apply to the part which is the covered work,
+but the work with which it is combined will remain governed by version
+3 of the GNU General Public License.
+
+  14. Revised Versions of this License.
+
+  The Free Software Foundation may publish revised and/or new versions of
+the GNU Affero General Public License from time to time.  Such new versions
+will be similar in spirit to the present version, but may differ in detail to
+address new problems or concerns.
+
+  Each version is given a distinguishing version number.  If the
+Program specifies that a certain numbered version of the GNU Affero General
+Public License "or any later version" applies to it, you have the
+option of following the terms and conditions either of that numbered
+version or of any later version published by the Free Software
+Foundation.  If the Program does not specify a version number of the
+GNU Affero General Public License, you may choose any version ever published
+by the Free Software Foundation.
+
+  If the Program specifies that a proxy can decide which future
+versions of the GNU Affero General Public License can be used, that proxy's
+public statement of acceptance of a version permanently authorizes you
+to choose that version for the Program.
+
+  Later license versions may give you additional or different
+permissions.  However, no additional obligations are imposed on any
+author or copyright holder as a result of your choosing to follow a
+later version.
+
+  15. Disclaimer of Warranty.
+
+  THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
+APPLICABLE LAW.  EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
+HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
+OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
+THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+PURPOSE.  THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
+IS WITH YOU.  SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
+ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
+
+  16. Limitation of Liability.
+
+  IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
+WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
+THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
+GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
+USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
+DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
+PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
+EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
+SUCH DAMAGES.
+
+  17. Interpretation of Sections 15 and 16.
+
+  If the disclaimer of warranty and limitation of liability provided
+above cannot be given local legal effect according to their terms,
+reviewing courts shall apply local law that most closely approximates
+an absolute waiver of all civil liability in connection with the
+Program, unless a warranty or assumption of liability accompanies a
+copy of the Program in return for a fee.
+
+                     END OF TERMS AND CONDITIONS
+
+            How to Apply These Terms to Your New Programs
+
+  If you develop a new program, and you want it to be of the greatest
+possible use to the public, the best way to achieve this is to make it
+free software which everyone can redistribute and change under these terms.
+
+  To do so, attach the following notices to the program.  It is safest
+to attach them to the start of each source file to most effectively
+state the exclusion of warranty; and each file should have at least
+the "copyright" line and a pointer to where the full notice is found.
+
+    <one line to give the program's name and a brief idea of what it does.>
+    Copyright (C) <year>  <name of author>
+
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU Affero General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU Affero General Public License for more details.
+
+    You should have received a copy of the GNU Affero General Public License
+    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+Also add information on how to contact you by electronic and paper mail.
+
+  If your software can interact with users remotely through a computer
+network, you should also make sure that it provides a way for users to
+get its source.  For example, if your program is a web application, its
+interface could display a "Source" link that leads users to an archive
+of the code.  There are many ways you could offer source, and different
+solutions will be better for different programs; see section 13 for the
+specific requirements.
+
+  You should also get your employer (if you work as a programmer) or school,
+if any, to sign a "copyright disclaimer" for the program, if necessary.
+For more information on this, and how to apply and follow the GNU AGPL, see
+<http://www.gnu.org/licenses/>.

bin/shared/mongodb-init-replica-set.js

@@ -0,0 +1,3 @@
+/* eslint-disable no-undef */
+
+rs.initiate({ _id: 'overleaf', members: [{ _id: 0, host: 'mongo:27017' }] })

develop/.gitignore

@@ -0,0 +1,3 @@
+/compiles/*
+!.gitkeep
+.env

develop/README.md

@@ -0,0 +1,90 @@
+# Overleaf Community Edition, development environment
+
+## Building and running
+
+In this `develop` directory, build the services:
+
+```shell
+bin/build
+```
+
+> [!NOTE]
+> If Docker is running out of RAM while building the services in parallel, create a `.env` file in this directory containing `COMPOSE_PARALLEL_LIMIT=1`.
+
+Then start the services:
+
+```shell
+bin/up
+```
+
+Once the services are running, open <http://localhost/launchpad> to create the first admin account.
+
+## TeX Live
+
+Compiling a PDF requires building a TeX Live image to handle the compilation inside Docker:
+
+```shell
+docker build texlive -t texlive-full
+```
+
+> [!NOTE]
+> To compile on a macOS host, you may need to override the path to the Docker socket by creating a `.env` file in this directory, containing
+> `DOCKER_SOCKET_PATH=/var/run/docker.sock.raw`
+
+## Development
+
+To avoid running `bin/build && bin/up` after every code change, you can run Overleaf
+Community Edition in _development mode_, where services will automatically update on code changes.
+
+To do this, use the included `bin/dev` script:
+
+```shell
+bin/dev
+```
+
+This will start all services using `nodemon`, which will automatically monitor the code and restart the services as necessary.
+
+To improve performance, you can start only a subset of the services in development mode by providing a space-separated list to the `bin/dev` script:
+
+```shell
+bin/dev [service1] [service2] ... [serviceN]
+```
+
+> [!NOTE]
+> Starting the `web` service in _development mode_ will only update the `web`
+> service when backend code changes. In order to automatically update frontend
+> code as well, make sure to start the `webpack` service in _development mode_
+> as well.
+
+If no services are named, all services will start in development mode.
+
+## Debugging
+
+When run in _development mode_ most services expose a debugging port to which
+you can attach a debugger such as
+[the inspector in Chrome's Dev Tools](chrome://inspect/) or one integrated into
+an IDE. The following table shows the port exposed on the **host machine** for
+each service:
+
+| Service            | Port |
+| ------------------ | ---- |
+| `web`              | 9229 |
+| `clsi`             | 9230 |
+| `chat`             | 9231 |
+| `contacts`         | 9232 |
+| `docstore`         | 9233 |
+| `document-updater` | 9234 |
+| `filestore`        | 9235 |
+| `notifications`    | 9236 |
+| `real-time`        | 9237 |
+| `references`       | 9238 |
+| `history-v1`       | 9239 |
+| `project-history`  | 9240 |
+
+To attach to a service using Chrome's _remote debugging_, go to
+<chrome://inspect/> and make sure _Discover network targets_ is checked. Next
+click _Configure..._ and add an entry `localhost:[service port]` for each of the
+services you want to attach a debugger to.
+
+After adding an entry, the service will show up as a _Remote Target_ that you
+can inspect and debug.

develop/bin/build

@@ -0,0 +1,3 @@
+#!/usr/bin/env bash
+
+docker compose build --pull "$@"

develop/bin/dev

@@ -0,0 +1,3 @@
+#!/usr/bin/env bash
+
+docker-compose -f docker-compose.yml -f docker-compose.dev.yml up --no-deps --detach "$@"

develop/bin/down

@@ -0,0 +1,3 @@
+#!/usr/bin/env bash
+
+docker compose down "$@"

develop/bin/logs

@@ -0,0 +1,9 @@
+#!/usr/bin/env bash
+
+docker compose logs --follow --tail 10 --no-color "$@" \
+  | ggrep --line-buffered --invert-match "global.gc" \
+  | ggrep --line-buffered --invert-match "health.check" \
+  | ggrep --line-buffered --invert-match "slow event loop" \
+  | ggrep --line-buffered --invert-match "process.memoryUsage" \
+  | ggrep --line-buffered --only-matching "[{].*" \
+  | bunyan --output short

develop/bin/shell

@@ -0,0 +1,3 @@
+#!/usr/bin/env bash
+
+docker compose exec -it "$@" /bin/bash

develop/bin/up

@@ -0,0 +1,3 @@
+#!/usr/bin/env bash
+
+docker compose up --detach "$@"

develop/dev.env

@@ -0,0 +1,21 @@
+CHAT_HOST=chat
+CLSI_HOST=clsi
+CONTACTS_HOST=contacts
+DOCSTORE_HOST=docstore
+DOCUMENT_UPDATER_HOST=document-updater
+FILESTORE_HOST=filestore
+GRACEFUL_SHUTDOWN_DELAY_SECONDS=0
+HISTORY_V1_HOST=history-v1
+LISTEN_ADDRESS=0.0.0.0
+MONGO_HOST=mongo
+MONGO_URL=mongodb://mongo/sharelatex?directConnection=true
+NOTIFICATIONS_HOST=notifications
+PROJECT_HISTORY_HOST=project-history
+REALTIME_HOST=real-time
+REDIS_HOST=redis
+REFERENCES_HOST=references
+SESSION_SECRET=foo
+WEBPACK_HOST=webpack
+WEB_API_PASSWORD=overleaf
+WEB_API_USER=overleaf
+WEB_HOST=web

develop/docker-compose.dev.yml

@@ -0,0 +1,150 @@
+services:
+  clsi:
+    command: ["node", "--watch", "app.js"]
+    environment:
+      - NODE_OPTIONS=--inspect=0.0.0.0:9229
+    ports:
+      - "127.0.0.1:9230:9229"
+    volumes:
+      - ../services/clsi/app:/overleaf/services/clsi/app
+      - ../services/clsi/app.js:/overleaf/services/clsi/app.js
+      - ../services/clsi/config:/overleaf/services/clsi/config
+
+  chat:
+    command: ["node", "--watch", "app.js"]
+    environment:
+      - NODE_OPTIONS=--inspect=0.0.0.0:9229
+    ports:
+      - "127.0.0.1:9231:9229"
+    volumes:
+      - ../services/chat/app:/overleaf/services/chat/app
+      - ../services/chat/app.js:/overleaf/services/chat/app.js
+      - ../services/chat/config:/overleaf/services/chat/config
+
+  contacts:
+    command: ["node", "--watch", "app.js"]
+    environment:
+      - NODE_OPTIONS=--inspect=0.0.0.0:9229
+    ports:
+      - "127.0.0.1:9232:9229"
+    volumes:
+      - ../services/contacts/app:/overleaf/services/contacts/app
+      - ../services/contacts/app.js:/overleaf/services/contacts/app.js
+      - ../services/contacts/config:/overleaf/services/contacts/config
+
+  docstore:
+    command: ["node", "--watch", "app.js"]
+    environment:
+      - NODE_OPTIONS=--inspect=0.0.0.0:9229
+    ports:
+      - "127.0.0.1:9233:9229"
+    volumes:
+      - ../services/docstore/app:/overleaf/services/docstore/app
+      - ../services/docstore/app.js:/overleaf/services/docstore/app.js
+      - ../services/docstore/config:/overleaf/services/docstore/config
+
+  document-updater:
+    command: ["node", "--watch", "app.js"]
+    environment:
+      - NODE_OPTIONS=--inspect=0.0.0.0:9229
+    ports:
+      - "127.0.0.1:9234:9229"
+    volumes:
+      - ../services/document-updater/app:/overleaf/services/document-updater/app
+      - ../services/document-updater/app.js:/overleaf/services/document-updater/app.js
+      - ../services/document-updater/config:/overleaf/services/document-updater/config
+
+  filestore:
+    command: ["node", "--watch", "app.js"]
+    environment:
+      - NODE_OPTIONS=--inspect=0.0.0.0:9229
+    ports:
+      - "127.0.0.1:9235:9229"
+    volumes:
+      - ../services/filestore/app:/overleaf/services/filestore/app
+      - ../services/filestore/app.js:/overleaf/services/filestore/app.js
+      - ../services/filestore/config:/overleaf/services/filestore/config
+
+  history-v1:
+    command: ["node", "--watch", "app.js"]
+    environment:
+      - NODE_OPTIONS=--inspect=0.0.0.0:9229
+    ports:
+      - "127.0.0.1:9239:9229"
+    volumes:
+      - ../services/history-v1/api:/overleaf/services/history-v1/api
+      - ../services/history-v1/app.js:/overleaf/services/history-v1/app.js
+      - ../services/history-v1/config:/overleaf/services/history-v1/config
+      - ../services/history-v1/storage:/overleaf/services/history-v1/storage
+      - ../services/history-v1/knexfile.js:/overleaf/services/history-v1/knexfile.js
+      - ../services/history-v1/migrations:/overleaf/services/history-v1/migrations
+
+  notifications:
+    command: ["node", "--watch", "app.js"]
+    environment:
+      - NODE_OPTIONS=--inspect=0.0.0.0:9229
+    ports:
+      - "127.0.0.1:9236:9229"
+    volumes:
+      - ../services/notifications/app:/overleaf/services/notifications/app
+      - ../services/notifications/app.js:/overleaf/services/notifications/app.js
+      - ../services/notifications/config:/overleaf/services/notifications/config
+
+  project-history:
+    command: ["node", "--watch", "app.js"]
+    environment:
+      - NODE_OPTIONS=--inspect=0.0.0.0:9229
+    ports:
+      - "127.0.0.1:9240:9229"
+    volumes:
+      - ../services/project-history/app:/overleaf/services/project-history/app
+      - ../services/project-history/app.js:/overleaf/services/project-history/app.js
+      - ../services/project-history/config:/overleaf/services/project-history/config
+
+  real-time:
+    command: ["node", "--watch", "app.js"]
+    environment:
+      - NODE_OPTIONS=--inspect=0.0.0.0:9229
+    ports:
+      - "127.0.0.1:9237:9229"
+    volumes:
+      - ../services/real-time/app:/overleaf/services/real-time/app
+      - ../services/real-time/app.js:/overleaf/services/real-time/app.js
+      - ../services/real-time/config:/overleaf/services/real-time/config
+
+  references:
+    command: ["node", "--watch", "app.js"]
+    environment:
+      - NODE_OPTIONS=--inspect=0.0.0.0:9229
+    ports:
+      - "127.0.0.1:9238:9229"
+    volumes:
+      - ../services/references/app:/overleaf/services/references/app
+      - ../services/references/config:/overleaf/services/references/config
+      - ../services/references/app.js:/overleaf/services/references/app.js
+
+  web:
+    command: ["node", "--watch", "app.js", "--watch-locales"]
+    environment:
+      - NODE_OPTIONS=--inspect=0.0.0.0:9229
+    ports:
+      - "127.0.0.1:9229:9229"
+    volumes:
+      - ../services/web/app:/overleaf/services/web/app
+      - ../services/web/app.mjs:/overleaf/services/web/app.mjs
+      - ../services/web/config:/overleaf/services/web/config
+      - ../services/web/locales:/overleaf/services/web/locales
+      - ../services/web/modules:/overleaf/services/web/modules
+      - ../services/web/public:/overleaf/services/web/public
+
+  webpack:
+    volumes:
+      - ../services/web/app:/overleaf/services/web/app
+      - ../services/web/config:/overleaf/services/web/config
+      - ../services/web/frontend:/overleaf/services/web/frontend
+      - ../services/web/locales:/overleaf/services/web/locales
+      - ../services/web/modules:/overleaf/services/web/modules
+      - ../services/web/public:/overleaf/services/web/public
+      - ../services/web/transform:/overleaf/services/web/transform
+      - ../services/web/types:/overleaf/services/web/types
+      - ../services/web/webpack-plugins:/overleaf/services/web/webpack-plugins

develop/docker-compose.yml

@@ -0,0 +1,182 @@
+volumes:
+  clsi-cache:
+  clsi-output:
+  filestore-public-files:
+  filestore-template-files:
+  filestore-uploads:
+  filestore-user-files:
+  mongo-data:
+  redis-data:
+  sharelatex-data:
+  web-data:
+  history-v1-buckets:
+
+services:
+  chat:
+    build:
+      context: ..
+      dockerfile: services/chat/Dockerfile
+    env_file:
+      - dev.env
+
+  clsi:
+    build:
+      context: ..
+      dockerfile: services/clsi/Dockerfile
+    env_file:
+      - dev.env
+    environment:
+      - DOCKER_RUNNER=true
+      - TEXLIVE_IMAGE=texlive-full # docker build texlive -t texlive-full
+      - COMPILES_HOST_DIR=${PWD}/compiles
+    user: root
+    volumes:
+      - ${PWD}/compiles:/overleaf/services/clsi/compiles
+      - ${DOCKER_SOCKET_PATH:-/var/run/docker.sock}:/var/run/docker.sock
+      - clsi-cache:/overleaf/services/clsi/cache
+      - clsi-output:/overleaf/services/clsi/output
+
+  contacts:
+    build:
+      context: ..
+      dockerfile: services/contacts/Dockerfile
+    env_file:
+      - dev.env
+
+  docstore:
+    build:
+      context: ..
+      dockerfile: services/docstore/Dockerfile
+    env_file:
+      - dev.env
+
+  document-updater:
+    build:
+      context: ..
+      dockerfile: services/document-updater/Dockerfile
+    env_file:
+      - dev.env
+
+  filestore:
+    build:
+      context: ..
+      dockerfile: services/filestore/Dockerfile
+    env_file:
+      - dev.env
+#    environment:
+#      - ENABLE_CONVERSIONS=true
+    volumes:
+      - filestore-public-files:/overleaf/services/filestore/public_files
+      - filestore-template-files:/overleaf/services/filestore/template_files
+      - filestore-uploads:/overleaf/services/filestore/uploads
+      - filestore-user-files:/overleaf/services/filestore/user_files
+
+  history-v1:
+    build:
+      context: ..
+      dockerfile: services/history-v1/Dockerfile
+    env_file:
+        - dev.env
+    environment:
+      OVERLEAF_EDITOR_ANALYTICS_BUCKET: "/buckets/analytics"
+      OVERLEAF_EDITOR_BLOBS_BUCKET: "/buckets/blobs"
+      OVERLEAF_EDITOR_CHUNKS_BUCKET: "/buckets/chunks"
+      OVERLEAF_EDITOR_PROJECT_BLOBS_BUCKET: "/buckets/project_blobs"
+      OVERLEAF_EDITOR_ZIPS_BUCKET: "/buckets/zips"
+      PERSISTOR_BACKEND: fs
+    volumes:
+        - history-v1-buckets:/buckets
+
+  mongo:
+    image: mongo:6.0
+    command: --replSet overleaf
+    ports:
+      - "127.0.0.1:27017:27017" # for debugging
+    volumes:
+      - mongo-data:/data/db
+      - ../bin/shared/mongodb-init-replica-set.js:/docker-entrypoint-initdb.d/mongodb-init-replica-set.js
+    environment:
+      MONGO_INITDB_DATABASE: sharelatex
+    extra_hosts:
+      # Required when using the automatic database setup for initializing the
+      # replica set. This override is not needed when running the setup after
+      # starting up mongo.
+      - mongo:127.0.0.1
+
+  notifications:
+    build:
+      context: ..
+      dockerfile: services/notifications/Dockerfile
+    env_file:
+      - dev.env
+
+  project-history:
+    build:
+      context: ..
+      dockerfile: services/project-history/Dockerfile
+    env_file:
+        - dev.env
+
+  real-time:
+    build:
+      context: ..
+      dockerfile: services/real-time/Dockerfile
+    env_file:
+      - dev.env
+ 
+  redis:
+    image: redis:5
+    ports:
+      - "127.0.0.1:6379:6379" # for debugging
+    volumes:
+      - redis-data:/data
+
+  references:
+    build:
+      context: ..
+      dockerfile: services/references/Dockerfile
+    env_file:
+      - dev.env
+
+  web:
+    build:
+      context: ..
+      dockerfile: services/web/Dockerfile
+      target: dev
+    env_file:
+      - dev.env
+    environment:
+      - APP_NAME=Overleaf Community Edition
+      - ENABLED_LINKED_FILE_TYPES=project_file,project_output_file,url
+      - EMAIL_CONFIRMATION_DISABLED=true
+      - NODE_ENV=development
+      - OVERLEAF_ALLOW_PUBLIC_ACCESS=true
+    command: ["node", "app.mjs"]
+    volumes:
+      - sharelatex-data:/var/lib/overleaf
+      - web-data:/overleaf/services/web/data
+    depends_on:
+      - mongo
+      - redis
+      - chat
+      - clsi
+      - contacts
+      - docstore
+      - document-updater
+      - filestore
+      - history-v1
+      - notifications
+      - project-history
+      - real-time
+      - references
+
+  webpack:
+    build:
+      context: ..
+      dockerfile: services/web/Dockerfile
+      target: webpack
+    command: ["npx", "webpack", "serve", "--config", "webpack.config.dev-env.js"]
+    ports:
+      - "127.0.0.1:80:3808"
+    volumes:
+      - ./webpack.config.dev-env.js:/overleaf/services/web/webpack.config.dev-env.js

develop/texlive/Dockerfile

@@ -0,0 +1,8 @@
+FROM debian:testing-slim
+
+RUN apt-get update
+RUN apt-cache depends texlive-full | grep "Depends: " | grep -v -- "-doc" | grep -v -- "-lang-" | sed 's/Depends: //' | xargs apt-get install -y --no-install-recommends
+RUN apt-get install -y --no-install-recommends fontconfig inkscape pandoc python3-pygments
+
+RUN useradd tex
+USER tex

develop/webpack.config.dev-env.js

@@ -0,0 +1,23 @@
+const { merge } = require('webpack-merge')
+
+const base = require('./webpack.config.dev')
+
+module.exports = merge(base, {
+  devServer: {
+    allowedHosts: 'auto',
+    devMiddleware: {
+      index: false,
+    },
+    proxy: [
+      {
+        context: '/socket.io/**',
+        target: 'http://real-time:3026',
+        ws: true,
+      },
+      {
+        context: ['!**/*.js', '!**/*.css', '!**/*.json'],
+        target: 'http://web:3000',
+      },
+    ],
+  },
+})

docker-compose.debug.yml

@@ -0,0 +1,23 @@
+version: '2.2'
+services:
+    sharelatex:
+        ports:
+            - 30000:30000
+            - 30150:30150
+            - 30120:30120
+            - 30050:30050
+            - 30420:30420
+            - 30030:30030
+            - 30160:30160
+            - 30360:30360
+            - 30130:30130
+            - 30100:30100
+            - 30540:30540
+            - 30640:30640
+            - 40000:40000
+
+            # Server Pro
+            - 30070:30070
+            - 30400:30400
+        environment:
+            DEBUG_NODE: 'true'

docker-compose.yml

@@ -0,0 +1,144 @@
+version: '2.2'
+services:
+    sharelatex:
+        restart: always
+        # Server Pro users:
+        # image: quay.io/sharelatex/sharelatex-pro
+        image: sharelatex/sharelatex
+        container_name: sharelatex
+        depends_on:
+            mongo:
+                condition: service_healthy
+            redis:
+                condition: service_started
+        ports:
+            - 80:80
+        stop_grace_period: 60s
+        volumes:
+            - ~/sharelatex_data:/var/lib/overleaf
+            ########################################################################
+            ####  Server Pro: Uncomment the following line to mount the docker  ####
+            ####             socket, required for Sibling Containers to work    ####
+            ########################################################################
+            # - /var/run/docker.sock:/var/run/docker.sock
+        environment:
+
+            OVERLEAF_APP_NAME: Overleaf Community Edition
+
+            OVERLEAF_MONGO_URL: mongodb://mongo/sharelatex
+
+            # Same property, unfortunately with different names in
+            # different locations
+            OVERLEAF_REDIS_HOST: redis
+            REDIS_HOST: redis
+
+            ENABLED_LINKED_FILE_TYPES: 'project_file,project_output_file,url'
+
+            # Enables Thumbnail generation using ImageMagick
+            ENABLE_CONVERSIONS: 'true'
+
+            # Disables email confirmation requirement
+            EMAIL_CONFIRMATION_DISABLED: 'true'
+
+            ## Set for SSL via nginx-proxy
+            #VIRTUAL_HOST: 103.112.212.22
+
+            # OVERLEAF_SITE_URL: http://overleaf.example.com
+            # OVERLEAF_NAV_TITLE: Overleaf Community Edition
+            # OVERLEAF_HEADER_IMAGE_URL: http://example.com/mylogo.png
+            # OVERLEAF_ADMIN_EMAIL: support@it.com
+
+            # OVERLEAF_LEFT_FOOTER: '[{"text": "Another page I want to link to can be found <a href=\"here\">here</a>"} ]'
+            # OVERLEAF_RIGHT_FOOTER: '[{"text": "Hello I am on the Right"} ]'
+
+            # OVERLEAF_EMAIL_FROM_ADDRESS: "hello@example.com"
+
+            # OVERLEAF_EMAIL_AWS_SES_ACCESS_KEY_ID:
+            # OVERLEAF_EMAIL_AWS_SES_SECRET_KEY:
+
+            # OVERLEAF_EMAIL_SMTP_HOST: smtp.example.com
+            # OVERLEAF_EMAIL_SMTP_PORT: 587
+            # OVERLEAF_EMAIL_SMTP_SECURE: false
+            # OVERLEAF_EMAIL_SMTP_USER:
+            # OVERLEAF_EMAIL_SMTP_PASS:
+            # OVERLEAF_EMAIL_SMTP_TLS_REJECT_UNAUTH: true
+            # OVERLEAF_EMAIL_SMTP_IGNORE_TLS: false
+            # OVERLEAF_EMAIL_SMTP_NAME: '127.0.0.1'
+            # OVERLEAF_EMAIL_SMTP_LOGGER: true
+            # OVERLEAF_CUSTOM_EMAIL_FOOTER: "This system is run by department x"
+
+            # ENABLE_CRON_RESOURCE_DELETION: true
+
+            ################
+            ## Server Pro ##
+            ################
+
+            ## Sandboxed Compiles: https://github.com/overleaf/overleaf/wiki/Server-Pro:-Sandboxed-Compiles
+            SANDBOXED_COMPILES: 'true'
+            SANDBOXED_COMPILES_SIBLING_CONTAINERS: 'true'
+            ### Bind-mount source for /var/lib/overleaf/data/compiles inside the container.
+            SANDBOXED_COMPILES_HOST_DIR: '/home/user/sharelatex_data/data/compiles'
+
+            ## Works with test LDAP server shown at bottom of docker compose
+            # OVERLEAF_LDAP_URL: 'ldap://ldap:389'
+            # OVERLEAF_LDAP_SEARCH_BASE: 'ou=people,dc=planetexpress,dc=com'
+            # OVERLEAF_LDAP_SEARCH_FILTER: '(uid={{username}})'
+            # OVERLEAF_LDAP_BIND_DN: 'cn=admin,dc=planetexpress,dc=com'
+            # OVERLEAF_LDAP_BIND_CREDENTIALS: 'GoodNewsEveryone'
+            # OVERLEAF_LDAP_EMAIL_ATT: 'mail'
+            # OVERLEAF_LDAP_NAME_ATT: 'cn'
+            # OVERLEAF_LDAP_LAST_NAME_ATT: 'sn'
+            # OVERLEAF_LDAP_UPDATE_USER_DETAILS_ON_LOGIN: 'true'
+
+            # OVERLEAF_TEMPLATES_USER_ID: "578773160210479700917ee5"
+            # OVERLEAF_NEW_PROJECT_TEMPLATE_LINKS: '[ {"name":"All Templates","url":"/templates/all"}]'
+
+
+            # OVERLEAF_PROXY_LEARN: "true"
+
+    mongo:
+        restart: always
+        image: mongo:6.0
+        container_name: mongo
+        command: '--replSet overleaf'
+        volumes:
+            - ~/mongo_data:/data/db
+            - ./bin/shared/mongodb-init-replica-set.js:/docker-entrypoint-initdb.d/mongodb-init-replica-set.js
+        environment:
+          MONGO_INITDB_DATABASE: sharelatex
+        extra_hosts:
+          # Required when using the automatic database setup for initializing the replica set.
+          # This override is not needed when running the setup after starting up mongo.
+          - mongo:127.0.0.1
+        healthcheck:
+            test: echo 'db.stats().ok' | mongosh localhost:27017/test --quiet
+            interval: 10s
+            timeout: 10s
+            retries: 5
+
+    redis:
+        restart: always
+        image: redis:6.2
+        container_name: redis
+        volumes:
+            - ~/redis_data:/data
+
+    # ldap:
+    #    restart: always
+    #    image: rroemhild/test-openldap
+    #    container_name: ldap
+
+    # See https://github.com/jwilder/nginx-proxy for documentation on how to configure the nginx-proxy container,
+    # and https://github.com/overleaf/overleaf/wiki/HTTPS-reverse-proxy-using-Nginx for an example of some recommended
+    # settings. We recommend using a properly managed nginx instance outside of the Overleaf Server Pro setup,
+    # but the example here can be used if you'd prefer to run everything with docker-compose
+
+    # nginx-proxy:
+    #     image: jwilder/nginx-proxy
+    #     container_name: nginx-proxy
+    #     ports:
+    #       - "80:80"
+    #       - "443:443"
+    #     volumes:
+    #       - /var/run/docker.sock:/tmp/docker.sock:ro
+    #       - /home/overleaf/tmp:/etc/nginx/certs

install_unibremen/docker/compose_cep/.env

@@ -1,23 +0,0 @@
-# Keycloak OpenID Connect Configuration
-OVERLEAF_OIDC_ISSUER=https://sso.fb1.uni-bremen.de/sso/realms/master
-OVERLEAF_OIDC_CLIENT_ID=overleaf_test
-OVERLEAF_OIDC_CLIENT_SECRET=REDACTED
-OVERLEAF_OIDC_IDENTITY_SERVICE_NAME='Log in with SSO FB1 Uni Bremen'
-
-
-# eMail Account Configuration
-OVERLEAF_EMAIL_PASSWORD=REDACTED
-OVERLEAF_EMAIL_FROM_ADDRESS=overleaf@uni-bremen.de
-OVERLEAF_EMAIL_SMTP_HOST=smtp.uni-bremen.de
-OVERLEAF_EMAIL_SMTP_PORT=465
-OVERLEAF_EMAIL_SMTP_SECURE=true
-OVERLEAF_EMAIL_SMTP_USER=overleaf
-
-# Other Overleaf Configurations Configuration
-OVERLEAF_SITE_URL=https://psintern.neuro.uni-bremen.de
-OVERLEAF_ADMIN_EMAIL=overleaf@uni-bremen.de
-OVERLEAF_APP_NAME="University of Bremen -- Overleaf"
-OVERLEAF_NAV_TITLE="University of Bremen -- Overleaf"
-OVERLEAF_CUSTOM_EMAIL_FOOTER="University of Bremen -- Overleaf"
-
-

install_unibremen/docker/compose_cep/overleafserver/compose.yaml

@@ -1,7 +1,6 @@
 services:
   overleafserver:
-    image: sharelatex/sharelatex:HEAD
+    image: sharelatex/sharelatex:5.4.0.2
-# 5.4.0.2
     container_name: overleafserver
     hostname: overleafserver
     restart: always

install_unibremen/services/web/frontend/js/features/settings/components/account-info-section.tsx

@@ -104,24 +104,6 @@ function AccountInfoSection() {
           canEdit={canUpdateNames}
           required={false}
         />
-        <ReadOrWriteFormGroup
-          id="ssh-public-key-input"
-          type="text"
-          label={"SSH Public Key"}
-          value={sshkeys.Public}
-          handleChange={() => {}}
-          canEdit={false}
-          required={false}
-        />
-        <ReadOrWriteFormGroup
-          id="ssh-private-key-input"
-          type="text"
-          label={"SSH Private Key"}
-          value={sshkeys.Private}
-          handleChange={() => {}}
-          canEdit={false}
-          required={false}
-        />
         {isSuccess ? (
           <OLFormGroup>
             <OLNotification
@@ -152,6 +134,30 @@ function AccountInfoSection() {
             </OLButton>
           </OLFormGroup>
         ) : null}
+        {sshkeys?.Public ? (
+         <OLFormGroup controlId="ssh-public-key-input">
+            <OLFormLabel>{"SSH Public Key"}</OLFormLabel>
+            <textarea
+              className="form-control"
+              readOnly
+              value={sshkeys.Public}
+              rows={4}
+              style={{ fontFamily: 'monospace', fontSize: '12px' }}
+            />
+          </OLFormGroup>
+        ) : null}
+        {sshkeys?.Private ? (
+         <OLFormGroup controlId="ssh-private-key-input">
+            <OLFormLabel>{"SSH Private Key"}</OLFormLabel>
+            <textarea
+              className="form-control"
+              readOnly
+              value={sshkeys.Private}
+              rows={6}
+              style={{ fontFamily: 'monospace', fontSize: '12px' }}
+            />
+          </OLFormGroup>
+        ) : null}
       </form>
     </>
   )

libraries/access-token-encryptor/.gitignore

@@ -0,0 +1,46 @@
+compileFolder
+
+Compiled source #
+###################
+*.com
+*.class
+*.dll
+*.exe
+*.o
+*.so
+
+# Packages #
+############
+# it's better to unpack these files and commit the raw source
+# git has its own built in compression methods
+*.7z
+*.dmg
+*.gz
+*.iso
+*.jar
+*.rar
+*.tar
+*.zip
+
+# Logs and databases #
+######################
+*.log
+*.sql
+*.sqlite
+
+# OS generated files #
+######################
+.DS_Store?
+ehthumbs.db
+Icon?
+Thumbs.db
+
+/node_modules/*
+data/*/*
+
+**.swp
+
+/log.json
+hash_folder
+
+.npmrc

libraries/access-token-encryptor/.mocharc.json

@@ -0,0 +1,6 @@
+{
+  "ui": "bdd",
+  "recursive": "true",
+  "reporter": "spec",
+  "require": "test/setup.js"
+}

libraries/access-token-encryptor/.nvmrc

@@ -0,0 +1 @@
+20.18.2

libraries/access-token-encryptor/LICENSE

@@ -0,0 +1,661 @@
+                    GNU AFFERO GENERAL PUBLIC LICENSE
+                       Version 3, 19 November 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+                            Preamble
+
+  The GNU Affero General Public License is a free, copyleft license for
+software and other kinds of works, specifically designed to ensure
+cooperation with the community in the case of network server software.
+
+  The licenses for most software and other practical works are designed
+to take away your freedom to share and change the works.  By contrast,
+our General Public Licenses are intended to guarantee your freedom to
+share and change all versions of a program--to make sure it remains free
+software for all its users.
+
+  When we speak of free software, we are referring to freedom, not
+price.  Our General Public Licenses are designed to make sure that you
+have the freedom to distribute copies of free software (and charge for
+them if you wish), that you receive source code or can get it if you
+want it, that you can change the software or use pieces of it in new
+free programs, and that you know you can do these things.
+
+  Developers that use our General Public Licenses protect your rights
+with two steps: (1) assert copyright on the software, and (2) offer
+you this License which gives you legal permission to copy, distribute
+and/or modify the software.
+
+  A secondary benefit of defending all users' freedom is that
+improvements made in alternate versions of the program, if they
+receive widespread use, become available for other developers to
+incorporate.  Many developers of free software are heartened and
+encouraged by the resulting cooperation.  However, in the case of
+software used on network servers, this result may fail to come about.
+The GNU General Public License permits making a modified version and
+letting the public access it on a server without ever releasing its
+source code to the public.
+
+  The GNU Affero General Public License is designed specifically to
+ensure that, in such cases, the modified source code becomes available
+to the community.  It requires the operator of a network server to
+provide the source code of the modified version running there to the
+users of that server.  Therefore, public use of a modified version, on
+a publicly accessible server, gives the public access to the source
+code of the modified version.
+
+  An older license, called the Affero General Public License and
+published by Affero, was designed to accomplish similar goals.  This is
+a different license, not a version of the Affero GPL, but Affero has
+released a new version of the Affero GPL which permits relicensing under
+this license.
+
+  The precise terms and conditions for copying, distribution and
+modification follow.
+
+                       TERMS AND CONDITIONS
+
+  0. Definitions.
+
+  "This License" refers to version 3 of the GNU Affero General Public License.
+
+  "Copyright" also means copyright-like laws that apply to other kinds of
+works, such as semiconductor masks.
+
+  "The Program" refers to any copyrightable work licensed under this
+License.  Each licensee is addressed as "you".  "Licensees" and
+"recipients" may be individuals or organizations.
+
+  To "modify" a work means to copy from or adapt all or part of the work
+in a fashion requiring copyright permission, other than the making of an
+exact copy.  The resulting work is called a "modified version" of the
+earlier work or a work "based on" the earlier work.
+
+  A "covered work" means either the unmodified Program or a work based
+on the Program.
+
+  To "propagate" a work means to do anything with it that, without
+permission, would make you directly or secondarily liable for
+infringement under applicable copyright law, except executing it on a
+computer or modifying a private copy.  Propagation includes copying,
+distribution (with or without modification), making available to the
+public, and in some countries other activities as well.
+
+  To "convey" a work means any kind of propagation that enables other
+parties to make or receive copies.  Mere interaction with a user through
+a computer network, with no transfer of a copy, is not conveying.
+
+  An interactive user interface displays "Appropriate Legal Notices"
+to the extent that it includes a convenient and prominently visible
+feature that (1) displays an appropriate copyright notice, and (2)
+tells the user that there is no warranty for the work (except to the
+extent that warranties are provided), that licensees may convey the
+work under this License, and how to view a copy of this License.  If
+the interface presents a list of user commands or options, such as a
+menu, a prominent item in the list meets this criterion.
+
+  1. Source Code.
+
+  The "source code" for a work means the preferred form of the work
+for making modifications to it.  "Object code" means any non-source
+form of a work.
+
+  A "Standard Interface" means an interface that either is an official
+standard defined by a recognized standards body, or, in the case of
+interfaces specified for a particular programming language, one that
+is widely used among developers working in that language.
+
+  The "System Libraries" of an executable work include anything, other
+than the work as a whole, that (a) is included in the normal form of
+packaging a Major Component, but which is not part of that Major
+Component, and (b) serves only to enable use of the work with that
+Major Component, or to implement a Standard Interface for which an
+implementation is available to the public in source code form.  A
+"Major Component", in this context, means a major essential component
+(kernel, window system, and so on) of the specific operating system
+(if any) on which the executable work runs, or a compiler used to
+produce the work, or an object code interpreter used to run it.
+
+  The "Corresponding Source" for a work in object code form means all
+the source code needed to generate, install, and (for an executable
+work) run the object code and to modify the work, including scripts to
+control those activities.  However, it does not include the work's
+System Libraries, or general-purpose tools or generally available free
+programs which are used unmodified in performing those activities but
+which are not part of the work.  For example, Corresponding Source
+includes interface definition files associated with source files for
+the work, and the source code for shared libraries and dynamically
+linked subprograms that the work is specifically designed to require,
+such as by intimate data communication or control flow between those
+subprograms and other parts of the work.
+
+  The Corresponding Source need not include anything that users
+can regenerate automatically from other parts of the Corresponding
+Source.
+
+  The Corresponding Source for a work in source code form is that
+same work.
+
+  2. Basic Permissions.
+
+  All rights granted under this License are granted for the term of
+copyright on the Program, and are irrevocable provided the stated
+conditions are met.  This License explicitly affirms your unlimited
+permission to run the unmodified Program.  The output from running a
+covered work is covered by this License only if the output, given its
+content, constitutes a covered work.  This License acknowledges your
+rights of fair use or other equivalent, as provided by copyright law.
+
+  You may make, run and propagate covered works that you do not
+convey, without conditions so long as your license otherwise remains
+in force.  You may convey covered works to others for the sole purpose
+of having them make modifications exclusively for you, or provide you
+with facilities for running those works, provided that you comply with
+the terms of this License in conveying all material for which you do
+not control copyright.  Those thus making or running the covered works
+for you must do so exclusively on your behalf, under your direction
+and control, on terms that prohibit them from making any copies of
+your copyrighted material outside their relationship with you.
+
+  Conveying under any other circumstances is permitted solely under
+the conditions stated below.  Sublicensing is not allowed; section 10
+makes it unnecessary.
+
+  3. Protecting Users' Legal Rights From Anti-Circumvention Law.
+
+  No covered work shall be deemed part of an effective technological
+measure under any applicable law fulfilling obligations under article
+11 of the WIPO copyright treaty adopted on 20 December 1996, or
+similar laws prohibiting or restricting circumvention of such
+measures.
+
+  When you convey a covered work, you waive any legal power to forbid
+circumvention of technological measures to the extent such circumvention
+is effected by exercising rights under this License with respect to
+the covered work, and you disclaim any intention to limit operation or
+modification of the work as a means of enforcing, against the work's
+users, your or third parties' legal rights to forbid circumvention of
+technological measures.
+
+  4. Conveying Verbatim Copies.
+
+  You may convey verbatim copies of the Program's source code as you
+receive it, in any medium, provided that you conspicuously and
+appropriately publish on each copy an appropriate copyright notice;
+keep intact all notices stating that this License and any
+non-permissive terms added in accord with section 7 apply to the code;
+keep intact all notices of the absence of any warranty; and give all
+recipients a copy of this License along with the Program.
+
+  You may charge any price or no price for each copy that you convey,
+and you may offer support or warranty protection for a fee.
+
+  5. Conveying Modified Source Versions.
+
+  You may convey a work based on the Program, or the modifications to
+produce it from the Program, in the form of source code under the
+terms of section 4, provided that you also meet all of these conditions:
+
+    a) The work must carry prominent notices stating that you modified
+    it, and giving a relevant date.
+
+    b) The work must carry prominent notices stating that it is
+    released under this License and any conditions added under section
+    7.  This requirement modifies the requirement in section 4 to
+    "keep intact all notices".
+
+    c) You must license the entire work, as a whole, under this
+    License to anyone who comes into possession of a copy.  This
+    License will therefore apply, along with any applicable section 7
+    additional terms, to the whole of the work, and all its parts,
+    regardless of how they are packaged.  This License gives no
+    permission to license the work in any other way, but it does not
+    invalidate such permission if you have separately received it.
+
+    d) If the work has interactive user interfaces, each must display
+    Appropriate Legal Notices; however, if the Program has interactive
+    interfaces that do not display Appropriate Legal Notices, your
+    work need not make them do so.
+
+  A compilation of a covered work with other separate and independent
+works, which are not by their nature extensions of the covered work,
+and which are not combined with it such as to form a larger program,
+in or on a volume of a storage or distribution medium, is called an
+"aggregate" if the compilation and its resulting copyright are not
+used to limit the access or legal rights of the compilation's users
+beyond what the individual works permit.  Inclusion of a covered work
+in an aggregate does not cause this License to apply to the other
+parts of the aggregate.
+
+  6. Conveying Non-Source Forms.
+
+  You may convey a covered work in object code form under the terms
+of sections 4 and 5, provided that you also convey the
+machine-readable Corresponding Source under the terms of this License,
+in one of these ways:
+
+    a) Convey the object code in, or embodied in, a physical product
+    (including a physical distribution medium), accompanied by the
+    Corresponding Source fixed on a durable physical medium
+    customarily used for software interchange.
+
+    b) Convey the object code in, or embodied in, a physical product
+    (including a physical distribution medium), accompanied by a
+    written offer, valid for at least three years and valid for as
+    long as you offer spare parts or customer support for that product
+    model, to give anyone who possesses the object code either (1) a
+    copy of the Corresponding Source for all the software in the
+    product that is covered by this License, on a durable physical
+    medium customarily used for software interchange, for a price no
+    more than your reasonable cost of physically performing this
+    conveying of source, or (2) access to copy the
+    Corresponding Source from a network server at no charge.
+
+    c) Convey individual copies of the object code with a copy of the
+    written offer to provide the Corresponding Source.  This
+    alternative is allowed only occasionally and noncommercially, and
+    only if you received the object code with such an offer, in accord
+    with subsection 6b.
+
+    d) Convey the object code by offering access from a designated
+    place (gratis or for a charge), and offer equivalent access to the
+    Corresponding Source in the same way through the same place at no
+    further charge.  You need not require recipients to copy the
+    Corresponding Source along with the object code.  If the place to
+    copy the object code is a network server, the Corresponding Source
+    may be on a different server (operated by you or a third party)
+    that supports equivalent copying facilities, provided you maintain
+    clear directions next to the object code saying where to find the
+    Corresponding Source.  Regardless of what server hosts the
+    Corresponding Source, you remain obligated to ensure that it is
+    available for as long as needed to satisfy these requirements.
+
+    e) Convey the object code using peer-to-peer transmission, provided
+    you inform other peers where the object code and Corresponding
+    Source of the work are being offered to the general public at no
+    charge under subsection 6d.
+
+  A separable portion of the object code, whose source code is excluded
+from the Corresponding Source as a System Library, need not be
+included in conveying the object code work.
+
+  A "User Product" is either (1) a "consumer product", which means any
+tangible personal property which is normally used for personal, family,
+or household purposes, or (2) anything designed or sold for incorporation
+into a dwelling.  In determining whether a product is a consumer product,
+doubtful cases shall be resolved in favor of coverage.  For a particular
+product received by a particular user, "normally used" refers to a
+typical or common use of that class of product, regardless of the status
+of the particular user or of the way in which the particular user
+actually uses, or expects or is expected to use, the product.  A product
+is a consumer product regardless of whether the product has substantial
+commercial, industrial or non-consumer uses, unless such uses represent
+the only significant mode of use of the product.
+
+  "Installation Information" for a User Product means any methods,
+procedures, authorization keys, or other information required to install
+and execute modified versions of a covered work in that User Product from
+a modified version of its Corresponding Source.  The information must
+suffice to ensure that the continued functioning of the modified object
+code is in no case prevented or interfered with solely because
+modification has been made.
+
+  If you convey an object code work under this section in, or with, or
+specifically for use in, a User Product, and the conveying occurs as
+part of a transaction in which the right of possession and use of the
+User Product is transferred to the recipient in perpetuity or for a
+fixed term (regardless of how the transaction is characterized), the
+Corresponding Source conveyed under this section must be accompanied
+by the Installation Information.  But this requirement does not apply
+if neither you nor any third party retains the ability to install
+modified object code on the User Product (for example, the work has
+been installed in ROM).
+
+  The requirement to provide Installation Information does not include a
+requirement to continue to provide support service, warranty, or updates
+for a work that has been modified or installed by the recipient, or for
+the User Product in which it has been modified or installed.  Access to a
+network may be denied when the modification itself materially and
+adversely affects the operation of the network or violates the rules and
+protocols for communication across the network.
+
+  Corresponding Source conveyed, and Installation Information provided,
+in accord with this section must be in a format that is publicly
+documented (and with an implementation available to the public in
+source code form), and must require no special password or key for
+unpacking, reading or copying.
+
+  7. Additional Terms.
+
+  "Additional permissions" are terms that supplement the terms of this
+License by making exceptions from one or more of its conditions.
+Additional permissions that are applicable to the entire Program shall
+be treated as though they were included in this License, to the extent
+that they are valid under applicable law.  If additional permissions
+apply only to part of the Program, that part may be used separately
+under those permissions, but the entire Program remains governed by
+this License without regard to the additional permissions.
+
+  When you convey a copy of a covered work, you may at your option
+remove any additional permissions from that copy, or from any part of
+it.  (Additional permissions may be written to require their own
+removal in certain cases when you modify the work.)  You may place
+additional permissions on material, added by you to a covered work,
+for which you have or can give appropriate copyright permission.
+
+  Notwithstanding any other provision of this License, for material you
+add to a covered work, you may (if authorized by the copyright holders of
+that material) supplement the terms of this License with terms:
+
+    a) Disclaiming warranty or limiting liability differently from the
+    terms of sections 15 and 16 of this License; or
+
+    b) Requiring preservation of specified reasonable legal notices or
+    author attributions in that material or in the Appropriate Legal
+    Notices displayed by works containing it; or
+
+    c) Prohibiting misrepresentation of the origin of that material, or
+    requiring that modified versions of such material be marked in
+    reasonable ways as different from the original version; or
+
+    d) Limiting the use for publicity purposes of names of licensors or
+    authors of the material; or
+
+    e) Declining to grant rights under trademark law for use of some
+    trade names, trademarks, or service marks; or
+
+    f) Requiring indemnification of licensors and authors of that
+    material by anyone who conveys the material (or modified versions of
+    it) with contractual assumptions of liability to the recipient, for
+    any liability that these contractual assumptions directly impose on
+    those licensors and authors.
+
+  All other non-permissive additional terms are considered "further
+restrictions" within the meaning of section 10.  If the Program as you
+received it, or any part of it, contains a notice stating that it is
+governed by this License along with a term that is a further
+restriction, you may remove that term.  If a license document contains
+a further restriction but permits relicensing or conveying under this
+License, you may add to a covered work material governed by the terms
+of that license document, provided that the further restriction does
+not survive such relicensing or conveying.
+
+  If you add terms to a covered work in accord with this section, you
+must place, in the relevant source files, a statement of the
+additional terms that apply to those files, or a notice indicating
+where to find the applicable terms.
+
+  Additional terms, permissive or non-permissive, may be stated in the
+form of a separately written license, or stated as exceptions;
+the above requirements apply either way.
+
+  8. Termination.
+
+  You may not propagate or modify a covered work except as expressly
+provided under this License.  Any attempt otherwise to propagate or
+modify it is void, and will automatically terminate your rights under
+this License (including any patent licenses granted under the third
+paragraph of section 11).
+
+  However, if you cease all violation of this License, then your
+license from a particular copyright holder is reinstated (a)
+provisionally, unless and until the copyright holder explicitly and
+finally terminates your license, and (b) permanently, if the copyright
+holder fails to notify you of the violation by some reasonable means
+prior to 60 days after the cessation.
+
+  Moreover, your license from a particular copyright holder is
+reinstated permanently if the copyright holder notifies you of the
+violation by some reasonable means, this is the first time you have
+received notice of violation of this License (for any work) from that
+copyright holder, and you cure the violation prior to 30 days after
+your receipt of the notice.
+
+  Termination of your rights under this section does not terminate the
+licenses of parties who have received copies or rights from you under
+this License.  If your rights have been terminated and not permanently
+reinstated, you do not qualify to receive new licenses for the same
+material under section 10.
+
+  9. Acceptance Not Required for Having Copies.
+
+  You are not required to accept this License in order to receive or
+run a copy of the Program.  Ancillary propagation of a covered work
+occurring solely as a consequence of using peer-to-peer transmission
+to receive a copy likewise does not require acceptance.  However,
+nothing other than this License grants you permission to propagate or
+modify any covered work.  These actions infringe copyright if you do
+not accept this License.  Therefore, by modifying or propagating a
+covered work, you indicate your acceptance of this License to do so.
+
+  10. Automatic Licensing of Downstream Recipients.
+
+  Each time you convey a covered work, the recipient automatically
+receives a license from the original licensors, to run, modify and
+propagate that work, subject to this License.  You are not responsible
+for enforcing compliance by third parties with this License.
+
+  An "entity transaction" is a transaction transferring control of an
+organization, or substantially all assets of one, or subdividing an
+organization, or merging organizations.  If propagation of a covered
+work results from an entity transaction, each party to that
+transaction who receives a copy of the work also receives whatever
+licenses to the work the party's predecessor in interest had or could
+give under the previous paragraph, plus a right to possession of the
+Corresponding Source of the work from the predecessor in interest, if
+the predecessor has it or can get it with reasonable efforts.
+
+  You may not impose any further restrictions on the exercise of the
+rights granted or affirmed under this License.  For example, you may
+not impose a license fee, royalty, or other charge for exercise of
+rights granted under this License, and you may not initiate litigation
+(including a cross-claim or counterclaim in a lawsuit) alleging that
+any patent claim is infringed by making, using, selling, offering for
+sale, or importing the Program or any portion of it.
+
+  11. Patents.
+
+  A "contributor" is a copyright holder who authorizes use under this
+License of the Program or a work on which the Program is based.  The
+work thus licensed is called the contributor's "contributor version".
+
+  A contributor's "essential patent claims" are all patent claims
+owned or controlled by the contributor, whether already acquired or
+hereafter acquired, that would be infringed by some manner, permitted
+by this License, of making, using, or selling its contributor version,
+but do not include claims that would be infringed only as a
+consequence of further modification of the contributor version.  For
+purposes of this definition, "control" includes the right to grant
+patent sublicenses in a manner consistent with the requirements of
+this License.
+
+  Each contributor grants you a non-exclusive, worldwide, royalty-free
+patent license under the contributor's essential patent claims, to
+make, use, sell, offer for sale, import and otherwise run, modify and
+propagate the contents of its contributor version.
+
+  In the following three paragraphs, a "patent license" is any express
+agreement or commitment, however denominated, not to enforce a patent
+(such as an express permission to practice a patent or covenant not to
+sue for patent infringement).  To "grant" such a patent license to a
+party means to make such an agreement or commitment not to enforce a
+patent against the party.
+
+  If you convey a covered work, knowingly relying on a patent license,
+and the Corresponding Source of the work is not available for anyone
+to copy, free of charge and under the terms of this License, through a
+publicly available network server or other readily accessible means,
+then you must either (1) cause the Corresponding Source to be so
+available, or (2) arrange to deprive yourself of the benefit of the
+patent license for this particular work, or (3) arrange, in a manner
+consistent with the requirements of this License, to extend the patent
+license to downstream recipients.  "Knowingly relying" means you have
+actual knowledge that, but for the patent license, your conveying the
+covered work in a country, or your recipient's use of the covered work
+in a country, would infringe one or more identifiable patents in that
+country that you have reason to believe are valid.
+
+  If, pursuant to or in connection with a single transaction or
+arrangement, you convey, or propagate by procuring conveyance of, a
+covered work, and grant a patent license to some of the parties
+receiving the covered work authorizing them to use, propagate, modify
+or convey a specific copy of the covered work, then the patent license
+you grant is automatically extended to all recipients of the covered
+work and works based on it.
+
+  A patent license is "discriminatory" if it does not include within
+the scope of its coverage, prohibits the exercise of, or is
+conditioned on the non-exercise of one or more of the rights that are
+specifically granted under this License.  You may not convey a covered
+work if you are a party to an arrangement with a third party that is
+in the business of distributing software, under which you make payment
+to the third party based on the extent of your activity of conveying
+the work, and under which the third party grants, to any of the
+parties who would receive the covered work from you, a discriminatory
+patent license (a) in connection with copies of the covered work
+conveyed by you (or copies made from those copies), or (b) primarily
+for and in connection with specific products or compilations that
+contain the covered work, unless you entered into that arrangement,
+or that patent license was granted, prior to 28 March 2007.
+
+  Nothing in this License shall be construed as excluding or limiting
+any implied license or other defenses to infringement that may
+otherwise be available to you under applicable patent law.
+
+  12. No Surrender of Others' Freedom.
+
+  If conditions are imposed on you (whether by court order, agreement or
+otherwise) that contradict the conditions of this License, they do not
+excuse you from the conditions of this License.  If you cannot convey a
+covered work so as to satisfy simultaneously your obligations under this
+License and any other pertinent obligations, then as a consequence you may
+not convey it at all.  For example, if you agree to terms that obligate you
+to collect a royalty for further conveying from those to whom you convey
+the Program, the only way you could satisfy both those terms and this
+License would be to refrain entirely from conveying the Program.
+
+  13. Remote Network Interaction; Use with the GNU General Public License.
+
+  Notwithstanding any other provision of this License, if you modify the
+Program, your modified version must prominently offer all users
+interacting with it remotely through a computer network (if your version
+supports such interaction) an opportunity to receive the Corresponding
+Source of your version by providing access to the Corresponding Source
+from a network server at no charge, through some standard or customary
+means of facilitating copying of software.  This Corresponding Source
+shall include the Corresponding Source for any work covered by version 3
+of the GNU General Public License that is incorporated pursuant to the
+following paragraph.
+
+  Notwithstanding any other provision of this License, you have
+permission to link or combine any covered work with a work licensed
+under version 3 of the GNU General Public License into a single
+combined work, and to convey the resulting work.  The terms of this
+License will continue to apply to the part which is the covered work,
+but the work with which it is combined will remain governed by version
+3 of the GNU General Public License.
+
+  14. Revised Versions of this License.
+
+  The Free Software Foundation may publish revised and/or new versions of
+the GNU Affero General Public License from time to time.  Such new versions
+will be similar in spirit to the present version, but may differ in detail to
+address new problems or concerns.
+
+  Each version is given a distinguishing version number.  If the
+Program specifies that a certain numbered version of the GNU Affero General
+Public License "or any later version" applies to it, you have the
+option of following the terms and conditions either of that numbered
+version or of any later version published by the Free Software
+Foundation.  If the Program does not specify a version number of the
+GNU Affero General Public License, you may choose any version ever published
+by the Free Software Foundation.
+
+  If the Program specifies that a proxy can decide which future
+versions of the GNU Affero General Public License can be used, that proxy's
+public statement of acceptance of a version permanently authorizes you
+to choose that version for the Program.
+
+  Later license versions may give you additional or different
+permissions.  However, no additional obligations are imposed on any
+author or copyright holder as a result of your choosing to follow a
+later version.
+
+  15. Disclaimer of Warranty.
+
+  THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
+APPLICABLE LAW.  EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
+HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
+OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
+THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+PURPOSE.  THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
+IS WITH YOU.  SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
+ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
+
+  16. Limitation of Liability.
+
+  IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
+WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
+THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
+GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
+USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
+DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
+PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
+EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
+SUCH DAMAGES.
+
+  17. Interpretation of Sections 15 and 16.
+
+  If the disclaimer of warranty and limitation of liability provided
+above cannot be given local legal effect according to their terms,
+reviewing courts shall apply local law that most closely approximates
+an absolute waiver of all civil liability in connection with the
+Program, unless a warranty or assumption of liability accompanies a
+copy of the Program in return for a fee.
+
+                     END OF TERMS AND CONDITIONS
+
+            How to Apply These Terms to Your New Programs
+
+  If you develop a new program, and you want it to be of the greatest
+possible use to the public, the best way to achieve this is to make it
+free software which everyone can redistribute and change under these terms.
+
+  To do so, attach the following notices to the program.  It is safest
+to attach them to the start of each source file to most effectively
+state the exclusion of warranty; and each file should have at least
+the "copyright" line and a pointer to where the full notice is found.
+
+    <one line to give the program's name and a brief idea of what it does.>
+    Copyright (C) <year>  <name of author>
+
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU Affero General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU Affero General Public License for more details.
+
+    You should have received a copy of the GNU Affero General Public License
+    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+Also add information on how to contact you by electronic and paper mail.
+
+  If your software can interact with users remotely through a computer
+network, you should also make sure that it provides a way for users to
+get its source.  For example, if your program is a web application, its
+interface could display a "Source" link that leads users to an archive
+of the code.  There are many ways you could offer source, and different
+solutions will be better for different programs; see section 13 for the
+specific requirements.
+
+  You should also get your employer (if you work as a programmer) or school,
+if any, to sign a "copyright disclaimer" for the program, if necessary.
+For more information on this, and how to apply and follow the GNU AGPL, see
+<http://www.gnu.org/licenses/>.

libraries/access-token-encryptor/README.md

@@ -0,0 +1,3 @@
+# Access Token Encryptor
+
+Used in third-party-references, to encrypt access tokens

libraries/access-token-encryptor/buildscript.txt

@@ -0,0 +1,10 @@
+access-token-encryptor
+--dependencies=None
+--docker-repos=gcr.io/overleaf-ops
+--env-add=
+--env-pass-through=
+--esmock-loader=False
+--is-library=True
+--node-version=20.18.2
+--public-repo=False
+--script-version=4.5.0

libraries/access-token-encryptor/index.js

@@ -0,0 +1 @@
+module.exports = require('./lib/js/AccessTokenEncryptor')

libraries/access-token-encryptor/lib/js/AccessTokenEncryptor.js

@@ -0,0 +1,164 @@
+const { promisify } = require('node:util')
+const crypto = require('node:crypto')
+
+const ALGORITHM = 'aes-256-ctr'
+
+const cryptoHkdf = promisify(crypto.hkdf)
+const cryptoRandomBytes = promisify(crypto.randomBytes)
+
+class AbstractAccessTokenScheme {
+  constructor(cipherLabel, cipherPassword) {
+    this.cipherLabel = cipherLabel
+    this.cipherPassword = cipherPassword
+  }
+
+  /**
+   * @param {Object} json
+   * @return {Promise<string>}
+   */
+  async encryptJson(json) {
+    throw new Error('encryptJson is not implemented')
+  }
+
+  /**
+   * @param {string} encryptedJson
+   * @return {Promise<Object>}
+   */
+  async decryptToJson(encryptedJson) {
+    throw new Error('decryptToJson is not implemented')
+  }
+}
+
+class AccessTokenSchemeWithGenericKeyFn extends AbstractAccessTokenScheme {
+  /**
+   * @param {Buffer} salt
+   * @return {Promise<Buffer>}
+   */
+  async keyFn(salt) {
+    throw new Error('keyFn is not implemented')
+  }
+
+  async encryptJson(json) {
+    const plainText = JSON.stringify(json)
+
+    const bytes = await cryptoRandomBytes(32)
+    const salt = bytes.slice(0, 16)
+    const iv = bytes.slice(16, 32)
+    const key = await this.keyFn(salt)
+
+    const cipher = crypto.createCipheriv(ALGORITHM, key, iv)
+    const cipherText =
+      cipher.update(plainText, 'utf8', 'base64') + cipher.final('base64')
+
+    return [
+      this.cipherLabel,
+      salt.toString('hex'),
+      cipherText,
+      iv.toString('hex'),
+    ].join(':')
+  }
+
+  async decryptToJson(encryptedJson) {
+    const [, salt, cipherText, iv] = encryptedJson.split(':', 4)
+    const key = await this.keyFn(Buffer.from(salt, 'hex'))
+
+    const decipher = crypto.createDecipheriv(
+      ALGORITHM,
+      key,
+      Buffer.from(iv, 'hex')
+    )
+    const plainText =
+      decipher.update(cipherText, 'base64', 'utf8') + decipher.final('utf8')
+    try {
+      return JSON.parse(plainText)
+    } catch (e) {
+      throw new Error('error decrypting token')
+    }
+  }
+}
+
+class AccessTokenSchemeV3 extends AccessTokenSchemeWithGenericKeyFn {
+  async keyFn(salt) {
+    const optionalInfo = ''
+    return await cryptoHkdf(
+      'sha512',
+      this.cipherPassword,
+      salt,
+      optionalInfo,
+      32
+    )
+  }
+}
+
+class AccessTokenEncryptor {
+  constructor(settings) {
+    /**
+     * @type {Map<string, AbstractAccessTokenScheme>}
+     */
+    this.schemeByCipherLabel = new Map()
+    for (const cipherLabel of Object.keys(settings.cipherPasswords)) {
+      if (!cipherLabel) {
+        throw new Error('cipherLabel cannot be empty')
+      }
+      if (cipherLabel.match(/:/)) {
+        throw new Error(
+          `cipherLabel must not contain a colon (:), got ${cipherLabel}`
+        )
+      }
+      const [, version] = cipherLabel.split('-')
+      if (!version) {
+        throw new Error(
+          `cipherLabel must contain version suffix (e.g. 2042.1-v42), got ${cipherLabel}`
+        )
+      }
+
+      const cipherPassword = settings.cipherPasswords[cipherLabel]
+      if (!cipherPassword) {
+        throw new Error(`cipherPasswords['${cipherLabel}'] is missing`)
+      }
+      if (cipherPassword.length < 16) {
+        throw new Error(`cipherPasswords['${cipherLabel}'] is too short`)
+      }
+
+      let scheme
+      switch (version) {
+        case 'v3':
+          scheme = new AccessTokenSchemeV3(cipherLabel, cipherPassword)
+          break
+        default:
+          throw new Error(`unknown version '${version}' for ${cipherLabel}`)
+      }
+      this.schemeByCipherLabel.set(cipherLabel, scheme)
+    }
+
+    /** @type {AbstractAccessTokenScheme} */
+    this.defaultScheme = this.schemeByCipherLabel.get(settings.cipherLabel)
+    if (!this.defaultScheme) {
+      throw new Error(`unknown default cipherLabel ${settings.cipherLabel}`)
+    }
+  }
+
+  promises = {
+    encryptJson: async json => await this.defaultScheme.encryptJson(json),
+    decryptToJson: async encryptedJson => {
+      const [label] = encryptedJson.split(':', 1)
+      const scheme = this.schemeByCipherLabel.get(label)
+      if (!scheme) {
+        throw new Error('unknown access-token-encryptor label ' + label)
+      }
+      return await scheme.decryptToJson(encryptedJson)
+    },
+  }
+
+  encryptJson(json, callback) {
+    this.promises.encryptJson(json).then(s => callback(null, s), callback)
+  }
+
+  decryptToJson(encryptedJson, callback) {
+    this.promises
+      .decryptToJson(encryptedJson)
+      .then(o => callback(null, o), callback)
+  }
+}
+
+module.exports = AccessTokenEncryptor

libraries/access-token-encryptor/package.json

@@ -0,0 +1,28 @@
+{
+  "name": "@overleaf/access-token-encryptor",
+  "version": "3.0.0",
+  "description": "",
+  "main": "index.js",
+  "scripts": {
+    "test": "npm run lint && npm run format && npm run types:check && npm run test:unit",
+    "lint": "eslint --ext .js --ext .cjs --ext .ts --max-warnings 0 --format unix .",
+    "lint:fix": "eslint --fix --ext .js --ext .cjs --ext .ts .",
+    "format": "prettier --list-different $PWD/'**/*.{js,cjs,ts}'",
+    "format:fix": "prettier --write $PWD/'**/*.{js,cjs,ts}'",
+    "test:ci": "npm run test:unit",
+    "test:unit": "mocha --exit test/**/*.{js,cjs}",
+    "types:check": "tsc --noEmit"
+  },
+  "author": "",
+  "license": "AGPL-3.0-only",
+  "dependencies": {
+    "lodash": "^4.17.21"
+  },
+  "devDependencies": {
+    "chai": "^4.3.6",
+    "chai-as-promised": "^7.1.1",
+    "mocha": "^11.1.0",
+    "sandboxed-module": "^2.0.4",
+    "typescript": "^5.0.4"
+  }
+}

libraries/access-token-encryptor/scripts/helpers/format-usage-stats.js

@@ -0,0 +1,27 @@
+function formatTokenUsageStats(STATS) {
+  const prettyStats = []
+  const sortedStats = Object.entries(STATS).sort((a, b) =>
+    a[0] > b[0] ? 1 : -1
+  )
+  const totalByName = {}
+  for (const [key, n] of sortedStats) {
+    const [name, version, collectionName, path, label] = key.split(':')
+    totalByName[name] = (totalByName[name] || 0) + n
+    prettyStats.push({ name, version, collectionName, path, label, n })
+  }
+  for (const row of prettyStats) {
+    row.percentage = ((100 * row.n) / totalByName[row.name])
+      .toFixed(2)
+      .padStart(6)
+  }
+
+  if (prettyStats.length === 0) {
+    console.warn('---')
+    console.warn('Found 0 access tokens.')
+    console.warn('---')
+  } else {
+    console.table(prettyStats)
+  }
+}
+
+module.exports = { formatTokenUsageStats }

libraries/access-token-encryptor/scripts/helpers/re-encrypt-tokens.js

@@ -0,0 +1,108 @@
+const _ = require('lodash')
+const { formatTokenUsageStats } = require('./format-usage-stats')
+
+const LOG_EVERY_IN_S = parseInt(process.env.LOG_EVERY_IN_S || '5', 10)
+const DRY_RUN = !process.argv.includes('--dry-run=false')
+
+/**
+ * @param {AccessTokenEncryptor} accessTokenEncryptor
+ * @param {string} encryptedJson
+ * @return {Promise<string>}
+ */
+async function reEncryptTokens(accessTokenEncryptor, encryptedJson) {
+  return await new Promise((resolve, reject) => {
+    accessTokenEncryptor.decryptToJson(encryptedJson, (err, json) => {
+      if (err) return reject(err)
+      accessTokenEncryptor.encryptJson(json, (err, reEncryptedJson) => {
+        if (err) return reject(err)
+        resolve(reEncryptedJson)
+      })
+    })
+  })
+}
+
+/**
+ * @param {AccessTokenEncryptor} accessTokenEncryptor
+ * @param {Collection} collection
+ * @param {Object} paths
+ * @param {Object} queryOptions
+ * @return {Promise<{}>}
+ */
+async function reEncryptTokensInCollection({
+  accessTokenEncryptor,
+  collection,
+  paths,
+  queryOptions,
+}) {
+  const { collectionName } = collection
+  const stats = {}
+
+  let processed = 0
+  let updatedNUsers = 0
+  let lastLog = 0
+  const logProgress = () => {
+    if (DRY_RUN) {
+      console.warn(
+        `processed ${processed} | Would have updated ${updatedNUsers} users`
+      )
+    } else {
+      console.warn(`processed ${processed} | Updated ${updatedNUsers} users`)
+    }
+  }
+
+  const projection = { _id: 1 }
+  for (const path of Object.values(paths)) {
+    projection[path] = 1
+  }
+  const cursor = collection.find(
+    {},
+    {
+      ...queryOptions,
+      projection,
+    }
+  )
+
+  for await (const doc of cursor) {
+    processed++
+
+    let update = null
+    for (const [name, path] of Object.entries(paths)) {
+      const blob = _.get(doc, path)
+      if (!blob) continue
+      // Schema: LABEL-VERSION:SALT:CIPHERTEXT:IV
+      const [label] = blob.split(':')
+      let [, version] = label.split('-')
+      version = version || 'v2'
+
+      const key = [name, version, collectionName, path, label].join(':')
+      stats[key] = (stats[key] || 0) + 1
+
+      if (version === 'v2') {
+        update = update || {}
+        update[path] = await reEncryptTokens(accessTokenEncryptor, blob)
+      }
+    }
+
+    if (Date.now() - lastLog >= LOG_EVERY_IN_S * 1000) {
+      logProgress()
+      lastLog = Date.now()
+    }
+    if (update) {
+      updatedNUsers++
+
+      const { _id } = doc
+      if (DRY_RUN) {
+        console.log('Would upgrade tokens for user', _id, Object.keys(update))
+      } else {
+        console.log('Upgrading tokens for user', _id, Object.keys(update))
+        await collection.updateOne({ _id }, { $set: update })
+      }
+    }
+  }
+  logProgress()
+  formatTokenUsageStats(stats)
+}
+
+module.exports = {
+  reEncryptTokensInCollection,
+}

libraries/access-token-encryptor/test/setup.js

@@ -0,0 +1,13 @@
+const chai = require('chai')
+const chaiAsPromised = require('chai-as-promised')
+const SandboxedModule = require('sandboxed-module')
+
+chai.use(chaiAsPromised)
+
+SandboxedModule.configure({
+  sourceTransformers: {
+    removeNodePrefix: function (source) {
+      return source.replace(/require\(['"]node:/g, "require('")
+    },
+  },
+})

libraries/access-token-encryptor/test/unit/js/AccessTokenEncryptorTests.js

@@ -0,0 +1,305 @@
+const chai = require('chai')
+chai.should()
+const { expect } = chai
+const modulePath = '../../../index.js'
+const SandboxedModule = require('sandboxed-module')
+
+describe('AccessTokenEncryptor', function () {
+  beforeEach(function () {
+    this.testObject = { hello: 'world' }
+    this.encrypted2015 =
+      '2015.1:473a66fb5d816bc716f278ab819d88a5:+mTg7O9sgUND8pNQFG6h2GE='
+    this.encrypted2016 =
+      '2016.1:76a7d64a444ccee1a515b49c44844a69:m5YSkexUsLjcF4gLncm72+k='
+    this.encrypted2019 =
+      '2019.1:627143b2ab185a020c8720253a4c984e:7gnY6Ez3/Y3UWgLHLfBtJsE=:bf75cecb6aeea55b3c060e1122d2a82d'
+    this.encrypted2023 =
+      '2023.1-v3:a6dd3781dd6ce93a4134874b505a209c:9TdIDAc8V9SeR0ffSn63Jj4=:d8b2de0b733c81b949993dce229abb4c'
+    this.badLabel = 'xxxxxx:c7a39310056b694c:jQf+Uh5Den3JREtvc82GW5Q='
+    this.badKey = '2015.1:d7a39310056b694c:jQf+Uh5Den3JREtvc82GW5Q='
+    this.badCipherText = '2015.1:c7a39310056b694c:xQf+Uh5Den3JREtvc82GW5Q='
+    this.settings = {
+      cipherLabel: '2023.1-v3',
+      cipherPasswords: {
+        '2023.1-v3': '44444444444444444444444444444444444444',
+      },
+    }
+    this.AccessTokenEncryptor = SandboxedModule.require(modulePath, {
+      globals: {
+        Buffer,
+      },
+    })
+    this.encryptor = new this.AccessTokenEncryptor(this.settings)
+  })
+
+  describe('invalid settings', function () {
+    it('should flag missing label', function () {
+      expect(
+        () =>
+          new this.AccessTokenEncryptor({
+            cipherLabel: '',
+            cipherPasswords: { '': '' },
+          })
+      ).to.throw(/cipherLabel cannot be empty/)
+    })
+
+    it('should flag invalid label with colon', function () {
+      expect(
+        () =>
+          new this.AccessTokenEncryptor({
+            cipherLabel: '2023:1-v2',
+            cipherPasswords: { '2023:1-v2': '' },
+          })
+      ).to.throw(/colon/)
+    })
+
+    it('should flag missing password', function () {
+      expect(
+        () =>
+          new this.AccessTokenEncryptor({
+            cipherPasswords: { '2023.1-v3': '' },
+            cipherVersions: { '2023.1-v3': 'v3' },
+          })
+      ).to.throw(/cipherPasswords.+ missing/)
+      expect(
+        () =>
+          new this.AccessTokenEncryptor({
+            cipherLabel: '2023.1-v3',
+            cipherPasswords: { '2023.1-v3': undefined },
+          })
+      ).to.throw(/cipherPasswords.+ missing/)
+    })
+
+    it('should flag short password', function () {
+      expect(
+        () =>
+          new this.AccessTokenEncryptor({
+            cipherLabel: '2023.1-v3',
+            cipherPasswords: { '2023.1-v3': 'foo' },
+          })
+      ).to.throw(/cipherPasswords.+ too short/)
+    })
+
+    it('should flag missing version', function () {
+      expect(
+        () =>
+          new this.AccessTokenEncryptor({
+            cipherLabel: '2023.1',
+            cipherPasswords: { 2023.1: '11111111111111111111111111111111' },
+          })
+      ).to.throw(/must contain version suffix/)
+      expect(
+        () =>
+          new this.AccessTokenEncryptor({
+            cipherLabel: '2023.1-',
+            cipherPasswords: { '2023.1-': '11111111111111111111111111111111' },
+          })
+      ).to.throw(/must contain version suffix/)
+    })
+
+    it('should flag invalid version', function () {
+      expect(
+        () =>
+          new this.AccessTokenEncryptor({
+            cipherLabel: '2023.1-v0',
+            cipherPasswords: {
+              '2023.1-v0': '11111111111111111111111111111111',
+            },
+          })
+      ).to.throw(/unknown version/)
+    })
+
+    it('should flag unknown default scheme', function () {
+      expect(
+        () =>
+          new this.AccessTokenEncryptor({
+            cipherLabel: '2000.1-v3',
+            cipherPasswords: {
+              '2023.1-v3': '11111111111111111111111111111111',
+            },
+          })
+      ).to.throw(/unknown default cipherLabel/)
+    })
+  })
+
+  describe('sync', function () {
+    describe('encrypt', function () {
+      it('should encrypt the object', function (done) {
+        this.encryptor.encryptJson(this.testObject, (err, encrypted) => {
+          expect(err).to.be.null
+          encrypted.should.match(
+            /^2023.1-v3:[0-9a-f]{32}:[a-zA-Z0-9=+/]+:[0-9a-f]{32}$/
+          )
+          done()
+        })
+      })
+
+      it('should encrypt the object differently the next time', function (done) {
+        this.encryptor.encryptJson(this.testObject, (err, encrypted1) => {
+          expect(err).to.be.null
+          this.encryptor.encryptJson(this.testObject, (err, encrypted2) => {
+            expect(err).to.be.null
+            encrypted1.should.not.equal(encrypted2)
+            done()
+          })
+        })
+      })
+    })
+
+    describe('decrypt', function () {
+      it('should decrypt the string to get the same object', function (done) {
+        this.encryptor.encryptJson(this.testObject, (err, encrypted) => {
+          expect(err).to.be.null
+          this.encryptor.decryptToJson(encrypted, (err, decrypted) => {
+            expect(err).to.be.null
+            expect(decrypted).to.deep.equal(this.testObject)
+            done()
+          })
+        })
+      })
+
+      it('should not be able to decrypt 2015 string', function (done) {
+        this.encryptor.decryptToJson(this.encrypted2015, (err, decrypted) => {
+          expect(err).to.exist
+          expect(err.message).to.equal(
+            'unknown access-token-encryptor label 2015.1'
+          )
+          expect(decrypted).to.not.exist
+          done()
+        })
+      })
+
+      it('should not be able to decrypt a 2016 string', function (done) {
+        this.encryptor.decryptToJson(this.encrypted2016, (err, decrypted) => {
+          expect(err).to.exist
+          expect(err.message).to.equal(
+            'unknown access-token-encryptor label 2016.1'
+          )
+          expect(decrypted).to.not.exist
+          done()
+        })
+      })
+
+      it('should not be able to decrypt a 2019 string', function (done) {
+        this.encryptor.decryptToJson(this.encrypted2019, (err, decrypted) => {
+          expect(err).to.exist
+          expect(err.message).to.equal(
+            'unknown access-token-encryptor label 2019.1'
+          )
+          expect(decrypted).to.not.exist
+          done()
+        })
+      })
+
+      it('should decrypt an 2023 string to get the same object', function (done) {
+        this.encryptor.decryptToJson(this.encrypted2023, (err, decrypted) => {
+          expect(err).to.be.null
+          expect(decrypted).to.deep.equal(this.testObject)
+          done()
+        })
+      })
+
+      it('should return an error when decrypting an invalid label', function (done) {
+        this.encryptor.decryptToJson(this.badLabel, (err, decrypted) => {
+          expect(err).to.be.instanceof(Error)
+          expect(decrypted).to.be.undefined
+          done()
+        })
+      })
+
+      it('should return an error when decrypting an invalid key', function (done) {
+        this.encryptor.decryptToJson(this.badKey, (err, decrypted) => {
+          expect(err).to.be.instanceof(Error)
+          expect(decrypted).to.be.undefined
+          done()
+        })
+      })
+
+      it('should return an error when decrypting an invalid ciphertext', function (done) {
+        this.encryptor.decryptToJson(this.badCipherText, (err, decrypted) => {
+          expect(err).to.be.instanceof(Error)
+          expect(decrypted).to.be.undefined
+          done()
+        })
+      })
+    })
+  })
+
+  describe('async', function () {
+    describe('encrypt', function () {
+      it('should encrypt the object', async function () {
+        const encrypted = await this.encryptor.promises.encryptJson(
+          this.testObject
+        )
+        encrypted.should.match(
+          /^2023.1-v3:[0-9a-f]{32}:[a-zA-Z0-9=+/]+:[0-9a-f]{32}$/
+        )
+      })
+
+      it('should encrypt the object differently the next time', async function () {
+        const encrypted1 = await this.encryptor.promises.encryptJson(
+          this.testObject
+        )
+        const encrypted2 = await this.encryptor.promises.encryptJson(
+          this.testObject
+        )
+        encrypted1.should.not.equal(encrypted2)
+      })
+    })
+
+    describe('decrypt', function () {
+      it('should decrypt the string to get the same object', async function () {
+        const encrypted = await this.encryptor.promises.encryptJson(
+          this.testObject
+        )
+        const decrypted = await this.encryptor.promises.decryptToJson(encrypted)
+        expect(decrypted).to.deep.equal(this.testObject)
+      })
+
+      it('should not be able to decrypt 2015 string', async function () {
+        await expect(
+          this.encryptor.promises.decryptToJson(this.encrypted2015)
+        ).to.eventually.be.rejectedWith(
+          'unknown access-token-encryptor label 2015.1'
+        )
+      })
+
+      it('should not be able to decrypt a 2016 string', async function () {
+        await expect(
+          this.encryptor.promises.decryptToJson(this.encrypted2016)
+        ).to.be.rejectedWith('unknown access-token-encryptor label 2016.1')
+      })
+
+      it('should not be able to decrypt a 2019 string', async function () {
+        await expect(
+          this.encryptor.promises.decryptToJson(this.encrypted2019)
+        ).to.be.rejectedWith('unknown access-token-encryptor label 2019.1')
+      })
+
+      it('should decrypt an 2023 string to get the same object', async function () {
+        const decrypted = await this.encryptor.promises.decryptToJson(
+          this.encrypted2023
+        )
+        expect(decrypted).to.deep.equal(this.testObject)
+      })
+
+      it('should return an error when decrypting an invalid label', async function () {
+        await expect(
+          this.encryptor.promises.decryptToJson(this.badLabel)
+        ).to.be.rejectedWith('unknown access-token-encryptor label xxxxxx')
+      })
+
+      it('should return an error when decrypting an invalid key', async function () {
+        await expect(
+          this.encryptor.promises.decryptToJson(this.badKey)
+        ).to.be.rejectedWith('unknown access-token-encryptor label 2015.1')
+      })
+
+      it('should return an error when decrypting an invalid ciphertext', async function () {
+        await expect(
+          this.encryptor.promises.decryptToJson(this.badCipherText)
+        ).to.be.rejectedWith('unknown access-token-encryptor label 2015.1')
+      })
+    })
+  })
+})

libraries/access-token-encryptor/tsconfig.json

@@ -0,0 +1,7 @@
+{
+  "extends": "../../tsconfig.backend.json",
+  "include": [
+    "**/*.js",
+    "**/*.cjs"
+  ]
+}

libraries/fetch-utils/.gitignore

@@ -0,0 +1,3 @@
+
+# managed by monorepo$ bin/update_build_scripts
+.npmrc

libraries/fetch-utils/.mocharc.json

@@ -0,0 +1,6 @@
+{
+  "ui": "bdd",
+  "recursive": "true",
+  "reporter": "spec",
+  "require": "test/setup.js"
+}

libraries/fetch-utils/.nvmrc

@@ -0,0 +1 @@
+20.18.2

libraries/fetch-utils/buildscript.txt

@@ -0,0 +1,10 @@
+fetch-utils
+--dependencies=None
+--docker-repos=gcr.io/overleaf-ops
+--env-add=
+--env-pass-through=
+--esmock-loader=False
+--is-library=True
+--node-version=20.18.2
+--public-repo=False
+--script-version=4.5.0

libraries/fetch-utils/index.js

@@ -0,0 +1,320 @@
+const _ = require('lodash')
+const { Readable } = require('node:stream')
+const OError = require('@overleaf/o-error')
+const fetch = require('node-fetch')
+const http = require('node:http')
+const https = require('node:https')
+
+/**
+ * @import { Response } from 'node-fetch'
+ */
+
+/**
+ * Make a request and return the parsed JSON response.
+ *
+ * @param {string | URL} url - request URL
+ * @param {any} [opts] - fetch options
+ * @return {Promise<any>} the parsed JSON response
+ * @throws {RequestFailedError} if the response has a failure status code
+ */
+async function fetchJson(url, opts = {}) {
+  const { json } = await fetchJsonWithResponse(url, opts)
+  return json
+}
+
+async function fetchJsonWithResponse(url, opts = {}) {
+  const { fetchOpts } = parseOpts(opts)
+  fetchOpts.headers = fetchOpts.headers ?? {}
+  fetchOpts.headers.Accept = fetchOpts.headers.Accept ?? 'application/json'
+
+  const response = await performRequest(url, fetchOpts)
+  if (!response.ok) {
+    const body = await maybeGetResponseBody(response)
+    throw new RequestFailedError(url, opts, response, body)
+  }
+
+  const json = await response.json()
+  return { json, response }
+}
+
+/**
+ * Make a request and return a stream.
+ *
+ * If the response body is destroyed, the request is aborted.
+ *
+ * @param {string | URL} url - request URL
+ * @param {any} [opts] - fetch options
+ * @return {Promise<Readable>}
+ * @throws {RequestFailedError} if the response has a failure status code
+ */
+async function fetchStream(url, opts = {}) {
+  const { stream } = await fetchStreamWithResponse(url, opts)
+  return stream
+}
+
+async function fetchStreamWithResponse(url, opts = {}) {
+  const { fetchOpts, abortController } = parseOpts(opts)
+  const response = await performRequest(url, fetchOpts)
+
+  if (!response.ok) {
+    const body = await maybeGetResponseBody(response)
+    throw new RequestFailedError(url, opts, response, body)
+  }
+
+  abortOnDestroyedResponse(abortController, response)
+
+  const stream = response.body
+  return { stream, response }
+}
+
+/**
+ * Make a request and discard the response.
+ *
+ * @param {string | URL} url - request URL
+ * @param {any} [opts] - fetch options
+ * @return {Promise<Response>}
+ * @throws {RequestFailedError} if the response has a failure status code
+ */
+async function fetchNothing(url, opts = {}) {
+  const { fetchOpts } = parseOpts(opts)
+  const response = await performRequest(url, fetchOpts)
+  if (!response.ok) {
+    const body = await maybeGetResponseBody(response)
+    throw new RequestFailedError(url, opts, response, body)
+  }
+  await discardResponseBody(response)
+  return response
+}
+
+/**
+ * Make a request and extract the redirect from the response.
+ *
+ * @param {string | URL} url - request URL
+ * @param {any} [opts] - fetch options
+ * @return {Promise<string>}
+ * @throws {RequestFailedError} if the response has a non redirect status code or missing Location header
+ */
+async function fetchRedirect(url, opts = {}) {
+  const { location } = await fetchRedirectWithResponse(url, opts)
+  return location
+}
+
+/**
+ * Make a request and extract the redirect from the response.
+ *
+ * @param {string | URL} url - request URL
+ * @param {object} opts - fetch options
+ * @return {Promise<{location: string, response: Response}>}
+ * @throws {RequestFailedError} if the response has a non redirect status code or missing Location header
+ */
+async function fetchRedirectWithResponse(url, opts = {}) {
+  const { fetchOpts } = parseOpts(opts)
+  fetchOpts.redirect = 'manual'
+  const response = await performRequest(url, fetchOpts)
+  if (response.status < 300 || response.status >= 400) {
+    const body = await maybeGetResponseBody(response)
+    throw new RequestFailedError(url, opts, response, body)
+  }
+  const location = response.headers.get('Location')
+  if (!location) {
+    const body = await maybeGetResponseBody(response)
+    throw new RequestFailedError(url, opts, response, body).withCause(
+      new OError('missing Location response header on 3xx response', {
+        headers: Object.fromEntries(response.headers.entries()),
+      })
+    )
+  }
+  await discardResponseBody(response)
+  return { location, response }
+}
+
+/**
+ * Make a request and return a string.
+ *
+ * @param {string | URL} url - request URL
+ * @param {any} [opts] - fetch options
+ * @return {Promise<string>}
+ * @throws {RequestFailedError} if the response has a failure status code
+ */
+async function fetchString(url, opts = {}) {
+  const { body } = await fetchStringWithResponse(url, opts)
+  return body
+}
+
+async function fetchStringWithResponse(url, opts = {}) {
+  const { fetchOpts } = parseOpts(opts)
+  const response = await performRequest(url, fetchOpts)
+  if (!response.ok) {
+    const body = await maybeGetResponseBody(response)
+    throw new RequestFailedError(url, opts, response, body)
+  }
+  const body = await response.text()
+  return { body, response }
+}
+
+class RequestFailedError extends OError {
+  constructor(url, opts, response, body) {
+    super('request failed', {
+      url,
+      method: opts.method ?? 'GET',
+      status: response.status,
+    })
+
+    this.response = response
+    if (body != null) {
+      this.body = body
+    }
+  }
+}
+
+function parseOpts(opts) {
+  const fetchOpts = _.omit(opts, ['json', 'signal', 'basicAuth'])
+  if (opts.json) {
+    setupJsonBody(fetchOpts, opts.json)
+  }
+  if (opts.basicAuth) {
+    setupBasicAuth(fetchOpts, opts.basicAuth)
+  }
+
+  const abortController = new AbortController()
+  fetchOpts.signal = abortController.signal
+  if (opts.signal) {
+    abortOnSignal(abortController, opts.signal)
+  }
+  if (opts.body instanceof Readable) {
+    abortOnDestroyedRequest(abortController, fetchOpts.body)
+  }
+  return { fetchOpts, abortController }
+}
+
+function setupJsonBody(fetchOpts, json) {
+  fetchOpts.body = JSON.stringify(json)
+  fetchOpts.headers = fetchOpts.headers ?? {}
+  fetchOpts.headers['Content-Type'] = 'application/json'
+}
+
+function setupBasicAuth(fetchOpts, basicAuth) {
+  fetchOpts.headers = fetchOpts.headers ?? {}
+  fetchOpts.headers.Authorization =
+    'Basic ' +
+    Buffer.from(`${basicAuth.user}:${basicAuth.password}`).toString('base64')
+}
+
+function abortOnSignal(abortController, signal) {
+  const listener = () => {
+    abortController.abort(signal.reason)
+  }
+  if (signal.aborted) {
+    abortController.abort(signal.reason)
+  }
+  signal.addEventListener('abort', listener)
+}
+
+function abortOnDestroyedRequest(abortController, stream) {
+  stream.on('close', () => {
+    if (!stream.readableEnded) {
+      abortController.abort()
+    }
+  })
+}
+
+function abortOnDestroyedResponse(abortController, response) {
+  response.body.on('close', () => {
+    if (!response.bodyUsed) {
+      abortController.abort()
+    }
+  })
+}
+
+async function performRequest(url, fetchOpts) {
+  let response
+  try {
+    response = await fetch(url, fetchOpts)
+  } catch (err) {
+    if (fetchOpts.body instanceof Readable) {
+      fetchOpts.body.destroy()
+    }
+    throw OError.tag(err, err.message, {
+      url,
+      method: fetchOpts.method ?? 'GET',
+    })
+  }
+  if (fetchOpts.body instanceof Readable) {
+    response.body.on('close', () => {
+      if (!fetchOpts.body.readableEnded) {
+        fetchOpts.body.destroy()
+      }
+    })
+  }
+  return response
+}
+
+async function discardResponseBody(response) {
+  // eslint-disable-next-line no-unused-vars
+  for await (const chunk of response.body) {
+    // discard the body
+  }
+}
+
+/**
+ * @param {Response} response
+ */
+async function maybeGetResponseBody(response) {
+  try {
+    return await response.text()
+  } catch (err) {
+    return null
+  }
+}
+
+// Define custom http and https agents with support for connect timeouts
+
+class ConnectTimeoutError extends OError {
+  constructor(options) {
+    super('connect timeout', options)
+  }
+}
+
+function withTimeout(createConnection, options, callback) {
+  if (options.connectTimeout) {
+    // Wrap createConnection in a timeout
+    const timer = setTimeout(() => {
+      socket.destroy(new ConnectTimeoutError(options))
+    }, options.connectTimeout)
+    const socket = createConnection(options, (err, stream) => {
+      clearTimeout(timer)
+      callback(err, stream)
+    })
+    return socket
+  } else {
+    // Fallback to default createConnection
+    return createConnection(options, callback)
+  }
+}
+
+class CustomHttpAgent extends http.Agent {
+  createConnection(options, callback) {
+    return withTimeout(super.createConnection.bind(this), options, callback)
+  }
+}
+class CustomHttpsAgent extends https.Agent {
+  createConnection(options, callback) {
+    return withTimeout(super.createConnection.bind(this), options, callback)
+  }
+}
+
+module.exports = {
+  fetchJson,
+  fetchJsonWithResponse,
+  fetchStream,
+  fetchStreamWithResponse,
+  fetchNothing,
+  fetchRedirect,
+  fetchRedirectWithResponse,
+  fetchString,
+  fetchStringWithResponse,
+  RequestFailedError,
+  ConnectTimeoutError,
+  CustomHttpAgent,
+  CustomHttpsAgent,
+}

libraries/fetch-utils/package.json

@@ -0,0 +1,33 @@
+{
+  "name": "@overleaf/fetch-utils",
+  "version": "0.1.0",
+  "description": "utilities for node-fetch",
+  "main": "index.js",
+  "scripts": {
+    "test": "npm run lint && npm run format && npm run types:check && npm run test:unit",
+    "lint": "eslint --ext .js --ext .cjs --ext .ts --max-warnings 0 --format unix .",
+    "lint:fix": "eslint --fix --ext .js --ext .cjs --ext .ts .",
+    "format": "prettier --list-different $PWD/'**/*.{js,cjs,ts}'",
+    "format:fix": "prettier --write $PWD/'**/*.{js,cjs,ts}'",
+    "test:ci": "npm run test:unit",
+    "test:unit": "mocha --exit test/**/*.{js,cjs}",
+    "types:check": "tsc --noEmit"
+  },
+  "author": "Overleaf (https://www.overleaf.com)",
+  "license": "AGPL-3.0-only",
+  "devDependencies": {
+    "@types/node-fetch": "^2.6.11",
+    "body-parser": "^1.20.3",
+    "chai": "^4.3.6",
+    "chai-as-promised": "^7.1.1",
+    "express": "^4.21.2",
+    "mocha": "^11.1.0",
+    "typescript": "^5.0.4"
+  },
+  "dependencies": {
+    "@overleaf/o-error": "*",
+    "lodash": "^4.17.21",
+    "node-fetch": "^2.7.0",
+    "selfsigned": "^2.4.1"
+  }
+}

libraries/fetch-utils/test/setup.js

@@ -0,0 +1,4 @@
+const chai = require('chai')
+const chaiAsPromised = require('chai-as-promised')
+
+chai.use(chaiAsPromised)

libraries/fetch-utils/test/unit/FetchUtilsTests.js

@@ -0,0 +1,414 @@
+const { expect } = require('chai')
+const { FetchError, AbortError } = require('node-fetch')
+const { Readable } = require('node:stream')
+const { once } = require('node:events')
+const { TestServer } = require('./helpers/TestServer')
+const selfsigned = require('selfsigned')
+const {
+  fetchJson,
+  fetchStream,
+  fetchNothing,
+  fetchRedirect,
+  fetchString,
+  RequestFailedError,
+  CustomHttpAgent,
+  CustomHttpsAgent,
+} = require('../..')
+
+const HTTP_PORT = 30001
+const HTTPS_PORT = 30002
+
+const attrs = [{ name: 'commonName', value: 'example.com' }]
+const pems = selfsigned.generate(attrs, { days: 365 })
+
+const PRIVATE_KEY = pems.private
+const PUBLIC_CERT = pems.cert
+
+const dns = require('node:dns')
+const _originalLookup = dns.lookup
+// Custom DNS resolver function
+dns.lookup = (hostname, options, callback) => {
+  if (hostname === 'example.com') {
+    // If the hostname is our test case, return the ip address for the test server
+    if (options?.all) {
+      callback(null, [{ address: '127.0.0.1', family: 4 }])
+    } else {
+      callback(null, '127.0.0.1', 4)
+    }
+  } else {
+    // Otherwise, use the default lookup
+    _originalLookup(hostname, options, callback)
+  }
+}
+
+describe('fetch-utils', function () {
+  before(async function () {
+    this.server = new TestServer()
+    await this.server.start(HTTP_PORT, HTTPS_PORT, {
+      key: PRIVATE_KEY,
+      cert: PUBLIC_CERT,
+    })
+    this.url = path => `http://example.com:${HTTP_PORT}${path}`
+    this.httpsUrl = path => `https://example.com:${HTTPS_PORT}${path}`
+  })
+
+  beforeEach(function () {
+    this.server.lastReq = undefined
+  })
+
+  after(async function () {
+    await this.server.stop()
+  })
+
+  describe('fetchJson', function () {
+    it('parses a JSON response', async function () {
+      const json = await fetchJson(this.url('/json/hello'))
+      expect(json).to.deep.equal({ msg: 'hello' })
+    })
+
+    it('parses JSON in the request', async function () {
+      const json = await fetchJson(this.url('/json/add'), {
+        method: 'POST',
+        json: { a: 2, b: 3 },
+      })
+      expect(json).to.deep.equal({ sum: 5 })
+    })
+
+    it('accepts stringified JSON as body', async function () {
+      const json = await fetchJson(this.url('/json/add'), {
+        method: 'POST',
+        body: JSON.stringify({ a: 2, b: 3 }),
+        headers: { 'Content-Type': 'application/json' },
+      })
+      expect(json).to.deep.equal({ sum: 5 })
+    })
+
+    it('throws a FetchError when the payload is not JSON', async function () {
+      await expect(fetchJson(this.url('/hello'))).to.be.rejectedWith(FetchError)
+    })
+
+    it('aborts the request if JSON parsing fails', async function () {
+      await expect(fetchJson(this.url('/large'))).to.be.rejectedWith(FetchError)
+      await expectRequestAborted(this.server.lastReq)
+    })
+
+    it('handles errors when the payload is JSON', async function () {
+      await expect(fetchJson(this.url('/json/500'))).to.be.rejectedWith(
+        RequestFailedError
+      )
+      await expectRequestAborted(this.server.lastReq)
+    })
+
+    it('handles errors when the payload is not JSON', async function () {
+      await expect(fetchJson(this.url('/500'))).to.be.rejectedWith(
+        RequestFailedError
+      )
+      await expectRequestAborted(this.server.lastReq)
+    })
+
+    it('supports abort signals', async function () {
+      await expect(
+        abortOnceReceived(
+          signal => fetchJson(this.url('/hang'), { signal }),
+          this.server
+        )
+      ).to.be.rejectedWith(AbortError)
+      await expectRequestAborted(this.server.lastReq)
+    })
+
+    it('supports basic auth', async function () {
+      const json = await fetchJson(this.url('/json/basic-auth'), {
+        basicAuth: { user: 'user', password: 'pass' },
+      })
+      expect(json).to.deep.equal({ key: 'verysecret' })
+    })
+
+    it("destroys the request body if it doesn't get consumed", async function () {
+      const stream = Readable.from(infiniteIterator())
+      await fetchJson(this.url('/json/ignore-request'), {
+        method: 'POST',
+        body: stream,
+      })
+      expect(stream.destroyed).to.be.true
+    })
+  })
+
+  describe('fetchStream', function () {
+    it('returns a stream', async function () {
+      const stream = await fetchStream(this.url('/large'))
+      const text = await streamToString(stream)
+      expect(text).to.equal(this.server.largePayload)
+    })
+
+    it('aborts the request when the stream is destroyed', async function () {
+      const stream = await fetchStream(this.url('/large'))
+      stream.destroy()
+      await expectRequestAborted(this.server.lastReq)
+    })
+
+    it('aborts the request when the request body is destroyed before transfer', async function () {
+      const stream = Readable.from(infiniteIterator())
+      const promise = fetchStream(this.url('/hang'), {
+        method: 'POST',
+        body: stream,
+      })
+      stream.destroy()
+      await expect(promise).to.be.rejectedWith(AbortError)
+      await wait(80)
+      expect(this.server.lastReq).to.be.undefined
+    })
+
+    it('aborts the request when the request body is destroyed during transfer', async function () {
+      const stream = Readable.from(infiniteIterator())
+      // Note: this test won't work on `/hang`
+      const promise = fetchStream(this.url('/sink'), {
+        method: 'POST',
+        body: stream,
+      })
+      await once(this.server.events, 'request-received')
+      stream.destroy()
+      await expect(promise).to.be.rejectedWith(AbortError)
+      await expectRequestAborted(this.server.lastReq)
+    })
+
+    it('handles errors', async function () {
+      await expect(fetchStream(this.url('/500'))).to.be.rejectedWith(
+        RequestFailedError
+      )
+      await expectRequestAborted(this.server.lastReq)
+    })
+
+    it('supports abort signals', async function () {
+      await expect(
+        abortOnceReceived(
+          signal => fetchStream(this.url('/hang'), { signal }),
+          this.server
+        )
+      ).to.be.rejectedWith(AbortError)
+      await expectRequestAborted(this.server.lastReq)
+    })
+
+    it('destroys the request body when an error occurs', async function () {
+      const stream = Readable.from(infiniteIterator())
+      await expect(
+        abortOnceReceived(
+          signal =>
+            fetchStream(this.url('/hang'), {
+              method: 'POST',
+              body: stream,
+              signal,
+            }),
+          this.server
+        )
+      ).to.be.rejectedWith(AbortError)
+      expect(stream.destroyed).to.be.true
+    })
+  })
+
+  describe('fetchNothing', function () {
+    it('closes the connection', async function () {
+      await fetchNothing(this.url('/large'))
+      await expectRequestAborted(this.server.lastReq)
+    })
+
+    it('aborts the request when the request body is destroyed before transfer', async function () {
+      const stream = Readable.from(infiniteIterator())
+      const promise = fetchNothing(this.url('/hang'), {
+        method: 'POST',
+        body: stream,
+      })
+      stream.destroy()
+      await expect(promise).to.be.rejectedWith(AbortError)
+      expect(this.server.lastReq).to.be.undefined
+    })
+
+    it('aborts the request when the request body is destroyed during transfer', async function () {
+      const stream = Readable.from(infiniteIterator())
+      // Note: this test won't work on `/hang`
+      const promise = fetchNothing(this.url('/sink'), {
+        method: 'POST',
+        body: stream,
+      })
+      await once(this.server.events, 'request-received')
+      stream.destroy()
+      await expect(promise).to.be.rejectedWith(AbortError)
+      await wait(80)
+      await expectRequestAborted(this.server.lastReq)
+    })
+
+    it("doesn't abort the request if the request body ends normally", async function () {
+      const stream = Readable.from('hello there')
+      await fetchNothing(this.url('/sink'), { method: 'POST', body: stream })
+    })
+
+    it('handles errors', async function () {
+      await expect(fetchNothing(this.url('/500'))).to.be.rejectedWith(
+        RequestFailedError
+      )
+      await expectRequestAborted(this.server.lastReq)
+    })
+
+    it('supports abort signals', async function () {
+      await expect(
+        abortOnceReceived(
+          signal => fetchNothing(this.url('/hang'), { signal }),
+          this.server
+        )
+      ).to.be.rejectedWith(AbortError)
+      await expectRequestAborted(this.server.lastReq)
+    })
+
+    it('destroys the request body when an error occurs', async function () {
+      const stream = Readable.from(infiniteIterator())
+      await expect(
+        abortOnceReceived(
+          signal =>
+            fetchNothing(this.url('/hang'), {
+              method: 'POST',
+              body: stream,
+              signal,
+            }),
+          this.server
+        )
+      ).to.be.rejectedWith(AbortError)
+      expect(stream.destroyed).to.be.true
+    })
+  })
+
+  describe('fetchString', function () {
+    it('returns a string', async function () {
+      const body = await fetchString(this.url('/hello'))
+      expect(body).to.equal('hello')
+    })
+
+    it('handles errors', async function () {
+      await expect(fetchString(this.url('/500'))).to.be.rejectedWith(
+        RequestFailedError
+      )
+      await expectRequestAborted(this.server.lastReq)
+    })
+  })
+
+  describe('fetchRedirect', function () {
+    it('returns the immediate redirect', async function () {
+      const body = await fetchRedirect(this.url('/redirect/1'))
+      expect(body).to.equal(this.url('/redirect/2'))
+    })
+
+    it('rejects status 200', async function () {
+      await expect(fetchRedirect(this.url('/hello'))).to.be.rejectedWith(
+        RequestFailedError
+      )
+      await expectRequestAborted(this.server.lastReq)
+    })
+
+    it('rejects empty redirect', async function () {
+      await expect(fetchRedirect(this.url('/redirect/empty-location')))
+        .to.be.rejectedWith(RequestFailedError)
+        .and.eventually.have.property('cause')
+        .and.to.have.property('message')
+        .to.equal('missing Location response header on 3xx response')
+      await expectRequestAborted(this.server.lastReq)
+    })
+
+    it('handles errors', async function () {
+      await expect(fetchRedirect(this.url('/500'))).to.be.rejectedWith(
+        RequestFailedError
+      )
+      await expectRequestAborted(this.server.lastReq)
+    })
+  })
+
+  describe('CustomHttpAgent', function () {
+    it('makes an http request successfully', async function () {
+      const agent = new CustomHttpAgent({ connectTimeout: 100 })
+      const body = await fetchString(this.url('/hello'), { agent })
+      expect(body).to.equal('hello')
+    })
+
+    it('times out when accessing a non-routable address', async function () {
+      const agent = new CustomHttpAgent({ connectTimeout: 10 })
+      await expect(fetchString('http://10.255.255.255/', { agent }))
+        .to.be.rejectedWith(FetchError)
+        .and.eventually.have.property('message')
+        .and.to.equal(
+          'request to http://10.255.255.255/ failed, reason: connect timeout'
+        )
+    })
+  })
+
+  describe('CustomHttpsAgent', function () {
+    it('makes an https request successfully', async function () {
+      const agent = new CustomHttpsAgent({
+        connectTimeout: 100,
+        ca: PUBLIC_CERT,
+      })
+      const body = await fetchString(this.httpsUrl('/hello'), { agent })
+      expect(body).to.equal('hello')
+    })
+
+    it('rejects an untrusted server', async function () {
+      const agent = new CustomHttpsAgent({
+        connectTimeout: 100,
+      })
+      await expect(fetchString(this.httpsUrl('/hello'), { agent }))
+        .to.be.rejectedWith(FetchError)
+        .and.eventually.have.property('code')
+        .and.to.equal('DEPTH_ZERO_SELF_SIGNED_CERT')
+    })
+
+    it('times out when accessing a non-routable address', async function () {
+      const agent = new CustomHttpsAgent({ connectTimeout: 10 })
+      await expect(fetchString('https://10.255.255.255/', { agent }))
+        .to.be.rejectedWith(FetchError)
+        .and.eventually.have.property('message')
+        .and.to.equal(
+          'request to https://10.255.255.255/ failed, reason: connect timeout'
+        )
+    })
+  })
+})
+
+async function streamToString(stream) {
+  let s = ''
+  for await (const chunk of stream) {
+    s += chunk
+  }
+  return s
+}
+
+async function* infiniteIterator() {
+  let i = 1
+  while (true) {
+    yield `chunk ${i++}\n`
+  }
+}
+
+/**
+ * @param {(signal: AbortSignal) => Promise<any>} func
+ * @param {TestServer} server
+ */
+async function abortOnceReceived(func, server) {
+  const controller = new AbortController()
+  const promise = func(controller.signal)
+  await once(server.events, 'request-received')
+  controller.abort()
+  return await promise
+}
+
+async function expectRequestAborted(req) {
+  if (!req.destroyed) {
+    try {
+      await once(req, 'close')
+    } catch (err) {
+      // `once` throws if req emits an 'error' event.
+      // We ignore `Error: aborted` when the request is aborted.
+      if (err.message !== 'aborted') {
+        throw err
+      }
+    }
+  }
+  expect(req.destroyed).to.be.true
+}
+
+const wait = ms => new Promise(resolve => setTimeout(resolve, ms))

libraries/fetch-utils/test/unit/helpers/TestServer.js

@@ -0,0 +1,130 @@
+const express = require('express')
+const bodyParser = require('body-parser')
+const { EventEmitter } = require('node:events')
+const http = require('node:http')
+const https = require('node:https')
+const { promisify } = require('node:util')
+
+class TestServer {
+  constructor() {
+    this.app = express()
+    this.events = new EventEmitter()
+
+    this.app.use(bodyParser.json())
+    this.app.use((req, res, next) => {
+      this.events.emit('request-received')
+      this.lastReq = req
+      next()
+    })
+
+    // Plain text endpoints
+
+    this.app.get('/hello', (req, res) => {
+      res.send('hello')
+    })
+
+    this.largePayload = 'x'.repeat(16 * 1024 * 1024)
+    this.app.get('/large', (req, res) => {
+      res.send(this.largePayload)
+    })
+
+    this.app.get('/204', (req, res) => {
+      res.status(204).end()
+    })
+
+    this.app.get('/empty', (req, res) => {
+      res.end()
+    })
+
+    this.app.get('/500', (req, res) => {
+      res.sendStatus(500)
+    })
+
+    this.app.post('/sink', (req, res) => {
+      req.on('data', () => {})
+      req.on('end', () => {
+        res.status(204).end()
+      })
+    })
+
+    // JSON endpoints
+
+    this.app.get('/json/hello', (req, res) => {
+      res.json({ msg: 'hello' })
+    })
+
+    this.app.post('/json/add', (req, res) => {
+      const { a, b } = req.body
+      res.json({ sum: a + b })
+    })
+
+    this.app.get('/json/500', (req, res) => {
+      res.status(500).json({ error: 'Internal server error' })
+    })
+
+    this.app.get('/json/basic-auth', (req, res) => {
+      const expectedAuth =
+        'Basic ' + Buffer.from('user:pass').toString('base64')
+      if (req.headers.authorization === expectedAuth) {
+        res.json({ key: 'verysecret' })
+      } else {
+        res.status(401).json({ error: 'unauthorized' })
+      }
+    })
+
+    this.app.post('/json/ignore-request', (req, res) => {
+      res.json({ msg: 'hello' })
+    })
+
+    // Never returns
+
+    this.app.get('/hang', (req, res) => {})
+    this.app.post('/hang', (req, res) => {})
+
+    // Redirect
+
+    this.app.get('/redirect/1', (req, res) => {
+      res.redirect('/redirect/2')
+    })
+    this.app.get('/redirect/2', (req, res) => {
+      res.send('body after redirect')
+    })
+    this.app.get('/redirect/empty-location', (req, res) => {
+      res.sendStatus(302)
+    })
+  }
+
+  start(port, httpsPort, httpsOptions) {
+    const startHttp = new Promise((resolve, reject) => {
+      this.server = http.createServer(this.app).listen(port, err => {
+        if (err) {
+          reject(err)
+        } else {
+          resolve()
+        }
+      })
+    })
+    const startHttps = new Promise((resolve, reject) => {
+      this.https_server = https
+        .createServer(httpsOptions, this.app)
+        .listen(httpsPort, err => {
+          if (err) {
+            reject(err)
+          } else {
+            resolve()
+          }
+        })
+    })
+    return Promise.all([startHttp, startHttps])
+  }
+
+  stop() {
+    const stopHttp = promisify(this.server.close).bind(this.server)
+    const stopHttps = promisify(this.https_server.close).bind(this.https_server)
+    this.server.closeAllConnections()
+    this.https_server.closeAllConnections()
+    return Promise.all([stopHttp(), stopHttps()])
+  }
+}
+
+module.exports = { TestServer }

libraries/fetch-utils/tsconfig.json

@@ -0,0 +1,7 @@
+{
+  "extends": "../../tsconfig.backend.json",
+  "include": [
+    "**/*.js",
+    "**/*.cjs"
+  ]
+}

libraries/logger/.gitignore

@@ -0,0 +1,3 @@
+node_modules
+
+.npmrc

libraries/logger/.mocharc.json

@@ -0,0 +1,3 @@
+{
+  "require": "test/setup.js"
+}

libraries/logger/.nvmrc

@@ -0,0 +1 @@
+20.18.2

libraries/logger/CHANGELOG.md

@@ -0,0 +1,10 @@
+## v3.1.1
+
+* Handle malformed requests in the req serializer
+
+## v3.0.0
+
+* Improve logging in Google Cloud Platform. Set environment variable `GCP_LOGGING=true` to enable.
+
+This version of the metrics module only works with versions of the `@overleaf/metrics` module greater than v4.0.0
+

libraries/logger/LICENSE.txt

@@ -0,0 +1,661 @@
+                    GNU AFFERO GENERAL PUBLIC LICENSE
+                       Version 3, 19 November 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+                            Preamble
+
+  The GNU Affero General Public License is a free, copyleft license for
+software and other kinds of works, specifically designed to ensure
+cooperation with the community in the case of network server software.
+
+  The licenses for most software and other practical works are designed
+to take away your freedom to share and change the works.  By contrast,
+our General Public Licenses are intended to guarantee your freedom to
+share and change all versions of a program--to make sure it remains free
+software for all its users.
+
+  When we speak of free software, we are referring to freedom, not
+price.  Our General Public Licenses are designed to make sure that you
+have the freedom to distribute copies of free software (and charge for
+them if you wish), that you receive source code or can get it if you
+want it, that you can change the software or use pieces of it in new
+free programs, and that you know you can do these things.
+
+  Developers that use our General Public Licenses protect your rights
+with two steps: (1) assert copyright on the software, and (2) offer
+you this License which gives you legal permission to copy, distribute
+and/or modify the software.
+
+  A secondary benefit of defending all users' freedom is that
+improvements made in alternate versions of the program, if they
+receive widespread use, become available for other developers to
+incorporate.  Many developers of free software are heartened and
+encouraged by the resulting cooperation.  However, in the case of
+software used on network servers, this result may fail to come about.
+The GNU General Public License permits making a modified version and
+letting the public access it on a server without ever releasing its
+source code to the public.
+
+  The GNU Affero General Public License is designed specifically to
+ensure that, in such cases, the modified source code becomes available
+to the community.  It requires the operator of a network server to
+provide the source code of the modified version running there to the
+users of that server.  Therefore, public use of a modified version, on
+a publicly accessible server, gives the public access to the source
+code of the modified version.
+
+  An older license, called the Affero General Public License and
+published by Affero, was designed to accomplish similar goals.  This is
+a different license, not a version of the Affero GPL, but Affero has
+released a new version of the Affero GPL which permits relicensing under
+this license.
+
+  The precise terms and conditions for copying, distribution and
+modification follow.
+
+                       TERMS AND CONDITIONS
+
+  0. Definitions.
+
+  "This License" refers to version 3 of the GNU Affero General Public License.
+
+  "Copyright" also means copyright-like laws that apply to other kinds of
+works, such as semiconductor masks.
+
+  "The Program" refers to any copyrightable work licensed under this
+License.  Each licensee is addressed as "you".  "Licensees" and
+"recipients" may be individuals or organizations.
+
+  To "modify" a work means to copy from or adapt all or part of the work
+in a fashion requiring copyright permission, other than the making of an
+exact copy.  The resulting work is called a "modified version" of the
+earlier work or a work "based on" the earlier work.
+
+  A "covered work" means either the unmodified Program or a work based
+on the Program.
+
+  To "propagate" a work means to do anything with it that, without
+permission, would make you directly or secondarily liable for
+infringement under applicable copyright law, except executing it on a
+computer or modifying a private copy.  Propagation includes copying,
+distribution (with or without modification), making available to the
+public, and in some countries other activities as well.
+
+  To "convey" a work means any kind of propagation that enables other
+parties to make or receive copies.  Mere interaction with a user through
+a computer network, with no transfer of a copy, is not conveying.
+
+  An interactive user interface displays "Appropriate Legal Notices"
+to the extent that it includes a convenient and prominently visible
+feature that (1) displays an appropriate copyright notice, and (2)
+tells the user that there is no warranty for the work (except to the
+extent that warranties are provided), that licensees may convey the
+work under this License, and how to view a copy of this License.  If
+the interface presents a list of user commands or options, such as a
+menu, a prominent item in the list meets this criterion.
+
+  1. Source Code.
+
+  The "source code" for a work means the preferred form of the work
+for making modifications to it.  "Object code" means any non-source
+form of a work.
+
+  A "Standard Interface" means an interface that either is an official
+standard defined by a recognized standards body, or, in the case of
+interfaces specified for a particular programming language, one that
+is widely used among developers working in that language.
+
+  The "System Libraries" of an executable work include anything, other
+than the work as a whole, that (a) is included in the normal form of
+packaging a Major Component, but which is not part of that Major
+Component, and (b) serves only to enable use of the work with that
+Major Component, or to implement a Standard Interface for which an
+implementation is available to the public in source code form.  A
+"Major Component", in this context, means a major essential component
+(kernel, window system, and so on) of the specific operating system
+(if any) on which the executable work runs, or a compiler used to
+produce the work, or an object code interpreter used to run it.
+
+  The "Corresponding Source" for a work in object code form means all
+the source code needed to generate, install, and (for an executable
+work) run the object code and to modify the work, including scripts to
+control those activities.  However, it does not include the work's
+System Libraries, or general-purpose tools or generally available free
+programs which are used unmodified in performing those activities but
+which are not part of the work.  For example, Corresponding Source
+includes interface definition files associated with source files for
+the work, and the source code for shared libraries and dynamically
+linked subprograms that the work is specifically designed to require,
+such as by intimate data communication or control flow between those
+subprograms and other parts of the work.
+
+  The Corresponding Source need not include anything that users
+can regenerate automatically from other parts of the Corresponding
+Source.
+
+  The Corresponding Source for a work in source code form is that
+same work.
+
+  2. Basic Permissions.
+
+  All rights granted under this License are granted for the term of
+copyright on the Program, and are irrevocable provided the stated
+conditions are met.  This License explicitly affirms your unlimited
+permission to run the unmodified Program.  The output from running a
+covered work is covered by this License only if the output, given its
+content, constitutes a covered work.  This License acknowledges your
+rights of fair use or other equivalent, as provided by copyright law.
+
+  You may make, run and propagate covered works that you do not
+convey, without conditions so long as your license otherwise remains
+in force.  You may convey covered works to others for the sole purpose
+of having them make modifications exclusively for you, or provide you
+with facilities for running those works, provided that you comply with
+the terms of this License in conveying all material for which you do
+not control copyright.  Those thus making or running the covered works
+for you must do so exclusively on your behalf, under your direction
+and control, on terms that prohibit them from making any copies of
+your copyrighted material outside their relationship with you.
+
+  Conveying under any other circumstances is permitted solely under
+the conditions stated below.  Sublicensing is not allowed; section 10
+makes it unnecessary.
+
+  3. Protecting Users' Legal Rights From Anti-Circumvention Law.
+
+  No covered work shall be deemed part of an effective technological
+measure under any applicable law fulfilling obligations under article
+11 of the WIPO copyright treaty adopted on 20 December 1996, or
+similar laws prohibiting or restricting circumvention of such
+measures.
+
+  When you convey a covered work, you waive any legal power to forbid
+circumvention of technological measures to the extent such circumvention
+is effected by exercising rights under this License with respect to
+the covered work, and you disclaim any intention to limit operation or
+modification of the work as a means of enforcing, against the work's
+users, your or third parties' legal rights to forbid circumvention of
+technological measures.
+
+  4. Conveying Verbatim Copies.
+
+  You may convey verbatim copies of the Program's source code as you
+receive it, in any medium, provided that you conspicuously and
+appropriately publish on each copy an appropriate copyright notice;
+keep intact all notices stating that this License and any
+non-permissive terms added in accord with section 7 apply to the code;
+keep intact all notices of the absence of any warranty; and give all
+recipients a copy of this License along with the Program.
+
+  You may charge any price or no price for each copy that you convey,
+and you may offer support or warranty protection for a fee.
+
+  5. Conveying Modified Source Versions.
+
+  You may convey a work based on the Program, or the modifications to
+produce it from the Program, in the form of source code under the
+terms of section 4, provided that you also meet all of these conditions:
+
+    a) The work must carry prominent notices stating that you modified
+    it, and giving a relevant date.
+
+    b) The work must carry prominent notices stating that it is
+    released under this License and any conditions added under section
+    7.  This requirement modifies the requirement in section 4 to
+    "keep intact all notices".
+
+    c) You must license the entire work, as a whole, under this
+    License to anyone who comes into possession of a copy.  This
+    License will therefore apply, along with any applicable section 7
+    additional terms, to the whole of the work, and all its parts,
+    regardless of how they are packaged.  This License gives no
+    permission to license the work in any other way, but it does not
+    invalidate such permission if you have separately received it.
+
+    d) If the work has interactive user interfaces, each must display
+    Appropriate Legal Notices; however, if the Program has interactive
+    interfaces that do not display Appropriate Legal Notices, your
+    work need not make them do so.
+
+  A compilation of a covered work with other separate and independent
+works, which are not by their nature extensions of the covered work,
+and which are not combined with it such as to form a larger program,
+in or on a volume of a storage or distribution medium, is called an
+"aggregate" if the compilation and its resulting copyright are not
+used to limit the access or legal rights of the compilation's users
+beyond what the individual works permit.  Inclusion of a covered work
+in an aggregate does not cause this License to apply to the other
+parts of the aggregate.
+
+  6. Conveying Non-Source Forms.
+
+  You may convey a covered work in object code form under the terms
+of sections 4 and 5, provided that you also convey the
+machine-readable Corresponding Source under the terms of this License,
+in one of these ways:
+
+    a) Convey the object code in, or embodied in, a physical product
+    (including a physical distribution medium), accompanied by the
+    Corresponding Source fixed on a durable physical medium
+    customarily used for software interchange.
+
+    b) Convey the object code in, or embodied in, a physical product
+    (including a physical distribution medium), accompanied by a
+    written offer, valid for at least three years and valid for as
+    long as you offer spare parts or customer support for that product
+    model, to give anyone who possesses the object code either (1) a
+    copy of the Corresponding Source for all the software in the
+    product that is covered by this License, on a durable physical
+    medium customarily used for software interchange, for a price no
+    more than your reasonable cost of physically performing this
+    conveying of source, or (2) access to copy the
+    Corresponding Source from a network server at no charge.
+
+    c) Convey individual copies of the object code with a copy of the
+    written offer to provide the Corresponding Source.  This
+    alternative is allowed only occasionally and noncommercially, and
+    only if you received the object code with such an offer, in accord
+    with subsection 6b.
+
+    d) Convey the object code by offering access from a designated
+    place (gratis or for a charge), and offer equivalent access to the
+    Corresponding Source in the same way through the same place at no
+    further charge.  You need not require recipients to copy the
+    Corresponding Source along with the object code.  If the place to
+    copy the object code is a network server, the Corresponding Source
+    may be on a different server (operated by you or a third party)
+    that supports equivalent copying facilities, provided you maintain
+    clear directions next to the object code saying where to find the
+    Corresponding Source.  Regardless of what server hosts the
+    Corresponding Source, you remain obligated to ensure that it is
+    available for as long as needed to satisfy these requirements.
+
+    e) Convey the object code using peer-to-peer transmission, provided
+    you inform other peers where the object code and Corresponding
+    Source of the work are being offered to the general public at no
+    charge under subsection 6d.
+
+  A separable portion of the object code, whose source code is excluded
+from the Corresponding Source as a System Library, need not be
+included in conveying the object code work.
+
+  A "User Product" is either (1) a "consumer product", which means any
+tangible personal property which is normally used for personal, family,
+or household purposes, or (2) anything designed or sold for incorporation
+into a dwelling.  In determining whether a product is a consumer product,
+doubtful cases shall be resolved in favor of coverage.  For a particular
+product received by a particular user, "normally used" refers to a
+typical or common use of that class of product, regardless of the status
+of the particular user or of the way in which the particular user
+actually uses, or expects or is expected to use, the product.  A product
+is a consumer product regardless of whether the product has substantial
+commercial, industrial or non-consumer uses, unless such uses represent
+the only significant mode of use of the product.
+
+  "Installation Information" for a User Product means any methods,
+procedures, authorization keys, or other information required to install
+and execute modified versions of a covered work in that User Product from
+a modified version of its Corresponding Source.  The information must
+suffice to ensure that the continued functioning of the modified object
+code is in no case prevented or interfered with solely because
+modification has been made.
+
+  If you convey an object code work under this section in, or with, or
+specifically for use in, a User Product, and the conveying occurs as
+part of a transaction in which the right of possession and use of the
+User Product is transferred to the recipient in perpetuity or for a
+fixed term (regardless of how the transaction is characterized), the
+Corresponding Source conveyed under this section must be accompanied
+by the Installation Information.  But this requirement does not apply
+if neither you nor any third party retains the ability to install
+modified object code on the User Product (for example, the work has
+been installed in ROM).
+
+  The requirement to provide Installation Information does not include a
+requirement to continue to provide support service, warranty, or updates
+for a work that has been modified or installed by the recipient, or for
+the User Product in which it has been modified or installed.  Access to a
+network may be denied when the modification itself materially and
+adversely affects the operation of the network or violates the rules and
+protocols for communication across the network.
+
+  Corresponding Source conveyed, and Installation Information provided,
+in accord with this section must be in a format that is publicly
+documented (and with an implementation available to the public in
+source code form), and must require no special password or key for
+unpacking, reading or copying.
+
+  7. Additional Terms.
+
+  "Additional permissions" are terms that supplement the terms of this
+License by making exceptions from one or more of its conditions.
+Additional permissions that are applicable to the entire Program shall
+be treated as though they were included in this License, to the extent
+that they are valid under applicable law.  If additional permissions
+apply only to part of the Program, that part may be used separately
+under those permissions, but the entire Program remains governed by
+this License without regard to the additional permissions.
+
+  When you convey a copy of a covered work, you may at your option
+remove any additional permissions from that copy, or from any part of
+it.  (Additional permissions may be written to require their own
+removal in certain cases when you modify the work.)  You may place
+additional permissions on material, added by you to a covered work,
+for which you have or can give appropriate copyright permission.
+
+  Notwithstanding any other provision of this License, for material you
+add to a covered work, you may (if authorized by the copyright holders of
+that material) supplement the terms of this License with terms:
+
+    a) Disclaiming warranty or limiting liability differently from the
+    terms of sections 15 and 16 of this License; or
+
+    b) Requiring preservation of specified reasonable legal notices or
+    author attributions in that material or in the Appropriate Legal
+    Notices displayed by works containing it; or
+
+    c) Prohibiting misrepresentation of the origin of that material, or
+    requiring that modified versions of such material be marked in
+    reasonable ways as different from the original version; or
+
+    d) Limiting the use for publicity purposes of names of licensors or
+    authors of the material; or
+
+    e) Declining to grant rights under trademark law for use of some
+    trade names, trademarks, or service marks; or
+
+    f) Requiring indemnification of licensors and authors of that
+    material by anyone who conveys the material (or modified versions of
+    it) with contractual assumptions of liability to the recipient, for
+    any liability that these contractual assumptions directly impose on
+    those licensors and authors.
+
+  All other non-permissive additional terms are considered "further
+restrictions" within the meaning of section 10.  If the Program as you
+received it, or any part of it, contains a notice stating that it is
+governed by this License along with a term that is a further
+restriction, you may remove that term.  If a license document contains
+a further restriction but permits relicensing or conveying under this
+License, you may add to a covered work material governed by the terms
+of that license document, provided that the further restriction does
+not survive such relicensing or conveying.
+
+  If you add terms to a covered work in accord with this section, you
+must place, in the relevant source files, a statement of the
+additional terms that apply to those files, or a notice indicating
+where to find the applicable terms.
+
+  Additional terms, permissive or non-permissive, may be stated in the
+form of a separately written license, or stated as exceptions;
+the above requirements apply either way.
+
+  8. Termination.
+
+  You may not propagate or modify a covered work except as expressly
+provided under this License.  Any attempt otherwise to propagate or
+modify it is void, and will automatically terminate your rights under
+this License (including any patent licenses granted under the third
+paragraph of section 11).
+
+  However, if you cease all violation of this License, then your
+license from a particular copyright holder is reinstated (a)
+provisionally, unless and until the copyright holder explicitly and
+finally terminates your license, and (b) permanently, if the copyright
+holder fails to notify you of the violation by some reasonable means
+prior to 60 days after the cessation.
+
+  Moreover, your license from a particular copyright holder is
+reinstated permanently if the copyright holder notifies you of the
+violation by some reasonable means, this is the first time you have
+received notice of violation of this License (for any work) from that
+copyright holder, and you cure the violation prior to 30 days after
+your receipt of the notice.
+
+  Termination of your rights under this section does not terminate the
+licenses of parties who have received copies or rights from you under
+this License.  If your rights have been terminated and not permanently
+reinstated, you do not qualify to receive new licenses for the same
+material under section 10.
+
+  9. Acceptance Not Required for Having Copies.
+
+  You are not required to accept this License in order to receive or
+run a copy of the Program.  Ancillary propagation of a covered work
+occurring solely as a consequence of using peer-to-peer transmission
+to receive a copy likewise does not require acceptance.  However,
+nothing other than this License grants you permission to propagate or
+modify any covered work.  These actions infringe copyright if you do
+not accept this License.  Therefore, by modifying or propagating a
+covered work, you indicate your acceptance of this License to do so.
+
+  10. Automatic Licensing of Downstream Recipients.
+
+  Each time you convey a covered work, the recipient automatically
+receives a license from the original licensors, to run, modify and
+propagate that work, subject to this License.  You are not responsible
+for enforcing compliance by third parties with this License.
+
+  An "entity transaction" is a transaction transferring control of an
+organization, or substantially all assets of one, or subdividing an
+organization, or merging organizations.  If propagation of a covered
+work results from an entity transaction, each party to that
+transaction who receives a copy of the work also receives whatever
+licenses to the work the party's predecessor in interest had or could
+give under the previous paragraph, plus a right to possession of the
+Corresponding Source of the work from the predecessor in interest, if
+the predecessor has it or can get it with reasonable efforts.
+
+  You may not impose any further restrictions on the exercise of the
+rights granted or affirmed under this License.  For example, you may
+not impose a license fee, royalty, or other charge for exercise of
+rights granted under this License, and you may not initiate litigation
+(including a cross-claim or counterclaim in a lawsuit) alleging that
+any patent claim is infringed by making, using, selling, offering for
+sale, or importing the Program or any portion of it.
+
+  11. Patents.
+
+  A "contributor" is a copyright holder who authorizes use under this
+License of the Program or a work on which the Program is based.  The
+work thus licensed is called the contributor's "contributor version".
+
+  A contributor's "essential patent claims" are all patent claims
+owned or controlled by the contributor, whether already acquired or
+hereafter acquired, that would be infringed by some manner, permitted
+by this License, of making, using, or selling its contributor version,
+but do not include claims that would be infringed only as a
+consequence of further modification of the contributor version.  For
+purposes of this definition, "control" includes the right to grant
+patent sublicenses in a manner consistent with the requirements of
+this License.
+
+  Each contributor grants you a non-exclusive, worldwide, royalty-free
+patent license under the contributor's essential patent claims, to
+make, use, sell, offer for sale, import and otherwise run, modify and
+propagate the contents of its contributor version.
+
+  In the following three paragraphs, a "patent license" is any express
+agreement or commitment, however denominated, not to enforce a patent
+(such as an express permission to practice a patent or covenant not to
+sue for patent infringement).  To "grant" such a patent license to a
+party means to make such an agreement or commitment not to enforce a
+patent against the party.
+
+  If you convey a covered work, knowingly relying on a patent license,
+and the Corresponding Source of the work is not available for anyone
+to copy, free of charge and under the terms of this License, through a
+publicly available network server or other readily accessible means,
+then you must either (1) cause the Corresponding Source to be so
+available, or (2) arrange to deprive yourself of the benefit of the
+patent license for this particular work, or (3) arrange, in a manner
+consistent with the requirements of this License, to extend the patent
+license to downstream recipients.  "Knowingly relying" means you have
+actual knowledge that, but for the patent license, your conveying the
+covered work in a country, or your recipient's use of the covered work
+in a country, would infringe one or more identifiable patents in that
+country that you have reason to believe are valid.
+
+  If, pursuant to or in connection with a single transaction or
+arrangement, you convey, or propagate by procuring conveyance of, a
+covered work, and grant a patent license to some of the parties
+receiving the covered work authorizing them to use, propagate, modify
+or convey a specific copy of the covered work, then the patent license
+you grant is automatically extended to all recipients of the covered
+work and works based on it.
+
+  A patent license is "discriminatory" if it does not include within
+the scope of its coverage, prohibits the exercise of, or is
+conditioned on the non-exercise of one or more of the rights that are
+specifically granted under this License.  You may not convey a covered
+work if you are a party to an arrangement with a third party that is
+in the business of distributing software, under which you make payment
+to the third party based on the extent of your activity of conveying
+the work, and under which the third party grants, to any of the
+parties who would receive the covered work from you, a discriminatory
+patent license (a) in connection with copies of the covered work
+conveyed by you (or copies made from those copies), or (b) primarily
+for and in connection with specific products or compilations that
+contain the covered work, unless you entered into that arrangement,
+or that patent license was granted, prior to 28 March 2007.
+
+  Nothing in this License shall be construed as excluding or limiting
+any implied license or other defenses to infringement that may
+otherwise be available to you under applicable patent law.
+
+  12. No Surrender of Others' Freedom.
+
+  If conditions are imposed on you (whether by court order, agreement or
+otherwise) that contradict the conditions of this License, they do not
+excuse you from the conditions of this License.  If you cannot convey a
+covered work so as to satisfy simultaneously your obligations under this
+License and any other pertinent obligations, then as a consequence you may
+not convey it at all.  For example, if you agree to terms that obligate you
+to collect a royalty for further conveying from those to whom you convey
+the Program, the only way you could satisfy both those terms and this
+License would be to refrain entirely from conveying the Program.
+
+  13. Remote Network Interaction; Use with the GNU General Public License.
+
+  Notwithstanding any other provision of this License, if you modify the
+Program, your modified version must prominently offer all users
+interacting with it remotely through a computer network (if your version
+supports such interaction) an opportunity to receive the Corresponding
+Source of your version by providing access to the Corresponding Source
+from a network server at no charge, through some standard or customary
+means of facilitating copying of software.  This Corresponding Source
+shall include the Corresponding Source for any work covered by version 3
+of the GNU General Public License that is incorporated pursuant to the
+following paragraph.
+
+  Notwithstanding any other provision of this License, you have
+permission to link or combine any covered work with a work licensed
+under version 3 of the GNU General Public License into a single
+combined work, and to convey the resulting work.  The terms of this
+License will continue to apply to the part which is the covered work,
+but the work with which it is combined will remain governed by version
+3 of the GNU General Public License.
+
+  14. Revised Versions of this License.
+
+  The Free Software Foundation may publish revised and/or new versions of
+the GNU Affero General Public License from time to time.  Such new versions
+will be similar in spirit to the present version, but may differ in detail to
+address new problems or concerns.
+
+  Each version is given a distinguishing version number.  If the
+Program specifies that a certain numbered version of the GNU Affero General
+Public License "or any later version" applies to it, you have the
+option of following the terms and conditions either of that numbered
+version or of any later version published by the Free Software
+Foundation.  If the Program does not specify a version number of the
+GNU Affero General Public License, you may choose any version ever published
+by the Free Software Foundation.
+
+  If the Program specifies that a proxy can decide which future
+versions of the GNU Affero General Public License can be used, that proxy's
+public statement of acceptance of a version permanently authorizes you
+to choose that version for the Program.
+
+  Later license versions may give you additional or different
+permissions.  However, no additional obligations are imposed on any
+author or copyright holder as a result of your choosing to follow a
+later version.
+
+  15. Disclaimer of Warranty.
+
+  THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
+APPLICABLE LAW.  EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
+HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
+OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
+THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+PURPOSE.  THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
+IS WITH YOU.  SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
+ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
+
+  16. Limitation of Liability.
+
+  IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
+WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
+THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
+GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
+USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
+DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
+PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
+EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
+SUCH DAMAGES.
+
+  17. Interpretation of Sections 15 and 16.
+
+  If the disclaimer of warranty and limitation of liability provided
+above cannot be given local legal effect according to their terms,
+reviewing courts shall apply local law that most closely approximates
+an absolute waiver of all civil liability in connection with the
+Program, unless a warranty or assumption of liability accompanies a
+copy of the Program in return for a fee.
+
+                     END OF TERMS AND CONDITIONS
+
+            How to Apply These Terms to Your New Programs
+
+  If you develop a new program, and you want it to be of the greatest
+possible use to the public, the best way to achieve this is to make it
+free software which everyone can redistribute and change under these terms.
+
+  To do so, attach the following notices to the program.  It is safest
+to attach them to the start of each source file to most effectively
+state the exclusion of warranty; and each file should have at least
+the "copyright" line and a pointer to where the full notice is found.
+
+    <one line to give the program's name and a brief idea of what it does.>
+    Copyright (C) <year>  <name of author>
+
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU Affero General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU Affero General Public License for more details.
+
+    You should have received a copy of the GNU Affero General Public License
+    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+Also add information on how to contact you by electronic and paper mail.
+
+  If your software can interact with users remotely through a computer
+network, you should also make sure that it provides a way for users to
+get its source.  For example, if your program is a web application, its
+interface could display a "Source" link that leads users to an archive
+of the code.  There are many ways you could offer source, and different
+solutions will be better for different programs; see section 13 for the
+specific requirements.
+
+  You should also get your employer (if you work as a programmer) or school,
+if any, to sign a "copyright disclaimer" for the program, if necessary.
+For more information on this, and how to apply and follow the GNU AGPL, see
+<http://www.gnu.org/licenses/>.

libraries/logger/buildscript.txt

@@ -0,0 +1,10 @@
+logger
+--dependencies=None
+--docker-repos=gcr.io/overleaf-ops
+--env-add=
+--env-pass-through=
+--esmock-loader=False
+--is-library=True
+--node-version=20.18.2
+--public-repo=False
+--script-version=4.5.0

libraries/logger/gcp-manager.js

@@ -0,0 +1,133 @@
+const bunyan = require('bunyan')
+
+/**
+ * When we copy log entry fields, omit some bunyan core fields that are not
+ * interesting, that have a special meaning in GCP, or that we will process
+ * separately.
+ */
+const ENTRY_FIELDS_TO_OMIT = [
+  'level',
+  'name',
+  'hostname',
+  'v',
+  'pid',
+  'msg',
+  'err',
+  'error',
+  'req',
+  'res',
+]
+
+/**
+ * Convert a bunyan log entry to a format that GCP understands
+ */
+function convertLogEntry(entry) {
+  const gcpEntry = omit(entry, ENTRY_FIELDS_TO_OMIT)
+
+  // Error information. In GCP, the stack trace goes in the message property.
+  // This enables the error reporting feature.
+  const err = entry.err || entry.error
+  if (err) {
+    if (err.info) {
+      Object.assign(gcpEntry, err.info)
+    }
+    if (err.code) {
+      gcpEntry.code = err.code
+    }
+    if (err.signal) {
+      gcpEntry.signal = err.signal
+    }
+    const stack = err.stack
+    if (stack && stack !== '(no stack)') {
+      gcpEntry.message = stack
+    } else if (err.message) {
+      gcpEntry.message = err.message
+    }
+    if (entry.name) {
+      gcpEntry.serviceContext = { service: entry.name }
+    }
+  }
+
+  // Log message
+  if (entry.msg) {
+    if (gcpEntry.message) {
+      // A message has already been extracted from the error. Keep the extra
+      // message in the msg property.
+      gcpEntry.msg = entry.msg
+    } else {
+      gcpEntry.message = entry.msg
+    }
+  }
+
+  // Severity
+  if (entry.level) {
+    gcpEntry.severity = bunyan.nameFromLevel[entry.level]
+  }
+
+  // HTTP request information
+  if (entry.req || entry.res || entry.responseTimeMs) {
+    const httpRequest = {}
+    if (entry.req) {
+      const req = entry.req
+      httpRequest.requestMethod = req.method
+      httpRequest.requestUrl = req.url
+      httpRequest.remoteIp = req.remoteAddress
+      if (req.headers) {
+        if (req.headers['content-length']) {
+          httpRequest.requestSize = parseInt(req.headers['content-length'], 10)
+        }
+        httpRequest.userAgent = req.headers['user-agent']
+        httpRequest.referer = req.headers.referer
+      }
+    }
+
+    if (entry.res) {
+      const res = entry.res
+      httpRequest.status = res.statusCode
+      if (res.headers && res.headers['content-length']) {
+        if (res.headers['content-length']) {
+          httpRequest.responseSize = parseInt(res.headers['content-length'], 10)
+        }
+      }
+    }
+
+    if (entry.responseTimeMs) {
+      const responseTimeSec = entry.responseTimeMs / 1000
+      httpRequest.latency = `${responseTimeSec}s`
+    }
+    gcpEntry.httpRequest = httpRequest
+  }
+
+  // Labels are indexed in GCP. We copy the project, doc and user ids to labels to enable fast filtering
+  const projectId =
+    gcpEntry.projectId ||
+    gcpEntry.project_id ||
+    (entry.req && entry.req.projectId)
+  const userId =
+    gcpEntry.userId || gcpEntry.user_id || (entry.req && entry.req.userId)
+  const docId =
+    gcpEntry.docId || gcpEntry.doc_id || (entry.req && entry.req.docId)
+  if (projectId || userId || docId) {
+    const labels = {}
+    if (projectId) {
+      labels.projectId = projectId
+    }
+    if (userId) {
+      labels.userId = userId
+    }
+    if (docId) {
+      labels.docId = docId
+    }
+    gcpEntry['logging.googleapis.com/labels'] = labels
+  }
+
+  return gcpEntry
+}
+
+function omit(obj, excludedFields) {
+  return Object.fromEntries(
+    Object.entries(obj).filter(([key]) => !excludedFields.includes(key))
+  )
+}
+
+module.exports = { convertLogEntry }

libraries/logger/index.js

@@ -0,0 +1 @@
+module.exports = require('./logging-manager.js')

libraries/logger/log-level-checker.js

@@ -0,0 +1,60 @@
+const { fetchString } = require('@overleaf/fetch-utils')
+const fs = require('node:fs')
+
+class LogLevelChecker {
+  constructor(logger, defaultLevel) {
+    this.logger = logger
+    this.defaultLevel = defaultLevel
+  }
+
+  start() {
+    // check for log level override on startup
+    this.checkLogLevel()
+    // re-check log level every minute
+    this.checkInterval = setInterval(this.checkLogLevel.bind(this), 1000 * 60)
+    this.checkInterval.unref()
+  }
+
+  stop() {
+    clearInterval(this.checkInterval)
+  }
+
+  async checkLogLevel() {
+    try {
+      const end = await this.getTracingEndTime()
+      if (end > Date.now()) {
+        this.logger.level('trace')
+      } else {
+        this.logger.level(this.defaultLevel)
+      }
+    } catch (e) {
+      this.logger.level(this.defaultLevel)
+    }
+  }
+
+  async getTracingEndTime() {
+    return 0
+  }
+}
+
+class FileLogLevelChecker extends LogLevelChecker {
+  async getTracingEndTime() {
+    const strEndTime = await fs.promises.readFile('/logging/tracingEndTime')
+    return parseInt(strEndTime, 10)
+  }
+}
+
+class GCEMetadataLogLevelChecker extends LogLevelChecker {
+  async getTracingEndTime() {
+    const options = {
+      headers: {
+        'Metadata-Flavor': 'Google',
+      },
+    }
+    const uri = `http://metadata.google.internal/computeMetadata/v1/project/attributes/${this.logger.fields.name}-setLogLevelEndTime`
+    const strEndTime = await fetchString(uri, options)
+    return parseInt(strEndTime, 10)
+  }
+}
+
+module.exports = { FileLogLevelChecker, GCEMetadataLogLevelChecker }

libraries/logger/logging-manager.js

@@ -0,0 +1,186 @@
+const Stream = require('node:stream')
+const bunyan = require('bunyan')
+const GCPManager = require('./gcp-manager')
+const Serializers = require('./serializers')
+const {
+  FileLogLevelChecker,
+  GCEMetadataLogLevelChecker,
+} = require('./log-level-checker')
+
+const LoggingManager = {
+  /**
+   * @param {string} name - The name of the logger
+   */
+  initialize(name) {
+    this.isProduction =
+      (process.env.NODE_ENV || '').toLowerCase() === 'production'
+    const isTest = (process.env.NODE_ENV || '').toLowerCase() === 'test'
+    this.defaultLevel =
+      process.env.LOG_LEVEL ||
+      (this.isProduction ? 'info' : isTest ? 'fatal' : 'debug')
+    this.loggerName = name
+    this.logger = bunyan.createLogger({
+      name,
+      serializers: {
+        err: Serializers.err,
+        error: Serializers.err,
+        req: Serializers.req,
+        res: Serializers.res,
+      },
+      streams: [this._getOutputStreamConfig()],
+    })
+    this._setupRingBuffer()
+    this._setupLogLevelChecker()
+    return this
+  },
+
+  /**
+   * @param {Record<string, any>|string} attributes - Attributes to log (nice serialization for err, req, res)
+   * @param {string} [message] - Optional message
+   * @signature `debug(attributes, message)`
+   * @signature `debug(message)`
+   */
+  debug(attributes, message, ...args) {
+    return this.logger.debug(attributes, message, ...args)
+  },
+
+  /**
+   * @param {Record<string, any>|string} attributes - Attributes to log (nice serialization for err, req, res)
+   * @param {string} [message]
+   * @signature `info(attributes, message)`
+   * @signature `info(message)`
+   */
+  info(attributes, message, ...args) {
+    return this.logger.info(attributes, message, ...args)
+  },
+
+  /**
+   * @param {Record<string, any>} attributes - Attributes to log (nice serialization for err, req, res)
+   * @param {string} [message]
+   */
+  error(attributes, message, ...args) {
+    if (this.ringBuffer !== null && Array.isArray(this.ringBuffer.records)) {
+      attributes.logBuffer = this.ringBuffer.records.filter(function (record) {
+        return record.level !== 50
+      })
+    }
+    this.logger.error(attributes, message, ...Array.from(args))
+  },
+
+  /**
+   * Alias to the error method.
+   * @param {Record<string, any>} attributes - Attributes to log (nice serialization for err, req, res)
+   * @param {string} [message]
+   */
+  err(attributes, message, ...args) {
+    return this.error(attributes, message, ...args)
+  },
+
+  /**
+   * @param {Record<string, any>|string} attributes - Attributes to log (nice serialization for err, req, res)
+   * @param {string} [message]
+   * @signature `warn(attributes, message)`
+   * @signature `warn(message)`
+   */
+  warn(attributes, message, ...args) {
+    return this.logger.warn(attributes, message, ...args)
+  },
+
+  /**
+   * @param {Record<string, any>} attributes - Attributes to log (nice serialization for err, req, res)
+   * @param {string} [message]
+   */
+  fatal(attributes, message) {
+    this.logger.fatal(attributes, message)
+  },
+
+  _getOutputStreamConfig() {
+    switch (process.env.LOGGING_FORMAT) {
+      case 'gke': {
+        const stream = new Stream.Writable({
+          objectMode: true,
+          write(entry, encoding, callback) {
+            const gcpEntry = GCPManager.convertLogEntry(entry)
+            // eslint-disable-next-line no-console
+            console.log(JSON.stringify(gcpEntry, bunyan.safeCycles()))
+            setImmediate(callback)
+          },
+        })
+        return { level: this.defaultLevel, type: 'raw', stream }
+      }
+      case 'gce': {
+        const { LoggingBunyan } = require('@google-cloud/logging-bunyan')
+        return new LoggingBunyan({
+          logName: this.loggerName,
+          serviceContext: { service: this.loggerName },
+        }).stream(this.defaultLevel)
+      }
+      default: {
+        return { level: this.defaultLevel, stream: process.stdout }
+      }
+    }
+  },
+
+  _setupRingBuffer() {
+    this.ringBufferSize = parseInt(process.env.LOG_RING_BUFFER_SIZE) || 0
+    if (this.ringBufferSize > 0) {
+      this.ringBuffer = new bunyan.RingBuffer({ limit: this.ringBufferSize })
+      this.logger.addStream({
+        level: 'trace',
+        type: 'raw',
+        stream: this.ringBuffer,
+      })
+    } else {
+      this.ringBuffer = null
+    }
+  },
+
+  _setupLogLevelChecker() {
+    const logLevelSource = (
+      process.env.LOG_LEVEL_SOURCE || 'file'
+    ).toLowerCase()
+
+    if (this.logLevelChecker) {
+      this.logLevelChecker.stop()
+      this.logLevelChecker = null
+    }
+
+    if (this.isProduction) {
+      switch (logLevelSource) {
+        case 'file':
+          this.logLevelChecker = new FileLogLevelChecker(
+            this.logger,
+            this.defaultLevel
+          )
+          break
+        case 'gce_metadata':
+          this.logLevelChecker = new GCEMetadataLogLevelChecker(
+            this.logger,
+            this.defaultLevel
+          )
+          break
+        case 'none':
+          break
+        default:
+          // eslint-disable-next-line no-console
+          console.log(`Unrecognised log level source: ${logLevelSource}`)
+      }
+      if (this.logLevelChecker) {
+        this.logLevelChecker.start()
+      }
+    }
+  },
+}
+
+LoggingManager.initialize('default')
+
+function handleWarning(err) {
+  LoggingManager.warn({ err }, 'Warning details')
+}
+
+process.on('warning', handleWarning)
+LoggingManager.removeWarningHandler = () => {
+  process.off('warning', handleWarning)
+}
+
+module.exports = LoggingManager

libraries/logger/package.json

@@ -0,0 +1,39 @@
+{
+  "name": "@overleaf/logger",
+  "homepage": "www.overleaf.com",
+  "description": "A centralised logging system for Overleaf",
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/overleaf/overleaf"
+  },
+  "main": "index.js",
+  "license": "AGPL-3.0-only",
+  "version": "3.1.1",
+  "scripts": {
+    "test": "npm run lint && npm run format && npm run types:check && npm run test:unit",
+    "format": "prettier --list-different $PWD/'**/*.{js,cjs,ts}'",
+    "format:fix": "prettier --write $PWD/'**/*.{js,cjs,ts}'",
+    "lint": "eslint --ext .js --ext .cjs --ext .ts --max-warnings 0 --format unix .",
+    "lint:fix": "eslint --fix --ext .js --ext .cjs --ext .ts .",
+    "test:ci": "npm run test:unit",
+    "test:unit": "mocha --exit test/**/*.{js,cjs}",
+    "types:check": "tsc --noEmit"
+  },
+  "dependencies": {
+    "@google-cloud/logging-bunyan": "^5.1.0",
+    "@overleaf/fetch-utils": "*",
+    "@overleaf/o-error": "*",
+    "bunyan": "^1.8.14"
+  },
+  "devDependencies": {
+    "chai": "^4.3.6",
+    "mocha": "^11.1.0",
+    "sandboxed-module": "^2.0.4",
+    "sinon": "^9.2.4",
+    "sinon-chai": "^3.7.0",
+    "typescript": "^5.0.4"
+  },
+  "peerDependencies": {
+    "@overleaf/metrics": "*"
+  }
+}

libraries/logger/serializers.js

@@ -0,0 +1,83 @@
+const OError = require('@overleaf/o-error')
+
+function errSerializer(err) {
+  if (!err) {
+    return err
+  }
+  let message = err.message
+  if (err.path) {
+    // filter paths from the message to avoid duplicate errors with different path in message
+    // (e.g. errors from `fs` methods which have a path attribute)
+    message = message.replace(` '${err.path}'`, '')
+  }
+  return {
+    message,
+    name: err.name,
+    stack: OError.getFullStack(err),
+    info: OError.getFullInfo(err),
+    code: err.code,
+    signal: err.signal,
+    path: err.path,
+  }
+}
+
+function reqSerializer(req) {
+  if (!req) {
+    return req
+  }
+  const headers = req.headers || {}
+  const entry = {
+    method: req.method,
+    url: req.originalUrl || req.url,
+    remoteAddress: getRemoteIp(req),
+    headers: {
+      referer: headers.referer || headers.referrer,
+      'user-agent': headers['user-agent'],
+      'content-length': headers['content-length'],
+    },
+  }
+  if (req.params) {
+    const projectId =
+      req.params.projectId || req.params.project_id || req.params.Project_id
+    const userId = req.params.userId || req.params.user_id
+    const docId = req.params.docId || req.params.doc_id
+    if (projectId) {
+      entry.projectId = projectId
+    }
+    if (userId) {
+      entry.userId = userId
+    }
+    if (docId) {
+      entry.docId = docId
+    }
+  }
+  return entry
+}
+
+function resSerializer(res) {
+  if (!res) {
+    return res
+  }
+  return {
+    statusCode: res.statusCode,
+    headers: {
+      'content-length': res.getHeader && res.getHeader('content-length'),
+    },
+  }
+}
+
+function getRemoteIp(req) {
+  if (req.ip) {
+    return req.ip
+  }
+  if (req.socket) {
+    if (req.socket.socket && req.socket.socket.remoteAddress) {
+      return req.socket.socket.remoteAddress
+    } else if (req.socket.remoteAddress) {
+      return req.socket.remoteAddress
+    }
+  }
+  return null
+}
+
+module.exports = { err: errSerializer, req: reqSerializer, res: resSerializer }

libraries/logger/test/setup.js

@@ -0,0 +1,16 @@
+const chai = require('chai')
+const sinonChai = require('sinon-chai')
+const SandboxedModule = require('sandboxed-module')
+
+// Chai configuration
+chai.should()
+chai.use(sinonChai)
+
+SandboxedModule.configure({
+  globals: { Buffer, JSON, console, process },
+  sourceTransformers: {
+    removeNodePrefix: function (source) {
+      return source.replace(/require\(['"]node:/g, "require('")
+    },
+  },
+})

libraries/logger/test/unit/log-level-checker-tests.js

@@ -0,0 +1,186 @@
+const Path = require('node:path')
+const { promisify } = require('node:util')
+const SandboxedModule = require('sandboxed-module')
+const sinon = require('sinon')
+const { expect } = require('chai')
+
+const MODULE_PATH = Path.join(__dirname, '../../log-level-checker.js')
+const DEFAULT_LEVEL = 'warn'
+const TRACE_LEVEL = 'trace'
+const TRACING_END_TIME_FILE = '/logging/tracingEndTime'
+const NOW = 10000
+const PAST = NOW - 1000
+const FUTURE = NOW + 1000
+
+const delay = promisify(setTimeout)
+
+describe('LogLevelChecker', function () {
+  beforeEach(function () {
+    this.logger = {
+      level: sinon.stub(),
+      fields: { name: 'myapp' },
+    }
+    this.FetchUtils = {
+      fetchString: sinon.stub(),
+    }
+    this.fetchLogLevelEndTimeStub = this.FetchUtils.fetchString.withArgs(
+      'http://metadata.google.internal/computeMetadata/v1/project/attributes/myapp-setLogLevelEndTime',
+      { headers: { 'Metadata-Flavor': 'Google' } }
+    )
+    this.fetchLogLevelEndTimeStub.resolves('')
+
+    this.fs = {
+      promises: {
+        readFile: sinon.stub(),
+      },
+    }
+
+    this.clock = sinon.useFakeTimers(NOW)
+
+    this.module = SandboxedModule.require(MODULE_PATH, {
+      requires: {
+        '@overleaf/fetch-utils': this.FetchUtils,
+        fs: this.fs,
+      },
+    })
+  })
+
+  afterEach(function () {
+    this.clock.restore()
+  })
+
+  describe('FileLogLevelChecker', function () {
+    beforeEach(function () {
+      this.logLevelChecker = new this.module.FileLogLevelChecker(
+        this.logger,
+        DEFAULT_LEVEL
+      )
+    })
+
+    describe('when the file is empty', function () {
+      setupTracingEndTimeFile('')
+      checkLogLevel()
+      expectLevelSetTo(DEFAULT_LEVEL)
+    })
+
+    describe("when the file can't be read", function () {
+      beforeEach(async function () {
+        this.fs.promises.readFile.rejects(new Error('Read error!'))
+      })
+      checkLogLevel()
+      expectLevelSetTo(DEFAULT_LEVEL)
+    })
+
+    describe('when the file has a timestamp in the future', function () {
+      setupTracingEndTimeFile(FUTURE.toString())
+      checkLogLevel()
+      expectLevelSetTo(TRACE_LEVEL)
+    })
+
+    describe('when the file has a timestamp in the past', function () {
+      setupTracingEndTimeFile(PAST.toString())
+      checkLogLevel()
+      expectLevelSetTo(DEFAULT_LEVEL)
+    })
+
+    describe('interval checker', function () {
+      beforeEach(function () {
+        this.fs.promises.readFile.resolves('')
+        this.logLevelChecker.start()
+      })
+
+      afterEach(function () {
+        this.logLevelChecker.stop()
+      })
+
+      it('checks the file every minute', async function () {
+        this.clock.tick(1000)
+        // Yield to the event loop
+        await delay(0)
+        expect(this.logger.level).to.have.been.calledOnceWithExactly(
+          DEFAULT_LEVEL
+        )
+        this.logger.level.reset()
+
+        // Trace until 1.5 minutes in the future
+        const traceUntil = NOW + 90000
+        this.fs.promises.readFile.resolves(traceUntil.toString())
+
+        this.clock.tick(61000)
+        await delay(0)
+        expect(this.logger.level).to.have.been.calledOnceWithExactly(
+          TRACE_LEVEL
+        )
+        this.logger.level.reset()
+
+        this.clock.tick(60000)
+        await delay(0)
+        expect(this.logger.level).to.have.been.calledOnceWithExactly(
+          DEFAULT_LEVEL
+        )
+      })
+    })
+  })
+
+  describe('GCEMetadataLogLevelChecker', function () {
+    beforeEach(function () {
+      this.logLevelChecker = new this.module.GCEMetadataLogLevelChecker(
+        this.logger,
+        DEFAULT_LEVEL
+      )
+    })
+
+    describe('when the response is empty', function () {
+      setupTracingEndTimeGCE('')
+      checkLogLevel()
+      expectLevelSetTo(DEFAULT_LEVEL)
+    })
+
+    describe('when the request errors', function () {
+      beforeEach(async function () {
+        this.FetchUtils.fetchString.rejects(new Error('Read error!'))
+      })
+      checkLogLevel()
+      expectLevelSetTo(DEFAULT_LEVEL)
+    })
+
+    describe('when the response is a timestamp in the future', function () {
+      setupTracingEndTimeGCE(FUTURE.toString())
+      checkLogLevel()
+      expectLevelSetTo(TRACE_LEVEL)
+    })
+
+    describe('when the response is a timestamp in the past', function () {
+      setupTracingEndTimeGCE(PAST.toString())
+      checkLogLevel()
+      expectLevelSetTo(DEFAULT_LEVEL)
+    })
+  })
+})
+
+function setupTracingEndTimeFile(contents) {
+  beforeEach(`set tracing end time in file to ${contents}`, function () {
+    this.fs.promises.readFile.withArgs(TRACING_END_TIME_FILE).resolves(contents)
+  })
+}
+
+function setupTracingEndTimeGCE(contents) {
+  beforeEach(
+    `set tracing end time in GCE metadata to ${contents}`,
+    function () {
+      this.fetchLogLevelEndTimeStub.resolves(contents)
+    }
+  )
+}
+
+function checkLogLevel() {
+  beforeEach('Check log level', async function () {
+    await this.logLevelChecker.checkLogLevel()
+  })
+}
+
+function expectLevelSetTo(level) {
+  it(`sets the log level to ${level}`, function () {
+    expect(this.logger.level).to.have.been.calledWith(level)
+  })
+}

libraries/logger/test/unit/logging-manager-tests.js

@@ -0,0 +1,202 @@
+const SandboxedModule = require('sandboxed-module')
+const bunyan = require('bunyan')
+const { expect } = require('chai')
+const path = require('node:path')
+const sinon = require('sinon')
+
+const MODULE_PATH = path.join(__dirname, '../../logging-manager.js')
+
+describe('LoggingManager', function () {
+  beforeEach(function () {
+    this.start = Date.now()
+    this.bunyanLogger = {
+      addStream: sinon.stub(),
+      debug: sinon.stub(),
+      error: sinon.stub(),
+      fatal: sinon.stub(),
+      info: sinon.stub(),
+      level: sinon.stub(),
+      warn: sinon.stub(),
+    }
+    this.Bunyan = {
+      createLogger: sinon.stub().returns(this.bunyanLogger),
+      RingBuffer: bunyan.RingBuffer,
+    }
+    this.stackdriverStreamConfig = { stream: 'stackdriver' }
+    this.stackdriverClient = {
+      stream: sinon.stub().returns(this.stackdriverStreamConfig),
+    }
+    this.GCPLogging = {
+      LoggingBunyan: sinon.stub().returns(this.stackdriverClient),
+    }
+    this.FileLogLevelChecker = {
+      start: sinon.stub(),
+      stop: sinon.stub(),
+    }
+    this.GCEMetadataLogLevelChecker = {
+      start: sinon.stub(),
+      stop: sinon.stub(),
+    }
+    this.LogLevelChecker = {
+      FileLogLevelChecker: sinon.stub().returns(this.FileLogLevelChecker),
+      GCEMetadataLogLevelChecker: sinon
+        .stub()
+        .returns(this.GCEMetadataLogLevelChecker),
+    }
+    this.LoggingManager = SandboxedModule.require(MODULE_PATH, {
+      requires: {
+        bunyan: this.Bunyan,
+        './log-level-checker': this.LogLevelChecker,
+      },
+    })
+    this.loggerName = 'test'
+    this.logger = this.LoggingManager.initialize(this.loggerName)
+  })
+
+  afterEach(function () {
+    this.LoggingManager.removeWarningHandler()
+  })
+
+  describe('initialize', function () {
+    beforeEach(function () {
+      this.Bunyan.createLogger.reset()
+    })
+
+    describe('not in production', function () {
+      beforeEach(function () {
+        this.logger = this.LoggingManager.initialize(this.loggerName)
+      })
+
+      it('should default to log level debug', function () {
+        this.Bunyan.createLogger.firstCall.args[0].streams[0].level.should.equal(
+          'debug'
+        )
+      })
+
+      it('should not instantiate a log level checker', function () {
+        expect(this.LoggingManager.logLevelChecker).not.to.exist
+      })
+    })
+
+    describe('in production', function () {
+      beforeEach(function () {
+        process.env.NODE_ENV = 'production'
+        this.logger = this.LoggingManager.initialize(this.loggerName)
+      })
+
+      afterEach(function () {
+        delete process.env.NODE_ENV
+      })
+
+      it('should default to log level info', function () {
+        this.Bunyan.createLogger.firstCall.args[0].streams[0].level.should.equal(
+          'info'
+        )
+      })
+
+      it('should set up a file log level checker', function () {
+        expect(this.logger.logLevelChecker).to.equal(this.FileLogLevelChecker)
+        expect(this.FileLogLevelChecker.start).to.have.been.called
+      })
+    })
+
+    describe('when LOG_LEVEL set in env', function () {
+      beforeEach(function () {
+        process.env.LOG_LEVEL = 'trace'
+        this.LoggingManager.initialize()
+      })
+
+      afterEach(function () {
+        delete process.env.LOG_LEVEL
+      })
+
+      it('should use custom log level', function () {
+        this.Bunyan.createLogger.firstCall.args[0].streams[0].level.should.equal(
+          'trace'
+        )
+      })
+    })
+  })
+
+  describe('bunyan logging', function () {
+    beforeEach(function () {
+      this.logArgs = [{ foo: 'bar' }, 'foo', 'bar']
+    })
+
+    it('should log debug', function () {
+      this.logger.debug(this.logArgs)
+      this.bunyanLogger.debug.should.have.been.calledWith(this.logArgs)
+    })
+
+    it('should log error', function () {
+      this.logger.error(this.logArgs)
+      this.bunyanLogger.error.should.have.been.calledWith(this.logArgs)
+    })
+
+    it('should log fatal', function () {
+      this.logger.fatal(this.logArgs)
+      this.bunyanLogger.fatal.should.have.been.calledWith(this.logArgs)
+    })
+
+    it('should log info', function () {
+      this.logger.info(this.logArgs)
+      this.bunyanLogger.info.should.have.been.calledWith(this.logArgs)
+    })
+
+    it('should log warn', function () {
+      this.logger.warn(this.logArgs)
+      this.bunyanLogger.warn.should.have.been.calledWith(this.logArgs)
+    })
+
+    it('should log err', function () {
+      this.logger.err(this.logArgs)
+      this.bunyanLogger.error.should.have.been.calledWith(this.logArgs)
+    })
+  })
+
+  describe('ringbuffer', function () {
+    beforeEach(function () {
+      this.logBufferMock = [
+        { msg: 'log 1' },
+        { msg: 'log 2' },
+        { level: 50, msg: 'error' },
+      ]
+    })
+
+    describe('when ring buffer size is positive', function () {
+      beforeEach(function () {
+        process.env.LOG_RING_BUFFER_SIZE = '20'
+        this.logger = this.LoggingManager.initialize(this.loggerName)
+        this.logger.ringBuffer.records = this.logBufferMock
+        this.logger.error({}, 'error')
+      })
+
+      afterEach(function () {
+        process.env.LOG_RING_BUFFER_SIZE = undefined
+      })
+
+      it('should include buffered logs in error log and filter out error logs in buffer', function () {
+        this.bunyanLogger.error.lastCall.args[0].logBuffer.should.deep.equal([
+          { msg: 'log 1' },
+          { msg: 'log 2' },
+        ])
+      })
+    })
+
+    describe('when ring buffer size is zero', function () {
+      beforeEach(function () {
+        process.env.LOG_RING_BUFFER_SIZE = '0'
+        this.logger = this.LoggingManager.initialize(this.loggerName)
+        this.logger.error({}, 'error')
+      })
+
+      afterEach(function () {
+        process.env.LOG_RING_BUFFER_SIZE = undefined
+      })
+
+      it('should not include buffered logs in error log', function () {
+        expect(this.bunyanLogger.error.lastCall.args[0].logBuffer).be.undefined
+      })
+    })
+  })
+})

libraries/logger/tsconfig.json

@@ -0,0 +1,7 @@
+{
+  "extends": "../../tsconfig.backend.json",
+  "include": [
+    "**/*.js",
+    "**/*.cjs"
+  ]
+}

libraries/metrics/.gitignore

@@ -0,0 +1,3 @@
+node_modules
+
+.npmrc

libraries/metrics/.npmignore

@@ -0,0 +1,4 @@
+/.circleci
+/.eslintrc
+/.nvmrc
+/.prettierrc

libraries/metrics/.nvmrc

@@ -0,0 +1 @@
+20.18.2

libraries/metrics/CHANGELOG.md

@@ -0,0 +1,9 @@
+## v4.1.0
+
+* Allows skipping the `sampleRate` argument.
+
+## v4.0.0
+
+* Send unmodified request and response to logger.
+
+This version of the metrics module only works with versions of the `@overleaf/logger` module greater than v3.0.0

libraries/metrics/LICENSE

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2024 Overleaf
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

libraries/metrics/README.md

@@ -0,0 +1,33 @@
+# overleaf/metrics-module
+
+Wrappers the [prom-client](https://github.com/siimon/prom-client) npm module to provide [Prometheus](https://prometheus.io/) metrics at `/metrics`.
+
+Use:
+
+```
+// Metrics must be initialized before importing anything else
+require('@overleaf/metrics/initialize')
+
+const express = require('express')
+const metrics = require('@overleaf/metrics')
+const app = express()
+metrics.injectMetricsRoute(app)
+```
+
+Request logging can be enabled:
+
+```
+const logger = require('@overleaf/logger')
+...
+app.use(metrics.http.monitor(logger))
+```
+
+The metrics module can be configured through the following environment variables:
+
+- `DEBUG_METRICS` - enables display of debugging messages to the console.
+- `GCP_OPENTELEMETRY` - enables OpenTelemetry tracing for GCP
+- `JAEGER_OPENTELEMETRY` - enables OpenTelemetry tracing for Jaeger (in the dev environment)
+- `METRICS_APP_NAME` - the app label for metrics and spans
+- `METRICS_COMPRESSION_LEVEL` - sets the [compression level](https://www.npmjs.com/package/compression#level) for `/metrics`
+- `STACKDRIVER_LOGGING` - toggles the request logging format
+- `UV_THREADPOOL_SIZE` - sets the libuv [thread pool](http://docs.libuv.org/en/v1.x/threadpool.html) size

libraries/metrics/buildscript.txt

@@ -0,0 +1,10 @@
+metrics
+--dependencies=None
+--docker-repos=gcr.io/overleaf-ops
+--env-add=
+--env-pass-through=
+--esmock-loader=False
+--is-library=True
+--node-version=20.18.2
+--public-repo=False
+--script-version=4.5.0

libraries/metrics/event_loop.js

@@ -0,0 +1,34 @@
+/*
+ * decaffeinate suggestions:
+ * DS102: Remove unnecessary code created because of implicit returns
+ * DS207: Consider shorter variations of null checks
+ * Full docs: https://github.com/decaffeinate/decaffeinate/blob/master/docs/suggestions.md
+ */
+module.exports = {
+  monitor(logger, interval, logThreshold) {
+    if (interval == null) {
+      interval = 1000
+    }
+    if (logThreshold == null) {
+      logThreshold = 100
+    }
+    const Metrics = require('./index')
+    // check for logger on startup to avoid exceptions later if undefined
+    if (logger == null) {
+      throw new Error('logger is undefined')
+    }
+    // monitor delay in setInterval to detect event loop blocking
+    let previous = Date.now()
+    const intervalId = setInterval(function () {
+      const now = Date.now()
+      const offset = now - previous - interval
+      if (offset > logThreshold) {
+        logger.warn({ offset }, 'slow event loop')
+      }
+      previous = now
+      return Metrics.timing('event-loop-millsec', offset)
+    }, interval)
+
+    return Metrics.registerDestructor(() => clearInterval(intervalId))
+  },
+}

libraries/metrics/http.js

@@ -0,0 +1,78 @@
+const Metrics = require('./index')
+
+function monitor(logger, level = 'debug') {
+  return function (req, res, next) {
+    const startTime = Date.now()
+    req.logger = new RequestLogger(logger, level)
+    const { end } = res
+    res.end = function (...args) {
+      end.apply(this, args)
+      const responseTimeMs = Date.now() - startTime
+      const requestSize = parseInt(req.headers['content-length'], 10)
+      const routePath = getRoutePath(req)
+
+      if (routePath != null) {
+        Metrics.timing('http_request', responseTimeMs, null, {
+          method: req.method,
+          status_code: res.statusCode,
+          path: routePath,
+        })
+        if (requestSize) {
+          Metrics.summary('http_request_size_bytes', requestSize, {
+            method: req.method,
+            status_code: res.statusCode,
+            path: routePath,
+          })
+        }
+      }
+      req.logger.addFields({ responseTimeMs })
+      req.logger.emit(req, res)
+    }
+    next()
+  }
+}
+
+function getRoutePath(req) {
+  if (req.route && req.route.path != null) {
+    return req.route.path
+      .toString()
+      .replace(/\//g, '_')
+      .replace(/:/g, '')
+      .slice(1)
+  }
+  if (req.swagger && req.swagger.apiPath != null) {
+    return req.swagger.apiPath
+  }
+  return null
+}
+
+class RequestLogger {
+  constructor(logger, level) {
+    this._logger = logger
+    this._level = level
+    this._info = {}
+  }
+
+  addFields(fields) {
+    Object.assign(this._info, fields)
+  }
+
+  setLevel(level) {
+    this._level = level
+  }
+
+  disable() {
+    this._disabled = true
+  }
+
+  emit(req, res) {
+    if (this._disabled) {
+      return
+    }
+    this.addFields({ req, res })
+    const url = req.originalUrl || req.url
+    this._logger[this._level](this._info, '%s %s', req.method, url)
+  }
+}
+
+module.exports.monitor = monitor

libraries/metrics/index.js

@@ -0,0 +1,182 @@
+/* eslint-disable no-console */
+
+const ExpressCompression = require('compression')
+const promClient = require('prom-client')
+const promWrapper = require('./prom_wrapper')
+
+const destructors = []
+
+require('./uv_threadpool_size')
+
+function registerDestructor(func) {
+  destructors.push(func)
+}
+
+function injectMetricsRoute(app) {
+  app.get(
+    '/metrics',
+    ExpressCompression({
+      level: parseInt(process.env.METRICS_COMPRESSION_LEVEL || '1', 10),
+    }),
+    function (req, res, next) {
+      res.set('Content-Type', promWrapper.registry.contentType)
+      promWrapper.registry
+        .metrics()
+        .then(metrics => {
+          res.end(metrics)
+        })
+        .catch(err => {
+          next(err)
+        })
+    }
+  )
+}
+
+function buildPromKey(key) {
+  return key.replace(/[^a-zA-Z0-9]/g, '_')
+}
+
+function sanitizeValue(value) {
+  return parseFloat(value)
+}
+
+function set(key, value, sampleRate = 1) {
+  console.log('counts are not currently supported')
+}
+
+function inc(key, sampleRate = 1, labels = {}) {
+  if (arguments.length === 2 && typeof sampleRate === 'object') {
+    labels = sampleRate
+  }
+
+  key = buildPromKey(key)
+  promWrapper.metric('counter', key, labels).inc(labels)
+  if (process.env.DEBUG_METRICS) {
+    console.log('doing inc', key, labels)
+  }
+}
+
+function count(key, count, sampleRate = 1, labels = {}) {
+  if (arguments.length === 3 && typeof sampleRate === 'object') {
+    labels = sampleRate
+  }
+
+  key = buildPromKey(key)
+  promWrapper.metric('counter', key, labels).inc(labels, count)
+  if (process.env.DEBUG_METRICS) {
+    console.log('doing count/inc', key, labels)
+  }
+}
+
+function summary(key, value, labels = {}) {
+  key = buildPromKey(key)
+  promWrapper.metric('summary', key, labels).observe(labels, value)
+  if (process.env.DEBUG_METRICS) {
+    console.log('doing summary', key, value, labels)
+  }
+}
+
+function timing(key, timeSpan, sampleRate = 1, labels = {}) {
+  if (arguments.length === 3 && typeof sampleRate === 'object') {
+    labels = sampleRate
+  }
+
+  key = buildPromKey('timer_' + key)
+  promWrapper.metric('summary', key, labels).observe(labels, timeSpan)
+  if (process.env.DEBUG_METRICS) {
+    console.log('doing timing', key, labels)
+  }
+}
+
+function histogram(key, value, buckets, labels = {}) {
+  key = buildPromKey('histogram_' + key)
+  promWrapper.metric('histogram', key, labels, buckets).observe(labels, value)
+  if (process.env.DEBUG_METRICS) {
+    console.log('doing histogram', key, buckets, labels)
+  }
+}
+
+class Timer {
+  constructor(key, sampleRate = 1, labels = {}, buckets = undefined) {
+    if (typeof sampleRate === 'object') {
+      // called with (key, labels, buckets)
+      if (arguments.length === 3) {
+        buckets = labels
+        labels = sampleRate
+      }
+
+      // called with (key, labels)
+      if (arguments.length === 2) {
+        labels = sampleRate
+      }
+
+      sampleRate = 1 // default value to pass to timing function
+    }
+
+    this.start = new Date()
+    key = buildPromKey(key)
+    this.key = key
+    this.sampleRate = sampleRate
+    this.labels = labels
+    this.buckets = buckets
+  }
+
+  // any labels passed into the done method override labels from constructor
+  done(labels = {}) {
+    const timeSpan = new Date() - this.start
+    if (this.buckets) {
+      histogram(this.key, timeSpan, this.buckets, { ...this.labels, ...labels })
+    } else {
+      timing(this.key, timeSpan, this.sampleRate, { ...this.labels, ...labels })
+    }
+    return timeSpan
+  }
+}
+
+function gauge(key, value, sampleRate = 1, labels = {}) {
+  if (arguments.length === 3 && typeof sampleRate === 'object') {
+    labels = sampleRate
+  }
+
+  key = buildPromKey(key)
+  promWrapper.metric('gauge', key, labels).set(labels, sanitizeValue(value))
+  if (process.env.DEBUG_METRICS) {
+    console.log('doing gauge', key, labels)
+  }
+}
+
+function globalGauge(key, value, sampleRate = 1, labels = {}) {
+  key = buildPromKey(key)
+  labels = { host: 'global', ...labels }
+  promWrapper.metric('gauge', key, labels).set(labels, sanitizeValue(value))
+}
+
+function close() {
+  for (const func of destructors) {
+    func()
+  }
+}
+
+module.exports.registerDestructor = registerDestructor
+module.exports.injectMetricsRoute = injectMetricsRoute
+module.exports.buildPromKey = buildPromKey
+module.exports.sanitizeValue = sanitizeValue
+module.exports.set = set
+module.exports.inc = inc
+module.exports.count = count
+module.exports.summary = summary
+module.exports.timing = timing
+module.exports.histogram = histogram
+module.exports.Timer = Timer
+module.exports.gauge = gauge
+module.exports.globalGauge = globalGauge
+module.exports.close = close
+module.exports.prom = promClient
+module.exports.register = promWrapper.registry
+
+module.exports.http = require('./http')
+module.exports.open_sockets = require('./open_sockets')
+module.exports.leaked_sockets = require('./leaked_sockets')
+module.exports.event_loop = require('./event_loop')
+module.exports.memory = require('./memory')
+module.exports.mongodb = require('./mongodb')

libraries/metrics/initialize.js

@@ -0,0 +1,105 @@
+/* eslint-disable no-console */
+
+/**
+ * This module initializes the metrics module. It should be imported once
+ * before any other module to support code instrumentation.
+ */
+
+const APP_NAME = process.env.METRICS_APP_NAME || 'unknown'
+const BUILD_VERSION = process.env.BUILD_VERSION
+const ENABLE_PROFILE_AGENT = process.env.ENABLE_PROFILE_AGENT === 'true'
+const GCP_OPENTELEMETRY = process.env.GCP_OPENTELEMETRY === 'true'
+const JAEGER_OPENTELEMETRY = process.env.JAEGER_OPENTELEMETRY === 'true'
+
+console.log('Initializing metrics')
+
+if (GCP_OPENTELEMETRY || JAEGER_OPENTELEMETRY) {
+  initializeOpenTelemetryInstrumentation()
+  initializeOpenTelemetryLogging()
+}
+
+if (ENABLE_PROFILE_AGENT) {
+  initializeProfileAgent()
+}
+
+initializePrometheus()
+initializePromWrapper()
+recordProcessStart()
+
+function initializeOpenTelemetryInstrumentation() {
+  console.log('Starting OpenTelemetry instrumentation')
+  const opentelemetry = require('@opentelemetry/sdk-node')
+  const {
+    getNodeAutoInstrumentations,
+  } = require('@opentelemetry/auto-instrumentations-node')
+  const { Resource } = require('@opentelemetry/resources')
+  const {
+    SemanticResourceAttributes,
+  } = require('@opentelemetry/semantic-conventions')
+
+  const resource = new Resource({
+    [SemanticResourceAttributes.SERVICE_NAME]: APP_NAME,
+    [SemanticResourceAttributes.SERVICE_NAMESPACE]: 'Overleaf',
+    'host.type': 'VM',
+  })
+
+  let exporter
+  if (GCP_OPENTELEMETRY) {
+    const GCP = require('@google-cloud/opentelemetry-cloud-trace-exporter')
+    exporter = new GCP.TraceExporter()
+  } else if (JAEGER_OPENTELEMETRY) {
+    const {
+      OTLPTraceExporter,
+    } = require('@opentelemetry/exporter-trace-otlp-http')
+    exporter = new OTLPTraceExporter({
+      url: `http://${process.env.JAEGER_HOST || 'jaeger'}:4318/v1/traces`,
+    })
+  } else {
+    return
+  }
+
+  const sdk = new opentelemetry.NodeSDK({
+    traceExporter: exporter,
+    logger: console,
+    instrumentations: [getNodeAutoInstrumentations()],
+    resource,
+  })
+  sdk.start()
+}
+
+function initializeOpenTelemetryLogging() {
+  const {
+    diag,
+    DiagConsoleLogger,
+    DiagLogLevel,
+  } = require('@opentelemetry/api')
+  diag.setLogger(new DiagConsoleLogger(), DiagLogLevel.INFO)
+}
+
+function initializeProfileAgent() {
+  console.log('Starting Google Profile Agent')
+  const profiler = require('@google-cloud/profiler')
+  profiler.start({
+    serviceContext: {
+      service: APP_NAME,
+      version: BUILD_VERSION,
+    },
+  })
+}
+
+function initializePrometheus() {
+  const os = require('node:os')
+  const promClient = require('prom-client')
+  promClient.register.setDefaultLabels({ app: APP_NAME, host: os.hostname() })
+  promClient.collectDefaultMetrics({ timeout: 5000, prefix: '' })
+}
+
+function initializePromWrapper() {
+  const promWrapper = require('./prom_wrapper')
+  promWrapper.setupSweeping()
+}
+
+function recordProcessStart() {
+  const metrics = require('.')
+  metrics.inc('process_startup')
+}

libraries/metrics/leaked_sockets.js

@@ -0,0 +1,254 @@
+/**
+ * This file monitors HTTP connections in Node.js and logs any potential socket leaks.
+ * It uses the `diagnostics_channel` module to intercept requests and reponses in the
+ * `http` module and tracks the lifetime of each http socket. If a socket is open for
+ * longer than a specified time, it is considered a potential leak and its details are
+ * logged along with the corresponding information from /proc/net/tcp.
+ */
+
+const fs = require('node:fs')
+const diagnosticsChannel = require('node:diagnostics_channel')
+
+const SOCKET_MONITOR_INTERVAL = 60 * 1000
+// set the threshold for logging leaked sockets in minutes, defaults to 15
+const MIN_SOCKET_LEAK_TIME =
+  (parseInt(process.env.LEAKED_SOCKET_AGE_THRESHOLD, 10) || 15) * 60 * 1000
+
+// Record HTTP events using diagnostics_channel
+diagnosticsChannel.subscribe('http.client.request.start', handleRequest)
+diagnosticsChannel.subscribe('http.server.request.start', handleRequest)
+diagnosticsChannel.subscribe('http.client.response.finish', handleResponse)
+diagnosticsChannel.subscribe('http.server.response.finish', handleResponse)
+
+function handleRequest({ request: req }) {
+  const socket = req?.socket
+  if (socket) {
+    recordRequest(req, socket)
+  }
+}
+
+function recordRequest(req, socket) {
+  const { method, protocol, path, url, rawHeaders, _header } = req
+  socket._ol_debug = {
+    method,
+    protocol,
+    url: url ?? path,
+    request: { headers: rawHeaders ?? _header, ts: new Date() },
+  }
+}
+
+function handleResponse({ request: req, response: res }) {
+  const socket = req?.socket || res?.socket
+  if (!socket || !res) {
+    return
+  }
+  if (!socket._ol_debug) {
+    // I don't know if this will ever happen, but if we missed the request,
+    // record it here.
+    recordRequest(req, socket)
+  }
+  const { statusCode, statusMessage, headers, _header } = res
+  Object.assign(socket._ol_debug, {
+    response: {
+      statusCode,
+      statusMessage,
+      headers: headers ?? _header,
+      ts: new Date(),
+    },
+  })
+}
+
+// Additional functions to log request headers with sensitive information redacted
+
+function flattenHeaders(rawHeaders) {
+  // Headers can be an array [KEY, VALUE, KEY, VALUE, ..]
+  // an object {key:value, key:value, ...}
+  // or a string of the headers separated by \r\n
+  // Flatten the array and object headers into the string form.
+  if (Array.isArray(rawHeaders)) {
+    return rawHeaders
+      .map((item, index) => (index % 2 === 0 ? `${item}: ` : `${item}\r\n`))
+      .join('')
+  } else if (typeof rawHeaders === 'object') {
+    return Object.entries(rawHeaders)
+      .map(([key, value]) => `${key}: ${value}\r\n`)
+      .join('')
+  } else if (typeof rawHeaders === 'string') {
+    return rawHeaders
+  } else {
+    return JSON.stringify(rawHeaders)
+  }
+}
+
+const REDACT_REGEX = /^(Authorization|Set-Cookie|Cookie):.*?\r/gim
+
+function redactObject(obj) {
+  const result = {}
+  for (const [key, value] of Object.entries(obj)) {
+    if (value == null) {
+      result[key] = null
+    } else if (key === 'headers') {
+      // remove headers with sensitive information
+      result[key] = flattenHeaders(value).replace(
+        REDACT_REGEX,
+        `$1: REDACTED\r`
+      )
+    } else if (
+      typeof value === 'object' &&
+      ['request', 'response'].includes(key)
+    ) {
+      result[key] = redactObject(value)
+    } else {
+      result[key] = value
+    }
+  }
+  return result
+}
+
+// Check if an old socket has crossed the threshold for logging.
+// We log multiple times with an exponential backoff so we can
+// see how long a socket hangs around.
+
+function isOldSocket(handle) {
+  const now = new Date()
+  const created = handle._ol_debug.request.ts
+  const lastLoggedAt = handle._ol_debug.lastLoggedAt ?? created
+  const nextLogTime = new Date(
+    created.getTime() +
+      Math.max(2 * (lastLoggedAt - created), MIN_SOCKET_LEAK_TIME)
+  )
+  return now >= nextLogTime
+}
+
+function logOldSocket(logger, handle, tcpinfo) {
+  const now = new Date()
+  const info = Object.assign(
+    {
+      localAddress: handle.localAddress,
+      localPort: handle.localPort,
+      remoteAddress: handle.remoteAddress,
+      remotePort: handle.remotePort,
+      tcpinfo,
+      age: Math.floor((now - handle._ol_debug.request.ts) / (60 * 1000)), // age in minutes
+    },
+    redactObject(handle._ol_debug)
+  )
+  handle._ol_debug.lastLoggedAt = now
+  if (tcpinfo) {
+    logger.error(info, 'old socket handle - tcp socket')
+  } else {
+    logger.warn(info, 'stale socket handle - no entry in /proc/net/tcp')
+  }
+}
+
+// Correlate socket handles with /proc/net/tcp entries using a key based on the
+// local and remote addresses and ports. This will allow us to distinguish between
+// sockets that are still open and sockets that have been closed and removed from
+// the /proc/net/tcp table but are still present in the node active handles array.
+
+async function getOpenSockets() {
+  // get open sockets remote and local address:port from /proc/net/tcp
+  const procNetTcp = '/proc/net/tcp'
+  const openSockets = new Map()
+  const lines = await fs.promises.readFile(procNetTcp, 'utf8')
+  for (const line of lines.split('\n')) {
+    const socket = parseProcNetTcp(line)
+    if (socket) {
+      openSockets.set(socket, line)
+    }
+  }
+  return openSockets
+}
+
+function keyFromSocket(socket) {
+  return `${socket.localAddress}:${socket.localPort} -> ${socket.remoteAddress}:${socket.remotePort}`
+}
+
+function decodeHexIpAddress(hex) {
+  // decode hex ip address to dotted decimal notation
+  const ip = parseInt(hex, 16)
+  const a = ip & 0xff
+  const b = (ip >> 8) & 0xff
+  const c = (ip >> 16) & 0xff
+  const d = (ip >> 24) & 0xff
+  return `${a}.${b}.${c}.${d}`
+}
+
+function decodeHexPort(hex) {
+  // decode hex port to decimal
+  return parseInt(hex, 16)
+}
+
+// Regex for extracting the local and remote addresses and ports from the /proc/net/tcp output
+// Example line:
+// 16: AB02A8C0:D9E2 86941864:01BB 01 00000000:00000000 02:000004BE 00000000     0        0 36802 2 0000000000000000 28 4 26 10 -1
+//     ^^^^^^^^^^^^^ ^^^^^^^^^^^^^
+//     local          remote
+
+const TCP_STATE_REGEX =
+  /^\s*\d+:\s+(?<localHexAddress>[0-9A-F]{8}):(?<localHexPort>[0-9A-F]{4})\s+(?<remoteHexAddress>[0-9A-F]{8}):(?<remoteHexPort>[0-9A-F]{4})/i
+
+function parseProcNetTcp(line) {
+  const match = line.match(TCP_STATE_REGEX)
+  if (match) {
+    const { localHexAddress, localHexPort, remoteHexAddress, remoteHexPort } =
+      match.groups
+    return keyFromSocket({
+      localAddress: decodeHexIpAddress(localHexAddress),
+      localPort: decodeHexPort(localHexPort),
+      remoteAddress: decodeHexIpAddress(remoteHexAddress),
+      remotePort: decodeHexPort(remoteHexPort),
+    })
+  }
+}
+
+let LeakedSocketsMonitor
+
+// Export the monitor and scanSockets functions
+
+module.exports = LeakedSocketsMonitor = {
+  monitor(logger) {
+    const interval = setInterval(
+      () => LeakedSocketsMonitor.scanSockets(logger),
+      SOCKET_MONITOR_INTERVAL
+    )
+    const Metrics = require('./index')
+    return Metrics.registerDestructor(() => clearInterval(interval))
+  },
+  scanSockets(logger) {
+    const debugSockets = process._getActiveHandles().filter(handle => {
+      return handle._ol_debug
+    })
+
+    // Bail out if there are no sockets with the _ol_debug property
+    if (debugSockets.length === 0) {
+      return
+    }
+
+    const oldSockets = debugSockets.filter(isOldSocket)
+
+    // Bail out if there are no old sockets to log
+    if (oldSockets.length === 0) {
+      return
+    }
+
+    // If there old sockets to log, get the connections from /proc/net/tcp
+    // to distinguish between sockets that are still open and sockets that
+    // have been closed and removed from the /proc/net/tcp table.
+    getOpenSockets()
+      .then(openSockets => {
+        oldSockets.forEach(handle => {
+          try {
+            const key = keyFromSocket(handle)
+            const tcpinfo = openSockets.get(key)
+            logOldSocket(logger, handle, tcpinfo)
+          } catch (err) {
+            logger.error({ err }, 'error in scanSockets')
+          }
+        })
+      })
+      .catch(err => {
+        logger.error({ err }, 'error getting open sockets')
+      })
+  },
+}

libraries/metrics/memory.js

@@ -0,0 +1,113 @@
+/*
+ * decaffeinate suggestions:
+ * DS102: Remove unnecessary code created because of implicit returns
+ * DS207: Consider shorter variations of null checks
+ * Full docs: https://github.com/decaffeinate/decaffeinate/blob/master/docs/suggestions.md
+ */
+// record memory usage each minute and run a periodic gc(), keeping cpu
+// usage within allowable range of 1ms per minute. Also, dynamically
+// adjust the period between gc()'s to reach a target of the gc saving
+// 4 megabytes each time.
+
+let MemoryMonitor
+const oneMinute = 60 * 1000
+const oneMegaByte = 1024 * 1024
+
+let CpuTimeBucket = 100 // current cpu time allowance in milliseconds
+const CpuTimeBucketMax = 100 // maximum amount of cpu time allowed in bucket
+const CpuTimeBucketRate = 10 // add this many milliseconds per minute
+
+let gcInterval = 1 // how many minutes between gc (parameter is dynamically adjusted)
+let countSinceLastGc = 0 // how many minutes since last gc
+const MemoryChunkSize = 4 // how many megabytes we need to free to consider gc worth doing
+
+const readyToGc = function () {
+  // update allowed cpu time
+  CpuTimeBucket = CpuTimeBucket + CpuTimeBucketRate
+  CpuTimeBucket =
+    CpuTimeBucket < CpuTimeBucketMax ? CpuTimeBucket : CpuTimeBucketMax
+  // update counts since last gc
+  countSinceLastGc = countSinceLastGc + 1
+  // check there is enough time since last gc and we have enough cpu
+  return countSinceLastGc > gcInterval && CpuTimeBucket > 0
+}
+
+const executeAndTime = function (fn) {
+  // time the execution of fn() and subtract from cpu allowance
+  const t0 = process.hrtime()
+  fn()
+  const dt = process.hrtime(t0)
+  const timeTaken = (dt[0] + dt[1] * 1e-9) * 1e3 // in milliseconds
+  CpuTimeBucket -= Math.ceil(timeTaken)
+  return timeTaken
+}
+
+const inMegaBytes = function (obj) {
+  // convert process.memoryUsage hash {rss,heapTotal,heapFreed} into megabytes
+  const result = {}
+  for (const k in obj) {
+    const v = obj[k]
+    result[k] = (v / oneMegaByte).toFixed(2)
+  }
+  return result
+}
+
+const updateMemoryStats = function (oldMem, newMem) {
+  countSinceLastGc = 0
+  const delta = {}
+  for (const k in newMem) {
+    delta[k] = (newMem[k] - oldMem[k]).toFixed(2)
+  }
+  // take the max of all memory measures
+  const savedMemory = Math.max(-delta.rss, -delta.heapTotal, -delta.heapUsed)
+  delta.megabytesFreed = savedMemory
+  // did it do any good?
+  if (savedMemory < MemoryChunkSize) {
+    gcInterval = gcInterval + 1 // no, so wait longer next time
+  } else {
+    gcInterval = Math.max(gcInterval - 1, 1) // yes, wait less time
+  }
+  return delta
+}
+
+module.exports = MemoryMonitor = {
+  monitor(logger) {
+    const interval = setInterval(() => MemoryMonitor.Check(logger), oneMinute)
+    const Metrics = require('./index')
+    return Metrics.registerDestructor(() => clearInterval(interval))
+  },
+
+  Check(logger) {
+    let mem
+    const Metrics = require('./index')
+    const memBeforeGc = (mem = inMegaBytes(process.memoryUsage()))
+    Metrics.gauge('memory.rss', mem.rss)
+    Metrics.gauge('memory.heaptotal', mem.heapTotal)
+    Metrics.gauge('memory.heapused', mem.heapUsed)
+    Metrics.gauge('memory.gc-interval', gcInterval)
+    // Metrics.gauge("memory.cpu-time-bucket", CpuTimeBucket)
+
+    logger.debug(mem, 'process.memoryUsage()')
+
+    if (global.gc != null && readyToGc()) {
+      const gcTime = executeAndTime(global.gc).toFixed(2)
+      const memAfterGc = inMegaBytes(process.memoryUsage())
+      const deltaMem = updateMemoryStats(memBeforeGc, memAfterGc)
+      logger.debug(
+        {
+          gcTime,
+          memBeforeGc,
+          memAfterGc,
+          deltaMem,
+          gcInterval,
+          CpuTimeBucket,
+        },
+        'global.gc() forced'
+      )
+      // Metrics.timing("memory.gc-time", gcTime)
+      Metrics.gauge('memory.gc-rss-freed', -deltaMem.rss)
+      Metrics.gauge('memory.gc-heaptotal-freed', -deltaMem.heapTotal)
+      return Metrics.gauge('memory.gc-heapused-freed', -deltaMem.heapUsed)
+    }
+  },
+}

libraries/metrics/mongodb.js

@@ -0,0 +1,84 @@
+const { Gauge, Summary } = require('prom-client')
+
+function monitor(mongoClient) {
+  const labelNames = ['mongo_server']
+  const poolSize = new Gauge({
+    name: 'mongo_connection_pool_size',
+    help: 'number of connections in the connection pool',
+    labelNames,
+    // Use this one metric's collect() to set all metrics' values.
+    collect,
+  })
+  const availableConnections = new Gauge({
+    name: 'mongo_connection_pool_available',
+    help: 'number of connections that are not busy',
+    labelNames,
+  })
+  const waitQueueSize = new Gauge({
+    name: 'mongo_connection_pool_waiting',
+    help: 'number of operations waiting for an available connection',
+    labelNames,
+  })
+  const maxPoolSize = new Gauge({
+    name: 'mongo_connection_pool_max',
+    help: 'max size for the connection pool',
+    labelNames,
+  })
+
+  const mongoCommandTimer = new Summary({
+    name: 'mongo_command_time',
+    help: 'time taken to complete a mongo command',
+    percentiles: [],
+    labelNames: ['status', 'method'],
+  })
+
+  if (mongoClient.on) {
+    mongoClient.on('commandSucceeded', event => {
+      mongoCommandTimer.observe(
+        {
+          status: 'success',
+          method: event.commandName === 'find' ? 'read' : 'write',
+        },
+        event.duration
+      )
+    })
+
+    mongoClient.on('commandFailed', event => {
+      mongoCommandTimer.observe(
+        {
+          status: 'failed',
+          method: event.commandName === 'find' ? 'read' : 'write',
+        },
+        event.duration
+      )
+    })
+  }
+
+  function collect() {
+    // Reset all gauges in case they contain values for servers that
+    // disappeared
+    poolSize.reset()
+    availableConnections.reset()
+    waitQueueSize.reset()
+    maxPoolSize.reset()
+
+    const servers = mongoClient.topology?.s?.servers
+    if (servers != null) {
+      for (const [address, server] of servers) {
+        // The server object is different between v4 and v5 (c.f. https://github.com/mongodb/node-mongodb-native/pull/3645)
+        const pool = server.s?.pool || server.pool
+        if (pool == null) {
+          continue
+        }
+
+        const labels = { mongo_server: address }
+        poolSize.set(labels, pool.totalConnectionCount)
+        availableConnections.set(labels, pool.availableConnectionCount)
+        waitQueueSize.set(labels, pool.waitQueueSize)
+        maxPoolSize.set(labels, pool.options.maxPoolSize)
+      }
+    }
+  }
+}
+
+module.exports = { monitor }

libraries/metrics/open_sockets.js

@@ -0,0 +1,99 @@
+/*
+ * decaffeinate suggestions:
+ * DS102: Remove unnecessary code created because of implicit returns
+ * DS205: Consider reworking code to avoid use of IIFEs
+ * DS207: Consider shorter variations of null checks
+ * Full docs: https://github.com/decaffeinate/decaffeinate/blob/master/docs/suggestions.md
+ */
+let OpenSocketsMonitor
+const seconds = 1000
+
+// In Node 0.10 the default is 5, which means only 5 open connections at one.
+// Node 0.12 has a default of Infinity. Make sure we have no limit set,
+// regardless of Node version.
+require('node:http').globalAgent.maxSockets = Infinity
+require('node:https').globalAgent.maxSockets = Infinity
+
+const SOCKETS_HTTP = require('node:http').globalAgent.sockets
+const SOCKETS_HTTPS = require('node:https').globalAgent.sockets
+const FREE_SOCKETS_HTTP = require('node:http').globalAgent.freeSockets
+const FREE_SOCKETS_HTTPS = require('node:https').globalAgent.freeSockets
+
+// keep track of set gauges and reset them in the next collection cycle
+const SEEN_HOSTS_HTTP = new Set()
+const SEEN_HOSTS_HTTPS = new Set()
+const FREE_SEEN_HOSTS_HTTP = new Set()
+const FREE_SEEN_HOSTS_HTTPS = new Set()
+
+function collectConnectionsCount(
+  sockets,
+  seenHosts,
+  status,
+  https,
+  emitLegacyMetric
+) {
+  const Metrics = require('./index')
+  Object.keys(sockets).forEach(host => seenHosts.add(host))
+  seenHosts.forEach(host => {
+    // host: 'HOST:PORT:'
+    const hostname = host.split(':')[0]
+    const openConnections = (sockets[host] || []).length
+    if (!openConnections) {
+      seenHosts.delete(host)
+    }
+    Metrics.gauge('sockets', openConnections, 1, {
+      path: hostname,
+      method: https,
+      status,
+    })
+    if (status === 'open' && emitLegacyMetric) {
+      // Emit legacy metric to keep old time series intact.
+      Metrics.gauge(
+        `${status}_connections.${https}.${hostname}`,
+        openConnections
+      )
+    }
+  })
+}
+
+module.exports = OpenSocketsMonitor = {
+  monitor(emitLegacyMetric) {
+    const interval = setInterval(
+      () => OpenSocketsMonitor.gaugeOpenSockets(emitLegacyMetric),
+      5 * seconds
+    )
+    const Metrics = require('./index')
+    return Metrics.registerDestructor(() => clearInterval(interval))
+  },
+
+  gaugeOpenSockets(emitLegacyMetric) {
+    collectConnectionsCount(
+      SOCKETS_HTTP,
+      SEEN_HOSTS_HTTP,
+      'open',
+      'http',
+      emitLegacyMetric
+    )
+    collectConnectionsCount(
+      SOCKETS_HTTPS,
+      SEEN_HOSTS_HTTPS,
+      'open',
+      'https',
+      emitLegacyMetric
+    )
+    collectConnectionsCount(
+      FREE_SOCKETS_HTTP,
+      FREE_SEEN_HOSTS_HTTP,
+      'free',
+      'http',
+      false
+    )
+    collectConnectionsCount(
+      FREE_SOCKETS_HTTPS,
+      FREE_SEEN_HOSTS_HTTPS,
+      'free',
+      'https',
+      false
+    )
+  },
+}

libraries/metrics/package.json

@@ -0,0 +1,45 @@
+{
+  "name": "@overleaf/metrics",
+  "version": "4.2.0",
+  "description": "A drop-in metrics and monitoring module for node.js apps",
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/overleaf/metrics-module.git"
+  },
+  "main": "index.js",
+  "dependencies": {
+    "@google-cloud/opentelemetry-cloud-trace-exporter": "^2.1.0",
+    "@google-cloud/profiler": "^6.0.0",
+    "@opentelemetry/api": "^1.4.1",
+    "@opentelemetry/auto-instrumentations-node": "^0.39.1",
+    "@opentelemetry/exporter-trace-otlp-http": "^0.41.2",
+    "@opentelemetry/resources": "^1.15.2",
+    "@opentelemetry/sdk-node": "^0.41.2",
+    "@opentelemetry/semantic-conventions": "^1.15.2",
+    "compression": "^1.7.4",
+    "prom-client": "^14.1.1",
+    "yn": "^3.1.1"
+  },
+  "devDependencies": {
+    "bunyan": "^1.0.0",
+    "chai": "^4.3.6",
+    "mocha": "^11.1.0",
+    "sandboxed-module": "^2.0.4",
+    "sinon": "^9.2.4",
+    "typescript": "^5.0.4"
+  },
+  "scripts": {
+    "lint": "eslint --ext .js --ext .cjs --ext .ts --max-warnings 0 --format unix .",
+    "lint:fix": "eslint --fix --ext .js --ext .cjs --ext .ts .",
+    "test:unit": "mocha --exit test/**/*.{js,cjs}",
+    "test:acceptance": "mocha --reporter spec --recursive --exit --grep=$MOCHA_GREP test/acceptance",
+    "test": "npm run lint && npm run format && npm run types:check && npm run test:unit",
+    "format": "prettier --list-different $PWD/'**/*.{js,cjs,ts}'",
+    "format:fix": "prettier --write $PWD/'**/*.{js,cjs,ts}'",
+    "test:ci": "npm run test:unit",
+    "types:check": "tsc --noEmit"
+  },
+  "peerDependencies": {
+    "@overleaf/logger": "*"
+  }
+}

libraries/metrics/prom_wrapper.js

@@ -0,0 +1,177 @@
+const logger = require('@overleaf/logger')
+const prom = require('prom-client')
+const registry = require('prom-client').register
+const metrics = new Map()
+
+const labelsKey = function (labels) {
+  let keys = Object.keys(labels)
+  if (keys.length === 0) {
+    return ''
+  }
+
+  keys = keys.sort()
+
+  let hash = ''
+  for (const key of keys) {
+    if (hash.length) {
+      hash += ','
+    }
+    hash += `${key}:${labels[key]}`
+  }
+
+  return hash
+}
+
+const labelsAsArgs = function (labels, labelNames) {
+  const args = []
+  for (const label of labelNames) {
+    args.push(labels[label] || '')
+  }
+  return args
+}
+
+const PromWrapper = {
+  ttlInMinutes: 0,
+  registry,
+
+  metric(type, name, labels, buckets) {
+    return metrics.get(name) || new MetricWrapper(type, name, labels, buckets)
+  },
+
+  collectDefaultMetrics: prom.collectDefaultMetrics,
+}
+
+class MetricWrapper {
+  constructor(type, name, labels, buckets) {
+    metrics.set(name, this)
+    this.name = name
+    this.instances = new Map()
+    this.lastAccess = new Date()
+
+    const labelNames = labels ? Object.keys(labels) : []
+    switch (type) {
+      case 'counter':
+        this.metric = new prom.Counter({
+          name,
+          help: name,
+          labelNames,
+        })
+        break
+      case 'histogram':
+        this.metric = new prom.Histogram({
+          name,
+          help: name,
+          labelNames,
+          buckets,
+        })
+        break
+      case 'summary':
+        this.metric = new prom.Summary({
+          name,
+          help: name,
+          maxAgeSeconds: 60,
+          ageBuckets: 10,
+          labelNames,
+        })
+        break
+      case 'gauge':
+        this.metric = new prom.Gauge({
+          name,
+          help: name,
+          labelNames,
+        })
+        break
+      default:
+        throw new Error(`Unknown metric type: ${type}`)
+    }
+  }
+
+  inc(labels, value) {
+    this._execMethod('inc', labels, value)
+  }
+
+  observe(labels, value) {
+    this._execMethod('observe', labels, value)
+  }
+
+  set(labels, value) {
+    this._execMethod('set', labels, value)
+  }
+
+  sweep() {
+    const thresh = new Date(Date.now() - 1000 * 60 * PromWrapper.ttlInMinutes)
+    this.instances.forEach((instance, key) => {
+      if (thresh > instance.time) {
+        if (process.env.DEBUG_METRICS) {
+          // eslint-disable-next-line no-console
+          console.log(
+            'Sweeping stale metric instance',
+            this.name,
+            { labels: instance.labels },
+            key
+          )
+        }
+        this.metric.remove(
+          ...labelsAsArgs(instance.labels, this.metric.labelNames)
+        )
+      }
+    })
+
+    if (thresh > this.lastAccess) {
+      if (process.env.DEBUG_METRICS) {
+        // eslint-disable-next-line no-console
+        console.log('Sweeping stale metric', this.name, thresh, this.lastAccess)
+      }
+      metrics.delete(this.name)
+      registry.removeSingleMetric(this.name)
+    }
+  }
+
+  _execMethod(method, labels, value) {
+    const key = labelsKey(labels)
+    if (key !== '') {
+      this.instances.set(key, { time: new Date(), labels })
+    }
+    this.lastAccess = new Date()
+    try {
+      this.metric[method](labels, value)
+    } catch (err) {
+      logger.warn(
+        { err, metric: this.metric.name, labels },
+        'failed to record metric'
+      )
+    }
+  }
+}
+
+let sweepingInterval
+PromWrapper.setupSweeping = function () {
+  if (sweepingInterval) {
+    clearInterval(sweepingInterval)
+  }
+  if (!PromWrapper.ttlInMinutes) {
+    if (process.env.DEBUG_METRICS) {
+      // eslint-disable-next-line no-console
+      console.log('Not registering sweep method -- empty ttl')
+    }
+    return
+  }
+  if (process.env.DEBUG_METRICS) {
+    // eslint-disable-next-line no-console
+    console.log('Registering sweep method')
+  }
+  sweepingInterval = setInterval(function () {
+    if (process.env.DEBUG_METRICS) {
+      // eslint-disable-next-line no-console
+      console.log('Sweeping metrics')
+    }
+    metrics.forEach((metric, key) => {
+      metric.sweep()
+    })
+  }, 60000)
+
+  const Metrics = require('./index')
+  Metrics.registerDestructor(() => clearInterval(sweepingInterval))
+}
+
+module.exports = PromWrapper

libraries/metrics/test/acceptance/metrics_tests.js

@@ -0,0 +1,343 @@
+const { promisify } = require('node:util')
+const os = require('node:os')
+const http = require('node:http')
+const { expect } = require('chai')
+const Metrics = require('../..')
+
+const HOSTNAME = os.hostname()
+const APP_NAME = 'test-app'
+const sleep = promisify(setTimeout)
+
+describe('Metrics module', function () {
+  before(function () {
+    process.env.METRICS_APP_NAME = 'test-app'
+    require('../../initialize')
+  })
+
+  describe('at startup', function () {
+    it('increments the process_startup counter', async function () {
+      await expectMetricValue('process_startup', 1)
+    })
+
+    it('collects default metrics', async function () {
+      const metric = await getMetric('process_cpu_user_seconds_total')
+      expect(metric).to.exist
+    })
+  })
+
+  describe('inc()', function () {
+    it('increments counts by 1', async function () {
+      Metrics.inc('duck_count')
+      await expectMetricValue('duck_count', 1)
+      Metrics.inc('duck_count')
+      Metrics.inc('duck_count')
+      await expectMetricValue('duck_count', 3)
+    })
+
+    it('escapes special characters in the key', async function () {
+      Metrics.inc('show.me the $!!')
+      await expectMetricValue('show_me_the____', 1)
+    })
+  })
+
+  describe('count()', function () {
+    it('increments counts by the given count', async function () {
+      Metrics.count('rabbit_count', 5)
+      await expectMetricValue('rabbit_count', 5)
+      Metrics.count('rabbit_count', 6)
+      Metrics.count('rabbit_count', 7)
+      await expectMetricValue('rabbit_count', 18)
+    })
+  })
+
+  describe('summary()', function () {
+    it('collects observations', async function () {
+      Metrics.summary('oven_temp', 200)
+      Metrics.summary('oven_temp', 300)
+      Metrics.summary('oven_temp', 450)
+      const sum = await getSummarySum('oven_temp')
+      expect(sum).to.equal(950)
+    })
+  })
+
+  describe('timing()', function () {
+    it('collects timings', async function () {
+      Metrics.timing('sprint_100m', 10)
+      Metrics.timing('sprint_100m', 20)
+      Metrics.timing('sprint_100m', 30)
+      const sum = await getSummarySum('timer_sprint_100m')
+      expect(sum).to.equal(60)
+    })
+  })
+
+  describe('histogram()', function () {
+    it('collects in buckets', async function () {
+      const buckets = [10, 100, 1000]
+      Metrics.histogram('distance', 10, buckets)
+      Metrics.histogram('distance', 20, buckets)
+      Metrics.histogram('distance', 100, buckets)
+      Metrics.histogram('distance', 200, buckets)
+      Metrics.histogram('distance', 1000, buckets)
+      Metrics.histogram('distance', 2000, buckets)
+      const sum = await getSummarySum('histogram_distance')
+      expect(sum).to.equal(3330)
+      await checkHistogramValues('histogram_distance', {
+        10: 1,
+        100: 3,
+        1000: 5,
+        '+Inf': 6,
+      })
+    })
+  })
+
+  describe('Timer', function () {
+    beforeEach('collect timings', async function () {
+      const buckets = [10, 100, 1000]
+      for (const duration of [1, 1, 1, 15, 15, 15, 105, 105, 105]) {
+        const withBuckets = new Metrics.Timer(
+          'height',
+          1,
+          { label_1: 'a' },
+          buckets
+        )
+        const withOutBuckets = new Metrics.Timer('depth', 1, { label_2: 'b' })
+        await sleep(duration)
+        withBuckets.done()
+        withOutBuckets.done({ label_3: 'c' })
+      }
+    })
+
+    it('with buckets', async function () {
+      await checkHistogramValues('histogram_height', {
+        10: 3,
+        100: 6,
+        1000: 9,
+        '+Inf': 9,
+      })
+      const labelNames = await getMetric('histogram_height').labelNames
+      expect(labelNames).to.deep.equal(['label_1'])
+    })
+
+    it('without buckets', async function () {
+      await checkSummaryValues('timer_depth', {
+        0.01: 1,
+        0.05: 1,
+        0.5: 15,
+        0.9: 105,
+        0.95: 105,
+        0.99: 105,
+        0.999: 105,
+      })
+      const labelNames = await getMetric('timer_depth').labelNames
+      expect(labelNames).to.deep.equal(['label_2', 'label_3'])
+    })
+  })
+
+  describe('gauge()', function () {
+    it('records values', async function () {
+      Metrics.gauge('water_level', 1.5)
+      await expectMetricValue('water_level', 1.5)
+      Metrics.gauge('water_level', 4.2)
+      await expectMetricValue('water_level', 4.2)
+    })
+  })
+
+  describe('globalGauge()', function () {
+    it('records values without a host label', async function () {
+      Metrics.globalGauge('tire_pressure', 99.99)
+      const { value, labels } = await getMetricValue('tire_pressure')
+      expect(value).to.equal(99.99)
+      expect(labels.host).to.equal('global')
+      expect(labels.app).to.equal(APP_NAME)
+    })
+  })
+
+  describe('open_sockets', function () {
+    const keyServer1 = 'open_connections_http_127_42_42_1'
+    const keyServer2 = 'open_connections_http_127_42_42_2'
+
+    let finish1, finish2, emitResponse1, emitResponse2
+    function resetEmitResponse1() {
+      emitResponse1 = new Promise(resolve => (finish1 = resolve))
+    }
+    resetEmitResponse1()
+    function resetEmitResponse2() {
+      emitResponse2 = new Promise(resolve => (finish2 = resolve))
+    }
+    resetEmitResponse2()
+
+    let server1, server2
+    before(function setupServer1(done) {
+      server1 = http.createServer((req, res) => {
+        res.write('...')
+        emitResponse1.then(() => res.end())
+      })
+      server1.listen(0, '127.42.42.1', done)
+    })
+    before(function setupServer2(done) {
+      server2 = http.createServer((req, res) => {
+        res.write('...')
+        emitResponse2.then(() => res.end())
+      })
+      server2.listen(0, '127.42.42.2', done)
+    })
+    after(function cleanupPendingRequests() {
+      finish1()
+      finish2()
+    })
+    after(function shutdownServer1(done) {
+      if (server1) server1.close(done)
+    })
+    after(function shutdownServer2(done) {
+      if (server2) server2.close(done)
+    })
+
+    let urlServer1, urlServer2
+    before(function setUrls() {
+      urlServer1 = `http://127.42.42.1:${server1.address().port}/`
+      urlServer2 = `http://127.42.42.2:${server2.address().port}/`
+    })
+    describe('gaugeOpenSockets()', function () {
+      beforeEach(function runGaugeOpenSockets() {
+        Metrics.open_sockets.gaugeOpenSockets(true)
+      })
+
+      describe('without pending connections', function () {
+        it('emits no open_connections', async function () {
+          await expectNoMetricValue(keyServer1)
+          await expectNoMetricValue(keyServer2)
+        })
+      })
+
+      describe('with pending connections for server1', function () {
+        before(function (done) {
+          http.get(urlServer1)
+          http.get(urlServer1)
+          setTimeout(done, 10)
+        })
+
+        it('emits 2 open_connections for server1', async function () {
+          await expectMetricValue(keyServer1, 2)
+        })
+
+        it('emits no open_connections for server2', async function () {
+          await expectNoMetricValue(keyServer2)
+        })
+      })
+
+      describe('with pending connections for server1 and server2', function () {
+        before(function (done) {
+          http.get(urlServer2)
+          http.get(urlServer2)
+          setTimeout(done, 10)
+        })
+
+        it('emits 2 open_connections for server1', async function () {
+          await expectMetricValue(keyServer1, 2)
+        })
+
+        it('emits 2 open_connections for server2', async function () {
+          await expectMetricValue(keyServer2, 2)
+        })
+      })
+
+      describe('when requests finish for server1', function () {
+        before(function (done) {
+          finish1()
+          resetEmitResponse1()
+          http.get(urlServer1)
+
+          setTimeout(done, 10)
+        })
+
+        it('emits 1 open_connections for server1', async function () {
+          await expectMetricValue(keyServer1, 1)
+        })
+
+        it('emits 2 open_connections for server2', async function () {
+          await expectMetricValue(keyServer2, 2)
+        })
+      })
+
+      describe('when all requests complete', function () {
+        before(function (done) {
+          finish1()
+          finish2()
+
+          setTimeout(done, 10)
+        })
+
+        it('emits no open_connections', async function () {
+          await expectNoMetricValue(keyServer1)
+          await expectNoMetricValue(keyServer2)
+        })
+      })
+    })
+  })
+})
+
+function getMetric(key) {
+  return Metrics.register.getSingleMetric(key)
+}
+
+async function getSummarySum(key) {
+  const metric = getMetric(key)
+  const item = await metric.get()
+  for (const value of item.values) {
+    if (value.metricName === `${key}_sum`) {
+      return value.value
+    }
+  }
+  return null
+}
+
+async function checkHistogramValues(key, values) {
+  const metric = getMetric(key)
+  const item = await metric.get()
+  const found = {}
+  for (const value of item.values) {
+    const bucket = value.labels.le
+    if (!bucket) continue
+    found[bucket] = value.value
+  }
+  expect(found).to.deep.equal(values)
+  return null
+}
+
+async function checkSummaryValues(key, values) {
+  const metric = getMetric(key)
+  const item = await metric.get()
+  const found = {}
+  for (const value of item.values) {
+    const quantile = value.labels.quantile
+    if (!quantile) continue
+    found[quantile] = value.value
+  }
+  for (const quantile of Object.keys(values)) {
+    expect(found[quantile]).to.be.within(
+      values[quantile] - 5,
+      values[quantile] + 15,
+      `quantile: ${quantile}`
+    )
+  }
+  return null
+}
+
+async function getMetricValue(key) {
+  const metrics = await Metrics.register.getMetricsAsJSON()
+  const metric = metrics.find(m => m.name === key)
+  return metric.values[0]
+}
+
+async function expectMetricValue(key, expectedValue) {
+  const value = await getMetricValue(key)
+  expect(value.value).to.equal(expectedValue)
+  expect(value.labels.host).to.equal(HOSTNAME)
+  expect(value.labels.app).to.equal(APP_NAME)
+}
+
+async function expectNoMetricValue(key) {
+  const metric = getMetric(key)
+  if (!metric) return
+  await expectMetricValue(key, 0)
+}

libraries/metrics/test/unit/js/event_loop.js

@@ -0,0 +1,44 @@
+/*
+ * decaffeinate suggestions:
+ * DS102: Remove unnecessary code created because of implicit returns
+ * Full docs: https://github.com/decaffeinate/decaffeinate/blob/master/docs/suggestions.md
+ */
+const chai = require('chai')
+const { expect } = chai
+const path = require('node:path')
+const modulePath = path.join(__dirname, '../../../event_loop.js')
+const SandboxedModule = require('sandboxed-module')
+const sinon = require('sinon')
+
+describe('event_loop', function () {
+  before(function () {
+    this.metrics = {
+      timing: sinon.stub(),
+      registerDestructor: sinon.stub(),
+    }
+    this.logger = {
+      warn: sinon.stub(),
+    }
+    return (this.event_loop = SandboxedModule.require(modulePath, {
+      requires: {
+        './index': this.metrics,
+      },
+    }))
+  })
+
+  describe('with a logger provided', function () {
+    before(function () {
+      return this.event_loop.monitor(this.logger)
+    })
+
+    return it('should register a destructor with metrics', function () {
+      return expect(this.metrics.registerDestructor.called).to.equal(true)
+    })
+  })
+
+  return describe('without a logger provided', function () {
+    return it('should throw an exception', function () {
+      return expect(this.event_loop.monitor).to.throw('logger is undefined')
+    })
+  })
+})

libraries/metrics/test/unit/js/http.js

@@ -0,0 +1,171 @@
+const Path = require('node:path')
+const SandboxedModule = require('sandboxed-module')
+const sinon = require('sinon')
+
+const MODULE_PATH = Path.join(__dirname, '../../../http.js')
+
+describe('http.monitor', function () {
+  beforeEach(function () {
+    this.req = {
+      method: 'POST',
+      url: '/project/1234/cleanup',
+      headers: {
+        'content-length': '123',
+      },
+      route: {
+        path: '/project/:id/cleanup',
+      },
+    }
+    this.originalResponseEnd = sinon.stub()
+    this.res = {
+      end: this.originalResponseEnd,
+    }
+    this.data = 'data'
+    this.logger = {
+      debug: sinon.stub(),
+      info: sinon.stub(),
+      warn: sinon.stub(),
+    }
+    this.Metrics = {
+      timing: sinon.stub(),
+      summary: sinon.stub(),
+    }
+    this.clock = sinon.useFakeTimers()
+
+    this.http = SandboxedModule.require(MODULE_PATH, {
+      requires: {
+        './index': this.Metrics,
+      },
+    })
+  })
+
+  afterEach(function () {
+    this.clock.restore()
+  })
+
+  describe('with the default options', function () {
+    beforeEach('set up the monitor', function (done) {
+      this.http.monitor(this.logger)(this.req, this.res, done)
+    })
+
+    describe('after a simple request', function () {
+      endRequest()
+      expectOriginalEndCalled()
+      expectMetrics()
+
+      it('logs the request at the DEBUG level', function () {
+        sinon.assert.calledWith(
+          this.logger.debug,
+          { req: this.req, res: this.res, responseTimeMs: 500 },
+          '%s %s',
+          this.req.method,
+          this.req.url
+        )
+      })
+    })
+
+    describe('when logging is disabled', function () {
+      beforeEach('disable logging', function () {
+        this.req.logger.disable()
+      })
+
+      endRequest()
+      expectOriginalEndCalled()
+      expectMetrics()
+
+      it("doesn't log the request", function () {
+        sinon.assert.notCalled(this.logger.debug)
+      })
+    })
+
+    describe('with custom log fields', function () {
+      beforeEach('add custom fields', function () {
+        this.req.logger.addFields({ a: 1, b: 2 })
+      })
+
+      endRequest()
+
+      it('logs the request with the custom log fields', function () {
+        sinon.assert.calledWith(
+          this.logger.debug,
+          { req: this.req, res: this.res, responseTimeMs: 500, a: 1, b: 2 },
+          '%s %s',
+          this.req.method,
+          this.req.url
+        )
+      })
+    })
+
+    describe('when setting the log level', function () {
+      beforeEach('set custom level', function () {
+        this.req.logger.setLevel('warn')
+      })
+
+      endRequest()
+
+      it('logs the request at the custom level', function () {
+        sinon.assert.calledWith(
+          this.logger.warn,
+          { req: this.req, res: this.res, responseTimeMs: 500 },
+          '%s %s',
+          this.req.method,
+          this.req.url
+        )
+      })
+    })
+  })
+
+  describe('with a different default log level', function () {
+    beforeEach('set up the monitor', function (done) {
+      this.http.monitor(this.logger, 'info')(this.req, this.res, done)
+    })
+
+    endRequest()
+
+    it('logs the request at that level', function () {
+      sinon.assert.calledWith(
+        this.logger.info,
+        { req: this.req, res: this.res, responseTimeMs: 500 },
+        '%s %s',
+        this.req.method,
+        this.req.url
+      )
+    })
+  })
+})
+
+function endRequest() {
+  beforeEach('end the request', function () {
+    this.clock.tick(500)
+    this.res.end(this.data)
+  })
+}
+
+function expectOriginalEndCalled() {
+  it('calls the original res.end()', function () {
+    sinon.assert.calledWith(this.originalResponseEnd, this.data)
+  })
+}
+
+function expectMetrics() {
+  it('records the response time', function () {
+    sinon.assert.calledWith(this.Metrics.timing, 'http_request', 500, null, {
+      method: this.req.method,
+      status_code: this.res.status_code,
+      path: 'project_id_cleanup',
+    })
+  })
+
+  it('records the request size', function () {
+    sinon.assert.calledWith(
+      this.Metrics.summary,
+      'http_request_size_bytes',
+      123,
+      {
+        method: this.req.method,
+        status_code: this.res.status_code,
+        path: 'project_id_cleanup',
+      }
+    )
+  })
+}

libraries/metrics/test/unit/js/mongodb.js

@@ -0,0 +1,89 @@
+const Metrics = require('../../..')
+
+const { expect } = require('chai')
+const prom = require('prom-client')
+
+describe('mongodb', function () {
+  beforeEach(function () {
+    prom.register.clear()
+    this.pool = {
+      totalConnectionCount: 8,
+      availableConnectionCount: 2,
+      waitQueueSize: 4,
+      options: { maxPoolSize: 10 },
+    }
+    this.servers = new Map([['server1', { s: { pool: this.pool } }]])
+
+    this.mongoClient = { topology: { s: { servers: this.servers } } }
+  })
+
+  it('handles an unconnected client', async function () {
+    const mongoClient = {}
+    Metrics.mongodb.monitor(mongoClient)
+    const metrics = await getMetrics()
+    expect(metrics).to.deep.equal({})
+  })
+
+  it('collects Mongo metrics', async function () {
+    Metrics.mongodb.monitor(this.mongoClient)
+    const metrics = await getMetrics()
+    expect(metrics).to.deep.equal({
+      'mongo_connection_pool_max:server1': 10,
+      'mongo_connection_pool_size:server1': 8,
+      'mongo_connection_pool_available:server1': 2,
+      'mongo_connection_pool_waiting:server1': 4,
+    })
+  })
+
+  it('handles topology changes', async function () {
+    Metrics.mongodb.monitor(this.mongoClient)
+    let metrics = await getMetrics()
+    expect(metrics).to.deep.equal({
+      'mongo_connection_pool_max:server1': 10,
+      'mongo_connection_pool_size:server1': 8,
+      'mongo_connection_pool_available:server1': 2,
+      'mongo_connection_pool_waiting:server1': 4,
+    })
+
+    // Add a server
+    this.servers.set('server2', this.servers.get('server1'))
+    metrics = await getMetrics()
+    expect(metrics).to.deep.equal({
+      'mongo_connection_pool_max:server1': 10,
+      'mongo_connection_pool_size:server1': 8,
+      'mongo_connection_pool_available:server1': 2,
+      'mongo_connection_pool_waiting:server1': 4,
+      'mongo_connection_pool_max:server2': 10,
+      'mongo_connection_pool_size:server2': 8,
+      'mongo_connection_pool_available:server2': 2,
+      'mongo_connection_pool_waiting:server2': 4,
+    })
+
+    // Delete a server
+    this.servers.delete('server1')
+    metrics = await getMetrics()
+    expect(metrics).to.deep.equal({
+      'mongo_connection_pool_max:server2': 10,
+      'mongo_connection_pool_size:server2': 8,
+      'mongo_connection_pool_available:server2': 2,
+      'mongo_connection_pool_waiting:server2': 4,
+    })
+
+    // Delete another server
+    this.servers.delete('server2')
+    metrics = await getMetrics()
+    expect(metrics).to.deep.equal({})
+  })
+})
+
+async function getMetrics() {
+  const metrics = await prom.register.getMetricsAsJSON()
+  const result = {}
+  for (const metric of metrics) {
+    for (const value of metric.values) {
+      const key = `${metric.name}:${value.labels.mongo_server}`
+      result[key] = value.value
+    }
+  }
+  return result
+}

libraries/metrics/tsconfig.json

@@ -0,0 +1,7 @@
+{
+  "extends": "../../tsconfig.backend.json",
+  "include": [
+    "**/*.js",
+    "**/*.cjs"
+  ]
+}

libraries/metrics/uv_threadpool_size.js

@@ -0,0 +1,5 @@
+if (!process.env.UV_THREADPOOL_SIZE) {
+  process.env.UV_THREADPOOL_SIZE = 16
+  // eslint-disable-next-line no-console
+  console.log(`Set UV_THREADPOOL_SIZE=${process.env.UV_THREADPOOL_SIZE}`)
+}

libraries/mongo-utils/.gitignore

@@ -0,0 +1,3 @@
+
+# managed by monorepo$ bin/update_build_scripts
+.npmrc

libraries/mongo-utils/.nvmrc

@@ -0,0 +1 @@
+20.18.2

libraries/mongo-utils/batchedUpdate.js

@@ -0,0 +1,307 @@
+// @ts-check
+/* eslint-disable no-console */
+const { ObjectId, ReadPreference } = require('mongodb')
+
+const READ_PREFERENCE_SECONDARY =
+  process.env.MONGO_HAS_SECONDARIES === 'true'
+    ? ReadPreference.secondary.mode
+    : ReadPreference.secondaryPreferred.mode
+
+const ONE_MONTH_IN_MS = 1000 * 60 * 60 * 24 * 31
+let ID_EDGE_PAST
+const ID_EDGE_FUTURE = objectIdFromMs(Date.now() + 1000)
+let BATCH_DESCENDING
+let BATCH_SIZE
+let VERBOSE_LOGGING
+let BATCH_RANGE_START
+let BATCH_RANGE_END
+let BATCH_MAX_TIME_SPAN_IN_MS
+
+/**
+ * @typedef {import("mongodb").Collection} Collection
+ * @typedef {import("mongodb-legacy").Collection} LegacyCollection
+ * @typedef {import("mongodb").Document} Document
+ * @typedef {import("mongodb").FindOptions} FindOptions
+ * @typedef {import("mongodb").UpdateFilter<Document>} UpdateDocument
+ */
+
+/**
+ * @typedef {Object} BatchedUpdateOptions
+ * @property {string} [BATCH_DESCENDING]
+ * @property {string} [BATCH_LAST_ID]
+ * @property {string} [BATCH_MAX_TIME_SPAN_IN_MS]
+ * @property {string} [BATCH_RANGE_END]
+ * @property {string} [BATCH_RANGE_START]
+ * @property {string} [BATCH_SIZE]
+ * @property {string} [VERBOSE_LOGGING]
+ */
+
+/**
+ * @param {BatchedUpdateOptions} options
+ */
+function refreshGlobalOptionsForBatchedUpdate(options = {}) {
+  options = Object.assign({}, options, process.env)
+
+  BATCH_DESCENDING = options.BATCH_DESCENDING === 'true'
+  BATCH_SIZE = parseInt(options.BATCH_SIZE || '1000', 10) || 1000
+  VERBOSE_LOGGING = options.VERBOSE_LOGGING === 'true'
+  if (options.BATCH_LAST_ID) {
+    BATCH_RANGE_START = objectIdFromInput(options.BATCH_LAST_ID)
+  } else if (options.BATCH_RANGE_START) {
+    BATCH_RANGE_START = objectIdFromInput(options.BATCH_RANGE_START)
+  } else {
+    if (BATCH_DESCENDING) {
+      BATCH_RANGE_START = ID_EDGE_FUTURE
+    } else {
+      BATCH_RANGE_START = ID_EDGE_PAST
+    }
+  }
+  BATCH_MAX_TIME_SPAN_IN_MS = parseInt(
+    options.BATCH_MAX_TIME_SPAN_IN_MS || ONE_MONTH_IN_MS.toString(),
+    10
+  )
+  if (options.BATCH_RANGE_END) {
+    BATCH_RANGE_END = objectIdFromInput(options.BATCH_RANGE_END)
+  } else {
+    if (BATCH_DESCENDING) {
+      BATCH_RANGE_END = ID_EDGE_PAST
+    } else {
+      BATCH_RANGE_END = ID_EDGE_FUTURE
+    }
+  }
+}
+
+/**
+ * @param {Collection | LegacyCollection} collection
+ * @param {Document} query
+ * @param {ObjectId} start
+ * @param {ObjectId} end
+ * @param {Document} projection
+ * @param {FindOptions} findOptions
+ * @return {Promise<Array<Document>>}
+ */
+async function getNextBatch(
+  collection,
+  query,
+  start,
+  end,
+  projection,
+  findOptions
+) {
+  if (BATCH_DESCENDING) {
+    query._id = {
+      $gt: end,
+      $lte: start,
+    }
+  } else {
+    query._id = {
+      $gt: start,
+      $lte: end,
+    }
+  }
+  return await collection
+    .find(query, findOptions)
+    .project(projection)
+    .sort({ _id: BATCH_DESCENDING ? -1 : 1 })
+    .limit(BATCH_SIZE)
+    .toArray()
+}
+
+/**
+ * @param {Collection | LegacyCollection} collection
+ * @param {Array<Document>} nextBatch
+ * @param {UpdateDocument} update
+ * @return {Promise<void>}
+ */
+async function performUpdate(collection, nextBatch, update) {
+  await collection.updateMany(
+    { _id: { $in: nextBatch.map(entry => entry._id) } },
+    update
+  )
+}
+
+/**
+ * @param {string} input
+ * @return {ObjectId}
+ */
+function objectIdFromInput(input) {
+  if (input.includes('T')) {
+    const t = new Date(input).getTime()
+    if (Number.isNaN(t)) throw new Error(`${input} is not a valid date`)
+    return objectIdFromMs(t)
+  } else {
+    return new ObjectId(input)
+  }
+}
+
+/**
+ * @param {ObjectId} objectId
+ * @return {string}
+ */
+function renderObjectId(objectId) {
+  return `${objectId} (${objectId.getTimestamp().toISOString()})`
+}
+
+/**
+ * @param {number} ms
+ * @return {ObjectId}
+ */
+function objectIdFromMs(ms) {
+  return ObjectId.createFromTime(ms / 1000)
+}
+
+/**
+ * @param {ObjectId} id
+ * @return {number}
+ */
+function getMsFromObjectId(id) {
+  return id.getTimestamp().getTime()
+}
+
+/**
+ * @param {ObjectId} start
+ * @return {ObjectId}
+ */
+function getNextEnd(start) {
+  let end
+  if (BATCH_DESCENDING) {
+    end = objectIdFromMs(getMsFromObjectId(start) - BATCH_MAX_TIME_SPAN_IN_MS)
+    if (getMsFromObjectId(end) <= getMsFromObjectId(BATCH_RANGE_END)) {
+      end = BATCH_RANGE_END
+    }
+  } else {
+    end = objectIdFromMs(getMsFromObjectId(start) + BATCH_MAX_TIME_SPAN_IN_MS)
+    if (getMsFromObjectId(end) >= getMsFromObjectId(BATCH_RANGE_END)) {
+      end = BATCH_RANGE_END
+    }
+  }
+  return end
+}
+
+/**
+ * @param {Collection | LegacyCollection} collection
+ * @return {Promise<ObjectId|null>}
+ */
+async function getIdEdgePast(collection) {
+  const [first] = await collection
+    .find({})
+    .project({ _id: 1 })
+    .sort({ _id: 1 })
+    .limit(1)
+    .toArray()
+  if (!first) return null
+  // Go one second further into the past in order to include the first entry via
+  // first._id > ID_EDGE_PAST
+  return objectIdFromMs(Math.max(0, getMsFromObjectId(first._id) - 1000))
+}
+
+/**
+ * @param {Collection | LegacyCollection} collection
+ * @param {Document} query
+ * @param {UpdateDocument | ((batch: Array<Document>) => Promise<void>)} update
+ * @param {Document} [projection]
+ * @param {FindOptions} [findOptions]
+ * @param {BatchedUpdateOptions} [batchedUpdateOptions]
+ */
+async function batchedUpdate(
+  collection,
+  query,
+  update,
+  projection,
+  findOptions,
+  batchedUpdateOptions
+) {
+  ID_EDGE_PAST = await getIdEdgePast(collection)
+  if (!ID_EDGE_PAST) {
+    console.warn(
+      `The collection ${collection.collectionName} appears to be empty.`
+    )
+    return 0
+  }
+  refreshGlobalOptionsForBatchedUpdate(batchedUpdateOptions)
+
+  findOptions = findOptions || {}
+  findOptions.readPreference = READ_PREFERENCE_SECONDARY
+
+  projection = projection || { _id: 1 }
+  let nextBatch
+  let updated = 0
+  let start = BATCH_RANGE_START
+
+  while (start !== BATCH_RANGE_END) {
+    let end = getNextEnd(start)
+    nextBatch = await getNextBatch(
+      collection,
+      query,
+      start,
+      end,
+      projection,
+      findOptions
+    )
+    if (nextBatch.length > 0) {
+      end = nextBatch[nextBatch.length - 1]._id
+      updated += nextBatch.length
+
+      if (VERBOSE_LOGGING) {
+        console.log(
+          `Running update on batch with ids ${JSON.stringify(
+            nextBatch.map(entry => entry._id)
+          )}`
+        )
+      } else {
+        console.error(`Running update on batch ending ${renderObjectId(end)}`)
+      }
+
+      if (typeof update === 'function') {
+        await update(nextBatch)
+      } else {
+        await performUpdate(collection, nextBatch, update)
+      }
+    }
+    console.error(`Completed batch ending ${renderObjectId(end)}`)
+    start = end
+  }
+  return updated
+}
+
+/**
+ * @param {Collection | LegacyCollection} collection
+ * @param {Document} query
+ * @param {UpdateDocument | ((batch: Array<Object>) => Promise<void>)} update
+ * @param {Document} [projection]
+ * @param {FindOptions} [findOptions]
+ * @param {BatchedUpdateOptions} [batchedUpdateOptions]
+ */
+function batchedUpdateWithResultHandling(
+  collection,
+  query,
+  update,
+  projection,
+  findOptions,
+  batchedUpdateOptions
+) {
+  batchedUpdate(
+    collection,
+    query,
+    update,
+    projection,
+    findOptions,
+    batchedUpdateOptions
+  )
+    .then(processed => {
+      console.error({ processed })
+      process.exit(0)
+    })
+    .catch(error => {
+      console.error({ error })
+      process.exit(1)
+    })
+}
+
+module.exports = {
+  READ_PREFERENCE_SECONDARY,
+  objectIdFromInput,
+  renderObjectId,
+  batchedUpdate,
+  batchedUpdateWithResultHandling,
+}

libraries/mongo-utils/buildscript.txt

@@ -0,0 +1,10 @@
+mongo-utils
+--dependencies=None
+--docker-repos=gcr.io/overleaf-ops
+--env-add=
+--env-pass-through=
+--esmock-loader=False
+--is-library=True
+--node-version=20.18.2
+--public-repo=False
+--script-version=4.5.0

libraries/mongo-utils/package.json

@@ -0,0 +1,30 @@
+{
+  "name": "@overleaf/mongo-utils",
+  "version": "0.0.1",
+  "description": "utilities to help working with mongo",
+  "main": "index.js",
+  "scripts": {
+    "test": "npm run lint && npm run format && npm run types:check && npm run test:unit",
+    "test:unit": "mocha --exit test/**/*.{js,cjs}",
+    "lint": "eslint --ext .js --ext .cjs --ext .ts --max-warnings 0 --format unix .",
+    "lint:fix": "eslint --fix --ext .js --ext .cjs --ext .ts .",
+    "format": "prettier --list-different $PWD/'**/*.{js,cjs,ts}'",
+    "format:fix": "prettier --write $PWD/'**/*.{js,cjs,ts}'",
+    "test:ci": "npm run test:unit",
+    "types:check": "tsc --noEmit"
+  },
+  "author": "Overleaf (https://www.overleaf.com)",
+  "license": "AGPL-3.0-only",
+  "dependencies": {
+    "mongodb": "6.12.0",
+    "mongodb-legacy": "6.1.3"
+  },
+  "devDependencies": {
+    "chai": "^4.3.6",
+    "mocha": "^11.1.0",
+    "sandboxed-module": "^2.0.4",
+    "sinon": "^9.2.4",
+    "sinon-chai": "^3.7.0",
+    "typescript": "^5.0.4"
+  }
+}

libraries/mongo-utils/test/setup.js

@@ -0,0 +1,11 @@
+const chai = require('chai')
+const sinonChai = require('sinon-chai')
+const SandboxedModule = require('sandboxed-module')
+
+// Chai configuration
+chai.should()
+chai.use(sinonChai)
+
+SandboxedModule.configure({
+  globals: { Buffer, JSON, console, process },
+})

libraries/mongo-utils/tsconfig.json

@@ -0,0 +1,7 @@
+{
+  "extends": "../../tsconfig.backend.json",
+  "include": [
+    "**/*.js",
+    "**/*.cjs"
+  ]
+}

libraries/o-error/.editorconfig

@@ -0,0 +1,9 @@
+root = true
+
+[*]
+charset = utf-8
+indent_style = space
+indent_size = 2
+end_of_line = lf
+insert_final_newline = true
+trim_trailing_whitespace = true