Compare commits
14 commits
Author | SHA1 | Date | |
---|---|---|---|
![]() |
989a83b82f | ||
![]() |
b3b787e149 | ||
![]() |
4514aee8b4 | ||
![]() |
4126b9a3c5 | ||
0a94338e7d | |||
![]() |
39a4cd22ce | ||
30ef082cc9 | |||
![]() |
3d03a7b0ea | ||
5eb1b8007c | |||
8d56ec925a | |||
e4536bc3b8 | |||
2ac07c4c12 | |||
a24a40efe5 | |||
![]() |
947ed940f3 |
6055 changed files with 43 additions and 840151 deletions
44
.github/ISSUE_TEMPLATE.md
vendored
44
.github/ISSUE_TEMPLATE.md
vendored
|
@ -1,44 +0,0 @@
|
||||||
<!--
|
|
||||||
|
|
||||||
Note: If you are using www.overleaf.com and have a problem,
|
|
||||||
or if you would like to request a new feature please contact
|
|
||||||
the support team at support@overleaf.com
|
|
||||||
|
|
||||||
This form should only be used to report bugs in the
|
|
||||||
Community Edition release of Overleaf.
|
|
||||||
|
|
||||||
-->
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<!-- BUG REPORT TEMPLATE -->
|
|
||||||
|
|
||||||
## Steps to Reproduce
|
|
||||||
<!-- Describe the steps leading up to when / where you found the bug. -->
|
|
||||||
<!-- Screenshots may be helpful here. -->
|
|
||||||
|
|
||||||
1.
|
|
||||||
2.
|
|
||||||
3.
|
|
||||||
|
|
||||||
## Expected Behaviour
|
|
||||||
<!-- What should have happened when you completed the steps above? -->
|
|
||||||
|
|
||||||
## Observed Behaviour
|
|
||||||
<!-- What actually happened when you completed the steps above? -->
|
|
||||||
<!-- Screenshots may be helpful here. -->
|
|
||||||
|
|
||||||
## Context
|
|
||||||
<!-- How has this issue affected you? What were you trying to accomplish? -->
|
|
||||||
|
|
||||||
## Technical Info
|
|
||||||
<!-- Provide any technical details that may be applicable (or N/A if not applicable). -->
|
|
||||||
|
|
||||||
* URL:
|
|
||||||
* Browser Name and version:
|
|
||||||
* Operating System and version (desktop or mobile):
|
|
||||||
* Signed in as:
|
|
||||||
* Project and/or file:
|
|
||||||
|
|
||||||
## Analysis
|
|
||||||
<!--- Optionally, document investigation of / suggest a fix for the bug, e.g. 'comes from this line / commit' -->
|
|
11
.github/PULL_REQUEST_TEMPLATE.md
vendored
11
.github/PULL_REQUEST_TEMPLATE.md
vendored
|
@ -1,11 +0,0 @@
|
||||||
## Description
|
|
||||||
<!-- Goal of the pull request -->
|
|
||||||
|
|
||||||
|
|
||||||
## Related issues / Pull Requests
|
|
||||||
<!-- Fixes #xyz, Contributes to #xyz, Related to #xyz-->
|
|
||||||
|
|
||||||
|
|
||||||
## Contributor Agreement
|
|
||||||
|
|
||||||
- [ ] I confirm I have signed the [Contributor License Agreement](https://github.com/overleaf/overleaf/blob/master/CONTRIBUTING.md#contributor-license-agreement)
|
|
6
.gitignore
vendored
6
.gitignore
vendored
|
@ -1,6 +0,0 @@
|
||||||
# docker image build
|
|
||||||
.dockerignore
|
|
||||||
|
|
||||||
# user defined files
|
|
||||||
.env
|
|
||||||
docker-compose.override.yml
|
|
|
@ -1,43 +0,0 @@
|
||||||
Contributing to Overleaf
|
|
||||||
==========================
|
|
||||||
|
|
||||||
Thank you for reading this! If you'd like to report a bug or join in the development
|
|
||||||
of Overleaf, then here are some notes on how to do that.
|
|
||||||
|
|
||||||
Reporting bugs and opening issues
|
|
||||||
---------------------------------
|
|
||||||
|
|
||||||
If you'd like to report a bug or open an issue, please **[check if there is an existing issue](https://github.com/overleaf/overleaf/issues).**
|
|
||||||
If there is then please add any more information that you have, or give it a 👍.
|
|
||||||
|
|
||||||
When submitting an issue please describe the issue as clearly as possible, including how to
|
|
||||||
reproduce the bug, which situations it appears in, what you expected to happen, and what actually happens.
|
|
||||||
If you can include a screenshot for front end issues that is very helpful.
|
|
||||||
|
|
||||||
**Note**: If you are using [www.overleaf.com](www.overleaf.com) and have a problem, or if you would like to request a new feature, please contact the Support team at support@overleaf.com. Raise an issue here only to report bugs in the Community Edition release of Overleaf.
|
|
||||||
|
|
||||||
Pull Requests
|
|
||||||
-------------
|
|
||||||
|
|
||||||
See [our wiki](https://github.com/overleaf/overleaf/wiki)
|
|
||||||
for how to manage the Overleaf development environment and for our developer guidelines.
|
|
||||||
|
|
||||||
We love pull requests, so be bold with them! Don't be afraid of going ahead
|
|
||||||
and changing something, or adding a new feature. We're very happy to work with you
|
|
||||||
to get your changes merged into Overleaf.
|
|
||||||
|
|
||||||
If you're looking for something to work on, have a look at the [open issues](https://github.com/overleaf/overleaf/issues).
|
|
||||||
|
|
||||||
Security
|
|
||||||
--------
|
|
||||||
|
|
||||||
Please see [our security policy](https://github.com/overleaf/overleaf/security/policy) if you would like to report a potential security vulnerability.
|
|
||||||
|
|
||||||
Contributor License Agreement
|
|
||||||
-----------------------------
|
|
||||||
|
|
||||||
Before we can accept any contributions of code, we need you to agree to our
|
|
||||||
[Contributor License Agreement](https://docs.google.com/forms/d/e/1FAIpQLSef79XH3mb7yIiMzZw-yALEegS-wyFetvjTiNBfZvf_IHD2KA/viewform?usp=sf_link).
|
|
||||||
This is to ensure that you own the copyright of your contribution, and that you
|
|
||||||
agree to give us a license to use it in both the open source version, and the version
|
|
||||||
of Overleaf running at www.overleaf.com, which may have additional changes.
|
|
661
LICENSE
661
LICENSE
|
@ -1,661 +0,0 @@
|
||||||
GNU AFFERO GENERAL PUBLIC LICENSE
|
|
||||||
Version 3, 19 November 2007
|
|
||||||
|
|
||||||
Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
|
|
||||||
Everyone is permitted to copy and distribute verbatim copies
|
|
||||||
of this license document, but changing it is not allowed.
|
|
||||||
|
|
||||||
Preamble
|
|
||||||
|
|
||||||
The GNU Affero General Public License is a free, copyleft license for
|
|
||||||
software and other kinds of works, specifically designed to ensure
|
|
||||||
cooperation with the community in the case of network server software.
|
|
||||||
|
|
||||||
The licenses for most software and other practical works are designed
|
|
||||||
to take away your freedom to share and change the works. By contrast,
|
|
||||||
our General Public Licenses are intended to guarantee your freedom to
|
|
||||||
share and change all versions of a program--to make sure it remains free
|
|
||||||
software for all its users.
|
|
||||||
|
|
||||||
When we speak of free software, we are referring to freedom, not
|
|
||||||
price. Our General Public Licenses are designed to make sure that you
|
|
||||||
have the freedom to distribute copies of free software (and charge for
|
|
||||||
them if you wish), that you receive source code or can get it if you
|
|
||||||
want it, that you can change the software or use pieces of it in new
|
|
||||||
free programs, and that you know you can do these things.
|
|
||||||
|
|
||||||
Developers that use our General Public Licenses protect your rights
|
|
||||||
with two steps: (1) assert copyright on the software, and (2) offer
|
|
||||||
you this License which gives you legal permission to copy, distribute
|
|
||||||
and/or modify the software.
|
|
||||||
|
|
||||||
A secondary benefit of defending all users' freedom is that
|
|
||||||
improvements made in alternate versions of the program, if they
|
|
||||||
receive widespread use, become available for other developers to
|
|
||||||
incorporate. Many developers of free software are heartened and
|
|
||||||
encouraged by the resulting cooperation. However, in the case of
|
|
||||||
software used on network servers, this result may fail to come about.
|
|
||||||
The GNU General Public License permits making a modified version and
|
|
||||||
letting the public access it on a server without ever releasing its
|
|
||||||
source code to the public.
|
|
||||||
|
|
||||||
The GNU Affero General Public License is designed specifically to
|
|
||||||
ensure that, in such cases, the modified source code becomes available
|
|
||||||
to the community. It requires the operator of a network server to
|
|
||||||
provide the source code of the modified version running there to the
|
|
||||||
users of that server. Therefore, public use of a modified version, on
|
|
||||||
a publicly accessible server, gives the public access to the source
|
|
||||||
code of the modified version.
|
|
||||||
|
|
||||||
An older license, called the Affero General Public License and
|
|
||||||
published by Affero, was designed to accomplish similar goals. This is
|
|
||||||
a different license, not a version of the Affero GPL, but Affero has
|
|
||||||
released a new version of the Affero GPL which permits relicensing under
|
|
||||||
this license.
|
|
||||||
|
|
||||||
The precise terms and conditions for copying, distribution and
|
|
||||||
modification follow.
|
|
||||||
|
|
||||||
TERMS AND CONDITIONS
|
|
||||||
|
|
||||||
0. Definitions.
|
|
||||||
|
|
||||||
"This License" refers to version 3 of the GNU Affero General Public License.
|
|
||||||
|
|
||||||
"Copyright" also means copyright-like laws that apply to other kinds of
|
|
||||||
works, such as semiconductor masks.
|
|
||||||
|
|
||||||
"The Program" refers to any copyrightable work licensed under this
|
|
||||||
License. Each licensee is addressed as "you". "Licensees" and
|
|
||||||
"recipients" may be individuals or organizations.
|
|
||||||
|
|
||||||
To "modify" a work means to copy from or adapt all or part of the work
|
|
||||||
in a fashion requiring copyright permission, other than the making of an
|
|
||||||
exact copy. The resulting work is called a "modified version" of the
|
|
||||||
earlier work or a work "based on" the earlier work.
|
|
||||||
|
|
||||||
A "covered work" means either the unmodified Program or a work based
|
|
||||||
on the Program.
|
|
||||||
|
|
||||||
To "propagate" a work means to do anything with it that, without
|
|
||||||
permission, would make you directly or secondarily liable for
|
|
||||||
infringement under applicable copyright law, except executing it on a
|
|
||||||
computer or modifying a private copy. Propagation includes copying,
|
|
||||||
distribution (with or without modification), making available to the
|
|
||||||
public, and in some countries other activities as well.
|
|
||||||
|
|
||||||
To "convey" a work means any kind of propagation that enables other
|
|
||||||
parties to make or receive copies. Mere interaction with a user through
|
|
||||||
a computer network, with no transfer of a copy, is not conveying.
|
|
||||||
|
|
||||||
An interactive user interface displays "Appropriate Legal Notices"
|
|
||||||
to the extent that it includes a convenient and prominently visible
|
|
||||||
feature that (1) displays an appropriate copyright notice, and (2)
|
|
||||||
tells the user that there is no warranty for the work (except to the
|
|
||||||
extent that warranties are provided), that licensees may convey the
|
|
||||||
work under this License, and how to view a copy of this License. If
|
|
||||||
the interface presents a list of user commands or options, such as a
|
|
||||||
menu, a prominent item in the list meets this criterion.
|
|
||||||
|
|
||||||
1. Source Code.
|
|
||||||
|
|
||||||
The "source code" for a work means the preferred form of the work
|
|
||||||
for making modifications to it. "Object code" means any non-source
|
|
||||||
form of a work.
|
|
||||||
|
|
||||||
A "Standard Interface" means an interface that either is an official
|
|
||||||
standard defined by a recognized standards body, or, in the case of
|
|
||||||
interfaces specified for a particular programming language, one that
|
|
||||||
is widely used among developers working in that language.
|
|
||||||
|
|
||||||
The "System Libraries" of an executable work include anything, other
|
|
||||||
than the work as a whole, that (a) is included in the normal form of
|
|
||||||
packaging a Major Component, but which is not part of that Major
|
|
||||||
Component, and (b) serves only to enable use of the work with that
|
|
||||||
Major Component, or to implement a Standard Interface for which an
|
|
||||||
implementation is available to the public in source code form. A
|
|
||||||
"Major Component", in this context, means a major essential component
|
|
||||||
(kernel, window system, and so on) of the specific operating system
|
|
||||||
(if any) on which the executable work runs, or a compiler used to
|
|
||||||
produce the work, or an object code interpreter used to run it.
|
|
||||||
|
|
||||||
The "Corresponding Source" for a work in object code form means all
|
|
||||||
the source code needed to generate, install, and (for an executable
|
|
||||||
work) run the object code and to modify the work, including scripts to
|
|
||||||
control those activities. However, it does not include the work's
|
|
||||||
System Libraries, or general-purpose tools or generally available free
|
|
||||||
programs which are used unmodified in performing those activities but
|
|
||||||
which are not part of the work. For example, Corresponding Source
|
|
||||||
includes interface definition files associated with source files for
|
|
||||||
the work, and the source code for shared libraries and dynamically
|
|
||||||
linked subprograms that the work is specifically designed to require,
|
|
||||||
such as by intimate data communication or control flow between those
|
|
||||||
subprograms and other parts of the work.
|
|
||||||
|
|
||||||
The Corresponding Source need not include anything that users
|
|
||||||
can regenerate automatically from other parts of the Corresponding
|
|
||||||
Source.
|
|
||||||
|
|
||||||
The Corresponding Source for a work in source code form is that
|
|
||||||
same work.
|
|
||||||
|
|
||||||
2. Basic Permissions.
|
|
||||||
|
|
||||||
All rights granted under this License are granted for the term of
|
|
||||||
copyright on the Program, and are irrevocable provided the stated
|
|
||||||
conditions are met. This License explicitly affirms your unlimited
|
|
||||||
permission to run the unmodified Program. The output from running a
|
|
||||||
covered work is covered by this License only if the output, given its
|
|
||||||
content, constitutes a covered work. This License acknowledges your
|
|
||||||
rights of fair use or other equivalent, as provided by copyright law.
|
|
||||||
|
|
||||||
You may make, run and propagate covered works that you do not
|
|
||||||
convey, without conditions so long as your license otherwise remains
|
|
||||||
in force. You may convey covered works to others for the sole purpose
|
|
||||||
of having them make modifications exclusively for you, or provide you
|
|
||||||
with facilities for running those works, provided that you comply with
|
|
||||||
the terms of this License in conveying all material for which you do
|
|
||||||
not control copyright. Those thus making or running the covered works
|
|
||||||
for you must do so exclusively on your behalf, under your direction
|
|
||||||
and control, on terms that prohibit them from making any copies of
|
|
||||||
your copyrighted material outside their relationship with you.
|
|
||||||
|
|
||||||
Conveying under any other circumstances is permitted solely under
|
|
||||||
the conditions stated below. Sublicensing is not allowed; section 10
|
|
||||||
makes it unnecessary.
|
|
||||||
|
|
||||||
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
|
|
||||||
|
|
||||||
No covered work shall be deemed part of an effective technological
|
|
||||||
measure under any applicable law fulfilling obligations under article
|
|
||||||
11 of the WIPO copyright treaty adopted on 20 December 1996, or
|
|
||||||
similar laws prohibiting or restricting circumvention of such
|
|
||||||
measures.
|
|
||||||
|
|
||||||
When you convey a covered work, you waive any legal power to forbid
|
|
||||||
circumvention of technological measures to the extent such circumvention
|
|
||||||
is effected by exercising rights under this License with respect to
|
|
||||||
the covered work, and you disclaim any intention to limit operation or
|
|
||||||
modification of the work as a means of enforcing, against the work's
|
|
||||||
users, your or third parties' legal rights to forbid circumvention of
|
|
||||||
technological measures.
|
|
||||||
|
|
||||||
4. Conveying Verbatim Copies.
|
|
||||||
|
|
||||||
You may convey verbatim copies of the Program's source code as you
|
|
||||||
receive it, in any medium, provided that you conspicuously and
|
|
||||||
appropriately publish on each copy an appropriate copyright notice;
|
|
||||||
keep intact all notices stating that this License and any
|
|
||||||
non-permissive terms added in accord with section 7 apply to the code;
|
|
||||||
keep intact all notices of the absence of any warranty; and give all
|
|
||||||
recipients a copy of this License along with the Program.
|
|
||||||
|
|
||||||
You may charge any price or no price for each copy that you convey,
|
|
||||||
and you may offer support or warranty protection for a fee.
|
|
||||||
|
|
||||||
5. Conveying Modified Source Versions.
|
|
||||||
|
|
||||||
You may convey a work based on the Program, or the modifications to
|
|
||||||
produce it from the Program, in the form of source code under the
|
|
||||||
terms of section 4, provided that you also meet all of these conditions:
|
|
||||||
|
|
||||||
a) The work must carry prominent notices stating that you modified
|
|
||||||
it, and giving a relevant date.
|
|
||||||
|
|
||||||
b) The work must carry prominent notices stating that it is
|
|
||||||
released under this License and any conditions added under section
|
|
||||||
7. This requirement modifies the requirement in section 4 to
|
|
||||||
"keep intact all notices".
|
|
||||||
|
|
||||||
c) You must license the entire work, as a whole, under this
|
|
||||||
License to anyone who comes into possession of a copy. This
|
|
||||||
License will therefore apply, along with any applicable section 7
|
|
||||||
additional terms, to the whole of the work, and all its parts,
|
|
||||||
regardless of how they are packaged. This License gives no
|
|
||||||
permission to license the work in any other way, but it does not
|
|
||||||
invalidate such permission if you have separately received it.
|
|
||||||
|
|
||||||
d) If the work has interactive user interfaces, each must display
|
|
||||||
Appropriate Legal Notices; however, if the Program has interactive
|
|
||||||
interfaces that do not display Appropriate Legal Notices, your
|
|
||||||
work need not make them do so.
|
|
||||||
|
|
||||||
A compilation of a covered work with other separate and independent
|
|
||||||
works, which are not by their nature extensions of the covered work,
|
|
||||||
and which are not combined with it such as to form a larger program,
|
|
||||||
in or on a volume of a storage or distribution medium, is called an
|
|
||||||
"aggregate" if the compilation and its resulting copyright are not
|
|
||||||
used to limit the access or legal rights of the compilation's users
|
|
||||||
beyond what the individual works permit. Inclusion of a covered work
|
|
||||||
in an aggregate does not cause this License to apply to the other
|
|
||||||
parts of the aggregate.
|
|
||||||
|
|
||||||
6. Conveying Non-Source Forms.
|
|
||||||
|
|
||||||
You may convey a covered work in object code form under the terms
|
|
||||||
of sections 4 and 5, provided that you also convey the
|
|
||||||
machine-readable Corresponding Source under the terms of this License,
|
|
||||||
in one of these ways:
|
|
||||||
|
|
||||||
a) Convey the object code in, or embodied in, a physical product
|
|
||||||
(including a physical distribution medium), accompanied by the
|
|
||||||
Corresponding Source fixed on a durable physical medium
|
|
||||||
customarily used for software interchange.
|
|
||||||
|
|
||||||
b) Convey the object code in, or embodied in, a physical product
|
|
||||||
(including a physical distribution medium), accompanied by a
|
|
||||||
written offer, valid for at least three years and valid for as
|
|
||||||
long as you offer spare parts or customer support for that product
|
|
||||||
model, to give anyone who possesses the object code either (1) a
|
|
||||||
copy of the Corresponding Source for all the software in the
|
|
||||||
product that is covered by this License, on a durable physical
|
|
||||||
medium customarily used for software interchange, for a price no
|
|
||||||
more than your reasonable cost of physically performing this
|
|
||||||
conveying of source, or (2) access to copy the
|
|
||||||
Corresponding Source from a network server at no charge.
|
|
||||||
|
|
||||||
c) Convey individual copies of the object code with a copy of the
|
|
||||||
written offer to provide the Corresponding Source. This
|
|
||||||
alternative is allowed only occasionally and noncommercially, and
|
|
||||||
only if you received the object code with such an offer, in accord
|
|
||||||
with subsection 6b.
|
|
||||||
|
|
||||||
d) Convey the object code by offering access from a designated
|
|
||||||
place (gratis or for a charge), and offer equivalent access to the
|
|
||||||
Corresponding Source in the same way through the same place at no
|
|
||||||
further charge. You need not require recipients to copy the
|
|
||||||
Corresponding Source along with the object code. If the place to
|
|
||||||
copy the object code is a network server, the Corresponding Source
|
|
||||||
may be on a different server (operated by you or a third party)
|
|
||||||
that supports equivalent copying facilities, provided you maintain
|
|
||||||
clear directions next to the object code saying where to find the
|
|
||||||
Corresponding Source. Regardless of what server hosts the
|
|
||||||
Corresponding Source, you remain obligated to ensure that it is
|
|
||||||
available for as long as needed to satisfy these requirements.
|
|
||||||
|
|
||||||
e) Convey the object code using peer-to-peer transmission, provided
|
|
||||||
you inform other peers where the object code and Corresponding
|
|
||||||
Source of the work are being offered to the general public at no
|
|
||||||
charge under subsection 6d.
|
|
||||||
|
|
||||||
A separable portion of the object code, whose source code is excluded
|
|
||||||
from the Corresponding Source as a System Library, need not be
|
|
||||||
included in conveying the object code work.
|
|
||||||
|
|
||||||
A "User Product" is either (1) a "consumer product", which means any
|
|
||||||
tangible personal property which is normally used for personal, family,
|
|
||||||
or household purposes, or (2) anything designed or sold for incorporation
|
|
||||||
into a dwelling. In determining whether a product is a consumer product,
|
|
||||||
doubtful cases shall be resolved in favor of coverage. For a particular
|
|
||||||
product received by a particular user, "normally used" refers to a
|
|
||||||
typical or common use of that class of product, regardless of the status
|
|
||||||
of the particular user or of the way in which the particular user
|
|
||||||
actually uses, or expects or is expected to use, the product. A product
|
|
||||||
is a consumer product regardless of whether the product has substantial
|
|
||||||
commercial, industrial or non-consumer uses, unless such uses represent
|
|
||||||
the only significant mode of use of the product.
|
|
||||||
|
|
||||||
"Installation Information" for a User Product means any methods,
|
|
||||||
procedures, authorization keys, or other information required to install
|
|
||||||
and execute modified versions of a covered work in that User Product from
|
|
||||||
a modified version of its Corresponding Source. The information must
|
|
||||||
suffice to ensure that the continued functioning of the modified object
|
|
||||||
code is in no case prevented or interfered with solely because
|
|
||||||
modification has been made.
|
|
||||||
|
|
||||||
If you convey an object code work under this section in, or with, or
|
|
||||||
specifically for use in, a User Product, and the conveying occurs as
|
|
||||||
part of a transaction in which the right of possession and use of the
|
|
||||||
User Product is transferred to the recipient in perpetuity or for a
|
|
||||||
fixed term (regardless of how the transaction is characterized), the
|
|
||||||
Corresponding Source conveyed under this section must be accompanied
|
|
||||||
by the Installation Information. But this requirement does not apply
|
|
||||||
if neither you nor any third party retains the ability to install
|
|
||||||
modified object code on the User Product (for example, the work has
|
|
||||||
been installed in ROM).
|
|
||||||
|
|
||||||
The requirement to provide Installation Information does not include a
|
|
||||||
requirement to continue to provide support service, warranty, or updates
|
|
||||||
for a work that has been modified or installed by the recipient, or for
|
|
||||||
the User Product in which it has been modified or installed. Access to a
|
|
||||||
network may be denied when the modification itself materially and
|
|
||||||
adversely affects the operation of the network or violates the rules and
|
|
||||||
protocols for communication across the network.
|
|
||||||
|
|
||||||
Corresponding Source conveyed, and Installation Information provided,
|
|
||||||
in accord with this section must be in a format that is publicly
|
|
||||||
documented (and with an implementation available to the public in
|
|
||||||
source code form), and must require no special password or key for
|
|
||||||
unpacking, reading or copying.
|
|
||||||
|
|
||||||
7. Additional Terms.
|
|
||||||
|
|
||||||
"Additional permissions" are terms that supplement the terms of this
|
|
||||||
License by making exceptions from one or more of its conditions.
|
|
||||||
Additional permissions that are applicable to the entire Program shall
|
|
||||||
be treated as though they were included in this License, to the extent
|
|
||||||
that they are valid under applicable law. If additional permissions
|
|
||||||
apply only to part of the Program, that part may be used separately
|
|
||||||
under those permissions, but the entire Program remains governed by
|
|
||||||
this License without regard to the additional permissions.
|
|
||||||
|
|
||||||
When you convey a copy of a covered work, you may at your option
|
|
||||||
remove any additional permissions from that copy, or from any part of
|
|
||||||
it. (Additional permissions may be written to require their own
|
|
||||||
removal in certain cases when you modify the work.) You may place
|
|
||||||
additional permissions on material, added by you to a covered work,
|
|
||||||
for which you have or can give appropriate copyright permission.
|
|
||||||
|
|
||||||
Notwithstanding any other provision of this License, for material you
|
|
||||||
add to a covered work, you may (if authorized by the copyright holders of
|
|
||||||
that material) supplement the terms of this License with terms:
|
|
||||||
|
|
||||||
a) Disclaiming warranty or limiting liability differently from the
|
|
||||||
terms of sections 15 and 16 of this License; or
|
|
||||||
|
|
||||||
b) Requiring preservation of specified reasonable legal notices or
|
|
||||||
author attributions in that material or in the Appropriate Legal
|
|
||||||
Notices displayed by works containing it; or
|
|
||||||
|
|
||||||
c) Prohibiting misrepresentation of the origin of that material, or
|
|
||||||
requiring that modified versions of such material be marked in
|
|
||||||
reasonable ways as different from the original version; or
|
|
||||||
|
|
||||||
d) Limiting the use for publicity purposes of names of licensors or
|
|
||||||
authors of the material; or
|
|
||||||
|
|
||||||
e) Declining to grant rights under trademark law for use of some
|
|
||||||
trade names, trademarks, or service marks; or
|
|
||||||
|
|
||||||
f) Requiring indemnification of licensors and authors of that
|
|
||||||
material by anyone who conveys the material (or modified versions of
|
|
||||||
it) with contractual assumptions of liability to the recipient, for
|
|
||||||
any liability that these contractual assumptions directly impose on
|
|
||||||
those licensors and authors.
|
|
||||||
|
|
||||||
All other non-permissive additional terms are considered "further
|
|
||||||
restrictions" within the meaning of section 10. If the Program as you
|
|
||||||
received it, or any part of it, contains a notice stating that it is
|
|
||||||
governed by this License along with a term that is a further
|
|
||||||
restriction, you may remove that term. If a license document contains
|
|
||||||
a further restriction but permits relicensing or conveying under this
|
|
||||||
License, you may add to a covered work material governed by the terms
|
|
||||||
of that license document, provided that the further restriction does
|
|
||||||
not survive such relicensing or conveying.
|
|
||||||
|
|
||||||
If you add terms to a covered work in accord with this section, you
|
|
||||||
must place, in the relevant source files, a statement of the
|
|
||||||
additional terms that apply to those files, or a notice indicating
|
|
||||||
where to find the applicable terms.
|
|
||||||
|
|
||||||
Additional terms, permissive or non-permissive, may be stated in the
|
|
||||||
form of a separately written license, or stated as exceptions;
|
|
||||||
the above requirements apply either way.
|
|
||||||
|
|
||||||
8. Termination.
|
|
||||||
|
|
||||||
You may not propagate or modify a covered work except as expressly
|
|
||||||
provided under this License. Any attempt otherwise to propagate or
|
|
||||||
modify it is void, and will automatically terminate your rights under
|
|
||||||
this License (including any patent licenses granted under the third
|
|
||||||
paragraph of section 11).
|
|
||||||
|
|
||||||
However, if you cease all violation of this License, then your
|
|
||||||
license from a particular copyright holder is reinstated (a)
|
|
||||||
provisionally, unless and until the copyright holder explicitly and
|
|
||||||
finally terminates your license, and (b) permanently, if the copyright
|
|
||||||
holder fails to notify you of the violation by some reasonable means
|
|
||||||
prior to 60 days after the cessation.
|
|
||||||
|
|
||||||
Moreover, your license from a particular copyright holder is
|
|
||||||
reinstated permanently if the copyright holder notifies you of the
|
|
||||||
violation by some reasonable means, this is the first time you have
|
|
||||||
received notice of violation of this License (for any work) from that
|
|
||||||
copyright holder, and you cure the violation prior to 30 days after
|
|
||||||
your receipt of the notice.
|
|
||||||
|
|
||||||
Termination of your rights under this section does not terminate the
|
|
||||||
licenses of parties who have received copies or rights from you under
|
|
||||||
this License. If your rights have been terminated and not permanently
|
|
||||||
reinstated, you do not qualify to receive new licenses for the same
|
|
||||||
material under section 10.
|
|
||||||
|
|
||||||
9. Acceptance Not Required for Having Copies.
|
|
||||||
|
|
||||||
You are not required to accept this License in order to receive or
|
|
||||||
run a copy of the Program. Ancillary propagation of a covered work
|
|
||||||
occurring solely as a consequence of using peer-to-peer transmission
|
|
||||||
to receive a copy likewise does not require acceptance. However,
|
|
||||||
nothing other than this License grants you permission to propagate or
|
|
||||||
modify any covered work. These actions infringe copyright if you do
|
|
||||||
not accept this License. Therefore, by modifying or propagating a
|
|
||||||
covered work, you indicate your acceptance of this License to do so.
|
|
||||||
|
|
||||||
10. Automatic Licensing of Downstream Recipients.
|
|
||||||
|
|
||||||
Each time you convey a covered work, the recipient automatically
|
|
||||||
receives a license from the original licensors, to run, modify and
|
|
||||||
propagate that work, subject to this License. You are not responsible
|
|
||||||
for enforcing compliance by third parties with this License.
|
|
||||||
|
|
||||||
An "entity transaction" is a transaction transferring control of an
|
|
||||||
organization, or substantially all assets of one, or subdividing an
|
|
||||||
organization, or merging organizations. If propagation of a covered
|
|
||||||
work results from an entity transaction, each party to that
|
|
||||||
transaction who receives a copy of the work also receives whatever
|
|
||||||
licenses to the work the party's predecessor in interest had or could
|
|
||||||
give under the previous paragraph, plus a right to possession of the
|
|
||||||
Corresponding Source of the work from the predecessor in interest, if
|
|
||||||
the predecessor has it or can get it with reasonable efforts.
|
|
||||||
|
|
||||||
You may not impose any further restrictions on the exercise of the
|
|
||||||
rights granted or affirmed under this License. For example, you may
|
|
||||||
not impose a license fee, royalty, or other charge for exercise of
|
|
||||||
rights granted under this License, and you may not initiate litigation
|
|
||||||
(including a cross-claim or counterclaim in a lawsuit) alleging that
|
|
||||||
any patent claim is infringed by making, using, selling, offering for
|
|
||||||
sale, or importing the Program or any portion of it.
|
|
||||||
|
|
||||||
11. Patents.
|
|
||||||
|
|
||||||
A "contributor" is a copyright holder who authorizes use under this
|
|
||||||
License of the Program or a work on which the Program is based. The
|
|
||||||
work thus licensed is called the contributor's "contributor version".
|
|
||||||
|
|
||||||
A contributor's "essential patent claims" are all patent claims
|
|
||||||
owned or controlled by the contributor, whether already acquired or
|
|
||||||
hereafter acquired, that would be infringed by some manner, permitted
|
|
||||||
by this License, of making, using, or selling its contributor version,
|
|
||||||
but do not include claims that would be infringed only as a
|
|
||||||
consequence of further modification of the contributor version. For
|
|
||||||
purposes of this definition, "control" includes the right to grant
|
|
||||||
patent sublicenses in a manner consistent with the requirements of
|
|
||||||
this License.
|
|
||||||
|
|
||||||
Each contributor grants you a non-exclusive, worldwide, royalty-free
|
|
||||||
patent license under the contributor's essential patent claims, to
|
|
||||||
make, use, sell, offer for sale, import and otherwise run, modify and
|
|
||||||
propagate the contents of its contributor version.
|
|
||||||
|
|
||||||
In the following three paragraphs, a "patent license" is any express
|
|
||||||
agreement or commitment, however denominated, not to enforce a patent
|
|
||||||
(such as an express permission to practice a patent or covenant not to
|
|
||||||
sue for patent infringement). To "grant" such a patent license to a
|
|
||||||
party means to make such an agreement or commitment not to enforce a
|
|
||||||
patent against the party.
|
|
||||||
|
|
||||||
If you convey a covered work, knowingly relying on a patent license,
|
|
||||||
and the Corresponding Source of the work is not available for anyone
|
|
||||||
to copy, free of charge and under the terms of this License, through a
|
|
||||||
publicly available network server or other readily accessible means,
|
|
||||||
then you must either (1) cause the Corresponding Source to be so
|
|
||||||
available, or (2) arrange to deprive yourself of the benefit of the
|
|
||||||
patent license for this particular work, or (3) arrange, in a manner
|
|
||||||
consistent with the requirements of this License, to extend the patent
|
|
||||||
license to downstream recipients. "Knowingly relying" means you have
|
|
||||||
actual knowledge that, but for the patent license, your conveying the
|
|
||||||
covered work in a country, or your recipient's use of the covered work
|
|
||||||
in a country, would infringe one or more identifiable patents in that
|
|
||||||
country that you have reason to believe are valid.
|
|
||||||
|
|
||||||
If, pursuant to or in connection with a single transaction or
|
|
||||||
arrangement, you convey, or propagate by procuring conveyance of, a
|
|
||||||
covered work, and grant a patent license to some of the parties
|
|
||||||
receiving the covered work authorizing them to use, propagate, modify
|
|
||||||
or convey a specific copy of the covered work, then the patent license
|
|
||||||
you grant is automatically extended to all recipients of the covered
|
|
||||||
work and works based on it.
|
|
||||||
|
|
||||||
A patent license is "discriminatory" if it does not include within
|
|
||||||
the scope of its coverage, prohibits the exercise of, or is
|
|
||||||
conditioned on the non-exercise of one or more of the rights that are
|
|
||||||
specifically granted under this License. You may not convey a covered
|
|
||||||
work if you are a party to an arrangement with a third party that is
|
|
||||||
in the business of distributing software, under which you make payment
|
|
||||||
to the third party based on the extent of your activity of conveying
|
|
||||||
the work, and under which the third party grants, to any of the
|
|
||||||
parties who would receive the covered work from you, a discriminatory
|
|
||||||
patent license (a) in connection with copies of the covered work
|
|
||||||
conveyed by you (or copies made from those copies), or (b) primarily
|
|
||||||
for and in connection with specific products or compilations that
|
|
||||||
contain the covered work, unless you entered into that arrangement,
|
|
||||||
or that patent license was granted, prior to 28 March 2007.
|
|
||||||
|
|
||||||
Nothing in this License shall be construed as excluding or limiting
|
|
||||||
any implied license or other defenses to infringement that may
|
|
||||||
otherwise be available to you under applicable patent law.
|
|
||||||
|
|
||||||
12. No Surrender of Others' Freedom.
|
|
||||||
|
|
||||||
If conditions are imposed on you (whether by court order, agreement or
|
|
||||||
otherwise) that contradict the conditions of this License, they do not
|
|
||||||
excuse you from the conditions of this License. If you cannot convey a
|
|
||||||
covered work so as to satisfy simultaneously your obligations under this
|
|
||||||
License and any other pertinent obligations, then as a consequence you may
|
|
||||||
not convey it at all. For example, if you agree to terms that obligate you
|
|
||||||
to collect a royalty for further conveying from those to whom you convey
|
|
||||||
the Program, the only way you could satisfy both those terms and this
|
|
||||||
License would be to refrain entirely from conveying the Program.
|
|
||||||
|
|
||||||
13. Remote Network Interaction; Use with the GNU General Public License.
|
|
||||||
|
|
||||||
Notwithstanding any other provision of this License, if you modify the
|
|
||||||
Program, your modified version must prominently offer all users
|
|
||||||
interacting with it remotely through a computer network (if your version
|
|
||||||
supports such interaction) an opportunity to receive the Corresponding
|
|
||||||
Source of your version by providing access to the Corresponding Source
|
|
||||||
from a network server at no charge, through some standard or customary
|
|
||||||
means of facilitating copying of software. This Corresponding Source
|
|
||||||
shall include the Corresponding Source for any work covered by version 3
|
|
||||||
of the GNU General Public License that is incorporated pursuant to the
|
|
||||||
following paragraph.
|
|
||||||
|
|
||||||
Notwithstanding any other provision of this License, you have
|
|
||||||
permission to link or combine any covered work with a work licensed
|
|
||||||
under version 3 of the GNU General Public License into a single
|
|
||||||
combined work, and to convey the resulting work. The terms of this
|
|
||||||
License will continue to apply to the part which is the covered work,
|
|
||||||
but the work with which it is combined will remain governed by version
|
|
||||||
3 of the GNU General Public License.
|
|
||||||
|
|
||||||
14. Revised Versions of this License.
|
|
||||||
|
|
||||||
The Free Software Foundation may publish revised and/or new versions of
|
|
||||||
the GNU Affero General Public License from time to time. Such new versions
|
|
||||||
will be similar in spirit to the present version, but may differ in detail to
|
|
||||||
address new problems or concerns.
|
|
||||||
|
|
||||||
Each version is given a distinguishing version number. If the
|
|
||||||
Program specifies that a certain numbered version of the GNU Affero General
|
|
||||||
Public License "or any later version" applies to it, you have the
|
|
||||||
option of following the terms and conditions either of that numbered
|
|
||||||
version or of any later version published by the Free Software
|
|
||||||
Foundation. If the Program does not specify a version number of the
|
|
||||||
GNU Affero General Public License, you may choose any version ever published
|
|
||||||
by the Free Software Foundation.
|
|
||||||
|
|
||||||
If the Program specifies that a proxy can decide which future
|
|
||||||
versions of the GNU Affero General Public License can be used, that proxy's
|
|
||||||
public statement of acceptance of a version permanently authorizes you
|
|
||||||
to choose that version for the Program.
|
|
||||||
|
|
||||||
Later license versions may give you additional or different
|
|
||||||
permissions. However, no additional obligations are imposed on any
|
|
||||||
author or copyright holder as a result of your choosing to follow a
|
|
||||||
later version.
|
|
||||||
|
|
||||||
15. Disclaimer of Warranty.
|
|
||||||
|
|
||||||
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
|
|
||||||
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
|
|
||||||
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
|
|
||||||
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
|
|
||||||
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
|
|
||||||
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
|
|
||||||
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
|
|
||||||
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
|
|
||||||
|
|
||||||
16. Limitation of Liability.
|
|
||||||
|
|
||||||
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
|
|
||||||
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
|
|
||||||
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
|
|
||||||
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
|
|
||||||
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
|
|
||||||
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
|
|
||||||
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
|
|
||||||
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
|
|
||||||
SUCH DAMAGES.
|
|
||||||
|
|
||||||
17. Interpretation of Sections 15 and 16.
|
|
||||||
|
|
||||||
If the disclaimer of warranty and limitation of liability provided
|
|
||||||
above cannot be given local legal effect according to their terms,
|
|
||||||
reviewing courts shall apply local law that most closely approximates
|
|
||||||
an absolute waiver of all civil liability in connection with the
|
|
||||||
Program, unless a warranty or assumption of liability accompanies a
|
|
||||||
copy of the Program in return for a fee.
|
|
||||||
|
|
||||||
END OF TERMS AND CONDITIONS
|
|
||||||
|
|
||||||
How to Apply These Terms to Your New Programs
|
|
||||||
|
|
||||||
If you develop a new program, and you want it to be of the greatest
|
|
||||||
possible use to the public, the best way to achieve this is to make it
|
|
||||||
free software which everyone can redistribute and change under these terms.
|
|
||||||
|
|
||||||
To do so, attach the following notices to the program. It is safest
|
|
||||||
to attach them to the start of each source file to most effectively
|
|
||||||
state the exclusion of warranty; and each file should have at least
|
|
||||||
the "copyright" line and a pointer to where the full notice is found.
|
|
||||||
|
|
||||||
<one line to give the program's name and a brief idea of what it does.>
|
|
||||||
Copyright (C) <year> <name of author>
|
|
||||||
|
|
||||||
This program is free software: you can redistribute it and/or modify
|
|
||||||
it under the terms of the GNU Affero General Public License as published by
|
|
||||||
the Free Software Foundation, either version 3 of the License, or
|
|
||||||
(at your option) any later version.
|
|
||||||
|
|
||||||
This program is distributed in the hope that it will be useful,
|
|
||||||
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
||||||
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
||||||
GNU Affero General Public License for more details.
|
|
||||||
|
|
||||||
You should have received a copy of the GNU Affero General Public License
|
|
||||||
along with this program. If not, see <http://www.gnu.org/licenses/>.
|
|
||||||
|
|
||||||
Also add information on how to contact you by electronic and paper mail.
|
|
||||||
|
|
||||||
If your software can interact with users remotely through a computer
|
|
||||||
network, you should also make sure that it provides a way for users to
|
|
||||||
get its source. For example, if your program is a web application, its
|
|
||||||
interface could display a "Source" link that leads users to an archive
|
|
||||||
of the code. There are many ways you could offer source, and different
|
|
||||||
solutions will be better for different programs; see section 13 for the
|
|
||||||
specific requirements.
|
|
||||||
|
|
||||||
You should also get your employer (if you work as a programmer) or school,
|
|
||||||
if any, to sign a "copyright disclaimer" for the program, if necessary.
|
|
||||||
For more information on this, and how to apply and follow the GNU AGPL, see
|
|
||||||
<http://www.gnu.org/licenses/>.
|
|
|
@ -1,3 +0,0 @@
|
||||||
/* eslint-disable no-undef */
|
|
||||||
|
|
||||||
rs.initiate({ _id: 'overleaf', members: [{ _id: 0, host: 'mongo:27017' }] })
|
|
3
develop/.gitignore
vendored
3
develop/.gitignore
vendored
|
@ -1,3 +0,0 @@
|
||||||
/compiles/*
|
|
||||||
!.gitkeep
|
|
||||||
.env
|
|
|
@ -1,90 +0,0 @@
|
||||||
# Overleaf Community Edition, development environment
|
|
||||||
|
|
||||||
## Building and running
|
|
||||||
|
|
||||||
In this `develop` directory, build the services:
|
|
||||||
|
|
||||||
```shell
|
|
||||||
bin/build
|
|
||||||
```
|
|
||||||
|
|
||||||
> [!NOTE]
|
|
||||||
> If Docker is running out of RAM while building the services in parallel, create a `.env` file in this directory containing `COMPOSE_PARALLEL_LIMIT=1`.
|
|
||||||
|
|
||||||
Then start the services:
|
|
||||||
|
|
||||||
```shell
|
|
||||||
bin/up
|
|
||||||
```
|
|
||||||
|
|
||||||
Once the services are running, open <http://localhost/launchpad> to create the first admin account.
|
|
||||||
|
|
||||||
## TeX Live
|
|
||||||
|
|
||||||
Compiling a PDF requires building a TeX Live image to handle the compilation inside Docker:
|
|
||||||
|
|
||||||
```shell
|
|
||||||
docker build texlive -t texlive-full
|
|
||||||
```
|
|
||||||
|
|
||||||
> [!NOTE]
|
|
||||||
> To compile on a macOS host, you may need to override the path to the Docker socket by creating a `.env` file in this directory, containing
|
|
||||||
> `DOCKER_SOCKET_PATH=/var/run/docker.sock.raw`
|
|
||||||
|
|
||||||
## Development
|
|
||||||
|
|
||||||
To avoid running `bin/build && bin/up` after every code change, you can run Overleaf
|
|
||||||
Community Edition in _development mode_, where services will automatically update on code changes.
|
|
||||||
|
|
||||||
To do this, use the included `bin/dev` script:
|
|
||||||
|
|
||||||
```shell
|
|
||||||
bin/dev
|
|
||||||
```
|
|
||||||
|
|
||||||
This will start all services using `nodemon`, which will automatically monitor the code and restart the services as necessary.
|
|
||||||
|
|
||||||
To improve performance, you can start only a subset of the services in development mode by providing a space-separated list to the `bin/dev` script:
|
|
||||||
|
|
||||||
```shell
|
|
||||||
bin/dev [service1] [service2] ... [serviceN]
|
|
||||||
```
|
|
||||||
|
|
||||||
> [!NOTE]
|
|
||||||
> Starting the `web` service in _development mode_ will only update the `web`
|
|
||||||
> service when backend code changes. In order to automatically update frontend
|
|
||||||
> code as well, make sure to start the `webpack` service in _development mode_
|
|
||||||
> as well.
|
|
||||||
|
|
||||||
If no services are named, all services will start in development mode.
|
|
||||||
|
|
||||||
## Debugging
|
|
||||||
|
|
||||||
When run in _development mode_ most services expose a debugging port to which
|
|
||||||
you can attach a debugger such as
|
|
||||||
[the inspector in Chrome's Dev Tools](chrome://inspect/) or one integrated into
|
|
||||||
an IDE. The following table shows the port exposed on the **host machine** for
|
|
||||||
each service:
|
|
||||||
|
|
||||||
| Service | Port |
|
|
||||||
| ------------------ | ---- |
|
|
||||||
| `web` | 9229 |
|
|
||||||
| `clsi` | 9230 |
|
|
||||||
| `chat` | 9231 |
|
|
||||||
| `contacts` | 9232 |
|
|
||||||
| `docstore` | 9233 |
|
|
||||||
| `document-updater` | 9234 |
|
|
||||||
| `filestore` | 9235 |
|
|
||||||
| `notifications` | 9236 |
|
|
||||||
| `real-time` | 9237 |
|
|
||||||
| `references` | 9238 |
|
|
||||||
| `history-v1` | 9239 |
|
|
||||||
| `project-history` | 9240 |
|
|
||||||
|
|
||||||
To attach to a service using Chrome's _remote debugging_, go to
|
|
||||||
<chrome://inspect/> and make sure _Discover network targets_ is checked. Next
|
|
||||||
click _Configure..._ and add an entry `localhost:[service port]` for each of the
|
|
||||||
services you want to attach a debugger to.
|
|
||||||
|
|
||||||
After adding an entry, the service will show up as a _Remote Target_ that you
|
|
||||||
can inspect and debug.
|
|
|
@ -1,3 +0,0 @@
|
||||||
#!/usr/bin/env bash
|
|
||||||
|
|
||||||
docker compose build --pull "$@"
|
|
|
@ -1,3 +0,0 @@
|
||||||
#!/usr/bin/env bash
|
|
||||||
|
|
||||||
docker-compose -f docker-compose.yml -f docker-compose.dev.yml up --no-deps --detach "$@"
|
|
|
@ -1,3 +0,0 @@
|
||||||
#!/usr/bin/env bash
|
|
||||||
|
|
||||||
docker compose down "$@"
|
|
|
@ -1,9 +0,0 @@
|
||||||
#!/usr/bin/env bash
|
|
||||||
|
|
||||||
docker compose logs --follow --tail 10 --no-color "$@" \
|
|
||||||
| ggrep --line-buffered --invert-match "global.gc" \
|
|
||||||
| ggrep --line-buffered --invert-match "health.check" \
|
|
||||||
| ggrep --line-buffered --invert-match "slow event loop" \
|
|
||||||
| ggrep --line-buffered --invert-match "process.memoryUsage" \
|
|
||||||
| ggrep --line-buffered --only-matching "[{].*" \
|
|
||||||
| bunyan --output short
|
|
|
@ -1,3 +0,0 @@
|
||||||
#!/usr/bin/env bash
|
|
||||||
|
|
||||||
docker compose exec -it "$@" /bin/bash
|
|
|
@ -1,3 +0,0 @@
|
||||||
#!/usr/bin/env bash
|
|
||||||
|
|
||||||
docker compose up --detach "$@"
|
|
|
@ -1,21 +0,0 @@
|
||||||
CHAT_HOST=chat
|
|
||||||
CLSI_HOST=clsi
|
|
||||||
CONTACTS_HOST=contacts
|
|
||||||
DOCSTORE_HOST=docstore
|
|
||||||
DOCUMENT_UPDATER_HOST=document-updater
|
|
||||||
FILESTORE_HOST=filestore
|
|
||||||
GRACEFUL_SHUTDOWN_DELAY_SECONDS=0
|
|
||||||
HISTORY_V1_HOST=history-v1
|
|
||||||
LISTEN_ADDRESS=0.0.0.0
|
|
||||||
MONGO_HOST=mongo
|
|
||||||
MONGO_URL=mongodb://mongo/sharelatex?directConnection=true
|
|
||||||
NOTIFICATIONS_HOST=notifications
|
|
||||||
PROJECT_HISTORY_HOST=project-history
|
|
||||||
REALTIME_HOST=real-time
|
|
||||||
REDIS_HOST=redis
|
|
||||||
REFERENCES_HOST=references
|
|
||||||
SESSION_SECRET=foo
|
|
||||||
WEBPACK_HOST=webpack
|
|
||||||
WEB_API_PASSWORD=overleaf
|
|
||||||
WEB_API_USER=overleaf
|
|
||||||
WEB_HOST=web
|
|
|
@ -1,150 +0,0 @@
|
||||||
services:
|
|
||||||
clsi:
|
|
||||||
command: ["node", "--watch", "app.js"]
|
|
||||||
environment:
|
|
||||||
- NODE_OPTIONS=--inspect=0.0.0.0:9229
|
|
||||||
ports:
|
|
||||||
- "127.0.0.1:9230:9229"
|
|
||||||
volumes:
|
|
||||||
- ../services/clsi/app:/overleaf/services/clsi/app
|
|
||||||
- ../services/clsi/app.js:/overleaf/services/clsi/app.js
|
|
||||||
- ../services/clsi/config:/overleaf/services/clsi/config
|
|
||||||
|
|
||||||
chat:
|
|
||||||
command: ["node", "--watch", "app.js"]
|
|
||||||
environment:
|
|
||||||
- NODE_OPTIONS=--inspect=0.0.0.0:9229
|
|
||||||
ports:
|
|
||||||
- "127.0.0.1:9231:9229"
|
|
||||||
volumes:
|
|
||||||
- ../services/chat/app:/overleaf/services/chat/app
|
|
||||||
- ../services/chat/app.js:/overleaf/services/chat/app.js
|
|
||||||
- ../services/chat/config:/overleaf/services/chat/config
|
|
||||||
|
|
||||||
contacts:
|
|
||||||
command: ["node", "--watch", "app.js"]
|
|
||||||
environment:
|
|
||||||
- NODE_OPTIONS=--inspect=0.0.0.0:9229
|
|
||||||
ports:
|
|
||||||
- "127.0.0.1:9232:9229"
|
|
||||||
volumes:
|
|
||||||
- ../services/contacts/app:/overleaf/services/contacts/app
|
|
||||||
- ../services/contacts/app.js:/overleaf/services/contacts/app.js
|
|
||||||
- ../services/contacts/config:/overleaf/services/contacts/config
|
|
||||||
|
|
||||||
docstore:
|
|
||||||
command: ["node", "--watch", "app.js"]
|
|
||||||
environment:
|
|
||||||
- NODE_OPTIONS=--inspect=0.0.0.0:9229
|
|
||||||
ports:
|
|
||||||
- "127.0.0.1:9233:9229"
|
|
||||||
volumes:
|
|
||||||
- ../services/docstore/app:/overleaf/services/docstore/app
|
|
||||||
- ../services/docstore/app.js:/overleaf/services/docstore/app.js
|
|
||||||
- ../services/docstore/config:/overleaf/services/docstore/config
|
|
||||||
|
|
||||||
document-updater:
|
|
||||||
command: ["node", "--watch", "app.js"]
|
|
||||||
environment:
|
|
||||||
- NODE_OPTIONS=--inspect=0.0.0.0:9229
|
|
||||||
ports:
|
|
||||||
- "127.0.0.1:9234:9229"
|
|
||||||
volumes:
|
|
||||||
- ../services/document-updater/app:/overleaf/services/document-updater/app
|
|
||||||
- ../services/document-updater/app.js:/overleaf/services/document-updater/app.js
|
|
||||||
- ../services/document-updater/config:/overleaf/services/document-updater/config
|
|
||||||
|
|
||||||
filestore:
|
|
||||||
command: ["node", "--watch", "app.js"]
|
|
||||||
environment:
|
|
||||||
- NODE_OPTIONS=--inspect=0.0.0.0:9229
|
|
||||||
ports:
|
|
||||||
- "127.0.0.1:9235:9229"
|
|
||||||
volumes:
|
|
||||||
- ../services/filestore/app:/overleaf/services/filestore/app
|
|
||||||
- ../services/filestore/app.js:/overleaf/services/filestore/app.js
|
|
||||||
- ../services/filestore/config:/overleaf/services/filestore/config
|
|
||||||
|
|
||||||
history-v1:
|
|
||||||
command: ["node", "--watch", "app.js"]
|
|
||||||
environment:
|
|
||||||
- NODE_OPTIONS=--inspect=0.0.0.0:9229
|
|
||||||
ports:
|
|
||||||
- "127.0.0.1:9239:9229"
|
|
||||||
volumes:
|
|
||||||
- ../services/history-v1/api:/overleaf/services/history-v1/api
|
|
||||||
- ../services/history-v1/app.js:/overleaf/services/history-v1/app.js
|
|
||||||
- ../services/history-v1/config:/overleaf/services/history-v1/config
|
|
||||||
- ../services/history-v1/storage:/overleaf/services/history-v1/storage
|
|
||||||
- ../services/history-v1/knexfile.js:/overleaf/services/history-v1/knexfile.js
|
|
||||||
- ../services/history-v1/migrations:/overleaf/services/history-v1/migrations
|
|
||||||
|
|
||||||
notifications:
|
|
||||||
command: ["node", "--watch", "app.js"]
|
|
||||||
environment:
|
|
||||||
- NODE_OPTIONS=--inspect=0.0.0.0:9229
|
|
||||||
ports:
|
|
||||||
- "127.0.0.1:9236:9229"
|
|
||||||
volumes:
|
|
||||||
- ../services/notifications/app:/overleaf/services/notifications/app
|
|
||||||
- ../services/notifications/app.js:/overleaf/services/notifications/app.js
|
|
||||||
- ../services/notifications/config:/overleaf/services/notifications/config
|
|
||||||
|
|
||||||
project-history:
|
|
||||||
command: ["node", "--watch", "app.js"]
|
|
||||||
environment:
|
|
||||||
- NODE_OPTIONS=--inspect=0.0.0.0:9229
|
|
||||||
ports:
|
|
||||||
- "127.0.0.1:9240:9229"
|
|
||||||
volumes:
|
|
||||||
- ../services/project-history/app:/overleaf/services/project-history/app
|
|
||||||
- ../services/project-history/app.js:/overleaf/services/project-history/app.js
|
|
||||||
- ../services/project-history/config:/overleaf/services/project-history/config
|
|
||||||
|
|
||||||
real-time:
|
|
||||||
command: ["node", "--watch", "app.js"]
|
|
||||||
environment:
|
|
||||||
- NODE_OPTIONS=--inspect=0.0.0.0:9229
|
|
||||||
ports:
|
|
||||||
- "127.0.0.1:9237:9229"
|
|
||||||
volumes:
|
|
||||||
- ../services/real-time/app:/overleaf/services/real-time/app
|
|
||||||
- ../services/real-time/app.js:/overleaf/services/real-time/app.js
|
|
||||||
- ../services/real-time/config:/overleaf/services/real-time/config
|
|
||||||
|
|
||||||
references:
|
|
||||||
command: ["node", "--watch", "app.js"]
|
|
||||||
environment:
|
|
||||||
- NODE_OPTIONS=--inspect=0.0.0.0:9229
|
|
||||||
ports:
|
|
||||||
- "127.0.0.1:9238:9229"
|
|
||||||
volumes:
|
|
||||||
- ../services/references/app:/overleaf/services/references/app
|
|
||||||
- ../services/references/config:/overleaf/services/references/config
|
|
||||||
- ../services/references/app.js:/overleaf/services/references/app.js
|
|
||||||
|
|
||||||
web:
|
|
||||||
command: ["node", "--watch", "app.js", "--watch-locales"]
|
|
||||||
environment:
|
|
||||||
- NODE_OPTIONS=--inspect=0.0.0.0:9229
|
|
||||||
ports:
|
|
||||||
- "127.0.0.1:9229:9229"
|
|
||||||
volumes:
|
|
||||||
- ../services/web/app:/overleaf/services/web/app
|
|
||||||
- ../services/web/app.mjs:/overleaf/services/web/app.mjs
|
|
||||||
- ../services/web/config:/overleaf/services/web/config
|
|
||||||
- ../services/web/locales:/overleaf/services/web/locales
|
|
||||||
- ../services/web/modules:/overleaf/services/web/modules
|
|
||||||
- ../services/web/public:/overleaf/services/web/public
|
|
||||||
|
|
||||||
webpack:
|
|
||||||
volumes:
|
|
||||||
- ../services/web/app:/overleaf/services/web/app
|
|
||||||
- ../services/web/config:/overleaf/services/web/config
|
|
||||||
- ../services/web/frontend:/overleaf/services/web/frontend
|
|
||||||
- ../services/web/locales:/overleaf/services/web/locales
|
|
||||||
- ../services/web/modules:/overleaf/services/web/modules
|
|
||||||
- ../services/web/public:/overleaf/services/web/public
|
|
||||||
- ../services/web/transform:/overleaf/services/web/transform
|
|
||||||
- ../services/web/types:/overleaf/services/web/types
|
|
||||||
- ../services/web/webpack-plugins:/overleaf/services/web/webpack-plugins
|
|
|
@ -1,182 +0,0 @@
|
||||||
volumes:
|
|
||||||
clsi-cache:
|
|
||||||
clsi-output:
|
|
||||||
filestore-public-files:
|
|
||||||
filestore-template-files:
|
|
||||||
filestore-uploads:
|
|
||||||
filestore-user-files:
|
|
||||||
mongo-data:
|
|
||||||
redis-data:
|
|
||||||
sharelatex-data:
|
|
||||||
web-data:
|
|
||||||
history-v1-buckets:
|
|
||||||
|
|
||||||
services:
|
|
||||||
chat:
|
|
||||||
build:
|
|
||||||
context: ..
|
|
||||||
dockerfile: services/chat/Dockerfile
|
|
||||||
env_file:
|
|
||||||
- dev.env
|
|
||||||
|
|
||||||
clsi:
|
|
||||||
build:
|
|
||||||
context: ..
|
|
||||||
dockerfile: services/clsi/Dockerfile
|
|
||||||
env_file:
|
|
||||||
- dev.env
|
|
||||||
environment:
|
|
||||||
- DOCKER_RUNNER=true
|
|
||||||
- TEXLIVE_IMAGE=texlive-full # docker build texlive -t texlive-full
|
|
||||||
- COMPILES_HOST_DIR=${PWD}/compiles
|
|
||||||
user: root
|
|
||||||
volumes:
|
|
||||||
- ${PWD}/compiles:/overleaf/services/clsi/compiles
|
|
||||||
- ${DOCKER_SOCKET_PATH:-/var/run/docker.sock}:/var/run/docker.sock
|
|
||||||
- clsi-cache:/overleaf/services/clsi/cache
|
|
||||||
- clsi-output:/overleaf/services/clsi/output
|
|
||||||
|
|
||||||
contacts:
|
|
||||||
build:
|
|
||||||
context: ..
|
|
||||||
dockerfile: services/contacts/Dockerfile
|
|
||||||
env_file:
|
|
||||||
- dev.env
|
|
||||||
|
|
||||||
docstore:
|
|
||||||
build:
|
|
||||||
context: ..
|
|
||||||
dockerfile: services/docstore/Dockerfile
|
|
||||||
env_file:
|
|
||||||
- dev.env
|
|
||||||
|
|
||||||
document-updater:
|
|
||||||
build:
|
|
||||||
context: ..
|
|
||||||
dockerfile: services/document-updater/Dockerfile
|
|
||||||
env_file:
|
|
||||||
- dev.env
|
|
||||||
|
|
||||||
filestore:
|
|
||||||
build:
|
|
||||||
context: ..
|
|
||||||
dockerfile: services/filestore/Dockerfile
|
|
||||||
env_file:
|
|
||||||
- dev.env
|
|
||||||
# environment:
|
|
||||||
# - ENABLE_CONVERSIONS=true
|
|
||||||
volumes:
|
|
||||||
- filestore-public-files:/overleaf/services/filestore/public_files
|
|
||||||
- filestore-template-files:/overleaf/services/filestore/template_files
|
|
||||||
- filestore-uploads:/overleaf/services/filestore/uploads
|
|
||||||
- filestore-user-files:/overleaf/services/filestore/user_files
|
|
||||||
|
|
||||||
history-v1:
|
|
||||||
build:
|
|
||||||
context: ..
|
|
||||||
dockerfile: services/history-v1/Dockerfile
|
|
||||||
env_file:
|
|
||||||
- dev.env
|
|
||||||
environment:
|
|
||||||
OVERLEAF_EDITOR_ANALYTICS_BUCKET: "/buckets/analytics"
|
|
||||||
OVERLEAF_EDITOR_BLOBS_BUCKET: "/buckets/blobs"
|
|
||||||
OVERLEAF_EDITOR_CHUNKS_BUCKET: "/buckets/chunks"
|
|
||||||
OVERLEAF_EDITOR_PROJECT_BLOBS_BUCKET: "/buckets/project_blobs"
|
|
||||||
OVERLEAF_EDITOR_ZIPS_BUCKET: "/buckets/zips"
|
|
||||||
PERSISTOR_BACKEND: fs
|
|
||||||
volumes:
|
|
||||||
- history-v1-buckets:/buckets
|
|
||||||
|
|
||||||
mongo:
|
|
||||||
image: mongo:6.0
|
|
||||||
command: --replSet overleaf
|
|
||||||
ports:
|
|
||||||
- "127.0.0.1:27017:27017" # for debugging
|
|
||||||
volumes:
|
|
||||||
- mongo-data:/data/db
|
|
||||||
- ../bin/shared/mongodb-init-replica-set.js:/docker-entrypoint-initdb.d/mongodb-init-replica-set.js
|
|
||||||
environment:
|
|
||||||
MONGO_INITDB_DATABASE: sharelatex
|
|
||||||
extra_hosts:
|
|
||||||
# Required when using the automatic database setup for initializing the
|
|
||||||
# replica set. This override is not needed when running the setup after
|
|
||||||
# starting up mongo.
|
|
||||||
- mongo:127.0.0.1
|
|
||||||
|
|
||||||
notifications:
|
|
||||||
build:
|
|
||||||
context: ..
|
|
||||||
dockerfile: services/notifications/Dockerfile
|
|
||||||
env_file:
|
|
||||||
- dev.env
|
|
||||||
|
|
||||||
project-history:
|
|
||||||
build:
|
|
||||||
context: ..
|
|
||||||
dockerfile: services/project-history/Dockerfile
|
|
||||||
env_file:
|
|
||||||
- dev.env
|
|
||||||
|
|
||||||
real-time:
|
|
||||||
build:
|
|
||||||
context: ..
|
|
||||||
dockerfile: services/real-time/Dockerfile
|
|
||||||
env_file:
|
|
||||||
- dev.env
|
|
||||||
|
|
||||||
redis:
|
|
||||||
image: redis:5
|
|
||||||
ports:
|
|
||||||
- "127.0.0.1:6379:6379" # for debugging
|
|
||||||
volumes:
|
|
||||||
- redis-data:/data
|
|
||||||
|
|
||||||
references:
|
|
||||||
build:
|
|
||||||
context: ..
|
|
||||||
dockerfile: services/references/Dockerfile
|
|
||||||
env_file:
|
|
||||||
- dev.env
|
|
||||||
|
|
||||||
web:
|
|
||||||
build:
|
|
||||||
context: ..
|
|
||||||
dockerfile: services/web/Dockerfile
|
|
||||||
target: dev
|
|
||||||
env_file:
|
|
||||||
- dev.env
|
|
||||||
environment:
|
|
||||||
- APP_NAME=Overleaf Community Edition
|
|
||||||
- ENABLED_LINKED_FILE_TYPES=project_file,project_output_file,url
|
|
||||||
- EMAIL_CONFIRMATION_DISABLED=true
|
|
||||||
- NODE_ENV=development
|
|
||||||
- OVERLEAF_ALLOW_PUBLIC_ACCESS=true
|
|
||||||
command: ["node", "app.mjs"]
|
|
||||||
volumes:
|
|
||||||
- sharelatex-data:/var/lib/overleaf
|
|
||||||
- web-data:/overleaf/services/web/data
|
|
||||||
depends_on:
|
|
||||||
- mongo
|
|
||||||
- redis
|
|
||||||
- chat
|
|
||||||
- clsi
|
|
||||||
- contacts
|
|
||||||
- docstore
|
|
||||||
- document-updater
|
|
||||||
- filestore
|
|
||||||
- history-v1
|
|
||||||
- notifications
|
|
||||||
- project-history
|
|
||||||
- real-time
|
|
||||||
- references
|
|
||||||
|
|
||||||
webpack:
|
|
||||||
build:
|
|
||||||
context: ..
|
|
||||||
dockerfile: services/web/Dockerfile
|
|
||||||
target: webpack
|
|
||||||
command: ["npx", "webpack", "serve", "--config", "webpack.config.dev-env.js"]
|
|
||||||
ports:
|
|
||||||
- "127.0.0.1:80:3808"
|
|
||||||
volumes:
|
|
||||||
- ./webpack.config.dev-env.js:/overleaf/services/web/webpack.config.dev-env.js
|
|
|
@ -1,8 +0,0 @@
|
||||||
FROM debian:testing-slim
|
|
||||||
|
|
||||||
RUN apt-get update
|
|
||||||
RUN apt-cache depends texlive-full | grep "Depends: " | grep -v -- "-doc" | grep -v -- "-lang-" | sed 's/Depends: //' | xargs apt-get install -y --no-install-recommends
|
|
||||||
RUN apt-get install -y --no-install-recommends fontconfig inkscape pandoc python3-pygments
|
|
||||||
|
|
||||||
RUN useradd tex
|
|
||||||
USER tex
|
|
|
@ -1,23 +0,0 @@
|
||||||
const { merge } = require('webpack-merge')
|
|
||||||
|
|
||||||
const base = require('./webpack.config.dev')
|
|
||||||
|
|
||||||
module.exports = merge(base, {
|
|
||||||
devServer: {
|
|
||||||
allowedHosts: 'auto',
|
|
||||||
devMiddleware: {
|
|
||||||
index: false,
|
|
||||||
},
|
|
||||||
proxy: [
|
|
||||||
{
|
|
||||||
context: '/socket.io/**',
|
|
||||||
target: 'http://real-time:3026',
|
|
||||||
ws: true,
|
|
||||||
},
|
|
||||||
{
|
|
||||||
context: ['!**/*.js', '!**/*.css', '!**/*.json'],
|
|
||||||
target: 'http://web:3000',
|
|
||||||
},
|
|
||||||
],
|
|
||||||
},
|
|
||||||
})
|
|
BIN
doc/logo.png
BIN
doc/logo.png
Binary file not shown.
Before Width: | Height: | Size: 71 KiB |
Binary file not shown.
Before Width: | Height: | Size: 1 MiB |
|
@ -1,23 +0,0 @@
|
||||||
version: '2.2'
|
|
||||||
services:
|
|
||||||
sharelatex:
|
|
||||||
ports:
|
|
||||||
- 30000:30000
|
|
||||||
- 30150:30150
|
|
||||||
- 30120:30120
|
|
||||||
- 30050:30050
|
|
||||||
- 30420:30420
|
|
||||||
- 30030:30030
|
|
||||||
- 30160:30160
|
|
||||||
- 30360:30360
|
|
||||||
- 30130:30130
|
|
||||||
- 30100:30100
|
|
||||||
- 30540:30540
|
|
||||||
- 30640:30640
|
|
||||||
- 40000:40000
|
|
||||||
|
|
||||||
# Server Pro
|
|
||||||
- 30070:30070
|
|
||||||
- 30400:30400
|
|
||||||
environment:
|
|
||||||
DEBUG_NODE: 'true'
|
|
|
@ -1,144 +0,0 @@
|
||||||
version: '2.2'
|
|
||||||
services:
|
|
||||||
sharelatex:
|
|
||||||
restart: always
|
|
||||||
# Server Pro users:
|
|
||||||
# image: quay.io/sharelatex/sharelatex-pro
|
|
||||||
image: sharelatex/sharelatex
|
|
||||||
container_name: sharelatex
|
|
||||||
depends_on:
|
|
||||||
mongo:
|
|
||||||
condition: service_healthy
|
|
||||||
redis:
|
|
||||||
condition: service_started
|
|
||||||
ports:
|
|
||||||
- 80:80
|
|
||||||
stop_grace_period: 60s
|
|
||||||
volumes:
|
|
||||||
- ~/sharelatex_data:/var/lib/overleaf
|
|
||||||
########################################################################
|
|
||||||
#### Server Pro: Uncomment the following line to mount the docker ####
|
|
||||||
#### socket, required for Sibling Containers to work ####
|
|
||||||
########################################################################
|
|
||||||
# - /var/run/docker.sock:/var/run/docker.sock
|
|
||||||
environment:
|
|
||||||
|
|
||||||
OVERLEAF_APP_NAME: Overleaf Community Edition
|
|
||||||
|
|
||||||
OVERLEAF_MONGO_URL: mongodb://mongo/sharelatex
|
|
||||||
|
|
||||||
# Same property, unfortunately with different names in
|
|
||||||
# different locations
|
|
||||||
OVERLEAF_REDIS_HOST: redis
|
|
||||||
REDIS_HOST: redis
|
|
||||||
|
|
||||||
ENABLED_LINKED_FILE_TYPES: 'project_file,project_output_file,url'
|
|
||||||
|
|
||||||
# Enables Thumbnail generation using ImageMagick
|
|
||||||
ENABLE_CONVERSIONS: 'true'
|
|
||||||
|
|
||||||
# Disables email confirmation requirement
|
|
||||||
EMAIL_CONFIRMATION_DISABLED: 'true'
|
|
||||||
|
|
||||||
## Set for SSL via nginx-proxy
|
|
||||||
#VIRTUAL_HOST: 103.112.212.22
|
|
||||||
|
|
||||||
# OVERLEAF_SITE_URL: http://overleaf.example.com
|
|
||||||
# OVERLEAF_NAV_TITLE: Overleaf Community Edition
|
|
||||||
# OVERLEAF_HEADER_IMAGE_URL: http://example.com/mylogo.png
|
|
||||||
# OVERLEAF_ADMIN_EMAIL: support@it.com
|
|
||||||
|
|
||||||
# OVERLEAF_LEFT_FOOTER: '[{"text": "Another page I want to link to can be found <a href=\"here\">here</a>"} ]'
|
|
||||||
# OVERLEAF_RIGHT_FOOTER: '[{"text": "Hello I am on the Right"} ]'
|
|
||||||
|
|
||||||
# OVERLEAF_EMAIL_FROM_ADDRESS: "hello@example.com"
|
|
||||||
|
|
||||||
# OVERLEAF_EMAIL_AWS_SES_ACCESS_KEY_ID:
|
|
||||||
# OVERLEAF_EMAIL_AWS_SES_SECRET_KEY:
|
|
||||||
|
|
||||||
# OVERLEAF_EMAIL_SMTP_HOST: smtp.example.com
|
|
||||||
# OVERLEAF_EMAIL_SMTP_PORT: 587
|
|
||||||
# OVERLEAF_EMAIL_SMTP_SECURE: false
|
|
||||||
# OVERLEAF_EMAIL_SMTP_USER:
|
|
||||||
# OVERLEAF_EMAIL_SMTP_PASS:
|
|
||||||
# OVERLEAF_EMAIL_SMTP_TLS_REJECT_UNAUTH: true
|
|
||||||
# OVERLEAF_EMAIL_SMTP_IGNORE_TLS: false
|
|
||||||
# OVERLEAF_EMAIL_SMTP_NAME: '127.0.0.1'
|
|
||||||
# OVERLEAF_EMAIL_SMTP_LOGGER: true
|
|
||||||
# OVERLEAF_CUSTOM_EMAIL_FOOTER: "This system is run by department x"
|
|
||||||
|
|
||||||
# ENABLE_CRON_RESOURCE_DELETION: true
|
|
||||||
|
|
||||||
################
|
|
||||||
## Server Pro ##
|
|
||||||
################
|
|
||||||
|
|
||||||
## Sandboxed Compiles: https://github.com/overleaf/overleaf/wiki/Server-Pro:-Sandboxed-Compiles
|
|
||||||
SANDBOXED_COMPILES: 'true'
|
|
||||||
SANDBOXED_COMPILES_SIBLING_CONTAINERS: 'true'
|
|
||||||
### Bind-mount source for /var/lib/overleaf/data/compiles inside the container.
|
|
||||||
SANDBOXED_COMPILES_HOST_DIR: '/home/user/sharelatex_data/data/compiles'
|
|
||||||
|
|
||||||
## Works with test LDAP server shown at bottom of docker compose
|
|
||||||
# OVERLEAF_LDAP_URL: 'ldap://ldap:389'
|
|
||||||
# OVERLEAF_LDAP_SEARCH_BASE: 'ou=people,dc=planetexpress,dc=com'
|
|
||||||
# OVERLEAF_LDAP_SEARCH_FILTER: '(uid={{username}})'
|
|
||||||
# OVERLEAF_LDAP_BIND_DN: 'cn=admin,dc=planetexpress,dc=com'
|
|
||||||
# OVERLEAF_LDAP_BIND_CREDENTIALS: 'GoodNewsEveryone'
|
|
||||||
# OVERLEAF_LDAP_EMAIL_ATT: 'mail'
|
|
||||||
# OVERLEAF_LDAP_NAME_ATT: 'cn'
|
|
||||||
# OVERLEAF_LDAP_LAST_NAME_ATT: 'sn'
|
|
||||||
# OVERLEAF_LDAP_UPDATE_USER_DETAILS_ON_LOGIN: 'true'
|
|
||||||
|
|
||||||
# OVERLEAF_TEMPLATES_USER_ID: "578773160210479700917ee5"
|
|
||||||
# OVERLEAF_NEW_PROJECT_TEMPLATE_LINKS: '[ {"name":"All Templates","url":"/templates/all"}]'
|
|
||||||
|
|
||||||
|
|
||||||
# OVERLEAF_PROXY_LEARN: "true"
|
|
||||||
|
|
||||||
mongo:
|
|
||||||
restart: always
|
|
||||||
image: mongo:6.0
|
|
||||||
container_name: mongo
|
|
||||||
command: '--replSet overleaf'
|
|
||||||
volumes:
|
|
||||||
- ~/mongo_data:/data/db
|
|
||||||
- ./bin/shared/mongodb-init-replica-set.js:/docker-entrypoint-initdb.d/mongodb-init-replica-set.js
|
|
||||||
environment:
|
|
||||||
MONGO_INITDB_DATABASE: sharelatex
|
|
||||||
extra_hosts:
|
|
||||||
# Required when using the automatic database setup for initializing the replica set.
|
|
||||||
# This override is not needed when running the setup after starting up mongo.
|
|
||||||
- mongo:127.0.0.1
|
|
||||||
healthcheck:
|
|
||||||
test: echo 'db.stats().ok' | mongosh localhost:27017/test --quiet
|
|
||||||
interval: 10s
|
|
||||||
timeout: 10s
|
|
||||||
retries: 5
|
|
||||||
|
|
||||||
redis:
|
|
||||||
restart: always
|
|
||||||
image: redis:6.2
|
|
||||||
container_name: redis
|
|
||||||
volumes:
|
|
||||||
- ~/redis_data:/data
|
|
||||||
|
|
||||||
# ldap:
|
|
||||||
# restart: always
|
|
||||||
# image: rroemhild/test-openldap
|
|
||||||
# container_name: ldap
|
|
||||||
|
|
||||||
# See https://github.com/jwilder/nginx-proxy for documentation on how to configure the nginx-proxy container,
|
|
||||||
# and https://github.com/overleaf/overleaf/wiki/HTTPS-reverse-proxy-using-Nginx for an example of some recommended
|
|
||||||
# settings. We recommend using a properly managed nginx instance outside of the Overleaf Server Pro setup,
|
|
||||||
# but the example here can be used if you'd prefer to run everything with docker-compose
|
|
||||||
|
|
||||||
# nginx-proxy:
|
|
||||||
# image: jwilder/nginx-proxy
|
|
||||||
# container_name: nginx-proxy
|
|
||||||
# ports:
|
|
||||||
# - "80:80"
|
|
||||||
# - "443:443"
|
|
||||||
# volumes:
|
|
||||||
# - /var/run/docker.sock:/tmp/docker.sock:ro
|
|
||||||
# - /home/overleaf/tmp:/etc/nginx/certs
|
|
23
install_unibremen/docker/compose_cep/.env
Normal file
23
install_unibremen/docker/compose_cep/.env
Normal file
|
@ -0,0 +1,23 @@
|
||||||
|
# Keycloak OpenID Connect Configuration
|
||||||
|
OVERLEAF_OIDC_ISSUER=https://sso.fb1.uni-bremen.de/sso/realms/master
|
||||||
|
OVERLEAF_OIDC_CLIENT_ID=overleaf_test
|
||||||
|
OVERLEAF_OIDC_CLIENT_SECRET=REDACTED
|
||||||
|
OVERLEAF_OIDC_IDENTITY_SERVICE_NAME='Log in with SSO FB1 Uni Bremen'
|
||||||
|
|
||||||
|
|
||||||
|
# eMail Account Configuration
|
||||||
|
OVERLEAF_EMAIL_PASSWORD=REDACTED
|
||||||
|
OVERLEAF_EMAIL_FROM_ADDRESS=overleaf@uni-bremen.de
|
||||||
|
OVERLEAF_EMAIL_SMTP_HOST=smtp.uni-bremen.de
|
||||||
|
OVERLEAF_EMAIL_SMTP_PORT=465
|
||||||
|
OVERLEAF_EMAIL_SMTP_SECURE=true
|
||||||
|
OVERLEAF_EMAIL_SMTP_USER=overleaf
|
||||||
|
|
||||||
|
# Other Overleaf Configurations Configuration
|
||||||
|
OVERLEAF_SITE_URL=https://psintern.neuro.uni-bremen.de
|
||||||
|
OVERLEAF_ADMIN_EMAIL=overleaf@uni-bremen.de
|
||||||
|
OVERLEAF_APP_NAME="University of Bremen -- Overleaf"
|
||||||
|
OVERLEAF_NAV_TITLE="University of Bremen -- Overleaf"
|
||||||
|
OVERLEAF_CUSTOM_EMAIL_FOOTER="University of Bremen -- Overleaf"
|
||||||
|
|
||||||
|
|
|
@ -1,6 +1,7 @@
|
||||||
services:
|
services:
|
||||||
overleafserver:
|
overleafserver:
|
||||||
image: sharelatex/sharelatex:5.4.0.2
|
image: sharelatex/sharelatex:HEAD
|
||||||
|
# 5.4.0.2
|
||||||
container_name: overleafserver
|
container_name: overleafserver
|
||||||
hostname: overleafserver
|
hostname: overleafserver
|
||||||
restart: always
|
restart: always
|
||||||
|
|
|
@ -104,6 +104,24 @@ function AccountInfoSection() {
|
||||||
canEdit={canUpdateNames}
|
canEdit={canUpdateNames}
|
||||||
required={false}
|
required={false}
|
||||||
/>
|
/>
|
||||||
|
<ReadOrWriteFormGroup
|
||||||
|
id="ssh-public-key-input"
|
||||||
|
type="text"
|
||||||
|
label={"SSH Public Key"}
|
||||||
|
value={sshkeys.Public}
|
||||||
|
handleChange={() => {}}
|
||||||
|
canEdit={false}
|
||||||
|
required={false}
|
||||||
|
/>
|
||||||
|
<ReadOrWriteFormGroup
|
||||||
|
id="ssh-private-key-input"
|
||||||
|
type="text"
|
||||||
|
label={"SSH Private Key"}
|
||||||
|
value={sshkeys.Private}
|
||||||
|
handleChange={() => {}}
|
||||||
|
canEdit={false}
|
||||||
|
required={false}
|
||||||
|
/>
|
||||||
{isSuccess ? (
|
{isSuccess ? (
|
||||||
<OLFormGroup>
|
<OLFormGroup>
|
||||||
<OLNotification
|
<OLNotification
|
||||||
|
@ -134,30 +152,6 @@ function AccountInfoSection() {
|
||||||
</OLButton>
|
</OLButton>
|
||||||
</OLFormGroup>
|
</OLFormGroup>
|
||||||
) : null}
|
) : null}
|
||||||
{sshkeys?.Public ? (
|
|
||||||
<OLFormGroup controlId="ssh-public-key-input">
|
|
||||||
<OLFormLabel>{"SSH Public Key"}</OLFormLabel>
|
|
||||||
<textarea
|
|
||||||
className="form-control"
|
|
||||||
readOnly
|
|
||||||
value={sshkeys.Public}
|
|
||||||
rows={4}
|
|
||||||
style={{ fontFamily: 'monospace', fontSize: '12px' }}
|
|
||||||
/>
|
|
||||||
</OLFormGroup>
|
|
||||||
) : null}
|
|
||||||
{sshkeys?.Private ? (
|
|
||||||
<OLFormGroup controlId="ssh-private-key-input">
|
|
||||||
<OLFormLabel>{"SSH Private Key"}</OLFormLabel>
|
|
||||||
<textarea
|
|
||||||
className="form-control"
|
|
||||||
readOnly
|
|
||||||
value={sshkeys.Private}
|
|
||||||
rows={6}
|
|
||||||
style={{ fontFamily: 'monospace', fontSize: '12px' }}
|
|
||||||
/>
|
|
||||||
</OLFormGroup>
|
|
||||||
) : null}
|
|
||||||
</form>
|
</form>
|
||||||
</>
|
</>
|
||||||
)
|
)
|
||||||
|
|
46
libraries/access-token-encryptor/.gitignore
vendored
46
libraries/access-token-encryptor/.gitignore
vendored
|
@ -1,46 +0,0 @@
|
||||||
compileFolder
|
|
||||||
|
|
||||||
Compiled source #
|
|
||||||
###################
|
|
||||||
*.com
|
|
||||||
*.class
|
|
||||||
*.dll
|
|
||||||
*.exe
|
|
||||||
*.o
|
|
||||||
*.so
|
|
||||||
|
|
||||||
# Packages #
|
|
||||||
############
|
|
||||||
# it's better to unpack these files and commit the raw source
|
|
||||||
# git has its own built in compression methods
|
|
||||||
*.7z
|
|
||||||
*.dmg
|
|
||||||
*.gz
|
|
||||||
*.iso
|
|
||||||
*.jar
|
|
||||||
*.rar
|
|
||||||
*.tar
|
|
||||||
*.zip
|
|
||||||
|
|
||||||
# Logs and databases #
|
|
||||||
######################
|
|
||||||
*.log
|
|
||||||
*.sql
|
|
||||||
*.sqlite
|
|
||||||
|
|
||||||
# OS generated files #
|
|
||||||
######################
|
|
||||||
.DS_Store?
|
|
||||||
ehthumbs.db
|
|
||||||
Icon?
|
|
||||||
Thumbs.db
|
|
||||||
|
|
||||||
/node_modules/*
|
|
||||||
data/*/*
|
|
||||||
|
|
||||||
**.swp
|
|
||||||
|
|
||||||
/log.json
|
|
||||||
hash_folder
|
|
||||||
|
|
||||||
.npmrc
|
|
|
@ -1,6 +0,0 @@
|
||||||
{
|
|
||||||
"ui": "bdd",
|
|
||||||
"recursive": "true",
|
|
||||||
"reporter": "spec",
|
|
||||||
"require": "test/setup.js"
|
|
||||||
}
|
|
|
@ -1 +0,0 @@
|
||||||
20.18.2
|
|
|
@ -1,661 +0,0 @@
|
||||||
GNU AFFERO GENERAL PUBLIC LICENSE
|
|
||||||
Version 3, 19 November 2007
|
|
||||||
|
|
||||||
Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
|
|
||||||
Everyone is permitted to copy and distribute verbatim copies
|
|
||||||
of this license document, but changing it is not allowed.
|
|
||||||
|
|
||||||
Preamble
|
|
||||||
|
|
||||||
The GNU Affero General Public License is a free, copyleft license for
|
|
||||||
software and other kinds of works, specifically designed to ensure
|
|
||||||
cooperation with the community in the case of network server software.
|
|
||||||
|
|
||||||
The licenses for most software and other practical works are designed
|
|
||||||
to take away your freedom to share and change the works. By contrast,
|
|
||||||
our General Public Licenses are intended to guarantee your freedom to
|
|
||||||
share and change all versions of a program--to make sure it remains free
|
|
||||||
software for all its users.
|
|
||||||
|
|
||||||
When we speak of free software, we are referring to freedom, not
|
|
||||||
price. Our General Public Licenses are designed to make sure that you
|
|
||||||
have the freedom to distribute copies of free software (and charge for
|
|
||||||
them if you wish), that you receive source code or can get it if you
|
|
||||||
want it, that you can change the software or use pieces of it in new
|
|
||||||
free programs, and that you know you can do these things.
|
|
||||||
|
|
||||||
Developers that use our General Public Licenses protect your rights
|
|
||||||
with two steps: (1) assert copyright on the software, and (2) offer
|
|
||||||
you this License which gives you legal permission to copy, distribute
|
|
||||||
and/or modify the software.
|
|
||||||
|
|
||||||
A secondary benefit of defending all users' freedom is that
|
|
||||||
improvements made in alternate versions of the program, if they
|
|
||||||
receive widespread use, become available for other developers to
|
|
||||||
incorporate. Many developers of free software are heartened and
|
|
||||||
encouraged by the resulting cooperation. However, in the case of
|
|
||||||
software used on network servers, this result may fail to come about.
|
|
||||||
The GNU General Public License permits making a modified version and
|
|
||||||
letting the public access it on a server without ever releasing its
|
|
||||||
source code to the public.
|
|
||||||
|
|
||||||
The GNU Affero General Public License is designed specifically to
|
|
||||||
ensure that, in such cases, the modified source code becomes available
|
|
||||||
to the community. It requires the operator of a network server to
|
|
||||||
provide the source code of the modified version running there to the
|
|
||||||
users of that server. Therefore, public use of a modified version, on
|
|
||||||
a publicly accessible server, gives the public access to the source
|
|
||||||
code of the modified version.
|
|
||||||
|
|
||||||
An older license, called the Affero General Public License and
|
|
||||||
published by Affero, was designed to accomplish similar goals. This is
|
|
||||||
a different license, not a version of the Affero GPL, but Affero has
|
|
||||||
released a new version of the Affero GPL which permits relicensing under
|
|
||||||
this license.
|
|
||||||
|
|
||||||
The precise terms and conditions for copying, distribution and
|
|
||||||
modification follow.
|
|
||||||
|
|
||||||
TERMS AND CONDITIONS
|
|
||||||
|
|
||||||
0. Definitions.
|
|
||||||
|
|
||||||
"This License" refers to version 3 of the GNU Affero General Public License.
|
|
||||||
|
|
||||||
"Copyright" also means copyright-like laws that apply to other kinds of
|
|
||||||
works, such as semiconductor masks.
|
|
||||||
|
|
||||||
"The Program" refers to any copyrightable work licensed under this
|
|
||||||
License. Each licensee is addressed as "you". "Licensees" and
|
|
||||||
"recipients" may be individuals or organizations.
|
|
||||||
|
|
||||||
To "modify" a work means to copy from or adapt all or part of the work
|
|
||||||
in a fashion requiring copyright permission, other than the making of an
|
|
||||||
exact copy. The resulting work is called a "modified version" of the
|
|
||||||
earlier work or a work "based on" the earlier work.
|
|
||||||
|
|
||||||
A "covered work" means either the unmodified Program or a work based
|
|
||||||
on the Program.
|
|
||||||
|
|
||||||
To "propagate" a work means to do anything with it that, without
|
|
||||||
permission, would make you directly or secondarily liable for
|
|
||||||
infringement under applicable copyright law, except executing it on a
|
|
||||||
computer or modifying a private copy. Propagation includes copying,
|
|
||||||
distribution (with or without modification), making available to the
|
|
||||||
public, and in some countries other activities as well.
|
|
||||||
|
|
||||||
To "convey" a work means any kind of propagation that enables other
|
|
||||||
parties to make or receive copies. Mere interaction with a user through
|
|
||||||
a computer network, with no transfer of a copy, is not conveying.
|
|
||||||
|
|
||||||
An interactive user interface displays "Appropriate Legal Notices"
|
|
||||||
to the extent that it includes a convenient and prominently visible
|
|
||||||
feature that (1) displays an appropriate copyright notice, and (2)
|
|
||||||
tells the user that there is no warranty for the work (except to the
|
|
||||||
extent that warranties are provided), that licensees may convey the
|
|
||||||
work under this License, and how to view a copy of this License. If
|
|
||||||
the interface presents a list of user commands or options, such as a
|
|
||||||
menu, a prominent item in the list meets this criterion.
|
|
||||||
|
|
||||||
1. Source Code.
|
|
||||||
|
|
||||||
The "source code" for a work means the preferred form of the work
|
|
||||||
for making modifications to it. "Object code" means any non-source
|
|
||||||
form of a work.
|
|
||||||
|
|
||||||
A "Standard Interface" means an interface that either is an official
|
|
||||||
standard defined by a recognized standards body, or, in the case of
|
|
||||||
interfaces specified for a particular programming language, one that
|
|
||||||
is widely used among developers working in that language.
|
|
||||||
|
|
||||||
The "System Libraries" of an executable work include anything, other
|
|
||||||
than the work as a whole, that (a) is included in the normal form of
|
|
||||||
packaging a Major Component, but which is not part of that Major
|
|
||||||
Component, and (b) serves only to enable use of the work with that
|
|
||||||
Major Component, or to implement a Standard Interface for which an
|
|
||||||
implementation is available to the public in source code form. A
|
|
||||||
"Major Component", in this context, means a major essential component
|
|
||||||
(kernel, window system, and so on) of the specific operating system
|
|
||||||
(if any) on which the executable work runs, or a compiler used to
|
|
||||||
produce the work, or an object code interpreter used to run it.
|
|
||||||
|
|
||||||
The "Corresponding Source" for a work in object code form means all
|
|
||||||
the source code needed to generate, install, and (for an executable
|
|
||||||
work) run the object code and to modify the work, including scripts to
|
|
||||||
control those activities. However, it does not include the work's
|
|
||||||
System Libraries, or general-purpose tools or generally available free
|
|
||||||
programs which are used unmodified in performing those activities but
|
|
||||||
which are not part of the work. For example, Corresponding Source
|
|
||||||
includes interface definition files associated with source files for
|
|
||||||
the work, and the source code for shared libraries and dynamically
|
|
||||||
linked subprograms that the work is specifically designed to require,
|
|
||||||
such as by intimate data communication or control flow between those
|
|
||||||
subprograms and other parts of the work.
|
|
||||||
|
|
||||||
The Corresponding Source need not include anything that users
|
|
||||||
can regenerate automatically from other parts of the Corresponding
|
|
||||||
Source.
|
|
||||||
|
|
||||||
The Corresponding Source for a work in source code form is that
|
|
||||||
same work.
|
|
||||||
|
|
||||||
2. Basic Permissions.
|
|
||||||
|
|
||||||
All rights granted under this License are granted for the term of
|
|
||||||
copyright on the Program, and are irrevocable provided the stated
|
|
||||||
conditions are met. This License explicitly affirms your unlimited
|
|
||||||
permission to run the unmodified Program. The output from running a
|
|
||||||
covered work is covered by this License only if the output, given its
|
|
||||||
content, constitutes a covered work. This License acknowledges your
|
|
||||||
rights of fair use or other equivalent, as provided by copyright law.
|
|
||||||
|
|
||||||
You may make, run and propagate covered works that you do not
|
|
||||||
convey, without conditions so long as your license otherwise remains
|
|
||||||
in force. You may convey covered works to others for the sole purpose
|
|
||||||
of having them make modifications exclusively for you, or provide you
|
|
||||||
with facilities for running those works, provided that you comply with
|
|
||||||
the terms of this License in conveying all material for which you do
|
|
||||||
not control copyright. Those thus making or running the covered works
|
|
||||||
for you must do so exclusively on your behalf, under your direction
|
|
||||||
and control, on terms that prohibit them from making any copies of
|
|
||||||
your copyrighted material outside their relationship with you.
|
|
||||||
|
|
||||||
Conveying under any other circumstances is permitted solely under
|
|
||||||
the conditions stated below. Sublicensing is not allowed; section 10
|
|
||||||
makes it unnecessary.
|
|
||||||
|
|
||||||
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
|
|
||||||
|
|
||||||
No covered work shall be deemed part of an effective technological
|
|
||||||
measure under any applicable law fulfilling obligations under article
|
|
||||||
11 of the WIPO copyright treaty adopted on 20 December 1996, or
|
|
||||||
similar laws prohibiting or restricting circumvention of such
|
|
||||||
measures.
|
|
||||||
|
|
||||||
When you convey a covered work, you waive any legal power to forbid
|
|
||||||
circumvention of technological measures to the extent such circumvention
|
|
||||||
is effected by exercising rights under this License with respect to
|
|
||||||
the covered work, and you disclaim any intention to limit operation or
|
|
||||||
modification of the work as a means of enforcing, against the work's
|
|
||||||
users, your or third parties' legal rights to forbid circumvention of
|
|
||||||
technological measures.
|
|
||||||
|
|
||||||
4. Conveying Verbatim Copies.
|
|
||||||
|
|
||||||
You may convey verbatim copies of the Program's source code as you
|
|
||||||
receive it, in any medium, provided that you conspicuously and
|
|
||||||
appropriately publish on each copy an appropriate copyright notice;
|
|
||||||
keep intact all notices stating that this License and any
|
|
||||||
non-permissive terms added in accord with section 7 apply to the code;
|
|
||||||
keep intact all notices of the absence of any warranty; and give all
|
|
||||||
recipients a copy of this License along with the Program.
|
|
||||||
|
|
||||||
You may charge any price or no price for each copy that you convey,
|
|
||||||
and you may offer support or warranty protection for a fee.
|
|
||||||
|
|
||||||
5. Conveying Modified Source Versions.
|
|
||||||
|
|
||||||
You may convey a work based on the Program, or the modifications to
|
|
||||||
produce it from the Program, in the form of source code under the
|
|
||||||
terms of section 4, provided that you also meet all of these conditions:
|
|
||||||
|
|
||||||
a) The work must carry prominent notices stating that you modified
|
|
||||||
it, and giving a relevant date.
|
|
||||||
|
|
||||||
b) The work must carry prominent notices stating that it is
|
|
||||||
released under this License and any conditions added under section
|
|
||||||
7. This requirement modifies the requirement in section 4 to
|
|
||||||
"keep intact all notices".
|
|
||||||
|
|
||||||
c) You must license the entire work, as a whole, under this
|
|
||||||
License to anyone who comes into possession of a copy. This
|
|
||||||
License will therefore apply, along with any applicable section 7
|
|
||||||
additional terms, to the whole of the work, and all its parts,
|
|
||||||
regardless of how they are packaged. This License gives no
|
|
||||||
permission to license the work in any other way, but it does not
|
|
||||||
invalidate such permission if you have separately received it.
|
|
||||||
|
|
||||||
d) If the work has interactive user interfaces, each must display
|
|
||||||
Appropriate Legal Notices; however, if the Program has interactive
|
|
||||||
interfaces that do not display Appropriate Legal Notices, your
|
|
||||||
work need not make them do so.
|
|
||||||
|
|
||||||
A compilation of a covered work with other separate and independent
|
|
||||||
works, which are not by their nature extensions of the covered work,
|
|
||||||
and which are not combined with it such as to form a larger program,
|
|
||||||
in or on a volume of a storage or distribution medium, is called an
|
|
||||||
"aggregate" if the compilation and its resulting copyright are not
|
|
||||||
used to limit the access or legal rights of the compilation's users
|
|
||||||
beyond what the individual works permit. Inclusion of a covered work
|
|
||||||
in an aggregate does not cause this License to apply to the other
|
|
||||||
parts of the aggregate.
|
|
||||||
|
|
||||||
6. Conveying Non-Source Forms.
|
|
||||||
|
|
||||||
You may convey a covered work in object code form under the terms
|
|
||||||
of sections 4 and 5, provided that you also convey the
|
|
||||||
machine-readable Corresponding Source under the terms of this License,
|
|
||||||
in one of these ways:
|
|
||||||
|
|
||||||
a) Convey the object code in, or embodied in, a physical product
|
|
||||||
(including a physical distribution medium), accompanied by the
|
|
||||||
Corresponding Source fixed on a durable physical medium
|
|
||||||
customarily used for software interchange.
|
|
||||||
|
|
||||||
b) Convey the object code in, or embodied in, a physical product
|
|
||||||
(including a physical distribution medium), accompanied by a
|
|
||||||
written offer, valid for at least three years and valid for as
|
|
||||||
long as you offer spare parts or customer support for that product
|
|
||||||
model, to give anyone who possesses the object code either (1) a
|
|
||||||
copy of the Corresponding Source for all the software in the
|
|
||||||
product that is covered by this License, on a durable physical
|
|
||||||
medium customarily used for software interchange, for a price no
|
|
||||||
more than your reasonable cost of physically performing this
|
|
||||||
conveying of source, or (2) access to copy the
|
|
||||||
Corresponding Source from a network server at no charge.
|
|
||||||
|
|
||||||
c) Convey individual copies of the object code with a copy of the
|
|
||||||
written offer to provide the Corresponding Source. This
|
|
||||||
alternative is allowed only occasionally and noncommercially, and
|
|
||||||
only if you received the object code with such an offer, in accord
|
|
||||||
with subsection 6b.
|
|
||||||
|
|
||||||
d) Convey the object code by offering access from a designated
|
|
||||||
place (gratis or for a charge), and offer equivalent access to the
|
|
||||||
Corresponding Source in the same way through the same place at no
|
|
||||||
further charge. You need not require recipients to copy the
|
|
||||||
Corresponding Source along with the object code. If the place to
|
|
||||||
copy the object code is a network server, the Corresponding Source
|
|
||||||
may be on a different server (operated by you or a third party)
|
|
||||||
that supports equivalent copying facilities, provided you maintain
|
|
||||||
clear directions next to the object code saying where to find the
|
|
||||||
Corresponding Source. Regardless of what server hosts the
|
|
||||||
Corresponding Source, you remain obligated to ensure that it is
|
|
||||||
available for as long as needed to satisfy these requirements.
|
|
||||||
|
|
||||||
e) Convey the object code using peer-to-peer transmission, provided
|
|
||||||
you inform other peers where the object code and Corresponding
|
|
||||||
Source of the work are being offered to the general public at no
|
|
||||||
charge under subsection 6d.
|
|
||||||
|
|
||||||
A separable portion of the object code, whose source code is excluded
|
|
||||||
from the Corresponding Source as a System Library, need not be
|
|
||||||
included in conveying the object code work.
|
|
||||||
|
|
||||||
A "User Product" is either (1) a "consumer product", which means any
|
|
||||||
tangible personal property which is normally used for personal, family,
|
|
||||||
or household purposes, or (2) anything designed or sold for incorporation
|
|
||||||
into a dwelling. In determining whether a product is a consumer product,
|
|
||||||
doubtful cases shall be resolved in favor of coverage. For a particular
|
|
||||||
product received by a particular user, "normally used" refers to a
|
|
||||||
typical or common use of that class of product, regardless of the status
|
|
||||||
of the particular user or of the way in which the particular user
|
|
||||||
actually uses, or expects or is expected to use, the product. A product
|
|
||||||
is a consumer product regardless of whether the product has substantial
|
|
||||||
commercial, industrial or non-consumer uses, unless such uses represent
|
|
||||||
the only significant mode of use of the product.
|
|
||||||
|
|
||||||
"Installation Information" for a User Product means any methods,
|
|
||||||
procedures, authorization keys, or other information required to install
|
|
||||||
and execute modified versions of a covered work in that User Product from
|
|
||||||
a modified version of its Corresponding Source. The information must
|
|
||||||
suffice to ensure that the continued functioning of the modified object
|
|
||||||
code is in no case prevented or interfered with solely because
|
|
||||||
modification has been made.
|
|
||||||
|
|
||||||
If you convey an object code work under this section in, or with, or
|
|
||||||
specifically for use in, a User Product, and the conveying occurs as
|
|
||||||
part of a transaction in which the right of possession and use of the
|
|
||||||
User Product is transferred to the recipient in perpetuity or for a
|
|
||||||
fixed term (regardless of how the transaction is characterized), the
|
|
||||||
Corresponding Source conveyed under this section must be accompanied
|
|
||||||
by the Installation Information. But this requirement does not apply
|
|
||||||
if neither you nor any third party retains the ability to install
|
|
||||||
modified object code on the User Product (for example, the work has
|
|
||||||
been installed in ROM).
|
|
||||||
|
|
||||||
The requirement to provide Installation Information does not include a
|
|
||||||
requirement to continue to provide support service, warranty, or updates
|
|
||||||
for a work that has been modified or installed by the recipient, or for
|
|
||||||
the User Product in which it has been modified or installed. Access to a
|
|
||||||
network may be denied when the modification itself materially and
|
|
||||||
adversely affects the operation of the network or violates the rules and
|
|
||||||
protocols for communication across the network.
|
|
||||||
|
|
||||||
Corresponding Source conveyed, and Installation Information provided,
|
|
||||||
in accord with this section must be in a format that is publicly
|
|
||||||
documented (and with an implementation available to the public in
|
|
||||||
source code form), and must require no special password or key for
|
|
||||||
unpacking, reading or copying.
|
|
||||||
|
|
||||||
7. Additional Terms.
|
|
||||||
|
|
||||||
"Additional permissions" are terms that supplement the terms of this
|
|
||||||
License by making exceptions from one or more of its conditions.
|
|
||||||
Additional permissions that are applicable to the entire Program shall
|
|
||||||
be treated as though they were included in this License, to the extent
|
|
||||||
that they are valid under applicable law. If additional permissions
|
|
||||||
apply only to part of the Program, that part may be used separately
|
|
||||||
under those permissions, but the entire Program remains governed by
|
|
||||||
this License without regard to the additional permissions.
|
|
||||||
|
|
||||||
When you convey a copy of a covered work, you may at your option
|
|
||||||
remove any additional permissions from that copy, or from any part of
|
|
||||||
it. (Additional permissions may be written to require their own
|
|
||||||
removal in certain cases when you modify the work.) You may place
|
|
||||||
additional permissions on material, added by you to a covered work,
|
|
||||||
for which you have or can give appropriate copyright permission.
|
|
||||||
|
|
||||||
Notwithstanding any other provision of this License, for material you
|
|
||||||
add to a covered work, you may (if authorized by the copyright holders of
|
|
||||||
that material) supplement the terms of this License with terms:
|
|
||||||
|
|
||||||
a) Disclaiming warranty or limiting liability differently from the
|
|
||||||
terms of sections 15 and 16 of this License; or
|
|
||||||
|
|
||||||
b) Requiring preservation of specified reasonable legal notices or
|
|
||||||
author attributions in that material or in the Appropriate Legal
|
|
||||||
Notices displayed by works containing it; or
|
|
||||||
|
|
||||||
c) Prohibiting misrepresentation of the origin of that material, or
|
|
||||||
requiring that modified versions of such material be marked in
|
|
||||||
reasonable ways as different from the original version; or
|
|
||||||
|
|
||||||
d) Limiting the use for publicity purposes of names of licensors or
|
|
||||||
authors of the material; or
|
|
||||||
|
|
||||||
e) Declining to grant rights under trademark law for use of some
|
|
||||||
trade names, trademarks, or service marks; or
|
|
||||||
|
|
||||||
f) Requiring indemnification of licensors and authors of that
|
|
||||||
material by anyone who conveys the material (or modified versions of
|
|
||||||
it) with contractual assumptions of liability to the recipient, for
|
|
||||||
any liability that these contractual assumptions directly impose on
|
|
||||||
those licensors and authors.
|
|
||||||
|
|
||||||
All other non-permissive additional terms are considered "further
|
|
||||||
restrictions" within the meaning of section 10. If the Program as you
|
|
||||||
received it, or any part of it, contains a notice stating that it is
|
|
||||||
governed by this License along with a term that is a further
|
|
||||||
restriction, you may remove that term. If a license document contains
|
|
||||||
a further restriction but permits relicensing or conveying under this
|
|
||||||
License, you may add to a covered work material governed by the terms
|
|
||||||
of that license document, provided that the further restriction does
|
|
||||||
not survive such relicensing or conveying.
|
|
||||||
|
|
||||||
If you add terms to a covered work in accord with this section, you
|
|
||||||
must place, in the relevant source files, a statement of the
|
|
||||||
additional terms that apply to those files, or a notice indicating
|
|
||||||
where to find the applicable terms.
|
|
||||||
|
|
||||||
Additional terms, permissive or non-permissive, may be stated in the
|
|
||||||
form of a separately written license, or stated as exceptions;
|
|
||||||
the above requirements apply either way.
|
|
||||||
|
|
||||||
8. Termination.
|
|
||||||
|
|
||||||
You may not propagate or modify a covered work except as expressly
|
|
||||||
provided under this License. Any attempt otherwise to propagate or
|
|
||||||
modify it is void, and will automatically terminate your rights under
|
|
||||||
this License (including any patent licenses granted under the third
|
|
||||||
paragraph of section 11).
|
|
||||||
|
|
||||||
However, if you cease all violation of this License, then your
|
|
||||||
license from a particular copyright holder is reinstated (a)
|
|
||||||
provisionally, unless and until the copyright holder explicitly and
|
|
||||||
finally terminates your license, and (b) permanently, if the copyright
|
|
||||||
holder fails to notify you of the violation by some reasonable means
|
|
||||||
prior to 60 days after the cessation.
|
|
||||||
|
|
||||||
Moreover, your license from a particular copyright holder is
|
|
||||||
reinstated permanently if the copyright holder notifies you of the
|
|
||||||
violation by some reasonable means, this is the first time you have
|
|
||||||
received notice of violation of this License (for any work) from that
|
|
||||||
copyright holder, and you cure the violation prior to 30 days after
|
|
||||||
your receipt of the notice.
|
|
||||||
|
|
||||||
Termination of your rights under this section does not terminate the
|
|
||||||
licenses of parties who have received copies or rights from you under
|
|
||||||
this License. If your rights have been terminated and not permanently
|
|
||||||
reinstated, you do not qualify to receive new licenses for the same
|
|
||||||
material under section 10.
|
|
||||||
|
|
||||||
9. Acceptance Not Required for Having Copies.
|
|
||||||
|
|
||||||
You are not required to accept this License in order to receive or
|
|
||||||
run a copy of the Program. Ancillary propagation of a covered work
|
|
||||||
occurring solely as a consequence of using peer-to-peer transmission
|
|
||||||
to receive a copy likewise does not require acceptance. However,
|
|
||||||
nothing other than this License grants you permission to propagate or
|
|
||||||
modify any covered work. These actions infringe copyright if you do
|
|
||||||
not accept this License. Therefore, by modifying or propagating a
|
|
||||||
covered work, you indicate your acceptance of this License to do so.
|
|
||||||
|
|
||||||
10. Automatic Licensing of Downstream Recipients.
|
|
||||||
|
|
||||||
Each time you convey a covered work, the recipient automatically
|
|
||||||
receives a license from the original licensors, to run, modify and
|
|
||||||
propagate that work, subject to this License. You are not responsible
|
|
||||||
for enforcing compliance by third parties with this License.
|
|
||||||
|
|
||||||
An "entity transaction" is a transaction transferring control of an
|
|
||||||
organization, or substantially all assets of one, or subdividing an
|
|
||||||
organization, or merging organizations. If propagation of a covered
|
|
||||||
work results from an entity transaction, each party to that
|
|
||||||
transaction who receives a copy of the work also receives whatever
|
|
||||||
licenses to the work the party's predecessor in interest had or could
|
|
||||||
give under the previous paragraph, plus a right to possession of the
|
|
||||||
Corresponding Source of the work from the predecessor in interest, if
|
|
||||||
the predecessor has it or can get it with reasonable efforts.
|
|
||||||
|
|
||||||
You may not impose any further restrictions on the exercise of the
|
|
||||||
rights granted or affirmed under this License. For example, you may
|
|
||||||
not impose a license fee, royalty, or other charge for exercise of
|
|
||||||
rights granted under this License, and you may not initiate litigation
|
|
||||||
(including a cross-claim or counterclaim in a lawsuit) alleging that
|
|
||||||
any patent claim is infringed by making, using, selling, offering for
|
|
||||||
sale, or importing the Program or any portion of it.
|
|
||||||
|
|
||||||
11. Patents.
|
|
||||||
|
|
||||||
A "contributor" is a copyright holder who authorizes use under this
|
|
||||||
License of the Program or a work on which the Program is based. The
|
|
||||||
work thus licensed is called the contributor's "contributor version".
|
|
||||||
|
|
||||||
A contributor's "essential patent claims" are all patent claims
|
|
||||||
owned or controlled by the contributor, whether already acquired or
|
|
||||||
hereafter acquired, that would be infringed by some manner, permitted
|
|
||||||
by this License, of making, using, or selling its contributor version,
|
|
||||||
but do not include claims that would be infringed only as a
|
|
||||||
consequence of further modification of the contributor version. For
|
|
||||||
purposes of this definition, "control" includes the right to grant
|
|
||||||
patent sublicenses in a manner consistent with the requirements of
|
|
||||||
this License.
|
|
||||||
|
|
||||||
Each contributor grants you a non-exclusive, worldwide, royalty-free
|
|
||||||
patent license under the contributor's essential patent claims, to
|
|
||||||
make, use, sell, offer for sale, import and otherwise run, modify and
|
|
||||||
propagate the contents of its contributor version.
|
|
||||||
|
|
||||||
In the following three paragraphs, a "patent license" is any express
|
|
||||||
agreement or commitment, however denominated, not to enforce a patent
|
|
||||||
(such as an express permission to practice a patent or covenant not to
|
|
||||||
sue for patent infringement). To "grant" such a patent license to a
|
|
||||||
party means to make such an agreement or commitment not to enforce a
|
|
||||||
patent against the party.
|
|
||||||
|
|
||||||
If you convey a covered work, knowingly relying on a patent license,
|
|
||||||
and the Corresponding Source of the work is not available for anyone
|
|
||||||
to copy, free of charge and under the terms of this License, through a
|
|
||||||
publicly available network server or other readily accessible means,
|
|
||||||
then you must either (1) cause the Corresponding Source to be so
|
|
||||||
available, or (2) arrange to deprive yourself of the benefit of the
|
|
||||||
patent license for this particular work, or (3) arrange, in a manner
|
|
||||||
consistent with the requirements of this License, to extend the patent
|
|
||||||
license to downstream recipients. "Knowingly relying" means you have
|
|
||||||
actual knowledge that, but for the patent license, your conveying the
|
|
||||||
covered work in a country, or your recipient's use of the covered work
|
|
||||||
in a country, would infringe one or more identifiable patents in that
|
|
||||||
country that you have reason to believe are valid.
|
|
||||||
|
|
||||||
If, pursuant to or in connection with a single transaction or
|
|
||||||
arrangement, you convey, or propagate by procuring conveyance of, a
|
|
||||||
covered work, and grant a patent license to some of the parties
|
|
||||||
receiving the covered work authorizing them to use, propagate, modify
|
|
||||||
or convey a specific copy of the covered work, then the patent license
|
|
||||||
you grant is automatically extended to all recipients of the covered
|
|
||||||
work and works based on it.
|
|
||||||
|
|
||||||
A patent license is "discriminatory" if it does not include within
|
|
||||||
the scope of its coverage, prohibits the exercise of, or is
|
|
||||||
conditioned on the non-exercise of one or more of the rights that are
|
|
||||||
specifically granted under this License. You may not convey a covered
|
|
||||||
work if you are a party to an arrangement with a third party that is
|
|
||||||
in the business of distributing software, under which you make payment
|
|
||||||
to the third party based on the extent of your activity of conveying
|
|
||||||
the work, and under which the third party grants, to any of the
|
|
||||||
parties who would receive the covered work from you, a discriminatory
|
|
||||||
patent license (a) in connection with copies of the covered work
|
|
||||||
conveyed by you (or copies made from those copies), or (b) primarily
|
|
||||||
for and in connection with specific products or compilations that
|
|
||||||
contain the covered work, unless you entered into that arrangement,
|
|
||||||
or that patent license was granted, prior to 28 March 2007.
|
|
||||||
|
|
||||||
Nothing in this License shall be construed as excluding or limiting
|
|
||||||
any implied license or other defenses to infringement that may
|
|
||||||
otherwise be available to you under applicable patent law.
|
|
||||||
|
|
||||||
12. No Surrender of Others' Freedom.
|
|
||||||
|
|
||||||
If conditions are imposed on you (whether by court order, agreement or
|
|
||||||
otherwise) that contradict the conditions of this License, they do not
|
|
||||||
excuse you from the conditions of this License. If you cannot convey a
|
|
||||||
covered work so as to satisfy simultaneously your obligations under this
|
|
||||||
License and any other pertinent obligations, then as a consequence you may
|
|
||||||
not convey it at all. For example, if you agree to terms that obligate you
|
|
||||||
to collect a royalty for further conveying from those to whom you convey
|
|
||||||
the Program, the only way you could satisfy both those terms and this
|
|
||||||
License would be to refrain entirely from conveying the Program.
|
|
||||||
|
|
||||||
13. Remote Network Interaction; Use with the GNU General Public License.
|
|
||||||
|
|
||||||
Notwithstanding any other provision of this License, if you modify the
|
|
||||||
Program, your modified version must prominently offer all users
|
|
||||||
interacting with it remotely through a computer network (if your version
|
|
||||||
supports such interaction) an opportunity to receive the Corresponding
|
|
||||||
Source of your version by providing access to the Corresponding Source
|
|
||||||
from a network server at no charge, through some standard or customary
|
|
||||||
means of facilitating copying of software. This Corresponding Source
|
|
||||||
shall include the Corresponding Source for any work covered by version 3
|
|
||||||
of the GNU General Public License that is incorporated pursuant to the
|
|
||||||
following paragraph.
|
|
||||||
|
|
||||||
Notwithstanding any other provision of this License, you have
|
|
||||||
permission to link or combine any covered work with a work licensed
|
|
||||||
under version 3 of the GNU General Public License into a single
|
|
||||||
combined work, and to convey the resulting work. The terms of this
|
|
||||||
License will continue to apply to the part which is the covered work,
|
|
||||||
but the work with which it is combined will remain governed by version
|
|
||||||
3 of the GNU General Public License.
|
|
||||||
|
|
||||||
14. Revised Versions of this License.
|
|
||||||
|
|
||||||
The Free Software Foundation may publish revised and/or new versions of
|
|
||||||
the GNU Affero General Public License from time to time. Such new versions
|
|
||||||
will be similar in spirit to the present version, but may differ in detail to
|
|
||||||
address new problems or concerns.
|
|
||||||
|
|
||||||
Each version is given a distinguishing version number. If the
|
|
||||||
Program specifies that a certain numbered version of the GNU Affero General
|
|
||||||
Public License "or any later version" applies to it, you have the
|
|
||||||
option of following the terms and conditions either of that numbered
|
|
||||||
version or of any later version published by the Free Software
|
|
||||||
Foundation. If the Program does not specify a version number of the
|
|
||||||
GNU Affero General Public License, you may choose any version ever published
|
|
||||||
by the Free Software Foundation.
|
|
||||||
|
|
||||||
If the Program specifies that a proxy can decide which future
|
|
||||||
versions of the GNU Affero General Public License can be used, that proxy's
|
|
||||||
public statement of acceptance of a version permanently authorizes you
|
|
||||||
to choose that version for the Program.
|
|
||||||
|
|
||||||
Later license versions may give you additional or different
|
|
||||||
permissions. However, no additional obligations are imposed on any
|
|
||||||
author or copyright holder as a result of your choosing to follow a
|
|
||||||
later version.
|
|
||||||
|
|
||||||
15. Disclaimer of Warranty.
|
|
||||||
|
|
||||||
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
|
|
||||||
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
|
|
||||||
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
|
|
||||||
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
|
|
||||||
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
|
|
||||||
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
|
|
||||||
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
|
|
||||||
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
|
|
||||||
|
|
||||||
16. Limitation of Liability.
|
|
||||||
|
|
||||||
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
|
|
||||||
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
|
|
||||||
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
|
|
||||||
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
|
|
||||||
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
|
|
||||||
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
|
|
||||||
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
|
|
||||||
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
|
|
||||||
SUCH DAMAGES.
|
|
||||||
|
|
||||||
17. Interpretation of Sections 15 and 16.
|
|
||||||
|
|
||||||
If the disclaimer of warranty and limitation of liability provided
|
|
||||||
above cannot be given local legal effect according to their terms,
|
|
||||||
reviewing courts shall apply local law that most closely approximates
|
|
||||||
an absolute waiver of all civil liability in connection with the
|
|
||||||
Program, unless a warranty or assumption of liability accompanies a
|
|
||||||
copy of the Program in return for a fee.
|
|
||||||
|
|
||||||
END OF TERMS AND CONDITIONS
|
|
||||||
|
|
||||||
How to Apply These Terms to Your New Programs
|
|
||||||
|
|
||||||
If you develop a new program, and you want it to be of the greatest
|
|
||||||
possible use to the public, the best way to achieve this is to make it
|
|
||||||
free software which everyone can redistribute and change under these terms.
|
|
||||||
|
|
||||||
To do so, attach the following notices to the program. It is safest
|
|
||||||
to attach them to the start of each source file to most effectively
|
|
||||||
state the exclusion of warranty; and each file should have at least
|
|
||||||
the "copyright" line and a pointer to where the full notice is found.
|
|
||||||
|
|
||||||
<one line to give the program's name and a brief idea of what it does.>
|
|
||||||
Copyright (C) <year> <name of author>
|
|
||||||
|
|
||||||
This program is free software: you can redistribute it and/or modify
|
|
||||||
it under the terms of the GNU Affero General Public License as published by
|
|
||||||
the Free Software Foundation, either version 3 of the License, or
|
|
||||||
(at your option) any later version.
|
|
||||||
|
|
||||||
This program is distributed in the hope that it will be useful,
|
|
||||||
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
||||||
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
||||||
GNU Affero General Public License for more details.
|
|
||||||
|
|
||||||
You should have received a copy of the GNU Affero General Public License
|
|
||||||
along with this program. If not, see <http://www.gnu.org/licenses/>.
|
|
||||||
|
|
||||||
Also add information on how to contact you by electronic and paper mail.
|
|
||||||
|
|
||||||
If your software can interact with users remotely through a computer
|
|
||||||
network, you should also make sure that it provides a way for users to
|
|
||||||
get its source. For example, if your program is a web application, its
|
|
||||||
interface could display a "Source" link that leads users to an archive
|
|
||||||
of the code. There are many ways you could offer source, and different
|
|
||||||
solutions will be better for different programs; see section 13 for the
|
|
||||||
specific requirements.
|
|
||||||
|
|
||||||
You should also get your employer (if you work as a programmer) or school,
|
|
||||||
if any, to sign a "copyright disclaimer" for the program, if necessary.
|
|
||||||
For more information on this, and how to apply and follow the GNU AGPL, see
|
|
||||||
<http://www.gnu.org/licenses/>.
|
|
|
@ -1,3 +0,0 @@
|
||||||
# Access Token Encryptor
|
|
||||||
|
|
||||||
Used in third-party-references, to encrypt access tokens
|
|
|
@ -1,10 +0,0 @@
|
||||||
access-token-encryptor
|
|
||||||
--dependencies=None
|
|
||||||
--docker-repos=gcr.io/overleaf-ops
|
|
||||||
--env-add=
|
|
||||||
--env-pass-through=
|
|
||||||
--esmock-loader=False
|
|
||||||
--is-library=True
|
|
||||||
--node-version=20.18.2
|
|
||||||
--public-repo=False
|
|
||||||
--script-version=4.5.0
|
|
|
@ -1 +0,0 @@
|
||||||
module.exports = require('./lib/js/AccessTokenEncryptor')
|
|
|
@ -1,164 +0,0 @@
|
||||||
const { promisify } = require('node:util')
|
|
||||||
const crypto = require('node:crypto')
|
|
||||||
|
|
||||||
const ALGORITHM = 'aes-256-ctr'
|
|
||||||
|
|
||||||
const cryptoHkdf = promisify(crypto.hkdf)
|
|
||||||
const cryptoRandomBytes = promisify(crypto.randomBytes)
|
|
||||||
|
|
||||||
class AbstractAccessTokenScheme {
|
|
||||||
constructor(cipherLabel, cipherPassword) {
|
|
||||||
this.cipherLabel = cipherLabel
|
|
||||||
this.cipherPassword = cipherPassword
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* @param {Object} json
|
|
||||||
* @return {Promise<string>}
|
|
||||||
*/
|
|
||||||
async encryptJson(json) {
|
|
||||||
throw new Error('encryptJson is not implemented')
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* @param {string} encryptedJson
|
|
||||||
* @return {Promise<Object>}
|
|
||||||
*/
|
|
||||||
async decryptToJson(encryptedJson) {
|
|
||||||
throw new Error('decryptToJson is not implemented')
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
class AccessTokenSchemeWithGenericKeyFn extends AbstractAccessTokenScheme {
|
|
||||||
/**
|
|
||||||
* @param {Buffer} salt
|
|
||||||
* @return {Promise<Buffer>}
|
|
||||||
*/
|
|
||||||
async keyFn(salt) {
|
|
||||||
throw new Error('keyFn is not implemented')
|
|
||||||
}
|
|
||||||
|
|
||||||
async encryptJson(json) {
|
|
||||||
const plainText = JSON.stringify(json)
|
|
||||||
|
|
||||||
const bytes = await cryptoRandomBytes(32)
|
|
||||||
const salt = bytes.slice(0, 16)
|
|
||||||
const iv = bytes.slice(16, 32)
|
|
||||||
const key = await this.keyFn(salt)
|
|
||||||
|
|
||||||
const cipher = crypto.createCipheriv(ALGORITHM, key, iv)
|
|
||||||
const cipherText =
|
|
||||||
cipher.update(plainText, 'utf8', 'base64') + cipher.final('base64')
|
|
||||||
|
|
||||||
return [
|
|
||||||
this.cipherLabel,
|
|
||||||
salt.toString('hex'),
|
|
||||||
cipherText,
|
|
||||||
iv.toString('hex'),
|
|
||||||
].join(':')
|
|
||||||
}
|
|
||||||
|
|
||||||
async decryptToJson(encryptedJson) {
|
|
||||||
const [, salt, cipherText, iv] = encryptedJson.split(':', 4)
|
|
||||||
const key = await this.keyFn(Buffer.from(salt, 'hex'))
|
|
||||||
|
|
||||||
const decipher = crypto.createDecipheriv(
|
|
||||||
ALGORITHM,
|
|
||||||
key,
|
|
||||||
Buffer.from(iv, 'hex')
|
|
||||||
)
|
|
||||||
const plainText =
|
|
||||||
decipher.update(cipherText, 'base64', 'utf8') + decipher.final('utf8')
|
|
||||||
try {
|
|
||||||
return JSON.parse(plainText)
|
|
||||||
} catch (e) {
|
|
||||||
throw new Error('error decrypting token')
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
class AccessTokenSchemeV3 extends AccessTokenSchemeWithGenericKeyFn {
|
|
||||||
async keyFn(salt) {
|
|
||||||
const optionalInfo = ''
|
|
||||||
return await cryptoHkdf(
|
|
||||||
'sha512',
|
|
||||||
this.cipherPassword,
|
|
||||||
salt,
|
|
||||||
optionalInfo,
|
|
||||||
32
|
|
||||||
)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
class AccessTokenEncryptor {
|
|
||||||
constructor(settings) {
|
|
||||||
/**
|
|
||||||
* @type {Map<string, AbstractAccessTokenScheme>}
|
|
||||||
*/
|
|
||||||
this.schemeByCipherLabel = new Map()
|
|
||||||
for (const cipherLabel of Object.keys(settings.cipherPasswords)) {
|
|
||||||
if (!cipherLabel) {
|
|
||||||
throw new Error('cipherLabel cannot be empty')
|
|
||||||
}
|
|
||||||
if (cipherLabel.match(/:/)) {
|
|
||||||
throw new Error(
|
|
||||||
`cipherLabel must not contain a colon (:), got ${cipherLabel}`
|
|
||||||
)
|
|
||||||
}
|
|
||||||
const [, version] = cipherLabel.split('-')
|
|
||||||
if (!version) {
|
|
||||||
throw new Error(
|
|
||||||
`cipherLabel must contain version suffix (e.g. 2042.1-v42), got ${cipherLabel}`
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
const cipherPassword = settings.cipherPasswords[cipherLabel]
|
|
||||||
if (!cipherPassword) {
|
|
||||||
throw new Error(`cipherPasswords['${cipherLabel}'] is missing`)
|
|
||||||
}
|
|
||||||
if (cipherPassword.length < 16) {
|
|
||||||
throw new Error(`cipherPasswords['${cipherLabel}'] is too short`)
|
|
||||||
}
|
|
||||||
|
|
||||||
let scheme
|
|
||||||
switch (version) {
|
|
||||||
case 'v3':
|
|
||||||
scheme = new AccessTokenSchemeV3(cipherLabel, cipherPassword)
|
|
||||||
break
|
|
||||||
default:
|
|
||||||
throw new Error(`unknown version '${version}' for ${cipherLabel}`)
|
|
||||||
}
|
|
||||||
this.schemeByCipherLabel.set(cipherLabel, scheme)
|
|
||||||
}
|
|
||||||
|
|
||||||
/** @type {AbstractAccessTokenScheme} */
|
|
||||||
this.defaultScheme = this.schemeByCipherLabel.get(settings.cipherLabel)
|
|
||||||
if (!this.defaultScheme) {
|
|
||||||
throw new Error(`unknown default cipherLabel ${settings.cipherLabel}`)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
promises = {
|
|
||||||
encryptJson: async json => await this.defaultScheme.encryptJson(json),
|
|
||||||
decryptToJson: async encryptedJson => {
|
|
||||||
const [label] = encryptedJson.split(':', 1)
|
|
||||||
const scheme = this.schemeByCipherLabel.get(label)
|
|
||||||
if (!scheme) {
|
|
||||||
throw new Error('unknown access-token-encryptor label ' + label)
|
|
||||||
}
|
|
||||||
return await scheme.decryptToJson(encryptedJson)
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
encryptJson(json, callback) {
|
|
||||||
this.promises.encryptJson(json).then(s => callback(null, s), callback)
|
|
||||||
}
|
|
||||||
|
|
||||||
decryptToJson(encryptedJson, callback) {
|
|
||||||
this.promises
|
|
||||||
.decryptToJson(encryptedJson)
|
|
||||||
.then(o => callback(null, o), callback)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
module.exports = AccessTokenEncryptor
|
|
|
@ -1,28 +0,0 @@
|
||||||
{
|
|
||||||
"name": "@overleaf/access-token-encryptor",
|
|
||||||
"version": "3.0.0",
|
|
||||||
"description": "",
|
|
||||||
"main": "index.js",
|
|
||||||
"scripts": {
|
|
||||||
"test": "npm run lint && npm run format && npm run types:check && npm run test:unit",
|
|
||||||
"lint": "eslint --ext .js --ext .cjs --ext .ts --max-warnings 0 --format unix .",
|
|
||||||
"lint:fix": "eslint --fix --ext .js --ext .cjs --ext .ts .",
|
|
||||||
"format": "prettier --list-different $PWD/'**/*.{js,cjs,ts}'",
|
|
||||||
"format:fix": "prettier --write $PWD/'**/*.{js,cjs,ts}'",
|
|
||||||
"test:ci": "npm run test:unit",
|
|
||||||
"test:unit": "mocha --exit test/**/*.{js,cjs}",
|
|
||||||
"types:check": "tsc --noEmit"
|
|
||||||
},
|
|
||||||
"author": "",
|
|
||||||
"license": "AGPL-3.0-only",
|
|
||||||
"dependencies": {
|
|
||||||
"lodash": "^4.17.21"
|
|
||||||
},
|
|
||||||
"devDependencies": {
|
|
||||||
"chai": "^4.3.6",
|
|
||||||
"chai-as-promised": "^7.1.1",
|
|
||||||
"mocha": "^11.1.0",
|
|
||||||
"sandboxed-module": "^2.0.4",
|
|
||||||
"typescript": "^5.0.4"
|
|
||||||
}
|
|
||||||
}
|
|
|
@ -1,27 +0,0 @@
|
||||||
function formatTokenUsageStats(STATS) {
|
|
||||||
const prettyStats = []
|
|
||||||
const sortedStats = Object.entries(STATS).sort((a, b) =>
|
|
||||||
a[0] > b[0] ? 1 : -1
|
|
||||||
)
|
|
||||||
const totalByName = {}
|
|
||||||
for (const [key, n] of sortedStats) {
|
|
||||||
const [name, version, collectionName, path, label] = key.split(':')
|
|
||||||
totalByName[name] = (totalByName[name] || 0) + n
|
|
||||||
prettyStats.push({ name, version, collectionName, path, label, n })
|
|
||||||
}
|
|
||||||
for (const row of prettyStats) {
|
|
||||||
row.percentage = ((100 * row.n) / totalByName[row.name])
|
|
||||||
.toFixed(2)
|
|
||||||
.padStart(6)
|
|
||||||
}
|
|
||||||
|
|
||||||
if (prettyStats.length === 0) {
|
|
||||||
console.warn('---')
|
|
||||||
console.warn('Found 0 access tokens.')
|
|
||||||
console.warn('---')
|
|
||||||
} else {
|
|
||||||
console.table(prettyStats)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
module.exports = { formatTokenUsageStats }
|
|
|
@ -1,108 +0,0 @@
|
||||||
const _ = require('lodash')
|
|
||||||
const { formatTokenUsageStats } = require('./format-usage-stats')
|
|
||||||
|
|
||||||
const LOG_EVERY_IN_S = parseInt(process.env.LOG_EVERY_IN_S || '5', 10)
|
|
||||||
const DRY_RUN = !process.argv.includes('--dry-run=false')
|
|
||||||
|
|
||||||
/**
|
|
||||||
* @param {AccessTokenEncryptor} accessTokenEncryptor
|
|
||||||
* @param {string} encryptedJson
|
|
||||||
* @return {Promise<string>}
|
|
||||||
*/
|
|
||||||
async function reEncryptTokens(accessTokenEncryptor, encryptedJson) {
|
|
||||||
return await new Promise((resolve, reject) => {
|
|
||||||
accessTokenEncryptor.decryptToJson(encryptedJson, (err, json) => {
|
|
||||||
if (err) return reject(err)
|
|
||||||
accessTokenEncryptor.encryptJson(json, (err, reEncryptedJson) => {
|
|
||||||
if (err) return reject(err)
|
|
||||||
resolve(reEncryptedJson)
|
|
||||||
})
|
|
||||||
})
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* @param {AccessTokenEncryptor} accessTokenEncryptor
|
|
||||||
* @param {Collection} collection
|
|
||||||
* @param {Object} paths
|
|
||||||
* @param {Object} queryOptions
|
|
||||||
* @return {Promise<{}>}
|
|
||||||
*/
|
|
||||||
async function reEncryptTokensInCollection({
|
|
||||||
accessTokenEncryptor,
|
|
||||||
collection,
|
|
||||||
paths,
|
|
||||||
queryOptions,
|
|
||||||
}) {
|
|
||||||
const { collectionName } = collection
|
|
||||||
const stats = {}
|
|
||||||
|
|
||||||
let processed = 0
|
|
||||||
let updatedNUsers = 0
|
|
||||||
let lastLog = 0
|
|
||||||
const logProgress = () => {
|
|
||||||
if (DRY_RUN) {
|
|
||||||
console.warn(
|
|
||||||
`processed ${processed} | Would have updated ${updatedNUsers} users`
|
|
||||||
)
|
|
||||||
} else {
|
|
||||||
console.warn(`processed ${processed} | Updated ${updatedNUsers} users`)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
const projection = { _id: 1 }
|
|
||||||
for (const path of Object.values(paths)) {
|
|
||||||
projection[path] = 1
|
|
||||||
}
|
|
||||||
const cursor = collection.find(
|
|
||||||
{},
|
|
||||||
{
|
|
||||||
...queryOptions,
|
|
||||||
projection,
|
|
||||||
}
|
|
||||||
)
|
|
||||||
|
|
||||||
for await (const doc of cursor) {
|
|
||||||
processed++
|
|
||||||
|
|
||||||
let update = null
|
|
||||||
for (const [name, path] of Object.entries(paths)) {
|
|
||||||
const blob = _.get(doc, path)
|
|
||||||
if (!blob) continue
|
|
||||||
// Schema: LABEL-VERSION:SALT:CIPHERTEXT:IV
|
|
||||||
const [label] = blob.split(':')
|
|
||||||
let [, version] = label.split('-')
|
|
||||||
version = version || 'v2'
|
|
||||||
|
|
||||||
const key = [name, version, collectionName, path, label].join(':')
|
|
||||||
stats[key] = (stats[key] || 0) + 1
|
|
||||||
|
|
||||||
if (version === 'v2') {
|
|
||||||
update = update || {}
|
|
||||||
update[path] = await reEncryptTokens(accessTokenEncryptor, blob)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if (Date.now() - lastLog >= LOG_EVERY_IN_S * 1000) {
|
|
||||||
logProgress()
|
|
||||||
lastLog = Date.now()
|
|
||||||
}
|
|
||||||
if (update) {
|
|
||||||
updatedNUsers++
|
|
||||||
|
|
||||||
const { _id } = doc
|
|
||||||
if (DRY_RUN) {
|
|
||||||
console.log('Would upgrade tokens for user', _id, Object.keys(update))
|
|
||||||
} else {
|
|
||||||
console.log('Upgrading tokens for user', _id, Object.keys(update))
|
|
||||||
await collection.updateOne({ _id }, { $set: update })
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
logProgress()
|
|
||||||
formatTokenUsageStats(stats)
|
|
||||||
}
|
|
||||||
|
|
||||||
module.exports = {
|
|
||||||
reEncryptTokensInCollection,
|
|
||||||
}
|
|
|
@ -1,13 +0,0 @@
|
||||||
const chai = require('chai')
|
|
||||||
const chaiAsPromised = require('chai-as-promised')
|
|
||||||
const SandboxedModule = require('sandboxed-module')
|
|
||||||
|
|
||||||
chai.use(chaiAsPromised)
|
|
||||||
|
|
||||||
SandboxedModule.configure({
|
|
||||||
sourceTransformers: {
|
|
||||||
removeNodePrefix: function (source) {
|
|
||||||
return source.replace(/require\(['"]node:/g, "require('")
|
|
||||||
},
|
|
||||||
},
|
|
||||||
})
|
|
|
@ -1,305 +0,0 @@
|
||||||
const chai = require('chai')
|
|
||||||
chai.should()
|
|
||||||
const { expect } = chai
|
|
||||||
const modulePath = '../../../index.js'
|
|
||||||
const SandboxedModule = require('sandboxed-module')
|
|
||||||
|
|
||||||
describe('AccessTokenEncryptor', function () {
|
|
||||||
beforeEach(function () {
|
|
||||||
this.testObject = { hello: 'world' }
|
|
||||||
this.encrypted2015 =
|
|
||||||
'2015.1:473a66fb5d816bc716f278ab819d88a5:+mTg7O9sgUND8pNQFG6h2GE='
|
|
||||||
this.encrypted2016 =
|
|
||||||
'2016.1:76a7d64a444ccee1a515b49c44844a69:m5YSkexUsLjcF4gLncm72+k='
|
|
||||||
this.encrypted2019 =
|
|
||||||
'2019.1:627143b2ab185a020c8720253a4c984e:7gnY6Ez3/Y3UWgLHLfBtJsE=:bf75cecb6aeea55b3c060e1122d2a82d'
|
|
||||||
this.encrypted2023 =
|
|
||||||
'2023.1-v3:a6dd3781dd6ce93a4134874b505a209c:9TdIDAc8V9SeR0ffSn63Jj4=:d8b2de0b733c81b949993dce229abb4c'
|
|
||||||
this.badLabel = 'xxxxxx:c7a39310056b694c:jQf+Uh5Den3JREtvc82GW5Q='
|
|
||||||
this.badKey = '2015.1:d7a39310056b694c:jQf+Uh5Den3JREtvc82GW5Q='
|
|
||||||
this.badCipherText = '2015.1:c7a39310056b694c:xQf+Uh5Den3JREtvc82GW5Q='
|
|
||||||
this.settings = {
|
|
||||||
cipherLabel: '2023.1-v3',
|
|
||||||
cipherPasswords: {
|
|
||||||
'2023.1-v3': '44444444444444444444444444444444444444',
|
|
||||||
},
|
|
||||||
}
|
|
||||||
this.AccessTokenEncryptor = SandboxedModule.require(modulePath, {
|
|
||||||
globals: {
|
|
||||||
Buffer,
|
|
||||||
},
|
|
||||||
})
|
|
||||||
this.encryptor = new this.AccessTokenEncryptor(this.settings)
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('invalid settings', function () {
|
|
||||||
it('should flag missing label', function () {
|
|
||||||
expect(
|
|
||||||
() =>
|
|
||||||
new this.AccessTokenEncryptor({
|
|
||||||
cipherLabel: '',
|
|
||||||
cipherPasswords: { '': '' },
|
|
||||||
})
|
|
||||||
).to.throw(/cipherLabel cannot be empty/)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('should flag invalid label with colon', function () {
|
|
||||||
expect(
|
|
||||||
() =>
|
|
||||||
new this.AccessTokenEncryptor({
|
|
||||||
cipherLabel: '2023:1-v2',
|
|
||||||
cipherPasswords: { '2023:1-v2': '' },
|
|
||||||
})
|
|
||||||
).to.throw(/colon/)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('should flag missing password', function () {
|
|
||||||
expect(
|
|
||||||
() =>
|
|
||||||
new this.AccessTokenEncryptor({
|
|
||||||
cipherPasswords: { '2023.1-v3': '' },
|
|
||||||
cipherVersions: { '2023.1-v3': 'v3' },
|
|
||||||
})
|
|
||||||
).to.throw(/cipherPasswords.+ missing/)
|
|
||||||
expect(
|
|
||||||
() =>
|
|
||||||
new this.AccessTokenEncryptor({
|
|
||||||
cipherLabel: '2023.1-v3',
|
|
||||||
cipherPasswords: { '2023.1-v3': undefined },
|
|
||||||
})
|
|
||||||
).to.throw(/cipherPasswords.+ missing/)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('should flag short password', function () {
|
|
||||||
expect(
|
|
||||||
() =>
|
|
||||||
new this.AccessTokenEncryptor({
|
|
||||||
cipherLabel: '2023.1-v3',
|
|
||||||
cipherPasswords: { '2023.1-v3': 'foo' },
|
|
||||||
})
|
|
||||||
).to.throw(/cipherPasswords.+ too short/)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('should flag missing version', function () {
|
|
||||||
expect(
|
|
||||||
() =>
|
|
||||||
new this.AccessTokenEncryptor({
|
|
||||||
cipherLabel: '2023.1',
|
|
||||||
cipherPasswords: { 2023.1: '11111111111111111111111111111111' },
|
|
||||||
})
|
|
||||||
).to.throw(/must contain version suffix/)
|
|
||||||
expect(
|
|
||||||
() =>
|
|
||||||
new this.AccessTokenEncryptor({
|
|
||||||
cipherLabel: '2023.1-',
|
|
||||||
cipherPasswords: { '2023.1-': '11111111111111111111111111111111' },
|
|
||||||
})
|
|
||||||
).to.throw(/must contain version suffix/)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('should flag invalid version', function () {
|
|
||||||
expect(
|
|
||||||
() =>
|
|
||||||
new this.AccessTokenEncryptor({
|
|
||||||
cipherLabel: '2023.1-v0',
|
|
||||||
cipherPasswords: {
|
|
||||||
'2023.1-v0': '11111111111111111111111111111111',
|
|
||||||
},
|
|
||||||
})
|
|
||||||
).to.throw(/unknown version/)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('should flag unknown default scheme', function () {
|
|
||||||
expect(
|
|
||||||
() =>
|
|
||||||
new this.AccessTokenEncryptor({
|
|
||||||
cipherLabel: '2000.1-v3',
|
|
||||||
cipherPasswords: {
|
|
||||||
'2023.1-v3': '11111111111111111111111111111111',
|
|
||||||
},
|
|
||||||
})
|
|
||||||
).to.throw(/unknown default cipherLabel/)
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('sync', function () {
|
|
||||||
describe('encrypt', function () {
|
|
||||||
it('should encrypt the object', function (done) {
|
|
||||||
this.encryptor.encryptJson(this.testObject, (err, encrypted) => {
|
|
||||||
expect(err).to.be.null
|
|
||||||
encrypted.should.match(
|
|
||||||
/^2023.1-v3:[0-9a-f]{32}:[a-zA-Z0-9=+/]+:[0-9a-f]{32}$/
|
|
||||||
)
|
|
||||||
done()
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
it('should encrypt the object differently the next time', function (done) {
|
|
||||||
this.encryptor.encryptJson(this.testObject, (err, encrypted1) => {
|
|
||||||
expect(err).to.be.null
|
|
||||||
this.encryptor.encryptJson(this.testObject, (err, encrypted2) => {
|
|
||||||
expect(err).to.be.null
|
|
||||||
encrypted1.should.not.equal(encrypted2)
|
|
||||||
done()
|
|
||||||
})
|
|
||||||
})
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('decrypt', function () {
|
|
||||||
it('should decrypt the string to get the same object', function (done) {
|
|
||||||
this.encryptor.encryptJson(this.testObject, (err, encrypted) => {
|
|
||||||
expect(err).to.be.null
|
|
||||||
this.encryptor.decryptToJson(encrypted, (err, decrypted) => {
|
|
||||||
expect(err).to.be.null
|
|
||||||
expect(decrypted).to.deep.equal(this.testObject)
|
|
||||||
done()
|
|
||||||
})
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
it('should not be able to decrypt 2015 string', function (done) {
|
|
||||||
this.encryptor.decryptToJson(this.encrypted2015, (err, decrypted) => {
|
|
||||||
expect(err).to.exist
|
|
||||||
expect(err.message).to.equal(
|
|
||||||
'unknown access-token-encryptor label 2015.1'
|
|
||||||
)
|
|
||||||
expect(decrypted).to.not.exist
|
|
||||||
done()
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
it('should not be able to decrypt a 2016 string', function (done) {
|
|
||||||
this.encryptor.decryptToJson(this.encrypted2016, (err, decrypted) => {
|
|
||||||
expect(err).to.exist
|
|
||||||
expect(err.message).to.equal(
|
|
||||||
'unknown access-token-encryptor label 2016.1'
|
|
||||||
)
|
|
||||||
expect(decrypted).to.not.exist
|
|
||||||
done()
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
it('should not be able to decrypt a 2019 string', function (done) {
|
|
||||||
this.encryptor.decryptToJson(this.encrypted2019, (err, decrypted) => {
|
|
||||||
expect(err).to.exist
|
|
||||||
expect(err.message).to.equal(
|
|
||||||
'unknown access-token-encryptor label 2019.1'
|
|
||||||
)
|
|
||||||
expect(decrypted).to.not.exist
|
|
||||||
done()
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
it('should decrypt an 2023 string to get the same object', function (done) {
|
|
||||||
this.encryptor.decryptToJson(this.encrypted2023, (err, decrypted) => {
|
|
||||||
expect(err).to.be.null
|
|
||||||
expect(decrypted).to.deep.equal(this.testObject)
|
|
||||||
done()
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
it('should return an error when decrypting an invalid label', function (done) {
|
|
||||||
this.encryptor.decryptToJson(this.badLabel, (err, decrypted) => {
|
|
||||||
expect(err).to.be.instanceof(Error)
|
|
||||||
expect(decrypted).to.be.undefined
|
|
||||||
done()
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
it('should return an error when decrypting an invalid key', function (done) {
|
|
||||||
this.encryptor.decryptToJson(this.badKey, (err, decrypted) => {
|
|
||||||
expect(err).to.be.instanceof(Error)
|
|
||||||
expect(decrypted).to.be.undefined
|
|
||||||
done()
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
it('should return an error when decrypting an invalid ciphertext', function (done) {
|
|
||||||
this.encryptor.decryptToJson(this.badCipherText, (err, decrypted) => {
|
|
||||||
expect(err).to.be.instanceof(Error)
|
|
||||||
expect(decrypted).to.be.undefined
|
|
||||||
done()
|
|
||||||
})
|
|
||||||
})
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('async', function () {
|
|
||||||
describe('encrypt', function () {
|
|
||||||
it('should encrypt the object', async function () {
|
|
||||||
const encrypted = await this.encryptor.promises.encryptJson(
|
|
||||||
this.testObject
|
|
||||||
)
|
|
||||||
encrypted.should.match(
|
|
||||||
/^2023.1-v3:[0-9a-f]{32}:[a-zA-Z0-9=+/]+:[0-9a-f]{32}$/
|
|
||||||
)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('should encrypt the object differently the next time', async function () {
|
|
||||||
const encrypted1 = await this.encryptor.promises.encryptJson(
|
|
||||||
this.testObject
|
|
||||||
)
|
|
||||||
const encrypted2 = await this.encryptor.promises.encryptJson(
|
|
||||||
this.testObject
|
|
||||||
)
|
|
||||||
encrypted1.should.not.equal(encrypted2)
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('decrypt', function () {
|
|
||||||
it('should decrypt the string to get the same object', async function () {
|
|
||||||
const encrypted = await this.encryptor.promises.encryptJson(
|
|
||||||
this.testObject
|
|
||||||
)
|
|
||||||
const decrypted = await this.encryptor.promises.decryptToJson(encrypted)
|
|
||||||
expect(decrypted).to.deep.equal(this.testObject)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('should not be able to decrypt 2015 string', async function () {
|
|
||||||
await expect(
|
|
||||||
this.encryptor.promises.decryptToJson(this.encrypted2015)
|
|
||||||
).to.eventually.be.rejectedWith(
|
|
||||||
'unknown access-token-encryptor label 2015.1'
|
|
||||||
)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('should not be able to decrypt a 2016 string', async function () {
|
|
||||||
await expect(
|
|
||||||
this.encryptor.promises.decryptToJson(this.encrypted2016)
|
|
||||||
).to.be.rejectedWith('unknown access-token-encryptor label 2016.1')
|
|
||||||
})
|
|
||||||
|
|
||||||
it('should not be able to decrypt a 2019 string', async function () {
|
|
||||||
await expect(
|
|
||||||
this.encryptor.promises.decryptToJson(this.encrypted2019)
|
|
||||||
).to.be.rejectedWith('unknown access-token-encryptor label 2019.1')
|
|
||||||
})
|
|
||||||
|
|
||||||
it('should decrypt an 2023 string to get the same object', async function () {
|
|
||||||
const decrypted = await this.encryptor.promises.decryptToJson(
|
|
||||||
this.encrypted2023
|
|
||||||
)
|
|
||||||
expect(decrypted).to.deep.equal(this.testObject)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('should return an error when decrypting an invalid label', async function () {
|
|
||||||
await expect(
|
|
||||||
this.encryptor.promises.decryptToJson(this.badLabel)
|
|
||||||
).to.be.rejectedWith('unknown access-token-encryptor label xxxxxx')
|
|
||||||
})
|
|
||||||
|
|
||||||
it('should return an error when decrypting an invalid key', async function () {
|
|
||||||
await expect(
|
|
||||||
this.encryptor.promises.decryptToJson(this.badKey)
|
|
||||||
).to.be.rejectedWith('unknown access-token-encryptor label 2015.1')
|
|
||||||
})
|
|
||||||
|
|
||||||
it('should return an error when decrypting an invalid ciphertext', async function () {
|
|
||||||
await expect(
|
|
||||||
this.encryptor.promises.decryptToJson(this.badCipherText)
|
|
||||||
).to.be.rejectedWith('unknown access-token-encryptor label 2015.1')
|
|
||||||
})
|
|
||||||
})
|
|
||||||
})
|
|
||||||
})
|
|
|
@ -1,7 +0,0 @@
|
||||||
{
|
|
||||||
"extends": "../../tsconfig.backend.json",
|
|
||||||
"include": [
|
|
||||||
"**/*.js",
|
|
||||||
"**/*.cjs"
|
|
||||||
]
|
|
||||||
}
|
|
3
libraries/fetch-utils/.gitignore
vendored
3
libraries/fetch-utils/.gitignore
vendored
|
@ -1,3 +0,0 @@
|
||||||
|
|
||||||
# managed by monorepo$ bin/update_build_scripts
|
|
||||||
.npmrc
|
|
|
@ -1,6 +0,0 @@
|
||||||
{
|
|
||||||
"ui": "bdd",
|
|
||||||
"recursive": "true",
|
|
||||||
"reporter": "spec",
|
|
||||||
"require": "test/setup.js"
|
|
||||||
}
|
|
|
@ -1 +0,0 @@
|
||||||
20.18.2
|
|
|
@ -1,10 +0,0 @@
|
||||||
fetch-utils
|
|
||||||
--dependencies=None
|
|
||||||
--docker-repos=gcr.io/overleaf-ops
|
|
||||||
--env-add=
|
|
||||||
--env-pass-through=
|
|
||||||
--esmock-loader=False
|
|
||||||
--is-library=True
|
|
||||||
--node-version=20.18.2
|
|
||||||
--public-repo=False
|
|
||||||
--script-version=4.5.0
|
|
|
@ -1,320 +0,0 @@
|
||||||
const _ = require('lodash')
|
|
||||||
const { Readable } = require('node:stream')
|
|
||||||
const OError = require('@overleaf/o-error')
|
|
||||||
const fetch = require('node-fetch')
|
|
||||||
const http = require('node:http')
|
|
||||||
const https = require('node:https')
|
|
||||||
|
|
||||||
/**
|
|
||||||
* @import { Response } from 'node-fetch'
|
|
||||||
*/
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Make a request and return the parsed JSON response.
|
|
||||||
*
|
|
||||||
* @param {string | URL} url - request URL
|
|
||||||
* @param {any} [opts] - fetch options
|
|
||||||
* @return {Promise<any>} the parsed JSON response
|
|
||||||
* @throws {RequestFailedError} if the response has a failure status code
|
|
||||||
*/
|
|
||||||
async function fetchJson(url, opts = {}) {
|
|
||||||
const { json } = await fetchJsonWithResponse(url, opts)
|
|
||||||
return json
|
|
||||||
}
|
|
||||||
|
|
||||||
async function fetchJsonWithResponse(url, opts = {}) {
|
|
||||||
const { fetchOpts } = parseOpts(opts)
|
|
||||||
fetchOpts.headers = fetchOpts.headers ?? {}
|
|
||||||
fetchOpts.headers.Accept = fetchOpts.headers.Accept ?? 'application/json'
|
|
||||||
|
|
||||||
const response = await performRequest(url, fetchOpts)
|
|
||||||
if (!response.ok) {
|
|
||||||
const body = await maybeGetResponseBody(response)
|
|
||||||
throw new RequestFailedError(url, opts, response, body)
|
|
||||||
}
|
|
||||||
|
|
||||||
const json = await response.json()
|
|
||||||
return { json, response }
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Make a request and return a stream.
|
|
||||||
*
|
|
||||||
* If the response body is destroyed, the request is aborted.
|
|
||||||
*
|
|
||||||
* @param {string | URL} url - request URL
|
|
||||||
* @param {any} [opts] - fetch options
|
|
||||||
* @return {Promise<Readable>}
|
|
||||||
* @throws {RequestFailedError} if the response has a failure status code
|
|
||||||
*/
|
|
||||||
async function fetchStream(url, opts = {}) {
|
|
||||||
const { stream } = await fetchStreamWithResponse(url, opts)
|
|
||||||
return stream
|
|
||||||
}
|
|
||||||
|
|
||||||
async function fetchStreamWithResponse(url, opts = {}) {
|
|
||||||
const { fetchOpts, abortController } = parseOpts(opts)
|
|
||||||
const response = await performRequest(url, fetchOpts)
|
|
||||||
|
|
||||||
if (!response.ok) {
|
|
||||||
const body = await maybeGetResponseBody(response)
|
|
||||||
throw new RequestFailedError(url, opts, response, body)
|
|
||||||
}
|
|
||||||
|
|
||||||
abortOnDestroyedResponse(abortController, response)
|
|
||||||
|
|
||||||
const stream = response.body
|
|
||||||
return { stream, response }
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Make a request and discard the response.
|
|
||||||
*
|
|
||||||
* @param {string | URL} url - request URL
|
|
||||||
* @param {any} [opts] - fetch options
|
|
||||||
* @return {Promise<Response>}
|
|
||||||
* @throws {RequestFailedError} if the response has a failure status code
|
|
||||||
*/
|
|
||||||
async function fetchNothing(url, opts = {}) {
|
|
||||||
const { fetchOpts } = parseOpts(opts)
|
|
||||||
const response = await performRequest(url, fetchOpts)
|
|
||||||
if (!response.ok) {
|
|
||||||
const body = await maybeGetResponseBody(response)
|
|
||||||
throw new RequestFailedError(url, opts, response, body)
|
|
||||||
}
|
|
||||||
await discardResponseBody(response)
|
|
||||||
return response
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Make a request and extract the redirect from the response.
|
|
||||||
*
|
|
||||||
* @param {string | URL} url - request URL
|
|
||||||
* @param {any} [opts] - fetch options
|
|
||||||
* @return {Promise<string>}
|
|
||||||
* @throws {RequestFailedError} if the response has a non redirect status code or missing Location header
|
|
||||||
*/
|
|
||||||
async function fetchRedirect(url, opts = {}) {
|
|
||||||
const { location } = await fetchRedirectWithResponse(url, opts)
|
|
||||||
return location
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Make a request and extract the redirect from the response.
|
|
||||||
*
|
|
||||||
* @param {string | URL} url - request URL
|
|
||||||
* @param {object} opts - fetch options
|
|
||||||
* @return {Promise<{location: string, response: Response}>}
|
|
||||||
* @throws {RequestFailedError} if the response has a non redirect status code or missing Location header
|
|
||||||
*/
|
|
||||||
async function fetchRedirectWithResponse(url, opts = {}) {
|
|
||||||
const { fetchOpts } = parseOpts(opts)
|
|
||||||
fetchOpts.redirect = 'manual'
|
|
||||||
const response = await performRequest(url, fetchOpts)
|
|
||||||
if (response.status < 300 || response.status >= 400) {
|
|
||||||
const body = await maybeGetResponseBody(response)
|
|
||||||
throw new RequestFailedError(url, opts, response, body)
|
|
||||||
}
|
|
||||||
const location = response.headers.get('Location')
|
|
||||||
if (!location) {
|
|
||||||
const body = await maybeGetResponseBody(response)
|
|
||||||
throw new RequestFailedError(url, opts, response, body).withCause(
|
|
||||||
new OError('missing Location response header on 3xx response', {
|
|
||||||
headers: Object.fromEntries(response.headers.entries()),
|
|
||||||
})
|
|
||||||
)
|
|
||||||
}
|
|
||||||
await discardResponseBody(response)
|
|
||||||
return { location, response }
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Make a request and return a string.
|
|
||||||
*
|
|
||||||
* @param {string | URL} url - request URL
|
|
||||||
* @param {any} [opts] - fetch options
|
|
||||||
* @return {Promise<string>}
|
|
||||||
* @throws {RequestFailedError} if the response has a failure status code
|
|
||||||
*/
|
|
||||||
async function fetchString(url, opts = {}) {
|
|
||||||
const { body } = await fetchStringWithResponse(url, opts)
|
|
||||||
return body
|
|
||||||
}
|
|
||||||
|
|
||||||
async function fetchStringWithResponse(url, opts = {}) {
|
|
||||||
const { fetchOpts } = parseOpts(opts)
|
|
||||||
const response = await performRequest(url, fetchOpts)
|
|
||||||
if (!response.ok) {
|
|
||||||
const body = await maybeGetResponseBody(response)
|
|
||||||
throw new RequestFailedError(url, opts, response, body)
|
|
||||||
}
|
|
||||||
const body = await response.text()
|
|
||||||
return { body, response }
|
|
||||||
}
|
|
||||||
|
|
||||||
class RequestFailedError extends OError {
|
|
||||||
constructor(url, opts, response, body) {
|
|
||||||
super('request failed', {
|
|
||||||
url,
|
|
||||||
method: opts.method ?? 'GET',
|
|
||||||
status: response.status,
|
|
||||||
})
|
|
||||||
|
|
||||||
this.response = response
|
|
||||||
if (body != null) {
|
|
||||||
this.body = body
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
function parseOpts(opts) {
|
|
||||||
const fetchOpts = _.omit(opts, ['json', 'signal', 'basicAuth'])
|
|
||||||
if (opts.json) {
|
|
||||||
setupJsonBody(fetchOpts, opts.json)
|
|
||||||
}
|
|
||||||
if (opts.basicAuth) {
|
|
||||||
setupBasicAuth(fetchOpts, opts.basicAuth)
|
|
||||||
}
|
|
||||||
|
|
||||||
const abortController = new AbortController()
|
|
||||||
fetchOpts.signal = abortController.signal
|
|
||||||
if (opts.signal) {
|
|
||||||
abortOnSignal(abortController, opts.signal)
|
|
||||||
}
|
|
||||||
if (opts.body instanceof Readable) {
|
|
||||||
abortOnDestroyedRequest(abortController, fetchOpts.body)
|
|
||||||
}
|
|
||||||
return { fetchOpts, abortController }
|
|
||||||
}
|
|
||||||
|
|
||||||
function setupJsonBody(fetchOpts, json) {
|
|
||||||
fetchOpts.body = JSON.stringify(json)
|
|
||||||
fetchOpts.headers = fetchOpts.headers ?? {}
|
|
||||||
fetchOpts.headers['Content-Type'] = 'application/json'
|
|
||||||
}
|
|
||||||
|
|
||||||
function setupBasicAuth(fetchOpts, basicAuth) {
|
|
||||||
fetchOpts.headers = fetchOpts.headers ?? {}
|
|
||||||
fetchOpts.headers.Authorization =
|
|
||||||
'Basic ' +
|
|
||||||
Buffer.from(`${basicAuth.user}:${basicAuth.password}`).toString('base64')
|
|
||||||
}
|
|
||||||
|
|
||||||
function abortOnSignal(abortController, signal) {
|
|
||||||
const listener = () => {
|
|
||||||
abortController.abort(signal.reason)
|
|
||||||
}
|
|
||||||
if (signal.aborted) {
|
|
||||||
abortController.abort(signal.reason)
|
|
||||||
}
|
|
||||||
signal.addEventListener('abort', listener)
|
|
||||||
}
|
|
||||||
|
|
||||||
function abortOnDestroyedRequest(abortController, stream) {
|
|
||||||
stream.on('close', () => {
|
|
||||||
if (!stream.readableEnded) {
|
|
||||||
abortController.abort()
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
function abortOnDestroyedResponse(abortController, response) {
|
|
||||||
response.body.on('close', () => {
|
|
||||||
if (!response.bodyUsed) {
|
|
||||||
abortController.abort()
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
async function performRequest(url, fetchOpts) {
|
|
||||||
let response
|
|
||||||
try {
|
|
||||||
response = await fetch(url, fetchOpts)
|
|
||||||
} catch (err) {
|
|
||||||
if (fetchOpts.body instanceof Readable) {
|
|
||||||
fetchOpts.body.destroy()
|
|
||||||
}
|
|
||||||
throw OError.tag(err, err.message, {
|
|
||||||
url,
|
|
||||||
method: fetchOpts.method ?? 'GET',
|
|
||||||
})
|
|
||||||
}
|
|
||||||
if (fetchOpts.body instanceof Readable) {
|
|
||||||
response.body.on('close', () => {
|
|
||||||
if (!fetchOpts.body.readableEnded) {
|
|
||||||
fetchOpts.body.destroy()
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
return response
|
|
||||||
}
|
|
||||||
|
|
||||||
async function discardResponseBody(response) {
|
|
||||||
// eslint-disable-next-line no-unused-vars
|
|
||||||
for await (const chunk of response.body) {
|
|
||||||
// discard the body
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* @param {Response} response
|
|
||||||
*/
|
|
||||||
async function maybeGetResponseBody(response) {
|
|
||||||
try {
|
|
||||||
return await response.text()
|
|
||||||
} catch (err) {
|
|
||||||
return null
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Define custom http and https agents with support for connect timeouts
|
|
||||||
|
|
||||||
class ConnectTimeoutError extends OError {
|
|
||||||
constructor(options) {
|
|
||||||
super('connect timeout', options)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
function withTimeout(createConnection, options, callback) {
|
|
||||||
if (options.connectTimeout) {
|
|
||||||
// Wrap createConnection in a timeout
|
|
||||||
const timer = setTimeout(() => {
|
|
||||||
socket.destroy(new ConnectTimeoutError(options))
|
|
||||||
}, options.connectTimeout)
|
|
||||||
const socket = createConnection(options, (err, stream) => {
|
|
||||||
clearTimeout(timer)
|
|
||||||
callback(err, stream)
|
|
||||||
})
|
|
||||||
return socket
|
|
||||||
} else {
|
|
||||||
// Fallback to default createConnection
|
|
||||||
return createConnection(options, callback)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
class CustomHttpAgent extends http.Agent {
|
|
||||||
createConnection(options, callback) {
|
|
||||||
return withTimeout(super.createConnection.bind(this), options, callback)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
class CustomHttpsAgent extends https.Agent {
|
|
||||||
createConnection(options, callback) {
|
|
||||||
return withTimeout(super.createConnection.bind(this), options, callback)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
module.exports = {
|
|
||||||
fetchJson,
|
|
||||||
fetchJsonWithResponse,
|
|
||||||
fetchStream,
|
|
||||||
fetchStreamWithResponse,
|
|
||||||
fetchNothing,
|
|
||||||
fetchRedirect,
|
|
||||||
fetchRedirectWithResponse,
|
|
||||||
fetchString,
|
|
||||||
fetchStringWithResponse,
|
|
||||||
RequestFailedError,
|
|
||||||
ConnectTimeoutError,
|
|
||||||
CustomHttpAgent,
|
|
||||||
CustomHttpsAgent,
|
|
||||||
}
|
|
|
@ -1,33 +0,0 @@
|
||||||
{
|
|
||||||
"name": "@overleaf/fetch-utils",
|
|
||||||
"version": "0.1.0",
|
|
||||||
"description": "utilities for node-fetch",
|
|
||||||
"main": "index.js",
|
|
||||||
"scripts": {
|
|
||||||
"test": "npm run lint && npm run format && npm run types:check && npm run test:unit",
|
|
||||||
"lint": "eslint --ext .js --ext .cjs --ext .ts --max-warnings 0 --format unix .",
|
|
||||||
"lint:fix": "eslint --fix --ext .js --ext .cjs --ext .ts .",
|
|
||||||
"format": "prettier --list-different $PWD/'**/*.{js,cjs,ts}'",
|
|
||||||
"format:fix": "prettier --write $PWD/'**/*.{js,cjs,ts}'",
|
|
||||||
"test:ci": "npm run test:unit",
|
|
||||||
"test:unit": "mocha --exit test/**/*.{js,cjs}",
|
|
||||||
"types:check": "tsc --noEmit"
|
|
||||||
},
|
|
||||||
"author": "Overleaf (https://www.overleaf.com)",
|
|
||||||
"license": "AGPL-3.0-only",
|
|
||||||
"devDependencies": {
|
|
||||||
"@types/node-fetch": "^2.6.11",
|
|
||||||
"body-parser": "^1.20.3",
|
|
||||||
"chai": "^4.3.6",
|
|
||||||
"chai-as-promised": "^7.1.1",
|
|
||||||
"express": "^4.21.2",
|
|
||||||
"mocha": "^11.1.0",
|
|
||||||
"typescript": "^5.0.4"
|
|
||||||
},
|
|
||||||
"dependencies": {
|
|
||||||
"@overleaf/o-error": "*",
|
|
||||||
"lodash": "^4.17.21",
|
|
||||||
"node-fetch": "^2.7.0",
|
|
||||||
"selfsigned": "^2.4.1"
|
|
||||||
}
|
|
||||||
}
|
|
|
@ -1,4 +0,0 @@
|
||||||
const chai = require('chai')
|
|
||||||
const chaiAsPromised = require('chai-as-promised')
|
|
||||||
|
|
||||||
chai.use(chaiAsPromised)
|
|
|
@ -1,414 +0,0 @@
|
||||||
const { expect } = require('chai')
|
|
||||||
const { FetchError, AbortError } = require('node-fetch')
|
|
||||||
const { Readable } = require('node:stream')
|
|
||||||
const { once } = require('node:events')
|
|
||||||
const { TestServer } = require('./helpers/TestServer')
|
|
||||||
const selfsigned = require('selfsigned')
|
|
||||||
const {
|
|
||||||
fetchJson,
|
|
||||||
fetchStream,
|
|
||||||
fetchNothing,
|
|
||||||
fetchRedirect,
|
|
||||||
fetchString,
|
|
||||||
RequestFailedError,
|
|
||||||
CustomHttpAgent,
|
|
||||||
CustomHttpsAgent,
|
|
||||||
} = require('../..')
|
|
||||||
|
|
||||||
const HTTP_PORT = 30001
|
|
||||||
const HTTPS_PORT = 30002
|
|
||||||
|
|
||||||
const attrs = [{ name: 'commonName', value: 'example.com' }]
|
|
||||||
const pems = selfsigned.generate(attrs, { days: 365 })
|
|
||||||
|
|
||||||
const PRIVATE_KEY = pems.private
|
|
||||||
const PUBLIC_CERT = pems.cert
|
|
||||||
|
|
||||||
const dns = require('node:dns')
|
|
||||||
const _originalLookup = dns.lookup
|
|
||||||
// Custom DNS resolver function
|
|
||||||
dns.lookup = (hostname, options, callback) => {
|
|
||||||
if (hostname === 'example.com') {
|
|
||||||
// If the hostname is our test case, return the ip address for the test server
|
|
||||||
if (options?.all) {
|
|
||||||
callback(null, [{ address: '127.0.0.1', family: 4 }])
|
|
||||||
} else {
|
|
||||||
callback(null, '127.0.0.1', 4)
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
// Otherwise, use the default lookup
|
|
||||||
_originalLookup(hostname, options, callback)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
describe('fetch-utils', function () {
|
|
||||||
before(async function () {
|
|
||||||
this.server = new TestServer()
|
|
||||||
await this.server.start(HTTP_PORT, HTTPS_PORT, {
|
|
||||||
key: PRIVATE_KEY,
|
|
||||||
cert: PUBLIC_CERT,
|
|
||||||
})
|
|
||||||
this.url = path => `http://example.com:${HTTP_PORT}${path}`
|
|
||||||
this.httpsUrl = path => `https://example.com:${HTTPS_PORT}${path}`
|
|
||||||
})
|
|
||||||
|
|
||||||
beforeEach(function () {
|
|
||||||
this.server.lastReq = undefined
|
|
||||||
})
|
|
||||||
|
|
||||||
after(async function () {
|
|
||||||
await this.server.stop()
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('fetchJson', function () {
|
|
||||||
it('parses a JSON response', async function () {
|
|
||||||
const json = await fetchJson(this.url('/json/hello'))
|
|
||||||
expect(json).to.deep.equal({ msg: 'hello' })
|
|
||||||
})
|
|
||||||
|
|
||||||
it('parses JSON in the request', async function () {
|
|
||||||
const json = await fetchJson(this.url('/json/add'), {
|
|
||||||
method: 'POST',
|
|
||||||
json: { a: 2, b: 3 },
|
|
||||||
})
|
|
||||||
expect(json).to.deep.equal({ sum: 5 })
|
|
||||||
})
|
|
||||||
|
|
||||||
it('accepts stringified JSON as body', async function () {
|
|
||||||
const json = await fetchJson(this.url('/json/add'), {
|
|
||||||
method: 'POST',
|
|
||||||
body: JSON.stringify({ a: 2, b: 3 }),
|
|
||||||
headers: { 'Content-Type': 'application/json' },
|
|
||||||
})
|
|
||||||
expect(json).to.deep.equal({ sum: 5 })
|
|
||||||
})
|
|
||||||
|
|
||||||
it('throws a FetchError when the payload is not JSON', async function () {
|
|
||||||
await expect(fetchJson(this.url('/hello'))).to.be.rejectedWith(FetchError)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('aborts the request if JSON parsing fails', async function () {
|
|
||||||
await expect(fetchJson(this.url('/large'))).to.be.rejectedWith(FetchError)
|
|
||||||
await expectRequestAborted(this.server.lastReq)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('handles errors when the payload is JSON', async function () {
|
|
||||||
await expect(fetchJson(this.url('/json/500'))).to.be.rejectedWith(
|
|
||||||
RequestFailedError
|
|
||||||
)
|
|
||||||
await expectRequestAborted(this.server.lastReq)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('handles errors when the payload is not JSON', async function () {
|
|
||||||
await expect(fetchJson(this.url('/500'))).to.be.rejectedWith(
|
|
||||||
RequestFailedError
|
|
||||||
)
|
|
||||||
await expectRequestAborted(this.server.lastReq)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('supports abort signals', async function () {
|
|
||||||
await expect(
|
|
||||||
abortOnceReceived(
|
|
||||||
signal => fetchJson(this.url('/hang'), { signal }),
|
|
||||||
this.server
|
|
||||||
)
|
|
||||||
).to.be.rejectedWith(AbortError)
|
|
||||||
await expectRequestAborted(this.server.lastReq)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('supports basic auth', async function () {
|
|
||||||
const json = await fetchJson(this.url('/json/basic-auth'), {
|
|
||||||
basicAuth: { user: 'user', password: 'pass' },
|
|
||||||
})
|
|
||||||
expect(json).to.deep.equal({ key: 'verysecret' })
|
|
||||||
})
|
|
||||||
|
|
||||||
it("destroys the request body if it doesn't get consumed", async function () {
|
|
||||||
const stream = Readable.from(infiniteIterator())
|
|
||||||
await fetchJson(this.url('/json/ignore-request'), {
|
|
||||||
method: 'POST',
|
|
||||||
body: stream,
|
|
||||||
})
|
|
||||||
expect(stream.destroyed).to.be.true
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('fetchStream', function () {
|
|
||||||
it('returns a stream', async function () {
|
|
||||||
const stream = await fetchStream(this.url('/large'))
|
|
||||||
const text = await streamToString(stream)
|
|
||||||
expect(text).to.equal(this.server.largePayload)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('aborts the request when the stream is destroyed', async function () {
|
|
||||||
const stream = await fetchStream(this.url('/large'))
|
|
||||||
stream.destroy()
|
|
||||||
await expectRequestAborted(this.server.lastReq)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('aborts the request when the request body is destroyed before transfer', async function () {
|
|
||||||
const stream = Readable.from(infiniteIterator())
|
|
||||||
const promise = fetchStream(this.url('/hang'), {
|
|
||||||
method: 'POST',
|
|
||||||
body: stream,
|
|
||||||
})
|
|
||||||
stream.destroy()
|
|
||||||
await expect(promise).to.be.rejectedWith(AbortError)
|
|
||||||
await wait(80)
|
|
||||||
expect(this.server.lastReq).to.be.undefined
|
|
||||||
})
|
|
||||||
|
|
||||||
it('aborts the request when the request body is destroyed during transfer', async function () {
|
|
||||||
const stream = Readable.from(infiniteIterator())
|
|
||||||
// Note: this test won't work on `/hang`
|
|
||||||
const promise = fetchStream(this.url('/sink'), {
|
|
||||||
method: 'POST',
|
|
||||||
body: stream,
|
|
||||||
})
|
|
||||||
await once(this.server.events, 'request-received')
|
|
||||||
stream.destroy()
|
|
||||||
await expect(promise).to.be.rejectedWith(AbortError)
|
|
||||||
await expectRequestAborted(this.server.lastReq)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('handles errors', async function () {
|
|
||||||
await expect(fetchStream(this.url('/500'))).to.be.rejectedWith(
|
|
||||||
RequestFailedError
|
|
||||||
)
|
|
||||||
await expectRequestAborted(this.server.lastReq)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('supports abort signals', async function () {
|
|
||||||
await expect(
|
|
||||||
abortOnceReceived(
|
|
||||||
signal => fetchStream(this.url('/hang'), { signal }),
|
|
||||||
this.server
|
|
||||||
)
|
|
||||||
).to.be.rejectedWith(AbortError)
|
|
||||||
await expectRequestAborted(this.server.lastReq)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('destroys the request body when an error occurs', async function () {
|
|
||||||
const stream = Readable.from(infiniteIterator())
|
|
||||||
await expect(
|
|
||||||
abortOnceReceived(
|
|
||||||
signal =>
|
|
||||||
fetchStream(this.url('/hang'), {
|
|
||||||
method: 'POST',
|
|
||||||
body: stream,
|
|
||||||
signal,
|
|
||||||
}),
|
|
||||||
this.server
|
|
||||||
)
|
|
||||||
).to.be.rejectedWith(AbortError)
|
|
||||||
expect(stream.destroyed).to.be.true
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('fetchNothing', function () {
|
|
||||||
it('closes the connection', async function () {
|
|
||||||
await fetchNothing(this.url('/large'))
|
|
||||||
await expectRequestAborted(this.server.lastReq)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('aborts the request when the request body is destroyed before transfer', async function () {
|
|
||||||
const stream = Readable.from(infiniteIterator())
|
|
||||||
const promise = fetchNothing(this.url('/hang'), {
|
|
||||||
method: 'POST',
|
|
||||||
body: stream,
|
|
||||||
})
|
|
||||||
stream.destroy()
|
|
||||||
await expect(promise).to.be.rejectedWith(AbortError)
|
|
||||||
expect(this.server.lastReq).to.be.undefined
|
|
||||||
})
|
|
||||||
|
|
||||||
it('aborts the request when the request body is destroyed during transfer', async function () {
|
|
||||||
const stream = Readable.from(infiniteIterator())
|
|
||||||
// Note: this test won't work on `/hang`
|
|
||||||
const promise = fetchNothing(this.url('/sink'), {
|
|
||||||
method: 'POST',
|
|
||||||
body: stream,
|
|
||||||
})
|
|
||||||
await once(this.server.events, 'request-received')
|
|
||||||
stream.destroy()
|
|
||||||
await expect(promise).to.be.rejectedWith(AbortError)
|
|
||||||
await wait(80)
|
|
||||||
await expectRequestAborted(this.server.lastReq)
|
|
||||||
})
|
|
||||||
|
|
||||||
it("doesn't abort the request if the request body ends normally", async function () {
|
|
||||||
const stream = Readable.from('hello there')
|
|
||||||
await fetchNothing(this.url('/sink'), { method: 'POST', body: stream })
|
|
||||||
})
|
|
||||||
|
|
||||||
it('handles errors', async function () {
|
|
||||||
await expect(fetchNothing(this.url('/500'))).to.be.rejectedWith(
|
|
||||||
RequestFailedError
|
|
||||||
)
|
|
||||||
await expectRequestAborted(this.server.lastReq)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('supports abort signals', async function () {
|
|
||||||
await expect(
|
|
||||||
abortOnceReceived(
|
|
||||||
signal => fetchNothing(this.url('/hang'), { signal }),
|
|
||||||
this.server
|
|
||||||
)
|
|
||||||
).to.be.rejectedWith(AbortError)
|
|
||||||
await expectRequestAborted(this.server.lastReq)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('destroys the request body when an error occurs', async function () {
|
|
||||||
const stream = Readable.from(infiniteIterator())
|
|
||||||
await expect(
|
|
||||||
abortOnceReceived(
|
|
||||||
signal =>
|
|
||||||
fetchNothing(this.url('/hang'), {
|
|
||||||
method: 'POST',
|
|
||||||
body: stream,
|
|
||||||
signal,
|
|
||||||
}),
|
|
||||||
this.server
|
|
||||||
)
|
|
||||||
).to.be.rejectedWith(AbortError)
|
|
||||||
expect(stream.destroyed).to.be.true
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('fetchString', function () {
|
|
||||||
it('returns a string', async function () {
|
|
||||||
const body = await fetchString(this.url('/hello'))
|
|
||||||
expect(body).to.equal('hello')
|
|
||||||
})
|
|
||||||
|
|
||||||
it('handles errors', async function () {
|
|
||||||
await expect(fetchString(this.url('/500'))).to.be.rejectedWith(
|
|
||||||
RequestFailedError
|
|
||||||
)
|
|
||||||
await expectRequestAborted(this.server.lastReq)
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('fetchRedirect', function () {
|
|
||||||
it('returns the immediate redirect', async function () {
|
|
||||||
const body = await fetchRedirect(this.url('/redirect/1'))
|
|
||||||
expect(body).to.equal(this.url('/redirect/2'))
|
|
||||||
})
|
|
||||||
|
|
||||||
it('rejects status 200', async function () {
|
|
||||||
await expect(fetchRedirect(this.url('/hello'))).to.be.rejectedWith(
|
|
||||||
RequestFailedError
|
|
||||||
)
|
|
||||||
await expectRequestAborted(this.server.lastReq)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('rejects empty redirect', async function () {
|
|
||||||
await expect(fetchRedirect(this.url('/redirect/empty-location')))
|
|
||||||
.to.be.rejectedWith(RequestFailedError)
|
|
||||||
.and.eventually.have.property('cause')
|
|
||||||
.and.to.have.property('message')
|
|
||||||
.to.equal('missing Location response header on 3xx response')
|
|
||||||
await expectRequestAborted(this.server.lastReq)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('handles errors', async function () {
|
|
||||||
await expect(fetchRedirect(this.url('/500'))).to.be.rejectedWith(
|
|
||||||
RequestFailedError
|
|
||||||
)
|
|
||||||
await expectRequestAborted(this.server.lastReq)
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('CustomHttpAgent', function () {
|
|
||||||
it('makes an http request successfully', async function () {
|
|
||||||
const agent = new CustomHttpAgent({ connectTimeout: 100 })
|
|
||||||
const body = await fetchString(this.url('/hello'), { agent })
|
|
||||||
expect(body).to.equal('hello')
|
|
||||||
})
|
|
||||||
|
|
||||||
it('times out when accessing a non-routable address', async function () {
|
|
||||||
const agent = new CustomHttpAgent({ connectTimeout: 10 })
|
|
||||||
await expect(fetchString('http://10.255.255.255/', { agent }))
|
|
||||||
.to.be.rejectedWith(FetchError)
|
|
||||||
.and.eventually.have.property('message')
|
|
||||||
.and.to.equal(
|
|
||||||
'request to http://10.255.255.255/ failed, reason: connect timeout'
|
|
||||||
)
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('CustomHttpsAgent', function () {
|
|
||||||
it('makes an https request successfully', async function () {
|
|
||||||
const agent = new CustomHttpsAgent({
|
|
||||||
connectTimeout: 100,
|
|
||||||
ca: PUBLIC_CERT,
|
|
||||||
})
|
|
||||||
const body = await fetchString(this.httpsUrl('/hello'), { agent })
|
|
||||||
expect(body).to.equal('hello')
|
|
||||||
})
|
|
||||||
|
|
||||||
it('rejects an untrusted server', async function () {
|
|
||||||
const agent = new CustomHttpsAgent({
|
|
||||||
connectTimeout: 100,
|
|
||||||
})
|
|
||||||
await expect(fetchString(this.httpsUrl('/hello'), { agent }))
|
|
||||||
.to.be.rejectedWith(FetchError)
|
|
||||||
.and.eventually.have.property('code')
|
|
||||||
.and.to.equal('DEPTH_ZERO_SELF_SIGNED_CERT')
|
|
||||||
})
|
|
||||||
|
|
||||||
it('times out when accessing a non-routable address', async function () {
|
|
||||||
const agent = new CustomHttpsAgent({ connectTimeout: 10 })
|
|
||||||
await expect(fetchString('https://10.255.255.255/', { agent }))
|
|
||||||
.to.be.rejectedWith(FetchError)
|
|
||||||
.and.eventually.have.property('message')
|
|
||||||
.and.to.equal(
|
|
||||||
'request to https://10.255.255.255/ failed, reason: connect timeout'
|
|
||||||
)
|
|
||||||
})
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
async function streamToString(stream) {
|
|
||||||
let s = ''
|
|
||||||
for await (const chunk of stream) {
|
|
||||||
s += chunk
|
|
||||||
}
|
|
||||||
return s
|
|
||||||
}
|
|
||||||
|
|
||||||
async function* infiniteIterator() {
|
|
||||||
let i = 1
|
|
||||||
while (true) {
|
|
||||||
yield `chunk ${i++}\n`
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* @param {(signal: AbortSignal) => Promise<any>} func
|
|
||||||
* @param {TestServer} server
|
|
||||||
*/
|
|
||||||
async function abortOnceReceived(func, server) {
|
|
||||||
const controller = new AbortController()
|
|
||||||
const promise = func(controller.signal)
|
|
||||||
await once(server.events, 'request-received')
|
|
||||||
controller.abort()
|
|
||||||
return await promise
|
|
||||||
}
|
|
||||||
|
|
||||||
async function expectRequestAborted(req) {
|
|
||||||
if (!req.destroyed) {
|
|
||||||
try {
|
|
||||||
await once(req, 'close')
|
|
||||||
} catch (err) {
|
|
||||||
// `once` throws if req emits an 'error' event.
|
|
||||||
// We ignore `Error: aborted` when the request is aborted.
|
|
||||||
if (err.message !== 'aborted') {
|
|
||||||
throw err
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
expect(req.destroyed).to.be.true
|
|
||||||
}
|
|
||||||
|
|
||||||
const wait = ms => new Promise(resolve => setTimeout(resolve, ms))
|
|
|
@ -1,130 +0,0 @@
|
||||||
const express = require('express')
|
|
||||||
const bodyParser = require('body-parser')
|
|
||||||
const { EventEmitter } = require('node:events')
|
|
||||||
const http = require('node:http')
|
|
||||||
const https = require('node:https')
|
|
||||||
const { promisify } = require('node:util')
|
|
||||||
|
|
||||||
class TestServer {
|
|
||||||
constructor() {
|
|
||||||
this.app = express()
|
|
||||||
this.events = new EventEmitter()
|
|
||||||
|
|
||||||
this.app.use(bodyParser.json())
|
|
||||||
this.app.use((req, res, next) => {
|
|
||||||
this.events.emit('request-received')
|
|
||||||
this.lastReq = req
|
|
||||||
next()
|
|
||||||
})
|
|
||||||
|
|
||||||
// Plain text endpoints
|
|
||||||
|
|
||||||
this.app.get('/hello', (req, res) => {
|
|
||||||
res.send('hello')
|
|
||||||
})
|
|
||||||
|
|
||||||
this.largePayload = 'x'.repeat(16 * 1024 * 1024)
|
|
||||||
this.app.get('/large', (req, res) => {
|
|
||||||
res.send(this.largePayload)
|
|
||||||
})
|
|
||||||
|
|
||||||
this.app.get('/204', (req, res) => {
|
|
||||||
res.status(204).end()
|
|
||||||
})
|
|
||||||
|
|
||||||
this.app.get('/empty', (req, res) => {
|
|
||||||
res.end()
|
|
||||||
})
|
|
||||||
|
|
||||||
this.app.get('/500', (req, res) => {
|
|
||||||
res.sendStatus(500)
|
|
||||||
})
|
|
||||||
|
|
||||||
this.app.post('/sink', (req, res) => {
|
|
||||||
req.on('data', () => {})
|
|
||||||
req.on('end', () => {
|
|
||||||
res.status(204).end()
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
// JSON endpoints
|
|
||||||
|
|
||||||
this.app.get('/json/hello', (req, res) => {
|
|
||||||
res.json({ msg: 'hello' })
|
|
||||||
})
|
|
||||||
|
|
||||||
this.app.post('/json/add', (req, res) => {
|
|
||||||
const { a, b } = req.body
|
|
||||||
res.json({ sum: a + b })
|
|
||||||
})
|
|
||||||
|
|
||||||
this.app.get('/json/500', (req, res) => {
|
|
||||||
res.status(500).json({ error: 'Internal server error' })
|
|
||||||
})
|
|
||||||
|
|
||||||
this.app.get('/json/basic-auth', (req, res) => {
|
|
||||||
const expectedAuth =
|
|
||||||
'Basic ' + Buffer.from('user:pass').toString('base64')
|
|
||||||
if (req.headers.authorization === expectedAuth) {
|
|
||||||
res.json({ key: 'verysecret' })
|
|
||||||
} else {
|
|
||||||
res.status(401).json({ error: 'unauthorized' })
|
|
||||||
}
|
|
||||||
})
|
|
||||||
|
|
||||||
this.app.post('/json/ignore-request', (req, res) => {
|
|
||||||
res.json({ msg: 'hello' })
|
|
||||||
})
|
|
||||||
|
|
||||||
// Never returns
|
|
||||||
|
|
||||||
this.app.get('/hang', (req, res) => {})
|
|
||||||
this.app.post('/hang', (req, res) => {})
|
|
||||||
|
|
||||||
// Redirect
|
|
||||||
|
|
||||||
this.app.get('/redirect/1', (req, res) => {
|
|
||||||
res.redirect('/redirect/2')
|
|
||||||
})
|
|
||||||
this.app.get('/redirect/2', (req, res) => {
|
|
||||||
res.send('body after redirect')
|
|
||||||
})
|
|
||||||
this.app.get('/redirect/empty-location', (req, res) => {
|
|
||||||
res.sendStatus(302)
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
start(port, httpsPort, httpsOptions) {
|
|
||||||
const startHttp = new Promise((resolve, reject) => {
|
|
||||||
this.server = http.createServer(this.app).listen(port, err => {
|
|
||||||
if (err) {
|
|
||||||
reject(err)
|
|
||||||
} else {
|
|
||||||
resolve()
|
|
||||||
}
|
|
||||||
})
|
|
||||||
})
|
|
||||||
const startHttps = new Promise((resolve, reject) => {
|
|
||||||
this.https_server = https
|
|
||||||
.createServer(httpsOptions, this.app)
|
|
||||||
.listen(httpsPort, err => {
|
|
||||||
if (err) {
|
|
||||||
reject(err)
|
|
||||||
} else {
|
|
||||||
resolve()
|
|
||||||
}
|
|
||||||
})
|
|
||||||
})
|
|
||||||
return Promise.all([startHttp, startHttps])
|
|
||||||
}
|
|
||||||
|
|
||||||
stop() {
|
|
||||||
const stopHttp = promisify(this.server.close).bind(this.server)
|
|
||||||
const stopHttps = promisify(this.https_server.close).bind(this.https_server)
|
|
||||||
this.server.closeAllConnections()
|
|
||||||
this.https_server.closeAllConnections()
|
|
||||||
return Promise.all([stopHttp(), stopHttps()])
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
module.exports = { TestServer }
|
|
|
@ -1,7 +0,0 @@
|
||||||
{
|
|
||||||
"extends": "../../tsconfig.backend.json",
|
|
||||||
"include": [
|
|
||||||
"**/*.js",
|
|
||||||
"**/*.cjs"
|
|
||||||
]
|
|
||||||
}
|
|
3
libraries/logger/.gitignore
vendored
3
libraries/logger/.gitignore
vendored
|
@ -1,3 +0,0 @@
|
||||||
node_modules
|
|
||||||
|
|
||||||
.npmrc
|
|
|
@ -1,3 +0,0 @@
|
||||||
{
|
|
||||||
"require": "test/setup.js"
|
|
||||||
}
|
|
|
@ -1 +0,0 @@
|
||||||
20.18.2
|
|
|
@ -1,10 +0,0 @@
|
||||||
## v3.1.1
|
|
||||||
|
|
||||||
* Handle malformed requests in the req serializer
|
|
||||||
|
|
||||||
## v3.0.0
|
|
||||||
|
|
||||||
* Improve logging in Google Cloud Platform. Set environment variable `GCP_LOGGING=true` to enable.
|
|
||||||
|
|
||||||
This version of the metrics module only works with versions of the `@overleaf/metrics` module greater than v4.0.0
|
|
||||||
|
|
|
@ -1,661 +0,0 @@
|
||||||
GNU AFFERO GENERAL PUBLIC LICENSE
|
|
||||||
Version 3, 19 November 2007
|
|
||||||
|
|
||||||
Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
|
|
||||||
Everyone is permitted to copy and distribute verbatim copies
|
|
||||||
of this license document, but changing it is not allowed.
|
|
||||||
|
|
||||||
Preamble
|
|
||||||
|
|
||||||
The GNU Affero General Public License is a free, copyleft license for
|
|
||||||
software and other kinds of works, specifically designed to ensure
|
|
||||||
cooperation with the community in the case of network server software.
|
|
||||||
|
|
||||||
The licenses for most software and other practical works are designed
|
|
||||||
to take away your freedom to share and change the works. By contrast,
|
|
||||||
our General Public Licenses are intended to guarantee your freedom to
|
|
||||||
share and change all versions of a program--to make sure it remains free
|
|
||||||
software for all its users.
|
|
||||||
|
|
||||||
When we speak of free software, we are referring to freedom, not
|
|
||||||
price. Our General Public Licenses are designed to make sure that you
|
|
||||||
have the freedom to distribute copies of free software (and charge for
|
|
||||||
them if you wish), that you receive source code or can get it if you
|
|
||||||
want it, that you can change the software or use pieces of it in new
|
|
||||||
free programs, and that you know you can do these things.
|
|
||||||
|
|
||||||
Developers that use our General Public Licenses protect your rights
|
|
||||||
with two steps: (1) assert copyright on the software, and (2) offer
|
|
||||||
you this License which gives you legal permission to copy, distribute
|
|
||||||
and/or modify the software.
|
|
||||||
|
|
||||||
A secondary benefit of defending all users' freedom is that
|
|
||||||
improvements made in alternate versions of the program, if they
|
|
||||||
receive widespread use, become available for other developers to
|
|
||||||
incorporate. Many developers of free software are heartened and
|
|
||||||
encouraged by the resulting cooperation. However, in the case of
|
|
||||||
software used on network servers, this result may fail to come about.
|
|
||||||
The GNU General Public License permits making a modified version and
|
|
||||||
letting the public access it on a server without ever releasing its
|
|
||||||
source code to the public.
|
|
||||||
|
|
||||||
The GNU Affero General Public License is designed specifically to
|
|
||||||
ensure that, in such cases, the modified source code becomes available
|
|
||||||
to the community. It requires the operator of a network server to
|
|
||||||
provide the source code of the modified version running there to the
|
|
||||||
users of that server. Therefore, public use of a modified version, on
|
|
||||||
a publicly accessible server, gives the public access to the source
|
|
||||||
code of the modified version.
|
|
||||||
|
|
||||||
An older license, called the Affero General Public License and
|
|
||||||
published by Affero, was designed to accomplish similar goals. This is
|
|
||||||
a different license, not a version of the Affero GPL, but Affero has
|
|
||||||
released a new version of the Affero GPL which permits relicensing under
|
|
||||||
this license.
|
|
||||||
|
|
||||||
The precise terms and conditions for copying, distribution and
|
|
||||||
modification follow.
|
|
||||||
|
|
||||||
TERMS AND CONDITIONS
|
|
||||||
|
|
||||||
0. Definitions.
|
|
||||||
|
|
||||||
"This License" refers to version 3 of the GNU Affero General Public License.
|
|
||||||
|
|
||||||
"Copyright" also means copyright-like laws that apply to other kinds of
|
|
||||||
works, such as semiconductor masks.
|
|
||||||
|
|
||||||
"The Program" refers to any copyrightable work licensed under this
|
|
||||||
License. Each licensee is addressed as "you". "Licensees" and
|
|
||||||
"recipients" may be individuals or organizations.
|
|
||||||
|
|
||||||
To "modify" a work means to copy from or adapt all or part of the work
|
|
||||||
in a fashion requiring copyright permission, other than the making of an
|
|
||||||
exact copy. The resulting work is called a "modified version" of the
|
|
||||||
earlier work or a work "based on" the earlier work.
|
|
||||||
|
|
||||||
A "covered work" means either the unmodified Program or a work based
|
|
||||||
on the Program.
|
|
||||||
|
|
||||||
To "propagate" a work means to do anything with it that, without
|
|
||||||
permission, would make you directly or secondarily liable for
|
|
||||||
infringement under applicable copyright law, except executing it on a
|
|
||||||
computer or modifying a private copy. Propagation includes copying,
|
|
||||||
distribution (with or without modification), making available to the
|
|
||||||
public, and in some countries other activities as well.
|
|
||||||
|
|
||||||
To "convey" a work means any kind of propagation that enables other
|
|
||||||
parties to make or receive copies. Mere interaction with a user through
|
|
||||||
a computer network, with no transfer of a copy, is not conveying.
|
|
||||||
|
|
||||||
An interactive user interface displays "Appropriate Legal Notices"
|
|
||||||
to the extent that it includes a convenient and prominently visible
|
|
||||||
feature that (1) displays an appropriate copyright notice, and (2)
|
|
||||||
tells the user that there is no warranty for the work (except to the
|
|
||||||
extent that warranties are provided), that licensees may convey the
|
|
||||||
work under this License, and how to view a copy of this License. If
|
|
||||||
the interface presents a list of user commands or options, such as a
|
|
||||||
menu, a prominent item in the list meets this criterion.
|
|
||||||
|
|
||||||
1. Source Code.
|
|
||||||
|
|
||||||
The "source code" for a work means the preferred form of the work
|
|
||||||
for making modifications to it. "Object code" means any non-source
|
|
||||||
form of a work.
|
|
||||||
|
|
||||||
A "Standard Interface" means an interface that either is an official
|
|
||||||
standard defined by a recognized standards body, or, in the case of
|
|
||||||
interfaces specified for a particular programming language, one that
|
|
||||||
is widely used among developers working in that language.
|
|
||||||
|
|
||||||
The "System Libraries" of an executable work include anything, other
|
|
||||||
than the work as a whole, that (a) is included in the normal form of
|
|
||||||
packaging a Major Component, but which is not part of that Major
|
|
||||||
Component, and (b) serves only to enable use of the work with that
|
|
||||||
Major Component, or to implement a Standard Interface for which an
|
|
||||||
implementation is available to the public in source code form. A
|
|
||||||
"Major Component", in this context, means a major essential component
|
|
||||||
(kernel, window system, and so on) of the specific operating system
|
|
||||||
(if any) on which the executable work runs, or a compiler used to
|
|
||||||
produce the work, or an object code interpreter used to run it.
|
|
||||||
|
|
||||||
The "Corresponding Source" for a work in object code form means all
|
|
||||||
the source code needed to generate, install, and (for an executable
|
|
||||||
work) run the object code and to modify the work, including scripts to
|
|
||||||
control those activities. However, it does not include the work's
|
|
||||||
System Libraries, or general-purpose tools or generally available free
|
|
||||||
programs which are used unmodified in performing those activities but
|
|
||||||
which are not part of the work. For example, Corresponding Source
|
|
||||||
includes interface definition files associated with source files for
|
|
||||||
the work, and the source code for shared libraries and dynamically
|
|
||||||
linked subprograms that the work is specifically designed to require,
|
|
||||||
such as by intimate data communication or control flow between those
|
|
||||||
subprograms and other parts of the work.
|
|
||||||
|
|
||||||
The Corresponding Source need not include anything that users
|
|
||||||
can regenerate automatically from other parts of the Corresponding
|
|
||||||
Source.
|
|
||||||
|
|
||||||
The Corresponding Source for a work in source code form is that
|
|
||||||
same work.
|
|
||||||
|
|
||||||
2. Basic Permissions.
|
|
||||||
|
|
||||||
All rights granted under this License are granted for the term of
|
|
||||||
copyright on the Program, and are irrevocable provided the stated
|
|
||||||
conditions are met. This License explicitly affirms your unlimited
|
|
||||||
permission to run the unmodified Program. The output from running a
|
|
||||||
covered work is covered by this License only if the output, given its
|
|
||||||
content, constitutes a covered work. This License acknowledges your
|
|
||||||
rights of fair use or other equivalent, as provided by copyright law.
|
|
||||||
|
|
||||||
You may make, run and propagate covered works that you do not
|
|
||||||
convey, without conditions so long as your license otherwise remains
|
|
||||||
in force. You may convey covered works to others for the sole purpose
|
|
||||||
of having them make modifications exclusively for you, or provide you
|
|
||||||
with facilities for running those works, provided that you comply with
|
|
||||||
the terms of this License in conveying all material for which you do
|
|
||||||
not control copyright. Those thus making or running the covered works
|
|
||||||
for you must do so exclusively on your behalf, under your direction
|
|
||||||
and control, on terms that prohibit them from making any copies of
|
|
||||||
your copyrighted material outside their relationship with you.
|
|
||||||
|
|
||||||
Conveying under any other circumstances is permitted solely under
|
|
||||||
the conditions stated below. Sublicensing is not allowed; section 10
|
|
||||||
makes it unnecessary.
|
|
||||||
|
|
||||||
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
|
|
||||||
|
|
||||||
No covered work shall be deemed part of an effective technological
|
|
||||||
measure under any applicable law fulfilling obligations under article
|
|
||||||
11 of the WIPO copyright treaty adopted on 20 December 1996, or
|
|
||||||
similar laws prohibiting or restricting circumvention of such
|
|
||||||
measures.
|
|
||||||
|
|
||||||
When you convey a covered work, you waive any legal power to forbid
|
|
||||||
circumvention of technological measures to the extent such circumvention
|
|
||||||
is effected by exercising rights under this License with respect to
|
|
||||||
the covered work, and you disclaim any intention to limit operation or
|
|
||||||
modification of the work as a means of enforcing, against the work's
|
|
||||||
users, your or third parties' legal rights to forbid circumvention of
|
|
||||||
technological measures.
|
|
||||||
|
|
||||||
4. Conveying Verbatim Copies.
|
|
||||||
|
|
||||||
You may convey verbatim copies of the Program's source code as you
|
|
||||||
receive it, in any medium, provided that you conspicuously and
|
|
||||||
appropriately publish on each copy an appropriate copyright notice;
|
|
||||||
keep intact all notices stating that this License and any
|
|
||||||
non-permissive terms added in accord with section 7 apply to the code;
|
|
||||||
keep intact all notices of the absence of any warranty; and give all
|
|
||||||
recipients a copy of this License along with the Program.
|
|
||||||
|
|
||||||
You may charge any price or no price for each copy that you convey,
|
|
||||||
and you may offer support or warranty protection for a fee.
|
|
||||||
|
|
||||||
5. Conveying Modified Source Versions.
|
|
||||||
|
|
||||||
You may convey a work based on the Program, or the modifications to
|
|
||||||
produce it from the Program, in the form of source code under the
|
|
||||||
terms of section 4, provided that you also meet all of these conditions:
|
|
||||||
|
|
||||||
a) The work must carry prominent notices stating that you modified
|
|
||||||
it, and giving a relevant date.
|
|
||||||
|
|
||||||
b) The work must carry prominent notices stating that it is
|
|
||||||
released under this License and any conditions added under section
|
|
||||||
7. This requirement modifies the requirement in section 4 to
|
|
||||||
"keep intact all notices".
|
|
||||||
|
|
||||||
c) You must license the entire work, as a whole, under this
|
|
||||||
License to anyone who comes into possession of a copy. This
|
|
||||||
License will therefore apply, along with any applicable section 7
|
|
||||||
additional terms, to the whole of the work, and all its parts,
|
|
||||||
regardless of how they are packaged. This License gives no
|
|
||||||
permission to license the work in any other way, but it does not
|
|
||||||
invalidate such permission if you have separately received it.
|
|
||||||
|
|
||||||
d) If the work has interactive user interfaces, each must display
|
|
||||||
Appropriate Legal Notices; however, if the Program has interactive
|
|
||||||
interfaces that do not display Appropriate Legal Notices, your
|
|
||||||
work need not make them do so.
|
|
||||||
|
|
||||||
A compilation of a covered work with other separate and independent
|
|
||||||
works, which are not by their nature extensions of the covered work,
|
|
||||||
and which are not combined with it such as to form a larger program,
|
|
||||||
in or on a volume of a storage or distribution medium, is called an
|
|
||||||
"aggregate" if the compilation and its resulting copyright are not
|
|
||||||
used to limit the access or legal rights of the compilation's users
|
|
||||||
beyond what the individual works permit. Inclusion of a covered work
|
|
||||||
in an aggregate does not cause this License to apply to the other
|
|
||||||
parts of the aggregate.
|
|
||||||
|
|
||||||
6. Conveying Non-Source Forms.
|
|
||||||
|
|
||||||
You may convey a covered work in object code form under the terms
|
|
||||||
of sections 4 and 5, provided that you also convey the
|
|
||||||
machine-readable Corresponding Source under the terms of this License,
|
|
||||||
in one of these ways:
|
|
||||||
|
|
||||||
a) Convey the object code in, or embodied in, a physical product
|
|
||||||
(including a physical distribution medium), accompanied by the
|
|
||||||
Corresponding Source fixed on a durable physical medium
|
|
||||||
customarily used for software interchange.
|
|
||||||
|
|
||||||
b) Convey the object code in, or embodied in, a physical product
|
|
||||||
(including a physical distribution medium), accompanied by a
|
|
||||||
written offer, valid for at least three years and valid for as
|
|
||||||
long as you offer spare parts or customer support for that product
|
|
||||||
model, to give anyone who possesses the object code either (1) a
|
|
||||||
copy of the Corresponding Source for all the software in the
|
|
||||||
product that is covered by this License, on a durable physical
|
|
||||||
medium customarily used for software interchange, for a price no
|
|
||||||
more than your reasonable cost of physically performing this
|
|
||||||
conveying of source, or (2) access to copy the
|
|
||||||
Corresponding Source from a network server at no charge.
|
|
||||||
|
|
||||||
c) Convey individual copies of the object code with a copy of the
|
|
||||||
written offer to provide the Corresponding Source. This
|
|
||||||
alternative is allowed only occasionally and noncommercially, and
|
|
||||||
only if you received the object code with such an offer, in accord
|
|
||||||
with subsection 6b.
|
|
||||||
|
|
||||||
d) Convey the object code by offering access from a designated
|
|
||||||
place (gratis or for a charge), and offer equivalent access to the
|
|
||||||
Corresponding Source in the same way through the same place at no
|
|
||||||
further charge. You need not require recipients to copy the
|
|
||||||
Corresponding Source along with the object code. If the place to
|
|
||||||
copy the object code is a network server, the Corresponding Source
|
|
||||||
may be on a different server (operated by you or a third party)
|
|
||||||
that supports equivalent copying facilities, provided you maintain
|
|
||||||
clear directions next to the object code saying where to find the
|
|
||||||
Corresponding Source. Regardless of what server hosts the
|
|
||||||
Corresponding Source, you remain obligated to ensure that it is
|
|
||||||
available for as long as needed to satisfy these requirements.
|
|
||||||
|
|
||||||
e) Convey the object code using peer-to-peer transmission, provided
|
|
||||||
you inform other peers where the object code and Corresponding
|
|
||||||
Source of the work are being offered to the general public at no
|
|
||||||
charge under subsection 6d.
|
|
||||||
|
|
||||||
A separable portion of the object code, whose source code is excluded
|
|
||||||
from the Corresponding Source as a System Library, need not be
|
|
||||||
included in conveying the object code work.
|
|
||||||
|
|
||||||
A "User Product" is either (1) a "consumer product", which means any
|
|
||||||
tangible personal property which is normally used for personal, family,
|
|
||||||
or household purposes, or (2) anything designed or sold for incorporation
|
|
||||||
into a dwelling. In determining whether a product is a consumer product,
|
|
||||||
doubtful cases shall be resolved in favor of coverage. For a particular
|
|
||||||
product received by a particular user, "normally used" refers to a
|
|
||||||
typical or common use of that class of product, regardless of the status
|
|
||||||
of the particular user or of the way in which the particular user
|
|
||||||
actually uses, or expects or is expected to use, the product. A product
|
|
||||||
is a consumer product regardless of whether the product has substantial
|
|
||||||
commercial, industrial or non-consumer uses, unless such uses represent
|
|
||||||
the only significant mode of use of the product.
|
|
||||||
|
|
||||||
"Installation Information" for a User Product means any methods,
|
|
||||||
procedures, authorization keys, or other information required to install
|
|
||||||
and execute modified versions of a covered work in that User Product from
|
|
||||||
a modified version of its Corresponding Source. The information must
|
|
||||||
suffice to ensure that the continued functioning of the modified object
|
|
||||||
code is in no case prevented or interfered with solely because
|
|
||||||
modification has been made.
|
|
||||||
|
|
||||||
If you convey an object code work under this section in, or with, or
|
|
||||||
specifically for use in, a User Product, and the conveying occurs as
|
|
||||||
part of a transaction in which the right of possession and use of the
|
|
||||||
User Product is transferred to the recipient in perpetuity or for a
|
|
||||||
fixed term (regardless of how the transaction is characterized), the
|
|
||||||
Corresponding Source conveyed under this section must be accompanied
|
|
||||||
by the Installation Information. But this requirement does not apply
|
|
||||||
if neither you nor any third party retains the ability to install
|
|
||||||
modified object code on the User Product (for example, the work has
|
|
||||||
been installed in ROM).
|
|
||||||
|
|
||||||
The requirement to provide Installation Information does not include a
|
|
||||||
requirement to continue to provide support service, warranty, or updates
|
|
||||||
for a work that has been modified or installed by the recipient, or for
|
|
||||||
the User Product in which it has been modified or installed. Access to a
|
|
||||||
network may be denied when the modification itself materially and
|
|
||||||
adversely affects the operation of the network or violates the rules and
|
|
||||||
protocols for communication across the network.
|
|
||||||
|
|
||||||
Corresponding Source conveyed, and Installation Information provided,
|
|
||||||
in accord with this section must be in a format that is publicly
|
|
||||||
documented (and with an implementation available to the public in
|
|
||||||
source code form), and must require no special password or key for
|
|
||||||
unpacking, reading or copying.
|
|
||||||
|
|
||||||
7. Additional Terms.
|
|
||||||
|
|
||||||
"Additional permissions" are terms that supplement the terms of this
|
|
||||||
License by making exceptions from one or more of its conditions.
|
|
||||||
Additional permissions that are applicable to the entire Program shall
|
|
||||||
be treated as though they were included in this License, to the extent
|
|
||||||
that they are valid under applicable law. If additional permissions
|
|
||||||
apply only to part of the Program, that part may be used separately
|
|
||||||
under those permissions, but the entire Program remains governed by
|
|
||||||
this License without regard to the additional permissions.
|
|
||||||
|
|
||||||
When you convey a copy of a covered work, you may at your option
|
|
||||||
remove any additional permissions from that copy, or from any part of
|
|
||||||
it. (Additional permissions may be written to require their own
|
|
||||||
removal in certain cases when you modify the work.) You may place
|
|
||||||
additional permissions on material, added by you to a covered work,
|
|
||||||
for which you have or can give appropriate copyright permission.
|
|
||||||
|
|
||||||
Notwithstanding any other provision of this License, for material you
|
|
||||||
add to a covered work, you may (if authorized by the copyright holders of
|
|
||||||
that material) supplement the terms of this License with terms:
|
|
||||||
|
|
||||||
a) Disclaiming warranty or limiting liability differently from the
|
|
||||||
terms of sections 15 and 16 of this License; or
|
|
||||||
|
|
||||||
b) Requiring preservation of specified reasonable legal notices or
|
|
||||||
author attributions in that material or in the Appropriate Legal
|
|
||||||
Notices displayed by works containing it; or
|
|
||||||
|
|
||||||
c) Prohibiting misrepresentation of the origin of that material, or
|
|
||||||
requiring that modified versions of such material be marked in
|
|
||||||
reasonable ways as different from the original version; or
|
|
||||||
|
|
||||||
d) Limiting the use for publicity purposes of names of licensors or
|
|
||||||
authors of the material; or
|
|
||||||
|
|
||||||
e) Declining to grant rights under trademark law for use of some
|
|
||||||
trade names, trademarks, or service marks; or
|
|
||||||
|
|
||||||
f) Requiring indemnification of licensors and authors of that
|
|
||||||
material by anyone who conveys the material (or modified versions of
|
|
||||||
it) with contractual assumptions of liability to the recipient, for
|
|
||||||
any liability that these contractual assumptions directly impose on
|
|
||||||
those licensors and authors.
|
|
||||||
|
|
||||||
All other non-permissive additional terms are considered "further
|
|
||||||
restrictions" within the meaning of section 10. If the Program as you
|
|
||||||
received it, or any part of it, contains a notice stating that it is
|
|
||||||
governed by this License along with a term that is a further
|
|
||||||
restriction, you may remove that term. If a license document contains
|
|
||||||
a further restriction but permits relicensing or conveying under this
|
|
||||||
License, you may add to a covered work material governed by the terms
|
|
||||||
of that license document, provided that the further restriction does
|
|
||||||
not survive such relicensing or conveying.
|
|
||||||
|
|
||||||
If you add terms to a covered work in accord with this section, you
|
|
||||||
must place, in the relevant source files, a statement of the
|
|
||||||
additional terms that apply to those files, or a notice indicating
|
|
||||||
where to find the applicable terms.
|
|
||||||
|
|
||||||
Additional terms, permissive or non-permissive, may be stated in the
|
|
||||||
form of a separately written license, or stated as exceptions;
|
|
||||||
the above requirements apply either way.
|
|
||||||
|
|
||||||
8. Termination.
|
|
||||||
|
|
||||||
You may not propagate or modify a covered work except as expressly
|
|
||||||
provided under this License. Any attempt otherwise to propagate or
|
|
||||||
modify it is void, and will automatically terminate your rights under
|
|
||||||
this License (including any patent licenses granted under the third
|
|
||||||
paragraph of section 11).
|
|
||||||
|
|
||||||
However, if you cease all violation of this License, then your
|
|
||||||
license from a particular copyright holder is reinstated (a)
|
|
||||||
provisionally, unless and until the copyright holder explicitly and
|
|
||||||
finally terminates your license, and (b) permanently, if the copyright
|
|
||||||
holder fails to notify you of the violation by some reasonable means
|
|
||||||
prior to 60 days after the cessation.
|
|
||||||
|
|
||||||
Moreover, your license from a particular copyright holder is
|
|
||||||
reinstated permanently if the copyright holder notifies you of the
|
|
||||||
violation by some reasonable means, this is the first time you have
|
|
||||||
received notice of violation of this License (for any work) from that
|
|
||||||
copyright holder, and you cure the violation prior to 30 days after
|
|
||||||
your receipt of the notice.
|
|
||||||
|
|
||||||
Termination of your rights under this section does not terminate the
|
|
||||||
licenses of parties who have received copies or rights from you under
|
|
||||||
this License. If your rights have been terminated and not permanently
|
|
||||||
reinstated, you do not qualify to receive new licenses for the same
|
|
||||||
material under section 10.
|
|
||||||
|
|
||||||
9. Acceptance Not Required for Having Copies.
|
|
||||||
|
|
||||||
You are not required to accept this License in order to receive or
|
|
||||||
run a copy of the Program. Ancillary propagation of a covered work
|
|
||||||
occurring solely as a consequence of using peer-to-peer transmission
|
|
||||||
to receive a copy likewise does not require acceptance. However,
|
|
||||||
nothing other than this License grants you permission to propagate or
|
|
||||||
modify any covered work. These actions infringe copyright if you do
|
|
||||||
not accept this License. Therefore, by modifying or propagating a
|
|
||||||
covered work, you indicate your acceptance of this License to do so.
|
|
||||||
|
|
||||||
10. Automatic Licensing of Downstream Recipients.
|
|
||||||
|
|
||||||
Each time you convey a covered work, the recipient automatically
|
|
||||||
receives a license from the original licensors, to run, modify and
|
|
||||||
propagate that work, subject to this License. You are not responsible
|
|
||||||
for enforcing compliance by third parties with this License.
|
|
||||||
|
|
||||||
An "entity transaction" is a transaction transferring control of an
|
|
||||||
organization, or substantially all assets of one, or subdividing an
|
|
||||||
organization, or merging organizations. If propagation of a covered
|
|
||||||
work results from an entity transaction, each party to that
|
|
||||||
transaction who receives a copy of the work also receives whatever
|
|
||||||
licenses to the work the party's predecessor in interest had or could
|
|
||||||
give under the previous paragraph, plus a right to possession of the
|
|
||||||
Corresponding Source of the work from the predecessor in interest, if
|
|
||||||
the predecessor has it or can get it with reasonable efforts.
|
|
||||||
|
|
||||||
You may not impose any further restrictions on the exercise of the
|
|
||||||
rights granted or affirmed under this License. For example, you may
|
|
||||||
not impose a license fee, royalty, or other charge for exercise of
|
|
||||||
rights granted under this License, and you may not initiate litigation
|
|
||||||
(including a cross-claim or counterclaim in a lawsuit) alleging that
|
|
||||||
any patent claim is infringed by making, using, selling, offering for
|
|
||||||
sale, or importing the Program or any portion of it.
|
|
||||||
|
|
||||||
11. Patents.
|
|
||||||
|
|
||||||
A "contributor" is a copyright holder who authorizes use under this
|
|
||||||
License of the Program or a work on which the Program is based. The
|
|
||||||
work thus licensed is called the contributor's "contributor version".
|
|
||||||
|
|
||||||
A contributor's "essential patent claims" are all patent claims
|
|
||||||
owned or controlled by the contributor, whether already acquired or
|
|
||||||
hereafter acquired, that would be infringed by some manner, permitted
|
|
||||||
by this License, of making, using, or selling its contributor version,
|
|
||||||
but do not include claims that would be infringed only as a
|
|
||||||
consequence of further modification of the contributor version. For
|
|
||||||
purposes of this definition, "control" includes the right to grant
|
|
||||||
patent sublicenses in a manner consistent with the requirements of
|
|
||||||
this License.
|
|
||||||
|
|
||||||
Each contributor grants you a non-exclusive, worldwide, royalty-free
|
|
||||||
patent license under the contributor's essential patent claims, to
|
|
||||||
make, use, sell, offer for sale, import and otherwise run, modify and
|
|
||||||
propagate the contents of its contributor version.
|
|
||||||
|
|
||||||
In the following three paragraphs, a "patent license" is any express
|
|
||||||
agreement or commitment, however denominated, not to enforce a patent
|
|
||||||
(such as an express permission to practice a patent or covenant not to
|
|
||||||
sue for patent infringement). To "grant" such a patent license to a
|
|
||||||
party means to make such an agreement or commitment not to enforce a
|
|
||||||
patent against the party.
|
|
||||||
|
|
||||||
If you convey a covered work, knowingly relying on a patent license,
|
|
||||||
and the Corresponding Source of the work is not available for anyone
|
|
||||||
to copy, free of charge and under the terms of this License, through a
|
|
||||||
publicly available network server or other readily accessible means,
|
|
||||||
then you must either (1) cause the Corresponding Source to be so
|
|
||||||
available, or (2) arrange to deprive yourself of the benefit of the
|
|
||||||
patent license for this particular work, or (3) arrange, in a manner
|
|
||||||
consistent with the requirements of this License, to extend the patent
|
|
||||||
license to downstream recipients. "Knowingly relying" means you have
|
|
||||||
actual knowledge that, but for the patent license, your conveying the
|
|
||||||
covered work in a country, or your recipient's use of the covered work
|
|
||||||
in a country, would infringe one or more identifiable patents in that
|
|
||||||
country that you have reason to believe are valid.
|
|
||||||
|
|
||||||
If, pursuant to or in connection with a single transaction or
|
|
||||||
arrangement, you convey, or propagate by procuring conveyance of, a
|
|
||||||
covered work, and grant a patent license to some of the parties
|
|
||||||
receiving the covered work authorizing them to use, propagate, modify
|
|
||||||
or convey a specific copy of the covered work, then the patent license
|
|
||||||
you grant is automatically extended to all recipients of the covered
|
|
||||||
work and works based on it.
|
|
||||||
|
|
||||||
A patent license is "discriminatory" if it does not include within
|
|
||||||
the scope of its coverage, prohibits the exercise of, or is
|
|
||||||
conditioned on the non-exercise of one or more of the rights that are
|
|
||||||
specifically granted under this License. You may not convey a covered
|
|
||||||
work if you are a party to an arrangement with a third party that is
|
|
||||||
in the business of distributing software, under which you make payment
|
|
||||||
to the third party based on the extent of your activity of conveying
|
|
||||||
the work, and under which the third party grants, to any of the
|
|
||||||
parties who would receive the covered work from you, a discriminatory
|
|
||||||
patent license (a) in connection with copies of the covered work
|
|
||||||
conveyed by you (or copies made from those copies), or (b) primarily
|
|
||||||
for and in connection with specific products or compilations that
|
|
||||||
contain the covered work, unless you entered into that arrangement,
|
|
||||||
or that patent license was granted, prior to 28 March 2007.
|
|
||||||
|
|
||||||
Nothing in this License shall be construed as excluding or limiting
|
|
||||||
any implied license or other defenses to infringement that may
|
|
||||||
otherwise be available to you under applicable patent law.
|
|
||||||
|
|
||||||
12. No Surrender of Others' Freedom.
|
|
||||||
|
|
||||||
If conditions are imposed on you (whether by court order, agreement or
|
|
||||||
otherwise) that contradict the conditions of this License, they do not
|
|
||||||
excuse you from the conditions of this License. If you cannot convey a
|
|
||||||
covered work so as to satisfy simultaneously your obligations under this
|
|
||||||
License and any other pertinent obligations, then as a consequence you may
|
|
||||||
not convey it at all. For example, if you agree to terms that obligate you
|
|
||||||
to collect a royalty for further conveying from those to whom you convey
|
|
||||||
the Program, the only way you could satisfy both those terms and this
|
|
||||||
License would be to refrain entirely from conveying the Program.
|
|
||||||
|
|
||||||
13. Remote Network Interaction; Use with the GNU General Public License.
|
|
||||||
|
|
||||||
Notwithstanding any other provision of this License, if you modify the
|
|
||||||
Program, your modified version must prominently offer all users
|
|
||||||
interacting with it remotely through a computer network (if your version
|
|
||||||
supports such interaction) an opportunity to receive the Corresponding
|
|
||||||
Source of your version by providing access to the Corresponding Source
|
|
||||||
from a network server at no charge, through some standard or customary
|
|
||||||
means of facilitating copying of software. This Corresponding Source
|
|
||||||
shall include the Corresponding Source for any work covered by version 3
|
|
||||||
of the GNU General Public License that is incorporated pursuant to the
|
|
||||||
following paragraph.
|
|
||||||
|
|
||||||
Notwithstanding any other provision of this License, you have
|
|
||||||
permission to link or combine any covered work with a work licensed
|
|
||||||
under version 3 of the GNU General Public License into a single
|
|
||||||
combined work, and to convey the resulting work. The terms of this
|
|
||||||
License will continue to apply to the part which is the covered work,
|
|
||||||
but the work with which it is combined will remain governed by version
|
|
||||||
3 of the GNU General Public License.
|
|
||||||
|
|
||||||
14. Revised Versions of this License.
|
|
||||||
|
|
||||||
The Free Software Foundation may publish revised and/or new versions of
|
|
||||||
the GNU Affero General Public License from time to time. Such new versions
|
|
||||||
will be similar in spirit to the present version, but may differ in detail to
|
|
||||||
address new problems or concerns.
|
|
||||||
|
|
||||||
Each version is given a distinguishing version number. If the
|
|
||||||
Program specifies that a certain numbered version of the GNU Affero General
|
|
||||||
Public License "or any later version" applies to it, you have the
|
|
||||||
option of following the terms and conditions either of that numbered
|
|
||||||
version or of any later version published by the Free Software
|
|
||||||
Foundation. If the Program does not specify a version number of the
|
|
||||||
GNU Affero General Public License, you may choose any version ever published
|
|
||||||
by the Free Software Foundation.
|
|
||||||
|
|
||||||
If the Program specifies that a proxy can decide which future
|
|
||||||
versions of the GNU Affero General Public License can be used, that proxy's
|
|
||||||
public statement of acceptance of a version permanently authorizes you
|
|
||||||
to choose that version for the Program.
|
|
||||||
|
|
||||||
Later license versions may give you additional or different
|
|
||||||
permissions. However, no additional obligations are imposed on any
|
|
||||||
author or copyright holder as a result of your choosing to follow a
|
|
||||||
later version.
|
|
||||||
|
|
||||||
15. Disclaimer of Warranty.
|
|
||||||
|
|
||||||
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
|
|
||||||
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
|
|
||||||
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
|
|
||||||
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
|
|
||||||
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
|
|
||||||
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
|
|
||||||
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
|
|
||||||
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
|
|
||||||
|
|
||||||
16. Limitation of Liability.
|
|
||||||
|
|
||||||
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
|
|
||||||
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
|
|
||||||
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
|
|
||||||
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
|
|
||||||
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
|
|
||||||
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
|
|
||||||
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
|
|
||||||
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
|
|
||||||
SUCH DAMAGES.
|
|
||||||
|
|
||||||
17. Interpretation of Sections 15 and 16.
|
|
||||||
|
|
||||||
If the disclaimer of warranty and limitation of liability provided
|
|
||||||
above cannot be given local legal effect according to their terms,
|
|
||||||
reviewing courts shall apply local law that most closely approximates
|
|
||||||
an absolute waiver of all civil liability in connection with the
|
|
||||||
Program, unless a warranty or assumption of liability accompanies a
|
|
||||||
copy of the Program in return for a fee.
|
|
||||||
|
|
||||||
END OF TERMS AND CONDITIONS
|
|
||||||
|
|
||||||
How to Apply These Terms to Your New Programs
|
|
||||||
|
|
||||||
If you develop a new program, and you want it to be of the greatest
|
|
||||||
possible use to the public, the best way to achieve this is to make it
|
|
||||||
free software which everyone can redistribute and change under these terms.
|
|
||||||
|
|
||||||
To do so, attach the following notices to the program. It is safest
|
|
||||||
to attach them to the start of each source file to most effectively
|
|
||||||
state the exclusion of warranty; and each file should have at least
|
|
||||||
the "copyright" line and a pointer to where the full notice is found.
|
|
||||||
|
|
||||||
<one line to give the program's name and a brief idea of what it does.>
|
|
||||||
Copyright (C) <year> <name of author>
|
|
||||||
|
|
||||||
This program is free software: you can redistribute it and/or modify
|
|
||||||
it under the terms of the GNU Affero General Public License as published by
|
|
||||||
the Free Software Foundation, either version 3 of the License, or
|
|
||||||
(at your option) any later version.
|
|
||||||
|
|
||||||
This program is distributed in the hope that it will be useful,
|
|
||||||
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
||||||
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
||||||
GNU Affero General Public License for more details.
|
|
||||||
|
|
||||||
You should have received a copy of the GNU Affero General Public License
|
|
||||||
along with this program. If not, see <http://www.gnu.org/licenses/>.
|
|
||||||
|
|
||||||
Also add information on how to contact you by electronic and paper mail.
|
|
||||||
|
|
||||||
If your software can interact with users remotely through a computer
|
|
||||||
network, you should also make sure that it provides a way for users to
|
|
||||||
get its source. For example, if your program is a web application, its
|
|
||||||
interface could display a "Source" link that leads users to an archive
|
|
||||||
of the code. There are many ways you could offer source, and different
|
|
||||||
solutions will be better for different programs; see section 13 for the
|
|
||||||
specific requirements.
|
|
||||||
|
|
||||||
You should also get your employer (if you work as a programmer) or school,
|
|
||||||
if any, to sign a "copyright disclaimer" for the program, if necessary.
|
|
||||||
For more information on this, and how to apply and follow the GNU AGPL, see
|
|
||||||
<http://www.gnu.org/licenses/>.
|
|
|
@ -1,10 +0,0 @@
|
||||||
logger
|
|
||||||
--dependencies=None
|
|
||||||
--docker-repos=gcr.io/overleaf-ops
|
|
||||||
--env-add=
|
|
||||||
--env-pass-through=
|
|
||||||
--esmock-loader=False
|
|
||||||
--is-library=True
|
|
||||||
--node-version=20.18.2
|
|
||||||
--public-repo=False
|
|
||||||
--script-version=4.5.0
|
|
|
@ -1,133 +0,0 @@
|
||||||
const bunyan = require('bunyan')
|
|
||||||
|
|
||||||
/**
|
|
||||||
* When we copy log entry fields, omit some bunyan core fields that are not
|
|
||||||
* interesting, that have a special meaning in GCP, or that we will process
|
|
||||||
* separately.
|
|
||||||
*/
|
|
||||||
const ENTRY_FIELDS_TO_OMIT = [
|
|
||||||
'level',
|
|
||||||
'name',
|
|
||||||
'hostname',
|
|
||||||
'v',
|
|
||||||
'pid',
|
|
||||||
'msg',
|
|
||||||
'err',
|
|
||||||
'error',
|
|
||||||
'req',
|
|
||||||
'res',
|
|
||||||
]
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Convert a bunyan log entry to a format that GCP understands
|
|
||||||
*/
|
|
||||||
function convertLogEntry(entry) {
|
|
||||||
const gcpEntry = omit(entry, ENTRY_FIELDS_TO_OMIT)
|
|
||||||
|
|
||||||
// Error information. In GCP, the stack trace goes in the message property.
|
|
||||||
// This enables the error reporting feature.
|
|
||||||
const err = entry.err || entry.error
|
|
||||||
if (err) {
|
|
||||||
if (err.info) {
|
|
||||||
Object.assign(gcpEntry, err.info)
|
|
||||||
}
|
|
||||||
if (err.code) {
|
|
||||||
gcpEntry.code = err.code
|
|
||||||
}
|
|
||||||
if (err.signal) {
|
|
||||||
gcpEntry.signal = err.signal
|
|
||||||
}
|
|
||||||
const stack = err.stack
|
|
||||||
if (stack && stack !== '(no stack)') {
|
|
||||||
gcpEntry.message = stack
|
|
||||||
} else if (err.message) {
|
|
||||||
gcpEntry.message = err.message
|
|
||||||
}
|
|
||||||
if (entry.name) {
|
|
||||||
gcpEntry.serviceContext = { service: entry.name }
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Log message
|
|
||||||
if (entry.msg) {
|
|
||||||
if (gcpEntry.message) {
|
|
||||||
// A message has already been extracted from the error. Keep the extra
|
|
||||||
// message in the msg property.
|
|
||||||
gcpEntry.msg = entry.msg
|
|
||||||
} else {
|
|
||||||
gcpEntry.message = entry.msg
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Severity
|
|
||||||
if (entry.level) {
|
|
||||||
gcpEntry.severity = bunyan.nameFromLevel[entry.level]
|
|
||||||
}
|
|
||||||
|
|
||||||
// HTTP request information
|
|
||||||
if (entry.req || entry.res || entry.responseTimeMs) {
|
|
||||||
const httpRequest = {}
|
|
||||||
if (entry.req) {
|
|
||||||
const req = entry.req
|
|
||||||
httpRequest.requestMethod = req.method
|
|
||||||
httpRequest.requestUrl = req.url
|
|
||||||
httpRequest.remoteIp = req.remoteAddress
|
|
||||||
if (req.headers) {
|
|
||||||
if (req.headers['content-length']) {
|
|
||||||
httpRequest.requestSize = parseInt(req.headers['content-length'], 10)
|
|
||||||
}
|
|
||||||
httpRequest.userAgent = req.headers['user-agent']
|
|
||||||
httpRequest.referer = req.headers.referer
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if (entry.res) {
|
|
||||||
const res = entry.res
|
|
||||||
httpRequest.status = res.statusCode
|
|
||||||
if (res.headers && res.headers['content-length']) {
|
|
||||||
if (res.headers['content-length']) {
|
|
||||||
httpRequest.responseSize = parseInt(res.headers['content-length'], 10)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if (entry.responseTimeMs) {
|
|
||||||
const responseTimeSec = entry.responseTimeMs / 1000
|
|
||||||
httpRequest.latency = `${responseTimeSec}s`
|
|
||||||
}
|
|
||||||
gcpEntry.httpRequest = httpRequest
|
|
||||||
}
|
|
||||||
|
|
||||||
// Labels are indexed in GCP. We copy the project, doc and user ids to labels to enable fast filtering
|
|
||||||
const projectId =
|
|
||||||
gcpEntry.projectId ||
|
|
||||||
gcpEntry.project_id ||
|
|
||||||
(entry.req && entry.req.projectId)
|
|
||||||
const userId =
|
|
||||||
gcpEntry.userId || gcpEntry.user_id || (entry.req && entry.req.userId)
|
|
||||||
const docId =
|
|
||||||
gcpEntry.docId || gcpEntry.doc_id || (entry.req && entry.req.docId)
|
|
||||||
if (projectId || userId || docId) {
|
|
||||||
const labels = {}
|
|
||||||
if (projectId) {
|
|
||||||
labels.projectId = projectId
|
|
||||||
}
|
|
||||||
if (userId) {
|
|
||||||
labels.userId = userId
|
|
||||||
}
|
|
||||||
if (docId) {
|
|
||||||
labels.docId = docId
|
|
||||||
}
|
|
||||||
gcpEntry['logging.googleapis.com/labels'] = labels
|
|
||||||
}
|
|
||||||
|
|
||||||
return gcpEntry
|
|
||||||
}
|
|
||||||
|
|
||||||
function omit(obj, excludedFields) {
|
|
||||||
return Object.fromEntries(
|
|
||||||
Object.entries(obj).filter(([key]) => !excludedFields.includes(key))
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
module.exports = { convertLogEntry }
|
|
|
@ -1 +0,0 @@
|
||||||
module.exports = require('./logging-manager.js')
|
|
|
@ -1,60 +0,0 @@
|
||||||
const { fetchString } = require('@overleaf/fetch-utils')
|
|
||||||
const fs = require('node:fs')
|
|
||||||
|
|
||||||
class LogLevelChecker {
|
|
||||||
constructor(logger, defaultLevel) {
|
|
||||||
this.logger = logger
|
|
||||||
this.defaultLevel = defaultLevel
|
|
||||||
}
|
|
||||||
|
|
||||||
start() {
|
|
||||||
// check for log level override on startup
|
|
||||||
this.checkLogLevel()
|
|
||||||
// re-check log level every minute
|
|
||||||
this.checkInterval = setInterval(this.checkLogLevel.bind(this), 1000 * 60)
|
|
||||||
this.checkInterval.unref()
|
|
||||||
}
|
|
||||||
|
|
||||||
stop() {
|
|
||||||
clearInterval(this.checkInterval)
|
|
||||||
}
|
|
||||||
|
|
||||||
async checkLogLevel() {
|
|
||||||
try {
|
|
||||||
const end = await this.getTracingEndTime()
|
|
||||||
if (end > Date.now()) {
|
|
||||||
this.logger.level('trace')
|
|
||||||
} else {
|
|
||||||
this.logger.level(this.defaultLevel)
|
|
||||||
}
|
|
||||||
} catch (e) {
|
|
||||||
this.logger.level(this.defaultLevel)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
async getTracingEndTime() {
|
|
||||||
return 0
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
class FileLogLevelChecker extends LogLevelChecker {
|
|
||||||
async getTracingEndTime() {
|
|
||||||
const strEndTime = await fs.promises.readFile('/logging/tracingEndTime')
|
|
||||||
return parseInt(strEndTime, 10)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
class GCEMetadataLogLevelChecker extends LogLevelChecker {
|
|
||||||
async getTracingEndTime() {
|
|
||||||
const options = {
|
|
||||||
headers: {
|
|
||||||
'Metadata-Flavor': 'Google',
|
|
||||||
},
|
|
||||||
}
|
|
||||||
const uri = `http://metadata.google.internal/computeMetadata/v1/project/attributes/${this.logger.fields.name}-setLogLevelEndTime`
|
|
||||||
const strEndTime = await fetchString(uri, options)
|
|
||||||
return parseInt(strEndTime, 10)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
module.exports = { FileLogLevelChecker, GCEMetadataLogLevelChecker }
|
|
|
@ -1,186 +0,0 @@
|
||||||
const Stream = require('node:stream')
|
|
||||||
const bunyan = require('bunyan')
|
|
||||||
const GCPManager = require('./gcp-manager')
|
|
||||||
const Serializers = require('./serializers')
|
|
||||||
const {
|
|
||||||
FileLogLevelChecker,
|
|
||||||
GCEMetadataLogLevelChecker,
|
|
||||||
} = require('./log-level-checker')
|
|
||||||
|
|
||||||
const LoggingManager = {
|
|
||||||
/**
|
|
||||||
* @param {string} name - The name of the logger
|
|
||||||
*/
|
|
||||||
initialize(name) {
|
|
||||||
this.isProduction =
|
|
||||||
(process.env.NODE_ENV || '').toLowerCase() === 'production'
|
|
||||||
const isTest = (process.env.NODE_ENV || '').toLowerCase() === 'test'
|
|
||||||
this.defaultLevel =
|
|
||||||
process.env.LOG_LEVEL ||
|
|
||||||
(this.isProduction ? 'info' : isTest ? 'fatal' : 'debug')
|
|
||||||
this.loggerName = name
|
|
||||||
this.logger = bunyan.createLogger({
|
|
||||||
name,
|
|
||||||
serializers: {
|
|
||||||
err: Serializers.err,
|
|
||||||
error: Serializers.err,
|
|
||||||
req: Serializers.req,
|
|
||||||
res: Serializers.res,
|
|
||||||
},
|
|
||||||
streams: [this._getOutputStreamConfig()],
|
|
||||||
})
|
|
||||||
this._setupRingBuffer()
|
|
||||||
this._setupLogLevelChecker()
|
|
||||||
return this
|
|
||||||
},
|
|
||||||
|
|
||||||
/**
|
|
||||||
* @param {Record<string, any>|string} attributes - Attributes to log (nice serialization for err, req, res)
|
|
||||||
* @param {string} [message] - Optional message
|
|
||||||
* @signature `debug(attributes, message)`
|
|
||||||
* @signature `debug(message)`
|
|
||||||
*/
|
|
||||||
debug(attributes, message, ...args) {
|
|
||||||
return this.logger.debug(attributes, message, ...args)
|
|
||||||
},
|
|
||||||
|
|
||||||
/**
|
|
||||||
* @param {Record<string, any>|string} attributes - Attributes to log (nice serialization for err, req, res)
|
|
||||||
* @param {string} [message]
|
|
||||||
* @signature `info(attributes, message)`
|
|
||||||
* @signature `info(message)`
|
|
||||||
*/
|
|
||||||
info(attributes, message, ...args) {
|
|
||||||
return this.logger.info(attributes, message, ...args)
|
|
||||||
},
|
|
||||||
|
|
||||||
/**
|
|
||||||
* @param {Record<string, any>} attributes - Attributes to log (nice serialization for err, req, res)
|
|
||||||
* @param {string} [message]
|
|
||||||
*/
|
|
||||||
error(attributes, message, ...args) {
|
|
||||||
if (this.ringBuffer !== null && Array.isArray(this.ringBuffer.records)) {
|
|
||||||
attributes.logBuffer = this.ringBuffer.records.filter(function (record) {
|
|
||||||
return record.level !== 50
|
|
||||||
})
|
|
||||||
}
|
|
||||||
this.logger.error(attributes, message, ...Array.from(args))
|
|
||||||
},
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Alias to the error method.
|
|
||||||
* @param {Record<string, any>} attributes - Attributes to log (nice serialization for err, req, res)
|
|
||||||
* @param {string} [message]
|
|
||||||
*/
|
|
||||||
err(attributes, message, ...args) {
|
|
||||||
return this.error(attributes, message, ...args)
|
|
||||||
},
|
|
||||||
|
|
||||||
/**
|
|
||||||
* @param {Record<string, any>|string} attributes - Attributes to log (nice serialization for err, req, res)
|
|
||||||
* @param {string} [message]
|
|
||||||
* @signature `warn(attributes, message)`
|
|
||||||
* @signature `warn(message)`
|
|
||||||
*/
|
|
||||||
warn(attributes, message, ...args) {
|
|
||||||
return this.logger.warn(attributes, message, ...args)
|
|
||||||
},
|
|
||||||
|
|
||||||
/**
|
|
||||||
* @param {Record<string, any>} attributes - Attributes to log (nice serialization for err, req, res)
|
|
||||||
* @param {string} [message]
|
|
||||||
*/
|
|
||||||
fatal(attributes, message) {
|
|
||||||
this.logger.fatal(attributes, message)
|
|
||||||
},
|
|
||||||
|
|
||||||
_getOutputStreamConfig() {
|
|
||||||
switch (process.env.LOGGING_FORMAT) {
|
|
||||||
case 'gke': {
|
|
||||||
const stream = new Stream.Writable({
|
|
||||||
objectMode: true,
|
|
||||||
write(entry, encoding, callback) {
|
|
||||||
const gcpEntry = GCPManager.convertLogEntry(entry)
|
|
||||||
// eslint-disable-next-line no-console
|
|
||||||
console.log(JSON.stringify(gcpEntry, bunyan.safeCycles()))
|
|
||||||
setImmediate(callback)
|
|
||||||
},
|
|
||||||
})
|
|
||||||
return { level: this.defaultLevel, type: 'raw', stream }
|
|
||||||
}
|
|
||||||
case 'gce': {
|
|
||||||
const { LoggingBunyan } = require('@google-cloud/logging-bunyan')
|
|
||||||
return new LoggingBunyan({
|
|
||||||
logName: this.loggerName,
|
|
||||||
serviceContext: { service: this.loggerName },
|
|
||||||
}).stream(this.defaultLevel)
|
|
||||||
}
|
|
||||||
default: {
|
|
||||||
return { level: this.defaultLevel, stream: process.stdout }
|
|
||||||
}
|
|
||||||
}
|
|
||||||
},
|
|
||||||
|
|
||||||
_setupRingBuffer() {
|
|
||||||
this.ringBufferSize = parseInt(process.env.LOG_RING_BUFFER_SIZE) || 0
|
|
||||||
if (this.ringBufferSize > 0) {
|
|
||||||
this.ringBuffer = new bunyan.RingBuffer({ limit: this.ringBufferSize })
|
|
||||||
this.logger.addStream({
|
|
||||||
level: 'trace',
|
|
||||||
type: 'raw',
|
|
||||||
stream: this.ringBuffer,
|
|
||||||
})
|
|
||||||
} else {
|
|
||||||
this.ringBuffer = null
|
|
||||||
}
|
|
||||||
},
|
|
||||||
|
|
||||||
_setupLogLevelChecker() {
|
|
||||||
const logLevelSource = (
|
|
||||||
process.env.LOG_LEVEL_SOURCE || 'file'
|
|
||||||
).toLowerCase()
|
|
||||||
|
|
||||||
if (this.logLevelChecker) {
|
|
||||||
this.logLevelChecker.stop()
|
|
||||||
this.logLevelChecker = null
|
|
||||||
}
|
|
||||||
|
|
||||||
if (this.isProduction) {
|
|
||||||
switch (logLevelSource) {
|
|
||||||
case 'file':
|
|
||||||
this.logLevelChecker = new FileLogLevelChecker(
|
|
||||||
this.logger,
|
|
||||||
this.defaultLevel
|
|
||||||
)
|
|
||||||
break
|
|
||||||
case 'gce_metadata':
|
|
||||||
this.logLevelChecker = new GCEMetadataLogLevelChecker(
|
|
||||||
this.logger,
|
|
||||||
this.defaultLevel
|
|
||||||
)
|
|
||||||
break
|
|
||||||
case 'none':
|
|
||||||
break
|
|
||||||
default:
|
|
||||||
// eslint-disable-next-line no-console
|
|
||||||
console.log(`Unrecognised log level source: ${logLevelSource}`)
|
|
||||||
}
|
|
||||||
if (this.logLevelChecker) {
|
|
||||||
this.logLevelChecker.start()
|
|
||||||
}
|
|
||||||
}
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
LoggingManager.initialize('default')
|
|
||||||
|
|
||||||
function handleWarning(err) {
|
|
||||||
LoggingManager.warn({ err }, 'Warning details')
|
|
||||||
}
|
|
||||||
|
|
||||||
process.on('warning', handleWarning)
|
|
||||||
LoggingManager.removeWarningHandler = () => {
|
|
||||||
process.off('warning', handleWarning)
|
|
||||||
}
|
|
||||||
|
|
||||||
module.exports = LoggingManager
|
|
|
@ -1,39 +0,0 @@
|
||||||
{
|
|
||||||
"name": "@overleaf/logger",
|
|
||||||
"homepage": "www.overleaf.com",
|
|
||||||
"description": "A centralised logging system for Overleaf",
|
|
||||||
"repository": {
|
|
||||||
"type": "git",
|
|
||||||
"url": "https://github.com/overleaf/overleaf"
|
|
||||||
},
|
|
||||||
"main": "index.js",
|
|
||||||
"license": "AGPL-3.0-only",
|
|
||||||
"version": "3.1.1",
|
|
||||||
"scripts": {
|
|
||||||
"test": "npm run lint && npm run format && npm run types:check && npm run test:unit",
|
|
||||||
"format": "prettier --list-different $PWD/'**/*.{js,cjs,ts}'",
|
|
||||||
"format:fix": "prettier --write $PWD/'**/*.{js,cjs,ts}'",
|
|
||||||
"lint": "eslint --ext .js --ext .cjs --ext .ts --max-warnings 0 --format unix .",
|
|
||||||
"lint:fix": "eslint --fix --ext .js --ext .cjs --ext .ts .",
|
|
||||||
"test:ci": "npm run test:unit",
|
|
||||||
"test:unit": "mocha --exit test/**/*.{js,cjs}",
|
|
||||||
"types:check": "tsc --noEmit"
|
|
||||||
},
|
|
||||||
"dependencies": {
|
|
||||||
"@google-cloud/logging-bunyan": "^5.1.0",
|
|
||||||
"@overleaf/fetch-utils": "*",
|
|
||||||
"@overleaf/o-error": "*",
|
|
||||||
"bunyan": "^1.8.14"
|
|
||||||
},
|
|
||||||
"devDependencies": {
|
|
||||||
"chai": "^4.3.6",
|
|
||||||
"mocha": "^11.1.0",
|
|
||||||
"sandboxed-module": "^2.0.4",
|
|
||||||
"sinon": "^9.2.4",
|
|
||||||
"sinon-chai": "^3.7.0",
|
|
||||||
"typescript": "^5.0.4"
|
|
||||||
},
|
|
||||||
"peerDependencies": {
|
|
||||||
"@overleaf/metrics": "*"
|
|
||||||
}
|
|
||||||
}
|
|
|
@ -1,83 +0,0 @@
|
||||||
const OError = require('@overleaf/o-error')
|
|
||||||
|
|
||||||
function errSerializer(err) {
|
|
||||||
if (!err) {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
let message = err.message
|
|
||||||
if (err.path) {
|
|
||||||
// filter paths from the message to avoid duplicate errors with different path in message
|
|
||||||
// (e.g. errors from `fs` methods which have a path attribute)
|
|
||||||
message = message.replace(` '${err.path}'`, '')
|
|
||||||
}
|
|
||||||
return {
|
|
||||||
message,
|
|
||||||
name: err.name,
|
|
||||||
stack: OError.getFullStack(err),
|
|
||||||
info: OError.getFullInfo(err),
|
|
||||||
code: err.code,
|
|
||||||
signal: err.signal,
|
|
||||||
path: err.path,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
function reqSerializer(req) {
|
|
||||||
if (!req) {
|
|
||||||
return req
|
|
||||||
}
|
|
||||||
const headers = req.headers || {}
|
|
||||||
const entry = {
|
|
||||||
method: req.method,
|
|
||||||
url: req.originalUrl || req.url,
|
|
||||||
remoteAddress: getRemoteIp(req),
|
|
||||||
headers: {
|
|
||||||
referer: headers.referer || headers.referrer,
|
|
||||||
'user-agent': headers['user-agent'],
|
|
||||||
'content-length': headers['content-length'],
|
|
||||||
},
|
|
||||||
}
|
|
||||||
if (req.params) {
|
|
||||||
const projectId =
|
|
||||||
req.params.projectId || req.params.project_id || req.params.Project_id
|
|
||||||
const userId = req.params.userId || req.params.user_id
|
|
||||||
const docId = req.params.docId || req.params.doc_id
|
|
||||||
if (projectId) {
|
|
||||||
entry.projectId = projectId
|
|
||||||
}
|
|
||||||
if (userId) {
|
|
||||||
entry.userId = userId
|
|
||||||
}
|
|
||||||
if (docId) {
|
|
||||||
entry.docId = docId
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return entry
|
|
||||||
}
|
|
||||||
|
|
||||||
function resSerializer(res) {
|
|
||||||
if (!res) {
|
|
||||||
return res
|
|
||||||
}
|
|
||||||
return {
|
|
||||||
statusCode: res.statusCode,
|
|
||||||
headers: {
|
|
||||||
'content-length': res.getHeader && res.getHeader('content-length'),
|
|
||||||
},
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
function getRemoteIp(req) {
|
|
||||||
if (req.ip) {
|
|
||||||
return req.ip
|
|
||||||
}
|
|
||||||
if (req.socket) {
|
|
||||||
if (req.socket.socket && req.socket.socket.remoteAddress) {
|
|
||||||
return req.socket.socket.remoteAddress
|
|
||||||
} else if (req.socket.remoteAddress) {
|
|
||||||
return req.socket.remoteAddress
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return null
|
|
||||||
}
|
|
||||||
|
|
||||||
module.exports = { err: errSerializer, req: reqSerializer, res: resSerializer }
|
|
|
@ -1,16 +0,0 @@
|
||||||
const chai = require('chai')
|
|
||||||
const sinonChai = require('sinon-chai')
|
|
||||||
const SandboxedModule = require('sandboxed-module')
|
|
||||||
|
|
||||||
// Chai configuration
|
|
||||||
chai.should()
|
|
||||||
chai.use(sinonChai)
|
|
||||||
|
|
||||||
SandboxedModule.configure({
|
|
||||||
globals: { Buffer, JSON, console, process },
|
|
||||||
sourceTransformers: {
|
|
||||||
removeNodePrefix: function (source) {
|
|
||||||
return source.replace(/require\(['"]node:/g, "require('")
|
|
||||||
},
|
|
||||||
},
|
|
||||||
})
|
|
|
@ -1,186 +0,0 @@
|
||||||
const Path = require('node:path')
|
|
||||||
const { promisify } = require('node:util')
|
|
||||||
const SandboxedModule = require('sandboxed-module')
|
|
||||||
const sinon = require('sinon')
|
|
||||||
const { expect } = require('chai')
|
|
||||||
|
|
||||||
const MODULE_PATH = Path.join(__dirname, '../../log-level-checker.js')
|
|
||||||
const DEFAULT_LEVEL = 'warn'
|
|
||||||
const TRACE_LEVEL = 'trace'
|
|
||||||
const TRACING_END_TIME_FILE = '/logging/tracingEndTime'
|
|
||||||
const NOW = 10000
|
|
||||||
const PAST = NOW - 1000
|
|
||||||
const FUTURE = NOW + 1000
|
|
||||||
|
|
||||||
const delay = promisify(setTimeout)
|
|
||||||
|
|
||||||
describe('LogLevelChecker', function () {
|
|
||||||
beforeEach(function () {
|
|
||||||
this.logger = {
|
|
||||||
level: sinon.stub(),
|
|
||||||
fields: { name: 'myapp' },
|
|
||||||
}
|
|
||||||
this.FetchUtils = {
|
|
||||||
fetchString: sinon.stub(),
|
|
||||||
}
|
|
||||||
this.fetchLogLevelEndTimeStub = this.FetchUtils.fetchString.withArgs(
|
|
||||||
'http://metadata.google.internal/computeMetadata/v1/project/attributes/myapp-setLogLevelEndTime',
|
|
||||||
{ headers: { 'Metadata-Flavor': 'Google' } }
|
|
||||||
)
|
|
||||||
this.fetchLogLevelEndTimeStub.resolves('')
|
|
||||||
|
|
||||||
this.fs = {
|
|
||||||
promises: {
|
|
||||||
readFile: sinon.stub(),
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
this.clock = sinon.useFakeTimers(NOW)
|
|
||||||
|
|
||||||
this.module = SandboxedModule.require(MODULE_PATH, {
|
|
||||||
requires: {
|
|
||||||
'@overleaf/fetch-utils': this.FetchUtils,
|
|
||||||
fs: this.fs,
|
|
||||||
},
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
afterEach(function () {
|
|
||||||
this.clock.restore()
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('FileLogLevelChecker', function () {
|
|
||||||
beforeEach(function () {
|
|
||||||
this.logLevelChecker = new this.module.FileLogLevelChecker(
|
|
||||||
this.logger,
|
|
||||||
DEFAULT_LEVEL
|
|
||||||
)
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('when the file is empty', function () {
|
|
||||||
setupTracingEndTimeFile('')
|
|
||||||
checkLogLevel()
|
|
||||||
expectLevelSetTo(DEFAULT_LEVEL)
|
|
||||||
})
|
|
||||||
|
|
||||||
describe("when the file can't be read", function () {
|
|
||||||
beforeEach(async function () {
|
|
||||||
this.fs.promises.readFile.rejects(new Error('Read error!'))
|
|
||||||
})
|
|
||||||
checkLogLevel()
|
|
||||||
expectLevelSetTo(DEFAULT_LEVEL)
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('when the file has a timestamp in the future', function () {
|
|
||||||
setupTracingEndTimeFile(FUTURE.toString())
|
|
||||||
checkLogLevel()
|
|
||||||
expectLevelSetTo(TRACE_LEVEL)
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('when the file has a timestamp in the past', function () {
|
|
||||||
setupTracingEndTimeFile(PAST.toString())
|
|
||||||
checkLogLevel()
|
|
||||||
expectLevelSetTo(DEFAULT_LEVEL)
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('interval checker', function () {
|
|
||||||
beforeEach(function () {
|
|
||||||
this.fs.promises.readFile.resolves('')
|
|
||||||
this.logLevelChecker.start()
|
|
||||||
})
|
|
||||||
|
|
||||||
afterEach(function () {
|
|
||||||
this.logLevelChecker.stop()
|
|
||||||
})
|
|
||||||
|
|
||||||
it('checks the file every minute', async function () {
|
|
||||||
this.clock.tick(1000)
|
|
||||||
// Yield to the event loop
|
|
||||||
await delay(0)
|
|
||||||
expect(this.logger.level).to.have.been.calledOnceWithExactly(
|
|
||||||
DEFAULT_LEVEL
|
|
||||||
)
|
|
||||||
this.logger.level.reset()
|
|
||||||
|
|
||||||
// Trace until 1.5 minutes in the future
|
|
||||||
const traceUntil = NOW + 90000
|
|
||||||
this.fs.promises.readFile.resolves(traceUntil.toString())
|
|
||||||
|
|
||||||
this.clock.tick(61000)
|
|
||||||
await delay(0)
|
|
||||||
expect(this.logger.level).to.have.been.calledOnceWithExactly(
|
|
||||||
TRACE_LEVEL
|
|
||||||
)
|
|
||||||
this.logger.level.reset()
|
|
||||||
|
|
||||||
this.clock.tick(60000)
|
|
||||||
await delay(0)
|
|
||||||
expect(this.logger.level).to.have.been.calledOnceWithExactly(
|
|
||||||
DEFAULT_LEVEL
|
|
||||||
)
|
|
||||||
})
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('GCEMetadataLogLevelChecker', function () {
|
|
||||||
beforeEach(function () {
|
|
||||||
this.logLevelChecker = new this.module.GCEMetadataLogLevelChecker(
|
|
||||||
this.logger,
|
|
||||||
DEFAULT_LEVEL
|
|
||||||
)
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('when the response is empty', function () {
|
|
||||||
setupTracingEndTimeGCE('')
|
|
||||||
checkLogLevel()
|
|
||||||
expectLevelSetTo(DEFAULT_LEVEL)
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('when the request errors', function () {
|
|
||||||
beforeEach(async function () {
|
|
||||||
this.FetchUtils.fetchString.rejects(new Error('Read error!'))
|
|
||||||
})
|
|
||||||
checkLogLevel()
|
|
||||||
expectLevelSetTo(DEFAULT_LEVEL)
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('when the response is a timestamp in the future', function () {
|
|
||||||
setupTracingEndTimeGCE(FUTURE.toString())
|
|
||||||
checkLogLevel()
|
|
||||||
expectLevelSetTo(TRACE_LEVEL)
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('when the response is a timestamp in the past', function () {
|
|
||||||
setupTracingEndTimeGCE(PAST.toString())
|
|
||||||
checkLogLevel()
|
|
||||||
expectLevelSetTo(DEFAULT_LEVEL)
|
|
||||||
})
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
function setupTracingEndTimeFile(contents) {
|
|
||||||
beforeEach(`set tracing end time in file to ${contents}`, function () {
|
|
||||||
this.fs.promises.readFile.withArgs(TRACING_END_TIME_FILE).resolves(contents)
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
function setupTracingEndTimeGCE(contents) {
|
|
||||||
beforeEach(
|
|
||||||
`set tracing end time in GCE metadata to ${contents}`,
|
|
||||||
function () {
|
|
||||||
this.fetchLogLevelEndTimeStub.resolves(contents)
|
|
||||||
}
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
function checkLogLevel() {
|
|
||||||
beforeEach('Check log level', async function () {
|
|
||||||
await this.logLevelChecker.checkLogLevel()
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
function expectLevelSetTo(level) {
|
|
||||||
it(`sets the log level to ${level}`, function () {
|
|
||||||
expect(this.logger.level).to.have.been.calledWith(level)
|
|
||||||
})
|
|
||||||
}
|
|
|
@ -1,202 +0,0 @@
|
||||||
const SandboxedModule = require('sandboxed-module')
|
|
||||||
const bunyan = require('bunyan')
|
|
||||||
const { expect } = require('chai')
|
|
||||||
const path = require('node:path')
|
|
||||||
const sinon = require('sinon')
|
|
||||||
|
|
||||||
const MODULE_PATH = path.join(__dirname, '../../logging-manager.js')
|
|
||||||
|
|
||||||
describe('LoggingManager', function () {
|
|
||||||
beforeEach(function () {
|
|
||||||
this.start = Date.now()
|
|
||||||
this.bunyanLogger = {
|
|
||||||
addStream: sinon.stub(),
|
|
||||||
debug: sinon.stub(),
|
|
||||||
error: sinon.stub(),
|
|
||||||
fatal: sinon.stub(),
|
|
||||||
info: sinon.stub(),
|
|
||||||
level: sinon.stub(),
|
|
||||||
warn: sinon.stub(),
|
|
||||||
}
|
|
||||||
this.Bunyan = {
|
|
||||||
createLogger: sinon.stub().returns(this.bunyanLogger),
|
|
||||||
RingBuffer: bunyan.RingBuffer,
|
|
||||||
}
|
|
||||||
this.stackdriverStreamConfig = { stream: 'stackdriver' }
|
|
||||||
this.stackdriverClient = {
|
|
||||||
stream: sinon.stub().returns(this.stackdriverStreamConfig),
|
|
||||||
}
|
|
||||||
this.GCPLogging = {
|
|
||||||
LoggingBunyan: sinon.stub().returns(this.stackdriverClient),
|
|
||||||
}
|
|
||||||
this.FileLogLevelChecker = {
|
|
||||||
start: sinon.stub(),
|
|
||||||
stop: sinon.stub(),
|
|
||||||
}
|
|
||||||
this.GCEMetadataLogLevelChecker = {
|
|
||||||
start: sinon.stub(),
|
|
||||||
stop: sinon.stub(),
|
|
||||||
}
|
|
||||||
this.LogLevelChecker = {
|
|
||||||
FileLogLevelChecker: sinon.stub().returns(this.FileLogLevelChecker),
|
|
||||||
GCEMetadataLogLevelChecker: sinon
|
|
||||||
.stub()
|
|
||||||
.returns(this.GCEMetadataLogLevelChecker),
|
|
||||||
}
|
|
||||||
this.LoggingManager = SandboxedModule.require(MODULE_PATH, {
|
|
||||||
requires: {
|
|
||||||
bunyan: this.Bunyan,
|
|
||||||
'./log-level-checker': this.LogLevelChecker,
|
|
||||||
},
|
|
||||||
})
|
|
||||||
this.loggerName = 'test'
|
|
||||||
this.logger = this.LoggingManager.initialize(this.loggerName)
|
|
||||||
})
|
|
||||||
|
|
||||||
afterEach(function () {
|
|
||||||
this.LoggingManager.removeWarningHandler()
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('initialize', function () {
|
|
||||||
beforeEach(function () {
|
|
||||||
this.Bunyan.createLogger.reset()
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('not in production', function () {
|
|
||||||
beforeEach(function () {
|
|
||||||
this.logger = this.LoggingManager.initialize(this.loggerName)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('should default to log level debug', function () {
|
|
||||||
this.Bunyan.createLogger.firstCall.args[0].streams[0].level.should.equal(
|
|
||||||
'debug'
|
|
||||||
)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('should not instantiate a log level checker', function () {
|
|
||||||
expect(this.LoggingManager.logLevelChecker).not.to.exist
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('in production', function () {
|
|
||||||
beforeEach(function () {
|
|
||||||
process.env.NODE_ENV = 'production'
|
|
||||||
this.logger = this.LoggingManager.initialize(this.loggerName)
|
|
||||||
})
|
|
||||||
|
|
||||||
afterEach(function () {
|
|
||||||
delete process.env.NODE_ENV
|
|
||||||
})
|
|
||||||
|
|
||||||
it('should default to log level info', function () {
|
|
||||||
this.Bunyan.createLogger.firstCall.args[0].streams[0].level.should.equal(
|
|
||||||
'info'
|
|
||||||
)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('should set up a file log level checker', function () {
|
|
||||||
expect(this.logger.logLevelChecker).to.equal(this.FileLogLevelChecker)
|
|
||||||
expect(this.FileLogLevelChecker.start).to.have.been.called
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('when LOG_LEVEL set in env', function () {
|
|
||||||
beforeEach(function () {
|
|
||||||
process.env.LOG_LEVEL = 'trace'
|
|
||||||
this.LoggingManager.initialize()
|
|
||||||
})
|
|
||||||
|
|
||||||
afterEach(function () {
|
|
||||||
delete process.env.LOG_LEVEL
|
|
||||||
})
|
|
||||||
|
|
||||||
it('should use custom log level', function () {
|
|
||||||
this.Bunyan.createLogger.firstCall.args[0].streams[0].level.should.equal(
|
|
||||||
'trace'
|
|
||||||
)
|
|
||||||
})
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('bunyan logging', function () {
|
|
||||||
beforeEach(function () {
|
|
||||||
this.logArgs = [{ foo: 'bar' }, 'foo', 'bar']
|
|
||||||
})
|
|
||||||
|
|
||||||
it('should log debug', function () {
|
|
||||||
this.logger.debug(this.logArgs)
|
|
||||||
this.bunyanLogger.debug.should.have.been.calledWith(this.logArgs)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('should log error', function () {
|
|
||||||
this.logger.error(this.logArgs)
|
|
||||||
this.bunyanLogger.error.should.have.been.calledWith(this.logArgs)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('should log fatal', function () {
|
|
||||||
this.logger.fatal(this.logArgs)
|
|
||||||
this.bunyanLogger.fatal.should.have.been.calledWith(this.logArgs)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('should log info', function () {
|
|
||||||
this.logger.info(this.logArgs)
|
|
||||||
this.bunyanLogger.info.should.have.been.calledWith(this.logArgs)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('should log warn', function () {
|
|
||||||
this.logger.warn(this.logArgs)
|
|
||||||
this.bunyanLogger.warn.should.have.been.calledWith(this.logArgs)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('should log err', function () {
|
|
||||||
this.logger.err(this.logArgs)
|
|
||||||
this.bunyanLogger.error.should.have.been.calledWith(this.logArgs)
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('ringbuffer', function () {
|
|
||||||
beforeEach(function () {
|
|
||||||
this.logBufferMock = [
|
|
||||||
{ msg: 'log 1' },
|
|
||||||
{ msg: 'log 2' },
|
|
||||||
{ level: 50, msg: 'error' },
|
|
||||||
]
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('when ring buffer size is positive', function () {
|
|
||||||
beforeEach(function () {
|
|
||||||
process.env.LOG_RING_BUFFER_SIZE = '20'
|
|
||||||
this.logger = this.LoggingManager.initialize(this.loggerName)
|
|
||||||
this.logger.ringBuffer.records = this.logBufferMock
|
|
||||||
this.logger.error({}, 'error')
|
|
||||||
})
|
|
||||||
|
|
||||||
afterEach(function () {
|
|
||||||
process.env.LOG_RING_BUFFER_SIZE = undefined
|
|
||||||
})
|
|
||||||
|
|
||||||
it('should include buffered logs in error log and filter out error logs in buffer', function () {
|
|
||||||
this.bunyanLogger.error.lastCall.args[0].logBuffer.should.deep.equal([
|
|
||||||
{ msg: 'log 1' },
|
|
||||||
{ msg: 'log 2' },
|
|
||||||
])
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('when ring buffer size is zero', function () {
|
|
||||||
beforeEach(function () {
|
|
||||||
process.env.LOG_RING_BUFFER_SIZE = '0'
|
|
||||||
this.logger = this.LoggingManager.initialize(this.loggerName)
|
|
||||||
this.logger.error({}, 'error')
|
|
||||||
})
|
|
||||||
|
|
||||||
afterEach(function () {
|
|
||||||
process.env.LOG_RING_BUFFER_SIZE = undefined
|
|
||||||
})
|
|
||||||
|
|
||||||
it('should not include buffered logs in error log', function () {
|
|
||||||
expect(this.bunyanLogger.error.lastCall.args[0].logBuffer).be.undefined
|
|
||||||
})
|
|
||||||
})
|
|
||||||
})
|
|
||||||
})
|
|
|
@ -1,7 +0,0 @@
|
||||||
{
|
|
||||||
"extends": "../../tsconfig.backend.json",
|
|
||||||
"include": [
|
|
||||||
"**/*.js",
|
|
||||||
"**/*.cjs"
|
|
||||||
]
|
|
||||||
}
|
|
3
libraries/metrics/.gitignore
vendored
3
libraries/metrics/.gitignore
vendored
|
@ -1,3 +0,0 @@
|
||||||
node_modules
|
|
||||||
|
|
||||||
.npmrc
|
|
|
@ -1,4 +0,0 @@
|
||||||
/.circleci
|
|
||||||
/.eslintrc
|
|
||||||
/.nvmrc
|
|
||||||
/.prettierrc
|
|
|
@ -1 +0,0 @@
|
||||||
20.18.2
|
|
|
@ -1,9 +0,0 @@
|
||||||
## v4.1.0
|
|
||||||
|
|
||||||
* Allows skipping the `sampleRate` argument.
|
|
||||||
|
|
||||||
## v4.0.0
|
|
||||||
|
|
||||||
* Send unmodified request and response to logger.
|
|
||||||
|
|
||||||
This version of the metrics module only works with versions of the `@overleaf/logger` module greater than v3.0.0
|
|
|
@ -1,21 +0,0 @@
|
||||||
The MIT License (MIT)
|
|
||||||
|
|
||||||
Copyright (c) 2024 Overleaf
|
|
||||||
|
|
||||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
||||||
of this software and associated documentation files (the "Software"), to deal
|
|
||||||
in the Software without restriction, including without limitation the rights
|
|
||||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
||||||
copies of the Software, and to permit persons to whom the Software is
|
|
||||||
furnished to do so, subject to the following conditions:
|
|
||||||
|
|
||||||
The above copyright notice and this permission notice shall be included in
|
|
||||||
all copies or substantial portions of the Software.
|
|
||||||
|
|
||||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
||||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
||||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
||||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
||||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
||||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
|
|
||||||
THE SOFTWARE.
|
|
|
@ -1,33 +0,0 @@
|
||||||
# overleaf/metrics-module
|
|
||||||
|
|
||||||
Wrappers the [prom-client](https://github.com/siimon/prom-client) npm module to provide [Prometheus](https://prometheus.io/) metrics at `/metrics`.
|
|
||||||
|
|
||||||
Use:
|
|
||||||
|
|
||||||
```
|
|
||||||
// Metrics must be initialized before importing anything else
|
|
||||||
require('@overleaf/metrics/initialize')
|
|
||||||
|
|
||||||
const express = require('express')
|
|
||||||
const metrics = require('@overleaf/metrics')
|
|
||||||
const app = express()
|
|
||||||
metrics.injectMetricsRoute(app)
|
|
||||||
```
|
|
||||||
|
|
||||||
Request logging can be enabled:
|
|
||||||
|
|
||||||
```
|
|
||||||
const logger = require('@overleaf/logger')
|
|
||||||
...
|
|
||||||
app.use(metrics.http.monitor(logger))
|
|
||||||
```
|
|
||||||
|
|
||||||
The metrics module can be configured through the following environment variables:
|
|
||||||
|
|
||||||
- `DEBUG_METRICS` - enables display of debugging messages to the console.
|
|
||||||
- `GCP_OPENTELEMETRY` - enables OpenTelemetry tracing for GCP
|
|
||||||
- `JAEGER_OPENTELEMETRY` - enables OpenTelemetry tracing for Jaeger (in the dev environment)
|
|
||||||
- `METRICS_APP_NAME` - the app label for metrics and spans
|
|
||||||
- `METRICS_COMPRESSION_LEVEL` - sets the [compression level](https://www.npmjs.com/package/compression#level) for `/metrics`
|
|
||||||
- `STACKDRIVER_LOGGING` - toggles the request logging format
|
|
||||||
- `UV_THREADPOOL_SIZE` - sets the libuv [thread pool](http://docs.libuv.org/en/v1.x/threadpool.html) size
|
|
|
@ -1,10 +0,0 @@
|
||||||
metrics
|
|
||||||
--dependencies=None
|
|
||||||
--docker-repos=gcr.io/overleaf-ops
|
|
||||||
--env-add=
|
|
||||||
--env-pass-through=
|
|
||||||
--esmock-loader=False
|
|
||||||
--is-library=True
|
|
||||||
--node-version=20.18.2
|
|
||||||
--public-repo=False
|
|
||||||
--script-version=4.5.0
|
|
|
@ -1,34 +0,0 @@
|
||||||
/*
|
|
||||||
* decaffeinate suggestions:
|
|
||||||
* DS102: Remove unnecessary code created because of implicit returns
|
|
||||||
* DS207: Consider shorter variations of null checks
|
|
||||||
* Full docs: https://github.com/decaffeinate/decaffeinate/blob/master/docs/suggestions.md
|
|
||||||
*/
|
|
||||||
module.exports = {
|
|
||||||
monitor(logger, interval, logThreshold) {
|
|
||||||
if (interval == null) {
|
|
||||||
interval = 1000
|
|
||||||
}
|
|
||||||
if (logThreshold == null) {
|
|
||||||
logThreshold = 100
|
|
||||||
}
|
|
||||||
const Metrics = require('./index')
|
|
||||||
// check for logger on startup to avoid exceptions later if undefined
|
|
||||||
if (logger == null) {
|
|
||||||
throw new Error('logger is undefined')
|
|
||||||
}
|
|
||||||
// monitor delay in setInterval to detect event loop blocking
|
|
||||||
let previous = Date.now()
|
|
||||||
const intervalId = setInterval(function () {
|
|
||||||
const now = Date.now()
|
|
||||||
const offset = now - previous - interval
|
|
||||||
if (offset > logThreshold) {
|
|
||||||
logger.warn({ offset }, 'slow event loop')
|
|
||||||
}
|
|
||||||
previous = now
|
|
||||||
return Metrics.timing('event-loop-millsec', offset)
|
|
||||||
}, interval)
|
|
||||||
|
|
||||||
return Metrics.registerDestructor(() => clearInterval(intervalId))
|
|
||||||
},
|
|
||||||
}
|
|
|
@ -1,78 +0,0 @@
|
||||||
const Metrics = require('./index')
|
|
||||||
|
|
||||||
function monitor(logger, level = 'debug') {
|
|
||||||
return function (req, res, next) {
|
|
||||||
const startTime = Date.now()
|
|
||||||
req.logger = new RequestLogger(logger, level)
|
|
||||||
const { end } = res
|
|
||||||
res.end = function (...args) {
|
|
||||||
end.apply(this, args)
|
|
||||||
const responseTimeMs = Date.now() - startTime
|
|
||||||
const requestSize = parseInt(req.headers['content-length'], 10)
|
|
||||||
const routePath = getRoutePath(req)
|
|
||||||
|
|
||||||
if (routePath != null) {
|
|
||||||
Metrics.timing('http_request', responseTimeMs, null, {
|
|
||||||
method: req.method,
|
|
||||||
status_code: res.statusCode,
|
|
||||||
path: routePath,
|
|
||||||
})
|
|
||||||
if (requestSize) {
|
|
||||||
Metrics.summary('http_request_size_bytes', requestSize, {
|
|
||||||
method: req.method,
|
|
||||||
status_code: res.statusCode,
|
|
||||||
path: routePath,
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
req.logger.addFields({ responseTimeMs })
|
|
||||||
req.logger.emit(req, res)
|
|
||||||
}
|
|
||||||
next()
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
function getRoutePath(req) {
|
|
||||||
if (req.route && req.route.path != null) {
|
|
||||||
return req.route.path
|
|
||||||
.toString()
|
|
||||||
.replace(/\//g, '_')
|
|
||||||
.replace(/:/g, '')
|
|
||||||
.slice(1)
|
|
||||||
}
|
|
||||||
if (req.swagger && req.swagger.apiPath != null) {
|
|
||||||
return req.swagger.apiPath
|
|
||||||
}
|
|
||||||
return null
|
|
||||||
}
|
|
||||||
|
|
||||||
class RequestLogger {
|
|
||||||
constructor(logger, level) {
|
|
||||||
this._logger = logger
|
|
||||||
this._level = level
|
|
||||||
this._info = {}
|
|
||||||
}
|
|
||||||
|
|
||||||
addFields(fields) {
|
|
||||||
Object.assign(this._info, fields)
|
|
||||||
}
|
|
||||||
|
|
||||||
setLevel(level) {
|
|
||||||
this._level = level
|
|
||||||
}
|
|
||||||
|
|
||||||
disable() {
|
|
||||||
this._disabled = true
|
|
||||||
}
|
|
||||||
|
|
||||||
emit(req, res) {
|
|
||||||
if (this._disabled) {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
this.addFields({ req, res })
|
|
||||||
const url = req.originalUrl || req.url
|
|
||||||
this._logger[this._level](this._info, '%s %s', req.method, url)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
module.exports.monitor = monitor
|
|
|
@ -1,182 +0,0 @@
|
||||||
/* eslint-disable no-console */
|
|
||||||
|
|
||||||
const ExpressCompression = require('compression')
|
|
||||||
const promClient = require('prom-client')
|
|
||||||
const promWrapper = require('./prom_wrapper')
|
|
||||||
|
|
||||||
const destructors = []
|
|
||||||
|
|
||||||
require('./uv_threadpool_size')
|
|
||||||
|
|
||||||
function registerDestructor(func) {
|
|
||||||
destructors.push(func)
|
|
||||||
}
|
|
||||||
|
|
||||||
function injectMetricsRoute(app) {
|
|
||||||
app.get(
|
|
||||||
'/metrics',
|
|
||||||
ExpressCompression({
|
|
||||||
level: parseInt(process.env.METRICS_COMPRESSION_LEVEL || '1', 10),
|
|
||||||
}),
|
|
||||||
function (req, res, next) {
|
|
||||||
res.set('Content-Type', promWrapper.registry.contentType)
|
|
||||||
promWrapper.registry
|
|
||||||
.metrics()
|
|
||||||
.then(metrics => {
|
|
||||||
res.end(metrics)
|
|
||||||
})
|
|
||||||
.catch(err => {
|
|
||||||
next(err)
|
|
||||||
})
|
|
||||||
}
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
function buildPromKey(key) {
|
|
||||||
return key.replace(/[^a-zA-Z0-9]/g, '_')
|
|
||||||
}
|
|
||||||
|
|
||||||
function sanitizeValue(value) {
|
|
||||||
return parseFloat(value)
|
|
||||||
}
|
|
||||||
|
|
||||||
function set(key, value, sampleRate = 1) {
|
|
||||||
console.log('counts are not currently supported')
|
|
||||||
}
|
|
||||||
|
|
||||||
function inc(key, sampleRate = 1, labels = {}) {
|
|
||||||
if (arguments.length === 2 && typeof sampleRate === 'object') {
|
|
||||||
labels = sampleRate
|
|
||||||
}
|
|
||||||
|
|
||||||
key = buildPromKey(key)
|
|
||||||
promWrapper.metric('counter', key, labels).inc(labels)
|
|
||||||
if (process.env.DEBUG_METRICS) {
|
|
||||||
console.log('doing inc', key, labels)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
function count(key, count, sampleRate = 1, labels = {}) {
|
|
||||||
if (arguments.length === 3 && typeof sampleRate === 'object') {
|
|
||||||
labels = sampleRate
|
|
||||||
}
|
|
||||||
|
|
||||||
key = buildPromKey(key)
|
|
||||||
promWrapper.metric('counter', key, labels).inc(labels, count)
|
|
||||||
if (process.env.DEBUG_METRICS) {
|
|
||||||
console.log('doing count/inc', key, labels)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
function summary(key, value, labels = {}) {
|
|
||||||
key = buildPromKey(key)
|
|
||||||
promWrapper.metric('summary', key, labels).observe(labels, value)
|
|
||||||
if (process.env.DEBUG_METRICS) {
|
|
||||||
console.log('doing summary', key, value, labels)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
function timing(key, timeSpan, sampleRate = 1, labels = {}) {
|
|
||||||
if (arguments.length === 3 && typeof sampleRate === 'object') {
|
|
||||||
labels = sampleRate
|
|
||||||
}
|
|
||||||
|
|
||||||
key = buildPromKey('timer_' + key)
|
|
||||||
promWrapper.metric('summary', key, labels).observe(labels, timeSpan)
|
|
||||||
if (process.env.DEBUG_METRICS) {
|
|
||||||
console.log('doing timing', key, labels)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
function histogram(key, value, buckets, labels = {}) {
|
|
||||||
key = buildPromKey('histogram_' + key)
|
|
||||||
promWrapper.metric('histogram', key, labels, buckets).observe(labels, value)
|
|
||||||
if (process.env.DEBUG_METRICS) {
|
|
||||||
console.log('doing histogram', key, buckets, labels)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
class Timer {
|
|
||||||
constructor(key, sampleRate = 1, labels = {}, buckets = undefined) {
|
|
||||||
if (typeof sampleRate === 'object') {
|
|
||||||
// called with (key, labels, buckets)
|
|
||||||
if (arguments.length === 3) {
|
|
||||||
buckets = labels
|
|
||||||
labels = sampleRate
|
|
||||||
}
|
|
||||||
|
|
||||||
// called with (key, labels)
|
|
||||||
if (arguments.length === 2) {
|
|
||||||
labels = sampleRate
|
|
||||||
}
|
|
||||||
|
|
||||||
sampleRate = 1 // default value to pass to timing function
|
|
||||||
}
|
|
||||||
|
|
||||||
this.start = new Date()
|
|
||||||
key = buildPromKey(key)
|
|
||||||
this.key = key
|
|
||||||
this.sampleRate = sampleRate
|
|
||||||
this.labels = labels
|
|
||||||
this.buckets = buckets
|
|
||||||
}
|
|
||||||
|
|
||||||
// any labels passed into the done method override labels from constructor
|
|
||||||
done(labels = {}) {
|
|
||||||
const timeSpan = new Date() - this.start
|
|
||||||
if (this.buckets) {
|
|
||||||
histogram(this.key, timeSpan, this.buckets, { ...this.labels, ...labels })
|
|
||||||
} else {
|
|
||||||
timing(this.key, timeSpan, this.sampleRate, { ...this.labels, ...labels })
|
|
||||||
}
|
|
||||||
return timeSpan
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
function gauge(key, value, sampleRate = 1, labels = {}) {
|
|
||||||
if (arguments.length === 3 && typeof sampleRate === 'object') {
|
|
||||||
labels = sampleRate
|
|
||||||
}
|
|
||||||
|
|
||||||
key = buildPromKey(key)
|
|
||||||
promWrapper.metric('gauge', key, labels).set(labels, sanitizeValue(value))
|
|
||||||
if (process.env.DEBUG_METRICS) {
|
|
||||||
console.log('doing gauge', key, labels)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
function globalGauge(key, value, sampleRate = 1, labels = {}) {
|
|
||||||
key = buildPromKey(key)
|
|
||||||
labels = { host: 'global', ...labels }
|
|
||||||
promWrapper.metric('gauge', key, labels).set(labels, sanitizeValue(value))
|
|
||||||
}
|
|
||||||
|
|
||||||
function close() {
|
|
||||||
for (const func of destructors) {
|
|
||||||
func()
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
module.exports.registerDestructor = registerDestructor
|
|
||||||
module.exports.injectMetricsRoute = injectMetricsRoute
|
|
||||||
module.exports.buildPromKey = buildPromKey
|
|
||||||
module.exports.sanitizeValue = sanitizeValue
|
|
||||||
module.exports.set = set
|
|
||||||
module.exports.inc = inc
|
|
||||||
module.exports.count = count
|
|
||||||
module.exports.summary = summary
|
|
||||||
module.exports.timing = timing
|
|
||||||
module.exports.histogram = histogram
|
|
||||||
module.exports.Timer = Timer
|
|
||||||
module.exports.gauge = gauge
|
|
||||||
module.exports.globalGauge = globalGauge
|
|
||||||
module.exports.close = close
|
|
||||||
module.exports.prom = promClient
|
|
||||||
module.exports.register = promWrapper.registry
|
|
||||||
|
|
||||||
module.exports.http = require('./http')
|
|
||||||
module.exports.open_sockets = require('./open_sockets')
|
|
||||||
module.exports.leaked_sockets = require('./leaked_sockets')
|
|
||||||
module.exports.event_loop = require('./event_loop')
|
|
||||||
module.exports.memory = require('./memory')
|
|
||||||
module.exports.mongodb = require('./mongodb')
|
|
|
@ -1,105 +0,0 @@
|
||||||
/* eslint-disable no-console */
|
|
||||||
|
|
||||||
/**
|
|
||||||
* This module initializes the metrics module. It should be imported once
|
|
||||||
* before any other module to support code instrumentation.
|
|
||||||
*/
|
|
||||||
|
|
||||||
const APP_NAME = process.env.METRICS_APP_NAME || 'unknown'
|
|
||||||
const BUILD_VERSION = process.env.BUILD_VERSION
|
|
||||||
const ENABLE_PROFILE_AGENT = process.env.ENABLE_PROFILE_AGENT === 'true'
|
|
||||||
const GCP_OPENTELEMETRY = process.env.GCP_OPENTELEMETRY === 'true'
|
|
||||||
const JAEGER_OPENTELEMETRY = process.env.JAEGER_OPENTELEMETRY === 'true'
|
|
||||||
|
|
||||||
console.log('Initializing metrics')
|
|
||||||
|
|
||||||
if (GCP_OPENTELEMETRY || JAEGER_OPENTELEMETRY) {
|
|
||||||
initializeOpenTelemetryInstrumentation()
|
|
||||||
initializeOpenTelemetryLogging()
|
|
||||||
}
|
|
||||||
|
|
||||||
if (ENABLE_PROFILE_AGENT) {
|
|
||||||
initializeProfileAgent()
|
|
||||||
}
|
|
||||||
|
|
||||||
initializePrometheus()
|
|
||||||
initializePromWrapper()
|
|
||||||
recordProcessStart()
|
|
||||||
|
|
||||||
function initializeOpenTelemetryInstrumentation() {
|
|
||||||
console.log('Starting OpenTelemetry instrumentation')
|
|
||||||
const opentelemetry = require('@opentelemetry/sdk-node')
|
|
||||||
const {
|
|
||||||
getNodeAutoInstrumentations,
|
|
||||||
} = require('@opentelemetry/auto-instrumentations-node')
|
|
||||||
const { Resource } = require('@opentelemetry/resources')
|
|
||||||
const {
|
|
||||||
SemanticResourceAttributes,
|
|
||||||
} = require('@opentelemetry/semantic-conventions')
|
|
||||||
|
|
||||||
const resource = new Resource({
|
|
||||||
[SemanticResourceAttributes.SERVICE_NAME]: APP_NAME,
|
|
||||||
[SemanticResourceAttributes.SERVICE_NAMESPACE]: 'Overleaf',
|
|
||||||
'host.type': 'VM',
|
|
||||||
})
|
|
||||||
|
|
||||||
let exporter
|
|
||||||
if (GCP_OPENTELEMETRY) {
|
|
||||||
const GCP = require('@google-cloud/opentelemetry-cloud-trace-exporter')
|
|
||||||
exporter = new GCP.TraceExporter()
|
|
||||||
} else if (JAEGER_OPENTELEMETRY) {
|
|
||||||
const {
|
|
||||||
OTLPTraceExporter,
|
|
||||||
} = require('@opentelemetry/exporter-trace-otlp-http')
|
|
||||||
exporter = new OTLPTraceExporter({
|
|
||||||
url: `http://${process.env.JAEGER_HOST || 'jaeger'}:4318/v1/traces`,
|
|
||||||
})
|
|
||||||
} else {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
const sdk = new opentelemetry.NodeSDK({
|
|
||||||
traceExporter: exporter,
|
|
||||||
logger: console,
|
|
||||||
instrumentations: [getNodeAutoInstrumentations()],
|
|
||||||
resource,
|
|
||||||
})
|
|
||||||
sdk.start()
|
|
||||||
}
|
|
||||||
|
|
||||||
function initializeOpenTelemetryLogging() {
|
|
||||||
const {
|
|
||||||
diag,
|
|
||||||
DiagConsoleLogger,
|
|
||||||
DiagLogLevel,
|
|
||||||
} = require('@opentelemetry/api')
|
|
||||||
diag.setLogger(new DiagConsoleLogger(), DiagLogLevel.INFO)
|
|
||||||
}
|
|
||||||
|
|
||||||
function initializeProfileAgent() {
|
|
||||||
console.log('Starting Google Profile Agent')
|
|
||||||
const profiler = require('@google-cloud/profiler')
|
|
||||||
profiler.start({
|
|
||||||
serviceContext: {
|
|
||||||
service: APP_NAME,
|
|
||||||
version: BUILD_VERSION,
|
|
||||||
},
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
function initializePrometheus() {
|
|
||||||
const os = require('node:os')
|
|
||||||
const promClient = require('prom-client')
|
|
||||||
promClient.register.setDefaultLabels({ app: APP_NAME, host: os.hostname() })
|
|
||||||
promClient.collectDefaultMetrics({ timeout: 5000, prefix: '' })
|
|
||||||
}
|
|
||||||
|
|
||||||
function initializePromWrapper() {
|
|
||||||
const promWrapper = require('./prom_wrapper')
|
|
||||||
promWrapper.setupSweeping()
|
|
||||||
}
|
|
||||||
|
|
||||||
function recordProcessStart() {
|
|
||||||
const metrics = require('.')
|
|
||||||
metrics.inc('process_startup')
|
|
||||||
}
|
|
|
@ -1,254 +0,0 @@
|
||||||
/**
|
|
||||||
* This file monitors HTTP connections in Node.js and logs any potential socket leaks.
|
|
||||||
* It uses the `diagnostics_channel` module to intercept requests and reponses in the
|
|
||||||
* `http` module and tracks the lifetime of each http socket. If a socket is open for
|
|
||||||
* longer than a specified time, it is considered a potential leak and its details are
|
|
||||||
* logged along with the corresponding information from /proc/net/tcp.
|
|
||||||
*/
|
|
||||||
|
|
||||||
const fs = require('node:fs')
|
|
||||||
const diagnosticsChannel = require('node:diagnostics_channel')
|
|
||||||
|
|
||||||
const SOCKET_MONITOR_INTERVAL = 60 * 1000
|
|
||||||
// set the threshold for logging leaked sockets in minutes, defaults to 15
|
|
||||||
const MIN_SOCKET_LEAK_TIME =
|
|
||||||
(parseInt(process.env.LEAKED_SOCKET_AGE_THRESHOLD, 10) || 15) * 60 * 1000
|
|
||||||
|
|
||||||
// Record HTTP events using diagnostics_channel
|
|
||||||
diagnosticsChannel.subscribe('http.client.request.start', handleRequest)
|
|
||||||
diagnosticsChannel.subscribe('http.server.request.start', handleRequest)
|
|
||||||
diagnosticsChannel.subscribe('http.client.response.finish', handleResponse)
|
|
||||||
diagnosticsChannel.subscribe('http.server.response.finish', handleResponse)
|
|
||||||
|
|
||||||
function handleRequest({ request: req }) {
|
|
||||||
const socket = req?.socket
|
|
||||||
if (socket) {
|
|
||||||
recordRequest(req, socket)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
function recordRequest(req, socket) {
|
|
||||||
const { method, protocol, path, url, rawHeaders, _header } = req
|
|
||||||
socket._ol_debug = {
|
|
||||||
method,
|
|
||||||
protocol,
|
|
||||||
url: url ?? path,
|
|
||||||
request: { headers: rawHeaders ?? _header, ts: new Date() },
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
function handleResponse({ request: req, response: res }) {
|
|
||||||
const socket = req?.socket || res?.socket
|
|
||||||
if (!socket || !res) {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
if (!socket._ol_debug) {
|
|
||||||
// I don't know if this will ever happen, but if we missed the request,
|
|
||||||
// record it here.
|
|
||||||
recordRequest(req, socket)
|
|
||||||
}
|
|
||||||
const { statusCode, statusMessage, headers, _header } = res
|
|
||||||
Object.assign(socket._ol_debug, {
|
|
||||||
response: {
|
|
||||||
statusCode,
|
|
||||||
statusMessage,
|
|
||||||
headers: headers ?? _header,
|
|
||||||
ts: new Date(),
|
|
||||||
},
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// Additional functions to log request headers with sensitive information redacted
|
|
||||||
|
|
||||||
function flattenHeaders(rawHeaders) {
|
|
||||||
// Headers can be an array [KEY, VALUE, KEY, VALUE, ..]
|
|
||||||
// an object {key:value, key:value, ...}
|
|
||||||
// or a string of the headers separated by \r\n
|
|
||||||
// Flatten the array and object headers into the string form.
|
|
||||||
if (Array.isArray(rawHeaders)) {
|
|
||||||
return rawHeaders
|
|
||||||
.map((item, index) => (index % 2 === 0 ? `${item}: ` : `${item}\r\n`))
|
|
||||||
.join('')
|
|
||||||
} else if (typeof rawHeaders === 'object') {
|
|
||||||
return Object.entries(rawHeaders)
|
|
||||||
.map(([key, value]) => `${key}: ${value}\r\n`)
|
|
||||||
.join('')
|
|
||||||
} else if (typeof rawHeaders === 'string') {
|
|
||||||
return rawHeaders
|
|
||||||
} else {
|
|
||||||
return JSON.stringify(rawHeaders)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
const REDACT_REGEX = /^(Authorization|Set-Cookie|Cookie):.*?\r/gim
|
|
||||||
|
|
||||||
function redactObject(obj) {
|
|
||||||
const result = {}
|
|
||||||
for (const [key, value] of Object.entries(obj)) {
|
|
||||||
if (value == null) {
|
|
||||||
result[key] = null
|
|
||||||
} else if (key === 'headers') {
|
|
||||||
// remove headers with sensitive information
|
|
||||||
result[key] = flattenHeaders(value).replace(
|
|
||||||
REDACT_REGEX,
|
|
||||||
`$1: REDACTED\r`
|
|
||||||
)
|
|
||||||
} else if (
|
|
||||||
typeof value === 'object' &&
|
|
||||||
['request', 'response'].includes(key)
|
|
||||||
) {
|
|
||||||
result[key] = redactObject(value)
|
|
||||||
} else {
|
|
||||||
result[key] = value
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return result
|
|
||||||
}
|
|
||||||
|
|
||||||
// Check if an old socket has crossed the threshold for logging.
|
|
||||||
// We log multiple times with an exponential backoff so we can
|
|
||||||
// see how long a socket hangs around.
|
|
||||||
|
|
||||||
function isOldSocket(handle) {
|
|
||||||
const now = new Date()
|
|
||||||
const created = handle._ol_debug.request.ts
|
|
||||||
const lastLoggedAt = handle._ol_debug.lastLoggedAt ?? created
|
|
||||||
const nextLogTime = new Date(
|
|
||||||
created.getTime() +
|
|
||||||
Math.max(2 * (lastLoggedAt - created), MIN_SOCKET_LEAK_TIME)
|
|
||||||
)
|
|
||||||
return now >= nextLogTime
|
|
||||||
}
|
|
||||||
|
|
||||||
function logOldSocket(logger, handle, tcpinfo) {
|
|
||||||
const now = new Date()
|
|
||||||
const info = Object.assign(
|
|
||||||
{
|
|
||||||
localAddress: handle.localAddress,
|
|
||||||
localPort: handle.localPort,
|
|
||||||
remoteAddress: handle.remoteAddress,
|
|
||||||
remotePort: handle.remotePort,
|
|
||||||
tcpinfo,
|
|
||||||
age: Math.floor((now - handle._ol_debug.request.ts) / (60 * 1000)), // age in minutes
|
|
||||||
},
|
|
||||||
redactObject(handle._ol_debug)
|
|
||||||
)
|
|
||||||
handle._ol_debug.lastLoggedAt = now
|
|
||||||
if (tcpinfo) {
|
|
||||||
logger.error(info, 'old socket handle - tcp socket')
|
|
||||||
} else {
|
|
||||||
logger.warn(info, 'stale socket handle - no entry in /proc/net/tcp')
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Correlate socket handles with /proc/net/tcp entries using a key based on the
|
|
||||||
// local and remote addresses and ports. This will allow us to distinguish between
|
|
||||||
// sockets that are still open and sockets that have been closed and removed from
|
|
||||||
// the /proc/net/tcp table but are still present in the node active handles array.
|
|
||||||
|
|
||||||
async function getOpenSockets() {
|
|
||||||
// get open sockets remote and local address:port from /proc/net/tcp
|
|
||||||
const procNetTcp = '/proc/net/tcp'
|
|
||||||
const openSockets = new Map()
|
|
||||||
const lines = await fs.promises.readFile(procNetTcp, 'utf8')
|
|
||||||
for (const line of lines.split('\n')) {
|
|
||||||
const socket = parseProcNetTcp(line)
|
|
||||||
if (socket) {
|
|
||||||
openSockets.set(socket, line)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return openSockets
|
|
||||||
}
|
|
||||||
|
|
||||||
function keyFromSocket(socket) {
|
|
||||||
return `${socket.localAddress}:${socket.localPort} -> ${socket.remoteAddress}:${socket.remotePort}`
|
|
||||||
}
|
|
||||||
|
|
||||||
function decodeHexIpAddress(hex) {
|
|
||||||
// decode hex ip address to dotted decimal notation
|
|
||||||
const ip = parseInt(hex, 16)
|
|
||||||
const a = ip & 0xff
|
|
||||||
const b = (ip >> 8) & 0xff
|
|
||||||
const c = (ip >> 16) & 0xff
|
|
||||||
const d = (ip >> 24) & 0xff
|
|
||||||
return `${a}.${b}.${c}.${d}`
|
|
||||||
}
|
|
||||||
|
|
||||||
function decodeHexPort(hex) {
|
|
||||||
// decode hex port to decimal
|
|
||||||
return parseInt(hex, 16)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Regex for extracting the local and remote addresses and ports from the /proc/net/tcp output
|
|
||||||
// Example line:
|
|
||||||
// 16: AB02A8C0:D9E2 86941864:01BB 01 00000000:00000000 02:000004BE 00000000 0 0 36802 2 0000000000000000 28 4 26 10 -1
|
|
||||||
// ^^^^^^^^^^^^^ ^^^^^^^^^^^^^
|
|
||||||
// local remote
|
|
||||||
|
|
||||||
const TCP_STATE_REGEX =
|
|
||||||
/^\s*\d+:\s+(?<localHexAddress>[0-9A-F]{8}):(?<localHexPort>[0-9A-F]{4})\s+(?<remoteHexAddress>[0-9A-F]{8}):(?<remoteHexPort>[0-9A-F]{4})/i
|
|
||||||
|
|
||||||
function parseProcNetTcp(line) {
|
|
||||||
const match = line.match(TCP_STATE_REGEX)
|
|
||||||
if (match) {
|
|
||||||
const { localHexAddress, localHexPort, remoteHexAddress, remoteHexPort } =
|
|
||||||
match.groups
|
|
||||||
return keyFromSocket({
|
|
||||||
localAddress: decodeHexIpAddress(localHexAddress),
|
|
||||||
localPort: decodeHexPort(localHexPort),
|
|
||||||
remoteAddress: decodeHexIpAddress(remoteHexAddress),
|
|
||||||
remotePort: decodeHexPort(remoteHexPort),
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
let LeakedSocketsMonitor
|
|
||||||
|
|
||||||
// Export the monitor and scanSockets functions
|
|
||||||
|
|
||||||
module.exports = LeakedSocketsMonitor = {
|
|
||||||
monitor(logger) {
|
|
||||||
const interval = setInterval(
|
|
||||||
() => LeakedSocketsMonitor.scanSockets(logger),
|
|
||||||
SOCKET_MONITOR_INTERVAL
|
|
||||||
)
|
|
||||||
const Metrics = require('./index')
|
|
||||||
return Metrics.registerDestructor(() => clearInterval(interval))
|
|
||||||
},
|
|
||||||
scanSockets(logger) {
|
|
||||||
const debugSockets = process._getActiveHandles().filter(handle => {
|
|
||||||
return handle._ol_debug
|
|
||||||
})
|
|
||||||
|
|
||||||
// Bail out if there are no sockets with the _ol_debug property
|
|
||||||
if (debugSockets.length === 0) {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
const oldSockets = debugSockets.filter(isOldSocket)
|
|
||||||
|
|
||||||
// Bail out if there are no old sockets to log
|
|
||||||
if (oldSockets.length === 0) {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
// If there old sockets to log, get the connections from /proc/net/tcp
|
|
||||||
// to distinguish between sockets that are still open and sockets that
|
|
||||||
// have been closed and removed from the /proc/net/tcp table.
|
|
||||||
getOpenSockets()
|
|
||||||
.then(openSockets => {
|
|
||||||
oldSockets.forEach(handle => {
|
|
||||||
try {
|
|
||||||
const key = keyFromSocket(handle)
|
|
||||||
const tcpinfo = openSockets.get(key)
|
|
||||||
logOldSocket(logger, handle, tcpinfo)
|
|
||||||
} catch (err) {
|
|
||||||
logger.error({ err }, 'error in scanSockets')
|
|
||||||
}
|
|
||||||
})
|
|
||||||
})
|
|
||||||
.catch(err => {
|
|
||||||
logger.error({ err }, 'error getting open sockets')
|
|
||||||
})
|
|
||||||
},
|
|
||||||
}
|
|
|
@ -1,113 +0,0 @@
|
||||||
/*
|
|
||||||
* decaffeinate suggestions:
|
|
||||||
* DS102: Remove unnecessary code created because of implicit returns
|
|
||||||
* DS207: Consider shorter variations of null checks
|
|
||||||
* Full docs: https://github.com/decaffeinate/decaffeinate/blob/master/docs/suggestions.md
|
|
||||||
*/
|
|
||||||
// record memory usage each minute and run a periodic gc(), keeping cpu
|
|
||||||
// usage within allowable range of 1ms per minute. Also, dynamically
|
|
||||||
// adjust the period between gc()'s to reach a target of the gc saving
|
|
||||||
// 4 megabytes each time.
|
|
||||||
|
|
||||||
let MemoryMonitor
|
|
||||||
const oneMinute = 60 * 1000
|
|
||||||
const oneMegaByte = 1024 * 1024
|
|
||||||
|
|
||||||
let CpuTimeBucket = 100 // current cpu time allowance in milliseconds
|
|
||||||
const CpuTimeBucketMax = 100 // maximum amount of cpu time allowed in bucket
|
|
||||||
const CpuTimeBucketRate = 10 // add this many milliseconds per minute
|
|
||||||
|
|
||||||
let gcInterval = 1 // how many minutes between gc (parameter is dynamically adjusted)
|
|
||||||
let countSinceLastGc = 0 // how many minutes since last gc
|
|
||||||
const MemoryChunkSize = 4 // how many megabytes we need to free to consider gc worth doing
|
|
||||||
|
|
||||||
const readyToGc = function () {
|
|
||||||
// update allowed cpu time
|
|
||||||
CpuTimeBucket = CpuTimeBucket + CpuTimeBucketRate
|
|
||||||
CpuTimeBucket =
|
|
||||||
CpuTimeBucket < CpuTimeBucketMax ? CpuTimeBucket : CpuTimeBucketMax
|
|
||||||
// update counts since last gc
|
|
||||||
countSinceLastGc = countSinceLastGc + 1
|
|
||||||
// check there is enough time since last gc and we have enough cpu
|
|
||||||
return countSinceLastGc > gcInterval && CpuTimeBucket > 0
|
|
||||||
}
|
|
||||||
|
|
||||||
const executeAndTime = function (fn) {
|
|
||||||
// time the execution of fn() and subtract from cpu allowance
|
|
||||||
const t0 = process.hrtime()
|
|
||||||
fn()
|
|
||||||
const dt = process.hrtime(t0)
|
|
||||||
const timeTaken = (dt[0] + dt[1] * 1e-9) * 1e3 // in milliseconds
|
|
||||||
CpuTimeBucket -= Math.ceil(timeTaken)
|
|
||||||
return timeTaken
|
|
||||||
}
|
|
||||||
|
|
||||||
const inMegaBytes = function (obj) {
|
|
||||||
// convert process.memoryUsage hash {rss,heapTotal,heapFreed} into megabytes
|
|
||||||
const result = {}
|
|
||||||
for (const k in obj) {
|
|
||||||
const v = obj[k]
|
|
||||||
result[k] = (v / oneMegaByte).toFixed(2)
|
|
||||||
}
|
|
||||||
return result
|
|
||||||
}
|
|
||||||
|
|
||||||
const updateMemoryStats = function (oldMem, newMem) {
|
|
||||||
countSinceLastGc = 0
|
|
||||||
const delta = {}
|
|
||||||
for (const k in newMem) {
|
|
||||||
delta[k] = (newMem[k] - oldMem[k]).toFixed(2)
|
|
||||||
}
|
|
||||||
// take the max of all memory measures
|
|
||||||
const savedMemory = Math.max(-delta.rss, -delta.heapTotal, -delta.heapUsed)
|
|
||||||
delta.megabytesFreed = savedMemory
|
|
||||||
// did it do any good?
|
|
||||||
if (savedMemory < MemoryChunkSize) {
|
|
||||||
gcInterval = gcInterval + 1 // no, so wait longer next time
|
|
||||||
} else {
|
|
||||||
gcInterval = Math.max(gcInterval - 1, 1) // yes, wait less time
|
|
||||||
}
|
|
||||||
return delta
|
|
||||||
}
|
|
||||||
|
|
||||||
module.exports = MemoryMonitor = {
|
|
||||||
monitor(logger) {
|
|
||||||
const interval = setInterval(() => MemoryMonitor.Check(logger), oneMinute)
|
|
||||||
const Metrics = require('./index')
|
|
||||||
return Metrics.registerDestructor(() => clearInterval(interval))
|
|
||||||
},
|
|
||||||
|
|
||||||
Check(logger) {
|
|
||||||
let mem
|
|
||||||
const Metrics = require('./index')
|
|
||||||
const memBeforeGc = (mem = inMegaBytes(process.memoryUsage()))
|
|
||||||
Metrics.gauge('memory.rss', mem.rss)
|
|
||||||
Metrics.gauge('memory.heaptotal', mem.heapTotal)
|
|
||||||
Metrics.gauge('memory.heapused', mem.heapUsed)
|
|
||||||
Metrics.gauge('memory.gc-interval', gcInterval)
|
|
||||||
// Metrics.gauge("memory.cpu-time-bucket", CpuTimeBucket)
|
|
||||||
|
|
||||||
logger.debug(mem, 'process.memoryUsage()')
|
|
||||||
|
|
||||||
if (global.gc != null && readyToGc()) {
|
|
||||||
const gcTime = executeAndTime(global.gc).toFixed(2)
|
|
||||||
const memAfterGc = inMegaBytes(process.memoryUsage())
|
|
||||||
const deltaMem = updateMemoryStats(memBeforeGc, memAfterGc)
|
|
||||||
logger.debug(
|
|
||||||
{
|
|
||||||
gcTime,
|
|
||||||
memBeforeGc,
|
|
||||||
memAfterGc,
|
|
||||||
deltaMem,
|
|
||||||
gcInterval,
|
|
||||||
CpuTimeBucket,
|
|
||||||
},
|
|
||||||
'global.gc() forced'
|
|
||||||
)
|
|
||||||
// Metrics.timing("memory.gc-time", gcTime)
|
|
||||||
Metrics.gauge('memory.gc-rss-freed', -deltaMem.rss)
|
|
||||||
Metrics.gauge('memory.gc-heaptotal-freed', -deltaMem.heapTotal)
|
|
||||||
return Metrics.gauge('memory.gc-heapused-freed', -deltaMem.heapUsed)
|
|
||||||
}
|
|
||||||
},
|
|
||||||
}
|
|
|
@ -1,84 +0,0 @@
|
||||||
const { Gauge, Summary } = require('prom-client')
|
|
||||||
|
|
||||||
function monitor(mongoClient) {
|
|
||||||
const labelNames = ['mongo_server']
|
|
||||||
const poolSize = new Gauge({
|
|
||||||
name: 'mongo_connection_pool_size',
|
|
||||||
help: 'number of connections in the connection pool',
|
|
||||||
labelNames,
|
|
||||||
// Use this one metric's collect() to set all metrics' values.
|
|
||||||
collect,
|
|
||||||
})
|
|
||||||
const availableConnections = new Gauge({
|
|
||||||
name: 'mongo_connection_pool_available',
|
|
||||||
help: 'number of connections that are not busy',
|
|
||||||
labelNames,
|
|
||||||
})
|
|
||||||
const waitQueueSize = new Gauge({
|
|
||||||
name: 'mongo_connection_pool_waiting',
|
|
||||||
help: 'number of operations waiting for an available connection',
|
|
||||||
labelNames,
|
|
||||||
})
|
|
||||||
const maxPoolSize = new Gauge({
|
|
||||||
name: 'mongo_connection_pool_max',
|
|
||||||
help: 'max size for the connection pool',
|
|
||||||
labelNames,
|
|
||||||
})
|
|
||||||
|
|
||||||
const mongoCommandTimer = new Summary({
|
|
||||||
name: 'mongo_command_time',
|
|
||||||
help: 'time taken to complete a mongo command',
|
|
||||||
percentiles: [],
|
|
||||||
labelNames: ['status', 'method'],
|
|
||||||
})
|
|
||||||
|
|
||||||
if (mongoClient.on) {
|
|
||||||
mongoClient.on('commandSucceeded', event => {
|
|
||||||
mongoCommandTimer.observe(
|
|
||||||
{
|
|
||||||
status: 'success',
|
|
||||||
method: event.commandName === 'find' ? 'read' : 'write',
|
|
||||||
},
|
|
||||||
event.duration
|
|
||||||
)
|
|
||||||
})
|
|
||||||
|
|
||||||
mongoClient.on('commandFailed', event => {
|
|
||||||
mongoCommandTimer.observe(
|
|
||||||
{
|
|
||||||
status: 'failed',
|
|
||||||
method: event.commandName === 'find' ? 'read' : 'write',
|
|
||||||
},
|
|
||||||
event.duration
|
|
||||||
)
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
function collect() {
|
|
||||||
// Reset all gauges in case they contain values for servers that
|
|
||||||
// disappeared
|
|
||||||
poolSize.reset()
|
|
||||||
availableConnections.reset()
|
|
||||||
waitQueueSize.reset()
|
|
||||||
maxPoolSize.reset()
|
|
||||||
|
|
||||||
const servers = mongoClient.topology?.s?.servers
|
|
||||||
if (servers != null) {
|
|
||||||
for (const [address, server] of servers) {
|
|
||||||
// The server object is different between v4 and v5 (c.f. https://github.com/mongodb/node-mongodb-native/pull/3645)
|
|
||||||
const pool = server.s?.pool || server.pool
|
|
||||||
if (pool == null) {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
const labels = { mongo_server: address }
|
|
||||||
poolSize.set(labels, pool.totalConnectionCount)
|
|
||||||
availableConnections.set(labels, pool.availableConnectionCount)
|
|
||||||
waitQueueSize.set(labels, pool.waitQueueSize)
|
|
||||||
maxPoolSize.set(labels, pool.options.maxPoolSize)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
module.exports = { monitor }
|
|
|
@ -1,99 +0,0 @@
|
||||||
/*
|
|
||||||
* decaffeinate suggestions:
|
|
||||||
* DS102: Remove unnecessary code created because of implicit returns
|
|
||||||
* DS205: Consider reworking code to avoid use of IIFEs
|
|
||||||
* DS207: Consider shorter variations of null checks
|
|
||||||
* Full docs: https://github.com/decaffeinate/decaffeinate/blob/master/docs/suggestions.md
|
|
||||||
*/
|
|
||||||
let OpenSocketsMonitor
|
|
||||||
const seconds = 1000
|
|
||||||
|
|
||||||
// In Node 0.10 the default is 5, which means only 5 open connections at one.
|
|
||||||
// Node 0.12 has a default of Infinity. Make sure we have no limit set,
|
|
||||||
// regardless of Node version.
|
|
||||||
require('node:http').globalAgent.maxSockets = Infinity
|
|
||||||
require('node:https').globalAgent.maxSockets = Infinity
|
|
||||||
|
|
||||||
const SOCKETS_HTTP = require('node:http').globalAgent.sockets
|
|
||||||
const SOCKETS_HTTPS = require('node:https').globalAgent.sockets
|
|
||||||
const FREE_SOCKETS_HTTP = require('node:http').globalAgent.freeSockets
|
|
||||||
const FREE_SOCKETS_HTTPS = require('node:https').globalAgent.freeSockets
|
|
||||||
|
|
||||||
// keep track of set gauges and reset them in the next collection cycle
|
|
||||||
const SEEN_HOSTS_HTTP = new Set()
|
|
||||||
const SEEN_HOSTS_HTTPS = new Set()
|
|
||||||
const FREE_SEEN_HOSTS_HTTP = new Set()
|
|
||||||
const FREE_SEEN_HOSTS_HTTPS = new Set()
|
|
||||||
|
|
||||||
function collectConnectionsCount(
|
|
||||||
sockets,
|
|
||||||
seenHosts,
|
|
||||||
status,
|
|
||||||
https,
|
|
||||||
emitLegacyMetric
|
|
||||||
) {
|
|
||||||
const Metrics = require('./index')
|
|
||||||
Object.keys(sockets).forEach(host => seenHosts.add(host))
|
|
||||||
seenHosts.forEach(host => {
|
|
||||||
// host: 'HOST:PORT:'
|
|
||||||
const hostname = host.split(':')[0]
|
|
||||||
const openConnections = (sockets[host] || []).length
|
|
||||||
if (!openConnections) {
|
|
||||||
seenHosts.delete(host)
|
|
||||||
}
|
|
||||||
Metrics.gauge('sockets', openConnections, 1, {
|
|
||||||
path: hostname,
|
|
||||||
method: https,
|
|
||||||
status,
|
|
||||||
})
|
|
||||||
if (status === 'open' && emitLegacyMetric) {
|
|
||||||
// Emit legacy metric to keep old time series intact.
|
|
||||||
Metrics.gauge(
|
|
||||||
`${status}_connections.${https}.${hostname}`,
|
|
||||||
openConnections
|
|
||||||
)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
module.exports = OpenSocketsMonitor = {
|
|
||||||
monitor(emitLegacyMetric) {
|
|
||||||
const interval = setInterval(
|
|
||||||
() => OpenSocketsMonitor.gaugeOpenSockets(emitLegacyMetric),
|
|
||||||
5 * seconds
|
|
||||||
)
|
|
||||||
const Metrics = require('./index')
|
|
||||||
return Metrics.registerDestructor(() => clearInterval(interval))
|
|
||||||
},
|
|
||||||
|
|
||||||
gaugeOpenSockets(emitLegacyMetric) {
|
|
||||||
collectConnectionsCount(
|
|
||||||
SOCKETS_HTTP,
|
|
||||||
SEEN_HOSTS_HTTP,
|
|
||||||
'open',
|
|
||||||
'http',
|
|
||||||
emitLegacyMetric
|
|
||||||
)
|
|
||||||
collectConnectionsCount(
|
|
||||||
SOCKETS_HTTPS,
|
|
||||||
SEEN_HOSTS_HTTPS,
|
|
||||||
'open',
|
|
||||||
'https',
|
|
||||||
emitLegacyMetric
|
|
||||||
)
|
|
||||||
collectConnectionsCount(
|
|
||||||
FREE_SOCKETS_HTTP,
|
|
||||||
FREE_SEEN_HOSTS_HTTP,
|
|
||||||
'free',
|
|
||||||
'http',
|
|
||||||
false
|
|
||||||
)
|
|
||||||
collectConnectionsCount(
|
|
||||||
FREE_SOCKETS_HTTPS,
|
|
||||||
FREE_SEEN_HOSTS_HTTPS,
|
|
||||||
'free',
|
|
||||||
'https',
|
|
||||||
false
|
|
||||||
)
|
|
||||||
},
|
|
||||||
}
|
|
|
@ -1,45 +0,0 @@
|
||||||
{
|
|
||||||
"name": "@overleaf/metrics",
|
|
||||||
"version": "4.2.0",
|
|
||||||
"description": "A drop-in metrics and monitoring module for node.js apps",
|
|
||||||
"repository": {
|
|
||||||
"type": "git",
|
|
||||||
"url": "https://github.com/overleaf/metrics-module.git"
|
|
||||||
},
|
|
||||||
"main": "index.js",
|
|
||||||
"dependencies": {
|
|
||||||
"@google-cloud/opentelemetry-cloud-trace-exporter": "^2.1.0",
|
|
||||||
"@google-cloud/profiler": "^6.0.0",
|
|
||||||
"@opentelemetry/api": "^1.4.1",
|
|
||||||
"@opentelemetry/auto-instrumentations-node": "^0.39.1",
|
|
||||||
"@opentelemetry/exporter-trace-otlp-http": "^0.41.2",
|
|
||||||
"@opentelemetry/resources": "^1.15.2",
|
|
||||||
"@opentelemetry/sdk-node": "^0.41.2",
|
|
||||||
"@opentelemetry/semantic-conventions": "^1.15.2",
|
|
||||||
"compression": "^1.7.4",
|
|
||||||
"prom-client": "^14.1.1",
|
|
||||||
"yn": "^3.1.1"
|
|
||||||
},
|
|
||||||
"devDependencies": {
|
|
||||||
"bunyan": "^1.0.0",
|
|
||||||
"chai": "^4.3.6",
|
|
||||||
"mocha": "^11.1.0",
|
|
||||||
"sandboxed-module": "^2.0.4",
|
|
||||||
"sinon": "^9.2.4",
|
|
||||||
"typescript": "^5.0.4"
|
|
||||||
},
|
|
||||||
"scripts": {
|
|
||||||
"lint": "eslint --ext .js --ext .cjs --ext .ts --max-warnings 0 --format unix .",
|
|
||||||
"lint:fix": "eslint --fix --ext .js --ext .cjs --ext .ts .",
|
|
||||||
"test:unit": "mocha --exit test/**/*.{js,cjs}",
|
|
||||||
"test:acceptance": "mocha --reporter spec --recursive --exit --grep=$MOCHA_GREP test/acceptance",
|
|
||||||
"test": "npm run lint && npm run format && npm run types:check && npm run test:unit",
|
|
||||||
"format": "prettier --list-different $PWD/'**/*.{js,cjs,ts}'",
|
|
||||||
"format:fix": "prettier --write $PWD/'**/*.{js,cjs,ts}'",
|
|
||||||
"test:ci": "npm run test:unit",
|
|
||||||
"types:check": "tsc --noEmit"
|
|
||||||
},
|
|
||||||
"peerDependencies": {
|
|
||||||
"@overleaf/logger": "*"
|
|
||||||
}
|
|
||||||
}
|
|
|
@ -1,177 +0,0 @@
|
||||||
const logger = require('@overleaf/logger')
|
|
||||||
const prom = require('prom-client')
|
|
||||||
const registry = require('prom-client').register
|
|
||||||
const metrics = new Map()
|
|
||||||
|
|
||||||
const labelsKey = function (labels) {
|
|
||||||
let keys = Object.keys(labels)
|
|
||||||
if (keys.length === 0) {
|
|
||||||
return ''
|
|
||||||
}
|
|
||||||
|
|
||||||
keys = keys.sort()
|
|
||||||
|
|
||||||
let hash = ''
|
|
||||||
for (const key of keys) {
|
|
||||||
if (hash.length) {
|
|
||||||
hash += ','
|
|
||||||
}
|
|
||||||
hash += `${key}:${labels[key]}`
|
|
||||||
}
|
|
||||||
|
|
||||||
return hash
|
|
||||||
}
|
|
||||||
|
|
||||||
const labelsAsArgs = function (labels, labelNames) {
|
|
||||||
const args = []
|
|
||||||
for (const label of labelNames) {
|
|
||||||
args.push(labels[label] || '')
|
|
||||||
}
|
|
||||||
return args
|
|
||||||
}
|
|
||||||
|
|
||||||
const PromWrapper = {
|
|
||||||
ttlInMinutes: 0,
|
|
||||||
registry,
|
|
||||||
|
|
||||||
metric(type, name, labels, buckets) {
|
|
||||||
return metrics.get(name) || new MetricWrapper(type, name, labels, buckets)
|
|
||||||
},
|
|
||||||
|
|
||||||
collectDefaultMetrics: prom.collectDefaultMetrics,
|
|
||||||
}
|
|
||||||
|
|
||||||
class MetricWrapper {
|
|
||||||
constructor(type, name, labels, buckets) {
|
|
||||||
metrics.set(name, this)
|
|
||||||
this.name = name
|
|
||||||
this.instances = new Map()
|
|
||||||
this.lastAccess = new Date()
|
|
||||||
|
|
||||||
const labelNames = labels ? Object.keys(labels) : []
|
|
||||||
switch (type) {
|
|
||||||
case 'counter':
|
|
||||||
this.metric = new prom.Counter({
|
|
||||||
name,
|
|
||||||
help: name,
|
|
||||||
labelNames,
|
|
||||||
})
|
|
||||||
break
|
|
||||||
case 'histogram':
|
|
||||||
this.metric = new prom.Histogram({
|
|
||||||
name,
|
|
||||||
help: name,
|
|
||||||
labelNames,
|
|
||||||
buckets,
|
|
||||||
})
|
|
||||||
break
|
|
||||||
case 'summary':
|
|
||||||
this.metric = new prom.Summary({
|
|
||||||
name,
|
|
||||||
help: name,
|
|
||||||
maxAgeSeconds: 60,
|
|
||||||
ageBuckets: 10,
|
|
||||||
labelNames,
|
|
||||||
})
|
|
||||||
break
|
|
||||||
case 'gauge':
|
|
||||||
this.metric = new prom.Gauge({
|
|
||||||
name,
|
|
||||||
help: name,
|
|
||||||
labelNames,
|
|
||||||
})
|
|
||||||
break
|
|
||||||
default:
|
|
||||||
throw new Error(`Unknown metric type: ${type}`)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
inc(labels, value) {
|
|
||||||
this._execMethod('inc', labels, value)
|
|
||||||
}
|
|
||||||
|
|
||||||
observe(labels, value) {
|
|
||||||
this._execMethod('observe', labels, value)
|
|
||||||
}
|
|
||||||
|
|
||||||
set(labels, value) {
|
|
||||||
this._execMethod('set', labels, value)
|
|
||||||
}
|
|
||||||
|
|
||||||
sweep() {
|
|
||||||
const thresh = new Date(Date.now() - 1000 * 60 * PromWrapper.ttlInMinutes)
|
|
||||||
this.instances.forEach((instance, key) => {
|
|
||||||
if (thresh > instance.time) {
|
|
||||||
if (process.env.DEBUG_METRICS) {
|
|
||||||
// eslint-disable-next-line no-console
|
|
||||||
console.log(
|
|
||||||
'Sweeping stale metric instance',
|
|
||||||
this.name,
|
|
||||||
{ labels: instance.labels },
|
|
||||||
key
|
|
||||||
)
|
|
||||||
}
|
|
||||||
this.metric.remove(
|
|
||||||
...labelsAsArgs(instance.labels, this.metric.labelNames)
|
|
||||||
)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
|
|
||||||
if (thresh > this.lastAccess) {
|
|
||||||
if (process.env.DEBUG_METRICS) {
|
|
||||||
// eslint-disable-next-line no-console
|
|
||||||
console.log('Sweeping stale metric', this.name, thresh, this.lastAccess)
|
|
||||||
}
|
|
||||||
metrics.delete(this.name)
|
|
||||||
registry.removeSingleMetric(this.name)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
_execMethod(method, labels, value) {
|
|
||||||
const key = labelsKey(labels)
|
|
||||||
if (key !== '') {
|
|
||||||
this.instances.set(key, { time: new Date(), labels })
|
|
||||||
}
|
|
||||||
this.lastAccess = new Date()
|
|
||||||
try {
|
|
||||||
this.metric[method](labels, value)
|
|
||||||
} catch (err) {
|
|
||||||
logger.warn(
|
|
||||||
{ err, metric: this.metric.name, labels },
|
|
||||||
'failed to record metric'
|
|
||||||
)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
let sweepingInterval
|
|
||||||
PromWrapper.setupSweeping = function () {
|
|
||||||
if (sweepingInterval) {
|
|
||||||
clearInterval(sweepingInterval)
|
|
||||||
}
|
|
||||||
if (!PromWrapper.ttlInMinutes) {
|
|
||||||
if (process.env.DEBUG_METRICS) {
|
|
||||||
// eslint-disable-next-line no-console
|
|
||||||
console.log('Not registering sweep method -- empty ttl')
|
|
||||||
}
|
|
||||||
return
|
|
||||||
}
|
|
||||||
if (process.env.DEBUG_METRICS) {
|
|
||||||
// eslint-disable-next-line no-console
|
|
||||||
console.log('Registering sweep method')
|
|
||||||
}
|
|
||||||
sweepingInterval = setInterval(function () {
|
|
||||||
if (process.env.DEBUG_METRICS) {
|
|
||||||
// eslint-disable-next-line no-console
|
|
||||||
console.log('Sweeping metrics')
|
|
||||||
}
|
|
||||||
metrics.forEach((metric, key) => {
|
|
||||||
metric.sweep()
|
|
||||||
})
|
|
||||||
}, 60000)
|
|
||||||
|
|
||||||
const Metrics = require('./index')
|
|
||||||
Metrics.registerDestructor(() => clearInterval(sweepingInterval))
|
|
||||||
}
|
|
||||||
|
|
||||||
module.exports = PromWrapper
|
|
|
@ -1,343 +0,0 @@
|
||||||
const { promisify } = require('node:util')
|
|
||||||
const os = require('node:os')
|
|
||||||
const http = require('node:http')
|
|
||||||
const { expect } = require('chai')
|
|
||||||
const Metrics = require('../..')
|
|
||||||
|
|
||||||
const HOSTNAME = os.hostname()
|
|
||||||
const APP_NAME = 'test-app'
|
|
||||||
const sleep = promisify(setTimeout)
|
|
||||||
|
|
||||||
describe('Metrics module', function () {
|
|
||||||
before(function () {
|
|
||||||
process.env.METRICS_APP_NAME = 'test-app'
|
|
||||||
require('../../initialize')
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('at startup', function () {
|
|
||||||
it('increments the process_startup counter', async function () {
|
|
||||||
await expectMetricValue('process_startup', 1)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('collects default metrics', async function () {
|
|
||||||
const metric = await getMetric('process_cpu_user_seconds_total')
|
|
||||||
expect(metric).to.exist
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('inc()', function () {
|
|
||||||
it('increments counts by 1', async function () {
|
|
||||||
Metrics.inc('duck_count')
|
|
||||||
await expectMetricValue('duck_count', 1)
|
|
||||||
Metrics.inc('duck_count')
|
|
||||||
Metrics.inc('duck_count')
|
|
||||||
await expectMetricValue('duck_count', 3)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('escapes special characters in the key', async function () {
|
|
||||||
Metrics.inc('show.me the $!!')
|
|
||||||
await expectMetricValue('show_me_the____', 1)
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('count()', function () {
|
|
||||||
it('increments counts by the given count', async function () {
|
|
||||||
Metrics.count('rabbit_count', 5)
|
|
||||||
await expectMetricValue('rabbit_count', 5)
|
|
||||||
Metrics.count('rabbit_count', 6)
|
|
||||||
Metrics.count('rabbit_count', 7)
|
|
||||||
await expectMetricValue('rabbit_count', 18)
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('summary()', function () {
|
|
||||||
it('collects observations', async function () {
|
|
||||||
Metrics.summary('oven_temp', 200)
|
|
||||||
Metrics.summary('oven_temp', 300)
|
|
||||||
Metrics.summary('oven_temp', 450)
|
|
||||||
const sum = await getSummarySum('oven_temp')
|
|
||||||
expect(sum).to.equal(950)
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('timing()', function () {
|
|
||||||
it('collects timings', async function () {
|
|
||||||
Metrics.timing('sprint_100m', 10)
|
|
||||||
Metrics.timing('sprint_100m', 20)
|
|
||||||
Metrics.timing('sprint_100m', 30)
|
|
||||||
const sum = await getSummarySum('timer_sprint_100m')
|
|
||||||
expect(sum).to.equal(60)
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('histogram()', function () {
|
|
||||||
it('collects in buckets', async function () {
|
|
||||||
const buckets = [10, 100, 1000]
|
|
||||||
Metrics.histogram('distance', 10, buckets)
|
|
||||||
Metrics.histogram('distance', 20, buckets)
|
|
||||||
Metrics.histogram('distance', 100, buckets)
|
|
||||||
Metrics.histogram('distance', 200, buckets)
|
|
||||||
Metrics.histogram('distance', 1000, buckets)
|
|
||||||
Metrics.histogram('distance', 2000, buckets)
|
|
||||||
const sum = await getSummarySum('histogram_distance')
|
|
||||||
expect(sum).to.equal(3330)
|
|
||||||
await checkHistogramValues('histogram_distance', {
|
|
||||||
10: 1,
|
|
||||||
100: 3,
|
|
||||||
1000: 5,
|
|
||||||
'+Inf': 6,
|
|
||||||
})
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('Timer', function () {
|
|
||||||
beforeEach('collect timings', async function () {
|
|
||||||
const buckets = [10, 100, 1000]
|
|
||||||
for (const duration of [1, 1, 1, 15, 15, 15, 105, 105, 105]) {
|
|
||||||
const withBuckets = new Metrics.Timer(
|
|
||||||
'height',
|
|
||||||
1,
|
|
||||||
{ label_1: 'a' },
|
|
||||||
buckets
|
|
||||||
)
|
|
||||||
const withOutBuckets = new Metrics.Timer('depth', 1, { label_2: 'b' })
|
|
||||||
await sleep(duration)
|
|
||||||
withBuckets.done()
|
|
||||||
withOutBuckets.done({ label_3: 'c' })
|
|
||||||
}
|
|
||||||
})
|
|
||||||
|
|
||||||
it('with buckets', async function () {
|
|
||||||
await checkHistogramValues('histogram_height', {
|
|
||||||
10: 3,
|
|
||||||
100: 6,
|
|
||||||
1000: 9,
|
|
||||||
'+Inf': 9,
|
|
||||||
})
|
|
||||||
const labelNames = await getMetric('histogram_height').labelNames
|
|
||||||
expect(labelNames).to.deep.equal(['label_1'])
|
|
||||||
})
|
|
||||||
|
|
||||||
it('without buckets', async function () {
|
|
||||||
await checkSummaryValues('timer_depth', {
|
|
||||||
0.01: 1,
|
|
||||||
0.05: 1,
|
|
||||||
0.5: 15,
|
|
||||||
0.9: 105,
|
|
||||||
0.95: 105,
|
|
||||||
0.99: 105,
|
|
||||||
0.999: 105,
|
|
||||||
})
|
|
||||||
const labelNames = await getMetric('timer_depth').labelNames
|
|
||||||
expect(labelNames).to.deep.equal(['label_2', 'label_3'])
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('gauge()', function () {
|
|
||||||
it('records values', async function () {
|
|
||||||
Metrics.gauge('water_level', 1.5)
|
|
||||||
await expectMetricValue('water_level', 1.5)
|
|
||||||
Metrics.gauge('water_level', 4.2)
|
|
||||||
await expectMetricValue('water_level', 4.2)
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('globalGauge()', function () {
|
|
||||||
it('records values without a host label', async function () {
|
|
||||||
Metrics.globalGauge('tire_pressure', 99.99)
|
|
||||||
const { value, labels } = await getMetricValue('tire_pressure')
|
|
||||||
expect(value).to.equal(99.99)
|
|
||||||
expect(labels.host).to.equal('global')
|
|
||||||
expect(labels.app).to.equal(APP_NAME)
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('open_sockets', function () {
|
|
||||||
const keyServer1 = 'open_connections_http_127_42_42_1'
|
|
||||||
const keyServer2 = 'open_connections_http_127_42_42_2'
|
|
||||||
|
|
||||||
let finish1, finish2, emitResponse1, emitResponse2
|
|
||||||
function resetEmitResponse1() {
|
|
||||||
emitResponse1 = new Promise(resolve => (finish1 = resolve))
|
|
||||||
}
|
|
||||||
resetEmitResponse1()
|
|
||||||
function resetEmitResponse2() {
|
|
||||||
emitResponse2 = new Promise(resolve => (finish2 = resolve))
|
|
||||||
}
|
|
||||||
resetEmitResponse2()
|
|
||||||
|
|
||||||
let server1, server2
|
|
||||||
before(function setupServer1(done) {
|
|
||||||
server1 = http.createServer((req, res) => {
|
|
||||||
res.write('...')
|
|
||||||
emitResponse1.then(() => res.end())
|
|
||||||
})
|
|
||||||
server1.listen(0, '127.42.42.1', done)
|
|
||||||
})
|
|
||||||
before(function setupServer2(done) {
|
|
||||||
server2 = http.createServer((req, res) => {
|
|
||||||
res.write('...')
|
|
||||||
emitResponse2.then(() => res.end())
|
|
||||||
})
|
|
||||||
server2.listen(0, '127.42.42.2', done)
|
|
||||||
})
|
|
||||||
after(function cleanupPendingRequests() {
|
|
||||||
finish1()
|
|
||||||
finish2()
|
|
||||||
})
|
|
||||||
after(function shutdownServer1(done) {
|
|
||||||
if (server1) server1.close(done)
|
|
||||||
})
|
|
||||||
after(function shutdownServer2(done) {
|
|
||||||
if (server2) server2.close(done)
|
|
||||||
})
|
|
||||||
|
|
||||||
let urlServer1, urlServer2
|
|
||||||
before(function setUrls() {
|
|
||||||
urlServer1 = `http://127.42.42.1:${server1.address().port}/`
|
|
||||||
urlServer2 = `http://127.42.42.2:${server2.address().port}/`
|
|
||||||
})
|
|
||||||
describe('gaugeOpenSockets()', function () {
|
|
||||||
beforeEach(function runGaugeOpenSockets() {
|
|
||||||
Metrics.open_sockets.gaugeOpenSockets(true)
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('without pending connections', function () {
|
|
||||||
it('emits no open_connections', async function () {
|
|
||||||
await expectNoMetricValue(keyServer1)
|
|
||||||
await expectNoMetricValue(keyServer2)
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('with pending connections for server1', function () {
|
|
||||||
before(function (done) {
|
|
||||||
http.get(urlServer1)
|
|
||||||
http.get(urlServer1)
|
|
||||||
setTimeout(done, 10)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('emits 2 open_connections for server1', async function () {
|
|
||||||
await expectMetricValue(keyServer1, 2)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('emits no open_connections for server2', async function () {
|
|
||||||
await expectNoMetricValue(keyServer2)
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('with pending connections for server1 and server2', function () {
|
|
||||||
before(function (done) {
|
|
||||||
http.get(urlServer2)
|
|
||||||
http.get(urlServer2)
|
|
||||||
setTimeout(done, 10)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('emits 2 open_connections for server1', async function () {
|
|
||||||
await expectMetricValue(keyServer1, 2)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('emits 2 open_connections for server2', async function () {
|
|
||||||
await expectMetricValue(keyServer2, 2)
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('when requests finish for server1', function () {
|
|
||||||
before(function (done) {
|
|
||||||
finish1()
|
|
||||||
resetEmitResponse1()
|
|
||||||
http.get(urlServer1)
|
|
||||||
|
|
||||||
setTimeout(done, 10)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('emits 1 open_connections for server1', async function () {
|
|
||||||
await expectMetricValue(keyServer1, 1)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('emits 2 open_connections for server2', async function () {
|
|
||||||
await expectMetricValue(keyServer2, 2)
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('when all requests complete', function () {
|
|
||||||
before(function (done) {
|
|
||||||
finish1()
|
|
||||||
finish2()
|
|
||||||
|
|
||||||
setTimeout(done, 10)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('emits no open_connections', async function () {
|
|
||||||
await expectNoMetricValue(keyServer1)
|
|
||||||
await expectNoMetricValue(keyServer2)
|
|
||||||
})
|
|
||||||
})
|
|
||||||
})
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
function getMetric(key) {
|
|
||||||
return Metrics.register.getSingleMetric(key)
|
|
||||||
}
|
|
||||||
|
|
||||||
async function getSummarySum(key) {
|
|
||||||
const metric = getMetric(key)
|
|
||||||
const item = await metric.get()
|
|
||||||
for (const value of item.values) {
|
|
||||||
if (value.metricName === `${key}_sum`) {
|
|
||||||
return value.value
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return null
|
|
||||||
}
|
|
||||||
|
|
||||||
async function checkHistogramValues(key, values) {
|
|
||||||
const metric = getMetric(key)
|
|
||||||
const item = await metric.get()
|
|
||||||
const found = {}
|
|
||||||
for (const value of item.values) {
|
|
||||||
const bucket = value.labels.le
|
|
||||||
if (!bucket) continue
|
|
||||||
found[bucket] = value.value
|
|
||||||
}
|
|
||||||
expect(found).to.deep.equal(values)
|
|
||||||
return null
|
|
||||||
}
|
|
||||||
|
|
||||||
async function checkSummaryValues(key, values) {
|
|
||||||
const metric = getMetric(key)
|
|
||||||
const item = await metric.get()
|
|
||||||
const found = {}
|
|
||||||
for (const value of item.values) {
|
|
||||||
const quantile = value.labels.quantile
|
|
||||||
if (!quantile) continue
|
|
||||||
found[quantile] = value.value
|
|
||||||
}
|
|
||||||
for (const quantile of Object.keys(values)) {
|
|
||||||
expect(found[quantile]).to.be.within(
|
|
||||||
values[quantile] - 5,
|
|
||||||
values[quantile] + 15,
|
|
||||||
`quantile: ${quantile}`
|
|
||||||
)
|
|
||||||
}
|
|
||||||
return null
|
|
||||||
}
|
|
||||||
|
|
||||||
async function getMetricValue(key) {
|
|
||||||
const metrics = await Metrics.register.getMetricsAsJSON()
|
|
||||||
const metric = metrics.find(m => m.name === key)
|
|
||||||
return metric.values[0]
|
|
||||||
}
|
|
||||||
|
|
||||||
async function expectMetricValue(key, expectedValue) {
|
|
||||||
const value = await getMetricValue(key)
|
|
||||||
expect(value.value).to.equal(expectedValue)
|
|
||||||
expect(value.labels.host).to.equal(HOSTNAME)
|
|
||||||
expect(value.labels.app).to.equal(APP_NAME)
|
|
||||||
}
|
|
||||||
|
|
||||||
async function expectNoMetricValue(key) {
|
|
||||||
const metric = getMetric(key)
|
|
||||||
if (!metric) return
|
|
||||||
await expectMetricValue(key, 0)
|
|
||||||
}
|
|
|
@ -1,44 +0,0 @@
|
||||||
/*
|
|
||||||
* decaffeinate suggestions:
|
|
||||||
* DS102: Remove unnecessary code created because of implicit returns
|
|
||||||
* Full docs: https://github.com/decaffeinate/decaffeinate/blob/master/docs/suggestions.md
|
|
||||||
*/
|
|
||||||
const chai = require('chai')
|
|
||||||
const { expect } = chai
|
|
||||||
const path = require('node:path')
|
|
||||||
const modulePath = path.join(__dirname, '../../../event_loop.js')
|
|
||||||
const SandboxedModule = require('sandboxed-module')
|
|
||||||
const sinon = require('sinon')
|
|
||||||
|
|
||||||
describe('event_loop', function () {
|
|
||||||
before(function () {
|
|
||||||
this.metrics = {
|
|
||||||
timing: sinon.stub(),
|
|
||||||
registerDestructor: sinon.stub(),
|
|
||||||
}
|
|
||||||
this.logger = {
|
|
||||||
warn: sinon.stub(),
|
|
||||||
}
|
|
||||||
return (this.event_loop = SandboxedModule.require(modulePath, {
|
|
||||||
requires: {
|
|
||||||
'./index': this.metrics,
|
|
||||||
},
|
|
||||||
}))
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('with a logger provided', function () {
|
|
||||||
before(function () {
|
|
||||||
return this.event_loop.monitor(this.logger)
|
|
||||||
})
|
|
||||||
|
|
||||||
return it('should register a destructor with metrics', function () {
|
|
||||||
return expect(this.metrics.registerDestructor.called).to.equal(true)
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
return describe('without a logger provided', function () {
|
|
||||||
return it('should throw an exception', function () {
|
|
||||||
return expect(this.event_loop.monitor).to.throw('logger is undefined')
|
|
||||||
})
|
|
||||||
})
|
|
||||||
})
|
|
|
@ -1,171 +0,0 @@
|
||||||
const Path = require('node:path')
|
|
||||||
const SandboxedModule = require('sandboxed-module')
|
|
||||||
const sinon = require('sinon')
|
|
||||||
|
|
||||||
const MODULE_PATH = Path.join(__dirname, '../../../http.js')
|
|
||||||
|
|
||||||
describe('http.monitor', function () {
|
|
||||||
beforeEach(function () {
|
|
||||||
this.req = {
|
|
||||||
method: 'POST',
|
|
||||||
url: '/project/1234/cleanup',
|
|
||||||
headers: {
|
|
||||||
'content-length': '123',
|
|
||||||
},
|
|
||||||
route: {
|
|
||||||
path: '/project/:id/cleanup',
|
|
||||||
},
|
|
||||||
}
|
|
||||||
this.originalResponseEnd = sinon.stub()
|
|
||||||
this.res = {
|
|
||||||
end: this.originalResponseEnd,
|
|
||||||
}
|
|
||||||
this.data = 'data'
|
|
||||||
this.logger = {
|
|
||||||
debug: sinon.stub(),
|
|
||||||
info: sinon.stub(),
|
|
||||||
warn: sinon.stub(),
|
|
||||||
}
|
|
||||||
this.Metrics = {
|
|
||||||
timing: sinon.stub(),
|
|
||||||
summary: sinon.stub(),
|
|
||||||
}
|
|
||||||
this.clock = sinon.useFakeTimers()
|
|
||||||
|
|
||||||
this.http = SandboxedModule.require(MODULE_PATH, {
|
|
||||||
requires: {
|
|
||||||
'./index': this.Metrics,
|
|
||||||
},
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
afterEach(function () {
|
|
||||||
this.clock.restore()
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('with the default options', function () {
|
|
||||||
beforeEach('set up the monitor', function (done) {
|
|
||||||
this.http.monitor(this.logger)(this.req, this.res, done)
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('after a simple request', function () {
|
|
||||||
endRequest()
|
|
||||||
expectOriginalEndCalled()
|
|
||||||
expectMetrics()
|
|
||||||
|
|
||||||
it('logs the request at the DEBUG level', function () {
|
|
||||||
sinon.assert.calledWith(
|
|
||||||
this.logger.debug,
|
|
||||||
{ req: this.req, res: this.res, responseTimeMs: 500 },
|
|
||||||
'%s %s',
|
|
||||||
this.req.method,
|
|
||||||
this.req.url
|
|
||||||
)
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('when logging is disabled', function () {
|
|
||||||
beforeEach('disable logging', function () {
|
|
||||||
this.req.logger.disable()
|
|
||||||
})
|
|
||||||
|
|
||||||
endRequest()
|
|
||||||
expectOriginalEndCalled()
|
|
||||||
expectMetrics()
|
|
||||||
|
|
||||||
it("doesn't log the request", function () {
|
|
||||||
sinon.assert.notCalled(this.logger.debug)
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('with custom log fields', function () {
|
|
||||||
beforeEach('add custom fields', function () {
|
|
||||||
this.req.logger.addFields({ a: 1, b: 2 })
|
|
||||||
})
|
|
||||||
|
|
||||||
endRequest()
|
|
||||||
|
|
||||||
it('logs the request with the custom log fields', function () {
|
|
||||||
sinon.assert.calledWith(
|
|
||||||
this.logger.debug,
|
|
||||||
{ req: this.req, res: this.res, responseTimeMs: 500, a: 1, b: 2 },
|
|
||||||
'%s %s',
|
|
||||||
this.req.method,
|
|
||||||
this.req.url
|
|
||||||
)
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('when setting the log level', function () {
|
|
||||||
beforeEach('set custom level', function () {
|
|
||||||
this.req.logger.setLevel('warn')
|
|
||||||
})
|
|
||||||
|
|
||||||
endRequest()
|
|
||||||
|
|
||||||
it('logs the request at the custom level', function () {
|
|
||||||
sinon.assert.calledWith(
|
|
||||||
this.logger.warn,
|
|
||||||
{ req: this.req, res: this.res, responseTimeMs: 500 },
|
|
||||||
'%s %s',
|
|
||||||
this.req.method,
|
|
||||||
this.req.url
|
|
||||||
)
|
|
||||||
})
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('with a different default log level', function () {
|
|
||||||
beforeEach('set up the monitor', function (done) {
|
|
||||||
this.http.monitor(this.logger, 'info')(this.req, this.res, done)
|
|
||||||
})
|
|
||||||
|
|
||||||
endRequest()
|
|
||||||
|
|
||||||
it('logs the request at that level', function () {
|
|
||||||
sinon.assert.calledWith(
|
|
||||||
this.logger.info,
|
|
||||||
{ req: this.req, res: this.res, responseTimeMs: 500 },
|
|
||||||
'%s %s',
|
|
||||||
this.req.method,
|
|
||||||
this.req.url
|
|
||||||
)
|
|
||||||
})
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
function endRequest() {
|
|
||||||
beforeEach('end the request', function () {
|
|
||||||
this.clock.tick(500)
|
|
||||||
this.res.end(this.data)
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
function expectOriginalEndCalled() {
|
|
||||||
it('calls the original res.end()', function () {
|
|
||||||
sinon.assert.calledWith(this.originalResponseEnd, this.data)
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
function expectMetrics() {
|
|
||||||
it('records the response time', function () {
|
|
||||||
sinon.assert.calledWith(this.Metrics.timing, 'http_request', 500, null, {
|
|
||||||
method: this.req.method,
|
|
||||||
status_code: this.res.status_code,
|
|
||||||
path: 'project_id_cleanup',
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
it('records the request size', function () {
|
|
||||||
sinon.assert.calledWith(
|
|
||||||
this.Metrics.summary,
|
|
||||||
'http_request_size_bytes',
|
|
||||||
123,
|
|
||||||
{
|
|
||||||
method: this.req.method,
|
|
||||||
status_code: this.res.status_code,
|
|
||||||
path: 'project_id_cleanup',
|
|
||||||
}
|
|
||||||
)
|
|
||||||
})
|
|
||||||
}
|
|
|
@ -1,89 +0,0 @@
|
||||||
const Metrics = require('../../..')
|
|
||||||
|
|
||||||
const { expect } = require('chai')
|
|
||||||
const prom = require('prom-client')
|
|
||||||
|
|
||||||
describe('mongodb', function () {
|
|
||||||
beforeEach(function () {
|
|
||||||
prom.register.clear()
|
|
||||||
this.pool = {
|
|
||||||
totalConnectionCount: 8,
|
|
||||||
availableConnectionCount: 2,
|
|
||||||
waitQueueSize: 4,
|
|
||||||
options: { maxPoolSize: 10 },
|
|
||||||
}
|
|
||||||
this.servers = new Map([['server1', { s: { pool: this.pool } }]])
|
|
||||||
|
|
||||||
this.mongoClient = { topology: { s: { servers: this.servers } } }
|
|
||||||
})
|
|
||||||
|
|
||||||
it('handles an unconnected client', async function () {
|
|
||||||
const mongoClient = {}
|
|
||||||
Metrics.mongodb.monitor(mongoClient)
|
|
||||||
const metrics = await getMetrics()
|
|
||||||
expect(metrics).to.deep.equal({})
|
|
||||||
})
|
|
||||||
|
|
||||||
it('collects Mongo metrics', async function () {
|
|
||||||
Metrics.mongodb.monitor(this.mongoClient)
|
|
||||||
const metrics = await getMetrics()
|
|
||||||
expect(metrics).to.deep.equal({
|
|
||||||
'mongo_connection_pool_max:server1': 10,
|
|
||||||
'mongo_connection_pool_size:server1': 8,
|
|
||||||
'mongo_connection_pool_available:server1': 2,
|
|
||||||
'mongo_connection_pool_waiting:server1': 4,
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
it('handles topology changes', async function () {
|
|
||||||
Metrics.mongodb.monitor(this.mongoClient)
|
|
||||||
let metrics = await getMetrics()
|
|
||||||
expect(metrics).to.deep.equal({
|
|
||||||
'mongo_connection_pool_max:server1': 10,
|
|
||||||
'mongo_connection_pool_size:server1': 8,
|
|
||||||
'mongo_connection_pool_available:server1': 2,
|
|
||||||
'mongo_connection_pool_waiting:server1': 4,
|
|
||||||
})
|
|
||||||
|
|
||||||
// Add a server
|
|
||||||
this.servers.set('server2', this.servers.get('server1'))
|
|
||||||
metrics = await getMetrics()
|
|
||||||
expect(metrics).to.deep.equal({
|
|
||||||
'mongo_connection_pool_max:server1': 10,
|
|
||||||
'mongo_connection_pool_size:server1': 8,
|
|
||||||
'mongo_connection_pool_available:server1': 2,
|
|
||||||
'mongo_connection_pool_waiting:server1': 4,
|
|
||||||
'mongo_connection_pool_max:server2': 10,
|
|
||||||
'mongo_connection_pool_size:server2': 8,
|
|
||||||
'mongo_connection_pool_available:server2': 2,
|
|
||||||
'mongo_connection_pool_waiting:server2': 4,
|
|
||||||
})
|
|
||||||
|
|
||||||
// Delete a server
|
|
||||||
this.servers.delete('server1')
|
|
||||||
metrics = await getMetrics()
|
|
||||||
expect(metrics).to.deep.equal({
|
|
||||||
'mongo_connection_pool_max:server2': 10,
|
|
||||||
'mongo_connection_pool_size:server2': 8,
|
|
||||||
'mongo_connection_pool_available:server2': 2,
|
|
||||||
'mongo_connection_pool_waiting:server2': 4,
|
|
||||||
})
|
|
||||||
|
|
||||||
// Delete another server
|
|
||||||
this.servers.delete('server2')
|
|
||||||
metrics = await getMetrics()
|
|
||||||
expect(metrics).to.deep.equal({})
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
async function getMetrics() {
|
|
||||||
const metrics = await prom.register.getMetricsAsJSON()
|
|
||||||
const result = {}
|
|
||||||
for (const metric of metrics) {
|
|
||||||
for (const value of metric.values) {
|
|
||||||
const key = `${metric.name}:${value.labels.mongo_server}`
|
|
||||||
result[key] = value.value
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return result
|
|
||||||
}
|
|
|
@ -1,7 +0,0 @@
|
||||||
{
|
|
||||||
"extends": "../../tsconfig.backend.json",
|
|
||||||
"include": [
|
|
||||||
"**/*.js",
|
|
||||||
"**/*.cjs"
|
|
||||||
]
|
|
||||||
}
|
|
|
@ -1,5 +0,0 @@
|
||||||
if (!process.env.UV_THREADPOOL_SIZE) {
|
|
||||||
process.env.UV_THREADPOOL_SIZE = 16
|
|
||||||
// eslint-disable-next-line no-console
|
|
||||||
console.log(`Set UV_THREADPOOL_SIZE=${process.env.UV_THREADPOOL_SIZE}`)
|
|
||||||
}
|
|
3
libraries/mongo-utils/.gitignore
vendored
3
libraries/mongo-utils/.gitignore
vendored
|
@ -1,3 +0,0 @@
|
||||||
|
|
||||||
# managed by monorepo$ bin/update_build_scripts
|
|
||||||
.npmrc
|
|
|
@ -1 +0,0 @@
|
||||||
20.18.2
|
|
|
@ -1,307 +0,0 @@
|
||||||
// @ts-check
|
|
||||||
/* eslint-disable no-console */
|
|
||||||
const { ObjectId, ReadPreference } = require('mongodb')
|
|
||||||
|
|
||||||
const READ_PREFERENCE_SECONDARY =
|
|
||||||
process.env.MONGO_HAS_SECONDARIES === 'true'
|
|
||||||
? ReadPreference.secondary.mode
|
|
||||||
: ReadPreference.secondaryPreferred.mode
|
|
||||||
|
|
||||||
const ONE_MONTH_IN_MS = 1000 * 60 * 60 * 24 * 31
|
|
||||||
let ID_EDGE_PAST
|
|
||||||
const ID_EDGE_FUTURE = objectIdFromMs(Date.now() + 1000)
|
|
||||||
let BATCH_DESCENDING
|
|
||||||
let BATCH_SIZE
|
|
||||||
let VERBOSE_LOGGING
|
|
||||||
let BATCH_RANGE_START
|
|
||||||
let BATCH_RANGE_END
|
|
||||||
let BATCH_MAX_TIME_SPAN_IN_MS
|
|
||||||
|
|
||||||
/**
|
|
||||||
* @typedef {import("mongodb").Collection} Collection
|
|
||||||
* @typedef {import("mongodb-legacy").Collection} LegacyCollection
|
|
||||||
* @typedef {import("mongodb").Document} Document
|
|
||||||
* @typedef {import("mongodb").FindOptions} FindOptions
|
|
||||||
* @typedef {import("mongodb").UpdateFilter<Document>} UpdateDocument
|
|
||||||
*/
|
|
||||||
|
|
||||||
/**
|
|
||||||
* @typedef {Object} BatchedUpdateOptions
|
|
||||||
* @property {string} [BATCH_DESCENDING]
|
|
||||||
* @property {string} [BATCH_LAST_ID]
|
|
||||||
* @property {string} [BATCH_MAX_TIME_SPAN_IN_MS]
|
|
||||||
* @property {string} [BATCH_RANGE_END]
|
|
||||||
* @property {string} [BATCH_RANGE_START]
|
|
||||||
* @property {string} [BATCH_SIZE]
|
|
||||||
* @property {string} [VERBOSE_LOGGING]
|
|
||||||
*/
|
|
||||||
|
|
||||||
/**
|
|
||||||
* @param {BatchedUpdateOptions} options
|
|
||||||
*/
|
|
||||||
function refreshGlobalOptionsForBatchedUpdate(options = {}) {
|
|
||||||
options = Object.assign({}, options, process.env)
|
|
||||||
|
|
||||||
BATCH_DESCENDING = options.BATCH_DESCENDING === 'true'
|
|
||||||
BATCH_SIZE = parseInt(options.BATCH_SIZE || '1000', 10) || 1000
|
|
||||||
VERBOSE_LOGGING = options.VERBOSE_LOGGING === 'true'
|
|
||||||
if (options.BATCH_LAST_ID) {
|
|
||||||
BATCH_RANGE_START = objectIdFromInput(options.BATCH_LAST_ID)
|
|
||||||
} else if (options.BATCH_RANGE_START) {
|
|
||||||
BATCH_RANGE_START = objectIdFromInput(options.BATCH_RANGE_START)
|
|
||||||
} else {
|
|
||||||
if (BATCH_DESCENDING) {
|
|
||||||
BATCH_RANGE_START = ID_EDGE_FUTURE
|
|
||||||
} else {
|
|
||||||
BATCH_RANGE_START = ID_EDGE_PAST
|
|
||||||
}
|
|
||||||
}
|
|
||||||
BATCH_MAX_TIME_SPAN_IN_MS = parseInt(
|
|
||||||
options.BATCH_MAX_TIME_SPAN_IN_MS || ONE_MONTH_IN_MS.toString(),
|
|
||||||
10
|
|
||||||
)
|
|
||||||
if (options.BATCH_RANGE_END) {
|
|
||||||
BATCH_RANGE_END = objectIdFromInput(options.BATCH_RANGE_END)
|
|
||||||
} else {
|
|
||||||
if (BATCH_DESCENDING) {
|
|
||||||
BATCH_RANGE_END = ID_EDGE_PAST
|
|
||||||
} else {
|
|
||||||
BATCH_RANGE_END = ID_EDGE_FUTURE
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* @param {Collection | LegacyCollection} collection
|
|
||||||
* @param {Document} query
|
|
||||||
* @param {ObjectId} start
|
|
||||||
* @param {ObjectId} end
|
|
||||||
* @param {Document} projection
|
|
||||||
* @param {FindOptions} findOptions
|
|
||||||
* @return {Promise<Array<Document>>}
|
|
||||||
*/
|
|
||||||
async function getNextBatch(
|
|
||||||
collection,
|
|
||||||
query,
|
|
||||||
start,
|
|
||||||
end,
|
|
||||||
projection,
|
|
||||||
findOptions
|
|
||||||
) {
|
|
||||||
if (BATCH_DESCENDING) {
|
|
||||||
query._id = {
|
|
||||||
$gt: end,
|
|
||||||
$lte: start,
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
query._id = {
|
|
||||||
$gt: start,
|
|
||||||
$lte: end,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return await collection
|
|
||||||
.find(query, findOptions)
|
|
||||||
.project(projection)
|
|
||||||
.sort({ _id: BATCH_DESCENDING ? -1 : 1 })
|
|
||||||
.limit(BATCH_SIZE)
|
|
||||||
.toArray()
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* @param {Collection | LegacyCollection} collection
|
|
||||||
* @param {Array<Document>} nextBatch
|
|
||||||
* @param {UpdateDocument} update
|
|
||||||
* @return {Promise<void>}
|
|
||||||
*/
|
|
||||||
async function performUpdate(collection, nextBatch, update) {
|
|
||||||
await collection.updateMany(
|
|
||||||
{ _id: { $in: nextBatch.map(entry => entry._id) } },
|
|
||||||
update
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* @param {string} input
|
|
||||||
* @return {ObjectId}
|
|
||||||
*/
|
|
||||||
function objectIdFromInput(input) {
|
|
||||||
if (input.includes('T')) {
|
|
||||||
const t = new Date(input).getTime()
|
|
||||||
if (Number.isNaN(t)) throw new Error(`${input} is not a valid date`)
|
|
||||||
return objectIdFromMs(t)
|
|
||||||
} else {
|
|
||||||
return new ObjectId(input)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* @param {ObjectId} objectId
|
|
||||||
* @return {string}
|
|
||||||
*/
|
|
||||||
function renderObjectId(objectId) {
|
|
||||||
return `${objectId} (${objectId.getTimestamp().toISOString()})`
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* @param {number} ms
|
|
||||||
* @return {ObjectId}
|
|
||||||
*/
|
|
||||||
function objectIdFromMs(ms) {
|
|
||||||
return ObjectId.createFromTime(ms / 1000)
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* @param {ObjectId} id
|
|
||||||
* @return {number}
|
|
||||||
*/
|
|
||||||
function getMsFromObjectId(id) {
|
|
||||||
return id.getTimestamp().getTime()
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* @param {ObjectId} start
|
|
||||||
* @return {ObjectId}
|
|
||||||
*/
|
|
||||||
function getNextEnd(start) {
|
|
||||||
let end
|
|
||||||
if (BATCH_DESCENDING) {
|
|
||||||
end = objectIdFromMs(getMsFromObjectId(start) - BATCH_MAX_TIME_SPAN_IN_MS)
|
|
||||||
if (getMsFromObjectId(end) <= getMsFromObjectId(BATCH_RANGE_END)) {
|
|
||||||
end = BATCH_RANGE_END
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
end = objectIdFromMs(getMsFromObjectId(start) + BATCH_MAX_TIME_SPAN_IN_MS)
|
|
||||||
if (getMsFromObjectId(end) >= getMsFromObjectId(BATCH_RANGE_END)) {
|
|
||||||
end = BATCH_RANGE_END
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return end
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* @param {Collection | LegacyCollection} collection
|
|
||||||
* @return {Promise<ObjectId|null>}
|
|
||||||
*/
|
|
||||||
async function getIdEdgePast(collection) {
|
|
||||||
const [first] = await collection
|
|
||||||
.find({})
|
|
||||||
.project({ _id: 1 })
|
|
||||||
.sort({ _id: 1 })
|
|
||||||
.limit(1)
|
|
||||||
.toArray()
|
|
||||||
if (!first) return null
|
|
||||||
// Go one second further into the past in order to include the first entry via
|
|
||||||
// first._id > ID_EDGE_PAST
|
|
||||||
return objectIdFromMs(Math.max(0, getMsFromObjectId(first._id) - 1000))
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* @param {Collection | LegacyCollection} collection
|
|
||||||
* @param {Document} query
|
|
||||||
* @param {UpdateDocument | ((batch: Array<Document>) => Promise<void>)} update
|
|
||||||
* @param {Document} [projection]
|
|
||||||
* @param {FindOptions} [findOptions]
|
|
||||||
* @param {BatchedUpdateOptions} [batchedUpdateOptions]
|
|
||||||
*/
|
|
||||||
async function batchedUpdate(
|
|
||||||
collection,
|
|
||||||
query,
|
|
||||||
update,
|
|
||||||
projection,
|
|
||||||
findOptions,
|
|
||||||
batchedUpdateOptions
|
|
||||||
) {
|
|
||||||
ID_EDGE_PAST = await getIdEdgePast(collection)
|
|
||||||
if (!ID_EDGE_PAST) {
|
|
||||||
console.warn(
|
|
||||||
`The collection ${collection.collectionName} appears to be empty.`
|
|
||||||
)
|
|
||||||
return 0
|
|
||||||
}
|
|
||||||
refreshGlobalOptionsForBatchedUpdate(batchedUpdateOptions)
|
|
||||||
|
|
||||||
findOptions = findOptions || {}
|
|
||||||
findOptions.readPreference = READ_PREFERENCE_SECONDARY
|
|
||||||
|
|
||||||
projection = projection || { _id: 1 }
|
|
||||||
let nextBatch
|
|
||||||
let updated = 0
|
|
||||||
let start = BATCH_RANGE_START
|
|
||||||
|
|
||||||
while (start !== BATCH_RANGE_END) {
|
|
||||||
let end = getNextEnd(start)
|
|
||||||
nextBatch = await getNextBatch(
|
|
||||||
collection,
|
|
||||||
query,
|
|
||||||
start,
|
|
||||||
end,
|
|
||||||
projection,
|
|
||||||
findOptions
|
|
||||||
)
|
|
||||||
if (nextBatch.length > 0) {
|
|
||||||
end = nextBatch[nextBatch.length - 1]._id
|
|
||||||
updated += nextBatch.length
|
|
||||||
|
|
||||||
if (VERBOSE_LOGGING) {
|
|
||||||
console.log(
|
|
||||||
`Running update on batch with ids ${JSON.stringify(
|
|
||||||
nextBatch.map(entry => entry._id)
|
|
||||||
)}`
|
|
||||||
)
|
|
||||||
} else {
|
|
||||||
console.error(`Running update on batch ending ${renderObjectId(end)}`)
|
|
||||||
}
|
|
||||||
|
|
||||||
if (typeof update === 'function') {
|
|
||||||
await update(nextBatch)
|
|
||||||
} else {
|
|
||||||
await performUpdate(collection, nextBatch, update)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
console.error(`Completed batch ending ${renderObjectId(end)}`)
|
|
||||||
start = end
|
|
||||||
}
|
|
||||||
return updated
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* @param {Collection | LegacyCollection} collection
|
|
||||||
* @param {Document} query
|
|
||||||
* @param {UpdateDocument | ((batch: Array<Object>) => Promise<void>)} update
|
|
||||||
* @param {Document} [projection]
|
|
||||||
* @param {FindOptions} [findOptions]
|
|
||||||
* @param {BatchedUpdateOptions} [batchedUpdateOptions]
|
|
||||||
*/
|
|
||||||
function batchedUpdateWithResultHandling(
|
|
||||||
collection,
|
|
||||||
query,
|
|
||||||
update,
|
|
||||||
projection,
|
|
||||||
findOptions,
|
|
||||||
batchedUpdateOptions
|
|
||||||
) {
|
|
||||||
batchedUpdate(
|
|
||||||
collection,
|
|
||||||
query,
|
|
||||||
update,
|
|
||||||
projection,
|
|
||||||
findOptions,
|
|
||||||
batchedUpdateOptions
|
|
||||||
)
|
|
||||||
.then(processed => {
|
|
||||||
console.error({ processed })
|
|
||||||
process.exit(0)
|
|
||||||
})
|
|
||||||
.catch(error => {
|
|
||||||
console.error({ error })
|
|
||||||
process.exit(1)
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
module.exports = {
|
|
||||||
READ_PREFERENCE_SECONDARY,
|
|
||||||
objectIdFromInput,
|
|
||||||
renderObjectId,
|
|
||||||
batchedUpdate,
|
|
||||||
batchedUpdateWithResultHandling,
|
|
||||||
}
|
|
|
@ -1,10 +0,0 @@
|
||||||
mongo-utils
|
|
||||||
--dependencies=None
|
|
||||||
--docker-repos=gcr.io/overleaf-ops
|
|
||||||
--env-add=
|
|
||||||
--env-pass-through=
|
|
||||||
--esmock-loader=False
|
|
||||||
--is-library=True
|
|
||||||
--node-version=20.18.2
|
|
||||||
--public-repo=False
|
|
||||||
--script-version=4.5.0
|
|
|
@ -1,30 +0,0 @@
|
||||||
{
|
|
||||||
"name": "@overleaf/mongo-utils",
|
|
||||||
"version": "0.0.1",
|
|
||||||
"description": "utilities to help working with mongo",
|
|
||||||
"main": "index.js",
|
|
||||||
"scripts": {
|
|
||||||
"test": "npm run lint && npm run format && npm run types:check && npm run test:unit",
|
|
||||||
"test:unit": "mocha --exit test/**/*.{js,cjs}",
|
|
||||||
"lint": "eslint --ext .js --ext .cjs --ext .ts --max-warnings 0 --format unix .",
|
|
||||||
"lint:fix": "eslint --fix --ext .js --ext .cjs --ext .ts .",
|
|
||||||
"format": "prettier --list-different $PWD/'**/*.{js,cjs,ts}'",
|
|
||||||
"format:fix": "prettier --write $PWD/'**/*.{js,cjs,ts}'",
|
|
||||||
"test:ci": "npm run test:unit",
|
|
||||||
"types:check": "tsc --noEmit"
|
|
||||||
},
|
|
||||||
"author": "Overleaf (https://www.overleaf.com)",
|
|
||||||
"license": "AGPL-3.0-only",
|
|
||||||
"dependencies": {
|
|
||||||
"mongodb": "6.12.0",
|
|
||||||
"mongodb-legacy": "6.1.3"
|
|
||||||
},
|
|
||||||
"devDependencies": {
|
|
||||||
"chai": "^4.3.6",
|
|
||||||
"mocha": "^11.1.0",
|
|
||||||
"sandboxed-module": "^2.0.4",
|
|
||||||
"sinon": "^9.2.4",
|
|
||||||
"sinon-chai": "^3.7.0",
|
|
||||||
"typescript": "^5.0.4"
|
|
||||||
}
|
|
||||||
}
|
|
|
@ -1,11 +0,0 @@
|
||||||
const chai = require('chai')
|
|
||||||
const sinonChai = require('sinon-chai')
|
|
||||||
const SandboxedModule = require('sandboxed-module')
|
|
||||||
|
|
||||||
// Chai configuration
|
|
||||||
chai.should()
|
|
||||||
chai.use(sinonChai)
|
|
||||||
|
|
||||||
SandboxedModule.configure({
|
|
||||||
globals: { Buffer, JSON, console, process },
|
|
||||||
})
|
|
|
@ -1,7 +0,0 @@
|
||||||
{
|
|
||||||
"extends": "../../tsconfig.backend.json",
|
|
||||||
"include": [
|
|
||||||
"**/*.js",
|
|
||||||
"**/*.cjs"
|
|
||||||
]
|
|
||||||
}
|
|
|
@ -1,9 +0,0 @@
|
||||||
root = true
|
|
||||||
|
|
||||||
[*]
|
|
||||||
charset = utf-8
|
|
||||||
indent_style = space
|
|
||||||
indent_size = 2
|
|
||||||
end_of_line = lf
|
|
||||||
insert_final_newline = true
|
|
||||||
trim_trailing_whitespace = true
|
|
Some files were not shown because too many files have changed in this diff Show more
Loading…
Add table
Add a link
Reference in a new issue