This updates the repo index/file view endpoints so annex files match the way
LFS files are rendered, making annexed files accessible via the web instead of
being black boxes only accessible by git clone.
This mostly just duplicates the existing LFS logic. It doesn't try to combine itself
with the existing logic, to make merging with upstream easier. If upstream ever
decides to accept, I would like to try to merge the redundant logic.
The one bit that doesn't directly copy LFS is my choice to hide annex-symlinks.
LFS files are always _pointer files_ and therefore always render with the "file"
icon and no special label, but annex files come in two flavours: symlinks or
pointer files. I've conflated both kinds to try to give a consistent experience.
The tests in here ensure the correct download link (/media, from the last PR)
renders in both the toolbar and, if a binary file (like most annexed files will be),
in the main pane, but it also adds quite a bit of code to make sure text files
that happen to be annexed are dug out and rendered inline like LFS files are.
Previously, Gitea's LFS support allowed direct-downloads of LFS content,
via http://$HOSTNAME:$PORT/$USER/$REPO/media/branch/$BRANCH/$FILE
Expand that grace to git-annex too. Now /media should provide the
relevant *content* from the .git/annex/objects/ folder.
This adds tests too. And expands the tests to try symlink-based annexing,
since /media implicitly supports both that and pointer-file-based annexing.
This makes HTTP symmetric with SSH clone URLs.
This gives us the fancy feature of _anonymous_ downloads,
so people can access datasets without having to set up an
account or manage ssh keys.
Previously, to access "open access" data shared this way,
users would need to:
1. Create an account on gitea.example.com
2. Create ssh keys
3. Upload ssh keys (and make sure to find and upload the correct file)
4. `git clone git@gitea.example.com:user/dataset.git`
5. `cd dataset`
6. `git annex get`
This cuts that down to just the last three steps:
1. `git clone https://gitea.example.com/user/dataset.git`
2. `cd dataset`
3. `git annex get`
This is significantly simpler for downstream users, especially for those
unfamiliar with the command line.
Unfortunately there's no uploading. While git-annex supports uploading
over HTTP to S3 and some other special remotes, it seems to fail on a
_plain_ HTTP remote. See https://github.com/neuropoly/gitea/issues/7
and https://git-annex.branchable.com/forum/HTTP_uploads/#comment-ce28adc128fdefe4c4c49628174d9b92.
This is not a major loss since no one wants uploading to be anonymous anyway.
To support private repos, I had to hunt down and patch a secret extra security
corner that Gitea only applies to HTTP for some reason (services/auth/basic.go).
This was guided by https://git-annex.branchable.com/tips/setup_a_public_repository_on_a_web_site/
Fixes https://github.com/neuropoly/gitea/issues/3
Co-authored-by: Mathieu Guay-Paquet <mathieu.guaypaquet@polymtl.ca>
[git-annex](https://git-annex.branchable.com/) is a more complicated cousin to
git-lfs, storing large files in an optional-download side content. Unlike lfs,
it allows mixing and matching storage remotes, so the content remote(s) doesn't
need to be on the same server as the git remote, making it feasible to scatter
a collection across cloud storage, old harddrives, or anywhere else storage can
be scavenged. Since this can get complicated, fast, it has a content-tracking
database (`git annex whereis`) to help find everything later.
The use-case we imagine for including it in Gitea is just the simple case, where
we're primarily emulating git-lfs: each repo has its large content at the same URL.
Our motivation is so we can self-host https://www.datalad.org/ datasets, which
currently are only hostable by fragilely scrounging together cloud storage --
and having to manage all the credentials associated with all the pieces -- or at
https://openneuro.org which is fragile in its own ways.
Supporting git-annex also allows multiple Gitea instance to be annex remotes for
each other, mirroring the content or otherwise collaborating the split up the
hosting costs.
Enabling
--------
TODO
HTTP
----
TODO
Permission Checking
-------------------
This tweaks the API in routers/private/serv.go to expose the calling user's
computed permission, instead of just returning HTTP 403.
This doesn't fit in super well. It's the opposite from how the git-lfs support is
done, where there's a complete list of possible subcommands and their matching
permission levels, and then the API compares the requested with the actual level
and returns HTTP 403 if the check fails.
But it's necessary. The main git-annex verbs, 'git-annex-shell configlist' and
'git-annex-shell p2pstdio' are both either read-only or read-write operations,
depending on the state on disk on either end of the connection and what the user
asked it to ask for, with no way to know before git-annex examines the situation.
So tell the level via GIT_ANNEX_READONLY and trust it to handle itself.
In the older Gogs version, the permission was directly read in cmd/serv.go:
```
mode, err = db.UserAccessMode(user.ID, repo)
```
- 966e925cf3/internal/cmd/serv.go (L334)
but in Gitea permission enforcement has been centralized in the API layer.
(perhaps so the cmd layer can avoid making direct DB connections?)
Deletion
--------
git-annex has this "lockdown" feature where it tries
really quite very hard to prevent you deleting its
data, to the point that even an rm -rf won't do it:
each file in annex/objects/ is nested inside a
folder with read-only permissions.
The recommended workaround is to run chmod -R +w when
you're sure you actually want to delete a repo. See
https://git-annex.branchable.com/internals/lockdown
So we edit util.RemoveAll() to do just that, so now
it's `chmod -R +w && rm -rf` instead of just `rm -rf`.
Backport #29105
`ctx.Error` only displays the text but `ctx.ServerError` renders the
usual error page.
Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
(cherry picked from commit a1c0b3a02e54e5fe879dabccb71fba9498b64051)
Backport #29347 by @carlosfelgueiras
Fixes#27188.
Introduces a check on the installation that tries to parse the FROM
address. If it fails, shows a new error message to the user.
Co-authored-by: Carlos Felgueiras <carlosfelgueiras@tecnico.ulisboa.pt>
Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
(cherry picked from commit 35db5a373babd9af157fd63eeb20d6da53320b73)
Follow-up of #2282 and #2296 (which tried to address #2278)
One of the issue with the previous PR is that when a conversation on the Files tab was marked as "resolved", it would fetch all the comments for that line (even the outdated ones, which should not be shown on this page - except when explicitly activated).
To properly fix this, I have changed `FetchCodeCommentsByLine` to `FetchCodeConversation`. Its role is to fetch all comments related to a given (review, path, line) and reverted my changes in the template (which were based on a misunderstanding).
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/2306
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Reviewed-by: Gusted <gusted@noreply.codeberg.org>
Co-authored-by: oliverpool <git@olivier.pfad.fr>
Co-committed-by: oliverpool <git@olivier.pfad.fr>
Backport #28935 by @silverwind
The `ToUTF8*` functions were stripping BOM, while BOM is actually valid
in UTF8, so the stripping must be optional depending on use case. This
does:
- Add a options struct to all `ToUTF8*` functions, that by default will
strip BOM to preserve existing behaviour
- Remove `ToUTF8` function, it was dead code
- Rename `ToUTF8WithErr` to `ToUTF8`
- Preserve BOM in Monaco Editor
- Remove a unnecessary newline in the textarea value. Browsers did
ignore it, it seems but it's better not to rely on this behaviour.
Fixes: https://github.com/go-gitea/gitea/issues/28743
Related: https://github.com/go-gitea/gitea/issues/6716 which seems to
have once introduced a mechanism that strips and re-adds the BOM, but
from what I can tell, this mechanism was removed at some point after
that PR.
Co-authored-by: silverwind <me@silverwind.io>
(cherry picked from commit b8e6cffd317401d980600e339eb21b15b9bc64c1)
Backport #28905 by @Anthony-Jhoiro
Currently, the `updateMirror` function which update the mirror interval
and enable prune properties is only executed by the `Edit` function. But
it is only triggered if `opts.MirrorInterval` is not null, even if
`opts.EnablePrune` is not null.
With this patch, it is now possible to update the enable_prune property
with a patch request without modifying the mirror_interval.
## Example request with httpie
### Currently:
**Does nothing**
```bash
http PATCH https://gitea.your-server/api/v1/repos/myOrg/myRepo "enable_prune:=false" -A bearer -a $gitea_token
```
**Updates both properties**
```bash
http PATCH https://gitea.your-server/api/v1/repos/myOrg/myRepo "enable_prune:=false" "mirror_interval=10m" -A bearer -a $gitea_token
```
### With the patch
**Updates enable_prune only**
```bash
http PATCH https://gitea.your-server/api/v1/repos/myOrg/myRepo "enable_prune:=false" -A bearer -a $gitea_token
```
Co-authored-by: Anthony Quéré <47711333+Anthony-Jhoiro@users.noreply.github.com>
(cherry picked from commit d6eb6c90f45ebe93709b90691be73ba5c31e843e)
Backport https://github.com/go-gitea/gitea/pull/28726 by @fuxiaohei
Fix Uploaded artifacts should be overwritten
https://github.com/go-gitea/gitea/issues/28549
When upload different content to uploaded artifact, it checks that
content size is not match in db record with previous artifact size, then
the new artifact is refused.
Now if it finds uploading content size is not matching db record when
receiving chunks, it updates db records to follow the latest size value.
(cherry picked from commit 7f0ce2dfc7f4a0c50f6895f6d478f5230089f1c7)
Backport #28877 by @KN4CK3R
Fixes#28875
If `RequireSignInView` is enabled, the ghost user has no access rights.
Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
(cherry picked from commit b7c944b9e4e9f847719fbce421b2f4fee7281187)
- Backport of #1981
- When the user is not found in `reloadparam`, early return when the
user is not found to avoid calling `IsUserVisibleToViewer` which in turn
avoids causing a NPE.
- This fixes the case that a 500 error and 404 error is shown on the
same page.
- Add integration test for non-existant user RSS.
- Regression by c6366089df
(cherry picked from commit f0e06962786ef8c417b0c6f07940c1909d3b91ba)
(cherry picked from commit 75d806690875a4fc38eb1e3c904096be34657011)
(cherry picked from commit 4d0a1e0637450865c7bbac69e42d92d63b95149c)
(cherry picked from commit 5f40a485da1b2c5f129f32e2ddc2065e3ba9ccd0)
(cherry picked from commit c4cb7812e39add6f7ff3d6f3f2d4e02c66435f0e)
Backport #26745Fixes#26548
This PR refactors the rendering of markup links. The old code uses
`strings.Replace` to change some urls while the new code uses more
context to decide which link should be generated.
The added tests should ensure the same output for the old and new
behaviour (besides the bug).
We may need to refactor the rendering a bit more to make it clear how
the different helper methods render the input string. There are lots of
options (resolve links / images / mentions / git hashes / emojis / ...)
but you don't really know what helper uses which options. For example,
we currently support images in the user description which should not be
allowed I think:
<details>
<summary>Profile</summary>
https://try.gitea.io/KN4CK3R
</details>
(cherry picked from commit 022552d5b6adc792d3cd16df7de6e52cb7b41a72)
Backport #28140 by @earl-warren
- Make use of the `form-fetch-action` for the merge button, which will
automatically prevent the action from happening multiple times and show
a nice loading indicator as user feedback while the merge request is
being processed by the server.
- Adjust the merge PR code to JSON response as this is required for the
`form-fetch-action` functionality.
- Resolves https://codeberg.org/forgejo/forgejo/issues/774
- Likely resolves the cause of
https://codeberg.org/forgejo/forgejo/issues/1688#issuecomment-1313044
(cherry picked from commit 4ec64c19507caefff7ddaad722b1b5792b97cc5a)
Co-authored-by: Earl Warren <109468362+earl-warren@users.noreply.github.com>
Co-authored-by: Gusted <postmaster@gusted.xyz>
(cherry picked from commit fbf29f29b5225be8e5e682e45b6977e7dda9b318)
Backport #28227 by @denyskon
When creating a pull request, allow directly assigning it to a project,
as it is already possible for issues.
After:
---------
Co-authored-by: Denys Konovalov <kontakt@denyskon.de>
Co-authored-by: delvh <dev.lh@web.de>
(cherry picked from commit 80e564087dc1a89b8b74abcdc879c2de444d3278)
Backport #27736 by @lng2020
Fix#27722Fix#27357Fix#25837Fix#28732
1. Fix the typo `BlockingByDependenciesNotPermitted`, which causes the
`not permitted message` not to show. The correct one is `Blocking` or
`BlockedBy`
2. Rewrite the perm check. The perm check uses a very tricky way to
avoid duplicate checks for a slice of issues, which is confusing. In
fact, it's also the reason causing the bug. It uses `lastRepoID` and
`lastPerm` to avoid duplicate checks, but forgets to assign the
`lastPerm` at the end of the code block. So I rewrote this to avoid this
trick.
3. It also reuses the `blocks` slice, which is even more confusing. So I
rewrote this too.
Co-authored-by: Nanguan Lin <70063547+lng2020@users.noreply.github.com>
(cherry picked from commit 571822b6eca07832f6b5ff633bec8ab8fdb32624)
Backport #28765 by @jackHay22
Fixes #28756
## Changes
- Require and check API token for `GET
/repos/{owner}/{repo}/subscription` in order to populate `ctx.Doer`.
Co-authored-by: Jack Hay <jack@allspice.io>
(cherry picked from commit cb33623bb6b0cf60417c4a1d84b8f42ef1714ea3)
Backport #28531 by @pulltheflower
- Fix the bug about admin/hooks API that `GET /admin/hooks` can only
fetch system_hooks, `POST /admin/hooks` can only create default_hooks.
Co-authored-by: vincent <38434877+pulltheflower@users.noreply.github.com>
(cherry picked from commit 3c79315cf2308e68caba5bd7f2f94aa3472d9425)
- The current architecture is inherently insecure, because you can
construct the 'secret' cookie value with values that are available in
the database. Thus provides zero protection when a database is
dumped/leaked.
- This patch implements a new architecture that's inspired from: [Paragonie Initiative](https://paragonie.com/blog/2015/04/secure-authentication-php-with-long-term-persistence#secure-remember-me-cookies).
- Integration testing is added to ensure the new mechanism works.
- Removes a setting, because it's not used anymore.
(cherry picked from commit eff097448b1ebd2a280fcdd55d10b1f6081e9ccd)
[GITEA] rework long-term authentication (squash) add migration
Reminder: the migration is run via integration tests as explained
in the commit "[DB] run all Forgejo migrations in integration tests"
(cherry picked from commit 4accf7443c1c59b4d2e7787d6a6c602d725da403)
(cherry picked from commit 99d06e344ebc3b50bafb2ac4473dd95f057d1ddc)
(cherry picked from commit d8bc98a8f021d381bf72790ad246f923ac983ad4)
(cherry picked from commit 6404845df9a63802fff4c5bd6cfe1e390076e7f0)
(cherry picked from commit 72bdd4f3b9f6509d1ff3f10ecb12c621a932ed30)
(cherry picked from commit 4b01bb0ce812b6c59414ff53fed728563d8bc9cc)
(cherry picked from commit c26ac318162b2cad6ff1ae54e2d8f47a4e4fe7c2)
(cherry picked from commit 8d2dab94a6)
Conflicts:
routers/web/auth/auth.go
https://codeberg.org/forgejo/forgejo/issues/2158
- https://github.com/NYTimes/gziphandler doesn't seems to be maintained
anymore and Forgejo already includes
https://github.com/klauspost/compress which provides a maintained and
faster gzip handler fork.
- Enables Jitter to prevent BREACH attacks, as this *seems* to be
possible in the context of Forgejo.
(cherry picked from commit cc2847241d82001babd8d40c87d03169f21c14cd)
(cherry picked from commit 99ba56a8761dd08e08d9499cab2ded1a6b7b970f)
Conflicts:
go.sum
https://codeberg.org/forgejo/forgejo/pulls/1581
(cherry picked from commit 711638193daa2311e2ead6249a47dcec47b4e335)
(cherry picked from commit 9c12a37fde6fa84414bf332ff4a066facdb92d38)
(cherry picked from commit 91191aaaedaf999209695e2c6ca4fb256b396686)
(cherry picked from commit 72be417f844713265a94ced6951f8f4b81d0ab1a)
(cherry picked from commit 98497c84da205ec59079e42274aa61199444f7cd)
(cherry picked from commit fba042adb5c1abcbd8eee6b5a4f735ccb2a5e394)
(cherry picked from commit dd2414f226)
Conflicts:
routers/web/web.go
https://codeberg.org/forgejo/forgejo/issues/2016
Backport #28587, the only conflict is the test file.
The CORS code has been unmaintained for long time, and the behavior is
not correct.
This PR tries to improve it. The key point is written as comment in
code. And add more tests.
Fix#28515Fix#27642Fix#17098
(cherry picked from commit 7a2786ca6cd84633784a2c9986da65a9c4d79c78)
Backport #28583 by @morphelinho
Follow #28184
Follow #28515
Fix problem with 405 method not allowed for CORS wrt OIDC
Co-authored-by: morphelinho <morphelinho@users.noreply.github.com>
(cherry picked from commit 4dd39eb54aed03e2939b4fda6ab683756d878f04)
- Backport of #2134
- It's possible that `canSoftDeleteContentHistory` is called without
`ctx.Doer` being set, such as an anonymous user requesting the
`/content-history/detail` endpoint.
- Add a simple condition to always set to `canSoftDelete` to false if an
anonymous user is requesting this, this avoids a panic in the code that
assumes `ctx.Doer` is set.
- Added integration testing.
(cherry picked from commit 0b5db0dcc608e9a9e79ead094a20a7775c4f9559)
- When a user requests a archive of a non-existant commit
`git.ErrNotExist` is returned, but was not gracefully handled resulting
in a 500 error.
- Doesn't exist in v1.22 due to it being refactored away in
cbf923e87b
- Adds integration test.
Backport #28555 by @fuxiaohei
Related to https://github.com/go-gitea/gitea/issues/28279
When merging artifact chunks, it lists chunks from storage. When storage
is minio, chunk's path contains `MINIO_BASE_PATH` that makes merging
break.
<del>So trim the `MINIO_BASE_PATH` when handle chunks.</del>
Update the chunk file's basename to retain necessary information. It
ensures that the directory in the chunk's path remains unaffected.
Co-authored-by: FuXiaoHei <fuxiaohei@vip.qq.com>
(cherry picked from commit 8ca32dc8733c0dad147cc7f734097dbe1113e9a9)
Backport #28491 by @appleboy
- Modify the `Password` field in `CreateUserOption` struct to remove the
`Required` tag
- Update the `v1_json.tmpl` template to include the `email` field and
remove the `password` field
Signed-off-by: Bo-Yi Wu <appleboy.tw@gmail.com>
Co-authored-by: Bo-Yi Wu <appleboy.tw@gmail.com>
(cherry picked from commit 411310d698e86bd639b31f2f5a8b856365b4590f)
Backport #28454 (the only conflict is caused by some comments)
* Close#24483
* Close#28123
* Close#23682
* Close#23149
(cherry picked from commit a3f403f438e7f5b5dca3a5042fae8e68a896b1e7)
Conflicts:
modules/setting/ui.go
trivial context conflict
Backport #28457 by @KN4CK3R
Recently Docker started to use the optional `POST /v2/token` endpoint
which should respond with a `404 Not Found` status code instead of the
current `405 Method Not Allowed`.
> Note: Not all token servers implement oauth2. If the request to the
endpoint returns 404 using the HTTP POST method, refer to Token
Documentation for using the HTTP GET method supported by all token
servers.
Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
(cherry picked from commit 74ab7980339da50e8a42db00677b457d5f340a5a)
It shows warnings although the setting is not set, this will surely be
fixed later but there is no sense in spaming the users right now. This
revert can be discarded when another fix lands in v1.21.
su -c "forgejo admin user generate-access-token -u root --raw --scopes 'all,sudo'" git
2023/12/12 15:54:45 .../setting/security.go:166:loadSecurityFrom() [W] Enabling Query API Auth tokens is not recommended. DISABLE_QUERY_AUTH_TOKEN will default to true in gitea 1.23 and will be removed in gitea 1.24.
This reverts commit 0e3a5abb69.
Conflicts:
routers/api/v1/api.go
Backport #28390 by @jackHay22
## Changes
- Add deprecation warning to `Token` and `AccessToken` authentication
methods in swagger.
- Add deprecation warning header to API response. Example:
```
HTTP/1.1 200 OK
...
Warning: token and access_token API authentication is deprecated
...
```
- Add setting `DISABLE_QUERY_AUTH_TOKEN` to reject query string auth
tokens entirely. Default is `false`
## Next steps
- `DISABLE_QUERY_AUTH_TOKEN` should be true in a subsequent release and
the methods should be removed in swagger
- `DISABLE_QUERY_AUTH_TOKEN` should be removed and the implementation of
the auth methods in question should be removed
## Open questions
- Should there be further changes to the swagger documentation?
Deprecation is not yet supported for security definitions (coming in
[OpenAPI Spec version
3.2.0](https://github.com/OAI/OpenAPI-Specification/issues/2506))
- Should the API router logger sanitize urls that use `token` or
`access_token`? (This is obviously an insufficient solution on its own)
Co-authored-by: Jack Hay <jack@allspice.io>
Co-authored-by: delvh <dev.lh@web.de>
(cherry picked from commit f144521aea0d7a08b9bd5f17e49bae4021bd7a45)
Fix#28056
Backport #28361
This PR will check whether the repo has zero branch when pushing a
branch. If that, it means this repository hasn't been synced.
The reason caused that is after user upgrade from v1.20 -> v1.21, he
just push branches without visit the repository user interface. Because
all repositories routers will check whether a branches sync is necessary
but push has not such check.
For every repository, it has two states, synced or not synced. If there
is zero branch for a repository, then it will be assumed as non-sync
state. Otherwise, it's synced state. So if we think it's synced, we just
need to update branch/insert new branch. Otherwise do a full sync. So
that, for every push, there will be almost no extra load added. It's
high performance than yours.
For the implementation, we in fact will try to update the branch first,
if updated success with affect records > 0, then all are done. Because
that means the branch has been in the database. If no record is
affected, that means the branch does not exist in database. So there are
two possibilities. One is this is a new branch, then we just need to
insert the record. Another is the branches haven't been synced, then we
need to sync all the branches into database.
(cherry picked from commit 87db4a47c8e22b7c2e4f2b9f9efc8df1e3622884)
