This updates the repo index/file view endpoints so annex files match the way
LFS files are rendered, making annexed files accessible via the web instead of
being black boxes only accessible by git clone.
This mostly just duplicates the existing LFS logic. It doesn't try to combine itself
with the existing logic, to make merging with upstream easier. If upstream ever
decides to accept, I would like to try to merge the redundant logic.
The one bit that doesn't directly copy LFS is my choice to hide annex-symlinks.
LFS files are always _pointer files_ and therefore always render with the "file"
icon and no special label, but annex files come in two flavours: symlinks or
pointer files. I've conflated both kinds to try to give a consistent experience.
The tests in here ensure the correct download link (/media, from the last PR)
renders in both the toolbar and, if a binary file (like most annexed files will be),
in the main pane, but it also adds quite a bit of code to make sure text files
that happen to be annexed are dug out and rendered inline like LFS files are.
Previously, Gitea's LFS support allowed direct-downloads of LFS content,
via http://$HOSTNAME:$PORT/$USER/$REPO/media/branch/$BRANCH/$FILE
Expand that grace to git-annex too. Now /media should provide the
relevant *content* from the .git/annex/objects/ folder.
This adds tests too. And expands the tests to try symlink-based annexing,
since /media implicitly supports both that and pointer-file-based annexing.
This moves the `annexObjectPath()` helper out of the tests and into a
dedicated sub-package as `annex.ContentLocation()`, and expands it with
`.Pointer()` (which validates using `git annex examinekey`),
`.IsAnnexed()` and `.Content()` to make it a more useful module.
The tests retain their own wrapper version of `ContentLocation()`
because I tried to follow close to the API modules/lfs uses, which in
terms of abstract `git.Blob` and `git.TreeEntry` objects, not in terms
of `repoPath string`s which are more convenient for the tests.
This makes HTTP symmetric with SSH clone URLs.
This gives us the fancy feature of _anonymous_ downloads,
so people can access datasets without having to set up an
account or manage ssh keys.
Previously, to access "open access" data shared this way,
users would need to:
1. Create an account on gitea.example.com
2. Create ssh keys
3. Upload ssh keys (and make sure to find and upload the correct file)
4. `git clone git@gitea.example.com:user/dataset.git`
5. `cd dataset`
6. `git annex get`
This cuts that down to just the last three steps:
1. `git clone https://gitea.example.com/user/dataset.git`
2. `cd dataset`
3. `git annex get`
This is significantly simpler for downstream users, especially for those
unfamiliar with the command line.
Unfortunately there's no uploading. While git-annex supports uploading
over HTTP to S3 and some other special remotes, it seems to fail on a
_plain_ HTTP remote. See https://github.com/neuropoly/gitea/issues/7
and https://git-annex.branchable.com/forum/HTTP_uploads/#comment-ce28adc128fdefe4c4c49628174d9b92.
This is not a major loss since no one wants uploading to be anonymous anyway.
To support private repos, I had to hunt down and patch a secret extra security
corner that Gitea only applies to HTTP for some reason (services/auth/basic.go).
This was guided by https://git-annex.branchable.com/tips/setup_a_public_repository_on_a_web_site/
Fixes https://github.com/neuropoly/gitea/issues/3
Co-authored-by: Mathieu Guay-Paquet <mathieu.guaypaquet@polymtl.ca>
Fixes https://github.com/neuropoly/gitea/issues/11
Tests:
* `git annex init`
* `git annex copy --from origin`
* `git annex copy --to origin`
over:
* ssh
for:
* the owner
* a collaborator
* a read-only collaborator
* a stranger
in a
* public repo
* private repo
And then confirms:
* Deletion of the remote repo (to ensure lockdown isn't messing with us: https://git-annex.branchable.com/internals/lockdown/#comment-0cc5225dc5abe8eddeb843bfd2fdc382)
------
To support all this:
* Add util.FileCmp()
* Patch withKeyFile() so it can be nested in other copies of itself
-------
Many thanks to Mathieu for giving style tips and catching several bugs,
including a subtle one in util.filecmp() which neutered it.
Co-authored-by: Mathieu Guay-Paquet <mathieu.guay-paquet@polymtl.ca>
[git-annex](https://git-annex.branchable.com/) is a more complicated cousin to
git-lfs, storing large files in an optional-download side content. Unlike lfs,
it allows mixing and matching storage remotes, so the content remote(s) doesn't
need to be on the same server as the git remote, making it feasible to scatter
a collection across cloud storage, old harddrives, or anywhere else storage can
be scavenged. Since this can get complicated, fast, it has a content-tracking
database (`git annex whereis`) to help find everything later.
The use-case we imagine for including it in Gitea is just the simple case, where
we're primarily emulating git-lfs: each repo has its large content at the same URL.
Our motivation is so we can self-host https://www.datalad.org/ datasets, which
currently are only hostable by fragilely scrounging together cloud storage --
and having to manage all the credentials associated with all the pieces -- or at
https://openneuro.org which is fragile in its own ways.
Supporting git-annex also allows multiple Gitea instance to be annex remotes for
each other, mirroring the content or otherwise collaborating the split up the
hosting costs.
Enabling
--------
TODO
HTTP
----
TODO
Permission Checking
-------------------
This tweaks the API in routers/private/serv.go to expose the calling user's
computed permission, instead of just returning HTTP 403.
This doesn't fit in super well. It's the opposite from how the git-lfs support is
done, where there's a complete list of possible subcommands and their matching
permission levels, and then the API compares the requested with the actual level
and returns HTTP 403 if the check fails.
But it's necessary. The main git-annex verbs, 'git-annex-shell configlist' and
'git-annex-shell p2pstdio' are both either read-only or read-write operations,
depending on the state on disk on either end of the connection and what the user
asked it to ask for, with no way to know before git-annex examines the situation.
So tell the level via GIT_ANNEX_READONLY and trust it to handle itself.
In the older Gogs version, the permission was directly read in cmd/serv.go:
```
mode, err = db.UserAccessMode(user.ID, repo)
```
- 966e925cf3/internal/cmd/serv.go (L334)
but in Gitea permission enforcement has been centralized in the API layer.
(perhaps so the cmd layer can avoid making direct DB connections?)
Deletion
--------
git-annex has this "lockdown" feature where it tries
really quite very hard to prevent you deleting its
data, to the point that even an rm -rf won't do it:
each file in annex/objects/ is nested inside a
folder with read-only permissions.
The recommended workaround is to run chmod -R +w when
you're sure you actually want to delete a repo. See
https://git-annex.branchable.com/internals/lockdown
So we edit util.RemoveAll() to do just that, so now
it's `chmod -R +w && rm -rf` instead of just `rm -rf`.
Backport #28576 by wxiaoguang
Regression of #28454 . Now the string is escaped HTML, so it doesn't
need `| Safe`.
Fix#28575
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
(cherry picked from commit acc8100d47bab3b13311bcf4c330b19e2153d4ca)
Backport #28555 by @fuxiaohei
Related to https://github.com/go-gitea/gitea/issues/28279
When merging artifact chunks, it lists chunks from storage. When storage
is minio, chunk's path contains `MINIO_BASE_PATH` that makes merging
break.
<del>So trim the `MINIO_BASE_PATH` when handle chunks.</del>
Update the chunk file's basename to retain necessary information. It
ensures that the directory in the chunk's path remains unaffected.
Co-authored-by: FuXiaoHei <fuxiaohei@vip.qq.com>
(cherry picked from commit 8ca32dc8733c0dad147cc7f734097dbe1113e9a9)
Backport #28552 by @6543
can we please PLEAS PLEASE only use raw SQL statements if it is relay
needed!!!
source is https://github.com/go-gitea/gitea/pull/28544 (before
refactoring)
Co-authored-by: 6543 <m.huber@kithara.com>
(cherry picked from commit 16263af9715afcfa249cf43157b2ae9b4d4765f4)
Backport #28520 by @framitdavid
There is an accessibility issue in the interface when attempting to
delete a repository. When I click on "Delete repository," a dialog box
appears, requiring confirmation to proceed with the repository deletion.
However, when I press the "Repo name" label, the wrong input field gains
focus. The focused field is located behind the dialog and is intended
for renaming the repository.
I am submitting these pull requests to ensure that the correct input
field is focused when the user clicks on the label. This change will
also facilitate the writing of tests using Playwright or Testing Library
to retrieve elements based on roles. This PR will also improve
acessibility of this area.
Co-authored-by: David Øvrelid <46874830+framitdavid@users.noreply.github.com>
(cherry picked from commit 932e282e153ca30f6afaac738469137b761cf904)
Backport #28491 by @appleboy
- Modify the `Password` field in `CreateUserOption` struct to remove the
`Required` tag
- Update the `v1_json.tmpl` template to include the `email` field and
remove the `password` field
Signed-off-by: Bo-Yi Wu <appleboy.tw@gmail.com>
Co-authored-by: Bo-Yi Wu <appleboy.tw@gmail.com>
(cherry picked from commit 411310d698e86bd639b31f2f5a8b856365b4590f)
Backport #28454 (the only conflict is caused by some comments)
* Close#24483
* Close#28123
* Close#23682
* Close#23149
(cherry picked from commit a3f403f438e7f5b5dca3a5042fae8e68a896b1e7)
Conflicts:
modules/setting/ui.go
trivial context conflict
Backport #28487 by @earl-warren
- When a repository is orphaned and has objects stored in any of the
storages such as repository avatar or attachments the delete function
would error, because the storage module wasn't initalized.
- Add code to initialize the storage module.
Refs: https://codeberg.org/forgejo/forgejo/pulls/1954
Co-authored-by: Earl Warren <109468362+earl-warren@users.noreply.github.com>
Co-authored-by: Gusted <postmaster@gusted.xyz>
(cherry picked from commit 8ee1ed877b0207d9d8733ac32270325c54659909)
Backport #28421 by wxiaoguang
Refactor the code and add tests, keep the old logic.
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
(cherry picked from commit 6cbb6f303a602e566f80b0e82ad73a333614efd5)
Backport #28441 by wxiaoguang
Fix#28319
It only polyfills if there is no "SubmitEvent" class, so it has no side
effect for most users.
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
(cherry picked from commit 6af698fb81b9928c5d88c6bbca4b57229db1c2f3)
Backport #28392 by @nekrondev
Windows-based shells will add a CRLF when piping the token into
ssh-keygen command resulting in
verification error. This resolves#21527.
Co-authored-by: nekrondev <heiko@noordsee.de>
Co-authored-by: Heiko Besemann <heiko.besemann@qbeyond.de>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
(cherry picked from commit b47482d58e86c636006f7b17b4d91786e6ed4d37)
Backport #28457 by @KN4CK3R
Recently Docker started to use the optional `POST /v2/token` endpoint
which should respond with a `404 Not Found` status code instead of the
current `405 Method Not Allowed`.
> Note: Not all token servers implement oauth2. If the request to the
endpoint returns 404 using the HTTP POST method, refer to Token
Documentation for using the HTTP GET method supported by all token
servers.
Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
(cherry picked from commit 74ab7980339da50e8a42db00677b457d5f340a5a)
Backport #28428 by @KN4CK3R
There could be a nil pointer exception if the file is not found because
that specific error is suppressed but not handled.
Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
(cherry picked from commit 97a0bf151a1fed2119b0ecb3c94f68462979c699)
Since a scheduled action is only run from the default branch, it
cannot be anything else.
Refs: https://codeberg.org/forgejo/forgejo/issues/1926
(cherry picked from commit eff0822856fd727915f6e6493a80844cffd7b02a)
- The transaction in combination with Git push was causing deadlocks if
you had the `push_update` queue set to `immediate`. This was the root
cause of slow integration tests in CI.
- Remove the sync branch code as this is already being done in the Git
post-receive hook.
- Add tests to proof the branch models are in sync even with this code
removed.
Backport of https://codeberg.org/forgejo/forgejo/pulls/1962
(cherry picked from commit a064065cb9a6e39597e38c37a405d066cfabf7f7)
This reverts commit 172fdd0d32.
This code was introduced for a test case that is no longer in use. It
should guard against the provider being null but that's not worth the
effort for deadcode. Just remove it.
Refs: https://codeberg.org/forgejo/forgejo/issues/1984
It will determine how anchors are created and will break existing
links otherwise.
Adapted from Revert "Make `user-content-* ` consistent with github (#26388)
It shows warnings although the setting is not set, this will surely be
fixed later but there is no sense in spaming the users right now. This
revert can be discarded when another fix lands in v1.21.
su -c "forgejo admin user generate-access-token -u root --raw --scopes 'all,sudo'" git
2023/12/12 15:54:45 .../setting/security.go:166:loadSecurityFrom() [W] Enabling Query API Auth tokens is not recommended. DISABLE_QUERY_AUTH_TOKEN will default to true in gitea 1.23 and will be removed in gitea 1.24.
This reverts commit 0e3a5abb69.
Conflicts:
routers/api/v1/api.go
To be rebuilt with latest golang version
---------
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
(cherry picked from commit 96d3fcf179d4eea98f62d380e04f246ff65694d7)
Backport #28390 by @jackHay22
## Changes
- Add deprecation warning to `Token` and `AccessToken` authentication
methods in swagger.
- Add deprecation warning header to API response. Example:
```
HTTP/1.1 200 OK
...
Warning: token and access_token API authentication is deprecated
...
```
- Add setting `DISABLE_QUERY_AUTH_TOKEN` to reject query string auth
tokens entirely. Default is `false`
## Next steps
- `DISABLE_QUERY_AUTH_TOKEN` should be true in a subsequent release and
the methods should be removed in swagger
- `DISABLE_QUERY_AUTH_TOKEN` should be removed and the implementation of
the auth methods in question should be removed
## Open questions
- Should there be further changes to the swagger documentation?
Deprecation is not yet supported for security definitions (coming in
[OpenAPI Spec version
3.2.0](https://github.com/OAI/OpenAPI-Specification/issues/2506))
- Should the API router logger sanitize urls that use `token` or
`access_token`? (This is obviously an insufficient solution on its own)
Co-authored-by: Jack Hay <jack@allspice.io>
Co-authored-by: delvh <dev.lh@web.de>
(cherry picked from commit f144521aea0d7a08b9bd5f17e49bae4021bd7a45)
Backport #28302 by @yp05327
Close#28287
## How to test it in local
convert Makefile L34 into:
```
cd .tmp/upstream-docs && git clean -f && git reset --hard && git fetch origin pull/28302/head:pr28302 && git switch pr28302
```
Co-authored-by: yp05327 <576951401@qq.com>
(cherry picked from commit 40d51188c039f6a674a37e67d0ea5504a7a3e282)
