Otherwise, git-annex tries to use its own default port (9417) and fails.
Fixes#52.
Reviewed-on: https://codeberg.org/forgejo-aneksajo/forgejo-aneksajo/pulls/55
Co-authored-by: Matthias Riße <m.risse@fz-juelich.de>
Co-committed-by: Matthias Riße <m.risse@fz-juelich.de>
This adds a new endpoint under `/git-annex-p2phttp` which acts as an
authenticating proxy to git-annex' p2phttp server. This makes it
possible to set `annex+<server-url>/git-annex-p2phttp` as
`remote.<name>.annexurl` and use git-annex fully over http(s) with the
normal credentials and access tokens provided by Forgejo.
Fixes#25.
Reviewed-on: https://codeberg.org/matrss/forgejo-aneksajo/pulls/42
Co-authored-by: Matthias Riße <m.risse@fz-juelich.de>
Co-committed-by: Matthias Riße <m.risse@fz-juelich.de>
Git-annex emits HEAD requests for keys while trying to drop them from a
repository that was cloned via http. Forgejo asked for authentication
for these HEAD requests. This meant that cloning and getting files was
possible without authentication, but dropping was not.
Since the response to a HEAD request is a subset of the response to a
GET request it is safe to make those unauthenticated as well. That is
what this change does, although limited to the
:username/:reponame/annex/objects endpoint.
Fixes#40.
Reviewed-on: https://codeberg.org/matrss/forgejo-aneksajo/pulls/41
Co-authored-by: Matthias Riße <m.risse@fz-juelich.de>
Co-committed-by: Matthias Riße <m.risse@fz-juelich.de>
Previously, an external renderer that matched on an annexed file would
only see its content streamed via `STDIN`, or a temporary file with a copy
of its content would be generated and passed-by-filepath (with
`IS_INPUT_FILE=true`). Whether that happens, is also subject to
`MAX_DISPLAY_FILE_SIZE` (which defaults to 8MB).
This was problematic, because annexed files tend to be large. Moreover,
if present, they already exist as write-protected files on the
file-system. Creating a copy is both expensive and serves no particular
purpose.
This commit changes how external renderers are called.
1) With `IS_INPUT_FILE=true`, the renderer is passed the true location
of an annex key, if present, and an empty path, if not.
2) The original, repository-relative path of the rendering target is
made available to the renderer via the `GITEA_RELATIVE_PATH`
environment variable.
To achieve a lean implementation, the `Blob` of the rendering target
is passed on to the `RenderContext` (because the implementation of
the annex-related functionality is centered on this dtype.
This change makes it less costly to increase `MAX_DISPLAY_FILE_SIZE`,
in order to make large, annexed files eligible for markup rendering,
because no content copies will be made any longer.
External renderers can now use the original file path, with the full
original filename, including extensions, for decision making. For
example, to detect particular compression formats based in a file name
extension, or to alter the rendering based on contextual information
encoded in the file path (e.g., a multi-file data structure with a
particular organization pattern).
Apart from the additional environment variable, there is no change to
the handling of renderers that take their input via `STDIN` (i.e.,
`IS_INPUT_FILE=false`).
Fixes#35.
Reviewed-on: https://codeberg.org/matrss/forgejo-aneksajo/pulls/36
Reviewed-by: matrss <matrss@noreply.codeberg.org>
Co-authored-by: Michael Hanke <michael.hanke@gmail.com>
Co-committed-by: Michael Hanke <michael.hanke@gmail.com>
Previously, trying to view files that were annexed, but missing, just
led to an uninformative error 500. This was rather confusing.
With these changes it now shows the pointer target instead of the
(missing) content of the file, and also indicates this situation in the
"stored with git-annex" message. For semantic correctness views for
missing files return a 404 instead of a 200, as they would with the
content present.
Fixes#7, fixes#13.
Reviewed-on: https://codeberg.org/matrss/forgejo-aneksajo/pulls/28
Co-authored-by: Matthias Riße <m.risse@fz-juelich.de>
Co-committed-by: Matthias Riße <m.risse@fz-juelich.de>
This updates the repo index/file view endpoints so annex files match the way
LFS files are rendered, making annexed files accessible via the web instead of
being black boxes only accessible by git clone.
This mostly just duplicates the existing LFS logic. It doesn't try to combine itself
with the existing logic, to make merging with upstream easier. If upstream ever
decides to accept, I would like to try to merge the redundant logic.
The one bit that doesn't directly copy LFS is my choice to hide annex-symlinks.
LFS files are always _pointer files_ and therefore always render with the "file"
icon and no special label, but annex files come in two flavours: symlinks or
pointer files. I've conflated both kinds to try to give a consistent experience.
The tests in here ensure the correct download link (/media, from the last PR)
renders in both the toolbar and, if a binary file (like most annexed files will be),
in the main pane, but it also adds quite a bit of code to make sure text files
that happen to be annexed are dug out and rendered inline like LFS files are.
Previously, Gitea's LFS support allowed direct-downloads of LFS content,
via http://$HOSTNAME:$PORT/$USER/$REPO/media/branch/$BRANCH/$FILE
Expand that grace to git-annex too. Now /media should provide the
relevant *content* from the .git/annex/objects/ folder.
This adds tests too. And expands the tests to try symlink-based annexing,
since /media implicitly supports both that and pointer-file-based annexing.
Usage of `path` was replaced by `path/filepath` in upstream forgejo, and
it made sense to use that as well where `path` was previously used. The
`setHeaderCacheForever` function and the `sendFile` method had their
signature changed.
This makes HTTP symmetric with SSH clone URLs.
This gives us the fancy feature of _anonymous_ downloads,
so people can access datasets without having to set up an
account or manage ssh keys.
Previously, to access "open access" data shared this way,
users would need to:
1. Create an account on gitea.example.com
2. Create ssh keys
3. Upload ssh keys (and make sure to find and upload the correct file)
4. `git clone git@gitea.example.com:user/dataset.git`
5. `cd dataset`
6. `git annex get`
This cuts that down to just the last three steps:
1. `git clone https://gitea.example.com/user/dataset.git`
2. `cd dataset`
3. `git annex get`
This is significantly simpler for downstream users, especially for those
unfamiliar with the command line.
Unfortunately there's no uploading. While git-annex supports uploading
over HTTP to S3 and some other special remotes, it seems to fail on a
_plain_ HTTP remote. See https://github.com/neuropoly/gitea/issues/7
and https://git-annex.branchable.com/forum/HTTP_uploads/#comment-ce28adc128fdefe4c4c49628174d9b92.
This is not a major loss since no one wants uploading to be anonymous anyway.
To support private repos, I had to hunt down and patch a secret extra security
corner that Gitea only applies to HTTP for some reason (services/auth/basic.go).
This was guided by https://git-annex.branchable.com/tips/setup_a_public_repository_on_a_web_site/
Fixes https://github.com/neuropoly/gitea/issues/3
Co-authored-by: Mathieu Guay-Paquet <mathieu.guaypaquet@polymtl.ca>
[git-annex](https://git-annex.branchable.com/) is a more complicated cousin to
git-lfs, storing large files in an optional-download side content. Unlike lfs,
it allows mixing and matching storage remotes, so the content remote(s) doesn't
need to be on the same server as the git remote, making it feasible to scatter
a collection across cloud storage, old harddrives, or anywhere else storage can
be scavenged. Since this can get complicated, fast, it has a content-tracking
database (`git annex whereis`) to help find everything later.
The use-case we imagine for including it in Gitea is just the simple case, where
we're primarily emulating git-lfs: each repo has its large content at the same URL.
Our motivation is so we can self-host https://www.datalad.org/ datasets, which
currently are only hostable by fragilely scrounging together cloud storage --
and having to manage all the credentials associated with all the pieces -- or at
https://openneuro.org which is fragile in its own ways.
Supporting git-annex also allows multiple Gitea instance to be annex remotes for
each other, mirroring the content or otherwise collaborating the split up the
hosting costs.
Enabling
--------
TODO
HTTP
----
TODO
Permission Checking
-------------------
This tweaks the API in routers/private/serv.go to expose the calling user's
computed permission, instead of just returning HTTP 403.
This doesn't fit in super well. It's the opposite from how the git-lfs support is
done, where there's a complete list of possible subcommands and their matching
permission levels, and then the API compares the requested with the actual level
and returns HTTP 403 if the check fails.
But it's necessary. The main git-annex verbs, 'git-annex-shell configlist' and
'git-annex-shell p2pstdio' are both either read-only or read-write operations,
depending on the state on disk on either end of the connection and what the user
asked it to ask for, with no way to know before git-annex examines the situation.
So tell the level via GIT_ANNEX_READONLY and trust it to handle itself.
In the older Gogs version, the permission was directly read in cmd/serv.go:
```
mode, err = db.UserAccessMode(user.ID, repo)
```
- 966e925cf3/internal/cmd/serv.go (L334)
but in Gitea permission enforcement has been centralized in the API layer.
(perhaps so the cmd layer can avoid making direct DB connections?)
Deletion
--------
git-annex has this "lockdown" feature where it tries
really quite very hard to prevent you deleting its
data, to the point that even an rm -rf won't do it:
each file in annex/objects/ is nested inside a
folder with read-only permissions.
The recommended workaround is to run chmod -R +w when
you're sure you actually want to delete a repo. See
https://git-annex.branchable.com/internals/lockdown
So we edit util.RemoveAll() to do just that, so now
it's `chmod -R +w && rm -rf` instead of just `rm -rf`.
- Add a permission check that the doer has write permissions to the head
repository if the the 'delete branch after merge' is enabled when
merging a pull request.
- Unify the checks in the web and API router to `DeleteBranchAfterMerge`.
- Added integration tests.
(cherry picked from commit 266e0b2ce9)
- When a truncated comment is detected in the RSS/Atom feeds, fetch the
comment from the database and use the original content.
- Added integration test.
- Resolves#5650
(cherry picked from commit f4a7132a89)
- On editting a team, only update the units if the team isn't the
'Owners' team. Otherwise the 'Owners' team end up having all of their
unit access modes set to 'None'; because the request form doesn't send
over any units, as it's simply not shown in the UI.
- Adds a database inconstency check and fix for the case where the
'Owners' team is affected by this bug.
- Adds unit test.
- Adds integration test.
- Resolves#5528
- Regression of https://github.com/go-gitea/gitea/pull/24012
(cherry picked from commit 9de9034400)
Port of https://github.com/go-gitea/gitea/pull/32204
(cherry picked from commit d6d3c96e6555fc91b3e2ef21f4d8d7475564bb3e)
Conflicts:
routers/api/v1/api.go
services/context/api.go
trivial context conflicts
(cherry picked from commit a052d2b602)
Conflicts:
routers/api/v1/user/user.go
trivial context conflict (search by email is not in v9.0)
Resolves#20475
(cherry picked from commit 7e68bc88238104d2ee8b5a877fc1ad437f1778a4)
Conflicts:
tests/integration/pull_create_test.go
add missing testPullCreateDirectly from
c63060b130d34e3f03f28f4dccbf04d381a95c17 Fix code owners will not be mentioned when a pull request comes from a forked repository (#30476)
Fix#31423
(cherry picked from commit f4b8f6fc40ce2869135372a5c6ec6418d27ebfba)
Conflicts:
models/fixtures/comment.yml
comment fixtures have to be shifted because there is one more in Forgejo
Multiple chunks are uploaded with type "block" without using
"appendBlock" and eventually out of order for bigger uploads.
8MB seems to be the chunk size
This change parses the blockList uploaded after all blocks to get the
final artifact size and order them correctly before calculating the
sha256 checksum over all blocks
Fixes#31354
(cherry picked from commit b594cec2bda6f861effedb2e8e0a7ebba191c0e9)
Conflicts:
routers/api/actions/artifactsv4.go
conflict because of Refactor AppURL usage (#30885) 67c1a07285008cc00036a87cef966c3bd519a50c
that was not cherry-picked in Forgejo
the resolution consist of removing the extra ctx argument
- [x] add architecture-specific removal support
- [x] Fix upload competition
- [x] Fix not checking input when downloading
docs: https://codeberg.org/forgejo/docs/pulls/874
### Release notes
- [ ] I do not want this change to show in the release notes.
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/5351
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Co-authored-by: Exploding Dragon <explodingfkl@gmail.com>
Co-committed-by: Exploding Dragon <explodingfkl@gmail.com>
(cherry picked from commit 89742c4913)
Remove unused CSRF options, decouple "new csrf protector" and "prepare"
logic, do not redirect to home page if CSRF validation falis (it
shouldn't happen in daily usage, if it happens, redirecting to home
doesn't help either but just makes the problem more complex for "fetch")
(cherry picked from commit 1fede04b83288d8a91304a83b7601699bb5cba04)
Conflicts:
options/locale/locale_en-US.ini
tests/integration/repo_branch_test.go
trivial context conflicts
(cherry picked from commit 1ae3b127fc)
A 500 status code was thrown when passing a non-existent target to the
create release API. This snapshot handles this error and instead throws
a 404 status code.
Discovered while working on #31840.
(cherry picked from commit f05d9c98c4cb95e3a8a71bf3e2f8f4529e09f96f)
PR for issue #31968
Replaces PR #31983 to comply with gitea's error definition
Failed authentications are now logged to level `Warning` instead of
`Info`.
(cherry picked from commit 64298dcb9e72a5a87a4680563d91fae5b90e0160)
Part of #27700
Removes all URLs from translation strings to easy up changing them in
the future and to exclude people injecting malicious URLs through
translations. First measure as long as #24402 is out of scope.
(cherry picked from commit 83f37f630246e381eefd650fc2d4b1f3976ea882)
Signed-off-by: Gergely Nagy <forgejo@gergo.csillger.hu>
Conflicts:
- options/locale/locale_en-US.ini
Resolved by manually applying the URL->%s changes to our translations.
- routers/web/admin/hooks.go
templates/repo/settings/protected_branch.tmpl
templates/status/500.tmpl
Manually resolved.
- templates/repo/settings/webhook/settings.tmpl
Applied the change to templates/webhook/shared-settings.tmpl
instead
Additional changes: Gitea-specific URLs have been replaced by their
Forgejo counterparts, lifted from the original translation text.
Fix#31916
In #30876, `sortOrder` has been changed into a map, but it is only
implemented in explore.
~~But it seems that size sort order has no effect from long long ago,~~
not directly caused by the PR above.
I think it is still caused by #29231.
In #29231, it merged the sort orders from
`templates/explore/repo_search.tmpl` and
`templates/admin/repo/search.tmpl`.
In `templates/admin/repo/search.tmpl`, it contains size sort orders, but
not in `templates/explore/repo_search.tmpl`, which is used in non-admin
pages.
So `order by size` is added from #29231, but the handler was not added.
---------
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
(cherry picked from commit 661a1e10f7abd3527d2abc027dec936022db9379)
- For WebAuthn Credential level 3, the `backup_eligible` and
`backup_state` flags are checked if they are consistent with the values
given on login. Forgejo never stored this data, so add a database
migration that makes all webauthn credentials 'legacy' and on the next
first use capture the values of `backup_eligible` and `backup_state`.
As suggested in https://github.com/go-webauthn/webauthn/discussions/219#discussioncomment-10429662
- Adds unit tests.
- Add E2E test.
- The Conan and Container packages use a different type of
authentication. It first authenticates via the regular way (api tokens
or user:password, handled via `auth.Basic`) and then generates a JWT
token that is used by the package software (such as Docker) to do the
action they wanted to do. This JWT token didn't properly propagate the
API scopes that the token was generated for, and thus could lead to a
'scope escalation' within the Conan and Container packages, read
access to write access.
- Store the API scope in the JWT token, so it can be propagated on
subsequent calls that uses that JWT token.
- Integration test added.
- Resolves#5128
