Previously, Gitea's LFS support allowed direct-downloads of LFS content,
via http://$HOSTNAME:$PORT/$USER/$REPO/media/branch/$BRANCH/$FILE
Expand that grace to git-annex too. Now /media should provide the
relevant *content* from the .git/annex/objects/ folder.
This adds tests too. And expands the tests to try symlink-based annexing,
since /media implicitly supports both that and pointer-file-based annexing.
The git repository must be closed after using it. Without this change
some tests started to fail due to the lingering repository running into
a timeout.
This moves the `annexObjectPath()` helper out of the tests and into a
dedicated sub-package as `annex.ContentLocation()`, and expands it with
`.Pointer()` (which validates using `git annex examinekey`),
`.IsAnnexed()` and `.Content()` to make it a more useful module.
The tests retain their own wrapper version of `ContentLocation()`
because I tried to follow close to the API modules/lfs uses, which in
terms of abstract `git.Blob` and `git.TreeEntry` objects, not in terms
of `repoPath string`s which are more convenient for the tests.
Usage of `path` was replaced by `path/filepath` in upstream forgejo, and
it made sense to use that as well where `path` was previously used. The
`setHeaderCacheForever` function and the `sendFile` method had their
signature changed.
This makes HTTP symmetric with SSH clone URLs.
This gives us the fancy feature of _anonymous_ downloads,
so people can access datasets without having to set up an
account or manage ssh keys.
Previously, to access "open access" data shared this way,
users would need to:
1. Create an account on gitea.example.com
2. Create ssh keys
3. Upload ssh keys (and make sure to find and upload the correct file)
4. `git clone git@gitea.example.com:user/dataset.git`
5. `cd dataset`
6. `git annex get`
This cuts that down to just the last three steps:
1. `git clone https://gitea.example.com/user/dataset.git`
2. `cd dataset`
3. `git annex get`
This is significantly simpler for downstream users, especially for those
unfamiliar with the command line.
Unfortunately there's no uploading. While git-annex supports uploading
over HTTP to S3 and some other special remotes, it seems to fail on a
_plain_ HTTP remote. See https://github.com/neuropoly/gitea/issues/7
and https://git-annex.branchable.com/forum/HTTP_uploads/#comment-ce28adc128fdefe4c4c49628174d9b92.
This is not a major loss since no one wants uploading to be anonymous anyway.
To support private repos, I had to hunt down and patch a secret extra security
corner that Gitea only applies to HTTP for some reason (services/auth/basic.go).
This was guided by https://git-annex.branchable.com/tips/setup_a_public_repository_on_a_web_site/
Fixes https://github.com/neuropoly/gitea/issues/3
Co-authored-by: Mathieu Guay-Paquet <mathieu.guaypaquet@polymtl.ca>
Multiple tests that worked fine on v1.20.4-1 started to fail after the
rebase onto v1.20.5-1. These tests are:
- TestGitAnnexPermissions/Private/Owner/HTTP/Init
- TestGitAnnexPermissions/Private/Owner/HTTP/Download
- TestGitAnnexPermissions/Private/Writer/HTTP/Init
- TestGitAnnexPermissions/Private/Writer/HTTP/Download
- TestGitAnnexPermissions/Private/Reader/HTTP/Init
- TestGitAnnexPermissions/Private/Reader/HTTP/Download
What these tests have in common is that they all operate on a private
repository via http with authentication.
They broke at some point between v1.20.4-1 and v1.20.5-1, so I did a
bisect between these two points running the offending tests. This
brought me to the conclusion that
ee48c0d5ea introduced the issue.
The thing is, this commit does not change any code, it only changes the
test environment. Among other things that didn't look as suspicious, it
changes the container image from a bespoke test_env image based on
debian bullseye to a node image based on debian bookworm. Obviously,
this means that there are many version differences between the two.
The first one I looked at was git. The previous bullseye image used a
manually installed git version 2.40.0, while the bookworm image has
2.39.2 installed. Updating git in the new image did not fix the issue,
however.
The next thing I looked at was the git-annex version. Bullseye had
8.20210223 installed and worked, while bookworm used 10.20230126 when
the tests broke. So I tried my luck upgrading to a more recent version
via neurodebian (10.20240227-1~ndall+1). This still worked fine on
bullseye and now also works fine on bookworm.
I have no idea why this specific version of git-annex broke the tests,
but at least there was a commit to pinpoint this to, which isn't always
the case with docker images silently changing beneath you...
Below are the versions as they are reported by git and git-annex:
bullseye (works):
git version 2.30.2
git-annex version: 8.20210223
build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite S3 WebDAV
dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.26 DAV-1.3.4 feed-1.3.0.1 ghc-8.8.4 http-client-0.6.4.1 persistent-sqlite-2.10.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.1.0
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external
operating system: linux x86_64
supported repository versions: 8
upgrade supported from repository versions: 0 1 2 3 4 5 6 7
bullseye + git-annex from neurodebian (works):
git version 2.30.2
git-annex version: 10.20240227-1~ndall+1
build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Benchmark Feeds Testsuite S3 WebDAV
dependency versions: aws-0.22.1 bloomfilter-2.0.1.0 cryptonite-0.29 DAV-1.3.4 feed-1.3.2.1 ghc-9.0.2 http-client-0.7.13.1 persistent-sqlite-2.13.1.0 torrent-10000.1.1 uuid-1.3.15 yesod-1.6.2.1
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external
operating system: linux x86_64
supported repository versions: 8 9 10
upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10
bookworm (fails):
git version 2.39.2
git-annex version: 10.20230126
build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Benchmark Feeds Testsuite S3 WebDAV
dependency versions: aws-0.22.1 bloomfilter-2.0.1.0 cryptonite-0.29 DAV-1.3.4 feed-1.3.2.1 ghc-9.0.2 http-client-0.7.13.1 persistent-sqlite-2.13.1.0 torrent-10000.1.1 uuid-1.3.15 yesod-1.6.2.1
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external
operating system: linux x86_64
supported repository versions: 8 9 10
upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10
bookworm + git-annex from neurodebian (works):
git version 2.39.2
git-annex version: 10.20240227-1~ndall+1
build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Benchmark Feeds Testsuite S3 WebDAV
dependency versions: aws-0.22.1 bloomfilter-2.0.1.0 cryptonite-0.29 DAV-1.3.4 feed-1.3.2.1 ghc-9.0.2 http-client-0.7.13.1 persistent-sqlite-2.13.1.0 torrent-10000.1.1 uuid-1.3.15 yesod-1.6.2.1
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external
operating system: linux x86_64
supported repository versions: 8 9 10
upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10
Fixes https://github.com/neuropoly/gitea/issues/11
Tests:
* `git annex init`
* `git annex copy --from origin`
* `git annex copy --to origin`
over:
* ssh
for:
* the owner
* a collaborator
* a read-only collaborator
* a stranger
in a
* public repo
* private repo
And then confirms:
* Deletion of the remote repo (to ensure lockdown isn't messing with us: https://git-annex.branchable.com/internals/lockdown/#comment-0cc5225dc5abe8eddeb843bfd2fdc382)
------
To support all this:
* Add util.FileCmp()
* Patch withKeyFile() so it can be nested in other copies of itself
-------
Many thanks to Mathieu for giving style tips and catching several bugs,
including a subtle one in util.filecmp() which neutered it.
Co-authored-by: Mathieu Guay-Paquet <mathieu.guay-paquet@polymtl.ca>
[git-annex](https://git-annex.branchable.com/) is a more complicated cousin to
git-lfs, storing large files in an optional-download side content. Unlike lfs,
it allows mixing and matching storage remotes, so the content remote(s) doesn't
need to be on the same server as the git remote, making it feasible to scatter
a collection across cloud storage, old harddrives, or anywhere else storage can
be scavenged. Since this can get complicated, fast, it has a content-tracking
database (`git annex whereis`) to help find everything later.
The use-case we imagine for including it in Gitea is just the simple case, where
we're primarily emulating git-lfs: each repo has its large content at the same URL.
Our motivation is so we can self-host https://www.datalad.org/ datasets, which
currently are only hostable by fragilely scrounging together cloud storage --
and having to manage all the credentials associated with all the pieces -- or at
https://openneuro.org which is fragile in its own ways.
Supporting git-annex also allows multiple Gitea instance to be annex remotes for
each other, mirroring the content or otherwise collaborating the split up the
hosting costs.
Enabling
--------
TODO
HTTP
----
TODO
Permission Checking
-------------------
This tweaks the API in routers/private/serv.go to expose the calling user's
computed permission, instead of just returning HTTP 403.
This doesn't fit in super well. It's the opposite from how the git-lfs support is
done, where there's a complete list of possible subcommands and their matching
permission levels, and then the API compares the requested with the actual level
and returns HTTP 403 if the check fails.
But it's necessary. The main git-annex verbs, 'git-annex-shell configlist' and
'git-annex-shell p2pstdio' are both either read-only or read-write operations,
depending on the state on disk on either end of the connection and what the user
asked it to ask for, with no way to know before git-annex examines the situation.
So tell the level via GIT_ANNEX_READONLY and trust it to handle itself.
In the older Gogs version, the permission was directly read in cmd/serv.go:
```
mode, err = db.UserAccessMode(user.ID, repo)
```
- 966e925cf3/internal/cmd/serv.go (L334)
but in Gitea permission enforcement has been centralized in the API layer.
(perhaps so the cmd layer can avoid making direct DB connections?)
Deletion
--------
git-annex has this "lockdown" feature where it tries
really quite very hard to prevent you deleting its
data, to the point that even an rm -rf won't do it:
each file in annex/objects/ is nested inside a
folder with read-only permissions.
The recommended workaround is to run chmod -R +w when
you're sure you actually want to delete a repo. See
https://git-annex.branchable.com/internals/lockdown
So we edit util.RemoveAll() to do just that, so now
it's `chmod -R +w && rm -rf` instead of just `rm -rf`.
This Fixes#3962 by adding `!important` to the margin of the heading in the rendered markdown.
In the current behaviour, the margin-top was always overridden by a global css-rule. This is prevented by this change.
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/4076
Reviewed-by: 0ko <0ko@noreply.codeberg.org>
Co-authored-by: Beowulf <beowulf@beocode.eu>
Co-committed-by: Beowulf <beowulf@beocode.eu>
(cherry picked from commit 187860bded)
Also convert a comment into a warning in the logs when the deletion of
an artifact cannot find the file in the destination storage.
The case were an error happens while deleting the file is not covered
as it would require to mock the storage.Copy function.
(cherry picked from commit e759794408)
Change the copy to use `ActionsArtifact.StoragePath` instead of the
`ArtifactPath`. Skip artifacts that are expired, and don't error if the
file to copy does not exist.
---
When trying to migrate actions artifact storage from local to MinIO, we
encountered errors that prevented the process from completing
successfully:
* The migration tries to copy the files using the per-run
`ArtifactPath`, instead of the unique `StoragePath`.
* Artifacts that have been marked expired and had their files deleted
would throw an error
* Artifacts that are pending, but don't have a file uploaded yet will
throw an error.
This PR addresses these cases, and allow the process to complete
successfully.
(cherry picked from commit 8de8972baf5d82ff7b58ed77d78e8e1869e64eb5)
(cherry picked from commit 4afbfd3946)
Backport #31188
Fixes issue when running `choco info pkgname` where `pkgname` is also a
substring of another package Id.
Relates to #31168
---
This might fix the issue linked, but I'd like to test it with more choco
commands before closing the issue in case I find other problems if
that's ok.
I'm pretty inexperienced with Go, so feel free to nitpick things.
Not sure I handled
[this](70f87e11b5/routers/api/packages/nuget/nuget.go (L135-L137))
in the best way, so looking for feedback on if I should fix the
underlying issue (`nil` might be a better default for `Value`?).
Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
(cherry picked from commit ca414a7ccf5e26272662e360c44ac50221a0f2d4)
Backport #31204 by @silverwind
Using `.segment` on the project columns is a major abuse of that class,
so remove it and instead set the border-radius directly on it.
Fixes: https://github.com/go-gitea/gitea/issues/31129
Co-authored-by: silverwind <me@silverwind.io>
(cherry picked from commit 298d05df3b79634a0364926f34fb02b73d442c31)
Do not try to create a new authorization grant when one exists already,
thus preventing a DB-related authorization issue.
Fix https://github.com/go-gitea/gitea/pull/30790#issuecomment-2118812426
---------
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
(cherry picked from commit 9c8c9ff6d10b35de8d2d7eae0fc2646ad9bbe94a)
(cherry picked from commit 07fe5a8b13)
This commit forces the resource owner (user) to always approve OAuth 2.0
authorization requests if the client is public (e.g. native
applications).
As detailed in [RFC 6749 Section 10.2](https://www.rfc-editor.org/rfc/rfc6749.html#section-10.2),
> The authorization server SHOULD NOT process repeated authorization
requests automatically (without active resource owner interaction)
without authenticating the client or relying on other measures to ensure
that the repeated request comes from the original client and not an
impersonator.
With the implementation prior to this patch, attackers with access to
the redirect URI (e.g., the loopback interface for
`git-credential-oauth`) can get access to the user account without any
user interaction if they can redirect the user to the
`/login/oauth/authorize` endpoint somehow (e.g., with `xdg-open` on
Linux).
Fixes#25061.
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
(cherry picked from commit 5c542ca94caa3587329167cfe9e949357ca15cf1)
(cherry picked from commit 1b088fade6)
See:
1b088fade6 Prevent automatic OAuth grants for public clients
07fe5a8b13 use existing oauth grant for public client
(cherry picked from commit 592469464b)
It is not possible to tell vulncheck that Forgejo is not affected by
CVE-2024-0406. Use a mirror of the repository to do that.
Refs: https://github.com/mholt/archiver/issues/404
(cherry picked from commit 3bfec270ac)
Conflicts:
go.sum
trivial context conflict
These changes were missed when cherry-picking the following
c9d0e63c202827756c637d9ca7bbde685c1984b7 Remove unnecessary "Str2html" modifier from templates (#29319)
Fixes: https://codeberg.org/forgejo/forgejo/issues/3623
(cherry picked from commit caadd1815a)
If the avatar file in storage does not exist, it is not an error and
the database can be updated.
See 1be797faba Fix bug on avatar
(cherry picked from commit d2c4d833f4)
The avatar must not be unset in the database if there is a failure to
remove the avatar file from storage (file or S3). The two operations
are wrapped in a transaction for that purpose and this test verifies
it is effective.
See 1be797faba Fix bug on avatar
(cherry picked from commit c139efb1e9)
