Commit graph

22215 commits

Author SHA1 Message Date
Matthias Riße
22855c6238 Cache git-annex UUID to repository path mappings (#54)
Always walking the filesystem and searching for UUIDs slowed p2phttp
operations down significantly on a production server with more than a
handful of repositories. This caching strategy ensures that only the
first call is rather slow, and subsequent ones should be much faster.

This should better be implemented as a background job, but for now this
is a simple solution to the problem.

Fixes #53.

Reviewed-on: https://codeberg.org/forgejo-aneksajo/forgejo-aneksajo/pulls/54
Co-authored-by: Matthias Riße <m.risse@fz-juelich.de>
Co-committed-by: Matthias Riße <m.risse@fz-juelich.de>
2025-02-08 16:52:15 +01:00
Matthias Riße
aa683cb1e2 Explicitly set http(s) default ports in annex.url (#55)
Otherwise, git-annex tries to use its own default port (9417) and fails.

Fixes #52.

Reviewed-on: https://codeberg.org/forgejo-aneksajo/forgejo-aneksajo/pulls/55
Co-authored-by: Matthias Riße <m.risse@fz-juelich.de>
Co-committed-by: Matthias Riße <m.risse@fz-juelich.de>
2025-02-08 16:52:15 +01:00
Matthias Riße
99d2de8956 Fix Forgejo version in published OCI images (#51)
The Forgejo version is derived from the git history, so the image build
needs to happen in the context of a full repository clone.

Also, the post-processing of the version string needs to remove the
second occurrence of "-g", as the first one is now part of the added
"-git-annex" part.

Reviewed-on: https://codeberg.org/forgejo-aneksajo/forgejo-aneksajo/pulls/51
Co-authored-by: Matthias Riße <m.risse@fz-juelich.de>
Co-committed-by: Matthias Riße <m.risse@fz-juelich.de>
2025-02-08 16:52:15 +01:00
Matthias Riße
4b75d40853 Add an OCI image build and publish workflow (#50)
Fixes #49.

Reviewed-on: https://codeberg.org/forgejo-aneksajo/forgejo-aneksajo/pulls/50
Co-authored-by: Matthias Riße <m.risse@fz-juelich.de>
Co-committed-by: Matthias Riße <m.risse@fz-juelich.de>
2025-02-08 16:52:15 +01:00
Matthias Riße
aa6674de6e Simplify git blob to annex key lookup (#43)
Fixes #27.

Reviewed-on: https://codeberg.org/matrss/forgejo-aneksajo/pulls/43
Co-authored-by: Matthias Riße <m.risse@fz-juelich.de>
Co-committed-by: Matthias Riße <m.risse@fz-juelich.de>
2025-02-08 16:52:15 +01:00
Matthias Riße
971f6f7bab Add git-annex p2phttp support (#42)
This adds a new endpoint under `/git-annex-p2phttp` which acts as an
authenticating proxy to git-annex' p2phttp server. This makes it
possible to set `annex+<server-url>/git-annex-p2phttp` as
`remote.<name>.annexurl` and use git-annex fully over http(s) with the
normal credentials and access tokens provided by Forgejo.

Fixes #25.

Reviewed-on: https://codeberg.org/matrss/forgejo-aneksajo/pulls/42
Co-authored-by: Matthias Riße <m.risse@fz-juelich.de>
Co-committed-by: Matthias Riße <m.risse@fz-juelich.de>
2025-02-08 16:52:14 +01:00
Matthias Riße
1dd3a341a7 Add tests for git annex drop (#47)
This adds some rudimentary tests that drop files in a repository's clone
as well as from a repository on Forgejo.

Fixes #4.

Reviewed-on: https://codeberg.org/matrss/forgejo-aneksajo/pulls/47
Co-authored-by: Matthias Riße <m.risse@fz-juelich.de>
Co-committed-by: Matthias Riße <m.risse@fz-juelich.de>
2025-02-08 16:52:14 +01:00
Matthias Riße
592b23f5cd Add git-annex' testremote to the test suite (#48)
The `git annex testremote` command runs a built-in set of tests against
a remote. It cannot hurt to check our implementation of a git-annex
remote against it too.

Reviewed-on: https://codeberg.org/matrss/forgejo-aneksajo/pulls/48
Co-authored-by: Matthias Riße <m.risse@fz-juelich.de>
Co-committed-by: Matthias Riße <m.risse@fz-juelich.de>
2025-02-08 16:52:14 +01:00
Matthias Riße
4d65d593ad Only upload to annex in doAnnexUploadTest (#46)
The previous implementation both uploaded to the annex and pushed to the
git repository. This meant that the tests checking that uploads without
permission fail actually could pass when the git push failed but the
git-annex upload didn't. The tests didn't catch the situation where
unauthorized users could modify the annex.

Reviewed-on: https://codeberg.org/matrss/forgejo-aneksajo/pulls/46
Co-authored-by: Matthias Riße <m.risse@fz-juelich.de>
Co-committed-by: Matthias Riße <m.risse@fz-juelich.de>
2025-02-08 16:52:14 +01:00
Matthias Riße
3cf2111dbf Use PATH when looking for git commands (#44)
This fixes some issues when using a git-annex that is installed in a
different location than where git is installed, e.g. when using the
git-annex-standalone release or one installed with nix.

Reviewed-on: https://codeberg.org/matrss/forgejo-aneksajo/pulls/44
Co-authored-by: Matthias Riße <m.risse@fz-juelich.de>
Co-committed-by: Matthias Riße <m.risse@fz-juelich.de>
2025-02-08 16:52:14 +01:00
Matthias Riße
cdc80b9fe6 Allow anonymous HEAD requests to annex/objects (#41)
Git-annex emits HEAD requests for keys while trying to drop them from a
repository that was cloned via http. Forgejo asked for authentication
for these HEAD requests. This meant that cloning and getting files was
possible without authentication, but dropping was not.

Since the response to a HEAD request is a subset of the response to a
GET request it is safe to make those unauthenticated as well. That is
what this change does, although limited to the
:username/:reponame/annex/objects endpoint.

Fixes #40.

Reviewed-on: https://codeberg.org/matrss/forgejo-aneksajo/pulls/41
Co-authored-by: Matthias Riße <m.risse@fz-juelich.de>
Co-committed-by: Matthias Riße <m.risse@fz-juelich.de>
2025-02-08 16:52:14 +01:00
Michael Hanke
8065657151 Elevate external markup renderer interface for annexed file content (#36)
Previously, an external renderer that matched on an annexed file would
only see its content streamed via `STDIN`, or a temporary file with a copy
of its content would be generated and passed-by-filepath (with
`IS_INPUT_FILE=true`). Whether that happens, is also subject to
`MAX_DISPLAY_FILE_SIZE` (which defaults to 8MB).

This was problematic, because annexed files tend to be large. Moreover,
if present, they already exist as write-protected files on the
file-system. Creating a copy is both expensive and serves no particular
purpose.

This commit changes how external renderers are called.

1) With `IS_INPUT_FILE=true`, the renderer is passed the true location
   of an annex key, if present, and an empty path, if not.
2) The original, repository-relative path of the rendering target is
   made available to the renderer via the `GITEA_RELATIVE_PATH`
   environment variable.

To achieve a lean implementation, the `Blob` of the rendering target
is passed on to the `RenderContext` (because the implementation of
the annex-related functionality is centered on this dtype.

This change makes it less costly to increase `MAX_DISPLAY_FILE_SIZE`,
in order to make large, annexed files eligible for markup rendering,
because no content copies will be made any longer.

External renderers can now use the original file path, with the full
original filename, including extensions, for decision making. For
example, to detect particular compression formats based in a file name
extension, or to alter the rendering based on contextual information
encoded in the file path (e.g., a multi-file data structure with a
particular organization pattern).

Apart from the additional environment variable, there is no change to
the handling of renderers that take their input via `STDIN` (i.e.,
`IS_INPUT_FILE=false`).

Fixes #35.

Reviewed-on: https://codeberg.org/matrss/forgejo-aneksajo/pulls/36
Reviewed-by: matrss <matrss@noreply.codeberg.org>
Co-authored-by: Michael Hanke <michael.hanke@gmail.com>
Co-committed-by: Michael Hanke <michael.hanke@gmail.com>
2025-02-08 16:52:14 +01:00
Matthias Riße
78ff09f63d Change the icon for annexed files to file-binary (#29)
Fixes #26.

Reviewed-on: https://codeberg.org/matrss/forgejo-aneksajo/pulls/29
Co-authored-by: Matthias Riße <m.risse@fz-juelich.de>
Co-committed-by: Matthias Riße <m.risse@fz-juelich.de>
2025-02-08 16:52:14 +01:00
Matthias Riße
4c60dfb6f0 Improve views for annexed but missing files (#28)
Previously, trying to view files that were annexed, but missing, just
led to an uninformative error 500. This was rather confusing.

With these changes it now shows the pointer target instead of the
(missing) content of the file, and also indicates this situation in the
"stored with git-annex" message. For semantic correctness views for
missing files return a 404 instead of a 200, as they would with the
content present.

Fixes #7, fixes #13.

Reviewed-on: https://codeberg.org/matrss/forgejo-aneksajo/pulls/28
Co-authored-by: Matthias Riße <m.risse@fz-juelich.de>
Co-committed-by: Matthias Riße <m.risse@fz-juelich.de>
2025-02-08 16:52:14 +01:00
Michael Hanke
5b2ae61a12 Add git-annex also to the rootless container (#24)
Same as 89f8aa0bf5, but for the rootless container.

Reviewed-on: https://codeberg.org/matrss/forgejo-aneksajo/pulls/24
Reviewed-by: matrss <matrss@noreply.codeberg.org>
Co-authored-by: Michael Hanke <michael.hanke@gmail.com>
Co-committed-by: Michael Hanke <michael.hanke@gmail.com>
2025-02-08 16:52:14 +01:00
Matthias Riße
e83f575da5 Git-annex web uploads (#21)
This implements support for uploading files into the annex using the web
interface.

If a repository is a git-annex-enabled repository all files will be
added to it using git annex add. This means that the repository's
configuration for what to put into the annex (annex.largefiles in
gitattributes) will be respected.

Plain git repositories without git-annex will work as before, directly
uploading to git.

Fixes #5.

Reviewed-on: https://codeberg.org/matrss/forgejo-aneksajo/pulls/21
Co-authored-by: Matthias Riße <m.risse@fz-juelich.de>
Co-committed-by: Matthias Riße <m.risse@fz-juelich.de>
2025-02-08 16:52:14 +01:00
Matthias Riße
246131cc69 Error if git-annex is enabled but missing (#16)
Copied from https://github.com/neuropoly/gitea/pull/47

This adds a check so that if `setting.Annex.Enabled` is true and git-annex is not in the PATH Forgejo will abort on startup with a reasonable error message.

Fixes #15.

Reviewed-on: https://codeberg.org/matrss/forgejo-aneksajo/pulls/16
Co-authored-by: Matthias Riße <m.risse@fz-juelich.de>
Co-committed-by: Matthias Riße <m.risse@fz-juelich.de>
2025-02-08 16:52:14 +01:00
Matthias Riße
e97e20427d Add git-annex to docker image 2025-02-08 16:52:14 +01:00
Matthias Riße
5ff7951560 Adapt patch to upstream changes
Test with different objectFormats.
2025-02-08 16:52:14 +01:00
Nick
56a9ea1fc3 git-annex: views for annex files
This updates the repo index/file view endpoints so annex files match the way
LFS files are rendered, making annexed files accessible via the web instead of
being black boxes only accessible by git clone.

This mostly just duplicates the existing LFS logic. It doesn't try to combine itself
with the existing logic, to make merging with upstream easier. If upstream ever
decides to accept, I would like to try to merge the redundant logic.

The one bit that doesn't directly copy LFS is my choice to hide annex-symlinks.
LFS files are always _pointer files_ and therefore always render with the "file"
icon and no special label, but annex files come in two flavours: symlinks or
pointer files. I've conflated both kinds to try to give a consistent experience.

The tests in here ensure the correct download link (/media, from the last PR)
renders in both the toolbar and, if a binary file (like most annexed files will be),
in the main pane, but it also adds quite a bit of code to make sure text files
that happen to be annexed are dug out and rendered inline like LFS files are.
2025-02-08 16:52:14 +01:00
Matthias Riße
82c9a86e83 Adapt patch to upstream changes
Use tests.FileCmp instead of util.FileCmp.
2025-02-08 16:52:14 +01:00
Matthias Riße
2845a75cbe Adapt patch to upstream changes
Test with different objectFormats.
2025-02-08 16:52:14 +01:00
Nick
2c0cb04776 git-annex: make /media/ download annexed content
Previously, Gitea's LFS support allowed direct-downloads of LFS content,
via http://$HOSTNAME:$PORT/$USER/$REPO/media/branch/$BRANCH/$FILE
Expand that grace to git-annex too. Now /media should provide the
relevant *content* from the .git/annex/objects/ folder.

This adds tests too. And expands the tests to try symlink-based annexing,
since /media implicitly supports both that and pointer-file-based annexing.
2025-02-08 16:52:14 +01:00
Matthias Riße
635770f8a4 Adapt patch to upstream changes
The git repository must be closed after using it. Without this change
some tests started to fail due to the lingering repository running into
a timeout.
2025-02-08 16:52:14 +01:00
Nick
0831aeccd1 git-annex: create modules/annex
This moves the `annexObjectPath()` helper out of the tests and into a
dedicated sub-package as `annex.ContentLocation()`, and expands it with
`.Pointer()` (which validates using `git annex examinekey`),
`.IsAnnexed()` and `.Content()` to make it a more useful module.

The tests retain their own wrapper version of `ContentLocation()`
because I tried to follow close to the API modules/lfs uses, which in
terms of abstract `git.Blob` and `git.TreeEntry` objects, not in terms
of `repoPath string`s which are more convenient for the tests.
2025-02-08 16:52:14 +01:00
Matthias Riße
0c81519d4e Adapt patch to upstream changes
The "context_service" import was changed to use the default name of just
"context". The patch set had to be adapted for that.
2025-02-08 16:52:14 +01:00
Matthias Riße
3c11eb1e26 Adapt patch to upstream changes
Usage of `path` was replaced by `path/filepath` in upstream forgejo, and
it made sense to use that as well where `path` was previously used. The
`setHeaderCacheForever` function and the `sendFile` method had their
signature changed.
2025-02-08 16:52:14 +01:00
Matthias Riße
24264cb45f Fix exit code check for git command
The err.IsExitCode method was changed to a function IsErrorExitCode
taking err as its first argument in
1e7a6483b8.
2025-02-08 16:52:14 +01:00
Matthias Riße
f0da2f268a Replace m.GetOptions with m.Methods
This applies the same changes that were done in
265cd70bdb to the git-annex specific
routes as well.
2025-02-08 16:52:13 +01:00
Nick
d8bf745749 git-annex: support downloading over HTTP
This makes HTTP symmetric with SSH clone URLs.

This gives us the fancy feature of _anonymous_ downloads,
so people can access datasets without having to set up an
account or manage ssh keys.

Previously, to access "open access" data shared this way,
users would need to:

  1. Create an account on gitea.example.com
  2. Create ssh keys
  3. Upload ssh keys (and make sure to find and upload the correct file)
  4. `git clone git@gitea.example.com:user/dataset.git`
  5. `cd dataset`
  6. `git annex get`

This cuts that down to just the last three steps:

  1. `git clone https://gitea.example.com/user/dataset.git`
  2. `cd dataset`
  3. `git annex get`

This is significantly simpler for downstream users, especially for those
unfamiliar with the command line.

Unfortunately there's no uploading. While git-annex supports uploading
over HTTP to S3 and some other special remotes, it seems to fail on a
_plain_ HTTP remote. See https://github.com/neuropoly/gitea/issues/7
and https://git-annex.branchable.com/forum/HTTP_uploads/#comment-ce28adc128fdefe4c4c49628174d9b92.

This is not a major loss since no one wants uploading to be anonymous anyway.

To support private repos, I had to hunt down and patch a secret extra security
corner that Gitea only applies to HTTP for some reason (services/auth/basic.go).

This was guided by https://git-annex.branchable.com/tips/setup_a_public_repository_on_a_web_site/

Fixes https://github.com/neuropoly/gitea/issues/3

Co-authored-by: Mathieu Guay-Paquet <mathieu.guaypaquet@polymtl.ca>
2025-02-08 16:52:13 +01:00
Nick
f42432a9e4 git-annex: add configuration setting [annex].ENABLED
Fixes https://github.com/neuropoly/gitea/issues/8

Co-authored-by: Mathieu Guay-Paquet <mathieu.guaypaquet@gmail.com>
2025-02-08 16:52:13 +01:00
Matthias Riße
cc193ec4ef Fix failing tests
Multiple tests that worked fine on v1.20.4-1 started to fail after the
rebase onto v1.20.5-1. These tests are:
- TestGitAnnexPermissions/Private/Owner/HTTP/Init
- TestGitAnnexPermissions/Private/Owner/HTTP/Download
- TestGitAnnexPermissions/Private/Writer/HTTP/Init
- TestGitAnnexPermissions/Private/Writer/HTTP/Download
- TestGitAnnexPermissions/Private/Reader/HTTP/Init
- TestGitAnnexPermissions/Private/Reader/HTTP/Download

What these tests have in common is that they all operate on a private
repository via http with authentication.

They broke at some point between v1.20.4-1 and v1.20.5-1, so I did a
bisect between these two points running the offending tests. This
brought me to the conclusion that
ee48c0d5ea introduced the issue.

The thing is, this commit does not change any code, it only changes the
test environment. Among other things that didn't look as suspicious, it
changes the container image from a bespoke test_env image based on
debian bullseye to a node image based on debian bookworm. Obviously,
this means that there are many version differences between the two.

The first one I looked at was git. The previous bullseye image used a
manually installed git version 2.40.0, while the bookworm image has
2.39.2 installed. Updating git in the new image did not fix the issue,
however.

The next thing I looked at was the git-annex version. Bullseye had
8.20210223 installed and worked, while bookworm used 10.20230126 when
the tests broke. So I tried my luck upgrading to a more recent version
via neurodebian (10.20240227-1~ndall+1). This still worked fine on
bullseye and now also works fine on bookworm.

I have no idea why this specific version of git-annex broke the tests,
but at least there was a commit to pinpoint this to, which isn't always
the case with docker images silently changing beneath you...

Below are the versions as they are reported by git and git-annex:

bullseye (works):

    git version 2.30.2
    git-annex version: 8.20210223
    build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite S3 WebDAV
    dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.26 DAV-1.3.4 feed-1.3.0.1 ghc-8.8.4 http-client-0.6.4.1 persistent-sqlite-2.10.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.1.0
    key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
    remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external
    operating system: linux x86_64
    supported repository versions: 8
    upgrade supported from repository versions: 0 1 2 3 4 5 6 7

bullseye + git-annex from neurodebian (works):

    git version 2.30.2
    git-annex version: 10.20240227-1~ndall+1
    build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Benchmark Feeds Testsuite S3 WebDAV
    dependency versions: aws-0.22.1 bloomfilter-2.0.1.0 cryptonite-0.29 DAV-1.3.4 feed-1.3.2.1 ghc-9.0.2 http-client-0.7.13.1 persistent-sqlite-2.13.1.0 torrent-10000.1.1 uuid-1.3.15 yesod-1.6.2.1
    key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
    remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external
    operating system: linux x86_64
    supported repository versions: 8 9 10
    upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10

bookworm (fails):

    git version 2.39.2
    git-annex version: 10.20230126
    build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Benchmark Feeds Testsuite S3 WebDAV
    dependency versions: aws-0.22.1 bloomfilter-2.0.1.0 cryptonite-0.29 DAV-1.3.4 feed-1.3.2.1 ghc-9.0.2 http-client-0.7.13.1 persistent-sqlite-2.13.1.0 torrent-10000.1.1 uuid-1.3.15 yesod-1.6.2.1
    key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
    remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external
    operating system: linux x86_64
    supported repository versions: 8 9 10
    upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10

bookworm + git-annex from neurodebian (works):

    git version 2.39.2
    git-annex version: 10.20240227-1~ndall+1
    build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Benchmark Feeds Testsuite S3 WebDAV
    dependency versions: aws-0.22.1 bloomfilter-2.0.1.0 cryptonite-0.29 DAV-1.3.4 feed-1.3.2.1 ghc-9.0.2 http-client-0.7.13.1 persistent-sqlite-2.13.1.0 torrent-10000.1.1 uuid-1.3.15 yesod-1.6.2.1
    key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
    remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external
    operating system: linux x86_64
    supported repository versions: 8 9 10
    upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10
2025-02-08 16:52:13 +01:00
Matthias Riße
3d7458b8cc Install git-annex in the testing workflow 2025-02-08 16:52:13 +01:00
Matthias Riße
37a8fa9b5c Adapt patch to upstream changes 2025-02-08 16:52:13 +01:00
Matthias Riße
d13faeccaa Adapt patch to upstream changes
A dead code check started to complain because FileCmp was only used in
tests. Moved the function to test_utils.
2025-02-08 16:52:13 +01:00
Matthias Riße
57de00e5b9 Adapt patch to upstream changes
Repository creation now expects an objectFormat to be specified for git.
2025-02-08 16:52:13 +01:00
Nick
52485edf64 git-annex tests
Fixes https://github.com/neuropoly/gitea/issues/11

Tests:

* `git annex init`
* `git annex copy --from origin`
* `git annex copy --to origin`

over:

* ssh

for:

* the owner
* a collaborator
* a read-only collaborator
* a stranger

in a

* public repo
* private repo

And then confirms:

* Deletion of the remote repo (to ensure lockdown isn't messing with us: https://git-annex.branchable.com/internals/lockdown/#comment-0cc5225dc5abe8eddeb843bfd2fdc382)

------

To support all this:

* Add util.FileCmp()
* Patch withKeyFile() so it can be nested in other copies of itself

-------

Many thanks to Mathieu for giving style tips and catching several bugs,
including a subtle one in util.filecmp() which neutered it.

Co-authored-by: Mathieu Guay-Paquet <mathieu.guay-paquet@polymtl.ca>
Co-authored-by: Matthias Riße <m.risse@fz-juelich.de>
2025-02-08 16:52:13 +01:00
Nick Guenther
1d01b58227 git-annex support
[git-annex](https://git-annex.branchable.com/) is a more complicated cousin to
git-lfs, storing large files in an optional-download side content.  Unlike lfs,
it allows mixing and matching storage remotes, so the content remote(s) doesn't
need to be on the same server as the git remote, making it feasible to scatter
a collection across cloud storage, old harddrives, or anywhere else storage can
be scavenged.  Since this can get complicated, fast, it has a content-tracking
database (`git annex whereis`) to help find everything later.

The use-case we imagine for including it in Gitea is just the simple case, where
we're primarily emulating git-lfs: each repo has its large content at the same URL.

Our motivation is so we can self-host https://www.datalad.org/ datasets, which
currently are only hostable by fragilely scrounging together cloud storage --
and having to manage all the credentials associated with all the pieces -- or at
https://openneuro.org which is fragile in its own ways.

Supporting git-annex also allows multiple Gitea instance to be annex remotes for
each other, mirroring the content or otherwise collaborating the split up the
hosting costs.

Enabling
--------

TODO

HTTP
----

TODO

Permission Checking
-------------------

This tweaks the API in routers/private/serv.go to expose the calling user's
computed permission, instead of just returning HTTP 403.

This doesn't fit in super well. It's the opposite from how the git-lfs support is
done, where there's a complete list of possible subcommands and their matching
permission levels, and then the API compares the requested with the actual level
and returns HTTP 403 if the check fails.

But it's necessary. The main git-annex verbs, 'git-annex-shell configlist' and
'git-annex-shell p2pstdio' are both either read-only or read-write operations,
depending on the state on disk on either end of the connection and what the user
asked it to ask for, with no way to know before git-annex examines the situation.
So tell the level via GIT_ANNEX_READONLY and trust it to handle itself.

In the older Gogs version, the permission was directly read in cmd/serv.go:

```
mode, err = db.UserAccessMode(user.ID, repo)
```
- 966e925cf3/internal/cmd/serv.go (L334)

but in Gitea permission enforcement has been centralized in the API layer.
(perhaps so the cmd layer can avoid making direct DB connections?)

Deletion
--------

git-annex has this "lockdown" feature where it tries
really quite very hard to prevent you deleting its
data, to the point that even an rm -rf won't do it:
each file in annex/objects/ is nested inside a
folder with read-only permissions.

The recommended workaround is to run chmod -R +w when
you're sure you actually want to delete a repo. See
https://git-annex.branchable.com/internals/lockdown

So we edit util.RemoveAll() to do just that, so now
it's `chmod -R +w && rm -rf` instead of just `rm -rf`.
2025-02-08 16:52:13 +01:00
Matthias Riße
2771c2e591 Fix name in package-lock.json
The frontend-checks job started failing because the declared name does
not match the repository name.
2025-02-08 16:52:13 +01:00
Matthias Riße
3c7f598230 Run testing workflow unconditionally 2025-02-08 16:52:13 +01:00
Earl Warren
6ef900899e fix(sec): Forgejo Actions web routes (#6839)
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/6839
2025-02-08 06:21:18 +00:00
Earl Warren
a9f0bb9f68 fix(sec): permission check for project issue (#6838)
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/6838
2025-02-08 06:20:29 +00:00
Gusted
3b4f1b3469
fix(sec): add tests for private issues on projects
- Add integration and unit tests to ensure that private issues on
projects are not shown in any way, shape or form when the doer has no
access to it.

(cherry picked from commit 55dcc1d06cb12ddb750a0289fbb6e212f93957a8)
2025-02-08 06:06:06 +00:00
Gusted
77fc232e5b
fix(sec): permission check for project issue
- Do an access check when loading issues for a project column, currently
this is not done and exposes the title, labels and existence of a
private issue that the viewer of the project board may not have access
to.
- The number of issues cannot be calculated in a efficient manner
and stored in the database because their number may vary depending on
the visibility of the repositories participating in the project. The
previous implementation used the pre-calculated numbers stored in each
project, which did not reflect that potential variation.
- The code is derived from https://github.com/go-gitea/gitea/pull/22865

(cherry picked from commit 2193afaeb9954a5778f5a47aafd0e6fbbf48d000)
2025-02-08 06:06:03 +00:00
Gusted
5a7d70658d
fix(sec): web route test edit and delete variable
Exhaustively test each combination of deleting and updating a action
action variable via the web route.
2025-02-08 06:04:14 +00:00
Gusted
5046a10aec
fix(sec): add tests for web route delete runner
Exhaustively test each combination of deleting and updating a action
runner via the web route. Although updating an action runner was not
impacted, its good to have a test nonetheless.
2025-02-08 06:04:14 +00:00
Gusted
77db7655e0
fix(sec): web route update and delete runner variables
The web route to update and delete variables of runners did not check if
the ID that was given belonged to the context it was requested in, this
made it possible to update and delete every existing runner variable of
a instance for any authenticated user.

The code has been reworked to always take into account the context of
the request (owner and repository ID).
2025-02-08 06:04:14 +00:00
Gusted
c324910c31
fix(sec): web route delete runner
The web route to delete action runners did not check if the ID that was
given belonged to the context it was requested in, this made it possible
to delete every existing runner of a instance by a authenticated user.

The code was reworked to ensure that the caller of the delete
runner function retrieved the runner by ID and then checks if it belongs
to the context it was requested in, although this is not an optimal
solution it is consistent with the context checking of other code for
runners.
2025-02-08 06:04:14 +00:00
0ko
57ad0b868d [v10.0/forgejo] i18n: update of translations from Codeberg Translate (#6834)
Backport: https://codeberg.org/forgejo/forgejo/pulls/6745

Feel free to leave merge to me. My strategy is to use squash-merge, but copy the description of the actual commit into squash description, so attribution is preserved.

https://codeberg.org/forgejo/forgejo/pulls/6745 was quite large, so I decided it will be ok to backport just one PR before a new release.

<!--start release-notes-assistant-->

## Release notes
<!--URL:https://codeberg.org/forgejo/forgejo-->
- Localization
  - [PR](https://codeberg.org/forgejo/forgejo/pulls/6834): <!--number 6834 --><!--line 0 --><!--description aTE4bjogdXBkYXRlIG9mIHRyYW5zbGF0aW9ucyBmcm9tIENvZGViZXJnIFRyYW5zbGF0ZQ==-->i18n: update of translations from Codeberg Translate<!--description-->
<!--end release-notes-assistant-->

Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/6834
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Co-authored-by: 0ko <0ko@noreply.codeberg.org>
Co-committed-by: 0ko <0ko@noreply.codeberg.org>
2025-02-08 00:01:24 +00:00
forgejo-backport-action
34d2a8531c [v10.0/forgejo] fix(ui): display verified icon for default gpg key (#6833)
**Backport:** https://codeberg.org/forgejo/forgejo/pulls/6803

## Description
Thank you for this amazing project!

I recently noticed that the icon next to the GPG key reference appears to be incorrect for commits signed by the default GPG key:

| Default GPG Key  | User GPG Key  |
|---------|---------|
| ![image](/attachments/ff27597a-d38c-48fc-8284-e16d4fd3ea2d) | ![image](/attachments/3c8c5540-dd29-45c9-a9f0-dc3b69677ef3) |

Looking into the commit history of the template file, I noticed that Forgejo-signed commits originally had a distinct icon:
[gitea-unlock-cog](b918609acc) --> [octicon-shield-lock](12ddc48c5c) --> octicon-unverified (current)

Since `octicon-unverified` is also used when a commit cannot be verified (.Verification.Warning), I find it misleading for successfully signed commits. This PR changes the icon to the verified variant for better clarity.

### Tests

1. Set up automatic commit signing by Forgejo ([guide](https://forgejo.org/docs/latest/admin/signing/#automatic-signing))
2. Trigger automatic commit signing in any of the following scenarios:
    - Repository Initialisation (should be the easiest)
    - Wiki Changes
    - CRUD actions using the editor or the API
    - Merges from Pull Requests
3. Open the commit signed by Forgejo
4. Verify that the icon next to the GPG key id is `octicon-verified`:
![image](/attachments/7b4eb81c-d33c-4daf-84dd-9f99ebb0b99d)

### Documentation

- [x] I did not document these changes and I do not expect someone else to do it.

### Release notes

- [x] I want the title to show in the release notes with a link to this pull request.

<!--start release-notes-assistant-->

## Release notes
<!--URL:https://codeberg.org/forgejo/forgejo-->
- User Interface bug fixes
  - [PR](https://codeberg.org/forgejo/forgejo/pulls/6803): <!--number 6803 --><!--line 0 --><!--description Zml4KHVpKTogZGlzcGxheSB2ZXJpZmllZCBpY29uIGZvciBkZWZhdWx0IGdwZyBrZXk=-->fix(ui): display verified icon for default gpg key<!--description-->
<!--end release-notes-assistant-->

Co-authored-by: shgew <shgew@noreply.codeberg.org>
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/6833
Reviewed-by: Gusted <gusted@noreply.codeberg.org>
Reviewed-by: Beowulf <beowulf@beocode.eu>
Co-authored-by: forgejo-backport-action <forgejo-backport-action@noreply.codeberg.org>
Co-committed-by: forgejo-backport-action <forgejo-backport-action@noreply.codeberg.org>
2025-02-07 13:16:23 +00:00