Previously, Gitea's LFS support allowed direct-downloads of LFS content,
via http://$HOSTNAME:$PORT/$USER/$REPO/media/branch/$BRANCH/$FILE
Expand that grace to git-annex too. Now /media should provide the
relevant *content* from the .git/annex/objects/ folder.
This adds tests too. And expands the tests to try symlink-based annexing,
since /media implicitly supports both that and pointer-file-based annexing.
The git repository must be closed after using it. Without this change
some tests started to fail due to the lingering repository running into
a timeout.
This moves the `annexObjectPath()` helper out of the tests and into a
dedicated sub-package as `annex.ContentLocation()`, and expands it with
`.Pointer()` (which validates using `git annex examinekey`),
`.IsAnnexed()` and `.Content()` to make it a more useful module.
The tests retain their own wrapper version of `ContentLocation()`
because I tried to follow close to the API modules/lfs uses, which in
terms of abstract `git.Blob` and `git.TreeEntry` objects, not in terms
of `repoPath string`s which are more convenient for the tests.
Usage of `path` was replaced by `path/filepath` in upstream forgejo, and
it made sense to use that as well where `path` was previously used. The
`setHeaderCacheForever` function and the `sendFile` method had their
signature changed.
This makes HTTP symmetric with SSH clone URLs.
This gives us the fancy feature of _anonymous_ downloads,
so people can access datasets without having to set up an
account or manage ssh keys.
Previously, to access "open access" data shared this way,
users would need to:
1. Create an account on gitea.example.com
2. Create ssh keys
3. Upload ssh keys (and make sure to find and upload the correct file)
4. `git clone git@gitea.example.com:user/dataset.git`
5. `cd dataset`
6. `git annex get`
This cuts that down to just the last three steps:
1. `git clone https://gitea.example.com/user/dataset.git`
2. `cd dataset`
3. `git annex get`
This is significantly simpler for downstream users, especially for those
unfamiliar with the command line.
Unfortunately there's no uploading. While git-annex supports uploading
over HTTP to S3 and some other special remotes, it seems to fail on a
_plain_ HTTP remote. See https://github.com/neuropoly/gitea/issues/7
and https://git-annex.branchable.com/forum/HTTP_uploads/#comment-ce28adc128fdefe4c4c49628174d9b92.
This is not a major loss since no one wants uploading to be anonymous anyway.
To support private repos, I had to hunt down and patch a secret extra security
corner that Gitea only applies to HTTP for some reason (services/auth/basic.go).
This was guided by https://git-annex.branchable.com/tips/setup_a_public_repository_on_a_web_site/
Fixes https://github.com/neuropoly/gitea/issues/3
Co-authored-by: Mathieu Guay-Paquet <mathieu.guaypaquet@polymtl.ca>
Multiple tests that worked fine on v1.20.4-1 started to fail after the
rebase onto v1.20.5-1. These tests are:
- TestGitAnnexPermissions/Private/Owner/HTTP/Init
- TestGitAnnexPermissions/Private/Owner/HTTP/Download
- TestGitAnnexPermissions/Private/Writer/HTTP/Init
- TestGitAnnexPermissions/Private/Writer/HTTP/Download
- TestGitAnnexPermissions/Private/Reader/HTTP/Init
- TestGitAnnexPermissions/Private/Reader/HTTP/Download
What these tests have in common is that they all operate on a private
repository via http with authentication.
They broke at some point between v1.20.4-1 and v1.20.5-1, so I did a
bisect between these two points running the offending tests. This
brought me to the conclusion that
ee48c0d5ea introduced the issue.
The thing is, this commit does not change any code, it only changes the
test environment. Among other things that didn't look as suspicious, it
changes the container image from a bespoke test_env image based on
debian bullseye to a node image based on debian bookworm. Obviously,
this means that there are many version differences between the two.
The first one I looked at was git. The previous bullseye image used a
manually installed git version 2.40.0, while the bookworm image has
2.39.2 installed. Updating git in the new image did not fix the issue,
however.
The next thing I looked at was the git-annex version. Bullseye had
8.20210223 installed and worked, while bookworm used 10.20230126 when
the tests broke. So I tried my luck upgrading to a more recent version
via neurodebian (10.20240227-1~ndall+1). This still worked fine on
bullseye and now also works fine on bookworm.
I have no idea why this specific version of git-annex broke the tests,
but at least there was a commit to pinpoint this to, which isn't always
the case with docker images silently changing beneath you...
Below are the versions as they are reported by git and git-annex:
bullseye (works):
git version 2.30.2
git-annex version: 8.20210223
build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite S3 WebDAV
dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.26 DAV-1.3.4 feed-1.3.0.1 ghc-8.8.4 http-client-0.6.4.1 persistent-sqlite-2.10.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.1.0
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external
operating system: linux x86_64
supported repository versions: 8
upgrade supported from repository versions: 0 1 2 3 4 5 6 7
bullseye + git-annex from neurodebian (works):
git version 2.30.2
git-annex version: 10.20240227-1~ndall+1
build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Benchmark Feeds Testsuite S3 WebDAV
dependency versions: aws-0.22.1 bloomfilter-2.0.1.0 cryptonite-0.29 DAV-1.3.4 feed-1.3.2.1 ghc-9.0.2 http-client-0.7.13.1 persistent-sqlite-2.13.1.0 torrent-10000.1.1 uuid-1.3.15 yesod-1.6.2.1
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external
operating system: linux x86_64
supported repository versions: 8 9 10
upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10
bookworm (fails):
git version 2.39.2
git-annex version: 10.20230126
build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Benchmark Feeds Testsuite S3 WebDAV
dependency versions: aws-0.22.1 bloomfilter-2.0.1.0 cryptonite-0.29 DAV-1.3.4 feed-1.3.2.1 ghc-9.0.2 http-client-0.7.13.1 persistent-sqlite-2.13.1.0 torrent-10000.1.1 uuid-1.3.15 yesod-1.6.2.1
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external
operating system: linux x86_64
supported repository versions: 8 9 10
upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10
bookworm + git-annex from neurodebian (works):
git version 2.39.2
git-annex version: 10.20240227-1~ndall+1
build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Benchmark Feeds Testsuite S3 WebDAV
dependency versions: aws-0.22.1 bloomfilter-2.0.1.0 cryptonite-0.29 DAV-1.3.4 feed-1.3.2.1 ghc-9.0.2 http-client-0.7.13.1 persistent-sqlite-2.13.1.0 torrent-10000.1.1 uuid-1.3.15 yesod-1.6.2.1
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external
operating system: linux x86_64
supported repository versions: 8 9 10
upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10
Fixes https://github.com/neuropoly/gitea/issues/11
Tests:
* `git annex init`
* `git annex copy --from origin`
* `git annex copy --to origin`
over:
* ssh
for:
* the owner
* a collaborator
* a read-only collaborator
* a stranger
in a
* public repo
* private repo
And then confirms:
* Deletion of the remote repo (to ensure lockdown isn't messing with us: https://git-annex.branchable.com/internals/lockdown/#comment-0cc5225dc5abe8eddeb843bfd2fdc382)
------
To support all this:
* Add util.FileCmp()
* Patch withKeyFile() so it can be nested in other copies of itself
-------
Many thanks to Mathieu for giving style tips and catching several bugs,
including a subtle one in util.filecmp() which neutered it.
Co-authored-by: Mathieu Guay-Paquet <mathieu.guay-paquet@polymtl.ca>
[git-annex](https://git-annex.branchable.com/) is a more complicated cousin to
git-lfs, storing large files in an optional-download side content. Unlike lfs,
it allows mixing and matching storage remotes, so the content remote(s) doesn't
need to be on the same server as the git remote, making it feasible to scatter
a collection across cloud storage, old harddrives, or anywhere else storage can
be scavenged. Since this can get complicated, fast, it has a content-tracking
database (`git annex whereis`) to help find everything later.
The use-case we imagine for including it in Gitea is just the simple case, where
we're primarily emulating git-lfs: each repo has its large content at the same URL.
Our motivation is so we can self-host https://www.datalad.org/ datasets, which
currently are only hostable by fragilely scrounging together cloud storage --
and having to manage all the credentials associated with all the pieces -- or at
https://openneuro.org which is fragile in its own ways.
Supporting git-annex also allows multiple Gitea instance to be annex remotes for
each other, mirroring the content or otherwise collaborating the split up the
hosting costs.
Enabling
--------
TODO
HTTP
----
TODO
Permission Checking
-------------------
This tweaks the API in routers/private/serv.go to expose the calling user's
computed permission, instead of just returning HTTP 403.
This doesn't fit in super well. It's the opposite from how the git-lfs support is
done, where there's a complete list of possible subcommands and their matching
permission levels, and then the API compares the requested with the actual level
and returns HTTP 403 if the check fails.
But it's necessary. The main git-annex verbs, 'git-annex-shell configlist' and
'git-annex-shell p2pstdio' are both either read-only or read-write operations,
depending on the state on disk on either end of the connection and what the user
asked it to ask for, with no way to know before git-annex examines the situation.
So tell the level via GIT_ANNEX_READONLY and trust it to handle itself.
In the older Gogs version, the permission was directly read in cmd/serv.go:
```
mode, err = db.UserAccessMode(user.ID, repo)
```
- 966e925cf3/internal/cmd/serv.go (L334)
but in Gitea permission enforcement has been centralized in the API layer.
(perhaps so the cmd layer can avoid making direct DB connections?)
Deletion
--------
git-annex has this "lockdown" feature where it tries
really quite very hard to prevent you deleting its
data, to the point that even an rm -rf won't do it:
each file in annex/objects/ is nested inside a
folder with read-only permissions.
The recommended workaround is to run chmod -R +w when
you're sure you actually want to delete a repo. See
https://git-annex.branchable.com/internals/lockdown
So we edit util.RemoveAll() to do just that, so now
it's `chmod -R +w && rm -rf` instead of just `rm -rf`.
The WIP prefix toggling link on the sidebar only supported toggling
the *first* prefix specified in
`[repository.pullrequest].WORK_IN_PROGRESS_PREFIXES`. If the pull
request had a title with any other prefix, the first prefix listed in
the config was added (and then removed on toggling it off).
This little change makes all of the prefixes available for the
JavaScript function that does the toggling, and changes said function to
find the used prefix first, and toggle that.
When adding the prefix, it will still default to adding the first one
listed in the configuration, but it will happily remove any others if
those are present.
Fixes#3377.
Signed-off-by: Gergely Nagy <forgejo@gergo.csillger.hu>
(cherry picked from commit a93a99eef3)
Motivation: The meaning of the button is apparent from the visual
position and the number icon. This is not exposed to a screenreader.
Naming it to "Finish Review" helps with to provide the meaning of the
button as well as the number in the label.
(cherry picked from commit 87d4746f5e)
Fixes: https://github.com/go-gitea/gitea/issues/30384
On repo settings page, there id `repo_name` was used 5 times on the same
page, some in modal and such. I think we are better off just
auto-generating these IDs in the future so that labels link up with
their form element.
Ideally this id generation would be done in backend in a subtemplate,
but seeing that we already have similar JS patches for checkboxes, I
took the easy path for now.
I also checked that these `#repo_name` were not in use in JS and the
only case where this id appears in JS is on the migration page where
it's still there.
---------
Co-authored-by: Giteabot <teabot@gitea.io>
(cherry picked from commit d4ec6b3d16496ce3b479d5a08f79823122dc2b7b)
Conflicts:
- templates/repo/settings/options.tmpl
Conflict resolved by manually removing all `id` and `for`
attributes from elements that had `repo_name` as their id.
(cherry picked from commit a01387f5b1)
Backport #30520 by @KN4CK3R
Fixes#28255
The new query uses the id field to sort by "newer". This most not be
correct (usually it is) but it's faster (see #28255).
If someone has a better idea, please propose changes.
Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
(cherry picked from commit ea2ea8ef28b6a2207ec00bafaf42d428612d69eb)
Follow https://github.com/go-gitea/gitea/pull/30357
When user push to default branch, the schedule trigger user will be the
user.
When disable then enable action units in settings, the schedule trigger
user will be action user.
When repo is a mirror, the schedule trigger user will be action user. (
before it will return error, fixed by #30357)
As scheduled job is a cron, the trigger user should be action user from
Gitea, not a real user.
---------
Co-authored-by: Giteabot <teabot@gitea.io>
(cherry picked from commit cb6814adad4dc81a683b50826a211ce7bce731d7)
Conflicts:
- services/actions/notifier_helper.go
Conflict resolved by keeping Forgejo's version of the line.
(cherry picked from commit 829c3c6838)
Backport #30584 by @wolfogre
Related to #30375.
It doesn't make sense to import `modules/web/middleware` and
`modules/setting` in `modules/web/session` since the last one is more
low-level.
And it looks like a workaround to call `DeleteLegacySiteCookie` in
`RegenerateSession`, so maybe we could reverse the importing by
registering hook functions.
Co-authored-by: Jason Song <i@wolfogre.com>
(cherry picked from commit 199397a852ec2d45524cefcc3c119fce4710560e)
Backport #30548 by @silverwind
Fixes https://github.com/go-gitea/gitea/issues/28114 and behaviour
matches vscode on desktop as well.
Co-authored-by: silverwind <me@silverwind.io>
(cherry picked from commit c9633f2d74490211ffd9fd6b3a17180e86fa1fb9)
Backport #30291 by @edwardzhanged
Add some logic in `convert.ToBranchProtection` to return only the names
associated with readAccess instead of returning all names. This will
ensure consistency in behavior between the frontend and backend.
Fixes: #27694
Co-authored-by: Edward Zhang <45360012+edwardzhanged@users.noreply.github.com>
Co-authored-by: techknowlogick <techknowlogick@gitea.com>
Co-authored-by: wenzhuo.zhang <wenzhuo.zhang@geely.com>
(cherry picked from commit d88958bb99eabc07dead6965e396755e7b6d947f)
Fixes: https://github.com/go-gitea/gitea/issues/30512
I think this does mean those tools would run on a potential `vendor`
directory, but I'm not sure we really support vendoring of dependencies
anymore.
`release` has a `vendor` prerequisite so likely the source tarballs
contain vendor files?
(cherry picked from commit 8e12ef911a1d10dedb03e3127c42ca76f9850aca)
Conflicts:
- Makefile
Manually adjusted the changes.
(cherry picked from commit 3918db10c8)
Backport #30511 by @yp05327
Fix#30508
ps: if `sourceID` is not set, `LoginName` will be ignored
Co-authored-by: yp05327 <576951401@qq.com>
(cherry picked from commit f9a025f6a358aa6e34408743a59f5081f397d47c)
Backport #30484 by @silverwind
Small tweak here to prevent this and likely other events from
overflowing in the timeline:
<img width="895" alt="Screenshot 2024-04-14 at 22 53 17"
src="https://github.com/go-gitea/gitea/assets/115237/001b4f6b-f649-44ff-b2f0-c8e0dedeb384">
Co-authored-by: silverwind <me@silverwind.io>
(cherry picked from commit 2efc81d200106cd8707dae1a1bf50ea586b4c846)
Unify the behaviors of "user create" and "user change-password".
Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
(cherry picked from commit 4c6e2da088cf092a9790df5c84b7b338508fede7)
Conflicts:
- cmd/admin_user_create.go
Resolved by favoring Gitea's version of the conflicting areas.
- docs/content/administration/command-line.en-us.md
Removed, Gitea specific.
(cherry picked from commit b122c6ef8b)
Backport #30375 by @jtran
Cookies may exist on "/subpath" and "/subpath/" for some legacy reasons
(eg: changed CookiePath behavior in code). The legacy cookie should be
removed correctly.
Co-authored-by: Jonathan Tran <jonnytran@gmail.com>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: Kyle D <kdumontnu@gmail.com>
(cherry picked from commit e64926c5193e9ccc30b34f187d96c74d104179ae)
