This makes it such that annexed files are treated like plain git files
in comparisons (e.g. the diff of a commit).
It also changes the image diff viewer to show a more reasonable error
message when one of the annexed files under comparison is missing.
Fixes#56.
Reviewed-on: https://codeberg.org/forgejo-aneksajo/forgejo-aneksajo/pulls/57
Co-authored-by: Matthias Riße <m.risse@fz-juelich.de>
Co-committed-by: Matthias Riße <m.risse@fz-juelich.de>
The previous implementation could lead to errors e.g. when a repository
was removed and a new one with a new UUID was created under the same
name. This now checks the validity of the retrieved cache entry every
time and invalidates the cache if necessary.
Reviewed-on: https://codeberg.org/forgejo-aneksajo/forgejo-aneksajo/pulls/60
Co-authored-by: Matthias Riße <m.risse@fz-juelich.de>
Co-committed-by: Matthias Riße <m.risse@fz-juelich.de>
This eliminates the wait time for the first p2phttp connection since
server startup at the cost of adding that time to the startup itself.
Reviewed-on: https://codeberg.org/forgejo-aneksajo/forgejo-aneksajo/pulls/59
Co-authored-by: Matthias Riße <m.risse@fz-juelich.de>
Co-committed-by: Matthias Riße <m.risse@fz-juelich.de>
Always walking the filesystem and searching for UUIDs slowed p2phttp
operations down significantly on a production server with more than a
handful of repositories. This caching strategy ensures that only the
first call is rather slow, and subsequent ones should be much faster.
This should better be implemented as a background job, but for now this
is a simple solution to the problem.
Fixes#53.
Reviewed-on: https://codeberg.org/forgejo-aneksajo/forgejo-aneksajo/pulls/54
Co-authored-by: Matthias Riße <m.risse@fz-juelich.de>
Co-committed-by: Matthias Riße <m.risse@fz-juelich.de>
This adds a new endpoint under `/git-annex-p2phttp` which acts as an
authenticating proxy to git-annex' p2phttp server. This makes it
possible to set `annex+<server-url>/git-annex-p2phttp` as
`remote.<name>.annexurl` and use git-annex fully over http(s) with the
normal credentials and access tokens provided by Forgejo.
Fixes#25.
Reviewed-on: https://codeberg.org/matrss/forgejo-aneksajo/pulls/42
Co-authored-by: Matthias Riße <m.risse@fz-juelich.de>
Co-committed-by: Matthias Riße <m.risse@fz-juelich.de>
The previous implementation both uploaded to the annex and pushed to the
git repository. This meant that the tests checking that uploads without
permission fail actually could pass when the git push failed but the
git-annex upload didn't. The tests didn't catch the situation where
unauthorized users could modify the annex.
Reviewed-on: https://codeberg.org/matrss/forgejo-aneksajo/pulls/46
Co-authored-by: Matthias Riße <m.risse@fz-juelich.de>
Co-committed-by: Matthias Riße <m.risse@fz-juelich.de>
Previously, an external renderer that matched on an annexed file would
only see its content streamed via `STDIN`, or a temporary file with a copy
of its content would be generated and passed-by-filepath (with
`IS_INPUT_FILE=true`). Whether that happens, is also subject to
`MAX_DISPLAY_FILE_SIZE` (which defaults to 8MB).
This was problematic, because annexed files tend to be large. Moreover,
if present, they already exist as write-protected files on the
file-system. Creating a copy is both expensive and serves no particular
purpose.
This commit changes how external renderers are called.
1) With `IS_INPUT_FILE=true`, the renderer is passed the true location
of an annex key, if present, and an empty path, if not.
2) The original, repository-relative path of the rendering target is
made available to the renderer via the `GITEA_RELATIVE_PATH`
environment variable.
To achieve a lean implementation, the `Blob` of the rendering target
is passed on to the `RenderContext` (because the implementation of
the annex-related functionality is centered on this dtype.
This change makes it less costly to increase `MAX_DISPLAY_FILE_SIZE`,
in order to make large, annexed files eligible for markup rendering,
because no content copies will be made any longer.
External renderers can now use the original file path, with the full
original filename, including extensions, for decision making. For
example, to detect particular compression formats based in a file name
extension, or to alter the rendering based on contextual information
encoded in the file path (e.g., a multi-file data structure with a
particular organization pattern).
Apart from the additional environment variable, there is no change to
the handling of renderers that take their input via `STDIN` (i.e.,
`IS_INPUT_FILE=false`).
Fixes#35.
Reviewed-on: https://codeberg.org/matrss/forgejo-aneksajo/pulls/36
Reviewed-by: matrss <matrss@noreply.codeberg.org>
Co-authored-by: Michael Hanke <michael.hanke@gmail.com>
Co-committed-by: Michael Hanke <michael.hanke@gmail.com>
This implements support for uploading files into the annex using the web
interface.
If a repository is a git-annex-enabled repository all files will be
added to it using git annex add. This means that the repository's
configuration for what to put into the annex (annex.largefiles in
gitattributes) will be respected.
Plain git repositories without git-annex will work as before, directly
uploading to git.
Fixes#5.
Reviewed-on: https://codeberg.org/matrss/forgejo-aneksajo/pulls/21
Co-authored-by: Matthias Riße <m.risse@fz-juelich.de>
Co-committed-by: Matthias Riße <m.risse@fz-juelich.de>
This updates the repo index/file view endpoints so annex files match the way
LFS files are rendered, making annexed files accessible via the web instead of
being black boxes only accessible by git clone.
This mostly just duplicates the existing LFS logic. It doesn't try to combine itself
with the existing logic, to make merging with upstream easier. If upstream ever
decides to accept, I would like to try to merge the redundant logic.
The one bit that doesn't directly copy LFS is my choice to hide annex-symlinks.
LFS files are always _pointer files_ and therefore always render with the "file"
icon and no special label, but annex files come in two flavours: symlinks or
pointer files. I've conflated both kinds to try to give a consistent experience.
The tests in here ensure the correct download link (/media, from the last PR)
renders in both the toolbar and, if a binary file (like most annexed files will be),
in the main pane, but it also adds quite a bit of code to make sure text files
that happen to be annexed are dug out and rendered inline like LFS files are.
This moves the `annexObjectPath()` helper out of the tests and into a
dedicated sub-package as `annex.ContentLocation()`, and expands it with
`.Pointer()` (which validates using `git annex examinekey`),
`.IsAnnexed()` and `.Content()` to make it a more useful module.
The tests retain their own wrapper version of `ContentLocation()`
because I tried to follow close to the API modules/lfs uses, which in
terms of abstract `git.Blob` and `git.TreeEntry` objects, not in terms
of `repoPath string`s which are more convenient for the tests.
This makes HTTP symmetric with SSH clone URLs.
This gives us the fancy feature of _anonymous_ downloads,
so people can access datasets without having to set up an
account or manage ssh keys.
Previously, to access "open access" data shared this way,
users would need to:
1. Create an account on gitea.example.com
2. Create ssh keys
3. Upload ssh keys (and make sure to find and upload the correct file)
4. `git clone git@gitea.example.com:user/dataset.git`
5. `cd dataset`
6. `git annex get`
This cuts that down to just the last three steps:
1. `git clone https://gitea.example.com/user/dataset.git`
2. `cd dataset`
3. `git annex get`
This is significantly simpler for downstream users, especially for those
unfamiliar with the command line.
Unfortunately there's no uploading. While git-annex supports uploading
over HTTP to S3 and some other special remotes, it seems to fail on a
_plain_ HTTP remote. See https://github.com/neuropoly/gitea/issues/7
and https://git-annex.branchable.com/forum/HTTP_uploads/#comment-ce28adc128fdefe4c4c49628174d9b92.
This is not a major loss since no one wants uploading to be anonymous anyway.
To support private repos, I had to hunt down and patch a secret extra security
corner that Gitea only applies to HTTP for some reason (services/auth/basic.go).
This was guided by https://git-annex.branchable.com/tips/setup_a_public_repository_on_a_web_site/
Fixes https://github.com/neuropoly/gitea/issues/3
Co-authored-by: Mathieu Guay-Paquet <mathieu.guaypaquet@polymtl.ca>
Fixes https://github.com/neuropoly/gitea/issues/11
Tests:
* `git annex init`
* `git annex copy --from origin`
* `git annex copy --to origin`
over:
* ssh
for:
* the owner
* a collaborator
* a read-only collaborator
* a stranger
in a
* public repo
* private repo
And then confirms:
* Deletion of the remote repo (to ensure lockdown isn't messing with us: https://git-annex.branchable.com/internals/lockdown/#comment-0cc5225dc5abe8eddeb843bfd2fdc382)
------
To support all this:
* Add util.FileCmp()
* Patch withKeyFile() so it can be nested in other copies of itself
-------
Many thanks to Mathieu for giving style tips and catching several bugs,
including a subtle one in util.filecmp() which neutered it.
Co-authored-by: Mathieu Guay-Paquet <mathieu.guay-paquet@polymtl.ca>
Co-authored-by: Matthias Riße <m.risse@fz-juelich.de>
[git-annex](https://git-annex.branchable.com/) is a more complicated cousin to
git-lfs, storing large files in an optional-download side content. Unlike lfs,
it allows mixing and matching storage remotes, so the content remote(s) doesn't
need to be on the same server as the git remote, making it feasible to scatter
a collection across cloud storage, old harddrives, or anywhere else storage can
be scavenged. Since this can get complicated, fast, it has a content-tracking
database (`git annex whereis`) to help find everything later.
The use-case we imagine for including it in Gitea is just the simple case, where
we're primarily emulating git-lfs: each repo has its large content at the same URL.
Our motivation is so we can self-host https://www.datalad.org/ datasets, which
currently are only hostable by fragilely scrounging together cloud storage --
and having to manage all the credentials associated with all the pieces -- or at
https://openneuro.org which is fragile in its own ways.
Supporting git-annex also allows multiple Gitea instance to be annex remotes for
each other, mirroring the content or otherwise collaborating the split up the
hosting costs.
Enabling
--------
TODO
HTTP
----
TODO
Permission Checking
-------------------
This tweaks the API in routers/private/serv.go to expose the calling user's
computed permission, instead of just returning HTTP 403.
This doesn't fit in super well. It's the opposite from how the git-lfs support is
done, where there's a complete list of possible subcommands and their matching
permission levels, and then the API compares the requested with the actual level
and returns HTTP 403 if the check fails.
But it's necessary. The main git-annex verbs, 'git-annex-shell configlist' and
'git-annex-shell p2pstdio' are both either read-only or read-write operations,
depending on the state on disk on either end of the connection and what the user
asked it to ask for, with no way to know before git-annex examines the situation.
So tell the level via GIT_ANNEX_READONLY and trust it to handle itself.
In the older Gogs version, the permission was directly read in cmd/serv.go:
```
mode, err = db.UserAccessMode(user.ID, repo)
```
- 966e925cf3/internal/cmd/serv.go (L334)
but in Gitea permission enforcement has been centralized in the API layer.
(perhaps so the cmd layer can avoid making direct DB connections?)
Deletion
--------
git-annex has this "lockdown" feature where it tries
really quite very hard to prevent you deleting its
data, to the point that even an rm -rf won't do it:
each file in annex/objects/ is nested inside a
folder with read-only permissions.
The recommended workaround is to run chmod -R +w when
you're sure you actually want to delete a repo. See
https://git-annex.branchable.com/internals/lockdown
So we edit util.RemoveAll() to do just that, so now
it's `chmod -R +w && rm -rf` instead of just `rm -rf`.
**Backport:** https://codeberg.org/forgejo/forgejo/pulls/6525
### Screenshot
#### Before
See #5069.
#### After
### Tests
- I added test coverage for Go changes...
- [x] in their respective `*_test.go` for unit tests.
- [ ] in the `tests/integration` directory if it involves interactions with a live Forgejo server.
- I added test coverage for JavaScript changes...
- [ ] in `web_src/js/*.test.js` if it can be unit tested.
- [ ] in `tests/e2e/*.test.e2e.js` if it requires interactions with a live Forgejo server (see also the [developer guide for JavaScript testing](https://codeberg.org/forgejo/forgejo/src/branch/forgejo/tests/e2e/README.md#end-to-end-tests)).
- Manual testing: see https://codeberg.org/forgejo/forgejo/pulls/6525#issuecomment-2578814. Thanks @earl-warren.
### Documentation
- [ ] I created a pull request [to the documentation](https://codeberg.org/forgejo/docs) to explain to Forgejo users how to use this change.
- [x] I did not document these changes and I do not expect someone else to do it.
### Release notes
- [ ] I do not want this change to show in the release notes.
- [ ] I want the title to show in the release notes with a link to this pull request.
- [ ] I want the content of the `release-notes/<pull request number>.md` to be be used for the release notes instead of the title.
Co-authored-by: Robert Wolff <mahlzahn@posteo.de>
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/6739
Reviewed-by: Robert Wolff <mahlzahn@posteo.de>
Co-authored-by: forgejo-backport-action <forgejo-backport-action@noreply.codeberg.org>
Co-committed-by: forgejo-backport-action <forgejo-backport-action@noreply.codeberg.org>
**Backport:** https://codeberg.org/forgejo/forgejo/pulls/6572
### What?
This fixes the inline file preview for rendered files (e.g., markdown). [Here, a live issue in v11](https://v11.next.forgejo.org/mahlzahn/test-inline-file-preview/issues/1) and [the same in v7 (with even more bugs)](https://v7.next.forgejo.org/mahlzahn/test-inline-file-preview/issues/1).
It fixes
1. the inline preview for possibly rendered files, when the link is specified with `?display=source`. This happens, e.g., if you are watching a (e.g., markdown) file in source and then want to link some of its lines.
2. the link to the source file inside the inline preview for possible rendered files (currently it links to the rendered version and then the `#L…` cannot point to the correct lines). This is done by always adding `?display=source` to the link.
### Screenshots
<details><summary>
#### Before
</summary>
</details>
<details><summary>
#### After
</summary>
</details>
### Tests
- I added test coverage for Go changes...
- [x] in their respective `*_test.go` for unit tests.
- [ ] in the `tests/integration` directory if it involves interactions with a live Forgejo server.
- I added test coverage for JavaScript changes...
- [ ] in `web_src/js/*.test.js` if it can be unit tested.
- [ ] in `tests/e2e/*.test.e2e.js` if it requires interactions with a live Forgejo server (see also the [developer guide for JavaScript testing](https://codeberg.org/forgejo/forgejo/src/branch/forgejo/tests/e2e/README.md#end-to-end-tests)).
I think that this minor edit does not need special tests. Some backend tests have been updated to reflect the addition of URL parameters.
#### Manual testing
- create a repository with a file that can be rendered with couple of lines inside, e.g., a markdown README.md
- go to the source of this file (e.g., `…/src/branch/main/README.md`)
- click on the `<> View Source` button (or add `?display=source` to the URL)
- click on one of the lines, then on the three dots, then on ”Reference in a new issue“
- continue creating the issue
### Documentation
- [ ] I created a pull request [to the documentation](https://codeberg.org/forgejo/docs) to explain to Forgejo users how to use this change.
- [x] I did not document these changes and I do not expect someone else to do it.
### Release notes
- [ ] I do not want this change to show in the release notes.
- [ ] I want the title to show in the release notes with a link to this pull request.
- [ ] I want the content of the `release-notes/<pull request number>.md` to be be used for the release notes instead of the title.
Co-authored-by: Robert Wolff <mahlzahn@posteo.de>
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/6685
Reviewed-by: Gusted <gusted@noreply.codeberg.org>
Co-authored-by: forgejo-backport-action <forgejo-backport-action@noreply.codeberg.org>
Co-committed-by: forgejo-backport-action <forgejo-backport-action@noreply.codeberg.org>
**Backport:** https://codeberg.org/forgejo/forgejo/pulls/6674
- The doctor commands to check the validity of existing usernames and
email addresses depend on functionality that have configurable behavior
depending on the values of the `[service]` settings, so load them when
running the doctor command.
- Resolves#6664
- No unit test due to the architecture of doctor commands.
Co-authored-by: Gusted <postmaster@gusted.xyz>
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/6678
Reviewed-by: Gusted <gusted@noreply.codeberg.org>
Co-authored-by: forgejo-backport-action <forgejo-backport-action@noreply.codeberg.org>
Co-committed-by: forgejo-backport-action <forgejo-backport-action@noreply.codeberg.org>
**Backport:** https://codeberg.org/forgejo/forgejo/pulls/6639
If a row in the two_factor table references a non existent user, it may contain a secret that has an invalid format. Such an orphaned row is never used and should be removed.
Improve the error message to suggest using the doctor to remove it.
Fixes: https://codeberg.org/forgejo/forgejo/issues/6637
## Testing
- make TAGS='sqlite sqlite_unlock_notify' watch
- make TAGS='sqlite sqlite_unlock_notify' forgejo
- sqlite3 data/gitea.db 'INSERT INTO two_factor VALUES( 0, 500, "", "", "", "", 0, 0)'
- ./forgejo doctor check --run check-db-consistency
```
[1] Check consistency of database
- [W] Found 1 Orphaned TwoFactor without existing User
OK
All done (checks: 1).
```
- ./forgejo doctor check --run check-db-consistency --fix
```
[1] Check consistency of database
- [I] Deleted 1 Orphaned TwoFactor without existing User
OK
All done (checks: 1).
```
## Checklist
The [contributor guide](https://forgejo.org/docs/next/contributor/) contains information that will be helpful to first time contributors. There also are a few [conditions for merging Pull Requests in Forgejo repositories](https://codeberg.org/forgejo/governance/src/branch/main/PullRequestsAgreement.md). You are also welcome to join the [Forgejo development chatroom](https://matrix.to/#/#forgejo-development:matrix.org).
### Tests
- I added test coverage for Go changes...
- [ ] in their respective `*_test.go` for unit tests.
- [x] in the `tests/integration` directory if it involves interactions with a live Forgejo server.
- I added test coverage for JavaScript changes...
- [ ] in `web_src/js/*.test.js` if it can be unit tested.
- [ ] in `tests/e2e/*.test.e2e.js` if it requires interactions with a live Forgejo server (see also the [developer guide for JavaScript testing](https://codeberg.org/forgejo/forgejo/src/branch/forgejo/tests/e2e/README.md#end-to-end-tests)).
### Documentation
- [ ] I created a pull request [to the documentation](https://codeberg.org/forgejo/docs) to explain to Forgejo users how to use this change.
- [x] I did not document these changes and I do not expect someone else to do it.
### Release notes
- [ ] I do not want this change to show in the release notes.
- [x] I want the title to show in the release notes with a link to this pull request.
- [ ] I want the content of the `release-notes/<pull request number>.md` to be be used for the release notes instead of the title.
<!--start release-notes-assistant-->
## Release notes
<!--URL:https://codeberg.org/forgejo/forgejo-->
- Bug fixes
- [PR](https://codeberg.org/forgejo/forgejo/pulls/6651): <!--number 6651 --><!--line 0 --><!--description Zml4OiB0ZWFjaCB0aGUgZG9jdG9yIGFib3V0IG9ycGhhbmVkIHR3b19mYWN0b3Igcm93cw==-->fix: teach the doctor about orphaned two_factor rows<!--description-->
<!--end release-notes-assistant-->
Co-authored-by: Earl Warren <contact@earl-warren.org>
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/6651
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Co-authored-by: forgejo-backport-action <forgejo-backport-action@noreply.codeberg.org>
Co-committed-by: forgejo-backport-action <forgejo-backport-action@noreply.codeberg.org>
When the Forgejo version is displayed in the footer, the metadata
should not be displayed. It was once an indication that Forgejo
includes all of Gitea. But since the hard fork the codebase diverged
and this is no longer accurate.
The metadata is still displayed in the API, admin panels or headers
for the sake of backward compatibility.
Refs: https://codeberg.org/forgejo/discussions/issues/244
(cherry picked from commit 2c81893c76)
Fix#32428.
Patch the regex to match `,`besides `.` `"` `'` `:` and space.
(cherry picked from commit fb75151fb14e866cab8d893c588f5673902b278b)
(cherry picked from commit 0efd7130f0)
Harden the current checks in place, I doubt these will ever hit (you can
prove easily by reading the current source code this cannot happen) but
just in case a new Go version does something weird or something else
goes catastrophicly wrong, this should add an extra defense-in-depth
layer.
`n != aeadKeySize` will panic a nil error, don't think it's needed to
add more logic to this, a nil error is enough to indicate that that
condition failed (given the other condition is `err != nil`).
Also move constant integers to being `const`, this helps reducing the
amount of instructions being done for the extra check.
This commit allows chaning default branch update style through global
and repository settings. The setting affects "Update branch" button
in PR view (button shows when some commits are ahead of master branch).
When default update style is set to "rebase", dropdown button updates branch
by rebase by default. When update style is set to other value, dropdown button
updates branch by merge. Any of these actions may be selected using dropdown
in any case.
Signed-off-by: George Bartolomey <george@bh4.ru>
Added support for searching content in a specific directory or file.
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/6143
Reviewed-by: Gusted <gusted@noreply.codeberg.org>
Reviewed-by: 0ko <0ko@noreply.codeberg.org>
Co-authored-by: Shiny Nematoda <snematoda.751k2@aleeas.com>
Co-committed-by: Shiny Nematoda <snematoda.751k2@aleeas.com>
- [Go 1.24](https://groups.google.com/g/golang-announce/c/vYMfuq_XO6w)
is currently out for rc1.
- Using it to test unit tests and integration testing it failed horribly
with strange panics and errors, it is caused by
ca63101df4
and Forgejo trying to access the wrong internal data structures that
have been changed in Go 1.24.
- Use the new data structure for Go 1.24 and above.
Fix: #32611
In some lfs server implementations, they require the ref attribute.
---------
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
(cherry picked from commit 7616aeb2ea2a02c15480dcd4a232e98081569690)
- The root cause is described in b4f1988a35
- Move to a fork of `github.com/gliderlabs/ssh` that exposes the
permissions that was chosen by `x/crypto/ssh` after succesfully
authenticating, this is the recommended mitigation by the Golang
security team. The fork exposes this, since `gliderlabs/ssh` instead
relies on context values to do so, which is vulnerable to the same
attack, although partially mitigated by the fix in `x/crypto/ssh` it
would not be good practice and defense deep to rely on it.
- Existing tests covers that the functionality is preserved.
- No tests are added to ensure it fixes the described security, the
exploit relies on non-standard SSH behavior it would be too hard to
craft SSH packets to exploit this.
- A softbreak was being preserved during the github legacy callout (this
is likely due to a change in Goldmark) while it should not. This caused
an `<br>` to be present between the attention title and attention content.
- Added unit test.
This fixes a TODO in the code to validate the RedirectURIs when adding
or editing an OAuth application in user settings.
This also includes a refactor of the user settings tests to only create
the DB once per top-level test to avoid reloading fixtures.
(cherry picked from commit 16a7d343d78807e39df124756e5d43a69a2203a3)
Conflicts:
services/forms/user_form.go
tests/integration/user_settings_test.go
simple conflicts
