The `checkValidity` function returns `false, err` if its `git config`
call returns an error. For renamed repositories the cached path no
longer exists, which makes `git config` return an error. When
`checkValidity` returns an error value it short-circuited the cache
lookup, instead of deleting the stale entry and retrying. This made the
`AnnexP2PHTTP` route 404.
This is fixed by ignoring the error from `checkValidity` entirely.
Fixes#88.
Reviewed-on: https://codeberg.org/forgejo-aneksajo/forgejo-aneksajo/pulls/89
Co-authored-by: Matthias Riße <m.risse@fz-juelich.de>
Co-committed-by: Matthias Riße <m.risse@fz-juelich.de>
The issue was that the caching mechanism for the UUID to repository path
association tried to be too smart and ended up buggy. This removes the
parts that skip updating the UUIDs for already-in-cache repository
paths. This change makes updating the cache more expensive, but since
https://codeberg.org/forgejo-aneksajo/forgejo-aneksajo/pulls/65 is
merged it should be fast enough to be fine.
Fixes#83.
Reviewed-on: https://codeberg.org/forgejo-aneksajo/forgejo-aneksajo/pulls/86
Co-authored-by: Matthias Riße <m.risse@fz-juelich.de>
Co-committed-by: Matthias Riße <m.risse@fz-juelich.de>
This changes the PR merge process such that annexed files contained in the to-be-merged commits are copied from the head repository to the base repository as part of the merge, similar to how it is done for LFS files.
Fixes#11.
The [contributor guide](https://forgejo.org/docs/next/contributor/) contains information that will be helpful to first time contributors. There also are a few [conditions for merging Pull Requests in Forgejo repositories](https://codeberg.org/forgejo/governance/src/branch/main/PullRequestsAgreement.md). You are also welcome to join the [Forgejo development chatroom](https://matrix.to/#/#forgejo-development:matrix.org).
- I added test coverage for Go changes...
- [ ] in their respective `*_test.go` for unit tests.
- [x] in the `tests/integration` directory if it involves interactions with a live Forgejo server.
- I added test coverage for JavaScript changes...
- [ ] in `web_src/js/*.test.js` if it can be unit tested.
- [ ] in `tests/e2e/*.test.e2e.js` if it requires interactions with a live Forgejo server (see also the [developer guide for JavaScript testing](https://codeberg.org/forgejo/forgejo/src/branch/forgejo/tests/e2e/README.md#end-to-end-tests)).
- [ ] I created a pull request [to the documentation](https://codeberg.org/forgejo/docs) to explain to Forgejo users how to use this change.
- [ ] I did not document these changes and I do not expect someone else to do it.
- [x] I do not want this change to show in the release notes.
- [ ] I want the title to show in the release notes with a link to this pull request.
- [ ] I want the content of the `release-notes/<pull request number>.md` to be be used for the release notes instead of the title.
Reviewed-on: https://codeberg.org/forgejo-aneksajo/forgejo-aneksajo/pulls/62
Co-authored-by: Matthias Riße <m.risse@fz-juelich.de>
Co-committed-by: Matthias Riße <m.risse@fz-juelich.de>
Globbing for the config files is marginally faster than walking the
directory and checking for config files. Replacing the expensive calls
to `git config` for each repository with reading the repository's config
as an ini file is two orders of magnitude faster.
All in all this reduces the required time initializing the cache for
approx. 3000 repositories from approx. 5s to 50ms.
The server startup now also logs how long the cache update took and the
cache update is only done if p2phttp support is not disabled, because
p2phttp support is currently the only feature that requires the UUID
cache.
Fixes#63, fixes#64.
The [contributor guide](https://forgejo.org/docs/next/contributor/) contains information that will be helpful to first time contributors. There also are a few [conditions for merging Pull Requests in Forgejo repositories](https://codeberg.org/forgejo/governance/src/branch/main/PullRequestsAgreement.md). You are also welcome to join the [Forgejo development chatroom](https://matrix.to/#/#forgejo-development:matrix.org).
- I added test coverage for Go changes...
- [ ] in their respective `*_test.go` for unit tests.
- [ ] in the `tests/integration` directory if it involves interactions with a live Forgejo server.
- I added test coverage for JavaScript changes...
- [ ] in `web_src/js/*.test.js` if it can be unit tested.
- [ ] in `tests/e2e/*.test.e2e.js` if it requires interactions with a live Forgejo server (see also the [developer guide for JavaScript testing](https://codeberg.org/forgejo/forgejo/src/branch/forgejo/tests/e2e/README.md#end-to-end-tests)).
- [ ] I created a pull request [to the documentation](https://codeberg.org/forgejo/docs) to explain to Forgejo users how to use this change.
- [x] I did not document these changes and I do not expect someone else to do it.
- [x] I do not want this change to show in the release notes.
- [ ] I want the title to show in the release notes with a link to this pull request.
- [ ] I want the content of the `release-notes/<pull request number>.md` to be be used for the release notes instead of the title.
Reviewed-on: https://codeberg.org/forgejo-aneksajo/forgejo-aneksajo/pulls/65
Co-authored-by: Matthias Riße <m.risse@fz-juelich.de>
Co-committed-by: Matthias Riße <m.risse@fz-juelich.de>
This makes it such that annexed files are treated like plain git files
in comparisons (e.g. the diff of a commit).
It also changes the image diff viewer to show a more reasonable error
message when one of the annexed files under comparison is missing.
Fixes#56.
Reviewed-on: https://codeberg.org/forgejo-aneksajo/forgejo-aneksajo/pulls/57
Co-authored-by: Matthias Riße <m.risse@fz-juelich.de>
Co-committed-by: Matthias Riße <m.risse@fz-juelich.de>
The previous implementation could lead to errors e.g. when a repository
was removed and a new one with a new UUID was created under the same
name. This now checks the validity of the retrieved cache entry every
time and invalidates the cache if necessary.
Reviewed-on: https://codeberg.org/forgejo-aneksajo/forgejo-aneksajo/pulls/60
Co-authored-by: Matthias Riße <m.risse@fz-juelich.de>
Co-committed-by: Matthias Riße <m.risse@fz-juelich.de>
This eliminates the wait time for the first p2phttp connection since
server startup at the cost of adding that time to the startup itself.
Reviewed-on: https://codeberg.org/forgejo-aneksajo/forgejo-aneksajo/pulls/59
Co-authored-by: Matthias Riße <m.risse@fz-juelich.de>
Co-committed-by: Matthias Riße <m.risse@fz-juelich.de>
Always walking the filesystem and searching for UUIDs slowed p2phttp
operations down significantly on a production server with more than a
handful of repositories. This caching strategy ensures that only the
first call is rather slow, and subsequent ones should be much faster.
This should better be implemented as a background job, but for now this
is a simple solution to the problem.
Fixes#53.
Reviewed-on: https://codeberg.org/forgejo-aneksajo/forgejo-aneksajo/pulls/54
Co-authored-by: Matthias Riße <m.risse@fz-juelich.de>
Co-committed-by: Matthias Riße <m.risse@fz-juelich.de>
This adds a new endpoint under `/git-annex-p2phttp` which acts as an
authenticating proxy to git-annex' p2phttp server. This makes it
possible to set `annex+<server-url>/git-annex-p2phttp` as
`remote.<name>.annexurl` and use git-annex fully over http(s) with the
normal credentials and access tokens provided by Forgejo.
Fixes#25.
Reviewed-on: https://codeberg.org/matrss/forgejo-aneksajo/pulls/42
Co-authored-by: Matthias Riße <m.risse@fz-juelich.de>
Co-committed-by: Matthias Riße <m.risse@fz-juelich.de>
The previous implementation both uploaded to the annex and pushed to the
git repository. This meant that the tests checking that uploads without
permission fail actually could pass when the git push failed but the
git-annex upload didn't. The tests didn't catch the situation where
unauthorized users could modify the annex.
Reviewed-on: https://codeberg.org/matrss/forgejo-aneksajo/pulls/46
Co-authored-by: Matthias Riße <m.risse@fz-juelich.de>
Co-committed-by: Matthias Riße <m.risse@fz-juelich.de>
Previously, an external renderer that matched on an annexed file would
only see its content streamed via `STDIN`, or a temporary file with a copy
of its content would be generated and passed-by-filepath (with
`IS_INPUT_FILE=true`). Whether that happens, is also subject to
`MAX_DISPLAY_FILE_SIZE` (which defaults to 8MB).
This was problematic, because annexed files tend to be large. Moreover,
if present, they already exist as write-protected files on the
file-system. Creating a copy is both expensive and serves no particular
purpose.
This commit changes how external renderers are called.
1) With `IS_INPUT_FILE=true`, the renderer is passed the true location
of an annex key, if present, and an empty path, if not.
2) The original, repository-relative path of the rendering target is
made available to the renderer via the `GITEA_RELATIVE_PATH`
environment variable.
To achieve a lean implementation, the `Blob` of the rendering target
is passed on to the `RenderContext` (because the implementation of
the annex-related functionality is centered on this dtype.
This change makes it less costly to increase `MAX_DISPLAY_FILE_SIZE`,
in order to make large, annexed files eligible for markup rendering,
because no content copies will be made any longer.
External renderers can now use the original file path, with the full
original filename, including extensions, for decision making. For
example, to detect particular compression formats based in a file name
extension, or to alter the rendering based on contextual information
encoded in the file path (e.g., a multi-file data structure with a
particular organization pattern).
Apart from the additional environment variable, there is no change to
the handling of renderers that take their input via `STDIN` (i.e.,
`IS_INPUT_FILE=false`).
Fixes#35.
Reviewed-on: https://codeberg.org/matrss/forgejo-aneksajo/pulls/36
Reviewed-by: matrss <matrss@noreply.codeberg.org>
Co-authored-by: Michael Hanke <michael.hanke@gmail.com>
Co-committed-by: Michael Hanke <michael.hanke@gmail.com>
This implements support for uploading files into the annex using the web
interface.
If a repository is a git-annex-enabled repository all files will be
added to it using git annex add. This means that the repository's
configuration for what to put into the annex (annex.largefiles in
gitattributes) will be respected.
Plain git repositories without git-annex will work as before, directly
uploading to git.
Fixes#5.
Reviewed-on: https://codeberg.org/matrss/forgejo-aneksajo/pulls/21
Co-authored-by: Matthias Riße <m.risse@fz-juelich.de>
Co-committed-by: Matthias Riße <m.risse@fz-juelich.de>
This updates the repo index/file view endpoints so annex files match the way
LFS files are rendered, making annexed files accessible via the web instead of
being black boxes only accessible by git clone.
This mostly just duplicates the existing LFS logic. It doesn't try to combine itself
with the existing logic, to make merging with upstream easier. If upstream ever
decides to accept, I would like to try to merge the redundant logic.
The one bit that doesn't directly copy LFS is my choice to hide annex-symlinks.
LFS files are always _pointer files_ and therefore always render with the "file"
icon and no special label, but annex files come in two flavours: symlinks or
pointer files. I've conflated both kinds to try to give a consistent experience.
The tests in here ensure the correct download link (/media, from the last PR)
renders in both the toolbar and, if a binary file (like most annexed files will be),
in the main pane, but it also adds quite a bit of code to make sure text files
that happen to be annexed are dug out and rendered inline like LFS files are.
This moves the `annexObjectPath()` helper out of the tests and into a
dedicated sub-package as `annex.ContentLocation()`, and expands it with
`.Pointer()` (which validates using `git annex examinekey`),
`.IsAnnexed()` and `.Content()` to make it a more useful module.
The tests retain their own wrapper version of `ContentLocation()`
because I tried to follow close to the API modules/lfs uses, which in
terms of abstract `git.Blob` and `git.TreeEntry` objects, not in terms
of `repoPath string`s which are more convenient for the tests.
This makes HTTP symmetric with SSH clone URLs.
This gives us the fancy feature of _anonymous_ downloads,
so people can access datasets without having to set up an
account or manage ssh keys.
Previously, to access "open access" data shared this way,
users would need to:
1. Create an account on gitea.example.com
2. Create ssh keys
3. Upload ssh keys (and make sure to find and upload the correct file)
4. `git clone git@gitea.example.com:user/dataset.git`
5. `cd dataset`
6. `git annex get`
This cuts that down to just the last three steps:
1. `git clone https://gitea.example.com/user/dataset.git`
2. `cd dataset`
3. `git annex get`
This is significantly simpler for downstream users, especially for those
unfamiliar with the command line.
Unfortunately there's no uploading. While git-annex supports uploading
over HTTP to S3 and some other special remotes, it seems to fail on a
_plain_ HTTP remote. See https://github.com/neuropoly/gitea/issues/7
and https://git-annex.branchable.com/forum/HTTP_uploads/#comment-ce28adc128fdefe4c4c49628174d9b92.
This is not a major loss since no one wants uploading to be anonymous anyway.
To support private repos, I had to hunt down and patch a secret extra security
corner that Gitea only applies to HTTP for some reason (services/auth/basic.go).
This was guided by https://git-annex.branchable.com/tips/setup_a_public_repository_on_a_web_site/
Fixes https://github.com/neuropoly/gitea/issues/3
Co-authored-by: Mathieu Guay-Paquet <mathieu.guaypaquet@polymtl.ca>
Fixes https://github.com/neuropoly/gitea/issues/11
Tests:
* `git annex init`
* `git annex copy --from origin`
* `git annex copy --to origin`
over:
* ssh
for:
* the owner
* a collaborator
* a read-only collaborator
* a stranger
in a
* public repo
* private repo
And then confirms:
* Deletion of the remote repo (to ensure lockdown isn't messing with us: https://git-annex.branchable.com/internals/lockdown/#comment-0cc5225dc5abe8eddeb843bfd2fdc382)
------
To support all this:
* Add util.FileCmp()
* Patch withKeyFile() so it can be nested in other copies of itself
-------
Many thanks to Mathieu for giving style tips and catching several bugs,
including a subtle one in util.filecmp() which neutered it.
Co-authored-by: Mathieu Guay-Paquet <mathieu.guay-paquet@polymtl.ca>
Co-authored-by: Matthias Riße <m.risse@fz-juelich.de>
[git-annex](https://git-annex.branchable.com/) is a more complicated cousin to
git-lfs, storing large files in an optional-download side content. Unlike lfs,
it allows mixing and matching storage remotes, so the content remote(s) doesn't
need to be on the same server as the git remote, making it feasible to scatter
a collection across cloud storage, old harddrives, or anywhere else storage can
be scavenged. Since this can get complicated, fast, it has a content-tracking
database (`git annex whereis`) to help find everything later.
The use-case we imagine for including it in Gitea is just the simple case, where
we're primarily emulating git-lfs: each repo has its large content at the same URL.
Our motivation is so we can self-host https://www.datalad.org/ datasets, which
currently are only hostable by fragilely scrounging together cloud storage --
and having to manage all the credentials associated with all the pieces -- or at
https://openneuro.org which is fragile in its own ways.
Supporting git-annex also allows multiple Gitea instance to be annex remotes for
each other, mirroring the content or otherwise collaborating the split up the
hosting costs.
Enabling
--------
TODO
HTTP
----
TODO
Permission Checking
-------------------
This tweaks the API in routers/private/serv.go to expose the calling user's
computed permission, instead of just returning HTTP 403.
This doesn't fit in super well. It's the opposite from how the git-lfs support is
done, where there's a complete list of possible subcommands and their matching
permission levels, and then the API compares the requested with the actual level
and returns HTTP 403 if the check fails.
But it's necessary. The main git-annex verbs, 'git-annex-shell configlist' and
'git-annex-shell p2pstdio' are both either read-only or read-write operations,
depending on the state on disk on either end of the connection and what the user
asked it to ask for, with no way to know before git-annex examines the situation.
So tell the level via GIT_ANNEX_READONLY and trust it to handle itself.
In the older Gogs version, the permission was directly read in cmd/serv.go:
```
mode, err = db.UserAccessMode(user.ID, repo)
```
- 966e925cf3/internal/cmd/serv.go (L334)
but in Gitea permission enforcement has been centralized in the API layer.
(perhaps so the cmd layer can avoid making direct DB connections?)
Deletion
--------
git-annex has this "lockdown" feature where it tries
really quite very hard to prevent you deleting its
data, to the point that even an rm -rf won't do it:
each file in annex/objects/ is nested inside a
folder with read-only permissions.
The recommended workaround is to run chmod -R +w when
you're sure you actually want to delete a repo. See
https://git-annex.branchable.com/internals/lockdown
So we edit util.RemoveAll() to do just that, so now
it's `chmod -R +w && rm -rf` instead of just `rm -rf`.
**Backport:** https://codeberg.org/forgejo/forgejo/pulls/8633
This reverts commit b2a3966e64.
weblate etc. are using this method and need to be updated before the change is enforced.
Co-authored-by: Earl Warren <contact@earl-warren.org>
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/8653
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Co-authored-by: forgejo-backport-action <forgejo-backport-action@noreply.codeberg.org>
Co-committed-by: forgejo-backport-action <forgejo-backport-action@noreply.codeberg.org>
**Backport:** https://codeberg.org/forgejo/forgejo/pulls/8596
- This reverts behavior that was partially unintentionally introduced in forgejo/forgejo#8143, symbolic links were no longer followed (if they escaped the asset folder) for local assets.
- Having symbolic links for user-added files is, to my understanding, a ,common usecase for NixOS and would thus have symbolic links in the asset folders. Avoiding symbolic links is not easy.
- The previous code used `http.Dir`, we cannot use that as it's not of the same type. The equivalent is `os.DirFS`.
- Unit test to prevent this regression from happening again.
Reported-by: bloxx12 (Matrix).
Co-authored-by: Gusted <postmaster@gusted.xyz>
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/8610
Reviewed-by: Gusted <gusted@noreply.codeberg.org>
Co-authored-by: forgejo-backport-action <forgejo-backport-action@noreply.codeberg.org>
Co-committed-by: forgejo-backport-action <forgejo-backport-action@noreply.codeberg.org>
**Backport:** https://codeberg.org/forgejo/forgejo/pulls/8550
- The `ACME_URL` setting is documented to default to Let's encrypt
production server if left empty, so do precisely that.
- Use a HTTP proxy to communicate with ACME if Forgejo is configured to
use that.
- Regression of forgejo/forgejo#7409 (previously certmagic took care of
setting these defaults).
- Resolvesforgejo/forgejo#8548
## Testing
1. Configure Forgejo's root URL to a public facing domain (that can pass a ACME challenge)
2. Configure Forgejo to use ACME by setting `[server].ENABLE_ACME = true` and `[server].ACME_ACCEPTTOS = true`.
3. Start Forgejo.
4. Observe that it's available via https.
Co-authored-by: Gusted <postmaster@gusted.xyz>
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/8552
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Co-authored-by: forgejo-backport-action <forgejo-backport-action@noreply.codeberg.org>
Co-committed-by: forgejo-backport-action <forgejo-backport-action@noreply.codeberg.org>
**Backport:** https://codeberg.org/forgejo/forgejo/pulls/8475closes#7946
- The `rpmsRepoPattern` regex has been fixed to handle releases with dots correctly. For example, the version `0.9.0-alt1.git.17.g2ba905d` is valid, just like `0.1.0-1.n1` mentioned in the issue (https://codeberg.org/forgejo/forgejo/issues/7946#issue-1628991)
- getEntries now returns entry names. In the integration tests, there were lines like:
```go
assert.Equal(t, []string{"", ""}, result.ProvideNames)
```
and it’s unclear how such test logic could have ever worked correctly (fixes problems with deps https://codeberg.org/forgejo/forgejo/issues/7946#issuecomment-5109795)
- ALT is an acronym for ALT Linux Team, so `Alt` was replaced with `ALT`. Strictly speaking, it should probably be `ALT Linux`, but since we use `Arch` instead of `Arch Linux`, this seems fine. Also, Distrowatch shows `Arch`/`ALT` in its dropdown, so it’s consistent.
- The strings `"Alt Linux Team"` and `"Sisyphus"` in the `Origin` and `Suite` fields have been replaced with `setting.AppName` and `"Unknown"`. `Unknown` is a valid value and is set by default, so this won’t cause any issues.
- The documentation link has been fixed: (404 docs.gitea.com/usage/packages/alt/ -> 200 forgejo.org/docs/latest/user/packages/alt/)
Co-authored-by: Maxim Slipenko <maks1ms@altlinux.org>
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/8480
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Co-authored-by: forgejo-backport-action <forgejo-backport-action@noreply.codeberg.org>
Co-committed-by: forgejo-backport-action <forgejo-backport-action@noreply.codeberg.org>
**Backport:** https://codeberg.org/forgejo/forgejo/pulls/8261
Query string tokenizer could return a list containing empty tokens when the query string was `\` or `"` (probably in other scenarios as well).
This seems undesirable and is what triggered #8260, but I'm posting this separately from that fix in case I'm wrong. Feel free to reject if so.
The actual change in behavior is that now searching for `\` or `"` behaves the same as if the query were empty (the bleve/elastic code checks that the tokenizer actually returned, anything rather than just query being non-empty).
### Tests
- I added test coverage for Go changes...
- [x] in their respective `*_test.go` for unit tests.
### Documentation
- [ ] I created a pull request [to the documentation](https://codeberg.org/forgejo/docs) to explain to Forgejo users how to use this change.
- [x] I did not document these changes and I do not expect someone else to do it.
### Release notes
- [x] I do not want this change to show in the release notes.
- [ ] I want the title to show in the release notes with a link to this pull request.
- [ ] I want the content of the `release-notes/<pull request number>.md` to be be used for the release notes instead of the title.
Co-authored-by: Danko Aleksejevs <danko@very.lv>
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/8398
Reviewed-by: Gusted <gusted@noreply.codeberg.org>
Co-authored-by: forgejo-backport-action <forgejo-backport-action@noreply.codeberg.org>
Co-committed-by: forgejo-backport-action <forgejo-backport-action@noreply.codeberg.org>
**Backport:** https://codeberg.org/forgejo/forgejo/pulls/7749
This adds pasted images to the dropzone. To provide the same experience
as when using the dropzone. This gives the possibility to preview and
delete the image. Additionally it provides a copy button to copy the
markdown code for inserting the image.
Fixes#4588
Co-authored-by: Beowulf <beowulf@beocode.eu>
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/8362
Reviewed-by: 0ko <0ko@noreply.codeberg.org>
Reviewed-by: Beowulf <beowulf@beocode.eu>
Co-authored-by: forgejo-backport-action <forgejo-backport-action@noreply.codeberg.org>
Co-committed-by: forgejo-backport-action <forgejo-backport-action@noreply.codeberg.org>
A fix for a bug introduced by me earlier, where attempting to parse an issue reference in an empty token would crash.
An empty token occurs if the search string is `\` or `"` (among other scenarios, probably). I'll make another PR that avoids having empty tokens (seems like a good idea).
### Tests
- I added test coverage for Go changes...
- [x] in their respective `*_test.go` for unit tests.
### Documentation
- [ ] I created a pull request [to the documentation](https://codeberg.org/forgejo/docs) to explain to Forgejo users how to use this change.
- [x] I did not document these changes and I do not expect someone else to do it.
### Release notes
- [x] I do not want this change to show in the release notes.
- [ ] I want the title to show in the release notes with a link to this pull request.
- [ ] I want the content of the `release-notes/<pull request number>.md` to be be used for the release notes instead of the title.
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/8260
Reviewed-by: Shiny Nematoda <snematoda@noreply.codeberg.org>
Co-authored-by: Danko Aleksejevs <danko@very.lv>
Co-committed-by: Danko Aleksejevs <danko@very.lv>
Two pull requests were merged at the same time
- https://codeberg.org/forgejo/forgejo/pulls/7699
- https://codeberg.org/forgejo/forgejo/pulls/7508
And added conflicting structs ActionRun modules/structs. That broke
the forgejo development branch and a quick fix was made to resolve
the name conflict.
- https://codeberg.org/forgejo/forgejo/pulls/8066
However that creates an undesirable duplication of two structures that
serve the same purpose but are different.
- Remove RepoActionRun and replace it with ActionRun
- convert.ToActionRun has one more argument, the doer, because it
is determined differently in the context of webhooks or API
### Tests
- No need because the two pull requests involved already have good coverage.
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/8250
Reviewed-by: Michael Kriese <michael.kriese@gmx.de>
Reviewed-by: klausfyhn <klausfyhn@noreply.codeberg.org>
Reviewed-by: Christopher Besch <mail@chris-besch.com>
Co-authored-by: Earl Warren <contact@earl-warren.org>
Co-committed-by: Earl Warren <contact@earl-warren.org>
## Motivation
The GLTF (`.gltf`, `.glb`) 3D model format is very popular for game development and visual productions.
For an indie game studio, it would be convenient for a team to view textured 3D models directly from the Forgejo interface (otherwise they need to be downloaded and opened). [Perforce](https://www.perforce.com/products/helix-dam), [Diversion](https://www.diversion.dev/), and GitHub all have this capability to differing extents.
Some discussion on 3D file support here: https://codeberg.org/forgejo/forgejo/issues/5188
## Changes
Adds a model viewer similar to [GitHub STL viewer](https://github.com/assimp/assimp/blob/master/test/models/STL/Spider_ascii.stl) for `.glb` model files, and lays some groundwork to support future files. Uses the [model-viewer](https://modelviewer.dev/) library by Google and three.js. The model viewer is interactive and can be rotated and scaled.
## How to Test
1) Create a new repository or use an existing one.
2) Upload a `.glb` file such as `tests/testdata/data/viewer/Unicode❤♻Test.glb` (CC0 1.0 Universal)
3) View the file in the repository.
- Similar to image files, the 3D model should be rendered in a viewer.
- Use mouse clicks to turn and zoom.
## Licenses
Libraries used for this change include three.js and @google/model-viewer, which are MIT and Apache-2.0 licenses respectively. Both of these are compatible with Forgejo's GPL3.0 license.
## Future Plans
1) `.gltf` was not attempted because it is a multiple file format, referencing other files in the same directory. Still need to experiment with this to see if it can work. `.glb` is a single file containing a `.gltf` and all of its other file/texture dependencies so was easier to implement.
2) The PR diff still shows the model as an unviewable bin file, but clicking the "View File" button takes you to a view screen where this model viewer is used. It would be nice to view the before and after of the model in two side-by-side model viewers, akin to reviewing a change in an image.
3) Also inserted stubs for adding contexts for GLTF, STL, OBJ, and 3MF. These ultimately don't do anything yet as only `.glb` files can be detected by the type sniffer of all of these.
## Checklist
The [contributor guide](https://forgejo.org/docs/next/contributor/) contains information that will be helpful to first time contributors. There also are a few [conditions for merging Pull Requests in Forgejo repositories](https://codeberg.org/forgejo/governance/src/branch/main/PullRequestsAgreement.md). You are also welcome to join the [Forgejo development chatroom](https://matrix.to/#/#forgejo-development:matrix.org).
### Tests
- I added test coverage for checking GLB file content using the first few bytes.
- [x] in their respective `typesniffer_test.go` for unit tests.
### Documentation
- [ ] I created a pull request [to the documentation](https://codeberg.org/forgejo/docs) to explain to Forgejo users how to use this change.
- [x] I did not document these changes and I do not expect someone else to do it.
### Release notes
- [ ] I do not want this change to show in the release notes.
- [ ] I want the title to show in the release notes with a link to this pull request.
- [ ] I want the content of the `release-notes/<pull request number>.md` to be be used for the release notes instead of the title.
<!--start release-notes-assistant-->
## Release notes
<!--URL:https://codeberg.org/forgejo/forgejo-->
- User Interface features
- [PR](https://codeberg.org/forgejo/forgejo/pulls/8111): <!--number 8111 --><!--line 0 --><!--description YWRkIG1vZGVsIHZpZXdlciBmb3IgYC5nbGJgIChHTFRGKSBtb2RlbCBpbiBmaWxlIHZpZXc=-->add model viewer for `.glb` (GLTF) model in file view<!--description-->
<!--end release-notes-assistant-->
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/8111
Reviewed-by: oliverpool <oliverpool@noreply.codeberg.org>
Co-authored-by: Alex Smith <amsmith.pro@pm.me>
Co-committed-by: Alex Smith <amsmith.pro@pm.me>
See #8222 for context.
## git.Blob.NewTruncatedReader
This introduce a new `NewTruncatedReader` method to return a blob-reader which silently truncates when the limit is reached (io.EOF will be returned).
Since the actual size is also returned `GetBlobContent` can pre-allocate a `[]byte` of the full-size (min of the asked size and the actual size) and call `io.ReadFull(rc, buf)` (instead of `util.ReadWithLimit(dataRc, int(limit))` which is convoluted and not used anywhere else).
### Tests
- I added test coverage for Go changes...
- [x] in their respective `*_test.go` for unit tests.
### Documentation
- [x] I did not document these changes and I do not expect someone else to do it.
### Release notes
- [x] I do not want this change to show in the release notes.
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/8223
Reviewed-by: Lucas <sclu1034@noreply.codeberg.org>
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Co-authored-by: oliverpool <git@olivier.pfad.fr>
Co-committed-by: oliverpool <git@olivier.pfad.fr>
This implements the fediverse handle markup as discussed in https://codeberg.org/forgejo/forgejo/issues/7942#issuecomment-5152173 by adding links to the https://fedirect.toolforge.org server.
It’s likely a temporary solution that will be reverted by proper federation implementation.
I wasn’t sure, but because I already had the implementation ready, I put the code here. I won’t be offended if we just close it.
(Also it relies on external server, that could be done configurable, but given that this is likely to be temporary it may not be worth the further implementation?)
### Tests
- I added test coverage for Go changes...
- [ ] in their respective `*_test.go` for unit tests.
- [x] in the `tests/integration` directory if it involves interactions with a live Forgejo server.
- I added test coverage for JavaScript changes...
- [ ] in `web_src/js/*.test.js` if it can be unit tested.
- [ ] in `tests/e2e/*.test.e2e.js` if it requires interactions with a live Forgejo server (see also the [developer guide for JavaScript testing](https://codeberg.org/forgejo/forgejo/src/branch/forgejo/tests/e2e/README.md#end-to-end-tests)).
### Documentation
- [ ] I created a pull request [to the documentation](https://codeberg.org/forgejo/docs) to explain to Forgejo users how to use this change.
- [ ] I did not document these changes and I do not expect someone else to do it.
### Release notes
- [ ] I do not want this change to show in the release notes.
- [x] I want the title to show in the release notes with a link to this pull request.
- [ ] I want the content of the `release-notes/<pull request number>.md` to be be used for the release notes instead of the title.
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/8185
Reviewed-by: Lucas <sclu1034@noreply.codeberg.org>
Co-authored-by: Robert Wolff <mahlzahn@posteo.de>
Co-committed-by: Robert Wolff <mahlzahn@posteo.de>
- Replace the current vfsgen with our own bindata generator.
- zstd is used instead of gzip. This reduces the size of the resulting binary by 2MiB, the size of the bundled assets were thus reduced from 13MiB to 11MiB.
- If [the browser accepts zstd encoding](https://caniuse.com/zstd), then the compressed bytes can be served directly, otherwise it falls back to being compressed by gzip if it's not disabled via `[server].ENABLE_GZIP`
- The compression and decompression speed is roughly 4 times faster.
- The generated filesystem is now of type `fs.Fs` instead of `http.FileSystem`, this slightly simplifies the generated code and
handling of the assets.
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/8143
Co-authored-by: Gusted <postmaster@gusted.xyz>
Co-committed-by: Gusted <postmaster@gusted.xyz>
This pull request enables the `mlkem768x25519-sha256` key exchange algorithm by default. Support for this has landed in go 1.24 and crypto/ssh 0.39.0 respectively.
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/8115
Reviewed-by: Gusted <gusted@noreply.codeberg.org>
Co-authored-by: networkException <git@nwex.de>
Co-committed-by: networkException <git@nwex.de>
# Fix OCI artifact uploads with`oras`
## Problem
ORAS (OCI Registry As Storage) artifact uploads were failing with several HTTP-related errors when pushing to Forgejo's container registry. This prevented users from storing OCI artifacts like `artifacthub-repo.yaml` in commands like `oras push [...] artifacthub-repo.yaml:application/vnd.cncf.artifacthub.repository-metadata.layer.v1.yaml`.
This has been discussed previously in https://github.com/go-gitea/gitea/issues/25846
## Root Causes and Fixes
### 1. Missing Content-Length for Empty Blobs
**Issue**: Empty blobs (size 0) were not getting the required `Content-Length: 0` header, causing ORAS to fail with "unknown response Content-Length".
**Fix**: Changed the condition in `setResponseHeaders` from `if h.ContentLength != 0` to `if h.ContentLength >= 0` to ensure the Content-Length header is always set for valid blob sizes.
```go
// Before
if h.ContentLength != 0 {
resp.Header().Set("Content-Length", strconv.FormatInt(h.ContentLength, 10))
}
// After
if h.ContentLength >= 0 {
resp.Header().Set("Content-Length", strconv.FormatInt(h.ContentLength, 10))
}
```
### 2. Content-Length Mismatch in JSON Error Responses
**Issue**: The `jsonResponse` function was calling `WriteHeader()` before writing JSON content, causing "wrote more than the declared Content-Length" errors when the HTTP stack calculated a different Content-Length than what was actually written.
**Fix**: Modified `jsonResponse` to buffer JSON content first, calculate the exact Content-Length, then write the complete response.
### 3. Incomplete HTTP Responses in Error Handling
**Issue**: The `apiError` function was only setting response headers without writing any response body, causing EOF errors when clients expected a complete HTTP response.
**Fix**: Updated `apiError` to write proper JSON error responses following the OCI Distribution Specification format with `code` and `message` fields.
### 4. Empty Config Blob Handling for OCI Artifacts
**Issue**: OCI artifacts often have empty config blobs (required by spec but contain no data). The JSON decoder was failing with EOF when trying to parse these empty configs.
**Fix**: Added EOF handling in `parseOCIImageConfig` to return a valid default metadata object for empty config blobs.
```go
if err := json.NewDecoder(r).Decode(&image); err != nil {
// Handle empty config blobs (common in OCI artifacts)
if err == io.EOF {
return &Metadata{
Type: TypeOCI,
Platform: DefaultPlatform,
}, nil
}
return nil, err
}
```
## Testing
Verified that ORAS artifact uploads now work correctly:
```bash
oras push registry/owner/package:artifacthub.io \
--config /dev/null:application/vnd.cncf.artifacthub.config.v1+yaml \
artifacthub-repo.yaml:application/vnd.cncf.artifacthub.repository-metadata.layer.v1.yaml
```
### Tests
- I added test coverage for Go changes...
- [x] in their respective `*_test.go` for unit tests.
- [ ] in the `tests/integration` directory if it involves interactions with a live Forgejo server.
- I added test coverage for JavaScript changes...
- [ ] in `web_src/js/*.test.js` if it can be unit tested.
- [ ] in `tests/e2e/*.test.e2e.js` if it requires interactions with a live Forgejo server (see also the [developer guide for JavaScript testing](https://codeberg.org/forgejo/forgejo/src/branch/forgejo/tests/e2e/README.md#end-to-end-tests)).
### Documentation
- [ ] I created a pull request [to the documentation](https://codeberg.org/forgejo/docs) to explain to Forgejo users how to use this change.
- [x] I did not document these changes and I do not expect someone else to do it.
### Release notes
- [ ] I do not want this change to show in the release notes.
- [x] I want the title to show in the release notes with a link to this pull request.
- [ ] I want the content of the `release-notes/<pull request number>.md` to be be used for the release notes instead of the title.
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/8070
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Co-authored-by: pat-s <patrick.schratz@gmail.com>
Co-committed-by: pat-s <patrick.schratz@gmail.com>
Currently, if you try to add an "external" link to a release in Forgejo,
the validation code checks for basic URL soundness and then specifically
checks that the URL is not an API URL.
In some cases, it may make sense to link to instance API URLs (like when
you want to create a release that links to several different repos'
packages). Relax this check so it only validates basic URL soundness.
Refs: https://codeberg.org/forgejo/forgejo/issues/7598
## Checklist
The [contributor guide](https://forgejo.org/docs/next/contributor/) contains information that will be helpful to first time contributors. There also are a few [conditions for merging Pull Requests in Forgejo repositories](https://codeberg.org/forgejo/governance/src/branch/main/PullRequestsAgreement.md). You are also welcome to join the [Forgejo development chatroom](https://matrix.to/#/#forgejo-development:matrix.org).
### Tests
- I added test coverage for Go changes...
- [ ] in their respective `*_test.go` for unit tests.
- [x] in the `tests/integration` directory if it involves interactions with a live Forgejo server.
- I added test coverage for JavaScript changes...
- [ ] in `web_src/js/*.test.js` if it can be unit tested.
- [ ] in `tests/e2e/*.test.e2e.js` if it requires interactions with a live Forgejo server (see also the [developer guide for JavaScript testing](https://codeberg.org/forgejo/forgejo/src/branch/forgejo/tests/e2e/README.md#end-to-end-tests)).
### Documentation
- [ ] I created a pull request [to the documentation](https://codeberg.org/forgejo/docs) to explain to Forgejo users how to use this change.
- [x] I did not document these changes and I do not expect someone else to do it.
See: https://codeberg.org/forgejo/docs/pulls/1161
### Release notes
- [ ] I do not want this change to show in the release notes.
- [x] I want the title to show in the release notes with a link to this pull request.
- [ ] I want the content of the `release-notes/<pull request number>.md` to be be used for the release notes instead of the title.
<!--start release-notes-assistant-->
## Release notes
<!--URL:https://codeberg.org/forgejo/forgejo-->
- Bug fixes
- [PR](https://codeberg.org/forgejo/forgejo/pulls/7644): <!--number 7644 --><!--line 0 --><!--description YWxsb3cgaW5zdGFuY2UgQVBJIFVSTHMgaW4gcmVsZWFzZSBhc3NldHM=-->allow instance API URLs in release assets<!--description-->
<!--end release-notes-assistant-->
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/7644
Reviewed-by: Gusted <gusted@noreply.codeberg.org>
Reviewed-by: Malte Jürgens <maltejur@noreply.codeberg.org>
Co-authored-by: John Moon <john.moon@vts-i.com>
Co-committed-by: John Moon <john.moon@vts-i.com>
Containers have been refactored to use the same serve method as other packages, ensuring consistent response handling.
fixes#7888
## Checklist
The [contributor guide](https://forgejo.org/docs/next/contributor/) contains information that will be helpful to first time contributors. There also are a few [conditions for merging Pull Requests in Forgejo repositories](https://codeberg.org/forgejo/governance/src/branch/main/PullRequestsAgreement.md). You are also welcome to join the [Forgejo development chatroom](https://matrix.to/#/#forgejo-development:matrix.org).
### Tests
- I added test coverage for Go changes...
- [ ] in their respective `*_test.go` for unit tests.
- [x] in the `tests/integration` directory if it involves interactions with a live Forgejo server.
- I added test coverage for JavaScript changes...
- [ ] in `web_src/js/*.test.js` if it can be unit tested.
- [ ] in `tests/e2e/*.test.e2e.js` if it requires interactions with a live Forgejo server (see also the [developer guide for JavaScript testing](https://codeberg.org/forgejo/forgejo/src/branch/forgejo/tests/e2e/README.md#end-to-end-tests)).
### Documentation
- [ ] I created a pull request [to the documentation](https://codeberg.org/forgejo/docs) to explain to Forgejo users how to use this change.
- [x] I did not document these changes and I do not expect someone else to do it.
### Release notes
- [ ] I do not want this change to show in the release notes.
- [x] I want the title to show in the release notes with a link to this pull request.
- [ ] I want the content of the `release-notes/<pull request number>.md` to be be used for the release notes instead of the title.
<!--start release-notes-assistant-->
## Release notes
<!--URL:https://codeberg.org/forgejo/forgejo-->
- Bug fixes
- [PR](https://codeberg.org/forgejo/forgejo/pulls/8037): <!--number 8037 --><!--line 0 --><!--description b21pdCBDb250ZW50LUxlbmd0aCBvbiAzMDcgcmVkaXJlY3RzIHdoZW4gc2VydmluZyBkaXJlY3QgbWFuaWZlc3QgZm9yIGNvbnRhaW5lcnM=-->omit Content-Length on 307 redirects when serving direct manifest for containers<!--description-->
<!--end release-notes-assistant-->
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/8037
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Co-authored-by: Julian Schlarb <julian.schlarb@denktmit.de>
Co-committed-by: Julian Schlarb <julian.schlarb@denktmit.de>
An attempt at solving #7956. This (and rebuilding the index) seems enough to ensure the issue *appears* among the results.
However, I couldn't figure out from [bleve docs](https://github.com/blevesearch/bleve/blob/master/docs/scoring.md) how to affect the scoring based on specific fields, or whether that is possible at all.
Disclaimer: I've never written Go before, sorry 😅 take it as a quick PoC more than anything.
### Tests
- I added test coverage for Go changes...
- [x] in their respective `*_test.go` for unit tests.
- [ ] in the `tests/integration` directory if it involves interactions with a live Forgejo server.
- I added test coverage for JavaScript changes...
- [ ] in `web_src/js/*.test.js` if it can be unit tested.
- [ ] in `tests/e2e/*.test.e2e.js` if it requires interactions with a live Forgejo server (see also the [developer guide for JavaScript testing](https://codeberg.org/forgejo/forgejo/src/branch/forgejo/tests/e2e/README.md#end-to-end-tests)).
### Documentation
- [ ] I created a pull request [to the documentation](https://codeberg.org/forgejo/docs) to explain to Forgejo users how to use this change.
- [x] I did not document these changes and I do not expect someone else to do it.
### Release notes
- [ ] I do not want this change to show in the release notes.
- [x] I want the title to show in the release notes with a link to this pull request.
- [ ] I want the content of the `release-notes/<pull request number>.md` to be be used for the release notes instead of the title.
<!--start release-notes-assistant-->
## Release notes
<!--URL:https://codeberg.org/forgejo/forgejo-->
- Features
- [PR](https://codeberg.org/forgejo/forgejo/pulls/7968): <!--number 7968 --><!--line 0 --><!--description QWRkIGlzc3VlIG51bWJlciB0byB0aGUgc2VhcmNoIGluZGV4LCByYW5rIG51bWJlciBhbmQgdGl0bGUgbWF0Y2hlcyBoaWdoZXIgKCM3OTU2KQ==-->Add issue number to the search index, rank number and title matches higher (#7956)<!--description-->
<!--end release-notes-assistant-->
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/7968
Reviewed-by: Shiny Nematoda <snematoda@noreply.codeberg.org>
Co-authored-by: Danko Aleksejevs <danko@very.lv>
Co-committed-by: Danko Aleksejevs <danko@very.lv>
