This moves the `annexObjectPath()` helper out of the tests and into a
dedicated sub-package as `annex.ContentLocation()`, and expands it with
`.Pointer()` (which validates using `git annex examinekey`),
`.IsAnnexed()` and `.Content()` to make it a more useful module.
The tests retain their own wrapper version of `ContentLocation()`
because I tried to follow close to the API modules/lfs uses, which in
terms of abstract `git.Blob` and `git.TreeEntry` objects, not in terms
of `repoPath string`s which are more convenient for the tests.
This makes HTTP symmetric with SSH clone URLs.
This gives us the fancy feature of _anonymous_ downloads,
so people can access datasets without having to set up an
account or manage ssh keys.
Previously, to access "open access" data shared this way,
users would need to:
1. Create an account on gitea.example.com
2. Create ssh keys
3. Upload ssh keys (and make sure to find and upload the correct file)
4. `git clone git@gitea.example.com:user/dataset.git`
5. `cd dataset`
6. `git annex get`
This cuts that down to just the last three steps:
1. `git clone https://gitea.example.com/user/dataset.git`
2. `cd dataset`
3. `git annex get`
This is significantly simpler for downstream users, especially for those
unfamiliar with the command line.
Unfortunately there's no uploading. While git-annex supports uploading
over HTTP to S3 and some other special remotes, it seems to fail on a
_plain_ HTTP remote. See https://github.com/neuropoly/gitea/issues/7
and https://git-annex.branchable.com/forum/HTTP_uploads/#comment-ce28adc128fdefe4c4c49628174d9b92.
This is not a major loss since no one wants uploading to be anonymous anyway.
To support private repos, I had to hunt down and patch a secret extra security
corner that Gitea only applies to HTTP for some reason (services/auth/basic.go).
This was guided by https://git-annex.branchable.com/tips/setup_a_public_repository_on_a_web_site/
Fixes https://github.com/neuropoly/gitea/issues/3
Co-authored-by: Mathieu Guay-Paquet <mathieu.guaypaquet@polymtl.ca>
Fixes https://github.com/neuropoly/gitea/issues/11
Tests:
* `git annex init`
* `git annex copy --from origin`
* `git annex copy --to origin`
over:
* ssh
for:
* the owner
* a collaborator
* a read-only collaborator
* a stranger
in a
* public repo
* private repo
And then confirms:
* Deletion of the remote repo (to ensure lockdown isn't messing with us: https://git-annex.branchable.com/internals/lockdown/#comment-0cc5225dc5abe8eddeb843bfd2fdc382)
------
To support all this:
* Add util.FileCmp()
* Patch withKeyFile() so it can be nested in other copies of itself
-------
Many thanks to Mathieu for giving style tips and catching several bugs,
including a subtle one in util.filecmp() which neutered it.
Co-authored-by: Mathieu Guay-Paquet <mathieu.guay-paquet@polymtl.ca>
[git-annex](https://git-annex.branchable.com/) is a more complicated cousin to
git-lfs, storing large files in an optional-download side content. Unlike lfs,
it allows mixing and matching storage remotes, so the content remote(s) doesn't
need to be on the same server as the git remote, making it feasible to scatter
a collection across cloud storage, old harddrives, or anywhere else storage can
be scavenged. Since this can get complicated, fast, it has a content-tracking
database (`git annex whereis`) to help find everything later.
The use-case we imagine for including it in Gitea is just the simple case, where
we're primarily emulating git-lfs: each repo has its large content at the same URL.
Our motivation is so we can self-host https://www.datalad.org/ datasets, which
currently are only hostable by fragilely scrounging together cloud storage --
and having to manage all the credentials associated with all the pieces -- or at
https://openneuro.org which is fragile in its own ways.
Supporting git-annex also allows multiple Gitea instance to be annex remotes for
each other, mirroring the content or otherwise collaborating the split up the
hosting costs.
Enabling
--------
TODO
HTTP
----
TODO
Permission Checking
-------------------
This tweaks the API in routers/private/serv.go to expose the calling user's
computed permission, instead of just returning HTTP 403.
This doesn't fit in super well. It's the opposite from how the git-lfs support is
done, where there's a complete list of possible subcommands and their matching
permission levels, and then the API compares the requested with the actual level
and returns HTTP 403 if the check fails.
But it's necessary. The main git-annex verbs, 'git-annex-shell configlist' and
'git-annex-shell p2pstdio' are both either read-only or read-write operations,
depending on the state on disk on either end of the connection and what the user
asked it to ask for, with no way to know before git-annex examines the situation.
So tell the level via GIT_ANNEX_READONLY and trust it to handle itself.
In the older Gogs version, the permission was directly read in cmd/serv.go:
```
mode, err = db.UserAccessMode(user.ID, repo)
```
- 966e925cf3/internal/cmd/serv.go (L334)
but in Gitea permission enforcement has been centralized in the API layer.
(perhaps so the cmd layer can avoid making direct DB connections?)
Deletion
--------
git-annex has this "lockdown" feature where it tries
really quite very hard to prevent you deleting its
data, to the point that even an rm -rf won't do it:
each file in annex/objects/ is nested inside a
folder with read-only permissions.
The recommended workaround is to run chmod -R +w when
you're sure you actually want to delete a repo. See
https://git-annex.branchable.com/internals/lockdown
So we edit util.RemoveAll() to do just that, so now
it's `chmod -R +w && rm -rf` instead of just `rm -rf`.
backport #28213
This PR will fix some missed checks for private repositories' data on
web routes and API routes.
(cherry picked from commit bc3d8bff73a5bd307dc825254b51bfedd722f078)
Backport #28205 by @CodeShakingSheep
In the [docs for email
setup](https://docs.gitea.com/administration/email-setup)
`mailer.IS_TLS_ENABLED` is mentioned which was replaced by
`mailer.PROTOCOL` in release 1.18.0 according to
https://blog.gitea.com/release-of-1.18.0/ . This change wasn't reflected
in the docs for email setup. I just replaced the deprecated mailer
setting.
Co-authored-by: CodeShakingSheep <19874562+CodeShakingSheep@users.noreply.github.com>
(cherry picked from commit 5ed0eefc9a9f3a0a493f28bcdb368b3517cf5c0e)
Backport #28191 by @yp05327
https://gitea.com/gitea/gitea-docusaurus/actions/runs/661/jobs/0#jobstep-9-39
I noticed that there are many warning logs in building docs.
It is causing 404 in docs.gitea.com now, so we need to fix it.
And there are also some other problems in v1.19 which can not be done in
this PR.
Co-authored-by: yp05327 <576951401@qq.com>
(cherry picked from commit 4b89c0f996d62bec6a03d129c0fde2e6ef332c0f)
Backport #28200
gitea doctor failed at checking and fixing 'delete-orphaned-repos',
because table name 'user' needs quoting to be correctly recognized by at
least PostgreSQL.
fixes#28199
(cherry picked from commit 7cae4dfc0048db02bef34ff1b8726e82b052fb85)
Backport #28184Fix#25473
Although there was `m.Post("/login/oauth/access_token", CorsHandler()...`,
it never really worked, because it still lacks the "OPTIONS" handler.
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
(cherry picked from commit 23838c2c2eaf596bffd5331406be99edc264883c)
Backport #28101 by @lng2020
Reverts go-gitea/gitea#27141close#28097
Co-authored-by: Nanguan Lin <70063547+lng2020@users.noreply.github.com>
(cherry picked from commit a2314ca9c5e17db140e66c7031cbfb97e04e41e5)
- "Gitea" is automatically being determined in Swagger and cannot be
overridden, therefor we override it after `swagger generate spec` is run.
- Resolves https://codeberg.org/forgejo/forgejo/issues/226
(cherry picked from commit 0b453807656b6fbe54284d865e59880460f3d253)
[BRANDING] Change Swagger title to Forgejo API
(cherry picked from commit 79f6a70b53421be9984a9ad96ebd0d06ab5af02e)
(cherry picked from commit 6cd47afe3366a615d40655a435275d5543910065)
(cherry picked from commit a7ae2e4c09431cd7a1f18c5b87b9dd87981b538d)
(cherry picked from commit 59f837e4661233dafc202ded9ad907dbdf7e3e69)
(cherry picked from commit c1c3d1f0e63b880baa32d285d3eba6034137b45a)
(cherry picked from commit 4b088713fb0a55c2c37f734ce82f13190ca412dd)
(cherry picked from commit 3df1d2137112a95279a8c84a8601e4c137d75d4f)
(cherry picked from commit a4a5fdbde4f9ebbf5ba6ba66c8f20a89e54860de)
(cherry picked from commit 57aa9ed710a95aa51ae3824873c3766b38da7850)
(cherry picked from commit 948b49bf9449af744a1f2e1324187897ed1c0809)
(cherry picked from commit be4929ab6a802d7d748f537ab8b798e30588c1d4)
(cherry picked from commit 9ecb507457d8a6aa60713762a1a5d2995ed986e5)
Conflicts:
templates/swagger/ui.tmpl
(cherry picked from commit 8e57e688637b9fbd3c33d3e38aed8768a2c07208)
(cherry picked from commit f0b3f621e34130c75b35c113ad509ab73118e123)
(cherry picked from commit 5b0a6d093ea5210a5ba27705b1d1636e23da8738)
(cherry picked from commit 2e0f6b5de8837a439ecadaf3434d006d7014016c)
(cherry picked from commit bb08076f5c26481d78855823c2daa1a76d3865fb)
(cherry picked from commit 5c9360e38a41a4494b1b1b247f9d57b1a0d91ef0)
(cherry picked from commit 2f277561489e1228af163fa4e5d153ceb7eec864)
(cherry picked from commit 6fce7eeb511d949249e045a995cc637d3788a1f0)
(cherry picked from commit 8d27e7bebd961468bd492c4d64e132a46d97f62b)
(cherry picked from commit fda2ee2aaedac8f394d9fc38085f768c775d2c38)
(cherry picked from commit fe0489a4d856655fdcf93c16a41995709c47f78f)
(cherry picked from commit fe9479b357d2ef0f5fd8247e8356ce7c75135b88)
(cherry picked from commit 1deadb2f35c8c0c87cd93b6f04675312b1e667bb)
(cherry picked from commit d1cb84b2f225c484755172735841e3b75abce409)
(cherry picked from commit 9de15d49f8ff0ef0803e8b3a7437a45358e924b9)
(cherry picked from commit 461fbe219c30de7e756ce2cb6dbc354417655a6f)
(cherry picked from commit f39bf3ba5b6325898968533e79fdfc5fe74c3810)
(cherry picked from commit b3f31c2f7bc5b4f5ed3f7fb6e6a6649331eb179a)
(cherry picked from commit b9f493c139bc40bb5aaf361b813019968be75968)
(cherry picked from commit 83781b16411d49f2f27b622ac3e806234b477984)
(cherry picked from commit 4438ee0434f88fc0dcb917304a5baf5b8c2d11a8)
(cherry picked from commit 56b6b368334fe3d11fb55b8e35bbfc314f45405a)
(cherry picked from commit d42aa19f27cb984693d3be7084059f5bd946ae3e)
(cherry picked from commit 4b7eb6a93bf9700e491e8bf99cf76b19f481cac4)
(cherry picked from commit 9d17e95951e17528cd912028544421264f531eca)
(cherry picked from commit 31d3f4bb01d0260da4b6e71b055b28f11dc6cce3)
(cherry picked from commit c428c3f14c770dd2f4a2f1320f7bf047933d1057)
(cherry picked from commit 07fbe2ba69)
Conflicts:
templates/swagger/v1_json.tmpl
remove trailing dot
Backport #28160 by @yp05327
Since #27054, Actions are enabled by default. so we should also edit the
document. 😃
ps: I think this should be backport to 1.21.0.
Co-authored-by: yp05327 <576951401@qq.com>
(cherry picked from commit 447422fe27b52c182a14519195dc11b663e315cd)
Backport #28143 by @wxiaoguang
Compare by ignoring spaces:
https://github.com/go-gitea/gitea/pull/28143/files?diff=split&w=1
When the form is going to be submitted, add the "is-loading" class to
show an indicator and avoid user UI events.
When the request finishes (success / error), remove the "is-loading"
class to make user can interact the UI.
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
(cherry picked from commit ffab076b72fb5822531f24f9e608f3b49bb2d324)
Backport #28100 by @lng2020
https://github.com/go-gitea/gitea/pull/27946 forgets to change them in
code. Sorry about that.
Co-authored-by: Nanguan Lin <70063547+lng2020@users.noreply.github.com>
(cherry picked from commit 56bedf2bccc7b9a98b94d1d5016231e7b68cd75d)
Backport #28085 by @wxiaoguang
Fix#28083 and fix the tests
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
(cherry picked from commit f7567f798d0d9dd3379051121b8b89abf09f938f)
Backport #27610 by @evantobin
Fixes#27598
In #27080, the logic for the tokens endpoints were updated to allow
admins to create and view tokens in other accounts. However, the same
functionality was not added to the DELETE endpoint. This PR makes the
DELETE endpoint function the same as the other token endpoints and adds
unit tests
Co-authored-by: Evan Tobin <me@evantob.in>
(cherry picked from commit 93ede4bc83ccb231b9ca67041318a0811d1d34dd)
Backport #28072
To avoid unnecessary database access, the `cacheTime` should always be
set if the revision has been checked.
Fix#28057
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
(cherry picked from commit 9f63d27ec4041897bb393a5f132af609a4988bf2)
Backport changelog for v1.21.0 as Giteabot doesn't seem to be in the
mood for it
---------
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: silverwind <me@silverwind.io>
(cherry picked from commit bc6477b36b774b0c4616a0ac7af76ba2d56da081)
Backport #28040 by @JakobDev
Fixes https://codeberg.org/forgejo/forgejo/issues/1758
For some weird reason we need to cast this nil.
Co-authored-by: JakobDev <jakobdev@gmx.de>
(cherry picked from commit 124a9957d086a054f45ca79ea5c5b6b6461a7f56)
Backport #28019 by @anudeepreddy
Hi,
This PR fixes#27988. The use of `path.join`(which uses `/` as the file
separator) to construct paths and comparing them with paths constructed
using `filepath.join`(which uses platform specific file separator) is
the root cause of this issue.
The desired behavior is to ignore attachments when dumping data
directory. Due to the what's mentioned above, the function
`addRecursiveExclude` is not actually ignoring the attachments directory
and is being written to the archive. The attachment directory is again
added to the archive (with different file separator as mentioned in the
issue) causing a duplicate entry on windows.
The solution is to use `filepath.join` in `addResursiveExclude` to
construct `currentAbsPath`.
Co-authored-by: Anudeep Reddy <anudeepc85@gmail.com>
(cherry picked from commit 00cd5ba6f4eb444085649aae6167bed32463e76b)
Backport #28023 by @6543
there was no check in place if a user could see a other user, if you
append e.g. `.rss`
(cherry picked from commit eef41489357a6b57e81f7c9a0a5580553f0f66ef)
Backport #27946 by @lng2020
As title. Some attachments and file sizes can easily be larger than
these limits
Co-authored-by: Nanguan Lin <70063547+lng2020@users.noreply.github.com>
(cherry picked from commit d4122712f7c1b9d5d1cce5d20695641dc3e5ac1c)
- Backport https://codeberg.org/forgejo/forgejo/pulls/1742
- While looking trough the logs for unrelated things I noticed errors
for directory size calculations in `pushUpdates` that were being caused
by a race condition in which git was making temporary file,
`filepath.WalkDir` noticed that but by the time the second lstat
came(`info.Info()`) it was already gone and it would error.
- Ignore temporary files created by Git.
- There are other cases but much much more rarer and not trivial to detect.
Examples:
...s/repository/push.go:96:pushUpdates() [E] Failed to update size for repository: updateSize: lstat [...]/objects/info/commit-graphs/tmp_graph_Wcy9kR: no such file or directory
...s/repository/push.go:96:pushUpdates() [E] Failed to update size for repository: updateSize: lstat [...]/packed-refs.lock: no such file or directory
(cherry picked from commit 16ce00772d4bfba929168533ad58c3a618f28353)
(cherry picked from commit 2aebef847ff998b8c2aa3aad12706698cef078c9)
