Query string tokenizer could return a list containing empty tokens when the query string was `\` or `"` (probably in other scenarios as well).
This seems undesirable and is what triggered #8260, but I'm posting this separately from that fix in case I'm wrong. Feel free to reject if so.
The actual change in behavior is that now searching for `\` or `"` behaves the same as if the query were empty (the bleve/elastic code checks that the tokenizer actually returned, anything rather than just query being non-empty).
### Tests
- I added test coverage for Go changes...
- [x] in their respective `*_test.go` for unit tests.
### Documentation
- [ ] I created a pull request [to the documentation](https://codeberg.org/forgejo/docs) to explain to Forgejo users how to use this change.
- [x] I did not document these changes and I do not expect someone else to do it.
### Release notes
- [x] I do not want this change to show in the release notes.
- [ ] I want the title to show in the release notes with a link to this pull request.
- [ ] I want the content of the `release-notes/<pull request number>.md` to be be used for the release notes instead of the title.
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/8261
Reviewed-by: Shiny Nematoda <snematoda@noreply.codeberg.org>
Co-authored-by: Danko Aleksejevs <danko@very.lv>
Co-committed-by: Danko Aleksejevs <danko@very.lv>
A fix for a bug introduced by me earlier, where attempting to parse an issue reference in an empty token would crash.
An empty token occurs if the search string is `\` or `"` (among other scenarios, probably). I'll make another PR that avoids having empty tokens (seems like a good idea).
### Tests
- I added test coverage for Go changes...
- [x] in their respective `*_test.go` for unit tests.
### Documentation
- [ ] I created a pull request [to the documentation](https://codeberg.org/forgejo/docs) to explain to Forgejo users how to use this change.
- [x] I did not document these changes and I do not expect someone else to do it.
### Release notes
- [x] I do not want this change to show in the release notes.
- [ ] I want the title to show in the release notes with a link to this pull request.
- [ ] I want the content of the `release-notes/<pull request number>.md` to be be used for the release notes instead of the title.
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/8260
Reviewed-by: Shiny Nematoda <snematoda@noreply.codeberg.org>
Co-authored-by: Danko Aleksejevs <danko@very.lv>
Co-committed-by: Danko Aleksejevs <danko@very.lv>
An attempt at solving #7956. This (and rebuilding the index) seems enough to ensure the issue *appears* among the results.
However, I couldn't figure out from [bleve docs](https://github.com/blevesearch/bleve/blob/master/docs/scoring.md) how to affect the scoring based on specific fields, or whether that is possible at all.
Disclaimer: I've never written Go before, sorry 😅 take it as a quick PoC more than anything.
### Tests
- I added test coverage for Go changes...
- [x] in their respective `*_test.go` for unit tests.
- [ ] in the `tests/integration` directory if it involves interactions with a live Forgejo server.
- I added test coverage for JavaScript changes...
- [ ] in `web_src/js/*.test.js` if it can be unit tested.
- [ ] in `tests/e2e/*.test.e2e.js` if it requires interactions with a live Forgejo server (see also the [developer guide for JavaScript testing](https://codeberg.org/forgejo/forgejo/src/branch/forgejo/tests/e2e/README.md#end-to-end-tests)).
### Documentation
- [ ] I created a pull request [to the documentation](https://codeberg.org/forgejo/docs) to explain to Forgejo users how to use this change.
- [x] I did not document these changes and I do not expect someone else to do it.
### Release notes
- [ ] I do not want this change to show in the release notes.
- [x] I want the title to show in the release notes with a link to this pull request.
- [ ] I want the content of the `release-notes/<pull request number>.md` to be be used for the release notes instead of the title.
<!--start release-notes-assistant-->
## Release notes
<!--URL:https://codeberg.org/forgejo/forgejo-->
- Features
- [PR](https://codeberg.org/forgejo/forgejo/pulls/7968): <!--number 7968 --><!--line 0 --><!--description QWRkIGlzc3VlIG51bWJlciB0byB0aGUgc2VhcmNoIGluZGV4LCByYW5rIG51bWJlciBhbmQgdGl0bGUgbWF0Y2hlcyBoaWdoZXIgKCM3OTU2KQ==-->Add issue number to the search index, rank number and title matches higher (#7956)<!--description-->
<!--end release-notes-assistant-->
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/7968
Reviewed-by: Shiny Nematoda <snematoda@noreply.codeberg.org>
Co-authored-by: Danko Aleksejevs <danko@very.lv>
Co-committed-by: Danko Aleksejevs <danko@very.lv>
- Use mock helper functions, instead of home-brew solutions.
- Disable cron jobs that are not important to be run during integration tests and might even interfere.
- Avoid sleeping unnecessary, if there's some requirement then sleep or retry until that requirement is met.
- Avoid trying to deliver webhooks that will always result in a failure.
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/7917
Reviewed-by: Michael Kriese <michael.kriese@gmx.de>
Co-authored-by: Gusted <postmaster@gusted.xyz>
Co-committed-by: Gusted <postmaster@gusted.xyz>
closes#6909
related to forgejo/design#14
# Description
Adds the following boolean operators for issues when using an indexer (with minor caveats)
- `+term`: `term` MUST be present for any result
- `-term`: negation; exclude results that contain `term`
- `"this is a term"`: matches the exact phrase `this is a term`
In all cases the special characters may be escaped by the prefix `\`
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/6952
Reviewed-by: 0ko <0ko@noreply.codeberg.org>
Reviewed-by: Otto <otto@codeberg.org>
Co-authored-by: Shiny Nematoda <snematoda.751k2@aleeas.com>
Co-committed-by: Shiny Nematoda <snematoda.751k2@aleeas.com>
- Make use of `test.MockVariableValue` to override variables for the
duration of the test.
- Don't needlessly call `onGiteaRun`, its only needed when a HTTP server
needs to be called by the code.
- When `onGiteaRun` is used, make use of the passed parameters, such as
the passed `*testing.T` variable and `*url.URL` (this also avoids
needing to serve the routers in the test code again).
- Use `(*url.URL).JoinPath` to craft new URLs.
- Don't override `setting.AppURL` & `setting.Database.LogSQL` when its
does not affect the test.
- Add empty fixture files for `FederatedUser` & `FederationHost` so they
are truncated and do not persist between tests.
Fix#31327
This is a quick patch to fix the bug.
Some parameters are using 0, some are using -1. I think it needs a
refactor to keep consistent. But that will be another PR.
(cherry picked from commit e4abaff7ffbbc5acd3aa668a9c458fbdf76f9573)
This updates the mapping definition of the elasticsearch issue indexer backend to use `long` instead of `integer`s wherever the go type is a `int64`. Without it larger instances could run into an issue.
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/3982
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Co-authored-by: Mai-Lapyst <mai-lapyst@noreply.codeberg.org>
Co-committed-by: Mai-Lapyst <mai-lapyst@noreply.codeberg.org>
This PR split the `Board` into two parts. One is the struct has been
renamed to `Column` and the second we have a `Template Type`.
But to make it easier to review, this PR will not change the database
schemas, they are just renames. The database schema changes could be in
future PRs.
---------
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: yp05327 <576951401@qq.com>
(cherry picked from commit 98751108b11dc748cc99230ca0fc1acfdf2c8929)
Conflicts:
docs/content/administration/config-cheat-sheet.en-us.md
docs/content/index.en-us.md
docs/content/installation/comparison.en-us.md
docs/content/usage/permissions.en-us.md
non existent files
options/locale/locale_en-US.ini
routers/web/web.go
templates/repo/header.tmpl
templates/repo/settings/options.tmpl
trivial context conflicts
More about codespell: https://github.com/codespell-project/codespell .
I personally introduced it to dozens if not hundreds of projects already and so far only positive feedback.
```
❯ grep lint-spell Makefile
@echo " - lint-spell lint spelling"
@echo " - lint-spell-fix lint spelling and fix issues"
lint: lint-frontend lint-backend lint-spell
lint-fix: lint-frontend-fix lint-backend-fix lint-spell-fix
.PHONY: lint-spell
lint-spell: lint-codespell
.PHONY: lint-spell-fix
lint-spell-fix: lint-codespell-fix
❯ git grep lint- -- .forgejo/
.forgejo/workflows/testing.yml: - run: make --always-make -j$(nproc) lint-backend checks-backend # ensure the "go-licenses" make target runs
.forgejo/workflows/testing.yml: - run: make lint-frontend
```
so how would you like me to invoke `lint-codespell` on CI? (without that would be IMHO very suboptimal and let typos sneak in)
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/3270
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Co-authored-by: Yaroslav Halchenko <debian@onerussian.com>
Co-committed-by: Yaroslav Halchenko <debian@onerussian.com>
Should resolve#30642.
Before this commit, we were treating an empty `?sort=` query parameter
as the correct sorting type (which is to sort issues in descending order
by their created UNIX time). But when we perform `sort=latest`, we did
not include this as a type so we would sort by the most recently updated
when reaching the `default` switch statement block.
This commit fixes this by considering the empty string, "latest", and
just any other string that is not mentioned in the switch statement as
sorting by newest.
(cherry picked from commit 9b7af4340c36d3e1888788499d16f83feeb1601b)
Noteable additions:
- `redefines-builtin-id` forbid variable names that shadow go builtins
- `empty-lines` remove unnecessary empty lines that `gofumpt` does not
remove for some reason
- `superfluous-else` eliminate more superfluous `else` branches
Rules are also sorted alphabetically and I cleaned up various parts of
`.golangci.yml`.
(cherry picked from commit 74f0c84fa4245a20ce6fb87dac1faf2aeeded2a2)
Conflicts:
.golangci.yml
apply the linter recommendations to Forgejo code as well
This PR uses `db.ListOptions` instead of `Paginor` to make the code
simpler.
And it also fixed the performance problem when viewing /pulls or
/issues. Before the counting in fact will also do the search.
---------
Co-authored-by: Jason Song <i@wolfogre.com>
Co-authored-by: silverwind <me@silverwind.io>
(cherry picked from commit 3f26fe2fa2c7141c9e622297e50a70f3e0003e4d)
also bleve did match on fuzzy search and the other way around. this also fix that bug.
(cherry picked from commit b9c57fb78e8e0d80d786d8e1da433b6c7ebf2f1c)
Conflicts:
tests/integration/repo_search_test.go
simple conflict resolution in the tests
(cherry picked from commit f8ab9dafb7a173a35e9308f8f784735b0f822439)
Conflicts:
routers/web/repo/fork.go
trivial context conflict, the file does not exist in Forgejo
meilisearch does not have an search option to contorl fuzzynes per query
right now:
- https://github.com/meilisearch/meilisearch/issues/1192
- https://github.com/orgs/meilisearch/discussions/377
- https://github.com/meilisearch/meilisearch/discussions/1096
so we have to create a workaround by post-filter the search result in
gitea until this is addressed.
For future works I added an option in backend only atm, to enable
fuzzynes for issue indexer too.
And also refactored the code so the fuzzy option is equal in logic to
code indexer
---
*Sponsored by Kithara Software GmbH*
Conflicts:
routers/web/repo/search.go
trivial context confict s/isMatch/isFuzzy/
just some refactoring bits towards replacing **util.OptionalBool** with
**optional.Option[bool]**
---------
Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
(cherry picked from commit f6656181e4a07d6c415927220efa2077d509f7c6)
Conflicts:
models/repo/repo_list_test.go
trivial shared fixture count conflicts
Fix#14459
The following users can add/remove review requests of a PR
- the poster of the PR
- the owner or collaborators of the repository
- members with read permission on the pull requests unit
(cherry picked from commit c42083a33950be6ee9f822c6d0de3c3a79d1f51b)
Conflicts:
models/repo/repo_list_test.go
tests/integration/api_nodeinfo_test.go
tests/integration/api_repo_test.go
shared fixture counts
(cherry picked from commit a1381d9146fba42cb97d72d38525fa3e721bfb03)
(cherry picked from commit 74714e02461fb47fcc0901211668e4529fac68d0)
(cherry picked from commit 7749dbfe6684498a47e3037088e7bef3542b6ce5)
(cherry picked from commit 437924971136eaed795f77edd3d3dfffa5f68103)
(cherry picked from commit a69f55bebf82a0b68bc0f66bc029eaea836cddb7)
(cherry picked from commit 24dd5fbfdbc27c887dbc24661c1005fb2e14e3c6)
(cherry picked from commit dda856d6b83936fd1c96c84544b086cbd8f63115)
(cherry picked from commit bc14f4fa97fffe82d1c666e961e313f88433cb9e)
(cherry picked from commit 78fef4f1379d8854901151d4bc62135c73db868e)
(cherry picked from commit 69e013cc515e2a50006d8d02f575ff6490d272ff)
(cherry picked from commit f173c6a2734b2dccf1424d27cd8e10fc296e44a4)
(cherry picked from commit 92f9d02547017770deafd1f715c32ae4479b8ded)
(cherry picked from commit c99d51e665370ceb71b96b3fb65184090c7e4442)
(cherry picked from commit aa0650fd2b42738a5e564c229c3eb63b8ca77f9b)
(cherry picked from commit 0a8ef91302368751df22a1967857283222bc097f)
(cherry picked from commit 7b54fe01c2ded0bbbcae6b89d9e330ca4f6ab744)
(cherry picked from commit 0e154f366f14d106d14f500f605380c29b5a3f21)
(cherry picked from commit 02d88ee16d23b9ebb04bf1af843fc5d2074783ce)
(cherry picked from commit 411924e0172a7b10de7513f2e7f60ab5341b13e4)
(cherry picked from commit f4e9ca6db59f2c5c638a0560d4ea99833d61520b)
(cherry picked from commit cd80126a23573dd5aea1e9674ee0bfa34c63ec5a)
(cherry picked from commit da626702f9743fc6e1dd77d21aff5fc3afe75912)
(cherry picked from commit 4b81d0bd046fef267bb10d2ca0cbd342c87fd4e2)
(cherry picked from commit 53ac2606694fa060879a0f4c82f6164c6f42a4d0)
(cherry picked from commit 984081f08d108acc47d312307b1c3beee3058202)
(cherry picked from commit 1c39bae3ec9b485f9969e29ed7ae8fe37b32da69)
It will fix#28268 .
<img width="1313" alt="image"
src="cb1e07d5-7a12-4691-a054-8278ba255bfc">
<img width="1318" alt="image"
src="4fd60820-97f1-4c2c-a233-d3671a5039e9">
## ⚠️ BREAKING ⚠️
But need to give up some features:
<img width="1312" alt="image"
src="281c0d51-0e7d-473f-bbed-216e2f645610">
However, such abandonment may fix#28055 .
## Backgroud
When the user switches the dashboard context to an org, it means they
want to search issues in the repos that belong to the org. However, when
they switch to themselves, it means all repos they can access because
they may have created an issue in a public repo that they don't own.
<img width="286" alt="image"
src="182dcd5b-1c20-4725-93af-96e8dfae5b97">
It's a confusing design. Think about this: What does "In your
repositories" mean when the user switches to an org? Repos belong to the
user or the org?
Whatever, it has been broken by #26012 and its following PRs. After the
PR, it searches for issues in repos that the dashboard context user owns
or has been explicitly granted access to, so it causes #28268.
## How to fix it
It's not really difficult to fix it. Just extend the repo scope to
search issues when the dashboard context user is the doer. Since the
user may create issues or be mentioned in any public repo, we can just
set `AllPublic` to true, which is already supported by indexers. The DB
condition will also support it in this PR.
But the real difficulty is how to count the search results grouped by
repos. It's something like "search issues with this keyword and those
filters, and return the total number and the top results. **Then, group
all of them by repo and return the counts of each group.**"
<img width="314" alt="image"
src="5206eb20-f8f5-49b9-b45a-1be2fcf679f4">
Before #26012, it was being done in the DB, but it caused the results to
be incomplete (see the description of #26012).
And to keep this, #26012 implement it in an inefficient way, just count
the issues by repo one by one, so it cannot work when `AllPublic` is
true because it's almost impossible to do this for all public repos.
1bfcdeef4c/modules/indexer/issues/indexer.go (L318-L338)
## Give up unnecessary features
We may can resovle `TODO: use "group by" of the indexer engines to
implement it`, I'm sure it can be done with Elasticsearch, but IIRC,
Bleve and Meilisearch don't support "group by".
And the real question is, does it worth it? Why should we need to know
the counts grouped by repos?
Let me show you my search dashboard on gitea.com.
<img width="1304" alt="image"
src="2bca2d46-6c71-4de1-94cb-0c9af27c62ff">
I never think the long repo list helps anything.
And if we agree to abandon it, things will be much easier. That is this
PR.
## TODO
I know it's important to filter by repos when searching issues. However,
it shouldn't be the way we have it now. It could be implemented like
this.
<img width="1316" alt="image"
src="99ee5f21-cbb5-4dfe-914d-cb796cb79fbe">
The indexers support it well now, but it requires some frontend work,
which I'm not good at. So, I think someone could help do that in another
PR and merge this one to fix the bug first.
Or please block this PR and help to complete it.
Finally, "Switch dashboard context" is also a design that needs
improvement. In my opinion, it can be accomplished by adding filtering
conditions instead of "switching".
Previously only the first term had to be matched. That default
Meilisearch behavior makes sense for e.g. some kind of autocomplete to
find and select a single result. But for filtering issues it means you
can't narrow down results by adding more terms.
This is also more consistent with other indexers and GitHub.
---
Reference:
https://www.meilisearch.com/docs/reference/api/search#matching-strategy
See https://github.com/go-gitea/gitea/pull/27718#issuecomment-1773743014
. Add a test to ensure its behavior.
Why this test uses `ProjectBoardID=0`? Because in `SearchOptions`,
`ProjectBoardID=0` means what it is. But in `IssueOptions`,
`ProjectBoardID=0` means there is no condition, and
`ProjectBoardID=db.NoConditionID` means the board ID = 0.
It's really confusing. Probably it's better to separate the db search
engine and the other issue search code. It's really two different
systems. As far as I can see, `IssueOptions` is not necessary for most
of the code, which has very simple issue search conditions.
1. remove unused function `MoveIssueAcrossProjectBoards`
2. extract the project board condition into a function
3. use db.NoCondition instead of -1. (BTW, the usage of db.NoCondition
is too confusing. Is there any way to avoid that?)
4. remove the unnecessary comment since the ctx refactor is completed.
5. Change `b.ID != 0` to `b.ID > 0`. It's more intuitive but I think
they're the same since board ID can't be negative.
This PR removed `unittest.MainTest` the second parameter
`TestOptions.GiteaRoot`. Now it detects the root directory by current
working directory.
---------
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
fix#27019
## testfixture yml
1. add issue20(a pr issue) in repo 23, org 17
2. add user15 to team 9
3. add four reviews about issue20
## test case
add two tests that are described with code comments
the code before pr #26784 failed the first test
<img width="479" alt="image"
src="1d9b5787-11b4-4c4d-931f-6a9869547f35">
current code failed the second test(as mentioned in #27019)
<img width="484" alt="image"
src="05608055-7587-43d1-bae1-92c688270819">
Any advice is appreciated.
---------
Co-authored-by: CaiCandong <50507092+CaiCandong@users.noreply.github.com>
Co-authored-by: Giteabot <teabot@gitea.io>
As described in the title.
Some points:
1. Why need those tests?
Because `buildIssueOverview` is not well tested, there are several
continuous bugs in the issue overview webpage.
2. Why in indexer_test.go?
It's hard to put those tests in `./modules/indexer/issue/db/db_test.go`
because those tests need 'real' data in db mocked by fixtures instead of
random data in `./modules/indexer/issue/internal/tests`. When using
'real' data(`unittest.PrepareTestDatabase`), `InitIssueIndexer` and the
package `init()` function of `indexer` are required to init indexer.
3. Why only db?
The other three indexer engines are well tested by random data and it's
okay to also test them with 'real' data in db mocked by fixtures. Any
follow-up PR is welcome.
4. Those tests are really basic, any more complicated tests are welcome.
5. I think it's also necessary to add tests in `TestAPISearchIssues`
in`api_test_issue.go` and `TestIssues` in `home_test.go`
Since the issue indexer has been refactored, the issue overview webpage
is built by the `buildIssueOverview` function and underlying
`indexer.Search` function and `GetIssueStats` instead of
`GetUserIssueStats`. So the function is no longer used.
I moved the relevant tests to `indexer_test.go` and since the search
option changed from `IssueOptions` to `SearchOptions`, most of the tests
are useless now.
We need more tests about the db indexer because those tests are highly
connected with the issue overview webpage and now this page has several
bugs.
Any advice about those test cases is appreciated.
---------
Co-authored-by: CaiCandong <50507092+CaiCandong@users.noreply.github.com>
