Commit graph

203 commits

Author SHA1 Message Date
6543
e83f2cbbac
Add doctor dbconsistency fix to delete repos with no owner (#27290)
to address #27273
replace #24873
2023-10-19 15:20:52 +02:00
JakobDev
3dc0c962bf
Delete repos of org when purge delete user (#27273)
Fixes https://codeberg.org/forgejo/forgejo/issues/1514

I had to remove `RenameOrganization` to avoid circular import.

We should really add some foreign keys to the database.
2023-10-19 13:16:11 +00:00
Lunny Xiao
9852c92e9a
Remove unnecessary parameter (#27671) 2023-10-18 15:03:10 +00:00
Lunny Xiao
cddf245c12
Replace more db.DefaultContext (#27628)
Target #27065
2023-10-15 17:46:06 +02:00
JakobDev
ebe803e514
Penultimate round of db.DefaultContext refactor (#27414)
Part of #27065

---------

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2023-10-11 04:24:07 +00:00
JakobDev
cc5df26680
Even more db.DefaultContext refactor (#27352)
Part of #27065

---------

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: delvh <dev.lh@web.de>
2023-10-03 10:30:41 +00:00
Dmitry Sharshakov
5e02e3b7ee
Add support for forking single branch (#25821)
Fixes #25117

Add UI for choosing branch to fork

Change default branch on single-branch forks


![image](https://github.com/go-gitea/gitea/assets/19504461/28505f69-a9a2-43a8-8b19-a0cdac3ddc5a)

---------

Co-authored-by: Denys Konovalov <kontakt@denyskon.de>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2023-09-29 09:48:39 +08:00
Lunny Xiao
673cf6af76
make writing main test easier (#27270)
This PR removed `unittest.MainTest` the second parameter
`TestOptions.GiteaRoot`. Now it detects the root directory by current
working directory.

---------

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2023-09-28 01:38:53 +00:00
JakobDev
7047df36d4
Another round of db.DefaultContext refactor (#27103)
Part of #27065

---------

Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
2023-09-25 13:17:37 +00:00
JakobDev
28f9b313ba
Updates to the API for archived repos (#27149) 2023-09-21 23:43:29 +00:00
Lunny Xiao
47b878858a
Search branches (#27055)
Resolve #25233 

<img width="1315" alt="图片"
src="https://github.com/go-gitea/gitea/assets/81045/3ba59b58-471a-4e1b-985c-87edac2268c0">

<img width="1297" alt="图片"
src="https://github.com/go-gitea/gitea/assets/81045/b6caa12f-323b-4f70-9c44-ef91cb71a26c">
2023-09-17 08:24:40 +00:00
JakobDev
f91dbbba98
Next round of db.DefaultContext refactor (#27089)
Part of #27065
2023-09-16 14:39:12 +00:00
JakobDev
c548dde205
More refactoring of db.DefaultContext (#27083)
Next step of #27065
2023-09-15 06:13:19 +00:00
JakobDev
76659b1114
Reduce usage of db.DefaultContext (#27073)
Part of #27065

This reduces the usage of `db.DefaultContext`. I think I've got enough
files for the first PR. When this is merged, I will continue working on
this.

Considering how many files this PR affect, I hope it won't take to long
to merge, so I don't end up in the merge conflict hell.

---------

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2023-09-14 17:09:32 +00:00
Nanguan Lin
da50be7360
Replace 'userxx' with 'orgxx' in all test files when the user type is org (#27052)
Currently 'userxx' and 'orgxx' are both used as username in test files
when the user type is org, which is confusing. This PR replaces all
'userxx' with 'orgxx' when the user type is org(`user.type==1`).
Some non-trivial changes
1. Rename `user3` dir to `org3` in `tests/git-repositories-meta` 
2. Change `end` in `issue reference` because 'org3' is one char shorter
than 'user3'

![ksnip_20230913-112819](https://github.com/go-gitea/gitea/assets/70063547/442988c5-4cf4-49b8-aa01-4dd6bf0ca954)
3. Change the search result number of `user/repo2` because
`user3/repo21` can't be searched now

![ksnip_20230913-112931](https://github.com/go-gitea/gitea/assets/70063547/d9ebeba4-479f-4110-9a85-825efbc981fd)
4. Change the first org name getting from API because the result is
ordered by alphabet asc and now `org 17` is before `org25`
![JW8U7NIO(J$H
_YCRB36H)T](https://github.com/go-gitea/gitea/assets/70063547/f55a685c-cf24-40e5-a87f-3a2327319548)
![)KFD411O4I8RB5ZOH7E0
Z3](https://github.com/go-gitea/gitea/assets/70063547/a0dc3299-249c-46f6-91cb-d15d4ee88dd5)

Other modifications are just find all and replace all.
Unit tests with SQLite are all passed.

---------

Co-authored-by: caicandong <1290147055@qq.com>
2023-09-14 02:59:53 +00:00
Lunny Xiao
e3ed67859a
Move some functions to service layer (#26969) 2023-09-08 21:09:23 +00:00
Lunny Xiao
4f32abaf94
move repository deletion to service layer (#26948)
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2023-09-08 04:51:15 +00:00
Lunny Xiao
b9df9fa2e2
Move createrepository from module to service layer (#26927)
Repository creation depends on many models, so moving it to service
layer is better.
2023-09-06 12:08:51 +00:00
Lunny Xiao
540bf9fa6d
Move notification interface to services layer (#26915)
Extract from #22266
2023-09-05 18:37:47 +00:00
Lunny Xiao
f064d716c3
Move feed notification service layer (#26908)
Extract from #22266
2023-09-05 13:00:52 +00:00
wxiaoguang
e8aae43f56
Move web/api context related testing function into a separate package (#26859)
Just like `models/unittest`, the testing helper functions should be in a
separate package: `contexttest`

And complete the TODO:

> // TODO: move this function to other packages, because it depends on
"models" package
2023-09-01 11:26:07 +00:00
Zettat123
98f2bf23bc
Sync tags when adopting repos (#26816)
Fixes #26138

Sync the tags into database when adopting repos

---------

Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
2023-08-31 01:42:42 +00:00
Chongyi Zheng
ad43486cd3
Fix some slice append usages (#26778)
Co-authored-by: delvh <dev.lh@web.de>
2023-08-29 15:47:26 +00:00
yp05327
f6e7798405
Add link to job details and tooltip to commit status in repo list in dashboard (#26326)
Tooltip:

![image](https://github.com/go-gitea/gitea/assets/18380374/237cb545-7844-424b-b995-1008eaaaedec)

Link to the target job:

![image](https://github.com/go-gitea/gitea/assets/18380374/0c11a97f-6517-47f2-8773-f381488c084e)
2023-08-21 15:26:10 +08:00
wxiaoguang
c6b92c84fe
Sync repo's IsEmpty status correctly (#26517)
Close #26509
2023-08-17 04:43:39 +00:00
delvh
bd6ef71854
Show branches and tags that contain a commit (#25180)
Now, you can see for a commit which existing branches and tags contain it.
You first have to click on the `load branches and tags` button, they are not preloaded by default.
All branches and tags are ordered descending by creation date.
You can even see without much hassle if the given commit is already part of the default branch.

Closes #25152 

## Screenshots

### Initial

![image](https://github.com/go-gitea/gitea/assets/51889757/84db2c0b-aaef-4f69-ab92-0b812793d2ad)

### Loaded

![image](https://github.com/go-gitea/gitea/assets/51889757/a9b84e66-8e44-4c55-b017-c37f4a45f41b)

---------

Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2023-07-27 12:47:41 +02:00
Zettat123
f3ed0ef692
Fix bugs in LFS meta garbage collection (#26122)
This PR

- Fix #26093. Replace `time.Time` with `timeutil.TimeStamp`
- Fix #26135. Add missing `xorm:"extends"` to `CountLFSMetaObject` for
LFS meta object query
- Add a unit test for LFS meta object garbage collection
2023-07-26 07:02:53 +00:00
Lunny Xiao
b167f35113
Add context parameter to some database functions (#26055)
To avoid deadlock problem, almost database related functions should be
have ctx as the first parameter.
This PR do a refactor for some of these functions.
2023-07-22 22:14:27 +08:00
Lunny Xiao
037c9895a7
Support copy protected branch from template repository (#25889)
Fix #14303
2023-07-21 12:32:47 +08:00
Lunny Xiao
01c04607c7
Fix bug when pushing to a pull request which enabled dismiss approval automatically (#25882)
Fix #25858 

The option `dissmiss stale approvals` was listed on protected branch but
never implemented. This PR fixes that.

<img width="1006" alt="图片"
src="https://github.com/go-gitea/gitea/assets/81045/60bfa968-4db7-4c24-b8be-2e5978f91bb9">


<img width="1021" alt="图片"
src="https://github.com/go-gitea/gitea/assets/81045/8dabc14d-2dfe-40c2-94ed-24fcbf6e0e8f">
2023-07-20 15:18:52 +08:00
wxiaoguang
236c645bf1
Refactor "Content" for file uploading (#25851)
Before: the concept "Content string" is used everywhere. It has some
problems:

1. Sometimes it means "base64 encoded content", sometimes it means "raw
binary content"
2. It doesn't work with large files, eg: uploading a 1G LFS file would
make Gitea process OOM

This PR does the refactoring: use "ContentReader" / "ContentBase64"
instead of "Content"

This PR is not breaking because the key in API JSON is still "content":
`` ContentBase64 string `json:"content"` ``
2023-07-18 18:14:47 +00:00
wxiaoguang
22eeede885
Do not "guess" the file encoding/BOM when using API to upload files (#25828)
Related issue: #18368

It doesn't seem right to "guess" the file encoding/BOM when using API to
upload files.

The API should save the uploaded content as-is.
2023-07-12 09:58:27 +00:00
wxiaoguang
f0bde0e4f9
Simplify the LFS GC logger usage (#25717)
Remove unnecessary `if opts.Logger != nil` checks.

* For "CLI doctor" mode, output to the console's "logger.Info".
* For "Web Task" mode, output to the default "logger.Debug", to avoid
flooding the server's log in a busy production instance.

Co-authored-by: Giteabot <teabot@gitea.io>
2023-07-06 16:52:41 +00:00
silverwind
88f835192d
Replace interface{} with any (#25686)
Result of running `perl -p -i -e 's#interface\{\}#any#g' **/*` and `make fmt`.

Basically the same [as golang did](2580d0e08d).
2023-07-04 18:36:08 +00:00
Lunny Xiao
65acd1e9ef
Fix branch commit message too long problem (#25588)
When branch's commit CommitMessage is too long, the column maybe too
short.(TEXT 16K for mysql).
This PR will fix it to only store the summary because these message will
only show on branch list or possible future search?
2023-06-30 17:03:05 +08:00
Lunny Xiao
6e19484f4d
Sync branches into databases (#22743)
Related #14180
Related #25233 
Related #22639
Close #19786
Related #12763 

This PR will change all the branches retrieve method from reading git
data to read database to reduce git read operations.

- [x] Sync git branches information into database when push git data
- [x] Create a new table `Branch`, merge some columns of `DeletedBranch`
into `Branch` table and drop the table `DeletedBranch`.
- [x] Read `Branch` table when visit `code` -> `branch` page
- [x] Read `Branch` table when list branch names in `code` page dropdown
- [x] Read `Branch` table when list git ref compare page
- [x] Provide a button in admin page to manually sync all branches.
- [x] Sync branches if repository is not empty but database branches are
empty when visiting pages with branches list
- [x] Use `commit_time desc` as the default FindBranch order by to keep
consistent as before and deleted branches will be always at the end.

---------

Co-authored-by: Jason Song <i@wolfogre.com>
2023-06-29 10:03:20 +00:00
Lunny Xiao
d44a415bf0
Add Adopt repository event and handler (#25497)
Fix #14304

---------

Co-authored-by: delvh <dev.lh@web.de>
2023-06-26 06:59:15 +00:00
wxiaoguang
4e2f1ee58d
Refactor web package and context package (#25298)
1. The "web" package shouldn't depends on "modules/context" package,
instead, let each "web context" register themselves to the "web"
package.
2. The old Init/Free doesn't make sense, so simplify it
* The ctx in "Init(ctx)" is never used, and shouldn't be used that way
* The "Free" is never called and shouldn't be called because the SSPI
instance is shared

---------

Co-authored-by: Giteabot <teabot@gitea.io>
2023-06-18 09:59:09 +02:00
yp05327
22a39bb961
Fix profile render when the README.md size is larger than 1024 bytes (#25131)
Fixes https://github.com/go-gitea/gitea/issues/25094

`GetBlobContent` will only get the first 1024 bytes, if the README.md
size is larger than 1024 bytes,
We can not render the rest of them.
After this fix, we should provide the limited size to read when call
`GetBlobContent`.

After:

![image](https://github.com/go-gitea/gitea/assets/18380374/22a42936-4cf8-40b4-a5c7-e384082beb0d)

---------

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2023-06-13 09:02:25 +00:00
Lunny Xiao
0d47b0ab1d
Remove duplicated functions when deleting a branch (#25128)
Extract from #22743

`DeleteBranch` will trigger a push update event, so that
`pull_service.CloseBranchPulls` has been invoked twice and
`AddDeletedBranch` is better to be moved to push update then even user
delete a branch via git command, it will also be triggered.

Co-authored-by: Giteabot <teabot@gitea.io>
2023-06-08 10:10:59 +00:00
Lunny Xiao
11598885b3
Use RepositoryList instead of []*Repository (#25074) 2023-06-05 15:25:47 +08:00
wxiaoguang
a90988d63f
Update repo's default branch when adding new files in an empty one (#25017)
Fix #25014

Only API needs this fix. On the Web UI, users could only add new file on
the default branch.
2023-05-31 17:07:51 +08:00
Denys Konovalov
275d4b7e3f
API endpoint for changing/creating/deleting multiple files (#24887)
This PR creates an API endpoint for creating/updating/deleting multiple
files in one API call similar to the solution provided by
[GitLab](https://docs.gitlab.com/ee/api/commits.html#create-a-commit-with-multiple-files-and-actions).

To archive this, the CreateOrUpdateRepoFile and DeleteRepoFIle functions
in files service are unified into one function supporting multiple files
and actions.

Resolves #14619
2023-05-29 17:41:35 +08:00
wxiaoguang
18f26cfbf7
Improve queue and logger context (#24924)
Before there was a "graceful function": RunWithShutdownFns, it's mainly
for some modules which doesn't support context.

The old queue system doesn't work well with context, so the old queues
need it.

After the queue refactoring, the new queue works with context well, so,
use Golang context as much as possible, the `RunWithShutdownFns` could
be removed (replaced by RunWithCancel for context cancel mechanism), the
related code could be simplified.

This PR also fixes some legacy queue-init problems, eg:

* typo : archiver: "unable to create codes indexer queue" => "unable to
create repo-archive queue"
* no nil check for failed queues, which causes unfriendly panic

After this PR, many goroutines could have better display name:

![image](https://github.com/go-gitea/gitea/assets/2114189/701b2a9b-8065-4137-aeaa-0bda2b34604a)

![image](https://github.com/go-gitea/gitea/assets/2114189/f1d5f50f-0534-40f0-b0be-f2c9daa5fe92)
2023-05-26 07:31:55 +00:00
Lunny Xiao
f9cfd6ce5b
Use the type RefName for all the needed places and fix pull mirror sync bugs (#24634)
This PR replaces all string refName as a type `git.RefName` to make the
code more maintainable.

Fix #15367
Replaces #23070 
It also fixed a bug that tags are not sync because `git remote --prune
origin` will not remove local tags if remote removed.

We in fact should use `git fetch --prune --tags origin` but not `git
remote update origin` to do the sync.

Some answer from ChatGPT as ref.

> If the git fetch --prune --tags command is not working as expected,
there could be a few reasons why. Here are a few things to check:
> 
>Make sure that you have the latest version of Git installed on your
system. You can check the version by running git --version in your
terminal. If you have an outdated version, try updating Git and see if
that resolves the issue.
> 
>Check that your Git repository is properly configured to track the
remote repository's tags. You can check this by running git config
--get-all remote.origin.fetch and verifying that it includes
+refs/tags/*:refs/tags/*. If it does not, you can add it by running git
config --add remote.origin.fetch "+refs/tags/*:refs/tags/*".
> 
>Verify that the tags you are trying to prune actually exist on the
remote repository. You can do this by running git ls-remote --tags
origin to list all the tags on the remote repository.
> 
>Check if any local tags have been created that match the names of tags
on the remote repository. If so, these local tags may be preventing the
git fetch --prune --tags command from working properly. You can delete
local tags using the git tag -d command.

---------

Co-authored-by: delvh <dev.lh@web.de>
2023-05-26 01:04:48 +00:00
Yarden Shoham
4810fe55e3
Add status indicator on main home screen for each repo (#24638)
It will show the calculated commit status state of the latest commit on
the default branch for each repository in the dashboard repo list

- Closes #15620

# Before

![image](https://github.com/go-gitea/gitea/assets/20454870/aa1326c7-43c0-458a-a798-3102c766bcf9)

# After

![image](https://github.com/go-gitea/gitea/assets/20454870/8658cc03-2224-442a-b1c8-bf64126e4575)

---------

Signed-off-by: Yarden Shoham <git@yardenshoham.com>
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: Giteabot <teabot@gitea.io>
2023-05-13 21:59:01 +00:00
wxiaoguang
82224c54e0
Improve avatar uploading / resizing / compressing, remove Fomantic card module (#24653)
Fixes: #8972
Fixes: #24263

And I think it also (partially) fix #24263 (no need to convert) ,
because users could upload any supported image format if it isn't larger
than AVATAR_MAX_ORIGIN_SIZE


The main idea: 

* if the uploaded file size is not larger than AVATAR_MAX_ORIGIN_SIZE,
use the origin
* if the resized size is larger than the origin, use the origin

Screenshots:

JPG:

<details>


![image](https://github.com/go-gitea/gitea/assets/2114189/70e98bb0-ecb9-4c4e-a89f-4a37d4e37f8e)

</details>

APNG:

<details>


![image](https://github.com/go-gitea/gitea/assets/2114189/9055135b-5e2d-4152-bd72-596fcb7c6671)


![image](https://github.com/go-gitea/gitea/assets/2114189/50364caf-f7f6-4241-a289-e485fe4cd582)

</details>

WebP (animated)

<details>


![image](https://github.com/go-gitea/gitea/assets/2114189/f642eb85-498a-49a5-86bf-0a7b04089ae0)

</details>

The only exception: if a WebP image is larger than MaxOriginSize and it
is animated, then current `webp` package can't decode it, so only in
this case it isn't supported. IMO no need to support such case: why a
user would upload a 1MB animated webp as avatar? crazy .....

---------

Co-authored-by: silverwind <me@silverwind.io>
2023-05-13 20:59:11 +02:00
wxiaoguang
6f9c278559
Rewrite queue (#24505)
# ⚠️ Breaking

Many deprecated queue config options are removed (actually, they should
have been removed in 1.18/1.19).

If you see the fatal message when starting Gitea: "Please update your
app.ini to remove deprecated config options", please follow the error
messages to remove these options from your app.ini.

Example:

```
2023/05/06 19:39:22 [E] Removed queue option: `[indexer].ISSUE_INDEXER_QUEUE_TYPE`. Use new options in `[queue.issue_indexer]`
2023/05/06 19:39:22 [E] Removed queue option: `[indexer].UPDATE_BUFFER_LEN`. Use new options in `[queue.issue_indexer]`
2023/05/06 19:39:22 [F] Please update your app.ini to remove deprecated config options
```

Many options in `[queue]` are are dropped, including:
`WRAP_IF_NECESSARY`, `MAX_ATTEMPTS`, `TIMEOUT`, `WORKERS`,
`BLOCK_TIMEOUT`, `BOOST_TIMEOUT`, `BOOST_WORKERS`, they can be removed
from app.ini.

# The problem

The old queue package has some legacy problems:

* complexity: I doubt few people could tell how it works.
* maintainability: Too many channels and mutex/cond are mixed together,
too many different structs/interfaces depends each other.
* stability: due to the complexity & maintainability, sometimes there
are strange bugs and difficult to debug, and some code doesn't have test
(indeed some code is difficult to test because a lot of things are mixed
together).
* general applicability: although it is called "queue", its behavior is
not a well-known queue.
* scalability: it doesn't seem easy to make it work with a cluster
without breaking its behaviors.

It came from some very old code to "avoid breaking", however, its
technical debt is too heavy now. It's a good time to introduce a better
"queue" package.

# The new queue package

It keeps using old config and concept as much as possible.

* It only contains two major kinds of concepts:
    * The "base queue": channel, levelqueue, redis
* They have the same abstraction, the same interface, and they are
tested by the same testing code.
* The "WokerPoolQueue", it uses the "base queue" to provide "worker
pool" function, calls the "handler" to process the data in the base
queue.
* The new code doesn't do "PushBack"
* Think about a queue with many workers, the "PushBack" can't guarantee
the order for re-queued unhandled items, so in new code it just does
"normal push"
* The new code doesn't do "pause/resume"
* The "pause/resume" was designed to handle some handler's failure: eg:
document indexer (elasticsearch) is down
* If a queue is paused for long time, either the producers blocks or the
new items are dropped.
* The new code doesn't do such "pause/resume" trick, it's not a common
queue's behavior and it doesn't help much.
* If there are unhandled items, the "push" function just blocks for a
few seconds and then re-queue them and retry.
* The new code doesn't do "worker booster"
* Gitea's queue's handlers are light functions, the cost is only the
go-routine, so it doesn't make sense to "boost" them.
* The new code only use "max worker number" to limit the concurrent
workers.
* The new "Push" never blocks forever
* Instead of creating more and more blocking goroutines, return an error
is more friendly to the server and to the end user.

There are more details in code comments: eg: the "Flush" problem, the
strange "code.index" hanging problem, the "immediate" queue problem.

Almost ready for review.

TODO:

* [x] add some necessary comments during review
* [x] add some more tests if necessary
* [x] update documents and config options
* [x] test max worker / active worker
* [x] re-run the CI tasks to see whether any test is flaky
* [x] improve the `handleOldLengthConfiguration` to provide more
friendly messages
* [x] fine tune default config values (eg: length?)

## Code coverage:

![image](https://user-images.githubusercontent.com/2114189/236620635-55576955-f95d-4810-b12f-879026a3afdf.png)
2023-05-08 19:49:59 +08:00
wxiaoguang
a6450494c3
Fix unclear IsRepositoryExist logic (#24374)
There was only one `IsRepositoryExist` function, it did: `has && isDir`

However it's not right, and it would cause 500 error when creating a new
repository if the dir exists.

Then, it was changed to `has || isDir`, it is still incorrect, it
affects the "adopt repo" logic.

To make the logic clear:

* IsRepositoryModelOrDirExist
* IsRepositoryModelExist
2023-04-28 14:14:26 -04:00
wxiaoguang
cf465b4721
Support uploading file to empty repo by API (#24357)
The uploading API already works (the only nit is the the IsEmpty flag is
out-of-sync, this PR also fixes it)

Close #14633
2023-04-26 21:36:26 -04:00