Commit Graph

3141 Commits

Author SHA1 Message Date
KN4CK3R a09b40de8d
Prevent double sanitize (#16386)
* Prevent double sanitize.
* Use SanitizeReaderToWriter.

At the moment `actualRender` uses `SanitizeReader` to sanitize the output. But `SanitizeReader` gets called in `markup.render` too so the output gets sanitized twice.

I moved the `SanitizeReader` call into `RenderRaw` because this method does not use `markup.render`. I would like to remove the `RenderRaw`/`RenderRawString` methods too because they are only called from tests, the fuzzer and the `/markup/raw` api endpoint. This endpoint is not in use so I think we could remove them. If we really in the future need a method to render markdown without PostProcessing we could achieve this with a more flexible `renderer.NeedPostProcess` method.
2021-11-19 18:46:47 +08:00
zeripath a85e75b2b1
Prevent deadlock in TestPersistableChannelQueue (#17717)
* Prevent deadlock in TestPersistableChannelQueue

There is a potential deadlock in TestPersistableChannelQueue due to attempting to
shutdown the test queue before it is ready.

Signed-off-by: Andrew Thornton <art27@cantab.net>

* prevent npe

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-11-19 01:13:25 +00:00
KN4CK3R f34151bdb2
Move user/org deletion to services (#17673) 2021-11-19 01:42:27 +08:00
Gusted d1f5584039
Simplify code for wrting SHA to name-rev (#17696) 2021-11-18 04:50:22 -05:00
Lunny Xiao 1f1ae57139
Move task from modules to services (#17680) 2021-11-18 14:47:57 +08:00
Lunny Xiao 3c3855a05c
Move user functions into user.go (#17659)
* Move user functions into user.go

* Fix test
2021-11-18 13:58:42 +08:00
Gusted c98dd7a3e0
Remove unnecessary variable assignments (#17695)
* Remove unnecessary variable assignments

As title

* enable ineffassign

Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2021-11-18 09:33:06 +08:00
KN4CK3R ea42d3c04e
Fixed commit count. (#17698)
Added "Tag" label.
Unified branch, tag and commit name.
2021-11-17 23:50:17 +00:00
zeripath 3c4724d70e
Add .gitattribute assisted language detection to blame, diff and render (#17590)
Use check attribute code to check the assigned language of a file and send that in to
chroma as a hint for the language of the file.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-11-17 20:37:00 +00:00
Lunny Xiao 5233051e64
Move some functions into services/repository (#17677) 2021-11-17 23:17:31 +08:00
wxiaoguang 750a8465f5
A better go code formatter, and now `make fmt` can run in Windows (#17684)
* go build / format tools
* re-format imports
2021-11-17 20:34:35 +08:00
Lunny Xiao 29cc169d20
Fix bug when read mysql database max lifetime (#17682) 2021-11-17 18:59:23 +08:00
zeripath bbffcc3aec
Multiple Escaping Improvements (#17551)
There are multiple places where Gitea does not properly escape URLs that it is building and there are multiple places where it builds urls when there is already a simpler function available to use this.
    
This is an extensive PR attempting to fix these issues.

1. The first commit in this PR looks through all href, src and links in the Gitea codebase and has attempted to catch all the places where there is potentially incomplete escaping.
2. Whilst doing this we will prefer to use functions that create URLs over recreating them by hand.
3. All uses of strings should be directly escaped - even if they are not currently expected to contain escaping characters. The main benefit to doing this will be that we can consider relaxing the constraints on user names and reponames in future. 
4. The next commit looks at escaping in the wiki and re-considers the urls that are used there. Using the improved escaping here wiki files containing '/'. (This implementation will currently still place all of the wiki files the root directory of the repo but this would not be difficult to change.)
5. The title generation in feeds is now properly escaped.
6. EscapePound is no longer needed - urls should be PathEscaped / QueryEscaped as necessary but then re-escaped with Escape when creating html with locales Signed-off-by: Andrew Thornton <art27@cantab.net>

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-11-16 18:18:25 +00:00
Lunny Xiao 7e1ae38097
Move migrations into services and base into modules/migration (#17663)
* Move migrtions into services and base into modules/migration

* Fix imports

* Fix lint
2021-11-16 23:25:33 +08:00
Lunny Xiao 48ccd325a1
Move some functions into services/repository (#17660) 2021-11-16 21:30:11 +08:00
wxiaoguang 81926d61db
Decouple unit test, remove intermediate `unittestbridge` package (#17662)
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2021-11-16 16:53:21 +08:00
silverwind 23bd7b1211
Add copy button to markdown code blocks (#17638)
* Add copy button to markdown code blocks

Done mostly in JS because I think it's better not to try getting buttons
past the markup sanitizer.

* add svg module tests

* fix sanitizer regexp

* remove outdated comment

* vertically center button in issue comments as well

* add comment to css

* fix undefined on view file line copy

* combine animation less files

* Update modules/markup/markdown/markdown.go

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>

* add test for different sizes

* add cloneNode and add tests for it

* use deep clone

* remove useless optional chaining

* remove the svg node cache

* unify clipboard copy string and i18n

* remove unused var

* remove unused localization

* minor css tweaks to the button

* comment tweak

* remove useless attribute

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2021-11-16 16:16:05 +08:00
Gusted ab1379743e
Fix nil checking on typed interface (#17598)
* Fix nil checking on typed interface

- Partially resoles #17596
- Resolves SA4023 errors.
- Ensure correctly that typed interface are nil.

* Remove unnecessary code

`NewBleveIndexer` will never return nil, even on errors.

* Patch `NewBleveIndexer`

* Fix low-level functions

* Remove deadcode

* Fix GetSession

* Close Elastic search when err isn't nil

* Update elastic_search.go

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2021-11-15 21:16:11 +08:00
Gusted 253d9e4158
Remove unnecassary calls to `filepath.Join` (#17608)
- Partialy resolvess #17596
- Resolves `badCall` errors from go-critic `badCall: suspicious Join on
1 argument`
- When only 1 argument is passed into `filepath.Join`, it won't do
anything special other than `filepath.Clean(...)` will be applied over
it.

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: 6543 <6543@obermui.de>
2021-11-15 14:02:53 +08:00
KN4CK3R 42ea0023a3
Add migration from GitBucket (#16767)
This PR adds [GitBucket](https://gitbucket.github.io/) as migration source.

Supported:
- Milestones
- Issues
- Pull Requests
- Comments
- Reviews
- Labels

There is no public usable instance so no integration tests added.
2021-11-14 20:11:10 +01:00
zeripath bab95c3a86
Correctly handle failed migrations (#17575)
* Correctly handle failed migrations

There is a bug in handling failed migrations whereby the migration task gets decoupled
from the migration repository. This leads to a failure of the task to get deleted with
the repository and also leads to the migration failed page resulting in a ISE.

This PR removes the zeroing out of the task id from the migration but also makes
the migration handler tolerate missing tasks much nicer.

Fix #17571

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-11-13 19:28:50 +08:00
wxiaoguang df64fa4865
Decouple unit test code from business code (#17623) 2021-11-12 22:36:47 +08:00
Lunny Xiao 90eb9fb889
Move EmailAddress & UserRedirect into models/user/ (#17607)
* Move EmailAddress into models/user/

* Fix test

* rename user_mail to user_email

* Fix test

* Move UserRedirect into models/user/

* Fix lint & test

* Fix lint

* Fix lint

* remove nolint comment

* Fix lint
2021-11-11 15:03:30 +08:00
Lunny Xiao 33fca2b537
Move webhook into models/webhook/ (#17579) 2021-11-10 13:13:16 +08:00
Lunny Xiao 99b2858e62
Move unit into models/unit/ (#17576)
* Move unit into models/unit/

* Rename unit.UnitType as unit.Type
2021-11-09 20:57:58 +01:00
Gusted ebaf4c48ea
Avoid double imports (#17569)
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2021-11-08 09:04:13 +02:00
Lunny Xiao 69b61d4373
Fix bug on admin subcommand (#17533)
* Fix bug on admin subcommand

* Add signals for all initDB

Co-authored-by: Lauris BH <lauris@nix.lv>
2021-11-07 11:11:27 +08:00
Wim bd613c704c
Fix ipv6 parsing for builtin ssh server (#17561) 2021-11-06 14:23:32 +08:00
Gusted bc6df18fb3
Use correct defaultValue for stracktrace (#17552) 2021-11-05 13:25:05 +02:00
Lunny Xiao a161c2ca58
Fix zero created time bug on commit api (#17546)
Fix #17543
2021-11-05 05:30:46 +00:00
zeripath 95da01c5cd
Add QueryEscape to general funcmap (#17526)
QueryEscape was only added to the text funcmap. Add this to the main template funcmap
too.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-11-02 23:00:30 +08:00
Lunny Xiao e69521f029
fix email with + when active (#17518)
Co-authored-by: zeripath <art27@cantab.net>
2021-11-02 06:26:13 +00:00
zeripath cb9c8184c9
Make Repo Code Indexer an Unique Queue (#17515)
The functioning of the code indexer queue really only makes sense as an unique queue
and doing this allows use to simplify the indexer data to simply delete the data if
the repo is no longer in the db.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-11-02 11:14:24 +08:00
wxiaoguang 599ff1c054
Only allow webhook to send requests to allowed hosts (#17482) 2021-11-01 16:39:52 +08:00
Richard Mahn 40c8451b7d
Properly determine CSV delimiter (#17459)
* Fixes #16558 CSV delimiter determiner

* Fixes #16558 - properly determine CSV delmiiter

* Moves quoteString to a new function

* Adds big test with lots of commas for tab delimited csv

* Adds comments

* Shortens the text of the test

* Removes single quotes from regexp as only double quotes need to be searched

* Fixes spelling

* Fixes check of length as it probalby will only be 1e4, not greater

* Makes sample size a const, properly removes truncated line

* Makes sample size a const, properly removes truncated line

* Fixes comment

* Fixes comment

* tests for FormatError() function

* Adds logic to find the limiter before or after a quoted value

* Simplifies regex

* Error tests

* Error tests

* Update modules/csv/csv.go

Co-authored-by: delvh <dev.lh@web.de>

* Update modules/csv/csv.go

Co-authored-by: delvh <dev.lh@web.de>

* Adds comments

* Update modules/csv/csv.go

Co-authored-by: delvh <dev.lh@web.de>

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: delvh <dev.lh@web.de>
2021-10-30 23:50:40 +08:00
Lunny Xiao 63c0dc89ef
Rename db Engines related functions (#17481)
* Rename db Engines related functions

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2021-10-30 22:32:11 +08:00
zeripath 0b4a8be26b
Ensure that restricted users can access repos for which they are members (#17460)
There is a small bug in the way that repo access is checked in
repoAssignment: Accessibility is checked by checking if the user has a
marked access to the repository instead of checking if the user has any
team granted access.

This PR changes this permissions check to use HasAccess() which does the
correct test. There is also a fix in the release api ListReleases where
it should return draft releases if the user is a member of a team with
write access to the releases.

The PR also adds a testcase.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-10-28 10:54:40 +08:00
Richard Mahn 8f9ac439ca
Fixes #16559 - Do not trim leading spaces for tab delimited (#17442)
* Fixes #16559 - Do not trim leading spaces for tab delimited

* Adds back semicolon delimited test

* Fixes linting

* Adds nolint directive to test because uses strings starting with spaces

Co-authored-by: zeripath <art27@cantab.net>
2021-10-26 16:46:56 -05:00
Lunny Xiao 812a9daab9
Upgrade go-github to v39 (#17437) 2021-10-26 08:19:21 +01:00
wxiaoguang 7c951fdd4a
In many cases user avatar link should be an absolute URL with http host (#17420) 2021-10-25 13:01:16 +08:00
qwerty287 3676fafdac
Add API to get/edit wiki (#17278)
* Add API to get/edit wiki

* Add swagger docs, various improvements

* fmt

* Fix lint and rm comment

* Add page parameter

* Add pagination to pages

* Add tests

* fmt

* Update func names

* Update error handling

* Update type name

* Fix lint

* Don't delete Home

* Update func name

* Update routers/api/v1/repo/wiki.go

Co-authored-by: delvh <dev.lh@web.de>

* Remove unnecessary check

* Fix lint

* Use English strings

* Update integrations/api_wiki_test.go

Co-authored-by: delvh <dev.lh@web.de>

* Update func and test names

* Remove unsed check and avoid duplicated error reports

* Improve error handling

* Return after error

* Document 404 error

* Update swagger

* Fix lint

* Apply suggestions from code review

Co-authored-by: delvh <dev.lh@web.de>

* Document file encoding

* fmt

* Apply suggestions

* Use convert

* Fix integration test

* simplify permissions

* unify duplicate key Title/Name

* improve types & return UTC timestamps

* improve types pt.2

- add WikiPageMetaData.LastCommit
- add WikiPageMetaData.HTMLURL
- replace WikiPageMetaData.Updated with .LastCommit.Committer.Created

also delete convert.ToWikiPage(), as it received too many arguments and
only had one callsite anyway. sorry for bad advice earlier 🙃

* WikiPage.Content is base64 encoded

* simplify error handling in wikiContentsByName()

* update swagger

* fix & DRY findWikiRepoCommit() error handling

ListWikiPages() previously wrote error twice when repo wiki didn't exist

* rename Content -> ContentBase64

* Fix test

* Fix tests

* Update var name

* suburl -> sub_url

Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: Norwin <git@nroo.de>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2021-10-25 11:43:40 +08:00
KN4CK3R 44f2c27d28
Fix CSV render error (#17406)
closed #17378 

Both errors from #17378 were caused by  #15175.

Problem 1 (error with added file):
`ToUTF8WithFallbackReader` creates a `MultiReader` from a `byte[2048]` and the remaining reader. `CreateReaderAndGuessDelimiter` tries to read 10000 bytes from this reader but only gets 2048 because that's the first reader in the `MultiReader`. Then the `if size < 1e4` thinks the input is at EOF and just returns that.

Problem 2 (error with changed file):
The blob reader gets defer closed. That was fine because the old version reads the whole file into memory. Now with the streaming version the close needs to defer after the method.
2021-10-25 00:42:32 +02:00
KN4CK3R f99d50fc9f
Read expected buffer size (#17409)
* Read expected buffer size.

* Changed name.
2021-10-24 22:12:43 +01:00
wxiaoguang 6c49517cbd
Fix issue markdown bugs (#17411)
* Bug fix: render Markdown http://AppURL/org/repo/issues/4?a=1&b=2#comment-123 test to HTML correctly, close #17394
* Bug fix: fix the positions of checkboxes in rendered HTML, close #17395
2021-10-23 21:38:12 +08:00
zeripath af96286f22
Stop sanitizing full name in API (#17396)
The API convert.toUser function makes the incorrect assumption that full names could
be rendered as is without being escaped. It therefore runs the names through
markup.Sanitize which leads to a double escape of user full names. This
pr stops this.

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2021-10-22 15:17:35 +08:00
wxiaoguang 960c322586
Refactor update checker to use AppState (#17387)
We have the `AppState` module now, it can store app related data easily. We do not need to create separate tables for each feature.

So the update checker can use `AppState` instead of a one-row dedicate table.

And the code of update checker is moved from `models` to `modules`.
2021-10-21 17:10:49 +01:00
wxiaoguang 83df0caf15
Sync gitea app path for git hooks and authorized keys when starting (#17335)
Gitea writes its own AppPath into git hook scripts. If Gitea's AppPath changes, then the git push will fail.

This PR:

* Introduce an AppState module, it can persist app states into database
* During GlobalInit, Gitea will check if the current AppPath is the same as last one. If they don't match, Gitea will sync git hooks.
* Refactor some code to make them more clear.
* Also, "Detect if gitea binary's name changed" #11341 is related, we call models.RewriteAllPublicKeys to update ssh authorized_keys file
2021-10-21 17:22:43 +08:00
wxiaoguang 2add8fe9be
Frontend refactor, PascalCase to camelCase, remove unused code (#17365)
* Frontend refactor, PascalCase to camelCase, remove unused code

* fix
2021-10-21 15:37:43 +08:00
zeripath 35b918f574
Offer rsa-sha2-512 and rsa-sha2-256 algorithms in internal SSH (#17281)
* Offer rsa-sha2-512 and rsa-sha2-256 algorithms in internal SSH

There is a subtle bug in the SSH library x/crypto/ssh which makes the incorrect
assumption that the public key type is the same as the signature algorithm type.

This means that only ssh-rsa signatures are offered by default.

This PR adds a workaround around this problem.

Fix #17175

Signed-off-by: Andrew Thornton <art27@cantab.net>

* as per review

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-10-20 15:55:33 -04:00
Richard Mahn 98f7013756
Prevent NPE in CSV diff rendering when column removed (#17018)
Fixes #16837 if a column is deleted.

We were clobbering the columns that were added by looping through the aline (base) and then when bline (head) was looped through, it clobbered what was in the "cells" array that is show in the diff, and then left a nil cell because nothing was shifted.

This fix properly shifts the cells, and properly puts the b cell either at its location or after, according to what the aline placed in the cells.

This includes test, adding a new test function since adding/removing cells works best with three columns, not two, which results in 4 columns of the resulting cells because it has a deleted column and an added column. If you try this locally, you can try those cases and others, such as adding a column.

There was no need to do anything special for the rows when `aline == 0 || bline == 0` so that was removed. This allows the same code to be used for removed or added lines, with the bcell text always being the RightCell, acell text being the LeftCell.

I still added the patch zeripath gave at https://github.com/go-gitea/gitea/issues/16837#issuecomment-913007382 so that just in case for some reason a cell is nil (which shouldn't happen now) it doesn't throw a 500 error, so the user can at least view the raw diff.

Also fixes in the [view.go](https://github.com/go-gitea/gitea/pull/17018/files#diff-43a7f4747c7ba8bff888c9be11affaafd595fd55d27f3333840eb19df9fad393L521) file how if a CSV file is empty (either created empty or if you edit it and remove all contents) it throws a huge 500 error when you then save it (when you view the file). Since we allow creating, saving and pushing empty files, we shouldn't throw an error on an empty CSV file, but just show its empty contents. This doesn't happen if it is a Markdown file or other type of file that is empty.
EDIT: Now handled in the markup/csv renderer code
2021-10-20 20:10:03 +01:00