Commit graph

290 commits

Author SHA1 Message Date
KN4CK3R
0e57ff7eee
Add generic set type (#21408)
This PR adds a generic set type to get rid of maps used as sets.

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2022-10-12 13:18:26 +08:00
zeripath
93df41f506
Fix slight bug in katex (#21171)
There is a small bug in #20571 whereby `$a a$b b$` will not be correctly
detected as a math inline block of `a a$b b`. This PR fixes this.

Also reenable test cases as per #21340 

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2022-10-05 19:55:36 +01:00
silverwind
ec0a06e52c
Upgrade chroma to v2.3.0 (#21259)
The behaviour of `PreventSurroundingPre` has changed in
https://github.com/alecthomas/chroma/pull/618 so that apparently it now
causes line wrapper tags to be no longer emitted, but we need some form
of indication to split the HTML into lines, so I did what
https://github.com/yuin/goldmark-highlighting/pull/33 did and added the
`nopWrapper`.

Maybe there are more elegant solutions but for some reason, just
splitting the HTML string on `\n` did not work.

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2022-09-26 13:50:03 +08:00
6543
c5e88fb03d
[API] teamSearch show teams with no members if user is admin (#21204)
close #21176
2022-09-19 20:02:29 +08:00
zeripath
88c2e24360
Add KaTeX rendering to Markdown. (#20571)
This PR adds mathematical rendering with KaTeX.

The first step is to add a Goldmark extension that detects the latex
(and tex) mathematics delimiters.

The second step to make this extension only run if math support is
enabled.

The second step is to then add KaTeX CSS and JS to the head which will
load after the dom is rendered.

Fix #3445

Signed-off-by: Andrew Thornton <art27@cantab.net>

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2022-09-14 00:33:37 +08:00
Kyle D
c8ded77680
Kd/ci playwright go test (#20123)
* Add initial playwright config

* Simplify Makefile

* Simplify Makefile

* Use correct config files

* Update playwright settings

* Fix package-lock file

* Don't use test logger for e2e tests

* fix frontend lint

* Allow passing TEST_LOGGER variable

* Init postgres database

* use standard gitea env variables

* Update playwright

* update drone

* Move empty env var to commands

* Cleanup

* Move integrations to subfolder

* tests integrations to tests integraton

* Run e2e tests with go test

* Fix linting

* install CI deps

* Add files to ESlint

* Fix drone typo

* Don't log to console in CI

* Use go test http server

* Add build step before tests

* Move shared init function to common package

* fix drone

* Clean up tests

* Fix linting

* Better mocking for page + version string

* Cleanup test generation

* Remove dependency on gitea binary

* Fix linting

* add initial support for running specific tests

* Add ACCEPT_VISUAL variable

* don't require git-lfs

* Add initial documentation

* Review feedback

* Add logged in session test

* Attempt fixing drone race

* Cleanup and bump version

* Bump deps

* Review feedback

* simplify installation

* Fix ci

* Update install docs
2022-09-02 15:18:23 -04:00
Jason Song
84447df4d3
Support Issue forms and PR forms (#20987)
* feat: extend issue template for yaml

* feat: support yaml template

* feat: render form to markdown

* feat: support yaml template for pr

* chore: rename to Fields

* feat: template unmarshal

* feat: split template

* feat: render to markdown

* feat: use full name as template file name

* chore: remove useless file

* feat: use dropdown of fomantic ui

* feat: update input style

* docs: more comments

* fix: render text without render

* chore: fix lint error

* fix: support use description as about in markdown

* fix: add field class in form

* chore: generate swagger

* feat: validate template

* feat: support is_nummber and regex

* test: fix broken unit tests

* fix: ignore empty body of md template

* fix: make multiple easymde editors work in one page

* feat: better UI

* fix: js error in pr form

* chore: generate swagger

* feat: support regex validation

* chore: generate swagger

* fix: refresh each markdown editor

* chore: give up required validation

* fix: correct issue template candidates

* fix: correct checkboxes style

* chore: ignore .hugo_build.lock in docs

* docs: separate out a new doc for merge templates

* docs: introduce syntax of yaml template

* feat: show a alert for invalid templates

* test: add case for a valid template

* fix: correct attributes of required checkbox

* fix: add class not-under-easymde for dropzone

* fix: use more back-quotes

* chore: remove translation in zh-CN

* fix EasyMDE statusbar margin

* fix: remove repeated blocks

* fix: reuse regex for quotes

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2022-09-02 15:58:49 +08:00
John Olheiser
a48d6ba4b4
Go 1.19 format (#20758)
* 1.19 gofumpt

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Change CSV test

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Commit whitespace fixes from @zeripath

Co-authored-by: zeripath <art27@cantab.net>

* Update emoji

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* bump swagger & fix generate-swagger

Signed-off-by: jolheiser <john.olheiser@gmail.com>
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Lauris BH <lauris@nix.lv>
2022-08-30 21:15:45 -05:00
Gary Wang
72b1fd7fdd
Should also support upper-case README files (#20581)
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2022-08-01 20:15:40 +08:00
Gary Wang
c35535ce07
Support localized README (#20508)
* Support localized README

* Slightly simplify getting the readme file and add some tests. Ensure that i18n also
works for docs/ etc.

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Update modules/markup/renderer.go

* Update modules/markup/renderer.go

* Update modules/markup/renderer.go

Co-authored-by: Andrew Thornton <art27@cantab.net>
2022-08-01 01:36:58 +03:00
Lucas Azevedo
599ae09a94
Use body text color in repository files table links (#20386)
Use body text color in for links in the repository files table
Issue/PR links (`.ref-issue`) will not be affected, as seen in other git services.

Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: Lauris BH <lauris@nix.lv>
2022-07-22 18:49:24 +08:00
Gusted
edd945bca3
Allow to specify colors for text in markup (#20363)
`<span style="color: red">Hello World!</span>` will now be accepted by
Bluemonday, other properties are still disallowed by Bluemonday.
2022-07-15 14:38:10 +08:00
Gusted
3f513f9e54
Fix NPE when using non-numeric (#20277)
- This code is only valid when `refNumeric` exist(otherwise we didn't find
such numeric PR and can skip that check) and give a free-pas to the  "BEFORE" check when
`ref` is nil.
- Resolves #20109
2022-07-07 17:46:49 +02:00
Gusted
5d3f99c7c6
Make better use of i18n (#20096)
* Prototyping

* Start work on creating offsets

* Modify tests

* Start prototyping with actual MPH

* Twiddle around

* Twiddle around comments

* Convert templates

* Fix external languages

* Fix latest translation

* Fix some test

* Tidy up code

* Use simple map

* go mod tidy

* Move back to data structure

- Uses less memory by creating for each language a map.

* Apply suggestions from code review

Co-authored-by: delvh <dev.lh@web.de>

* Add some comments

* Fix tests

* Try to fix tests

* Use en-US as defacto fallback

* Use correct slices

* refactor (#4)

* Remove TryTr, add log for missing translation key

* Refactor i18n

- Separate dev and production locale stores.
- Allow for live-reloading in dev mode.

Co-authored-by: zeripath <art27@cantab.net>

* Fix live-reloading & check for errors

* Make linter happy

* live-reload with periodic check (#5)

* Fix tests

Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: zeripath <art27@cantab.net>
2022-06-26 22:19:22 +08:00
Wim
cb50375e2b
Add more linters to improve code readability (#19989)
Add nakedret, unconvert, wastedassign, stylecheck and nolintlint linters to improve code readability

- nakedret - https://github.com/alexkohler/nakedret - nakedret is a Go static analysis tool to find naked returns in functions greater than a specified function length.
- unconvert - https://github.com/mdempsky/unconvert - Remove unnecessary type conversions
- wastedassign - https://github.com/sanposhiho/wastedassign -  wastedassign finds wasted assignment statements.
- notlintlint -  Reports ill-formed or insufficient nolint directives
- stylecheck - https://staticcheck.io/docs/checks/#ST - keep style consistent
  - excluded: [ST1003 - Poorly chosen identifier](https://staticcheck.io/docs/checks/#ST1003) and [ST1005 - Incorrectly formatted error string](https://staticcheck.io/docs/checks/#ST1005)
2022-06-20 12:02:49 +02:00
wxiaoguang
157b405753
Remove legacy git code (ver < 2.0), fine tune markup tests (#19930)
* clean git support for ver < 2.0

* fine tune tests for markup (which requires git module)

* remove unnecessary comments

* try to fix tests

* try test again

* use const for GitVersionRequired instead of var

* try to fix integration test

* Refactor CheckAttributeReader to make a *git.Repository version

* update document for commit signing with Gitea's internal gitconfig

* update document for commit signing with Gitea's internal gitconfig

Co-authored-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2022-06-16 23:47:44 +08:00
Lunny Xiao
b01dce2a6e
Allow render HTML with css/js external links (#19017)
* Allow render HTML with css/js external links

* Fix bug because of filename escape chars

* Fix lint

* Update docs about new configuration item

* Fix bug of render HTML in sub directory

* Add CSP head for displaying iframe in rendering file

* Fix test

* Apply suggestions from code review

Co-authored-by: delvh <dev.lh@web.de>

* Some improvements

* some improvement

* revert change in SanitizerDisabled of external renderer

* Add sandbox for iframe and support allow-scripts and allow-same-origin

* refactor

* fix

* fix lint

* fine tune

* use single option RENDER_CONTENT_MODE, use sandbox=allow-scripts

* fine tune CSP

* Apply suggestions from code review

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>

Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2022-06-16 11:33:23 +08:00
Sandro Santilli
52c2e82813
Custom regexp external issues (#17624)
* Implement custom regular expression for external issue tracking.

Signed-off-by: Alexander Beyn <malex@fatelectrons.org>

* Fix syntax/style

* Update repo.go

* Set metas['regexp']

* gofmt

* fix some tests

* fix more tests

* refactor frontend

* use LRU cache for regexp

* Update modules/markup/html_internal_test.go

Co-authored-by: Alexander Beyn <malex@fatelectrons.org>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2022-06-10 13:39:53 +08:00
Lauris BH
f92b7a6331
Add support for rendering terminal output with colors (#19497) 2022-06-09 00:46:39 +03:00
zeripath
ac88f21ecc
Automatically render wiki TOC (#19873)
Automatically add sidebar in the wiki view containing a TOC for the wiki page.
Make the TOC collapsable

Signed-off-by: Andrew Thornton <art27@cantab.net>
2022-06-08 16:59:16 +08:00
zeripath
1d04e8641d
Set Setpgid on child git processes (#19865)
When Gitea is running as PID 1 git will occassionally orphan child processes leading
to (defunct) processes. This PR simply sets Setpgid to true on these child processes
meaning that these defunct processes will also be correctly reaped.

Fix #19077

Signed-off-by: Andrew Thornton <art27@cantab.net>
2022-06-03 15:36:18 +01:00
wxiaoguang
d242511e86
Remove legacy unmaintained packages, refactor to support change default locale (#19308)
Remove two unmaintained vendor packages `i18n` and `paginater`. Changes:
* Rewrite `i18n` package with a more clear fallback mechanism. Fix an unstable `Tr` behavior, add more tests.
* Refactor the legacy `Paginater` to `Paginator`, test cases are kept unchanged.

Trivial enhancement (no breaking for end users):
* Use the first locale in LANGS setting option as the default, add a log to prevent from surprising users.
2022-04-03 17:46:48 +08:00
wxiaoguang
65f17bfc31
Refactor legacy unknwon/com package, improve golangci lint (#19284)
The main purpose is to refactor the legacy `unknwon/com` package.
1. Remove most imports of `unknwon/com`, only `util/legacy.go` imports the legacy `unknwon/com`
2. Use golangci's depguard to process denied packages
3. Fix some incorrect values in golangci.yml, eg, the version should be quoted string `"1.18"`
4. Use correctly escaped content for `go-import` and `go-source` meta tags
5. Refactor `com.Expand` to our stable (and the same fast) `vars.Expand`, our `vars.Expand` can still return partially rendered content even if the template is not good (eg: key mistach).
2022-04-01 16:47:50 +08:00
6543
3e88af898a
Make git.OpenRepository accept Context (#19260)
* OpenRepositoryCtx -> OpenRepository
* OpenRepository -> openRepositoryWithDefaultContext, only for internal usage
2022-03-30 03:13:41 +08:00
zeripath
97625b44e7
Provide configuration to allow camo-media proxying (#12802)
* Provide configuration to allow camo-media proxying

Fix #916

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2022-03-29 10:27:37 +02:00
Robert Kaussow
909804b717
Bump goldmark to v1.4.11 (#19201)
* Bump goldmark to v1.4.11

* add testcase

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: Andrew Thornton <art27@cantab.net>
2022-03-24 11:47:37 -04:00
6543
45f8d97131
nit fix (#19116) 2022-03-17 20:04:36 +02:00
Lunny Xiao
b24e8d38af
Support ignore all santize for external renderer (#18984)
* Support ignore all santize for external renderer

* Update docs

* Apply suggestions from code review

Co-authored-by: silverwind <me@silverwind.io>

* Fix doc

Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: 6543 <6543@obermui.de>
2022-03-06 16:41:54 +08:00
Alexander Neumann
fd273b05b9
Correctly link URLs to users/repos with dashes, dots or underscores (#18890)
* Add tests for references with dashes

This commit adds tests for full URLs referencing repos names and user
names containing a dash.

* Extend regex to match URLs to repos/users with dashes
2022-02-26 00:26:43 +01:00
Gusted
72256c16a8
Prevent NPE on partial match of compare URL and allow short SHA1 compare URLs (#18472)
* Don't panic & allow shorter sha1

- Don't panic when the full regex isn't matched and allow the usage of a
shorter sha1 being used.
- Resolves #18471

* Update modules/markup/html.go

Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2022-01-31 01:48:47 +02:00
zeripath
b34923d919
Update Goldmark to Goldmark 1.4.4 (#18420)
* Update Goldmark to Goldmark 1.4.4
* nolint the deprecation

Signed-off-by: Andrew Thornton <art27@cantab.net>
2022-01-29 21:17:21 +08:00
6543
54e9ee37a7
format with gofumpt (#18184)
* gofumpt -w -l .

* gofumpt -w -l -extra .

* Add linter

* manual fix

* change make fmt
2022-01-20 18:46:10 +01:00
Jelle Hulter
bbd30787d3
Updated Chroma to v0.10.0 (#18270) 2022-01-20 17:59:15 +08:00
zeripath
5cb0c9aa0d
Propagate context and ensure git commands run in request context (#17868)
This PR continues the work in #17125 by progressively ensuring that git
commands run within the request context.

This now means that the if there is a git repo already open in the context it will be used instead of reopening it.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2022-01-19 23:26:57 +00:00
Ben Niemann
2871e58a01
Remove unneeded debug messages to stdout. (#18298)
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2022-01-16 23:42:06 +08:00
Gusted
ff2fd08228
Simplify parameter types (#18006)
Remove repeated type declarations in function definitions.
2021-12-20 04:41:31 +00:00
KN4CK3R
b748acf2a0
Fixed emoji alias not parsed in links (#16221)
* Do not skip links.

* Restrict text in links to emojis.

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: zeripath <art27@cantab.net>
2021-12-15 23:49:12 +00:00
wxiaoguang
6d4172987e
Fix markdown URL parsing (#17924)
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: zeripath <art27@cantab.net>
2021-12-11 19:21:36 +02:00
zeripath
01087e9eef
Make Requests Processes and create process hierarchy. Associate OpenRepository with context. (#17125)
This PR registers requests with the process manager and manages hierarchy within the processes.

Git repos are then associated with a context, (usually the request's context) - with sub commands using this context as their base context.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-11-30 20:06:32 +00:00
KN4CK3R
a09b40de8d
Prevent double sanitize (#16386)
* Prevent double sanitize.
* Use SanitizeReaderToWriter.

At the moment `actualRender` uses `SanitizeReader` to sanitize the output. But `SanitizeReader` gets called in `markup.render` too so the output gets sanitized twice.

I moved the `SanitizeReader` call into `RenderRaw` because this method does not use `markup.render`. I would like to remove the `RenderRaw`/`RenderRawString` methods too because they are only called from tests, the fuzzer and the `/markup/raw` api endpoint. This endpoint is not in use so I think we could remove them. If we really in the future need a method to render markdown without PostProcessing we could achieve this with a more flexible `renderer.NeedPostProcess` method.
2021-11-19 18:46:47 +08:00
wxiaoguang
750a8465f5
A better go code formatter, and now make fmt can run in Windows (#17684)
* go build / format tools
* re-format imports
2021-11-17 20:34:35 +08:00
silverwind
23bd7b1211
Add copy button to markdown code blocks (#17638)
* Add copy button to markdown code blocks

Done mostly in JS because I think it's better not to try getting buttons
past the markup sanitizer.

* add svg module tests

* fix sanitizer regexp

* remove outdated comment

* vertically center button in issue comments as well

* add comment to css

* fix undefined on view file line copy

* combine animation less files

* Update modules/markup/markdown/markdown.go

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>

* add test for different sizes

* add cloneNode and add tests for it

* use deep clone

* remove useless optional chaining

* remove the svg node cache

* unify clipboard copy string and i18n

* remove unused var

* remove unused localization

* minor css tweaks to the button

* comment tweak

* remove useless attribute

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2021-11-16 16:16:05 +08:00
Richard Mahn
40c8451b7d
Properly determine CSV delimiter (#17459)
* Fixes #16558 CSV delimiter determiner

* Fixes #16558 - properly determine CSV delmiiter

* Moves quoteString to a new function

* Adds big test with lots of commas for tab delimited csv

* Adds comments

* Shortens the text of the test

* Removes single quotes from regexp as only double quotes need to be searched

* Fixes spelling

* Fixes check of length as it probalby will only be 1e4, not greater

* Makes sample size a const, properly removes truncated line

* Makes sample size a const, properly removes truncated line

* Fixes comment

* Fixes comment

* tests for FormatError() function

* Adds logic to find the limiter before or after a quoted value

* Simplifies regex

* Error tests

* Error tests

* Update modules/csv/csv.go

Co-authored-by: delvh <dev.lh@web.de>

* Update modules/csv/csv.go

Co-authored-by: delvh <dev.lh@web.de>

* Adds comments

* Update modules/csv/csv.go

Co-authored-by: delvh <dev.lh@web.de>

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: delvh <dev.lh@web.de>
2021-10-30 23:50:40 +08:00
wxiaoguang
6c49517cbd
Fix issue markdown bugs (#17411)
* Bug fix: render Markdown http://AppURL/org/repo/issues/4?a=1&b=2#comment-123 test to HTML correctly, close #17394
* Bug fix: fix the positions of checkboxes in rendered HTML, close #17395
2021-10-23 21:38:12 +08:00
Lunny Xiao
4a57c9ea17
Fix some lints (#17337)
Fix some linting problems.
2021-10-17 20:47:12 +01:00
wxiaoguang
d8e06a90f0
Open markdown image links in new window (#17287) 2021-10-11 20:12:06 +08:00
zeripath
623d2dd411
Prevent panic in Org mode HighlightCodeBlock (#17140)
When rendering source in org mode there is a mistake in the highlight code that
causes a panic.

This PR fixes this.

Fix #17139

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-09-24 14:29:32 +01:00
Eng Zer Jun
f2e7d5477f
refactor: move from io/ioutil to io and os package (#17109)
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-09-22 13:38:34 +08:00
zeripath
976db2a8b7
Do not show issue context popup on external issues (#17050)
The issues pop-up context cannot work for external issues - therefore do not show
these.

Fix #17047

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-09-15 09:45:27 +01:00
zeripath
c9c0475f4d
In Render tolerate not being passed a context (#16842)
* In Render tolerate not being passed a context

It is possible for RenderString to be passed to an external renderer if markdown
is set to be rendered by an external renderer. No context is currently sent to these
meaning that this will error out.

Fix #16835

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Add Context to Repo calls for RenderString

All calls from routers can easily add the context - so add it.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-08-28 21:15:56 +01:00
zeripath
5fbccad906
Fix NPE in fuzzer (#16680)
The fuzzer found an issue with the issue pattern processor where there is a spurious
path.Clean which does not need to be there. This PR also sets the default AppURL for
the fuzzer too.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-08-13 00:22:05 +02:00
zeripath
1fc7d6d6ad
Fix table alignment in markdown (#16596)
Set the TableOptions in markdown to allow alignment of the tables to work correctly

Fix #15959

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-08-02 17:11:18 +01:00
Lars Lehtonen
2344007fa3
modules/markup/markdown: fix dropped test error (#16438) 2021-07-18 14:09:34 +02:00
Josef Fröhle
8d962daed6
cleanup code issueFullPattern in modules/markup (#16419)
fix #16415
2021-07-15 22:33:56 +02:00
6543
46a4c6835d
Fix external renderer (#16401)
* fix external renderer

* use GBackground context as fallback

* no fallback, return error

Co-authored-by: Lauris BH <lauris@nix.lv>
2021-07-12 17:13:59 -04:00
6543
91162bbaea
Update bluemonday to v1.0.15 (#16379)
* update github.com/microcosm-cc/bluemonday

* add exec flag to contrib/update_dependencies.sh

* Fix TESTS
2021-07-09 03:30:31 +02:00
luzpaz
e0296b6a6d
Fix various documentation, user-facing, and source comment typos (#16367)
* Fix various doc, user-facing, and source comment typos

Found via `codespell -q 3 -S ./options/locale,./vendor -L ba,pullrequest,pullrequests,readby`
2021-07-08 13:38:13 +02:00
zeripath
32fd11395b
Fix relative links in postprocessed images (#16334)
If a pre-post-processed file contains relative img tags these need to be updated
and joined correctly with the prefix. Finally, the node attributes need to be updated.

Fix #16308

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
2021-07-04 10:26:04 +01:00
zeripath
302e8b6d02
Prevent zombie processes (#16314)
Unfortunately go doesn't always ensure that execd processes are completely
waited for. On linux this means that zombie processes can occur.

This PR ensures that these are waited for by using signal notifier in serv and
passing a context elsewhere.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-06-30 22:07:23 +02:00
6543
65548359cc
Add custom emoji support (#16004) 2021-06-29 16:28:38 +02:00
KN4CK3R
c9c7afda1a
Add sanitizer rules per renderer (#16110)
* Added sanitizer rules per renderer.

* Updated documentation.

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-06-23 17:09:51 -04:00
Lunny Xiao
f2babf3346
Add code block highlight to orgmode back (#14222)
Fix missed orgmode code block hightlight

Co-authored-by: zeripath <art27@cantab.net>
2021-06-23 22:45:47 +02:00
zeripath
d55b5eb0d3
Use html.Parse rather than html.ParseFragment (#16223)
* Use html.Parse rather than html.ParseFragment
  There have been a few issues with html.ParseFragment - just use html.Parse instead.

* Skip document node

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-06-22 00:12:22 +02:00
zeripath
196593e2e9
More efficiently parse shas for shaPostProcessor (#16101)
* More efficiently parse shas for shaPostProcessor

The shaPostProcessor currently repeatedly calls git rev-parse --verify on both backends
which is fine if there is only one thing that matches a sha - however if there are
multiple things then this becomes wildly inefficient.

This PR provides functions for both backends which are much faster to use.

Fix #16092

* Add ShaExistCache to RenderContext

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
2021-06-21 00:39:12 +02:00
zeripath
0db1048c3a
Run processors on whole of text (#16155)
There is an inefficiency in the design of our processors which means that Emoji
and other processors run in order n^2 time.

This PR forces the processors to process the entirety of text node before passing
back up. The fundamental inefficiency remains but it should be significantly
ameliorated.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-06-17 11:35:05 +01:00
zeripath
f4d3bf7867
issue-keyword class is being incorrectly stripped off spans (#16163)
Bluemonday sanitizer regexp rules are not additive, so the addition of the icons,
emojis and chroma syntax policy has led to this being stripped.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-06-16 03:02:03 +02:00
KN4CK3R
21cde5c439
Fix data URI scramble (#16098)
* Removed unused method.

* No prefix for data uris.

* Added test to prevent regressions.
2021-06-07 18:55:26 +02:00
KN4CK3R
3607f79d78
Fixed assert statements. (#16089) 2021-06-07 07:27:09 +02:00
Lunny Xiao
b6762e2306
Fix regression of renderer (#16091)
* Fix regression of renderer

* Fix render setting load twice bug
2021-06-06 18:50:07 -04:00
KN4CK3R
6021fbfe7a
Make tasklist checkboxes clickable (#15791)
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Lauris BH <lauris@nix.lv>
2021-05-23 17:14:03 +03:00
KN4CK3R
e818e9150f
Fixed individual markdown tests. (#15802) 2021-05-09 15:29:49 +02:00
silverwind
d4f28fd4ad
Fix URL of gitea emoji (#15770)
Fixes regression from #15219
2021-05-07 17:34:33 +02:00
Lunny Xiao
9d99f6ab19
Refactor renders (#15175)
* Refactor renders

* Some performance optimization

* Fix comment

* Transform reader

* Fix csv test

* Fix test

* Fix tests

* Improve optimaziation

* Fix test

* Fix test

* Detect file encoding with reader

* Improve optimaziation

* reduce memory usage

* improve code

* fix build

* Fix test

* Fix for go1.15

* Fix render

* Fix comment

* Fix lint

* Fix test

* Don't use NormalEOF when unnecessary

* revert change on util.go

* Apply suggestions from code review

Co-authored-by: zeripath <art27@cantab.net>

* rename function

* Take NormalEOF back

Co-authored-by: zeripath <art27@cantab.net>
2021-04-19 18:25:08 -04:00
Lunny Xiao
66f0fd0959
Add NeedPostProcess for Parser interface to improve performance of csv render (#15153) 2021-04-13 15:06:31 +08:00
zeripath
c680eb2cc7
Links in markdown should be absolute to the repository not the server (#15088)
* Links in markdown should be absolute to the repository not the server

Fix #15075

Signed-off-by: Andrew Thornton <art27@cantab.net>

* match github

Signed-off-by: Andrew Thornton <art27@cantab.net>

* add testcase

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: 6543 <6543@obermui.de>
2021-04-11 00:26:28 +08:00
zeripath
b9ed3cbc26
Upgrade to bluemonday 1.0.7 (#15379)
* Upgrade to bluemonday 1.0.7

Fix #15349

Signed-off-by: Andrew Thornton <art27@cantab.net>

* resolve unit test

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-04-09 19:13:06 -04:00
6543
9c4601bdf8
Code Formats, Nits & Unused Func/Var deletions (#15286)
* _ to unused func options

* rm useless brakets

* rm trifial non used models functions

* rm dead code

* rm dead global vars

* fix routers/api/v1/repo/issue.go

* dont overload import module
2021-04-09 09:40:34 +02:00
zeripath
04196b7658
Update to bluemonday-1.0.6 (#15294)
Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-04-05 17:38:31 -04:00
KN4CK3R
0c6137617f
Add Tabular Diff for CSV files (#14661)
Implements request #14320 The rendering of CSV files does match the diff style.

* Moved CSV logic into base package.

* Added method to create a tabular diff.

* Added CSV compare context.

* Added CSV diff template.

* Use new table style in CSV markup.

* Added file size limit for CSV rendering.

* Display CSV parser errors in diff.

* Lazy read single file.

* Lazy read rows for full diff.

* Added unit tests for various CSV changes.
2021-03-29 22:44:28 +02:00
zeripath
b68eb54f95
Clusterfuzz found another way (#15160)
Clusterfuzz found another way so I found another way to stop it

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-03-26 01:21:00 +02:00
zeripath
a587a28434
Fix another clusterfuzz identified issue (#15096)
* Fix another clusterfuzz identified issue

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-03-22 02:45:01 +01:00
zeripath
c0c052bdbb
another clusterfuzz spotted issue (#15032)
Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-03-18 22:37:11 +01:00
zeripath
71aca93dec
Remove extraneous logging (#15020)
Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-03-18 04:25:52 +01:00
zeripath
ed31ddc29a
Fix several render issues (#14986)
* Fix an issue with panics related to attributes
* Wrap goldmark render in a recovery function
* Reduce memory use in render emoji
* Use a pipe for rendering goldmark - still needs more work and a limiter

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lauris BH <lauris@nix.lv>
2021-03-16 00:20:05 +01:00
zeripath
164e35ead3
Make sure sibling images get a link too (#14979)
* Make sure sibling images get a link too

Due a problem with the ast.Walker in the our transformer in goldmark
an image with a sibling image will not be transformed to gain a parent
link. This PR fixes this.

Fix #12925

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-03-14 16:36:51 +00:00
zeripath
aa4f9180e4
Clarify the suffices and prefixes of setting.AppSubURL and setting.AppURL (#12999)
Also removes some unnecessary uses of fmt.Sprintf and adds documentation
strings

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-02-19 22:36:43 +01:00
zeripath
7ab6c77b41
Remove NULs byte arrays passed to PostProcess (#14587)
PostProcess is supposed to be parsing and handling HTML
fragments, but on fuzzing it appears that there is a weird
issue with NUL elements that could cause a memory address
error in downstream libraries.

The simplest solution is to strip out the weird NULs - they
should not be there in any case and would be stripped out
anyway.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-02-18 02:32:14 +01:00
zeripath
172229966c
Prevent panic on fuzzer provided string (#14405)
* Prevent panic on fuzzer provided string

The fuzzer has found that providing a <body> tag with an attribute to
PostProcess causes a panic. This PR removes any rendered html or body
tags from the output.

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Placate lint

* placate lint again

Signed-off-by: Andrew Thornton <art27@cantab.net>

* minor cleanup

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-01-20 23:10:50 +08:00
Nuno Silva
44ff1d6a1e
Render links for commit hashes followed by comma (#14224)
Regex test cases: https://regex101.com/r/mVbPxM/2/

fixes #14223
2021-01-03 23:11:10 +08:00
Lunny Xiao
11555d850b
Fix bug of link query order on markdown render (#14156)
* Fix bug of link query order on markdown render

* Fix bluemonday bug and fix one wrong test

Co-authored-by: 6543 <6543@obermui.de>
2020-12-29 00:28:27 +08:00
a1012112796
34df4e5df5
Add mentionable teams to tributeValues and change team mention rules to gh's style (#13198)
* Add mentionable teams to tributeValues

Signed-off-by: a1012112796 <1012112796@qq.com>

* Apply suggestions from code review

Co-authored-by: silverwind <me@silverwind.io>

* Change team mention rules to gh's style

* use org's avator as team avator in ui

Signed-off-by: a1012112796 <1012112796@qq.com>

* Update modules/markup/html.go

* Update models/issue.go

Co-authored-by: Lauris BH <lauris@nix.lv>

* Update models/issue.go

* fix a small nit and update test code

Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: 6543 <6543@obermui.de>
2020-12-21 16:39:28 +01:00
silverwind
41e8b834af
Markdown task list improvements (#13952)
* Markdown task list improvements

- Remove `.ui` class and wrappers to prevent fomantic from messing with it.
- Change rendered HTML to match GitHub.
- Add custom styling for the checkboxes.

* fix unittest

Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2020-12-13 09:05:50 +08:00
mrsdizzie
3512c7e40f
Fix crash in short link processor (#13839)
Fixes #13819
2020-12-04 03:01:42 +01:00
Jimmy Praet
056b8f5b52
Clickable links in pull request (and issue) titles (#13695)
* clickable links in pull request (and issue) titles #13658

reuses the existing logic to render clickable links in commit messages

* dedicated RenderIssueTitle function

applied patch from @mrsdizzie
2020-12-03 11:50:47 +01:00
kolaente
64133126cd
Update golangci-lint to version 1.31.0 (#13102)
This PR updates golangci-lint to the latest version 1.31.0.

The upgrade introduced a new check for which I've fixed or disabled most cases.

Signed-off-by: kolaente <k@knt.li>
2020-10-11 21:27:20 +01:00
Pranav Nachnekar
1827f892de
fix: media links in org files not liked to media files (#12997)
* fix: media links in org files not liked to media files

* fix: write directly to io.Writer r

as suggested by code review

Co-authored-by: zeripath <art27@cantab.net>

Co-authored-by: zeripath <art27@cantab.net>
2020-10-01 11:22:34 -04:00
John Olheiser
b2bda11595
Fix markdown meta parsing (#12817)
* Fix meta parsing and add minimal test

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Add license

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Require first separator and add more tests

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Make fmt

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Go lets this work gracefully 🙏

Signed-off-by: jolheiser <john.olheiser@gmail.com>

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-09-12 21:48:47 -04:00
zeripath
f91bb745d6
Don't replace underscores in auto-generated IDs in goldmark (#12805)
Fix #12196

Signed-off-by: Andrew Thornton <art27@cantab.net>
2020-09-12 12:00:48 -04:00
John Olheiser
26c4a049da
Issue templates directory (#11450)
* Issue templates

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Add some comments, appease the linter

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Add docs and re-use dir candidates

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Add default labels to issue templates

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Generate swagger

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Suggested changes

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Update issue.go

* Suggestions

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Extract metadata from legacy if possible

Signed-off-by: jolheiser <john.olheiser@gmail.com>

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2020-09-11 10:48:39 -04:00
zeripath
9fdb4f887b
Update to go-org 1.3.2 (#12728)
* Update to go-org 1.3.2

Fix #12727

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Fix unit test

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-09-05 11:45:10 -04:00
silverwind
ee047312a1
Fix emoji replacements, make emoji images consistent (#12567)
- Fix emoji not being replaced in issue title change text
- Make the image attributes consistent, add alt, remove align

Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2020-08-23 21:44:53 -04:00
zeripath
74bd9691c6
Re-attempt to delete temporary upload if the file is locked by another process (#12447)
Replace all calls to os.Remove/os.RemoveAll by retrying util.Remove/util.RemoveAll and remove circular dependencies from util.

Fix #12339

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: silverwind <me@silverwind.io>
2020-08-11 21:05:34 +01:00
zeripath
e770c2b850
Detect full references to issues and pulls in commit messages (#12399)
Fix #10269

Signed-off-by: Andrew Thornton <art27@cantab.net>
2020-08-06 20:20:05 +01:00
silverwind
e61c09ed73
Add loading spinners and mermaid error handling (#12358)
- Add loading spinners on editor and mermaid renderers
- Add error handling and inline error box for mermaid
- Fix Mermaid rendering by using the .init api
2020-08-04 20:56:37 +01:00
mrsdizzie
ea1ed802a3
Fix emoji detection in certain cases (#12320)
* Fix emoji detection certain cases

Previous tests weren't complicated enough so there were some situations where emojis were't detected properly. Find the earliest occurance in addition to checking for the longest combination.

Fixes #12312

* ok spell bot

Co-authored-by: Lauris BH <lauris@nix.lv>
2020-07-25 16:40:04 +03:00
mrsdizzie
af7ffaa279
Server-side syntax highlighting for all code (#12047)
* Server-side syntax hilighting for all code

This PR does a few things:

* Remove all traces of highlight.js
* Use chroma library to provide fast syntax hilighting directly on the server
* Provide syntax hilighting for diffs
* Re-style both unified and split diffs views
* Add custom syntax hilighting styling for both regular and arc-green

Fixes #7729
Fixes #10157
Fixes #11825
Fixes #7728
Fixes #3872
Fixes #3682

And perhaps gets closer to #9553

* fix line marker

* fix repo search

* Fix single line select

* properly load settings

* npm uninstall highlight.js

* review suggestion

* code review

* forgot to call function

* fix test

* Apply suggestions from code review

suggestions from @silverwind thanks

Co-authored-by: silverwind <me@silverwind.io>

* code review

* copy/paste error

* Use const for highlight size limit

* Update web_src/less/_repository.less

Co-authored-by: Lauris BH <lauris@nix.lv>

* update size limit to 1MB and other styling tweaks

* fix highlighting for certain diff sections

* fix test

* add worker back as suggested

Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Lauris BH <lauris@nix.lv>
2020-07-01 00:34:03 +03:00
silverwind
2447ffc74a
Disable all typographic replacements in markdown renderer (#11871)
* Disable all typographic replacements in markdown renderer

Previously we only disabled some of them. This disables all the default
replacements that goldmark's typographer extension offers, matching
GitHub's renderer.

Ref: https://github.com/yuin/goldmark#typographer-extension
Fixes: https://github.com/go-gitea/gitea/issues/11001

* remove typographer extension completely

* fix test

* really fix test

Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2020-06-13 02:10:17 -04:00
mrsdizzie
4c1ff57f1a
Update emoji regex (#11584)
When matching emoji, use a regex built from the data we have instead of something generic using unicode ranges. A generic regex can't tell the difference between two separate emoji next to each other or one emoji that is built out of two separate emoji next to each other.

This means that emoji that are next to each other without space in between will be now accurately spanned individually with proper title etc...
2020-05-29 17:08:36 +01:00
zeripath
814ca9ffea
Allow different HardBreaks settings for documents and comments (#11515)
GH has different HardBreaks behaviour for markdown comments and documents.

Comments have hard breaks and documents have soft breaks - therefore Gitea's rendering will always be different from GH's if we only provide one setting.

Here we split the setting in to two - one for documents and one for comments and other things.

Signed-off-by: Andrew Thornton art27@cantab.net

Changes to index.js as per @silverwind 
Co-authored-by: silverwind <me@silverwind.io>

Changes to docs as per @guillep2k 
Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>
2020-05-24 09:14:26 +01:00
zeripath
742e26f5a5
Prevent 500 with badly formed task list (#11328)
Fix #11317

Signed-off-by: Andrew Thornton <art27@cantab.net>
2020-05-11 00:14:49 +01:00
zeripath
ce66ca7f9f
Restore checkbox rendering and prevent poor sanitization of spans (#11277)
* Add test

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Restore checkbox rendering and prevent poor sanitization of spans

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Also fix preview context

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Also fix preview context

Signed-off-by: Andrew Thornton <art27@cantab.net>
2020-05-03 17:17:24 -03:00
mrsdizzie
4ed7d2a2bb
Remove last traces of has-emoji class (#11263)
Now that emojify.js has been removed, get rid of all instances of has-emoji class that was only used for that. Support for rendering shortcodes should remain in all of these places so it should still work the same.

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Lauris BH <lauris@nix.lv>
2020-05-01 20:58:45 +03:00
mrsdizzie
4563eb873d
Support unicode emojis and remove emojify.js (#11032)
* Support unicode emojis and remove emojify.js

This PR replaces all use of emojify.js and adds unicode emoji support to various areas of gitea.

This works in a few ways:

First it adds emoji parsing support into gitea itself. This allows us to

 * Render emojis from valid alias (😄)
 * Detect unicode emojis and let us put them in their own class with proper aria-labels and styling
 * Easily allow for custom "emoji"
 * Support all emoji rendering and features without javascript
 * Uses plain unicode and lets the system render in appropriate emoji font
 * Doesn't leave us relying on external sources for updates/fixes/features

That same list of emoji is also used to create a json file which replaces the part of emojify.js that populates the emoji search tribute. This file is about 35KB with GZIP turned on and I've set it to load after the page renders to not hinder page load time (and this removes loading emojify.js also)

For custom "emoji" it uses a pretty simple scheme of just looking for /emojis/img/name.png where name is something a user has put in the "allowed reactions" setting we already have. The gitea reaction that was previously hard coded into a forked copy of emojify.js is included and works as a custom reaction under this method.

The emoji data sourced here is from https://github.com/github/gemoji which is the gem library Github uses for their emoji rendering (and a data source for other sites). So we should be able to easily render any emoji and :alias: that Github can, removing any errors from migrated content. They also update it as well, so we can sync when there are new unicode emoji lists released.

I've included a slimmed down and slightly modified forked copy of https://github.com/knq/emoji to make up our own emoji module. The code is pretty straight forward and again allows us to have a lot of flexibility in what happens.

I had seen a few comments about performance in some of the other threads if we render this ourselves, but there doesn't seem to be any issue here. In a test it can parse, convert, and render 1,000 emojis inside of a large markdown table in about 100ms on my laptop (which is many more emojis than will ever be in any normal issue). This also prevents any flickering and other weirdness from using javascript to render some things while using go for others.

Not included here are image fall back URLS. I don't really think they are necessary for anything new being written in 2020. However, managing the emoji ourselves would allow us to add these as a feature later on if it seems necessary.

Fixes: https://github.com/go-gitea/gitea/issues/9182
Fixes: https://github.com/go-gitea/gitea/issues/8974
Fixes: https://github.com/go-gitea/gitea/issues/8953
Fixes: https://github.com/go-gitea/gitea/issues/6628
Fixes: https://github.com/go-gitea/gitea/issues/5130

* add new shared function emojiHTML

* don't increase emoji size in issue title

* Update templates/repo/issue/view_content/add_reaction.tmpl

Co-Authored-By: 6543 <6543@obermui.de>

* Support for emoji rendering in various templates

* Render code and review comments as they should be

* Better way to handle mail subjects

* insert unicode from tribute selection

* Add template helper for plain text when needed

* Use existing replace function I forgot about

* Don't include emoji greater than Unicode Version 12

Only include emoji and aliases in JSON

* Update build/generate-emoji.go

* Tweak regex slightly to really match everything including random invisible characters. Run tests for every emoji we have

* final updates

* code review

* code review

* hard code gitea custom emoji to match previous behavior

* Update .eslintrc

Co-Authored-By: silverwind <me@silverwind.io>

* disable preempt

Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>
2020-04-28 15:05:39 -03:00
zeripath
0e799c26ba
Prevent panic during wrappedConn close at hammertime (#11219)
* Prevent panic during wrappedConn close at hammertime

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Update modules/graceful/server.go

* Fix extraneous debug in goldmark.go

Signed-off-by: Andrew Thornton <art27@cantab.net>
2020-04-26 21:01:06 -03:00
zeripath
9f959ac064
Make TaskCheckBox render correctly (#11214)
* Fix checkbox rendering

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Normalize checkbox rendering

Signed-off-by: Andrew Thornton <art27@cantab.net>

* set the checkboxes to readonly instead of disabled

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: Lauris BH <lauris@nix.lv>
2020-04-26 02:09:08 -03:00
zeripath
812cfd0ad9
Use markdown frontmatter to provide Table of contents, language and frontmatter rendering (#11047)
* Add control for the rendering of the frontmatter
* Add control to include a TOC
* Add control to set language - allows control of ToC header and CJK glyph choice.

Signed-off-by: Andrew Thornton art27@cantab.net
2020-04-24 14:22:36 +01:00
silverwind
7e20f1cb5b
enable ENABLE_HARD_LINE_BREAK by default (#11162)
enable ENABLE_HARD_LINE_BREAK by default

This matches GitHub's rendering.

Fix #11155
2020-04-21 23:13:56 +01:00
zeripath
0be25e2150
Handle yaml frontmatter (#11016)
Add goldmark-meta to render yaml frontmatter as a table

Fix #5377

Signed-off-by: Andrew Thornton <art27@cantab.net>
2020-04-09 11:54:50 +01:00
techknowlogick
d00ebf445b
upgrade to most recent bluemonday (#11007)
* upgrade to most recent bluemonday

* make vendor

* update tests for bluemonday

* update tests for bluemonday

* update tests for bluemonday
2020-04-07 23:08:47 +03:00
zeripath
cf4f2cfa3e
Fix internal server error on checkboxes (#10845)
Annoyingly goldmarks SetAttributeString requires that
the value of the attribute is still a []byte but does
not make it clear in the documentation.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2020-03-26 23:47:34 +00:00
zeripath
f036de5a94
Completely fix task-list checkbox styling (#10798)
Continuing on from #10668 this PR makes goldmark apply the task-list styling to task-lists.
2020-03-22 22:25:38 +00:00
Alexander Scheel
b8551f8532
Fix task-list checkbox styling (#10668)
* Fix task-list checkbox styling

The pandoc renderer will append the class "task-list" to the ul element
wrapping a li with one or more check-boxes. This allows us to select for
them, removing their list-style-type property. However, goldmark and the
gfm spec doesn't specify the "task-list" class name, so we can't use
that to fix the issue there.

Signed-off-by: Alexander Scheel <alexander.m.scheel@gmail.com>

* Update to goldmark v1.1.25

This version adds the missing space after a checkbox.

Resolves: #9656

Signed-off-by: Alexander Scheel <alexander.m.scheel@gmail.com>

Co-authored-by: zeripath <art27@cantab.net>
2020-03-08 21:17:03 +02:00
zeripath
154b137b6d
Relax sanitization as per https://github.com/jch/html-pipeline (#10527)
Looking at github/markup#245 it is clear that GH uses https://github.com/jch/html-pipeline to sanitize. This PR relaxes our sanitization to more closely match this.

Fixes #10471
and likely others...
2020-02-28 20:05:12 +00:00
zeripath
9ad2aa8f4d
Use media links for img in post-process (#10515)
* use media links for img in post-process

* do not visit text of anchors
2020-02-28 01:16:05 -03:00
6543
e57ac841de
Fix potential bugs (#10513)
* use e if it is an option
* potential nil so check err first
* check err first
* m == nil already checked
2020-02-27 20:10:27 -03:00
mrsdizzie
b5f28d154d
Don't manually replace whitespace during render (#10291)
* Don't manually replace whitespace during render

For historical reasons Gitea manually alters the urlPrefix and replaces
a whitespace with a +. This Works for URLs, but we're also passing
urlPrefix to git calls and adding the + is breaking the tree path.

Goldmark will automatically convert a white space to the proper %20, so
we should leave the string as is which lets us pass it to git unmodified
and then let Goldmark fix it.

Also fixed separate bug in URLJoin I noticed while testing where it will
silently discard sections of a path that have # in them (possibly
others). We should just escape it first.

Fixes 10156

* Escape elems as well

* Revert "Escape elems as well"

This reverts commit 8bf49596fe.

* restart ci

* remove changes to URLJoin

* restart ci

Co-authored-by: techknowlogick <matti@mdranta.net>
2020-02-17 07:46:51 +02:00
John Olheiser
53f9dbfc7b Don't convert ellipsis in markdown (#9905)
* Don't convert ellipsis

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Formatting

Co-Authored-By: zeripath <art27@cantab.net>

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: Antoine GIRARD <sapk@users.noreply.github.com>
Co-authored-by: zeripath <art27@cantab.net>
2020-01-22 16:30:20 +01:00
John Olheiser
7d7ab1eeae Issue/PR Context Popups (#9822)
* Add data-index attribute to issue anchors

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Init JS

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Add required data to anchor

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Finish popup

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Revert changes to html.go

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Better octicon contexts

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Split out popup function for re-use

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Style changes, test fixes, and cross-reference support

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Prefer em to px

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Move label margin to base CSS

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Move JS to separate file.

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Move JS to features and fix module

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Remove query-string and hash

Co-Authored-By: silverwind <me@silverwind.io>

Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: Antoine GIRARD <sapk@users.noreply.github.com>
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: zeripath <art27@cantab.net>
2020-01-19 23:39:21 -05:00
zeripath
948dec3d75
Allow hyphen in language name (#9873) 2020-01-19 21:36:00 +00:00
Moritz
f740943df4 Fix markdown anchor links (#9673)
Co-authored-by: Lauris BH <lauris@nix.lv>
2020-01-16 12:23:48 +01:00
zeripath
27757714d0 Change markdown rendering from blackfriday to goldmark (#9533)
* Move to goldmark

Markdown rendering moved from blackfriday to the goldmark.

Multiple subtle changes required to the goldmark extensions to keep
current rendering and defaults.

Can go further with goldmark linkify and have this work within markdown
rendering making the link processor unnecessary.

Need to think about how to go about allowing extensions - at present it
seems that these would be hard to do without recompilation.

* linter fixes

Co-authored-by: Lauris BH <lauris@nix.lv>
2019-12-31 03:53:28 +02:00
John Olheiser
e6ceb6880a Prefix all user-generated IDs in markup (#9477)
* Prefix all user-generated IDs in markup
* Add user-content- to IDs in unit-tests
* fixup markdown_test.go
* update the hrefs for the wiki test
* Add blackfriday extension regex

Signed-off-by: jolheiser <john.olheiser@gmail.com>
2019-12-23 22:38:50 +00:00
Alexander Scheel
ee7df7ba8c Markdown: Sanitizier Configuration (#9075)
* Support custom sanitization policy

Allowing the gitea administrator to configure sanitization policy allows
them to couple external renders and custom templates to support more
markup. In particular, the `pandoc` renderer allows generating KaTeX
annotations, wrapping them in `<span>` elements with class `math` and
either `inline` or `display` (depending on whether or not inline or
block mode was requested).

This iteration gives the administrator whitelisting powers; carefully
crafted regexes will thus let through only the desired attributes
necessary to support their custom markup.

Resolves: #9054

Signed-off-by: Alexander Scheel <alexander.m.scheel@gmail.com>

* Document new sanitization configuration

 - Adds basic documentation to app.ini.sample,
 - Adds an example to the Configuration Cheat Sheet, and
 - Adds extended information to External Renderers section.

Signed-off-by: Alexander Scheel <alexander.m.scheel@gmail.com>

* Drop extraneous length check in newMarkupSanitizer(...)

Signed-off-by: Alexander Scheel <alexander.m.scheel@gmail.com>

* Fix plural ELEMENT and ALLOW_ATTR in docs

These were left over from their initial names. Make them singular to
conform with the current expectations.

Signed-off-by: Alexander Scheel <alexander.m.scheel@gmail.com>
2019-12-07 14:49:04 -05:00
John Olheiser
9a33adcdd3 Allow kbd tags (#9245)
* Allow kbd tags

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Add test

Signed-off-by: jolheiser <john.olheiser@gmail.com>
2019-12-03 14:02:41 -05:00
guillep2k
6a90c7e3dd Alternate syntax for cross references (#9116)
* Add support for local vs. remote xrefs

* Add doc for references

* Docs: fix cases not currently supported

* One more doc fix

* Doc: mentions for teams and orgs

* Change !num ref concept, no change in functionality

* Fix test

* Improve table of issue reference types

* Fix paragraph mark
2019-12-01 15:57:05 +02:00
guillep2k
f25fd5c8eb Fix team links in HTML rendering (#9127)
* Fix team links in HTML rendering

* Fix check and lint
2019-11-24 11:34:44 -05:00
zeripath
675f275237
Fix race condition in ReplaceSanitizer (#9123) 2019-11-22 14:46:39 +00:00
guillep2k
b15f26b1cf Close/reopen issues by keywords in titles and comments (#8866)
* Add close/reopen from comment functionality

* Fix comment

* Rewrite closing/reopening template

* Check xref permissions, move action to services/pull

* Fix RefIsPull field

* Add xref tests

* Fix xref unique filter

* Only highlight keywords for actionable xrefs

* Fix xref neuter filter

* Fix check return status

* Restart CI
2019-11-18 15:13:07 +02:00
guillep2k
7b97e04555 Convert EOL to UNIX-style to render MD properly (#8925)
* Convert EOL to UNIX-style to render MD properly

* Update modules/markup/markdown/markdown.go

Co-Authored-By: zeripath <art27@cantab.net>

* Fix lint optimization

* Check for empty content before conversion

* Update modules/util/util.go

Co-Authored-By: zeripath <art27@cantab.net>

* Improved checks and tests

* Add paragraph render test

* Improve speed even more, improve tests

* Small improvement by @gary-kim

* Fix test for DOS

* More improvements

* Restart CI
2019-11-13 03:27:11 +01:00
Lauris BH
05e7715c4b Update go-org to optimize code (#8824) 2019-11-05 16:39:03 +08:00
Lauris BH
086a46994a Rewrite markdown rendering to blackfriday v2 and rewrite orgmode rendering to go-org (#8560)
* Rewrite markdown rendering to blackfriday v2.0

* Fix style

* Fix go mod with golang 1.13

* Fix blackfriday v2 import

* Inital orgmode renderer migration to go-org

* Vendor go-org dependency

* Ignore errors :/

* Update go-org to latest version

* Update test

* Fix go-org test

* Remove unneeded code

* Fix comments

* Fix markdown test

* Fix blackfriday regression rendering HTML block
2019-10-31 01:06:25 +00:00
guillep2k
cea8ea5ae6 Support inline rendering of CUSTOM_URL_SCHEMES (#8496)
* Support inline rendering of CUSTOM_URL_SCHEMES

* Fix lint

* Add tests

* Fix lint
2019-10-15 02:31:09 +01:00
guillep2k
15809d81f7 Rewrite reference processing code in preparation for opening/closing from comment references (#8261)
* Add a markdown stripper for mentions and xrefs

* Improve comments

* Small code simplification

* Move reference code to modules/references

* Fix typo

* Make MarkdownStripper return [][]byte

* Implement preliminary keywords parsing

* Add FIXME comment

* Fix comment

* make fmt

* Fix permissions check

* Fix text assumptions

* Fix imports

* Fix lint, fmt

* Fix unused import

* Add missing export comment

* Bypass revive on implemented interface

* Move mdstripper into its own package

* Support alphanumeric patterns

* Refactor FindAllMentions

* Move mentions test to references

* Parse mentions from reference package

* Refactor code to implement renderizable references

* Fix typo

* Move patterns and tests to the references package

* Fix nil reference

* Preliminary rendering attempt of closing keywords

* Normalize names, comments, general tidy-up

* Add CSS style for action keywords

* Fix permission for admin and owner

* Fix golangci-lint

* Fix golangci-lint
2019-10-13 23:29:10 +01:00
jaqra
7e17424c7e Make link last commit massages in repository home page and commit tables (#8006)
* Make link last commit massages in repository home page and commit tables

* Use RenderCommitMessageLink instead surround with a

* deleted __debug_bin file

* Exclude email to link from latest commit title

* Exclude email processor from commit table

Co-Authored-By: mrsdizzie <info@mrsdizzie.com>

* Add class parameter to a html element creator functions.
Make links underline dashed that are not commit

* fix tests

* Show dashed underline when also not hovered
2019-09-10 12:03:30 +03:00
Rinat
b660a732ae feat: highlight issue references with : (#8101)
* feat: highlight issue references with :

e.g. #1287: my commit msg
e.g. ABC-1234: my commit msg

* ref: update model regex to consistent with issueNumericPattern

* test: check highlight issue with : in commits messages
2019-09-05 20:39:54 -04:00
Tamal Saha
171b359877 Use gitea forked macaron (#7933)
Signed-off-by: Tamal Saha <tamal@appscode.com>
2019-08-23 12:40:29 -04:00
Norwin
867f46f78e Detect delimiter in CSV rendering (#7869)
* detect csv delimiter in csv rendering

fixes #7868

* make linter happy

* fix failing testcase & use ints where possible

* expose markup type to template

previously all markup had the .markdown class, which is incorrect,
as it applies markdown CSS & JS logic to CSV rendering

* fix build (missing `make css`)

* ignore quoted csv content for delimiter scoring

also fix html generation
2019-08-16 01:09:50 +03:00
Gary Kim
7eed11e5e9 Check commit message hashes before making links (#7713)
* Check commit message hashes before making links

Previously, when formatting commit messages, anything
that looked like SHA1 hashes was turned into a link
using regex. This meant that certain phrases or numbers
such as `777777` or `deadbeef` could be recognized as a commit
even if the repository has no commit with those hashes.

This change will make it so that anything that looks
like a SHA1 hash using regex will then also be checked
to ensure that there is a commit in the repository
with that hash before making a link.

Signed-off-by: Gary Kim <gary@garykim.dev>

* Use gogit to check if commit exists

This commit modifies the commit hash check
in the render for commit messages to use
gogit for better performance.

Signed-off-by: Gary Kim <gary@garykim.dev>

* Make code cleaner

Signed-off-by: Gary Kim <gary@garykim.dev>

* Use rev-parse to check if commit exists

Signed-off-by: Gary Kim <gary@garykim.dev>

* Add and modify tests for checking hashes in html link rendering

Signed-off-by: Gary Kim <gary@garykim.dev>

* Return error in sha1CurrentPatternProcessor

Co-Authored-By: mrsdizzie <info@mrsdizzie.com>

* Import Gitea log module

Signed-off-by: Gary Kim <gary@garykim.dev>

* Revert "Return error in sha1CurrentPatternProcessor"

This reverts commit 28f561cac4.

Signed-off-by: Gary Kim <gary@garykim.dev>

* Add debug logging to sha1CurrentPatternProcessor

This will log errors by the git command run in
sha1CurrentPatternProcessor if the error is one
that was unexpected.

Signed-off-by: Gary Kim <gary@garykim.dev>
2019-08-14 16:04:55 +08:00
Christian Muehlhaeuser
54d96c79b5 Removed unnecessary conversions (#7557)
No need to convert to the same type.
2019-07-23 19:50:39 +01:00
Lunny Xiao
8ad64574ee Fix markdown invoke sequence (#7513) 2019-07-18 16:23:27 -04:00