8000 Merge branch 'master' into ruby-skip · rubychan/coderay@cec02d0 · GitHub
[go: up one dir, main page]
Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Commit cec02d0

Browse files
kornykorny
committed
Merge branch 'master' into ruby-skip
2 parents 3fc5573 + 0a1f500 commit cec02d0

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

60 files changed

+126
-3093
lines changed

‎.travis.yml

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,10 @@ rvm:
22
- 1.8.7
33
- ree
44
- 1.9.3
5-
- 2.0.0
5+
- 2.0
6+
- 2.1
7+
- 2.2
8+
- 2.3.0
69
- ruby-head
710
- jruby-18mode
811
- jruby-19mode
@@ -19,3 +22,4 @@ matrix:
1922
- rvm: rbx-18mode
2023
- rvm: rbx-19mode
2124
script: "rake test" # test:scanners"
25+
sudo: false

‎Changes.textile

Lines changed: 69 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -2,42 +2,80 @@ h1=. CodeRay Version History
22

33
p=. _This files lists all changes in the CodeRay library since the 0.9.8 release._
44

5+
h2. Changes in 1.1.1
6+
7+
* SQL scanner: Allow @$@ signs in SQL identifiers [#164, thanks to jasir and Ben Basson]
8+
* SQL scanner: Fix open strings [#163, thanks to Adam]
9+
* Ruby scanner: Accept number literal suffixes @r@ and @i@ (Ruby 2.1)
10+
* Ruby scanner: Accept quoted hash keys like @{ "a": boss }@ (Ruby 2.2)
11+
* Ruby scanner: Accept save navigation operator @&.@ (Ruby 2.3)
12+
* Ruby scanner: Accept squiggly heredoc @<<~@ (Ruby 2.3)
13+
* Diff scanner: Prevent running out of regexp stack.
14+
* HTML encoder: You can keep tabs intact now by setting @tab_width: false@.
15+
516
h2. Changes in 1.1
617

7-
* New scanner: Lua [#21, #22, thanks to Quintus]
8-
* New scanner: Sass [#93]
9-
* New scanner: Go [#28, thanks to Eric Guo and Nathan Youngman]
10-
* New scanner: Taskpaper [#39, thanks to shimomura]
18+
New scanners:
19+
20+
* Go [#28, thanks to Eric Guo and Nathan Youngman]
21+
* Lua [#21, #22, thanks to Quintus]
22+
* Sass [#93]
23+
* Taskpaper [#39, thanks to shimomura]
24+
25+
More new stuff:
26+
27+
* @.xaml@ file type [#121, thanks to Kozman Bálint]
28+
* recognize @Guardfile@, @Vagrantfile@, and @Appraisals@ as Ruby files [#121, thanks to Kozman Bálint]
29+
* new token kind @:id@ for CSS/Sass [#27]
30+
* new token kind @:done@ for Taskpaper [#39]
31+
* new token kind @:map@ for Lua, introducing a nice nested-shades trick [#22, thanks to Quintus and Nathan Youngman]
32+
* new token kind @:unknown@ for Debug scanner
33+
* new DebugLint encoder that checks for empty tokens and correct nesting
34+
35+
Improvements:
36+
37+
* CSS scanner uses @:id@ and @:tag@ now [#27]
1138
* Diff scanner: Highlight inline changes in multi-line changes [#99]
1239
* JavaScript scanner: Highlight multi-line comments in diff correctly
13-
* Ruby scanner: Accept %i and %I symbol lists (Ruby 2.0) [thanks to Nathan Youngman]
14-
* Ruby scanner: Accept keywords as Ruby 1.9 hash keys [#126]
40+
* JSON scanner: simplify key/value heuristic, using look-ahead instead of a stack
1541
* HTML scanner displays style tags and attributes now [#145]
16-
* Remove double-click toggle handler from HTML table output
17-
* Fixes to CSS scanner (floats, pseudoclasses, nth-child) [#143]
18-
* Fixed empty tokens and unclosed token groups in HTML, CSS, Diff, Goovy, PHP, Raydebug, Ruby, SQL, and YAML scanners [#144]
19-
* Added @:keep_state@ functionality to more scanners [#116]
20-
* CSS scanner uses @:id@ and @:tag@ now [#27]
21-
* Removed @Tokens#dump@, @Tokens.load@, @Tokens::Undumping@, and @zlib@ dependency. Nobody was using this, right?
22-
* Add .xaml file type [#121, thanks to Kozman Bálint]
23-
* @CodeRay::TokenKinds@ should not be frozen [#130, thanks to Gavin Kistner]
24-
* New token type @:id@ for CSS/Sass [#27]
25-
* New token type @:done@ for Taskpaper [#39]
26-
* New token type @:map@ for Lua, introducing a nice nested-shades trick [#22, thanks to Quintus and Nathan Youngman]
27-
* New token type @:unknown@ for Debug scanner
28-
* Display line numbers in HTML @:table@ mode even for single-line code (remove special case) [#41, thanks to Ariejan de Vroom]
29-
* Override Bootstrap's @pre { word-break: break-all }@ styling for line numbers [#102, thanks to lightswitch05]
30-
* Fixed @:docstring@ token type style
31-
* Fixed several problems related to Hash caches and dynamic Symbol creation that might have been exploited by an attacker [#148]
32-
* @PluginHost@ now works with Strings instead of Symbols internally (to avoid using @#to_sym@)
33-
* The @Debug@ scanner maps unknown token kinds to @:unknown@ (to avoid creating Symbols based on possibly unsafe input)
34-
* The @Raydebug@ scanner highlights unknown token kinds as @:plain@
35-
* @Plugin@ does not warn about fallback when default is defined
36-
* Fixed @HTML@ encoder when output is a StringIO (eg. when using @-HTML@)
37-
* @HTML@ encoder will not warn about unclosed token groups at the end of the stream
38-
* @Debug@ encoder refactored; use @DebugLint@ if you want strict checking now
39-
* @Debug@ encoder will not warn about errors in the token stream
40-
* New @DebugLint@ encoder that checks for empty tokens and correct nesting
42+
* Ruby scanner: Accept @%i(…)@ and @%I(…)@ symbol lists (Ruby 2.0) [thanks to Nathan Youngman]
43+
* Ruby scanner: Accept keywords as Ruby hash keys [#126]
44+
* performance improvements to several scanners and encoders, especially Terminal and HTML
45+
* added @:keep_state@ functionality to more scanners so they work nicely with diff now [#116]
46+
* refactoring and cleanup to achieve better "Code Climate" ratings (but I don't really care)
47+
* updated and cleaned up the documentation,
48+
* documented list of TokenKinds
49+
* Alpha style: tweaked colors for @.binary@, @.local-variable@, and @.predefined-type@
50+
* @rake generate@ supports Git now instead of Subversion
51+
52+
Removed:
53+
54+
* @Tokens#dump@, @Tokens.load@, @Tokens::Undumping@, and @zlib@ dependency
55+
* double-click toggle handler from HTML table output
56+
* @rake_helpers@, @sample@ directories and several other ancient garbage
57+
58+
Fixes:
59+
60+
* fixes to CSS scanner (floats, pseudoclasses, nth-child) [#143]
61+
* fixed empty tokens and unclosed token groups in HTML, CSS, Diff, Goovy, PHP, Raydebug, Ruby, SQL, and YAML scanners [#144]
62+
* fixed @:docstring@ token type style
63+
* fixed several infinite Hash caches and dynamic Symbol creation that might have been exploited by an attacker [#148]
64+
* fixed HTML encoder when output is a StringIO (eg. when using @-HTML@ as a command line parameter)
65+
* TokenKinds should not be frozen [#130, thanks to Gavin Kistner]
66+
* display line numbers in HTML @:table@ mode even for single-line code (remove special case) [#41, thanks to Ariejan de Vroom]
67+
* override Bootstrap's @pre { word-break: break-all }@ styling for line numbers [#102, thanks to lightswitch05]
68+
* HTML encoder will not warn about unclosed token groups at the end of the stream
69+
* fixed problem with coderay/version.rb being loaded twice
70+
71+
Internals:
72+
73+
* The Debug scanner maps unknown token kinds to @:unknown@ (to avoid creating Symbols based on possibly unsafe input).
74+
* The Raydebug scanner highlights unknown token kinds as @:plain@.
75+
* The Debug encoder refactored; use DebugLint if you want strict checking now..
76+
* The Debug encoder will not warn about errors in the token stream.
77+
* Plugin does not warn about fallback when default is defined.
78+
* PluginHost now works with Strings instead of Symbols internally (to avoid using @#to_sym@).
4179

4280
h2. Changes in 1.0.9
4381

@@ -447,6 +485,3 @@ The helper classes were cleaned up; see above for details.
447485

448486
* *CHANGED* @Plugin@ API was simplified and stripped of all unnecessary features.
449487
* *CHANGED* Moved @GZip@ and @FileType@ libraries into @CodeRay@; cleaned them up.
450-
451-
452-

‎Gemfile

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6,11 +6,13 @@ gemspec
66
# Add dependencies to develop your gem here.
77
# Include everything needed to run rake, tests, features, etc.
88
group :development do
9-
gem "bundler", ">= 1.0.0"
9+
gem "bundler"
1010
gem "rake"
1111
gem "RedCloth", RUBY_PLATFORM == 'java' ? ">= 4.2.7" : ">= 4.0.3"
12-
gem "term-ansicolor", '~> 1.2.2'
13-
gem "shoulda-context", "~> 1.1.2"
12+
gem "term-ansicolor"
13+
gem 'tins', '~> 1.6.0'
14+
gem "shoulda-context"
15+
gem "test-unit"
1416
gem "json" if RUBY_VERSION < '1.9'
1517
gem "rdoc"
1618
end

‎README.markdown

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ You put your code in, and you get it back colored; Keywords, strings, floats, co
1616

1717
### Dependencies
1818

19-
CodeRay needs Ruby 1.8.7, 1.9.3 or 2.0. It also runs on JRuby.
19+
CodeRay needs Ruby 1.8.7, 1.9.3 or 2.0+. It also runs on JRuby.
2020

2121
## Example Usage
2222

@@ -28,4 +28,4 @@ html = CodeRay.scan("puts 'He 10000 llo, world!'", :ruby).div(:line_numbers => :table)
2828

2929
## Documentation
3030

31-
See [http://coderay.rubychan.de/doc/](http://coderay.rubychan.de/doc/).
31+
See [rubydoc](http://rubydoc.info/gems/coderay).

‎bin/coderay

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ defaults:
3535
3636
common:
3737
coderay file.rb # highlight file to terminal
38-
coderay file.rb > file.html # highlight file to HTML page
38+
coderay file.rb -page > file.html # highlight file to HTML page
3939
coderay file.rb -div > file.html # highlight file to HTML snippet
4040
4141
configure output:

‎lib/coderay/encoders/html.rb

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,8 @@ module Encoders
2525
# == Options
2626
#
2727
# === :tab_width
28-
# Convert \t characters to +n+ spaces (a number.)
28+
# Convert \t characters to +n+ spaces (a number or false.)
29+
# false will keep tab characters untouched.
2930
#
3031
# Default: 8
3132
#
@@ -180,7 +181,7 @@ def setup options
180181

181182
@break_lines = (options[:break_lines] == true)
182183

183-
@HTML_ESCAPE = HTML_ESCAPE.merge("\t" => ' ' * options[:tab_width])
184+
@HTML_ESCAPE = HTML_ESCAPE.merge("\t" => options[:tab_width] ? ' ' * options[:tab_width] : "\t")
184185

185186
@opened = []
186187
@last_opened = nil

‎lib/coderay/scanners/diff.rb

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -100,7 +100,7 @@ def scan_tokens encoder, options
100100
next
101101
elsif match = scan(/-/)
102102
deleted_lines_count += 1
103-
if options[:inline_diff] && deleted_lines_count == 1 && (changed_lines_count = 1 + check(/.*(?:\n\-.*)*/).count("\n")) && match?(/(?>.*(?:\n\-.*){#{changed_lines_count - 1}}(?:\n\+.*){#{changed_lines_count}})$(?!\n\+)/)
103+
if options[:inline_diff] && deleted_lines_count == 1 && (changed_lines_count = 1 + check(/.*(?:\n\-.*)*/).count("\n")) && changed_lines_count <= 100_000 && match?(/(?>.*(?:\n\-.*){#{changed_lines_count - 1}}(?:\n\+.*){#{changed_lines_count}})$(?!\n\+)/)
104104
deleted_lines = Array.new(changed_lines_count) { |i| skip(/\n\-/) if i > 0; scan(/.*/) }
105105
inserted_lines = Array.new(changed_lines_count) { |i| skip(/\n\+/) ; scan(/.*/) }
106106

‎lib/coderay/scanners/go.rb

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,6 @@
11
module CodeRay
22
module Scanners
33

4-
# Scanner for Go, copy from c
54
class Go < Scanner
65

76
register_for :go

‎lib/coderay/scanners/ruby.rb

Lines changed: 13 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -165,15 +165,19 @@ def scan_tokens encoder, options
165165
end
166166

167167
elsif match = scan(/ ' (?:(?>[^'\\]*) ')? | " (?:(?>[^"\\\#]*) ")? /mx)
168-
encoder.begin_group :string
169168
if match.size == 1
169+
kind = check(self.class::StringState.simple_key_pattern(match)) ? :key : :string
170+
encoder.begin_group kind
170171
encoder.text_token match, :delimiter
171-
state = self.class::StringState.new :string, match == '"', match # important for streaming
172+
state = self.class::StringState.new kind, match == '"', match # important for streaming
172173
else
174+
kind = value_expected == true && scan(/:/) ? :key : :string
175+
encoder.begin_group kind
173176
encoder.text_token match[0,1], :delimiter
174177
encoder.text_token match[1..-2], :content if match.size > 2
175178
encoder.text_token match[-1,1], :delimiter
176-
encoder.end_group :string
179+
encoder.end_group kind
180+
encoder.text_token ':', :operator if kind == :key
177181
value_expected = false
178182
end
179183

@@ -192,11 +196,14 @@ def scan_tokens encoder, options
192196
encoder.text_token match, :error
193197
method_call_expected = false
194198
else
195-
encoder.text_token match, self[1] ? :float : :integer # TODO: send :hex/:octal/:binary
199+
kind = self[1] ? :float : :integer # TODO: send :hex/:octal/:binary
200+
match << 'r' if match !~ /e/i && scan(/r/)
201+
match << 'i' if scan(/i/)
202+
encoder.text_token match, kind
196203
end
197204
value_expected = false
198205

199-
elsif match = scan(/ [-+!~^\/]=? | [:;] | [*|&]{1,2}=? | >>? /x)
206+
elsif match = scan(/ [-+!~^\/]=? | [:;] | &\. | [*|&]{1,2}=? | >>? /x)
200207
value_expected = true
201208
encoder.text_token match, :operator
202209

@@ -209,7 +216,7 @@ def scan_tokens encoder, options
209216
encoder.end_group kind
210217
heredocs ||= [] # create heredocs if empty
211218
heredocs << self.class::StringState.new(kind, quote != "'", delim,
212-
self[1] == '-' ? :indented : :linestart)
219+
self[1] ? :indented : :linestart)
213220
value_expected = false
214221

215222
elsif value_expected && match = scan(/#{patterns::FANCY_STRING_START}/o)

‎lib/coderay/scanners/ruby/patterns.rb

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -114,7 +114,7 @@ module Ruby::Patterns # :nodoc: all
114114
# NOTE: This is not completely correct, but
115115
# nobody needs heredoc delimiters ending with \n.
116116
HEREDOC_OPEN = /
117-
<< (-)? # $1 = float
117+
<< ([-~])? # $1 = float
118118
(?:
119119
( [A-Za-z_0-9]+ ) # $2 = delim
120120
|

‎lib/coderay/scanners/ruby/string_state.rb

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,14 @@ class StringState < Struct.new :type, :interpreted, :delim, :heredoc,
3737
end
3838
end
3939

40+
def self.simple_key_pattern delim
41+
if delim == "'"
42+
/ (?> (?: [^\\']+ | \\. )* ) ' : /mx
43+
else
44+
/ (?> (?: [^\\"\#]+ | \\. | \#\$[\\"] | \#\{[^\{\}]+\} | \#(?!\{) )* ) " : /mx
45+
end
46+
end
47+
4048
def initialize kind, interpreted, delim, heredoc = false
4149
if heredoc
4250
pattern = heredoc_pattern delim, interpreted, heredoc == :indented

‎lib/coderay/scanners/sql.rb

Lines changed: 17 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,12 @@ class SQL < Scanner
5757

5858
STRING_PREFIXES = /[xnb]|_\w+/i
5959

60+
STRING_CONTENT_PATTERN = {
61+
'"' => / (?: [^\\"] | "" )+ /x,
62+
"'" => / (?: [^\\'] | '' )+ /x,
63+
'`' => / (?: [^\\`] | `` )+ /x,
64+
}
65+
6066
def scan_tokens encoder, options
6167

6268
state = :initial
@@ -90,7 +96,7 @@ def scan_tokens encoder, options
9096
state = :string
9197
encoder.text_token match, :delimiter
9298

93-
elsif match = scan(/ @? [A-Za-z_][A-Za-z_0-9]* /x)
99+
elsif match = scan(/ @? [A-Za-z_][A-Za-z_0-9\$]* /x)
94100
encoder.text_token match, name_expected ? :ident : (match[0] == ?@ ? :variable : IDENT_KIND[match])
95101
name_expected = false
96102

@@ -115,40 +121,26 @@ def scan_tokens encoder, options
115121
end
116122

117123
elsif state == :string
118-
if match = scan(/[^\\"'`]+/)
119-
string_content << match
120-
next
124+
if match = scan(STRING_CONTENT_PATTERN[string_type])
125+
encoder.text_token match, :content
121126
elsif match = scan(/["'`]/)
122127
if string_type == match
123128
if peek(1) == string_type # doubling means escape
124-
string_content << string_type << getch
125-
next
126-
end
127-
unless string_content.empty?
128-
encoder.text_token string_content, :content
129-
string_content = ''
129+
encoder.text_token match + getch, :content
130+
else
131+
encoder.text_token match, :delimiter
132+
encoder.end_group :string
133+
state = :initial
134+
string_type = nil
130135
end
131-
encoder.text_token match, :delimiter
132-
encoder.end_group :string
133-
state = :initial
134-
string_type = nil
135136
else
136-
string_content << match
137+
encoder.text_token match, :content
137138
end
138139
elsif match = scan(/ \\ (?: #{ESCAPE} | #{UNICODE_ESCAPE} ) /mox)
139-
unless string_content.empty?
140-
encoder.text_token string_content, :content
141-
string_content = ''
142-
end
143140
encoder.text_token match, :char
144141
elsif match = scan(/ \\ . /mox)
145-
string_content << match
146-
next
142+
encoder.text_token match, :content
147143
elsif match = scan(/ \\ | $ /x)
148-
unless string_content.empty?
149-
encoder.text_token string_content, :content
150-
string_content = ''
151-
end
152144
encoder.text_token match, :error unless match.empty?
153145
encoder.end_group :string
154146
state = :initial

‎lib/coderay/version.rb

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
11
module CodeRay
2-
VERSION = '1.1.0'
2+
VERSION = '1.1.1'
33
end

‎rake_tasks/test.rake

Lines changed: 0 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,4 @@
11
namespace :test do
2-
desc 'run all sample tests'
3-
task :samples do
4-
ruby './test/samples/suite.rb'
5-
end
6-
72
desc 'run functional tests'
83
task :functional do
94
ruby './test/functional/suite.rb'
@@ -85,4 +80,3 @@ Please rename or remove it and run again to use the GitHub repository:
8580
end
8681

8782
task :test => %w(test:functional test:units test:exe)
88-
task :samples => 'test:samples'

‎test/samples/README

Lines changed: 0 additions & 1 deletion
This file was deleted.

‎test/samples/cache.actual

Lines changed: 0 additions & 2 deletions
This file was deleted.

‎test/samples/cache.expected

Lines changed: 0 additions & 2 deletions
This file was deleted.

0 commit comments

Comments
 (0)
0