sum in #unnormalize #193Description
Describe the bug
I'm not completely familiar with this repo so please enlighten me if I'm wrong. I suspect sum is calculated incorrectly in #unnormalize. rv.bytesize is added multiple times over, even for matches that has already been substituted.
rexml/lib/rexml/parsers/baseparser.rb
Lines 550 to 569 in e3f747f
How to reproduce
require "rexml/parsers/baseparser"
entity_less_than = "<"
entitiy_length = 100
filler_text = "A"
filler_length = 100
feed = "#{entity_less_than * entitiy_length}#{filler_text * filler_length}"
base_parser = REXML::Parsers::BaseParser.new("")
base_parser.unnormalize(feed) # => "<" * 100 + "A" * 100Error:
❯ bundle exec ruby test.rb
/Users/vikiv/.gem/ruby/3.1.2/gems/rexml-3.3.4/lib/rexml/parsers/baseparser.rb:560:in `block in unnormalize': entity expansion has grown too large (RuntimeError)
from /Users/vikiv/.gem/ruby/3.1.2/gems/rexml-3.3.4/lib/rexml/parsers/baseparser.rb:552:in `each'
from /Users/vikiv/.gem/ruby/3.1.2/gems/rexml-3.3.4/lib/rexml/parsers/baseparser.rb:552:in `unnormalize'
from test.rb:13:in `<main>'
Suggestion/fix
diff --git a/lib/rexml/parsers/baseparser.rb b/lib/rexml/parsers/baseparser.rb
index 28810bf..e6d6ae6 100644
--- a/lib/rexml/parsers/baseparser.rb
+++ b/lib/rexml/parsers/baseparser.rb
@@ -549,7 +549,7 @@ module REXML
matches.collect!{|x|x[0]}.compact!
if matches.size > 0
sum = 0
- matches.each do |entity_reference|
+ matches.uniq.each do |entity_reference|
unless filter and filter.include?(entity_reference)
entity_value = entity( entity_reference, entities )
if entity_value
@@ -557,7 +557,7 @@ module REXML
rv.gsub!( re, entity_value )
sum += rv.bytesize
if sum > Security.entity_expansion_text_limit
- raise "entity expansion has grown too large"
+ # raise "entity expansion has grown too large"
end
else
er = DEFAULT_ENTITIES[entity_reference]
@@ -566,6 +566,7 @@ module REXML
end
end
rv.gsub!( Private::DEFAULT_ENTITIES_PATTERNS['amp'], '&' )
+ puts "Sum of entity expansions: #{sum}"
end
rv
endResult:
# Before change
❯ bundle exec ruby test.rb
Sum of entity expansions: 20000
# After change
❯ bundle exec ruby test.rb
Sum of entity expansions: 200