The time-consuming issue of assigning values to flow.response.content #8040

Karmylr · 2026-01-08T14:10:23Z

Karmylr
Jan 8, 2026

The timing starts after the program processes the response body and ends after the assignment is completed,

start_time = time.time()
response_body = flow.response.content
end_time = time.time()
print(f'It takes {end_time - start_time:.4f} seconds to obtain {len(response_body)} bytes of data from flow.response.content.')

data = msg.SerializeToString()
length = len(data)
start_time = time.time()
flow.response.content = data
end_time = time.time()
print(f'It takes {end_time - start_time:.4f} seconds to assign {length} bytes of data to flow.response.content.')

result:

It takes 0.0047 seconds to obtain 1221005 bytes of data from flow.response.content.
It takes 3.8846 seconds to assign 1211880 bytes of data to flow.response.content.

This is much slower than obtaining the response body. Is this normal?
(I am not a professional, so I don't know what more information is needed. If you need me to provide more information, please feel free to say.)

Answered by Prinzhorn

Jan 8, 2026

You would need to provide more context. This would be expected if the content-encoding is gzip or some other compression, because compression is a lot slower than decompression by design.

(.content will transparently decode/encode .raw_content depending on the content-encoding header)

Depending on what you're doing (if the client will accept uncompressed data) it could be faster to strip the header and set the raw bytes.

View full answer

Prinzhorn · 2026-01-08T17:29:56Z

Prinzhorn
Jan 8, 2026
Collaborator

You would need to provide more context. This would be expected if the content-encoding is gzip or some other compression, because compression is a lot slower than decompression by design.

(.content will transparently decode/encode .raw_content depending on the content-encoding header)

Depending on what you're doing (if the client will accept uncompressed data) it could be faster to strip the header and set the raw bytes.

6 replies

Prinzhorn Jan 9, 2026
Collaborator

That doesn't sound right either. You're saying mitmproxy is much slower at brotli encoding than you are? Or are you now only measuring the assignment and not the time it takes for you to do brotli? Could you please share your code?

Karmylr Jan 9, 2026
Author

That doesn't sound right either. You're saying mitmproxy is much slower at brotli encoding than you are? Or are you now only measuring the assignment and not the time it takes for you to do brotli? Could you please share your code?

msg = response_browse.YT_browse_response() # protobuf
try:
    msg.ParseFromString(flow.response.content)
except DecodeError:
    return
# prune some fields of buffer here

flow.response.raw_content = brotli.compress(
    msg.SerializeToString(),
    quality=1,
    lgwin=22
)

faster than directly flow.response.content = msg.SerializeToString().

Prinzhorn Jan 9, 2026
Collaborator

How much faster are we talking? Here's the mitmproxy implementation:

mitmproxy/mitmproxy/http.py

Lines 360 to 383 in e6aa924

    
           def set_content(self, value: bytes | None) -> None: 
        
               if value is None: 
        
                   self.raw_content = None 
        
                   return 
        
               if not isinstance(value, bytes): 
        
                   raise TypeError( 
        
                       f"Message content must be bytes, not {type(value).__name__}. " 
        
                       "Please use .text if you want to assign a str." 
        
                   ) 
        
               ce = self.headers.get("content-encoding") 
        
               try: 
        
                   self.raw_content = encoding.encode(value, ce or "identity") 
        
               except ValueError: 
        
                   # So we have an invalid content-encoding? 
        
                   # Let's remove it! 
        
                   del self.headers["content-encoding"] 
        
                   self.raw_content = value 
        
               if "transfer-encoding" in self.headers: 
        
                   # https://httpwg.org/specs/rfc7230.html#header.content-length 
        
                   # don't set content-length if a transfer-encoding is provided 
        
                   pass 
        
               else: 
        
                   self.headers["content-length"] = str(len(self.raw_content))

mitmproxy/mitmproxy/net/encoding.py

Lines 93 to 136 in e6aa924

    
           def encode( 
        
               decoded: None | str | bytes, encoding, errors="strict" 
        
           ) -> None | str | bytes: 
        
               """ 
        
               Encode the given input object 
        
               Returns: 
        
                   The encoded value 
        
               Raises: 
        
                   ValueError, if encoding fails. 
        
               """ 
        
               if decoded is None: 
        
                   return None 
        
               encoding = encoding.lower() 
        
               global _cache 
        
               cached = ( 
        
                   isinstance(decoded, bytes) 
        
                   and _cache.decoded == decoded 
        
                   and _cache.encoding == encoding 
        
                   and _cache.errors == errors 
        
               ) 
        
               if cached: 
        
                   return _cache.encoded 
        
               try: 
        
                   try: 
        
                       encoded = custom_encode[encoding](decoded) 
        
                   except KeyError: 
        
                       encoded = codecs.encode(decoded, encoding, errors)  # type: ignore 
        
                   if encoding in ("gzip", "deflate", "deflateraw", "br", "zstd"): 
        
                       _cache = CachedDecode(encoded, encoding, errors, decoded) 
        
                   return encoded 
        
               except TypeError: 
        
                   raise 
        
               except Exception as e: 
        
                   raise ValueError( 
        
                       "{} when encoding {} with {}: {}".format( 
        
                           type(e).__name__, 
        
                           repr(decoded)[:10], 
        
                           repr(encoding), 
        
                           repr(e), 
        
                       ) 
        
                   )

mitmproxy/mitmproxy/net/encoding.py

Lines 225 to 233 in e6aa924

    
           custom_encode = { 
        
               "none": identity, 
        
               "identity": identity, 
        
               "gzip": encode_gzip, 
        
               "deflate": encode_deflate, 
        
               "deflateraw": encode_deflate, 
        
               "br": encode_brotli, 
        
               "zstd": encode_zstd, 
        
           }

mitmproxy/mitmproxy/net/encoding.py

Lines 176 to 177 in e6aa924

    
           def encode_brotli(content: bytes) -> bytes: 
        
               return brotli.compress(content)

Karmylr Jan 18, 2026
Author

That is to say, mitmproxy uses brotli compression by default for br? Could it be that I set the quality to 1, which resulted in faster speed?

Prinzhorn Jan 26, 2026
Collaborator

Yeah, we're using the default options:

https://github.com/google/brotli/blob/5fa73e23bee34f84148719576a7a434f0fc43dc8/python/brotli.py#L25C1-L25C74

Uh oh!

The time-consuming issue of assigning values to flow.response.content #8040

Uh oh!

Uh oh!

Karmylr Jan 8, 2026

Replies: 1 comment · 6 replies

Uh oh!

Uh oh!

Prinzhorn Jan 8, 2026 Collaborator

Uh oh!

Prinzhorn Jan 9, 2026 Collaborator

Uh oh!

Uh oh!

Karmylr Jan 9, 2026 Author

Uh oh!

Uh oh!

Prinzhorn Jan 9, 2026 Collaborator

Uh oh!

Karmylr Jan 18, 2026 Author

Uh oh!

Prinzhorn Jan 26, 2026 Collaborator

Karmylr
Jan 8, 2026

Replies: 1 comment 6 replies

Prinzhorn
Jan 8, 2026
Collaborator

Prinzhorn Jan 9, 2026
Collaborator

Karmylr Jan 9, 2026
Author

Prinzhorn Jan 9, 2026
Collaborator

Karmylr Jan 18, 2026
Author

Prinzhorn Jan 26, 2026
Collaborator