Give users the ability to disable gzip content encoding to increase throughput #3822
Description
- OS: all
- Java version: 1.8 (probably all)
- google-cloud-java version(s): 1.23.0
Steps to reproduce
- Upload to GCS using the cloud storage library.
- Observe poor throughput (20 MiB/s)_ and that lots of time is spent compressing the content.
On GCE I'm able to achieve 70 MiB/s if I disable gzip encoding when uploading to GCS. With gzip encoding it's much worse at 20 MiB/s. However disabling gzip encoding isn't an option exposed in the interface. This makes google-cloud-java poor in comparison to the REST api it's wrapping.
This is a request to expose the gzip encoding of the stream as a storage option. See code in com.google.cloud.storage.spi.v1.HttpStorageRpc.create there is no code to call StorageRequest.setDisableGZipContent (which defaults to false) on the underling com.google.api.services.storage.StorageRequest:
Storage.Objects.Insert insert = storage.objects()
.insert(storageObject.getBucket(), storageObject,
new InputStreamContent(storageObject.getContentType(), content));
insert.getMediaHttpUploader().setDirectUploadEnabled(true);
setEncryptionHeaders(insert.getRequestHeaders(), ENCRYPTION_KEY_PREFIX, options);
return insert.
... .execute();
Can we please provide this feature to maintain comparable performance with the com.google.api.services.storage.* client?
thanks
Code snippet
Uploading an object to GCS and observe performance, then do so with gzip disabled:
/// USING THE OLD CLIENT
Storage client = getService();
Storage.Objects.Insert insertRequest =
client.objects().insert(bucketName, objectMetadata, contentStream);
/// Currently NO ability to this in google-cloud-java, need this for much faster throughput on GCE.
insertRequest.setDisableGZipContent(true);
insertRequest.getMediaHttpUploader().setDisableGZipContent(true);
insertRequest.getMediaHttpUploader().setDirectUploadEnabled(true);
long start = System.currentTimeMillis();
insertRequest.execute();
long end = System.currentTimeMillis();
System.out.println( String.format("%.02f MiB/s ",(length / (1024*1024.0) )/ ((end - start) / 1000.0)));
Any additional information below
Also, perhaps document somewhere that in order to get any reasonable performance on Java8 it's necessary to disable Galios/Counter mode in java.security: https://stackoverflow.com/questions/25992131/slow-aes-gcm-encryption-and-decryption-with-java-8u20