Slow download performance for Storage API #3929
Description
The new version of the storage client (com.google.cloud:google-cloud-storage:1.52.0) appears download storage content at a MUCH slower rate than the legacy client (com.google.apis:google-api-services-storage:v1-rev141-1.25.0).
Legacy client (~40MB/s):
@Test
public void download() throws Exception {
Storage storage = buildStorage();
Storage.Objects.Get get = storage.objects().get(BUCKET_NAME, BUCKET_PATH);
StorageObject storageObject = get.execute();
File tempFile = createTempFile();
try (OutputStream out = new FileOutputStream(tempFile)) {
Stopwatch stopwatch = Stopwatch.createStarted();
get.getMediaHttpDownloader().setDirectDownloadEnabled(true);
get.executeMediaAndDownloadTo(out);
long elapsedSeconds = stopwatch.stop().elapsed(TimeUnit.SECONDS);
double fileSizeMb = storageObject.getSize().doubleValue() / 1024 / 1024;
double throughput = fileSizeMb / elapsedSeconds;
log.info("Completed download: elapsed={}s,fileSize={}MB,throughput={}MB/s",
elapsedSeconds, fileSizeMb, throughput);
assertTrue("Expected at least 20MB/s", throughput > 20);
}
}
New client (~10MB/s):
@Test
public void download() throws Exception {
Storage storage = StorageOptions.getDefaultInstance().getService();
Blob blob = storage.get(BLOB_ID);
Path tempFile = createTempFile();
Stopwatch stopwatch = Stopwatch.createStarted();
blob.downloadTo(tempFile);
long elapsedSeconds = stopwatch.stop().elapsed(TimeUnit.SECONDS);
double fileSizeMb = blob.getSize().doubleValue() / 1024 / 1024;
double throughput = fileSizeMb / elapsedSeconds;
log.info("Completed download: elapsed={}s,fileSize={}MB,throughput={}MB/s",
elapsedSeconds, fileSizeMb, throughput);
assertTrue("Expected at least 20MB/s", throughput > 20);
}
I'm attaching a couple of test cases that I've ran from a GCE instance (Ubuntu 16.04 with Java 1.8.0_191):