8000 logging: make flush wait for writes by pongad · Pull Request #1815 · googleapis/google-cloud-java · GitHub
[go: up one dir, main page]
Skip to content

logging: make flush wait for writes #1815

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
garrettjonesgoogle merged 6 commits into from
Mar 31, 2017
Merged

logging: make flush wait for writes #1815

garrettjonesgoogle merged 6 commits into from
Mar 31, 2017

Conversation

@pongad
Copy link
Contributor
@pongad pongad commented Mar 28, 2017

Updates 1795.

This PR still isn't completely correct,
since it does not force any RPC to immediately be issued.
However, flush should now correctly wait for RPCs
representing prior calls to publish to complete
and any failures to be reported to ErrorManager
before returning.

This PR is WIP due to lack of tests.
They are forth-coming,
but all existing tests pass.

Fun fact:
I wrote this PR three times,
using synchronized, ReentrantLock, then Monitor
mirroring the Monitor class's documentation:
http://google.github.io/guava/releases/21.0/api/docs/com/google/common/util/concurrent/Monitor.html

@googlebot googlebot added the cla: yes This human has signed the Contributor License Agreement. label Mar 28, 2017
michaelbausor
michaelbausor reviewed Mar 28, 2017
View reviewed changes
google-cloud-logging/src/main/java/com/google/cloud/logging/LoggingHandler.java Outdated
}
}
});
flushMonitor.enterWhenUninterruptibly(noFlushers);

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

pongad added 2 commits March 29, 2017 17:12
@pongad
WIP: logging: make flush wait for writes
f2d3c73 
Updates 1795.

This PR still isn't completely correct,
since it does not force any RPC to immediately be issued.
However, flush should now correctly wait for RPCs
representing prior calls to publish to complete
and any failures to be reported to ErrorManager
before returning.

This PR is WIP due to lack of tests.
They are forth-coming,
but all existing tests pass.

Fun fact:
I wrote this PR three times,
using `synchronized`, `ReentrantLock`, then `Monitor`
mirroring the Monitor class's documentation:
http://google.github.io/guava/releases/21.0/api/docs/com/google/common/util/concurrent/Monitor.html
@pongad
allow concurrent writes and flushes
af0cbae
@pongad
Copy link
Contributor Author
pongad commented Mar 29, 2017

@michaelbausor @garrettjonesgoogle PTAL

@coveralls
Copy link
coveralls commented Mar 29, 2017

Coverage Status

Coverage increased (+0.005%) to 80.807% when pulling af0cbae on pongad:logging-sync into 38bace4 on GoogleCloudPlatform:master.

@pongad
Copy link
Contributor Author
pongad commented Mar 29, 2017

@michaelbausor @garrettjonesgoogle Actually, please hold off. I'm no longer convinced this is right.

Consider the following events:

  • A publish RPC is sent, its future recorded in the table
  • The RPC fails. We invoke a callback, roughly calling callback.onFailure(throwable).
  • Since we only need the callback and the throwable, the future itself might not be needed. The future gets GCed
  • Someone calls flush, since the future is already GCed, flush returns.
  • Then, the callback calls reportError.

Thus, the flush might return before the error could be reported.

@pongad
Copy link
Contributor Author
pongad commented Mar 29, 2017

@michaelbausor @garrettjonesgoogle I hope this is right now. Still needs tests and force-RPC-to-fire thing.

@coveralls
Copy link
coveralls commented Mar 29, 2017

Coverage Status

Coverage decreased (-0.004%) to 80.798% when pulling baf2d43 on pongad:logging-sync into 38bace4 on GoogleCloudPlatform:master.

@michaelbausor
Copy link
Contributor
michaelbausor commented Mar 29, 2017

Looks good!

garrettjonesgoogle
garrettjonesgoogle reviewed Mar 29, 2017
View reviewed changes
google-cloud-logging/src/main/java/com/google/cloud/logging/LoggingHandler.java Outdated

@Override
public void flush() {
// BUG(1795): flush is broken, need support from batching implementation.

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

google-cloud-logging/src/main/java/com/google/cloud/logging/LoggingHandler.java Outdated
reportError(null, new Exception(t), ErrorManager.FLUSH_FAILURE);
}
} finally {
onSuccess(null);

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

google-cloud-logging/src/main/java/com/google/cloud/logging/LoggingHandler.java
pendingWrites.add(writeFuture);
} finally {
writeLock.unlock();
}

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

google-cloud-logging/src/main/java/com/google/cloud/logging/LoggingHandler.java Outdated
public void flush() {
// BUG(1795): flush is broken, need support from batching implementation.

ArrayList<ApiFuture<Void>> writes = new ArrayList<>();

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

@coveralls
Copy link
coveralls commented Mar 30, 2017

Coverage Status

Coverage increased (+0.004%) to 80.806% when pulling 5378b39 on pongad:logging-sync into 38bace4 on GoogleCloudPlatform:master.

garrettjonesgoogle
garrettjonesgoogle reviewed Mar 30, 2017
View reviewed changes
google-cloud-logging/src/test/java/com/google/cloud/logging/LoggingHandlerTest.java
assertTrue(flushWaiter.isAlive());

// With the RPC completed, flush should return, and the thread should terminate.
mockRpc.set(null);

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

google-cloud-logging/src/test/java/com/google/cloud/logging/LoggingHandlerTest.java
});
flushWaiter.start();

// flushWaiter should be waiting for mockRpc to complete.

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

@pongad pongad changed the title WIP: logging: make flush wait for writes logging: make flush wait for writes Mar 31, 2017
@pongad
Copy link
Contributor Author
pongad commented Mar 31, 2017

@garrettjonesgoogle PTAL

@coveralls
Copy link
coveralls commented Mar 31, 2017

Coverage Status

Coverage increased (+0.02%) to 80.82% when pulling 91f3a33 on pongad:logging-sync into 38bace4 on GoogleCloudPlatform:master.

@garrettjonesgoogle garrettjonesgoogle merged commit d726f41 into googleapis:master Mar 31, 2017
@pongad pongad deleted the logging-sync branch April 3, 2017 04:43
garrettjonesgoogle pushed a commit that referenced this pull request Apr 11, 2017
@rybosome @garrettjonesgoogle
Rebase master (#1900)
05990c8 
* updating 'PubSubExample' to latest api (#1808)

* updating pubsub sample to latest api

* Pubsub update (#1818)

* Update GAPIC layer
* Manual updates to support pubsub changes

* Update spi classes (#1817)

* Update README.md (#1825) [ci skip]

* Add DURABLE_REDUCED_AVAILABILITY storage class (#1834)

Otherwise we crash when servers return this value.

* logging: make flush wait for writes (#1815)

This PR still isn't completely correct,
since it does not force any RPC to immediately be issued.
However, flush should now correctly wait for RPCs
representing prior calls to publish to complete
and any failures to be reported to ErrorManager
before returning.

* use new PartitionKey implementation (#1841)

bumping gax version to 0.8.0 for the new implementation.
reflects googleapis/gapic-generator#1153 .

* Release 0.11.0

* Updating version in README files. [ci skip]

* Update version to 0.11.1-SNAPSHOT (#1843)

* Added a missing @test annotation (#1842)

* pubsub: make Subscriber use ApiService (#1824)

Fixes #1761.

* SPI: Adding @experimentalapi back to logging client classes (#1844)

* Bumping NL, Translate to beta (#1848)

* Release 0.11.1

* Updating version in README files. [ci skip]

* Update version to 0.11.2-SNAPSHOT (#1852)

* pubsub: remove obsolete doc references (#1823)

This PR removes references to the deprecated code of the docs.
The emulator section is rewritten.

Fixes #1789.

* adding functions to manage life cycle of resources in ITComputeTest (#1768)

*adding functions to ITComputeTest, in order to make sure resources
created during tests can be properly deleted even if tests fail or
become timed out.
* refactor and incorporate feedbacks
* implement `add` function with compile-time type checking
* add `remove` method to remove a resource from managed resources
* use Id's as handles to resources
* fix copyright header
* rename class name
* modify remove function, pass delete function to each add method
* address comments

* Add Speech v1. (#1858)

* Make logging overrides the default channel provider (#1820)

* Added more unit tests for SessionPool (#1862)

* Rename Translate title to Translation [ci skip] (#1867)

* Release 0.11.2

* Updating version in README files. [ci skip]

* Update pom.xml version to 0.11.3-SNAPSHOT (#1870)

* pubsub: acquire FlowController before releasing (#1831)

* Revert "pubsub: acquire FlowController before releasing (#1831)" (#1872)

This reverts commit 3717ac6.

This change brings up another serious bug.
If the number of messages we pull in one RPC is greater than the number size of the semaphore, we deadlock forever.
Will redo this later.

* pubsub: make deprecated methods package-private (#1861)

add back mistakenly deleted test

Fixes #1828.

* pubsub: rename newBuilder to defaultBuilder (#1873)

Fixes #1853.

* GAE (Flex_Java/Flex_Custom/Flex_Compat/Std_Java8), GCE, GKE testing app for gcj (#1859)

Appengine tests for #1752

* Replace a constant of type Set with ImmutableSet (#1876)

* Language v1beta2 Release (#1878)

* Language v1beta2 Release
* Language v1 update

* Regenerating SPI: use setEndpoint (#1879)

Reflects googleapis/gapic-generator#1172.
Updates #1835.

* remove last use of setPort/setServiceAddress (#1880)

Fixes #1835.

* new code snippet  for push subscription + cleanup of deprecated snippets (#1875)

* cleaning up PubSubExample
adding snippet for creating a subscription with a push endpoint

* updating start, end tags for snippets,
adding async pull snippet tag

* Release 0.12.0

* Updating version in README files. [ci skip]

* Update version to 0.12.1-SNAPSHOT (#1886)

* Update version of google-auth-java to 0.6.1 (#1888)

* Update version of google-auth-java to 0.6.1

Latest version of google-auth-java contains a fix for auth token refresh failures.

* Remove harcoded auth dependencies

* Don't use `UrlFetchTransport` in App Engine Flex environment  (#1893)

* Don't use `UrlFetchTransport` in App Engine Flex environment #1492

* Add annotations to specify GCP launch stage (#1889)

The Google Cloud Platform launch stage
(https://cloud.google.com/terms/launch-stages) is a signifier of the level of
access and support that can be expected of a particular feature. These
annotations will be used to clearly demarcate features as being in a state
other than General Availability to help set user expectations accordingly.

* Add Identity Access Management (IAM) to the Storage API (#1812)

Adds support for bucket-level IAM (currently in limited alpha). More information
about IAM in Google Cloud Storage can be found at
https://cloud.google.com/storage/docs/access-control/iam
meltsufin pushed a commit that referenced this pull request Dec 22, 2025
@cloud-java-bot
chore: Update generation configuration at Thu Jun 5 02:30:55 UTC 2025 (…
591ce2f 
…#1815)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

2 more reviewers

@garrettjonesgoogle garrettjonesgoogle garrettjonesgoogle left review comments

@michaelbausor michaelbausor michaelbausor left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

cla: yes This human has signed the Contributor License Agreement.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@pongad @coveralls @michaelbausor @garrettjonesgoogle @googlebot
0