"SELECT * FROM ks1.table1" fails: Failed decoding result column "ck2" of type timestamp: days=2147458792; must have magnitude <= 999999999 #255
Comments
|
Since it is flaky failure. Installation detailsKernel Version: 5.15.0-1040-aws Cluster size: 3 nodes (i3.large) Scylla Nodes used in this run:
OS / Image: Test: Logs and commands
Logs:
|
|
The Installation detailsKernel Version: 5.15.0-1039-aws Cluster size: 3 nodes (i3.large) Scylla Nodes used in this run:
OS / Image: Test: Logs and commands
Logs:
|
|
Looks very similar to the following closed bug: scylladb/scylladb#12239 |
|
Looks like a duplicate of It was fixed in Gemini, and not sure if it's in 1.8.6 version |
|
It seems this issue is about the "timestamp", the other was about "time". |
|
@eliransin please assign it, adding it as a blocker until we will have some diagnostic from your team. |
|
@vponomaryov - can you try with gemini pre 1.8.4, to see if it happens? I'm quite sure it's a change in Gemini. It may or may not be a bug that we need to fix in Scylla side, but I'd like to ensure it's not a regression, first of all. |
Yes, this bug was caught using the Either Scylla or python driver have bug: Scylla allows to write values which driver complains about. From the To summarize:
|
|
@cvybhu is this in your plans for this Q? |
No, I haven't seen this before. To me this looks like a python driver bug. The rust driver allows to insert and select a timestamp equal to use scylla::{Session, SessionBuilder};
#[tokio::main]
async fn main() {
let session: Session = SessionBuilder::new()
.known_node("127.0.0.1:9042")
.build()
.await
.unwrap();
session.query("CREATE KEYSPACE IF NOT EXISTS ks WITH REPLICATION = {'class' : 'SimpleStrategy', 'replication_factor' : 1}", &[]).await.unwrap();
session
.query(
"CREATE TABLE IF NOT EXISTS ks.t (p timestamp PRIMARY KEY)",
&[],
)
.await
.unwrap();
session
.query("INSERT INTO ks.t (p) VALUES (?)", (i64::MAX,))
.await
.unwrap();
session
.query("INSERT INTO ks.t (p) VALUES (?)", (i64::MIN,))
.await
.unwrap();
let ps: Vec<i64> = session
.query("SELECT p FROM ks.t", &[])
.await
.unwrap()
.rows_typed::<(i64,)>()
.unwrap()
.map(|parse_res| parse_res.unwrap())
.map(|p: (i64,)| p.0)
.collect();
for p in ps {
println!("p: {}", p)
}
}# Output:
p: 9223372036854775807
p: -9223372036854775808OTOH [cqlsh 6.0.9 | Scylla 5.2.3-0.20230608.ea08d409f155 | CQL spec 3.3.1 | Native protocol v4]
Use HELP for help.
cqlsh> SELECT p FROM ks.t;
/home/jancio/.local/lib/python3.11/site-packages/cqlsh/cqlsh.py:403: DateOverFlowWarning: Some timestamps are larger than Python datetime can represent. Timestamps are displayed in milliseconds from epoch.
p
----------------------
9223372036854775807
-9223372036854775808
(2 rows)The CQL spec defines To me it doesn't look like a Scylla bug. Scylla correctly writes and reads timestamp values, it's the driver that complains about "out of range values", so the bug is in the driver. /cc @avelanarius |
It's not the first time we are running into such a warning, since in some cases it surfaced a real scylla issue (like using the wrong clock), I would argue we shouldn't remove the warning from cqlsh/python-driver In this case I do agree it's not scylla core fault and caused by a change in scylla-bench, one if those were already fixed in scylla-bench, if a user puts a timestamp of 292471209 years into the future, I think he would like to know it... |
|
The same problem was reported in Cassandra eight years ago - see https://issues.apache.org/jira/browse/CASSANDRA-10625. They fixed cqlsh (see apache/cassandra@trunk...pauloricardomg:trunk-10625) to print the warning that @cvybhu saw above instead of failing, but didn't fix the driver. Maybe there are additional cases (not just cqlsh...) that need to be fixed? But this issue should probably be reopened in the driver repository, not here. I think Scylla's handling of the "timestamp" column type isn't the problem here. |
I agree it's should be move to python-driver (and might be a problem also on other drivers that convert to date objects) |
|
This is a real bug in the Python driver, that has been known for nine years (since https://issues.apache.org/jira/browse/CASSANDRA-10625) and unfortunately instead of fixing it we ended up hacking cqlsh and now gemini to work around it. But it just needs to be fixed. Basically, if The timestamp returned by Scylla for totimestamp(d) is valid, but the Scylla driver doesn't agree to accept it from the server (!), saying: (note that the date |
|
Unfortunately, it a timestamp of more than 999999999 days isn't the only problem, even lower-magnitude timestamps cause problems. I tried in the above example to set d to from: |
|
I believe these errors are present because we overflow python |
In Python `datetime.datetime` type year has to be in range [MINYEAR, MAXYEAR]. This range is not the same as possible timestamps in scylla. Previously if timestamp was outside this range it made driver raise an Exception. It was not correct behavior. There was a work around implemented in cqlsh. This commit introduces a `Datetime` type to accommodate ranges outside datetime.[MIN|MAX]YEAR. For Datetimes that cannot be represented as a datetime.datetime (because datetime.MINYEAR, datetime.MAXYEAR), this type falls back to printing milliseconds_from_epoch offset. Fixes: scylladb#255
In Python `datetime.datetime` type year has to be in range [MINYEAR, MAXYEAR]. This range is not the same as possible timestamps in scylla. Previously if timestamp was outside this range it made driver raise an Exception. It was not correct behavior. There was a work around implemented in cqlsh. This commit introduces a `Datetime` type to accommodate ranges outside datetime.[MIN|MAX]YEAR. For Datetimes that cannot be represented as a datetime.datetime (because datetime.MINYEAR, datetime.MAXYEAR), this type falls back to printing milliseconds_from_epoch offset. Fixes: scylladb#255
In Python `datetime.datetime` type year has to be in range [MINYEAR, MAXYEAR]. This range is not the same as possible timestamps in scylla. Previously if timestamp was outside this range it made driver raise an Exception. It was not correct behavior. There was a work around implemented in cqlsh. This commit introduces a `Datetime` type to accommodate ranges outside datetime.[MIN|MAX]YEAR. For Datetimes that cannot be represented as a datetime.datetime (because datetime.MINYEAR, datetime.MAXYEAR), this type falls back to printing milliseconds_from_epoch offset. Fixes: scylladb#255
In Python `datetime.datetime` type year has to be in range [MINYEAR, MAXYEAR]. This range is not the same as possible timestamps in scylla. Previously if timestamp was outside this range it made driver raise an Exception. It was not correct behavior. There was a work around implemented in cqlsh. This commit introduces a `Datetime` type to accommodate ranges outside datetime.[MIN|MAX]YEAR. For Datetimes that cannot be represented as a datetime.datetime (because datetime.MINYEAR, datetime.MAXYEAR), this type falls back to printing milliseconds_from_epoch offset. Fixes: scylladb#255
In Python `datetime.datetime` type year has to be in range [MINYEAR, MAXYEAR]. This range is not the same as possible timestamps in scylla. Previously if timestamp was outside this range it made driver raise an Exception. It was not correct behavior. There was a work around implemented in cqlsh. This commit introduces a `Datetime` type to accommodate ranges outside datetime.[MIN|MAX]YEAR. For Datetimes that cannot be represented as a datetime.datetime (because datetime.MINYEAR, datetime.MAXYEAR), this type falls back to printing milliseconds_from_epoch offset. Fixes: scylladb#255
In Python `datetime.datetime` type year has to be in range [MINYEAR, MAXYEAR]. This range is not the same as possible timestamps in scylla. Previously if timestamp was outside this range it made driver raise an Exception. It was not correct behavior. There was a work around implemented in cqlsh. This commit introduces a `Datetime` type to accommodate ranges outside datetime.[MIN|MAX]YEAR. For Datetimes that cannot be represented as a datetime.datetime (because datetime.MINYEAR, datetime.MAXYEAR), this type falls back to printing milliseconds_from_epoch offset. Fixes: scylladb#255
In Python `datetime.datetime` type year has to be in range [MINYEAR, MAXYEAR]. This range is not the same as possible timestamps in scylla. Previously if timestamp was outside this range it made driver raise an Exception. It was not correct behavior. There was a work around implemented in cqlsh. This commit introduces a `Datetime` type to accommodate ranges outside datetime.[MIN|MAX]YEAR. For Datetimes that cannot be represented as a datetime.datetime (because datetime.MINYEAR, datetime.MAXYEAR), this type falls back to printing milliseconds_from_epoch offset. Fixes: scylladb#255
|
The fix proposed in #310 is a breaking change that may break user code - which we should not do. Given that the issue is not a big problem, unlikely to be an issue for any client (as it is just an unlikely edge-case) and fixing it creates other problems I think we can let upstream deal with it and close the PR and issue (as wontfix) unless someone has any objections. |
|
We are not merging the fix in #310 to not break backwards compatibility, but if someone needs datetimes outside datetime.[MIN|MAX]YEAR feel free to cherry-pick. |
Issue description
Running the
rebuild_streaming_errnemesis in thegeminiCI job usinggemini-v1.8.6, where we walk through the test tables and check for data presence, we get following error:Impact
Describe the impact this issue causes to the user.
How frequently does it reproduce?
Happens in about
50% casesusinggemini-v1.8.6in the gemini CI job if no data mismatches get caught.Installation details
Kernel Version: 5.15.0-1040-aws
Scylla version (or git commit hash):
5.2.6-20230730.58acf071bf28with build-id17961be569f8503b27ff284a8de1e00a9d83811eCluster size: 3 nodes (i3.large)
Scylla Nodes used in this run:
OS / Image:
ami-0f1f419420235978b(aws: us-east-1)Test:
vp-gemini-3h-with-nemesis-testTest id:
985fe5dc-f634-47d6-b517-814916df7a5eTest name:
scylla-staging/valerii/vp-gemini-3h-with-nemesis-testTest config file(s):
Logs and commands
$ hydra investigate show-monitor 985fe5dc-f634-47d6-b517-814916df7a5e$ hydra investigate show-logs 985fe5dc-f634-47d6-b517-814916df7a5eLogs:
Jenkins job URL
Argus
The text was updated successfully, but these errors were encountered: