LATENESS/GC downstream of recursive views #5492

bens-secret-alt-account · 2026-01-22T16:30:15Z

bens-secret-alt-account
Jan 22, 2026

Hi all, I've been absolutely loving feldera, thank you for an amazing project. I have a question about GC which I'd appreciate a gut check on. The TLDR of my question is -- would it theoretically be possible to add LATENESS GC to views which are downstream (not inside) of recursive queries, if I am confident my application assumptions are safe?

I have an application that ingests a large volume of payload records associated with nodes in a tree/forest. The node graph itself is relatively small and can be retained indefinitely, but the payloads are extremely large and must be garbage-collected to keep memory reasonably bounded.

Each node has a numeric height (incrementing coordinate along parent edges) and at most one parent, with no cycles. Separately, there is an input stream of finalization updates that effectively advances the “finalized height” monotonically over time. It does this by writing seq, node_id pairs. The application guarantees that seq is a monotonically increasing integer, and that the node_id is always an ancestor of the previously finalized ID. Once the system has finalized height Hf, my DBSP should never again need payload for any node with height < Hf.

The rough shape of the SQL is something like this:

-- Tree metadata (small; I'm ok keeping this indefinitely)
CREATE TABLE nodes (
  node_id   BIGINT NOT NULL,
  parent_id BIGINT,
  height    BIGINT NOT NULL
) WITH ('append_only' = 'true');

-- Big payload table keyed by height/time, needs GC
CREATE TABLE node_payload (
  height  BIGINT NOT NULL LATENESS 0,
  node_id BIGINT NOT NULL,
  payload VARCHAR NOT NULL
) WITH ('append_only' = 'true');

-- Finalization stream -- seq is monotonic, finalized_id is always an ancestor of pervious
CREATE TABLE finalize_updates (
  seq          BIGINT NOT NULL LATENESS 0,
  finalized_id BIGINT NOT NULL
) WITH ('append_only' = 'true');

-- Latest finalized pointer (singleton)
CREATE VIEW finalized AS
SELECT ARG_MAX(finalized_id, seq) AS finalized_id,
       MAX(seq)                   AS seq
FROM finalize_updates;

-- Recursive view: ancestors of the current finalized node.
DECLARE RECURSIVE VIEW finalized_ancestors(node_id BIGINT);

CREATE LOCAL VIEW fa_base AS
SELECT finalized_id AS node_id
FROM finalized;

CREATE LOCAL VIEW fa_step AS
SELECT n.parent_id AS node_id
FROM finalized_ancestors a
JOIN nodes n ON n.node_id = a.node_id
WHERE n.parent_id IS NOT NULL;

CREATE VIEW finalized_ancestors AS
SELECT * FROM fa_base
UNION
SELECT * FROM fa_step;

-- Scalar finalized height, derived through recursion.
-- (MAX(height) over ancestors is the finalized node's height in a tree.)
CREATE VIEW finalized_height AS
SELECT MAX(n.height) AS finalized_height
FROM finalized_ancestors a
JOIN nodes n ON n.node_id = a.node_id;

-- Payload retention is STILL purely height-based.
CREATE VIEW finalized_payloads AS
SELECT p.*
FROM node_payload p
CROSS JOIN finalized_height f
WHERE p.height <= f.finalized_height;

LATENESS finalized_payloads.height 0; -- this is what I really want to work

Since Feldera allows LATENESS annotations on views, my original assumption was that the above would work and allow the circuit to discard the finalized_payloads to an external sink. However, after digging in, my understanding now is that compiler validation disables GC for all views which have any dependency on a recursive view whatsoever.

My question is: is that an overly conservative decision on the compiler's part? If I try to remove the compiler GC validation or insert integrate_trace_retain_keys into the circuit myself--not within the recursive view, but on the downstream JOIN--is that theoretically safe given my application's assumptions? Or am I missing something more fundamental about how recursion works?

Answered by mihaibudiu

Jan 22, 2026

I ran the compiler on this program and it did produce a GC operator for the join with node_payload.
The finalized_payloads view is not an input to any other computation, so the lateness does not help anything - lateness acts downstream of a view.

View full answer

mihaibudiu · 2026-01-22T18:31:14Z

mihaibudiu
Jan 22, 2026
Maintainer

I ran the compiler on this program and it did produce a GC operator for the join with node_payload.
The finalized_payloads view is not an input to any other computation, so the lateness does not help anything - lateness acts downstream of a view.

0 replies

mihaibudiu · 2026-01-22T18:31:44Z

mihaibudiu
Jan 22, 2026
Maintainer

0 replies

mihaibudiu · 2026-01-22T18:35:08Z

mihaibudiu
Jan 22, 2026
Maintainer

Note that GC only works for incremental circuits, you need to supply the -i flag to the compiler.
The algorithm will not detect GC opportunities inside recursive circuits, but otherwise does not do anything special for recursive circuits.

0 replies

bens-secret-alt-account · 2026-01-22T21:42:49Z

bens-secret-alt-account
Jan 22, 2026
Author

Derp -- I tried to make a simpler example for readability but didn't make sure my issue repro'd 🤦 I must have something else going wrong in my actual example that I misconstrued was coming from the JOIN. That's awesome news for me, thank you so much!

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

LATENESS/GC downstream of recursive views #5492

Uh oh!

{{title}}

Uh oh!

Replies: 4 comments

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

LATENESS/GC downstream of recursive views #5492

Uh oh!

bens-secret-alt-account Jan 22, 2026

Replies: 4 comments

Uh oh!

mihaibudiu Jan 22, 2026 Maintainer

Uh oh!

mihaibudiu Jan 22, 2026 Maintainer

Uh oh!

mihaibudiu Jan 22, 2026 Maintainer

Uh oh!

bens-secret-alt-account Jan 22, 2026 Author

bens-secret-alt-account
Jan 22, 2026

mihaibudiu
Jan 22, 2026
Maintainer

mihaibudiu
Jan 22, 2026
Maintainer

mihaibudiu
Jan 22, 2026
Maintainer

bens-secret-alt-account
Jan 22, 2026
Author