Replies: 6 comments 13 replies
-
|
We have to clarify the terminology. Feldera can do view maintenance with any granularity. There is no gap between streaming, micro batch, and batch, they all work in exactly the same way. You decide when to feed inputs (accumulating batches) and when you want to see the output (letting feldera acumulate batches). Feldera does offer no particular benefits for ad-hoc queries which are run only once. There is nothing special you need to do to implement the diagram you have described. Will look at the paper you describe for details. |
Beta Was this translation helpful? Give feedback.
-
|
In fact, feldera is even more efficient with batched inputs. |
Beta Was this translation helpful? Give feedback.
-
|
I agree. And I think we should minimize the additional storage cost of z-set, as a base table could be consumed over 100 times in production batch processing. |
Beta Was this translation helpful? Give feedback.
-
|
The same delta join algorithm is used by Feldera, but I wouldn't call it "stateless". It has to maintain a copy of all input relations of the join. Materialize calls these "arrangements." The delta join algorithm has been known for many decades and is essentially the optimal way to perform incremental joins. |
Beta Was this translation helpful? Give feedback.
-
Sorry, not sure I understand. Feldera shouldn't need to consume the full table each time, only ingesting changes after initial full load. And in any case, storage cost should not depend on how frequently the data is loaded, it should only be a function of the current size of the table. |
Beta Was this translation helpful? Give feedback.
-
For a streaming database, it will always store the input table, and the delta-join operation would not introduce another inner state. That's why we can say it appears as |
Beta Was this translation helpful? Give feedback.
-
|
Certainly, all the internal state maintained by Feldera is in addition to the data in the database. That state can exceed the size of the database. This is inherent in IVM, which is a trade-off between time and space. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @Myasuka! Let me know if I understand your idea correctly:
Is this what you had in mind? |
Beta Was this translation helpful? Give feedback.
-
|
@ryzhyk Thanks for your response. |
Beta Was this translation helpful? Give feedback.
-
|
This makes sense. As @mihaibudiu pointed out, IVM may require maintaining a lot of intermediate state, in addition to the input tables; however there is no fundamental reason this state cannot live in disaggregated storage. We are starting to implement support for DeltaLake, initially in the form of input and output adapters, but potentially also as a third-tier data store in the future. |
Beta Was this translation helpful? Give feedback.
-
|
My next blog post, which will be published this week, touches on the topic you mention. It will describe how the work performed by a DBSP circuit can be decoupled from the input and output deltas using buffers. But I don't talk about optimizations, only about the actual buffering mechanism. Thank you for the pointer to the paper on "incrementability". If I read it right, translating the proposal of the paper in the language of DBSP would simply involve breaking a DBSP circuit with a single "logical clock" into multiple circuits separated by buffers, using independent clocks, under the assumption that some parts of the plan can benefit from larger batches. The clock of the output buffer can be driven by the users, i.e., the equivalent of the triggers in the paper. The paper only uses the metric of work, but does not discuss about idle time. If the system has nothing else to do, perhaps processing smaller batches is a good idea (if you don't pay by CPU usage). The more general problem requires an on-line algorithm - since you don't necessarily know when the user wants to inspect the data. For choosing a good size for the input batches for one circuit, I suspect a simple control mechanism could work: try different sizes, and see which one works best; it can probably use a simple regression model. It would be a fun experiment to reproduce the results in the paper using our system, and I suspect it wouldn't entail too much work. Could be a good intern/research project. So far we have done very little with regards to tuning. |
Beta Was this translation helpful? Give feedback.
-
|
Compared with the simple "incrementability" model paper, I think another Tempura paper would be more complex and general. As far as I know, the Tempura paper is one of the basic work of enterprise Incremental processing engine. |
Beta Was this translation helpful? Give feedback.
-
|
I read the Tempura paper, very interesting work. I also like the name of the system. Most of the Tempura paper seems to address the offline version of the problem, where the "future" deltas are know by the optimizer, but the paper does talk about how to adapt this in an online setting.
Let me point out several things contrasting DBSP to Tempura:
I hope this makes sense, and that I interpreted correctly the Tempura paper. Thank you for sharing it! |
Beta Was this translation helpful? Give feedback.
-
|
Sorry to join the discussion late; I'll share some info from my side(use cases, problems, and potential solutions -- some use cases have been simplified so that I can discuss them in public). 
The picture shows three cases, I'll describe them separately later. The first caseThe first case is the common warehouse scenario. The user will ingest data into Hive/DataLake Table, and then use Spark to process it. Problem:
User's Desire:
Potential Solution:
The Second Case:The second case is generally used in recommendation scenarios with high real-time requirements. It can be considered the real-time pipeline of the first case. However, there are also some problems: Issues:
User's Desire:
Possible Solutions:None for now. If possible, the user hopes to process the first and second cases in a single pipeline. The Third Case:The third use case is generally an online service that consumes upstream data, joins external dimension tables, and sends the processed results downstream. Issues:
User's Desire:
Possible Solutions:
Other QuestionsStreaming-Streaming join with retractionIIUC the delta join is more efficient in batch mode, but has a litter effect in streaming-streaming join, is this understanding right? DBSP and TempuraRegarding the mentioned Tempura, my understanding is that DBSP can cover the part that Tempura does. I'll briefly explain my understanding:
|
Beta Was this translation helpful? Give feedback.
-
|
This is a long message. I will reply in bits. DBSP is a language to describe streaming systems, which can be implemented relatively easily (there are only 4 kinds of operators), maybe 5 if you are picky. Our paper also gives an algorithm to convert a query Q (a standard point-in-time query written in SQL or Datalog) into a DBSP circuit which maintains the results of the query incrementally. The algorithm input is a plan for Q, the output is a circuit in DBSP implementing (\uparrow Q)^\Delta (the incremental lifted query). For each plan of Q you will get a different DBSP circuit, but if the input plans are equivalent, the DBSP circuits will also be 100% equivalent, providing the same exact input-output behavior. But DBSP as a language is more powerful, it can express queries that are not just of the form (\uparrow Q)^\Delta. The simplest example is a program consisting of just a delay. There is no query Q such that (\uparrow Q)^\Delta = z^{-1}. A DBSP program is supposed to run forever: you build it and you send data through it; it will produce results, and update its internal state on every input. The DBSP theory doesn't give you any solution for converting a DBSP program and its internal state into another DBSP program that is equivalent, such that you can substitute the new one for the original one from some point onwards. Neither does our implementation. If you want to switch to a different DBSP program that computes the same function, you can build the new DBSP circuit and you have to feed again all the inputs you have ever fed the original circuit. Then the new circuit can take over from the original circuit. However, if your DBSP program implements a lifted incremental query Q, then you can do better: it is enough to feed to the new circuit only the integrals of the input tables computed so far. My understanding is that this is what Tempura does. We can do this too, but you must save these integrals somewhere when you switch circuits. And this may be very hard for applications where these integrals would grow forever (like some streaming applications). For some special DBSP programs (the ones without any delay operators inside) you can just switch the implementation at any time; such programs do not maintain any internal state, so the replacement can take over at any time. Such programs may be produced from queries Q that only contain linear operators. These are all worst-case options. You can do better for some programs. For the example of a program that is equivalent to just one delay, it's enough to feed the previous input, and then the replacement can can continue from that point on. |
Beta Was this translation helpful? Give feedback.
-
|
IIUC, DBSP should be able to do the above case-1, and case-2(use the final MV as the final result), and this way of IVM seems to be relatively better in terms of maintainability as well as performance, case-3 is also possible, but need to have a suitable storage system. Thank you a lot for this wonderful work, I'm also trying to use DBSP to solve some of our problems(now is some poc demo), but this can only happen in my part-time, so the progress may be slow. |
Beta Was this translation helpful? Give feedback.
-
|
(1) Case 3 above is very interesting. What you need is a wrapper that exposes an external indexed collection as a DBSP Z-set. It should clearly be doable. An important question is whether the external data set changes, and when it changes how DBSP finds out what the changes are. (2) Regarding blog posts, so far I haven't even started talking about DBSP, only about the preliminaries, I still have quite a few to write. I try to make them each bite-size so they are easy to digest. (3) Regarding your use of DBSP for solving your problems, I am sure you will encounter problems - we do all the time. Please file issues for any problem found, including documentation and usability problems, and we'll be happy to fix them. Hopefully we can iron out the kinks and make this into an easy to use tool. The Rust DBSP layer wasn't designed necessarily for human consumption (it is the target of our compiler), so some design decisions may seem strange. |
Beta Was this translation helpful? Give feedback.
-
|
For use case-3, it is something like real-time fraud detection. The dimension data in HBase may change over time. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for sharing the use cases, these are very interesting. We should also make a distinction between the DBSP theory and the DBSP Rust library. The theory does not care about stuff like scale out or fault-tolerance, whereas the implementation clearly does. The Rust library is also continuously evolving, so we have to distinguish between what it does today from what it can do in the future. And the SQL compiler is also evolving. I am saying this because whether DBSP Rust/Feldera can implement these use cases also may depend on the nature of the data (size, frequency, etc), the nature of the computation (Spark and Flink can each express algorithms the other one cannot) and the maturity of our implementation. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Currently, Feldera focus on the streaming area to act as a continuous query engine. However, for the BI area, people don't rely on the micro-second-level or even minute-level latency to make a decision, that's why T+1 batch style ETL is more suitable and have a much larger market. If we can make the batch processing to generate fresher output without high cost, I think that could make a real difference to the world.
And for the cost analysis of query execution, we can use the model in the SIGMOD '20 paper Thrifty Query Execution via Incrementability. Long running streaming engine have to pay high cost for retraction operations.
Since incremental materialized view (IVM) is very suitable for batch processing, and the DBSP already support to make any incremental materialized view possible. From my point of view, it's a great chance to make the change.
If we refer to the product of snowflake's dynamic tables, I think we can change the architecture to the following:
Beta Was this translation helpful? Give feedback.
All reactions