[go: up one dir, main page]
Scribd
Upload
5 views16 pages

Why Postgres

The document discusses the challenges of storing and retrieving data for monitoring traffic using OTNEngine, emphasizing the need for real-time analytics and efficient data management. It highlights the limitations of MySQL and LFS while advocating for Postgres due to its fault tolerance and dynamic schema capabilities. The main flow of data processing is described, showcasing the performance of a system capable of handling large volumes of frames efficiently.

Uploaded by

amr.shams2015.as
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
5 views16 pages

Why Postgres

The document discusses the challenges of storing and retrieving data for monitoring traffic using OTNEngine, emphasizing the need for real-time analytics and efficient data management. It highlights the limitations of MySQL and LFS while advocating for Postgres due to its fault tolerance and dynamic schema capabilities. The main flow of data processing is described, showcasing the performance of a system capable of handling large volumes of frames efficiently.

Uploaded by

amr.shams2015.as
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 16

First, We need a

problem to Solve.
This is US
And we had the chance to be here
and to impalement OTNEngine
And to display the analytics and
data has some challenges

Need to find a way to store and retrieve the data


- Would be able to run for long time (as long as the
session)
- Can write real time traffic
- Can read huge traffic for analytics and
- Has a catchy name
Our user want to monitor traffic and
see the progress
So how to monitor data and see the
history of records
Focal Points for frames Monitoring
• Number of frames generated
• What are the headers most important value
• Did it fit the allocated space
• How long did the graph stall
• Did the user get board?
• How do these numbers compare to what is
expected
• How do these number up to the user expectation
This creates a lot of data
Not all data are important for each user
Sampling frames generated in one hour (approx): 360K frame
Samples from frames are trimmed to the header only so the size reduces by 63
Those headers will be aggregated with some meta data like page number
After aggreges we are left with 360k header per hour
Each row has 256 measures of viewability metrics
Real-time accumulation and
aggregation
Consume header of frames as fast as possible
Serving API calls continuously
Single machine with
- 6 physical cores
- Low latency
- Overprovisioned
What didn’t work: MySQL
• Original DB choice
• Performed very well, when daily volume was < 1% of
current volume
• Impossible to add new columns to database
What didn’t work LFS
• Very fast compared to any db.
• Not Optimized for Random Access
• Needed local disk with very fast I/O
What did work: Postgres
• Fault tolerance against interrupts
• Can have a dynamic schema (json data-
type)
• Support multiple connection
Main Flow
DB fast?
• No loops needed (used array to insert records)
• Connection pool (for parallel writes to db)
• Non-blocking code (used async + threads)
• Prepared statement (fancy words)
• In an 2.9 GHz 6-Core Intel Core i9
• Write 10k frames in 40 ms
• Read 10k frames in 50 ms
Saving Frames Payload
What didn’t work: Compressing
Payload
• Our Data is pRPS
• Reduce in both read and write
• From 40 ms to 250 to the same size
• With this rate it is estimated to have
200GB in 24

You might also like