[go: up one dir, main page]
Scribd
Upload
22 views10 pages

Message Passing - Blocking & Non-Blocking API

Uploaded by

iitjee.prime
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
22 views10 pages

Message Passing - Blocking & Non-Blocking API

Uploaded by

iitjee.prime
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 10

Asynchronous / Synchronous API

Request Handling Workflow


Accept the incoming request –
For a new HTTP connection, an underlying TCP connection
must be established first.
Read the request
• Reading the raw bytes from the socket (I/O-bound) &
• Parsing the actual HTTP request (CPU-bound)
• If request contains entity such as POST parameters or a
file upload, this additional content must be read as well.
Depending on the implementation, the web server either
• Buffers the entity until loaded completely - content
offloading and is important for slow connections
• Pipes it directly to the application server - decreases
latencies.
Dispatch the request to the Application Level
• The parsed request is issued to the application server.
• We use decoupled components, generally a network-
based task, using Messaging (or alternatives - RPC).
• Write the generated response to the socket
• Once response is generated (generated HTML file from
the App Server or a static image from the file system), it
can be returned to the client by writing to the socket.
Web server can either
• Buffer the response and thus provide offloading for the
application servers or
• Pipe the generated response directly to the client.
Finish the request - Depending on the connection state
negotiated and the HTTP defaults, the web server either:
• Closes the connection, or
• It starts from the beginning and awaits the next request
to be sent from the client.

Server's Performance Concerning Web Infrastructures


• Request throughput (#/sec)
• Raw data throughput (Mbps)
• Response times (ms)
• Number of concurrent connections (#)
Performance Statistics to be observed locally on the Server:
• CPU utilization
• Memory usage
• Number of open socket / file handles
• Number of threads / processes
Aim
• To handle as many requests:
• In parallel as possible,
• As fast as possible &
• With as few resources as necessary.
How does Non-Blocking IO Work
• Applications may have to handle 10000s of req./sec
• Peak loads can come up to 20K requests per second.
• Thread-per-connection models do not suffice.
How does the caller of a non-blocking / asynchronous API
gets notified when data is ready?
• Caller must be notified that the data is ready.
• The only “listener” that is natively available in most
computers is the hardware interrupt processor.
• Hardware interrupts do not scale well and lack
flexibility.
• Constant checking, like an infinite loop that polls to see
if data is ready.
• An infinite loop must be taking quite some CPU time.
Blocking IO
• Calling an API that requests data from IO will cause the
running thread to “block”,
• When a thread is blocked in Linux, it goes in a Sleep
state by the kernel until data has returned to the caller.
Why non-blocking IO?
• Fewer threads to handle same amount of IO requests.
• A thread costs around 1MB, and there are some costs
due to context switching.
Types of blocking
• CPU-bound blocking
• IO-bound blocking
CPU-bound blocking
Some CPU intensive task it performs takes more time than
“instantly”.

CPU-bound blocking

IO-bound blocking
Wait for data to return from an IO source, such as a network
or a hard drive.

IO-bound blocking
Non-blocking IO
• When data has returned from IO, the caller will be
notified
• Done with a callback function that has access to the
returned data.
Callback
Network IO and Sockets
• At kernel level a socket is used as an abstraction to
communicate with a NIC.
• Socket takes care of reading and writing data to / from
NIC, NIC sends data over the UTP cable to the internet.
• For example, if you go to a URL in your browser:
• At low level the data in your HTTP request is written
to a socket using the send(2) system call.
• When a response is returned, response data can be
read from that socket using the recv(2) system call.
• So when data has returned from network IO, it is ready
to be read from the socket.
Non-Blocking IO under the hood
• Use an infinite loop that constantly checks (polls) if data
is returned from IO called Event Loop
• It checks if data is ready to read from a network socket.
• Sockets are implemented as file descriptors (FD) on
UNIX systems.
• All sockets are file descriptors but converse is not true
• So technically, FD is checked for ready data.
• The list of FDs that you want to check for ready data is
generally called the Interest List.
Optimizations to Event Loop
Each (major) OS provides kernel level APIs to help create
an event loop
• Linux - epoll or io_uring,
• BSD - kqueue &
• Windows - IOCP.
Each of these APIs is able to check FDs for ready data with
a computational complexity of around
O(No_of_Events_Occurred).
In other words, you can monitor 100,000s of FDs, but the
API’s execution speed only depends on the amount of
events that occur in the current iteration of the event loop.
Conclusion
Applications that need to handle high event rates mostly use
non-blocking IO models implemented with event loops.
For best performance, the event loop is built using kernel
APIs such as kqueue, io_uring, epoll and IOCP.
Hardware interrupts and Signals are less suited for non-
blocking when handling large amounts of events per second.
Server Architectures
Two competitive server architectures based on:

• Threads or
• Events.
Internals of an Event-Driven Architecture

• A single-threaded event loop consumes event after


event from the queue and sequentially executes
associated event handler code.
• New events are emitted by external sources such as
socket or file I/O notifications.
• Event handlers trigger I/O actions that eventually
result in new events later.
• Processing an event either requires:
• Registered event handler code for specific events, or
• Based on the execution of a callback associated to
the event in advance.
• Instead of sequential operations, an event-driven
program uses a cascade of asynchronous calls and
callbacks that get executed on events.
• This notion often makes the flow of control less
obvious and complicates debugging.
• The usage of event-driven server architectures has
historically depended on the availability of:
• Asynch. / non-blocking I/O operations on OS level &
• Suitable high performance event notification
interfaces such as epoll and kqueue.

You might also like