Real-time testing

Real-time testing is the process of testing real-time computer systems.

Software testing is performed to detect and help correct bugs (errors) in computer software. Testing involves ensuring not only that the software is error-free but that it provides the required functionality to the user. Static and conventional methods of testing can detect bugs, but such techniques may not ensure correct results in real time software systems.
Real-time software systems have strict timing constraints and have a deterministic behavior. These systems have to schedule their tasks such that the timing constraints imposed on them are met. Conventional static way of analysis is not adequate to deal with such timing constraints, hence additional real-time testing is important.^[1]

Strategy

Test case design for real time testing can be proposed in four steps^[2]

Task testing

In the very first step, each task is tested individually with conventional static testing. This testing is performed only to discover the errors in logic or syntax of the program. Order of the events doesn't matter as task testing doesn't deal with timing constraints and time properties of events.

Behavioral testing

Using the system models designed with the help of automated testing tools, it is possible to simulate behavior of real time system and impact of concurrent external events on its behavior.

Intertask testing

Once the testing with the individual task is done, then task is supposed to be error free in coding and behavioral area. Time-related constraints are tested with intertask testing. To reveal the errors in communication, asynchronous tasks are tested with variable data rates and different payloads.

System testing

In this testing, software and hardware are integrated and full range of system tests are conducted to discover errors, if any, during software and hardware interfacing.

Tools for real time testing

As testing of real time systems is becoming more important, there are some tools designed for such testing.

MSC

Message Sequence Charts is an internationally accepted standard for capturing requirements.^[3] MSC provides a graphical 2-D language often required for collecting requirements through some interaction scenarios.

SDL

Specification and Description Language is a standard used for design and analysis. SDL^[4] supports the specification of complex software systems and has been extensively applied across a broad array of domains from telecommunications, automation, through to general software development

TTCN

Testing and Test Control Notation is the only internationally standard testing language. TTCN3^[5] provides a broader applicability, as compared to earlier versions of TTCN, which were primarily focused on OSI protocols only.

These three standards together are used for testing of real time applications. It is necessary that requirements be satisfied with these models and test cases generated must capture the functional and real time information needed to test systems. Also, the changes in the requirements of design and new information about the real time properties of systems should be fed into models so that its impact can be found out.

To accurately capture the real time properties of a given test system and to ensure that requirements and models are used to generate realistic and enforceable timing information, it is essential that the language itself (TTCN-3) has a well understood and semantically sound model of time.

TTCN-3

Background

TTCN-3 is the only currently available, internationally standardized testing language. Prior to TTCN3, its earlier versions were having limited functionality and limited scope over OSI protocol. But, TTCN3 is an advanced version and has broader applicability.
Characteristics of TTCN3 are:

ability to specify dynamic concurrent testing
operations for message-based and procedure based communication
the ability to specify data and signature templates with powerful matching mechanisms
type and value parameterization
the assignment and handling of test verdicts
test suite parameterization and test case selection mechanisms

The reason for using TTCN3 for real time testing is because of its timers. These timers are defined in function test suites. There are no any global kind timers used in TTCN3. These timers can be started, stopped and checked using simple functions like timer.start, timer.stop, and timer.read.

Snapshot Semantics is a technique in TTCN3 (also in TTCN2), which deals with the message passed during communication by system to system or implementation under test. When a series of responses are received by system under test, then snapshot is taken and they are evaluated in order of their arrival. So, each time around a set of attributes, a snapshot is taken and only those events are evaluated which are present in snapshot.

But this technique is not efficient as some events and their attribute information might get lost while the snapshot is taken. Some events might get recorded on processing queue, but not on snapshot. Such events can never get processed. Also, if the test executer equipment is not fast enough, then it can not communicate properly with the system under test. So, faults might get generated during such test evaluation.

Linux tools^[6]^[7]

rt-tests: https://git.kernel.org/pub/scm/utils/rt-tests/rt-tests.git/
cycletest by SUSE: https://documentation.suse.com/sle-rt/15-SP1/html/SLE-RT-all/art-slert-hwtest.html

References

^ Tsai, J.J.P.; Fang, K.-Y.; Bi, Y.-D. (1990). "On real-time software testing and debugging". Proceedings., Fourteenth Annual International Computer Software and Applications Conference. pp. 512–518. doi:10.1109/CMPSAC.1990.139423. ISBN 978-0-8186-2054-6.
^ Pressman, Roger S. (2005). Software engineering : a practitioner's approach (6th ed.). Boston, Mass.: McGraw-Hill. ISBN 0-07-285318-2. OCLC 53848343.
^ TTCN-3 Test Case Generation from Message Sequence Charts (PDF).
^ "SDL Forum Society - Home page". sdl-forum.org. Retrieved 2021-03-08.
^ "Home". TTCN-3. Retrieved 2021-03-08.
^ Dahlke, Patrick (2018-02-06). "Realtime Linux". Medium. Retrieved 2021-03-10.
^ "Realtime Testing Best Practices - eLinux.org". elinux.org. Retrieved 2021-03-10.

[real_time_testing_&_debugging_IEEE_paper-1] Tsai, J.J.P.; Fang, K.-Y.; Bi, Y.-D. (1990). "On real-time software testing and debugging". Proceedings., Fourteenth Annual International Computer Software and Applications Conference. pp. 512–518. doi:10.1109/CMPSAC.1990.139423. ISBN 978-0-8186-2054-6.

[2] Pressman, Roger S. (2005). Software engineering : a practitioner's approach (6th ed.). Boston, Mass.: McGraw-Hill. ISBN 0-07-285318-2. OCLC 53848343.

[MSC-3] TTCN-3 Test Case Generation from Message Sequence Charts (PDF).

[4] "SDL Forum Society - Home page". sdl-forum.org. Retrieved 2021-03-08.

[5] "Home". TTCN-3. Retrieved 2021-03-08.

[6] Dahlke, Patrick (2018-02-06). "Realtime Linux". Medium. Retrieved 2021-03-10.

[7] "Realtime Testing Best Practices - eLinux.org". elinux.org. Retrieved 2021-03-10.

[1]

[2]

[3]

[4]

[5]

[6]

[7]