SoC Verification Methodology

Prof. Chien-Nan Liu TEL: 03-4227151 ext:4534 Email: jimmy@ee.ncu.edu.tw

Outline
Verification Overview Verification Strategies Tools for Verification SoC Verification Flow

l l l l

What is Verification ?
l

A process used to demonstrate the functional correctness of a design To making sure your are verifying that you are indeed implementing what you want To ensure that the result of some transformation is as expected
Transformation

Specification

Verification

Netlist
3

Source : Writing Test Benches Functional Verification of HDL Models by Janick Bergeron, KAP, 2000.

Verification Problems
l l l l

Was the spec correct ? Did the design team understand the spec? Was the blocks implemented correctly? Were the interfaces between the blocks correct? Does it implement the desired functionality?
4

Testing v.s. Verification

l

Testing verifies manufacturing

Verify that the design was manufactured correctly

HW Design

Manufacturing

Specification

Verification

Netlist

Testing

Silicon

Source : Writing Test Benches Functional Verification of HDL Models by Janick Bergeron, KAP, 2000.

SoC Design Verification

l

Using pre-defined and pre-verified building block can effectively reduce the productivity gap
Block (IP) based design approach Platform based design approach

But 60 % to 80 % of design effort is now dedicated to verification

An Industrial Example
System Simulation ASIC Testbenches Extended Simulation Equivalence Checking Emulation Support Emulation Software Verification Design BEH Model Bottleneck !! High Level Design

Timing Analysis DFT Synthesis

RTL and Block Test

Source : Functional Verification on Large ASICs by Adrian Evans, etc., 35th DAC, June 1998.

Verification Complexity
l

For a single flip-flop:

Number of states = 2 Number of test patterns required = 4

For a Z80 microprocessor (~5K gates)

Has 208 register bits and 13 primary inputs Possible state transitions = 2bits+inputs = 2221 At 1M IPS would take 1053 years to simulate all

transitions
l

For a chip with 20M gates

??????

*IPS = Instruction Per Second

When is Verification Complete ?

l

Some answers from real designers:

When we run out of time or money When we need to ship the product When we have exercised each line of the HDL code When we have tested for a week and not found a

new bug We have no idea!!

l

Designs are often too complex to ensure full functional coverage

The number of possible vectors greatly exceeds the

time available for test

9

Typical Verification Experience

B u g s p e r w e e k Functional testing Purgatory Tapeout

Weeks

10

Error-Free Design ?
l

As the number of errors left to be found decreases, the time and cost to identify them increases Verification can only show the presence of errors, not their absence Two important questions to be solved:
How much is enough? When will it be done?
11

Reference Book
l

System-on-a-Chip Verification Methodology and Techniques

by Prakash Rashinkar Peter Paterson Leena Singh Cadence Design Systems Inc., USA published by Kluwer Academic Publishers, 2001

12

Outline
Verification Overview Verification Strategies Tools for Verification SoC Verification Flow

l l l l

13

Verification Approaches
l

Top-down verification approach

From system to individual components

Bottom-up verification approach

From individual components to system

Platform-based verification approach

Verify the developed IPs in an existing platform

System interface-based verification approach

Model each block at the interface level Suitable for final integration verification

14

Advs. of Bottom-up Approach

l l

Locality Catching bugs is easier and faster with foundational IPs (sub-blocks) Design the SoC chip with these highly confidence bug-free IPs

15

Verification Environment

Testbench Design under Verification DUV

Input Pattern (Stimulus)

Output (Response)

16

Terminology
l

Verification environment
Commonly referred as testbench (environment)

Definition of a testbench
A verification environment containing a set of

components [such as bus functional models (BFMs), bus monitors, memory modules] and the interconnect of such components with the designunder-verification (DUV)
l

Verification (test) suites (stimuli, patterns, vectors)

Test signals and the expected response under given

testbenches
17

Testbench Design
l l

Auto or semi-auto stimulus generator is preferred Automatic response checking is highly recommended May be designed with the following techniques
Testbench in HDL Tesebench in programming language interface (PLI) Waveform-based Transaction-based Specification-based

18

Types of Verification Tests (1/2)

l

Random testing
Try to create scenarios that engineers do

not anticipate
l

Functional testing
User-provided functional patterns

l l l

Compliances testing Corner case testing Real code testing (application SW)
Avoid misunderstanding the spec.
19

Types of Verification Tests (2/2)

l

Regression testing
Ensure that fixing a bug will not introduce

another bug(s) Regression test system should be automated

Add new tests Check results and generate report Distribute simulation over multiple computer

Time-consuming process when verification

suites become large

20

Bug Tracking
l

A central database collecting known bugs and fixes Avoid debugging the same bug multiple times Good bug report system helps knowledge accumulation

21

Bug Rate Tracking

l

Bug rate usually follow a well-defined curve The position on the curve decides the most-effective verification approach Help determine whether the SoC is ready to tape-out

22

Bug Rate Tracking Example

100 90 80 70 60 50 40 30 20 10 0 0 2 4 6 8 10 12

# of bugs reported

Time
23

Adversarial Testing
l

For original designers

Focus on proving the design works correctly

Separate verification team

Focus on trying to prove the design is broken Keep up with the latest tools and

methodologies
l

The balanced combination of these two gives the best results

24

Verification Plan
l l

Verification plan is a part of the design reports Contents

Test strategy for both blocks and top-level module Testbench components BFM, bus monitors, ... Required verification tools and flows Simulation environment including block diagram Key features needed to be verified in both levels Regression test environment and procedure Clear criteria to determine whether the verification

is successfully complete
25

Benefits of Verification Plan

l

Verification plan enables

Developing the testbench environment early Developing the test suites early Developing the verification environment in parallel

with the design task by a separate team

Focusing the verification effort to meet the

product shipment criteria

Forcing designers to think through the time-

consuming activities before performing them

26

Outline
Verification Overview Verification Strategies Tools for Verification SoC Verification Flow

l l l l

27

Tools for Verification (1/4)

l

Simulation
Event-driven: Timing accurate Cycle-based: Faster simulation time (5x 100x) Transaction-based: Require bus functional model (BFM) of each design Only check the transactions between components Have faster speed by raising the level of abstraction
28

Event-Driven Simulation
l l l

Timing accurate Good debugging environment Simulation speed is slower

Schedule New Events

Advance Time

More events for this time N N Time wheel empty

Get Current Event

Event Evaluation

Propagate Events

Finish
29

Cycle-Based Simulation
l

Perform evaluations just before the clock edge

Repeatedly triggered events

Clock Input Output

are evaluated only once in a clock cycle

l l l l

Faster simulation time (5x 100x) Only works for synchronous designs Only cycle-accurate Require other tools (ex: STA) to check timing problems
30

Simulation-Based Verification
l

Still the primary approach for functional verification

In both gate-level and register-transfer level (RTL)

Test cases
User-provided (often) Randomly generated

Hard to gauge how well a design has been tested

Often results in a huge test bench to test large designs

Near-term improvements
Faster simulators

Compiled code, cycle-based, emulation, Testbench tools Make the generation of pseudo-random patterns better/easier
l

Incremental improvements wont be enough

31

Tools for Verification (2/4)

l

Code coverage
Qualify a particular test suite when applied to a

specific design Verification Navigator, CoverMeter,

l

Testbench (TB) automation

A platform (language) providing powerful constructs

for generating stimulus and checking response VERA, Specman Elite,

l

Analog/mixed-signal (AMS) simulation

Build behavior model of analog circuits for system

simulation Verilog-A, VHDL-A,

32

Coverage-Driven Verification
l

Coverage reports can indicate how much of the design has been exercised
Point out what areas need additional verification

Optimize regression suite runs

Redundancy removal (to minimize the test suites) Minimizes the use of simulation resources

Quantitative sign-off (the end of verification process) criterion Verify more but simulate less
33

The Rate of Bug Detection

Functional Testing purgatory release

Rate of dugs found

with coverage without coverage

Time
source : Verification Methodology Manual For Code Coverage In HDL Designs by Dempster and Stuart 34

Coverage Analysis
l l

Dedicated tools are required besides the simulator Several commercial tools for measuring Verilog and VHDL code coverage are available VCS (Synopsys) NC-Sim (Cadence) Verification navigator (TransEDA) Basic idea is to monitor the actions during simulation Require supports from the simulator PLI (programming language interface) VCD (value change dump) files
35

l l

Analysis Results
n

Verification Navigator (TransEDA)

Untested code line will be highlighted

36

Testbench Automation
l

Require both generator and predictor in an integrated environment Generator: constrained random patterns
Ex: keep A in [10 100]; keep A + B == 120; Pure random data is useless Variations can be directed by weighting options Ex: 60% fetch, 30% data read, 10% write

Predictor: generate the estimated outputs

Require a behavioral model of the system Not designed by same designers to avoid

containing the same errors

37

Analog Behavioral Modeling

l

A mathematical model written in Hardware Description Language Emulate circuit block functionality by sensing and responding to circuit conditions Available Analog/Mixed-Signal HDL:
Verilog-A VHDL-A Verilog-AMS VHDL-AMS

38

Mixed Signal Simulation

l

Available simulators:
Cadence Antrim Mentor Synopsys

39

Tools for Verification (3/4)

l

Static technologies
Lint checking:

A static check of the design code Identify simple errors in the early design cycle Static timing analysis: Check timing-related problems without input patterns Faster and more complete if applicable Formal verification: Check functionality only Theoretically promise 100% coverage but design size is often limited due to high resource requirement
40

HDL Linter
l

Fast static RTL code checker

Preprocessor of the synthesizer RTL purification (RTL DRC) Syntax, semantics, simulation Check for built-in or user-specified rules Testability checks Reusability checks Shorten design cycle Avoid error code that increases design iterations
41

Inspection
l

For designers, finding bugs by careful inspection is often faster then that by simulation Inspection process
Design (specification, architecture) review Code (implementation) review
Line-by-line fashion At the sub-block level

Lint-liked tools can help spot defects without simulation

Nova ExploreRTL, VN-Check, ProVerilog,
42

What is STA ?
l l

STA = static timing analysis STA is a method for determining if a circuit meets timing constraints without having to simulate No input patterns are required
100% coverage if applicable

FF1 CLK

FF2

Reference :Synopsys

43

Formal Verification
l

Ensure the consistency with specification for all possible inputs (100% coverage) Primary applications
Equivalence Checking Model Checking

Solve the completeness problem in simulationbased methods Cannot handle large designs due to its high complexity Valuable, but not a general solution
44

Equivalence Checking
Synthesis RTL or Netlist

RTL or Netlist Equivalence Checking

l

Gate-level to gate-level
Ensure that some netlist post-processing did not change the

functionality of the circuit

l

RTL to gate-level
Verify that the netlist correctly implements the original RTL code

RTL to RTL
Verify that two RTL descriptions are logically identical
45

Equivalence Checking
l

Compare two descriptions to check their equivalence Gaining acceptance in practice

Abstract, Avant!, Cadence, Synopsys, Verplex,

Verysys,
l

But the hard bugs are usually in both descriptions Target implementation errors, not design errors
Similar to check C v.s. assembly language
46

Model Checking
RTL Coding Specification Interpretation Assertions
l

RTL Model Checking

Formally prove or disprove some assertions (properties) of the design The assertions of the design should be provided by users first
Described using a formal language
47

Model Checking
l

Enumerates all states in the STG of the design to check those assertions Gaining acceptance, but not widely used
Abstract, Chrysalis, IBM, Lucent, Verplex,

Verysys,
l

Barrier: low capacity (~100 register bits)

Require extraction (of FSM controllers) or

abstraction (of the design) Both tend to cause costly false errors
48

New Approaches
l

The two primary verification methods both have their drawbacks and limitations
Simulation: time consuming Formal verification: memory explosion

We need alternate approaches to fill the productivity gap

Verification bottleneck is getting worse

Semi-formal approaches may be the solution

Combine the two existing approaches Completeness (formal) + lower complexity (simulation)
49

Tools for Verification (4/4)

l

Hardware modeling
Pre-developed simulation models for other

hardware in a system Smart Model (Synopsys)

l

Emulation
Test the system in a hardware-liked fashion

Rapid prototyping
FPGA ASIC test chip
50

Assistant Hardware
l

Rule of thumb
10 cycles/s for software simulator 1K cycles/s for hardware accelerator 100K cycles/s for ASIC emulator

Hardware accelerator
To speed up logic simulation by mapping

gate level netlist into specific hardware Examples: IKOS, Axis, .

51

Emulation
l

Emulation
Verify designs using real hardware (like FPGA?) Better throughput in handling complex designs Inputs should be the gate-level netlist Synthesizable testbenches required Require expensive emulators Software-driven verification
Verify HW using SW Verify SW using HW

Interfaced with real HW components Examples: Aptix, Quicktum, Mentors AVS .

52

Prototyping
l

FPGA
Performance degradation Limited design capacity (utilization) Partitioning and routing problems

Emulation system
FPGA + Logic modeler Automatic partitioning and routing under

EDA SW control More expensive

53

Limited Production
l

l l

Even after robust verification process and prototyping, its still not guaranteed to be bug-free Engineering samples A limited production for new macro is necessary
1 to 4 customers Small volume Reducing the risk of supporting problems

Same as real cases but more expensive

54

Comparing Verification Options

Source : System-on-a-chip Verification Methodology and Techniques by P. Rashinkar, etc., KAP, 2001.

55

Outline
Verification Overview Verification Strategies Tools for Verification SoC Verification Flow

l l l l

56

SOC Verification Methodology

Source : System-on-a-chip Verification Methodology and Techniques by P. Rashinkar, etc., KAP, 2001.

57

System Verification Steps

l l l l

Verify the leaf IPs Verify the interface among IPs Run a set of complex applications Prototype the full chip and run the application software Decide when to release for mass production

58

Interesting Observations
l

90% of ASICs work at the first silicon but only 50% work in the target system
Do not perform system level verification (with

software)
l

If a SoC design consisting of 10 blocks

P(work)= .910 =.35

If a SoC design consisting of 2 new blocks and 8 pre-verified robust blocks

P(work) = .92 * .988 =.69

To achieve 90% of first-silicon success SoC

P(work) = .9910 =.90
59

Interface Verification
l

Inter-block interfaces
Point-to-point On-chip bus (OCB) Simplified models are used
BFM, bus monitors, bus checkers

60

Check System Functionality (1/2)

l

Verify the whole system by using full functional models

Test the system as it will be used in the

real world
l

Running real application codes (such as boot OS) for higher design confidence
RTL simulation is not fast enough to

execute real applications

61

Check System Functionality (2/2)

l

Solutions
Move to a higher level of abstraction for

system functional verification Formal verification Use assistant hardware:

Hardware accelerator ASIC emulator Rapid-prototyping(FPGA) Logic modeler
62

HW/SW Co-Simulation
l

Couple a software execution environment with a hardware simulator Simulate the system at higher levels
Software normally executed on an Instruction Set

Simulator (ISS) A Bus Interface Model (BIM) converts software operations into detailed pin operations
l l l

Allows two engineering groups to talk together Allows earlier integration Provide a significant performance improvement for system verification
Have gained more and more popularity
63

System-Level Testbench
l

All functionality stated in the spec. should be covered The success and failure conditions must be defined for each test Pay particular attention to:
Corner cases Boundary conditions Design discontinuities Error conditions Exception handling

Source : System-on-a-chip Verification Methodology and Techniques by P. Rashinkar, etc., KAP, 2001.

64

Timing Verification
l

STA (static timing analysis) is the fastest approach for synchronous designs
Avoid any timing exceptions is extremely important

Gate-level simulation (dynamic timing analysis)

Most useful in verifying timing of asynchronous

logic, multi-cycle paths and false paths

Much slower run-time performance Gate-level sign-off

65

Design Sign-off
l

Sign-off is the final step in the design process It determines whether the design is ready to be taped out for fabrication No corrections can be made after this step The design team needs to be confident that the design is 100% correct
Many items need to be checked

Source : System-on-a-chip Verification Methodology and Techniques by P. Rashinkar, etc., KAP, 2001.

66

Traditional Sign-off
l

Traditional sign-off simulation

Gate level simulation with precise timing under a

given parallel verification vectors Verify functionality and timing at the same time (dynamic timing analysis, DTA) Parallel verification vectors also serve as the manufacturing test patterns
l

Problems
Infeasible for million gates design (take too much

simulation time) Fault coverage is low (60% typically) Critical timing paths may not be exercised
67

SoC Sign-off Scheme

l

Formal verification used to verify functionality STA for timing verification Gate level simulation
Supplement for formal verification and STA

l l

Full scan BIST for manufacturing test

68