The HDF Group
Intro to HDF5
Katie Antypas
NERSC
Tutorial in part from HDF Group
NUG 2010
Berkeley, CA
Oct 18th, 2010
2010
NUG 2010
www.hdfgroup.org
Serial I/O
processors
Each processor sends its data to the
master who then writes the data to a file
Advantages ?
File
Disadvantages ?
Parallel I/O Multi-file
processors
File
File
File
File
File
File
Each processor writes its own data to a separate file
Advantages ?
Disadvantages ?
Parallel I/O Single-file
processors
File
Each processor writes its own data to the same file using
MPI-IO mapping
Advantages ?
Disadvantages ?
What is a High Level Parallel I/O
Library?
An API which helps to express scientific
simulation data in a more natural way
Multi-dimensional data, labels and tags, noncontiguous data, typed data
Typically sits on top of MPI-IO layer and
can use MPI-IO optimizations
Offer
Simplicity for visualization and analysis
Portable formats - can run on one machine and
take output to another
Longevity - output will last and be accessible
with library tools and no need to remember
version number of code
Common Storage Formats
ASCII:
Slow
Takes more space!
Inaccurate
Binary
Non-portable (eg. byte ordering and types sizes)
Not future proof
Parallel I/O using MPI-IO
Self-Describing formats
Many NERSC users
at this level. We
would like to
encourage users to
transition to a higher
IO library
NetCDF/HDF4, HDF5, Parallel NetCDF
Example in HDF5: API implements Object DB model in portable file
Parallel I/O using: pHDF5/pNetCDF (hides MPI-IO)
Community File Formats
FITS, HDF-EOS, SAF, PDB, Plot3D
Modern Implementations built on top of HDF, NetCDF, or other selfdescribing object-model API
But what about
performance?
Hand tuned I/O for a particular application and
architecture will likely perform better, but
Purpose of I/O libraries is not only portability, longevity,
simplicity, but productivity
Using own binary file format forces user to understand
layers below the application to get optimal IO
performance
Every time code is ported to a new machine or
underlying file system is changed or upgraded, user is
required to make changes to improve IO performance
Let other people do the work
HDF5 can be optimized for given platforms and file systems by
library developers
Goal is for shared file performance to be close enough
HDF5 File is a Container of Objects
HDF5 groups
and links
organize
data objects.
Experiment Notes:
Serial Number: 99378920
Date: 3/13/09
Configuration: Standard 3
Viz
SimOut
www.hdfgroup.org
HDF5 Dataset
Metadata
Data
Dataspace
3
Dim_1 = 4
Dim_2 = 5
Dim_3 = 7
Integer
Time = 32.4
Chunked
Pressure = 987
Compressed
Temp = 56
www.hdfgroup.org
HDF5 Dataset
3
Datatype:
Dataspace:
16-byte integer
Rank = 2
Dimensions = 5 x 3
10
www.hdfgroup.org
HDF5 Datatypes
The HDF5 datatype describes how to interpret
individual data elements.
HDF5 datatypes include:
integer, float, unsigned, bitfield,
user-definable (e.g., 13-bit integer)
variable length types (e.g., strings)
references to objects/dataset regions
enumerations - names mapped to integers
opaque
compound (similar to C structs)
11
www.hdfgroup.org
HDF5 Pre-defined Datatype Identifiers
HDF5 defines set of Datatype Identifiers per HDF5
session.
For example:
C Type
HDF5 File Type
HDF5 Memory Type
int
H5T_STD_I32BE
H5T_STD_I32LE
H5T_NATIVE_INT
float
H5T_IEEE_F32BE
H5T_IEEE_F32LE
H5T_NATIVE_FLOAT
double
H5T_IEEE_F64BE
H5T_IEEE_F64LE
H5T_NATIVE_DOUBLE
12
www.hdfgroup.org
HDF5 Defined Types
For portability, the HDF5 library has its own defined
types:
hid_t:
hsize_t:
herr_t:
object identifiers (native integer)
size used for dimensions (unsigned long or
unsigned long long)
function return value
For C, include hdf5.h in your HDF5 application.
13
www.hdfgroup.org
Basic Functions
H5Fcreate (H5Fopen)
create (open) File
H5Screate_simple/H5Screate create fileSpace
H5Dcreate (H5Dopen)
H5Sselect_hyperslab
select subsections of data
H5Dread, H5Dwrite
access Dataset
H5Dclose
H5Sclose
H5Fclose
create (open) Dataset
close Dataset
close fileSpace
close File
NOTE: Order not strictly specified.
14
www.hdfgroup.org
Logistics
Log into franklin or carver
ssh franklin.nersc.gov or ssh carver.nersc.gov
cp /project/projectdirs/training/pHDF5_examples.tar
$SCRATCH
cd $SCRATCH
tar xvf pHDF5_examples.tar
Here you will find the code examples, submission
scripts and detailed instructions in
instructions_carver.txt or instructions_franklin.txt
15
www.hdfgroup.org
The HDF Group
Example :
write_grid_rows.c
(or fortran90 version if you prefer)
16
www.hdfgroup.org
Example 1: Writing dataset by rows
P0
P1
File
P2
P3
NX
NY
17
www.hdfgroup.org
Writing by rows: Output of h5dump
HDF5 grid_rows.h5" {
GROUP "/" {
DATASET "dataset1" {
DATATYPE H5T_IEEE_F64LE
DATASPACE SIMPLE { ( 8, 5 ) / ( 8, 5 ) }
DATA {
18, 18, 18, 18, 18,
18, 18, 18, 18, 18,
19, 19, 19, 19, 19,
19, 19, 19, 19, 19,
20, 20, 20, 20, 20,
20, 20, 20, 20, 20,
21, 21, 21, 21, 21,
21, 21, 21, 21, 21
}
}
}
18
www.hdfgroup.org
Initialize the file for parallel access
/* first initialize MPI */
/* create access property list */
plist_id = H5Pcreate(H5P_FILE_ACCESS);
/* necessary for parallel access */
status = H5Pset_fapl_mpio(plist_id,
MPI_COMM_WORLD, MPI_INFO_NULL);
/* Create an hdf5 file */
file_id = H5Fcreate(FILENAME, H5F_ACC_TRUNC,
H5P_DEFAULT, plist_id);
status = H5Pclose(plist_id);
19
www.hdfgroup.org
Create file filespace and dataset
/* initialize local grid data */
/* Create the filespace */
dimsf[0] = NX;
dimsf[1] = NY;
filespace = H5Screate_simple(RANK, dimsf,NULL);
/* create a dataset */
dset_id = H5Dcreate(file_id, "dataset1,
H5T_NATIVE_DOUBLE, filespace, H5P_DEFAULT,
H5P_DEFAULT, H5P_DEFAULT);
20
www.hdfgroup.org
Create Property List
/* Create property list for collective dataset
write. */
plist_id = H5Pcreate(H5P_DATASET_XFER);
/* The other option is HDFD_MPIO_INDEPENDENT */
H5Pset_dxpl_mpio(plist_id,H5FD_MPIO_COLLECTIVE);
21
www.hdfgroup.org
Calculate Offsets
P0
P1
File
P2
NX
P3
NY
Every processor has a 2d array, which holds the number of
blocks to write and the starting offset
count[0], count[1]
offet[0][offset[1]
22
www.hdfgroup.org
Example 1: Writing dataset by rows
Process 1
File
Memory
offset[1]
count[1]
count[0]
offset[0]
count[0] = dimsf[0]/num_procs
count[1] = dimsf[1];
offset[0] = my_proc * count[0];
offset[1] = 0;
/* = 2 */
23
www.hdfgroup.org
Writing and Reading Hyperslabs
Distributed memory model: data is split among
processes
PHDF5 uses HDF5 hyperslab model
Each process defines memory and file
hyperslabs
Each process executes partial write/read call
Collective calls
Independent calls
24
www.hdfgroup.org
Create a Memory Space select hyperslab
/* Create the local memory space */
memspace = H5Screate_simple(RANK, count, NULL);
filespace = H5Dget_space (dset_id);
/* Create the hyperslab -- says how you want to
lay out data */
status = H5Sselect_hyperslab(filespace,
H5S_SELECT_SET, offset, NULL, count, NULL);
25
www.hdfgroup.org
Write Data
Identifier for dataset
dataset1
Datatype
status = H5Dwrite(dset_id, H5T_NATIVE_DOUBLE,
memspace, filespace, plist_id, grid_data);
Access Properties:
We choose collective.
This is where other
optimizations could be
added.
Data buffer
Then close every dataspace and file space that was opened
26
www.hdfgroup.org
How to Compile PHDF5 Applications
h5pcc HDF5 C compiler command
Similar to mpicc
h5pfc HDF5 F90 compiler command
Similar to mpif90
To compile:
% h5pcc h5prog.c
% h5pfc h5prog.f90
27
www.hdfgroup.org
Example 2: Writing dataset by columns
P0
File
P1
28
www.hdfgroup.org
Writing by columns: Output of h5dump,
HDF5 grid_cols.h5" {
GROUP "/" {
DATASET dataset1" {
DATATYPE H5T_IEEE_F64LE
DATASPACE SIMPLE { ( 4, 6 ) / ( 8, 6 ) }
DATA {
1, 2, 10, 20, 100, 200,
1, 2, 10, 20, 100, 200,
1, 2, 10, 20, 100, 200,
1, 2, 10, 20, 100, 200
}
}
}
}
29
www.hdfgroup.org
Example 2: Writing dataset by columns
P0
NX
File
P1
NY
More complicated pattern, describe data layout with 4 arrays
offset[] - starting position
stride[] - spacing to the next element
count[] - how many times to write a contiguous block
block[] - how many contiguous elements to write
30
www.hdfgroup.org
Example 2: Writing dataset by column
Memory
Process 0
File
P0 offset[1]
block[0]
dimsm[0]
dimsm[1]
Process 1
block[1]
P1 offset[1]
stride[1]
31
www.hdfgroup.org
Example 2: Writing dataset by column
/* Each process defines hyperslab in
the file */
count[0] = 1;
count[1] = dimsm[1];
offset[0] = 0;
offset[1] = my_proc;
stride[0] = 1;
stride[1] = 2;
block[0] = dimsm[0];
block[1] = 1;
/* Each process selects hyperslab.
filespace = H5Dget_space(dset_id); */
H5Sselect_hyperslab(filespace,
H5S_SELECT_SET, offset, stride,
count, block);
32
www.hdfgroup.org
Example 3: Writing dataset by pattern
Memory
File
Process 0
Process 1
NX
Process 2
Process 3
NY
33
www.hdfgroup.org
Writing by Pattern: Output of h5dump
HDF5 grid_pattern.h5" {
GROUP "/" {
DATASET Dataset1" {
DATATYPE H5T_IEEE_F64LE
DATASPACE SIMPLE { ( 8, 4 ) / ( 8, 4 ) }
DATA {
1, 3, 1, 3,
2, 4, 2, 4,
1, 3, 1, 3,
2, 4, 2, 4,
1, 3, 1, 3,
2, 4, 2, 4,
1, 3, 1, 3,
2, 4, 2, 4
}
}
}
34
www.hdfgroup.org
Example 3: Writing dataset by pattern
Memory
File
Process 0
Process 1
NX
Process 2
Process 3
NY
More complicated pattern, describe data layout with 4 arrays
offset[] - starting position
stride[] - spacing to the next element
count[] - how many times to write a contiguous block
block[] - how many contiguous elements to write
35
www.hdfgroup.org
Example 3: Writing dataset by pattern
Memory
File
stride[1]
Process 2
stride[0]
offset[0]
offset[1]
count[0]
count[1]
stride[0]
stride[1]
=
=
=
=
=
=
0;
1;
4;
2;
2;
2;
count[1]
offset[1]
36
www.hdfgroup.org
Example 3: Writing by pattern
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
/* Each process defines dataset in memory and
* writes it to the hyperslab in the file.
*/
count[0] = 4;
count[1] = 2;
stride[0] = 2;
stride[1] = 2;
if(my_proc == 0) {
offset[0] = 0;
offset[1] = 0;
}
if(my_proc == 1) {
offset[0] = 1;
offset[1] = 0;
}
if(my_proc == 2) {
offset[0] = 0;
offset[1] = 1;
}
if(my_proc == 3) {
offset[0] = 1;
offset[1] = 1;
}
37
www.hdfgroup.org
Example 4: Writing dataset by chunks
P0
P1
P2
File
P3
NX
NY
38
www.hdfgroup.org
Example 4: Writing dataset by chunks
File
P0
P1
P2
P3
NX
NY
More complicated pattern, describe data layout with 4 arrays
offset[] - starting position
stride[] - spacing to the next element
count[] - how many times to write a contiguous block
block[] - how many contiguous elements to write
39
www.hdfgroup.org
Writing by Chunks: Output of h5dump
HDF5 write_chunks.h5" {
GROUP "/" {
DATASET Dataset1" {
DATATYPE H5T_IEEE_F64LE
DATASPACE SIMPLE { ( 8, 4 ) / ( 8, 4 ) }
DATA {
1, 1, 2, 2,
1, 1, 2, 2,
1, 1, 2, 2,
1, 1, 2, 2,
3, 3, 4, 4,
3, 3, 4, 4,
3, 3, 4, 4,
3, 3, 4, 4
}
}
}
40
www.hdfgroup.org
Example 4: Writing dataset by chunks
File
Process 2: Memory
offset[1]
chunk_dims[1]
offset[0]
chunk_dims[0]
block[0]
block[0]
block[1]
offset[0]
offset[1]
=
=
=
=
chunk_dims[0];
chunk_dims[1];
chunk_dims[0];
0;
block[1]
41
www.hdfgroup.org
Example 4: Writing by chunks
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
count[0] = 1;
count[1] = 1 ;
stride[0] = 1;
stride[1] = 1;
block[0] = chunk_dims[0];
block[1] = chunk_dims[1];
if(mpi_rank == 0) {
offset[0] = 0;
offset[1] = 0;
}
if(mpi_rank == 1) {
offset[0] = 0;
offset[1] = chunk_dims[1];
}
if(mpi_rank == 2) {
offset[0] = chunk_dims[0];
offset[1] = 0;
}
if(mpi_rank == 3) {
offset[0] = chunk_dims[0];
offset[1] = chunk_dims[1];
}
42
www.hdfgroup.org
Fortran Tips and Tricks
Fortran interfaces require an extra initialization
and finalize call:
CALL h5open_f(error)
CALL h5close_f(error)
Some differences in argument order to API
from C version
Remember Fortran arrays start at 1 not 0.
Remember row and column order switched
from C programs. See write_grid_rows.f90 for
example
43
www.hdfgroup.org
Problem 1: Writing dataset by rows 3d
P0
P1
File
P2
P3
44
www.hdfgroup.org
Problem 2: Writing dataset by cols 3d
P0
File
P1
45
www.hdfgroup.org
HDF5 Compile Scripts
h5pcc HDF5 C compiler command
h5pfc HDF5 F90 compiler command
To compile:
% h5pcc h5prog.c
% h5pfc h5prog.f90
46
www.hdfgroup.org
The HDF Group
Parallel HDF5 in a little more
detail
47
www.hdfgroup.org
MPI-IO vs. HDF5
MPI-IO is an Input/Output API.
It treats the data file as a linear byte stream
and each MPI application needs to provide its
own file view and data representations to
interpret those bytes.
All data stored are machine dependent except
the external32 representation.
External32 is defined in Big Endianness
Little endian machines have to do the data
conversion in both read or write operations.
64bit sized data types may lose information.
48 www.hdfgroup.org
MPI-IO vs. HDF5 Cont.
HDF5 is a data management software.
It stores the data and metadata according to
the HDF5 data format definition.
HDF5 file is self-described.
Each machine can store the data in its own
native representation for efficient I/O without
loss of data precision.
Any necessary data representation conversion
is done by the HDF5 library automatically.
49 www.hdfgroup.org
Examples of PHDF5 API
Examples of PHDF5 collective API
File operations: H5Fcreate, H5Fopen, H5Fclose
Objects creation: H5Dcreate, H5Dopen, H5Dclose
Objects structure: H5Dextend (increase dimension
sizes)
Array data transfer can be collective or
independent
Dataset operations: H5Dwrite, H5Dread
Collectiveness is indicated by function parameters, not
by function names as in MPI API
50
www.hdfgroup.org
What Does PHDF5 Support ?
After a file is opened by the processes of a
communicator
All parts of file are accessible by all processes
All objects in the file are accessible by all
processes
Multiple processes may write to the same data
array
Each process may write to individual data array
51
www.hdfgroup.org
Collective vs. Independent Calls
MPI definition of collective call
All processes of the communicator must
participate in the right order. E.g.,
Process1
call A(); call B();
call A(); call B();
Process2
call A(); call B(); **right**
call B(); call A(); **wrong**
Independent means not collective
Collective is not necessarily synchronous
52
www.hdfgroup.org
Programming Restrictions
Most PHDF5 APIs are collective
PHDF5 opens a parallel file with a
communicator
Returns a file-handle
Future access to the file via the file-handle
All processes must participate in collective
PHDF5 APIs
Different files can be opened via different
communicators
53
www.hdfgroup.org
Programming model for creating and accessing a file
HDF5 uses access template object
(property list) to control the file access
mechanism
General model to access HDF5 file in
parallel:
Setup MPI-IO access template (access
property list)
Open File
Access Data
Close File
54
www.hdfgroup.org
Setup MPI-IO access template
Each process of the MPI communicator creates an
access template and sets it up with MPI parallel
access information
C:
herr_t H5Pset_fapl_mpio(hid_t plist_id,
MPI_Comm comm, MPI_Info info);
F90:
h5pset_fapl_mpio_f(plist_id, comm, info)
integer(hid_t) :: plist_id
integer
:: comm, info
plist_id is a file access property list identifier
55
www.hdfgroup.org
C Example Parallel File Create
23
24
26
27
28
29
33
34
35
->36
->37
38
->42
49
50
51
52
54
comm = MPI_COMM_WORLD;
info = MPI_INFO_NULL;
/*
* Initialize MPI
*/
MPI_Init(&argc, &argv);
/*
* Set up file access property list for MPI-IO access
*/
plist_id = H5Pcreate(H5P_FILE_ACCESS);
H5Pset_fapl_mpio(plist_id, comm, info);
file_id = H5Fcreate(H5FILE_NAME, H5F_ACC_TRUNC,
H5P_DEFAULT, plist_id);
/*
* Close the file.
*/
H5Fclose(file_id);
MPI_Finalize();
56
www.hdfgroup.org
F90 Example Parallel File Create
23
24
26
29
30
32
34
35
->37
->38
40
41
->43
45
46
49
51
52
54
56
comm = MPI_COMM_WORLD
info = MPI_INFO_NULL
CALL MPI_INIT(mpierror)
!
! Initialize FORTRAN predefined datatypes
CALL h5open_f(error)
!
! Setup file access property list for MPI-IO access.
CALL h5pcreate_f(H5P_FILE_ACCESS_F, plist_id, error)
CALL h5pset_fapl_mpio_f(plist_id, comm, info, error)
!
! Create the file collectively.
CALL h5fcreate_f(filename, H5F_ACC_TRUNC_F, file_id,
error, access_prp = plist_id)
!
! Close the file.
CALL h5fclose_f(file_id, error)
!
! Close FORTRAN interface
CALL h5close_f(error)
CALL MPI_FINALIZE(mpierror)
57
www.hdfgroup.org
Creating and Opening Dataset
All processes of the communicator open/
close a dataset by a collective call
C: H5Dcreate or H5Dopen; H5Dclose
F90: h5dcreate_f or h5dopen_f; h5dclose_f
All processes of the communicator must
extend an unlimited dimension dataset
before writing to it
C: H5Dextend
F90: h5dextend_f
58
www.hdfgroup.org
C Example: Create Dataset
56
57
58
59
60
61
62
63
64
65
66
->67
68
70
71
72
73
74
file_id = H5Fcreate();
/*
* Create the dataspace for the dataset.
*/
dimsf[0] = NX;
dimsf[1] = NY;
filespace = H5Screate_simple(RANK, dimsf, NULL);
/*
* Create the dataset with default properties collective.
*/
dset_id = H5Dcreate(file_id, dataset1, H5T_NATIVE_INT,
filespace, H5P_DEFAULT);
H5Dclose(dset_id);
/*
* Close the file.
*/
H5Fclose(file_id);
59
www.hdfgroup.org
F90 Example: Create Dataset
43 CALL h5fcreate_f(filename, H5F_ACC_TRUNC_F, file_id,
error, access_prp = plist_id)
73 CALL h5screate_simple_f(rank, dimsf, filespace, error)
76 !
77 ! Create the dataset with default properties.
78 !
->79 CALL h5dcreate_f(file_id, dataset1, H5T_NATIVE_INTEGER,
filespace, dset_id, error)
90 !
91 ! Close the dataset.
92 CALL h5dclose_f(dset_id, error)
93 !
94 ! Close the file.
95 CALL h5fclose_f(file_id, error)
60
www.hdfgroup.org
Accessing a Dataset
All processes that have opened dataset may
do collective I/O
Each process may do independent and
arbitrary number of data I/O access calls
C: H5Dwrite and H5Dread
F90: h5dwrite_f and h5dread_f
61
www.hdfgroup.org
Programming model for dataset access
Create and set dataset transfer property
C: H5Pset_dxpl_mpio
H5FD_MPIO_COLLECTIVE
H5FD_MPIO_INDEPENDENT (default)
F90: h5pset_dxpl_mpio_f
H5FD_MPIO_COLLECTIVE_F
H5FD_MPIO_INDEPENDENT_F (default)
Access dataset with the defined transfer
property
62
www.hdfgroup.org
C Example: Collective write
95
96
97
98
->99
100
/*
* Create property list for collective dataset write.
*/
plist_id = H5Pcreate(H5P_DATASET_XFER);
H5Pset_dxpl_mpio(plist_id, H5FD_MPIO_COLLECTIVE);
63
www.hdfgroup.org
F90 Example: Collective write
88
89
90
->91
92
93
94
95
96
! Create property list for collective dataset write
!
CALL h5pcreate_f(H5P_DATASET_XFER_F, plist_id, error)
CALL h5pset_dxpl_mpio_f(plist_id, &
H5FD_MPIO_COLLECTIVE_F, error)
!
! Write the dataset collectively.
!
CALL h5dwrite_f(dset_id, H5T_NATIVE_INTEGER, data, &
error, &
file_space_id = filespace, &
mem_space_id = memspace, &
xfer_prp = plist_id)
64
www.hdfgroup.org
Writing and Reading Hyperslabs
Distributed memory model: data is split among
processes
PHDF5 uses HDF5 hyperslab model
Each process defines memory and file
hyperslabs
Each process executes partial write/read call
Collective calls
Independent calls
65
www.hdfgroup.org
HDF5 Properties
Properties (also known as Property Lists)
are characteristics of HDF5 objects that can
be modified
Default properties handle most needs
By changing properties one can take
advantage of the more powerful features in
HDF5
66
www.hdfgroup.org
Storage Properties
Data elements
stored physically
adjacent to each
other
Better access time
for subsets;
extensible
Improves storage
efficiency,
transmission speed
67
www.hdfgroup.org
HDF5 Attributes (optional)
An HDF5 attribute has a name and a value
Attributes typically contain user metadata
Attributes may be associated with
- HDF5 groups
- HDF5 datasets
- HDF5 named datatypes
An attributes value is described by a datatype and a
dataspace
Attributes are analogous to datasets except
- they are NOT extensible
- they do NOT support compression or partial I/O
68
www.hdfgroup.org
Dataset Creation Property List
Better access time
for subsets;
extensible
Improves storage
efficiency,
transmission speed
H5P_DEFAULT: contiguous
Dataset creation property list: information on how to
organize data in storage.
69
www.hdfgroup.org
Steps to Create a Group
1. Decide where to put it root group
Obtain location ID
2. Define properties or use H5P_DEFAULT
5. Create group in file.
4. Close the group.
70
www.hdfgroup.org
Example: Create a Group
/ (root)
A
4x6 array of
integers
file.h5
71
www.hdfgroup.org
Code: Create a Group
hid_t file_id, group_id;
...
/* Open file.h5 */
file_id = H5Fopen (file.h5, H5F_ACC_RDWR,
H5P_DEFAULT);
/* Create group "/B" in file. */
group_id = H5Gcreate (file_id,"B", H5P_DEFAULT,
H5P_DEFAULT, H5P_DEFAULT);
/* Close group and file. */
status = H5Gclose (group_id);
status = H5Fclose (file_id);
72
www.hdfgroup.org
The HDF Group
Intermediate Parallel HDF5
73
www.hdfgroup.org
Outline
Performance
Parallel tools
74
www.hdfgroup.org
My PHDF5 Application I/O is slow
If my application I/O performance is slow, what
can I do?
Use larger I/O data sizes
Independent vs. Collective I/O
Specific I/O system hints
Increase Parallel File System capacity
75
www.hdfgroup.org
Write Speed vs. Block Size
MB/Sec
TFLOPS: HDF5 Write vs MPIO Write
(File size 3200MB, Nodes: 8)
120
100
80
60
40
20
0
HDF5 Write
MPIO Write
16
32
Block Size (MB)
76 www.hdfgroup.org
Independent Vs Collective Access
User reported
Independent data
transfer mode was much
slower than the
Collective data transfer
mode
Data array was tall and
thin: 230,000 rows by 6
columns
:
:
:
230,000 rows
:
:
:
77
www.hdfgroup.org
Independent vs. Collective write
6 processes, IBM p-690, AIX, GPFS
# of Rows
Data Size
(MB)
Independent
(Sec.)
Collective
(Sec.)
16384
0.25
8.26
1.72
32768
0.50
65.12
1.80
65536
1.00
108.20
2.68
122918
1.88
276.57
3.11
150000
2.29
528.15
3.63
180300
2.75
881.39
4.12
78
www.hdfgroup.org
Independent vs. Collective write (cont.)
Performance (non-contiguous)
1000
900
800
Time (s)
700
600
Independent
500
Collective
400
300
200
100
0
0.00
0.50
1.00
1.50
2.00
2.50
3.00
Data space size (MB)
79
www.hdfgroup.org
Effects of I/O Hints: IBM_largeblock_io
GPFS at LLNL Blue
4 nodes, 16 tasks
Total data size 1024MB
I/O buffer size 1MB
Tasks
16
16
IBM_largeblock_io=false IBM_largeblock_io=true
MPI-IO
PHDF5
MPI-IO
PHDF5
write (MB/S)
60
48
354
294
read (MB/S)
44
39
256
248
80
www.hdfgroup.org
Effects of I/O Hints: IBM_largeblock_io
GPFS at LLNL ASCI Blue machine
4 nodes, 16 tasks
Total data size 1024MB
I/O buffer size 1MB
400
350
300
250
200
150
16 write
16 read
100
50
0
MPI-IO
PHDF5
MPI-IO
PHDF5
IBM_largeblock_io=false IBM_largeblock_io=true
81
www.hdfgroup.org
Parallel Tools
ph5diff
Parallel version of the h5diff tool
h5perf
Performance measuring tools showing I/
O performance for different I/O API
82
www.hdfgroup.org
ph5diff
An parallel version of the h5diff tool
Supports all features of h5diff
An MPI parallel tool
Manager process (proc 0)
coordinates each the remaining processes
(workers) to diff one dataset at a time;
collects any output from each worker and
prints them out.
Works best if there are many datasets in the
files with few differences.
Available in v1.8.
83
www.hdfgroup.org
h5perf
An I/O performance measurement tool
Test 3 File I/O API
POSIX I/O (open/write/read/close)
MPIO (MPI_File_{open,write,read,close})
PHDF5
H5Pset_fapl_mpio (using MPI-IO)
H5Pset_fapl_mpiposix (using POSIX I/O)
84
www.hdfgroup.org
h5perf: Some features
Check (-c) verify data correctness
Added 2-D chunk patterns in v1.8
-h shows the help page.
85
www.hdfgroup.org
h5perf: example output 1/3
% mpirun -np 4 h5perf
Number of processors = 4
Transfer Buffer Size: 131072 bytes, File size: 1.00 MBs
# of files: 1, # of datasets: 1, dataset size: 1.00 MBs
IO API = POSIX
Write (1 iteration(s)):
Maximum Throughput: 18.75 MB/s
Average Throughput: 18.75 MB/s
Minimum Throughput: 18.75 MB/s
Write Open-Close (1 iteration(s)):
Maximum Throughput: 10.79 MB/s
Average Throughput: 10.79 MB/s
Minimum Throughput: 10.79 MB/s
Read (1 iteration(s)):
Maximum Throughput: 2241.74 MB/s
Average Throughput: 2241.74 MB/s
Minimum Throughput: 2241.74 MB/s
Read Open-Close (1 iteration(s)):
Maximum Throughput: 756.41 MB/s
Average Throughput: 756.41 MB/s
Minimum Throughput: 756.41 MB/s
86 www.hdfgroup.org
h5perf: example output 2/3
% mpirun -np 4 h5perf
IO API = MPIO
Write (1 iteration(s)):
Maximum Throughput: 611.95 MB/s
Average Throughput: 611.95 MB/s
Minimum Throughput: 611.95 MB/s
Write Open-Close (1 iteration(s)):
Maximum Throughput: 16.89 MB/s
Average Throughput: 16.89 MB/s
Minimum Throughput: 16.89 MB/s
Read (1 iteration(s)):
Maximum Throughput: 421.75 MB/s
Average Throughput: 421.75 MB/s
Minimum Throughput: 421.75 MB/s
Read Open-Close (1 iteration(s)):
Maximum Throughput: 109.22 MB/s
Average Throughput: 109.22 MB/s
Minimum Throughput: 109.22 MB/s
87 www.hdfgroup.org
h5perf: example output 3/3
% mpirun -np 4 h5perf
IO API = PHDF5 (w/MPI-I/O driver)
Write (1 iteration(s)):
Maximum Throughput: 304.40 MB/s
Average Throughput: 304.40 MB/s
Minimum Throughput: 304.40 MB/s
Write Open-Close (1 iteration(s)):
Maximum Throughput: 15.14 MB/s
Average Throughput: 15.14 MB/s
Minimum Throughput: 15.14 MB/s
Read (1 iteration(s)):
Maximum Throughput: 1718.27 MB/s
Average Throughput: 1718.27 MB/s
Minimum Throughput: 1718.27 MB/s
Read Open-Close (1 iteration(s)):
Maximum Throughput: 78.06 MB/s
Average Throughput: 78.06 MB/s
Minimum Throughput: 78.06 MB/s
Transfer Buffer Size: 262144 bytes, File size: 1.00 MBs
# of files: 1, # of datasets: 1, dataset size: 1.00 MBs
88 www.hdfgroup.org
Useful Parallel HDF Links
Parallel HDF information site
http://www.hdfgroup.org/HDF5/PHDF5/
Parallel HDF5 tutorial available at
http://www.hdfgroup.org/HDF5/Tutor/
HDF Help email address
help@hdfgroup.org
89
www.hdfgroup.org
The HDF Group
Questions?
End of Part IV
90
www.hdfgroup.org
HDF5 Groups
Used to organize collections
Every file starts with a root group
Similar to UNIX directories
Path to object defines it
Objects can be shared:
/A/k and /B/l are the same temp
A
k
B
l
temp
= Group
= Dataset
91
www.hdfgroup.org
HDF5 Dataset with Compound Datatype
3
int8
int4
V V V
V V V
int16
Compound
Datatype:
Dataspace: Rank = 2
Dimensions = 5 x 3
92
www.hdfgroup.org
Link Creation/Dataset Access Properties
Link Creation:
Creating intermediate groups
Dataset Access:
Retrieve the raw data chunk cache parameters
93
www.hdfgroup.org
Group Properties
Link Creation
Creating intermediate groups
Group Creation
Creation order tracking and indexing for links in
a group.
Set Number of links and length of link names in
a group.
Group Access (not used)
94
www.hdfgroup.org
Compile option: -show
-show: displays the compiler commands and options
without executing them
% h5cc show Sample_c.c
Will show the correct paths and libraries used by
the installed HDF5 library.
Will show the correct flags to specify when
building an application with that HDF5 library.
95
www.hdfgroup.org
The HDF Group
Other General HDF5 Slides
96
www.hdfgroup.org
Help
The HDF Group Page:
HDF5 Home Page:
HDF Helpdesk:
HDF Mailing Lists:
http://hdfgroup.org/
http://hdfgroup.org/HDF5/
help@hdfgroup.org
http://hdfgroup.org/services/support.html
ASTROSIM Summer School
97
www.hdfgroup.org
HDF5 is designed
for high volume and/or complex data
for every size and type of system (portable)
for flexible, efficient storage and I/O
to enable applications to evolve in their use of
HDF5 and to accommodate new models
to support long-term data preservation
98
www.hdfgroup.org
HDF5 Home Page
HDF5 home page: http://hdfgroup.org/HDF5/
Two releases: HDF5 1.8 and HDF5 1.6
HDF5 source code:
Written in C, and includes optional C++, Fortran 90 APIs,
and High Level APIs
Contains command-line utilities (h5dump, h5repack,
h5diff, ..) and compile scripts
HDF pre-built binaries:
When possible, include C, C++, F90, and High Level
libraries. Check ./lib/libhdf5.settings file.
Built with and require the SZIP and ZLIB external libraries
99
www.hdfgroup.org
HDF5 Technology
HDF5 (Abstract) Data Model
Defines the building blocks for data organization and
specification
Files, Groups, Datasets, Attributes, Datatypes, Dataspaces,
HDF5 Library (C, Fortran 90, C++ APIs)
Also Java Language Interface and High Level Libraries
HDF5 Binary File Format
Bit-level organization of HDF5 file
Defined by HDF5 File Format Specification
Tools For Accessing Data in HDF5 Format
h5dump, h5repack, HDFView,
100
www.hdfgroup.org
HDF5 File
An HDF5 file is a
container that
holds data
objects.
lat|lon|temp
||
12|23|3.1
15|24|4.2
17|21|3.6
101
www.hdfgroup.org
HDF5 Datasets
HDF5 Datasets organize and contain your
raw data values. They consist of:
Your raw data
Metadata describing the data:
- The information to interpret the data (Datatype)
- The information to describe the logical layout of the
data elements (Dataspace)
- Characteristics of the data (Properties)
- Additional optional information that describes the
data (Attributes)
102
www.hdfgroup.org
HDF5 Abstract Data Model Summary
The Objects in the Data Model are the building
blocks for data organization and specification
Files, Groups, Links, Datasets, Datatypes,
Dataspaces, Attributes,
Projects using HDF5 map their data concepts to
these HDF5 Objects
103
www.hdfgroup.org
HDF5 Library
Tools
HDF5 Software Layers & Storage
API
High Level h5dump
tool
APIs
Language
Interfaces
C, Fortran, C++
Internals
Virtual File
Layer
h5repack HDFview
tool
tool
Java Interface
HDF5 Data Model
Objects
Tunable Properties
Groups, Datasets, Attributes,
Memory
Mgmt
Datatype
Conversion
Filters
Split
Files
Posix
I/O
Chunk Size, I/O Driver,
Chunked
Storage
Version
Compatibility
and so
on
Custom
MPI I/O
Storage
I/O Drivers
HDF5 File
Format
File
Split
Files
File on
Parallel
Filesystem
104
Other
www.hdfgroup.org
Useful Tools For New Users
h5dump:
Tool to dump or display contents of HDF5 files
h5pcc,, h5pfc:
Scripts to compile applications
HDFView:
Java browser to view HDF4 and HDF5 files
http://www.hdfgroup.org/hdf-java-html/hdfview/
105
www.hdfgroup.org
HDF5 is like
106
www.hdfgroup.org
h5dump Utility
h5dump [options] [file]
-H, --header
-d <names>
-g <names>
-p
Display header only no data
Display the specified dataset(s).
Display the specified group(s) and
all members.
Display properties.
<names> is one or more appropriate object names.
107
www.hdfgroup.org
Example of h5dump Output
HDF5 "dset.h5" {
GROUP "/" {
DATASET "dset" {
DATATYPE { H5T_STD_I32BE }
DATASPACE { SIMPLE ( 4, 6 ) / ( 4, 6 ) }
DATA {
1, 2, 3, 4, 5, 6,
7, 8, 9, 10, 11, 12,
13, 14, 15, 16, 17, 18,
19, 20, 21, 22, 23, 24
}
/
}
dset
}
}
108
www.hdfgroup.org
Pre-defined Native Datatypes
Examples of predefined native types in C:
H5T_NATIVE_INT
H5T_NATIVE_FLOAT
H5T_NATIVE_UINT
H5T_NATIVE_LONG
H5T_NATIVE_CHAR
(int)
(float )
(unsigned int)
(long )
(char )
NOTE: Memory types.
Different for each machine.
Used for reading/writing.
109
www.hdfgroup.org
Other Common Functions
DataSpaces:
H5Sselect_hyperslab
H5Sselect_elements
H5Dget_space
Groups:
H5Gcreate, H5Gopen, H5Gclose
Attributes:
H5Acreate, H5Aopen_name,
H5Aclose, H5Aread, H5Awrite
Property lists:
H5Pcreate, H5Pclose
H5Pset_chunk, H5Pset_deflate
110
www.hdfgroup.org
HDF = Hierarchical Data Format
HDF5 is the second HDF format
Development started in 1996
First release was in 1998
HDF4 is the first HDF format
Originally called HDF
Development started in 1987
Still supported by The HDF Group
111
www.hdfgroup.org
HDF5 Dataspaces
Two roles:
Dataspace contains spatial information (logical
layout) about a dataset
stored in a file
Rank and dimensions
Permanent part of dataset
definition
Subsets: Dataspace describes applications data
buffer and data elements participating in I/O
112
www.hdfgroup.org
