Linux Clusters Ins.
tute:
Cluster Stack Basics
Bre$
Zimmerman,
University
of
Oklahoma
Senior
Systems
Analyst,
OU
Supercompu<ng
Center
for
Educa<on
and
Research
(OSCER)
A Bunch of Computers
Users
can
login
to
any
node
Filesystems
arent
shared
between
nodes
Work
is
run
wherever
you
can
nd
space
Nodes
maintained
individually
4-8 August 2014
Whats wrong with a bunch of nodes?
Compe<<on
for
resources
Size
and
type
of
problem
is
limited
Nodes
get
out
of
sync
Problems
for
users
Diculty
in
management
4-8 August 2014
Cluster Approach
Shared
lesystems
Job
management
Nodes
dedicated
to
compute
Consistent
environment
Interconnect
4-8 August 2014
Whats right about the cluster approach?
Easier
to
use
Maximize
eciency
Can
do
bigger
and
be$er
problems
Nodes
can
be
used
coopera<vely
4-8 August 2014
The Types of Nodes
Login
Users
login
here
Compiling
Edi<ng
SubmiTng
and
Monitoring
jobs
Compute
Users
might
login
here
Run
jobs
as
directed
by
the
scheduler
Support
Users
dont
login
here
Do
all
the
other
stu
4-8 August 2014
What a cluster needs the mundane
Network
services
NTP,
DNS,
DHCP
Shared
Storage
--
NFS
Logging
Consolidated
Syslog
as
a
star<ng
point
Licensing
FlexLM
and
the
like
Database
User
and
Administra<ve
Data
Boot/Provisioning
PXE,
build
system
Authen<ca<on
LDAP
4-8 August 2014
What a cluster needs -- Specialized
Interconnect
An
ideally
low-latency
network
Job
manager
Resource
manager/
scheduler
Parallel
Storage
Get
around
the
limita<ons
of
NFS
4-8 August 2014
Network Services
NTP
Network
Time
Protocol,
provides
clock
synchroniza<on
across
all
nodes
in
the
cluster
DHCP
Dynamic
Host
Congura<on
Protocol,
allows
central
congura<on
of
host
networking
DNS
Provides
name
to
address
transla<on
for
the
cluster
NFS
Basic
UNIX
network
lesystem
4-8 August 2014
Logging
Syslog
The
classic
system
for
UNIX
logging
Applica<on
has
to
opt
to
emit
messages
Monitoring
Ac<ve
monitoring
to
catch
condi<ons
elec<ve
monitoring
doesnt
catch
Resource
manager
Nagios/cac</zabbix/ganglia
IDS
Intrusion
detec<on
Monitoring
targe<ng
misuse/a$acks
on
the
cluster
4-8 August 2014
10
Basic services, con.nued
Licensing
FlexNet/FlexLM
or
equivalent,
mediates
access
to
a
pool
of
shared
licenses.
Database
Administra<ve
use
for
logging/monitoring,
dynamic
congura<on.
Requirements
of
user
so`ware.
Boot/Provisioning
For
example
PXE/Cobbler,
PXE/Image
or
part
of
a
cluster
management
suite
4-8 August 2014
11
Authen.ca.on
Flat
les
--
passwd,
group,
shadow
entries
NIS
--
network
access
to
central
at
les
LDAP
--
Read/Write
access
to
a
dynamic
tree
structure
of
account
and
other
informa<on
Host
equivalency
4-8 August 2014
12
Cluster Networking
Hardware
Management
Lights
out
management
External
Public
interfaces
to
the
cluster
Internal
General
node
to
node
communica<on
Storage
Access
to
network
lesystems
Interconnect
high-speed,
low-latency
for
mul<-
node
jobs
Some
of
these
can
share
a
medium
4-8 August 2014
13
Interconnect
In
the
most
recent
Top
500
list
(h$p://top500.org)
there
were
224
installa<ons
relying
on
Inniband,
100
using
Gigabit
Ethernet,
and
88
using
10
Gigabit
Ethernet
Ethernet
Latency
of
50-125
s
(GbE),
5-50
s
(10GbE),
~5
s
RoCEE
Inniband
Latency
of
1.3
s
(QDR)
.7
s
(FDR-10/
FDR),
.5
s
(EDR)
4-8 August 2014
14
Parallel Filesystem
Lustre
-
h$p://lustre.org/
PanFS
-
h$p://www.panasas.com/
GPFS
-
h$p://www-03.ibm.com/so`ware/products/en/
so`ware
Parallel
lesystems
take
the
general
approach
of
separa<ng
lesystem
metadata
from
the
storage.
Lustre
and
PanFS
have
dedicated
nodes
for
metadata
(MDS
or
director
blades).
GPFS
distributes
metadata
throughout
the
cluster
4-8 August 2014
15
Cluster Management
Automates
the
building
of
a
cluster
Some
way
to
easily
maintain
cluster
system
consistency
The
ability
to
automate
cluster
maintenance
tasks
Oer
some
way
to
monitor
cluster
health
and
performance
4-8 August 2014
16
Cluster Managemement SoNware
The
resource
manager
knows
the
state
of
the
various
resources
on
the
cluster
and
maintains
a
list
of
the
jobs
that
are
reques<ng
resources
The
scheduler,
using
the
informa<on
from
the
resource
manager
selects
jobs
from
the
queue
for
execu<on
Rocks
(h$p://www.rocksclusters.org/wordpress/)
Bright
Cluster
Manager
(
h$p://www.brightcompu<ng.com/Bright-Cluster-Manager)
xCAT
(Extreme
Cluster/Cloud
Administra<on
Toolkit)
(
h$p://sourceforge.net/p/xcat/wiki/Main_Page/
4-8 August 2014
17
Congura.on Management
While
it
is
true
that
boo<ng
with
a
central
boot
server
can
make
it
easier
to
make
sure
the
OS
on
each
compute
node
(or,
at
least,
each
type
of
compute
node)
has
an
iden<cal
setup/install,
there
are
s<ll
les
which
wind
up
being
more
dynamic.
Some
such
les
are
password/group/shadow
and
hosts
les.
Rsync
Cfengine
Chef
Puppet
Salt
4-8 August 2014
18
SoNware Installa.on and Management
All
linux
distros
have
some
sort
of
package
management
tool.
For
Redhat/CentOS/Scien<c
based
clusters,
this
is
rpm
and
yum.
Debian
has
dpkg
and
apt
In
any
case
pre-packaged
so`ware
tends
to
assume
that
it
is
going
to
be
installed
in
a
specic
place
on
the
machine
and
that
it
will
be
the
only
version
of
that
so`ware
on
the
machine.
One
a
cluster,
it
may
be
necessary
to
look
at
so`ware
installa<on
dierently
from
a
standard
linux
machine
Install
to
global
lesystem
Keep
boot
image
as
small
as
possible
Maintain
mul<ple
versions
4-8 August 2014
19
SoNware installa.on and management
There
are
a
couple
of
tools
useful
for
naviga<ng
the
dicul<es
of
maintaining
user
environments
when
dealing
with
mul<ple
versions
of
so`ware
or
so`ware
in
non-standard
loca<ons.
So`Env
(h$p://h$p://www.lcrc.anl.gov/info/So`ware/So`env)
Useful
for
packaging
sta<c
user
environment
required
by
packages
Modules
(h$p://modules.sourceforge.net/)
Can
be
used
to
make
dynamic
changes
to
a
users
environment.
4-8 August 2014
20
Resource Manager/Scheduler
Accepts
job
submissions,
maintains
a
queue
of
jobs
Allocates
nodes/resources
and
starts
jobs
on
compute
nodes
Schedules
wai<ng
jobs
Available
op<ons
SGE
(Sun
Grid
Engine)
LSF
/
Openlava
(Load
Sharing
Facility)
PBS
(Portable
Batch
System)
OpenPBS
Torque
SLURM
4-8 August 2014
21
Best Prac.ces
Here
is
a
quick
overview
of
the
general
func<ons
to
secure
a
cluster
Risk
Avoidance
Deterrence
Preven<on
Detec<on
Recovery
The
priority
of
these
will
depend
on
your
security
approach
4-8 August 2014
22
Risk Avoidance
Provide
the
minimum
of
services
necessary
Grant
the
least
privileges
necessary
Install
the
minimum
so`ware
necessary
The
simpler
the
environment,
the
fewer
the
vectors
available
for
a$ack.
4-8 August 2014
23
Deterrence
Limit
the
discoverability
of
the
cluster
Publish
acceptable
use
policies
Preven.on
Fix
known
issues
(patching)
Congure
services
for
minimal
func<onality
Restrict
user
access
and
authority
Document
ac<ons
and
changes
4-8 August 2014
24
Detec.on
Monitor
the
cluster
Integrate
feedback
from
the
users
Set
alerts
and
automated
response
Recovery
Backups
Documenta<on
Dene
acceptable
loss
4-8 August 2014
25
