Setting up Lustre Router - Hosts and Clients

by Jesse Stroik — last modified Mar 20, 2014 09:53 PM

 

Important Note: The definitive source for Lustre documentation is the Lustre Operations Manual available at https://wiki.hpdd.intel.com/display/PUB/Documentation.

These documents are copied from internal SSEC working documentation that may be useful for some, but be we provide no guarantee of accuraccy, correctness, or safety. Use at your own risk.

These notes are for hosts bridgng between networks, such as infinband to ethernet. Significant portions of the notes have been removed as they are specific to SSEC procedures.

Install Software on the Router

 
The router only requires lustre client software, not server.
 

Setting up the Lustre Networks

 
This sets up the lnet module to load and start the proper networks to communicate on. it needs to at least include all networks the router needs. The most common examples will be the current IB network it is connected to, and the SSEC public network. Int his example, I'll use the SSEC IB and SSEC public nets.
 
options lnet 'ip2nets="o2ib0 172.16.*.*; tcp0 128.104.108.0/22"'
 
 
 

Routing configuration

 
Set the /etc/modprobe.d/lustre.conf files accordingly. Create the file if it does not exist.

 Router

 
options lnet ip2nets="tcp0 128.104.*.*; o2ib0 172.16.*.*" forwarding=enabled
 NOTE: ORDER MATTERS with tcp0 and o2ib0 definitions
 

Clients

 


Need to have lustre client installed

 

 Simple example for client:
options lnet networks="tcp" routes="o2ib0 128.104.108.34@tcp0"
 After creating lustre.conf:

service --add lustre  service --add lnet  chkconfig lustre on  chkconfig lnet on  reboot

After the reboot, you can create mount points and add lustre mounts. Here is an example of adding fjord to /etc/fstab.

172.16.24.5@o2ib:/fjord /fjord                  lustre defaults,_netdev,user_xattr      0 0

MDS

 
options lnet ip2nets="o2ib0 172.16.*.*" routes="tcp0 172.16.24.13@o2ib0 172.16.24.[220-229]@o2ib0"

 where 172.16.24.13 is your router. multiple can be speicifed, and you can use the syntax:

routes="network-type1 N.N.N.[N-N]@network-type2

 Where N is a number.

 

OSTs

 

options lnet ip2nets="o2ib0 172.16.*.*" routes="tcp0 172.16.24.13@o2ib0"

 

 Checking

 
After you attempt to mount, run:
 
lfs df -h

And see if every OST is active and available.

It may take a few minutes for the mount points to work properly after you get everything else up, because it needs to determine which OSTs are up and sane, and it needs to determine which routers are active.