
  Copyright (c) 2013 Intel Corporation.  All rights reserved.
  Copyright (c) 2006-2011. QLogic Corporation. All rights reserved.
  Copyright (c) 2003-2006, PathScale, Inc. All rights reserved.
 
  This software is available to you under a choice of one of two
  licenses.  You may choose to be licensed under the terms of the GNU
  General Public License (GPL) Version 2, available from the file
  COPYING in the main directory of this source tree, or the
  OpenIB.org BSD license below:
 
      Redistribution and use in source and binary forms, with or
      without modification, are permitted provided that the following
      conditions are met:
 
       - Redistributions of source code must retain the above
         copyright notice, this list of conditions and the following
         disclaimer.
 
       - Redistributions in binary form must reproduce the above
         copyright notice, this list of conditions and the following
         disclaimer in the documentation and/or other materials
         provided with the distribution.
 
  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
  EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
  MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
  NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
  BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
  ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
  CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
  SOFTWARE.
 
================================================================================

OFED Support
------------
OFED 3.5 or above should be installed on the node. Prior versions of OFED
have an older QLogic IB driver (ib_qib) and do not fully support all the PSM
features with this release.

Building PSM
------------
Build requires that GNU GCC compiler is installed on the machine doing the
build, if compiling code to mic card, the cross compiler x86_64-k1om-linux-gcc
must also installed. If compiling mic version or mic card version, scif must
installed on the system. Root privileges are required to install the runtime
libraries and development header files into standard system location.

There are three mechanisms for building PSM:
  1. Use provided Makefiles to build and install
  2. Generate tarballs and install
  3. Generate rpms and install

Get this code, and cd to the top directory where this README file is.

Building from Makefiles
-----------------------
Run make on the command line. This will build the libraries. By default,
the Makefile will auto-detect whether libuuid and the uuid.h header file
are installed. If so, it will use the system's libuuid. Otherwise, PSM
will be compiled with the libuuid included with PSM.
	$ make
To force compiling with the included libuuid, use the USE_PSM_UUID variable:
        $ make USE_PSM_UUID=1

Note, Makefile will also auto-detect if scif(/usr/include/scif.h) presents,
and build either mic version or Xeon version, mic version enables symmetric
mode to talk to processes on mic card.
To force compiling Xeon version when scif is installed:
	$ make PSM_HAVE_SCIF=0

To compile PSM mic-card version:
	$ make MIC=1 CC=x86_64-k1om-linux-gcc

Install the libraries and header files on the system (as root):
	$ make install

The libraries will be installed in either /usr/lib or /usr/lib64, depending on
the architecture of the machine, and the header files will be installed in 
/usr/include.
This behavior can be altered by using the "DESTDIR" and "LIBDIR" variables on
the "make install" command line. "DESTDIR" will add a leading path component
to the overall install path and "LIBDIR" will change the path where libraries
will be installed. For example, "make DESTDIR=/tmp/psm-install install" will
install all files (libraries and headers) into "/tmp/psm-install/usr/...",
"make DESTDIR=/tmp/psm-install LIBDIR=/libraries install" will install the
libraries in "/tmp/psm-install/libraries" and the headers in
"/tmp/psm-install/usr/include", and "make LIBDIR=/tmp/libs install" will
install the libraries in "/tmp/libs" and the headers in "/usr/include".

Generating PSM tarballs
-----------------------
1. generate Xeon tarballs:
	$ make infinipath-tarballs
2. generate mic-host tarballs (must have scif installed):
	$ make mic-tarballs
3. generate mic-card tarballs (must have scif installed):
	$ make miccard-tarballs
4. generate all tarballs above (must have scif installed):
	$ make tarballs

Generating PSM source and binary RPMs
-------------------------------------
1. generate Xeon rpms:
	$ make infinipath-rpms
2. generate mic-host rpms (must have scif installed):
	$ make mic-rpms
3. generate mic-card rpms (must have scif installed):
	$ make miccard-rpms
4. generate all rpms above (must have scif installed):
	$ make rpms

Please note, in any case, you can build debugging version of the code by
adding PSM_DEBUG=1 on make command line.

If you run into any issues during build please post any questions with as much 
information as possible to the psm-devel mailing list. Please include the 
distribution you are building on, the compiler type and version as well as any 
relevant error messages.

MPI Libraries supported
-----------------------
A large number of open source (OpenMPI, MVAPICH, MVAPICH2) and Vendor MPI
implementations support PSM for optimized communication on QLogic Truescale
Infiniband HCAs. Vendor MPI implementations (HP-MPI, Intel MPI 4.0 with PMI,
Platform/Scali MPI) require that the PSM runtime libraries be installed and
available on each node. Usually a configuration file or a command line switch
to mpirun needs to be specified to utilize the PSM transport.

OpenMPI support
---------------
It is recommended to use the OpenMPI v1.5 development branch. Prior versions
of OpenMPI have an issue with support PSM network transports mixed with standard
Verbs transport (BTL openib). This prevents an OpenMPI installation with
network modules available for PSM and Verbs to work correctly on nodes with
no QLogic IB hardware. This has been fixed in the latest development branch
allowing a single OpenMPI installation to target IB hardware via PSM or Verbs
as well as alternate transports seamlessly.

PSM header and runtime files need to be installed on a node where the OpenMPI
build is performed. All compute nodes additionally should have the PSM runtime
libraries available on them. OpenMPI provides a standard configure, make and
make install mechanism which will detect and build the relevant PSM network
modules for OpenMPI once the header and runtime files are detected. Further
information on compiling and running MPI applications with OpenMPI on PSM is
available in the QLogic OFED User guide available at:

http://driverdownloads.qlogic.com/QLogicDriverDownloads_UI/SearchByProduct.aspx?ProductCategory=301&Product=1116&Os=65

MVAPICH and MVAPICH2 support
----------------------------
Both MVAPICH and MVAPICH2 support PSM transport for optimized communication on
QLogic Truescale IB hardware. MVAPICH2 1.4 and MVAPICH 1.2 versions are
recommended. PSM header and runtime files need to be installed on a node where 
MVAPICH builds are performed. All compute nodes additionally should have the 
PSM runtime libraries available on them.

MVAPICH provides a shell script in it's top level directory called
make.mvapich.psm to configur, make and install MVAPICH with PSM support.

MVAPICH2 provides a standard configure and make infrastructure. In order to
MVAPICH2 for PSM the following should be performed from the top level directory:

    - ./configure --prefix=<path_to_install_mvapich2> --with-device=ch3:psm
    - make
    - make install

Further information on compiling and running MPI applications with MVAPICH on 
PSM is available in the QLogic OFED User guide available at:

http://driverdownloads.qlogic.com/QLogicDriverDownloads_UI/SearchByProduct.aspx?ProductCategory=301&Product=1116&Os=65

Submitting comments, bugs, questions
------------------------------------
The best way to report bugs, send comments or ask questions is to sign up to the
developers mailing list: psm-devel@qlogic.com. Because of spam only subscribers
are allowed to post to the list. Please ensure that you subscribe with and post
from the same email address else posts will be blocked as spam. To subscribe 
send the following in the BODY of an email to majordomo@qlogic.com:

  subscribe psm-devel

Majordomo will reply back with instructions on how to confirm your subscription.
The mailing list can be used to report bugs, send comments and ask questions.

