
What is EZTrace?
=============================================
EZTrace is a tool that aims at generating automatically execution traces from
HPC (High Performance Computing) programs. It generates execution trace files
that can be interpreted by visualization tools such as ViTE.


Requirements
=============================================
In order to run EZTrace, you need the following software:

  * autoconf 2.63;

  * libelf or libbfd. Otherwise, only eztrace.old would be installed and the
    functionality would be limited, see the FAQ section.
    On Debian, libelf can be installed from command line by the following
    command: apt-get install libelf-dev

  * [Optional] Any MPI implementation.

Getting EZTrace
=============================================
* You can get the latest stable release on EZTrace website:
  http://eztrace.gforge.inria.fr/

* Current development version is available via GIT
  git clone git://scm.gforge.inria.fr/eztrace/eztrace.git

After getting the latest development version (from GIT), you need to run
'./bootstrap' and only then build the tool.


Building EZTrace
=============================================
At first, you need to configure EZTrace by invoking the configure script:

$ ./configure --prefix=<WHERE_YOU_INSTALL_EZTRACE>

Options to configure. You can pass several options to the configure script for
specifying where it should find the needed libraries:
--with-litl=$LITL_ROOT or ARG(=no): specify where LiTL is installed
--with-gtg=$GTG_ROOT: specify where GTG is installed.
--with-mpi=$MPI_ROOT: specify where MPI is installed. The mpi.h file must be
  located in the $MPI_ROOT/include/ directory.
--with-mpi-include=<PATH_TO_MPI.H>: specify the directory that contains the
  mpi.h file.
--with-papi=$PAPI_ROOT: specify where PAPI is installed. The papi.h file must be
  located in the $PAPI_ROOT/include directory and libpapi should be in 
  $PAPI_ROOT/lib

Once EZTrace is configured, just type:

$ make
$ make install


Using EZTrace
=============================================

Choosing the functions to instrument
------------------------------------
First, you should select the functions you want to instrument in your
application. For this, you can set the EZTRACE_TRACE environment variable to the
list of plugins that should be used:

$ export EZTRACE_TRACE="pthread mpi"

You can get the list of available plugins with the eztrace_avail command:

$ eztrace_avail
3       stdio   Module for stdio functions (read, write, select, poll, etc.)
2       pthread Module for PThread synchronization functions (mutex, semaphore,
                spinlock, etc.)
6       papi    Module for PAPI Performance counters
1       omp     Module for OpenMP parallel regions
4       mpi     Module for MPI functions
5       memory  Module for memory functions (malloc, free, etc.)

You can get the list of selected modules with the eztrace_loaded command:
$ export EZTRACE_TRACE="pthread mpi"
$ eztrace_loaded
2       pthread Module for PThread synchronization functions (mutex, semaphore,
                spinlock, etc.)
4       mpi     Module for MPI functions

NB: you can also select the modules to load when invoking eztrace using the -t
option. For instance:

$ eztrace -t "memory pthread" ./my_program

Creating execution traces
-------------------------
Once the list of plugins is selected, you can run your application with eztrace.
For example:

$ eztrace ./my_program my_arg1 my_arg2

This command line executes your program and generates a trace file in the /tmp
directory (usually the file is named /tmp/<username>_eztrace_log_rank_#rank).


Using EZTrace with MPI
----------------------
EZTrace needs to instrument each MPI process. Thus, you can run the following 
command:

$ mpirun -np 16 eztrace ./my_program

When your application ends. Each process writes a file named
/tmp/<username>_eztrace_log_rank_<rank>.


Changing The output directory
-----------------------------
By default, each process saves its trace in the local /tmp directory. You can
change this by setting EZTRACE_TRACE_DIR or by using the -o option in eztrace:
$ export EZTRACE_TRACE_DIR=$HOME/traces
or
$ mpirun -np 16 eztrace -o $HOME/traces ./my_program


Merging execution traces
------------------------
Once the execution traces are recorded, you can merge and convert them into a
file format that can be read by your visualization software:

$ eztrace_convert -o my_paje.trace /tmp/<username>_eztrace_log_rank_0
/tmp/<username>_eztrace_log_rank_1

This converts the trace files into the Paje format. If gtg is installed with OTF
support (this is enabled by default), you can choose to convert into the OTF
file format with the -t OTF option:

$ eztrace_convert -t OTF /tmp/<username>_eztrace_log_rank_0
/tmp/<username>_eztrace_log_rank_1


Filtering events
----------------
You can select the plugins to use for the conversion phase by using the
EZTRACE_TRACE variable: if your traces contains MPI and OpenMP events, setting
EZTRACE_TRACE to mpi and calling eztrace_convert creates an output trace that
only contains MPI events.


Computing statistics
--------------------
Instead of creating a merged trace file, you can tell EZTrace to compute
statistics on the recorded traces:

$ eztrace_stats /tmp/<username>_eztrace_log_rank_0
/tmp/<username>_eztrace_log_rank_1
[...]
PThread:
-------
6 locks acquired

MPI:
---
27 messages sent
MPI_RECV:10 calls
MPI_BARRIER:11 calls
163 events handled


Defining custom plugins
------------------------
Since EZTrace works with plugins, you can create one and instrument the
functions that you want. An example of plugin is available in the example
directory.

Once your plugin is created, you should tell EZTrace where to find it. For this,
just set the EZTRACE_LIBRARY_PATH to the appropriate directory(-ies):

$ export EZTRACE_LIBRARY_PATH=<PATH_TO_PLUGIN1>:<PATH_TO_PLUGIN2>
$ eztrace_avail
  3       stdio   Module for stdio functions (read, write, select, poll, etc.)
  2       pthread Module for PThread synchronization functions (mutex,
                  semaphore, spinlock, etc.)
  6       papi    Module for PAPI Performance counters
  1       omp     Module for OpenMP parallel regions
  4       mpi     Module for MPI functions
  5       memory  Module for memory functions (malloc, free, etc.)
  99 plugin1 Example module for libplugin1
  98 plugin2 Example module for the plugin2 library

You can also use the -l option when invoking eztrace:
$ eztrace -l <PATH_TO_PLUGIN1>:<PATH_TO_PLUGIN2> ./my_program

Generating custom plugins
--------------------------
You can generate one plugin and instrument the functions you want to.
In order to generate your plugin, you need to create a file containing:

* The name of the library you want to trace (libNAME.so)
* A brief description of the library (optional)
* An ID to identify the module (0? is reserved for eztrace internal use. Thus,
  you can use any between 10 and ff)
* The prototype of the functions you want to instrument

Basically, the file should look like that:

BEGIN_MODULE
NAME example_lib
DESC "module for the example library"
ID 99
int example_do_event(int n)
double example_function1(double* array, int array_size)
END_MODULE

Now use eztrace_create_plugin to generate the plugin source code:

$ eztrace_create_plugin example.tpl
New Module
Module name: 'example_lib'
Module description: '"module for the example library"'
Module id: '99'
        emulate record_state for 'example_do_event'
Function 'example_do_event' done
        emulate record_state for 'example_function1'
Function 'example_function1' done
End of Module example_lib

The source code is generated in the output directory. Just type:

$ make

Now set the EZTRACE_LIBRARY_PATH to the appropriate directory and you are good
to go.

You can also specify (in the example.tpl file) the way a function is depicted in
the output trace. For instance:

int submit_job(int* array, int array_size)
BEGIN
 ADD_VAR("job counter", 1)
END

Specifies that when the submit_job function is called, the output trace should
increment the "job counter" variable. You can now track the value of a variable!

The test/module_generator directory contains several scripts that demonstrate
the various commands available.


Environment variables
=====================
Here is a list of the environment variables that you can use for tuning EZTrace.

* General-purpose variables:
  - EZTRACE_TRACE_DIR: specifies the directory in which the traces are created
    (default: /tmp). You can also use the -o option in eztrace.
  - EZTRACE_LIBRARY_PATH: specifies the directories in which eztrace can find
    eztrace modules (default: none) You can also use the -l option in eztrace.
  - EZTRACE_TRACE: specifies the list of eztrace modules to load (default: the
    list of available modules)  You can also use the -t option in eztrace.
  - EZTRACE_BUFFER_SIZE: specifies the size of the buffer in which eztrace
    stores events (default: 32MB)
  - EZTRACE_FLUSH: specifies the behavior of eztrace when the event buffer is
    full. If set to 1, the buffer is flushed. This permits to record trace
    larger than EZTRACE_BUFFER_SIZE, but this may impact the application
    performance. If set to 0, additional event won't be recorded. The trace is
    thus truncated, but there is no impact on performance. (default: 0).
    You can also enable buffer flushing with the -f option in eztrace.
  - EZTRACE_SIGALARM: ask EZTrace to stop the application every x ms in order to
    collect information (such as hardware counters). (default: 0)

* Error-handling variables
  - EZTRACE_NO_SIGNAL_HANDLER: disables eztrace signal handling (default: 0)

* Hardware counters-related variables:
  - EZTRACE_PAPI_COUNTERS: selects hardware events to trace using the PAPI
    library, e.g. export EZTRACE_PAPI_COUNTERS="PAPI_L3_TCM PAPI_FP_INS". Please
    note that the list of supported events as well as the number of events,
    which can be traced simultaneously, vary depending on the processor type.
    This information can be retrieved using 'papi_avail' and the processor
    documentation.
  - EZTRACE_PAPI_SAMPLE_INTERVAL: select the minimum interval (in microseconds)
    between each check of the PAPI counters (default: 100)

* MPI-related variables:
  - EZTRACE_MPI_DUMP_MESSAGES: tells EZTrace to dump the list of messages into
    a file. You can then compute your own statistics on MPI messages.
* CUDA-related variables:
  - EZTRACE_CUDA_CUPTI_DISABLED: disable the use of CUPTI in EZTrace. This
    option disables the recording of events that happen on the GPU.
* submodules variables:
  - EZTRACE_USE_GETCPU: enable/disable the getcpu module (default: disabled)
  - EZTRACE_GETCPU_INTERVAL: specify the interval (in microsecond) between each
    probe of the getcpu module (default: 10000)

Known limitations
=================

  * The CUDA module has not been tested on multi-GPU architecture. It is likely
    to fail in such situation
  * If EZTrace is compiled with a particular MPI implementation (OpenMPI for
    instance) it won't work if you run your application with another (MPICH2
    for instance). So make sure your application uses the same MPI
    implementation as EZTrace.

		Frequently Asked Questions
=============================================

Q: What would happen if I do not install neither libelf nor libbfd before the
   installation of EZTrace?

A: In that case, the configuration of EZTrace would succeed with a warning
   regarding the missing library. And, after the installation only eztrace.old
   would be available. The EZTrace launcher (eztrace) that uses pptrace would
   not work.

Q: When I run my MPI application with EZTrace, all the processes generate
   /tmp/<username>_eztrace_log_rank_1 files. What's going wrong?

A: This happens when EZTrace fails to intercept calls to MPI_Init (or 
   MPI_Init_thread). There may be several reasons for that:
     * EZTrace MPI module was not compiled. For intercepting calls to MPI
       functions, you need the MPI module in your installation (look for the
       $EZTRACE_ROOT/lib/libeztrace-autostart-mpi.so file). If you don't see
       that file, it means that something went wrong during the configuration
       of EZTrace, so check for errors or warnings in you config.log file.
     * You specified the list of modules to use and the MPI module wasn't in
       there. Check your EZTRACE_TRACE variable or use eztrace_loaded.

   If you still experience problems, please contact EZTrace development team so
   that we can fix your problem.

Q: What if I don't want to trace my whole application, but only a small part of
   it?

A: Then, you can call eztrace_start() and eztrace_stop() in your code to specify
   the part to be traced. You will need to #include <eztrace.h> and to link with
   libeztrace. Then, just run your application as usual
   (i.e. $ ./my_program my_arg1)

Q: I need to trace my program while using gdb, how can I do that?

A: Just add the -d option to eztrace to enable gdb:
   $ eztrace -d ./my_program my_arg1 my_arg2
   Please note that this should be applied only when a bug occurs while using
   EZTrace.

Q: I want my trace to be saved in a specific directory, how can I do that?

A: By default, EZTrace saves the trace in the /tmp directory. You can change
   this by setting EZTRACE_TRACE_DIR or by providing the -o option to eztrace.

Q: What if I don't care about OpenMP and I only want to see MPI communication?

A: You can set EZTRACE_TRACE to the list of "modules" you want to activate.
   By default, all the available modules are enabled, but you can tell EZTrace
   to trace only MPI or OpenMP functions:

   $ export EZTRACE_TRACE=mpi
   $ export EZTRACE_TRACE=omp
   $ export EZTRACE_TRACE="omp mpi"

Q: Can EZTrace generate an OTF trace file so that I can visualize it with
   Vampir?

A: Yes, since EZTrace relies on GTG for writing output traces, it can generate
   OTF trace files. When converting the trace with eztrace_convert, just add the
   -t OTF option:

   $ eztrace_convert -t OTF /tmp/<username>_eztrace_log_rank_0
     /tmp/<username>_eztrace_log_rank_1

Q: I don't have any error during installation, but I can't find the eztrace
   program. All I've got is an eztrace.preload file. What happened ?

A: The eztrace program relies on libelf or libbfd for instrumenting applications.
   If neither of these libraries were found during configure, the program is not
   created.
   You can either fall back to using eztrace.old (in that case, only
   instrumentation through shared libraries is supported); or you can install
   either libelf (on Debian: install the libelf-dev package) or libbfd (on
   Debian: install the binutils-dev package).

Troubleshooting
=============================================
If you encounter a bug or want some explanation about EZTrace, feel free to ask
to the developers:

* On the developer mailing list <eztrace-devel@lists.gforge.inria.fr>

* On our IRC channel:
  - server: chat.freenode.net
  - channel: #eztrace
