Collective Operations Time per Rank

Key:-c [--collop_time_per_rank]

Available options: -C [--communicators], -M [--comm_id]

This charts shows the time each rank spent in MPI collective operations. Extend the chart using available options:

Example 1 (-c)

$ aps-report ./aps_result_<postfix> -c
| Collective Operations Time and Communicators for all ranks
|--------------------------------------------------------------
| Rank     LifeTime(sec) CollOp Time(sec)  CollOp Time(%)
|---------------------------------------------------------
  0003              3.48             2.32           66.73
  0002              3.48             2.21           63.38
  0001              3.48             2.08           59.72
  0000              4.53             2.52           55.67
|=========================================================
| TOTAL            14.96             9.12           60.98
| AVG               3.74             2.28           60.98
|

Example 2 (-c -M)

This charts shows the time each rank spent in MPI collective operation in a specific communicator.

$ aps-report ./aps_result_<postfix> -c -M 1
| Collective Operations Time and Communicators for all ranks
|--------------------------------------------------------------
| Rank     LifeTime(sec) CollOp Time(sec)  CollOp Time(%)
|---------------------------------------------------------
  0003              3.48             2.32           66.73
  0002              3.48             2.21           63.38
  0001              3.48             2.08           59.72
  0000              4.53             2.52           55.67
|=========================================================
| TOTAL            14.96             9.12           60.98
| AVG               3.74             2.28           60.98

Example 3 (-c -C)

This chart shows the time spent by each MPI collective operation in all communicators.

$ aps-report ./aps_result_<postfix> -c -C
 Collective Operations Time and Communicators for all ranks
|--------------------------------------------------------------
|--------------------------------------------------------------------------------------------------------------------------
|           Function        Time(sec)          Time(%)       Volume(MB)        Volume(%)            Calls    AvrMsgSize(B) 
|--------------------------------------------------------------------------------------------------------------------------
       MPI_Allreduce             3.61            24.11          4157.97            30.00           198016            22018
        MPI_Alltoall             2.99            20.00          4157.99            30.00           219396            19873
          MPI_Gather             1.28             8.55          2772.00            20.00           220228            13198
          MPI_Reduce             1.24             8.27          2771.97            20.00           195384            14876
           MPI_Bcast             0.01             0.04             0.00             0.00              612                2
|==========================================================================================================================
| TOTAL                          9.12            60.98         13859.93           100.00           833636            17433
|
|--------------------------------------------------------------------------------------------------------------------------
|           Function        Time(sec)          Time(%)       Volume(MB)        Volume(%)            Calls       MsgSize(B)
|--------------------------------------------------------------------------------------------------------------------------
       MPI_Allreduce             3.61            24.11          4157.97           100.00           198016              ALL
 Comm.: Id       Size
         1         4             3.11            86.11          2771.98            66.67           132192              ALL
         0         2             0.50            13.89          1385.99            33.33            65824              ALL
|--------------------------------------------------------------------------------------------------------------------------
        MPI_Alltoall             2.99            20.00          4157.99           100.00           219396              ALL
 Comm.: Id       Size
         1         4             2.63            87.74          2771.99            66.67           146264              ALL
         0         2             0.37            12.26          1386.00            33.33            73132              ALL
|--------------------------------------------------------------------------------------------------------------------------
          MPI_Gather             1.28             8.55          2772.00           100.00           220228              ALL
 Comm.: Id       Size
         1         4             0.99            77.15          2079.00            75.00           147096              ALL
         0         2             0.29            22.85           693.00            25.00            73132              ALL
|--------------------------------------------------------------------------------------------------------------------------
          MPI_Reduce             1.24             8.27          2771.97           100.00           195384              ALL
 Comm.: Id       Size
         1         4             0.96            77.61          2078.98            75.00           130256              ALL
         0         2             0.28            22.39           692.99            25.00            65128              ALL
|--------------------------------------------------------------------------------------------------------------------------
           MPI_Bcast             0.01             0.04             0.00           100.00              612              ALL
 Comm.: Id       Size
         1         4             0.01            95.90             0.00            50.21              380              ALL
         0         2             0.00             4.10             0.00            49.79              232              ALL
|==========================================================================================================================

Chart Entries

-c and -c -M Columns
-c -C Columns
Rows
Rank Rank number
LifeTime(sec) Total execution time of the rank (in seconds)
CollOpTime(sec) Time spent in MPI collective operations (in seconds)
CollOpTime(%) Percentage of the MPI collective operations time in the application lifetime
Rank Rank number
LifeTime(sec) Total execution time of the rank (in seconds)
CollOpTime(sec) Time spent in MPI collective operations (in seconds)
CollOpTime(%) Percentage of the MPI collective operations time in the application lifetime
Function Function name
Time(sec) Total time of the function execution in all ranks (in seconds)
Time(%) Percentage of the function execution time in the application lifetime
Idle(sec) The time when the collective operation was idle (in seconds)
Idle(%) Percentage of the time when the operation was idle
Volume(MB) Amount of data transferred by the function in all ranks (in megabytes)
Volume(%) Percentage of the data transferred by the function
Calls Number of calls of this function in all ranks
AvrMsgSize(B) Average size of messages
TOTAL Total values for all columns
AVG Average values for all columns