Complex Collective Operation Patterns

TACO does not provide many of the complex collective communication patterns like all-to-all communications that can be found e.g. in the MPI standard. The reason is quite simple: the semantics of most of these operations can be directly achieved with a very few lines of code in TACO. TACO provides a powerful concept for this purpose, namely so-called group pointers that can be passed as parameters to TACO's functors.

The GroupPtr class is basically an extension of ObjectPtr and provides an implementation of TACO's collective methods. This allows us to implement collective methods that cause themselves the execution of collective methods.

To illustrate this further, consider a scan() operation that computes a partial sum for all members in the group. The following piece of code then implements the complete scan() algorithm by means of a masked collective operation:

class X: public GroupMember<X>
{
public: ...
    int getValue() const { return value; }
    bool lower(int r) { return (rank() <= r); }
    void scan(GroupPtr<X> group)
    {
        partialSum = group.reduce(
           m2f(&X::lower, rank()), // mask
           m2f(&X::getValue), // method to call
           plus<int>()); // STL reduction method
    }
private:
    int value; int partialSum;
};
...
GroupOf<X> group(...);
group.step(m2f(&X::scan, group));

We simply create a group and perform the scan in a single logical step(), which causes all group members to compute the partial sum on the group that we passed as parameter to the scan() method, i.e. the own group. This particular step() operation is a collective operation that triggers many strongly pipelined collective operations and waits until all are completed. Alternatively, we can also apply a map() operation to the group when we do not need to synchronize on the end of the complete scan. When we intend to perform many scan operations on the same group, we can improve the performance by creating an appropriate selection for each group member that holds all members with lower ranks. In this case we do not need to apply masked operations but each member operates on its own selection.

It is quite obvious, that other collective operations like all-to-all operations are very similar. Thus any arbitrary collective method can potentially implement very sophisticated collective operation patterns with modest programming effort.