4.11. Execution Objects

Although passing whole arrays and cell sets into a worklet is a convenient way to provide data to a worklet that is not divided by the input or output domain, they are sometimes not the best structures to represent data. Thus, all worklets support a another type of argument called an execution object, or exec object for short, that provides a user-defined object directly to each invocation of the worklet. This is defined by an ExecObject tag in the ControlSignature.

Later in this chapter in Section 4.11.3 (Designing Execution Objects) we will see how to implement an execution object that is provided to each instance of a worklet. However, before that we will explore some of the execution objects created by other builtin VTK‑m objects such as vtkm::cont::ArrayHandle. These objects are used internally by VTK‑m when implementing the functionality of other ControlSignature arguments. They also are often used as building blocks when constructing your own execution objects.

4.11.1. Interfaces to the Execution Environment

One of the main functions of VTK‑m classes like vtkm::cont::ArrayHandle and vtkm::cont::CellSet is to allow data to be defined in the control environment and then be used in the execution environment. When using these objects with filters, worklets, or algorithms, this transition is handled automatically. However, it is also possible to invoke the transfer for a known device.

Each class may have its own functions for transferring data from control environment to execution environment. They typically take a vtkm::cont::DeviceAdapterId to specify the device and a vtkm::cont::Token to define the time during which the data must remain valid. These methods return an object that must be passed to the execution environment running on the same device to be used. We will start by describing the vtkm::cont::ArrayHandle object, which manages transferring basic arrays between environments. Most other execution objects are built from vtkm::cont::ArrayHandle objects.

The vtkm::cont::ArrayHandle class manages the transition from control to execution with a set of three methods that allocate, transfer, and ready the data in one operation. These methods all start with the prefix Prepare and are meant to be called before some operation happens in the execution environment. The methods are as follows.

The vtkm::cont::ArrayHandle::PrepareForInput() and vtkm::cont::ArrayHandle::PrepareForInPlace() methods each take two arguments. The first argument is the device adapter tag where execution will take place (see Section 2.10.1 (Device Adapter Tag) for more information on device adapter tags). The second argument is a reference to a vtkm::cont::Token, which scopes the returned array portal, as described in Section 4.11.2 (Specifying Object Scope with Tokens). vtkm::cont::ArrayHandle::PrepareForOutput() takes three arguments: the size of the space to allocate, the device adapter tag, and a reference to a vtkm::cont::Token object.

Each of these Prepare methods returns an array portal that can be used in the execution environment. vtkm::cont::ArrayHandle::PrepareForInput() returns an object of type vtkm::cont::ArrayHandle::ReadPortalType whereas PrepareForInPlace and PrepareForOutput each return an object of type vtkm::cont::ArrayHandle::WritePortalType.

Although these Prepare methods are called in the control environment, the returned array portal can only be used in the execution environment. Thus, the portal must be passed to an invocation of the execution environment.

Most of the time, the passing of vtkm::cont::ArrayHandle data to the execution environment is handled automatically by VTK‑m. The most common need to call one of these Prepare methods is to build execution objects, described below.

The following example is a contrived example for preparing arrays for the execution environment. It is contrived because it would be easier to create a worklet or transform array handle to have the same effect, and in those cases VTK‑m would take care of the transfers internally. More realistic examples are given later.

Example 4.114 Using an execution array portal from an vtkm::cont::ArrayHandle.
 1template<typename InputPortalType, typename OutputPortalType>
 2struct DoubleFunctor : public vtkm::exec::FunctorBase
 3{
 4  InputPortalType InputPortal;
 5  OutputPortalType OutputPortal;
 6
 7  VTKM_CONT
 8  DoubleFunctor(InputPortalType inputPortal, OutputPortalType outputPortal)
 9    : InputPortal(inputPortal)
10    , OutputPortal(outputPortal)
11  {
12  }
13
14  VTKM_EXEC
15  void operator()(vtkm::Id index) const
16  {
17    this->OutputPortal.Set(index, 2 * this->InputPortal.Get(index));
18  }
19};
20
21template<typename T, typename Device>
22void DoubleArray(vtkm::cont::ArrayHandle<T> inputArray,
23                 vtkm::cont::ArrayHandle<T> outputArray,
24                 Device)
25{
26  vtkm::Id numValues = inputArray.GetNumberOfValues();
27
28  vtkm::cont::Token token;
29  auto inputPortal = inputArray.PrepareForInput(Device{}, token);
30  auto outputPortal = outputArray.PrepareForOutput(numValues, Device{}, token);
31  // Token is now attached to inputPortal and outputPortal. Those two portals
32  // are guaranteed to be valid until token goes out of scope at the end of
33  // this function.
34
35  DoubleFunctor<decltype(inputPortal), decltype(outputPortal)> functor(inputPortal,
36                                                                       outputPortal);
37
38  vtkm::cont::DeviceAdapterAlgorithm<Device>::Schedule(functor, numValues);
39}

Other classes have their own Prepare- algorithms to get an execution object for a particular device. For example, all the subclasses of vtkm::cont::CellSet have a function named PrepareForInput() (e.g., vtkm::cont::CellSetExplicit::PrepareForInput() and vtkm::cont::CellSetStructured::PrepareForInput()). These take a vtkm::cont::DeviceAdapterId, a pair of tags specifying the visit and incident topology, and a vtkm::cont::Token. The returned object is the same connectivity object described in Section 4.10.3 (Whole Cell Sets).

4.11.2. Specifying Object Scope with Tokens

One of the problems with receiving execution objects from other managed objects is that it is difficult to ensure that returned execution object remains valid. For example, if you were to use vtkm::cont::ArrayHandle::PrepareForInput() to get an array portal for a vtkm::cont::ArrayHandle, that array portal would become invalid if the array were freed. If some code were to use that array portal, it would result in undefined behavior.

To prevent something like this from occurring, VTK‑m uses an object called vtkm::cont::Token. A vtkm::cont::Token is a simple non-copyable object that gets attached to other VTK‑m objects such as vtkm::cont::ArrayHandle. While the vtkm::cont::Token is attached, certain operations on the target object will block.

class Token

A token to hold the scope of an ArrayHandle or other object.

A Token is an object that is held in the stack or state of another object and is used when creating references to resouces that may be used by other threads. For example, when preparing an ArrayHandle or ExecutionObject for a device, a Token is given. The returned object will be valid as long as the Token remains in scope.

As described in Section 4.11.1 (Interfaces to the Execution Environment), whenever an execution object is created, a vtkm::cont::Token object must be provided. That vtkm::cont::Token is attached to the source object. While it is attached, the source object prevents any changes that could invalidate the execution object. For example, when a vtkm::cont::Token is used to create an array portal, while the given token object exists, the returned portal is guaranteed to be valid and any conflicting operations on the vtkm::cont::ArrayHandle will block. Once the vtkm::cont::Token is destroyed, the associated array portal may become invalid. It is best to structure code such that the token and the execution object are in the same scope.

Example 4.115 Using a vtkm::cont::Token to lock a vtkm::cont::ArrayHandle while a portal is accessing it.
 1  vtkm::cont::ArrayHandle<vtkm::FloatDefault> arrayHandle;
 2  // Fill array with interesting stuff...
 3
 4  std::vector<vtkm::FloatDefault> externalData;
 5  externalData.reserve(static_cast<std::size_t>(arrayHandle.GetNumberOfValues()));
 6  {
 7    vtkm::cont::Token token;
 8    auto arrayPortal = arrayHandle.ReadPortal(token);
 9    // token is attached to arrayHandle. arrayHandle cannot invalidate arrayPortal
10    // while token exists.
11
12    for (vtkm::Id index = 0; index < arrayPortal.GetNumberOfValues(); ++index)
13    {
14      externalData.push_back(arrayPortal.Get(index));
15    }
16
17    // Error! This will block because of token and therefore cause a deadlock!
18    //arrayHandle.ReleaseResources();
19  }
20  // Token is destroyed. We can delete the array.
21  arrayHandle.ReleaseResources();

A vtkm::cont::Token typically releases objects when it is destroyed by going out of scope. If there is a reason to detach a token before it is destroyed, this can be done with the vtkm::cont::Token::DetachFromAll() method.

void vtkm::cont::Token::DetachFromAll()

Detaches this Token from all resources to allow them to be used elsewhere or deleted.

Did You Know?

When a token is destroyed or detached, it does not immediately invalidate the execution objects it is associated with. This is both good and bad. It is good in that it simplifies code that is not managing objects on multiple threads so that scopes do not have to be continually created and destroyed. However, it is bad in that there is no automatic check that an object is being protected by a token. The code might appear to be working but then fail under different circumstances. Thus, be careful about using objects in multithreaded environments.

Common Errors

A vtkm::cont::Token adds safety to prevent an object from being invalidated while it is still being used. However, a vtkm::cont::Token will cause other code to block if necessary. This creates the possibility of deadlock, which can happen even in a single thread. Thus, a vtkm::cont::Token should live just as long as needed and no more.

4.11.3. Designing Execution Objects

It is possible to create your own execution objects. These objects can be passed to a worklet using an ExecObject tag in the ControlEnvironment. VTK‑m makes it straightforward to create your own execution objects. These execution objects will have a management object in the control environment and then will create an execution object for a particular device.

The execution object you create must be a subclass of vtkm::cont::ExecutionObjectBase.

struct ExecutionObjectBase

Base ExecutionObjectBase for execution objects to inherit from so that you can use an arbitrary object as a parameter in an execution environment function.

Any subclass of ExecutionObjectBase must implement a PrepareForExecution method that takes a device adapter tag and a vtkm::cont::Token and then returns an object for that device. The object must be valid as long as the Token is in scope.

Subclassed by vtkm::cont::AtomicArray< T >, vtkm::cont::CellLocatorBase, vtkm::cont::CellLocatorPartitioned, vtkm::cont::ColorTable, vtkm::cont::ExecutionAndControlObjectBase, vtkm::cont::PointLocatorBase, vtkm::rendering::Texture2D< NumComponents >::Texture2DSampler, vtkm::worklet::StableSortIndices::IndirectSortPredicateExecObject< KeyArrayType >, vtkm::worklet::StableSortIndices::IndirectUniquePredicateExecObject< KeyArrayType >, vtkm::worklet::internal::TetrahedralizeTables, vtkm::worklet::internal::TriangulateTables

Your execution object must implement a PrepareForExecution() method declared with VTKM_CONT. PrepareForExecution should take two arguments. The first argument is the device adapter tag (usually a vtkm::cont::DeviceAdapterId). The second argument is a vtkm::cont::Token object that should be used to scope any execution objects created internally.

The PrepareForExecution function creates an execution object that can be passed from the control environment to the execution environment and be usable in the execution environment. Any method of the produced object used within the worklet must be declared with VTKM_EXEC or VTKM_EXEC_CONT.

An execution object can refer to an array, but the array reference must be through an array portal for the execution environment. This can be retrieved from the vtkm::cont::ArrayHandle::PrepareForInput() method as described in Section 4.11.1 (Interfaces to the Execution Environment). Other VTK‑m data objects, such as the subclasses of vtkm::cont::CellSet, have similar methods.

Returning to the example we have in Section 4.10.1 (Whole Arrays), we are computing triangle quality quickly by looking up a value in a table. In Example 4.111, the table is passed directly to the worklet as a whole array. However, there is some additional code involved to get the appropriate index into the table for a given triangle. Let us say that we want to have the ability to compute triangle quality in many different worklets. Rather than pass in a raw array, it would be better to encapsulate the functionality in an object.

We can do that by creating an execution object with a PrepareForExecution() method that creates an object that has the table stored inside and methods to compute the triangle quality. The following example uses the table built in Example 4.111 to create such an object.

Example 4.116 Using ExecObject to access a lookup table in a worklet.
 1class TriangleQualityTableExecutionObject
 2{
 3  using TableArrayType = vtkm::cont::ArrayHandle<vtkm::Float32>;
 4  using TablePortalType = typename TableArrayType::ReadPortalType;
 5  TablePortalType TablePortal;
 6
 7public:
 8  VTKM_CONT
 9  TriangleQualityTableExecutionObject(const TablePortalType& tablePortal)
10    : TablePortal(tablePortal)
11  {
12  }
13
14  template<typename T>
15  VTKM_EXEC vtkm::Float32 GetQuality(const vtkm::Vec<T, 3>& point1,
16                                     const vtkm::Vec<T, 3>& point2,
17                                     const vtkm::Vec<T, 3>& point3) const
18  {
19    return detail::LookupTriangleQuality(this->TablePortal, point1, point2, point3);
20  }
21};
22
23class TriangleQualityTable : public vtkm::cont::ExecutionObjectBase
24{
25public:
26  VTKM_CONT TriangleQualityTableExecutionObject
27  PrepareForExecution(vtkm::cont::DeviceAdapterId device, vtkm::cont::Token& token) const
28  {
29    return TriangleQualityTableExecutionObject(
30      detail::GetTriangleQualityTable().PrepareForInput(device, token));
31  }
32};
33
34struct TriangleQualityWorklet2 : vtkm::worklet::WorkletVisitCellsWithPoints
35{
36  using ControlSignature = void(CellSetIn cells,
37                                FieldInPoint pointCoordinates,
38                                ExecObject triangleQualityTable,
39                                FieldOutCell triangleQuality);
40  using ExecutionSignature = _4(CellShape, _2, _3);
41  using InputDomain = _1;
42
43  template<typename CellShape,
44           typename PointCoordinatesType,
45           typename TriangleQualityTableType>
46  VTKM_EXEC vtkm::Float32 operator()(
47    CellShape shape,
48    const PointCoordinatesType& pointCoordinates,
49    const TriangleQualityTableType& triangleQualityTable) const
50  {
51    if (shape.Id != vtkm::CELL_SHAPE_TRIANGLE)
52    {
53      this->RaiseError("Only triangles are supported for triangle quality.");
54      return vtkm::Nan32();
55    }
56    else
57    {
58      return triangleQualityTable.GetQuality(
59        pointCoordinates[0], pointCoordinates[1], pointCoordinates[2]);
60    }
61  }
62};
63
64//
65// Later in the associated Filter class...
66//
67
68    TriangleQualityTable triangleQualityTable;
69
70    vtkm::cont::ArrayHandle<vtkm::Float32> triangleQualities;
71
72    this->Invoke(TriangleQualityWorklet2{},
73                 inputDataSet.GetCellSet(),
74                 inputPointCoordinatesField,
75                 triangleQualityTable,
76                 triangleQualities);