ASTM Phantom Test

This software is a module for 3D Slicer to perform the accuracy test of a tracking system as described in the standard ASTM F2554 in its latest version (2022).

This test relies on a calibration object, hereafter referred to as the "phantom", of known dimensions measured by a CMM. Performing measurements on such a phantom provides a reliable assessment of the tracking system accuracy and precision.

Table of contents

• Material
• Required software
  • PLUS Toolkit
  • 3D Slicer
• Parameter files
  • Pointer file
  • Working volume file
  • Phantom file
• Usage
  • Setup
  • Launching the software
  • Running the tests
  • Getting the results
• Troubleshooting

Material

To perform the test, the following items are necessary:

• A phantom with a distribution of divots and a reference tracked array rigidly attached. See the standard for requirements and recommendations regarding phantom design and manufacturing.
• The tested tracking system, typically including
  • A pointer composed of a tracked array and a shaft with a pointy end fitting the divots.
  • A tracker which spatially locates the various arrays aforementionned. Various technologies can be utilized (optical, magnetic, etc).
  • A computer to receive and analyze the tracking data. The present software is to be installed on this computer, alongside the various supporting tools and libraries. Since this software also provides visual guidance to perform the test, a computer monitor is recommended.

Required software

PLUS Toolkit

PLUS Toolkit is a free, open-source set of software tools for Computer-Assisted Surgery, which includes a wrapper for SDK's from many manufacturers of tracking systems. This enables a standardization of the tracking data streaming from the tracker to the computer, thanks to an OpenIGTLink server (i.e. the Plus Server).

Installation

Pre-built installers for more than twenty systems are available for Windows from the Download page. To know which installer to choose, the user may refer to the table at the bottom of that page. The version must be 2.9.0.202207x or more recent. Since the 2.9 release is not stable yet, one can access it from the Latest Development Snapshot.

⚠️ Beside the wrappers, PLUS Toolkit also includes the SDK for most systems. For Atracsys trackers though, the SDK needs to be installed separately on the computer. Also, the Atracsys SDK version is required to be 4.5.2 or more recent.

To install on Linux or Mac OS, please refer to the Developer's guide.

Configuration file

The Plus Server relies on a configuration file, in XML format, to set the tracking parameters. These configuration files are located in /config for pre-built versions and in /PlusLibData/ConfigFiles for compiled versions. Plus Toolkit already comes with two examples of configuration file for the ASTM Phantom Test: one for Atracsys trackers (PlusDeviceSet_Server_AstmPhantomTest_Atracsys.xml) and one for NDI Polaris trackers (PlusDeviceSet_Server_AstmPhantomTest_NDI_Polaris.xml). The user is welcome to duplicate and customize these configuration files to meet their needs, but some parameters, listed below, must be correctly set in order to successfully run the ASTM Phantom Test.

• Device > Type describes what tracker is used and must be in accordance with the version of PLUS Toolkit installed. For more information on what type to use, please refer to the Configuration page.

• The Device > ToolReferenceFrame gives a name to the reference frame of the tracking coordinates, typically the tracker.

• DataSources provides the inputs i.e. the description of the tracked arrays. For our ASTM Phantom Test, two are necessary:

  • the array attached to the phantom with the id Phantom. This id must remain unchanged.
  • the array of the pointer with the id Pointer. This id must remain unchanged. Each array can be for ACTIVE tracking (with emitting markers) or PASSIVE tracking (with reflective markers). GeometryFile is set as the path (relative to the xml configuration file) of the geometry of the array (which enables the detection and tracking). For Atracsys trackers for example, the geometry is contained in an .ini file as such:

  [geometry]
  count=4
  id=101
  [fiducial0]
  x=-24.3012
  y=-29.5377
  z=0.14917
  [fiducial1]
  x=-66.7241
  y=14.0559
  z=-6.80029
  [fiducial2]
  x=20.8942
  y=3.55241
  z=1.4325
  [fiducial3]
  x=70.131
  y=11.9294
  z=5.21875

  However, each manufacturer has its own format for geometry files (e.g. NDI uses binary ROM files), so please refer to the tracker's manual for more information.

• The OutputChannel will be composed of the two streams of our DataSources so Phantom and Pointer

• The Plus Server is an OpenIGTLink server, which sends messages of type TRANSFORM. These represent 4x4 matrices describing the transform from one item to another. For the ASTM Phantom Test, we need three transforms:

  • the transform from the pointer (Pointer) to the phantom (Phantom), which results in PointerToPhantom
  • the transform from the phantom (Phantom) to the tracker (Tracker), which results in PhantomToTracker
  • the transform from the pointer (Pointer) to the tracker (Tracker), which results in PointerToTracker
    DO NOT change the names of the output transforms, as these are hardcoded in the Slicer module.

  The option SendValidTransformsOnly has to be set to FALSE, which helps the device detect when an item gets out of tracking (the corresponding transform attribute then goes from "VALID" to "MISSING").

• Other device-specific parameters can be set, depending on the manufacturer and/or the tracker. For example, the origin of the tracker coordinate system is supposed to be at the center of the device. For Atracsys trackers, the device parameter SymmetriseCoordinates="1" is then required to move the origin of the tracker from the left camera to the center.

For a better understanding of Plus configuration files, their structure is detailed in the Configuration page of the User Manual.

3D Slicer

3D Slicer (or "Slicer" for short) is a free, open-source software dedicated to medical image analysis. One of strengths of Slicer is its modularity, as it is possible to develop extensions to further expand its features or use its platform to create a dedicated software. The latter is the approach chosen for this project. Our Slicer module sets up an OpenIGTLink client, connects to the Plus Server and receives and analyzes the tracking data to perform the ASTM Phantom Test.

Installation

From Slicer's download page, download and install the latest stable release corresponding to the computer OS (5.0.3 or above). Then, run Slicer and click on Install Slicer Extensions from the welcome panel.

In the Extensions Manager, head to the Install Extensions tab (1) and browse or look for the AstmPhantomTest extension (2). Once found, simply click on Install (3) and restart Slicer as required (4).

The ASTM Phantom Test module is now installed and a shortcut for it can be set in the main menu bar by returning to Application Settings > Modules.

Keeping the module up-to-date

It is important to keep the module up-to-date in order to benefit from future improvements and bug fixes. Thus, it is recommended to set Slicer to automatically check (and even install) module updates for you. To do this, head back to the Extensions Manager, click on the wrench icon (1) and check the desired options (2).

Doing so, updates will be notified as a yellow dot on the Extensions Manager button (see below on the left). Browsing the extensions will then show you a restart prompt but also offer the possibility to cancel the update (see below on the right).

If you want to manage the updates yourself, you can also manually check and install the update from the Manage Extensions tab (1), click on Check for updates (2) and if an update is available, it will show up along with an Update button (3).

⚠️ Updating the module will require to restart Slicer to take effect.

Parameter files

The module relies on several parameter files to accomodate for the used hardware. Those parameter files may be duplicated and customized to accomodate for specific tools or requirements. The parameter files are located in the module folder, whose path (hereafter coined module_path) can be found via Application Settings > Modules.

Pointer file

Located in module_path\Resources\ptr, this parameter file contains the maximum tilt angle (MAXTILT, in degrees) beyond which the pointer manufacturer does not guarantee tracking. This value typically depends on the type of tracking technology and that of the fiducials/markers attached to the pointer. The parameter file also describes the pointer rotation axes (ROLL, PITCH, YAW) in the coordinate system of the pointer. ⚠️ These axes need to match those set for the working volume (see example in the Troubleshooting section). Finally, the file contains the pointer height (HEIGHT, in mm) to accomodate for pointer tracking while the phantom nears the top of the working volume. This consists in placing the top target location for the phantom (TL) with a downward offset of HEIGHT + the elevation of the highest divot (e.g, #47) from the central divot (CTR). If HEIGHT is set to 0, then there is no compensation.

Working volume file

Located in module_path\Resources\wv, this parameter file contains various information:

• the coordinates of the various locations that the phantom should be placed at in the working volume. Beside the center location (CL), all other locations lie at the edges of the working volume, as described in the ASTM standard. The four other locations are placed on the outer boundaries of the back plane. BL is located at the bottom, TL at the top, LL at the left and RL at the right. All these coordinates are expressed in the referential of the tracker.

• the actual working volume is described by the NODES coordinates (again in the tracker referential frame). The working volume is assumed to be composed of a succession of quadrilateral planes, but their number can vary.

• the moving tolerance is the threshold that separates actual pointer motion from the slight "wiggle" that typically occurs with most tracking technologies even when the pointer tip is static. Since the magnitude of this wiggle often depends on the distance to the tracker, the range for the moving tolerance is given by two extreme values. MOVTOLMIN sets the minimum threshold when the pointer is the closest possible to the tracker (e.g, 0.4mm at 920mm in depth) and MOVTOLMAX the maximum when the pointer is the farthest possible (e.g, 1.0mm at 2850mm in depth). The moving tolerance is automatically set by the module during the tests within the provided range. Nonetheless, if the user experiences trouble acquiring a divot because the program keeps detecting tip motion when there is none, the moving tolerance can be manually increased live (see troubleshoot).

• the working volume file also describes the pointer rotation axes (ROLL, PITCH, YAW) in the coordinate system of the tracker. ⚠️ These axes need to match those set for the pointer (see example in the Troubleshooting section).

• the model name of the tracker (MODEL) is also given in the working volume file. The name has to match one of the models included in module_path\Resources\models. For example, MODEL = ftk500 will prompt the software to load the 3D model ftk500_RAS.stl.

Phantom file

Located in module_path\Resources\gt, this parameter file describes the divots on the phantom and their use. All the divots coordinates are listed (POINT id,X,Y,Z) and given in the referential frame of the phantom. This referential frame is defined by the first three divots provided by REF in the following order O, X and Y.

The sequence of divots used for the multi-point test is given by SEQ and the id of the central divot (used for all the other tests) is given by CTR.

REF = 1 19 18
SEQ = 20 18 12 6 1 7 13 19 25 22 28 30 32 34 37 35 40 42 44 46
CTR = 20
MODEL = phantom2022

POINT 1
X 0.00
Y 0.00
Z 0.00
POINT 2
X 0.01
Y 14.44
Z 0.00
POINT 3
X 14.45
Y 0.00
Z -0.01
...

The model name of the phantom (MODEL) is also given in the parameter file. The name has to match one of the models included in module_path\Resources\models. For example, MODEL = phantom2018 will prompt the software to load the 3D model phantom2018_RAS.stl. Two models are already included, corresponding to the last two revisions of the standard from 2018 and 2022.

Usage

Setup

The tracker and the phantom are set up so that:

1. the tracker is installed according to the manufacturer's specifications (typically in orientation). Then, it is connected to the computer and turned on. For more details about this connection (e.g. cabling, network configuration, drivers), please refer to the tracker manual.
2. the phantom is inially placed within the first half of the working volume facing the tracker.
3. during most measurements, both the phantom and the pointer must keep their tracked array (approximately) oriented towards the tracker so as to ensure an optimal tracking accuracy. The only exception for the pointer is during the rotation tests. The only exception for the phantom is during the single point tests with extreme orientations.
4. the operator can manipulate the pointer with respect to the phantom with ease, without occluding the device lines of sight. The operator should also be able to monitor the progress of the tests on the screen monitor. See example of setup below.
5. if possible, sound should be enabled on the computer as audio clues are given to signal progression during the tests, which sometimes alleviates the need to look at the screen (especially during the multi-point test).

Launching the software

Plus Server

The Plus Server is launched by running PlusServerLauncher.exe from the bin repository of PLUS Toolkit.

Once launched as illustrated below, the path to the configuration files folder (1) is automatically selected as the one included in the PLUS Toolkit repository. If the requested configuration file is located elsewhere, the user may manually select another folder and refresh it thanks to the two buttons next to the input field.

Then, the user shall select from the dropdown menu (2) the appropriate configuration file, correctly formatted as explained in the Configuration file subsection. Finally, the server is launched by click on the Launch Server button (3) and the message Connecting to devices. appears.

If the server has sucessfully started, it appears as shown below.

Slicer

Start Slicer by clicking a shortcut to it. If no shortcut to Slicer was created during its installation, the application may still be launched by running the Slicer executable from its repository.

Once Slicer is started, the ASTM Phantom Test module can be accessed via the dropdown menu in the Tracking category or via the shortcut in the menu bar (if previously created).

Running the tests

Preliminary steps

When the module starts, a dialog window pops up and the user has to select the folder where the output files (including the log) will be saved.

On the left panel, the tracking status of the pointer and the reference array (attached to the phantom) is displayed. These labels correspond to the three transforms streamed by the Plus Server and show up either as OK or MISSING.

The program won't start until both the pointer and the phantom reference array are being tracked (even briefly), i.e. until all transforms are displayed as OK.

Once the program is started, the working volume and the target locations are previewed from the top and the front of the scene. The choice in working volume may be changed while setting the parameters in the left panel.

The first parameter the operator is expected to input is the tracker serial number (1).

Once filled in, the other parameters are unlocked and the user can choose the appropriate pointer file (2), phantom file (3) and working volume file (4). The user may also choose the point acquisition mode (5), the recalibration option (6) and has to input his id in the Operator field (7).

There are three point acquisition modes available:

• 1-frame: the point coordinates are those measured in a single frame in the middle of the acquisition. In this mode, the acquisition length is set to 0.5 sec.
• mean: the point coordinates are the mean of those measured across N frames (default is 30). In this mode, the point acquisition lasts N frames.
• median: the point coordinates are the median of those measured across N frames (default is 30). In this mode, the point acquisition lasts N frames.

The option of Recalibrate at each location is also offered, which implies that a new calibration be performed at each selected location in the working volume. If this option is not checked, the initial calibration will be used for the remainder of the procedure. Note that, as per the standard, recalibrations are not required but recommended.

Filling in the operator id triggers the initial phantom calibration, which estimates the geometrical relationship between the coordinate system of the phantom (and thereby its divots) and the reference array attached to it. To perform the calibration, the user needs to successively pick all the divots indicated in the phantom file by REF with a minimum of three divots (more can be added beyond the first three defining the phantom coordinate system).

For any point acquisition, the targets are indicated by a red sphere and their id. Once the pointer hit the target (the correct divot), the point acquisition starts, indicated by the target becoming smaller and greener. The duration of the acquisition depends on the point acquisition mode. The point acquisition is done when the target becomes large and green, and a "pop" sound is played. Removing the pointer from the divot then prompts the program to show the next target, if any. Removing the pointer before the point acquisition is done will reset the current target.

Once the calibration is over, the choice in locations is unlocked (8). By default, all five locations in the working volume are enabled as recommended by the standard, but the user can, at any time during the session, disable and skip undone locations if desired. Likewise, by default all seven tests are enabled (9) as recommended by the standard, but the user can disable certain tests before the phantom hits a target location in the working volume.

The moving tolerance slidebar (10) allows the user to monitor and adjust the sensibility to pointer motion. Finally, the operator may interact with the 3D scene with the mouse and the optimal view can be automatically reset with the Reset Camera button (11).

To start the tests, the user must first place the phantom at one of the designated locations in the working volume. To help the user in this task, the interface switches back to "working volume guidance". In the top and front views (left and right respectively), the user can now see where the phantom is located with respect to the target locations. Once the phantom is stabily placed at one of the enabled locations, a specific sound is played and the first test starts.

Single Point Tests

Referred to as the Single Point Accuracy and Precision Test in the ASTM standard, this test aims at assessing the performance of the tracking system for single point measurements. To do so, the user must repeatedly acquire the central divot for a certain number of times. The progression of the test is displayed in the top right corner.

The single point test is to be performed with three different phantom orientations: Extreme Left, Extreme Right and Normal. The two extreme orientations correspond to the most extreme left and right rotations of the phantom while maintaining tracking of the attached reference element. The normal orientation is with the phantom rotated such that the attached reference element is optimally located by the tracker.

Rotation Tests

As described in the ASTM standard, these Rotation Tests aim at assessing the stability in measurement of a single point while the pointer rotates around specific axes (roll, pitch, yaw). The definition of these axes are defined in the pointer file and in the working volume file in their respective coordinate system.

To start the test, the user must lodge the pointer in the central divot, which will toggle the display of the pointer angles with respect to each rotation axes. The current rotation axis to be measured is highlighted by the symbols > and <. The pointer must be rotated only around that specific axis, i.e. the other angles shall remain as close to 0 as possible during the test.

To start the actual measurements, the user must first rotate the pointer as detailed above until it is not tracked anymore. Then, performing a counter rotation, the pointer is tracked again, which will trigger the program to begin saving the measurements. The user shall continue rotating that way until they reach the opposite end of the angular range, i.e. until the tracking stops again, which will stop the measurements.

This routine is to be repeated for each axis corresponding to the enabled rotation tests in the parameter panel.

Certain rotations may lead to the pointer colliding with parts of the phantom or escaping the divot before it gets out of tracking. It is preferable to prematurly stop the measurements (and hence the rotation test) slightly before reaching these dead-ends. To artificially set the pointer out of tracking, swiftly occlude the pointer with the hand while remaining in the central divot.

Multi-Point Test

As explained in the ASTM standard, this test consists in picking various points on the phantom and comparing the resulting point cloud to the ground truth given by the phantom file. For a same set of acquisitions, this comparison is done in two ways: comparing point-to-point distances and actual point cloud registration. The results constitute an assessment of the accuracy of the tracking system in multi-point picking.

To perform the test, the user only needs to measure with the pointer a sequence of points, defined in the phantom file by SEQ. During the test, the current target is shown in red and the acquired points become green.

Getting the results

Once all the enabled tests for all the enabled locations are done, the program generates various files in the output folder.

• a report in HTML format (to be open with any internet browser), that contains all the statistical analysis of the measurements for each test.
• a json file containing all the parameters and the actual measurements. This is meant to perform some more analysis if desired.
• a log file, which contains all the events that occured during the session. The log is written in real-time, so even if the program crashes, the events are saved. The log file is common for all the sessions performed on a same day.

Besides, a Matlab script (AnalyzePhantomJSON.m) is also provided that parses all the recorded measurements and display the results.

Troubleshooting

1. I'm having trouble acquiring a divot although my pointer is static (i.e. placed in a divot).

   • Check that there is no occlusion of any of the markers.
   • Check the environment for interferences. For optical systems, use the manufacturer's tools to visually assess in the camera images the absence of reflections that could affect the detection of the markers, typically on the phantom.
   • Increase the moving tolerance by increments of 10% of the total range. If the maximum tolerance is reached, you may slightly increase that maximum value in the working volume file and restart Slicer. However, increasing the moving tolerance further will hinder the reliability of the tests.

2. The Plus Server Launcher does not connect to my device or gives me errors.

   • Check that the installed version of PLUS Toolkit is the appropriate one for my system.
   • Check that the SDK version of my system is compatible with PLUS Toolkit. In doubt, you may contact the PLUS Toolkit development team.
   • Check that the device is powered on and connected to the computer as per the manufacturer's guidelines.
   • Check that the computer is well configured for the tracking system as per the manufacturer's guidelines (typically network configuration if connected through ethernet).
   • Check that the configuration file has been correctly written.

3. The status for the phantom and/or the pointer remains MISSING despite being visible by the tracker. => There is a problem of communication between Slicer and the Plus Server.

   • Check that the Plus Server is launched with the appropriate configuration file and successfully connected to the tracking system.
   • In Slicer, head to the Transforms module via the dropdown menu and check the Active Transform list at the top. If "PhantomToTracker" or "PointerToTracker" is missing, check that the geometry files are correct and that their paths in the configuration file are valid.

4. I can't naturally trigger the out of tracking of the pointer during the rotation tests.

   • See the recommendation for artificially ending the rotation measurements prematurly.

5. The pointer is misoriented in the rendering and/or the angle values are not near 0 when "facing" the tracker.

   • There is a mismatch between the rotation axes in the tracker's coordinate system (in the working volume file) and the rotation axes in the pointer's coordinate system (in the pointer file). Those axes should be the same in the world's coordinate system and they are important as they allow for 1) a correct interpretation of the pointer rotations with respect to the tracker and 2) a correct orientation of the 3D pointer model in the display. Below is an illustrated example of rotation axes in both coordinate systems:

6. I imported my own 3D models in STL format for the phantom, the pointer and/or tracker, and they are misoriented in the 3D scene.
   • Slicer uses the RAS convention for axes, which differs from the typical X,Y,Z spatial axes. For a correct interpretation of the STL file, it must be edited (e.g, using Notepad) and its first line shall include SPACE=RAS. For example, the first few lines of the STL can be:

   solid SPACE=RAS
    facet normal 0 0.987688 -0.156434
     outer loop
      vertex -150 50 0
      vertex 150 50 0
      ...
   

   • Finally, the model for the pointer is not meant to be replaced. If you still want to do it, you can replace the model pointer_RAS.stl by the one of your choosing, but make sure that the origin and axes are the same and that the RAS space is specified as described above.