Tracking Competition

Many tracking technologies for Augmented Reality have been proposed so far. Some have made their way into commercial products, some are freely available as open‐source software, and others are still in the development phase. To allow a fair comparison of the state‐of‐the art a tracking competition was organized at ISMAR’08 in Cambridge. With its success, we will be continuing this event at ISMAR 09 to help stimulate the state of the art of tracking. Tracking quality is to be evaluated on a concrete and reasonably realistic industrial task. The task is inspired by a logistics scenario where a worker is given a list of items to pick in a storage room. These items are only described by their 3D‐coordinate in a global reference coordinate frame. The worker has to be guided to the objects by visually overlaying the object location onto the worker’s view. As the objects to be picked decrease in size, and the picking areas are in different corners of a large room, both locally accurate and wide‐area tracking is needed. We make no restrictions on the type of tracking equipment used, and the competition is open to both academic and commercial participants. Participants must bring their own hardware and we will not provide software for the visualization. However, we estimate that a talented undergraduate student could write an appropriate visualization in a short time, provided the hardware, the tracking system and registration tools are available.


Whats New?

Compared to the 2008 contest, a new points system for ranking contestants is introduced. Its purpose is to make the contest both more exciting and more relevant for real‐world scenarios. Contestants can earn bonus points in a new accuracy contest and for fast contest runs. By penalizing wrongly picked (as opposed to not picked) objects, participants are encouraged to make use of tracking accuracy information.


Outcome and Prize

Compared to the 2008 contest, a new points system for ranking contestants is introduced. Its purpose is to make the contest both more exciting and more relevant for real‐world scenarios. Contestants can earn bonus points in a new accuracy contest and for fast contest runs. By penalizing wrongly picked (as opposed to not picked) objects, participants are encouraged to make use of tracking accuracy information.


When

The tracking competition takes place during the main conference (October 20—22nd). More details about the schedule will be made available after the registration deadline.


Registration

In order to register for the tracking competition, participating teams have to submit a one‐page summary of their approach. The summary should cover the following topics:

  • What tracking technology will be used
  • What needs to be installed in the room
  • A short description of the mobile setup
  • Contact Details

The deadline for proposals is July, 19th 2009. It is to be expected that only a limited number of teams can participate. The summaries are to be set to: tracking@ismar09.org


Further Information

For further information, we have created forum at http://ismar‐society.org/forum/index.php Please use this forum when asking questions about the general contest rules, so that everybody can profit from the answers. We will also post updates on the competition there. Note: You need to register before you can access the forum.

Detailed Competition Rules


1. Envirnoment

  1. The competition takes place in a large room. In order to simulate a storage room environment, the room will be separated into different areas by moveable walls, shelves or similar objects. In particular, there will be no single place from which the whole room can be observed. More details will be made available later.
  2. The organizers will define a global coordinate frame in the room using geodesic equipment. For registration with the participant’s tracking system, a number of reference points, spread throughout the whole room, will be provided. The coordinates of these reference points are provided as a text file to the participants before the setup phase. The shape of a reference point is shown in the following picture:
  3. The unit used for all coordinates is meters.
  4. Participants are not allowed to make permanent modifications (e.g. drilling holes) to the room or its equipment
  5. The room will contain a number of designated picking areas from which the objects are to be picked. The extent of these areas will be clearly marked. Participants are not allowed to place markers or any other objects into these areas. Examples of picking areas are shown in the following images:

2. Visualization System

  1. The organizers will not provide binaries or source code for visualization.
  2. The location of the object to pick must be visualized on a tracked mobile display by highlighting it in a video image or by otherwise overlaying it onto the user’s view.
  3. The location of the object must be visualized by showing a simple 3D (e.g. sphere) or 2D (e.g. crosshair, circle) object registered at the given coordinates.
  4. If the object to pick is outside the user’s view, an arrow must be displayed to indicate the direction to look to find the object.
  5. Any other type of output (e.g. textual) is not allowed. In particular, the visualization must not make use of a map of the environment.
  6. The system must provide a mechanism to select the next object, such as a button or speech input.
  7. It is allowed to integrate a simple tracking quality indicator

3. Hardware Requirements

  1. The display hardware must be mobile such that it can be moved by a single user while performing the picking task. It is not permitted to install display devices (e.g. projectors) in the environment.
  2. There are no other restrictions on the type of tracking or display equipment used.

4. Contest Procedure

  1. Participants will have some amount of time for the setup and testing. Details about the schedule will be made available after the registration deadline.
  2. The setup phase is supervised by the organizers
  3. For a dry run of their systems, participants will be provided with coordinates of a few known objects.
  4. The system will be tested by the organizers after setup, to make sure that it complies with the rules described in section 2

5. Contest Run

  1. At the beginning of the contest run, participants will receive the list of objects on a USB memory stick. The memory stick holds a Unix‐format text file named Contest.dat that contains the 3D coordinates of the objects. For each object, the file contains a line with three floating point numbers, separated by spaces. An example file is provided upon request. Participants are not allowed to display the contents of the file in any other way than described in section 2.
  2. The contest run is performed by a single person (the user) selected by the participant.
  3. Only the mobile equipment by the user may be moved during the contest run. All other equipment must stay in place.
  4. During the contest run, the user has to pick the objects in the given order.
  5. All users start the contest run at the same location and have to return to it after having picked all objects.
  6. The contest run is supervised by the organizers who will note the objects that were picked and measure the required time.
  7. The timing starts when the location of the first object is displayed at the starting location and ends when the user has returned to the starting location.
  8. The organizers will videotape the contest run to present the results to the ISMAR audience at the end of the conference.

6. Accuracy Contest

  1. The contest will include a planar area with no objects. Instead of picking an object, participants need to draw a cross at the given location.
  2. The distance to the real coordinate will be measured to create an accuracy‐ranking of the participants.

7. Determining the Winner

  1. To determine the winner, a point system is used. The team with the highest number of points wins
  2. Points will be allotted according to the following system:
    • One point for each correctly picked object
    • Two negative points for each wrongly picked (as opposed to not picked) object
    • 3 bonus points for the fastest run, 2 for the second and 1 point for the third‐fastest run
    • 3 bonus points for the winner of the accuracy contest, 2 for the second and 1 point for the third‐most accurate team
  3. The installation of equipment (such as sensors or fiducials) in the room during the contest run is penalized with two negative points.
  4. In case multiple contestants obtain the same number of points, the timing of the contest run is used to create a ranking among these contestants.

Sponsors