Sphero Dead Reckoning and CV Tracking Dataset

A Sphero Mini is a Bluetooth ball that can be controlled by a smartphone (or in our case a laptop). The Sphero Mini can be controlled by sending movement instructions to the Sphero consisting of a direction and speed. In this dataset, we placed a camera on top of a table to create a top-down view of the Sphero moving on the floor. The Sphero was instructed to move in a spiral trajectory from the bottom-right corner to the center of the area. The dataset contains the video recording of the Sphero moving, the input instructions given to the Sphero, and the sensor data retrieved from the Sphero. The dataset was used to evaluate the sensor fusion from various sources.

The dataset was recorded on Friday, November 27, 2020, at the Vrije Universiteit Brussel. The dataset was used in the paper: https://arxiv.org/abs/2101.05198

Area

The dataset was recorded in a 260cm (W) x 200cm (H) area on the floor. The origin (0, 0) is at the bottom-right corner of the video frames.

Trajectory

The input trajectory is defined and visualised in the input data. A spiral trajectory was given to the Sphero toy. The Sphero toy was instructed to move in a spiral trajectory from the bottom-right corner to the center of the area.

Camera

Processing

The processed data is available in the processed/ folder. The processing steps are as follows:

Video processing

Wrap the video frames to a top-down view

The video frames are wrapped to a top-down view. Yellow markers were placed on the ground to help with the wrapping process (processed/video_frames_wrapped/).

Convert the video frames to HSV

The video frames are converted to HSV to help with the object detection process (processed/video_frames_hsv/).

Convert the video frames to black and white

The (HSV) video frames are converted to black and white to help with the object detection process (processed/video_frames_bw/).

Position detection

The position of the Sphero is detected in each frame of the video recording. The X-Y coordinates of the Sphero are saved in processed/video_final.csv. Note that the origin (0, 0) is at the bottom-right corner. The position is converted from pixels (the pixel location in the video frame) to centimeters (the real-world location of the Sphero).

Sensor processing

All sensor data from the Sphero toy were processed to match the orientation of the video frames.

Files

• misc/: Images and other files that are not part of the dataset itself.
• raw/: Contains the raw data files generated by OpenHPS.
  • raw/input_frames/: Contains the raw input frames (i.e, the instructions that were given to the Sphero). Each file is a JSON file containing an OpenHPS data frame.
  • raw/sensor_frames/: Contains the raw output sensor data from the Sphero. Each file is a JSON file containing an OpenHPS data frame.
  • raw/video_frames/: Contains the individual frames of the video recording.
  • raw/output.avi: All the frames of the video recording stitched together.
  • raw/dataset_info.json: Contains information about the dataset (primarily the camera and sensor frequency).
• processed/: Contains the processed data files processed by OpenHPS.
  • processed/video_frames_wrapped/: Contains the invidiual frames of the video recording wrapped to a top-down view (step 1).
  • processed/video_frames_hsv/: Contains the individual frames of the video recording in HSV (step 2).
  • processed/video_frames_bw/: Contains the individual frames of the video recording in black and white (step 3).
  • processed/video.csv: Contains the X-Y pixel coordinates of the Sphero in each frame of the video recording.
  • processed/video_final.csv: Contains the final processed video X-Y coordinates in cm. (*.pdf, *.svg are the visualisations of these coordinates)
  • processed/input_final.csv: Contains the final processed input data. Input data are the movement instructions given to the Sphero. (*.pdf, *.svg are the visualisations of these coordinates)
  • processed/sensor_position_final.csv: Contains the position that is reported by the Sphero (internally computed).
  • processed/sensor_velocity_final.csv: Contains the position based on the velocity that is reported by the Sphero.
  • processed/final.csv: Contains the final processed data. This file contains the X-Y coordinates of the Sphero based on the video processing, input instructions, and sensor data. (*.pdf, *.svg are the visualisations of these coordinates)
• results/: Contains results and experiments based on the processed data.
• README.md: This file (source).

Columns

Raw data

input_frames/

Raw OpenHPS data frames containing the input instructions given to the Sphero. For more info check https://openhps.org.

sensor_frames/

Raw OpenHPS data frames containing the sensor output retrieved from the Sphero. For more info check https://openhps.org.

sensor_velocity_raw.csv

• timestamp: Sensor frame timestamp
• x: Raw processed position based on heading + velocity
• y: Raw processed position based on heading + velocity
• heading: Heading in degrees
• velocity: Linear velocity of the Sphero (cm/s)

sensor_position_raw.csv

• timestamp: Sensor frame timestamp
• x: Internally computed X-coordinate of the Sphero (cm)
• y: Internally computed Y-coordinate of the Sphero (cm)
• euler_x: Euler angle X (degrees)
• euler_y: Euler angle Y (degrees)
• euler_z: Euler angle Z (degrees)

Processed data

video_final.csv

Position according to the video processing

• timestamp: Video frame timestamp
• x: X-coordinate of the Sphero (cm)
• y: Y-coordinate of the Sphero (cm)

input_final.csv

Position according to the input instructions

• timestamp: Input frame timestamp
• x: X-coordinate of the Sphero (cm)
• y: Y-coordinate of the Sphero (cm)

sensor_position_final.csv

Internally calculated position by the Sphero

• timestamp: Sensor frame timestamp
• x: X-coordinate of the Sphero (cm)
• y: Y-coordinate of the Sphero (cm)

Results

• timestamp: Timestamp of the data frame
• x: X-coordinate of the Sphero (cm)
• y: Y-coordinate of the Sphero (cm)
• video: True or false if the video frames were included to compute the position (only for deadzone calculations)

License

The dataset is available under the MIT License.