Asynchonous API

Device Discovery

class pupil_labs.realtime_api.discovery.Network

Bases: object

async close()

Return type:

None

property devices

Return type:

Tuple[DiscoveredDeviceInfo, …]

async wait_for_new_device(timeout_seconds=None)

Return type:

Optional[DiscoveredDeviceInfo]

async pupil_labs.realtime_api.discovery.discover_devices(timeout_seconds=None)

Use Bonjour to find devices in the local network that serve the Realtime API.

Parameters:

timeout_seconds (Optional[float]) – Stop after timeout_seconds. If None, run discovery forever.

Return type:

AsyncIterator[DiscoveredDeviceInfo]

pupil_labs.realtime_api.discovery.is_valid_service_name(name)

Return type:

bool

Remote Control

class pupil_labs.realtime_api.device.Device(*args, **kwargs)

Bases: DeviceBase

async close()

async get_calibration()

Raises:

pupil_labs.realtime_api.device.DeviceError – if the request fails

Return type:

ndarray

async get_status()

Raises:

pupil_labs.realtime_api.device.DeviceError – if the request fails

Return type:

Status

async get_template()

Gets the template currently selected on device

Raises:

pupil_labs.realtime_api.device.DeviceError – if the template can’t be fetched.

Return type:

Template

async get_template_data(format='simple')

Gets the template data entered on device

Parameters:

format (str) – “simple” | “api” “api” returns the data as is from the api eg. {“item_uuid”: [“42”]} “simple” returns the data parsed eg. {“item_uuid”: 42}

Raises:

pupil_labs.realtime_api.device.DeviceError – if the template’s data could not be fetched

async post_template_data(template_answers, format='simple')

Sets the data for the currently selected template

Parameters:

format (str) – “simple” | “api” “api” accepts the data as in realtime api format eg. {“item_uuid”: [“42”]} “simple” accepts the data in parsed format eg. {“item_uuid”: 42}

Raises:

pupil_labs.realtime_api.device.DeviceError – if the data can not be sent. ValueError: if invalid data type.

Return type:

None

async recording_cancel()

Raises:

pupil_labs.realtime_api.device.DeviceError – if the recording could not be started Possible reasons include - Recording not running

async recording_start()

Raises:

pupil_labs.realtime_api.device.DeviceError – if the recording could not be started. Possible reasons include - Recording already running - Template has required fields - Low battery - Low storage - No wearer selected - No workspace selected - Setup bottom sheets not completed

Return type:

str

async recording_stop_and_save()

Raises:

pupil_labs.realtime_api.device.DeviceError – if the recording could not be started Possible reasons include - Recording not running - template has required fields

async send_event(event_name, event_timestamp_unix_ns=None)

Raises:

pupil_labs.realtime_api.device.DeviceError – if sending the event fails

Return type:

Event

async status_updates()

Return type:

AsyncIterator[Union[Phone, Hardware, Sensor, Recording, NetworkDevice]]

exception pupil_labs.realtime_api.device.DeviceError

Bases: Exception

class pupil_labs.realtime_api.device.StatusUpdateNotifier(device, callbacks)

Bases: object

async receive_updates_start()

Return type:

None

async receive_updates_stop()

pupil_labs.realtime_api.device.UpdateCallback

Type annotation for synchronous and asynchronous callbacks

Alias of Union[Callable[[Component], None], Callable[[Component], Awaitable[None]]]

pupil_labs.realtime_api.device.UpdateCallbackAsync

Type annotation for asynchronous update callbacks

Alias of Callable[[Component], Awaitable[None]]

pupil_labs.realtime_api.device.UpdateCallbackSync

Type annotation for synchronous update callbacks

Alias of Callable[[Component], None]

Streaming

Gaze Data

namedtuple pupil_labs.realtime_api.streaming.gaze.EyestateGazeData(x, y, worn, pupil_diameter_left, eyeball_center_left_x, eyeball_center_left_y, eyeball_center_left_z, optical_axis_left_x, optical_axis_left_y, optical_axis_left_z, pupil_diameter_right, eyeball_center_right_x, eyeball_center_right_y, eyeball_center_right_z, optical_axis_right_x, optical_axis_right_y, optical_axis_right_z, timestamp_unix_seconds)

Bases: NamedTuple

EyestateGazeData(x, y, worn, pupil_diameter_left, eyeball_center_left_x, eyeball_center_left_y, eyeball_center_left_z, optical_axis_left_x, optical_axis_left_y, optical_axis_left_z, pupil_diameter_right, eyeball_center_right_x, eyeball_center_right_y, eyeball_center_right_z, optical_axis_right_x, optical_axis_right_y, optical_axis_right_z, timestamp_unix_seconds)

Fields:

0.  x (float) – Alias for field number 0

1.  y (float) – Alias for field number 1

2.  worn (bool) – Alias for field number 2

3.  pupil_diameter_left (float) – Alias for field number 3

4.  eyeball_center_left_x (float) – Alias for field number 4

5.  eyeball_center_left_y (float) – Alias for field number 5

6.  eyeball_center_left_z (float) – Alias for field number 6

7.  optical_axis_left_x (float) – Alias for field number 7

8.  optical_axis_left_y (float) – Alias for field number 8

9.  optical_axis_left_z (float) – Alias for field number 9

10.  pupil_diameter_right (float) – Alias for field number 10

11.  eyeball_center_right_x (float) – Alias for field number 11

12.  eyeball_center_right_y (float) – Alias for field number 12

13.  eyeball_center_right_z (float) – Alias for field number 13

14.  optical_axis_right_x (float) – Alias for field number 14

15.  optical_axis_right_y (float) – Alias for field number 15

16.  optical_axis_right_z (float) – Alias for field number 16

17.  timestamp_unix_seconds (float) – Alias for field number 17

property datetime

classmethod from_raw(data)

Return type:

EyestateGazeData

property timestamp_unix_ns

namedtuple pupil_labs.realtime_api.streaming.gaze.GazeData(x, y, worn, timestamp_unix_seconds)

Bases: NamedTuple

GazeData(x, y, worn, timestamp_unix_seconds)

Fields:

0.  x (float) – Alias for field number 0

1.  y (float) – Alias for field number 1

2.  worn (bool) – Alias for field number 2

3.  timestamp_unix_seconds (float) – Alias for field number 3

property datetime

classmethod from_raw(data)

Return type:

GazeData

property timestamp_unix_ns

namedtuple pupil_labs.realtime_api.streaming.gaze.Point(x, y)

Bases: NamedTuple

Point(x, y)

Fields:

0.  x (float) – Alias for field number 0

1.  y (float) – Alias for field number 1

class pupil_labs.realtime_api.streaming.gaze.RTSPGazeStreamer(*args, **kwargs)

Bases: RTSPRawStreamer

async receive()

Return type:

AsyncIterator[Union[GazeData, DualMonocularGazeData, EyestateGazeData]]

async pupil_labs.realtime_api.streaming.gaze.receive_gaze_data(url, *args, **kwargs)

Return type:

AsyncIterator[Union[GazeData, DualMonocularGazeData, EyestateGazeData]]

IMU Data

namedtuple pupil_labs.realtime_api.streaming.imu.Data3D(x, y, z)

Bases: NamedTuple

Data3D(x, y, z)

Fields:

0.  x (float) – Alias for field number 0

1.  y (float) – Alias for field number 1

2.  z (float) – Alias for field number 2

namedtuple pupil_labs.realtime_api.streaming.imu.IMUData(gyro_data, accel_data, quaternion, timestamp_unix_seconds)

Bases: NamedTuple

IMUData(gyro_data, accel_data, quaternion, timestamp_unix_seconds)

Fields:

0.  gyro_data (Data3D) – Alias for field number 0

1.  accel_data (Data3D) – Alias for field number 1

2.  quaternion (Quaternion) – Alias for field number 2

3.  timestamp_unix_seconds (float) – Alias for field number 3

property datetime

property timestamp_unix_nanoseconds

property timestamp_unix_ns

pupil_labs.realtime_api.streaming.imu.IMUPacket_to_IMUData(imu_packet)

Return type:

IMUData

namedtuple pupil_labs.realtime_api.streaming.imu.Quaternion(x, y, z, w)

Bases: NamedTuple

Quaternion(x, y, z, w)

Fields:

0.  x (float) – Alias for field number 0

1.  y (float) – Alias for field number 1

2.  z (float) – Alias for field number 2

3.  w (float) – Alias for field number 3

class pupil_labs.realtime_api.streaming.imu.RTSPImuStreamer(*args, **kwargs)

Bases: RTSPRawStreamer

async receive()

Return type:

AsyncIterator[IMUData]

async pupil_labs.realtime_api.streaming.imu.receive_imu_data(url, *args, **kwargs)

Return type:

AsyncIterator[IMUData]

Scene Video

pupil_labs.realtime_api.streaming.video.BGRBuffer

Type annotation for raw BGR image buffers of the scene camera

class pupil_labs.realtime_api.streaming.video.RTSPVideoFrameStreamer(*args, **kwargs)

Bases: RTSPRawStreamer

async receive()

Return type:

AsyncIterator[VideoFrame]

property sprop_parameter_set_payloads

Raises:

pupil_labs.realtime_api.streaming.base.SDPDataNotAvailableError –

Return type:

Optional[List[ByteString]]

namedtuple pupil_labs.realtime_api.streaming.video.VideoFrame(av_frame, timestamp_unix_seconds)

Bases: NamedTuple

VideoFrame(av_frame, timestamp_unix_seconds)

Fields:

0.  av_frame (VideoFrame) – Alias for field number 0

1.  timestamp_unix_seconds (float) – Alias for field number 1

bgr_buffer()

Return type:

ndarray[Any, dtype[uint8]]

property datetime

property timestamp_unix_ns

to_ndarray(*args, **kwargs)

async pupil_labs.realtime_api.streaming.video.receive_video_frames(url, *args, **kwargs)

Return type:

AsyncIterator[VideoFrame]

pupil_labs.realtime_api.streaming.nal_unit.extract_payload_from_nal_unit(unit)

• prepend NAL unit start code to payload if necessary

• handle fragmented units (of type FU-A)

Inspired by https://github.com/runtheops/rtsp-rtp/blob/master/transport/primitives/nal_unit.py Rewritten due to license incompatibility.

Return type:

ByteString

Raw RTSP Data

namedtuple pupil_labs.realtime_api.streaming.base.RTSPData(raw, timestamp_unix_seconds)

Bases: NamedTuple

RTSPData(raw, timestamp_unix_seconds)

Fields:

0.  raw (ByteString) – Alias for field number 0

1.  timestamp_unix_seconds (float) – Alias for field number 1

_asdict()

Return a new dict which maps field names to their values.

_field_defaults = {}

Type:    dict

_fields = ('raw', 'timestamp_unix_seconds')

Type:    tuple

classmethod _make(iterable)

Make a new RTSPData object from a sequence or iterable

_replace(**kwds)

Return a new RTSPData object replacing specified fields with new values

property datetime

property timestamp_unix_ns

class pupil_labs.realtime_api.streaming.base.RTSPRawStreamer(*args, **kwargs)

Bases: object

Forwards all arguments to aiortsp.rtsp.reader.RTSPReader

property encoding

Raises:

pupil_labs.realtime_api.streaming.base.SDPDataNotAvailableError –

property reader

async receive()

Return type:

AsyncIterator[RTSPData]

exception pupil_labs.realtime_api.streaming.base.SDPDataNotAvailableError

Bases: Exception

exception pupil_labs.realtime_api.streaming.base._UnknownClockoffsetError

Bases: Exception

class pupil_labs.realtime_api.streaming.base._WallclockRTSPReader(*args, **kwargs)

Bases: RTSPReader

_calculate_clock_offset(sr_pkt)

absolute_timestamp_from_packet(packet)

Returns the Unix epoch timestamp for the input packet

Uses the cached clock offset between the NTP and relative timestamp provided by the RTCP packets.

get_primary_media()

get_rtpmap()

handle_rtcp(rtcp)

Handle an incoming RTP packet

relative_timestamp_from_packet(packet)

async pupil_labs.realtime_api.streaming.base.receive_raw_rtsp_data(url, *args, **kwargs)

Return type:

AsyncIterator[RTSPData]

Time Echo Protocol

Manual time offset estimation via the Pupil Labs Time Echo protocol

The Realtime Network API host device timestamps its data with nanoseconds since the Unix epoch (January 1, 1970, 00:00:00 UTC). This clock is kept in sync by the operating system through NTP (Network Time Protocol). For some use cases, this sync is not good enough. For more accurate time syncs, the Time Echo protocol allows the estimation of the direct offset between the host’s and the client’s clocks.

The Time Echo protocol works in the following way:

1. The API host (Neon / Pupil Invisible Companion app) opens a TCP server at an specific port

2. The client connects to the host address and port

3. The client sends its current time (t1) in milliseconds as an uint64 in network byte order to the host

4. The host responds with the time echo, two uint64 values in network byte order 1. The first value is equal to the sent client time (t1) 2. The second value corresponds to the host’s time in milliseconds (tH)

5. The client calculates the duration of steps 3 and 4 (roundtrip time) by measuring the client time before sending the request (t1) and after receiving the echo (t2)

6. The protocol assumes that the transport duration is symmetric. It will assume that tH was measured at the same time as the midpoint betwee t1 and t2.

7. To calculate the offset between the host’s and client’s clock, we subtract tH from the client’s midpoint (t1 + t2) / 2:

   offset_ms = ((t1 + t2) / 2) - tH
   

8. This measurement can be repeated multiple times to make the time offset estimation more robust.

To convert client to host time, subtract the offset:

host_time_ms = client_time_ms() - offset_ms

This is particularly helpful to accurately timestamp local events, e.g. a stimulus presentation.

To convert host to client time, add the offset:

client_time_ms = host_time_ms() + offset_ms

This is particularly helpful to convert the received data into the client’s time domain.

---

class pupil_labs.realtime_api.time_echo.Estimate(measurements)

Bases: object

Provides easy access to statistics over a collection of measurements

property mean

Return type:

float

property median

Return type:

float

property std

Return type:

float

namedtuple pupil_labs.realtime_api.time_echo.TimeEcho(roundtrip_duration_ms, time_offset_ms)

Bases: NamedTuple

Measurement of a single time echo

Fields:

0.  roundtrip_duration_ms (int) – Round trip duration of the time echo, in milliseconds

1.  time_offset_ms (int) – Time offset between host and client, in milliseconds

namedtuple pupil_labs.realtime_api.time_echo.TimeEchoEstimates(roundtrip_duration_ms, time_offset_ms)

Bases: NamedTuple

Provides estimates for the roundtrip duration and time offsets

Fields:

0.  roundtrip_duration_ms (Estimate) – Alias for field number 0

1.  time_offset_ms (Estimate) – Alias for field number 1

pupil_labs.realtime_api.time_echo.TimeFunction

Returns time in milliseconds

Alias of Callable[[], int]

class pupil_labs.realtime_api.time_echo.TimeOffsetEstimator(address, port)

Bases: object

async estimate(number_of_measurements=100, sleep_between_measurements_seconds=None, time_fn_ms=<function 'time_ms'>)

Return type:

Optional[TimeEchoEstimates]

async static request_time_echo(time_fn_ms, reader, writer)

Request a time echo, measure the roundtrip time, and estimate the time offset

Return type:

TimeEcho

pupil_labs.realtime_api.time_echo.time_ms()

Return milliseconds since Unix epoch (January 1, 1970, 00:00:00 UTC)