[ orcid=0000-0003-0314-7107, [email protected], url=https://maximvdw.be/profile/card, ]

[ orcid=0000-0001-9916-0837, [email protected], url=https://beatsigner.com, ]

1 Introduction and Related Work

In Augmented reality (AR), virtual objects are superimposed onto the physical world. Since its early days, AR has significantly advanced to portable devices such as smartphones and head-mounted displays [1], enabling its use in regular environments such as office buildings to superimpose virtual information to the physical world.

Superimposed virtual objects or information can be anchored to physical objects and walls within these physical environments [2], [3]. These virtual objects can range from textual information such as timers or instructions to images, videos or interactive elements [4]. However, there is a lack of AR solutions that enable collaboration on the creation of these AR environments without the need for proprietary applications.

A conceptual content sharing solution for AR, offering a peer-to-peer and client-server collaboration solution using events that indicate changes to the virtual objects, was proposed in [5]. While the solution proposed in [5] supports collaboration in AR, it still requires a complex and proprietary peer-to-peer communication framework and does not tackle the problem of different devices using a different frame of reference [6].

We propose an interoperable solution that enables users to create their personal AR environments that can be discovered in the physical world by other users using AR-enabled devices. Unlike other existing work such as XSpace [7], we aim for an synthetic and semantic interoperable solution that decentralises the collaborative aspect of AR environments, while also enabling the sharing of crucial environmental information such as common reference spaces. Our interoperable solution should enable the unambiguous description of the virtual environment in a way that can be accessed and understood by other applications. In this paper we focus on the syntactic- and processing interoperability [8], [9] of the virtual environments. The semantic interoperability of 3D environments through common vocabularies is limited to illustrate the basic concept.

2 Solution

The goal of our proposed solution is to allow multiple AR devices to contribute to a single shared AR environment or virtual space belonging to a user. We assume that the AR device used to contribute to augmented environments is a smart device that has access to the Web and can broadcast an RF signal. In the general architecture of our proposed solution, we let AR devices broadcast a semantic Bluetooth Low Energy (BLE) beacon advertisement containing the URI of a specific resource. We utilise the advertising protocol of SemBeacon [10] that advertises an AltBeacon advertisement and Eddystone-URL compatible scan response to broadcast semantic data URIs. In addition, SemBeacon has a BLE v5.0 advertisement that prevents the need to shorten the URI to fit within the PDU. The use of SemBeacon to advertise the environment URI is illustrated in Figure 1. This environment resource contains information about the personal environment(s) owned by the user. Other devices can receive these advertisements when in proximity to the AR device and then access the URI to retrieve more information. For each environment, we have a link to a public inbox that can be used by other users to link their own modifications to the environment. Any modifications made to the superimposed space are stored in a Solid Pod owned by the user who made the modifications, which enables users to both contribute to the same environment as well as control the access rights of modifications made to these environments.

On the left-hand side of the architecture shown in Figure 1, we have a Solid Pod for user 1. The Pod contains all environments owned or modified by the user. An AR device connects to the user’s Solid Pod through a Solid application that authenticates the user, allowing it to modify the resources when editing a virtual environment. In order to enable the discovery of these virtual spaces, the AR device broadcasts the *.ttl file of the environment it is currently in via BLE advertisements. This resource contains all the information about the environment, such as its location, any identifiable features and all virtual objects placed relative within this environment. While the URI that is broadcasted is public, the access to read this URI is controllable through access control lists implemented in Solid. SemBeacon offers a broadcasted flag to indicate if a URI is publicly accessible to prevent applications that do not have access to the resource from attempting to access it.

When another user (e.g. user 2) wants to modify the environment of user 1, they create a new resource including the modifications and additions to virtual objects or detectable features (e.g. markers). The application will then notify user 1 about these changes by referencing the user1-office/data.ttl file in the inbox [11] container of the environment that is being modified. Solid also enables the access rights of these modifications, more information about access control for environments and modifications is provided in Section 2.3.

<> a seas:Room ; rdfs:label "Our Lab"@en ;
  ldp:inbox <./inbox/> ;
  vcard:address [ ... ] .
:printer_marker a fidmark:AruCo ;
  fidmark:markerIdentifier 12 .
:printer_info a sosa:FeatureOfInterest ;
  poso:hasPosition [ 
    poso:isRelativeTo :printer_marker ;
    ... ] ;
  omg:hasGeometry [ ... ] .

An alternative architecture is illustrated in Figure 2. In this scenario, a fixed Bluetooth beacon is placed in a room, broadcasting the URI of a single environment. This scenario can be used for public physical environments, such as a meeting room or laboratory, enabling collaboration in AR. Similar to personal environments shown in Figure 1, users store their changes to an environment in their Solid Pod and reference these changes in the inbox of the environment.

2.1 Usage

Our solution is depicted in Figure 3 where we showcase the flow of our architecture previously illustrated in Figure 1. Two users with AR devices have their own Solid Pod. User 1 will create an environment (A) on their Pod and subscribe to the inbox container of this environment. Once the environment is ready, the AR device will use the SemBeacon specification to advertise the URI of the environment to enable its discovery.

When another user, such as user 2 discovers the resource URI the AR application will access the environment to visualise the augmented objects. If user 2 makes a modification such as adding virtual objects, these modifications are stored in the Pod of user 2 as environment A′, ensuring ownership of this contribution and enabling user 2 to choose the access rights to this modification. To keep up-to-date with changes in the original environment, the application will listen for changes in the inbox of environment A. All users who are subscribed to this inbox will receive notifications whenever a new modification is added.

An inbox uses the LDP⁵ vocabulary to index all resources within this container. A user who wishes to accept contributors for their environment should configure this inbox container with public append access rights, giving other users the opportunity to append a new resource to the container. Each inbox item represents an action using the schema.org vocabulary [12].

@prefix card: <https://user2.solidweb.org/profile/card#> .
@prefix office: <https://user2.solidweb.org/environments/user1-office/data.ttl#> .

<> a schema:CreateAction ;
  schema:description "Created a new object 'painting' with label 'User 2'"@en ;
  schema:agent <https://ar-app.com/id> ;  # AR application that created the action
  schema:creator card:me ;                # Owner of the modification
  schema:object office:painting ; schema:result office:painting .

Listing [lst:inbox-example] demonstrates an individual appended inbox item to the Solid Pod of user 1. In the example we see a schema:CreateAction indicating that user 2 created a new painting object. Users can listen for notifications in the inbox to automatically apply changes to virtual objects in the shared AR environment as they are made [11].

Positioning virtual objects is done using the POSO ontology [13], which allows the description of (virtual) objects to be placed relative to other objects using the poso:isRelativeTo predicate. When a user wants to place an object in an environment using an absolute position that is not relative to a marker or detectable feature, the same predicate can be used to indicate that the absolute position is relative to the environment.

The final AR environment combining all modifications made by other users is created by the AR application that has access to the Solid Pod. Our architecture allows users to easily ignore all contributions of another user or agent. Individual modifications or contributions can be rejected by ignoring the individual inbox items where these changes are referenced. Future work might address the moderation of individual contributions by applying quality assessment crowdsourcing techniques [14].

2.2 Reference Frame

AR uses one or more reference frames [6] to anchor virtual objects in the physical environment and to determine an absolute or relative position. This can be done through feature detection [2] that creates anchors based on visual patterns or artificial features such as fiducial markers [3].

In order to enable interoperable applications to contribute to the same environment, all applications need to operate within the same reference frame. Our solution assumes that contributions to an environment not only include virtual objects, but also additional anchor points such as markers and detectable features that are contributed by multiple users [15]. By semantically describing these reference frames as objects on itself, the POSO ontology can be used to position virtual objects relative to these anchors.

2.3 Access Control

Solid enables users to have control over the access rights of their data and resources. In the scope of our solution, this entails the data that is shared in the AR environment (i.e. virtual objects and anchors) by both the user who owns the environment and other users who make that own modifications to the environment.

Each modification to an environment is made in the Solid Pod of the user who made the modification, giving them full control on the access rights to these modifications – as well as control on the actions which are normally created in the original environment to notify the original owner of the environment (see Section 2.1).

3 Conclusion and Future Work

We presented a solution enabling the collaboration and sharing of personal augmented reality (AR) environments by using Solid Pods. Our solution allows these personal AR spaces to be discoverable within the physical space that these environments are augmenting by means of semantic beacons that advertise a URI [10]. While our proposed solution allows users to choose which collaborators to accept for the environment through the inbox system, it is currently impossible to accept individual contributions from a single collaborator. Future work might address this issue by applying crowdsourcing techniques [14] to the data that is produced by other users.

While we focused on the passive discovery using BLE advertisements to enable seamless discovery of the linked data, similar architectures can be designed using other discovery methods such as a QR-code on the door of the environment.

In the future, we might evaluate the performance differences between keeping multiple versions of a virtual environment in different Pods (as proposed in our solution) versus using a stream of events that indicate changes to the environment, similar to the proposed work in [16].