EUD4XR: D2.1 - AUTHORING AND RUN-TIME SUPPORT (initial version)

Spano D., Carcangiu A., Mereu J., Manca M., Santoro C., Simeoli L.

EUD, VR, AR, XR, ECA Rules, intelligent conversational agent, ICA, chatbot 

EUD4XR aims to empower end-user developers, EUDs, with an intuitive and practical tool for customizing XR experiences by setting the behavior of the involved virtual and real objects. Our proposed system supports the customization of the interactions through a rule-based language based on an event-condition-action (ECA) scheme. It is integrated with an intelligent conversational agent (ICA) tasked with supporting and guiding the user when help is needed. In this document, we describe a proof-of-concept implementation of our approach that includes the following components:  Rule Engine: this module monitors the extended environment, detects events occurring in the system, accesses the APIs of virtual and physical objects that make up the scene, and commands the execution of the actions at runtime.  Intelligent Conversational Agent or ICA: is a chatbot implemented through a large language model, LLM, and refined using prompt engineering. The ICA must support the user on: o Providing information about the environment and its objects. o Customizing the environment by adding a virtual object. o Creating or editing a rule.  Rule Authoring UI: to support users in configuring the environment, EUD4XR provides an immersive interface that allows both direct editing of ECA rules in situ without requiring coding skills, and interaction with an intelligent chatbot. Through this interface, the user can create, edit and manage ECA rules, but also request and receive assistance from the chatbot to define or correct rules according to their needs. Section 2 provides an overview of the framework supporting the components that are the subject of the deliverable D2.1. Section 3 lists and explains the rule engine's requirements, while Section 4 describes its implementation. Sections 5 and 6 define the requirements and the implementation of rule authoring UI. Next, Section 7 illustrates the intelligent conversational agent: it first reviews the current state of the art, then reports on the result of a pilot test, and finally details its requirements and design.