Unify Media and UX with timed variables

Media (moving pictures) and UX (user experience) are both strong and long-standing traditions for audiovisual experiences, yet with very different technical foundations. In particular, the two traditions appear as polar opposites regarding state representation and how rendering (in time) is modelled. For example, media content is serialized into static frames on a timeline - ahead of distribution, whereas UX state may include a variety of heterogeneous, dynamic data sources and user interactivity. In media, rendering is modelled as playback along a timeline, performed by common purpose media players, while in UX, rendering is driven by application code triggering state transitions in data and programming variables. Differences aside, there is also a strong need to combine media and UX traditions. For example, UX-based games borrow media concepts such as instant replay and employ media technologies for recording and distribution of live gameplay. Conversely, there is broad interest in integration of UX into media experiences, often to address growing demands for interactivity, adaptability, personalization, accessibility etc. This is particularly relevant for WP4 in the Media SFI. However, due to the fundamental incompatibility between media and UX, integration is often complex and time consuming. We argue though, that this incompatibility is not given by nature, but stems from incompatibilities in fundamental concepts. Our hypothesis is that a unifying concept may be derived for media and UX, showing them as two sides of the same coin. This project aims to propose such a technical concept and evaluate relevance and implications with respect to opportunities for innovation, as well as cost saving and complexity reduction for select applications from the media and UX space.