When LLMs meets immersive reality
Presented Aalto University,Finland, in collaboration with Khalifa University, UAE
Experience real-time semantic immersive streaming at MWC Doha 2025! Our end-to-end demo integrates听360掳听environmental听sensing,听LLM-powered听3D听content听generation,听and听immersive听VR teleoperation, allowing users to explore remote environments visually and with tactile feedback.
Visitors can experience a live teleoperation of a 3D environment, with a digital twin that adapts and responds to user movements, creating a truly immersive encounter.
our system combines five key components:
1.听VR User听Interface
- First-personVR control听via听HMD听and听joystick
- Dualvisualization:听360掳听video听+听dynamically听generated听3D听scene
2.听On-Board Computer & 360 Camera
- Omnidirectional video capture and sensor telemetry (GPS, IMU, altimeter)
- Image captioning through a Vision Language Model
- Semantic annotations packaged in JSON and sent to the cloud
3.听Cloud Server
- Multi-agent LLM architecture:
- LLM 1: Generates scene descriptions semantically
- LLM 2: Converts descriptions into A-Frame-based 3D content for WebXR
- Supports in-context learning and generates dynamic immersive 3D content
4.听Edge Cloud Server
- Low-latency delivery of 360掳 video and 3D content
- Hosts WebXR interface and stores pre-generated content for fast rendering
- Acts as MQTT broker for telemetry and haptic feedback
Visit Us at MWC Doha 2025
Discover how GenAI enables semantic immersive streaming. Explore VR teleoperation and interactive 3D worlds at our booth.
