Obviousness

Obviousness Analysis of US Patent 10,200,654 under 35 U.S.C. § 103

The independent claims of US Patent 10,200,654 describe a system and method for real-time manipulation and interaction with multiple dynamic and synchronized video streams in an augmented reality (AR) or multi-dimensional space. The core innovation, as articulated by the patent's background, aims to overcome limitations in prior AR systems that were primarily single-user and video chat systems that did not allow for synchronized, interactive augmented objects in shared streams.

This analysis identifies a combination of prior art references that would render the independent claims (Claim 1 and Claim 32) and many dependent claims obvious to a person having ordinary skill in the art (PHOSITA) at the time of the invention's priority date (February 27, 2013).

Independent Claims

Independent Claim 1 (System) details a system comprising:

• A data input device (video camera) providing a real-world video stream of a participant in a video chat or conference.
• A multiple stream sync module that receives an augmented video stream (containing 3D virtual objects from a database) from another participant.
• This module multiplexes the received augmented video with the local real-world video in real time to generate a multiplexed video stream (real-world video + 3D virtual objects from the other participant).
• It extracts at least one 3D virtual object, and based on participant action, translates its 3D coordinates within the multi-dimensional space defined by the real-world video.
• A display device shows the multiplexed video stream, allowing both the local participant and the other participant to translate the coordinates of the 3D virtual objects within the shared 3D space.

Independent Claim 32 (Method) outlines a corresponding method for achieving the functionalities of Claim 1, including providing real-world video, receiving and multiplexing augmented video streams, extracting and translating virtual objects in 3D, displaying the combined stream, and enabling both participants to manipulate the virtual objects.

Prior Art Combination and Motivation

A combination of the following prior art references would render Independent Claims 1 and 32, as well as many dependent claims, obvious:

1. US20120320158A1 to Microsoft Corporation ("Interactive and shared surfaces")
2. "Zugara's ZugSTAR: Video Conferencing + Augmented Reality = A Shared AR Experience," We Are Organized Chaos, published Nov. 4, 2009 (and related ZugSTAR publications)
3. Nilsson, S., Gustafsson, T., & Carleberg, P. (2009). "Hands Free Interaction with Virtual Information in a Real Environment: Eye Gaze as an Interaction Tool in an Augmented Reality System." PsychNology Journal, 7(2), 175-196.

Detailed Analysis of Obviousness:

1. Elements Taught by US20120320158A1 (Microsoft):
This patent, cited by the examiner, discloses systems and methods for creating and interacting with augmented reality (AR) content on shared surfaces. It teaches:

• Generating augmented reality (AR) content (i.e., virtual objects) corresponding to a physical surface and displaying it.
• Detecting user interaction with the AR content.
• Performing actions based on detected user interaction, including updating AR content and providing the AR content to a second user.
• Remote collaboration on a shared interactive surface where "Each user may see the AR content and interact with it, and their interactions may be reflected on the shared interactive surface."
• User interaction with AR content can include "tapping, swiping, pushing, pulling, translating, rotating, or scaling the AR content."

From this, US20120320158A1 clearly teaches a multi-user, interactive augmented reality experience where multiple users can view and manipulate (including translating 3D coordinates) shared virtual objects.

2. Elements Taught by Zugara's "ZugSTAR: Video Conferencing + Augmented Reality = A Shared AR Experience" (2009):
This non-patent prior art, directly associated with the assignee (Zugara Inc.) and inventors of US10200654, explicitly addresses the combination of video conferencing with AR for a shared experience, predating the filing date of US10200654. It teaches:

• The concept of "Video Conferencing + Augmented Reality = A Shared AR Experience."
• "real-time, multi-user augmented reality applications."
• Enabling "two-way augmented reality video chat that allows users in two different physical locations (using two different webcams and computers) to view the same shared augmented reality content at the same time and interact with it."
• States unequivocally that "Not only do they share the augmented reality experience, but they can both interact with the virtual object(s) in real-time."

The ZugSTAR publications directly disclose the use of webcams to provide real-world video streams in a video chat context, the synchronization and sharing of augmented content, and the ability for multiple participants to interact with and manipulate shared virtual objects in real-time. This directly addresses the core novelty claimed by US10200654.

3. Elements Taught by Nilsson et al. (2009):
This article describes "eye gaze as an interaction tool in an Augmented Reality System." This directly anticipates or renders obvious Claim 21 and 48 of US10200654, which claim controlling virtual objects using pupil tracking.

Motivation to Combine:

A PHOSITA in 2013, faced with the recognized limitations of existing single-user AR systems and basic, non-augmented video chat platforms (as noted in US10200654's background), would have had a clear and compelling motivation to combine these technologies.

1. Enhancing Remote Collaboration: US20120320158A1 already highlighted the benefits of "remote collaboration on a shared interactive surface." A natural progression for a PHOSITA would be to integrate such interactive AR with live video communication (i.e., "video chat or conference") to facilitate more engaging and intuitive remote collaborative experiences where participants can see each other and jointly manipulate shared virtual objects in a real-world context.
2. Addressing Known Deficiencies: The problem the patent purports to solve—the inability of current video chat systems to synchronize and allow interaction with augmented objects in shared streams—is precisely the motivation for combining video conferencing with AR.
3. Prior Art Suggests the Combination: The existence of "Zugara's ZugSTAR" in 2009 explicitly demonstrated this combination. As a product/concept from the inventors' own company, it serves as strong evidence that the idea of combining video conferencing with shared, interactive AR was well-known and being pursued by those skilled in the art at the time. The title itself, "Video Conferencing + Augmented Reality = A Shared AR Experience," explicitly teaches the combination.
4. Routine Implementation of Known Technologies: Implementing mechanisms for synchronization in time (T) and 3D coordinates (X, Y, Z), reducing latency, and supporting multiple platforms are routine engineering considerations for developing real-time, networked multimedia applications, especially when aiming for a seamless user experience. Similarly, integrating various input methods (hand gestures, voice, pupil tracking from Nilsson et al.) into an interactive AR system would be obvious to a PHOSITA.

Conclusion:

The combination of US20120320158A1 and the Zugara ZugSTAR publications (from 2009) would render Independent Claims 1 and 32 of US10200654 obvious. The ZugSTAR prior art explicitly describes a system that combines video conferencing and augmented reality to create a real-time, multi-user, shared AR experience where both participants can interact with and manipulate virtual objects. This directly anticipates or renders obvious the core elements of the independent claims, especially the critical aspect of shared, interactive manipulation of augmented objects in a synchronized video stream. US20120320158A1 further details the mechanisms for multi-user interaction with shared AR content, including translation of objects. The motivation for combining these elements is clearly articulated by the need to overcome the limitations of existing AR and video chat systems for enhanced remote collaboration, a need explicitly stated in the background of US10200654.

Furthermore, Nilsson et al. (2009) specifically teaches the use of pupil tracking for interaction in AR systems, making the related dependent claims obvious in combination with the shared AR system described by Zugara ZugSTAR and US20120320158A1. Other dependent claims related to general video conferencing features (e.g., chat rooms, participant management, transport systems) or standard AR implementation details (e.g., virtual object databases, overlaying objects, various data input devices) represent routine engineering choices for a PHOSITA in designing such a combined system.