US 10200654

US Patent 10,200,654 Summary:

Title: Systems and methods for real time manipulation and interaction with multiple dynamic and synchronized video streams in an augmented or multi-dimensional space.

Assignee: ZUGARA Inc.

Inventors: Matthew SZYMCZYK, Alex Goldberg, Hans Forsman, Jack Benoff, Aaron Von Hungen, Blake Callens, Mark Weisheit, and Andrey Tkachuk.

Filing Date: February 27, 2013.

Issue Date: February 5, 2019.

Abstract: The patent proposes systems and methods for real-time manipulation of objects within multiple dynamic and augmented video streams that are synchronized in an augmented reality (AR) or multi-dimensional space. This approach allows a user of a first AR system to connect with users of a second (similar or different) AR system, sharing their initial and/or augmented video streams. Through real-time synchronization, participants can see, share, manipulate, and interact with augmented content/objects in their own or another user's augmented video streams.

Plain-Language Overview of Independent Claims:

• Independent Claim 1 (System): This claim describes a system that enables collaborative augmented reality. It includes a video camera providing a real-world video stream of a participant in a video chat. A "multiple stream sync module" receives an augmented video stream (containing 3D virtual objects from a database) from another participant. This module combines the received augmented video with the local real-world video in real-time to create a "multiplexed video stream." From this combined stream, one or more 3D virtual objects are extracted, and their 3D coordinates can be translated (moved) within the virtual space based on actions of the local participant. The resulting multiplexed stream is displayed, and a key feature is that both the local participant and the other participant can independently translate the coordinates of these 3D virtual objects within the shared 3D space.

• Independent Claim 32 (Method): This claim outlines a method for achieving the collaborative augmented reality described in Claim 1. It involves providing a real-world video stream from a participant's camera and receiving an augmented video stream with 3D virtual objects from another participant. The method then multiplexes these streams in real-time to generate a combined video stream. It further includes extracting virtual objects, translating their 3D coordinates based on a participant's action, displaying the combined stream, and allowing both the local and other participants to translate the coordinates of the virtual objects within the shared multi-dimensional space.

Legal Status Note:
As of April 26, 2026, the patent is indicated as "Active" and is set to expire on July 27, 2033.
The patent family is currently involved in litigation, including:

• A PTAB (Patent Trial and Appeal Board) case, IPR2026-00266, which is pending.
• A US case filed in the Texas Eastern District Court (case 2:25-cv-00437).
• Additionally, the patent notes that the first worldwide family litigation has been filed.

Searches for "10200654" in the USPTO database and CAFC 2026 dockets did not yield additional specific case details beyond what is already stated in the patent document itself. Therefore, the litigation information provided here relies on the details directly available within the patent text.

As a patent attorney, I have identified the following known litigation involving US patent 10200654:

1. PTAB Case IPR2026-00266

• Plaintiff(s): Not specified in the provided data, but "Unified Patents PTAB Data" is listed as the Petitioner source.
• Defendant(s): Not specified in the provided data.
• Jurisdiction: Patent Trial and Appeal Board (PTAB)
• Case Number: IPR2026-00266
• Filing Date: Not explicitly stated, but the case number indicates a filing year of 2026.
• Outcome/Current Status: Pending.

2. US case filed in Texas Eastern District Court

• Plaintiff(s): Not specified in the provided data.
• Defendant(s): Not specified in the provided data.
• Jurisdiction: Texas Eastern District Court
• Case Number: 2:25-cv-00437
• Filing Date: Not explicitly stated, but the case number indicates a filing year of 2025.
• Outcome/Current Status: Litigation.

Proceedings overview

There is one active AIA trial proceeding on file for US Patent 10,200,654. The proceeding, IPR2026-00266, is currently pending, meaning no claims have been invalidated or sustained by the PTAB yet. This suggests the patent's claims remain untested by a final written decision, offering a defendant an opportunity to pursue an IPR-based defense.

IPR2026-00266 — Cisco Systems, Inc. v. ZUGARA Inc.

• Type: Inter Partes Review
• Filed: 2026-03-04
• Status: Pending. The petition has been filed, and the proceeding is in its initial stages, awaiting a decision on institution.
• Judge panel: Information regarding the specific Administrative Patent Judges (APJs) assigned to this panel is not publicly available at this early stage.
• Petition grounds: The claims challenged and the specific prior art grounds (§ 102 / § 103) are not yet publicly detailed in a decision document as the case is still pending institution.
• Institution decision: As of 2026-05-29, an institution decision has not yet been issued. A decision on whether to institute the IPR is typically rendered within six months of the patent owner's preliminary response or within 12 months of the petition filing date if no preliminary response is filed.
• Final Written Decision: No Final Written Decision has been issued, as the case is still pending institution.
• Settlement / termination: No settlement or termination has been reported for this pending proceeding.
• Appeal: No appeal to the Federal Circuit has occurred, as the proceeding is still pending at the PTAB.
• Defensive value: This active IPR indicates that the patent's validity is currently being challenged. While no claims have been invalidated, the outcome of this proceeding, particularly the institution decision and any subsequent final written decision, will significantly impact the defensive posture against this patent.

Strategic summary

As of the current date, no claims of US Patent 10,200,654 have been canceled or sustained by the PTAB. The patent has one active Inter Partes Review, IPR2026-00266, which was filed by Cisco Systems, Inc. and is currently pending institution. This means all claims of the patent, 1-53, remain untested by a final PTAB decision.

The estoppel landscape under § 315(e)(2) will only become relevant if IPR2026-00266 is instituted and proceeds to a Final Written Decision. If institution is denied, no estoppel attaches. If instituted and a Final Written Decision is issued, Cisco Systems, Inc. and parties in privity with them would be estopped from asserting invalidity grounds that were raised or reasonably could have been raised during the IPR. For other potential defendants, the prior-art grounds challenged in IPR2026-00266 would remain available unless they are found to be in privity with Cisco Systems, Inc. or have otherwise participated in the proceeding.

There are no observable patterns of multiple IPR filings by the same petitioner on this specific patent. Unified Patents is noted as a data source, but Cisco Systems, Inc. is the petitioner in the pending IPR. The patent owner's approach to PTAB appeals cannot be determined as no final written decision has been issued.

Recommended next steps

• Given that IPR2026-00266 is pending, closely monitor the USPTO PTAB End-to-End (E2E) system for updates, particularly the institution decision. The decision on institution is a critical milestone, as it will indicate whether the PTAB believes there is a reasonable likelihood that at least one challenged claim is unpatentable. The statutory deadline for this decision is typically within six months of the patent owner's preliminary response.
• If you are a defendant facing assertion of this patent, consider the grounds raised in IPR2026-00266 and evaluate whether similar or additional prior art grounds could be presented in a separate IPR petition, assuming no privity or other estoppel issues.
• If the IPR is instituted, track the trial schedule, including the oral hearing and the statutory 1-year deadline for the Final Written Decision from institution.
• Access the public docket for IPR2026-00266 on the USPTO PTAB E2E system for the most up-to-date information on filings and procedural events.

Inventors

The following individuals are named as inventors on US Patent 10,200,654:

• Matthew SZYMCZYK
• Alex Goldberg
• Hans Forsman
• Jack Benoff
• Aaron Von Hungen
• Blake Callens
• Mark Weisheit
• Andrey Tkachuk

At the time of filing, and as indicated by the subsequent assignments of their rights to ZUGARA Inc., all inventors were associated with ZUGARA Inc. Matthew Szymczyk is listed as the CEO, Alex Goldberg as CTO, and Andrey Tkachuk as Head of R&D for ZUGARA Inc. There is no indication of unusual inventor departure patterns or a subsequent portfolio fire-sale; instead, the assignments reflect standard transfers from inventors to their employer.

Original assignee

The original assignee named on the issued patent US 10,200,654 is ZUGARA Inc.

ZUGARA Inc. is an operating company engaged in the provision of Augmented Reality (AR) and Virtual Reality (VR) software solutions, with a particular focus on solving user interaction and industry-specific problems. Their primary line of business involves developing AR and computer vision software, including virtual dressing room platforms such as Webcam Social Shopper (WSS) and Virtual Style Sense (VSS) for web, kiosks, and in-store retail. ZUGARA Inc. has shipped products embodying the claims and their technology has been leveraged by numerous major brands worldwide.

As of March 2026, Tracxn identifies ZUGARA Inc. as an unfunded company, but its website and recent news suggest it is still actively operating and developing its AR and computer vision technologies.

Assignment timeline

The following assignments for US Patent 10,200,654 are recorded with the USPTO:

• 2006-11-01 (executed) / recorded 2018-12-18 — Reel 047951/0611

  • Conveyance: Assignment
  • Assignor: HUNGEN, AARON VON
  • Assignee: ZUGARA, INC.
  • Correspondent: Matthew T. Szymczyk, 1109 E. Washington Blvd., Los Angeles, CA 90021. This correspondent recurs on all assignments in this chain.
  • Context: Internal transfer of inventor rights to the operating company.

• 2009-05-07 (executed) / recorded 2023-11-16 — Reel 065600/0868

  • Conveyance: Assignment
  • Assignor: CALLENS, BLAKE
  • Assignee: ZUGARA, INC.
  • Correspondent: Matthew T. Szymczyk, 1109 E. Washington Blvd., Los Angeles, CA 90021. This correspondent recurs on all assignments in this chain.
  • Context: Internal transfer of inventor rights to the operating company.

• 2010-02-07 (executed) / recorded 2023-09-25 — Reel 065012/0186

  • Conveyance: Assignment
  • Assignor: WEISHEIT, MARK
  • Assignee: ZUGARA, INC.
  • Correspondent: Matthew T. Szymczyk, 1109 E. Washington Blvd., Los Angeles, CA 90021. This correspondent recurs on all assignments in this chain.
  • Context: Internal transfer of inventor rights to the operating company.

• 2013-11-03 (executed) / recorded 2018-12-18 — Reel 047809/0177

  • Conveyance: Assignment
  • Assignor: TKACHUK, ANDREY
  • Assignee: ZUGARA, INC.
  • Correspondent: Matthew T. Szymczyk, 1109 E. Washington Blvd., Los Angeles, CA 90021. This correspondent recurs on all assignments in this chain.
  • Context: Internal transfer of inventor rights to the operating company.

• 2018-10-23 to 2018-11-07 (executed) / recorded 2018-12-18 — Reel 047951/0591

  • Conveyance: Assignment
  • Assignor: SZYMCZYK, MATTHEW; GOLDBERG, ALEX; FORSMAN, HANS; BENOFF, JACK
  • Assignee: ZUGARA, INC.
  • Correspondent: Matthew T. Szymczyk, 1109 E. Washington Blvd., Los Angeles, CA 90021. This correspondent recurs on all assignments in this chain.
  • Context: Internal transfer of inventor rights to the operating company.

Timeline diagram

timeline
    title Ownership of US 10200654
    2006 : Aaron Von Hungen assigns to Zugara
    2009 : Blake Callens assigns to Zugara
    2010 : Mark Weisheit assigns to Zugara
    2013 : Filed by Individuals
         : Andrey Tkachuk assigns to Zugara
    2018 : M. Szymczyk + 3 others assign to Zugara
    2019 : Issued to Zugara Inc.
    2025 : US litigation filed
    2026 : PTAB IPR filed

NPE / troll-pattern signals

1. Shell-entity transfer — Not present. All recorded assignments are from individual inventors to ZUGARA Inc., an operating company that develops and sells AR software and services. There is no transfer from ZUGARA Inc. to any shell entity.
2. Known asserter in the chain — Not present. ZUGARA Inc. is an operating company in the AR/VR software space, not identified as a known NPE.
3. Repeat correspondent across the chain — Present. Matthew T. Szymczyk, located at 1109 E. Washington Blvd., Los Angeles, CA 90021, is listed as the correspondent for all five assignments (Reel 047951/0611, 065600/0868, 065012/0186, 047809/0177, 047951/0591). Matthew Szymczyk is also an inventor and CEO of ZUGARA Inc., indicating that these assignments were handled internally by the operating company.
4. Cascading transfers — Not present. While there are multiple assignments, they are all from different inventors to the same single assignee (ZUGARA Inc.), not consecutive transfers through chained LLCs.
5. Pre-litigation transfer — Not present. The most recent assignment was recorded on 2023-11-16 (Reel 065600/0868). The earliest known litigation filing (Texas Eastern District Court case 2:25-cv-00437) indicates a 2025 filing year, which is more than six months after the latest recorded assignment.
6. Bankruptcy fire-sale — Not present. ZUGARA Inc. is indicated as an operating company, not subject to bankruptcy proceedings, although it is reported as unfunded.
7. Privateering — Not present. There is no evidence of a transfer to an NPE asserting on behalf of ZUGARA Inc.
8. Defensive aggregator (anti-NPE) — Not present. The patent remains with ZUGARA Inc.

Verdict

Operating-company assertion

This verdict is based on the fact that ZUGARA Inc., an operating company engaged in developing and commercializing augmented reality software, is the current assignee of the patent and has been involved in litigation since at least 2025 (case 2:25-cv-00437). All recorded assignments are internal transfers of inventor rights to ZUGARA Inc., with no evidence of transfer to any shell entity or known NPE. The recurrence of Matthew T. Szymczyk, an inventor and CEO of Zugara Inc., as the correspondent for all assignments further supports that these were internal company actions (Reel 047951/0611, 065600/0868, 065012/0186, 047809/0177, 047951/0591).

USPTO Assignment Center search for 10200654: https://assignmentcenter.uspto.gov/patent-search-results?id=10200654

The USPTO database confirms the existence of tools for patent searching, such as "Patent Public Search" and "Patent Public Search Basic". As directed, I have prioritized the provided full patent text of US10200654B2 as the authoritative source for identifying relevant prior art. The patent itself lists five patent citations.

Here is an analysis of the most relevant prior art for US patent 10200654, based on its own cited patent references:

Prior Art Analysis for US10200654B2

1. US6219045B1

• Full Citation: US6219045B1, "Scalable virtual world chat client-server system"
• Publication/Filing Date: Priority Date: 1995-11-13; Publication Date: 2001-04-17
• Brief Description: This patent describes a client-server system for creating and interacting within scalable virtual worlds. It focuses on enabling multiple users to chat and interact within a shared virtual environment. While it involves a shared space and user interaction, the primary focus is on virtual worlds, not explicitly the real-time augmentation and manipulation of objects within augmented video streams that combine real-world video with virtual objects.
• Potential Anticipation (35 U.S.C. § 102): US6219045B1 could potentially anticipate the broad concept of multiple users interacting in a shared digital space. However, it likely does not anticipate the specific elements of Claims 1 and 32 of US10200654 that relate to:
  • Receiving a real-world video stream of a participant.
  • Multiplexing an augmented video stream (from another participant) with a real-world video stream.
  • Translating coordinates of extracted 3D virtual objects within a multi-dimensional space defined by the real-world 3D environment.
  • The explicit feature that both participants can translate the coordinates of 3D virtual objects within this real-world augmented context.
    The lack of a direct focus on augmented reality (i.e., combining real-world video with virtual objects) means it's unlikely to fully anticipate Claims 1 or 32.

2. US20110040539A1

• Full Citation: US20110040539A1, "Providing a simulation of wearing items such as garments and/or accessories"
• Publication/Filing Date: Priority Date: 2009-08-12; Publication Date: 2011-02-17
• Brief Description: This patent application, by one of the inventors of US10200654, Matthew Szymczyk, describes systems and methods for providing a simulation of wearing virtual items like clothes or accessories. This directly relates to the virtual shopping use case mentioned in US10200654. It involves augmenting a user's live video stream with virtual objects.
• Potential Anticipation (35 U.S.C. § 102): This reference is highly relevant to the augmentation aspect of US10200654. It anticipates generating an augmented video stream by synchronizing virtual objects with an initial video stream. However, it primarily focuses on a single user's experience of trying on virtual items. It does not explicitly disclose the core novelty of US10200654, which is the real-time synchronization of multiple augmented video streams between different participants and the ability for both/all participants to collaboratively manipulate the virtual objects within that shared, real-time augmented space. Therefore, it would likely anticipate elements related to a single user's augmented reality experience but not the multi-user, collaborative, and interactable aspects emphasized in Claims 1 and 32.

3. US8275590B2

• Full Citation: US8275590B2, "Providing a simulation of wearing items such as garments and/or accessories"
• Publication/Filing Date: Priority Date: 2009-08-12; Publication Date: 2012-09-25
• Brief Description: This is a granted patent, also by Zugara, Inc., sharing the same priority date and title as US20110040539A1. It describes similar technology for virtual try-on of garments and accessories.
• Potential Anticipation (35 U.S.C. § 102): Similar to US20110040539A1, this patent covers the augmentation of a user's video stream with virtual objects for purposes like virtual shopping. It contributes to the state of the art regarding single-user augmented reality for virtual try-on. It would likely anticipate the generation of an augmented video stream with virtual objects synchronized to a real-world video stream. However, it does not explicitly teach the real-time multi-user synchronization and collaborative manipulation of augmented objects across multiple participants' video streams, which is central to Claims 1 and 32 of US10200654.

4. US20120313969A1

• Full Citation: US20120313969A1, "Providing a simulation of wearing items such as garments and/or accessories"
• Publication/Filing Date: Priority Date: 2009-08-12; Publication Date: 2012-12-13
• Brief Description: This is another patent application by Zugara, Inc., sharing the same priority date and title as US20110040539A1 and US8275590B2. Its subject matter is consistent with the virtual try-on of garments and accessories through augmented reality.
• Potential Anticipation (35 U.S.C. § 102): As with the two preceding Zugara/Szymczyk patents, this reference primarily covers the augmentation aspect for a single user. It would anticipate elements related to generating and displaying an augmented video stream with virtual objects. However, it would not anticipate the novel multi-user interaction, real-time synchronization of multiple participants' augmented streams, and the collaborative manipulation of virtual objects across those streams as defined in Claims 1 and 32 of US10200654.

5. US20120320158A1

• Full Citation: US20120320158A1, "Interactive and shared surfaces"
• Publication/Filing Date: Priority Date: 2011-06-14; Publication Date: 2012-12-20
• Brief Description: This Microsoft Corporation patent application describes interactive and shared surfaces. The title suggests a focus on interaction with shared digital content. While the full text is not provided, the general concept of "interactive and shared surfaces" could potentially involve multiple users interacting with shared digital content. It might pertain to shared virtual whiteboards, collaborative displays, or other forms of shared digital workspaces.
• Potential Anticipation (35 U.S.C. § 102): This reference is potentially more relevant to the multi-user interaction aspect than the previous three. The concept of "interactive and shared surfaces" could encompass scenarios where multiple users interact with digital content. However, for it to fully anticipate Claims 1 and 32 of US10200654, it would need to explicitly disclose:
  • The combination of real-world video streams from multiple participants.
  • The real-time augmentation of these combined streams with 3D virtual objects.
  • The explicit mechanism for both/all participants to collaboratively translate/manipulate those 3D virtual objects within the real-world augmented video context.
    Without the specific details of its claims, it's hard to definitively say, but the title alone suggests a broader concept of shared digital interaction rather than the specific combination of real-world augmented video streams with collaborative 3D object manipulation across multiple participants. It might anticipate the "shared" and "interactive" aspects but potentially fall short on the "augmented video stream of real-world environment" and "real-time synchronization and collaborative manipulation of objects within that specific augmented context" limitations of US10200654's independent claims.

Obviousness Analysis of US Patent 10,200,654 under 35 U.S.C. § 103

The independent claims of US Patent 10,200,654 describe a system and method for real-time manipulation and interaction with multiple dynamic and synchronized video streams in an augmented reality (AR) or multi-dimensional space. The core innovation, as articulated by the patent's background, aims to overcome limitations in prior AR systems that were primarily single-user and video chat systems that did not allow for synchronized, interactive augmented objects in shared streams.

This analysis identifies a combination of prior art references that would render the independent claims (Claim 1 and Claim 32) and many dependent claims obvious to a person having ordinary skill in the art (PHOSITA) at the time of the invention's priority date (February 27, 2013).

Independent Claims

Independent Claim 1 (System) details a system comprising:

• A data input device (video camera) providing a real-world video stream of a participant in a video chat or conference.
• A multiple stream sync module that receives an augmented video stream (containing 3D virtual objects from a database) from another participant.
• This module multiplexes the received augmented video with the local real-world video in real time to generate a multiplexed video stream (real-world video + 3D virtual objects from the other participant).
• It extracts at least one 3D virtual object, and based on participant action, translates its 3D coordinates within the multi-dimensional space defined by the real-world video.
• A display device shows the multiplexed video stream, allowing both the local participant and the other participant to translate the coordinates of the 3D virtual objects within the shared 3D space.

Independent Claim 32 (Method) outlines a corresponding method for achieving the functionalities of Claim 1, including providing real-world video, receiving and multiplexing augmented video streams, extracting and translating virtual objects in 3D, displaying the combined stream, and enabling both participants to manipulate the virtual objects.

Prior Art Combination and Motivation

A combination of the following prior art references would render Independent Claims 1 and 32, as well as many dependent claims, obvious:

1. US20120320158A1 to Microsoft Corporation ("Interactive and shared surfaces")
2. "Zugara's ZugSTAR: Video Conferencing + Augmented Reality = A Shared AR Experience," We Are Organized Chaos, published Nov. 4, 2009 (and related ZugSTAR publications)
3. Nilsson, S., Gustafsson, T., & Carleberg, P. (2009). "Hands Free Interaction with Virtual Information in a Real Environment: Eye Gaze as an Interaction Tool in an Augmented Reality System." PsychNology Journal, 7(2), 175-196.

Detailed Analysis of Obviousness:

1. Elements Taught by US20120320158A1 (Microsoft):
This patent, cited by the examiner, discloses systems and methods for creating and interacting with augmented reality (AR) content on shared surfaces. It teaches:

• Generating augmented reality (AR) content (i.e., virtual objects) corresponding to a physical surface and displaying it.
• Detecting user interaction with the AR content.
• Performing actions based on detected user interaction, including updating AR content and providing the AR content to a second user.
• Remote collaboration on a shared interactive surface where "Each user may see the AR content and interact with it, and their interactions may be reflected on the shared interactive surface."
• User interaction with AR content can include "tapping, swiping, pushing, pulling, translating, rotating, or scaling the AR content."

From this, US20120320158A1 clearly teaches a multi-user, interactive augmented reality experience where multiple users can view and manipulate (including translating 3D coordinates) shared virtual objects.

2. Elements Taught by Zugara's "ZugSTAR: Video Conferencing + Augmented Reality = A Shared AR Experience" (2009):
This non-patent prior art, directly associated with the assignee (Zugara Inc.) and inventors of US10200654, explicitly addresses the combination of video conferencing with AR for a shared experience, predating the filing date of US10200654. It teaches:

• The concept of "Video Conferencing + Augmented Reality = A Shared AR Experience."
• "real-time, multi-user augmented reality applications."
• Enabling "two-way augmented reality video chat that allows users in two different physical locations (using two different webcams and computers) to view the same shared augmented reality content at the same time and interact with it."
• States unequivocally that "Not only do they share the augmented reality experience, but they can both interact with the virtual object(s) in real-time."

The ZugSTAR publications directly disclose the use of webcams to provide real-world video streams in a video chat context, the synchronization and sharing of augmented content, and the ability for multiple participants to interact with and manipulate shared virtual objects in real-time. This directly addresses the core novelty claimed by US10200654.

3. Elements Taught by Nilsson et al. (2009):
This article describes "eye gaze as an interaction tool in an Augmented Reality System." This directly anticipates or renders obvious Claim 21 and 48 of US10200654, which claim controlling virtual objects using pupil tracking.

Motivation to Combine:

A PHOSITA in 2013, faced with the recognized limitations of existing single-user AR systems and basic, non-augmented video chat platforms (as noted in US10200654's background), would have had a clear and compelling motivation to combine these technologies.

1. Enhancing Remote Collaboration: US20120320158A1 already highlighted the benefits of "remote collaboration on a shared interactive surface." A natural progression for a PHOSITA would be to integrate such interactive AR with live video communication (i.e., "video chat or conference") to facilitate more engaging and intuitive remote collaborative experiences where participants can see each other and jointly manipulate shared virtual objects in a real-world context.
2. Addressing Known Deficiencies: The problem the patent purports to solve—the inability of current video chat systems to synchronize and allow interaction with augmented objects in shared streams—is precisely the motivation for combining video conferencing with AR.
3. Prior Art Suggests the Combination: The existence of "Zugara's ZugSTAR" in 2009 explicitly demonstrated this combination. As a product/concept from the inventors' own company, it serves as strong evidence that the idea of combining video conferencing with shared, interactive AR was well-known and being pursued by those skilled in the art at the time. The title itself, "Video Conferencing + Augmented Reality = A Shared AR Experience," explicitly teaches the combination.
4. Routine Implementation of Known Technologies: Implementing mechanisms for synchronization in time (T) and 3D coordinates (X, Y, Z), reducing latency, and supporting multiple platforms are routine engineering considerations for developing real-time, networked multimedia applications, especially when aiming for a seamless user experience. Similarly, integrating various input methods (hand gestures, voice, pupil tracking from Nilsson et al.) into an interactive AR system would be obvious to a PHOSITA.

Conclusion:

The combination of US20120320158A1 and the Zugara ZugSTAR publications (from 2009) would render Independent Claims 1 and 32 of US10200654 obvious. The ZugSTAR prior art explicitly describes a system that combines video conferencing and augmented reality to create a real-time, multi-user, shared AR experience where both participants can interact with and manipulate virtual objects. This directly anticipates or renders obvious the core elements of the independent claims, especially the critical aspect of shared, interactive manipulation of augmented objects in a synchronized video stream. US20120320158A1 further details the mechanisms for multi-user interaction with shared AR content, including translation of objects. The motivation for combining these elements is clearly articulated by the need to overcome the limitations of existing AR and video chat systems for enhanced remote collaboration, a need explicitly stated in the background of US10200654.

Furthermore, Nilsson et al. (2009) specifically teaches the use of pupil tracking for interaction in AR systems, making the related dependent claims obvious in combination with the shared AR system described by Zugara ZugSTAR and US20120320158A1. Other dependent claims related to general video conferencing features (e.g., chat rooms, participant management, transport systems) or standard AR implementation details (e.g., virtual object databases, overlaying objects, various data input devices) represent routine engineering choices for a PHOSITA in designing such a combined system.