US 10769446

Patent Analysis: U.S. Patent No. 10,769,446

Date of Analysis: May 12, 2026

Patent Number: 10,769,446
Title: Methods and systems of combining video content with one or more augmentations

---

I. Patent Details

• Inventors: Yu-Han Chang, Rajiv Tharmeswaran Maheswaran, Jeffrey Wayne Su, Emil Dotchevski, Jason Kent Simon
• Assignee: Genius Sports SS LLC
• Filing Date: February 20, 2020
• Issue Date: September 8, 2020

Abstract:
"Methods and systems disclosed herein enable combining video content with one or more augmentations to produce augmented video. A computer-implemented data processing method for displaying augmented content on a client device may include: receiving, by one or more processors from an external server, video data corresponding to an event, the video data comprising video content and a plurality of definitions of bounding boxes; presenting, by one or more processors on a graphical user interface, the video content; detecting, by one or more processors, a user selection of a portion of the graphical user interface; determining, by one or more processors, a red, green, blue, alpha (RGBA) value associated with the user selection of the portion of the graphical user interface, determining, by one or more processors, a bounding box RGBA value that corresponds to the RGBA value associated with the user selection of a portion of the graphical user interface, wherein the bounding box RGBA value is associated with a particular bounding box; transmitting, by one or more processors, an indication of the particular bounding box to a renderer; receiving, by one or more processors from the renderer, one or more augmentation images; and presenting, by one or more processors, the one or more augmentation images on the graphical user interface based on video frame data and location data associated with the augmentation image."

---

II. Plain-Language Summary of Independent Claims

This patent protects a system and method for displaying interactive, augmented reality-style graphics over live or recorded video, particularly for sports. The core innovation lies in how a user can select an object on the screen (like a player) and have relevant information or graphics appear, seamlessly integrated with the video.

• Independent Claim 1: This claim describes a method for displaying augmented content on a device. It works by:

  1. Receiving video data of an event, which includes both the video itself and a set of defined "bounding boxes" that outline objects within the video (e.g., players on a field).
  2. Showing the video to the user.
  3. When the user touches or clicks on a specific part of the video, the system identifies the exact bounding box that was selected.
  4. Information about the selected bounding box is sent to a "renderer."
  5. The renderer provides an "augmentation image" (e.g., a graphic with player stats) which is then displayed on the screen, correctly positioned with the moving object in the video.

• Independent Claim 9: This claim describes the system that performs the method of Claim 1. It outlines a client device (like a smartphone or computer) equipped with processors and memory to:

  1. Receive video and associated "transparent bounding boxes" from a server.
  2. Display the video on a screen.
  3. Detect when a user selects a part of the screen.
  4. Identify the specific transparent bounding box that corresponds to the user's selection.
  5. Communicate with a renderer to get and then display the appropriate augmented images.

• Independent Claim 15: This claim focuses on the server-side operations. It describes a computer-readable medium (like a server's storage) with instructions to:

  1. Send video and bounding box data for an event to a user's device.
  2. Receive a signal back from the user's device indicating which bounding box has been selected.
  3. Based on this selection, determine which object in the video was chosen.
  4. Check the current "augmentation state" for that object (e.g., if any information is already being displayed).
  5. Select a new or updated augmentation image based on this state.
  6. Determine the precise timing (video frame) and location to display this new image.
  7. Send the new augmentation image and its placement data back to the user's device for display.

---

III. Legal Status and Litigation

A search of the public dockets of the U.S. Court of Appeals for the Federal Circuit (CAFC) for appeals related to U.S. Patent No. 10,769,446, conducted on May 12, 2026, did not yield any results. This suggests that, as of this date, the patent has not been the subject of an appeal to this court. A comprehensive litigation search would require checking district court records. According to the USPTO's public records, the patent is currently listed as "Active."

Based on a thorough review of publicly available litigation databases as of May 12, 2026, there is no record of U.S. Patent No. 10,769,446 being involved in any legal disputes.

It's important to note that the current assignee, Genius Sports SS LLC, and the original assignee, Second Spectrum Inc., have been involved in other intellectual property and commercial disputes. For instance, Genius Sports has been in a high-profile legal battle with Sportradar over exclusive rights to sports data. More recently, Panda Interactive has filed a lawsuit against both Genius Sports and Sportradar, initially for patent infringement and later adding antitrust claims. This case, originally filed in October 2023 in the Texas Eastern District Court, alleges that the companies used their control over official sports data to create anti-competitive conditions in the sports betting market.

However, these legal actions do not specifically name U.S. Patent No. 10,769,446 as a patent-in-suit. Searches of specialized patent litigation databases and federal court records confirm that this particular patent has not been the subject of litigation to date.

Proceedings overview

There are no PTAB trial proceedings (IPR, PGR, or CBM) on file for U.S. Patent No. 10,769,446. Consequently, all claims remain as originally granted, and the patent has not yet been tested in an inter partes review. This presents a clean slate for a potential defendant, who would be the first to challenge the patent's validity before the PTAB.

[No Proceedings Filed]

• Type: N/A
• Filed: N/A
• Status: A search of the USPTO's Patent Trial and Appeal Board (PTAB) End to End (E2E) system and other public dockets as of May 12, 2026, shows that no IPR, PGR, or CBM proceedings have ever been filed against U.S. Patent No. 10,769,446.
• Judge panel: N/A
• Petition grounds: N/A
• Institution decision: N/A
• Final Written Decision: N/A
• Settlement / termination: N/A
• Appeal: N/A
• Defensive value: The patent's validity has not been challenged or affirmed by the PTAB. This means a defendant has the full range of prior art available to build a case for invalidity without facing arguments that the Board has already considered and rejected similar challenges.

Strategic summary

• Claim Status: All claims of U.S. Patent No. 10,769,446 are currently UNTESTED at the PTAB. No claims have been canceled or confirmed. The patent's original scope remains intact.

• Estoppel Landscape: For a company facing an assertion of this patent, the estoppel landscape is entirely open. Since no prior IPRs have been filed, the limitations of 35 U.S.C. § 315(e)(2) do not apply. A defendant is free to file an IPR petition based on any prior art patents or printed publications it deems relevant, without restriction from previous proceedings. The prior art cited during prosecution, particularly US 9,495,788 and US 9,986,227, provides a starting point, but a challenger would need to find art that more directly teaches or suggests the specific method of using unique RGBA values for identifying user-selected bounding boxes.

• Pattern Signals: The absence of PTAB challenges to date suggests the patent has not been heavily litigated or licensed, or that potential licensees have opted for business resolutions rather than validity challenges. The current assignee, Genius Sports SS LLC, is a major operating company in the sports data and technology field, not a non-practicing entity (NPE). This indicates the patent is likely viewed as a strategic asset protecting a core technology, rather than a tool for a pure licensing campaign.

Recommended next steps

As there have been no PTAB proceedings filed against U.S. Patent No. 10,769,446, a defendant's first course of action should be to conduct a comprehensive prior art search. The goal of this search would be to identify patents or printed publications that were not considered by the USPTO examiner and that disclose the key elements of the patent's independent claims (1, 9, and 15).

Specifically, the search should focus on prior art that teaches or suggests:

1. Receiving video data that includes both the video stream and pre-defined bounding boxes for objects within the video.
2. Detecting a user selection on a screen displaying the video.
3. Using a unique identifier, such as a color or RGBA value associated with the bounding box, to determine which object the user selected.
4. Sending this identifier to a renderer or server to retrieve and display context-specific augmented content.

Given the patent's 2014 priority date, a successful invalidity challenge would require identifying art predating this that combines these elements. If strong prior art is found, filing the first IPR against this patent could be a highly effective defensive strategy.

Inventors

Based on the initial patent assignment, the inventors were all affiliated with the original assignee, Second Spectrum, Inc.

• Yu-Han Chang
• Rajiv Tharmeswaran Maheswaran
• Jeffrey Wayne Su
• Emil Dotchevski
• Jason Kent Simon

There is no public information to suggest any unusual departure patterns of the inventors following the patent application.

Original Assignee

Second Spectrum, Inc.

The original assignee was Second Spectrum, Inc., a California-based company known for providing data analytics, visualizations, and video augmentation services to sports leagues, including the NBA and the English Premier League. The company's products, which include player tracking and real-time video analysis, directly embody the technologies described in this patent. In May 2021, Second Spectrum was acquired by Genius Sports Group, and it appears to operate as a subsidiary or integrated unit.

Assignment Timeline

• 2020-02-19 (recorded 2020-02-25) — Reel 053673/0748
  • Conveyance: Assignment of Assignor's Interest
  • Assignor: Yu-Han Chang, Rajiv Tharmeswaran Maheswaran, Jeffrey Wayne Su, Emil Dotchevski, Jason Kent Simon
  • Assignee: Second Spectrum, Inc.
  • Correspondent: Lewis Roca Rothgerber Christie LLP, Glendale, CA
  • Context: Routine assignment from inventors to their employer at the time of filing.
• 2021-06-02 (recorded 2023-01-23) — Reel 064434/0833
  • Conveyance: Merger
  • Assignor: Second Spectrum, Inc.
  • Assignee: GENIUS SPORTS SS, LLC
  • Correspondent: KILPATRICK TOWNSEND & STOCKTON LLP, Atlanta, GA
  • Context: Transfer of assets resulting from the acquisition of Second Spectrum, Inc. by Genius Sports Group.
• 2022-12-14 (recorded 2024-05-01) — Reel 065261/0861
  • Conveyance: Security Interest
  • Assignor: GENIUS SPORTS SS, LLC
  • Assignee: CITIBANK, N.A.
  • Correspondent: John C. Ralls, Pearlstone, Dallas, TX
  • Context: The patent was pledged as collateral to secure a financial agreement, a standard business practice for operating companies.

Timeline diagram

timeline
    title Ownership of US 10769446
    2020 : Filed and assigned to Second Spectrum Inc
    2021 : Genius Sports acquires Second Spectrum Inc
    2022 : Pledged as collateral to Citibank NA
    2023 : Merger with Genius Sports SS LLC recorded
    2024 : Security interest recorded

NPE / troll-pattern signals

1. Shell-entity transfer: Not present. The assignment from Second Spectrum to Genius Sports SS, LLC was the result of a merger between two well-established operating companies in the sports technology sector.
2. Known asserter in the chain: Not present. Neither Second Spectrum, Inc. nor Genius Sports SS, LLC are on public lists of patent assertion entities.
3. Repeat correspondent across the chain: Not present. The three recorded assignments were handled by three different law firms, which is consistent with distinct events: initial patent prosecution, a corporate merger, and a financing deal.
4. Cascading transfers: Not present. The transfers are separated by significant time and correspond to standard corporate activities (acquisition and financing).
5. Pre-litigation transfer: Not present. A search of litigation databases shows no record of this patent being asserted in any lawsuit.
6. Bankruptcy fire-sale: Not present. The transfer was part of a strategic acquisition, not a bankruptcy proceeding.
7. Privateering: Not present. The patent remains with the acquiring operating company, Genius Sports.
8. Defensive aggregator (anti-NPE): Not present. The patent is held by an operating company, not a defensive aggregator.

Verdict

Insufficient data

The assignment history shows a clear and conventional chain of ownership from the inventors to their employer, Second Spectrum, Inc., and then to its acquirer, Genius Sports SS, LLC. Both entities are operating companies that commercialize the technology covered by the patent. While there is no evidence of litigation, the "Operating-company assertion" category is not a perfect fit as there is no active assertion. The patent is currently held as a standard corporate asset, and there are no signals whatsoever of non-practicing entity (NPE) or "patent troll" activity.

Verify at: USPTO Patent Assignment Search

Here is a detailed analysis of the prior art cited in U.S. Patent No. 10,769,446, focusing on the most relevant references and their potential impact on the patent's claims.

Analysis of Prior Art for U.S. Patent No. 10,769,446

The following analysis details the prior art references cited during the prosecution of U.S. Patent No. 10,769,446. Each reference is assessed for its relevance to the patent's independent claims, particularly claims 1, 9, and 15, which describe a method, system, and non-transitory computer-readable medium for combining video content with augmentations based on user interaction with defined bounding boxes.

---

Most Relevant Prior Art

1. U.S. Patent No. 9,495,788 B2 ("the '788 patent")

• Full Citation: US 9,495,788 B2, "System and method for augmenting a broadcast video with secondary content related to a user-selected object"

• Assignee: Rovi Guides, Inc.

• Filed: March 13, 2013

• Issued: November 15, 2016

• Brief Description: The '788 patent describes a system that identifies objects in a video stream, associates them with supplementary information, and allows a user to select an object on-screen to retrieve and display that information. It explicitly mentions identifying an object's location, receiving a user selection corresponding to that location, and presenting related content.

• Potential Anticipation Analysis:

  • Claim 1 (Method): This reference appears to disclose most elements of claim 1. It teaches displaying a video, identifying objects (which implies a form of bounding box or coordinate system), detecting a user's selection of an object, and displaying augmented information related to that object. The key difference may lie in the specific method of identifying the selected box using RGBA values, which is a specific implementation detail in the '446 patent. However, the broader process is substantially described.
  • Claim 9 (System): The system described in the '788 patent, involving a user device that receives video, processes user input, and displays augmented content, aligns closely with the system defined in claim 9.
  • Claim 15 (Server-Side Method): The '788 patent discloses a system where a server can provide the video and associated metadata (e.g., object locations and related content) to a client device. It describes the client sending a selection back and the system providing the relevant secondary content. This mirrors the server-client interaction described in claim 15.

  Conclusion: The '788 patent is highly relevant and could be argued to anticipate the core concept of claims 1, 9, and 15. The novelty of the '446 patent may rest on the specific technical implementation, such as the use of RGBA values for identifying transparent bounding boxes.

2. U.S. Patent No. 9,986,227 B2 ("the '227 patent")

• Full Citation: US 9,986,227 B2, "Augmented reality system including a broadcast production system for a live event"

• Assignee: Replay Technologies Ltd.

• Filed: October 26, 2015

• Issued: May 29, 2018

• Brief Description: This patent details a system for creating augmented reality views of live events, particularly sports. It involves capturing video from multiple angles, generating a 3D model of the event, and overlaying graphical information. The system allows users to interact with the broadcast, for example, by selecting a player to see their statistics.

• Potential Anticipation Analysis:

  • Claim 1 (Method): The '227 patent describes presenting video of an event and allowing user selection of objects (players) to display augmentations (statistics). This covers the fundamental process of claim 1. While it focuses on a 3D reconstructed environment, the core interaction model is similar. The method for linking a user's screen tap to a specific player is not detailed as using RGBA values but is functionally equivalent.
  • Claim 9 (System): The client device in the '227 patent's system is designed to receive video, detect user input on displayed objects, and render augmented reality content, which is analogous to the system described in claim 9.
  • Claim 15 (Server-Side Method): The '227 patent's architecture relies on a server-side component (the "broadcast production system") to process video feeds, identify objects, and manage the data for augmentations, which is then sent to client devices. This aligns with the server-side process of claim 15.

  Conclusion: The '227 patent is also highly relevant. It describes a sophisticated system for real-time sports video augmentation with user interaction. The patentability of the '446 invention likely hinges on the specific, more efficient method of using invisible bounding boxes and RGBA values for object identification, as opposed to the more complex 3D modeling and object recognition described in the '227 patent.

3. U.S. Patent Application Publication No. 2013/0212530 A1 ("Mistry et al.")

• Full Citation: US 2013/0212530 A1, "System and Method for Presenting Supplemental Information for Television Content"

• Applicant: Pranav Mistry et al.

• Filed: February 13, 2012

• Published: August 15, 2013

• Brief Description: This application discloses a system that allows a user to interact with video content displayed on a television using a second-screen device (e.g., a smartphone). The user can point their device at an object or person on the TV screen, and the system identifies the object and displays related information on the second screen.

• Potential Anticipation Analysis:

  • Claim 1 (Method): Mistry et al. describe identifying objects within video frames, allowing a user to select an object, and presenting supplemental content. The core interactive loop is present. The mechanism for selection involves image recognition from the second device's camera rather than a direct screen tap on a bounding box, which is a key distinction.
  • Claim 9 (System): The system involves a client device for interaction and display, but it operates as a second screen, which differs from the integrated display system described in claim 9 of the '446 patent.
  • Claim 15 (Server-Side Method): A server component is described for processing the video, identifying objects, and providing the supplemental information. This has a strong functional overlap with claim 15.

  Conclusion: Mistry et al. is relevant as it describes the concept of selecting objects in a video to get more information. However, its reliance on a two-screen system and a different selection mechanism (camera-based pointing) differentiates it from the '446 patent's single-screen, direct-touch, bounding-box-based method. It weakens the non-obviousness of the general concept but may not directly anticipate the specific claims.

Less Relevant Prior Art

The remaining references cited against the patent, while related to the general fields of video analysis, augmented reality, and data visualization, are less directly relevant to the specific combination of claim elements.

• U.S. Patent No. 9,491,424 B2 (KudMost et al.): Focuses on synchronizing video content with external data sources but does not detail the specific user interaction mechanism of selecting objects within the video via bounding boxes to trigger augmentations.
• U.S. Patent No. 8,935,720 B2 (Ellis et al.): Relates to providing interactive television advertising and supplementary content, but the method of interaction and object identification is not as specific as that claimed in the '446 patent.
• U.S. Patent No. 9,697,643 B2 (Hsu et al.): Describes generating augmented reality views by inserting virtual objects into a real-world video feed, which is a different problem from overlaying informational graphics onto existing objects within a video.
• U.S. Patent Application Publication No. 2013/0014072 A1 (Gilley et al.): Pertains to a system for presenting an "enhancement layer" over video content, which can be user-customized. While it involves overlaying information, the specific method of identifying selectable objects using invisible, color-coded bounding boxes is not disclosed.

Overall Assessment

The core invention of US 10,769,446 resides in a specific, efficient method for enabling user interaction with objects in a video stream. By pre-defining transparent bounding boxes, each with a unique identifier (encoded as an RGBA value), the system can quickly determine which object a user has selected without complex image processing on the client device at the time of interaction. The client simply has to read the RGBA value at the point of user input and send that identifier to a renderer or server.

The prior art, particularly US 9,495,788 B2 and US 9,986,227 B2, clearly establishes the broader concept of selecting objects in a video (especially a sports broadcast) to view related data. The patentability of the '446 patent's claims, therefore, likely depends on the novelty and non-obviousness of using this specific RGBA-based bounding box identification technique as the enabling mechanism for this interaction. An argument could be made that this is a specific, inventive implementation that offers advantages in speed and processing efficiency over the more general object recognition or 3D modeling systems described in the prior art.

Obviousness Analysis of U.S. Patent No. 10,769,446

To: In-house Counsel
From: Senior Patent Analyst
Date: May 12, 2026
Subject: Obviousness Analysis of U.S. Patent No. 10,769,446

I. Executive Summary

This analysis concludes that the independent claims of U.S. Patent No. 10,769,446 ("the '446 patent") are likely invalid as obvious under 35 U.S.C. § 103. The core invention—using invisible, color-coded bounding boxes to identify user-selected objects in a video for augmentation—represents an obvious implementation choice for a well-established concept.

The primary argument for obviousness combines U.S. Patent No. 9,495,788 B2 ("the '788 patent") with the well-known computer graphics technique of "ID Buffering" or "Color Picking." The '788 patent teaches the broader system of displaying augmented information in response to a user selecting an object within a video. The use of color-coded IDs to handle the selection mechanism was a common, efficient, and widely understood technique in the field of computer graphics at the time of the invention. A Person of Ordinary Skill in the Art (POSITA) would have been motivated to combine these teachings to create the claimed system for reasons of computational efficiency and straightforward implementation.

II. Definition of a Person of Ordinary Skill in the Art (POSITA)

For the purposes of this analysis, a POSITA at the time of the invention (priority date of Feb. 28, 2014) would be an individual with a Bachelor's degree in Computer Science or a related field, and approximately 2-5 years of professional experience in software development, particularly in areas involving computer graphics, user interface (UI) design, and/or video processing. Such a person would be familiar with standard algorithms for object detection and selection (hit-testing) in graphical environments.

III. Obviousness Analysis

The claims of the '446 patent are directed to a method, system, and non-transitory computer-readable medium for augmenting video content. The key inventive concept appears to be the specific mechanism for identifying which object a user has selected: rendering invisible bounding boxes over objects, each with a unique RGBA color value, and then reading the RGBA value at the coordinates of the user's touch or click to identify the object.

Primary Combination: U.S. Patent No. 9,495,788 B2 ("the '788 patent") in view of the Known Technique of Color Picking/ID Buffering

1. Base Reference: The '788 patent

The '788 patent, assigned to Rovi Guides, Inc., clearly discloses the foundational elements of the '446 patent's claims. It teaches a system that:

• Receives and displays a video stream (e.g., a sports broadcast).
• Identifies objects of interest within the video.
• Allows a user to select a specific object on the screen.
• In response to the selection, displays secondary content related to that object.

This teaching establishes a clear framework for interactive, augmented video. However, the '788 patent does not specify the precise technical method used for linking the user's input (a tap or click) to the identified object. It creates the problem that the '446 patent purports to solve: how to efficiently and accurately perform this "hit detection" on a video stream.

2. Secondary Teaching: The Well-Known Technique of Color Picking / ID Buffering

At the time of the invention, "color picking" was a well-established and widely used technique in computer graphics for identifying objects selected by a mouse click or screen tap. The method involves rendering a second, hidden scene (an "ID buffer") where each selectable object is drawn as a solid, unique color. When the user clicks, the system looks up the color of the pixel at the click coordinates in this hidden buffer. Since each object has a unique color, this color value serves as a direct and unambiguous identifier for the selected object. This technique was known to be significantly faster and less computationally intensive than geometric methods like ray-tracing, especially in complex scenes.

3. Motivation to Combine

A POSITA, tasked with building the interactive video system described in the '788 patent, would need to implement a selection mechanism. The most common and performance-oriented challenge in such a system is real-time processing. A method that requires complex on-the-fly analysis of video frames to identify which object a user tapped would be slow and inefficient on client devices, especially the mobile devices of that era.

The color-picking/ID buffer technique offers a direct and elegant solution. A POSITA would have been motivated to use this known technique for the following reasons:

• Computational Efficiency: It offloads the complex object identification to a simple color lookup, which is extremely fast and has a low processing overhead on the client device.
• Simplicity of Implementation: The logic is straightforward: render a hidden layer of colored identifiers and map the user's input coordinates to that layer.
• Scalability: It works well regardless of the complexity or number of objects on the screen.

Applying this technique to the system of the '788 patent would logically lead to the exact method described in the '446 patent. The "bounding boxes" of the '446 patent are the simple shapes rendered in the ID buffer, and the "RGBA value" is the unique color used as an identifier. Making these bounding boxes invisible by setting the alpha channel to zero is an obvious and necessary step to ensure the ID buffer does not obstruct the user's view of the primary video content.

Claim-by-Claim Analysis (Mapping onto Independent Claim 1)

Claim Element	How the Combination Renders it Obvious
"...receiving... video data... comprising video content and a plurality of definitions of bounding boxes;"	The '788 patent teaches receiving video and identifying objects within it. Defining bounding boxes is a standard way to represent the location of identified objects.
"...presenting... the video content;"	Directly taught by the '788 patent.
"...detecting... a user selection of a portion of the graphical user interface;"	Directly taught by the '788 patent.
"...determining... a red, green, blue, alpha (RGBA) value associated with the user selection..."	This is the core step of the well-known color-picking/ID buffer technique, which a POSITA would be motivated to use to implement the selection mechanism of the '788 patent.
"...determining... a bounding box RGBA value that corresponds..."	The '446 patent's "bounding box RGBA value" is simply the unique color ID used in the color-picking technique. A POSITA would obviously assign a unique color to each bounding box to make the technique work.
"...transmitting... an indication of the particular bounding box to a renderer;"	This is the logical result of the identification step. Once the object is identified via its unique color/bounding box, that ID is sent to the part of the system responsible for fetching and displaying the augmentation, as taught by the '788 patent.
"...receiving... one or more augmentation images;"	This is the "secondary content" or "supplemental information" described in the '788 patent.
"...presenting... the one or more augmentation images..."	The end goal of the system taught by the '788 patent is to display this information to the user.

This analysis applies with equal force to independent claims 9 (the system) and 15 (the server-side method), which are different forms of the same underlying inventive concept.

IV. Conclusion

The '446 patent claims a specific implementation of an interactive video system. The prior art, particularly the '788 patent, already disclosed the high-level concept of such a system. The specific implementation method—using invisible, color-coded bounding boxes for object selection—was a well-known, efficient, and common technique in the field of computer graphics. The combination of these elements would have been obvious to a person of ordinary skill in the art seeking to build a responsive and performant interactive video application. Therefore, the claims of U.S. Patent No. 10,769,446 are likely invalid for obviousness.

Patent Term and Family Status

Based on a review of the United States Patent and Trademark Office (USPTO) records for U.S. Patent No. 10,769,446 ('446 patent), the following information details its term, related applications, and family history.

Patent Term Adjustment (PTA) / Extension (PTE)

• Patent Term Adjustment (PTA): The '446 patent was granted a Patent Term Adjustment of 0 days. The USPTO determined that there were no administrative delays during the prosecution of this application that would warrant an extension of the patent term.
• Patent Term Extension (PTE): There is no indication of any Patent Term Extension (PTE) being sought or granted for this patent. PTE is typically associated with delays in regulatory review for products like pharmaceuticals and medical devices and is not applicable here.

Projected Expiration Date

The '446 patent is a continuation of a series of applications that claim priority back to U.S. Provisional Application No. 61/946,143, filed on February 28, 2014. The 20-year term of a patent is calculated from the filing date of the earliest non-provisional application in its family chain. In this case, that date is February 27, 2015, from U.S. Patent Application No. 14/634,070 (now U.S. Patent No. 9,984,249).

• Earliest Non-Provisional Filing Date: February 27, 2015
• Standard 20-Year Term: Expires February 27, 2035
• Patent Term Adjustment (PTA): + 0 days

The projected expiration date for U.S. Patent No. 10,769,446 is February 27, 2035.

Continuity and Family Data

This patent is part of a larger family of applications, indicating a strategy to build a comprehensive portfolio around this technology.

Continuation History:

U.S. Patent No. 10,769,446, which issued from application 16/795,834, is part of a chain of "continuing applications." This means it shares the same core specification as its parent applications but typically claims different aspects of the invention.

• This application (16/795,834) is a continuation of:
  • U.S. Application No. 16/675,799, filed November 6, 2019 (now U.S. Patent No. 10,713,494)
• Which is a continuation of:
  • U.S. Application No. 15/586,379, filed May 4, 2017 (now U.S. Patent No. 10,521,671)
• Which is a continuation of:
  • U.S. Application No. 14/634,070, filed February 27, 2015 (now U.S. Patent No. 9,984,249)
• Which claims priority to:
  • U.S. Provisional Application No. 61/946,143, filed February 28, 2014.

Divisional Applications:

There are no divisional applications directly related to application 16/795,834 noted in the USPTO records.

Patent Family Summary:

The core invention is protected by a family of U.S. patents stemming from the original 2015 non-provisional application. This family includes at least:

• U.S. Patent No. 9,984,249
• U.S. Patent No. 10,521,671
• U.S. Patent No. 10,713,494
• U.S. Patent No. 10,769,446 (the patent in question)
• U.S. Patent No. 11,120,271 (Continuation of '446)
• U.S. Patent No. 11,373,405 (Continuation of '446)
• U.S. Patent No. 11,861,906 (Continuation of '446)
• U.S. Patent No. 12,394,201 (Continuation of '446)
• U.S. Patent No. 12,190,585 (Continuation of '446)

This extensive family of patents indicates a deliberate strategy to protect various aspects and improvements of the core technology related to analyzing video and overlaying augmentations.

Defensive Disclosure and Prior Art Derivations for U.S. Patent No. 10,769,446

Publication Date: May 12, 2026
Subject Matter: Enhancements, variations, and alternative embodiments for systems and methods of combining video content with one or more augmentations.

This document discloses further inventions and improvements that build upon, or provide alternative implementations for, the technology described in U.S. Patent No. 10,769,446 ("the '446 patent"). The purpose of this disclosure is to place these concepts into the public domain, thereby establishing them as prior art for any future patent applications.

The core concept of the '446 patent involves a client-server architecture where video content is presented with a synchronized, invisible layer of bounding boxes. Each box is assigned a unique identifier (e.g., an RGBA color value), allowing a user's selection on the screen to be rapidly mapped to a specific object. This triggers the display of augmented information. The following disclosures expand upon this concept across various dimensions.

---

Axis 1: Material & Component Substitution

1.1. Unique Identifier Encoding via Non-Visible Light and Steganography

Enabling Description: Instead of using RGBA values in a parallel, invisible frame buffer (the "ID buffer"), object identifiers are encoded directly into the Luma (Y') channel of the YCbCr color space of the video stream itself. By subtly modulating the luma value of pixels within a bounding box by an imperceptible amount (e.g., ±1 on an 8-bit scale), a unique binary ID can be embedded. The client device, upon user selection, would not sample a separate buffer but would read the luma values of the pixels at the selected coordinates, decode the embedded ID, and request the corresponding augmentation. This method eliminates the need for a secondary video-like stream for the bounding boxes, reducing bandwidth. An alternative uses steganography, embedding the object ID in the least significant bits (LSB) of the color data of the video frame's pixels within the bounding box.

sequenceDiagram
    participant User
    participant ClientDevice as Client Device
    participant Renderer as Augmentation Renderer

    User->>+ClientDevice: Taps screen at (x,y)
    ClientDevice->>ClientDevice: Read Luma/LSB values at (x,y)
    ClientDevice->>ClientDevice: Decode Object ID from pixel data
    ClientDevice->>+Renderer: Request Augmentation for Object ID
    Renderer-->>-ClientDevice: Return Augmentation Data
    ClientDevice->>User: Display Augmentation over Video

1.2. Haptic Feedback Grids as Bounding Box Proxies

Enabling Description: For applications on devices with advanced haptic feedback systems, the bounding box information is used to create a "haptic texture map." Instead of a visual selection, the user drags their finger across the screen. As their finger enters a region corresponding to a tagged object, the device provides a distinct haptic feedback pattern (e.g., a specific vibration frequency or texture). The haptic pattern's unique signature, tied to the object ID, is used to identify the selection upon a secondary action, like a firm press or double-tap. This method is particularly useful for accessibility or in environments where the user's visual attention cannot be diverted.

stateDiagram-v2
    [*] --> NoInteraction
    NoInteraction --> FingerDown: User touches screen
    FingerDown --> Dragging: User moves finger
    Dragging --> HapticFeedback: Finger enters bounding box area
    HapticFeedback --> Dragging: Finger leaves bounding box area
    HapticFeedback --> SelectionMade: User performs confirmation action (e.g., double-tap)
    SelectionMade --> NoInteraction: Augmentation displayed

1.3. Compressed Vector-Based Bounding Box Definitions

Enabling Description: Rather than sending a full-frame ID buffer or pixel-level data, the server transmits a separate, time-synchronized metadata stream containing vectorized representations of the bounding boxes. These vectors could be defined using Scalable Vector Graphics (SVG) path data or a compact binary format describing polygons. The client-side renderer reconstructs these invisible vector shapes on a canvas overlaid on the video. When a user taps the screen, the client performs a point-in-polygon test against the vector data to determine which object was selected. This significantly reduces the data overhead compared to a full-frame RGBA buffer, especially for high-resolution video with few tracked objects.

graph TD
    A[Server: Encodes Video] --> B(Server: Generates Vector Bounding Box Data);
    B --> C{Metadata Stream (SVG/Custom Format)};
    A --> D{Video Stream};
    C --> E[Client Device];
    D --> E;
    E --> F[Render Video];
    E --> G[Render Invisible Vector Shapes];
    F & G --> H(Composite View);
    I[User Tap] --> H;
    H --> J{Point-in-Polygon Test};
    J --> K[Identify Selected Object ID];
    K --> L[Request/Display Augmentation];

---

Axis 2: Operational Parameter Expansion

2.1. High-Frequency (240fps+) Video for Slow-Motion Analysis

Enabling Description: In high-speed sports analysis (e.g., golf swing, baseball pitch), the system processes video at 240 frames per second or higher. The bounding box data stream is time-coded with sub-millisecond precision to match each video frame. To handle the data rate, the server pre-processes the video to generate keyframes for object location, and the client-side renderer interpolates the bounding box positions between these keyframes using a kinematic model (e.g., a Kalman filter) that predicts object trajectory. When the user pauses the video and scrubs frame-by-frame through a slow-motion replay, the bounding boxes remain perfectly synchronized, allowing for precise selection and retrieval of frame-accurate performance metrics.

graph TD
    A[High-FPS Video (240Hz)] --> B(Server: Keyframe Extraction);
    B --> C(Server: Bounding Box Generation @ Keyframes);
    C --> D[Client: Receives Video & Sparse Box Data];
    D --> E{Client: Interpolates Box Positions (Kalman Filter)};
    E --> F(Client: Renders Video & Synced Boxes @ 240Hz);
    G[User Interaction: Pause, Scrub] --> F;
    F --> H(Display Frame-Specific Augmentation);

2.2. Distributed, Low-Bandwidth "Edge Rendering" System

Enabling Description: In a low-bandwidth environment (e.g., mobile network in a stadium), the full video stream is not sent to a central server. Instead, on-site edge computing nodes, connected to the broadcast cameras, perform the object detection and generate the bounding box metadata. The client device receives a compressed video stream from a standard CDN, and a separate, very low-bandwidth data stream containing the bounding box definitions from the nearest edge node. The augmentation content (e.g., player stats) is also cached at the edge. When a user selects a bounding box, the request is sent to the local edge node, not a distant cloud server, which returns the augmentation. This minimizes latency and reduces the core network load.

sequenceDiagram
    participant UserDevice
    participant EdgeNode
    participant CloudServer

    CloudServer-->>EdgeNode: Pre-cache Augmentation Data
    EdgeNode->>UserDevice: Low-bandwidth Bounding Box Stream
    CloudServer-->>UserDevice: Compressed Video Stream (via CDN)

    UserDevice->>UserDevice: User selects object
    UserDevice->>EdgeNode: Request Augmentation (Object ID)
    EdgeNode-->>UserDevice: Return Aug-Data (Low Latency)

2.3. Nanoscale Microscopy Video Augmentation

Enabling Description: The system is adapted for real-time analysis of video from an electron microscope. The "objects" are cellular structures, organelles, or nanoparticles. The bounding box definitions are generated by a real-time image analysis algorithm trained to identify these microscopic features. A researcher viewing the live feed can tap on a specific mitochondrion, for example. The system uses its pre-assigned unique ID (from the RGBA buffer) to retrieve and overlay data from other sensors connected to the microscopy equipment, such as fluorescence intensity, chemical signatures from a spectrometer, or its movement vector over the last 1000 frames. This allows for immediate, in-context data correlation during live experiments.

graph TD
    A[Electron Microscope Video Feed] --> B(Image Analysis Server);
    B --> C{Real-time Organelle Identification};
    C --> D(Generates RGBA ID Mask);
    A --> E[Researcher's Display];
    D --> E;
    E --> F(User Taps on a Mitochondrion);
    F --> G{Read RGBA Value};
    G --> H{Request Data for ID};
    I[Spectrometer/Sensor Data] --> H;
    H --> J(Overlay Fluorescence Data on Display);

---

Axis 3: Cross-Domain Application

3.1. Aerospace: Augmented Reality for Aircraft Maintenance

Enabling Description: A maintenance technician points a tablet at an aircraft engine. The tablet's camera feed is processed in real-time. A server, using a 3D CAD model of the engine, identifies key components (e.g., fuel lines, sensor connections, turbine blades) and sends bounding polygons for each visible component back to the tablet. The technician taps on a specific hydraulic line. The client device identifies the component via the invisible ID layer and overlays its maintenance history, required torque specifications, and the next scheduled inspection date directly onto the video feed.

graph TD
    subgraph On-Site Technician
        A(Tablet Camera) --> B{Live Video Feed};
    end
    subgraph Maintenance Server
        C(3D CAD Model) -- Identifies --> D{Component Recognition};
        D -- Generates --> E(Bounding Polygon Data);
    end
    subgraph Tablet Display
        B -- Renders --> F[Video Display];
        E -- Renders --> G(Invisible ID Overlay);
        F & G --> H((Composite View));
    end
    I(Technician Taps Component) --> H;
    H --> J{ID Lookup};
    J --> K[Request Maintenance Data];
    L[Maintenance Database] --> K;
    K --> M(Display Specs & History);
    M -- Overlays --> H;

3.2. Agriculture (AgTech): Precision Livestock Monitoring

Enabling Description: Video feeds from cameras in a large-scale dairy farm are streamed to a central monitoring system. An animal recognition algorithm places a unique bounding box around each cow. A veterinarian or farm manager can view the live feed on a tablet. By tapping on a specific cow, they can instantly pull up an augmented reality overlay showing that cow's health records, milk yield, last insemination date, and any current health alerts (e.g., from an IoT-enabled collar). The selection is made by decoding the RGBA value of the transparent bounding box associated with that animal in the video frame.

sequenceDiagram
    participant FarmCamera
    participant RecognitionServer
    participant VetTablet
    participant FarmDB

    loop Live Feed
        FarmCamera->>+RecognitionServer: Send Video Frame
        RecognitionServer->>RecognitionServer: Identify Cows, Assign IDs
        RecognitionServer-->>-VetTablet: Stream Video + Bounding Box Data
    end
    VetTablet->>VetTablet: Vet Taps a Cow
    VetTablet->>+FarmDB: Request Record for Cow ID
    FarmDB-->>-VetTablet: Return Health & Production Data
    VetTablet->>VetTablet: Overlay Data on Video

3.3. Consumer Electronics: Interactive Cooking Tutorials

Enabling Description: A smart display in a kitchen streams a cooking class. As the chef introduces ingredients and uses equipment, an AI on the server identifies each item (e.g., "salt," "whisk," "mixing bowl") and creates a corresponding time-synced bounding box. The user, following along, can tap on the whisk in the video. The system identifies the "whisk" object and displays an augmentation with a link to purchase that specific whisk online, or a pop-up video demonstrating the proper whisking technique. The selection mechanism uses an invisible RGBA overlay, ensuring the primary video is unobstructed.

graph TD
    A[Cooking Video Stream] --> B(Server-side Object Recognition);
    B -- Identifies --> C(Ingredients & Utensils);
    C --> D(Generate Bounding Box Metadata);
    A & D --> E[User's Smart Display];
    E --> F(User Taps on 'Whisk');
    F --> G{Decode ID from Tap Coords};
    G --> H[Request 'Whisk' Augmentation];
    H --> I(Display Purchase Link / Technique Video);

---

Axis 4: Integration with Emerging Tech

4.1. AI-Driven Predictive Bounding Boxes & Augmentation

Enabling Description: A machine learning model analyzes player and object trajectories from the spatiotemporal data. It predicts the screen position of all tracked objects for the next N frames (e.g., N=10). The server sends not just the current bounding box data, but also the predicted trajectory vectors for each box. The client device uses these vectors to extrapolate the box positions, ensuring smooth, accurate tracking even with network latency or dropped packets in the metadata stream. A second AI model analyzes user interaction patterns (e.g., which player stats are most frequently requested) and pre-fetches the most likely augmentation data to the client device before the user even makes a selection, resulting in near-instantaneous display of information.

graph TD
    subgraph Server
        A[Live Spatiotemporal Data] --> B(Predictive Tracking Model);
        B --> C(Generates Bounding Boxes + Trajectory Vectors);
        A --> D(User Behavior Model);
        D --> E(Pre-fetches Probable Augmentations);
    end
    subgraph Client
        C --> F[Receive Boxes & Vectors];
        E --> G[Cache Augmentations];
        F --> H{Render & Extrapolate Box Positions};
        I[User Taps Player] --> H;
        H --> J{Lookup ID};
        J --> K[Retrieve Augmentation from Cache];
        K --> L(Display Instantly);
    end

4.2. IoT-Enhanced Live Event Augmentation

Enabling Description: During a live auto race, each car is equipped with IoT sensors transmitting real-time data (tire pressure, engine temperature, G-forces). This data is ingested by a central server. The system described in the '446 patent is used to track the cars on the video broadcast. The bounding box for each car is linked not to a static database, but to the live IoT data stream for that vehicle. When a viewer taps on a car, the augmentation overlay displays a live telemetry dashboard for that specific vehicle, showing real-time G-force in a corner or a warning if tire temperatures are critical. The renderer acts as an aggregator, pulling video-frame data, bounding box IDs, and live IoT data together to generate the composite view.

sequenceDiagram
    participant CarIoT as "Car IoT Sensors"
    participant CentralServer as "Data Ingestion Server"
    participant Renderer
    participant ClientDevice

    CarIoT->>CentralServer: Stream Real-Time Telemetry
    CentralServer->>Renderer: Forward Telemetry by Car ID
    Renderer->>ClientDevice: Send Video + Bounding Boxes

    activate ClientDevice
    ClientDevice->>Renderer: User selects Car (sends Bounding Box ID)
    activate Renderer
    Renderer->>Renderer: Correlate ID with Live Telemetry
    Renderer-->>ClientDevice: Send Augmentation (Live Data)
    deactivate Renderer
    ClientDevice->>ClientDevice: Display Real-Time Telemetry Overlay
    deactivate ClientDevice

4.3. Blockchain-Verified Authenticity for Digital Collectibles

Enabling Description: In a system for viewing or trading digital art or sports memorabilia (e.g., NFTs), a video showcases the item. A bounding box is placed around the digital asset. When a user taps the asset, the client sends the object's unique ID to a server. The server uses this ID to query a public blockchain to verify the asset's authenticity, ownership history, and provenance. The augmentation returned to the user is a visually secure, non-editable graphic displaying the "Verified on Blockchain" status along with the transaction history. This provides immediate, trustworthy validation of digital goods within an interactive video context.

graph TD
    A[Video of Digital Asset] --> B{Server Adds Bounding Box ID};
    B --> C[Stream to User];
    subgraph User Interaction
        C --> D(User Taps on Asset);
        D --> E{Client Extracts ID};
    end
    E --> F[Server Receives ID];
    F --> G(Query Blockchain for ID);
    G --> H[Retrieve Provenance Data];
    H --> I(Generate Verification Augmentation);
    I --> J[Send Augmentation to Client];
    J --> K(Display on User's Screen);

---

Axis 5: The "Inverse" or Failure Mode

5.1. Graceful Degradation Mode for Low-Power/Low-Bandwidth

Enabling Description: The client device monitors its own performance (e.g., CPU load, frame-rate drops) and network quality (e.g., latency, jitter). If performance degrades below a set threshold, the augmentation system automatically switches to a "low-fidelity" mode. In this mode, instead of requesting and rendering complex animated graphics, the client requests a simple text-only version of the augmentation data. For example, instead of a player's animated shot chart, it would display "Player X: 6/10 FG, 3/5 3PT." The invisible bounding boxes may also be simplified from complex polygons to simple rectangles to reduce the client-side rendering load for the hit-test. This ensures the core functionality remains available without crashing or lagging the device.

stateDiagram-v2
    state "High-Fidelity Mode" as HiFi
    state "Low-Fidelity Mode" as LoFi

    [*] --> HiFi : Initial State
    HiFi --> LoFi: Detects High CPU / Low Bandwidth
    LoFi --> HiFi: Detects Improved Performance

    HiFi: Renders Animated Graphics
    HiFi: Uses Complex Bounding Polygons
    LoFi: Renders Text-Only Data
    LoFi: Uses Simple Bounding Rectangles

5.2. Haptic/Audio Feedback on Selection Failure

Enabling Description: To provide clear feedback when an augmentation fails to load, the system incorporates a failure-mode response. When a user selects a valid bounding box, the client sends the request to the renderer. If the renderer fails to respond within a timeout period (e.g., 500ms) or returns an error, the client device provides immediate non-visual feedback. This can be a short, sharp vibration (haptic alert) and/or a subtle, non-intrusive audio cue (e.g., a soft "click" or "error" tone). This differentiates a failed data fetch from a tap on an non-interactive area of the screen, improving the user experience by managing expectations and communicating system status.

sequenceDiagram
    participant User
    participant ClientDevice
    participant Renderer

    User->>+ClientDevice: Tap on Bounding Box
    ClientDevice->>+Renderer: Request Augmentation
    alt Successful Response
        Renderer-->>ClientDevice: Augmentation Data
        ClientDevice-->>-User: Display Augmentation
    else Timeout or Error
        Renderer--xClientDevice: Error / No Response
        ClientDevice->>ClientDevice: Trigger Haptic Pulse
        ClientDevice->>ClientDevice: Play Audio Cue
        ClientDevice-->>-User: Display "Info Unavailable"
    end

5.3. Static Fallback Cache System

Enabling Description: Before an event (e.g., a sports game) begins, the server pushes a lightweight "static fallback cache" to all client devices. This cache contains basic, non-real-time augmentation data for key objects (e.g., player names, jersey numbers, and basic season stats). During the live event, if the client requests a dynamic, real-time augmentation and the connection to the renderer fails, the client's augmentation module will automatically check its local cache for the selected object's ID. If a match is found, it will display the pre-loaded static data with a visual indicator (e.g., a "cached" icon) to signify that the information is not live. This ensures that the user receives some relevant information even during network interruptions.

graph TD
    A[Pre-Game] --> B(Server Pushes Static Data Cache to Client);
    C[Live Game] --> D{User Taps Player};
    D --> E{Request Live Augmentation from Server};
    E -- Fails --> F{Check Local Cache for Player ID};
    F -- ID Found --> G[Display Static Data];
    E -- Succeeds --> H[Display Live Data];

---

Combination Prior Art Scenarios

1. Combination with WebGL and ONNX.js: The core method of the '446 patent is combined with open web standards. A web browser-based client receives the video stream and bounding box data. The bounding box ID buffer is rendered to an off-screen <canvas> element using WebGL for GPU-accelerated performance. Upon user click, the RGBA value is read from this canvas. The object ID is then fed into a client-side ONNX.js (Open Neural Network Exchange for JavaScript) runtime, which executes a pre-trained machine learning model to generate a context-aware augmentation locally, reducing server-side load. For example, the model could predict a basketball player's shot probability from their current on-court position (derived from the bounding box) and display it.

2. Combination with GStreamer and RTP: A broadcast system uses the open-source GStreamer framework for video processing. A custom GStreamer plugin is developed to analyze the video, perform object detection (e.g., using a YOLO model), and embed the bounding box data as a separate data track within the Real-time Transport Protocol (RTP) stream (per RFC 2326), alongside the audio and video tracks. A client application, also built with GStreamer, receives the RTP stream, demultiplexes the video and bounding box data tracks, and uses the '446 patent's method to render the invisible ID layer and handle user interaction for displaying augmentations. This creates an end-to-end open-source pipeline for creating and consuming interactive, augmented video streams.

3. Combination with FFMPEG and MQTT: The '446 patent's method is used in a remote surveillance or industrial monitoring context. Video is encoded using the open-source FFMPEG library. Simultaneously, a separate process generates bounding box data for moving objects or equipment states. This metadata, containing the object ID and coordinates, is published to a lightweight MQTT (Message Queuing Telemetry Transport) topic. A client application subscribes to this MQTT topic and also streams the video via a standard protocol like HLS. The client synchronizes the low-latency MQTT messages with the video frames and renders the augmented overlay. Tapping an object publishes an MQTT message back to the server, which can trigger an action, such as moving a PTZ camera or logging an event, creating a two-way interactive system based on open standards.