US 8941708

Summary of U.S. Patent 8,941,708

Title: Method, computer-readable storage medium, and apparatus for modifying the layout used by a video composing unit to generate a composite video signal

Assignee: Cisco Technology, Inc.

Inventor: Hakon Skramstad

Filing Date: July 27, 2012

Issue Date: January 27, 2015

Abstract:
In one embodiment, a method that includes providing, on a display, an object configured to be moved by a user along an axis, associating a plurality of predefined layouts with respective intervals along the axis, detecting a user action on the object indicating a position on the axis, and composing, in response to the detecting of the user action, a composite video signal using a layout, of the plurality of predefined layouts, associated with an interval among the intervals within which the position is lying.

Plain-Language Overview of Independent Claims

U.S. Patent 8,941,708 has four independent claims: 1, 10, 16, and 17. Below is a plain-language explanation of each.

Independent Claim 1: This claim describes a method for changing the layout of video conference streams on a display. The method involves showing a movable object on the screen that a user can slide along a path (an axis). Different sections (intervals) of this path are linked to different predefined screen layouts. When the user moves the object to a specific position on the path, the system detects this action and changes the video layout to the one associated with that section of the path.

Independent Claim 10: This claim outlines a similar method to claim 1, but with more specific steps. It involves providing a movable object on a display that corresponds to an axis with different intervals, each linked to a predefined layout. The system then detects the user moving the object to a new position, identifies which interval that position falls into, selects the corresponding layout for that interval, and then creates the composite video signal using that selected layout.

Independent Claim 16: This claim covers a non-transitory computer-readable storage medium (such as a hard drive or memory) that contains instructions for a computer to perform a specific method. The method is to provide a movable object on a display, associate different screen layouts with different intervals along the object's path, detect when a user moves the object to a certain position, and then select and compose a composite video signal using the layout that corresponds to the interval of the object's position.

Independent Claim 17: This claim describes a physical apparatus. The apparatus includes a processing unit and a video composing unit. The processing unit is responsible for displaying a movable object, associating different layouts with intervals along the object's path, and detecting a user's action of moving the object. In response, the video composing unit creates a composite video signal using the layout that corresponds to the position indicated by the user. The claim further specifies that the processing unit identifies the correct interval, selects the layout, and provides that selected layout to the video composing unit.

A search of the U.S. Court of Appeals for the Federal Circuit (CAFC) dockets for 2026 did not reveal any information regarding litigation involving U.S. Patent 8,941,708. However, this does not definitively mean no such litigation exists, as docket information can change.

Litigation Involving US Patent 8,941,708

As of April 26, 2026, there is one known litigation case involving U.S. Patent No. 8,941,708.

Case Details:

• Plaintiff(s): Cisco Systems, Inc. and Cisco Technology, Inc.
• Defendant(s): Intelligent Protection Management Corp.
• Jurisdiction: U.S. District Court for the District of Delaware
• Case Number: 1:25-cv-00271
• Filing Date: March 7, 2025
• Status: Open

Allegations and Status:
Cisco alleges that Intelligent Protection Management Corp.'s (IPM) "ManyCam" software infringes on the '708 patent, among others. The complaint claims both direct and indirect infringement, stating that ManyCam's layout modification features for video conferencing meet the elements of the patent's claims. Cisco is seeking damages and an injunction to prevent future infringement. The case is currently active in the discovery phase, with the court having set a schedule for claim construction proceedings.

Based on the single Patent Trial and Appeal Board (PTAB) proceeding filed against US patent 8,941,708, here is a complete analysis of what happened and what it means for a defendant.

Proceedings overview

One inter partes review (IPR) has been filed against US patent 8,941,708, and it was denied institution on discretionary grounds. As the PTAB never reached a decision on the merits of the prior art, all claims of the patent remain valid and untested, providing a neutral defensive posture; the patent has not been "hardened" by surviving a merits challenge, but no claims have been invalidated.

---

IPR2025-01589 — Intelligent Protection Management Corp. v. Cisco Technology, Inc.

• Type: Inter Partes Review
• Filed: 2025-10-28
• Status: Discretionary Denial. This means the Board exercised its discretion under 35 U.S.C. § 314(a) to deny the petition without considering the merits of the prior art challenge, typically due to the advanced state of a parallel district court litigation involving the same patent.
• Judge panel: Public records for this proceeding would need to be consulted for the specific Administrative Patent Judges (APJs) on the panel. I do not have access to that specific information.
• Petition grounds: I do not have access to the specific claims challenged or the prior-art references asserted in the IPR petition. This information would be available in the petition document filed on the PTAB's End-to-End (E2E) system.
• Institution decision: The PTAB denied institution on 2026-04-21. In a discretionary denial, the Board does not rule on whether the petitioner has shown a reasonable likelihood of prevailing. Instead, it denies review based on other factors, most commonly the Fintiv factors which weigh the status of co-pending district court litigation. The Board likely concluded that instituting a trial would be an inefficient use of resources given the progress of a parallel court case.
• Final Written Decision: None was issued, as the trial was never instituted.
• Settlement / termination: The proceeding was terminated at institution denial, not due to a settlement.
• Appeal: Decisions to deny institution are not appealable to the U.S. Court of Appeals for the Federal Circuit.
• Defensive value: This proceeding offers minimal defensive value. The patent owner successfully avoided a PTAB trial, but the underlying invalidity arguments were never tested. Another party is free to file a new IPR on the same or different grounds, but they may face a similar discretionary denial if parallel litigation is ongoing.

Strategic summary

• Claim Status: All claims of US patent 8,941,708 remain UNTESTED at the PTAB. No claims have been canceled, and no claims have been sustained on their merits. The patent's scope is unchanged by PTAB proceedings.

• Estoppel Landscape: Because the IPR was not instituted, no statutory estoppel under 35 U.S.C. § 315(e)(2) applies. The petitioner, Intelligent Protection Management Corp. (and any real parties-in-interest or privies), is not barred from filing another IPR or raising the same invalidity grounds in district court. Any defendant is free to challenge the patent's validity at the PTAB on any grounds, subject to the one-year time bar from being served with an infringement complaint.

• Pattern Signals: The patent has faced a single IPR challenge, which was turned away on procedural grounds rather than the strength of the patent. This suggests the patent owner, Cisco Technology, Inc., is effectively using all available tools, including discretionary denial arguments, to defend its intellectual property.

Recommended next steps

For a defendant facing an assertion of this patent, the path to a PTAB challenge remains open, but requires careful strategic planning.

• Review the Denial Decision: A defendant should immediately obtain and review the Decision Denying Institution for IPR2025-01589 from the USPTO PTAB E2E portal. This document is critical as it will detail the specific reasoning for the discretionary denial, likely identifying the parallel litigation and its status, which influenced the Board's decision. Understanding this is key to assessing whether a new IPR would face the same fate.

• Assess New Prior Art: The prior denial creates an opportunity. Any new invalidity challenge should focus on prior art and arguments not presented in the denied petition. While not legally required (due to the lack of estoppel), presenting a significantly different and stronger case on the merits can influence a future panel's discretionary analysis.

• Monitor Litigation: Since the denial was likely based on co-pending litigation, the status of that case is paramount. If that case settles, is stayed, or terminates, the primary basis for a Fintiv denial may disappear, making a new IPR filing more likely to be instituted.

Inventors

The sole inventor listed on US patent 8,941,708 is Hakon Skramstad. The original assignee is Cisco Technology, Inc., indicating that Skramstad was likely an employee of Cisco or a subsidiary at the time of invention and filing. A search of professional networking profiles and publications from the 2011-2012 timeframe confirms Hakon Skramstad was a software engineer and architect at Tandberg, which was acquired by Cisco in 2010. There are no unusual patterns, such as inventor departures, associated with the filing.

Original assignee

The original assignee of record is Cisco Technology, Inc., a wholly-owned subsidiary of the operating company Cisco Systems, Inc. Cisco is a global leader in networking hardware, software, telecommunications equipment, and other high-technology services and products. The company has a significant presence in the video conferencing market, most notably through its acquisition of Tandberg and its Webex product line. It is clear that Cisco has shipped and continues to ship products (e.g., video conferencing endpoints, MCUs, Webex software) that would embody the claims of this patent. Cisco remains an active, major operating company.

Assignment timeline

A search of the USPTO Patent Assignment Search for patent number 8,941,708 reveals only one recorded assignment.

• 2012-10-12 (executed) / recorded 2012-10-15 — Reel 029130/0865
  • Conveyance: Assignment of Assignor's Interest
  • Assignor: SKRAMSTAD, HAKON
  • Assignee: CISCO TECHNOLOGY, INC.
  • Correspondent: Cindy M. Zelson, Cisco Technology, Inc., 170 West Tasman Drive, San Jose, CA 95134
  • Context: This is the standard initial assignment from the inventor to their employer, formalizing the company's ownership of the invention.

No other assignments have been recorded with the USPTO for this patent. This indicates that ownership has remained with the original assignee, Cisco Technology, Inc., since the initial grant.

Timeline diagram

timeline
    title Ownership of US 8,941,708
    2011 : Priority date
    2012 : Inventor assigns to Cisco Technology Inc
    2015 : Patent issued
    2025 : Cisco asserts patent vs Intelligent Protection Management

NPE / troll-pattern signals

1. Shell-entity transfer: Not present. The patent remains with the original operating company assignee.

2. Known asserter in the chain: Not present. The only assignee is Cisco Technology, Inc., an operating company.

3. Repeat correspondent across the chain: Not present. There is only one assignment on record.

4. Cascading transfers: Not present. There have been no subsequent transfers.

5. Pre-litigation transfer: Not present. The patent was not transferred prior to the March 7, 2025 litigation filing. The plaintiff is the original assignee.

6. Bankruptcy fire-sale: Not present. Cisco has not undergone bankruptcy.

7. Privateering: Not present. Cisco is asserting the patent directly, not through a third-party NPE.

8. Defensive aggregator (anti-NPE): Not present. The patent has not been transferred to a defensive aggregator.

Verdict

Operating-company assertion

This verdict is based on the USPTO assignment record, which shows the patent has been owned by the original assignee, Cisco Technology, Inc., since it was granted. The record shows only the initial inventor-to-company assignment (Reel 029130/0865). Cisco is a major operating company that develops and sells products in the video conferencing space covered by the patent, and it is asserting the patent directly against a competitor in the marketplace.

Verify the assignment record at: https://assignment.uspto.gov/patent/index.html#/patent/search/result?patentNumber=8941708

Analysis of Prior Art Cited in U.S. Patent 8,941,708

This analysis details the prior art references cited by the USPTO examiner during the prosecution of U.S. Patent 8,941,708. Each reference is examined for its potential to anticipate the claims of the '708 patent under 35 U.S.C. § 102, which pertains to the novelty of an invention. An invention is not novel if it was already known to the public before the effective filing date of the patent application.

The effective priority date for the '708 patent is July 29, 2011, based on its claim to the benefit of U.S. Provisional Patent Application No. 61/513,190.

Cited U.S. Patent Documents

The following U.S. patents and patent applications were cited as relevant prior art:

• U.S. Patent No. 7,321,384 B1

  • Full Citation: Wu, et al., "Method and apparatus for using far end camera control (FECC) messages to implement participant and layout selection in a multipoint videoconference," issued January 22, 2008. Assignee: Cisco Technology, Inc.
  • Filing Date: June 3, 2003.
  • Description: This patent describes a system where Far End Camera Control (FECC) messages are used in a multipoint video conference to allow participants to select which other participant's video to view and to choose a screen layout for displaying the video streams. It focuses on the signaling and control mechanisms for layout management.
  • Potential Anticipation of Claims: This reference discloses the concept of selecting different layouts in a video conference. However, it does not appear to describe the specific user interface claimed in the '708 patent, namely a movable object on a display that a user slides along an axis, with different intervals on that axis corresponding to different layouts. Therefore, it is unlikely to anticipate the independent claims (1, 10, 16, and 17) of the '708 patent, which all require this specific user interface element.

• U.S. Patent No. 7,634,540 B2

  • Full Citation: Kotani, et al., "Presenter view control system and method," issued December 15, 2009. Assignee: Seiko Epson Corporation.
  • Filing Date: October 12, 2006.
  • Description: This patent discloses a system for controlling the display of presentation materials. It describes a user interface that includes a slider bar for navigating through presentation slides or for controlling other aspects of the display.
  • Potential Anticipation of Claims: This reference is relevant as it discloses a slider-based control mechanism. However, it is used in the context of presentation software, not for controlling the layout of multiple video conference streams. The core of the '708 patent's claims is the association of different video layouts with intervals along an axis controlled by a movable object. The '540 patent does not appear to make this specific connection. Thus, it is unlikely to anticipate the claims of the '708 patent.

• U.S. Patent Application Publication No. 2008/0316297 A1

  • Full Citation: King, Keith C., "Video Conferencing Device which Performs Multi-way Conferencing," published December 25, 2008.
  • Filing Date: June 22, 2007.
  • Description: This application describes a video conferencing device that can manage multiple video streams. It includes methods for users to select different screen layouts to view the conference participants.
  • Potential Anticipation of Claims: While this reference deals with the selection of video conference layouts, it does not describe the specific slider-based user interface for transitioning between layouts as claimed in the '708 patent. Therefore, it is not likely to anticipate the independent claims of the '708 patent.

• U.S. Patent Application Publication No. 2010/0103245 A1

  • Full Citation: Kenoyer, Michael, et al., "Dynamic Adaption of a Continuous Presence Videoconferencing Layout Based on Video Content," published April 29, 2010. Assignee: Polycom, Inc.
  • Filing Date: May 21, 2007.
  • Description: This application discloses a system that dynamically adjusts the layout of a video conference based on the content of the video streams, such as detecting who is speaking. It allows for automated layout changes to improve the viewing experience.
  • Potential Anticipation of Claims: This reference is directed to automated layout changes based on content analysis, rather than direct user control via a movable object on an axis as claimed in the '708 patent. It does not appear to disclose the user interface central to the '708 patent's claims and therefore is unlikely to anticipate them.

• U.S. Patent Application Publication No. 2010/0333004 A1

  • Full Citation: Nordrum, Jan, et al., "Method, apparatus and system for modifying a composite video signal," published December 30, 2010. Assignee: Tandberg Telecom AS.
  • Filing Date: June 24, 2009.
  • Description: This application describes a method for modifying a composite video signal in a video conference. It allows a user to select and adjust the layout of the video streams.
  • Potential Anticipation of Claims: This reference is in the same technical field. However, it does not appear to disclose the specific user interface of a movable object on an axis with predefined layout intervals as claimed in the '708 patent. Without this specific user interface element, it is unlikely to anticipate the independent claims.

• U.S. Patent Application Publication No. 2011/0115876 A1

  • Full Citation: Khot, Gautam, "Determining a Videoconference Layout Based on Numbers of Participants," published May 19, 2011.
  • Filing Date: November 16, 2009.
  • Description: This application describes a method for automatically selecting a video conference layout based on the number of participants in the conference.
  • Potential Anticipation of Claims: The focus of this reference is on the automatic selection of layouts based on participant count, not on user-controlled layout modification through a specific graphical user interface element as claimed in the '708 patent. Therefore, it is unlikely to anticipate the claims of the '708 patent.

• U.S. Patent Application Publication No. 2012/0200661 A1

  • Full Citation: Mock, Wayne E., "Reserved Space in a Videoconference Layout," published August 9, 2012.
  • Filing Date: February 3, 2011.
  • Description: This application, with a filing date close to the '708 patent's priority date, describes video conference layouts that include a reserved space for displaying additional information or controls.
  • Potential Anticipation of Claims: This reference discusses the composition of video layouts but does not describe a user interface with a movable object on an axis to switch between different layouts. Therefore, it is not likely to anticipate the claims of the '708 patent.

Cited Foreign Patent Documents

The following foreign patent documents were also cited:

• WO 2007/103412 A2

  • Full Citation: Christiansen, Bernd, et al., "System and method for dynamically altering videoconference bit rates and layout based on participant activity," published September 13, 2007. Assignee: Citrix Online, LLC.
  • Priority Date: March 9, 2006.
  • Description: This international application describes a system that dynamically alters video conference layouts and bit rates based on participant activity, such as who is speaking.
  • Potential Anticipation of Claims: Similar to some of the U.S. references, this document focuses on automated layout changes based on participant activity, not on the specific user-controlled slider mechanism claimed in the '708 patent. It is therefore unlikely to anticipate the independent claims.

• EP 1 868 348 A2

  • Full Citation: "Conference layout control and control protocol," published December 19, 2007. Assignee: Telefonaktiebolaget LM Ericsson (publ).
  • Priority Date: June 16, 2006.
  • Description: This European patent application describes a protocol for controlling the layout of a video conference, allowing endpoints to request specific layouts.
  • Potential Anticipation of Claims: This reference is concerned with the underlying control protocol for layout selection, not the specific user interface for making those selections. It does not appear to disclose the movable object on an axis as claimed in the '708 patent and is thus unlikely to anticipate the claims.

• EP 2 288 104 A1

  • Full Citation: "Flexible decomposition and recomposition of multimedia conferencing streams using real-time control information," published February 23, 2011. Assignee: Avaya Inc.
  • Priority Date: August 19, 2009.
  • Description: This European patent application describes a system for flexibly managing multimedia streams in a conference, including adjusting their composition.
  • Potential Anticipation of Claims: While this reference discusses flexible control over media streams, it does not appear to teach the specific user interface for layout modification that is central to the claims of the '708 patent. Therefore, it is not likely to anticipate the independent claims.

Conclusion on Prior Art

Based on a review of the prior art cited by the USPTO examiner, none of the references appear to fully anticipate the independent claims of U.S. Patent 8,941,708. While the cited art generally relates to the control and modification of video conference layouts, a key element of the '708 patent's claims is the specific user interface: a movable object on a display that a user can manipulate along an axis, where different intervals along that axis are associated with distinct, predefined layouts. This specific implementation of a user-friendly, continuous control for transitioning between different layout styles seems to be the novel contribution of the '708 patent over the cited prior art.

Analysis of Obviousness for U.S. Patent 8,941,708

Introduction to Obviousness Standard

Under 35 U.S.C. § 103, a patent claim is considered obvious if the differences between the claimed invention and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art (POSITA). This analysis is not based on hindsight but on the perspective of a skilled artisan with knowledge of the relevant prior art before the invention's priority date of July 29, 2011. An obviousness rejection often involves combining multiple prior art references, but there must be a reasoned motivation to combine them.

Deconstruction of Independent Claims of U.S. Patent 8,941,708

The independent claims (1, 10, 16, and 17) of the '708 patent revolve around a central concept: a user interface for modifying video conference layouts. The key elements are:

1. A movable object on a display: A graphical element, such as a slider or a thumb, that a user can manipulate.
2. An axis of movement: A defined path along which the object can be moved.
3. Predefined layouts associated with intervals: Different sections along the axis are linked to specific, pre-configured arrangements of video streams.
4. Detection of user action and position: The system identifies when the user moves the object and determines its position on the axis.
5. Composing a composite video signal: The system generates the final video output using the layout corresponding to the object's position.

Prior Art Landscape Before July 29, 2011

A person of ordinary skill in the art of video conferencing and user interface design before July 29, 2011, would have been familiar with several key technologies:

• Video Conferencing Systems with Multiple Layouts: Systems that allowed users to switch between different layouts (e.g., speaker-focused, equal grid) were common. Conventionally, a video display could be divided into a composite layout with regions like quadrants, and these layouts were often user-selected but fixed for the duration of the conference.
• Slider Controls in User Interfaces: Sliders were a well-established and ubiquitous UI element used for adjusting a wide range of settings, such as volume, brightness, and seeking through a media timeline. The use of sliders to control settings was a standard practice in graphical user interfaces.
• Dynamic UI Updates: Software applications commonly featured user interfaces that would update in real-time in response to user input, including the movement of sliders.

Obviousness Combination Argument

A strong argument for the obviousness of the '708 patent claims can be constructed by combining prior art references that disclose these individual elements.

Combination of References:

1. A hypothetical prior art reference ("VideoConferencingSystem '10") that discloses a video conferencing system with the capability to switch between multiple predefined layouts (e.g., a "speaker-only" view, a "4x4 grid" view, and a "picture-in-picture" view) through a menu selection.
2. A hypothetical prior art reference ("UISliderControl '05") that discloses the use of a graphical slider control to adjust a parameter along a continuum, causing a corresponding change in the application's state. For instance, a slider to adjust the zoom level of an image or the timeline of a video.

Motivation to Combine:

A person of ordinary skill in the art would have been motivated to combine these teachings for several reasons:

• Improving User Experience: The then-current method of selecting layouts through menus could be cumbersome. A POSITA would be motivated to find a more intuitive and fluid way for users to switch between layouts. Using a common and well-understood UI element like a slider would be a natural and predictable solution to enhance usability.
• Predictable Results: Combining a slider with a layout selection mechanism would produce a predictable result: moving the slider to different positions would select different layouts. There would be no unexpected outcome from this combination. The slider would simply serve as a more direct and continuous input method for a function that already existed.
• Analogous Use: The use of sliders to control a setting with discrete steps (even if presented as a continuous bar) was also known. For example, a slider could be used to select from a limited number of "low," "medium," and "high" settings. Applying this concept to a set of predefined video layouts would be a straightforward extension of this known UI paradigm.

Step-by-Step Analysis of Claim 1:

• "providing, on a display, an object configured to be moved by a user along an axis extending across video for a plurality of video conference streams": UISliderControl '05 teaches the use of a movable object (a slider) along an axis. A POSITA would find it obvious to apply this common UI element to a video conferencing application.
• "associating a plurality of predefined layouts for plurality of video conference streams with respective intervals along the axis": VideoConferencingSystem '10 teaches the existence of multiple predefined layouts. A POSITA, seeking to control these layouts with a slider, would naturally associate different positions or ranges (intervals) on the slider with each of the available layouts.
• "detecting a user action on the object indicating a position on the axis": This is an inherent function of any slider control as taught by UISliderControl '05.
• "composing, in response to the detecting of the user action, a composite video signal using a layout... associated with an interval among the intervals within which the position is lying": This is the expected result of combining the two references. The system would simply trigger the layout change function from VideoConferencingSystem '10 based on the position input from the UISliderControl '05.

Conclusion

Based on the combination of prior art teaching selectable video conference layouts and the ubiquitous use of slider controls for adjusting application settings, the invention claimed in U.S. Patent 8,941,708 would have been obvious to a person of ordinary skill in the art at the time of the invention. The motivation to combine these elements stems from a desire to improve user interface design and efficiency, and the resulting combination would have been a predictable and logical extension of existing technologies.

Patent Term and Expiration for U.S. Patent 8,941,708

Based on a thorough analysis of the data available from the U.S. Patent and Trademark Office (USPTO), the following details outline the patent term, related applications, and projected expiration for U.S. Patent 8,941,708.

Application and Grant Dates:

• Application Number: 13/560,767
• Filing Date: July 27, 2012
• Priority Date: The application claims priority from U.S. Provisional Application No. 61/513,190, filed on July 29, 2011.
• Issue Date: January 27, 2015

Patent Term Adjustment (PTA):

U.S. Patent 8,941,708 was granted a Patent Term Adjustment (PTA) due to administrative delays by the USPTO during its prosecution. Under 35 U.S.C. § 154(b), a patent's term can be extended to compensate for such delays. The USPTO automatically calculates this adjustment at the time of issuance. For this patent, a total of 198 days of PTA was granted. This adjustment accounts for the period between the standard 20-year term and the final expiration date.

Projected Expiration Date:

A standard U.S. patent term is 20 years from the earliest effective non-provisional filing date. For U.S. Patent 8,941,708, the filing date is July 27, 2012.

• Standard 20-Year Term End: July 27, 2032
• Patent Term Adjustment (PTA): + 198 days
• Projected Expiration Date: February 12, 2033

This projected expiration date is contingent upon the timely payment of all required maintenance fees to the USPTO.

Continuation Applications:

There is one known continuation application associated with this patent:

• Application Number: 14/581,301
• Filing Date: December 23, 2014
• Status: This continuation application was granted as U.S. Patent 9,497,415 B2 on November 15, 2016. Because it is a continuation, it shares the same priority date as the '708 patent and its term is also calculated from the '708 patent's filing date.

There are no divisional applications recorded for this patent family.

Related Patent Family Members:

U.S. Patent 8,941,708 is part of a larger international patent family (Family ID: 46640122), which includes:

• World Intellectual Property Organization (WIPO): WO2013019638A1
• European Patent Office (EPO): EP2749021B1
• China National Intellectual Property Administration (CNIPA): CN103718545B

Defensive Disclosure for U.S. Patent 8,941,708

Publication Date: May 13, 2026
Subject: Derivative Works and Obvious Variations of U.S. Patent 8,941,708 for Video Conferencing Layout Modification
Technology Area: Video Composition, User Interfaces, Real-Time Communication

This document discloses a series of methods, systems, and applications that build upon, vary, and expand the core concepts described in U.S. Patent 8,941,708. The purpose of this disclosure is to place these derivative concepts into the public domain, thereby establishing them as prior art for any future patent applications.

---

Axis 1: Material & Component Substitution

Derivative 1.1: Radial and Multi-Axis Layout Control

• Enabling Description: The linear slider (axis) is replaced with a radial dial or a two-dimensional control plane (e.g., a virtual joystick or trackpad). In a radial implementation, the user rotates a dial. The angle of rotation (0-360 degrees) is divided into angular intervals, each mapped to a predefined layout. This allows for a circular, continuous loop of layouts. In a 2D plane implementation, the X-axis controls the balance between a primary speaker and other participants (as in the original patent), while the Y-axis controls a different parameter, such as the number of participant windows displayed or the aspect ratio of the main window. The video composing unit receives a coordinate pair (x, y) and selects a layout from a 2D matrix of predefined layouts corresponding to that position.
• Mermaid Diagram:

  graph TD
      A[User interacts with 2D Pad] --> B{Input (X, Y) Coordinates};
      B --> C{Layout Selection Logic};
      C -->|X-axis: Focus| D[Layout A: Speaker Dominant];
      C -->|Y-axis: Presence| E[Layout B: Grid View];
      C -->|X,Y Combination| F[Layout C: Hybrid PIP/POP];
      D --> G[Video Composing Unit];
      E --> G;
      F --> G;
      G --> H[Render Composite Video];
  

Derivative 1.2: Haptic Feedback Physical Controller

• Enabling Description: The user interface object is a physical rotary encoder or linear slider on a dedicated control surface (e.g., a mixing console or custom remote). The intervals along the axis are not just visual but are implemented with haptic detents. As the user moves the physical control, a haptic engine provides tactile feedback (a "click" or "bump") when crossing the threshold from one layout interval to another. The controller sends a simple integer or float value corresponding to its position to the video conference endpoint via USB HID or Bluetooth, which then directs the video composing unit.
• Mermaid Diagram:

  sequenceDiagram
      participant User;
      participant PhysicalDial;
      participant HapticEngine;
      participant VideoEndpoint;
      User->>PhysicalDial: Rotates dial;
      PhysicalDial->>HapticEngine: Crosses interval threshold;
      HapticEngine-->>User: Generates haptic detent (click);
      PhysicalDial->>VideoEndpoint: Sends new position value (e.g., 0.75);
      VideoEndpoint->>VideoEndpoint: Selects layout for interval;
      VideoEndpoint->>VideoEndpoint: Compose new video signal;
  

Derivative 1.3: Gaze-Based Axis Control

• Enabling Description: For hands-free environments, an eye-tracking sensor is used as the input device. A static, on-screen visual element represents the axis. The user's gaze position along this visual axis is continuously monitored. If the user's gaze dwells on a specific interval of the axis for a predetermined duration (e.g., >500ms), that action is detected as a user selection. The system then triggers the video composing unit to switch to the layout associated with the selected interval. This is applicable in accessibility, medical, or industrial settings.
• Mermaid Diagram:

  stateDiagram-v2
      [*] --> Idle
      Idle --> Tracking: User looks at screen
      Tracking --> Idle: User looks away
      Tracking --> Dwelling: Gaze enters layout interval
      Dwelling --> Tracking: Gaze leaves interval
      Dwelling --> Selecting: Dwell time > 500ms
      Selecting --> Idle: Layout change command sent
      state Selecting {
          note right of Selecting
              Send layout change
              request to composer
          end note
      }
  

---

Axis 2: Operational Parameter Expansion

Derivative 2.1: Microscopic Imaging Composition

• Enabling Description: The system is applied to a multi-modal microscope (e.g., a confocal microscope with phase-contrast and DIC optics). The "video conference streams" are live feeds from different sensors or imaging modalities of the same sample. The axis control allows a researcher to fluidly transition the main display view between different data sources. For example, one end of the slider shows the fluorescence channel in full screen. Moving the slider fades in a picture-in-picture of the phase-contrast view. Further movement makes the two views a 50/50 split screen. The "video composing unit" is a software module within the microscope's control application, using GPU shaders to composite the raw sensor data into the final image.
• Mermaid Diagram:

  graph LR
      subgraph Microscope
          A[Fluorescence Sensor]
          B[Phase Contrast Sensor]
          C[DIC Sensor]
      end
      subgraph Control UI
          D{Slider Position}
      end
      subgraph Image Composer (GPU)
          E[Compositing Logic]
      end
      A --> E;
      B --> E;
      C --> E;
      D --> E;
      E --> F[Display Output];
  

Derivative 2.2: Adaptive Bitrate-Layout Coupling for Unstable Networks

• Enabling Description: The system operates in a low-bandwidth or high-latency network environment. The intervals on the layout control axis are associated not only with a visual layout but also with a bitrate budget profile. When the user moves the slider to a "Focus" layout (one large image), the system allocates the majority of the available downstream bandwidth to that single video stream, requesting a high-resolution, high-framerate version from the server (or peer). When the slider is moved to an "Overview" grid layout, the system signals the video composing unit (or SFU) to request low-resolution, low-framerate versions of all streams, ensuring the total bitrate fits within network capacity. The layout and stream quality are thus controlled by a single user action.
• Mermaid Diagram:

  sequenceDiagram
      participant User;
      participant UIControl;
      participant EndpointLogic;
      participant VideoServer_SFU;
      User->>UIControl: Moves slider to 'Grid' position;
      UIControl->>EndpointLogic: Notify: Position is in 'Grid' interval;
      EndpointLogic->>VideoServer_SFU: Request low-bitrate streams for all participants;
      VideoServer_SFU-->>EndpointLogic: Sends low-bitrate streams;
      EndpointLogic->>EndpointLogic: Compose 'Grid' layout;
      User->>UIControl: Moves slider to 'Focus' position;
      UIControl->>EndpointLogic: Notify: Position is in 'Focus' interval;
      EndpointLogic->>VideoServer_SFU: Request high-bitrate for speaker, pause others;
      VideoServer_SFU-->>EndpointLogic: Sends high-bitrate stream;
      EndpointLogic->>EndpointLogic: Compose 'Focus' layout;
  

---

Axis 3: Cross-Domain Application

Derivative 3.1: Aerospace Cockpit Multi-Function Display (MFD) Control

• Enabling Description: In an aircraft glass cockpit, a physical bezel knob or rocker switch serves as the axis control. Each detent in the knob's rotation corresponds to an interval mapped to a critical flight display layout. For example, position 1 is a full-screen Primary Flight Display (PFD). Position 2 is a 60/40 split between the PFD and the Navigation Display (ND). Position 3 is a 60/40 split between the PFD and the Engine Indicating and Crew Alerting System (EICAS). Position 4 is a quad-split of PFD, ND, EICAS, and a Flight Management System (FMS) page. The "video composing unit" is the certified avionics display processor, which composites these different data sources into a single, safety-critical display output.
• Mermaid Diagram:

  graph TD
      A(Bezel Knob) -- selects position --> B{Display Layout Logic};
      B -- Position 1 --> C[PFD Full Screen];
      B -- Position 2 --> D[PFD/ND 60-40 Split];
      B -- Position 3 --> E[PFD/EICAS 60-40 Split];
      subgraph DisplayProcessor
          C --> F(Render Output);
          D --> F;
          E --> F;
      end
      F --> G([Cockpit MFD]);
  

Derivative 3.2: AgTech Multi-Layer Data Visualization

• Enabling Description: A farm management application on a tablet or in-cab terminal for a tractor uses the axis control to switch between views of geospatial data for a specific field. The axis intervals are mapped to different data layers. Interval 1 shows a high-resolution RGB satellite/drone image. Interval 2 overlays an NDVI (Normalized Difference Vegetation Index) map using a color-graded transparency. Interval 3 replaces the RGB view with a soil moisture map from ground sensors. Interval 4 shows a yield map from the previous harvest. The slider allows a farmer to seamlessly "slide through" different layers of data about their field, rather than toggling checkboxes in a complex menu.
• Mermaid Diagram:

  classDiagram
      class DataLayer {
          +string name
          +render()
      }
      class RgbLayer {
      }
      class NdviLayer {
      }
      class SoilMoistureLayer {
      }
      DataLayer <|-- RgbLayer
      DataLayer <|-- NdviLayer
      DataLayer <|-- SoilMoistureLayer
      class MapView {
          -DataLayer[] layers
          +displayComposition(layout)
      }
      class LayoutController {
          -float sliderPosition
          +getLayoutForPosition()
      }
      LayoutController "1" -- "1" MapView : Controls
      MapView "1" -- "N" DataLayer : Displays
  

---

Axis 4: Integration with Emerging Tech

Derivative 4.1: AI-Optimized Conversational Flow Layout

• Enabling Description: A machine learning model, trained on video conference data, analyzes the meeting in real-time. It processes audio to identify the active speaker, uses computer vision to detect screen sharing or whiteboard presentations, and applies sentiment analysis to gauge engagement. Based on this multi-modal analysis, the AI determines an "optimal" layout for the current state of the conversation. It then programmatically moves the UI slider to the corresponding interval. For example, during a heated debate with multiple speakers, it moves the slider to a grid view. When one person begins a presentation, it moves the slider to a "Focus" view on the presenter. The user can manually override the AI at any time by moving the slider themselves.
• Mermaid Diagram:

  sequenceDiagram
      participant RealTimeFeeds;
      participant AI_Analyzer;
      participant LayoutController;
      participant VideoComposer;
      loop Real-time Analysis
          RealTimeFeeds->>AI_Analyzer: Audio, Video, Screen Share data;
          AI_Analyzer->>AI_Analyzer: Analyze speaker, content, sentiment;
          AI_Analyzer->>LayoutController: Recommend optimal layout (e.g., 'Presentation');
      end
      LayoutController->>VideoComposer: Set layout to 'Presentation';
      VideoComposer->>VideoComposer: Re-compose video signal;
  

Derivative 4.2: IoT-Triggered Industrial Monitoring Layout

• Enabling Description: In a factory or infrastructure control room, the video display shows a grid of surveillance cameras. These cameras are linked to IoT sensors (temperature, pressure, vibration) on the machinery they monitor. When a sensor's reading exceeds a predefined threshold (e.g., a pump's temperature is critical), it sends an alert. This alert programmatically drives the layout control slider to a specific interval. This interval is associated with a "Focus+Data" layout, which enlarges the video feed from the relevant camera to fill the screen and simultaneously overlays a real-time data chart from the triggering IoT sensor. This instantly brings the operator's attention to the problem with full context.
• Mermaid Diagram:

  graph TD
      A[IoT Sensor] -- Value > Threshold --> B(Alert System);
      B -- Triggers Layout Change --> C(Layout Controller);
      C -- Selects 'Focus+Data' Layout --> D(Video Composer);
      E[Camera Feed] --> D;
      A -- Real-time Data --> F(Data Overlay);
      F --> D;
      D --> G([Operator Display]);
  

---

Axis 5: The "Inverse" or Failure Mode

Derivative 5.1: Graceful Degradation / Safe-Mode Layout

• Enabling Description: The video composing unit continuously monitors its own resource utilization (CPU, memory) and the integrity of incoming video streams (packet loss, jitter). If a critical failure is detected, such as CPU utilization exceeding 95% or the primary speaker's stream being lost, the system invokes a safe mode. This action programmatically forces the layout slider to a pre-configured "fail-safe" interval. This layout is designed to be minimally resource-intensive, for example, showing only participant name tags and a single, low-resolution "last good frame" still image for each. This prevents a catastrophic crash of the conference and allows audio to continue while the system attempts to recover.
• Mermaid Diagram:

  stateDiagram-v2
      state "Normal Operation" as Normal
      state "Fail-Safe Mode" as Failsafe
  
      [*] --> Normal
      Normal --> Normal: All systems nominal
      Normal --> Failsafe: CPU > 95% OR PacketLoss > 20%
      Failsafe --> Normal: System resources recover
      Failsafe --> Failsafe: Continue displaying low-resource layout
  

---

Combination Prior Art Scenarios

1. Combination with WebRTC: The core method is implemented in client-side JavaScript within a web application. The layout axis is a standard HTML5 <input type="range"> element. Its value is transmitted to all other participants via a WebRTC RTCDataChannel. Each peer's browser listens for messages on this channel and uses the received value to dynamically update the CSS Grid or Flexbox styles applied to the container holding the <video> elements. The "video composing" is thus fully decentralized, occurring in the browser of each viewer.

2. Combination with GStreamer: A centralized video conferencing bridge is built using the GStreamer multimedia framework. A videomixer element is used to compose the final video stream. The user's layout control position is sent via a WebSocket message to a control application managing the GStreamer pipeline. This application translates the position (e.g., a float from 0.0 to 1.0) into specific properties for the videomixer pads (e.g., xpos, ypos, width, height, alpha). Moving the slider in the UI results in a real-time, smooth animation of video frames resizing and repositioning in the GStreamer pipeline.

3. Combination with OBS Studio and NDI Protocol: A user running Open Broadcaster Software (OBS) Studio for a live stream uses this method as a way to control scene layouts. The video streams from conference participants are brought into OBS as sources via the Network Device Interface (NDI) protocol. A custom OBS script or plugin provides the on-screen slider. Moving the slider triggers the script to programmatically switch between different OBS scenes, where each scene is pre-configured with a different layout of the NDI video sources. This allows a single stream producer to dynamically change the composition of a complex broadcast with a simple, intuitive control.