Derivative works — US Patent 9036010

As a Senior Patent Strategist and Research Engineer specializing in Defensive Publishing, the goal is to expand the prior art landscape around US Patent 9036010, "Transport of stereoscopic image data over a display interface." The following defensive disclosure details derivative variations based on the patent's core inventive concepts, aiming to render future incremental improvements obvious or non-novel.

Defensive Disclosure for US Patent 9036010

Derivative 1: Optical-Fiber-Based DDI with FPGA/ASIC Multiplexing

Derivation Axis: Material & Component Substitution

Enabling Description:
This derivative implements the core mechanism of transporting multiplexed stereoscopic image data and signaling over a digital display interface by substituting the electrical High Definition Multimedia Interface (HDMI) with a high-bandwidth optical fiber interface, such as a Fibre Channel or a custom optical transport layer utilizing Vertical-Cavity Surface-Emitting Lasers (VCSELs) for short-reach and Distributed Feedback (DFB) lasers for long-reach transmission. The formatter and processor functionalities, as described in claims 1 and 12, are implemented in custom Application-Specific Integrated Circuits (ASICs) or reconfigurable Field-Programmable Gate Arrays (FPGAs) rather than general-purpose processors.

On the source side, the ASIC/FPGA-based formatter receives uncompressed stereoscopic image components (e.g., left/right eye data or 2D+depth) and multiplexes them directly into high-speed optical data streams. This multiplexing occurs at the physical layer, potentially encoding the stereoscopic information within specific wavelength channels (Dense Wavelength Division Multiplexing - DWDM) or time-division multiplexed sub-frames of the optical signal. Signaling information (identifying the stereoscopic mode, encoding scheme, and depth data location) is embedded as dedicated control packets within the optical data stream or on a separate low-speed optical control channel, ensuring resilience against electrical noise and extending transmission distances.

On the sink side, a corresponding ASIC/FPGA-based processor demultiplexes the incoming optical stream, extracting the stereoscopic image data components and interpreting the embedded signaling information. The optical-to-electrical conversion is handled by photodiodes and transimpedance amplifiers, feeding directly into the FPGA/ASIC for real-time demultiplexing and reconstruction of the 3D image for display. This architecture supports extremely high data rates beyond current HDMI capabilities while maintaining uncompressed pixel integrity.

flowchart TD
    SUBGRAPH Source Device
        A[Stereoscopic Image Source] --> B(ASIC/FPGA Formatter)
        B -- Multiplexes Left/Right or 2D+Depth --> C{Optical Transmitter (VCSEL/DFB)}
    END
    C -- Optical Fiber Link --> D{Optical Receiver (Photodiode)}
    SUBGRAPH Sink Device
        D --> E(ASIC/FPGA Processor)
        E -- Demultiplexes & Decodes --> F[3D Display Renderer]
    END
    B -- Signaling Packets --> C
    D -- Signaling Packets --> E

Derivative 2: Terahertz (THz) Wireless DDI for High-Throughput Stereoscopic Data

Derivation Axis: Operational Parameter Expansion

Enabling Description:
This derivative extends the digital display interface to operate at Terahertz (THz) frequencies for wireless transmission of uncompressed stereoscopic image data, enabling significantly higher bandwidth and lower latency than conventional wireless display interfaces. The formatter (source) and processor (sink) from claims 1 and 12 are adapted for THz transceivers.

On the source side, the formatter multiplexes stereoscopic image components into a THz-optimized data format, potentially employing advanced modulation schemes like Quadrature Amplitude Modulation (QAM) with high constellation orders (e.g., 256-QAM or 1024-QAM) to maximize spectral efficiency. This multiplexed data stream is then fed to a THz transmitter, which converts the electrical signals into directed THz beams using array antennas for beamforming and spatial multiplexing. Signaling information, indicating the stereoscopic format and multiplexing characteristics, is embedded in THz control frames or transmitted using a robust, lower-rate THz sub-carrier.

The THz display interface operates at peak data rates of several hundreds of gigabits per second, far exceeding the "known data carrying capacity" of current wired interfaces, but maintaining the principle of utilizing existing (or newly available in THz spectrum) capacity for multiplexed stereoscopic content. This allows for multi-view autostereoscopic displays or ultra-high-resolution stereoscopic content (e.g., 8K per eye). The sink-side THz receiver and processor perform the inverse operations, demultiplexing the THz signal, extracting stereoscopic data, and decoding the signaling information for rendering. The operational parameters include carrier frequencies from 100 GHz to 10 THz, with power levels calibrated for short-range indoor transmission (e.g., 1-10 meters line-of-sight) to mitigate atmospheric absorption.

sequenceDiagram
    participant S as Source Device (THz Formatter)
    participant T as THz Transmitter (Antenna Array)
    participant R as THz Receiver (Antenna Array)
    participant D as Sink Device (THz Processor)

    S->>T: Multiplexed Stereo Data + Signaling (Electrical)
    T->>R: THz Beam (Ultra-high Bandwidth, Uncompressed)
    R->>D: Demultiplexed Stereo Data + Signaling (Electrical)
    D->>D: Demultiplexing, Decoding, 3D Rendering

Derivative 3: Stereoscopic Data Transport in High-Definition Medical Endoscopy Systems

Derivation Axis: Cross-Domain Application (Medical Imaging)

Enabling Description:
This derivative applies the patent's principles to high-definition medical endoscopy, specifically for real-time stereoscopic visualization during minimally invasive surgery. The "first audio-visual device" is an endoscopic camera control unit, and the "second audio-visual device" is a medical-grade 3D display or heads-up display worn by the surgeon. The "digital display interface" is a specialized, shielded medical data bus (e.g., a custom PCIe-over-fiber link or a specialized HDMI-compliant interface designed for medical environments).

The endoscopic camera generates two distinct video streams (left and right eye views, or a 2D view with depth map generated via structured light or time-of-flight sensors). The formatter in the camera control unit multiplexes these stereoscopic components into the data stream using existing high-resolution 2D image formats (e.g., 4K UHD 2D streams), leveraging their deep color modes to carry the additional stereoscopic information (e.g., a 10-bit color channel might carry 8-bit image data plus 2-bit depth data, or 5-bit left/5-bit right intensity data). Signaling information, critical for precise synchronization and surgical guidance, is embedded in blanking intervals (similar to HDMI Data Island Packets) to inform the medical display about the stereoscopic format, depth scale, and any critical diagnostic overlays. This allows surgeons to perceive depth accurately within the surgical field, improving precision and reducing procedural risks.

flowchart TD
    A[Stereoscopic Endoscope Camera] --> B(Camera Control Unit - Formatter)
    B -- Multiplexed Stereo Data (e.g., 2D+Depth in Deep Color) --> C{Medical Display Interface (Fiber/Shielded Cable)}
    C --> D(3D Medical Display - Processor)
    D -- Synchronized Output --> E[Surgeon's 3D View]
    B -- Signaling (Depth Scale, Format) --> C
    D -- Decodes Signaling --> D

Derivative 4: Autonomously Guided Vehicle (AGV) Robotic Vision

Derivation Axis: Cross-Domain Application (Industrial Automation/Robotics)

Enabling Description:
This derivative implements the stereoscopic data transport for real-time 3D perception in industrial Autonomous Guided Vehicles (AGVs) or collaborative robots. The "first audio-visual device" is a robot's perception module (e.g., containing stereo cameras or LiDAR-fusion) and the "second audio-visual device" is the robot's onboard processing unit or a remote control station. The "digital display interface" is an industrial Ethernet link (e.g., PROFINET, EtherCAT) or a high-speed wireless industrial communication standard (e.g., 5G NR-U, Wi-Fi 6E).

The perception module's formatter multiplexes stereoscopic data (e.g., a 2D visual stream + real-time point cloud depth data, or semantic segmentation maps) into the industrial network packets. Instead of traditional pixel data, the "data elements" here represent slices of compressed depth maps, feature vectors, or grid occupancy data, multiplexed within standard high-bandwidth Ethernet frames. Signaling information, conveyed via custom EtherType packets or specific IP/UDP headers, specifies the 3D data format, coordinate system (e.g., robot base frame, world frame), sensor fusion parameters, and dynamic region-of-interest indicators. This enables the AGV's navigation system or remote operator to accurately perceive the robot's environment in 3D, crucial for obstacle avoidance, object manipulation, and path planning in dynamic industrial settings.

flowchart LR
    A[Stereo Vision/LiDAR Sensors] --> B(Robot Perception Module - Formatter)
    B -- Multiplexed 3D Data (e.g., 2D+Depth as Industrial Ethernet Packets) --> C{Industrial Ethernet/Wireless Link}
    C --> D(Robot Control Unit / Remote Station - Processor)
    D -- 3D Environmental Model --> E[AGV Navigation / Operator Interface]
    B -- Signaling (Coordinate System, ROI) --> C
    D -- Decodes Signaling --> D

Derivative 5: AI-Driven Dynamic Bandwidth Allocation for Stereoscopic Streaming

Derivation Axis: Integration with Emerging Tech (AI-driven optimization)

Enabling Description:
This derivative enhances the patent's core concept by integrating AI-driven optimization to dynamically allocate bandwidth for stereoscopic image components based on content complexity, network conditions, and user viewing preferences. The formatter (source) and processor (sink) from claims 1 and 12 incorporate embedded AI inference engines.

On the source side, an AI module within the formatter continuously analyzes the incoming stereoscopic content (e.g., scene depth complexity, motion vectors, regions of interest) and monitors interface telemetry (e.g., available bandwidth, error rates). Based on this analysis, the AI dynamically determines the optimal multiplexing strategy: for low-complexity scenes, it might allocate more bits to the 2D image and fewer to depth; for fast-moving, high-depth scenes, it might prioritize depth data or switch to a full left/right stream if capacity allows. The AI also generates adaptive signaling information that informs the sink device not only about the current mode and characteristics but also the rate and quality of each stereoscopic component, which can change frame-by-frame. This signaling is carried in auxiliary data elements (e.g., HDMI Data Island Packets or custom transport stream metadata).

On the sink side, the processor's AI module interprets this dynamic signaling and adjusts its demultiplexing and rendering pipeline in real-time. For instance, if the AI determines that network congestion requires reduced depth data, the sink's AI might use a local depth estimation algorithm to interpolate missing depth information, ensuring a continuous, albeit adaptively scaled, 3D experience. This allows the system to operate efficiently across varying conditions without requiring a fixed stereoscopic format, maximizing visual quality while adhering to the "known data carrying capacity" by intelligently distributing the load.

sequenceDiagram
    participant SC as Stereo Content Source
    participant SF as Source Formatter (with AI)
    participant DDI as Digital Display Interface
    participant SP as Sink Processor (with AI)
    participant D as 3D Display

    SC->>SF: Raw Stereo Components
    SF->>SF: AI analyzes content & telemetry
    SF->>SF: AI determines dynamic multiplexing strategy
    SF->>DDI: Dynamically Multiplexed Stereo Data
    SF->>DDI: Dynamic Signaling (Mode, Characteristics, Rates)
    DDI->>SP: Dynamically Multiplexed Stereo Data & Signaling
    SP->>SP: AI interprets dynamic signaling & adjusts pipeline
    SP->>SP: AI interpolates/enhances if needed
    SP->>D: Adaptive 3D Rendered Output

Derivative 6: IoT-Enabled Context-Aware 3D Display Personalization

Derivation Axis: Integration with Emerging Tech (IoT sensors for real-time monitoring)

Enabling Description:
This derivative extends the patent's concept by integrating IoT sensors to gather real-time context about the viewing environment and user, dynamically adjusting the stereoscopic display parameters. The display interface part on both the source and sink (claims 1 and 12) communicates with a local IoT network.

On the source side, the formatter receives user-specific 3D preferences (e.g., preferred depth level, convergence point) and environmental data (e.g., ambient light, viewing distance, number of viewers) from IoT sensors (e.g., smart cameras, ambient light sensors, proximity sensors). An IoT gateway connected to the source device aggregates this data. The formatter then uses this real-time contextual information to adapt the multiplexing of stereoscopic components. For example, if a single viewer is close to the screen, it might enable a higher depth fidelity profile, or if multiple viewers are present, it might adjust for autostereoscopic comfort zones. This adaptive strategy results in modified stereoscopic image data streams. The signaling information sent via auxiliary data elements (e.g., HDMI Data Island Packets) includes not only the stereoscopic format but also dynamic metadata related to the context-aware adjustments, such as current depth budget, parallax adjustments, or individualized rendering instructions.

On the sink side, the processor receives this contextual metadata via the interface and further refines the 3D rendering based on its own local IoT sensor data (e.g., eye-tracking for precise gaze-contingent rendering, biometric feedback for viewer comfort). The result is a personalized, contextually optimized 3D viewing experience that dynamically adapts to the user and environment, all while leveraging the patent's efficient data transport methods.

graph TD
    subgraph Source Device (AV1)
        AVS[Stereo Content Source] --> F(Formatter)
        F -- Multiplexed Stereo Data --> DDI
    end

    subgraph IoT Network
        IS1[Ambient Light Sensor] --> IOTG(IoT Gateway)
        IS2[Proximity Sensor] --> IOTG
        IS3[User Preferences DB] --> IOTG
    end

    IOTG -- Contextual Data --> F
    F -- Context-aware Signaling --> DDI[Digital Display Interface]

    subgraph Sink Device (AV2)
        DDI --> P(Processor)
        P -- Personalized 3D Output --> Display[3D Display]
        ES[Eye-Tracking Sensor] --> P
    end

Derivative 7: Fail-Safe Stereoscopic Degradation (2D Fallback)

Derivation Axis: The "Inverse" or Failure Mode

Enabling Description:
This derivative focuses on the "inverse" or fail-safe operation mode of the stereoscopic display interface. The system is designed to gracefully degrade stereoscopic content to 2D in the event of interface errors, bandwidth limitations, or sink device capability failures, ensuring continuous, albeit reduced, functionality. The formatter and processor of claims 1 and 12 are augmented with error detection, health monitoring, and fallback logic.

On the source side, the formatter continuously monitors the display interface (e.g., CRC errors, dropped packets, DDC/EDID capability negotiation failures). If critical errors are detected in the "second portion" of the interface carrying stereoscopic data (e.g., depth information loss), or if the sink device signals a degraded state, the formatter initiates a fail-safe sequence. It gracefully transitions from the "second mode" (multiplexed stereoscopic) to the "first mode" (2D image generation) as described in claim 1. This is achieved by ceasing to multiplex the stereoscopic components and instead sending only the 2D base image data within the standard data elements. Signaling information, explicitly indicating the transition to "2D Fallback Mode" and the reason for the fallback, is sent via robust auxiliary data elements (e.g., dedicated HDMI Data Island Packets marked with a high priority).

On the sink side, the processor continuously monitors incoming signaling and data integrity. Upon receiving the "2D Fallback Mode" signal or detecting a severe degradation in the stereoscopic data stream (e.g., corrupted depth maps), it automatically switches its operation from demultiplexing stereoscopic components to simply extracting and displaying the 2D image data. This ensures that the user always receives a viewable image, preventing a black screen or distorted 3D experience in failure scenarios. The system can be configured to attempt re-negotiation for 3D transmission once conditions improve.

stateDiagram
    [*] --> Initializing
    Initializing --> NegotiatingCapabilities: System Boot/Reconnect
    NegotiatingCapabilities --> Full3DMode: Capabilities OK, Bandwidth OK
    NegotiatingCapabilities --> Legacy2DMode: Sink No 3D, or Insufficient Bandwidth
    Full3DMode --> DegradationDetected: Interface Errors, BW Loss, Sink Issue
    DegradationDetected --> Graceful2DFallback: Source Initiates Fallback
    Graceful2DFallback --> Full3DMode: Conditions Recovered, Renegotiate
    Graceful2DFallback --> Legacy2DMode: Persistent Failure or Manual Override
    Legacy2DMode --> Full3DMode: Manual 3D Enable, Capabilities Re-established

Combination Prior Art Scenarios

Here are three scenarios combining US 9036010's principles with existing open-source standards to demonstrate broader applicability and potential obviousness of future improvements.

1. HDMI 2.1 with VESA Display Stream Compression (DSC) for Depth Data

Scenario Description:
US Patent 9036010 describes leveraging existing interface capacity, like HDMI deep color modes, to multiplex stereoscopic data and using Data Island Packets for signaling. This scenario combines the patent's core idea with the HDMI 2.1 standard and the VESA Display Stream Compression (DSC) standard. HDMI 2.1 significantly increases bandwidth (up to 48 Gbps) and introduces features like Fixed Rate Link (FRL) and Enhanced Audio Return Channel (eARC). VESA DSC is an open-source, visually lossless compression standard.

Combination: An implementation sends 2D image data over the primary TMDS/FRL channels of an HDMI 2.1 interface. The associated depth information, or the second eye's image data (for L/R formats), is compressed using VESA DSC before being multiplexed into either:
a) The higher-capacity "deep color" pixel packing modes of HDMI 2.1, or
b) Dedicated auxiliary data channels that HDMI 2.1's FRL can carry, beyond what was explicitly defined as Data Island Packets in earlier HDMI versions.
The signaling information, as taught by US 9036010, is embedded within HDMI 2.1 InfoFrames (an evolution of Data Island Packets) to specify the DSC profile used for the stereoscopic components (e.g., DSC bitrate, specific slicing, predictor mode), the 3D format (2D+Depth, L/R), and the decoding parameters. This allows for higher effective stereoscopic resolution or frame rates over the same HDMI 2.1 link by applying a known compression standard to the less critical or redundant 3D components.

2. DisplayPort Alt Mode over USB-C with Side-by-Side Frame Packing and Metadata Packets

Scenario Description:
US Patent 9036010 outlines multiplexing stereoscopic components and using auxiliary data for signaling. DisplayPort (DP) is another digital display interface, and USB-C with DisplayPort Alt Mode allows DP signals to be carried over a USB-C cable. The patent mentions "schemes which send left eye image data and right eye image data" and "signaling information...carried in a horizontal or vertical blanking period and for a High Definition Multimedia Interface (HDMI) the signaling information can be sent in a Data Island Packet."

Combination: A system uses DisplayPort Alt Mode over USB-C to transmit stereoscopic video. Instead of solely relying on deep color modes, the system employs a "side-by-side" or "top-and-bottom" frame packing method, a common open-source approach for 3D content, where the left and right eye images are scaled and placed adjacent within a single standard 2D video frame. The key innovation from US 9036010 is applied to metadata transmission: custom DisplayPort "Secondary Data Packets" (analogous to HDMI Data Island Packets) sent during the Horizontal Blanking Interval (HBI) or Vertical Blanking Interval (VBI) are used to carry rich signaling information. This signaling explicitly defines:
a) The precise geometry of the side-by-side/top-and-bottom packing (e.g., scaling factor, aspect ratio adjustments).
b) Parallax and convergence metadata for the 3D display to optimize depth perception.
c) Any embedded depth maps that complement the 2D side-by-side images, multiplexed into unused pixel data capacity within the blanking interval or allocated specific portions of the active frame (similar to the WOWvx format mentioned in the patent's background, but explicitly using DP's metadata capabilities for signaling).
This combines a common open 3D framing method with the patent's intelligent signaling over auxiliary channels for enhanced capability and flexibility.

3. OpenCAPI/CCIX as a Heterogeneous DDI for Distributed 3D Processing

Scenario Description:
US Patent 9036010 describes a "digital display interface" and refers to "a first audio-visual device" and "a second audio-visual device." While it implicitly assumes a dedicated display interface, the principles can extend to more generalized, high-speed interconnects. OpenCAPI (Open Coherent Accelerator Processor Interface) and CCIX (Cache Coherent Interconnect for Accelerators) are open-source, high-speed, cache-coherent interconnect standards designed for heterogeneous computing, linking CPUs with accelerators (like GPUs, FPGAs).

Combination: Consider a scenario where a high-performance rendering server (first AV device) streams complex volumetric stereoscopic data to a specialized 3D display controller (second AV device) for an advanced autostereoscopic or holographic display. Instead of a traditional display interface, an OpenCAPI or CCIX link is used as the "digital display interface."
The formatter on the rendering server multiplexes the stereoscopic data, which in this case might be raw point clouds, voxel data, or light field parameters (rather than pixel data), into the memory-mapped coherent transactions of OpenCAPI/CCIX. The "deep color modes" analogy extends to utilizing the high-bandwidth, low-latency, and cache-coherent nature of these interconnects to effectively stream large chunks of 3D volumetric data. Signaling information, indicating the 3D data format (e.g., voxel resolution, point cloud density, light field array parameters), rendering instructions, and synchronization metadata, is communicated via specific memory-mapped registers or doorbell mechanisms accessible through the coherent interconnect. This allows for distributed, high-fidelity 3D rendering where the initial stereoscopic components are not necessarily pixels but fundamental 3D scene descriptors, efficiently transported over a highly advanced, open-standard, heterogeneous computing interconnect.