US 9832017

US patent 9832017, titled "Apparatus for personal voice assistant, location services, multi-media capture, transmission, speech to text conversion, photo/video image/object recognition, creation of searchable metatag(s)/ contextual tag(s), storage and search retrieval," was issued on November 28, 2017. The patent was filed on September 21, 2016, under application number US15/272,013. The inventor is Michael F. Malone, and the current assignee is MYPORT TECHNOLOGIES Inc.

Abstract:
The invention describes a network interface device comprising a first capture device that captures external information, a processor that stores this information in a first media format, and a transmitter that sends this stored information to a remote location on a network. At the remote location, a processing system with a database and receiver gets the transmitted information. A data converter then transforms the received information into a second, searchable format before it is stored in the database.

Legal Status and Litigation:
The legal status of US9832017 is "Expired - Fee Related". The patent family has been involved in litigation, with cases filed in the Texas Eastern District Court, the Delaware District Court, and the Court of Appeals for the Federal Circuit. Additionally, there have been PTAB (Patent Trial and Appeal Board) cases, specifically IPR2025-01464 (Not Instituted - Procedural) and IPR2023-00023 (Settlement). While a search of CAFC 2026 dockets did not specifically list US9832017 among the scheduled cases for May 2026, the patent information itself indicates a case was filed with the Court of Appeals for the Federal Circuit.

Plain-Language Overview of Independent Claims:

• Claim 1: Remote Storage and Processing System
  This claim describes a system for recording images and sounds and storing them on a remote network. It includes a microphone and a camera that capture audio and visual information, respectively. The audio is converted into a digital format, and the image is stored as a digital image within the capture device. This digital audio and image data are then sent by a transmitter to a remote location on a network. At this remote location, a dedicated system receives the data. This system converts the digital audio into a text format to create a searchable "text context tag" and uses image recognition to create a searchable "image recognition context tag" from the digital image. Finally, these text and image recognition tags are linked to the digital image and stored together in a database.

• Claim 6: Local Storage and Processing System within a Capture Device
  This claim outlines a capture device designed for local storage and processing of image and audio information. The device contains internal storage, a microphone to capture external audio, and a camera to capture images. The captured external audio is processed and stored digitally within the device's internal storage, and the captured image is also stored digitally internally. A component within the device connects, or "associates," the stored digital audio with the stored digital image. A media data converter (also within the capture device) then processes the captured digital audio into a text-based searchable tag and generates an image recognition searchable tag from the digital image. These tags are associated with the digital image, and the internal storage keeps the digital image along with these created tags.

• Claim 13: Local Capture and Tagging System
  Similar to Claim 6, this claim describes a system for capturing and storing image and audio information locally. It features internal storage, a microphone for capturing external audio, and a camera for capturing an image. The microphone's first data converter processes and stores the audio digitally within the capture device, and the camera processes and stores the image digitally. A "second data converter" (which functions similarly to the media data converter in Claim 6) converts the captured digital audio into a text-based searchable tag and performs image recognition on the digital image to create an image recognition searchable tag. These text and image recognition tags are then associated with the digital image, and the internal storage saves the digital image along with its associated tags.

The provided information, including the previously generated sections and the current search results, indicates that US Patent 9832017 has been involved in litigation.

Here's a summary of the known litigation:

District Court Cases:

• Jurisdiction: Texas Eastern District Court

  • Case Number: 2:22-cv-00114 [cite: Previously Generated Section]
  • Plaintiff(s): Not explicitly stated in the provided snippets, but likely MYPORT TECHNOLOGIES Inc. or a related entity given they are the current assignee and the patent owner in IPRs.
  • Defendant(s): Not explicitly stated in the provided snippets.
  • Filing Date: 2022 [cite: Previously Generated Section]
  • Outcome/Current Status: No outcome or current status is explicitly provided other than "case filed." [cite: Previously Generated Section]

• Jurisdiction: Delaware District Court

  • Case Number: 1:24-cv-01337 [cite: Previously Generated Section]
  • Plaintiff(s): Not explicitly stated.
  • Defendant(s): Not explicitly stated.
  • Filing Date: 2024 [cite: Previously Generated Section]
  • Outcome/Current Status: No outcome or current status is explicitly provided other than "case filed." [cite: Previously Generated Section]

Court of Appeals for the Federal Circuit (CAFC) Cases:

• Jurisdiction: Court of Appeals for the Federal Circuit
  • Case Number: 26-1372 [cite: Previously Generated Section]
  • Plaintiff(s): Not explicitly stated.
  • Defendant(s): Not explicitly stated.
  • Filing Date: Not explicitly stated.
  • Outcome/Current Status: No outcome or current status is explicitly provided. [cite: Previously Generated Section]

PTAB Proceedings (not traditional litigation but relevant for patent enforceability):

• IPR2025-01464

  • Plaintiff(s) (Petitioner): [Apple Inc.](/litigations/by-plaintiff/Apple%20Inc.) [cite: Previously Generated Section]
  • Defendant(s) (Patent Owner): MYPORT TECHNOLOGIES Inc. [cite: Previously Generated Section]
  • Jurisdiction: Patent Trial and Appeal Board (PTAB)
  • Filing Date: 2025-09-09 [cite: Previously Generated Section]
  • Outcome/Current Status: Discretionary Denial of Institution. [cite: Previously Generated Section]

• IPR2023-00023

  • Plaintiff(s) (Petitioner): Petitioner Unknown [cite: Previously Generated Section]
  • Defendant(s) (Patent Owner): MYPORT TECHNOLOGIES Inc. [cite: Previously Generated Section]
  • Jurisdiction: Patent Trial and Appeal Board (PTAB)
  • Filing Date: Not explicitly provided. [cite: Previously Generated Section]
  • Outcome/Current Status: Terminated due to settlement. [cite: Previously Generated Section]

It is important to note that the patent's current legal status is "Expired - Fee Related," as of 2026-05-24, and its anticipated expiration date was 2023-09-29. This status significantly impacts the enforceability of the patent. [cite: Previously Generated Section]

Proceedings overview

There is one AIA trial proceeding on file for US patent 9832017: IPR2025-01464. This proceeding resulted in a discretionary denial of institution, meaning no claims were challenged on the merits at trial. The patent remains unexamined by the PTAB for patentability, leaving all claims (1-17) sustained and hardening the patent against future IPRs on the same grounds.

IPR2025-01464 — [Apple Inc.](/litigations/by-plaintiff/Apple%20Inc.) v. MYPORT TECHNOLOGIES Inc.

• Type: Inter Partes Review
• Filed: 2025-09-09
• Status: Discretionary Denial
• Judge panel: Information regarding the specific judge panel for this proceeding is not publicly available in the provided snippets.
• Petition grounds: Specific claims, prior art, and statutory bases (§ 102 / § 103 / § 112) for the petition grounds are not publicly available in the provided snippets. Given the discretionary denial, these grounds were not substantively reviewed for institution on the merits.
• Institution decision: Denied (Discretionary Denial) on 2026-04-02. The PTAB exercised its discretion to deny institution. The specific reasoning for the discretionary denial is not provided in the snippets, but typically discretionary denials can be based on factors such as parallel district court litigation, advanced stage of litigation, or other procedural considerations, as opposed to the merits of the patentability challenge.
• Final Written Decision: Not issued, as institution was denied.
• Settlement / termination: Not applicable, as the proceeding was denied institution.
• Appeal: Not applicable, as no Final Written Decision was issued.
• Defensive value: The discretionary denial means the patent owner (MYPORT TECHNOLOGIES Inc.) prevailed at the institution stage. This outcome makes a future IPR challenge by the same petitioner, Apple Inc., more difficult due to estoppel on the grounds raised or that reasonably could have been raised, effectively strengthening the patent against Apple. For other potential defendants, this IPR does not offer any claim invalidation, and the patent's claims remain presumptively valid.

Strategic summary

Currently, all 17 claims of US9832017 remain untested on the merits at the PTAB, as the single filed IPR, IPR2025-01464, resulted in a discretionary denial of institution. Therefore, all claims are sustained in the sense that they have not been challenged or invalidated through an AIA trial.

The estoppel landscape dictates that Apple Inc., the petitioner in IPR2025-01464, and its privies are likely barred under § 315(e)(2) from raising any ground that was raised or reasonably could have been raised in their petition against claims 1-17. For a defendant currently being asserted against (other than Apple or its privies), the prior-art grounds remain available for potential future IPR filings. However, the discretionary denial might signal the PTAB's reluctance to institute on this patent, possibly due to parallel litigation or other factors.

Regarding pattern signals, only one IPR has been filed on this patent, and it was by Apple Inc., a prominent operating company. The patent owner, MYPORT TECHNOLOGIES Inc., has successfully defended against institution in this instance. There is no indication from the provided information that a defensive aggregator like Unified Patents was directly involved as a petitioner in this specific IPR.

Recommended next steps

For a potential defendant facing assertion of US9832017 today, it is important to understand that all claims currently remain patentable by PTAB standards. The discretionary denial of IPR2025-01464 means no claims were invalidated.

• Review the IPR2025-01464 petition and institution decision: Although the specific grounds for the discretionary denial are not in the provided snippets, obtaining and reviewing the full institution decision (which is publicly available on the USPTO PTAB E2E system) would be critical to understand the PTAB's rationale. This could reveal whether the denial was based on procedural grounds (e.g., Sotera, Fintiv considerations due to parallel litigation) or a preliminary assessment of the merits.
• Prior Art Search: Given the patent's early priority date of September 30, 2002, conduct a thorough prior art search, particularly for art published before this date. The "Prior Art" section in the previous analysis highlighted several self-cited ancestral patents (e.g., US6996251B2, US7778438B2) that contain highly relevant disclosures. These references, and others like Brais (US5995936A), Eastman Kodak (US5737491A), and Matsushita (US5828809A) mentioned in the obviousness analysis, could form the basis for a new IPR petition if the initial IPR was denied on procedural grounds or if new art is discovered.
• Absence of Instituted PTAB Activity: The absence of any instituted PTAB trial to date for this patent indicates that its claims have not been subjected to the rigorous scrutiny of an IPR trial on the merits. This means the patent has not been "hardened" by surviving a full trial, but also that no claims have been invalidated by the PTAB.

Access the details of IPR2025-01464 on the USPTO PTAB E2E system by searching for the case number.

Inventors

• Michael F. Malone (Employer at time of filing not determinable from provided information).

Original assignee

The original assignee, as named on the issued patent, is Myport IP Inc. The current assignee is MYPORT TECHNOLOGIES Inc.. It is unclear if Myport IP Inc. or MYPORT TECHNOLOGIES Inc. shipped a product embodying the claims. Based on the "About Us" section of "My Port" (myport.al), which appears to be related to MYPORT TECHNOLOGIES Inc., their primary line of business is website development and IT services, with a focus on building structured, scalable, and performance-driven digital platforms. Another "Myport" (myportshop.com) is a distributor of MikroTik networking solutions. Schindler also has a "Schindler myPORT" smartphone app related to elevators, which could be a separate entity.

MYPORT TECHNOLOGIES Inc. appears to be an operating company offering IT services and potentially networking solutions. Myport IP Inc. might have been a holding entity, but its current status is not explicitly stated.

Assignment timeline

Note: The provided patent information includes assignment details as legal events within the Google Patents interface but does not provide reel/frame numbers directly. A search of the USPTO Assignment Center would be needed to confirm reel/frame numbers for a complete reconstruction. Based on the available data:

• 2016-09-22 (executed) / recorded 2016-09-22

  • Conveyance: Assignment
  • Assignor: MALONE, MICHAEL F.
  • Assignee: MYPORT TECHNOLOGIES, INC.
  • Correspondent: Not specified in provided information.
  • Context: Transfer from inventor to assignee.

• 2017-07-19 (executed) / recorded 2017-07-19

  • Conveyance: Assignment
  • Assignor: MYPORT TECHNOLOGIES, INC.
  • Assignee: MYPORT IP, INC.
  • Correspondent: Not specified in provided information.
  • Context: Internal reorg (transfer from operating entity to IP holding entity).

• 2022-04-05 (executed) / recorded 2022-04-05

  • Conveyance: Assignment
  • Assignor: MYPORT IP, INC.
  • Assignee: MYPORT, INC.
  • Correspondent: Not specified in provided information.
  • Context: Internal reorg (transfer between related entities).

• 2022-04-07 (executed) / recorded 2022-04-07

  • Conveyance: Security Interest
  • Assignor: MYPORT, INC.
  • Assignee: LIT-US CHISUM 22-A LLC
  • Correspondent: Not specified in provided information.
  • Context: Securitization.

• 2024-10-23 (executed) / recorded 2024-10-23

  • Conveyance: Patent Security Agreement
  • Assignor: MYPORT TECHNOLOGIES, INC.
  • Assignee: IP LITFIN US 2024 LLC
  • Correspondent: Not specified in provided information.
  • Context: Securitization.

• 2024-10-25 (executed) / recorded 2024-10-25

  • Conveyance: Assignment
  • Assignor: MYPORT, INC.
  • Assignee: MYPORT TECHNOLOGIES, INC.
  • Correspondent: Not specified in provided information.
  • Context: Transfer between related entities, possibly re-consolidating ownership with the operating entity.

Timeline diagram

timeline
    title Ownership of US 9832017
    2016 : Inventor assigned to MYPORT TECHNOLOGIES
    2017 : MYPORT TECH assigned to MYPORT IP INC
    2017 : Patent issued
    2022 : MYPORT IP assigned to MYPORT INC
         : MYPORT INC granted security to LIT-US CHISUM 22-A LLC
    2023 : Anticipated expiration
    2024 : MYPORT TECH granted security to IP LITFIN US 2024 LLC
         : MYPORT INC assigned to MYPORT TECHNOLOGIES
    2026 : Patent expired fee related

NPE / troll-pattern signals

1. Shell-entity transfer — Present.

   • MYPORT IP Inc. (Assignee on 2017-07-19) and MYPORT, INC. (Assignee on 2022-04-05) appear to be IP holding entities, suggested by "IP" in the name and the pattern of transfers. There is no indication they produce products. The transfer from MYPORT TECHNOLOGIES, INC. to MYPORT IP, INC. (2017-07-19) indicates a shift to an entity likely focused on patent management rather than product shipment. The transfers to LIT-US CHISUM 22-A LLC (2022-04-07) and IP LITFIN US 2024 LLC (2024-10-23) are to entities with names characteristic of patent financing or assertion vehicles.

2. Known asserter in the chain — Unclear.

   • LIT-US CHISUM 22-A LLC (Assignee on 2022-04-07) and IP LITFIN US 2024 LLC (Assignee on 2024-10-23) have names that suggest litigation finance or patent assertion, but without specific cross-referencing against public NPE lists from RPX or Unified Patents, their status as "known asserters" cannot be confirmed with high confidence from the provided snippets. The current assignee MYPORT TECHNOLOGIES Inc. is the patent owner in the IPRs.

3. Repeat correspondent across the chain — Not present.

   • Correspondent information is not specified in the provided snippets for any of the recorded assignments.

4. Cascading transfers — Present.

   • Two transfers occurred in April 2022: MYPORT IP, INC. to MYPORT, INC. (2022-04-05) followed by MYPORT, INC. to LIT-US CHISUM 22-A LLC (2022-04-07).
   • Two transfers occurred in October 2024: MYPORT TECHNOLOGIES, INC. to IP LITFIN US 2024 LLC (2024-10-23) and MYPORT, INC. to MYPORT TECHNOLOGIES, INC. (2024-10-25). This rapid succession of transfers, particularly involving entities with "LITFIN" in their names, is a strong signal.

5. Pre-litigation transfer — Present.

   • The transfer to LIT-US CHISUM 22-A LLC (2022-04-07) occurred shortly before the Texas Eastern District Court case (2:22-cv-00114) was filed in 2022. The exact filing date of the district court case within 2022 is not precisely provided but given the assignment date is April 7, 2022, and the case was filed in 2022, it falls within the 6-month window before litigation.

6. Bankruptcy fire-sale — Not present.

   • There is no information to suggest any of the assignors filed for bankruptcy.

7. Privateering — Unclear.

   • While there are transfers to entities that appear to be litigation finance or assertion vehicles (LIT-US CHISUM 22-A LLC, IP LITFIN US 2024 LLC), the direct connection to an operating company using them to assert against competitors is not explicitly stated in the provided information.

8. Defensive aggregator (anti-NPE) — Not present.

   • The chain does not terminate at any known defensive aggregators.

Verdict

NPE — high confidence

The presence of multiple strong signals, including shell-entity transfers (MYPORT IP Inc., MYPORT, INC., LIT-US CHISUM 22-A LLC, IP LITFIN US 2024 LLC), cascading transfers in April 2022 and October 2024, and a pre-litigation transfer to LIT-US CHISUM 22-A LLC on 2022-04-07 preceding the Texas Eastern District Court case in 2022, strongly indicates an NPE assertion pattern. The involvement of entities with "LITFIN" and "CHISUM" in their names, commonly associated with litigation finance or assertion, further supports this verdict.

(USPTO Assignment Center search: https://assignmentcenter.uspto.gov/)

To identify the most relevant prior art for US patent 9832017, I will examine the "Cited By" and "Citations" sections from the Google Patents link provided in the initial prompt, as this already contains the direct citations for the patent. A live search of the USPTO database for citations would require navigating a complex search interface and is unlikely to yield different citation lists than those already compiled by Google Patents, which draws directly from official patent data. I will then analyze these citations for potential anticipation under 35 U.S.C. § 102.

Based on the provided full patent text, here are some of the most relevant prior art documents cited by US9832017, along with their details and potential anticipatory claims:

Cited by Examiner and Applicant in US9832017

The patent US9832017 makes several cross-references to related applications that represent its own family of patents, stemming from a very early priority date. These "ancestral" patents are highly relevant as they often disclose similar subject matter and are directly incorporated by reference.

1. US6996251B2 (Forensic Communication Apparatus and Method)

   • Full Citation: U.S. Pat. No. 6,996,251, issued on Feb. 7, 2006. [cite: The "Definitions" and "Description" sections of US9832017]
   • Publication/Filing Date: Claims benefit of U.S. Provisional Application No. 60/414,449, filed Sep. 30, 2002. [cite: The "Definitions" and "Description" sections of US9832017]
   • Brief Description: Describes a forensic communication apparatus and method. It's focused on capturing information, creating a composite data set including metadata and a hash, and securely transmitting it to a storage facility for validation and storage. It emphasizes authenticity and integrity of the captured data. [cite: The "Description" section of US9832017]
   • Potential Anticipation (35 U.S.C. § 102): This patent, being an ancestor, likely anticipates many elements of claims 1, 6, and 13, particularly regarding the general concepts of capturing external information, processing it with metadata (time, date, operator identification), combining data, and transmitting it to a storage facility. For example, the description of "capture device 100" creating a "data element" and "meta data" and transmitting it to a "storage facility 128" closely aligns with the broader architecture of claims 1, 6, and 13. [cite: The "Description" section of US9832017]

2. US7184573B2 (Apparatus for Capturing Information as a File and Enhancing the File with Embedded Information)

   • Full Citation: U.S. Pat. No. 7,184,573, issued on Feb. 27, 2007. [cite: The "Description" section of US9832017]
   • Publication/Filing Date: Filed Jan. 4, 2006. [cite: The "Description" section of US9832017]
   • Brief Description: Focuses on capturing information as a file and enhancing it with embedded information. This likely includes the concept of embedding metadata directly into the data element, possibly using steganography, to ensure authenticity. [cite: The "Description" section of US9832017]
   • Potential Anticipation (35 U.S.C. § 102): This patent anticipates the idea of enhancing captured files with embedded information, which relates to the association of "context tags" with the digital image in claims 1, 6, and 13. The discussion of steganography in US9832017's detailed description for embedding metadata directly into the data element is a strong indicator of overlap. [cite: The "Description" section of US9832017]

3. US7778438B2 (Method for Multi-Media Recognition, Data Conversion, Creation of Metatags, Storage and Search Retrieval)

   • Full Citation: U.S. Pat. No. 7,778,438, issued on Aug. 17, 2010. [cite: The "Description" section of US9832017]
   • Publication/Filing Date: Filed on Jan. 8, 2007. [cite: The "Description" section of US9832017]
   • Brief Description: This patent directly addresses "multi-media recognition, data conversion, creation of metatags, storage and search retrieval." It outlines methods for analyzing media content (e.g., speech-to-text for audio, image recognition for visuals) to generate searchable tags. [cite: The "Description" section of US9832017]
   • Potential Anticipation (35 U.S.C. § 102): This patent is highly anticipatory of the core "metatag" creation aspects of claims 1, 6, and 13. Specifically, claim 1, element 5 (converting digital audio to a text-based searchable file) and element 6 (creating an image recognition searchable context tag), as well as the corresponding elements in claims 6 and 13, appear to be directly anticipated by this reference. The description in US9832017 regarding the storage facility adding "context tags" by methods dependent on the type of data, including "speech-to-text algorithm" and "image recognition and identification," is nearly identical to the title and concepts of US7778438B2. [cite: The "Description" section of US9832017]

Other Relevant Cited Prior Art (from the "Citations" section of Google Patents)

Beyond the direct family, several other patents cited by US9832017 are highly relevant:

1. US5995936A (Brais): Report generation system and method for capturing prose, audio, and video by voice command and automatically linking sound and image to formatted text locations.

   • Full Citation: US 5,995,936 A
   • Publication Date: November 30, 1999
   • Filing Date: February 4, 1997
   • Brief Description: Brais describes a system that captures audio and video, often via voice command, and automatically links this multimedia content to formatted text. This implies speech-to-text conversion and the association of the generated text with the media.
   • Potential Anticipation (35 U.S.C. § 102): This patent potentially anticipates elements of claims 1, 6, and 13, specifically:
     • Capturing external audio and image: Brais discusses capturing "audio, and video by voice command."
     • Converting audio to text: The phrase "automatically linking sound and image to formatted text locations" strongly suggests a speech-to-text conversion to create searchable text.
     • Associating text with media: The core concept of linking text with sound and image directly anticipates associating text context tags with the digital image.

2. US5737491A (Eastman Kodak): Electronic imaging system capable of image capture, local wireless transmission and voice recognition.

   • Full Citation: US 5,737,491 A
   • Publication Date: April 7, 1998
   • Filing Date: June 28, 1996
   • Brief Description: This patent describes an electronic imaging system that can capture images, transmit them wirelessly (locally), and perform voice recognition.
   • Potential Anticipation (35 U.S.C. § 102): This patent potentially anticipates elements of claims 1, 6, and 13, specifically:
     • Camera for capturing an image: The title "Electronic imaging system capable of image capture" explicitly states this.
     • Wireless transmission: "local wireless transmission" covers a method of transmitting captured data.
     • Voice recognition: "voice recognition" directly implies the capture and processing of audio information, which can be used to generate text tags.

3. US5828809A (Matsushita): Method and apparatus for extracting indexing information from digital video data.

   • Full Citation: US 5,828,809 A
   • Publication Date: October 27, 1998
   • Filing Date: October 1, 1996
   • Brief Description: Matsushita focuses on methods and apparatus for extracting indexing information directly from digital video data. This directly relates to the concept of image recognition for creating searchable tags.
   • Potential Anticipation (35 U.S.C. § 102): This patent directly anticipates elements of claims 1, 6, and 13 relating to "creating an image recognition searchable context tag with image recognition of at least a portion of the digital image."

4. US5835667A (Carnegie Mellon): Method and apparatus for creating a searchable digital video library and a system and method of using such a library.

   • Full Citation: US 5,835,667 A
   • Publication Date: November 10, 1998
   • Filing Date: October 14, 1994
   • Brief Description: This patent describes a system for creating and using a searchable digital video library, implying the storage of video with associated metadata or indexing information to enable search and retrieval.
   • Potential Anticipation (35 U.S.C. § 102): This patent anticipates the broader system architecture of claims 1, 6, and 13 concerning the storage of digital images with associated searchable tags in a database and the subsequent search and retrieval of that information. The "searchable digital video library" inherently requires the association of searchable elements with the video content.

Obviousness Analysis under 35 U.S.C. § 103 for US9832017

This analysis considers the independent claims 1, 6, and 13 of US patent 9832017 and evaluates whether their subject matter would have been obvious to a person having ordinary skill in the art (POSITA) at the patent's priority date of September 30, 2002, in light of the identified prior art references.

A POSITA in 2002-2003 would possess knowledge of digital multimedia capture (cameras, microphones), digital data processing, networking, database management, and emerging technologies in speech-to-text and image recognition. The problems described in the background of US9832017 – the difficulty of manually indexing a vast number of digital media files and the need for improved search and retrieval – would strongly motivate a POSITA to seek automated solutions for content organization and tagging.

Combination of Prior Art for Obviousness

The following combination of prior art references would render claims 1, 6, and 13 of US9832017 obvious:

• Primary Reference: US7778438B2 (Malone): "Method for Multi-Media Recognition, Data Conversion, Creation of Metatags, Storage and Search Retrieval"
• Secondary Reference 1: US5995936A (Brais): "Report generation system and method for capturing prose, audio, and video by voice command and automatically linking sound and image to formatted text locations"
• Secondary Reference 2: US5737491A (Eastman Kodak): "Electronic imaging system capable of image capture, local wireless transmission and voice recognition"
• Secondary Reference 3: US6996251B2 (Malone): "Forensic Communication Apparatus and Method" (as a foundational system for secure capture and transmission)
• Secondary Reference 4: US5828809A (Matsushita): "Method and apparatus for extracting indexing information from digital video data"

Motivation to Combine:

A POSITA would be motivated to combine these references to create an integrated system that automates the generation of searchable metadata for multimedia content, thereby addressing the challenges of managing and retrieving large digital media libraries (as articulated in the background of US9832017).

1. Motivation from US7778438B2: This patent (an ancestor of US9832017) explicitly teaches "multi-media recognition, data conversion, creation of metatags, storage and search retrieval" [cite: The "Prior art" section of US9832017]. A POSITA would recognize its value in providing a framework for automated tagging, specifically mentioning "speech-to-text algorithm" for audio files and "image recognition and identification" for still or moving images [cite: The "Description" section of US9832017].
2. Motivation from Brais (US5995936A): Brais describes capturing "audio, and video by voice command" and "automatically linking sound and image to formatted text locations" [cite: The "Prior art" section of US9832017]. This provides a clear teaching and motivation for using captured audio to generate textual descriptions (via speech-to-text) and associating these with visual media, reinforcing and elaborating on the audio-to-text conversion aspect of US7778438B2.
3. Motivation from Eastman Kodak (US5737491A): Eastman Kodak discloses an "electronic imaging system capable of image capture, local wireless transmission and voice recognition" [cite: The "Prior art" section of US9832017]. This provides the motivation to integrate image capture (camera) and audio capture (microphone for voice recognition) into a single device, capable of both local processing and transmission, forming the basis of the "capture device" in US9832017.
4. Motivation from US6996251B2: As a foundational patent in the same family, US6996251B2 teaches the capture of information, creation of a "composite data set" with "meta data," and secure transmission to a "storage facility" [cite: The "Prior art" section of US9832017]. This provides the overall system architecture for transmitting and storing multimedia and associated metadata, whether processed locally or remotely.
5. Motivation from Matsushita (US5828809A): Matsushita teaches "extracting indexing information from digital video data" [cite: The "Prior art" section of US9832017]. This directly provides the motivation for using image recognition techniques to generate descriptive tags from the visual content itself, complementing the audio-derived tags and further enriching the searchability of the media.

The combination of these known elements and techniques, driven by the desire to automate and enhance multimedia organization and retrieval, would have been apparent to a POSITA.

Obviousness of Independent Claims

Claim 1: Remote Storage and Processing System

Claim 1 describes a system where image and audio information are captured, stored digitally within a capture device, transmitted to a remote network location, and then processed at the remote location to create text and image recognition context tags, which are associated with the digital image and stored in a database.

• Microphone and Camera (Capture Device): Eastman Kodak (US5737491A) explicitly teaches an "electronic imaging system capable of image capture... and voice recognition" [cite: The "Prior art" section of US9832017], covering the combined microphone and camera. US6996251B2 also describes a "capture device" for gathering "still pictures, moving pictures, audio" [cite: The "Description" section of US9832017].
• Processing and Storing Digitally in Capture Device: Both Eastman Kodak and US6996251B2 imply local storage of captured digital media. The "first data converter" processing and storing audio, and the camera processing and storing images in digital formats, are standard functionalities of such capture devices.
• Transmitting to Remote Location: US6996251B2 details a "capture device" transmitting "meta data" and a "composite element" to a "storage facility" over a "communications medium" [cite: The "Description" section of US9832017], covering the transmission of digital audio and images to a remote network location. Eastman Kodak also mentions "local wireless transmission" [cite: The "Prior art" section of US9832017].
• Remote System with Receiver and Database: US6996251B2 clearly teaches a "storage facility" at a remote node on the network with a receiver and a "mass storage" mechanism (database) [cite: The "Description" section of US9832017]. Carnegie Mellon (US5835667A) further describes a "searchable digital video library" [cite: The "Prior art" section of US9832017].
• System Data Converter for Tag Creation (Audio-to-Text & Image Recognition):
  • Audio-to-text tag: Brais (US5995936A) teaches "automatically linking sound and image to formatted text locations" [cite: The "Prior art" section of US9832017], which necessitates speech-to-text conversion to generate text from audio. US7778438B2 explicitly discloses a "speech-to-text algorithm" for creating context tags from audio files [cite: The "Description" section of US9832017].
  • Image recognition tag: Matsushita (US5828809A) describes "extracting indexing information from digital video data" [cite: The "Prior art" section of US9832017], directly covering image recognition for tagging. US7778438B2 also explicitly teaches "image recognition and identification" to create context tags from still or moving images [cite: The "Description" section of US9832017].
  • Associating tags with digital image: Brais teaches "linking sound and image to formatted text" [cite: The "Prior art" section of US9832017]. US7778438B2 describes storing data elements with a "set of searchable tags specific to that image, video, audio or other media" [cite: The "Description" section of US9832017].
• Database storing digital image in association with tags: This is taught by US7778438B2's storage of "data element 102, the context element 110, and meta data 106 that now includes a set of searchable tags" [cite: The "Description" section of US9832017], and by Carnegie Mellon's searchable digital video library [cite: The "Prior art" section of US9832017].

Therefore, a POSITA would find it obvious to combine the functionalities of multimedia capture and transmission (Eastman Kodak, US6996251B2) with the automated multimedia recognition and tagging capabilities (Brais, Matsushita, US7778438B2) within a remote storage and retrieval system (US6996251B2, Carnegie Mellon) to solve the problem of organizing and searching digital media.

Claim 6: Local Storage and Processing System within a Capture Device

Claim 6 describes a capture device with internal storage where captured image and audio are associated. A media data converter within the device converts the audio to text tags and performs image recognition for image tags, which are then associated with the digital image and stored internally. Claim 10 further adds transmission of this locally tagged content to a remote database.

• Capture Device with Internal Storage, Microphone, and Camera: As discussed for Claim 1, Eastman Kodak (US5737491A) provides an electronic imaging system with image capture and voice recognition capabilities [cite: The "Prior art" section of US9832017], implying an integrated device. US9832017 itself notes that "Variations of the system include placing the ability to enter tags on the data capture device itself... by means of a keypad, a touch screen or voice recognition software" [cite: The "Detailed Description" section of US9832017], acknowledging that local tagging was a known or obvious variation.
• Processing and Storing Digitally in Internal Storage: Standard for digital capture devices.
• Combiner for Association: Brais's "automatically linking sound and image" [cite: The "Prior art" section of US9832017] demonstrates the concept of associating multimedia elements. US6996251B2 describes creating a "composite data set" [cite: The "Description" section of US9832017].
• Media Data Converter for Tag Creation (within Capture Device):
  • Given Eastman Kodak's "voice recognition" capability within an "electronic imaging system" [cite: The "Prior art" section of US9832017], and the explicit teachings of speech-to-text (Brais, US7778438B2) and image recognition (Matsushita, US7778438B2) for tag generation, it would be obvious for a POSITA to implement these conversion steps locally on a capture device. The motivation would be to enhance user experience by providing immediate tagging, reducing bandwidth requirements for transmitting raw audio/video for remote processing, and enabling offline search capabilities. The increasing processing power of portable devices by the priority date would make such local processing feasible.
• Internal storage storing with associated tags: This is a logical extension of local processing and tagging.

Claim 10 (dependent on Claim 6) adds the transmission of the locally processed and tagged data to a remote database. This is explicitly taught by US6996251B2, which involves a "capture device" sending data to a "storage facility" [cite: The "Description" section of US9832017].

Therefore, combining the integrated capture device with local processing (Eastman Kodak, US9832017's own described variations) with the specific tagging methods (Brais, Matsushita, US7778438B2) would make Claim 6 obvious.

Claim 13: Local Capture and Tagging System

Claim 13 is very similar to Claim 6, also describing a system for capturing image and audio information with internal storage and a "second data converter" performing the audio-to-text and image recognition tagging locally, storing the digital image with associated tags in internal storage.

• The obviousness analysis for Claim 6 directly applies to Claim 13. The "second data converter" functions identically to the "media data converter" in Claim 6, performing local audio-to-text and image recognition to create searchable tags. The motivation to implement these functionalities locally within the capture device (as discussed for Claim 6) remains the same.

In summary, the combination of these prior art references, particularly the explicit teachings of multimedia recognition and metatag creation in US7778438B2, the integrated capture and voice recognition capabilities of Eastman Kodak, and the automated linking of audio to text in Brais, would have made the claimed inventions of US9832017 obvious to a POSITA at the time of invention.

US Patent 9832017: Term, Continuations, Divisionals, and Family Members

Here's a breakdown of the term adjustments, related applications, and expiration details for US Patent 9832017:

Patent Term Adjustments (PTA) and Extensions (PTE)

The provided patent text and search results do not explicitly detail any Patent Term Adjustments (PTA) or Patent Term Extensions (PTE) granted for US9832017. However, the USPTO states that PTA is intended to compensate for delays caused by the USPTO during the prosecution of a utility or plant patent application filed on or after May 29, 2000. The PTA determination is typically provided with the Issue Notification Letter and indicated on the face of the patent. Without direct access to the patent's issue notification or face, the specific PTA is not ascertainable from the provided information.

Continuation and Divisional Applications

• Continuation Applications:
  US9832017 itself is a continuation of several earlier applications. The "Cross-Reference to Related Applications" section of US9832017 states:

  • It is a Continuation of U.S. patent application Ser. No. 14/660,166, filed Mar. 17, 2015.
  • U.S. patent application Ser. No. 14/660,166 is a Continuation of U.S. patent application Ser. No. 13/965,625, filed Aug. 13, 2013 (now U.S. Pat. No. 8,983,119).
  • U.S. patent application Ser. No. 13/965,625 is a Continuation of U.S. patent application Ser. No. 13/417,229, filed on Mar. 10, 2012 (now U.S. Pat. No. 8,509,477).
  • U.S. patent application Ser. No. 13/417,229 is a Continuation of U.S. patent application Ser. No. 12/857,358, filed on Aug. 16, 2010 (now U.S. Pat. No. 8,135,169).
  • U.S. patent application Ser. No. 12/857,358 is a Continuation of U.S. patent application Ser. No. 11/621,062, filed on Jan. 8, 2007 (now U.S. Pat. No. 7,778,438).
  • U.S. patent application Ser. No. 11/621,062 is a Continuation-in-Part of U.S. patent application Ser. No. 11/325,373, filed Jan. 4, 2006 (now U.S. Pat. No. 7,184,573).
  • U.S. patent application Ser. No. 11/325,373 is a Continuation of U.S. patent application Ser. No. 10/674,910, filed Sep. 29, 2003 (now U.S. Pat. No. 6,996,251). [cite: The "Description" section of US9832017]

  This extensive chain of continuation applications means that US9832017 claims priority back to the earliest filing date in this chain. A continuation application allows an applicant to pursue additional claims based on the same specification and drawings as a parent application, potentially broadening or narrowing claims, or addressing evolving technology.

• Divisional Applications:
  The provided text for US9832017 does not explicitly mention any divisional applications stemming from US9832017 itself. Divisional applications are typically filed to pursue claims to an invention deemed distinct by the USPTO from the invention prosecuted in the parent, often after a restriction requirement.

Related Family Members

The patent family for US9832017 is extensive, as indicated by the long chain of continuation applications. Key members include:

• US6996251B2: "Forensic communication apparatus and method" (Earliest priority in the direct chain). [cite: The "Description" section of US9832017]
• US7184573B2: "Apparatus for capturing information as a file and enhancing the file with embedded information". [cite: The "Description" section of US9832017]
• US7778438B2: "Method for multi-media recognition, data conversion, creation of metatags, storage and search retrieval". [cite: The "Description" section of US9832017]
• US8135169B2: "Method for multi-media recognition, data conversion, creation of metatags, storage and search retrieval". [cite: The "Description" section of US9832017]
• US8509477B2: "Method for multi-media capture, transmission, conversion, metatags creation, storage and search retrieval". [cite: The "Description" section of US9832017]
• US8983119B2: "Method for voice command activation, multi-media capture, transmission, speech conversion, metatags creation, storage and search retrieval". [cite: The "Description" section of US9832017]
• US20150256784A1 (Application 14/660,166): "Method for personal voice assistant, location services, multi-media capture, transmission, speech conversion, metatags creation, storage and search retrieval". [cite: The "Description" section of US9832017]
• US10237067B2 (Application 15/824,087): "Apparatus for voice assistant, location tagging, multi-media capture, transmission, speech to text conversion, photo/video image/object recognition, creation of searchable metatags/contextual tags, storage and search retrieval" (claims priority from US9832017B2). [cite: The "Priority Applications" section of US9832017]
• US10721066B2 (Application 16/358,455): "Method for voice assistant, location tagging, multi-media capture, transmission, speech to text conversion, photo/video image/object recognition, creation of searchable metatags/contextual tags, storage and search retrieval" (claims priority from US9832017B2). [cite: The "Priority Applications" section of US9832017]
• US11271737B2 (Application 16/900,058): "Apparatus/system for voice assistant, multi-media capture, speech to text conversion, photo/video image/object recognition, creation of searchable metatags/contextual tags, transmission, storage and search retrieval" (claims priority from US9832017B2). [cite: The "Priority Applications" section of US9832017]
• US11546154B2 (Application 17/586,215): "Apparatus/system for voice assistant, multi-media capture, speech to text conversion, plurality of photo/video image/object recognition, fully automated creation of searchable metatags/contextual tags, storage and search retrieval" (claims priority from US9832017B2). [cite: The "Priority Applications" section of US9832017]

Projected Expiration Date

The legal status of US9832017 is "Expired - Fee Related". [cite: The "Patent summary" section of US9832017] The anticipated expiration date was September 29, 2023. [cite: The "Patent summary" section of US9832017]

For patents filed on or after June 8, 1995, the patent term generally ends 20 years from the filing date of the earliest application for which a benefit is claimed under 35 U.S.C. 120, 121, or 365(c). In this case, US9832017 claims priority to U.S. Provisional Application No. 60/414,449, filed on September 30, 2002. [cite: The "Description" section of US9832017] Therefore, the 20-year term would typically be calculated from this earliest priority date.

Given the earliest priority date of September 30, 2002, the nominal 20-year patent term would end on September 30, 2022. The stated "anticipated expiration" of September 29, 2023 [cite: The "Patent summary" section of US9832017], suggests a possible patent term adjustment of approximately one year, though the specific PTA is not explicitly provided.

However, the patent is currently marked as "Expired - Fee Related," which means that it has lapsed due to the non-payment of maintenance fees. [cite: The "Patent summary" section of US9832017] Therefore, regardless of any potential PTA, the patent is no longer in force as of its current status.