Skip to main content
Ingestion Process
Updated over 2 months ago

Loop processes documents from carriers and clients, transforming them into linked pairs of payable invoices and shipment job entities. Our system conducts audits, ensuring accuracy and reliability while providing clients with an intuitive platform to manage their operations.


Artifacts & Entities

At Loop, we call the documents we receive from carriers and clients "artifacts." We use the term "artifact" instead of "document" because it covers all bases—whether we’re dealing with PDF files or API responses. Each artifact has a type and a format:

  • Common types: bills of lading, invoices, and delivery receipts

  • Common formats: PDFs to JSON and JPEGs.

To extract data from any artifact, Loop will first convert it into a PDF format.

However, if we receive structured data, such as JSON shipment records through our API, Loop can utilize that data directly without requiring extraction.

To get the most out of these artifacts, we extract data to create structured information we call entities:

  • A payable invoice entity comes from invoice artifacts, allowing clients to approve, pay, export, and more. Each invoice artifact is linked to its corresponding payable invoice entity.

  • Similarly, we create shipment jobs from artifacts like shipment records and bills of lading. A shipment job represents a clear unit of work that can be audited, with all relevant records assigned to it.


Processing Steps

#1 Screening

During screening, we examine the artifact and decide whether it is worth processing. We screen artifacts based on things such as carrier names. Screened-in artifacts continue processing further to categorization. Screened-out artifacts stay in our system unprocessed.

Screening rules are configurable by the client. Only organizations that are in scope for a given client get screened in.

#2 Categorization

We categorize each artifact as one of our defined artifact types (invoice, bill of lading, delivery receipt, etc) through the categorization process. If an artifact is a multipage packet we categorize each page in the packet. If two pages following each other in the packet are categorized as the same type we check if they are continuations of each other or if they are standalone artifacts.

#3 Assignment

After an artifact is categorized we try to match it by assigning it to an entity, payable invoice or shipment job, that exists in our system already. The outcome of the assignment is either matched to an existing entity or "no match"

  • Invoice artifacts are assigned to invoice entities

  • all other artifacts are assigned to shipment jobs

The "no match" process is explained in the processing pipeline section below

#4 Entity normalization

During normalization, extracted data from an artifact is used to update an existing entity it matched to during assignment or create a new entity.

#5 Linking

After both the payable invoice and the shipment job entities have been normalized, Loop links them together.

#6 Audit

The audit has two parts:

  • Document audit: based on client configurations we check for the presence of certain data points, artifacts, or matches of data points between artifacts on a job<>invoice pair.

    • For example, we might check if a bill of lading is present, if the bill of lading number is present, and if it matches a certain number on the invoice.

  • Rate audit: we map the invoice line items to the expected charges and calculate a variance, whether and how much a carrier overcharges or undercharges for a given shipment job.


Processing Pipelines

Now that we established the steps, let's see how they all come together in our two use cases: TMS clients and Non-TMS clients.

TMS clients

  1. Artifacts are categorized

  2. Assignment

    1. Invoice bundles are assigned to invoiced

      1. New invoice entity is created upon no match

    2. Shipment records and non-invoice bundles are assigned to shipment jobs

      1. Only shipments record artifacts create a new shipment upon no match, other artifacts do not create a shipment and are left unassigned

  3. Normalization

    1. Invoice artifacts create or update invoice entities

    2. Shipment records create or update shipments

  4. Invoice entities and shipments are linked

Non-TMS clients

  1. Artifacts are categorized

  2. Assignment

    1. Invoice bundles are assigned to invoiced

      1. New invoice entity is created upon no match

    2. Non-invoice bundles are assigned to shipment jobs

      1. No shipments are created upon no match

  3. Normalization

    1. Invoice artifacts create or update invoice entities

    2. Non invoice artifacts update shipments

  4. Invoice entities and shipments are linked

    1. If no matching shipment job is found for an invoice entity, a new shipment is created from the invoice entity and artifacts within the invoice bundle, such as the bill of lading.

Did this answer your question?