Open Contracts (Demo)

The Free and Open Source Document Analytics Platform

---

Backend CI/CD
Meta

TLDR: What Does it Do?

OpenContracts is an Apache-2 Licensed enterprise document analytics tool. It supports multiple formats - including PDF and txt-based formats (with more on the way). It also supports multiple document ingestion pipelines with a pluggable architecture designed to make supporting new formats and ingestion engines easy - see our Docling Integration for an example. Writing your own custom document analytics tools where the results get displayed beautifully over the original document is easy. We also support mass document data extraction with a LlamaIndex wrapper.

PDF-Annotation and Analysis:

TXT-Based Format Annotation and Analysis:

Data Extract:

Rapidly Deployable Bespoke Analytics

[DEVELOPING] Document Management

🗺️ Roadmap

Timeline	Milestone	Description
🌟 January 2025	Document Format Expansion	LlamaParse integration for enhanced document processing Native Markdown viewing and highlighting support
📊 February 2025	Benchmarking Suite	Comprehensive benchmarks for extract features Benchmark Tools
🚀 March 2025	Extract Enhancement	Improved default data extract algorithms Enhanced accuracy and speed
🌊 April 2025	Streaming & Filtering	Real-time response streaming for chats and queries Better UX for asking questions of entier corpus Improved search functionality
📚 May 2025	Government Data Integration	Code of Federal Regulations (CFR) integration Additional government data sources
🏗️ Summer 2025	Enterprise DMS Features	Document versioning Intelligent file matching

💡 This roadmap represents our current plans and is subject to change based on community feedback and evolving requirements.

Ok, now tell me more. What Does it Do?

OpenContracts provides several key features:

1. Manage Documents - Manage document collections (Corpuses)
2. Layout Parser - Automatically extracts layout features from PDFs
3. Automatic Vector Embeddings - generated for uploaded PDFs and extracted layout blocks
4. Pluggable microservice analyzer architecture - to let you analyze documents and automatically annotate them
5. Pluggable Parsing Pipelines - to let you support new document formats and ingestion engines.
6. Human Annotation Interface - to manually annotated documents, including multi-page annotations.
7. LlamaIndex Integration - Use our vector stores (powered by pgvector) and any manual or automatically annotated features to let an LLM intelligently answer questions.
8. Data Extract - ask multiple questions across hundreds of documents using complex LLM-powered querying behavior. Our sample implementation uses LlamaIndex + Marvin.
9. Custom Data Extract - Custom data extract pipelines can be used on the frontend to query documents in bulk.

Key Docs

We recommend you browse our docs via our Mkdocs Site. You can also view the docs in the repo:

1. Quickstart Guide - You'll probably want to get started quickly. Setting up locally should be pretty painless if you're already running Docker.
2. Basic Walkthrough - Check out the walkthrough to step through basic usage of the application for document and annotation management.
3. PDF Annotation Data Format Overview - You may be interested how we map text to PDFs visually and the underlying data format we're using.
4. Django + Pgvector Powered Hybrid Vector Database We've used the latest open source tooling for vector storage in postgres to make it almost trivially easy to combine structured metadata and vector embeddings with an API-powered application.
5. LlamaIndex Integration Walkthrough - We wrote a wrapper for our backend database and vector store to make it simple to load our parsed annotations, embeddings and text into LlamaIndex. Even better, if you have additional annotations in the document, the LLM can access those too.
6. Write Custom Data Extractors - Custom data extract tasks (which can use LlamaIndex or can be totally bespoke) are automatically loaded and displayed on the frontend to let user's select how to ask questions and extract data from documents.

Architecture and Data Flows at a Glance

Core Data Standard

The core idea here - besides providing a platform to analyze contracts - is an open and standardized architecture that makes data extremely portable. Powering this is a set of data standards to describe the text and layout blocks on a PDF page:

Modern, Pluggable Document Processing Pipeline

OpenContracts features a powerful, modular pipeline system for processing documents. The architecture supports easy creation and integration of custom parsers, embedders, and thumbnail generators:

Each pipeline component inherits from a base class that defines a clear interface:

• Parsers: Extract text and structure from documents
• Embedders: Generate vector embeddings for semantic search
• Thumbnailers: Create visual previews of documents

Learn more about:

• Pipeline Architecture Overview
• Docling Parser
• NLM-Ingest Parser

The modular design makes it easy to add custom processors - just inherit from the appropriate base class and implement the required methods. See our pipeline documentation for details on creating your own components.

Limitations

At the moment, we only support PDF and text-based formats (like plaintext and MD). With our new parsing pipeline, we can easily support other ooxml office formats like docx and xlsx, HOWEVER, open source viewers and editors are a rarity. One possible route is to leverage the many ooxml --> MD tools that now exist. This will be a reasonably good solution for the majority of documents once we add a markdown viewer and annotator (see our roadmap).

Acknowledgements

Special thanks to AllenAI's PAWLS project and Nlmatics nlm-ingestor. They've pioneered a number of features and flows, and we are using their code in some parts of the application.

NLmatics was also the creator of and inspiration for our data extract grid and parsing pipeline UI/UX:

The company was ahead of its time, and, while the product is no longer available, OpenContracts aims to take some of its best and most innovative features and make them open source and available to the masses!