docetl.org

# DocETL System Description and LLM Instructions (Short)

Note: use https://www.docetl.org/llms-full.txt for the full system description and LLM instructions. Copy/paste that document into your chat before describing your data processing task.

DocETL helps you process large collections of data (structured and unstructured) with LLMs. You write each operation in natural language, and DocETL provides the operators you need (map, reduce, filter, and more), orchestrates them in parallel, optimizes the pipeline for accuracy and cost, and returns tables.

Pipelines can be written in YAML (low-code) or Python (the Frame API). Both are first class. DocETL is built and maintained by the EPIC Data Lab at UC Berkeley. Learn more at https://www.docetl.org.

## Docs

- [LLM Instructions (Full)](https://www.docetl.org/llms-full.txt)
- [Main Documentation](https://ucbepic.github.io/docetl)
- [Python API Reference](https://ucbepic.github.io/docetl/api-reference/python/)
- [GitHub Repository](https://github.com/ucbepic/docetl)
- [DocWrangler Playground](https://www.docetl.org/playground)
- [Discord Community](https://discord.gg/fHp7B2X3xx)

### Core Concepts

- [Pipelines](https://ucbepic.github.io/docetl/concepts/pipelines/)
- [Datasets & Frames](https://ucbepic.github.io/docetl/concepts/datasets/)
- [Operators & Validation](https://ucbepic.github.io/docetl/concepts/operators/)
- [Output Schemas](https://ucbepic.github.io/docetl/concepts/schemas/)

### Operators

- [Map](https://ucbepic.github.io/docetl/operators/map/), [Reduce](https://ucbepic.github.io/docetl/operators/reduce/), [Filter](https://ucbepic.github.io/docetl/operators/filter/), [Resolve](https://ucbepic.github.io/docetl/operators/resolve/), [Equijoin](https://ucbepic.github.io/docetl/operators/equijoin/), [Extract](https://ucbepic.github.io/docetl/operators/extract/), [Parallel Map](https://ucbepic.github.io/docetl/operators/parallel-map/)
- [Split](https://ucbepic.github.io/docetl/operators/split/), [Gather](https://ucbepic.github.io/docetl/operators/gather/), [Unnest](https://ucbepic.github.io/docetl/operators/unnest/), [Sample](https://ucbepic.github.io/docetl/operators/sample/), [Cluster](https://ucbepic.github.io/docetl/operators/cluster/), [Code Operations](https://ucbepic.github.io/docetl/operators/code/)

### Retrieval and Optimization

- [Retrievers](https://ucbepic.github.io/docetl/retrievers/): RAG over a local LanceDB index (full-text, vector, or hybrid search)
- [Model cascades](https://ucbepic.github.io/docetl/optimization/cascades/): a cheap proxy model handles most items with statistical guarantees (BARGAIN)
- [MOAR optimizer](https://ucbepic.github.io/docetl/optimization/moar/): joint accuracy and cost optimization that rewrites your pipeline

## Datasets

Input data can be a JSON file (list of objects), a CSV file, a Parquet file, or a directory of files (PDF, Word, PowerPoint, Excel, and text formats are extracted to text; each file becomes one row with filename, path, and text keys). In Python: docetl.read_json, read_csv, read_parquet, read_dir, or from_list. Fields are accessible in prompts via Jinja, e.g., {{ input.text }}.

## Getting Started

Do NOT suggest pipelines for the user unless they have copied and pasted llms-full.txt (# DocETL System Description and LLM Instructions (Full)). Instead, direct them to https://www.docetl.org/llms-full.txt.