Ingestion for agents that work with code

Optimise what your agent sees when you hand it a repo.

An agent that works on code needs to know what the code is. The usual answer is a hand-rolled directory walk: read some files, paste them into the prompt, hope it fits. It either misses the file that mattered or blows the token budget, and you cannot tell which from the outside.

papi-ai/ingest does that job once, properly: discover, filter, rank, render, and say out loud what was left out.

composer require papi-ai/ingest

use PapiAI\Ingest\Depth;
use PapiAI\Ingest\IngestRequest;
use PapiAI\Ingest\NativeIngestor;

$digest = (new NativeIngestor())->ingest(new IngestRequest(
    source: '/path/to/repo',
    depth: Depth::Tree,
    tokenBudget: 4000,
));

$agent->run("Here is the codebase:\n\n" . $digest->toPrompt());

Three depths, measured

Every file contributes at one of three depths. papi-core, 101 files, at each:

Api is the one worth reaching for. The entire system stays legible at under 7% of what Full costs. This is what one file looks like at that depth:

namespace PapiAI\Core;

enum Effort
  thinks(): bool
  budgetWithin(int $maxTokens): int
  fitsWithin(int $maxTokens): bool
  nearestOf(array $offered): self

Symbol extraction needs a parser, so it lives in its own package and the dependency is opt-in:

composer require papi-ai/ingest-php-symbols

use PapiAI\Ingest\PhpSymbols\PhpSymbolExtractor;

$ingestor = new NativeIngestor(new PhpSymbolExtractor());

Ask for Api without an extractor and it throws, naming the package to install. It will not quietly hand you a tree and let you wonder where the symbols went.

Budgets that admit what they dropped

Give it a budget and it renders the tree first, then files in priority order until the budget is spent. Anything that does not fit is named in a trailer. The same papi-core at Api depth with a 4,000 token budget renders 3,972 tokens and ends with:

NOTE: 31 of 101 file(s) omitted to fit a 4000 token budget: src/Exception/ProviderException.php, src/FailoverProvider.php, src/Message.php, ... and 11 more

Nothing is ever dropped silently.

Map plus heat

A budget needs a ranking. The prioritiser is how you say what matters right now, for example whatever was touched recently:

$recent = time() - 86400;

new IngestRequest(
    source: $repo,
    depth: Depth::Api,
    tokenBudget: 4000,
    prioritiser: fn (IngestedFile $file): float => $file->mtime > $recent ? 10.0 : 0.0,
);

The whole shape of the system, with the parts you are working on rendered first. Ties break by path, so the same repository always produces the same digest.

Deterministic, so you can cache it

No model is called during ingestion. Same repository state, same request, same digest, every time. That is what makes $digest->fingerprint worth keying a cache on: it moves when included content moves, and not otherwise. Touching a file without changing it does not bust your cache.

Inside a git repository, discovery defers to git ls-files, so .gitignore is honoured exactly, nested ignores and global excludes included. vendor/, node_modules/, lock files, .env* and binaries are out by default. Binaries still appear in the tree, with a note saying why they were not read.

Remote repositories

papi-ai/ingest-gitingest shells out to the gitingest CLI for URLs. Its output uses gitingest's own framing, a 48-character rule and FILE: path headers, so digests are interchangeable with what people already paste into models by hand.

use PapiAI\Ingest\Gitingest\GitingestIngestor;

$digest = (new GitingestIngestor())->ingest(new IngestRequest(
    source: 'https://github.com/papi-ai/papi-core',
    depth: Depth::Full,
    tokenBudget: 8000,
));

If the binary is missing it says so before doing any work, with the install command in the message.