Register, prioritize, and contextualize papers.

The registry#

LemPaperRegistry loads the paper manifest from ~/.lem/papers.json by default; point Lem at an alternate registry with LEM_PAPERS_PATH. An absent registry lists as an empty set with a note; one that is there but cannot be read or decoded is refused with “paper registry <path> cannot be read: <cause>”, never listed as empty.

Each paper records an id and title; optional local, github_url, and overleaf_url sources; priority (urgent, high, medium, normal, low); status (backlog, research, in_progress, blocked, writing, review, ready, done, submitted, integrated, paused); an optional deadline, tasks, tags, and target_venue. The target venue (neurips, icml, iclr) is authoritative for the venue bucket when set; otherwise Lem falls back to preamble and title detection.

Inspect and import#

import is mutating: it encodes and rewrites papers.json. Import, context refresh and provenance audit are owned by the CLI and the shared LemCore entry points; the retired standalone utility executables are not required.

Discovery scans the directories LEM_PAPER_ROOTS declares (colon-separated, ~ expanded); when it declares none, the paper-repos directory of the Wisent workspace under ~/Documents/CodingProjects. Each root, its children and their children are read, and a directory is a paper only when one of its .tex files opens a document with \begin{document}; a figures or sections directory is read and passed over by that test, never skipped by its name. A directory or .tex file discovery cannot read, and a path whose file type the file system cannot report (a root, a child, the built PDF beside the manuscript, a .git entry above it), refuses the whole scan with “paper discovery could not read <path>: <cause>”, so an unreadable manuscript is never a paper that silently disappeared from the list, an unexaminable PDF is never a paper not built yet and an unexaminable .git is never a directory outside any checkout. A root that is not there holds no papers.

A discovered paper's GitHub page comes from the origin remote of the Git checkout holding it: the nearest directory at or above it with a .git entry, a directory or the file a worktree keeps. A directory outside any checkout, or a checkout without an origin remote, has none; a Git command that fails inside a checkout refuses with the command and what Git wrote.

A registered paper with a GitHub page but no local source is cloned under ~/.lem/repositories/<owner>/<name> and read like a discovered one. If the clone, or the discovery of the clone, fails, the paper keeps its registered record and Lem writes “lem: <title> keeps its registered record; localizing it failed: <cause>” to stderr; a manuscript that does not build with latexmk is still localized, without a PDF, and Lem writes “lem: <title> is localized without a PDF; building it failed: <cause>”.

The backend's `POST /sync/<provider>` syncs every registered paper and `POST /papers/<id>/sync/<provider>` one of them; each paper answers synced, skipped or failed with a message. A paper id the registry does not hold answers 404 `no paper <id> is registered`, and a provider Lem does not sync with is not a sync route. A paper is skipped only for what it does not have: no local source, no .git entry (a directory, or the file a worktree keeps), no remote the provider serves, or for an Overleaf push no Overleaf project or GitHub repository. A Git command that fails (listing the remotes, reading a remote's URL, the current branch or its upstream, a fetch or a fast-forward) and a .git entry the file system cannot examine fail the paper with that cause, so a checkout Lem could not read is never reported as one with nothing to sync. A fast-forward runs only when the checked-out branch tracks one of the provider's remotes; a detached HEAD or a branch with no upstream is fetched only.

swift run LemCLI papers                 # every paper's metadata
swift run LemCLI paper <id-or-title>    # one paper by exact id or title substring
swift run LemCLI import                 # discover papers and merge them into the registry

Per-paper context#

refresh-context fetches and persists venue-fit context for each authored manuscript. The target-venue bucket is the conference's own listed papers for every corpus year Lem holds, newest year first; Lem keeps no award table of its own. A corpus file is read as Lem ships it: a JSON object whose `papers` each carry `title` and may carry `award`, `description`, `arxiv`, `url`, `venue`, `year`, `provenance`, `introduction_word_count`, `bibliography_reference_count`, `metrics_provenance` and `section_measurements` (`title`, `normalized_title`, `word_count`, `snippet`, `is_empty_stub`); a file that cannot be read or is not such a corpus is named in the context message (`corpus file <path> cannot be read: …` or `… is not a corpus Lem can read: …`) instead of counting as a year without papers, and a local corpus directory that cannot be listed is said on stderr. The similar bucket is Semantic Scholar's default page of its answer to the manuscript's title, addressed as Lem's bundled search-apis.json declares the endpoint, ranked by Pareto fronts of Semantic Scholar's own relevance order and citation count, so no similarity score, citation floor or candidate cut is chosen; a candidate without a citation count is named as unranked. A cached answer that cannot be read is named and searched again. How many papers each bucket keeps is declared by LEM_CONTEXT_VENUE_PAPERS and LEM_CONTEXT_SIMILAR_PAPERS; unset, it keeps every paper with full text and says so on stderr. Use library for external papers independent of a manuscript; use citation-harvest for one manuscript's bound literature corpus.

Paper prompts include every stored context reference, every listed author, every context suggestion and the context-status message, including grouping failures without a target venue. No separate prompt count hides incomplete retrieval.

Explicit refresh-context --force and Desktop Kontekst → Reload context group measured reference and manuscript sections through Lem's configured Brama model. The judgment reads complete observed section bodies and titles, not overlap scores, repetition thresholds or token-length filters. Each normalized heading appears exactly once in the returned partition. Unknown, repeated or omitted identities, empty explanations and malformed JSON refuse the grouping and preserve the response in the diagnostic. Lem does not replace a failed judgment with heuristic matching.

The saved context includes section_grouping (model, groups, member identities, comparison contexts and reasons) or section_grouping_failure. The same judgment selects contexts where a family is a justified comparison from the supplied papers, with no frequency threshold or minimum sample size; an empty contexts list explicitly abstains. The decoder refuses repeated context names or contexts without nonempty measured sections in that family. A bucket-specific suggestion is withheld when another measured bucket is not an approved comparison. CLI refresh reports incomplete and exits unsuccessfully on a grouping failure, even when reference counts were met. Desktop shows the same cause without clipping its message. Read-only loading never calls Brama; legacy section suggestions without a judgment are withheld with a refresh instruction. Scalar metrics, measured word counts and fetched reference bodies are retained.

Real qualification: node tests/context/grouping/journey.mjs in the Lem checkout, with LEM_BIN, LEM_TEST_BUILD_RECEIPT (source_revision and binary_sha256 from the actual build), LEM_TEST_SUCCESS_EXIT, and LEM_CONTEXT_TEST_FIXTURE. The dedicated fixture contains papers.json, home/.lem with real local corpora and Brama/Skarbiec configuration pointers, and expectations.json keyed by lowercase paper UUID. Each expectation declares together and apart arrays of heading identities, comparison_contexts mapping heading identities to approved context-name arrays, and current_words keyed by manuscript mass metric. Include synonymous headings, unrelated headings sharing words, and justified and rejected context comparisons. The journey copies the dedicated fixture into .build/context-grouping, calls the real CLI and real Brama, checks persisted groups, comparison contexts and unchanged measurements, proves cached reads do not rewrite judgments, then damages only its copied Brama configuration and verifies refusal and recovery. Commands, streams, statuses, build identity and stored contexts remain in report.json. This is not a Desktop interaction or a simulated model-response test.

The backend's `GET /papers/<id>/venue-recommendations` recommends venues from the paper's title and the abstract of its main manuscript. The abstract is the manuscript's abstract environment read as plain text by the same LaTeX reader the paper context uses. Each venue in Lem's bundled catalog scores its declared base score plus, for every signal term found in the title and abstract, the term's declared weight times the times it occurs; venues no signal matches are not listed, and each listed venue carries the standing sources the catalog records for it (none when it records none). A bundled catalog that is missing or does not decode stops Lem naming the file, never an empty list of venues. A paper the registry does not hold is a not-found refusal, like the figures route — never an empty answer under a made-up id. A paper without a local manuscript, or whose manuscript has no abstract environment, is recommended from its title alone; a manuscript that is there but cannot be read is refused with “manuscript source <path> cannot be read: <cause>” rather than recommended from an empty abstract. Every paper route is declared once as the path it answers (`papers/{}/reviews/{}`), and a request matches by that declaration's segments and identifiers. `swift test --filter VenueRecommendationsRouteTests` exercises the route on a registry under `.build`.

swift run LemCLI refresh-context [--force] [--limit <n>]

Provenance audit#

LemProvenanceAuditor audits text provenance across a paper's sources; its report lives at ~/.lem/provenance-audit.json by default.

When Git holds no commit details for a manuscript, provenance reads the lines as `git blame` reports them, or line by line when blame fails. The report's coverage error names what stopped attribution: `Provenance coverage failed: source file <path> could not be read: <cause>` for a manuscript the audit cannot open, and `git blame of <path> failed, so no line carries a commit or author: <cause>` appended when the manuscript was read but Git could not blame it — lines without a commit are never reported as a manuscript nobody authored. `swift test --filter FallbackLinesTests` exercises both on a manuscript under `.build`.

Provenance is cached under `~/.lem/provenance-cache/<paper id>.json` with the manuscript's and the transcript roots' signatures. An absent cache is simply recomputed; a cache that is there but cannot be read or decoded is recomputed too and named on stderr — `lem: provenance cache <path> cannot be read, so provenance is recomputed: <cause>` — and a cache that cannot be written is named as `lem: provenance cache <path> could not be written, so the next call recomputes: <cause>`, so a broken cache never hides behind a slow audit. The signatures are measurements: a manuscript, transcript root or `.jsonl` transcript whose size or modification date cannot be read is never keyed as a zero that a later, readable state would match; the cache is not consulted or written for that call and stderr says `lem: provenance cache for <title> not consulted, computing without it: provenance cache signature of <path> could not be measured: <cause>`.

Transcript tree signatures include each file's standardized path as well as its modification time and size, so renaming a transcript invalidates cached evidence that refers to its old location. An enumeration failure stops the measurement with the failed path and filesystem cause instead of accepting a partial tree. `swift test --filter CacheSignatureTests` includes a real filesystem rename case; it does not qualify the desktop audit journey.

Git blame drains stdout and stderr concurrently. Launch and pipe-read failures name the manuscript and cause; a non-successful exit names the termination reason, status and retained stderr, and stdout that is not UTF-8 is refused. These failures flow into the provenance coverage error rather than becoming an empty successful blame.

Each blamed commit's time comes from `git show`; it bounds which transcript evidence may explain the commit's lines. A commit Git cannot show is named in the coverage error as `git could not show when <n> blamed commits were made, so transcript evidence for their lines has no time bound: <commit>: <git's exit and stderr>; …` and is never read as a commit made at time zero. Lines not committed yet carry no commit to show. A blame field Git did not report (a manuscript Git could not blame, a line not yet committed) is absent from the saved line and its readable block rather than an empty author, and a saved coverage error whose count disagrees with the units it lists is refused naming both.

A transcript root, pattern file or transcript the search cannot read is not read as "no match". Each is named on stderr as `lem: transcript read failed: <path>: <cause>`, and the report's coverage error appends `Transcript roots could not be read, so their evidence is missing: <path>: <cause>; …`. The cause is the reader's own: for a root Git cannot list, `git in <root> exited with status <status>: <git's stderr>`. Units left unattributed by such a pass are a search that could not run, not a paper nobody wrote — fix the named root and run the audit again.

Every such cause (an unreadable manuscript, a failed blame, a commit Git cannot show, a transcript root that cannot be read) is kept with the paper's provenance as `causes`. The missing units are counted again after cached Overleaf evidence is applied, and the coverage error still names each cause after that count, even when no unit is left missing. A provenance cache written before `causes` existed cannot be decoded; it is named on stderr and recomputed.

The transcript search looks for each paper's title, LaTeX `\title`, folder, main source and repository URL. A main source that cannot be read stops the transcript index refresh as `the manuscript <path> could not be read for its title: <cause>` instead of searching for that paper without its LaTeX title.

Set LEM_PROVENANCE_DEBUG_TEXT to the passage to trace attribution for matching source lines. Diagnostics include every matching probe, candidate counts and match counts; probes are not shortened to a fixed-size preview. These diagnostics explain the search and are not a replacement for the persisted provenance report.

Set LEM_PROVENANCE_DEBUG_LINES to comma-separated source line numbers to inspect each candidate attribution with its origin, timestamp, event type, tool and matching text (`not recorded` when the transcript did not record one). This line trace includes every candidate alongside raw and valid counts, without a fixed evidence preview, and names any entry of the variable that is not a line number. A candidate is valid when it falls within the line's source time: it was found because the event holds its unit's whole word sequence, so no word count or overlap is asked of it again. A line no unit attributes is completed only from a neighbouring line's transcript message (or the message before or after it) that holds the line's whole word sequence, as written or as plain text; Lem does not widen a line with its neighbours to make it match.

Lem Desktop colours the paper's PDF by provenance and by Pangram's word labels. Each run of text is placed where the PDF holds it exactly once: from each word, Lem searches the shortest run of following words that occurs only once in the PDF. A phrase the paper repeats is left for the line-context pass instead of being pinned to its first copy, and no fragment length decides what is searched for.

When a paper changes or its view is refreshed, Desktop clears the prior authorship data and accepts results only from the active request. A cancelled or superseded request cannot replace the new paper's highlights or error. Failed word-annotation reads show their reason in the authorship banner while retaining the current section provenance; a missing paper is a backend not-found error, not an empty authorship report.

A word's Pangram label is the label Pangram documents for the window that covers it (`AI-Generated`, `AI-Assisted`, `Human Written`), or, for a result without windows, the whole-text verdict Weles writes (`ai_generated`, `ai_assisted`, `human`, the class Pangram gave the largest share of the text). Any other label is shown as unknown with the window's AI likelihood beside it; no likelihood cutoff turns a number into a label. Pangram's words are laid on the paper's words by the same rule the PDF placement uses: from each word Pangram read, the shortest run of following words the paper holds exactly once is placed there, so a phrase the paper repeats or a word the paper does not hold places nothing; no search window or match score decides it. A whole-text signal labels every word from its first placed word up to the next whole-text signal's.

Pangram's verdict (`--pangram-abstracts`, `POST /ai-text/detect`, the Desktop Pangram panel) is asked for through Stado's Weles workload like every browser action: Lem writes a `weles-browser-task` plan for `pangram_analyze_text` with the text as `params.pangram_text` (and `pangram_require_account`, `pangram_analyze_url` when given), runs `stado workload run weles-browser-task --plan <file> --json` on an eligible Weles host or the one `LEM_WELES_TARGET` names, and reads the verdict from the run's `pangram` output. Lem keeps no Weles checkout, runs no trajectory itself and reads no recording directory; a result names `run_id`, `host`, `plan_path`, the stado `command`, the run's retained streams and `exit_code`. A refusal is Stado's own sentence; a run that ended failed is `Weles run <id> on <host> ended failed (exit <n>)` with the ban signal's signal and reason appended when the run left one (`(ban signal <file> could not be read by Weles: <error>)` when Weles could not read it); a run that left no `pangram` output is `Weles run <id> has no readable pangram output: the run left no such output (it left: …)`. A request that names its paper (`paper_id`; the audit and the Desktop panel always do) has its verdict and text kept under `~/.lem/pangram-runs/<paper id>/<run id>/` (`pangram_result.json`, `text.txt`, `ban_signal.json`), reported as `store_path`; a verdict that cannot be kept there fails naming the paper and the cause.

The paper's word annotations (`GET /papers/<id>/word-annotations`, the Desktop authorship colouring) read those kept runs: each run is one Pangram signal over its text, per window when the result carries `windows` and over the whole text otherwise. A kept run whose `pangram_result.json` or `text.txt` cannot be read fails the read as `Pangram run record <path> cannot be read: <cause>`; a paper with no kept run has no Pangram signal, which is not an error.

swift run LemCLI audit [--paper <id-or-title>] [--no-report] [--force] [--transcripts] [--resolve-overleaf-missing]
swift run LemCLI provenance   # the latest provenance report, if present