A native home for authored papers and external literature.

What Lem owns#

Lem is the research-paper operations package. It keeps a registry of manuscripts being authored and a separate global library of external papers being read. Per-manuscript operations scrape venue-fit context, audit text provenance, check and harvest citations, and generate publication figures.

The canonical engine is the LemCore Swift library. The command-line interface, loopback backend, MCP server, and native macOS app all use the same manuscript and literature stores. The native interface lives separately in lem-desktop.

Venue recommendations (the backend's papers/<id>/venue-recommendations, shown in the app's venue panel) only recommend a venue a published ranking lists — the CORE/ICORE conference ranking (portal.core.edu.au) or Google Scholar Metrics (scholar.google.com) — so no predatory venue is recommended. A venue whose signals the paper matches but that no such ranking lists is answered apart under unranked_venues with the signals it matched.

Executables and targets#

The Swift package Lem (macOS) declares these products:

• LemCLI — the command-line interface; prints JSON to stdout.
• LemMCP — a read-only stdio MCP server for agents and integrations.
• LemBackend — a loopback HTTP server for the desktop app and integrations.
• LemCore — the shared library all of the above depend on.
• Registry discovery/import, paper-context refresh and provenance audit are CLI operations: `lem-cli import`, `lem-cli refresh-context` and `lem-cli audit`, not separate utility executables.

Build and run#

Build the whole package with swift build, or one product with swift build --target <name>.

swift build
swift run LemCLI papers    # authored manuscripts
swift run LemCLI library   # external literature

Services and credentials#

Model calls route through Brama, the LLM gateway, and --model selects a route alias. Credentials come from Skarbiec: for each spending invocation Lem signs an Ed25519 workload proof and redeems a one-time, thirty-second bearer — no code path in the package reads a credential from the process environment. A paper agent must read its conversation history before starting the agent request: an unreadable history or conversation index fails the run with the actual cause, rather than being replaced with an empty conversation. Plain and streaming runs retain that failure; streaming also records an error event. If the failure itself cannot be persisted, backend diagnostics name the run, failed persistence operation and underlying cause.

Lem never dials Brama or Skarbiec itself. With no `url` in `~/.lem/brama.json`, every model request (reading, note judgement, section grouping, figure text) is handed to `stado capabilities call inference --consumer lem --json`, which asks the first alternative in Stado's selected order that answers it — Brama, or a statement recorded for that exact request — so Lem names no model alternative and reads no Brama credential, and an alternative Stado gains later is used with no change. When none answers, the call is refused with `no inference alternative answered `stado capabilities call inference --consumer lem --json` (exit 75): …` followed by Stado's sentence, which names each alternative's cause and the `stado capabilities attestation record inference --consumer lem <sha> --checked …` command that answers the kept request; without Stado, with `stado … could not start: …; install Stado, or name the gateway in ~/.lem/brama.json`. Images are drawn on Brama's image routes, which Stado's alternatives do not carry, so with no gateway named a raster figure is refused naming that file. Credentials come from what `stado capabilities resolve secrets --consumer lem` hands out: Skarbiec, or the owner-only file store declared in its place. The real journey is `tests/stado/CapabilityResolutionTests.swift`.

The local Lem backend binds a system-assigned loopback port unless `LEM_BACKEND_PORT` declares one, then publishes the bound URL in `~/.lem/backend-address`. Clients reread the announcement per request. Address-file read failures include the path and underlying cause; malformed URL content is refused before sending a request. An occupied declared port remains a listener error, not a fallback to another port.

The backend reads a request until its head and as many body bytes as `Content-Length` names have arrived, however many reads that takes; a head without `Content-Length` has no body. A body framed any other way is refused with 400 before any route runs, never read as ending at the head: `a request body sent with Transfer-Encoding <coding> is not read; send Content-Length`, `Content-Length "<value>" is not a byte count`, or `Content-Length is declared twice, as <n> and <m>; send it once`. A connection that closes before the declared body arrives is answered 400 `incomplete_request`.

The account the backend reports (`GET /account`, Desktop Settings) is Lem's own `~/.lem/account.json`, or the file LEM_ACCOUNT_PATH names: `user`, `organization` and `billing`, each optional. No file is a local, signed-out Lem; a file that cannot be read or decoded is reported with auth_state `unavailable` and the message `<path> cannot be read as Lem's account: <cause>`, never as no account. The account's contents come from that file only, never from another program's files or from environment variables. Saving an organization without both an id and a name is refused with 400 `the organization needs both an id and a name to be saved` instead of being dropped.

Desktop backend startup preserves the actual process-launch error in its failure display. Existing-backend lookup drains both pgrep output streams concurrently: no matching process is distinct from a failed lookup, invalid output, or a successful lookup without a process identifier. Lookup and log-opening failures are reported with their underlying cause; startup does not treat those diagnostics as proof that the backend is available.

When the desktop cannot read papers or literature from the backend, a failed local-store read remains visible beside the backend failure. The shared failure reporter receives both causes as well. A successful local literature read still reports the backend failure and identifies that the screen is using the local library; it does not erase the service failure.

The Oko agenda — the organization's `work_items` and `work_item_checklist_items`, which every catalog read (`lem-cli papers`, `GET /papers`, Lem Desktop) lays over the registry, `PATCH /papers/<id>` (Lem Desktop's paper edits) and `lem-cli roadmap --push` write back — is reached the one way a product reaches the shared Oko database: the installed `oko` program as the signed-in member, through Stado. Lem runs `oko database read --table work_items --query select=*&external_id=like.________-____-____-____-____________` (and the checklist table) to load it, and `oko database request --method PATCH|DELETE|POST --path <table>?<filter> --prefer return=minimal --body` to write, the body on standard input; row security scopes what Lem may touch. An agenda item's external id is its Lem id, a UUID, so the read selects ids of a UUID's shape (`_` is any one character to PostgREST's `like`); the rows other programs keep in the same table under ids of their own form (Oko's roadmap pushes `roadmap:<product>:<slug>`) are not agenda items and are not read. Lem holds no database address, no key and no organization id of its own: the rows' `org_id` is the one oko answers with. A machine without `oko` on PATH keeps Lem's own catalog and prints `lem: the Oko agenda is not read on this machine: oko database read … cannot run: no executable named oko is on PATH; install Oko or set LEM_OKO_AGENDA_PATH to a local agenda file`; any other failure — a refused command (`oko database read --table work_items --query … exited <status>: <what oko said>`), an answer that is not an array of rows, or a row missing a column (`Oko work_items row <id> cannot be read: external_id is missing or not text`) — fails the catalog load, the update or the roadmap push by name instead of reading the agenda as empty.

A local agenda file (LEM_OKO_AGENDA_PATH) holds `items`, each written as the work item row it stands for: `id`, `title`, `source`, `paperTitle`, `status`, `priority`, `githubURL`, `overleafURL` and `checklist` are required, because Oko's work items table declares those columns `not null`; `finishOrder`, `includeInPaperQueue` and `deadline` may be left out. Each checklist item carries `id`, `title` and `isDone`; `dueDate`, `createdAt`, `completedAt` and `source` may be left out. A missing required field, or a deadline or due date that is not an ISO 8601 time, refuses the file with the decoder's sentence naming the field, instead of reading the item as a backlog item of medium priority.

Credential routing, Git pushes, citation insertion, Pangram and Overleaf trajectories, and figure evaluation drain child stdout and stderr concurrently. Child completion, not a Lem polling deadline, ends the operation; complete output and process errors remain available to CLI and desktop consumers. An output-read error is reported as failure rather than substituted with empty success. Agent event appends also propagate file-close failures: a successful write followed by a failed close is not a successful append, and simultaneous write/close failures retain the event-file path and both causes. With LEM_TEST_SOURCE_REVISION set to the full clean source commit, `swift test --filter EventPersistenceTests` exercises the public event store against an isolated real filesystem and retains refusal, persisted bytes and recovery evidence under `.build/agent-events-*`. This store journey does not force a close failure or qualify native agent error presentation.

The paper agent runs through `oko agent <kind> <prompt> --json`. The kind is LEM_AGENT_KIND, else Oko's own agent kind setting (oko.agent.kind); with neither set the run refuses as `no agent kind is configured: set LEM_AGENT_KIND, or Oko's agent kind …`, and Lem picks no harness itself. Lem passes the kind as configured and Oko accepts or refuses it; the run records the kind as its agent. The answer is the agent's text events; a run with none fails with its last error, or with `the agent wrote no assistant text` followed by any harness output Oko could not read, which the run screen also shows as status lines.

A paper review (`POST /papers/<id>/reviews`) asks the paper agent for exactly one JSON object: a summary (readiness, text and counts by severity) and findings (severity, type, optional section, lines, quote and suggestion, problem and confidence). An answer that is anything else fails the review with 502 `Reviewer agent did not answer one review JSON object (<decoder's cause>): <answer>`; Lem does not search the answer for an object inside it. A reviewer reply in a discussion is given the finding with only the fields it records, or is told the discussion is about the review as a whole.

A compute job (`POST /papers/<id>/compute/jobs`) uploads a snapshot of the paper repository: exactly the files Git's index holds (`git ls-files --cached`), so an untracked file — a key, an environment file — never leaves the machine until it is added, and Lem keeps no list of file names it guesses are secret. Links, and paths that run through one, are left out. Every file carries one fixed modification time, so the archive's SHA-256 is its content's; a directory that is not a repository fails with Git's own refusal, and a failed tool names itself, its exit status and its stderr.

A streaming paper agent measures the paper's LaTeX sources (modification time and size of every .tex, .bib, .sty and .cls file) and its PDF before and after the run, and rebuilds the PDF with latexmk when the sources changed; a PDF the filesystem reports as not existing is not built yet and has no signature. A PDF whose metadata cannot be read for any other reason (a folder on its path that cannot be searched, a volume that is gone) is refused as `paper source <path> cannot be read: <cause>` rather than taken for an unbuilt one. A source the backend cannot measure is named in a run status event — “Paper sources could not be measured before the run, so changes will not be detected: paper source <path> cannot be read: <cause>” — and the comparison is skipped for that run; after the run the same refusal fails the run with its cause. Nothing signs an unreadable file as empty or as zero. `swift test --filter SourceSignatureTests` exercises the signatures on a paper directory under `.build`.

Source-tree enumeration failures name the failed directory rather than accepting a partial signature. PDF rebuilds drain stdout and stderr concurrently before interpreting the child's ending; a failed process names the executable, termination reason, exit status and both streams. An absent PATH is refused before starting a build tool. SourceSignatureTests also checks that removing the paper directory produces a named failure instead of an empty signature; it does not qualify a streaming agent or native UI journey.

Workspace paths#

Browser work — the Overleaf version-history scan and the Overleaf push to GitHub — runs through Stado's Weles workload: Lem writes a `weles-browser-task` plan under `~/.lem/weles-plans/` (the action, the page, the objective and the action's own `params`) and runs `stado workload run weles-browser-task --plan <file> --json`; Stado places the run on an eligible Weles host, or on the one `LEM_WELES_TARGET` names, checks the action against that host's allowlist, routes the sign-in through Skarbiec and waits for the run to end. Lem keeps no Weles checkout path, queue table, service key or environment file; a refusal names the stado command, its exit status and what it said, or the run's id, host and outcome with the run's retained standard error.

Lem reads Stado's `--json` report field by field: `host`, `action`, `run_id`, `ok`, `exit_code` (null for a run a signal ended), the trajectory's `result`, and the run's named `outputs`. Missing required fields are refused by name. `output_errors` is optional: Weles includes it when recording files or directories cannot be read, keyed by relative recording paths. The version-history scan reads `outputs.overleaf_version_history_summary`, not `result`. A returned output is read directly even when other recordings failed. If the requested output is absent, Lem names the run and output and includes all observed recording-read causes in sorted path order; it does not guess which output a failed path represents from a shared name prefix. Without recorded errors, the refusal lists the output names the run actually returned. A failed run names its host, ending, plan and retained standard error.

The scan then reads the summary once. A summary that is missing, not JSON or not a JSON object is that target's failure, `Weles Overleaf scan summary <path> cannot be read: <cause>` (the file error, `not JSON: <parse error>` or `the scan summary is JSON but not an object`), listed under the target in the scan result and the audit report; it is never read as a scan that found no evidence, so `no exact UI match found` means the summary was read and held no probe for the phrase.

If no target supplies matching evidence and any target fails, the scan returns `failed` with the target diagnostics and does not cache `noMatch`. This includes workload refusals, retained failure markers, unreadable summaries and unresolved project identifiers. A known project identifier alone cannot turn a failed scan into a completed search. `noMatch` is cached only after all attempted targets complete without such failures; a later target that actually supplies matching evidence can still complete the scan.

`swift test --filter ScanOutcomeJourneyTests` exercises the public scan API with real Stado/Weles dependencies. Run it with a fresh isolated HOME beneath Lem's `.build` and `LEM_SCAN_OUTCOME_EXPECTATION` naming a JSON document with `revision` (the exact clean Lem source revision), `matching` and `refused`. Each scenario contains a serialized Lem `paper` with an Overleaf URL, a `phrase`, and a `state` (`completed` or `failed` respectively); `refused.cause` is the expected actual refusal. Use caller-owned projects: one containing the phrase in its history and one the workload genuinely cannot read. The journey assigns fresh local paper identities, retains requests, outcomes, source commands and cache readback under `.build/scan-outcome-*`, requires matching evidence to persist as completed, and requires the refused scan to leave no no-match cache. Missing inputs or source mismatch fail qualification; this journey does not qualify CLI rendering or native UI.

The Semantic Scholar key comes from Skarbiec only, never from a file another program writes or from the process environment; context refresh and citation-harvest discovery use the same grant. When Skarbiec does not hand it over, Lem says why once on stderr (`lem: Semantic Scholar runs unauthenticated: <cause>`, the cause being that the Brama configuration cannot be loaded, Skarbiec's own refusal (including that neither skarbiec.url nor Stado's directory names a broker), or an empty key) and calls Semantic Scholar unauthenticated, which works and is rate-limited sooner.

First use: bring your research#

The CLI walkthrough (`lem-cli onboarding`, or `--reset` to replay it) and Lem Desktop's first-run overlay both lead to the same LemCore importer. The importer accepts an unpacked LaTeX project directory, a UTF-8 BibTeX file whose every entry contains a title and an arXiv id in `eprint` or `url`, or one PDF accompanied by its arXiv id or URL.

A project is adopted in place and must contain a `.tex` document with `\\begin{document}` at the root or within two directory levels. BibTeX entries are resolved through arXiv into the literature library; the exact raw entries, source path, source digest, and keys remain attached as import provenance. A supplied PDF is copied to the literature asset store. ZIP, tar, gzip, bzip2, xz, zstd, 7z, and rar archives are refused explicitly.

Lem validates the complete selected input before committing a registry. Existing source identities remain unchanged, so replaying onboarding or importing the same input reports `unchanged` rather than adding a duplicate. A refusal leaves the prior registries and registered assets intact.

The desktop first-use overlay keeps the caught cause when loading the journey, saving a step, recording completion, or delivering events fails. Settings replay also preserves its failure cause. An imported paper remains imported if recording completion fails: a rejected completion verdict, a thrown recording error, and undelivered events after recorded completion are distinct outcomes, not instructions to import the same research again.

lem-cli import --project /path/to/unpacked-project
lem-cli import --bibtex /path/to/library.bib
lem-cli import --pdf /path/to/paper.pdf --arxiv 2404.02905