Project structure

You rarely need to read the framework to use it, but knowing where things live helps when a run surprises you. core/ is the runner every channel shares; modeling/, evaluation/ and data/ are the channels; output/ is where runs land. The tree below is the map.

LLMClick/
├── main.py                       # `uvicorn main:app` from the repository root; the app itself is core/api/app.py
├── configs/                      # one YAML = one custom model; a directory per family (llm/, embedding/)
├── samples/                      # tiny rows per recipe so the shipped configs run offline
├── core/                         # the framework, shared by every channel
│   ├── stage.py                  #   Stage: one fingerprinted, rerunnable unit of work
│   ├── pipeline.py               #   Pipeline: runs a recipe's stages; build()/load() turn a config into one
│   ├── progress.py               #   Progress, BarProgress (terminal), StateProgress (API jobs)
│   ├── registry.py               #   Registry, RECIPES, CHANNELS
│   ├── config/                   #   schema.py (BaseConfig), experiment.py (directories, manifest)
│   ├── cli.py                    #   llmclick run|validate|recipes|serve
│   ├── settings.py               #   process settings from the environment and environment/.env (LLMCLICK_ENV)
│   ├── api/                      #   app.py; routers: config_channel, job_channel, system_channel; store.py, guard.py
│   └── utils/                    #   files (hash, lock), device
├── modeling/                     # Modeling channel: config -> trained, validated model
│   ├── config.py                 #   ModelingConfig: tracker, validation bounds
│   ├── stages.py                 #   TrainStage, ValidateStage (templates every recipe fills in)
│   ├── pipeline.py               #   ModelingPipeline: training always brings validation
│   ├── tracker.py                #   wandb replay
│   ├── tuning/                   #   in-process fine-tuning shared by the families below
│   │   ├── backbone.py           #     Backbone: base of every model catalog entry
│   │   ├── method.py             #     TrainingMethod: base of every training method
│   │   ├── loop.py               #     the one training loop (accumulation, schedule, resume)
│   │   ├── stages.py             #     data, train, validate
│   │   ├── sources.py            #     row sources: local_jsonl, huggingface
│   │   └── config.py pipeline.py #     TuningConfig, TuningPipeline
│   ├── llm/                      #   family: generative language models
│   │   ├── models/               #     Model Catalog: base.py (LLMBackbone), transformer.py, hybrid.py
│   │   │   └── heads/            #       decision heads on top of a backbone: base.py (DecisionHead), readout, pointer
│   │   ├── methods/              #     Training Method: base.py (LLMMethod), sft, instruction, dpo, grpo
│   │   │   └── decision/         #       base.py (DecisionMethod: rows, metrics, calibration), sft, cispo
│   │   └── config.py pipeline.py #     one config class and one registered recipe per method
│   ├── embedding/                #   family: text embedding models
│   │   ├── models/               #     Model Catalog: base.py (EmbeddingBackbone), bi_encoder.py
│   │   ├── methods/              #     Training Method: contrastive.py
│   │   └── config.py pipeline.py
│   ├── tts/                      #   family: text-to-speech (tts_piper)
│   │   ├── piper.py              #     the Piper program (GPL) as a separate process: speak, fit, export
│   │   ├── audio.py asr.py       #     WAV measurements; Whisper transcripts and word error rate
│   │   └── config.py stages.py pipeline.py
│   └── wakeword/                 #   family: wake word detection (wakeword)
│       ├── features.py           #     openWakeWord's frozen front end on onnxruntime: audio -> embeddings every 80 ms
│       ├── detector.py           #     the DNN head: trained with torch, exported and scored in ONNX
│       └── config.py stages.py pipeline.py
├── data/                         # Data channel: seeds -> prompts -> evolve -> respond -> verify -> select (data_synthetic)
│   ├── teachers.py               #   anthropic, openai, ollama, local behind one complete()
│   ├── config.py rows.py ...     #   one module per stage; rows.py: natural keys, atomic files, per-item caches
│   └── speech/                   #   data_speech: synthesize, verify, select after the shared seeds and prompts
├── evaluation/                   # Evaluation channel: config -> scores of a trained model, report.json
│   ├── source.py                 #   SourceExperiment: an experiment's config read back, its checkpoint located
│   ├── stages.py                 #   rows, score, report (evaluation_custom)
│   ├── benchmark.py              #   presets, one score stage per benchmark over lm-evaluation-harness (evaluation_benchmark)
│   ├── compare.py                #   runs, verdicts against the base, polygon SVG, Markdown summary (evaluation_compare)
│   ├── decision.py               #   jevbench source, decide without and with reasoning, ECE, tiers, latency (evaluation_decision)
│   ├── embedding.py              #   mteb encoder wrapper, one stage per MTEB task (evaluation_embedding)
│   ├── wakeword.py               #   clips, score traces, miss rate and false accepts per hour over thresholds (evaluation_wakeword)
│   └── config.py pipeline.py     #   (README: design and milestones)
├── tests/core/  tests/modeling/
├── docs/                         # api-spec.md, screen-spec.md
├── environment/                  # Dockerfile, docker-compose.yml, .env.sample
│   └── piper/                    #   the Piper program's own locked environment (GPL), built by setup.sh
└── output/
    ├── _stages/<stage>-<fingerprint>/   # what each stage built; shared by every experiment with the same inputs
    └── <name>-<hash>/                   # one experiment: config.yaml, manifest.json, pipeline.log, links to its stages

evaluation/ scores what modeling/ built: evaluation_custom measures any checkpoint of an experiment (validate, train, base, or a directory) on any rows with the metrics of the recipe that trained it; evaluation_benchmark scores it on public benchmarks through lm-evaluation-harness (uv sync --extra eval), one cached stage per benchmark, with presets for the small-model panel and Korean; evaluation_compare puts several checkpoints, the base model among them, through the same benchmarks and rows and reports the differences with their intervals, a polygon chart and a Markdown summary; evaluation_decision measures a decision model on Jev rows or the JevBench tiers without and with reasoning (accuracy, NLL, ECE, per tier against chance, latency); evaluation_embedding runs an embedding model on MTEB tasks; evaluation_wakeword measures a wake word detector's miss rate under noise and false accepts per hour of audio without the name. All write report.json with the overlap between the evaluation items and the experiment's training rows (exact matches and n-gram near-duplicates) next to the scores, send the report to wandb with tracker.enabled, and are listed by GET /api/evaluations (evaluation/README.md has the details). data/ produces rows: data_synthetic makes them from seeds through a teacher model (Anthropic, OpenAI-format, Ollama or a local checkpoint), evolves and verifies them, removes duplicates and leaks, and writes a recipe's rows with a manifest; every stage is cached by its inputs and safe to rerun, so a crash or a spent budget resumes where it stopped (data/README.md). Loading open datasets (data_open) is designed there and not built. A channel is a package that registers recipes; it joins by adding its name to CHANNELS in core/registry.py and reuses Stage, Pipeline, Progress and Experiment as they are.

Source: README.md. This page is generated; edit the source and run npm run sync.