API

MkDocs builds this page from the docstrings. Fit math, abliteration, and the quality gates have the longer notes. Other public functions have one line.

Fit

Weights + KV size against a card budget.

The check is weights + KV, not the whole CUDA context. Headroom (default 3 GB on a 16 or 24 GB card) is what remains for compute buffers and the desktop. A quant that fails this on-GPU check can still be kept when an offload split fits in VRAM and RAM. See voxprint_forge.placement.

Arch dataclass

Layer counts and head sizes used to estimate KV bytes.

kv_bytes(ctx, cache)

KV cache bytes for this context and cache type.

Full-attention layers use the whole context. Sliding-window layers use min(ctx, swa_window). q8_0 is 34 bytes per 32 values and q4_0 is 18. f16 is 2 bytes per value. The fit gate compares weights plus this estimate to the card budget minus headroom.

gib(num_bytes)

Convert a byte count to gibibytes.

budget_bytes(budget_gb, headroom_gb)

Maximum weights plus KV on the card.

Headroom stays free. On a 16 or 24 GB tier the plan uses 3 GB, so weights and KV together must fit in 13 or 21 GiB.

fits_budget(weights_bytes, kv_bytes, *, budget_gb, headroom_gb)

True when weights plus KV fit on the card after headroom.

The budget is (budget_gb - headroom_gb) GiB. Headroom (3 GB on the 16 and 24 GB tiers) stays free for compute buffers and the desktop. This is the on-GPU fit gate. A quant that fails it can still be kept when an offload split fits.

min_vram_gb(weights_plus_kv_gib, headroom_gb)

Smallest suite card (16 or 24) that holds weights plus KV and headroom.

Anything above 24 GB is rounded up to the next whole gigabyte. The value is what the registry records as min_vram_gb for the tier.

quant_rank(name)

Higher means a better quant when several fit the same tier.

quant_bpw(name)

Bits per weight used by the preflight size estimate.

estimate_weights_bytes(params_b, quant)

Parameter count times bits per weight.

params_b is billions of parameters. Preflight uses this before a file exists. The fit stage later replaces it with the real GGUF size.

VRAM and RAM splits tried by the fit stage before a smaller quant is accepted.

Order is the preference order. gpu keeps weights and KV on the card. The later splits move something into RAM: KV (--no-kv-offload), MoE experts (--n-cpu-moe), a diffusion text encoder, or sequential loading so the peak is one component rather than the sum. An offload that fits is preferred to a lower quant whose only advantage is that it passed the on-GPU gate.

Split dataclass

One VRAM/RAM placement: name, device bytes, RAM bytes, and llama-server flags.

llm_splits(weights, kv, *, moe_layers=0, expert_fraction=0.0)

GPU, KV-in-RAM, and (for MoE) --n-cpu-moe.

diffusion_splits(transformer, encoder, vae=0)

Full load, encoder in RAM, and sequential load (peak is the larger resident set).

options_for(weights, kv, *, moe_layers=0, expert_fraction=0.0, encoder_gib=0.0)

Preference order for one quant: GPU, then KV in RAM, MoE experts, encoder, sequential.

Diffusion splits do not include the KV cache. The cache stays on the card with the transformer, so those options gain kv bytes of VRAM here.

split_fits(split, *, budget_gb, headroom_gb, ram_bytes)

True when device bytes fit the card and RAM bytes fit the machine.

The card limit is budget_bytes, which already subtracts headroom. This is the formula the fit stage uses before it measures a split.

first_fitting(splits, *, budget_gb, headroom_gb, ram_bytes)

Prefer gpu. The next fitting offload wins over trying a smaller quant.

Pick a quant that fits a card budget, then parse llama-server buffer sizes.

Candidate dataclass

One quant file: name, weight bytes, path, and whether it was reused or quantized.

BufferReport dataclass

CUDA device totals. devices is one entry per CUDA* buffer, excluding *_Host.

Measurement dataclass

One context's measured weights, KV, and whether it fit.

fit_contexts(tier, plan)

16 GB is gated at the shortest context. 24 GB must fit every context.

fitting_candidates(candidates, arch, contexts, *, budget_gb, headroom_gb, cache)

Quants whose weights plus KV fit the card, best rank first.

This is the on-GPU shortlist. The fit stage still tries an offload of a higher quant before it accepts a lower quant from this list.

choose_cache(candidates, arch, contexts, *, budget_gb, headroom_gb)

Prefer q8_0 KV. Use q4_0 only when no quant fits with q8_0.

parse_buffers(log)

Sum CUDA* model and KV buffers. CPU_Mapped and *_Host are not the card.

placement_bytes(log)

CUDA model bytes, CUDA KV bytes, and host bytes that an offload must hold.

CPU_Mapped is the weight file mapping, not an extra allocation, so it is left out. CPU and *_Host KV buffers, and non-mapped CPU model buffers, count.

measurement_fits(weights_bytes, kv_bytes, *, budget_gb, headroom_gb)

True when measured CUDA weights plus KV fit the card budget.

Host and *_Host buffers are not the card. The gate is the same weights-plus-KV test as fits_budget, including headroom. nvidia-smi usage is recorded separately and is not the gate.

tier_min_vram(weights_bytes, kv_bytes, headroom_gb)

Smallest 16 or 24 GB card that holds this peak plus headroom.

Abliteration

Difference-of-means refusal-direction ablation.

One direction is the normalised gap between mean residual activations on harmful prompts and harmless prompts, taken at the layer where that gap is largest. Writing matrices (attention output and MLP down projection) are orthogonalised against it: W := W - r r^T W.

On an 80 GB GPU a 24-33B bf16 model can stay resident. The layer path still moves one decoder block at a time so a 24 GB card can run the same code. Heretic is not imported or invoked.

AblationResult dataclass

Layer index, direction norm, how many matrices changed, and resident or layer mode.

orthogonalize_weight(weight, direction)

Project the refusal direction out of a [out, in] matrix.

The update is W := W - r r^T W with r a unit vector. A matrix whose output size does not match the direction is returned unchanged.

difference_of_means(harmful, harmless)

Unit refusal direction from mean residuals.

harmful and harmless are [n, hidden] last-token residuals. The direction is their mean gap, divided by its length. A zero gap is an error: there is nothing to remove.

select_direction(harmful_by_layer, harmless_by_layer)

Layer whose mean harmful-minus-harmless gap is longest, and that unit direction.

Each map is layer index to [n, hidden] residuals. The chosen layer is the largest gap, which is the direction the orthogonalisation step removes.

decoder_layers(model)

Decoder blocks from model.layers or model.model.layers.

apply_orthogonalization(model, direction)

Remove the refusal direction from writing matrices and return how many changed.

Writing matrices are the leaves in WRITE_LEAVES (attention output and MLP down projection) whose output size matches the direction. Each is updated as W := W - r r^T W. Zero matches is an error.

collect_last_token(model, encode, prompts, *, device, mode)

Last-token residual at every decoder layer for these prompts.

abliterate_module(model, encode, harmful, harmless, *, device=None, mode='layer')

Pick the refusal direction and orthogonalise this module. Returns the result record.

gpu_can_hold(weight_bytes, free_vram_bytes, margin_bytes=6 * 1024 ** 3)

True when free VRAM covers the weights plus the margin.

choose_mode(weight_bytes, free_vram_bytes)

resident when the GPU can hold the weights, otherwise layer.

save_ablated(model, tokenizer, dest)

Write the ablated model and tokenizer into dest.

load_hf(source, *, offload_dir, dtype=torch.bfloat16)

Load a local causal LM and tokenizer, optionally offloading layers to disk.

encode_with_tokenizer(tokenizer)

Return a prompt encoder that uses this tokenizer.

abliterate_directory(source, dest, harmful, harmless, *, weight_bytes, free_vram_bytes, offload_dir=None)

Abliterate a local model directory and save the result.

resident keeps the model on the GPU when weight_bytes fits in free VRAM with margin. layer moves one decoder block at a time so a smaller card can run the same orthogonalisation. The returned record is the layer, the direction norm, the matrix count, and the mode.

Quality record

Light refusal, perplexity, and prose checks.

These numbers are the quality record on the model card. They do not choose the quant. The fit gate does that, and an offload that fits is preferred to a smaller quant. The corpora are original Apache-2.0 text.

data_dir()

Directory of the original eval corpora shipped with Forge.

read_lines(path, limit)

Non-comment lines from path, capped at limit.

is_refusal(text)

True when the sample contains a refusal phrase.

The phrases are a fixed list in English and Russian (for example "i can't", "i cannot", "не могу"). This is a string check on the model sample, not a classifier. A tier's refusal rate is the fraction of prompts this marks.

parse_ppl(log)

Read the final perplexity estimate from a llama-perplexity log.

refusal_rate(runner, model, prompts, *, cache)

Fraction of the prompts that ran whose sample is a refusal.

Each prompt is one llama-cli sample. After four prompts, stop when the Wilson 95% interval lies entirely above or entirely below one half. The rate divides by the prompts that ran, or is 0 when the list is empty. The fit stage does not use this number; it is the eval quality record on the card.

perplexity(runner, model, corpus, *, chunks, cache)

Perplexity of the corpus from llama-perplexity.

prose_sample(runner, model, prompt, *, cache)

One prose sample for the card.

evaluate_model(runner, model, *, cache, refusal_limit, ppl_chunks, prose_samples)

Perplexity, EN and RU refusal rates, and prose samples for one GGUF.

Refusal prompts and the perplexity corpus are the original text under data/. The result is recorded on the tier. It does not pick the quant: the fit gate does that, and an offload that fits is preferred to a smaller quant.

Model manifest

Shared voxprint-model.json manifest (schema voxprint-model/1).

The schema is provisional until VOXPrint Components reviews its MDL change. The Plotweaver run export in manifest.py is a different file and stays.

ManifestInput dataclass

Fields known before the published file list is hashed.

RemoteFile dataclass

One file listed on a Hub repo.

RemoteRepo dataclass

One published voxprint-* repo the backfill can turn into a manifest.

Listing dataclass

Repos the backfill can write, and repos it skipped.

schema_path()

Path of schemas/voxprint-model.schema.json in a repo checkout.

gb_to_gib(gb)

Convert a decimal GB card budget to the GiB value this manifest records.

MDL treats a 24 GB card as about 23.5 GiB. The value is gb * 1000 / 1024, rounded to the nearest 0.5, so 24 becomes 23.5 and 16 becomes 15.5.

default_cc(engine)

Minimum CUDA compute capability for one runtime engine.

build_document(*, model_id, display_name, app_ids, task, license, files, mirrors, min_vram_gb, recommended_vram_gb, gpu_min_cc, engine, version_req, base_model, card_url, flags=(), on_demand=True, forge_version='', tag='', date='', onnx_prebuilt=None, post_install=None, prebuilt=None)

Build one voxprint-model/1 document. VRAM arguments are decimal GB.

File rows may carry onnx_prebuilt. A model-level onnx_prebuilt or post_install (the owning app's CLI only) is copied when given.

write_manifest(directory, document)

Write voxprint-model.json into directory and return its path.

check_document(document)

Raise StageError when document is not a voxprint-model/1 object.

hash_tree(root)

Hash files under root. voxprint-model.json is omitted.

rows_from_artifacts(artifacts)

File rows for a manifest. The manifest artifact itself is omitted.

manifest_artifact(path)

Artifact for an already written voxprint-model.json.

refresh_manifest(artifacts, path)

Replace the manifest artifact after its bytes change.

hub_mirrors(*, hf_repo='', kaggle_ref='', hf_revision='main', kaggle_revision='private')

Hugging Face and Kaggle mirror entries. Empty refs are left out.

drive_mirror(model_id, date)

Drive mirror for Voxprint/model-forge/<date>/<model>.

card_url_for(mirrors)

Hub card URL for the mirror that holds the descriptor.

An empty string when there is no hf mirror. A marked ours mirror or the Mitroshenkov87 mirror wins over an upstream author listed first. One hf mirror is that repo: publish writes the card there.

retarget_mirrors(path, mirrors, *, card_url=None)

Rewrite mirrors on an existing manifest and refresh its digest.

The default card URL is our repo, the mirror that holds the descriptor.

infer_task(repo_id, paths)

Pick a task string from a repo id and its file paths.

infer_app_ids(task)

Suite app ids for a task. Text models serve Plotweaver and Audiobook Builder.

infer_manifest(slug, licence, *, title='')

Best-effort manifest fields when a publish step has only a slug.

document_from_repo(repo, *, date='')

Build a manifest for one already published repo. Nothing is uploaded.

backfill(repos, dest)

Write voxprint-model.json for each repo under dest. Does not upload.

list_published_repos(user=HF_AUTHOR)

Read user/voxprint-* model repos from the Hub. Does not upload.

The index dataset is not a model repo and is skipped. A repo with no licence tag, or a file that has no sha256, is returned in skipped.

find_manifests(root)

Every voxprint-model.json under root.

write_index(paths, dest, *, revisions=None, commits=None, serial=None, head=None, generated_at=None)

Write index.json and return its rows. This does not upload.

The file is the signed-index object {head, serial, generated_at, models}. serial defaults to the published serial plus one (at least 2), and head to the dataset commit this index replaces.

revisions pins the descriptor commit for a repo. Every other repo is resolved with commits (the Hugging Face commit of voxprint-model.json by default). The mirror's revision is not a fallback.

index_document(rows, *, serial, head, generated_at=None)

The signed-index object. generated_at defaults to now, in UTC.

next_serial(published)

One more than the published serial, never below the bundled pin plus one.

check_index(document)

Raise StageError unless document is a signable index object.

per_file_sources(document, upstream_paths, *, policy='strict-priority')

Return a copy of document with mirror_policy and per-file sources.

Each file gets our forge mirror as its source. A path in upstream_paths (same bytes upstream) lists the author's mirror first. Paths, sizes, hashes and mirror revisions are unchanged.

published_index_state()

Serial and dataset commit of the published index. No token is passed.

A bare-list index (before signing) has no serial and returns None.

sign_index(path, *, key_file=None)

Write index.json.sig beside path: base64 Ed25519 over the exact bytes.

The format matches voxprint-mdl index sign. The key is a PEM private key read from key_file or VOXPRINT_MDL_INDEX_KEY_FILE; it is never printed. The signature is checked against the derived public key before the file is written.

run_build(source, dest, *, revisions=None, commits=None, serial=None)

Build index.json from local manifests. Does not upload.

revision_map(items)

Parse repeatable REPO=SHA pins. SHA is 40 lowercase hex digits.

descriptor_commit(repo)

Last commit that touches voxprint-model.json on the published ref.

repo is a model repo, or a :func:hosted_path in the index dataset.

No token is passed. A missing file or a value that is not a full commit id is an error. This does not read a mirror revision.

push_index(path, *, upload=None, key_file=None, published_serial='fetch')

Sign index.json and upload it with index.json.sig in one commit.

The CLI calls this only after --push. There is no upload on the default path. A bare list, or a serial that is not above the published one, is refused before anything is signed.

run_backfill(dest, user=HF_AUTHOR)

Write local manifests for published repos. Does not upload.

is_hosted(document)

True when no forge repo holds the descriptor, so the index dataset does.

That is an official-source model: it has an hf or url mirror, none is marked ours, and none is under Mitroshenkov87.

hosted_path(model_id)

Row-side name of a hosted descriptor: datasets/<index>/descriptors/<id>/voxprint-model.json.

host_descriptors(source, *, upload=None)

Upload official-source descriptors into the index dataset in one commit.

Only documents that :func:is_hosted accepts are sent, to descriptors/<id>/voxprint-model.json. The CLI calls this only after --push. Returns the dataset commit to pin with index build --revision datasets/Mitroshenkov87/voxprint-model-index=SHA.

Plan and pipeline

Load plans/plan.toml. Disabled models stay in the file and are not run.

PinnedFile dataclass

One pinned source, reuse, or sidecar file.

Tier dataclass

A card budget and the quants that may fill it.

ModelPlan dataclass

One model: source, architecture, tiers, and optional offload fields.

source_files()

Pinned files whose role is source.

reuse_files()

Pinned files whose role is reuse.

Plan dataclass

The loaded run: budgets, contexts, and models.

enabled_models()

Models with enabled set.

load_plan(path)

Load and check a plan TOML file.

format_estimates(plan)

Text estimate of the run, including the llama.cpp build.

Run fetch, ablation, quantize, fit, eval, card, and upload for one model.

Host dataclass

Runner, downloader, and remote used by one pipeline process.

retry_stage(stage)

Stage command that restarts stage. imatrix restarts at quantize.

clear_from_stage(layout, model_id, stage)

Delete markers from stage through upload so the stage command runs that work again.

run_all(plan, host, repo, layout, *, jobs=1)

Run every enabled model and write the manifest.

run_model(plan, model, host, repo, layout, *, stop_after, budgets=None, progress=None, gate=None)

Run one model from fetch through stop_after.

default_ablate(source, dest, weight_bytes, free_vram)

Abliterate source into dest and return the ablation record.

forge command line. --json prints one object; logs stay on stderr.

build_parser()

The forge argument parser. The CLI reference is generated from this.

main(argv=None)

Run the forge CLI and return the process exit code.

Diffusion INT4

Pinned tools, models, and time estimates for the diffusion-int4 stage.

Licence checks below were read from the pinned trees on 2026-10-11. ComfyUI-GGUF LICENSE and Wan2.2 LICENSE.txt are Apache-2.0. llama.cpp tag b3962 is MIT.

StepEstimate dataclass

One resumable step and its planning time on an A100 80GB.

An H100 80GB uses the same table except the quant steps, which are 1.8x faster.

skip_wan()

True when FORGE_SKIP_WAN=1 omits the Wan 2.2 steps.

planned_steps()

Steps that run for this invocation, in order.

expected_minutes()

A100 planning total. Wan minutes are the same on an H100.

A100 (sm_80) and H100 (sm_90) planning and FP8 baseline mode.

An H100 80GB is planned at 1.8x the A100 quant step. FP8 tensor cores exist from sm_89 upward. On sm_80 the FP8 baseline is weight-only BF16.

GpuProfile dataclass

One detected device, or the A100 planning default when nothing was probed.

as_dict()

JSON object stored on the eval comparison.

parse_compute_cap(text)

Parse 8.0 or 9.0 from a capability string.

sm_tag(major, minor)

sm_80 for compute capability 8.0.

fp8_mode(major, minor)

native from sm_89 up, otherwise emulated.

planning_name(major, minor)

a100 for sm_80, h100 for sm_90, otherwise other.

profile_for_cap(major, minor)

Build the eval record for a known compute capability.

resolve_cap(*, probe)

Capability from FORGE_COMPUTE_CAP, then an optional nvidia-smi probe.

device_fp8_report(*, probe)

Eval field describing whether the FP8 baseline is native or emulated.

planning_gpu()

Which minute table to use. Unset detection stays on the A100 table.

step_minutes(step, gpu)

Minutes for one step. Only quant steps shrink on an H100.

total_minutes(steps, gpu)

Sum of :func:step_minutes.

Run the diffusion-int4 steps in order, with markers, heartbeats, and the disk floor.

step_functions(done)

Map a step id to the function that performs it.

run(ctx, *, done, watch, only=None)

Run every selected step, skipping markers that already exist.

format_estimates()

A100 and H100 planning tables for the selected steps.

Download the Wan experts before a rental run.

--dry-run does not call this module. A licence that is not Apache-2.0 stops the download.

fetch_wan(dest, token)

Snapshot Wan2.2-T2V-A14B at the pinned revision.

fetch_weights(dest, token, *, forge_out=None, licence=None)

Download the Wan experts and return the environment paths.