Interface LlamaCppEmbeddingsParams

Note that the modelPath is the only required parameter. For testing you can set this in the environment variable LLAMA_PATH.

interface LlamaCppEmbeddingsParams {
    modelPath: string;
    batchSize?: number;
    contextSize?: number;
    embedding?: boolean;
    f16Kv?: boolean;
    gbnf?: string;
    gpuLayers?: number;
    jsonSchema?: object;
    logitsAll?: boolean;
    maxTokens?: number;
    prependBos?: boolean;
    seed?: null | number;
    temperature?: number;
    threads?: number;
    topK?: number;
    topP?: number;
    trimWhitespaceSuffix?: boolean;
    useMlock?: boolean;
    useMmap?: boolean;
    vocabOnly?: boolean;
}

Hierarchy (view full)

  • LlamaBaseCppInputs
  • Toolkit
    • LlamaCppEmbeddingsParams

Properties

modelPath batchSize? contextSize? embedding? f16Kv? gbnf? gpuLayers? jsonSchema? logitsAll? maxTokens? prependBos? seed? temperature? threads? topK? topP? trimWhitespaceSuffix? useMlock? useMmap? vocabOnly?

Properties

modelPath

modelPath: string

Path to the model on the filesystem.

OptionalbatchSize

batchSize?: number

Prompt processing batch size.

OptionalcontextSize

contextSize?: number

Text context size.

Optionalembedding

embedding?: boolean

Embedding mode only.

Optionalf16Kv

f16Kv?: boolean

Use fp16 for KV cache.

Optionalgbnf

gbnf?: string

GBNF string to be used to format output. Also known as grammar.

OptionalgpuLayers

gpuLayers?: number

Number of layers to store in VRAM.

OptionaljsonSchema

jsonSchema?: object

JSON schema to be used to format output. Also known as grammar.

OptionallogitsAll

logitsAll?: boolean

The llama_eval() call computes all logits, not just the last one.

OptionalmaxTokens

maxTokens?: number

OptionalprependBos

prependBos?: boolean

Add the begining of sentence token.

Optionalseed

seed?: null | number

If null, a random seed will be used.

Optionaltemperature

temperature?: number

The randomness of the responses, e.g. 0.1 deterministic, 1.5 creative, 0.8 balanced, 0 disables.

Optionalthreads

threads?: number

Number of threads to use to evaluate tokens.

OptionaltopK

topK?: number

Consider the n most likely tokens, where n is 1 to vocabulary size, 0 disables (uses full vocabulary). Note: only applies when temperature > 0.

OptionaltopP

topP?: number

Selects the smallest token set whose probability exceeds P, where P is between 0 - 1, 1 disables. Note: only applies when temperature > 0.

OptionaltrimWhitespaceSuffix

trimWhitespaceSuffix?: boolean

Trim whitespace from the end of the generated text Disabled by default.

OptionaluseMlock

useMlock?: boolean

Force system to keep model in RAM.

OptionaluseMmap

useMmap?: boolean

Use mmap if possible.

OptionalvocabOnly

vocabOnly?: boolean

Only load the vocabulary, no weights.

Settings

Member Visibility

On This Page

Properties