LLM

class sycamore.llms.llms.LLM(model_name: str, default_mode: LLMMode, cache: Cache | None = None, default_llm_kwargs: dict[str, Any] | None = None)

Abstract representation of an LLM instance. and should be subclassed to implement specific LLM providers.

default_mode() → LLMMode

Returns the default execution mode for the llm

format_image(image: Image) → dict[str, Any]

Returns a dictionary containing the specified image suitable for use in an LLM message.

abstract generate(*, prompt: RenderedPrompt, llm_kwargs: dict | None = None, model: LLMModel | None = None) → str

Generates a response from the LLM for the given prompt and LLM parameters.

async generate_async(*, prompt: RenderedPrompt, llm_kwargs: dict | None = None, model: LLMModel | None = None) → str

Generates a response from the LLM for the given prompt and LLM parameters asynchronously.

generate_batch(*, prompts: list[RenderedPrompt], llm_kwargs: dict | None = None, model: LLMModel | None = None) → list[str]

Generates a series of responses from the LLM for the given series of prompts. Order is preserved.

generate_metadata(*, prompt: RenderedPrompt, model: LLMModel | None = None, llm_kwargs: dict | None = None) → dict

Generates a response from the LLM for the given prompt and LLM parameters and returns metadata.

TODO: Implement generic_generate(model: LLMModel, ...) and generic_generate_args(model_class, kwargs)

to specify default arguments during model construction. The former should cache the client if possible. Then we can call generate on any model rather than only ones in the same family. Potentially get rid of the model argument to generate* at the same time to simplify the implementations.

get_metadata(model, kwargs, response_text, wall_latency, in_tokens, out_tokens) → dict

Generate metadata for the LLM response.

abstract is_chat_mode() → bool

Returns True if the LLM is in chat mode, False otherwise.

enum sycamore.llms.llms.LLMMode(value)

Valid values are as follows:

SYNC = <LLMMode.SYNC: 1>

ASYNC = <LLMMode.ASYNC: 2>

BATCH = <LLMMode.BATCH: 3>

class sycamore.llms.openai.OpenAI(model_name: OpenAIModels | OpenAIModel | str, api_key: str | None = None, client_wrapper: OpenAIClientWrapper | None = None, params: OpenAIClientWrapper | None = None, default_mode: LLMMode = LLMMode.ASYNC, cache: Cache | None = None, default_llm_kwargs: dict[str, Any] | None = None, **kwargs)

Bases: LLM

An LLM interface to OpenAI models.

Parameters:

• model_name -- The name of the OpenAI model to use. This can be an instance of OpenAIModels, an instance of OpenAIModel, or a string. If a string is provided, it must be the name of the model.

• api_key -- The API key to use for the OpenAI client. If not provided, the key will be read from the OPENAI_API_KEY environment variable.

• client_wrapper -- An instance of OpenAIClientWrapper to use for the OpenAI client. If not provided, a new instance will be created using the provided parameters.

• params -- An instance of OpenAIClientParameters to use for the OpenAI client. If not provided, a new instance will be created using the provided parameters.

• cache -- An instance of Cache to use for caching responses. If not provided, no caching will be used.

• default_mode -- Default execution mode for the llm

• **kwargs -- Additional parameters to pass to the OpenAI client.

format_image(image: Image, format: str = 'PNG') → dict[str, Any]

Returns a dictionary containing the specified image suitable for use in an LLM message.

generate(*, prompt: RenderedPrompt, llm_kwargs: dict | None = None, model: LLMModel | None = None) → str

Generates a response from the LLM for the given prompt and LLM parameters.

async generate_async(*, prompt: RenderedPrompt, llm_kwargs: dict | None = None, model: LLMModel | None = None) → str

Generates a response from the LLM for the given prompt and LLM parameters asynchronously.

generate_batch(*, prompts: list[RenderedPrompt], llm_kwargs: dict | None = None, model: LLMModel | None = None) → list[str]

Generates a series of responses from the LLM for the given series of prompts. Order is preserved.

generate_metadata(*, prompt: RenderedPrompt, model: LLMModel | None = None, llm_kwargs: dict | None = None) → dict

Generates a response from the LLM for the given prompt and LLM parameters and returns metadata.

TODO: Implement generic_generate(model: LLMModel, ...) and generic_generate_args(model_class, kwargs)

to specify default arguments during model construction. The former should cache the client if possible. Then we can call generate on any model rather than only ones in the same family. Potentially get rid of the model argument to generate* at the same time to simplify the implementations.

is_chat_mode()

Returns True if the LLM is in chat mode, False otherwise.

enum sycamore.llms.openai.OpenAIClientType(value)

Bases: Enum

Valid values are as follows:

OPENAI = <OpenAIClientType.OPENAI: 0>

AZURE = <OpenAIClientType.AZURE: 1>

class sycamore.llms.anthropic.Anthropic(model_name: AnthropicModels | AnthropicModel | str, default_mode: LLMMode = LLMMode.ASYNC, cache: Cache | None = None, default_llm_kwargs: dict[str, Any] | None = None, client_args: dict[str, Any] = {})

Bases: LLM

This is an LLM implementation that uses the AWS Claude API to generate text.

Parameters:

• model_name -- The name of the Claude model to use.

• cache -- A cache object to use for caching results.

default_mode() → LLMMode

Returns the default execution mode for the llm

format_image(image: Image) → dict[str, Any]

Returns a dictionary containing the specified image suitable for use in an LLM message.

generate(*, prompt: RenderedPrompt, llm_kwargs: dict | None = None, model: LLMModel | None = None) → str

Generates a response from the LLM for the given prompt and LLM parameters.

async generate_async(*, prompt: RenderedPrompt, llm_kwargs: dict | None = None, model: LLMModel | None = None) → str

Generates a response from the LLM for the given prompt and LLM parameters asynchronously.

generate_batch(*, prompts: list[RenderedPrompt], llm_kwargs: dict | None = None, model: LLMModel | None = None) → list[str]

Generates a series of responses from the LLM for the given series of prompts. Order is preserved.

generate_metadata(*, prompt: RenderedPrompt, model: LLMModel | None = None, llm_kwargs: dict | None = None) → dict

Generates a response from the LLM for the given prompt and LLM parameters and returns metadata.

TODO: Implement generic_generate(model: LLMModel, ...) and generic_generate_args(model_class, kwargs)

to specify default arguments during model construction. The former should cache the client if possible. Then we can call generate on any model rather than only ones in the same family. Potentially get rid of the model argument to generate* at the same time to simplify the implementations.

is_chat_mode() → bool

Returns True if the LLM is in chat mode, False otherwise.

class sycamore.llms.gemini.Gemini(model_name: GeminiModels | GeminiModel | str, default_mode: LLMMode = LLMMode.ASYNC, cache: Cache | None = None, api_key: str | None = None, default_llm_kwargs: dict[str, Any] | None = None, disable_helicone: bool = True)

Bases: LLM

This is an LLM implementation that uses the Google Gemini API to generate text.

Parameters:

• model_name -- The name of the Gemini model to use.

• cache -- A cache object to use for caching results.

default_mode() → LLMMode

Returns the default execution mode for the llm

generate(*, prompt: RenderedPrompt, llm_kwargs: dict | None = None, model: LLMModel | None = None) → str

Generates a response from the LLM for the given prompt and LLM parameters.

async generate_async(*, prompt: RenderedPrompt, llm_kwargs: dict | None = None, model: LLMModel | None = None) → str

Generates a response from the LLM for the given prompt and LLM parameters asynchronously.

generate_metadata(*, prompt: RenderedPrompt, model: LLMModel | None = None, llm_kwargs: dict | None = None) → dict

Generates a response from the LLM for the given prompt and LLM parameters and returns metadata.

TODO: Implement generic_generate(model: LLMModel, ...) and generic_generate_args(model_class, kwargs)

to specify default arguments during model construction. The former should cache the client if possible. Then we can call generate on any model rather than only ones in the same family. Potentially get rid of the model argument to generate* at the same time to simplify the implementations.

is_chat_mode() → bool

Returns True if the LLM is in chat mode, False otherwise.

class sycamore.llms.bedrock.Bedrock(model_name: BedrockModels | BedrockModel | str, cache: Cache | None = None, default_llm_kwargs: dict[str, Any] | None = None)

Bases: LLM

This is an LLM implementation that uses the AWS Bedrock API to generate text.

Parameters:

• model_name -- The name of the Bedrock model to use.

• cache -- A cache object to use for caching results.

format_image(image: Image) → dict[str, Any]

Returns a dictionary containing the specified image suitable for use in an LLM message.

generate(*, prompt: RenderedPrompt, llm_kwargs: dict | None = None, model: LLMModel | None = None) → str

Generates a response from the LLM for the given prompt and LLM parameters.

generate_metadata(*, prompt: RenderedPrompt, model: LLMModel | None = None, llm_kwargs: dict | None = None) → dict

Generates a response from the LLM for the given prompt and LLM parameters and returns metadata.

TODO: Implement generic_generate(model: LLMModel, ...) and generic_generate_args(model_class, kwargs)

to specify default arguments during model construction. The former should cache the client if possible. Then we can call generate on any model rather than only ones in the same family. Potentially get rid of the model argument to generate* at the same time to simplify the implementations.

is_chat_mode() → bool

Returns True if the LLM is in chat mode, False otherwise.