Documentation Index
Fetch the complete documentation index at: https://docs-v1.agno.com/llms.txt
Use this file to discover all available pages before exploring further.
Code
cookbook/models/vllm/structured_output.py
You are viewing v1 docs. For the latest documentation, visit docs.agno.com
Documentation Index
Fetch the complete documentation index at: https://docs-v1.agno.com/llms.txt
Use this file to discover all available pages before exploring further.
from typing import List
from agno.agent import Agent
from agno.models.vllm import vLLM
from pydantic import BaseModel, Field
class MovieScript(BaseModel):
name: str = Field(..., description="Give a name to this movie")
setting: str = Field(
..., description="Provide a nice setting for a blockbuster movie."
)
ending: str = Field(
...,
description="Ending of the movie. If not available, provide a happy ending.",
)
genre: str = Field(
...,
description="Genre of the movie. If not available, select action, thriller or romantic comedy.",
)
characters: List[str] = Field(..., description="Name of characters for this movie.")
storyline: str = Field(
..., description="3 sentence storyline for the movie. Make it exciting!"
)
agent = Agent(
model=vLLM(
id="NousResearch/Nous-Hermes-2-Mistral-7B-DPO", top_k=20, enable_thinking=False
),
description="You write movie scripts.",
response_model=MovieScript,
)
agent.print_response("Llamas ruling the world")
Create a virtual environment
Terminal and create a python virtual environment.python3 -m venv .venv
source .venv/bin/activate
Start vLLM server
vllm serve NousResearch/Nous-Hermes-2-Mistral-7B-DPO \
--enable-auto-tool-choice \
--tool-call-parser hermes \
--dtype float16 \
--max-model-len 8192 \
--gpu-memory-utilization 0.9
Was this page helpful?