Building with a large language model means assembling more than a model. You need an API for inference, a framework for orchestration, a retrieval layer for your own data, and a way to test what comes out the other side. The catalog of options in 2025 and 2026 is huge, and picking wrong usually means a rewrite.
This is a working map of the LLM tools developers actually reach for: APIs, prompt engineering tools, RAG frameworks, fine-tuning platforms, evaluation harnesses, and open source LLM tools you can run on your own hardware. Real names, real features, real pricing.
The Six Layers of a Modern LLM Stack
“LLM tools” covers everything from a raw API call to a full observability platform. In practice, most teams choose one tool per layer:
- LLM APIs — OpenAI, Anthropic, Google Gemini, Mistral, DeepSeek, and hosted open-weight providers like Together AI and Fireworks.
- Prompt engineering tools — playgrounds, versioned prompt managers, and testing harnesses such as Promptfoo and Langfuse.
- RAG frameworks — LlamaIndex, LangChain and LangGraph, Haystack, and Dify for connecting models to private data.
- Fine-tuning platforms — Hugging Face TRL, Unsloth, Axolotl, and LLaMA-Factory.
- Evaluation and observability tools — Ragas, DeepEval, Arize Phoenix, Braintrust, and Langfuse.
- Open source LLM tools for local inference — Ollama, LM Studio, Jan, llama.cpp, and vLLM.
Each layer has a default. Knowing it saves a week of comparison spreadsheets before you write a line of code.
LLM APIs: Still the Cheapest Way to Start
For most products, hosted APIs remain the right first move. You avoid GPU procurement, model serving, and quantization headaches, and you pay only for tokens you burn.
List prices in late 2026 (per million tokens, from published provider rate tables) cluster into three bands.
Budget tier. OpenAI’s GPT-5 nano lists at $0.05 input and $0.40 output. Google’s Gemini Flash-Lite class sits around $0.07 to $0.10 input and $0.30 to $0.40 output. These handle classification, extraction, and high-volume chat at pennies per thousand requests.
Mid tier. GPT-5 runs $1.25 in and $10 out. Anthropic’s Claude Haiku 4.5 is $1 in and $5 out. Gemini 2.5 Flash lands near $0.30 in and $2.50 out. This band carries most production workloads.
Flagship tier. Frontier reasoning models push $3 to $15 per million input tokens, with output often four to five times that. Reserve them for hard tasks where the extra cost is invisible against the value of a correct answer.
Cost control is now part of the category. LiteLLM is an open source gateway that gives you one OpenAI-compatible endpoint across a hundred-plus providers, with budget limits, retries, and fallback routing. If you touch more than one model vendor, it is one of the cheapest LLM tools to adopt: a proxy, not a platform migration.
Model launches and price cuts move fast. We track every one on the LLM updates page.
Prompt Engineering Tools: Treat Prompts Like Code
Prompt engineering stopped being a text editor exercise. Teams version prompts, run regression suites against them, and block deploys when quality drops.
Promptfoo is the standard open source entry point. You write test cases and assertions in YAML, then run them across 30-plus providers — OpenAI, Anthropic, Gemini, Bedrock, and local models — in a single pass. Assertions cover semantic similarity, regex, JSON validity, LLM-as-a-judge scoring, and toxicity. It plugs into GitHub Actions, so prompt changes behave like any other pull request. The CLI is free; Promptfoo Cloud adds shared dashboards and team controls on paid plans. Red teaming plugins generate adversarial vectors for prompt injection, jailbreaks, and PII leakage. The Promptfoo provider docs list every supported model.
Langfuse handles the other half: prompt management in production. Prompts live in a versioned store, get pulled at runtime, and every call is traced with cost, latency, and token counts. Its free cloud tier includes 50,000 observations per month, and the MIT-licensed code can be self-hosted without those limits.
DSPy takes a different route. Instead of hand-tuning wording, you declare modules and metrics, and the framework compiles the prompts from data. It suits teams rerunning the same retrieval or classification prompt through dozens of iterations.
Braintrust and Maxim AI are the commercial prompt engineering tools aimed at product teams — playgrounds, experiment tracking, and evaluation workflows shared between engineers and non-technical reviewers.
RAG Frameworks: LLM Tools for Developers With Private Data
Retrieval-augmented generation is where a lot of LLM tools for developers converge. The framework you pick decides how much control you keep over chunking, indexing, and reranking.
LlamaIndex is the depth pick. It supports over 160 file formats, LlamaParse handles messy enterprise PDFs, and the connector library covers nearly every data source worth indexing. LlamaCloud adds managed infrastructure with a free tier for prototyping. The tradeoff is a steeper learning curve than a thinner framework gives you.
LangChain and LangGraph are the breadth pick. LangChain has passed 143,000 GitHub stars and ships the largest integration surface in the category. LangGraph, stable since late 2025, is the stateful graph runtime for multi-step agentic workflows: cycles, branching, human-in-the-loop interrupts. LangGraph Platform pricing runs Developer free, Plus at $39 per month, Enterprise custom.
Haystack from deepset is the pipeline pick: explicit, readable components where every connection is visible in code. Apache-2.0 licensed, EU-based, a good fit for data-sensitive teams. deepset Studio offers 100 free pipeline hours.
Dify is the visual option — a source-available platform with a drag-and-drop builder, built-in RAG, and agent workflows, for teams that want a working app before they want a codebase.
Small projects often need no framework at all. One call to one model with one document attached is a provider SDK and a vector store. Frameworks earn their weight when state, retries, and multi-step tool use enter the picture.
Fine-Tuning Platforms: When Prompting Isn’t Enough
Fine-tuning is not the default. Prompting plus retrieval solves most problems cheaper. But domain jargon, strict output formats, and latency budgets justify it, and the tooling got dramatically cheaper in 2025.
Unsloth is the fastest path to a custom LoRA. It reports roughly 2x faster training with lower memory use than a standard Hugging Face loop, and it fits many 7B to 70B fine-tunes on a single consumer GPU. Free and open source.
Axolotl prioritizes flexibility: supervised tuning, LoRA, QLoRA, and full-parameter updates across architectures, all configured from YAML. LLaMA-Factory is the same idea with a web UI, which matters when a domain expert needs to launch a run without touching a terminal.
Hugging Face is the platform layer. The hub hosts over 500,000 models, and TRL, PEFT, and Transformers cover training, parameter-efficient fine-tuning, and inference. The free tier covers public work; PRO runs $9 per month for higher rate limits and private model support.
Managed fine-tuning exists on every major API — OpenAI, Anthropic, and Google all accept training data through their platforms — but per-token training charges add up. A few epochs over a 50,000-example dataset can cost more than a year of prompting the same model. Run the math first. Our about page explains how we verify these numbers.
Evaluation and Observability: The Layer People Skip
The most common failure in production LLM systems is shipping changes nobody measured. Evaluation tools close that gap.
Ragas scores RAG pipelines specifically — faithfulness, answer relevance, context precision — using your retrieved chunks and generated answers. DeepEval behaves like pytest for LLM outputs, with assertion-style metrics you drop into a CI run. Both are open source.
Arize Phoenix is an OpenTelemetry-native observability stack: traces, datasets, experiments, and evaluations, self-hostable at no cost. Langfuse covers the same ground with the widest adoption in the open source category — MIT-licensed, ClickHouse-backed, used by more than 50,000 companies, with a free tier of 50,000 observations monthly.
LangSmith is LangChain’s hosted equivalent, while Braintrust and Helicone compete on CI-integrated evaluation and cost tracking. Most teams pick one tracing platform and one eval harness; they solve different problems.
The habit worth building: a golden dataset of 50 to 200 real inputs, a scoring pass on every model or prompt change, and a blocking check in CI. That one routine catches more regressions than any leaderboard ever will, and it works the same whether you are calling a hosted model or serving open weights in a datacenter.
A note on vector databases, since they show up under every RAG comparison. Pinecone, Weaviate, Qdrant, Milvus, and pgvector all work. If you already run Postgres, pgvector removes a whole operational dependency. If you expect billions of vectors, pick a dedicated store and benchmark it with your own embedding model before committing.
Open Source LLM Tools for Running Models Locally
Local inference stopped being a hobbyist project. Open source LLM tools now cover the full path from download to production endpoint.
Ollama is the developer default — two commands to a running model, more than 100 models in the library (Llama, Mistral, Qwen, DeepSeek, Phi), and an OpenAI-compatible REST API on localhost:11434. Point any existing SDK at it and swap providers later with a base URL change. macOS, Linux, and Windows, with GPU acceleration. Ollama is free.
LM Studio gives the same capability behind a polished desktop GUI with a Hugging Face-backed model browser and local server mode. Jan and text-generation-webui cover the fully offline and power-user ends of the spectrum — the latter supports GGUF, GPTQ, AWQ, and EXL2 formats plus built-in LoRA training.
For serving at scale, vLLM provides high-throughput inference with PagedAttention, and llama.cpp is the C++ engine underneath most of the tools above. Together they cover a laptop prototype and a Kubernetes deployment.
Free LLM tools are the sensible default for development anyway: build and test locally against open weights, then route to a hosted API only when you need frontier quality or guaranteed uptime.
How to Pick the Best LLM Tools for Your Project
Shortlist by constraint, not by hype:
- Startups and MVPs: one hosted API, LiteLLM for routing, Promptfoo for tests, Langfuse free tier for traces. Four tools, near-zero fixed cost.
- Teams with private documents: LlamaIndex or Haystack for retrieval, a vector store, Ragas for retrieval scoring.
- Agent-heavy products: LangGraph for state, Langfuse for tracing every tool call, DeepEval for regression suites.
- Cost-sensitive, high-volume workloads: open weights through Ollama or vLLM, or budget API tiers for classification and extraction.
- Regulated or air-gapped environments: self-hosted Haystack or LlamaIndex, Langfuse self-hosted, local inference through llama.cpp.
Whatever you choose, keep the seams clean. Every LLM tool behind an interface you control can be replaced in a quarter. Every one wired directly into business logic cannot.
FAQ: LLM Tools
What are LLM tools?
LLM tools are the software layers used to build, deploy, and monitor large language model applications: provider APIs, prompt engineering tools, RAG frameworks, fine-tuning platforms, evaluation harnesses, and local inference runtimes. Most production stacks combine one tool from each layer.
What is the best LLM tool for developers in 2025?
There is no single winner. For most teams the best starting set is a hosted API such as OpenAI or Anthropic, LlamaIndex or LangChain for retrieval, Promptfoo for prompt testing, and Langfuse for tracing. The best LLM tools are the ones that match your data, budget, and latency constraints.
Are there free LLM tools that work in production?
Yes. Promptfoo, Langfuse, Arize Phoenix, Ragas, DeepEval, Haystack, and LiteLLM are all open source with free tiers or fully self-hosted use. Ollama runs models locally at no cost. Paid plans generally add hosted infrastructure, collaboration features, and higher limits rather than core capability.
Which open source LLM tools run locally?
Ollama, LM Studio, Jan, text-generation-webui, llama.cpp, and vLLM all run models on your own hardware. Ollama is the fastest start and exposes an OpenAI-compatible API, while vLLM and llama.cpp handle production serving. None of them send prompts to a third party.
Do I need a framework to build with an LLM?
Usually not at first. A provider SDK plus a vector database covers most single-call and single-document use cases. Reach for a RAG framework or an agent runtime when you need state, retries, multiple tools, or complex retrieval pipelines across many sources.
Picking among LLM tools gets easier once you write down the constraint you are actually solving for — cost, privacy, latency, or accuracy. Everything else is preference. Questions about a specific stack? Contact us.
