For those who’re following AI Brokers, then you definitely might need seen that LangChain has created a pleasant ecosystem with LangChain, LangGraph, LangSmith & LangServe. Leveraging these, we will construct, deploy, consider, and monitor Agentic AI programs. Whereas constructing an AI Agent, I simply thought to myself, “Why not present a easy demo to point out the intertwined working of LangGraph & LangSmith?”. That is gonna be useful as AI Brokers typically want a number of LLM calls and now have larger prices related to them. This combo will assist monitor the bills and likewise consider the system utilizing customized datasets. With none additional ado, let’s dive in.  

LangGraph for AI Brokers

Merely put, AI Brokers are LLMs with the potential to suppose/cause and will entry instruments to deal with their shortcomings or acquire entry to real-time data. LangGraph is an Agentic AI framework based mostly on LangChain to construct these AI Brokers. LangGraph helps construct graph-based Brokers; additionally, the creation of Agentic workflows is simplified with many inbuilt capabilities already current within the LangGraph/LangChain libraries. 

Learn extra: What’s LangGraph?

What’s LangSmith?

LangSmith is a monitoring and analysis platform by LangChain. It’s framework-agnostic, designed to work with any Agentic framework, resembling LangGraph, and even with Brokers constructed utterly from scratch. LangSmith will be simply configured to hint the runs and likewise monitor the bills of the Agentic system. It additionally helps operating experiments on the system, like altering the immediate and fashions within the system, and evaluating the outcomes. It has predefined evaluators like helpfulness, correctness, and hallucinations. You too can select to outline your individual evaluators. Let’s take a look at the LangSmith platform to get a greater thought of it.  

Learn extra: Final Information to LangSmith

The LangSmith Platform

Let’s first enroll/register to take a look at the platform: https://www.langchain.com/langsmith 

That is how the platform seems with a number of tabs: 

• Tracing Initiatives: Retains a monitor of a number of initiatives together with their traces or units of runs. Right here, the prices, errors, latency, and plenty of different issues are tracked.  
• Monitoring: Right here you possibly can set alerts to warn you, as an example, if the system fails or the latency is above the set threshold.  
• Dataset & Experiments: Right here, you possibly can run experiments utilizing human-crafted datasets or use the platform to create AI-generated datasets for testing your system. You too can change your mannequin to see how the efficiency varies.  
• Prompts: Right here you possibly can retailer a number of prompts and later change the wording or sequence of directions to see how your outcomes are altering.  

LangSmith in Motion

Notice: We’ll solely construct easy brokers for this tutorial to concentrate on the LangSmith facet of issues.  

Let’s construct a basic math expression-solving agent that makes use of a easy instrument after which allow traceability. After which we’ll verify the LangSmith dashboard to see what will be tracked utilizing the platform.  

Getting the API keys: 

1. Go to the Langsmith dashboard and click on on the ‘Setup Observability’ Button. You then’ll see this display screen. https://www.langchain.com/langsmith 

Now, click on on the ‘Generate API Key’ possibility and preserve the LangSmith key helpful.  

2. Now go to Google AI Studio to get your palms on the Gemini API key: https://aistudio.google.com/api-keys  

Click on on ‘Create API key’ on the right-top and create a mission if it doesn’t exist already, and preserve the important thing helpful. 

Python Code

Notice: I’ll be utilizing Google Colab for operating the code.

Installations 

!pip set up -q langgraph langsmith langchain  !pip set up -q langchain-google-genai

Notice: Ensure that to restart the session earlier than persevering with from right here.  

Setting the atmosphere 

Move the API keys when prompted.  

from getpass import getpass  LANGCHAIN_API_KEY=getpass('Enter LangSmith API Key: ')  GOOGLE_API_KEY=getpass('Enter Gemini API Key: ')  import os  os.environ['LANGCHAIN_TRACING_V2'] = 'true'  os.environ['LANGCHAIN_API_KEY'] = LANGCHAIN_API_KEY  os.environ['LANGCHAIN_PROJECT'] = 'Testing'

Notice: It’s really helpful to trace totally different initiatives with totally different mission names; right here, I’m naming it ‘Testing’. 

Establishing and operating the agent 

• Right here, we’re utilizing a easy instrument that the agent can use to unravel math expressions 
• We’re utilizing the in-built create_react_agent from LangGraph, the place we have now to outline the mannequin, give entry to instruments, and we’re good to go. 

from langgraph.prebuilt import create_react_agent  from langchain_google_genai import ChatGoogleGenerativeAI  def solve_math_problem(expression: str) -> str:     """Remedy a math drawback."""     strive:      # Consider the mathematical expression      outcome = eval(expression, {"__builtins__": {}})      return f"The reply is {outcome}."     besides Exception:      return "I could not resolve that expression."  # Initialize the Gemini mannequin with API key  mannequin = ChatGoogleGenerativeAI(     mannequin="gemini-2.5-flash",     google_api_key=GOOGLE_API_KEY  )  # Create the agent  agent = create_react_agent(     mannequin=mannequin,     instruments=[solve_math_problem],     immediate=(      "You're a Math Tutor AI. "      "When a person asks a math query, cause by means of the steps clearly "      "and use the instrument `solve_math_problem` for numeric calculations. "      "At all times clarify your reasoning earlier than giving the ultimate reply."     ),  )  # Run the agent  response = agent.invoke(     {"messages": [{"role": "user", "content": "What is (12 + 8) * 3?"}]}  )  print(response)

Output: 

We are able to see that the agent used the instrument’s response ‘The reply is 60’ and didn’t hallucinate whereas answering the query. Now let’s verify the LangSmith dashboard.  

LangSmith Dashboard

Tracing Initiatives tab 

We are able to see that the mission has been created with the identify ‘testing’; you possibly can click on on it to see detailed logs.  

Right here it exhibits the run-wise: 

• Complete Tokens 
• Complete Value 
• Latency 
• Enter 
• Output 
• Time when the code was executed 

Notice: I’m utilizing the free tier of Gemini right here, so I can use the important thing freed from price in line with the each day limits.  

Monitoring tab 

• Right here you possibly can see a dashboard with the initiatives, runs, and whole prices.  

LLM as a decide

LangSmith permits the creation of a dataset utilizing a easy dictionary with enter and output keys. This dataset with the anticipated output can be utilized to judge an AI system’s generated outputs on metrics like helpfulness, correctness, and hallucinations.  

We’ll use an identical math agent, create the dataset, and consider our agentic system.  

Notice: I’ll be utilizing OpenAI API (gpt-4o-mini) for the demo right here, that is to keep away from API Restrict points with the free-tier Gemini API.  

Installations 

!pip set up -q openevals langchain-openai 

Surroundings Setup 

import os  from google.colab import userdata  os.environ['OPENAI_API_KEY']=userdata.get('OPENAI_API_KEY')

Defining the Agent 

from langsmith import Consumer, wrappers  from openevals.llm import create_llm_as_judge  from openevals.prompts import CORRECTNESS_PROMPT  from langchain_openai import ChatOpenAI  from langgraph.prebuilt import create_react_agent  from langchain_core.instruments import instrument  from typing import Dict, Record  import requests  # STEP 1: Outline Instruments for the Agent =====  @instrument  def solve_math_problem(expression: str) -> str:     """Remedy a math drawback."""     strive:      # Consider the mathematical expression      outcome = eval(expression, {"__builtins__": {}})      return f"The reply is {outcome}."     besides Exception:      return "I could not resolve that expression."  # STEP 2: Create the LangGraph ReAct Agent =====  def create_math_agent():     """Create a ReAct agent with instruments."""     # Initialize the LLM     mannequin = ChatOpenAI(mannequin="gpt-4o-mini", temperature=0)     # Outline the instruments     instruments = [solve_math_problem]     # Create the ReAct agent utilizing LangGraph's prebuilt operate     agent = create_react_agent(      mannequin=mannequin,      instruments=[solve_math_problem],      immediate=(          "You're a Math Tutor AI. "          "When a person asks a math query, cause by means of the steps clearly "          "and use the instrument `solve_math_problem` for numeric calculations. "          "At all times clarify your reasoning earlier than giving the ultimate reply."      ),     )     return agent

Creating the dataset 

• Let’s create a dataset with easy and exhausting math expressions that we will later use to run experiments.  

shopper = Consumer()  dataset = shopper.create_dataset(     dataset_name="Math Dataset",     description="Arduous numeric + blended arithmetic expressions to judge the solver agent."  )  examples = [     # Simple check     {      "inputs": {"question": "12 + 7"},      "outputs": {"answer": "The answer is 19."},     },     {      "inputs": {"question": "100 - 37"},      "outputs": {"answer": "The answer is 63."},     },     # Mixed operators and parentheses     {      "inputs": {"question": "(3 + 5) * 2 - 4 / 2"},      "outputs": {"answer": "The answer is 14.0."},     },     {      "inputs": {"question": "2 * (3 + (4 - 1)*5) / 3"},      "outputs": {"answer": "The answer is 14.0."},     },     # Large numbers & multiplication     {      "inputs": {"question": "98765 * 4321"},      "outputs": {"answer": "The answer is 426,373,565."},     },     {      "inputs": {"question": "123456789 * 987654321"},      "outputs": {"answer": "The answer is 121,932,631,112,635,269."},     },     # Division, decimals, rounding     {      "inputs": {"question": "22 / 7"},      "outputs": {"answer": "The answer is approximately 3.142857142857143."},     },     {      "inputs": {"question": "5 / 3"},      "outputs": {"answer": "The answer is 1.6666666666666667."},     },     # Exponents, roots     {      "inputs": {"question": "2 ** 10 + 3 ** 5"},      "outputs": {"answer": "The answer is 1128."},     },     {      "inputs": {"question": "sqrt(2) * sqrt(8)"},      "outputs": {"answer": "The answer is 4.0."},     },     # Edge / error / “unanswerable” cases     {      "inputs": {"question": "5 / 0"},      "outputs": {"answer": "I couldn’t solve that expression."},     },     {      "inputs": {"question": "abc + 5"},      "outputs": {"answer": "I couldn’t solve that expression."},     },     {      "inputs": {"question": ""},      "outputs": {"answer": "I couldn’t solve that expression."},     },  ]  shopper.create_examples(     dataset_id=dataset.id,     examples=examples)

Nice! We created a dataset with 13 data:

Defining the goal operate 

• This operate invokes the agent and returns the response 

def goal(inputs: Dict) -> Dict:     agent = create_math_agent()     agent_input = {      "messages": [{"role": "user", "content": inputs["question"]}]     }     outcome = agent.invoke(agent_input)     final_message = outcome["messages"][-1]     reply = final_message.content material if hasattr(final_message, 'content material') else str(final_message)     return {"reply": reply}

Defining the Evaluator 

• We use the pre-built llm_as_judge operate and likewise import the immediate from the openevals library. 
• We’re utilizing 4o-mini for now to maintain the prices low, however a reasoning mannequin will be higher fitted to this process.  

def correctness_evaluator(inputs: Dict, outputs: Dict, reference_outputs: Dict) -> Dict:     evaluator = create_llm_as_judge(      immediate=CORRECTNESS_PROMPT,      mannequin="openai:gpt-4o-mini",      feedback_key="correctness",     )     eval_result = evaluator(      inputs=inputs,      outputs=outputs,      reference_outputs=reference_outputs     )     return eval_result

Working the analysis 

experiment_results = shopper.consider(     goal,     knowledge="Math Dataset",     evaluators=[correctness_evaluator],     experiment_prefix="langgraph-math-agent",     max_concurrency=2,  )

Output:

A hyperlink will probably be generated after the run. On click on, you’ll be redirected to LangSmith’s ‘Datasets & Experiments’ tab, the place you possibly can see the outcomes of the experiment.  

Now we have efficiently experimented with utilizing LLM as a Decide. That is insightful when it comes to discovering edge instances, prices, and token utilization.  

The errors listed here are principally mismatched because of the usage of commas or the presence of lengthy decimals. This may be solved by altering the analysis immediate or attempting a reasoning mannequin. Or simplify including commas and guaranteeing decimal formatting on the instrument degree itself.  

Conclusion

And there you’ve got it! We’ve efficiently proven the intertwined working of LangGraph for constructing our agent and LangSmith for tracing and evaluating it. This combo is extremely highly effective for monitoring bills and guaranteeing your agent performs as anticipated with customized datasets. Whereas we targeted on tracing and experiments, LangSmith’s capabilities don’t cease there. You too can discover highly effective options like A/B testing totally different prompts in manufacturing, including human-in-the-loop suggestions on to traces, and creating automations to streamline your debugging workflow. 

Incessantly Requested Questions