AI Evaluation Engineer
- Siri Info Solutions Inc.
23 Hours Ago
NA
W2, C2C
Tampa-FL
8-10 Years
Required Skills: GenAI, LLMs, RAG, prompts, tool calling, AI agents, Python, APIs, JSON, datasets, evaluation framework, AI evaluation, LangSmith, Langfuse, Ragas, DeepEval, MLflow
Job Description
We are seeking an experienced AI Evaluation Engineer to define and implement evaluation strategies and success metrics for AI agents. In this role, you will be responsible for assessing task completion, correctness, groundedness, hallucination, tool usage, safety, consistency, and overall user experience of AI-driven systems. You will build comprehensive test scenarios, including edge and adversarial cases, and develop automated evaluation pipelines leveraging deterministic checks, LLM-as-a-Judge techniques, and custom evaluators. Your analysis will involve detailed examination of agent traces, prompts, retrievals, tool calls, and outputs to identify failure modes and performance bottlenecks. You will also conduct regression testing spanning changes in models, prompts, tools, and workflows. Establishing quality thresholds, monitoring dashboards, and release gates will be key to maintaining robust AI agent deployments. This role requires close collaboration with AI engineering, product, business, and risk teams to ensure continuous improvement in agent functionality and safety.
Key Responsibilities:Responsibilities:
- Define evaluation strategies and success metrics for AI agents.
- Evaluate task completion, correctness, groundedness, hallucination, tool usage, safety, consistency, and user experience.
- Build test scenarios, edge cases, and adversarial cases.
- Develop automated eval pipelines using deterministic checks, LLM-as-a-Judge, and custom evaluators.
- Analyze agent traces, prompts, retrieval, tool calls, and outputs to identify failure patterns.
- Perform regression testing across model, prompt, tool, and workflow changes.
- Establish quality thresholds, dashboards, and release gates.
- Partner with AI engineering, product, business, and risk teams to continuously improve agent performance.
