Agent Evaluation Engineer — Build-Time Framework
Guarde esta oferta e mantenha a sua pesquisa organizada
Crie uma conta gratuita para guardar ofertas, criar alertas e voltar a este anúncio a partir do seu painel.
Ao continuar, aceita os nossos Termos & Política de Privacidade.
ResponsibilitiesDesign and implement build-time evaluation frameworks for agentic workflows using LangGraph or comparable orchestration frameworksCreate deterministic and LLM-as-judge grading pipelines covering reasoning, trajectory accuracy, and output qualityDevelop test harnesses for multi-turn conversational simulations and context-retention scoringDefine reliability assessment methods including multi-trial metrics (pass@k, pass^k)Implement CI/CD deployment gates that enforce quality thresholds and block releases not meeting standardsIntegrate staging validation, shadow-mode traffic comparison, and A/B rollout control in deployment pipelinesLeverage AWS AgentCore Evaluations for on-demand and online scoring components connected to production feedbackConvert production incidents into reusable regression cases for continuous quality improvementCollaborate with engineering and DevOps teams to embed evaluation gates into automated workflowsRequirements4+ years of experience in building automated testing or evaluation frameworks for ML, LLM, or agentic systemsProven hands-on experience designing multi-layer evaluation suites with deterministic and LLM-based gradersExpertise with CI/CD pipelines and implementing metric-based quality gates for automated deploymentsPractical knowledge of LangGraph or similar agent orchestration frameworksStrong background in designing simulation-based evaluation strategies and conversation-level testsNice to haveExperience with AWS AgentCore Evaluations API (CreateEvaluation, custom evaluators)Familiarity with shadow-mode, canary, or A/B deployment practices for ML-based platformsBackground in transforming production failures into build-time regression tests for agent workflowsWe offerCompetitive compensation depending on experience and skillsVariety of projects within one companyBeing a part of a project following engineering excellence standardsIndividual career path and professional growth opportunitiesInternal events and communitiesFlexible work hours
-
braga, braga, Portugal EPAM Systems Tempo integralEPAM Systems, Inc. is seeking an Agent Evaluation Engineer to build evaluation frameworks for AI agents, combining deterministic and LLM-based grading with CI/CD deployment gates.You will design multi-layer evaluation suites, simulate multi-turn conversations, and establish reliability metrics, including metrics and staging validations. The role is fully...
-
Agent Evaluation Engineer — Build-Time Framework
Há 18 horas
braga, braga, Portugal EPAM Systems Tempo integralWe're looking for an Agent Evaluation Engineer — Build-Time Framework & Deployment Gates to join our team in Portugal in a fully remote working mode. In this role, you will design and maintain an evaluation framework for AI agents, ensuring quality and compliance through automated tests and CI/CD deployment gates. You will develop multi-layer evaluation...
-
Braga, Braga, Portugal EPAM Systems, Inc. Tempo integral 70 000 € - 100 000 € ContratoEPAM Systems, Inc. is seeking an Agent Evaluation Engineer to build evaluation frameworks for AI agents, combining deterministic and LLM-based grading with CI/CD deployment gates. You will design multi-layer evaluation suites, simulate multi-turn conversations, and establish reliability metrics, including pass@k metrics and staging validations. The role is...
-
Agent Evaluation Engineer — Build-Time Framework
Há 2 semanas
Braga, Braga, Portugal Epam Systems, Inc. Tempo integralWe're looking for an Agent Evaluation Engineer — Build-Time Framework & Deployment Gates to join our team in Portugal in a fully remote working mode.In this role, you will design and maintain an evaluation framework for AI agents, ensuring quality and compliance through automated tests and CI/CD deployment gates.You will develop multi-layer evaluation...
-
Senior ML
Há 18 horas
braga, braga, Portugal EPAM Systems Tempo integralWe're looking for a Senior ML / Evaluation Engineer to join our team in Portugal in a fully remote working mode. In this role, you will own the design and implementation of advanced evaluation frameworks for an Enterprise Agent Development Platform—a production-grade, cloud-native ecosystem enabling scalable, secure AI agent deployment. You will create...
-
AI Agent Evaluation Analyst
Há 3 dias
Braga, Braga, Portugal Mindrift Tempo integral 31 180 € - 44 408 € Contrato1 week ago Be among the first 25 applicants Get AI-powered advice on this job and more exclusive features. This opportunity is only for candidates currently residing in the specified country. Your location may affect eligibility and rates. Please submit your resume in English and indicate your level of English proficiency. At Mindrift, innovation meets...
-
Remote Lead Engineer, AI Quality
Há 1 dia
Braga, Braga, Portugal Circle Internet Financial Tempo integral 107 000 € - 161 000 € ContratoCircle Internet Financial is hiring a Lead Engineer for the AI Quality engineering team. You will build evaluation frameworks, observability tooling, and diagnostic infrastructure to ensure AI agents perform reliably and cost-efficiently. This is a hands-on, player-coach role with leadership responsibilities. You’ll lead the team, set technical direction,...
-
Braga, Braga, Portugal Siemens Tempo integral 134 000 € - 187 000 € ContratoCircle is building the world’s leading AI-powered, all-in-one platform for digital businesses, with a fully remote team and a strong focus on high-impact AI quality engineering. We are seeking a Lead Engineer to build evaluation frameworks, observability tooling, and diagnostic infrastructure for AI agents, while growing and leading the AI Quality...
-
Braga, Braga, Portugal Mindrift Tempo integral 29 529 € - 35 435 € ContratoThis opportunity is only for candidates currently residing in the specified country. Your location may affect eligibility and rates. Please submit your resume in English and indicate your level of English proficiency. At Mindrift, innovation meets opportunity. We believe in using the power of collective human intelligence to ethically shape the future of...
-
Remote Lead Engineer
Há 1 dia
Braga, Braga, Portugal CIRCLE Tempo integral 90 000 € - 150 000 € ContratoCircle is seeking a Lead Engineer for AI Quality to build evaluation frameworks, observability tooling, and diagnostic infrastructure for its AI agents. This hands-on coach role also leads the AI Quality engineering team and collaborates with AI Core engineering to deliver reliable, scalable AI features. You will design datasets, run experiments across...
