Empregos de remote agent evaluation engineer build-time framework em Aveiro

EPAM Systems, Inc. is seeking an Agent Evaluation Engineer to build evaluation frameworks for AI agents, combining deterministic and LLM-based grading with CI/CD deployment gates. You will design multi-layer evaluation suites, simulate multi-turn conversations, and establish reliability metrics, including pass@k metrics and staging validations. The role is...

EPAM Systems in Portugal seeks an Agent Evaluation Engineer to design and maintain a build-time evaluation framework for AI agents. You will implement automated tests, staging validations, and deployment gates integrated with production feedback to ensure robustness. You will develop multi-layer evaluation suites blending deterministic checks with...

We're looking for an Agent Evaluation Engineer — Build-Time Framework & Deployment Gates to join our team in Portugal in a fully remote working mode. In this role, you will design and maintain an evaluation framework for AI agents, ensuring quality and compliance through automated tests and CI/CD deployment gates. You will develop multi-layer evaluation...

Jobgether is seeking an AI Evaluation Engineer to design evaluation, testing, and observability infrastructure for production-grade AI agents. You will create scalable evaluation pipelines, automate tests, and implement robust regression and adversarial scenarios, all in a fully remote, globally distributed setting. You will own the evaluation architecture,...

An innovative AI company in Lisbon seeks part-time contributors for Quality Assurance roles focused on AI agents. Ideal applicants will have strong analytical thinking, attention to detail, and the ability to communicate findings in English. This freelance opportunity allows you to work flexibly on exciting AI projects while contributing to the future of...

This position is listed on behalf of a partner company, which manages all applications and next steps. Our partner is looking for an AI Evaluation Engineer based in Portugal. As a founding AI Evaluation Engineer, you will design and build the evaluation, testing, and observability infrastructure that ensures production-grade AI agents perform reliably,...

1 week ago Be among the first 25 applicantsGet AI-powered advice on this job and more exclusive features.This opportunity is only for candidates currently residing in the specified country. Your location may affect eligibility and rates. Please submit your resume in English and indicate your level of English proficiency.At Mindrift, innovation meets...

Circle Internet Financial is hiring a Lead Engineer for the AI Quality engineering team. You will build evaluation frameworks, observability tooling, and diagnostic infrastructure to ensure AI agents perform reliably and cost-efficiently. This is a hands-on, player-coach role with leadership responsibilities. You’ll lead the team, set technical direction,...

We're looking for a Senior ML / Evaluation Engineer to join our team in Portugal in a fully remote working mode. In this role, you will own the design and implementation of advanced evaluation frameworks for an Enterprise Agent Development Platform—a production-grade, cloud-native ecosystem enabling scalable, secure AI agent deployment. You will create...

EPAM Systems, Inc. is seeking a Senior ML / Evaluation Engineer to design and implement advanced evaluation frameworks for an Enterprise Agent Development Platform. You will build LLM-as-judge evaluators and deterministic validators to enforce quality before release. You will collaborate with platform, orchestration, and DevOps teams to ensure scalable,...

1 week ago Be among the first 25 applicants Get AI-powered advice on this job and more exclusive features. This opportunity is only for candidates currently residing in the specified country. Your location may affect eligibility and rates. Please submit your resume in English and indicate your level of English proficiency. At Mindrift, innovation meets...

Part-time AI company in Lisbon is looking for remote contributors for AI agent quality assurance roles. Requires analytical thinking, attention to detail, and English communication skills.

Circle is building the world’s leading AI-powered, all-in-one platform for digital businesses, with a fully remote team and a strong focus on high-impact AI quality engineering. We are seeking a Lead Engineer to build evaluation frameworks, observability tooling, and diagnostic infrastructure for AI agents, while growing and leading the AI Quality...

Seeking experienced developers to create and refine coding tests for AI systems. Requires Python and Full-Stack expertise, with a B2 English level. Part-time, project-based role.

Seeking experienced developers to create coding tests for AI systems. Requires Python, Full-Stack experience, and English proficiency. Project-based, part-time role.

YO AI Labs is seeking a highly skilled Full-Stack Software Engineer to support an AI training project focused on software development, debugging, code evaluation, and problem-solving. The role is fully remote, contractor-based, approx. 15 hours per week, with a flexible schedule. You will work across frontend and backend, build features, APIs, and data...

EPAM Systems in Portugal is seeking a Senior ML / Evaluation Engineer to own the design and implementation of advanced evaluation frameworks for a cloud-native AI agent platform. You will craft evaluation strategies, deploy CI/CD gates, and integrate LLM-as-judge and deterministic evaluators into automated pipelines. You will work with AWS AgentCore...

Remote Crew is seeking a Senior AI Engineer to own the intelligence layer of a compliant cloud orchestration platform. You will architect autonomous AI agents that operate across AWS, Azure, and GCP, turning black-box AI into predictable, type-safe enterprise tools. You'll build scalable microservices in Python and Go, apply LangChain/LangGraph, and...

EPAM Systems, Inc. is seeking an Observability Engineer — AgentCore Telemetry to own observability and telemetry for a cloud-native enterprise AI platform. You will design tracing, metrics, and logging solutions using OpenTelemetry and AWS services to ensure health, performance, and enterprise compliance. You will instrument Python-based AI runtimes,...

At Mindera, we believe in building software that improves lives. As a Senior AI Engineer for AI Systems you own the machinery that makes our accelerators reliable and reusable: the agent harness, orchestration, the platform-agnostic core, the evaluation infrastructure, and the observability that tells us whether a change made things better or worse. You are...