Test Engineer-AI/LLM

4 days ago Be among the first 25 applicantsOPPO US Research Center is seeking a meticulous and innovative AI/LLM Test Engineer to join our cutting-edge AI team. In this critical role, you will evaluate the performance, reliability, and safety of Large Language Models (LLMs) in real-world product scenarios and test end-to-end generative AI solutions. Your work will directly shape how users experience AI-powered features by ensuring robustness, accuracy, and alignment with product goals. This is a unique opportunity to pioneer testing methodologies for next-generation AI systems at the forefront of technology.RequirementsCore Testing & EvaluationDesign and execute performance tests for LLMs across diverse product use cases (e.g., chatbots, content generation etc.)Develop automated test frameworks to evaluate LLM outputs for accuracy, bias, safety, and coherenceConduct end-to-end testing of integrated generative AI solutions, including APIs, data pipelines, and user interfacesOptimization & ValidationCollaborate with ML engineers to validate fine-tuned models and optimize prompts for target scenariosAnalyze model failures, edge cases, and adversarial inputs to identify risks and improvement areasBenchmark LLM performance against industry standards and product-specific KPIsCollaboration & Quality AssurancePartner with product, engineering, and research teams to define test requirements and acceptance criteriaDocument defects, performance metrics, and test results to drive data-driven improvementsAdvocate for AI ethics and safety through rigorous testing of fairness, bias mitigation, and content moderationInnovation & ToolingBuild scalable tools for synthetic test data generation, prompt variation testing, and automated evaluation workflowsStay current with advancements in generative AI testing, including red-teaming techniques and evaluation frameworks (e.g., HELM, Dynabench)Propose novel testing strategies for emerging challenges (e.g., hallucinations, context drift)Basic Qualifications:Bachelor's degree in Computer Science, Data Science, Engineering, or a related technical field, or equivalent practical experience1+ years of experience in software testing, data science, or ML validation, with exposure to AI/ML systemsProficiency in Python and testing frameworks (e.g., PyTest, Selenium)Hands-on experience evaluating LLMs in production environments (e.g., GPT, Claude, Llama, Gemini)Strong analytical skills for dissecting model behavior, statistical performance, and failure modesFamiliarity with cloud platforms (GCP, Azure, or AWS) and MLOps tooling (e.g., MLflow, Weights & Biases)Experience with version control (Git) and agile development methodologiesPreferred Qualifications:Master's degree in AI, Machine Learning, or a related fieldExpertise in prompt engineering, LLM fine-tuning (e.g., LoRA, RLHF), or optimization techniquesExperience with automated evaluation tools (e.g., LangChain, TruLens) or LLM-specific test suitesKnowledge of data pipelines, SQL/NoSQL databases, and API testing (e.g., Postman)Background in statistics, quantitative analysis, or data visualization for test insightsContributions to AI safety/ethics initiatives or open-source LLM evaluation projectsExperience testing mobile-integrated AI solutions (Android/iOS)BenefitsOPPO is proud to be an equal opportunity workplace. We are committed to equal employment opportunity regardless of race, color, ancestry, religion, sex, national origin, sexual orientation, age, citizenship, marital status, disability, gender identity or Veteran status. We also consider qualified applicants regardless of criminal histories, consistent with legal requirements.The US base salary range for this full-time position is $100,000-$200,000 + bonus + long term incentives benefits. Our salary ranges are determined by role, level, and location.Seniority levelSeniority levelMid-Senior levelEmployment typeEmployment typeFull-timeJob functionJob functionProduct ManagementIndustriesIT Services and IT ConsultingReferrals increase your chances of interviewing at OPPO by 2xGet notified about new Test Engineer jobs in Palo Alto, CA.Software Test Engineer, Pixel Cross-Device ExperiencesMountain View, CA $102,000 - $146,000 1 week agoPalo Alto, CA $140,000 - $170,000 8 hours agoCampbell, CA $178,000 - $190,000 2 days agoSan Jose, CA $100,500 - $173,250 2 months agoMenlo Park, CA $169,000 - $236,000 1 hour agoSunnyvale, CA $122,000 - $174,000 1 day agoWe’re unlocking community knowledge in a new way. Experts add insights directly into each article, started with the help of AI. #J-18808-Ljbffr

Created: 2025-09-17