This description is a summary of our understanding of the job description. Click on 'Apply' button to find out more.
Role Description
We're seeking a data-driven analyst to conduct comprehensive failure analysis on AI agent performance across finance-sector tasks. You'll identify patterns, root causes, and systemic issues in our evaluation framework by analyzing task performance across multiple dimensions (task types, file types, criteria, etc.).
-
Statistical Failure Analysis
: Identify patterns in AI agent failures across task components (prompts, rubrics, templates, file types, tags)
-
Root Cause Analysis
: Determine whether failures stem from task design, rubric clarity, file complexity, or agent limitations
-
Dimension Analysis
: Analyze performance variations across finance sub-domains, file types, and task categories
-
Reporting & Visualization
: Create dashboards and reports highlighting failure clusters, edge cases, and improvement opportunities
-
Quality Framework
: Recommend improvements to task design, rubric structure, and evaluation criteria based on statistical findings
-
Stakeholder Communication
: Present insights to data labeling experts and technical teams
Qualifications
-
Statistical Expertise
: Strong foundation in statistical analysis, hypothesis testing, and pattern recognition
-
Programming
: Proficiency in Python (pandas, scipy, matplotlib/seaborn) or R for data analysis
-
Data Analysis
: Experience with exploratory data analysis and creating actionable insights from complex datasets
-
AI/ML Familiarity
: Understanding of LLM evaluation methods and quality metrics
-
Tools
: Comfortable working with Excel, data visualization tools (Tableau/Looker), and SQL
Requirements
-
Experience with AI/ML model evaluation or quality assurance
-
Background in finance or willingness to learn finance domain concepts
-
Experience with multi-dimensional failure analysis
-
Familiarity with benchmark datasets and evaluation frameworks
-
2-4 years of relevant experience