BI Analyst, Report Developer
- GTECH LLC
1 Hours Ago
30-48 per Hourly
W2, C2C, 1099
Malvern-PA
4-10 Years
Required Skills: SQL, AI
Job Description
Responsibilities
1. Deterministic Testing & Data Validation
- Validate generative AI tool outputs for structured, rules-based use cases by reconciling results against trusted data sources and established SQL-based metrics.
- Ensure consistency, explainability, and auditability of outputs by confirming alignment with existing data pipelines and query logic
- Expand and maintain test coverage across prioritized use cases to establish a robust, high-confidence baseline for the platform
- Partner with data engineering and analytics teams to identify and resolve discrepancies in underlying data or logic
2. Non-Deterministic Testing & Scenario Evaluation
- Design and execute scenario-based testing for more complex, AI-driven outputs where direct validation is not always possible
- Evaluate results based on intent accuracy, reasonableness, and confidence thresholds rather than exact match validation
- Prioritize testing across higher-risk and high-impact use cases using curated question sets and real-world scenarios
- Identify patterns in output variability and drive iterative refinement to improve reliability and user trust
3. Human-in-the-Loop Review & Continuous Monitoring
- Conduct ongoing review of generative AI tool interactions post-launch, validating outputs and ensuring quality across all user scenarios
- Identify edge cases, inconsistencies, and emerging risks, and escalate findings to product and engineering teams
- Synthesize insights from testing and live usage to inform enhancements, training data improvements, and governance practices
- Serve as an accountable reviewer, providing a critical control point for responsible AI deployment and continuous improvement
Qualifications
Required Skills & Experience
- Robust SQL skills required.
- Robust analytical background with experience in data validation, SQL, and analytics workflows
- Ability to assess outputs both quantitatively (data accuracy) and qualitatively (reasonableness, business context)
- Demonstrated critical thinking and sound judgment, especially in ambiguous or non-deterministic environments
- Experience working with large datasets, reporting tools, or analytics platforms
Preferred Qualifications
- Exposure to AI/ML or generative AI tools and associated testing or validation frameworks
- Experience in scenario-based testing, UAT, or model validation
- Familiarity with financial services, retirement, or plan sponsor analytics
