**Who we are**: Third Way Health helps medical practices across the United States improve the patient experience while reducing the administrative burden on practice owners. We enable practices to enhance the experience of their patients by providing them with a leading patient engagement platform and a world-class team of service representatives. What unites us is our passion for supporting independent physicians and helping patients from all backgrounds have a better healthcare experience. **About the position**: **Key Responsibilities**: - Collaborating with Product team members (e.g., product managers, software engineers, system administrators, analysts) to establish targets and processes for continuously improving system reliability, with a focus on voice interactions and AI-generated responses. - Embedding quality engineering principles, including defining data standards and acceptance criteria, to help “shift testing left” in the product development process. - Designing, executing, and documenting test plans, system monitoring, and data sync routines to ensure that our voice UI and LLM-based functionalities meet functional and non-functional requirements as internal and external software configurations change. - Executing structured test cases for voice-based interactions, ensuring accuracy, clarity, and consistency in AI-generated responses. - Evaluating voice UI performance across different languages, accents, and speech patterns to ensure accessibility and inclusivity for a diverse patient population. - Partnering with engineers and business owners to identify, triage, and address issues with AI-driven interactions, data quality, and software performance, and then implement improvements to preempt similar defects. - Contributing as an integral member of Agile teams, including daily stand-ups, sprint planning, and retrospectives. - Staying up to date with voice testing methodologies, NLP advancements, and security best practices for LLM evaluation, continuously working to enhance test coverage and system resilience. **Qualifications**: - Bachelor's degree in a relevant field such as Information Systems, Linguistics, or Human-Computer Interaction. - Experience working with voice UI testing tools and platforms, with a focus on evaluating speech generation and response quality. - Familiarity with LLM security testing, including methods for jailbreaking guardrails and adversarial testing of AI-generated responses. - Strong analytical skills, with the ability to evaluate voice-based interactions, identify sources of errors, and implement improvements. - Ability to work independently and as part of a team, with strong collaboration skills.