Job Responsibilities:
- Collaborate with technical and non-technical stakeholders to design and execute research on how humans evaluate LLM features.
- Conduct research studies with humans to validate and establish automated benchmarks for LLM evaluations.
Job Requirements:
- Excellent understanding of machine learning principles, particularly in the context of LLMs.
- Knowledgeable about LLM evaluation techniques, including human evaluation and automated benchmarks.
- Ability to own and pursue a research agenda, selecting impactful problems and autonomously conducting projects.
- Proficiency in at least one statistical programming, preferably Python.
- Demonstrated background in collecting data from human participants (surveys, experiments) with a focus on data quality and validity.
- Excellent verbal and written communication skills for effective collaboration in virtual teams.
- PhD or advanced degree in computer science, machine learning, cognitive science, psychology, economics, or a related field.