
ITエンジニア / 東京都
【JAPAN AI】AI Evaluation Scientist / English
ITエンジニアの求人(社名は面談時にご案内)
- 外国籍OK
- 土日休み
- 給与
- 年収 800万円〜1600万円
- 勤務地
- 東京都
- 雇用形態
- 正社員
- 仕事Role & Expectations]
- 時間10:00〜19:00
- 給与年収 800万円〜1600万円
この仕事について
Role & Expectations]
この求人のポイント
- 勤務地
- 東京都
- 年収の目安
- 初年度 800万円〜1600万円
- 転勤
- 求人票に記載なし
- 寮・社宅
- 求人票に記載なし
- この求人の条件
- 土日休み
勤務地エリアの情報

16445件東京都の掲載求人
1,280円東京都の最低賃金(時給)
仕事内容と労働条件
- 仕事内容
- [Role & Expectations]
As an AI Evaluation Scientist, you will lead the design, construction, and operation of the AI agent quality-evaluation infrastructure.
・Research and develop evaluation metrics — scientifically define "what constitutes quality" through LLM-as-Judge calibration, reward modeling, and benchmark design
・Design and build automated evaluation pipelines — integrate research outcomes into production CI/CD to deliver scalable quality gates
・Red teaming and safety verification — automate adversarial testing and build policy compliance verification frameworks
・Drive quality improvement through statistical experimental design — quantitatively verify the effectiveness of prompt strategies and model changes through A/B tests and significance testing
・Ensure the quality of products used in production by ~200 companies through a "science of quality" approach
[Job Description]
As an AI Evaluation Scientist, you will lead the design, construction, and operation of the AI agent Evaluation Infrastructure.
・Evaluation Metric Research & Development
・Research and implement LLM-as-Judge calibration methods (rubric design, bias detection, proper scoring rules)
・Design, build, and validate evaluation benchmarks (construct validity, contamination detection)
・Research the application of reward modeling / preference learning to evaluation
・Select and design evaluation metrics (win rate, task success, factuality, harm detection)
・Design, build, and maintain evaluation sets (synthetic data + real logs)
・Automated Evaluation Pipeline Design & Development
・Design and implement scalable automated evaluation pipelines
・Integrate evaluation pipelines into CI/CD and build quality gates
・Design agent evaluation harnesses (multi-turn, tool use, long-context support)
・Ensure reproducibility and reliability of evaluation pipelines
・Safety & Quality Verification
・Research and implement automated red teaming (automated adversarial testing)
・Build safety and policy compliance verification frameworks
・Research and implement hallucination detection and calibration methods
・Design and execute prompt / tool regression tests
・Statistical Analysis & Experimental Design
・Design and analyze statistical experiments (A/B tests, significance testing)
・Visualize quality trends and automate regression detection
・Create quality reports and improvement proposals
[Key Results (KR/Metrics)]
・Evaluation coverage rate (test case coverage)… - 応募資格
- [You May Be a Good Fit If You]
・Education & Experience
‐Master's degree or higher (or equivalent practical experience) in Computer Science, Machine Learning, Statistics, Mathematics, Physics, Psychometrics, or related fields
‐3+ years of practical experience as an ML Engineer, Data Scientist, Research Engineer, or in ML/AI evaluation-related roles
・Technical Skills
‐Deep knowledge of LLM / generative AI evaluation methods (benchmark design, LLM-as-Judge, quantitative output quality measurement, hallucination detection, etc.)
‐Practical knowledge of statistics and experimental design (hypothesis testing, A/B testing, confidence intervals, effect sizes, etc.)
‐Experience building ML / evaluation pipelines in Python
‐Practical experience with machine learning frameworks (PyTorch, JAX, TensorFlow, etc.)
‐Experience designing and implementing evaluation metrics (task-specific metric design beyond precision/recall)
・Language requirement (at least one of the following):
‐Japanese: Fluent — able to discuss product development without friction
‐English: Business level
This position is a research and development role responsible for AI output Evaluation Science. Research or implementation experience in ML model evaluation / LLM evaluation is required.
【歓迎】
[Strong Candidates May Also Have]
・Publication experience at top ML/NLP conferences (NeurIPS, ICML, ICLR, ACL, EMNLP, etc.)
・Research or implementation experience with reward modeling / preference learning (RLHF, DPO, etc.)
・Experience with LLM-as-Judge calibration and rubric design
・Knowledge or experience in AI safety, Responsible AI, and red teaming
・Experience with benchmark design and validity verification (IRT, construct validity)
・Experience evaluating multi-agent workflows, tool use, and long-context scenarios
・Large-scale data processing experience (Spark / BigQuery, etc.)
・Experience integrating ML / evaluation pipelines into CI/CD
・Ability to read, comprehend, and reproduce research papers
・Technical communication ability in English - 就業場所
- 東京都 新宿区
Hybrid work : 3 days in office, 2 days remote
Flexible working hours : Core time is negotiable
Flexibility : Future consideration for more flexible work styles is possible - 就業時間・休憩
- 10:00〜19:00
- 休日・休暇
- 土日祝休み
年間休日120日
年末年始休暇・有給休暇・夏季休暇・慶弔休暇
完全週休二日制
所定休日:土・日・祝日
休暇:年次有給休暇、夏季休暇(3日)、年末年始休暇(12月31日〜1月3日)、慶弔休暇 - 賃金
- Monthly: ¥571,429~¥1,142,857 (incl. 45h fixed overtime)
Stock options available
Reviews & bonuses: twice/year
OT beyond 45h paid separately
Negotiable based on experience and skills
■賞与
2回 - 待遇・福利厚生
- 社会保険完備・交通費支給・住宅手当・健康診断
[AI Tool Usage Support]
Company covers the cost of using AI tools such as JAPAN AI SaaS services, Cursor, ChatGPT, Claude, etc.
Development Tool Support
If a desired development tool is paid, the cost is covered (up to ¥30,000 per year)
[Book Purchase Assistance]
Company covers the cost of purchasing books for learning, such as technical books (up to ¥30,000 per half-year)
Language Learning / Qualification Support
Company covers the cost of Japanese or English learning programs and qualification acquisition
[Refresh Allowance]
Company covers the cost of services used for personal refreshment (up to ¥5,000 per month)
e.g., gym, yoga, chiropractic, aquarium, movies, theme park tickets, etc.
[Housing Allowance]
Housing allowance provided for those living in designated areas (up to ¥30,000 per month) - 受動喫煙対策
- 禁煙
- 選考・応募書類
- 応募 → Periodとの面談 → 企業の選考(1. Application Review 2. Coding Assessment 3. Interviews (4–5 rounds) 4. Offer )
- 募集者
- 社名は面談時にお伝えします
- 取扱い
- 株式会社Period(有料職業紹介事業 13-ユ-315555)
- 掲載日
- 2026年10月21日
記載の労働条件は掲載時点のものです。詳しくは面談・面接の際にご確認ください。
選考の流れ
- 応募(このページのフォーム)
- Periodとの面談(オンライン・日本語または英語)
- 企業との面接
- 内定・入社の手続き
同じエリア・職種の求人
- PM(WEBアプリ開発) / 年収850万円から / 完全週休2日制東京都 / 年収 850万円〜1500万円 / 正社員
- 【未経験可・将来的にはPM / PMOへ】年休129日 / 賞与2回東京都 / 年収 300万円〜450万円 / 正社員
- 【東京)週2リモート可フロントエンドエンジニア(テックリード / マネージャー候補) / 自社メディア開発東京都 / 年収 740万円〜950万円 / 正社員
- 【Engineer】ソフトウェアエンジニア(テックリード/オープンポジション)東京都 / 年収 850万円〜1050万円 / 正社員
- 【Engineer】SRE_テックリード(Embedded SRE)東京都 / 年収 1000万円〜1450万円 / 正社員
- 【Engineer】エンジニアリングマネージャー_施設プラットフォーム領域東京都 / 年収 900万円〜1400万円 / 正社員
APPLY
この仕事に応募する
送信後、1営業日以内にPeriodからご連絡します。日本語・英語・ベトナム語どれでも大丈夫です。
応募する求人【JAPAN AI】AI Evaluation Scientist / EnglishITエンジニアの求人(社名は面談時にご案内)(ID: pc527920)
