Artificial Intelligence Engineer Job in India – BlueRose Technologies

BlueRose Technologies is hiring an Artificial Intelligence Engineer to test, evaluate and improve advanced AI agent systems. The role focuses on measuring the quality, reliability and performance of AI agents used in real-world applications.

The engineer will create evaluation datasets and multi-turn scenarios, analyze agent responses and identify areas where AI systems can be improved. The role also involves working with business documentation and complex data to create effective evaluation cases.

Key Responsibilities:

Test and evaluate AI agents and Generative AI systems.
Create evaluation datasets, Q&A sets and multi-turn conversations.
Measure tool-calling performance, accuracy and reliability.
Benchmark AI agent performance and identify issues.
Analyze business documents and complex datasets.
Improve AI agent quality and overall user outcomes.

Requirements:

3–5+ years of experience in AI/ML, Generative AI or related fields.
Strong Python programming skills.
Good understanding of APIs and system integrations.
Knowledge of LLMs and Agentic AI systems.
Experience with AI/LLM evaluation and benchmarking.
Strong analytical and problem-solving skills.
Ability to convert business requirements into useful AI evaluation scenarios.