Hackathon (after a long time!): Portkey AI Builder Challenge

Table of Contents

Hackathon Presentation

The Problem Statement

There is huge value to be unlocked for enabling Enterprise agents for non-tech teams in a safe and optimal way. Most Enterprise teams waste budget on frontier models for tasks that don’t need them—GPT-4 for copy, Claude Opus for FAQs. Many Non-technical teams default to expensive models without the expertise to optimize. They need a smarter solution.

80% potential cost savings by using right-sized models.

Our Solution: Cost-Quality Optimization via Historical Replay

We built Intelligent Model Optimization Through Historical Analysis — a system that helps teams find the right model for each task by auto-validating on their traces (available via products like Portkey).

How it works:

1. Auto-Capture AI Traces — Integrate Portkey for prompt-completion logging
2. Configure LLM Judge & Guardrails — Define quality metrics & minimum thresholds
3. Benchmark LLM Alternatives — Test different models on your actual workload
4. Generate Recommendations — Receive data-driven guidance on optimal model selection

The system balances saving costs while maintaining quality. Imagine reports like these automatically generated for your agents :

Key Learnings

Enabling AI agents for non-tech teams is a massive opportunity. The key is balancing three parameters:

• Accuracy — Output must meet business standards
• Guardrails : Ensure the agents can be enabeld in a safe & secure way
• Cost — Budget constraints are real

Results

We won!