glacial-lightweight System Requirements Document

1. Introduction

The "glacial-lightweight" project aims to develop a lightweight LLM operations monitor tailored for small AI companies. This tool will provide essential observability features without the complexity and cost associated with large enterprise platforms. It will focus on recording every prompt, model, and version while highlighting quality regressions, rising token costs, latency spikes, and failed evaluations.

2. System Overview

The glacial-lightweight system is designed to integrate with popular AI platforms such as OpenAI, Anthropic, Gemini, and Ollama. It will offer prompt-version comparison, golden test sets, cost and latency dashboards, basic human feedback mechanisms, and alert systems through Slack or email. Additionally, it will support exportable evaluation reports, aligning with the micro-SaaS pattern of providing niche analytics dashboards focused on key performance indicators.

3. Functional Requirements as Story Points

• As a User, I should be able to integrate the tool with OpenAI, Anthropic, Gemini, and Ollama.
• As a User, I should be able to compare different prompt versions.
• As a User, I should be able to create and manage golden test sets.
• As a User, I should be able to view cost and latency dashboards.
• As a User, I should be able to provide basic human feedback on model performance.
• As a User, I should receive Slack or email alerts for quality regressions, rising token costs, latency spikes, and failed evaluations.
• As a User, I should be able to export evaluation reports.

4. User Personas

• AI Developer: Responsible for integrating and monitoring AI models using the tool.
• Data Scientist: Analyzes model performance and provides feedback.
• Operations Manager: Oversees cost and latency metrics and manages alerts.
• Quality Assurance Specialist: Ensures the accuracy and reliability of model outputs.

5. Core User Flows

• AI Developer integrates the tool with AI platforms -> Configures prompt-version comparisons -> Sets up golden test sets.
• Data Scientist reviews cost and latency dashboards -> Provides feedback on model performance.
• Operations Manager receives alerts -> Analyzes cost and latency metrics -> Takes corrective actions.
• Quality Assurance Specialist exports evaluation reports -> Reviews for quality regressions and failed evaluations.

6. Visuals Colors and Theme

• primary: #1E90FF (Dodger Blue)
• primary_light: #63B8FF (Light Sky Blue)
• secondary: #FFD700 (Gold)
• accent: #FF4500 (Orange Red)
• highlight: #32CD32 (Lime Green)
• bg: #F0F8FF (Alice Blue)
• surface: #FFFFFF (White)
• text: #000000 (Black)
• text_muted: #696969 (Dim Gray)
• border: #D3D3D3 (Light Gray)

7. Signature Design Concept

Interactive Data Flow Visualization

The homepage will feature an interactive data flow visualization where users can see real-time data streams from different AI platforms. Each stream will be represented as a dynamic, flowing line that users can hover over to reveal detailed metrics such as prompt versions, cost, and latency. The visualization will use motion/react for smooth animations and transitions, creating an engaging and informative experience.

LANDING HERO MOTION BRIEF

The landing page will showcase a continuous loop of data streams converging into a central dashboard. As users hover over each stream, the dashboard will dynamically update to display specific metrics. This interaction will highlight the tool's ability to consolidate and analyze data from multiple sources, emphasizing its efficiency and focus on key performance indicators.

8. Interaction Model & Motion Direction

• Interaction Model: Animated
• The landing page will feature moderate scroll-triggered reveals and hover transitions, enhancing user engagement without overwhelming them with excessive motion.

9. Non-Functional Requirements

• The system must be scalable to accommodate growing data volumes from multiple AI platforms.
• The system should ensure data security and privacy, adhering to relevant regulations.
• The system must provide real-time updates and alerts with minimal latency.

10. Tech Stack

• Frontend: React for Web
• Backend: Python, FastAPI
• Database: MySQL or MariaDB
• AI Models: OpenAI, Anthropic, Gemini, Ollama
• AI Tools: Litellm, Langchain
• Local Orchestration: Docker, docker-compose
• Server-side Orchestration: Kubernetes

11. Assumptions and Constraints

• The tool will initially support integrations with OpenAI, Anthropic, Gemini, and Ollama.
• The system will focus on providing essential observability features without the complexity of full enterprise platforms.
• The tool will be designed for small AI companies with limited resources.

12. Glossary

• LLM: Large Language Model
• KPI: Key Performance Indicator
• Micro-SaaS: A small, focused software-as-a-service product that addresses a specific need or niche.