🔍 LangFuse Integration
🔍

LangFuse

Modern LLM Observability Platform - Real-time tracing, automatic cost tracking, user feedback, and A/B testing for your AI agents.

Automatic Token Tracking
Built-in Cost Calculation
User Feedback Collection
A/B Testing

🎯 Why LangFuse for SuperOptiX?

LangFuse is purpose-built for LLM applications, providing features that general ML platforms don't offer

Real-time Tracing

Detailed token usage and latency tracking

Cost Tracking

Automatic cost calculation per request

User Feedback

Built-in feedback collection and scoring

A/B Testing

LLM prompt and model comparison

Production LLM

Live monitoring and debugging

Real-time UI

Excellent dashboard experience

📦 Installation & Setup

1. Install LangFuse SDK

uv pip install langfuse

2. Start LangFuse Locally (Docker)

# docker-compose.yml
version: '3.8'
services:
  langfuse:
    image: langfuse/langfuse:latest
    container_name: langfuse
    ports:
      - "3000:3000"
    environment:
      - LANGFUSE_SECRET_KEY=your-secret-key
      - LANGFUSE_PUBLIC_KEY=your-public-key
      - LANGFUSE_HOST=http://localhost:3000
    restart: unless-stopped

# Start with: docker compose up -d

3. Configure Agent Playbook

observability:
  enabled: true
  backends:
    - langfuse
  langfuse:
    public_key: "pk-lf-your-public-key"
    secret_key: "sk-lf-your-secret-key"
    host: "http://localhost:3000"
    project: "superoptix-agents"
    tags:
      agent_type: "developer"
      tier: "genies"
      environment: "development"

🚀 Complete Workflow

1

Initialize Project

super init langfuse_demo
cd langfuse_demo
2

Pull Agent

super agent pull developer --tier genies
3

Start LangFuse

docker compose up -d
curl http://localhost:3000/api/public/health  # Verify running
4

Compile Agent

super agent compile developer
5

Run Agent with LangFuse

super agent run developer --goal "Write factorial function" --observe langfuse
6

View in LangFuse UI

# Open http://localhost:3000 in your browser

📊 What LangFuse Tracks

LLM Metrics

  • Input/output tokens per request
  • Real-time cost per request
  • Latency and throughput
  • Model and provider information
  • Full conversation history

Quality & Feedback

  • User feedback scores
  • Automated quality scoring
  • A/B test results
  • Error rates and debugging
  • Custom evaluation metrics

⚙️ Advanced Features

User Feedback Integration

observability:
  langfuse:
    feedback:
      enabled: true
      score_range: [1, 5]
      comment_enabled: true
      categories:
        - "accuracy"
        - "helpfulness"
        - "code_quality"

A/B Testing Configuration

observability:
  langfuse:
    ab_testing:
      enabled: true
      variants:
        - name: "baseline"
          config:
            temperature: 0.7
            model: "llama3.1:8b"
        - name: "optimized"
          config:
            temperature: 0.5
            model: "llama3.1:70b"

🎯 When to Use LangFuse

Choose LangFuse when:

  • You're building LLM applications
  • You need real-time cost tracking
  • You want user feedback integration
  • You're doing prompt engineering
  • You need A/B testing for LLMs
  • You want production LLM monitoring

Consider other options:

MLFlow

For traditional ML experiments, model versioning, artifact management

Weights & Biases

For research projects, beautiful visualizations, hyperparameter sweeps

SuperOptiX Native

For local development, no setup required, works offline

📚 Additional Resources

LangFuse Documentation

Official LangFuse docs

LangFuse Cloud

Hosted solution

SuperOptiX LangFuse Guide

Complete integration guide

MLFlow vs LangFuse

Comparison guide

🔍 Ready for LLM Observability?

Start using LangFuse with SuperOptiX for real-time cost tracking and A/B testing!