CCF AIOps 2025 Challenge - LLM-Based Microservice Root Cause Analysis

CCF AIOps 2025 挑战赛 - 基于大模型的微服务根因分析

---

Competition Result / 比赛成绩: Semi-final 13th Place / 复赛第13名

Overview / 概述

[EN] This repository contains the solution for the 2025 CCF AIOps Challenge, which focuses on root cause analysis (RCA) in microservice systems. The task requires identifying the faulty component (pod, service, or node) when an anomaly occurs, using multi-modal observability data including metrics, logs, and traces.

Our approach leverages a multi-agent LLM pipeline where specialized agents analyze different data sources in parallel and a final "Judge Agent" synthesizes their findings to produce a structured root cause diagnosis.

[CN] 本仓库包含 2025 CCF AIOps 挑战赛 的参赛方案代码。赛题聚焦于微服务系统的根因分析（RCA），要求在异常发生时，利用多模态可观测数据（指标、日志、链路追踪）定位故障组件（Pod、服务或节点）。

我们的方案采用 多Agent大模型流水线，由专门的 Agent 并行分析不同数据源，最终由"决策 Agent"综合所有发现，生成结构化的根因诊断结果。

Architecture / 架构

                         ControllerAgent
                              |
              ┌───────────────┼───────────────┐
              v               v               v
         MetricAgent      LogAgent       TraceAgent
         (Stage 1)       (Stage 1)       (Stage 1)
              |               |               |
              └───────────────┼───────────────┘
                              v
                         JudgeAgent
                         (Stage 2)
                              |
                              v
                      RCAFormatterAgent
                         (Stage 3)

• Stage 1 - Data Summarization / 数据总结: Three specialized agents (MetricAgent, LogAgent, TraceAgent) analyze metrics, logs, and traces concurrently, each producing a structured summary of key anomalies.
• Stage 2 - Root Cause Judgment / 根因判断: The JudgeAgent receives all summaries and applies structured reasoning (Chain-of-Thought) following an SRE diagnostic playbook to determine the root cause.
• Stage 3 - Output Formatting / 输出格式化: The RCAFormatterAgent converts the detailed judgment into the competition's required submission format.

Project Structure / 项目结构

.
├── Dockerfile              # Docker build file
├── run.sh                  # One-click run script
├── domain.conf             # Domain configuration
├── src/
│   ├── main.py             # Entry point, orchestrates parallel analysis
│   ├── agent_new.py        # Agent implementations (Metric/Log/Trace/Judge/Formatter)
│   ├── llm_wrapper.py      # LLM API wrapper with retry logic
│   ├── prompts.py          # All prompt templates for each agent
│   └── requirements.txt    # Python dependencies
└── data/
    ├── input_time.json     # Anomaly event list with time windows
    ├── trace/              # Distributed tracing data
    │   ├── trace_raw/      # Raw trace call chains
    │   ├── trace_error/    # Traces with HTTP 4xx/5xx or gRPC errors
    │   ├── trace_delay/    # Traces with latency anomalies
    │   └── trace_3xx_error/# Traces with HTTP 3xx redirections
    ├── log/                # Log data
    │   ├── log_raw/        # Raw log entries
    │   ├── log_warning/    # Warning-level log templates
    │   └── log_error/      # Error-level log templates
    └── metric/             # Metric data
        ├── all_metric/     # Combined pod/node/service metrics
        ├── selected_apm/   # Pre-filtered APM metrics
        ├── selected_infra/ # Pre-filtered infrastructure metrics
        └── tidb/           # TiDB cluster metrics

Note / 注意: The data/ directory in this repository contains only sample files for demonstration. The full dataset is provided by the competition organizers.

本仓库 data/ 目录仅包含少量样例文件用于展示数据格式，完整数据集由比赛主办方提供。

Tech Stack / 技术栈

• Language / 语言: Python 3.12
• LLM: Qwen2.5-VL-72B-Instruct (via SiliconFlow API)
• Key Libraries / 主要依赖: openai, pandas, tqdm, tenacity
• Deployment / 部署: Docker

Quick Start / 快速开始

Prerequisites / 前置条件

• Docker installed and running / 已安装并运行 Docker
• An LLM API key (e.g., SiliconFlow) / 一个大模型 API Key

Steps / 步骤

1. Configure API Key / 配置 API Key

   Edit src/main.py and replace the placeholder with your API key:

   编辑 src/main.py，将占位符替换为你的 API Key：

   API_KEY = "your-api-key-here"
   API_BASE = "https://api.siliconflow.cn/"

2. Prepare Data / 准备数据

   Place the full competition dataset in the data/ directory following the structure above.

   将完整的比赛数据集放入 data/ 目录，按上述结构组织。

3. Run / 运行

   chmod +x run.sh
   ./run.sh

   The script builds a Docker image and runs the analysis. Results are saved to answer.jsonl.

   脚本会构建 Docker 镜像并运行分析，结果保存到 answer.jsonl。

Key Features / 主要特点

• Multi-Agent Parallel Analysis / 多Agent并行分析: Metric, log, and trace agents run concurrently for efficiency.
• Structured SRE Playbook / 结构化SRE诊断手册: The Judge Agent follows a priority-based diagnostic strategy (node -> TiDB -> service/pod) inspired by real SRE practices.
• Checkpoint Resume / 断点续传: Supports resuming from previous runs if interrupted.
• Robust JSON Parsing / 健壮的JSON解析: Multi-strategy extraction of JSON from LLM responses.
• Retry with Backoff / 指数退避重试: Built-in retry logic for LLM API calls and task-level retries.

Authors / 作者

• zhongwei
• wangxijing

License / 许可证

This project is for academic and research purposes.

本项目仅供学术研究使用。