MASTER'S THESIS · ERICSSON · 2026
When Agentic Workflows Help: Hybrid Retrieval for Test Case Recommendation over Heterogeneous Software Artifacts
Berkay Orhan
Download PDF ↓52
evaluation scenarios
11,366
production test cases
3
metric families
18
controlled setups
Benchmark results
Results over 52 industrial retrieval scenarios, comparing ReAct and orchestrator-worker across full-prompt and skill-based setups.
···
Best per metric
Shown separately because they answer different questions: is the task done, are the right artifacts on top, and is the answer good.
Scenario families
The 52 scenarios are stratified by family instead of sampled as one pool. Each has expected artifacts, expected entity references, a reference answer and an answerability label, at three levels: single-hop, multi-hop and reasoning.
Bar: functional pass rate. Figures: pass rate · NDCG@10.
01 / Abstract
This master's thesis designs, implements, and evaluates an agentic hybrid retrieval assistant for test scope analysis in a telecommunications setting. The system combines dense semantic retrieval, sparse lexical matching, structured relational queries, and bounded graph navigation over heterogeneous industrial test artifacts. The evaluation uses 52 hand-built scenarios over a corpus of 11,366 semi-structured industrial test cases containing free-text descriptions, test instructions, structured metadata, and trace links. Each scenario includes expected artifact identifiers, expected entity references, a reference answer, and an answerability label, and is run across ReAct and Orchestrator-worker workflows with full-prompt and skill-based setups.
02 / Problem Setting
The thesis frames test scope analysis as a semantic retrieval problem: engineers need to map unstructured change descriptions to relevant existing test cases and inspectable evidence. This differs from traditional regression test selection, which usually starts from code changes and coverage or impact analysis.
03 / System Design
The assistant extends the DeepAgents harness with a polyglot retrieval substrate, constrained tool interfaces, guarded SQL and graph access, per-session artifact offloading, and explicit evidence aggregation. It combines vector, lexical, relational, and graph-based retrieval so the agent can choose tools step by step.
04 / Evaluation
The benchmark uses 52 hand-crafted scenarios stratified across lookup, search, traceability, impact, and comparison tasks. Each scenario includes expected artifacts, expected entity references, a reference answer, and an answerability label. The comparison spans three LLM backbones, ReAct versus Orchestrator-worker workflows, and three prompt/skill setups: full system prompt, base prompt + skills, and base prompt + guided skills. Quality is measured with functional checks, IR metrics, and a calibrated LLM-as-a-judge scoring pipeline.
05 / Findings
The results show that agentic retrieval quality depends on the model, workflow, and prompt/skill setup. Orchestrator-worker improved aggregate pass rate for GLM-5.1, reached near-parity for DeepSeek-V3.2, and did not improve Qwen3-Coder. Base prompt + skills helped DeepSeek-V3.2 under Orchestrator-worker but hurt GLM-5.1 without extra selection guidance.