Test/uv 16309 span flush race + fix - #1901

This PR addresses intermittent missing tool-call spans in AgentRunHistory by flushing OpenTelemetry spans before collecting execution data.

Changes:

• Adds OpenTelemetry race reproduction tests.
• Flushes spans before reading and clearing execution data.

Suppressed comments (3)

packages/uipath/src/uipath/eval/runtime/runtime.py:979

• This flush is only a snapshot barrier: BatchSpanProcessor.force_flush() drains spans already queued when it runs. If the detached task described above ends after this call, its on_end() queues the span after the flush, and the following clear() can discard the execution's data without another flush. Please synchronize with or await execution-owned background tasks (or otherwise wait for their spans) before reading and clearing; this line alone does not close that race.

        self.trace_manager.flush_spans()

packages/uipath/tests/cli/eval/test_execution_span_race.py:97

• These tests call processor.force_flush() directly and never exercise _get_and_clear_execution_data(), so they would still pass if the new self.trace_manager.flush_spans() line were removed. Add a test through UiPathEvalRuntime (or a focused test of this method) that verifies the production read/clear path flushes the processor before get_spans().

    processor.force_flush()

    spans_at_read_time = exporter.get_spans(EXECUTION_ID)
    assert {s.name for s in spans_at_read_time} == {"root", "tool_call"}

packages/uipath/tests/cli/eval/test_execution_span_race.py:123

• input.value is an OpenTelemetry span attribute, and dict values are not valid attribute values; the SDK drops this field and logs an invalid-attribute warning. That makes the real-OTel reproduction noisy and loses the tool arguments from the history. Encode the value as a string, as the other evaluator span tests do.

            "input.value": {"query": "uipath"},

• Files reviewed: 2/2 changed files
• Comments generated: 2
• Review effort level: Lite

Suppressed comments (4)

packages/uipath/src/uipath/eval/evaluators/legacy_trajectory_evaluator.py:152

• WorkloadExecution.workload_output explicitly allows dictionaries, so {} is a valid falsey final output. This truthiness check omits it from the legacy prompt, leaving a tool-free run without the final response section. Check against None instead of truthiness to preserve valid falsey outputs.

        if workload_output:

packages/uipath/src/uipath/eval/evaluators/legacy_trajectory_evaluator.py:156

• As in the LLM-judge evaluator, this appends workload_output whenever it is truthy, including when the rendered history already contains tool calls. The stated fix is a no-tool fallback, so this silently changes existing legacy trajectory prompts and may duplicate or inflate them for normal tool runs. Please either restrict the append to empty rendered history or clarify and cover the intentional broader contract.

        if workload_output:
            final_output_section = f"Agent Final Response:\n{workload_output}"
            agent_run_history = (
                f"{agent_run_history}\n\n{final_output_section}"
                if agent_run_history

packages/uipath/src/uipath/eval/evaluators/llm_judge_trajectory_evaluator.py:93

• WorkloadExecution.workload_output explicitly allows dictionaries, so {} is a valid falsey final output. In the no-tool case this branch skips it and _get_actual_output() still returns an empty string, leaving that valid output absent from the judge prompt. Use is not None so the optional parameter's None sentinel is distinguished from a falsey result.

        if final_output:

packages/uipath/src/uipath/eval/evaluators/llm_judge_trajectory_evaluator.py:97

• The PR description scopes this change as a fallback for runs with no tool-call spans, but this condition appends workload_output even when history already contains tool calls. That changes every mixed/tool trajectory prompt (and can duplicate or substantially expand the input) rather than only fixing the empty-history case. Please either gate this on an empty rendered history or explicitly document and test the broader behavior; the correct choice is ambiguous from the stated scope.

        if final_output:
            final_output_section = f"Agent Final Response:\n{final_output}"
            history = (
                f"{history}\n\n{final_output_section}"
                if history

• Files reviewed: 5/5 changed files
• Comments generated: 1
• Review effort level: Lite

Suppressed comments (1)

packages/uipath/tests/cli/eval/test_execution_span_race.py:200

• This test creates a UiPathTraceManager, which registers a default BatchSpanProcessor and retains both processors in the global delegating processor, but it never shuts them down. The default worker thread therefore remains alive and the old processors stay attached to subsequent tests/jobs, potentially receiving and queuing unrelated spans. Wrap the test body in try/finally and call trace_manager.shutdown() so the processors are unregistered and their workers are stopped.

    trace_manager = UiPathTraceManager()
    span_exporter = ExecutionSpanExporter()
    span_collector = ExecutionSpanCollector()
    span_processor = ExecutionSpanProcessor(span_exporter, span_collector)
    _pin_batch_schedule(span_processor)
    trace_manager.add_span_processor(span_processor)

• Files reviewed: 5/5 changed files
• Comments generated: 0 new
• Review effort level: Lite

• Files reviewed: 5/5 changed files
• Comments generated: 1
• Review effort level: Lite