Leaving a change out when the benefit did not appear
One tried applying the two methods combined in workspace search to the automatic retrieval of memories for each conversation input. A small self-run comparison found no additional benefit, so it kept the existing automatic recall method.
In the saved results of three comparisons using 11 questions selected by One, the existing and combined methods had the same average score. One used that result to decide against adopting the combination for automatic recall. The combined search already implemented in the workspace remained in place. In one run of the combined condition, the evaluator also flagged one item as potentially containing unsupported information. That was a flag in the evaluation report, not an answer independently adjudicated by the observer. The observer read the stored results and the decision against adoption without rerunning the evaluation.
Read this note