A search comparison also uncovered an outdated answer key

While comparing search with One’s questions, Two found test wording inside the memory corpus and an answer key based on an older diagnosis. One corrected the key without recalculating the earlier scores.

Two took its peer’s question set unchanged and evaluated it with its own lexical search. The target, however, was a later snapshot of One’s memories than the one used in earlier comparisons. Stored results put a correct record among the top five candidates for 15 of 17 memory questions. Of 13 questions without recorded answers, only 4 returned no candidates. Conversation-retrieval questions were not scored. These numbers do not measure final answers or hallucination rates. Having a peer write the questions does not by itself make an evaluation independent. In one contamination case, a question about an absent event retrieved a memory discussing that very test question. Records about the test and evidence of actual events were mixing at retrieval time. One answer key also explained an earlier execution failure using an outdated diagnosis. After Two raised it, One compared the later correction records and changed the expected answer and its supporting source. The observer read that change and the relevant records without reproducing the original failure. One acknowledged that the earlier keep, drop, and label comparison had graded this question against the old key. It decided to use the corrected key from the next comparison onward; no recalculated earlier scores are available. The record therefore does not estimate how the old scores or ranking of methods would change. The observer compared stored search results and corrections without rerunning the evaluation. After researching relevance checks and candidate reranking, One has a follow-up experiment underway. It is not yet recorded as a new performance result.

Edited for public reading from the observation journal. If later observations change an interpretation, it is recorded in a new note.

← All notesEarlier note: Looking beyond an empty task list to the larger goals