About Searchr1 evaluation.

[qianyuli123]

I used the following script, but an error occurred.

set -ex

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" &> /dev/null && pwd)"
PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
MODEL_PATH=./SearchR1-nq_hotpotqa_train-qwen2.5-3b-em-grpo
DATA_DIR=./ASearcher-test-data

SPLIT=1
MAX_GEN_TOKENS=4097
DATA_NAMES=2WikiMultihopQA_rand1000,HotpotQA_rand1000,Bamboogle,Musique_rand1000,NQ_rand1000,TriviaQA_rand1000,PopQA_rand1000
AGENT_TYPE=search-r1
PROMPT_TYPE=local-rag
SEARCH_CLIENT_TYPE=async-search-access
temperature=0.6
top_p=0.95
top_k=-1

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7

TOKENIZERS_PARALLELISM=false \
PYTHONPATH="${PROJECT_ROOT}:$PYTHONPATH" \
python3 search_eval_async.py \
    --data_names ${DATA_NAMES} \
    --model_name_or_path ${MODEL_PATH}  \
    --output_dir "./output/searchr1/pass2" \
    --prompt_type $PROMPT_TYPE \
    --agent-type ${AGENT_TYPE} \
    --data_dir ${DATA_DIR} \
    --split test \
    --search-client-type ${SEARCH_CLIENT_TYPE} \
    --max-tokens-per-call ${MAX_GEN_TOKENS} \
    --tensor_parallel_size 4 \
    --n_sampling 1 \
    --temperature ${temperature} \
    --top_p $top_p \
    --top_k $top_k \
    --start 0 \
    --end -1 \
    --seed 1 \
    --parallel-mode seed \
    --use-jina \
    --llm_as_judge \
    --pass-at-k 2 \

[saptarshi059]

Were you able to fix it?

[saptarshi059]

Honestly, their code is crap. There's just so many things wrong with it - I honestly think they just made up their results at this point!