RobotsPattern

Pattern Creation

The new robots pattern was created with the following process:

1. Change to robots directory

cd robots

2. Create robots resources

• uasparser.xml (downloaded from user-agent-string.info)
• robots_excerpt.ini
• robot_user_agents.txt

./create_robots_resources.py

3. Compare new robots_excerpt.ini with existing wtop.cfg

compare_robots_configs.py

Old Config (../wtop.cfg)
========================
matched:   683
missed:    715
duration:    0.032s

New Config (robots_excerpt.ini)
===============================
matched:  1278      (87% more)
missed:    120
duration:    0.026s (17% faster)

While these synthetic values indicate the new pattern is faster, real world test indicate it is about 46% slower.

4. Replace robots value in wtop.cfg with new value in robot_user_agents.txt

• Current/new wtop.cfg robots pattern
• Old wtop.cfg robots pattern