UBB.Developers Forums Thread Like Summary - [7.x] Robots (Crawlers) List for UBB.threads

Robots (Crawlers) List for UBB.threads - UPDATED 2022-07-10
formerly named "Search Engine Spiders List for UBB.threads"

About:
The advantage to using this list is that Robots (Crawlers and Search Engine Spiders) get put into the correct "Robots (Crawlers)" group when viewing your forum's Who's Online page.

This translation list is not used anywhere else in UBB.threads except on the Who's Online page at /ubbthreads.php/online. Having a long list of Robots will not slow down your forums.

Details:
Robots List: 2022-07-10
Total Agent Strings: 912
Sources:
https://udger.com/resources/ua-list/crawlers (2022-07-10) and
https://github.com/monperrus/crawler-user-agents (2022-05-03)
I've simply converted, combined, and cleaned the data for use with UBB.threads.

How to install:
1) Go to Control Panel > Display Options > Who's Online Settings.
2) Copy/paste the text from the newest list in to the "Robots (Crawlers)" box at the bottom of the page.
3) Click the "Update General Display Options" button.
done.

Notes:
Always use the newest list. Older lists contain old robots and Inactive crawlers.
Installs prior to 7.6.2 did not include a list of robot agents.
• robots_20141014-UBBT762.txt Fresh Installs of UBB.threads 7.6.2 to 7.7.3 are pre-populated with this list.
• robots_20200114-UBBT774.txt Fresh Installs of UBB.threads 7.7.4 are pre-populated with this list.
• robots_20200924-UBBT775.txt Fresh Installs of UBB.threads 7.7.5 are pre-populated with this list.
• robots_20220710-UBBT800.txt Fresh Installs of UBB.threads 8.0.0+ are pre-populated with this list.
• robots_20220710-CATEGORIES.txt is the same robots list as the stock list, plus it includes a category name for many of the robots, such as Search, Marketing, Monitoring, Link Checker, Tool, etc.

Having problems using this list on an older version of UBB.threads? Remove any blank lines from the top/bottom of your copied list.

Although the source information for you to create your own updates and conversions is in the OP, I plan on updating this post every couple of months, making your job as an UBBT forum admin much easier.

---
EDIT: user-agent-string.info no longer provides a list of user agent strings without a subscription. this means that until another resource for this data is found, this list will remain as it currently is.

This list has been updated 2014-10-14.

Some further reading regarding http/https referer data:
https://yoast.com/web-https/

SteveS, are you confident your Amazon stuff is not related caching of content within your site by AWS Cloud Computing /Route 53? https://aws.amazon.com/

Additional reading at:
https://en.wikipedia.org/wiki/Amazon_Route_53

The IP 72.21.217.n has been used as a proxy for a User Agent of MSIE-6, which is in itself highly deprecated. Headers can also be consistent with either a battened-down proxy or a bot.

Additional reading at:
"amazonaws.com plays host to wide variety of bad bots"
https://www.webmasterworld.com/search_engine_spiders/3828718.htm

Changelog 2020-01-14
The Robots (Crawlers) List for UBB.threads has been updated in OP

Changelog 2020-09-24
The Robots (Crawlers) List for UBB.threads has been updated in OP

This list has been updated 2022-07-09.