|
Many robots also pass a special [[user-agent]] to the web server when fetching content.[{{cite web |url=http://www.user-agents.org/ |title=List of User-Agents (Spiders, Robots, Browser) |publisher=User-agents.org |access-date=2013-12-29 |archive-url=https://web.archive.org/web/20140107154205/http://user-agents.org/ |archive-date=2014-01-07 |url-status=live }}] A web administrator could also configure the server to automatically return failure (or [[Cloaking|pass alternative content]]) when it detects a connection using one of the robots.[{{cite web |url=https://httpd.apache.org/docs/2.2/howto/access.html |title=Access Control - Apache HTTP Server |publisher=Httpd.apache.org |access-date=2013-12-29 |archive-url=https://web.archive.org/web/20131229110831/http://httpd.apache.org/docs/2.2/howto/access.html |archive-date=2013-12-29 |url-status=live }}][{{cite web |url=http://www.iis.net/configreference/system.webserver/security/requestfiltering/filteringrules/filteringrule/denystrings |title=Deny Strings for Filtering Rules : The Official Microsoft IIS Site |publisher=Iis.net |date=2013-11-06 |access-date=2013-12-29 |archive-url=https://web.archive.org/web/20140101112730/http://www.iis.net/configreference/system.webserver/security/requestfiltering/filteringrules/filteringrule/denystrings |archive-date=2014-01-01 |url-status=live }}] |
|
Many robots also pass a special [[user-agent]] to the [[web server]] when fetching content.[{{cite web |url=http://www.user-agents.org/ |title=List of User-Agents (Spiders, Robots, Browser) |publisher=User-agents.org |access-date=2013-12-29 |archive-url=https://web.archive.org/web/20140107154205/http://user-agents.org/ |archive-date=2014-01-07 |url-status=live }}] A web administrator could also configure the server to automatically return failure (or [[Cloaking|pass alternative content]]) when it detects a connection using one of the robots.[{{cite web |url=https://httpd.apache.org/docs/2.2/howto/access.html |title=Access Control - Apache HTTP Server |publisher=Httpd.apache.org |access-date=2013-12-29 |archive-url=https://web.archive.org/web/20131229110831/http://httpd.apache.org/docs/2.2/howto/access.html |archive-date=2013-12-29 |url-status=live }}][{{cite web |url=http://www.iis.net/configreference/system.webserver/security/requestfiltering/filteringrules/filteringrule/denystrings |title=Deny Strings for Filtering Rules : The Official Microsoft IIS Site |publisher=Iis.net |date=2013-11-06 |access-date=2013-12-29 |archive-url=https://web.archive.org/web/20140101112730/http://www.iis.net/configreference/system.webserver/security/requestfiltering/filteringrules/filteringrule/denystrings |archive-date=2014-01-01 |url-status=live }}] |
|
Some sites, such as [[Google]], host a humans.txt file that displays information meant for humans to read.[{{Cite web |url=https://www.google.com/humans.txt |title=Google humans.txt |access-date=October 3, 2019 |archive-date=January 24, 2017 |archive-url=https://web.archive.org/web/20170124121422/https://www.google.com/humans.txt |url-status=live }}] Some sites such as [[GitHub]] redirect humans.txt to an ''About'' page.[{{Cite web |url=https://github.com/humans.txt |title=Github humans.txt |website=[[GitHub]] |access-date=October 3, 2019 |archive-date=May 30, 2016 |archive-url=https://web.archive.org/web/20160530160942/https://github.com/humans.txt |url-status=live }}] |
|
Some sites, such as [[Google]], host a humans.txt file that displays information meant for humans to read.[{{Cite web |url=https://www.google.com/humans.txt |title=Google humans.txt |access-date=October 3, 2019 |archive-date=January 24, 2017 |archive-url=https://web.archive.org/web/20170124121422/https://www.google.com/humans.txt |url-status=live }}] Some sites such as [[GitHub]] redirect humans.txt to an ''About'' page.[{{Cite web |url=https://github.com/humans.txt |title=Github humans.txt |website=[[GitHub]] |access-date=October 3, 2019 |archive-date=May 30, 2016 |archive-url=https://web.archive.org/web/20160530160942/https://github.com/humans.txt |url-status=live }}] |
Like
0
Dislike
0
Love
0
Funny
0
Angry
0
Sad
0
Wow
0
