• 

  Google open-sources its robots.txt parser and submits a proposal to IETF to make the Robots Exclusion Protocol an official standard (Paul Sawers/VentureBeat)

  

  techmeme.com

  Paul Sawers / VentureBeat: Google open-sources its robots.txt parser and submits a proposal to IETF to make the Robots Exclusion Protocol an official standard  —  Google wants to turn the decades-old Robots Exclusion Protocol (REP) into an official internet standard — and it's making its own robots.txt parser open source as part of the push.

  • Add  

  1

More like this (1)

• 

  Google wants to make the 25-year-old robots.txt protocol an internet standard

  

  thenextweb.com

  Google Wants to Make The 25-Year-Old robots.txt Protocol An Internet Standard — Google has outlined plans...

  Google Wants to Make The 25-Year-Old robots.txt Protocol An Internet Standard — Google has outlined plans to turn robots exclusion protocol (REP) — better known as robots.txt — into an internet standard after 25 years. Here’s the official blog post announcement from Google.

  • Add  

  1