ment 7/2 '19 posted
0
Google, Robots.txt 사용을 위한 공식 표준 수립을 원한다

• 초안 제안은 "1994 년에 만들어진 규칙을 변경하지는 않지만 본질적으로 로봇에 대해 정의되지 않은 모든 시나리오를 정의합니다.
• 이것은 모든 URI 기반 전송 프로토콜이 로봇을 사용할 수 있음을 의미합니다.
• 또한 "개발자는 적어도 로봇의 처음 500 킬로바이트를 파싱해야 합니다.
• 사이트 소유자가 자신의 로봇을 업데이트할 수 있도록 24 시간 또는 캐싱 지시 값의 최대 캐싱 시간을 설정합니다.
• GitHub 저장소에 언급된 바와 같이, 더 중요한 것은,이 라이브러리는 오픈 소스로 출시되고 있으며, 개발자는 Google 로봇을 더 잘 반영하는 도구를 구축할 수 있습니다.


Google has declared it is open sourcing its robots.txt parser…
Google aims to standardise robots.txt, 25 years on [devclass.com]
Google has proposed an official internet standard for the rules included in robots.txt files.Those…
Google Wants to Establish an Official Standard for Using Robots.txt [www.searchenginejournal.com]
For 25 years, the Robots Exclusion Protocol (REP) has been one of the most basic and critical…
Formalizing the Robots Exclusion Protocol Specification [webmasters.googleblog.com]
Google wants to turn the decades-old Robots Exclusion Protocol (REP) into an official…
Google open-sources robots.txt parser in push to make Robots Exclusion Protocol an official standard [venturebeat.com]
Google announced this morning it has posted a Request for Comments to the Internet Engineering Task…
Google posts draft to formalize Robots Exclusion Protocol Specification [searchengineland.com]
Google's webmaster channel is on a series of posts every hour around the Robots Exclusion…
Google Works To Make Robots Exclusion Protocol An Official Standard [www.seroundtable.com]
Google Robots.txt Parser and Matcher Library The repository contains Google's robots.txt parser and…
google/robotstxt: The repository contains Google's robots.txt parser and matcher as a C++ library (compliant to C++11). [github.com]


newest repliespopular replies
Open Wiki - Feel free to edit it. -
7/2 '19 answered
0



permanent link
Open Wiki - Feel free to edit it. -
7/2 '19 answered
0



permanent link

Related

Featured

Important Notice: Discontinuation of Our Wiki Service

Dear Valued Users,

We hope this message finds you well. We are writing to inform you of an important upcoming change to our services. After careful consideration and analysis, we have made the difficult decision to discontinue our wiki site, effective one month from 2024/04/30.

This was not an easy decision to make, as we understand the importance of the resources and community that have been built around our wiki. However, due to evolving market conditions and strategic realignments, we must redirect our resources and efforts towards other ventures that will allow us to serve you better in the future.

What This Means for You:

  • Access to Content: You will continue to have full access to all content on the wiki until the discontinuation date. We encourage you to save or migrate any content you wish to retain.
  • Final Date of Service: The wiki will become inaccessible starting 1st June 2024, after which all content will be permanently removed.

We are immensely grateful for the support and contributions of our user community throughout the years. The wiki is being retired, but our team will be back soon with new services and apps.

Thank you for being a part of our journey.

Warm regards,

editoy Inc.