The robots META tag is similar to robots.txt, its placed in the head section of an HTML page.
In robots meta tag -

The “NAME” attribute is “ROBOTS”.
The “CONTENT” attribute are: “INDEX”, “NOINDEX”, “FOLLOW”, “NOFOLLOW”.

Do not index the content of a page, scan it for links to follow.
<META NAME=”ROBOTS” CONTENT=”NOINDEX, FOLLOW”>
Indexing content of a page […]

• 0 Comments

Sitemap consists of lists of URLs of a site with metadata (when it was last updated, how often it usually changes, and how important it is, relative to other URLs in the site) about each URL, which helps informing search engines about site pages that are available for crawling, so that search engines can […]

• 0 Comments

Web Robots (Crawlers, or Spiders) are programs that traverse the Web automatically. Search engines use them to index the web content.
Whenever search engine robotos wants to crawl the website, it looks for robots.txt which is the file where we write the instructions to the robots/crawlers about what it should crawl and […]

• 0 Comments