How do you inform search engines that your website existed and inform them to index your website?

When i started working on Search Engine Optimization ( SEO ) for 3 ecommerce sites in 2006, this was the first question i had in mind.

Following are the ways of getting your website indexed by search engines -

• Submit site to dmoz -
  Go to dmoz - open directory project and submit site to dmoz , search engines (like google, yahoo etc.) get the data from dmoz - open directory and they crawl the websites.
• Submit site to google -
  Go to google search and submit site to google search engine, this will inform google to crawl your website.
• Submit site to yahoo -
  Go to yahoo search and submit site to yahoo search, this will inform yahoo to crawl your website
• Submit site to live -
  Go to live search and submit site to live search, this will inform live search to crawl your website
• Create sitemap and ping search engines
• Mention the sitemap path in robots.txt
  

Above links leads to the same page, but it differs with www.
But search engine treats both links are different, i have seen in few cases where we link many a times we ignore www. and in some cases we do include www. in the links.

So what are the impacts?

1. Search engines keep both the versions of the URLs, when people click on search engine results links which leads to our site with different versions of these URLs, it will drastically affect the page rank and traffic.
2. These URLs look like different documents to crawlers and create excessive crawling on our website.

How do we instruct search engine to treat both the URL’s as same, Google webmasters tool has a option to set the preferred domain

So whats the advantage of set preferred domain ? If i set my preferred domain as ganeshhs.com and next time if Google comes and crawls my website, and if it finds any link starting with www.ganeshhs.com it will follow it as ganeshhs.com and when Google displays my website links in search results it will show the links as ganeshhs.com

It also helps us to fix the external site referrals,  few guys started provide links to my website, if suppose their referral link is http://www.ganeshhs.com/category/search-engine-optimization-seo where as my actual article URL was http://ganeshhs.com/category/search-engine-optimization-seo and when google crawls our website through that referral link it will keep the right version of domain what we preferred.

In some cases, we come across situations where we keep links to external sites. But what are the impacts of this?

1. Part of page rank is shared to external website -
   When we link to other websites, our part of our website page rank is shared to
   those external sites, and we may end up sending the search engine crawlers to other side.
2. Leading Search Engine Crawlers to crawl external website -
   Crawler entered our website to crawl more pages, it will help us to have more indexes in Search Engines, but what did we end up keeping external links, we created a way to Crawler to leave our website and crawl the external websites.

We have to keep external links, but how do we prevent the above scenario -

When i started first project with zend framework may 2007, there were very few articles/tutorials and my first point of getting info was using search engine, then i realised it would be a great idea if my articles list in search engine and my first eye was on search engine optimization.

Looking at my website analytics i noticed that my recent posts on zend lucene search had more number of unique visits which also increased my website daily visits to average of 100 visits (with more unique visits), and also i started getting backlinks from other websites(namely http://www.phpimpact.com/ etc.) which also contributed for this page rank.

More essentially keywords(relevant to the context of the website/article) helps the articles to get indexed by search engines, following lists some of the blog articles and keywords i targeted and their stats in search engines Yahoo!/Google -

In the previous article Zend Lucene Search - part3 - retrieving the indexed data i talked about retrieving the search results. When we search, highlighting the searched keyword in the search result is one of the important aspect which most search engines follow, in this article i will be writing about highlighting the search results retrieved using the zend lucene search.

<?phprequire_once ‘Zend/Search/Lucene.php’;$queryStr= "php";

$query = Zend_Search_Lucene_Search_QueryParser::parse($queryStr);

$index = Zend_Search_Lucene::open("/var/www/lucene-data/blog-index");

$results = $index->find($query);

echo "Index contains ".$index->count()." documents.\n\n";

if($index->count())

{

$count = 0;

foreach ($results as $result)

{

$data[$count]["article_url"]         = $result->url;

$data[$count]["article_title"]        = $query->highlightMatches($result->title);

$data[$count]["article_description"]        = $query->highlightMatches($result->contents);

$data[$count]["article_created_date_time"]    = $result->postedDateTime;

$data[$count]["article_id"]             = $result->articleId;

$count++;

}

}

print_R($data);

?>

This program is same as in the Zend Lucene Search - part3 - retrieving the indexed data only one thing differs is now i am calling highlightMatches for the search results returned.

Related articles:
Zend Lucene Search - part1 - creating index
Zend Lucene Search - part2 - Real time indexing
Zend Lucene Search - part3 - retrieving the indexed data

<?php

require_once ‘Zend/Search/Lucene.php’;$query = "php";

$index = Zend_Search_Lucene::open("/var/www/lucene-data/blog-index");

$results = $index->find($query);

echo "Index contains ".$index->count()." documents.\n\n";

if($index->count())

{

$count = 0;

foreach ($results as $result)

{

$data[$count]["article_url"]         = $result->url;

$data[$count]["article_title"]        = $query->highlightMatches($result->title);

$data[$count]["article_description"]        = $query->highlightMatches($result->contents);

$data[$count]["article_created_date_time"]    = $result->postedDateTime;

$data[$count]["article_id"]             = $result->articleId;

$count++;

}

}

print_R($data);

?>

To retrieve the index data, first thing we need to do is to open the indexed path.

$index = Zend_Search_Lucene::open("/var/www/lucene-data/blog-index");

Suppose if user search input is -

$query = "php";

We have to use the find method of zend search lucene -

$results = $index->find($query);

To retrieve the total records resulted in the search result, we have to use count method of zend lucene search -

echo "Index contains ".$index->count()." documents.\n\n";

To limit the search result count we have to use setResultSetLimit of zend lucene search -

$index->setResultSetLimit(10);

Related articles:
Zend Lucene Search - part1 - creating index
Zend Lucene Search - part2 - Real time indexing
Zend Lucene Search - part4 - Search Results Highlighting

 <?phprequire_once 'Zend/Search/Lucene.php';$index = Zend_Search_Lucene::open("/var/www/lucene-data/blog-index");

$doc = new Zend_Search_Lucene_Document();

$doc->addField(Zend_Search_Lucene_Field::Keyword('url',

"http://ganeshhs.com/url-3"));

$doc->addField(Zend_Search_Lucene_Field::UnIndexed('articleId',

3));

$doc->addField(Zend_Search_Lucene_Field::UnIndexed('postedDateTime',

"20007-12-29 01:40:00"));

$doc->addField(Zend_Search_Lucene_Field::Text('title',

"Porting PHP to Javascript : php js"));

$doc->addField(Zend_Search_Lucene_Field::UnStored('contents',

"During graduation got interested in web technology, to kick start i started reading html, javascript."));

$doc->addField(Zend_Search_Lucene_Field::Text('category',

"Javascript"));

$index->addDocument($doc);

$index->commit();

$index->optimize();

Related articles:
Zend Lucene Search - part1 - creating index
Zend Lucene Search - part3 - retrieving the indexed data
Zend Lucene Search - part4 - Search Results Highlighting

Conventionally most of the site search are powered by database driven.

Lets consider my blog site, if anyone comes to my site and wants to search for any keyword, if i have to give search results i may have to look into articles table, comments table, executing SQL queries against 2 tables is acceptable, but if we go to any e-commerce application, we may have to search against lot of categories and products, since database queries are costlier, it consumes more resources. One more important point is we cannot get more relevant results first, in general we cannot rank the search results.

Apache Lucene is a high-performance, full-featured text search engine library written entirely in Java. This is being used in most of web2.0 websites. Zend_Search_Lucene was derived from the Apache Lucene project.

<?php//Index the blog articles

require_once 'Zend/Search/Lucene.php';

$articlesData =    array (0 => array( "url"           => "http://ganeshhs.com/url-1",

"title"	     => "Google suggest : pick right search keyword",

"contents"	 => "Picking the right keywords for the websites is the success of search engine marketing. When i started search engine optimization, i used to use overture keyword selector tool and check the search counts what other users have searched. "

"category"	     => "Google",

"postedDateTime" => "2007-12-26 12:20:00",

"articleId"            	     => 1),

1 => array( "url"           => "http://ganeshhs.com/url-2",

"title"	     => "zend framework tutorial | part 9 Zend Auth",

"contents"	 => "Zend Auth is easy to set up and provides a system that secures our site with an easy to use  authentication mechanism. Zend Auth(Zend_Auth) provides an API for authentication. "

"category"	     => "zend-framework",

"postedDateTime" => "2007-12-26 12:20:00",

"articleId"	     => 2));

if(is_array($articlesData) && count($articlesData))

{

$index = Zend_Search_Lucene::create('/var/www/lucene-data/blog-index');

foreach($articlesData as $articleData)

{

$doc = new Zend_Search_Lucene_Document();

$doc->addField(Zend_Search_Lucene_Field::Keyword('url',

$articleData["url"]));

$doc->addField(Zend_Search_Lucene_Field::UnIndexed('articleId',

$articleData["articleId"]));

$doc->addField(Zend_Search_Lucene_Field::UnIndexed('postedDateTime',

$articleData["postedDateTime"]));

$doc->addField(Zend_Search_Lucene_Field::Text('title',

$articleData["title"]));

$doc->addField(Zend_Search_Lucene_Field::UnStored('contents',

$articleData["contents"]));

$doc->addField(Zend_Search_Lucene_Field::Text('category',

$articleData["category"]));

echo "

Adding: ". $articleData["title"] ."\n";

$index->addDocument($doc);

}

$index->commit();

$index->optimize();

}

?>

$index = Zend_Search_Lucene::create(’/var/www/lucene-data/blog-index’);
Specifies the path of zend lucene index where the documents will be store.

For each iteration, we are creating a document-

$doc = new Zend_Search_Lucene_Document();

Once the document is created we need to add the fields and contents to the document -

Here since the URL is unique to the article we are indexing it as a Keyword field type.

we may need blog article id and blog create date time in the display part, it wont be used for search so we are storing it as UnIndexed field type.

Title is stored as text field type.

Content/Description is indexed but not stored in index. Because description occupies more space and creates a larger index on disk, so if we need to search but not redisplay the data, UnStored field type is preferred.

$doc->addField(Zend_Search_Lucene_Field::Keyword(’url’,

$articleData[”url”]));$doc->addField(Zend_Search_Lucene_Field::UnIndexed(’articleId’,

$articleData[”articleId”]));

$doc->addField(Zend_Search_Lucene_Field::UnIndexed(’postedDateTime’,

$articleData[”postedDateTime”]));

$doc->addField(Zend_Search_Lucene_Field::Text(’title’,

$articleData[”title”]));

$doc->addField(Zend_Search_Lucene_Field::UnStored(’contents’,

$articleData[”contents”]));

$doc->addField(Zend_Search_Lucene_Field::Text(’category’,

$articleData[”category”]));

Once the document is created and fields are added we need to add the document to the index -

$index->addDocument($doc);

After all the iterations we can commit the index-

$index->commit();

Following command is used to optimize the index -

$index->optimize();

Understanding Field Types -

Related articles:
Zend Lucene Search - part2 - Real time indexing
Zend Lucene Search - part3 - retrieving the indexed data
Zend Lucene Search - part4 - Search Results Highlighting

Recently came across a article about Google experiment, its a experiment any user can take, its aimed at improving the search experience.

Experiments allows us to join any one of following experiment at a time -

1. Alternate views for search results.
2. Keyword suggestions.
3. Keyboard shortcuts.
4. Left-hand search navigation.
5. Right-hand contextual search navigation.

This experiment with keyword suggestions is same as google suggest, i preferred taking that as my first experiment.

But when i started my career i was bit disappointed when i realised javascript won’t work if the javascript is disabled in the browser, so my senior in first company suggested me to always do server side validation using php.

But things changed rapidly, in the current firm we use lot of javascript/ajax applications, according to many surveys of most internet users do use javascript enabled browser.

Smarty is pretty interesting concept to bring liveliness to the static html, where we do dynamic programming in templates. Sometimes i always felt its always better if smarty has functions similar to php, and register_modifier opens up that option where we register php functions that are not available in smarty.

Having server side validation with client side validation is better approach, even the web tech giants like yahoo and google does the same, if you want to notice then try disabling javascript in your browser and open gmail.com or mail.yahoo.com it shows the basic edition of mail.

Well sometimes i came across writting few functions similar to PHP for email validation and URL validation. I always thought why can’t we use PHP in Javascript as much similar finding dynamic behavior in HTML(smarty and other template engines provides that), here is a PHP JS  which does the same, its a project aimed porting PHP to Javascript. It has lot of PHP functions being ported to Javascript version.