User-agent: * Disallow: /*? Disallow: /index.php/ Disallow: /catalog/product_compare/ Disallow: /catalog/category/view/ Disallow: /catalog/product/view/ Disallow: /wishlist/ Disallow: /admin/ Disallow: /catalogsearch/ Disallow: /checkout/ Disallow: /onestepcheckout/ Disallow: /customer/ Disallow: /review/product/ Disallow: /sendfriend/ Disallow: /enable-cookies/ Disallow: /LICENSE.txt Disallow: /LICENSE.html Disallow: /skin/ Disallow: /js/ Disallow: /directory/ #Lets consider each groups of commands separately. #Stop crawling user account and checkout pages by search engine robot: Disallow: /checkout/ Disallow: /onestepcheckout/ Disallow: /customer/ Disallow: /customer/account/ Disallow: /customer/account/login/ #Blocking native catalog and search pages: Disallow: /catalogsearch/ Disallow: /catalog/product_compare/ Disallow: /catalog/category/view/ Disallow: /catalog/product/view/ #Sometimes Webmasters block pages with filters.. Disallow: /*?dir* Disallow: /*?dir=desc Disallow: /*?dir=asc Disallow: /*?limit=all Disallow: /*?mode* #More reasonable to use canonical tag on these pages. #Blocking CMS directories. Disallow: /app/ Disallow: /bin/ Disallow: /dev/ Disallow: /lib/ Disallow: /phpserver/ Allow: /pub/media/catalog/product/ Disallow: /pub/ #These commands are not necessary. Search engines are smart enough to avoid including CMS files in their index. #Blocking duplicate content: Disallow: /tag/ Disallow: /rewiew/ #Don’t forget about domain and sitemap pointing: Host: www.wirisi.com Sitemap: https://www.wirisi.com/sitemap.xml