What Is A Google Crawl?
To start, let’s define what a Google crawl is and why that is so important to the business owner. To gather the information hosted all over the ‘world wide web’ for organized search results a search engine like Google must deploy software referred to as a spider (or a crawler or a bot). By using these spiders to find and review web pages (including listings, and citations), search engines are able to crawl, index, rank, and then serve this information in their’ SERPs or search engine result pages.
- “Crawling” = Google crawls to discover all website pages
- “Indexing” = Google learns what all pages are about, including the page title at an absolute minimum
- “Serving” = Google responds to a search with results including some advertisements and then website pages, rank order or position determined by various signals it counts
- Approximately 90% of all people looking for something online search, and approximately 86% still use Google to do it
Spiders consume pages, listings, citations, etc. into an index where it is all compared and ranked for the most accurate SERP (i.e. search engine result page) possible. Notice that we did not mention websites so much as we have pages, including listing & citations pages. Google wants to index & compare all web pages! Google applies rank to web pages for their keywords, NOT just to a website or a website’s home page and that’s important for a business to understand. Also notice we mentioned “indexes” and not index. It’s because there are different indexes for different contexts. For instance, a local service-oriented search will bring an index of results relevant to that geography.
It’s why we MUST prioritize the keywords that a prospective lead would search in our pages, posts, and all content creation. For the business wishing to be visible when people are actually looking for its products or services, the effort has to begin with content. Content that a search engine can easily identify, consume and organize into its SERPs!
Rebuilding The Google Index – The Business Opportunity!
According to Google, their spiders crawl the web regularly in order to rebuild their index of pages and related keywords. NOTE: Google says it constantly rebuilds its’ index, meaning that it is always looking for the most timely information it can find! This is important because many business owners think they can set up a website and be done with it. Wrong!
Search engines demand much more of the content they recommend in search engine result pages (SERPs). And because search engines are constantly rebuilding, the business owner must share the most current information and knowledge in order to remain relevant in these SERPs.
- Is mobile- friendliness important? Yes!
- Do inbound links matter? Some, when they are considered to be relevant to the subject and also trusted?!
- Does social activity matter? Some, particularly when it drives traffic out from the network to a website page!
- Is Google looking at how often a page is visited and how long users stay? Yes, because it is evidence the page is valuable to visitors.
- Does Google lose interest in pages? To underestimate the enormous amount of competition is foolish, so we must consider what signals confirm that “this website page remains worthy of a high ranking position for its keywords”!
In a competitive world for traffic and leads, search engine relevance is NO static exercise. Freshness matters and being timely with relevant content is certainly a competitive signal. Blogging about “what’s new”, for instance, can motivate a search engine to crawl one website more often than another that never adds new content. The more often a website is crawled, the more pages, posts, and keywords it will see consumed into the search engine indexes!
Google Crawl: Deep Spider Crawls and Fresh Spider Crawls
The search indexes are in constant motion (in fact, they are kinetic) and Google crawls the web at varying depths and on several different schedules. Some are believed to be deep crawls and other fresh crawls. It is believed that there are comprehensive crawls (the deep crawls), which occur on a per month basis, and then there are intermediate or random crawls (fresh crawls), which occur more often, but don’t go as deep or index as much.
Some folks refer to the constant motion of the indexes as “the Google Dance”, but the fact is new information is ALWAYS being indexed. Website owners can raise the chance of being crawled deeper and more often, by adding new content frequently. In other words, blog and update or add new useful content!
Google’s spider is capable of comparing previous per page crawls to the current visit at light speed. If it identifies new content often it will schedule more frequent visits. Think of it like a web presence is being judged on each visit. Why? Because it’s highly inefficient and expensive for Google to crawl a website and its pages over and over when nothing changes. NOTE: there is only so much crawl bandwidth and search engines only succeed when they satisfy their search traffic. In order to satisfy the search, a business must provide the best user experience possible, which includes the most timely and accurate information possible. ALSO NOTE: due to the “Google Dance” and the constant re-evaluation of website pages, there is never a guarantee a page will remain indexed in the same position from one day to the next… it means, in terms of rank or position, that pages will move up and down!
Motivate Google To Help Your Business: Update Your Pages & Blog!
In all, website pages will be re-evaluated over and over. Like anything else in life worth having, rank and position are earned. So a website needs to be current, consistent, and on-topic! And while certain things can help (i.e. page submissions into Google Search Console), focus the most on great new content that people will appreciate, use, link to, and share! WHY?: Because Google is looking for the best new content. Help Google help you: write often and write well!
See Also:
