Loading...

Follow Portent | SEO Techniques Blog on Feedspot

Continue with Google
Continue with Facebook
or

Valid

Redesigning your website presents a significant risk from an SEO standpoint if not managed appropriately.

What can be years of optimizations to your infrastructure, design, and content are going to be overwritten, and it’s hard to predict if the organic rankings you’ve earned are going to be swept away. `

If your redesign strategy includes taking SEO into account post-launch, you’ll probably end up in for a world of hurt.

Here’s a look at organic traffic from a brand that came to us after launching a new beautifully designed site, wondering what happened to their SEO traffic.

Don’t be that website.

If your brand heavily relies on organic search to drive traffic, engagement, and conversion metrics, the risks from a redesign are real, and the impact can be devastating.

But through careful consideration throughout the redesign process, you can account for potential pitfalls and mitigate your chances of a drop in organic search ranking and traffic post-launch.

Follow along as I outline how we approach this at Portent.

How to Approach SEO Strategy in your Next Redesign

For many websites, organic search is the channel that brings in the most traffic, and inherently, conversions- whether that be leads or transactions and revenue.

Adding to that, organic traffic is the acquisition channel most at-risk through a significant website redesign.

SEO isn’t a tactic to employ after the website launches to clean up loose ends. Approaching your redesign with SEO in mind from the very beginning of the project is vital to ensure your channel’s requirements are baked into the result.

The key to preserving organic rankings through a redesign is two-fold:

1. Marketers must focus on minimizing risk pre-launch

2. Marketers must have a response plan to threats post-launch

With that in mind, our approach to minimizing risk and building a response plan requires us to include an SEO-minded team member in the project from the very start, identify and address gaps in the marketing stack throughout the redesign process, and quantify the impact on organic KPIs post-launch.

This approach lines up with three stages of the project:

  • The planning and design stage (before any code is written)
  • The development stage (when the website is being built)
  • The post-launch stage (after the dust settles)

Let’s explore each of these stages further.

Get SEO Involved Early

The best way to handle potential SEO issues in a redesign project is to prevent them from existing in the first place. SEO for a website redesign starts long before the first line of code is written.

Get your SEO team a seat at the table from the very first meeting.

Their role on the project is to find solutions to infrastructure and content issues that may crop up. Providing a list of SEO requirements and expecting a designer or developer to take them into consideration isn’t enough.

SEO must be hands-on throughout the process.

Don’t wait for infrastructure decisions to be made for you. You may end up with a funky hosting plan, three subdomains, and two of them running on Wix.

(We’ve seen it happen.)

As an SEO, here are some of the questions to consider when kicking off a redesign process:

  • Is the new CMS or framework SEO-friendly?
  • Will you need to prerender JavaScript?
  • Does the new information architecture include your essential landing pages?
  • How many URLs are going to change?
Find the Gaps

As your new website starts coming together, you should ask yourself the question, “is this website more or less optimized than before?”

To answer this, you need to conduct two SEO audits of the website’s infrastructure and content: pre-launch and post-launch.

The goal of the pre-launch audit is to find all of the big problems that you can’t afford to launch with. Auditing a website that isn’t finished yet may seem premature, but it’s a great exercise; it allows you to correct any show-stopping bugs you may find.

This audit is where your SEO team will do most of the work involved with a typical redesign:

  • Redirecting old URLs to new URLs
  • Migrating title and meta description tags
  • Correcting broken links and unnecessary redirects
  • Testing mobile rendering
  • Ensuring canonicalization

The pre-launch audit should also communicate gaps between the two websites in a few key areas:

Site speed

Does the new platform have fewer site speed optimizations?

Content

Does the new content satisfy queries better than before? Are the same Featured Snippets targeted?

Site structure

Is site navigation more or less descriptive than before? Do your important pages still have smart internal links?

Conversion

Do you expect the conversion rate to be higher or lower?

Your post-launch audit should uncover any new bugs and make sure the website is being crawled and indexed correctly.

Some important factors to review post-launch are:

  • Robots.txt (It’s incredibly common for websites to go live disallowing all crawling)
  • Sitemap submission in Google and Bing search consoles
  • The index coverage report in Google Search Console
  • Checking redirect implementation

Doing two thorough audits goes a long way toward minimizing risk.

Any major threat to your rankings will be identified and hopefully addressed before launch, and everything else will be a known quantity. At this point, you should have a pretty good idea of which way your site’s performance will go.

Measure the Impact

Before you launch your new site, make sure your web analytics and rank tracking are recording reliable baseline data for your KPIs.

Also, confirm the new website has your web analytics implemented correctly. You don’t want to launch with all of your conversion goals broken.

I find these metrics and trends the most useful when gauging post-launch performance:

Organic users by landing page

If you didn’t change your URL structure, this report will be incredibly helpful in narrowing down performance gaps.

Organic users by website section

This report will help you find problems with the design or internal linking structure for sections that aren’t doing well.

Non-brand keyword rankings

For each important landing page, add the non-brand keywords contributing the most traffic to a rank tracker. If any of these rankings take a dive after launch, you’ll know which topics you need to prioritize.

Conversion rate by landing page

If your sales copy or CTA links had a drastic change in the new design, this report would let you know which pages will need their offers reconsidered.

Bounce rate and exit rate by website section

Increases in either after the launch might indicate usability problems with the new design for that section.

Common SEO Pitfalls

There are SEO problems so common to redesigns that I’ve seen one in nearly every launch I’ve cleaned up.

Don’t shoot yourself in the foot by making these mistakes:

Unnecessarily changing the URL structure

The best way to map old URLs to new URLs is to not change them at all. Plus, you won’t break year-over-year reports in Google Analytics. Unless you have a good reason, don’t change your URLs in a redesign.

Not redirecting URLs with backlinks

If you have to change your URLs for a new website, make sure you aren’t throwing your backlinks away. Redirect your old URLs to keep the link authority flowing into your site.

Not checking robots.txt on launch day

If your traffic flatlines after launching the new site, this is probably why. Make sure your robots.txt file is configured correctly.

Using uncompressed images

Please don’t make your users download 4 MB of images on every page. Use the right image format and level of compression to keep your images crisp and as small as necessary.

Introducing unnecessary subdomains

Keep your content in one place. Adding a subdomain to your site will split link authority and guarantee a migration project in the future. Always base a new website on a single platform that can do everything you need.

Time to Go Live

Eventually, it’s time to go live with your redesign.

It can be a nervewracking time for every party involved, but at the end of the day, it’s going to happen.

And while redesigning your website can have profound effects (both positive and negative) on site performance, there are ways to mitigate your risk through the process you take.

Be sure to:

  • Get SEO involved from the start
  • Find the gaps
  • Measure the impact

Sticking to this strategy can set yourself up to limit website problems that could devastate your organic traffic.

The post Use This SEO Strategy for Your Next Website Redesign appeared first on Portent.

Read Full Article
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 

Authority and rank metrics like Moz Domain Authority can be incredibly useful for all of your SEO campaigns but when taken at face value or misinterpreted, these metrics can lead you astray of important goals and leave you chasing ROI shadows.

The point of any ranking or authority metric is to mimic a search engine ranking algorithm. Companies like Moz have invested heavily and built these calculations on the backs of crawlers that index pages, count links, evaluate topicality, and use machine learning to provide insights into how well your website might rank against a competitor.

There are quite a few ranking metrics on the market today. Some of the more well-known ones include Majestic’s Trust Flow and Citation Flow, Ahrefs’ Domain Rating, and Moz’s Domain Authority.

Let’s dive into one of the most used authority metrics out there: Moz Domain Authority.

What is Moz Domain Authority?

The Moz Domain Authority, or DA, is a proprietary metric that attempts to predict how well your website will rank on a scale of 1-100.

DA is a good indicator of a website’s potential for organic visibility. Moz also calculates the Page Authority (PA) for a specific URL, scoring it the same way that it does for the domain as a whole.

You can quickly check your DA with the MozBar Chrome extension.

MozBar comes in a few different flavors. The default view gives you DA scores and linking breakdowns for all the pages in a search results page, or once you click through to a page, the DA score and analysis for that page.

A lighter, more simplified view is the DA mode which only shows the score of the page you’re on in the extension icon.

What Does Moz DA Measure?

Moz DA measures links: links, links, and more links.

The Moz score measures inbound links from other sites to your specific pages, inbound links to any of your sub-domains, and inbound links to anywhere on your root domain. It also adds up the number of unique domains and the total quantity of links pointing at your site.

From the Moz Domain Authority explainer:

“Domain Authority is calculated by evaluating multiple factors, including linking root domains and number of total links, into a single DA score. This score can then be used when comparing websites or tracking the “ranking strength” of a website over time. Domain Authority is not a metric used by Google in determining search rankings and has no effect on the SERPs.”

Moz is taking all of the link data from their Link Explorer tool for a given website and matching that up against other domains in their index.

Did you see the catch there?

The score is based only on the Moz index of the web, not Google’s or anyone else’s. Their metric is only as good as its index, which got a lot better last year when Link Explorer replaced Open Site Explorer (OSE), increasing the size of their index 35x and the freshness of their index 30x.

Moz DA 2.0 Update

You may have heard recently about a shake-up with Domain Authority. Moz overhauled the way that they were calculating DA, and marketers and webmasters went into a tizzy. Some sites’ scores went up while others’ went down.

The most important thing to know is that Domain Authority got better. The size of Moz’s link data set got bigger and fresher while they worked in ways to better interpret the link data by incorporating their Spam Score and adding a dash of machine-learning.

So, if you were closely monitoring your Domain Authority scores and they went up on or around March 5, 2019, keep doing what you’re doing. Maybe even more of it.

If your DA dropped off a cliff, you weren’t penalized — it was because your links stink. Moz got significantly better at assessing the quality of the inbound links pointing at your website, and their DA is now a more accurate reflection of Google’s algorithm.

I recommend conducting a little backlink audit and updating that disavow file as soon as you get the chance.

Is Domain Authority a Ranking Factor?

No.

Domain Authority is not a Google ranking factor.

DA is a Moz metric.

Google doesn’t use it. Bing doesn’t use it. DuckDuckGo doesn’t use it.

But, as Googler John Mueller cheekily pointed out during an AMA last year, DA does exist, and Google is aware of it:

How Should You Use Domain Authority?

Domain Authority and similar authority and ranking scores are great as a comparative metric. They should never be used as an absolute measurement of how well your website is doing in organic search. Keep a close eye on your scores and your competitors’ scores over time, especially when there are coordinated campaigns with dedicated outreach efforts.

If you have a link building campaign running alongside on-page optimizations of sections of your website, you’ll be able to show the correlation between on- and off-page factors driving in high-quality traffic.

Then, add DA over time to a reporting dashboard that highlights all the traffic being driven to your campaign’s target pages.

Moz Domain Authority should only be used as a part of your overall tracking strategy; it should never be the sole measurement.

And after the 2.0 refresh, it is one of the best reflections of how Google might be analyzing and ranking your website. As long as you know how to use it (and how not to use it), DA will be a reliable indicator of how well you will perform against your competitors in the SERPs.

The post Is Moz Domain Authority Important for SEO? appeared first on Portent.

Read Full Article
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 

This post does not pick on Google. It picks on us. We’re passive. Lazy. Dependent on Google to give us SEO advice. We jump on bandwagons and apply hacky fixes. Then we say we’re practicing “advanced SEO” because we know how to use noopener.

Advanced SEO isn’t about scouring webmaster hangouts, applying the latest Google patches, and then calling it a day. You want to be “advanced”? Understand five principles:

1: Google Recommendations Are Not SEO Recommendations

Rule number one trumps all other rules.

Apply Google’s recommendations, and a website is visible. Our job isn’t to make sure a site is visible. Any competent developer can do that.

An SEO’s job is to ensure a website can sustainably compete for organic rankings and conversion-generating traffic.

An advanced SEO is always aware that Google is not making SEO recommendations.

2: Always Read The Fine Print

Everyone talks about javascript and its impact on SEO. Search engines tell us “We can render javascript.”

They can! But javascript is risky. Rendered does not equal ranked.

That’s one example. There are plenty more. Here’s Google talking about 301s and links. I added the highlighting:

Read all of Google’s guidance with skepticism and rule #1 in mind. Watch for “could” and “might” versus “do” and “does.”

Read the fine print.

3: Remove Problems. Don’t Hide Them.

Now and then, the search engines give us Remedies: Stuff like rel=canonical. Search engine support for remedies comes and goes.

Google said nofollow would work for link sculpting. Then they said otherwise, sparking the worst nerd riot in history. Folks pelted the stage at SMX Advanced with branded stress balls and iPhone cases. Horrifying.

Google supported rel=next/prev. Until they didn’t.

Remedies change. Shit that just works, though? That’s forever. Don’t mitigate. Fix.

4: Reduce Abstraction

Don’t put more stuff between your content and browsers/bots.

Prerendering and hybrid solutions create additional layers. Redirects create extra hops. Avoid them whenever possible.

Reduce abstraction. Give Google a direct path to your content.

5: Learn The Tools

You can be “advanced” if you don’t know how to analyze a log file, or do fancy natural language processing (I can’t). But you need to be prepared to learn.

That sounds trite. But “learning” doesn’t mean reading the latest blog posts. It means digging into the command line, hacking around with some code, and learning how search engines work.

Read someone’s great GREP tutorial(cough cough).

Learn on-page SEO and natural language processing. Understand how to measure shingling (and what it is, too).

I can’t list everything because I’m still figuring it out myself.

Corollary: Advanced SEO Doesn’t Replace The Basics

By the way: Advanced isn’t a replacement. It’s an amplifier. It makes the basic, nuts-and-bolts SEO tactics more effective.

So spend all the time you want geeking out about natural language processing and isomorphic javascript (I do).

But you’d better spend even more time on title tags, internal linking, and the words you use on a page.

If you skip the basics, all the fancy-schmancy advanced tactics in the world are worth exactly nothing. You’re giving Google a chocolate chip cookie without chocolate chips. That’s bound to piss anyone off.

Advanced SEO means giving me my chocolate chip cookie.

End of rant.

The post Five Principles Of Advanced SEO appeared first on Portent.

Read Full Article
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 

Link building is hard, I get it.

Fussy search engines, spammy savvy marketers selling guest posts, black hat link builders burning bridges — it’s tough. Simply counting links isn’t enough when it comes to performance — you need other measures of success. When you hit it out of the park, your outreach campaign deserves to be celebrated.

But how do you know when your link building work is successful, and, more importantly, how do you prove it?

By using the following KPIs, you can measure and report on the success of your link building efforts. When combined, they present a powerful case for whether your link building campaign was a shining, glorious, singing unicorn, or a damp, tone-deaf chihuahua. Trust me, you’ll have both along the way.

Let’s look at the data before I get metaphorical again, shall we? Here are a handful of KPIs you should monitor as you measure the performance of your link building efforts.

Quantitative KPIs for Link Building

There is security behind hard data, and with link building, you can find performance progress right in the numbers.

Here are the data-driven KPIs that are hard to argue with when progress is made.

#1 Organic Search Traffic

Up and to the right is precisely the kind of movement you want to see for the site that you have been consistently running link building campaigns for. It is important to note that if link building is not the only SEO work you are doing, there are likely other factors playing a role in the movement you see: on-site technical SEO and content improvements will also impact organic search, so make sure to factor that in when measuring the impact of link building.

When reporting this growth, be sure to annotate in Google Analytics when link building campaigns begin, and focus on percentage change of improvement over an extended period of time.

#2 Referral Traffic

Sizeable referral traffic doesn’t always happen with link building. More traffic is a good thing, and qualified traffic that stays on your site is even better. If you want referral traffic, it’s important to mix up your link building strategies.

Guest posting and HARO links rarely generate much referral traffic. HARO links just source a quote, so no one clicks on them. And site editors discourage their readers from leaving the site (I’ve seen a lot of hidden anchor text in posts lately), so guest post links don’t tend to generate much traffic, by design.

If you get 15 referrals in a month from a guest post, you should stand up by your desk and do a little dance. High referral traffic for a guest post is a unicorn—it’s rare. In a survey of over 500 guest post bloggers, more than 35% of reported guest posts only got 10 or fewer referrals.

Content promotion, on the other hand, has the potential to perform well, generating a lot of referral traffic if done right.

Content promotion is driven by highly promotable (read: “newsworthy”) data-driven content in the form of a report, tool, or long-form writing. Promoting that content through Digital PR efforts is much more likely to bring significant referral traffic. This occurs when your content gets picked up and covered by a media outlet of any kind, and they link to your site as the source of this newsworthy content. When reporting content promotion results, be sure to include:

  • The media outlets that covered your content
  • Domain Authority of each site that covered your content
  • Referral traffic for each link
  • Traffic behavior (time of page, bounce rate, pages visited)
  • Total number of links placed

The graph above shows the potential impact that content promotion and digital PR can have. During the three-year life of this site, the first digital PR campaign we launched was responsible for 33% of the site’s lifetime referral traffic. It’s important to note that in this campaign’s case, a majority of the links earned (177) were no-follow. While not quite as ideal from an SEO perspective, no-follow links shouldn’t be underestimated. If we were keeping score (which, c’mon, you know we are), no-follow links should be considered “half links.” They can be used as part of a more long-term link building strategy since a mixture of do-follow and no-follow links create a healthy backlink profile.

In this example for our client, we received a ton of no-follow brand mentions. Brand mentions have been proven to be a ranking factor, so they should be considered a success. Be sure to report no-follow brand mentions too, even if they don’t link to your site (but be sure to follow up anyway and ask for a link).

Utility and resource link building, along with similar link building strategies might garner referral traffic over time as users find and click on the links, especially if they are helpful and on moderate-to-highly-trafficked sites. The great thing about utility link building is that it can often generate conversions because the links target consumers at the lower end of the conversion funnel and drive them to pages aimed at conversion. While improving organic traffic is the ultimate goal of link building, this qualified referral traffic is a great cherry on top.

#3 Traffic to Target Pages

As mentioned previously, in every campaign, whether it is guest posting, content promotion, or any other form of link building, you should have target pages to which you build all your links. The traffic, visitor engagement, and organic rankings for these pages should be monitored and reported on.

Be sure to report changes in traffic and behavior over time to these pages. Below is a snapshot of traffic to Portent’s SERP Preview Tool. You can see where traffic over time has increased to this page by 48% YoY which would make any stakeholder happy, especially if a page is designed to convert users.

Qualitative KPIs for Link Building

While the quantitative KPIs mentioned above do a nice job of putting hard numbers to performance, there are a couple of qualitative factors to monitor when tracking the performance of your link building efforts.

#4 Relevance

Although it is not nearly as measurable as other KPIs, the relevance of the links you build should be an important focus for any link building campaign. The domains on which you build links on should be relevant to your website’s brand and industry. Landing a link on a competitor’s blog is the sparkly, magic kingdom every link building team should aim for, but it’s hard to get there. Linking on pages within your direct industry is a close second on the list of coveted link placements.

The text surrounding your backlink should be natural, highly relevant, and full of naturally-occurring keywords (relevance). The anchor text that directs the reader to the content on your target page should inform the reader of exactly what information they’re going to get when they click on your link. Guest post content should be informative and well-written (and hopefully your target page is too).

#5 Domain Authority, Domain Rank, Trust Flow

We all learned a big lesson recently about how tenuous it is to rely on Domain Authority (DA) as a primary metric. With Moz’s DA algorithm change on March 5th of this year, marketers in the SEO world got a big wake-up call. While DA still gauges the authority of a site in a very reliable way, it is important to take into account the relevance of the page when evaluating whether a link you place is valuable. To a small HVAC company, for instance, a link on Inc. Magazine with a DA of 92 (on a scale of one to 100) is not nearly as valuable as a link on their biggest competitor’s blog in the HVAC industry with a DA of only 32.

Domain Rank (DR) on Ahrefs and the Trust Flow on Majestic are both similar gauges of your referring domain’s authority, as well as your ability to rank as compared to their competitors.

Use DA/DR/Trust Flow to point out the authority of the sites you’re placing backlinks on, and to point out that you’re building a natural backlink profile. Make sure you always take this information with a grain of salt, realizing that they are relative, proprietary metrics. You should work to build links on sites from 10-100 DA and DR, and relevance should always be your primary KPI here.

Relevance and authority are what I would call soft KPIs. Your DA will rise very slowly (it’s like watching water come to a boil if you do it right), and it shouldn’t be a primary (or even a secondary) indicator of success. Monitor how your DA tracks in comparison to your competitors to get the most out of any DA numbers.

Go forth, armed with cold, hard data

It isn’t hard to prove the success of a good link building campaign. Link building is a long-term digital marketing strategy and should complement the on-site technical SEO and content efforts you put forth. So don’t get discouraged if your efforts are slow to reveal themselves. Resilience is the most important character trait of a good link builder and patience is a must. Use carefully-selected link targets, mark your progress, and prove your hard-won success with a handful of the KPIs outlined here. And always remember to celebrate those unicorns—whether or not they can sing Queen.

The post 5 KPIs for Measuring Outreach and Link Building Efforts appeared first on Portent.

Read Full Article
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 

Does a bear poop in the woods? With javascript and SEO, the answer is just as clear, if a little more complicated.

Javascript-driven sites aren’t bad for indexation. Google can crawl a site that populates content client-side.

Javascript-driven client-side content is bad for SEO. Javascript-driven sites make Google work harder. At the very least, Google renders them more slowly. In the SERPs, that’s a competitive disadvantage.

To demonstrate (Ian rubs his hands together) I get to use quadrant diagrams.

If you already know how javascript works, what client-side rendering is, and how Google handles client-side rendering, print this, stick it to your forehead, and move on:

quadrant

The javascript/SEO quadrant

For us mere mortals, here’s a fuller explanation:

Two Types Of Javascript

There are two ways client-side javascript—javascript executed by a web browser—can interact with web content:

UI enhancement changes the way the browser interacts with content rendered on the server. Examples include tabbed, drop down navigation, and (sigh) carousels.

Client-side rendering delivers pages and content separately. Your web browser uses javascript to merge the two.

This post talks about client-side javascript rendering and why it’s bad for SEO.

Client vs. Server-Side

Every web page is an interaction between a client (a browser, like Chrome, or a bot, like Google) and a server.

Generating a web page involves three steps:

  1. Fetch the page template (the layout)
  2. Fetch the content
  3. Merge the content with the template

Server- and client-side rendering perform these three steps differently.

Server-side rendering does all three steps on the server, then sends the result to the client. The client has everything it needs, renders the full page, and goes on its merry way.

Client-side rendering uses javascript to split the labor: It sends the template to the browser or bot, then sends the content separately. The browser uses javascript to merge the content and the template. Client-side rendering has advantages: It’s speedy (if you do it right). It’s a great way to build interactive applications. But it requires more work by the client.

Here’s our quadrant diagram so far:

quadrant-server-client

Server and client rendering

Static Content vs. Dynamic Interface

Some pages are just stuff: Words and pictures and links and buttons. Clicking those links and buttons send me to another page or display a form. They don’t profoundly modify the page itself. That’s static content, and it’s what you browse 90% of the time: Articles, product pages, blog posts, news, etc.

Other pages change a lot depending on my actions: A text editor, a multi-faceted search, or a page where content continually updates. A page like this is a dynamic interface. The Portent Title Generator, built by some incredible agency (cough) is an example:

javascript-title-generator

A dynamic interface using javascript

Hopefully, your SEO strategy doesn’t hinge on dynamic content. If you’re going to succeed in SEO, you need to get your static content indexed and optimized.

Static vs. dynamic is the next part of the quadrant diagram:

quadrant-all-four

Static, dynamic, client- and server-side

When you combine static/dynamic and server-side/client-side, you get a feel for where and how javascript can make SEO more difficult.

When Javascript Hurts SEO

Javascript is terrible for SEO when you use client-side rendering for static content:

quadrant

The javascript/SEO quadrant

Here’s why:

Static content is what you need indexed. If you can’t get a key product into the rankings, if your blog post is invisible, you’re hosed. Fortunately, Google crawls and indexes javascript-driven static content. All good.

You also need static content optimized: You need higher rankings, and that content is how you’ll get there. The trouble starts here. Google uses two-stage rendering on javascript-powered websites: It crawls the site now, renders content later. Here’s how Google’s engineers put it:

“The rendering of JavaScript powered websites in Google Search is deferred until Googlebot has resources available to process that content.”

That’s in Google’s own words at io2018. Check the video at 14:11.

Two learnings:

  • Google needs extra resources to fully crawl, render and index javascript-powered, client-side rendered pages
  • Google felt it necessary to point out that fact

Client-side rendering doesn’t hurt indexation. It hurts SEO. There’s a difference. As I said, Google can crawl javascript content, and it does. But two-step rendering puts client-side content at a competitive disadvantage. All these quadrant diagrams are making me giddy:

quadrant2

Indexing vs. SEO

If you’re doing SEO, you can’t afford to end up in the bottom-right box.

If you must use client-side rendering on static content, here are two ways to reduce the damage:

Mitigation

If you must use javascript, mitigate it using prerendering or hybrid rendering.

Prerendering and user-agent detection

Prerendering works like this:

  1. Render a server-side version of each page on your site
  2. Store that
  3. When a client visits, check the user agent
  4. If the client is a search bot, deliver the prerendered content instead of the javascript-rendered content

The logic is licking-your-own-eyeball-from-the-inside tortured: If you can deliver prerendered content, why not just do that from the start? But, if you must, try Puppeteer to do prerendering, or a service like prerender.io, which does all the work for you.

Hybrid rendering

Hybrid rendering generates the first page/content server-side, then delivers remaining content client-side. Sort of. Most javascript libraries, such as Angular, support this. I think.

If you search for “hybrid rendering,” you’ll find seven million pages, each with a slightly different definition of “hybrid rendering.” For our purposes, assume it means “Deliver the most important content, then the other stuff.”

For example, you could use it for filtering. Coursera lets you filter courses without javascript:

But the interface gets speedier, and the results richer, if your browser supports javascript:

That’s not the best example. TRUST ME that hybrid rendering mixes javascript-driven and static content, delivering static content first.

When To Use Which

For static content, use server-side rendering or, if you must, prerendering. If you want to optimize content that’s in a dynamic interface (like Coursera’s course list), use hybrid rendering.

ONE LAST QUADRANT DIAGRAM:

Why Mitigation Sucks

My rule: If Google gives you ways to mitigate a thing, don’t do that thing at all.

You know your doctor can set a bone. That doesn’t mean you go out of your way to break your leg for giggles.

Google can handle javascript-driven sites. That doesn’t mean you go out of your way to render content using javascript.

If nothing else, remember that Google changes their mind.

But I am not a javascript hater. In some cases, javascript-driven pages make a ton of sense.

When You Should Use Javascript Rendering

Build a client-side javascript-driven website when interactivity is more important than rankings. Apps and app-like websites, aggregators, and filters require client-side javascript rendering. Then use hybrid rendering to deliver critical content to Google.

When You Shouldn’t Use Javascript Rendering

Don’t use javascript for static content. If content isn’t interactive—a basic product page, a blog post, news articles, and any other content that doesn’t have to instantly respond to user input—it doesn’t need client-side javascript.

That doesn’t include carousels and other stuff. That’s UI enhancement, not content delivery. Done right, it’s perfectly OK.

Testing

This will bunch up the undergarments of many SEOs, developers, search scientists, and engineers: Don’t test.

Tests make you feel better. They show you that Google can indeed render the content. Great! Hooray for you!

No. Boo for you! Because testing verifies indexing and rendering. It does not verify that you’re competitive.

If you’re using client-side javascript to deliver static content you’ve failed the test. Stop. Change it.

Ask Yourself Why

There are two lessons here:

  1. Javascript can be bad for SEO
  2. There’s a difference between SEO and indexation

If you want to compete in the rankings, don’t use client-side rendering to deliver static content, or any content for which you want to rank. Use javascript to drive app-like experiences. When you’re considering using javascript to deliver content, do a very honest assessment of the pluses and minuses.

Then remember this handy quadrant diagram. I put a lot of time into this:

Read Full Article
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 

Florida 2 Algorithm Update What’s Going On?

On March 12th, Google released what being referred to as the Florida 2 algorithm update. Website owners are already noticing significant shifts across their keyword rankings. While Google’s algorithm updates vary in terms of how often they receive broad notice, the Florida 2 update is one that every marketer needs to be paying close attention to.

Who Was Impacted by the Florida 2 Algorithm Update?

Google makes several broad ranking algorithm updates each year, but only confirms updates that result in widespread impact. Google did confirm Florida 2, and there are SEOs out there already calling it the “biggest update [Google has made] in years.” Unlike last August’s Medic Update, Florida 2 isn’t targeting a specific niche or vertical, which means the entire search community needs to be paying attention as we try to better understand the changes Google is making to its search algorithm.

While it’s still too early for our team to pinpoint what exactly is being impacted by Florida 2, we’re going to keep a very close eye on where things fall out over the next several days (and weeks).

Here’s what we’ve seen so far:

  • Indiscriminate swings in site traffic & ranking, with some websites reporting zero traffic after the update.
  • Evidence of traffic increases for site owners who are prioritizing quality content and page speed.
  • A worldwide impact – this is not a niche specific or region specific update.
  • Potential adjustments in how Google is interpreting particular search queries.
  • Backlink quality possibly being a very important factor.
  • Short term keyword ranking changes (declines in ranking that then back to “normal” after a few hours).
My Rankings Took a Hit. What Can I Do?

In short? Nothing. But don’t panic.

As with any Google algorithm update, websites will see increases or declines in their rankings. There is no quick fix for sites or web pages that experience negative results from the update; don’t make the mistake of aggressively changing aspects of your site without fully understanding the broader impact those changes will have. If you are being negatively impacted by Florida 2 (or any other algorithm update), your best bet is continuing to focus on offering the best content you can, as that is what Google always seeks to reward.

For advice on how to produce great content, a good starting point is to review Google’s Search Quality Guidelines. For recommendations on copywriting tips and strategies check out the copywriting section of the Portent blog.

We’ll continue to update this post as we gather more information.

The post Google’s Latest Broad Core Ranking Update: Florida 2 appeared first on Portent.

Read Full Article
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 

Log file analysis is a lost art. But it can save your SEO butt. This post is half-story, half-tutorial about how web server log files helped solve a problem, how we analyzed them, and what we found. Read on, and you’ll learn to use grep plus Screaming Frog to saw through logs and do some serious, old school organic search magic (Saw. Logs. Get it?!!! I’m allowed two dad jokes per day. That’s one).

If you don’t know what a log file is or why they matter, read What Is A Log File? first.

The Problem

We worked with a client who rebuilt their site. Organic traffic plunged and didn’t recover.

The site was gigantic: Millions of pages. We figured Google Search Console would explode with dire warnings. Nope. GSC reported that everything was fine. Bing Webmaster Tools said the same. A site crawl might help but would take a long time.

Phooey.

We needed to see how Googlebot, BingBot, and others were hitting our client’s site. Time for some log file analysis.

This tutorial requires command-line-fu. On Linux or OS X, open Terminal. On a PC, use a utility like cmder.net.

Tools We Used

My favorite log analysis tool is Screaming Frog Log File Analyser. It’s inexpensive, easy to learn, and has more than enough oomph for complex tasks.

But our client’s log file snapshot was over 20 gigabytes of data. Screaming Frog didn’t like that at all:

screaming-frog-log-choke-compressed

Screaming Frog says "WTF?"

Understandable. We needed to reduce the file size, first. Time for some grep.

Get A Grep

OK, that’s dad joke number two. I’ll stop.

The full, 20-gig log file included browser and bot traffic. All we needed was the bots. How do you filter a mongobulous log file?

  • Open it in Excel (hahahahahahaahahahah)
  • Import it into a database (maybe, but egads)
  • Open it in a text editor (and watch your laptop melt to slag)
  • Use a zippy, command line filtering program. That’s grep

“grep” stands for “global regular expression print.” Grep’s parents really didn’t like it. So we all use the acronym, instead. Grep lets you sift through large text files, searching for lines that contain specific strings. Here’s the important part: It does all that without opening the file. Your computer can process a lot more data if it doesn’t have to show it to you. So grep is super-speedy.

Here’s the syntax for a typical grep command:

grep [options][thing to find] [files to search for the thing]

Here’s an example: It searches every file that ends in “*.log” in the current folder, looking for lines that include “Googlebot,” then writes those lines to a file called botsonly.txt:

grep -h -i ‘Googlebot’ *.log >> botsonly.txt

The -h means “Don’t record the name of the file where you found this text.” We want a standard log file. Adding the filename at the start of every line would mess that up.

The -i means “ignore case.”

Googlebot is the string to find.

*.log says “search every file in this folder that ends with .log”

The >> botsonly.txt isn’t a grep command. It’s a little Linux trick. >> writes the output of a command to a file instead of the screen, in this case to botsonly.txt.

For this client, we wanted to grab multiple bots: Google, Bing, Baidu, DuckDuckBot, and Yandex. So we added -e. That lets us search for multiple strings:

grep -h -i -e 'Googlebot' -e 'Bingbot' -e 'Baiduspider' -e 'DuckDuckBot' -e 'YandexBot' *.log >> botsonly.txt

Every good LINUX nerd out there just choked and spat Mountain Dew on their keyboards. You can replace this hackery with some terribly elegant regular expression that accomplishes the same thing with fewer characters. I am not elegant.

Breaking down the full command:

h: Leaves out filenames
i: Case insensitive (I’m too lazy to figure out the case for each bot)
e: Filter for multiple factors, one factor after each instance of -e
>>: Write the results to a file

Bots crawl non-page resources, too, like javascript. I didn’t need those, so I filtered them out:

grep -h -v *.js botsonly.txt >> botsnojs.txt

-v inverts the match, finding all lines that do not include the search string. So, the grep command above searched botsonly.txt and wrote all lines that did not include .js to a new, even smaller file, called botsnojs.txt.

Result: A Smaller Log File

I started with a 20-gigabyte log file that contained a bazillion lines.

After a few minutes, I had a one-gigabyte log file with under a million lines. Log file analysis step one: Complete.

Analyze in Screaming Frog

Time for Screaming Frog Log File Analyser. Note that this is not their SEO Spider. It’s a different tool.

I opened Screaming Frog, then drag-and-dropped the log file. Poof.

If your log file uses relative addresses—if it doesn’t have your domain name for each request—then Screaming Frog prompts you to enter your site URL.

What We Found

Google was going bonkers. Why? Every page on the client’s site had a link to an inquiry form.

A teeny link. A link with ‘x’ as the anchor text, actually, because it was a leftover from the old code. The link pointed at:

inquiry_form?subject_id=[id]

If you’re an SEO, you just cringed. It’s duplicate content hell: Hundreds of thousands of pages, all pointing at the same inquiry form, all at unique URLs. And Google saw them all:

log-files-inquiry-form-compressed

Inquiry Forms Run Amok

60% of Googlebot events hit inquiry_form?subject_id= pages. The client’s site was burning crawl budget.

The Fix(es): Why Log Files Matter

First, we wanted to delete the links. That couldn’t happen. Then, we wanted to change all inquiry links to use fragments:

inquiry_form#subject_id=[id]

Google ignores everything after the ‘#.’ Problem solved!

Nope. The development team was slammed. So we tried a few less-than-ideal quick fixes:

  • robots.txt
  • meta robots
  • We tried rel canonical No, we didn’t, because rel canonical was going to work about as well as trying to pee through a Cheerio in a hurricane (any parents out there know whereof I speak).

Each time, we waited a few days, got a new log file snippet, filtered it, and analyzed it.

We expected Googlebot to follow the various robots directives. It didn’t. Google kept cheerfully crawling every inquiry_form URL, expending crawl budget, and ignoring 50% of our client’s site.

Thanks to the logs, though, we were able to quickly analyze bot behavior and know whether a fix was working. We didn’t have to wait weeks for improvements (or not) in organic traffic or indexation data in Google Search Console.

A Happy Ending

The logs showed that quick fixes weren’t fixing anything. If we were going to resolve this problem, we had to switch to URL fragments. Our analysis made a stronger case. The client raised the priority of this recommendation. The development team got the resources they needed and changed to fragments.

That was in January:

fragments-implemented-compressed

Immediate Lift From URL Fragments

These results are bragworthy. But the real story is the log files. They let us do faster, more accurate analysis, diagnose the problem, and then test solutions far faster than otherwise possible.

If you think your site has a crawl issue, look at the logs. Waste no time. You can thank me later.

I nerd out about this all the time. If you have a question, leave it below in the comments, or hit me up at @portentint.

Or, if you want to make me feel important, reach out on LinkedIn.

The post Log File Analysis For SEO appeared first on Portent.

Read Full Article
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 

Advanced SEO cuts through abstraction. It doesn’t add layers of fixes and workarounds to mask SEO problems. Instead, it removes the problems themselves.

This deck is my talk from Digital Summit 2019. There are a lot of slides. If you’ve seen me speak you’re used to that. If not, don’t let it scare you. Every slide has a single link, idea, or tip. It’s a fast read that I’ve hopefully crammed with useful stuff.

Ask questions in the comments below, or find me on Twitter: @portentint

Or on LinkedIn: https://www.linkedin.com/in/ianlurie/

The post Advanced SEO: Digital Summit Slide Deck appeared first on Portent.

Read Full Article
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 

I have this friend. Some neanderthal hacked their site and sprayed spam links all over their blog, 2008-style.

So this friend had two problems:

  1. Find all the spam links
  2. Detect new spam links as they occur

 I  My friend came up with a relatively simple process using ScreamingFrog. Here it is:

Yeah, yeah, it was us. I won’t detail how. It’s embarrassing, and I don’t want to get beat up on the nerd playground.

This process uses Screaming Frog. You can use another crawler, but Screaming Frog is a bucket of awesome.

The Process

Here’s what you’ll do:

  1. Crawl your site
  2. Clean up the bad links
  3. Do a “clean crawl”
  4. A few days/weeks later, do a new crawl and compare that to the previous one, looking for new, suspicious links
  5. Repeat 2 and 3
Step 1: Crawl

First, crawl your site:

  1. Open Screaming Frog (I know, I know, it’s obvious, but I like to be comprehensive in my step-by-step).
  2. Click Configuration >> Spider
  3. In the “Basic” tab, uncheck “Check Images,” “Check CSS,” and “Check JavaScript.” Uncheck “Check SWF,” if you’re still using Flash. But you’re not, right? Your Spider configuration should look like this: Basic Screaming Frog Spider Configuration

    Basic Screaming Frog Spider Configuration

  4. Make sure ‘Follow External “nofollow”’ is not checked.
  5. Optional: If your site has tens of thousands or more pages, and your computer has a solid state drive, consider going to Configuration >> System >> Storage and switching to Database Storage.
  6. Start your crawl. If you have a big site, have a coffee or something. Your computer’s fan may start to shriek in protest. It’s OK. Just keep a fire extinguisher handy.
  7. Set up an exclusion filter. See “Set Up An Exclusion Filter,” below.
  8. Save the completed crawl. Your computer worked hard. It’s the least you can do.
Set Up An Exclusion Filter

Most crawls generate a long list of external links. You can make life easier by filtering out links to Twitter, LinkedIn, etc. In theory, links to those sites are legitimate. A good filter is a sanity-saver. It can reduce the list of links you have to review by 70–80%. Screaming Frog has a handy exclusion tool for that very purpose.

  1. In Screaming Frog, click Configuration >> Exclude
  2. Type whatever URLs you want to exclude

Screaming Frog’s filter uses regular expressions. Here’s what our regex genius, Matthew Henry, came up with:

https?://(?:[^/\.]+\.)*domainname\.com/.* will filter www.domainname.com and domainname.com.

For example, https?://(?:[^/\.]+\.)*twitter\.com/.* filters out https://www.twitter.com/whatever and https://twitter.com/whatever

Here’s how the exclusion filter looks once you’ve entered a few domains:

Screaming Frog Exclusion Filter

Screaming Frog Exclusion Filter

You can use the “Test” tab to see whether your filter works.

You can see our filter list here. If you want to, download it, copy it, and paste it right into the Exclude field. Note this list is perfect for us, but you’ll need to customize yours over time. Or live with a less-relevant exclusion filter based on our outgoing link profile.

Save your exclusion filter!!! You’re going to run regular crawls. Save the filter. Don’t lose it.

What You’ve Got

With the crawl complete, you’ve got a complete list of external links, minus the filtered domains. Click the “External Links” tab, and you’ll see something like this:

external-links-tab

The External Links Tab

Time to get to work.

Step 2: Clean Up The Bad Links

You’re going to have to look at all those links.

In case you’re wondering, yes, that’s hellaciously tedious. We tried all kinds of automation. None of it was accurate, and we need as close to 100% accuracy as possible. We forged ahead. You will, too.

Don’t give up. After this first cleanup, you can automate detection.

  1. In the External Links tab, clean things up a bit. Look to the right of the column labels. Click the “+” sign.
  2. I always uncheck “Content,” “Status,” and “Crawl Depth.” It makes skimming through the links easier and keeps your exports cleaner. change-columns

    Changing Columns

  3. Start scrolling through the list of links.
  4. As you scroll, look at the info pane at the bottom left of Screaming Frog.

The info pane shows the link type, the source page, and the target. It also shows (and this is important) the anchor and ALT text. Look at those, and the spam jumps out at you. Look at this example:

reviewing-a-link

Reviewing A Link

That’s a link from the HTML version of Conversation Marketing. It points to Cooper.com. The anchor text is “Cooper.com.” I was suspicious, but a glance at the “from” page showed it was legit. We sifted through all the links, looking for:

  • Moronic anchor text
  • Links to irrelevant content. You can usually figure that out by looking at the source and destination URLs
  • Links to SEO agencies we didn’t recognize

By the way, if you’re an SEO agency, don’t hack other marketing companies’ sites for links. It’s like jacking a police car full of angry, mace-bearing trainees all anxious to prove themselves. There are only a couple of endings. None are good for you. The only reason I’m not listing you all in this post and subscribing your webmaster address to every porn site on the internet is the very, very slim chance someone else placed these links.

When you find a spam link, note the “From” page. That’s the linking page on your site—you’ll go there to delete the link. Now, remove all those links! It’s very satisfying.

Examples

Here are three of the links we caught, and why:

  • A link to another SEO agency pointed at www.agencyname.com/seo-[city] page with the anchor text “SEO-[cityname]” from a 2014 blog post about SEO analytics. It made zero sense. That was easy.
  • A link to a greeting card company (?!!!) from a blog post about digital marketing strategy to the “leaving cards” page on their site with the anchor text “leaving cards.” Okaaaayyyy.
  • A link to an SEO agency with the anchor text “[city in Australia] SEO.” I’m sure [city] is beautiful, but we’d use better anchor text than that if we suddenly decided to start reviewing Australian SEO agencies.
Why Not Use Link Data?

You can use the link metrics provided by tools like Moz, ahrefs, and Majestic to score the spamminess of a link. That can save you oodles of time, and we tried it. We discovered that many of the target pages appeared legitimate — for example, one of the links we found pointed at a site with a spam score of 1%.

If a spam link points at a perfectly normal page, link metrics won’t flag it.

Step 2a (Optional): Update Your Exclusion List

If you find many external links to a single legitimate domain, add that domain to your exclusion list. It’ll make future crawls and reviews easier, and keep crawl files under control.

After our first review, we copied all domains we knew were OK, used some search-and-replace, and added those domains to our exclusion list. It cut the next crawl export in half.

What You’ve Got

You now have a clean site. You can do a clean crawl.

Step 3: Run and save another crawl

Now, run another crawl using the same exclusion filter. You saved the filter, right?

Once the crawl’s done:

  1. Clean things up a bit. Look to the right of the column labels. Click the “+” sign.
  2. Uncheck everything except “Address.”
  3. Click the “Export” button. It’s next to the Filter drop-down. Save the result.

I convert the result from a csv to a text file. It’s a one-column list. Why get complicated?

You’ll compare your next crawl to this one and never, ever have to hand-review thousands of links again.

Step 4: Run A New Crawl And Compare

Run a new crawl and save it, just as you did in Step 3. I keep my old crawls and organize files by date. Compulsive. I know.

Now for the fun part! You’re going to compare the most recent crawl to the new one, looking for new links. Don’t fret — you don’t have to do this by hand. While computers suck at finding spam links, they excel at finding differences between files.

Tons of tools let you compare files. If you want something simple, I like Mergely.

Here’s what a comparison of the last and latest crawls looks like in Mergely:

mergely

Mergely File Comparison

The highlighted line is a new link. Easy!

Mergely might bog down with humungous files, though.

So I use the command line. On Linux, this command works like a charm:

‘comm –13 [oldfile] [newfile]’

That command compares oldfile to newfile, showing new stuff in newfile. Try File Compare (FC) on Windows. Your results may vary.

Here’s what a comparison of the last and latest crawls looks like in comm:

comm

Comm File Comparison

It’s fussier than Mergely, so you may get some false positives.

Review any new links. If they’re spam, you know what to do.

This is an excellent time to update your exclusion filter, too. See Step 2a, above.

Step 5: Repeat

Save the last crawl as your new baseline. When you run the next crawl, you’ll compare it to the last one. And so on. Repeat steps 3–5 as desired.

I run crawls every two to three weeks. If you have a faster-growing site, run crawls more often. It’ll make step 5 easier by reducing potential differences and delivering shorter lists of new links.

Enterprise-scale Alternatives

Our site is about 5700 pages. With the exclusion filter, our crawl generated a list of 2300 links. Hand-reviewing those isn’t all that bad. I divided it into chunks of 100, passed them around the office, and we finished reasonably quickly. If your site is millions of pages, you may need to use a crawler like Deepcrawl or OnCrawl.

You may need to look at machine learning as a spam link detection tool (there I said “machine learning” so this is now a legitimate marketing article). However, machine learning gets sticky when you’re sniffing for spam links that point at not-spammy pages.

Worth it?

Is all this work worth it?

Google won’t penalize you for trash links pointing to other sites. Probably. The linked sites don’t benefit from these links. Much.

It boils down to pettiness. I’m a petty person. I have an intense hatred of sleaze. Every spammer we found had “acquired” links from other sites. We contacted all those site owners. Then we reported every linked site to Google.

I have no idea if it’ll have an impact. But I sure feel better.

The post How to: Check Your Site For Spam Links Using Screaming Frog appeared first on Portent.

Read Full Article
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 

I have this friend. Some neanderthal hacked their site and sprayed spam links all over their blog, 2008-style.

So this friend had two problems:

  1. Find all the spam links
  2. Detect new spam links as they occur

 I  My friend came up with a relatively simple process using ScreamingFrog. Here it is:

Yeah, yeah, it was us. I won’t detail how. It’s embarrassing, and I don’t want to get beat up on the nerd playground.

This process uses Screaming Frog. You can use another crawler, but Screaming Frog is a bucket of awesome.

The Process

Here’s what you’ll do:

  1. Crawl your site
  2. Clean up the bad links
  3. Do a “clean crawl”
  4. A few days/weeks later, do a new crawl and compare that to the previous one, looking for new, suspicious links
  5. Repeat 2 and 3
Step 1: Crawl

First, crawl your site:

  1. Open Screaming Frog (I know, I know, it’s obvious, but I like to be comprehensive in my step-by-step).
  2. Click Configuration >> Spider
  3. In the “Basic” tab, uncheck “Check Images,” “Check CSS,” and “Check JavaScript.” Uncheck “Check SWF,” if you’re still using Flash. But you’re not, right? Your Spider configuration should look like this: Basic Screaming Frog Spider Configuration

    Basic Screaming Frog Spider Configuration

  4. Make sure ‘Follow External “nofollow”’ is not checked.
  5. Optional: If your site has tens of thousands or more pages, and your computer has a solid state drive, consider going to Configuration >> System >> Storage and switching to Database Storage.
  6. Start your crawl. If you have a big site, have a coffee or something. Your computer’s fan may start to shriek in protest. It’s OK. Just keep a fire extinguisher handy.
  7. Set up an exclusion filter. See “Set Up An Exclusion Filter,” below.
  8. Save the completed crawl. Your computer worked hard. It’s the least you can do.
Set Up An Exclusion Filter

Most crawls generate a long list of external links. You can make life easier by filtering out links to Twitter, LinkedIn, etc. In theory, links to those sites are legitimate. A good filter is a sanity-saver. It can reduce the list of links you have to review by 70–80%. Screaming Frog has a handy exclusion tool for that very purpose.

  1. In Screaming Frog, click Configuration >> Exclude
  2. Type whatever URLs you want to exclude

Screaming Frog’s filter uses regular expressions. Here’s what our regex genius, Matthew Henry, came up with:

https?://(?:[^/\.]+\.)*domainname\.com/.* will filter www.domainname.com and domainname.com.

For example, https?://(?:[^/\.]+\.)*twitter\.com/.* filters out https://www.twitter.com/whatever and https://twitter.com/whatever

Here’s how the exclusion filter looks once you’ve entered a few domains:

Screaming Frog Exclusion Filter

Screaming Frog Exclusion Filter

You can use the “Test” tab to see whether your filter works.

You can see our filter list here. If you want to, download it, copy it, and paste it right into the Exclude field. Note this list is perfect for us, but you’ll need to customize yours over time. Or live with a less-relevant exclusion filter based on our outgoing link profile.

Save your exclusion filter!!! You’re going to run regular crawls. Save the filter. Don’t lose it.

What You’ve Got

With the crawl complete, you’ve got a complete list of external links, minus the filtered domains. Click the “External Links” tab, and you’ll see something like this:

external-links-tab

The External Links Tab

Time to get to work.

Step 2: Clean Up The Bad Links

You’re going to have to look at all those links.

In case you’re wondering, yes, that’s hellaciously tedious. We tried all kinds of automation. None of it was accurate, and we need as close to 100% accuracy as possible. We forged ahead. You will, too.

Don’t give up. After this first cleanup, you can automate detection.

  1. In the External Links tab, clean things up a bit. Look to the right of the column labels. Click the “+” sign.
  2. I always uncheck “Content,” “Status,” and “Crawl Depth.” It makes skimming through the links easier and keeps your exports cleaner. change-columns

    Changing Columns

  3. Start scrolling through the list of links.
  4. As you scroll, look at the info pane at the bottom left of Screaming Frog.

The info pane shows the link type, the source page, and the target. It also shows (and this is important) the anchor and ALT text. Look at those, and the spam jumps out at you. Look at this example:

reviewing-a-link

Reviewing A Link

That’s a link from the HTML version of Conversation Marketing. It points to Cooper.com. The anchor text is “Cooper.com.” I was suspicious, but a glance at the “from” page showed it was legit. We sifted through all the links, looking for:

  • Moronic anchor text
  • Links to irrelevant content. You can usually figure that out by looking at the source and destination URLs
  • Links to SEO agencies we didn’t recognize

By the way, if you’re an SEO agency, don’t hack other marketing companies’ sites for links. It’s like jacking a police car full of angry, mace-bearing trainees all anxious to prove themselves. There are only a couple of endings. None are good for you. The only reason I’m not listing you all in this post and subscribing your webmaster address to every porn site on the internet is the very, very slim chance someone else placed these links.

When you find a spam link, note the “From” page. That’s the linking page on your site—you’ll go there to delete the link. Now, remove all those links! It’s very satisfying.

Examples

Here are three of the links we caught, and why:

  • A link to another SEO agency pointed at www.agencyname.com/seo-[city] page with the anchor text “SEO-[cityname]” from a 2014 blog post about SEO analytics. It made zero sense. That was easy.
  • A link to a greeting card company (?!!!) from a blog post about digital marketing strategy to the “leaving cards” page on their site with the anchor text “leaving cards.” Okaaaayyyy.
  • A link to an SEO agency with the anchor text “[city in Australia] SEO.” I’m sure [city] is beautiful, but we’d use better anchor text than that if we suddenly decided to start reviewing Australian SEO agencies.
Why Not Use Link Data?

You can use the link metrics provided by tools like Moz, ahrefs, and Majestic to score the spamminess of a link. That can save you oodles of time, and we tried it. We discovered that many of the target pages appeared legitimate — for example, one of the links we found pointed at a site with a spam score of 1%.

If a spam link points at a perfectly normal page, link metrics won’t flag it.

Step 2a (Optional): Update Your Exclusion List

If you find many external links to a single legitimate domain, add that domain to your exclusion list. It’ll make future crawls and reviews easier, and keep crawl files under control.

After our first review, we copied all domains we knew were OK, used some search-and-replace, and added those domains to our exclusion list. It cut the next crawl export in half.

What You’ve Got

You now have a clean site. You can do a clean crawl.

Step 3: Run and save another crawl

Now, run another crawl using the same exclusion filter. You saved the filter, right?

Once the crawl’s done:

  1. Clean things up a bit. Look to the right of the column labels. Click the “+” sign.
  2. Uncheck everything except “Address.”
  3. Click the “Export” button. It’s next to the Filter drop-down. Save the result.

I convert the result from a csv to a text file. It’s a one-column list. Why get complicated?

You’ll compare your next crawl to this one and never, ever have to hand-review thousands of links again.

Step 4: Run A New Crawl And Compare

Run a new crawl and save it, just as you did in Step 3. I keep my old crawls and organize files by date. Compulsive. I know.

Now for the fun part! You’re going to compare the most recent crawl to the new one, looking for new links. Don’t fret — you don’t have to do this by hand. While computers suck at finding spam links, they excel at finding differences between files.

Tons of tools let you compare files. If you want something simple, I like Mergely.

Here’s what a comparison of the last and latest crawls looks like in Mergely:

mergely

Mergely File Comparison

The highlighted line is a new link. Easy!

Mergely might bog down with humungous files, though.

So I use the command line. On Linux, this command works like a charm:

‘comm –13 [oldfile] [newfile]’

That command compares oldfile to newfile, showing new stuff in newfile. Try File Compare (FC) on Windows. Your results may vary.

Here’s what a comparison of the last and latest crawls looks like in comm:

comm

Comm File Comparison

It’s fussier than Mergely, so you may get some false positives.

Review any new links. If they’re spam, you know what to do.

This is an excellent time to update your exclusion filter, too. See Step 2a, above.

Step 5: Repeat

Save the last crawl as your new baseline. When you run the next crawl, you’ll compare it to the last one. And so on. Repeat steps 3–5 as desired.

I run crawls every two to three weeks. If you have a faster-growing site, run crawls more often. It’ll make step 5 easier by reducing potential differences and delivering shorter lists of new links.

Enterprise-scale Alternatives

Our site is about 5700 pages. With the exclusion filter, our crawl generated a list of 2300 links. Hand-reviewing those isn’t all that bad. I divided it into chunks of 100, passed them around the office, and we finished reasonably quickly. If your site is millions of pages, you may need to use a crawler like Deepcrawl or OnCrawl.

You may need to look at machine learning as a spam link detection tool (there I said “machine learning” so this is now a legitimate marketing article). However, machine learning gets sticky when you’re sniffing for spam links that point at not-spammy pages.

Worth it?

Is all this work worth it?

Google won’t penalize you for trash links pointing to other sites. Probably. The linked sites don’t benefit from these links. Much.

It boils down to pettiness. I’m a petty person. I have an intense hatred of sleaze. Every spammer we found had “acquired” links from other sites. We contacted all those site owners. Then we reported every linked site to Google.

I have no idea if it’ll have an impact. But I sure feel better.

The post How to: Check Your Site For Spam Links Using Screaming Frog appeared first on Portent.

Read Full Article

Read for later

Articles marked as Favorite are saved for later viewing.
close
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 

Separate tags by commas
To access this feature, please upgrade your account.
Start your free month
Free Preview