Joey Hoer's Blog

Creating Medium-style Permalinks in Jekyll

2019-08-13T11:10:44-04:00

As I’ve previously written, information that is subject to change has no place in a URL. This is because a URL is an agreement to serve a specific piece of content from a predictable location for as long as possible. As Tim Berners-Lee wrote, “cool URIs don’t change.” With this in mind, designing a human-friendly bulletproof URL structure can be challenging.

Tim Berners-Lee recommends that webmasters avoid using the subject (i.e. post title or topic) in the URL. Despite the fact that using the subject often is human readable, the title has the potential to change over time. For instance, we know that keywords within headers and URLs impact search rankings and impact a user’s perception of credibility. For this reason, and whenever content changes, content authors and Search Engine Optimizers may want to experiment with title and keyword changes to improve search rankings.

We want a URL structure that is human readable, provides the flexibility to be changed periodically, but continues to serve the same content even if the URL changes.

Fortunately, we already have an example of a website that achieves just that—Medium. Medium assigns each post an ID which is appended to the post title.

Medium Post ID adbec59c7cf4

Minimal URL: https://medium.com/@torgo/adbec59c7cf4
Augmented URL: https://medium.com/@torgo/another-title-adbec59c7cf4
Full URL: https://medium.com/@torgo/in-defense-of-the-url-adbec59c7cf4

As can be seen here, the “title” portion of the URL is inconsequential—only the post ID matters. By appending the ID to the end of the URL, Medium ensures that keywords are closer to the root domain, and are less likely to be truncated within search results or elsewhere. I find this approach particularly elegant, so I attempted to recreate this URL structure within Jekyll.

The first step was to create a static ID that could be used within the URL. Because Jekyll has no persistent database, almost every value within Jekyll is subject to change. My goal here was to select a post attribute that would (almost) never change to use as the static ID. The attribute that I deemed least likely to change was the post date.

The creation date of the document—the date the URI is created—is one thing which will not change. … That is one thing with which it is good to start a URI. If a document is in any way dated, even though it will be of interest for generations, then the date is a good starter.

– Cool URIs don’t change, Tim Berners-Lee

The only time I could see the date becoming an issue is if I ever decided to change a post date, or if I ever published two posts with the exact same post date. Both scenarios are unlikely, but if either situation ever does occur it will require special consideration.

Once I had decided to use date as the static ID, I had to decide how I wanted to display that date within the URL. I could have used any combination ofyear-month-day, but the less specific I am with the date, the more likely I am to have a collision (e.g. if two posts are published on the same day). I could have easily added hour-minute-second to the date to be more specific, and that probably would work fine, but I didn’t want the date so blatantly displayed within the URL. After all, these are permalinks and I strive to may my content evergreen. I don’t want users thinking my content is outdated because the URL contains an old date.

It seems like Medium is using an ID in base64 notation, so I decided to encode the date using MD5 and then truncate the hash to a limited number of characters. To achieve this, I wrote a small Jekyll extension called jekyll-hashpermalink which adds a :static_id variable that could be used within the Jekyll permalink.

permalink: '/:categories/:title-:static_id'

Having added this new variable, I was able to append a static ID to my URLs, but this still did not solve the problem. If I changed the post title, the old URL would become invalid. I want a solution where the title can change without invalidating the URL. Effectively, I need the title to be ignored, such that only the static ID is used to serve content.

To achieve this next step, I had to change the way Jekyll creates the file structure when creating posts during compilation by overwriting the Jekyll::Document destination method. Instead of using the full url, which is Jekyll’s default destination, I replace this with just the static_id.

Once the files were writing to the correct location, they could be accessed using the static_id as the URL, but they could no longer be accessed with the full URL (including their title). To resolve this problem, I had to configure my server to serve content based on the static ID suffix rather than the full URL.

RewriteCond %{REQUEST_URI} -([^-]*)$
RewriteRule ^ %1/index.html

Caution: It may be pertinent to ignore specific subdirectories (e.g. the assets directory), or be more specific with the patten match to prevent unintended URLs from being rewritten.

Finally, I could successfully access all of my content using the static ID, and I could change the title at will without needing to manually define a 301 redirect after each change.

Duplicate Content

To prevent duplicate content issues, I ensured that a canonical link was present on the page that points to the URL with the latest title.

<link rel="canonical" href="{{ page.url | replace:'index.html','' | prepend: site.baseurl | prepend: site.url }}">

Always Accurate

I also added a tiny bit of JavaScript to rewrite the URL in browser, using history.replaceState, so that the URL the visitor sees will always match the current correct canonical URL, regardless of the URL they used to access the page.

// Shareable URLs
// Automatically replace URL with canonical URL
// Strip query parameters from the URL
canonical_pathname=document.querySelector('link[rel=canonical]').href.replace(/^.*\/\/[^\/]+/, '');
(function(l){window.history.replaceState({},'',(canonical_pathname||l.pathname)+l.hash)})(location)

Unfolding the Fold

2019-02-11T14:01:22-05:00

When discussing web design, I often hear clients refer to “the fold.” Generally there is some important element that they would like displayed prominently on a page, and they will request that the element is placed “above the fold.” As a frontend developer experienced in responsive web design, knowing that viewpoint sizes vary greatly, this concept of the fold is difficult to grasp. When we say “the fold”, what exactly are we referring to? How can we respectfully share our expertise with clients to arrive at a common understanding of how the fold applies to modern web design?

What is “the fold”?

The term “above the fold” originated in newspaper print design and refers to the content that is visible on the front page of the newspaper when it is folded. Usually this section of the page includes an intriguing headline accompanied by a large image. The goal of this content is to entice readers to pick up the paper and open it.

On the web, “the fold” similarly refers to the content visible on the page without scrolling. People who believe in “the digital fold” will say that the intention is the same—placing compelling content “above the fold” encourages visitors to read on and become more engaged with the website.

The Bottom Line

Content should be prioritized such that “above the fold” content demonstrates immediate value to the visitor, reaffirming their decision to visit the page.

That is not to say that all content, even the most important content must be placed above the fold. Simply that the first content which is displayed to the user should give them confidence that the page they are visiting contains relevant information to the content which they are seeking.

The Digital Fold

The main problem with the digital fold is that it is inconsistent. When a newspaper designer creates a layout for the front page, they know exactly how that content will appear to the reader—the medium is fixed. The printed broadsheet will always be 749mm by 597mm, and those physical dimensions will never change. This assumption cannot be made on the web. On the web, the fold will be very different on a hand-held device in landscape orientation when compared to an ultrawide desktop monitor in portrait orientation. A browser window may cover the entire screen, or just a portion of the screen. Simply put, the digital fold is unpredictable. With all of these various viewports (browser window sizes) being used to display the same content, it is impractical to believe that content which appears “above the fold” can be controlled.

Note: Modern browsers support viewport units which can allow designs to be created relative to the “above the fold” section. However, the usability of such a design may be questionable when presented within different contexts (viewports).

It’s easy to say “there is no fold.” Afterall, there is no fold. However, true as that may be, stating that fact does nothing to address very real concerns that the first content a user sees on a page is a determining factor in their assessment of the page’s credibility.

It is important to understand that everybody scrolls, as scrolling is a natural affordance of a webpage. However, the longer it takes a user to locate relevant content on the page, the more likely they are to abandon the page. For this reason, when conversations around “above the fold” content arise, it is important to steer the conversation towards a content planning and prioritization exercise rather than towards a design discussion, as the act of prioritizing content such that it is likely to quickly answer a visitor’s question will produce better results than attempting to perfect above the fold content/design across every viewport size.

The Effective Use of Story Points

2018-12-27T20:55:07-05:00

When managing a project of known scope, understanding the time required to complete the project (or a piece of the project) with available resources is critical to the ability to plan effectively and appropriately set expectations with stakeholders.

Naturally, workers of differing skill levels will complete a given task at different rates—i.e. a highly skilled worker will work more quickly than a low skilled worker. This fundamental truth of labor can present a significant challenge when estimating timelines in man-hours, as estimates produced by one worker may not hold true for another.

Story points may be used as an alternative to man-hours when planning projects, but before we start using story points, we must understand what story points are and why they are useful.

What is a story point?

At the most basic level, a story point is a unit of measurement. Story points measure the effort¹ required to complete a piece of work.

More accurately story points are an estimate of effort, as story points consider the fact that attempting to predict the future has inherent uncertainties. Story points do not attempt to be precise at a micro level (i.e. tasks), but allow more accurate predictions to be made at a macro level (i.e. projects).

Note: The term “estimate” is used in this article to refer to level-of-effort estimates used in project planning, these types of estimates are not intended to be used directly in a sales context and do not estimate monetary cost.

Note: For the remainder of this article I will use “points” and “story points” interchangeably.

What are story points used for?

Story points are typically used in Agile project management to make predictions about the amount of time required to complete a project with known resources (i.e. a given set of workers of known skill levels).

Story points are about time. There, I’ve said it, and can’t be more clear than that. … The primary reason for estimating product backlog items is so that predictions can be made about how much functionality can be delivered by what date. If we want to estimate what can be delivered by when, we’re talking about time.

– Story Points Are Still About Effort

Making timeline predictions with story points

To predict the remaining time required to complete a project using story points we must have two things:

We must know how much work must be done (i.e. the project estimate)
We must understand the team’s velocity

“Velocity” is the number of points a team has historically completed over a given amount of time (commonly a one-week or two-week “sprint”).

Note: I like to refer to the number of points an individual has completed over a given amount of time as that individual’s “throughput”. Though “individual velocity” may be a more correct term within Agile, I’ll use “throughput” in this article to differentiate between team velocity and individual velocity.

In the simplest terms, we can predict duration by dividing the total number of points by the team’s historical velocity.

estimated duration = \frac{total points}{velocity}

For example, if there are 100 story points remaining within a project, and our team can complete 20 points per week, we can expect that the project will require 5 weeks to complete.

\frac{100 points}{20 points/week} = 5 weeks

Story Points vs. Man-Hours

At this point, you might be wondering how using story points in this way is different from using man-hours—after all, if we have a project estimated at 100 hours, and our team can devote 20 hours per week to a project, then we can also predict the project will require 5 weeks to complete.

The crux of the issue is that, as we know, workers of differing skill levels will complete a given task at different rates. Estimating in man-hours tightly couples the skill set of an individual worker to the estimate. For this reason, hours-based estimates must be re-estimated when team assignments change (e.g. when team members are added or removed during the project), or when challenges emerge that impede the team’s performance.

Story points decouple estimates from individual workers, providing an abstraction to managers which offers more information about remaining effort. This allows managers to make more informed decisions when allocating resources. Using this method, level-of-effort estimates need not change when the team changes; all that must be done to recalculate the predicted timeline is to update the team’s velocity.

Let’s consider an example: Parker, an expert programmer, and Pat, a new employee, are both tasked individually with configuring a web server. The following table shows the actual hours each employee spent on the task.

Task	Parker’s Hours	Pat’s Hours
Configure a web server	4 hours	8 hours

Morgan, their manager, is asked by a new client, “How long will it take to configure a new web server?” … To answer this question accurately, Morgan must understand who is assigned to the task. This underscores the fact that duration is always a function of the resources assigned to a task. If resources change, when using hours-based estimates, the timeline required for completion must also be re-estimated.

Taking this example further—the client makes a request to build a new microsite. We want Parker and Pat to work on the project together, and we understand that this task is roughly 10 times the effort of configuring a web server. How quickly can we deliver this project? … Stumped? This too illustrates the difficulty of using man-hours for estimating completion.

Let’s try using story points to answer this question instead. First, we must have an agreed upon baseline to calibrate our story points. We will use the task of configuring a web server as the baseline, and assign 4 points to this task.

Task	Story Points
Configure a web server	4 points
Build a microsite	40 points

Note: The number of points assigned to the baseline task is not important, so long as we can measure other tasks accurately in relation to that task.

Remembering that duration is always a function of the resources assigned to the task, we must also determine how many story points both Parker and Pat are able to complete within a given time frame (throughput). Based on their previous work configuring the server, we can infer the following:

Team Member	Throughput
Parker	1 point/hour
Pat	0.5 point/hour

Note: The conversion rate between points and hours again is insignificant, so long as it is consistent across the team.

We can use this information to determine the team’s velocity by combining Parker and Pat’s throughput.

1 point/hour + 0.5 points/hour = 1.5 points/hour

We now have all the information we need to answer the question “how long will it take Ted and Pat to build a microsite?”

\frac{40 points}{1.5 points/hour} = 26.67 hours

In this way, managers may more easily and accurately predict timelines. Additionally, managers may reassign resources without having to reestimate individual tasks.

For instance, suppose Morgan needs to reassign Parker at the last minute, and replaces Parker with Jamie, an experienced programmer. Jamie can configure a server in 6 hours, and therefor has a throughput of 0.75 points/hour. Let’s see how this impacts the timeline:

\frac{40 points}{1.25 points/hour} = 32 hours

Morgan can now appropriately set stakeholders’ expectations or assign additional resources to the project.

Points to Actual Hours Conversion

Recall that story points are a measure of effort. Story points make the assumption that, over time, the average velocity of the team—the amount of effort they’re able to invest—will be reasonably constant. Assuming point value estimates are consistent across the team, estimates based on points and historical velocity should be fairly accurate and should not fluctuate wildly. Additionally, the longer this process is used, the more accurate it becomes.

Unfortunately, time is not a precise measurement of effort—as we’ve seen, one “one point” task might require 30 minutes to complete while another “one point” task might require two hours to complete, depending upon a number of factors. More accurately, one point equals a range of possible values representing a normal distribution centralized on a mean with some standard deviation. For this reason, story points cannot be directly translated into actual hours. It is possible to determine the mean, though it may not always be accurate.

The most important attribute of a story point is that it is a common unit of measurement across the entire team. A unit of measurement must remain constant to be an effective tool. If point estimates vary between team members then velocity cannot be tracked accurately, nor can velocity be used to make future predictions about expected timeline. Story points are meaningless if the team does not have general consensus on the value of a story point.

Calibrating the Team

Estimating in story points requires commitment from the entire team. Everyone must agree on what “1 story point” represents in terms of effort. Before a team can effectively use story points, the team must be “calibrated” so each member shares a common understanding of the value of a point. Selecting a baseline that is commonly understood will make calibrating the team easier.

When a team is properly calibrated, a points estimate should be able to be validated with any team member, and that team member should arrive at (roughly) the same point value assigned. Planning poker, a variation of the Wideband Delphi method, is one effective method for calibrating a team.

Another approach to calibrate the team is to use the highest or lowest performer to set the baseline. It may be easier for the team to agree “it should only take a high performer X hours to complete this task.” This was demonstrated in the previous example where Parker had a 1-to-1 points-to-hours ratio.

Communicating with Stakeholders

Presenting clients estimates in hours can establish expectations which may be difficult to revise. Because story points abstract hours estimates, clients cannot draw conclusions between story points and hours. However story points do offer the client the ability to compare the relative size of one task with another.

While this approach may effectively move focus away from hours estimates, it does not address the client’s primary concerns—namely, the price of a project, and the delivery date. If work is priced commensurate to the value the customer receives, if timeline is adequately communicated, and if there is trust established based on a history of meeting deadlines, then inputs (man-hours) will be irrelevant to the client.

The opacity of story points makes them an ineffective tool for managing client expectations. Timelines should always be delivered to the client as delivery dates rather than as a number of hours or number of points.

Story points do not measure complexity. Complexity is merely one factor which is considered when determining the level of effort—in addition to risk and uncertainty. ↩

Thoughts on AMP

2018-09-26T14:06:18-04:00

Google has a long standing tradition of introducing changes to it’s search engine designed to enhance user experience, and Google’s dominance as a search provider essentially mandates the adoption of Google’s prescribed best practices. AMP promises to make the web faster, but embracing AMP might be more complicated than you would expect.

What is AMP?

Accelerated Mobile Pages (AMP) is a web content publishing format, developed by Google, which aims to improve user experience on mobile devices by optimizing webpages for performance. This is accomplished using a non-standard subset of HTML which provides only limited access to JavaScript and other native web technologies. The most common implementation of AMP is to use separate page templates for AMP, though this is not a requirement of AMP.

Here is a typical AMP setup:

Original/Canonical URL: https://www.wired.com/story/google-artificial-intelligence-monopoly/
AMP URL: https://www.wired.com/story/google-artificial-intelligence-monopoly/amp
AMP Cache (Google)¹: https://www.google.com/amp/s/www.wired.com/story/google-artificial-intelligence-monopoly/amp

Note: In May 2018 Google announced plans to incorporate new technology into AMP that would allow content creators to use their own URLs within Google’s cache.

Despite the fact that Google is primarily responsible for both the AMP format and an AMP cache implementation within Google search results, it is important to understand that the format and cache implementation are technically independent elements. Other platforms like Twitter and Cloudflare offer their own AMP cache implementations, but for the purposes of this article, “AMP” refers to Google’s AMP cache implementation.

AMP is a polarizing topic which has received widespread criticism from the web development community. Some of the criticism stems from the fact that Google has created a “closed” format which is not governed by an open standards organization; this allows the AMP team to move quickly, but does not create an inviting atmosphere for community involvement. Additional controversy is precipitated by Google’s use of their dominant market position to drive the adoption of the format by giving AMP pages preferential treatment within mobile search results. Furthermore, Google’s management of content served from the AMP cache, which retains users within Google’s own domain, has also fueled frustrations.

Why does AMP exist?

Colin Chapman, the founder of Lotus Cars had a famous philosophy: “simplify, then add lightness.“ He contended that while ”adding power makes you faster on the straights, subtracting weight makes you faster everywhere.” This is the lens through which we examine website performance—less is more.

Unfortunately, with ever-increasing pressure to drive results, website managers are driven to seek out any opportunity that claims to show a positive return on investment. With a multitude of providers each claiming that adding their service will improve conversion and provide more powerful user insights, content creators are easily convinced that adding more to their websites is the answer to their problems. In this way too, website managers can easily demonstrate progress to their supervisors. As these additions accumulate, site performance decreases, leading to an unpleasant user experience. By this point, the damage has been done—it’s difficult to justify getting rid of anything that’s claiming to produce a positive return on investment, even when those claims are difficult or impossible to substantiate. As it turns out, convincing content creators to add functionality to their website is easy, convincing them to remove functionality—which is generally a requirement of improving performance—is hard.

So here we are. While web practitioners have long been aware of the importance of site speed, incentives and penalties like Google’s Site Speed algorithm update (2010) did little to curb the ever-increasing size of webpages. Since that update, the average number of assets requested on a mobile webpage increased by over 140% (from 29 to 70). Over the same period, the size of the average mobile webpage increased over 725% (from 200KB to nearly 1700KB). Despite the fact that there is plenty of data which supports the idea that better page performance equates to more time onsite and increased conversion rates, most organizations have not defined a performance budget to educate decisions and protect site speed and user experience, so webpages grow larger while the user experience becomes slower.

By failing to improve performance, content creators are missing out on opportunities. Instead, tech giants introduced their own solutions which redistribute content with improved performance—Facebook (via Instant Articles), Google and Twitter (via AMP), and Apple (via the News app)—which allow them to keep users on their platforms longer, and make more money through their own advertising.

[T]he emphasis on speed… with things like Instant Articles, native is making the browser-based web look like a relic even just for publishing articles… [which] might pose a problem even for my overwhelmingly-text work. [My website’s] pages load fast, but the pages I link to often don’t. I worry that the inherent slowness of the web and ill-considered trend toward over-produced web design is going to start hurting traffic to [my site].

– Facebook Introduces Instant Articles by John Gruber

The depressing truth of why AMP exists is that Google gave the community a chance to make the web faster—we didn’t. Instead we broke another piece of the web. In it’s brilliance, AMP is a simplification masquerading as an enhancement.

Benefits of AMP

AMP’s biggest advantage is the restrictions it draws on how much stuff you can cram into a single page.

— How Fast Is AMP Really? by Tim Kadlec

Google has encouraged adoption of the AMP format by offering content creators increased readership through heightened visibility within search results. This occurs in a number of ways:

A lightning bolt icon is displayed alongside AMP content within mobile search results.
AMP content is eligible to be promoted into AMP carousels within mobile search results (including the “Top Stories” carousel which is prominently displayed).
AMP content is cached by Google, served through it’s CDN, and preloaded within search results, allowing content to load “instantly”.

Speculation: Presumably the preferred placement in mobile search also impacts ranking in non-mobile search results, as Google announced that the “mobile index” would become the primary source of site rank in November 2016.

Criticisms of AMP

Critics of AMP believe that these benefits are an unfair anti competitive behavior due to the fact that non-AMP pages, which can be made to be faster than their AMP counterparts, do not get access to the same benefits as AMP pages. The result of preloading AMP content within search results is what allows AMP pages to load instantly; publishers can’t compete with that.

There is an axiom: “There is no right way to do the wrong thing.” I wonder now if the reverse is also true, or if Google has managed to do the right thing the wrong way. Google was able to leverage it’s search dominance to address performance in a way that no one else could, by offering business owners valuable incentives. However, because only AMP pages receive preferential treatment, only AMP pages are “instant”. Until business owners are convinced that overall performance is important, the web at large will continue to grow larger and load slower, which is the real problem that needs to be addressed.

AMP hasn’t solved the core problem; it has merely hidden it a little bit. … the incentives being placed on AMP content seem to be accomplishing exactly what you would think: they’re incentivizing AMP, not performance.

— How Fast Is AMP Really? by Tim Kadlec

In fact, even Malte Ubl, creator and tech lead of the AMP Project, has said “make AMP your own site.” The solution to addressing the problem of performance is to make your site faster, not to focus on AMP.

Content creators should also understand the trade-offs they are making when choosing to implement AMP. Unless content creators are willing to rebuild their existing websites, eliminating any functionality not supported by AMP, then publishers will need to assume the costs of creating and maintaining a second AMP version of each of their webpages. (Does this remind anyone else of m-dot mobile sites?) By using a closed format, and allowing Google to host their content, publishers are ceding control of their content to Google. Due to the nature of Google’s current AMP cache implementation, whereby AMP content is served from the google.com domain, publishers are accepting that their website (the content origin) may receive less traffic.

How does Google benefit from AMP?

Notice: This section contains speculation.

I don’t find it too hard to believe that Google can or will use AMP formatted content to fuel it’s AI initiatives. Scraping the web is a complicated business—enforcing a limited standard would allow Google to crawl and scrape content more easily, and use this data to power Google’s “Featured Snippets,” which are used to power the conversational UI of Google Assistant. Google confirmed that AMP links may be displayed in featured snippets of mobile search results in September 2017.

We also know that slow sites lead to fewer page views which lead to fewer ad impressions which results in fewer dollars in the pockets of ad platforms like Google. Google has a vested interest in maintaining a dominant position in digital advertising. In fact, the AMP format with all of it’s limitations on JavaScript and functionality, does support a variety of ads.

Of course, by writing the code for these new ad formats, [Google is] also putting itself in the middle of how those ads will be implemented, giving Google an ongoing place at the table for how the next generation of the mobile web will monetize.

— Ingrid Lunden

Speed without AMP

As a parting thought for any reader who is considering implementing AMP, I’d like to share some wisdom from a very entertaining talk by Maciej Cegłowski called The Website Obesity Crisis which succinctly describes how to build a fast website:

Nutritionists used to be big on this concept of a food pyramid. I think we need one for the web, to remind ourselves of what a healthy site should look like. … Here is what I recommend for a balanced website …:

A solid base of text worth reading, formatted with a healthy dose of markup.

Some images, in moderation, to illustrate and punch up the visual design.

A dollop of CSS.

And then, very sparingly and only if you need it, JavaScript.

… my simple two-step secret to improving the performance of any website.

Make sure that the most important elements of the page download and render first.

Stop there.

To visit Google’s cached AMP page, you must be on a mobile device. ↩

Static Site URL Shortener

2017-10-30T10:27:17-04:00

On Apache servers, a RedirectMap may be used to import bulk redirects. This provides an ideal solution for creating a self-hosted short URL service for any website, particularly static sites generated with Jekyll.

To implement a URL shortener using Jekyll and Apache, begin by simply adding a new key (e.g. shorturl) to the front matter of any relevant posts. For example:

---
shorturl: example
---

Next, create a new file (e.g. shorturls.txt) which contains the following liquid code:

{% for post in site.posts %}{% if post.shorturl %}{{ post.shorturl }} {{ site.url }}{{ post.url }}
{% endif %}{% endfor %}{% for page in site.pages %}{% if page.shorturl %}{{ page.shorturl }} {{ site.url }}{{ page.url | remove: "index.html" }}
{% endif %}{% endfor %}

Build the site. Once the site is built, a new text file should exist resembling the following:

example https://www.example.com/this-is-an-example

Once the redirect map file is created, instruct Apache to use the file by adding the RedirectMap to the Apache configuration (typically in httpd.conf, or an included vhost) by using the following:

<IfModule mod_rewrite.c>
  RewriteEngine On
  RewriteBase /
  
  # IMPORTANT: Make sure this file exists
  # RewriteMap shorturl txt:/path/to/shorturls.txt
  RewriteMap shorturl dbm:/path/to/shorturls.map

  # URL shortener with fallback
  RewriteRule ^(.*)$ ${shorturl:$1|https://example.com/$1} [R=302,L]
</IfModule>

Note: The RewriteRule above uses 302 redirects when redirecting, and allows non-existent paths to fall through/fallback to the unshortened URL.

This may be setup on a primary domain (which may introduce conflicts as new URLs are created), or this may be setup on a secondary domain (e.g. short domain) or subdomain to reduce the possibility of naming collisions.

It may be desirable to prevent direct access to the redirect map file; this can be accomplished with a simple RedirectRule:

# Disable direct access to short URL map
RewriteCond %{REQUEST_URI} ^/shorturls\. [NC]
RewriteRule ^ - [R=404]

Faster Redirects with DBM files

As the number of redirects grows, this solution may become slower (though the number of redirects would need to be very large). In any case, the httxt2dbm command can be used to create dbm files for use with RewriteMap. A dbm database file contains key/value pairs which map the redirect data; this works exactly the same way as a txt map, but it is much faster, because the dbm file is indexed. Rather than processing each line of the file until a match is found (which happens for txt files), the dbm file allows Apache to jump right to the relevant entry. This can greatly improve performance for large data sets.

httxt2dbm -i shorturls.txt -o shorturls.map

Note that with some dbm types, more than one file is generated, with a common base name. For example, you may have two files named mapfile.map.dir and mapfiile.map.pag. This is normal, and you need only use the base name mapfile.map in your RewriteMap directive.

– https://httpd.apache.org/docs/current/rewrite/rewritemap.html#dbm

This command may be introduced into build logic, allowing the created dbm file(s) to be created automatically whenever content changes.

Advanced nth-child Selectors

2017-10-05T10:56:51-04:00

Frontend developers have long known that CSS nth-child selectors may be combined to yield interesting and practical results. Most commonly, this technique is applied to select all items in a list when there are more than N children, less than N children, exactly N children, or between N and M children. However these simple selectors may also be combined in other ways to produce even more advanced selectors.

Modulo Selector

The CSS modulo selector can be used to select only the remaining items (quotient) after dividing a list of items by a number n. This can be used in conjunction with a grid system to ensure that all rows are filled evenly by hiding extra elements. This is particularly useful for responsive designs in which the number of columns changes between viewports, yet the desire is to keep all rows filled evenly.

Note: I recommend hiding only unnecessary elements, and I discourage loading more content than necessary on any device, particularly on mobile devices.

See the Pen CSS Modulo Quantity Query Selector by Joey Hoer (joeyhoer).

This technique is accomplished by selecting the final element in a complete group (i.e. full row) using nth-child(an) (where a is the divisor), and then counting backwards from the last element using nth-last-child(b) (where b is a number between 2 and the divisor) until that element is matched. Once the element is found, the general sibling combinator (~) is used to select the remaining items in the list.

/* Modulo 4 */
ul > *::nth-child(4n):nth-last-child(2) ~ *,
ul > *::nth-child(4n):nth-last-child(3) ~ *,
ul > *::nth-child(4n):nth-last-child(4) ~ * { }

Note: We begin with 2 because nth-last-child(1) will always be the last-child, and when nth-child(an) matches last-child, there is no remainder.

This approach works great except when n is 0, as there is then no element to match. This means that the first group is never selected. To overcome this challenge, an additional less-than quantity query may used to select items in an incomplete first group/row.

Middle Element Selector

The CSS middle element selector is used to select the middle element in a list of elements. This is particularly useful when used with the general sibling combinator (~) to select items in the second half of a list. This technique can be used add a different background to half of a list, and to anchor menus to the right rather than the left and prevent overflow on the right side of the screen.

See the Pen CSS Middle Child Selector by Joey Hoer (joeyhoer).

Unfortunately, this technique produces a lot of generated code, and while the required code is very repetitive and therefore quite compressible, it is inelegant because it is difficult to maintain and scale because a maximum number of elements must be known in advance.

This technique is accomplished using :nth-child(b) and :nth-last-child(b) in equal measure by incrementing b from the start until the median is matched. Because this technique requires a unique selector for each distinct quantity, the total number of required selectors is at least half of the maximum list size. If a list minimum is also known, the number of overall selectors required may be reduced further.

/* Exact middle element of a list of 6–14 elements */
ul > *:nth-child(3):nth-last-child(3),
ul > *:nth-child(4):nth-last-child(4),
ul > *:nth-child(5):nth-last-child(5),
ul > *:nth-child(6):nth-last-child(6),
ul > *:nth-child(7):nth-last-child(7) {}

Lists with an even count present an issue, as they do not have a median. In this case, the developer must choose either to select the preceding element, the following element, both elements or neither. Selecting both elements requires an additional CSS selector, effectively doubling the number of selectors required.

These selectors may combined with the general sibling combinator (~) to select the items in the second half of the list. Because the general sibling combinator is not inclusive (meaning that the current element is not selected) it will typically be desired to target the “after” method when selecting even numbered lists, and add an additional selector to make the list inclusive. This technique also doubles the number of selectors required.

There is no way to select only the first half of a list using this method.

Half of Elements Selector

To work around the limitations of the middle element selector, the CSS half of elements selector can select both the first half of a list, and the second half of a list using a fraction of the number of required selectors.

See the Pen CSS Half Selector by Joey Hoer (joeyhoer).

This technique is accomplished using :nth-child(-n+b) and :nth-last-child(n+b), incrementing b until the median is matched. The n value overlaps either in a positive or negative direction from the median, based on the signs associated with the nth-child and nth-last-child equations. Because this technique requires a unique selector for each distinct quantity, the total number of required selectors is at least half of the maximum list size. Fortunately, this technique is also very repetitive and compressible.

/* First half of elements in list of 1–14 elements */
ul > *:nth-child(-n+1):nth-last-child(n+1),
ul > *:nth-child(-n+2):nth-last-child(n+2),
ul > *:nth-child(-n+3):nth-last-child(n+3),
ul > *:nth-child(-n+4):nth-last-child(n+4),
ul > *:nth-child(-n+5):nth-last-child(n+5),
ul > *:nth-child(-n+6):nth-last-child(n+6),
ul > *:nth-child(-n+7):nth-last-child(n+7),
ul > *:nth-child(-n+8):nth-last-child(n+8) {}

Lists with an odd count present an issue, as they cannot be divided evenly. In this case, the developer must choose whether to select the median element; doing so does not increase the number of selectors required.

The URL is Dead, Long Live the URL

2017-08-20T19:16:03-04:00

I care about URLs. I have spent time ensuring the URLs for this site are clean, concise, uniform, readable, and easily sharable. I subscribe to the idea that cool URIs don’t change and that URLs are UI. Even while I recognize that the “appification” of the web has minimized the importance of URLs, and I expect URLs to become increasingly niche as computer interfaces become conversational, the URL remains the defining feature of “the web” as we know it today, and I believe we should design URLs to embolden visitors with confidence.

The web has always been a nebulous concept, but at its center is the idea that everything can be linked.

– Rethinking What We Mean by ‘Mobile Web’, John Gruber

The Long, Slow Death of the URL

Anything that reduces the prevalence and usefulness of cross-site linking is a direct attack on the founding principle of the Web.

– Fighting Linkrot

In September 2013, iOS 7’s Mobile Safari began hiding all but the domain of visited URLs. Eight months later in April 2014, the Chrome team experimented with burying the URL. The experiment was met with widespread criticism from the development community—Paul Irish called it a “very bad change [which] runs anti-thetical to Chrome’s goals”—and the feature was “backburnered” a month later, and declared “dead” in 2015. Nevertheless, when OS 10.10 was released in October 2014, Apple introduced the “Smart Search” field to Safari 8 which brought the URL hiding features of iOS 7 to the Mac.

Shortly after the advent of installable PWAs in Chrome in 2015, the Chrome team launched Lighthouse, a tool which checks whether a website meets Google’s criteria for PWA compatibility. The tool initially encouraged site owners to configure the site to “allow launching without address bar”—a recommendation which was eventually removed after a lengthy discussion (once again involving Paul Irish) on GitHub and heated discussion around the web.

While that battle seems to have been won, for the time being, the URL fell under attack again when Google released AMP (Accelerated Mobile Pages) in February 2016. AMP content is cached on Google’s servers (to improve performance), and uses a non-standard URL format—once again, this garnered widespread criticism from the development community. In February 2017, Google updated AMP to allow users “to access, copy, and share the canonical URL of an AMP document … [via] an anchor button.” The experience remains suboptimal however, and has been dubbed by John Gruber as begrudging UI. iOS 11 may work around this issue; in iOS 11, when AMP pages are shared via the built-in share sheet, the canonical URL will be shared instead of the AMP URL. According to an AMP Tech Lead, Google “specifically requested Apple (and other browser vendors) to do this … [as] AMP’s policy states that platforms should share the canonical URL of an article whenever technically possible.”

The killer feature of the web—URLs—are being treated as something undesirable because they aren’t part of native apps. That’s not a failure of the web; that’s a failure of native apps.

– Regressive Web Apps, Jeremy Keith

Anyone witnessing this must ask themselves: “Why are we, the development community, so passionately protective over URLs? And why are these large internet companies so seemingly hellbent on devaluing them?”

The answer lies in conversational UI.

Defending the URL

Those who defend the URL do so because URLs are universal. URLs work in every conceivable browser, on every conceivable device, in command line utilities without any UI at all, on billboards and business cards … URLs are extremely portable.

The portability of the URL is part of what has made it such an important social tool. In 2012, the Atlantic estimated that 56.5 percent of their social referrals came from “Dark Social” (i.e. direct URL sharing). At the same time, Chartbeat told the Atlantic that direct URL shares accounted for 69 percent of overall social referrals on sites that Chartbeat managed. More recent reports from RadiumOne suggests Dark Social may now account for as much as 84 percent of social referrals.

The URL and the ability for anyone to mint a new one and then propagate it is what makes the web so resilient, so empowering, and so interesting! That I don’t need to ask anyone permission to create a new website or webpage is a kind of ideological freedom that few generations in history have known!

– The death of the URL, Chris Messina

URLs are powerful tools that work everywhere. Except … when it comes to voice.

I’m sure you hear URLs on the radio. You may have even used them in a conversation, but ultimately anything beyond a domain name becomes incredibly difficult to communicate audibly. Those who look forward to conversational UI will argue that URLs don’t actually work on “every conceivable device” but only in text-based mediums.

Conversational UI

Jakob Nielsen predicted that speech could be the death of the URL in URL as UI in 1999:

Domain names may die … In the long term, it is not appropriate to require unique words to identify every single entity in the world. That’s not how human language works.

When I look at virtual “personal assistants” like Amazon’s Alexa, Google Assistant, Apple’s Siri, Microsoft’s Cortana, and Samsung’s Viv, it’s easy to see why the URL it starting to be treated like a second class citizen. Conversational UI will become a huge part of human life in the not-so-distant future, and that is the true villain of the URL. The awesome and terrifying power of conversational UI is that data is aggregated, stripped of sources, and regurgitated as fact. Speed is valued over clarity. The reckless devaluation of the URL is the first step towards a future, where queries are spoken and responses can truly be accessed anywhere, anyhow … and (if we are not careful) where fiction becomes fact.

And here's what happens if you ask Google Home "is Obama planning a coup?" pic.twitter.com/MzmZqGOOal
— Rory Cellan-Jones (@ruskin147) March 5, 2017

Our ability to understand who we’re talking to on the web is underpinned by search engines.

– Rethinking URL bars as primary browser UI

Why is Google leading this charge? Because they are the only one’s who can. The cataloging and indexation of data has always been Google’s domain. Now, with forays into AI and neural networking, Google is in a position to turn the web into a closed API for clean, aggregate data.

Isn’t [Google] trying to guarantee the open Web stays alive? Not necessarily. [Google]’s goal is to gather as much rich data as possible, and build AI. Their mission is to have an AI provide timely and personalized information to us, not specifically to have websites provide information. Any GOOG concerted efforts are aligned to the AI mission.

– The Web Began Dying in 2014, Here’s How

We see examples of this in Google’s “Featured Snippets”, which have been referred to as Google’s “One True Answer”, “Direct Answer” and “Rank #0”. Featured Snippets scrape content from websites and display only the relevant data directly at the top of Google’s search results and power Google’s Assistant. Some argue that this eliminates the need to visit the originating page. While the URL is displayed, the value of the link is diminished, the content has already been delivered, the “knowledge” gained. In September 2017, Google confirmed that AMP links may be displayed in featured snippets of mobile search results, providing searchers even less incentive to visit the originating website. The heart of the matter is can we trust Google?

A big corporation (in [this] case Google) decides what is good for [us] and what [we] can find … and buy and use.

– The death of the URL, Michael Bumann

By removing our ability to navigate, choose, and share freely … [these companies are] exchanging our freedom for a promise that they’ll keep us safe, give us everything we need, and do all the choosing of what’s “good enough” for us …

– The death of the URL, Chris Messina

Well “You can’t blame me”, says the media man
… I just point my camera at what the people want to see
Man it’s a two way mirror and you can’t blame me”

– Cookie Jar by Jack Johnson

Fortunately, Google is taking responsibility and actively trying to cleanup Featured Snippets. As a consumer, I admire Google’s initiative. I appreciate getting answers at the touch of a button, at the tip of the tongue, without having to pour over pages of content, links, and advertising just to get at the good bits. It saves me valuable time, and on the web, time is capital. At Google, content is cheap.

URLs are the web. They are critical to assessing credibility in the digital age.

Microsoft Research conducted an eyetracking study of search engine use that found that people spend 24% of their gaze time looking at the URLs in the search results.

We found that searchers are particularly interested in the URL when they are assessing the credibility of a destination. If the URL looks like garbage, people are less likely to click …

– URL as UI, Jakob Nielsen

The war on the URL is far from over. If we want the URL to live on, it is our responsibility, as web practitioners, to design URLs that are easily spoken and easily remembered.

Designing URLs for Humans

Note: This section contains technical details of how to implement URL designs in Apache 2.4 ; be aware of your server type and version, as they may be different.

URLs are prime real estate. Invest in your URLs. Make URLs a priority.

URLs are for humans. Design them for humans.

Any regular semi-technical user of your site should be able to navigate 90% of your app based off memory of the URL structure. In order to achieve this, your URLs will need to be pragmatic.

– URL Design, Kyle Neath

Designing URLs mostly means leaving information out. When designing URLs there are two basic elements to keep in mind: simplicity and consistency. These basic elements can be applied to each of the following rules for URL design. URLs should be:

short
easy-to-type
hierarchical (visualize the site structure)
malleable (enable navigation by removing segments of the URL)
persistent

I want people to be able to copy URLs. I want people to be able to hack URLs. I'm not ashamed of my URLs …I'm downright proud.
— Jeremy Keith (@adactio) May 23, 2016

URL Length

The full, indexed URL (including the scheme and host) should be between 75–115 characters in length to allow the full URL to be displayed in SERPs and preserve the readability and shareability of links.

In 2007, a MarketingSherpa study found that shorter URLs were 250% more likely to be clicked in search results when directly compared with longer URLs.

URL Simplicity

Use only lowercase letters, digits and hyphens (i.e. dashes, -) in URLs.

Use short, full, and commonly known words. If a path segment has a dash or special character in it, the word is probably too long.

– URL Design, Kyle Neath

Hyphens vs Underscores

While they are technically permitted as “unreserved characters” in URIs, do not use tildes or underscores (i.e. ~ or _) in URLs, and use periods only when file extensions are required.

Hyphens feel native in the context of URLs as they are allowed in hostnames and domain names, while other characters (such as underscores) are not. Additionally, hyphens are easier to type—no need to use the ⇧ key—and hyphens are not obscured by underlines. Also, Google recommends using hyphens instead of underscores.

While emoji domains are cool (I own a few myself: 🌩.ws and ☄.ws), non-ASCII characters are difficult to type and sometimes difficult to differentiate. If you are targeting an English speaking audience, use only unreserved ASCII characters in URLs.

When using a post or product title (page subject) in the URL, replace spaces and special characters with hyphens. Typically apostrophes and quotes may be removed altogether. Multiple sequential hyphens should be collapsed to a single hyphen.

Lowercase URLs

Browsers and search engines treat each unique URL as if it contains unique content, because it is technically possible, through web server configuration, to serve different content at each distinct URL. If distinct URLs are not appropriately redirected or canonicalized, browser caching, edge caching and search engine results may be impacted. As best practice, redirection should be preferred over canonicalization.

While URLs themselves are case sensitive (to support file systems which are case sensitive [e.g UNIX]), domain names and hostnames make no distinction between upper and lower case. Use only lowercase letters in URLs to standardize their appearance and avoid technical issues.

To ensure that all URLs are appropriately redirected to their lowercase equivalent, include the following RewriteMap in your site’s Apache configuration files (e.g. httpd.conf or vhosts.conf:

# Add map to force lowercase URLs
RewriteMap lowercase int:tolower

Then, in your .htaccess file, add the following redirect:

# Lowercase URLs
# Skip case sensitive files/directories manually
# Skip downloads directory
RewriteCond %{REQUEST_URI} !^/downloads/
# Skip dotfiles
# Important for letsencrypt challenge authentication
RewriteCond %{REQUEST_URI} !/\.
RewriteCond $1 [A-Z]
RewriteRule (.*) ${lowercase:%{REQUEST_URI}} [R=301,L]

Apache’s mod_speling may also be used to resolve this issue by letting the server decide which file to serve.

Remove File Extensions

URLs should be independent of implementation. File extensions generally denote the underlying technology powering a website—information which is irrelevant to end users.

Note: Files which are intended to be downloaded typically should include an extension.

One benefit of using extensionless URLs is that if you decide to migrate from one language to another (e.g. ASP to PHP), the URLs will not need to be changed or redirected (which always results in a slight SEO penalty and dip in organic traffic).

To enable Apache to serve extensionless URLs, add the following RedirectRule to your .htaccess file:

# Support extensionless paths
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !-l
RewriteRule ^([^\.]+[^/])/*$ $1.html [L]

In this setup, the files will be available both at the extensioned path (with the .html) and without. If you do not canonicalize these files, search engines may flag these distinct URLs as duplicate content.

To resolve this, redirect or canonicalize the URLs, or disable access to the extensioned URL altogether:

# Disable direct access to files with extensions
RewriteCond %{REQUEST_URI} \.(p?html?|php)$ [NC,OR]
# Disable direct access to index files
RewriteCond %{REQUEST_URI} /index$ [NC]
RewriteCond %{ENV:REDIRECT_STATUS} ^$
RewriteRule ^ - [R=404]

Another solution is to enable content negotiation in your web server. Your web server will attempt to serve the correct file type, based on Accept headers provided in the request and naming conventions followed on the server.

Trailing Slash

Adding a trailing slash to URLs is largely a matter of preference; there is no distinct technical or SEO advantage to trailing slashes. Be consistent. Avoid mixing URLs with trailing slashes and non-trailing slashes on your site, and be sure to redirect or canonicalize URLs appropriately.

Trailing slashes present some important technical considerations for relative URLs. For example:

With Trailing Slash: resource relative to /location/ is /location/resource
Without Trailing Slash: resource relative to /location is /resource

This behavior mirrors UNIX file systems in which:

Slashes denote directories (as / is a directory separator)
File names do not require a file extension, but cannot contain slashes

Using a trailing slash may make some content more portable, easier to organize, and could save some characters when linking to content via relative URLs. However, a site/page would need to be architected very carefully to take advantage of these benefits.

The question you must ask yourself is: should “directories” always include a trailing slash, or should path segments only include the slashes between hierarchical ancestors? For example, if we navigate upwards from the URL http://example.com/collection/item, which of the following URLs is cleaner?

http://example.com/collection
http://example.com/collection/

Essentially, a trailing slash indicates that a URL is part of a hierarchical data structure, meaning the URL contains children or may contain children in the future. If “directories” should always be labeled as such, with a trailing slash, then URLs should probably include the trailing slash, so that URLs will not need to change as a site expands.

Keep in mind, bare domain names are supposed to include a trailing slash, as according to RFC 2616 §5.1.2, “if [no absolute path] is present in the original URI, it must be given as /”

You can confirm this by curling a bare domain name:

$ curl -sILo /dev/null -w %{url_effective} google.com
http://www.google.com/

However, modern browsers strip the slash from bare domain names.

And yet, they still appear in Google search results…

At the time of writing, Google, Wikipedia, GitHub, Stack Overflow, Twitter and Medium omit the trailing slash, while Facebook, WordPress,reddit and Apple include the trailing slash.

A subjective opinion is that a trailing slash on a URL is like a period at the end of a sentence—it balances the URL and provides closure. On the other hand, omitting the trailing slash is one less character to type.

Because neither solution is objectively superior, I have included Apache rewrites for both solutions below.

To add trailing slashes to URLs:

# Add trailing slashes to URLs
# Ignore files with extensions
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !-l
RewriteCond %{REQUEST_URI} !(\..*|/)$
RewriteRule .* $0/ [R=301]

To remove trailing slashes from URLs:

# Remove trailing slashes from URLs
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !-l
RewriteRule ^(.*)/+$ $1 [R=301]

Collapse Multiple Slashes

In most web server implementations, URIs map to an on disk path (this can vary depending upon configuration). Because modern operating systems do not assign any special meaning to sequential path separators (i.e. /), paths which include multiple path separators (e.g. /path/to/file and /path//to///file) map to the same file. Therefore the following URLs (which are valid according to RFC3986 §3.3) map to the same location by default:

https://example.com/path/to/file
https://example.com/path//to///file

While the same content may be served at both URLs, browsers will not munge the URL by automatically collapsing multiple slashes. This is because it is possible, through web server configuration, to serve different content at both URLs. Browsers and search engines must account for these potential differences and treat each unique URL as unique content unless they are explicitly told not to do so. Because of this, multiple slashes in URLs may affect browser caching and edge caching as well as search engine results if the URL is not appropriately redirected or labeled with a relevant canonical URL.

Multiple slashes also present issues with relative URLs. In relative URLs, “dot-segments” are used to traverse “the URI path hierarchy” (e.g. the .. path segment moves up one “level” in the path hierarchy—equivalent to “parent directory” in the filesystem). If the base element is not correctly defined on a page, multiple path separators will affect the ability to properly navigate the path hierarchy with relative URLs.

As such, it is best practice to collapse multiple slashes in URLs.

In Apache, multiple slashes may be collapsed with the following RewriteRule:

# Collapse multiple slashes in URLs
RewriteCond %{THE_REQUEST} //
RewriteRule .* /$0 [R=301,NE]

Stripping URL Parameters (Query String)

Many third party services (like email providers, social sites and affiliates) add query parameters onto URLs shared through their platform or service, so their service will be identified in your site’s analytics, and (assuming you’ve installed the corresponding JavaScript “pixel”) so they can track your traffic themselves.

While this benefits site owners and affiliates, these URLs sacrifice user experience. These additional parameters lengthen the URL, making it difficult to read and unpleasant to share. These parameters may also be inadvertently shared when visitors copy and paste URLs, which can skew analytics data.

Even if you do not directly advertise your site through such services, because of the nature of the web, anyone can link to your site through one of these services and these parameters will be displayed to your visitors.

Help users understand the content of the current view by removing unnecessary query parameters from URLs.

One method to scrub query strings from URLs is to use your web server to redirect to the same URL without the query string:

# Remove query string from the URL
# Note: In Apache 2.4 or later, the "QSD" option (qsdiscard)
# can be used to remove the query string opposed to
# appending a "?" the the substitution string
RewriteCond %{QUERY_STRING} .
RewriteRule (.*) /$1? [R=301,L]

The disadvantage of redirecting at the server level is that once the page has been redirected, the query string is no longer available to JavaScript based analytics tools (like Google Analytics). Ideally, you would clean the URL and track the data correctly in analytics tools.

A better solution is to strip the query string from the URL once the page has loaded and a pageview has been tracked. This is possible using JavaScript’s history API.

// Shareable URLs
// Strip query parameters from the URL
(function(l){window.history.replaceState({},'',l.pathname+l.hash)})(location)

This removes the entire query string. Sometimes some URL parameters will be required for rendering a unique view; in that case, you will need a solution that allows these necessary query parameters to be whitelisted. This can be accomplished with a more advanced JavaScript solution like Fresh URL.

Redirect to secure www

Use https for the SEO benefits; it’s just so easy that there is really no excuse.

In 2015, Chrome 46 introduced changes to mixed content warnings in the address bar to “encourage site operators to switch to HTTPS sooner rather than later”, also announcing a plan to eventually label all HTTP pages as non-secure. In 2016, Chrome 56 began labeling HTTP pages with password and credit card fields as “not secure” and reiterated the intent to move toward two states, secure and not secure, and has been marching steadily towards that goal. Google recommends using “HTTPS everywhere” as a long term solution. In 2018, with the release of Chrome 68, Chrome began marking all HTTP sites as “not secure”.

Additionally, http2 (which introduces performance benefits) is currently supported only over HTTPS and “powerful features” of HTML5 require HTTPS. These features include:

Geolocation
Video and Microphone
Device motion / orientation
Fullscreen
Pointer locking
Service Workers
AppCache
Notifications
Web Bluetooth

Not using HTTPS can also impact your site’s analytics data. When a visitor arrives on an HTTP site from an HTTPS referrer via a link, the referrer is stripped; this traffic is labeled “dark” traffic (i.e. visits for which the origin cannot be determined) which Google Analytics records as “direct” acquisition. In contrast, traveling from an HTTP or HTTPS site to an HTTPS site will properly attribute the referral data in Google Analytics. While it is possible for external site owners to work around this by establishing a referrer policy, may sites do not.

HTTPS also prevents “man in the middle” attacks and third party content injection by malicious agents and internet service providers (ISPs) which can insert advertisements, notices, tracking code and malware and hijack traffic. The threat is real. Accordingly, HTTPS is required for government agencies.

Most domains should use www, because www prepares a site to handle the challenges of a multi-server architecture with multiple subdomains.

At the time of writing, Google, Wikipedia, Facebook, reddit and Apple redirect to www, while GitHub, Stack Overflow, Twitter, Medium and WordPress redirect to non-www.

Below I have combined the https and www redirects to reduce the number of redirects. Redirects slow page load time and leak SEO value—eliminate unnecessary redirects whenever possible.

# Redirect insecure, www traffic to secure, www
RewriteCond %{HTTPS} !on [OR]
RewriteCond %{HTTP_HOST} !^www\. [NC]
RewriteCond %{HTTP_HOST} ^(?:www\.)?(.*) [NC]
RewriteRule (.*) https://www.%1/$1 [R=301,L]

If you really don’t care about the benefits of www:

# Redirect insecure, www traffic to secure, no-www
RewriteCond %{HTTPS} !on [OR]
RewriteCond %{HTTP_HOST} ^www\. [NC]
RewriteCond %{HTTP_HOST} ^(?:www\.)?(.*) [NC]
RewriteRule (.*) https://%1/$1 [R=301,L]

URL Hierarchy

Hierarchical URLs make it easier for users to navigate and search a site via “virtual breadcrumbs”, enable site owners to gather more detailed analytics data, enable easier URL migrations via redirection, and may enable more efficient crawling from search engines.

Namespaced URLs

One way to easily add hierarchy to URLs is through namespacing. Namespacing allows you to create new “top level sections” by organizing content into specific verticals. For example, rather than having blog posts live at the same level as your main content, namespace all blog content to /blog/. This keeps blog URLs isolated from the rest of your content. It’s also easy for users to remember this convention with continued usage.

Github does a great job of namespacing their URLs. For example:

https://github.com/joeyhoer: Every path beyond this URL belongs to me
https://github.com/joeyhoer/starter: Every path beyond this URL belongs to my project starter
https://github.com/joeyhoer/starter/issues: Every path beyond this URL references an issue on my project starter

The simplicity and consistency in Github’s URL design has allowed them to scale to millions of users with millions of repositories.

URL Malleability

URLs should be malleable (“hackable”)—by enabling users to navigate down the URL tree (to the left) by removing segments from the URL—and easily guessable.

Pages should always work without query strings attached.

I like to think of querystrings as the knobs of URLs — something to tweak your current view and fine tune it to your liking.

– URL Design, Kyle Neath

I’ve written at length about why query parameters should be used for facets and sorts. Query parameters are great for URL malleability, because even if a user attempts to hack a URL and “gets it wrong”, they’ll still be on a page with relevant content.

URL Persistence

Any URL that has ever been exposed to the Internet should live forever: never let any URL die … Remember, people follow links because they want something on your site: the best possible introduction and more valuable than any advertising for attracting new customers.

– Fighting Linkrot, Jakob Nielsen

Once a URL has been minted, you have made a commitment to serve relevant content at that location for as long as possible.

It is the duty of a Webmaster to allocate URIs which you will be able to stand by in 2 years, in 20 years, in 200 years. This needs thought, and organization, and commitment.

– Cool URIs don’t change, Tim Berners-Lee

Once you’ve launched a new URL, links to that location are no longer under your control. You never know where links to your URLs may exist.

A URL is an agreement to serve something from a predictable location for as long as possible. Once your first visitor hits a URL you’ve implicitly entered into an agreement that if they bookmark the page or hit refresh, they’ll see the same thing.

Don’t change your URLs after they’ve been publicly launched. If you absolutely must change your URLs, add redirects …

– URL Design, Kyle Neath

When content is not maintained (i.e. when content is removed/archived) during an upgrade and URLs are not appropriately redirected, links from various locations die. Linkrot refers to the tendency for links to increasingly point to resources that have become permanently unavailable over time. The average life span of a web page is 9.3 years.

Linkrot equals lost business: make sure all URLs live forever and continue to point to relevant pages.

– URL as UI, Jakob Nielsen

URLs that no longer return relevant content erode the credibility of your site and all sites linking to that data.

When someone follows a link and it breaks, they generally lose confidence in the owner of the server. They also are frustrated - emotionally and practically from accomplishing their goal.

Enough people complain all the time about dangling links that I hope the damage is obvious. I hope it also obvious that the reputation damage is to the maintainer of the server whose document vanished.

– Cool URIs don’t change, Tim Berners-Lee

By the way, changing URLs can also have an impact on social sharing.

Naming URLs

URLs change when information changes. If information is subject to change, it has no place in the URL. The first step in selecting a naming convention for URLs is identifying common attributes, unique to each piece of content, which will not change over time.

Tip: For ecommerce websites, SKUs are typically the best example of a unique identifier that should not change.

Using the date which the content is published is a good place to start:

The creation date of the document—the date the URI is created—is one thing which will not change. … That is one thing with which it is good to start a URI. If a document is in any way dated, even though it will be of interest for generations, then the date is a good starter.

– Cool URIs don’t change, Tim Berners-Lee

Using the date as the sole indicator for the URL presents a number of challenges. Namely, the date does not convey any relevant information about the content of the linked page—information important for search engines and users making decisions about the quality of a link. Dates also do not scale well for sites which may have multiple pieces of content being published on the same day, the same hour, the same minute, and sometimes even in the same instant. Additionally, users are quick to judge content relevancy based on the date posted and have a prejudice against outdated content. “Information architects” (typically UX designers) avoid dating URLs because they want content to appear evergreen (always relevant); instead they choose to use the post title/subject as the URL.

Using the post title in the URL presents it’s own issues:

leave out … [the] subject. … It always looks good at the time but changes surprisingly fast. … This is a tempting way of organizing a web site—and indeed a tempting way of organizing anything … but has serious drawbacks in the long term. … When you use a topic name in a URI you are binding yourself to some classification. You may in the future prefer a different one. Then, the URI will be liable to break.

– Cool URIs don’t change, Tim Berners-Lee

As an example, imagine you’ve written a piece of evergreen content titled “Google’s Various Entreprises” and you discover that you’ve misspelled “Enterprises”, and then Google changes its name to Alphabet, and perhaps you decide later that “Various Ventures” has a nice sound and would be better suited to SEO. What happens to the URL then? Either the URL must change (and be redirected) or your content’s address is permanently mislabeled.

Medium and Stack Overflow have both come up with interesting solutions to this issue. Both companies assign an ID to each post and include the ID in the URL along with the current post title. The important thing to note is that in these romanticized URLs the title in the URL is entirely optional. For example:

Stack Overflow Question ID 10712555

Minimal URL: https://stackoverflow.com/q/10712555/
Augmented URL: https://stackoverflow.com/questions/10712555/another-title/
Full URL: https://stackoverflow.com/questions/10712555/gitignore-all-files-of-extension-in-directory/

Medium Post ID adbec59c7cf4

Minimal URL: https://medium.com/@torgo/adbec59c7cf4
Augmented URL: https://medium.com/@torgo/another-title-adbec59c7cf4
Full URL: https://medium.com/@torgo/in-defense-of-the-url-adbec59c7cf4

As long as the ID appears in the correct location in the URL, the URL redirects to the correct content. This solution works because it creates persistent URLs that are flexible enough to support romanticized SEO titles which may change over time while enabling scalability.

reddit also takes as similar approach, while YouTube uses an 11 character base64 encoded ID for each uploaded video. This technique allows these companies to easily scale into the foreseeable future.

Migrating URLs via Redirection

A solid URL migration strategy is a critical element of any site migration. Do not neglect to redirect URLs appropriately when changing technology and migrating content.

One-Time Use URLs

When a user completes and action, they are often presented with a success page/message. Ideally, every user-targeted view on a site would have a unique URL that can be saved. Typically, refreshing a “success page” will return the user to a previous state in the conversion process, meaning success pages cannot be saved or shared. One time use URLs should be avoided. Allow users to refresh the success page to retrieve their order number, subscription information, etc. Better yet, add useful information to that page, just in case the user needs to change something.

In the past, the development community loved to create URLs which could never be re-used. I like to call them POST-specific URLs—they’re the URLs you see in your address bar after you submit a form, but when you try to copy & pasting the URL into a new tab you get an error.

There’s no excuse for these URLs at all. Post-specific URLs are for redirects and APIs—not end-users.

– URL Design, Kyle Neath

Fixing Vertical Videos

2017-07-25T11:46:32-04:00

In April 2017, Redditors made a compelling argument against blurred pillarboxing. The primary complaint of this approach is that it increases the file size, which poses a challenge for mobile devices where data is scarce and expensive, thereby increasing the load time. Additionally, pillarboxing makes the video smaller and difficult to see.

Vertical video presented in any widescreen medium experience similar issues. When viewed on a laptop or desktop (typically 8:5), vertical video embedded in a web page alongside content may either look too small or push content far down the page. In the most dramatic examples (where a vertical video “covers” the screen, nearly 65% of the video will remain unseen.

History

This technique was popularized by television news where vertical video is routinely edited in with widescreen 16:9 video. Maintaining a consistent aspect ratio throughout the program has a more professional feel and can prevent some televisions from stretching the video to fill the screen (which might happen when videos are framed by black bars).

Today, more than half of YouTube views come from mobile devices, while globally, 65% of Facebook video views occur on mobile.. According to Cisco, mobile video traffic accounted for 60 percent of total mobile data traffic in 2016. Cisco also predicts over three-fourths (78 percent) of the world’s mobile data traffic will be video by 2021. Because people naturally hold their phones in “portrait orientation”, more photos and videos are inevitably being produced in portrait. Likewise, more content is being produced to target these audiences.

Complaints

The internet loves to complain about the use of vertical video, particularity when it comes to blurred pillarboxing, and rightly so.

By adding that blurry border … the image size [increased] by 16/9*16/9, or a factor of 3.16 … because image compression of large blurry areas is rather efficient, the file size is probably about doubled. Which uses twice as much data for mobile users (and makes it load half as fast).

Also, it makes it impossible to see the video clip full size on cellphones, decreasing the size to 1/3 its original size.

In summary, you’ve doubled the file size, doubled the loading time, and shrunk the image size to 1/3. I’m so glad I see this all the time now.

– /u/sloppyjoes7

Additionally:

Not to mention the it ruins the auto crop when viewing the video on a phone.

– /u/Sid6po1nt7

The topic has even spawned a number of parody videos, including: Vertical Video Syndrome - A PSA, Turn Your Phone!, and Turn Your Phone 90 Degrees.

This “issue” is so pervasive that apps like Horizon Camera have been developed to attempt to resolve the “problem”.

Biology

Perhaps unsurprisingly, humans experience the world in “widescreen”. According to a study conducted by NASA, human vision crops to around a 16:10 aspect ratio.

This diagram shows the normal field of view of a pair of human eyes. The central white portion represents the region seen by both eyes. The gray portions, right and left, represent the regions seen by the right and left eyes, respectively. The cut-off by the brows, cheeks, and nose is shown by the black area. Head and eyes are motionless in this case.

– Source: Ruch and Fulton, eds. [25].

In addition to our physical facial features which obstruct our vision, humans have a natural blind spot in each eye due to lack of receptors (rods or cones) where the optic nerve and blood vessels leave the eye. This can be visualized with the following demonstration:

Close your right eye, focus on the black dot, and slowly move your eyes closer to the screen until the cross disappears.

Our brain compensates for these blind spots by “filling in” missing visual information with the surrounding content to create a complete picture of our environment. This is a demonstration of our brain’s natural tendency towards creating uniformity in our environment. Marte Otten, psychology researcher at the University of Amsterdam and lead author of research on the uniformity illusion states that “under some circumstances, a large part of the periphery may become a visual illusion. This effect seems to hold for many basic visual features, indicating that this ‘filling in’ is a general, and fundamental, perceptual mechanism” … “which gives rise to a rich, but illusory, experience of peripheral vision.” While this research was limited to specific tests of the periphery, it is not such a stretch to believe that framing vertical videos with blurred areas makes for a more immersive experience.

An example of the uniformity illusion applied to blurriness can be seen in the following demonstration:

See the Pen Uniformity Illusion Blur by Joey Hoer (joeyhoer).

This shows that at worst, blurred pillarboxing provides a more immersive viewing experience, and at best, blurred pillarboxing provides a more comfortable viewing experience by reducing eye stain and cognitive load.

A Solution

Vertical videos are poorly suited to wide screens, while widescreen videos are poorly suited to the “tall” screens found on mobile devices. A true responsive video solution would provide an optimal experience in both environments. We need a solution that does not add weight, that does not make videos too small or too large.

Fortunately, JavaScript and canvas allow us to crate blurred pillarboxing in realtime, using the original video source; that means no unnecessary data over the wire. Additionally, because we can “paint” pixels on the canvas however we choose, we have the ability to create a truly responsive viewing experience that is wide for devices in landscape orientation (like laptops), and tall for devices in portrait orientation.

The following demonstration uses a small amount of JavaScript to create an optimal viewing experience for vertical videos on all devices:

See the Pen Vertical Video Bars by Joey Hoer (joeyhoer).

Tip: Always constrain embedded media to the height of the viewport in CSS using vh units (e.g. max-height: 100vh), leaving “padding” between the media and the viewport using calc().

The Curious Case of Route 53 and New Relic Compatibility

2017-04-02T18:52:58-04:00

Last week I received an email from a client outlining an issue they were encountering with Amazon’s Route 53 health checks which lead to a discovery about how Amazon’s string based health checks operate and how that can create issues with the performance monitoring tool New Relic.

The Incident

I was sitting at my desk when an unsuspecting email appeared in my inbox:

Joey,

Something changed and Amazon can no longer reliably monitor the site’s health. See the first graph below. This started around 1AM last Friday. Since then you can see that the health percent from each test point is less than 60% and thus we can no longer rely on this to monitor the site. The monitoring used to be perfectly spot on, so something has changed.

I’ve attached a screenshot of the health check configuration. Why this is failing makes no sense, but I’ve tripled checked. Amazon’s test points are whitelisted on the firewall and have no problems resolving the correct load balancer IP address, but for whatever reason it cannot find the [search string] on the main page.

Also attached is a second screenshot of the status of all the most recent checks and their result for reference.

Thanks,
Jeff

The Clues

While I’m not particularly well versed in Amazon’s health check system, I could tell based on the screenshots that the health check was configured to use a search string which is an approach commonly used in other health check systems to ensure that a 200 response isn’t, in fact, a blank page or error page. I knew that I could reproduce this health check myself with the following command:

curl -sL https://www.site.com | grep $search-string

However, running this command multiple times did not reproduce the issue. The search string was found every time.

So I made a cursory search of Google to learn more about Amazon’s Route 53, and found:

[When using] health checks with string matching… The string must appear entirely in the first 5120 bytes of the response body

Ah-ha! So Route 53 has a byte limit restriction on the string matching health check. Armed with that information, I was able to amend the previous manual check, and this time I was able to reproduce the issue.

curl -sL https://www.site.com | head -c 5120 | grep $search-string

Now, I know from experience (and based on Jeff’s email) that this site has two web servers behind a load balancer. Fortunately for me, our server configuration exposes which server responds to a particular request, so I was able to see that while responses from “Web 1” were succeeding, responses from “Web 2” were failing.

Looking at the responses from both servers, it became clear that there was JavaScript at the top of each response that was different between the two servers. In the response from “Web 2” there was more JavaScript. So much JavaScript the the page title (which included the first occurrence of search string on the page) was pushed just beyond Amazon’s 5120 byte limit.

This was concerning; given our automated deployment process, having different code across the two servers should not be possible.

After a moment, I realized that the JavaScript was a requirement of a performance monitoring tool we use: New Relic. …but the question remains, if it wasn’t included in the codebase, why was it different? How did it get on the page?

The Culprit

So I fired up Google again, this time searching for more information on New Relic’s JavaScript, and found New Relic documentation reports “the PHP agent will automatically inject the Browser JavaScript into your pages.”

Great, so now we just need to check which version of New Relic’s PHP agent is installed on each server. As I suspected, the versions across the two server’s are indeed different:

Server	Agent	Version
Web 1	newrelic-php5.x86_64	6.5.0.166-1
Web 2	newrelic-php5.x86_64	7.1.0.187-1

As it turned out, our hosting provider had upgraded the New Relic Agent while investigating an issue the week prior.

As a temporary solution I suggested checking for another unique string that appear prior to the New Relic JavaScript and was unlikely to appear on a blank page or error page.

Ultimately, I learned that given Route 53’s 5120 byte limit to string matches, it may be difficult to use newer versions of New Relic’s agent along with Route 53 unless modifications are to the header of the document, like adding a unique string very early on in the document.

Case closed.

Robots Directives

2016-11-28T21:45:24-05:00

Search engines are powerful tools, yet it is surprisingly simple to provide direction to search engines and ensure that content is shown (and hidden) appropriately in search results. Likewise, it is surprisingly easy to make mistakes and instruct a search engine to show (or hide) content unintentionally. Understanding the basic tools that search engines provide content authors to guide search robots enables you to have important content appear in search results, and prevents you from making costly mistakes.

Indexing Methods

Search engines use three basic actions to discover new content to present in search results:

Crawl: page content is read by search engines
Follow: in-page links are traversed by search engines
Index: page is recorded and may appear in search results

Search engines provide content authors with tools to control behavior when new content is discovered: the robots.txt file and the robots tag.

The robots tag controls indexing and following, while the robots.txt file controls crawling (evidence of which can be seen in the Google search results: “A description for this result is not available because of this site’s robots.txt”).

There are a number of directives (described by Google’s robots.txt specification as “guidelines for a crawler or group of crawlers”) that allow content authors to prevent search engines from crawling, following and indexing content; by default, there are no restrictions for crawling; pages are treated as crawlable [and] indexable… unless permission is specifically denied.

The primary directives for restricting search engine behavior are:

disallow: Do not crawl, access (i.e. read content from) the specified paths.
noindex: Do not index this page (i.e do not show in search results).
nofollow: Do not follow (i.e. attempt to crawl) links on the page.

Crawl Directives in `robots.txt`

The robots.txt is a plain text file accessible at the top-level directory of the host (domain) encoded in UTF-8.

Note: Crawlers will not check for robots.txt files in subdirectories.

In standard UNIX fashion, comments can be included at any location in the file using the “#” character; all remaining characters on that line are treated as a comment and are ignored.

Interestingly, “a maximum file size may be enforced per crawler. Content which exceeds the maximum file size may be ignored. Google currently enforces a size limit of 500 kilobytes”.

The robots.txt file is broken up into sections (called “groups”), each defined/separated by a user-agent record.

Only one group of group-member records is valid for a particular crawler… All other groups of records are ignored by the crawler… The order of the groups within the robots.txt file is irrelevant.

– Google’s robots.txt specification: Order of precedence for user-agents

Non-human Readable Files

Ensure that search engines can crawl CSS and JavaScript files.

Disallowing crawling of JavaScript or CSS files in your site’s robots.txt directly harms how well our algorithms render and index your content and can result in suboptimal rankings.

– Updating our technical Webmaster Guidelines

If resources like JavaScript or CSS in separate files are blocked (say, with robots.txt) so that Googlebot can’t retrieve them, our indexing systems won’t be able to see your site like an average user. We recommend allowing Googlebot to retrieve JavaScript and CSS so that your content can be indexed better. This is especially important for mobile websites, where external resources like CSS and JavaScript help our algorithms understand that the pages are optimized for mobile.

– Understanding web pages better

Robots Tag

The robots tag may be defined either directly in the HTML as a meta tag, or in the HTTP X-Robots-Tag header.

<meta name="robots" content="noindex, nofollow">

X-Robots-Tag: noindex, nofollow

The quantity and order of robots directives does not matter, if multiple directives exist, the most restrictive directive will be used:

If competing directives are encountered by our crawlers we will use the most restrictive directive we find.

– Robots meta tag and X-Robots-Tag HTTP header specifications

Noindex a Section of a Site

Using the X-Robots-Tag HTTP header, it is possible to prevent search engines from indexing sections of a site, and/or particular file types:

# Prevent indexing of all pages with `/category/` in the URL
RewriteCond %{THE_REQUEST} /category/
RewriteRule ^ - [ENV=NOINDEX:true]
<IfModule mod_headers.c>
    Header set X-Robots-Tag "noindex" env=NOINDEX
</IfModule>

# Prevent indexing of image files (.png, .jpeg, .jpg, .gif)
<Files ~ "\.(png|jpe?g|gif)$">
  Header set X-Robots-Tag "noindex"
</Files>

De-indexing Catch 22

Even when the robots.txt is set to disallow, and the header of every page includes a noindex, nofollow directive, pages which have already been indexed may continue to appear in the index – “stuck” in perpetuity. This happens because the robots.txt file, when set to disallow, instructs the crawler not to crawl (i.e. read the contents of) the page, so the noindex directive is never read and the search engine does not know to remove the pages from the index. As Google Search Console’s Remove URLs tool instructs, the best way to de-index the site is to allow the search engine to crawl the site.

To block a page from appearing in Google Search results permanently… add a NOINDEX tag to the page and allow it to be crawled by Googlebot.

Solution:

Allow search engines to crawl pages by remove disallow from robots.txt
Add a noindex HTTP header or meta tag to the pages
Claim the domain (in Google Search Console) and submit the URLs for removal

NOTE: To remove an entire domain from Google Search results, enter a single forward slash (“/”) into the prompt in Google Search Console’s Remove URLs tool.

Once you’ve successfully removed the URLs from Google Search results, the SERP page should appear empty.

Canonical Links

2016-11-25T00:36:00-05:00

Canonical links are frequently misunderstood by developers, marketers, content authors and SEOs. When used correctly, rel=canonical is a powerful tool that can improve search results for visitors. However, when used incorrectly, canonical links can be detrimental to search results, and may in some cases be ignored by search engines entirely.

What is Canonicalization?

Canonical links allow authors to specify a preferred version of a web page. In other words, these links instruct search engines of which URL should appear in search results for a particular piece of content.

When authoring content on the web, it is common to present the same material at different URLs. This frequently occurs when a site is accessible via more than one protocol or subdomain and when index files are directly accessible. For example:

http://example.com
http://example.com/index.html
http://www.example.com
http://www.example.com/index.html
https://example.com
https://example.com/index.html
https://www.example.com
https://www.example.com/index.html

However, this may also occur when content is organized into different categories or collections, and when all or part of a body of content is syndicated on other sites.

This duplicate content presents a problem for search engines, which aim to improve user experience. If a search engine was to index each of the URLs above, a visitor could be presented with repetitive content in search results, leading to a poor user experience. Therefore, search engines “[try] hard to index and show pages with distinct information.”

When a search engine encounters “substantive blocks” of duplicate content, the search engine must select a single version of the content to show in search results.

Note: The concept of a “duplicate content penalty” is largely a myth. As Google notes:

There’s no such thing as a “duplicate content penalty.” At least, not in the way most people mean when they say that.

– Demystifying the “duplicate content penalty”

Duplicate content on a site is not grounds for action on that site unless it appears that the intent of the duplicate content is to be deceptive and manipulate search engine results.

…Deceptive practices like this can result in a poor user experience, when a visitor sees substantially the same content repeated within a set of search results.

– Google Search Console Help: Duplicate content

Through the use of canonical links, authors can effectively inform search engines to index content only at one URL.

Defining a Canonical Link

The rel=canonical link may be added via either an HTML tag or HTTP header.

<link rel="canonical" href="http://www.example.com/canonical.html" />

Link: <http://www.example.com/canonical.html>; rel="canonical"

Note: Any HTML link may be specified as an HTTP header according to RFC5988§5.

Canonical Indexing

The rel=canonical link is an SEO hint not a directive. While canonicalized URLs are not supposed to be indexed or show up in search results, there are some circumstances where search engines may choose not to honor canonical links.

Are Non-Canonical Pages Indexed?

For all practical purposes – no. If Google honors a rel=canonical tag, then the non-canonical page is not eligible for ranking. It will not have a unique cached copy, and it will not appear in the public index via a “site:” search.

– Rel=Confused? Answers to Your Rel=Canonical Questions

Nevertheless, Google may ignore a canonical URL if:

Content is significantly different on the canonicalized URL.
Multiple canonical links exist on a URL.
A canonical loop exists wherein a canonicalized paged links back to the original page.
The canonicalized page is redirected.
- According to Yahoo, canonical links are transitive, but this can result in unexpected behavior.
The target does not exist.
- The URL is empty, broken, or results in an error or “soft 404”.
Google believes the link is malicious.
The link does not appear in the <head>, or if the <head> contains unusual content.

Note: The above list outlines some of the known instances in which Google may elect to disregard a canonical link. Other examples may exist.

Content Discrepancies

Canonical links are designed to connect content that is substantively similar. If the canonicalized content does not closely match the original content, search engines are unlikely to respect the link.

A large portion of the duplicate page’s content should be present on the canonical version… for example, if [two pages are] only topically similar but not extremely close in exact words, the canonical designation might be disregarded by search engines.

– 5 common mistakes with rel=canonical

Note: While not recommended, it is possible to use rel=canonical on dissimilar content; doing so may cause search engines to stop trusting your site’s canonical links.

Link Quantity

The number of canonical links that exist for a specific URL is significant. When multiple canonical declarations exist for a URL, Google will “likely ignore all the rel=canonical hints”.

If multiple canonical links do exist for a URL, be careful to ensure that the links do not contradict one another by verifying that all canonical links point to the same destination.

Tip: Use the HTML or the HTTP method of declaring canonical links consistently across a site to avoid overlap and accidental conflicts.

Link Placement

The position of canonical links in the HTML document is also significant. Ensure that the rel=canonical tag appears as close to the beginning of the document as possible, and strive to maintain valid HTML – particularly in the document head.

The rel=canonical link tag should only appear in the <head> of an HTML document. Additionally, to avoid HTML parsing issues, it’s good to include the rel=canonical as early as possible in the <head>. When we encounter a rel=canonical designation in the <body>, it’s disregarded.

– 5 common mistakes with rel=canonical

Matt Cutts explains why this is the case:

If Google trusted rel=canonical in the HTML body, we’d see far more attacks where people would drop a rel=canonical on part of a web page to try to hijack it.

…so now we come to another corner case… we probably won’t trust a rel=canonical if we see weird stuff in your HEAD section… [because] we may assume that someone forgot to close the HEAD section[, and] we don’t allow rel=canonical in the BODY.

– A rel=canonical corner case

Broken and Malicious Links

An even poorer user experience than indexing duplicate content would be indexing the wrong content, no content, or malicious content. Matt Cutts outlines scenarios in which Google would ignore such links:

We take rel=canonical URLs as a strong hint, but in some cases we won’t use them:

For example, if we think you’re shooting yourself in the foot by accident (pointing a rel=canonical toward a non-existent/404 page), we’d reserve the right not to use the destination URL you specify with rel=canonical.

Another example where we might not go with your rel=canonical preference: if we think your website has been hacked and the hacker added a malicious rel=canonical.

Canonicalization vs Redirection

Because search engines can choose to ignore canonical links, redirection is preferred; as Matt Cutts explains, “regarding 301 redirects vs rel=canonical, in general I would use 301 redirects… they’re more widely supported.” He goes on to say, “the rel=canonical is more appropriate for when you can’t get to the server headers… if you can do 301 redirects… do [that]. If you don’t have the ability or option to do 301 redirects… rel=canonical makes sense.”

Google echoes this in the Search Console Help guidelines “Use canonical URLs” the use of “301 redirects to send traffic from [undesirable] URLs to your preferred URL. A server-side 301 redirect is the best way to ensure that users and search engines are directed to the correct page.”

Note: Because canonicalized pages are actually separate entities, visitor behavior is recorded separately for each unique URL in analytics tools. This may or may not be desirable.

Barring any technical reasons preventing redirection, another acceptable reason to use a canonical link over a redirect is if redirection would negatively impact user experience (e.g. categorization with breadcrumbs, pagination).

Canonicalized Pagination

A common mistake when canonicalizing pages is to canonicalize paginated content to the first page of content. Because the content of the first page is not representative of or substantively similar to the remaining paginated content, linking these pages to the first page essentially instructs search engines not to index a majority of the paginated content. This may lead search engines to ignore the canonical links.

Because “searchers commonly prefer to view a whole article or category on a single page,” Google recommends linking to a “View All” version of the content, if one exists and does not present any user experience issues. In fact, “if [Google thinks] this is what the searcher is looking for, [they] try to show the View All page in search results.”

While facets can and likely should be canonicalized, Google recommends using the Google Search Console Parameter Handling tool to “tell Google about any parameters you would like ignored.” As Google notes, “ignoring certain parameters can reduce duplicate content in Google’s index, and make your site more crawlable.”

Note: Search engines impose a crawl budget on sites (i.e. content is crawled and updated over time, not immediately). Providing search engines with valuable information about how your site is structured will allow search engines to index content more efficiently.

Combining with Robots Directives

Implementing rel=canonical in combination with robots directives like noindex is not advised, because noindex prevents pages from pass PageRank to the canonical version. As Google’s John Mueller reports:

You should not combine the noindex with a re-canonical pointing at an indexable URL (the rel=canonical says they’re equivalent, the noindex says they’re pretty much opposites)… pick one, but not both.

– John Mueller’s comment on canonical

And again:

[When using rel=canonical,] only use the rel=canonical link element… One reason for this is that we sometimes find a non-canonical URL first. If this URL has a noindex robots meta tag, we might decide not to index anything until we crawl and index the canonical URL.

– John Mueller’s answer to Canonical conflicts with NOINDEX?

The URL Structure of Faceted Navigation

2016-10-16T16:08:35-04:00

Faceted navigation is the collection of UI elements and functionality which provide the ability to filter and refine category views. There is some debate in the SEO, UX and general web development communities about the best way to present faceted navigation in the URL.

Faceted navigation, such as filtering by color or price range, can be helpful for your visitors, but it’s often not search-friendly since it creates many combinations of URLs with duplicative content.

– Faceted navigation best (and 5 of the worst) practices

The correct way to denote facets in a URL is through the use of query parameters. However, some believe that virtual subdirectories present a better alternative for SEO and UX. We’ll compare the different options for including facets in a URL, starting with simplistic examples of each method.

Query parameters: https://example.com/category?option=true
Virtual subdirectories: https://example.com/category/filter/option/true

Tip: Both query parameters and virtual subdirectories (path segments) are susceptible to duplicate content issues if not managed appropriately; canonicalizing URLs can help avoid duplicate content penalties.

To determine which URL structure is best for SEO, let’s weigh the pros and cons of each option according to the “best and worst” practices of faceted navigation as defined by Google.

Uniqueness

In an ideal state, unique content – whether an individual product/article or a category of products/articles – would have only one accessible URL.

As it pertains to URL uniqueness, neither query parameters nor virtual subdirectories are superior.

Google treats each distinct URL (including those with query parameters) as a unique URL with unique content. This is because it is technically possible for a website to serve different content at each distinct URL, and for URLs that include query parameters, it is likely that content may change based on these values.

Tip: This also applies to http and https URLs and URLs that can be visited with and without a subdomain (e.g. www); canonicalizing or redirecting these URLs will help avoid duplicate content penalties.

Multiple Selection

One challenge faced when structuring faceted URLs is correctly denoting taxonomy for multiple values and multiple options. As more facets are added, the URL structure becomes increasingly complicated. The following examples illustrate two common approaches to this problem:

Query parameters
- https://example.com/category?color=black,white
- https://example.com/category?color=black,white&size=8,10
Virtual subdirectories
- https://example.com/category/filter/color/black,white/
- https://example.com/category/filter/color/black,white/size/8,10/

These approaches raise concerns about valid URL encoding.

… [The danger of hierarchical classification as a general solution lies] in the philosophy of meaning. … Because the relationships between subjects are web-like rather than tree-like, even for people who agree on a web may pick a different tree representation.

– Cool URIs don’t change, Tim Berners-Lee

Non-Standard URL Encoding

Google advises against using “non-standard URL encoding for parameters… instead of key=value& pairs.”

Google provides two “worst practice” examples where key-value pairs are marked incorrectly with : and , rather than =, and where multiple parameters are appended with [] and ,, rather than &.

If key-value pairs are marked correctly, there are valid special characters that may be used within a URL.

The comma, for example, is an allowed path character (i.e. “pchar”) as part of the sub delimiters (i.e. “sub-delims”) defined in RFC 3986 § 2.2 for both query parameters and path segments. As RFC 3986 § 3.3 states:

the semicolon (“;”) and equals (“=”) reserved characters are often used to delimit parameters and parameter values applicable to that segment. The comma (“,”) reserved character is often used for similar purposes.

While these characters may be used in both path and query segments, it is uncommon to see these characters in path segments because delimiting options in a path segment does not impart a clear hierarchy.

The heart of the question is whether or not facets are hierarchical data.

According to the Wikipedia entry on “Faceted classification”, facets are not hierarchical:

Hierarchical classification refers to the classification of objects using one single hierarchical taxonomy. Faceted classification may actually employ hierarchy in one or more of its facets, but allows for the use of more than one taxonomy to classify objects.

As we’ve seen in our examples above, the “multiple taxonomies” presented by facets is not well suited to inclusion in the path segment of the URL. Returning to the point of using non-standard URL encoding, according to RFC 3986 § 3.3:

The path component contains data, usually organized in hierarchical form, that, along with data in the non-hierarchical query component (Section 3.4), serves to identify a resource…

RFC 3986 § 3.4 continues:

The query component contains non-hierarchical data

This makes it clear that facets should not appear in path segments, but should appear as query parameters. In fact, Google instructs:

Use parameters (when possible) with standard encoding and key=value pairs.

Using virtual subdirectories to denote facets is non-standard and Google is able to make better assumptions of page contents when facets are conveyed through query parameters. Google even provides a URL Parameters tool in Google Search Console that allows site administrators to instruct Google on how to interpret query parameters; no such tool exists for virtual subdirectories.

Immutable values

Another “worst practice” as defined by Google is “using directories or file paths rather than parameters to list values that don’t change page content.”

In a directory based faceted URL structure, the following URLs would all serve the same content:

https://example.com/category/filter/option/
https://example.com/category/filter/
https://example.com/category/

This issue is less apparent when using query parameters because there is a clear delineation between hierarchy and facets using key-value pairs. This issue may be resolved via canonicalization, but is not considered best practice as noted by Google; best practice is to use query parameters, because “URL parameters allow more flexibility for search engines to determine how to crawl efficiently.”

Regardless of which URL structure is used, facets should always be presented in a unified manner (e.g. alphabetical order), so multiple URLs are not indexed for the same content. Take the following URLs as an example:

https://example.com/category?test=0,1
https://example.com/category?test=1,0

Both of these URLs would display the same content. To reduce the total number of unique links on a site, and thus duplicate content from being indexed, only one of the above should be used consistently across a site.

Again, redirection or canonicalization can help search engines index this content correctly if it is referenced elsewhere.

User Experience

As a front-facing component of websites, URLs are an important part of the user experience. The URL acts as a reference point for the current view, and advanced users may use the URL as a “virtual breadcrumb trail” to navigate backwards through your site’s hierarchy. Maintaining a human parsible URL is an important and non-trivial endeavor.

Note: I’ve written an extensive post on the importance of designing URLs as part of a site’s UI.

Logical Additions

Google advises against “appending URL parameters without logic”. Unnecessary parameters should be stripped to maintain a human parsible URL structure, when possible. Google recommends removing user session information from the URL and storing that data in cookies instead. Keeping the URL free of unnecessary data not only helps users understand the content present in the current view, it also aides SEO, as Google notes:

Extraneous URL parameters only increase duplication, causing less efficient crawling and indexing.

Good UX Is Good SEO

2016-10-05T22:47:50-04:00

Google has a long standing tradition of encouraging a better web by updating its algorithms to benefit sites that offer a positive user experience (UX) and penalize those that don’t. As Google Webmaster Matt Cutts explains, “we try to help people make the web a better experience, so people will be on the web longer, and people will be happier … for example, we never show popup ads on Google. Even though it might have meant a little bit more money up front, because we also knew it would also annoy users and make them less likely to come back.”

Here are a few examples of Google algorithm updates that have focused on making the web a better place:

Site Speed

In August 2010 Google introduced the site speed ranking signal and “encouraged [site owners] to start looking at site speed — not only to improve ranking in search engines, but also to improve everyone’s experience on the Internet.”

Faster sites create happy users and we’ve seen in our internal studies that when a site responds slowly, visitors spend less time there. But faster sites don’t just improve user experience; recent data shows that improving site speed also reduces operating costs.

– Using site speed in web search ranking

In 2018, Google doubled down on site speed with “Speed Update”, making page speed a ranking factor for mobile searches—which are now the primary source of site rank.

People want to be able to find answers to their questions as fast as possible — studies show that people really care about the speed of a page. … We encourage developers to think broadly about how performance affects a user’s experience of their page and to consider a variety of user experience metrics.

PageSpeed Insights

In May 2014 Google’s PageSpeed Insights was updated to offer suggestions for improving user experience.

Poor usability can diminish the benefits of a fast page load.

– Making your site more mobile-friendly with PageSpeed Insights

Modern Technology

In July 2014 Google began displaying search result indicators for pages that used unsupported technology (i.e. Flash).

A common annoyance for web users is when websites require browser technologies that are not supported by their device.

– Promoting modern websites for modern devices in Google search results

HTTPS

In August 2014 Google began using HTTPS as a ranking signal, and in December 2015 Google began crawling and indexing HTTPS pages by default, even for HTTPS URLs that were not linked elsewhere.

At Google, user security has always been a top priority. Over the years, we’ve worked hard to promote a more secure web and to provide a better browsing experience for users. …we started giving a slight ranking boost to HTTPS URLs in search results last year. Browsing the web should be a private experience between the user and the website, and must not be subject to eavesdropping, man-in-the-middle attacks, or data modification. This is why we’ve been strongly promoting HTTPS everywhere.

– Indexing HTTPS pages by default

We’d like to encourage all website owners to switch from HTTP to HTTPS to keep everyone safe on the web.

– HTTPS as a raking signal

Mobile Friendly

In Febuary 2015 Google announced the mobile-friendly update and in April 2015 the new signal was introduced, boosting the ranking of pages using responsive web design.

The desktop version of a site might be difficult to view and use on a mobile device. The version that’s not mobile-friendly requires the user to pinch or zoom in order to read the content. Users find this a frustrating experience and are likely to abandon the site. Alternatively, the mobile-friendly version is readable and immediately usable.

– Mobile Friendly Websites

Mobile-first Indexing

In November 2016 Google announced that a new “mobile index” would become the primary source of site rank, further benefiting sites with a well optimized mobile experience.

Today, most people are searching on Google using a mobile device. … To make our results more useful, we’ve begun experiments to make our index mobile-first. … Of course, while our index will be built from mobile documents, we’re going to continue to build a great search experience for all users, whether they come from mobile or desktop devices.

Doorway Pages

In March 2015 Google announced a ranking adjustment to better address doorway pages – sites or pages created to rank highly for specific search queries – as part of a continual effort by Google’s Search Quality team to “minimize the impact of webspam on users.”

[I]f a user clicks on one result, doesn’t like it, and then tries the next result in the search results page and is taken to that same site that they didn’t like, that’s a really frustrating experience.

– An update on doorway pages

App Pop Overs

In September 2015 Google announced a change to their mobile-friendly algorithms to exclude web pages that show an app install interstitial that hides a significant amount of content.

Sometimes a user may tap on a search result on a mobile device and see an app install interstitial that hides a significant amount of content and prompts the user to install an app. Our analysis shows that it is not a good search experience and can be frustrating for users because they are expecting to see the content of the web page.

– Mobile-friendly web pages using app banners

[A] full page interstitial can interrupt the user from reaching their desired content. …Our analysis found that 69% of the visits abandoned [on our interstitial] page. …Based on these results, we decided to permanently retire the interstitial …and we are sharing this with the hope that you will reconsider the use of promotional interstitials. Let’s remove friction and make the mobile web more useful and usable!

– Google+: A case study on App Download Interstitials

Interstitials

In August 2016 Google announced a search penalty for pages where content is not easily accessible to a user due to “intrusive interstitials”.

[C]ontent may be visually obscured by an interstitial. This can frustrate users because they are unable to easily access the content that they were expecting…

Pages that show intrusive interstitials provide a poorer experience to users than other pages where content is immediately accessible.

– Helping users easily access content on mobile

The Appropriate Use of Modals

2016-08-24T02:48:23-04:00

Modal windows are designed to enhance user experience by presenting new content situationally. By definition, these interstitials obstruct user flow by preventing interaction with the main application. This UI element generally requires a user interaction to initiate its display; however, a page load, scroll event, or an arbitrary timeout is not a justifiable “interaction” for the presentation of a modal window. Presenting a modal window without an appropriate prequalifying event not only disrupts user flow and concentration, it also introduces frustration and confusion.

Those who forget history…

In the early day’s of the web, pop-up advertisements plagued web surfers. The problem became so prolific that browser vendors were forced (by user demand) to implement pop-up blockers to protect users from intrusive advertising aimed at cheap conversion.

Firefox 0.1 (originally named Phoenix), released September 23, 2002, introduced the first pop-up blocker. Shortly afterwards, on November 13, 2002, Opera 7.0 Beta 1 followed suit. By August 2004, all major browsers included built-in pop-up blockers. These built-in pop-up blockers still exist today.

Matt Korostoff recounts the overuse of pop-up windows for advertisements during the Internet boom:

There was a time when any web page could pop open as many windows as it wanted without your consent… popup blockers became so ubiquitous and effective that now advertisers hardly bother with them anymore.

Let us not forget, old-style window.open() popups weren’t originally invented to display ads. They’ve been largely eliminated, not because they stopped being a useful, but simply because advertisers used them with such abandon that the rest of the web was becoming unusable. We broke that feature of the web because we couldn’t be trusted to use it responsibly.

– Popup ads are back

The prosaic pop-up window has been replaced by the modern modal, reintroducing the frustration and confusion of the Y2K era to today’s users. In many ways, the modal is more contemptible than the pop-up. The pop-up, while relentless, does not directly prevent a user from interacting with an application; modals, on the other hand, actively block users from accessing information and content. The modal also typically demands a higher level of engagement than a simple click, despite not having established prerequisite trust, and – if not implemented correctly – presents a number of accessibility concerns.

Modals are popups

Modals (i.e. pop-overs, pop-ups, or interstitials) have become increasingly used to coerce unsuspecting users into conversions. This erodes user trust and leads to a poor overall experience (often initiated during the user’s introduction to a site).

Using the word “modal” is actually an inaccurate description of the common form most prevalent on the web today. A more appropriate term is “interstitial”, which describes a barrier placed before the expected content, and sits – literally – “in between” the user and the desired content.

Content Barriers

Undoubtedly, the most important aspect of a site – to a visitor – is the content the site contains. Interstitials block this content by forcing the user to interact with an unexpected interruption. This diverts the visitors attention away from their goal, and ultimately, away from any conversion opportunities on the site.

This may seem like a minor inconvenience, but as Jakob Nielsen reports, “annoyances matter, because they compound.” This is particularly true if a user’s first experience with a site is negative.

Even if no single annoyance stops users in their tracks or makes them leave the site, the combined negative impact of the annoyances will make users feel less satisfied. … [E]liminating annoyances increases customer satisfaction and user loyalty, and thus improves the long-term business value of the site.

– Does User Annoyance Matter?

This is particularly relevant for ecommerce sites where the main goal is to convert users as quickly as possible, as Telegraph senior designer Andy Booth explains.

If [I] could only focus on one thing to improve conversions… [it would be] the [ecommerce] landing page. The average user will make up their mind if they trust your website within three seconds at best – so you better hope your page is fully loaded and shows off what your trying to sell. Don’t hide it behind unnecessary distractions or email signups… If you are an unknown site, you will be judged on how trustworthy you are.

– Andy Booth Net Magazine, Issue 278, April 2016 Page 12

Obstructing content not only presents usability concerns by frustrating users and needlessly delaying conversion, it also erodes user trust.

Trust and Reciprocity

When aspiring to “true conversion”, trust is extremely important. Trust may be the most valuable commodity of the modern web. You can’t ask someone for their trust, you must earn it, and once you have earned a user’s trust, you gain privileges others don’t.

The Nielsen Norman Group has performed in-depth analysis of trust and user commitment:

The more information or effort a site asks for, the more trust and comfort the user must have. In our rush to collect and convert… users get put off and abandon the site because [we haven’t] covered the basic levels of commitment.

– Hierarchy of Trust: The 5 Experiential Levels of Commitment

Requesting a user’s information before offering the user any substantial value contributes to an overall negative impression of a brand. If you do collect a user’s information, think of creative ways to offer the user something in return – perhaps by auto-populating other forms with the information you’ve collected.

If you want your users to trust you with their information and come back to you repeatedly, plant the reciprocity seed by being nice to them upfront and minimizing their interaction cost. Ask as little of your users as possible. On the web and elsewhere, start by giving before taking, and people will reciprocate.

The Reciprocity Principle: Give Before You Take in Web Design

Asking a user for data is one thing, tricking a user into providing data is another. Interstitials, depending upon the implementation, walk a line of deception by presenting accessibility concerns.

Accessibility Concerns

Interstitials are difficult to execute properly without affecting accessibility in some way. Users with visual or cognitive impairments may struggle with interstitials, and may not realize or understand how to dismiss an interstitial. This experience can force users into divulging more information than they feel comfortable with, and may cause users to abandon a site.

Dark patterns are insincere and deceptive user interface elements designed to coerce users into performing an action unintentionally. Interstitials can present many dark patterns including “forced disclosure”, “privacy zuckering” and “road blocking”¹.

Prioritizing conversions or short-term metrics leads designers to pressure people into doing things they don’t actually want to do and can easily cross the ethical boundaries towards dark patterns. It’s time to reassess priorities and long-term goals: you may be getting a few extra clicks now, but in the long run you’re losing your users’ trust and respect.

– Needy Design Patterns: Please-Don’t-Go Popups & Get-Back-to-Me Tabs

User Feedback

Users find modals so aggravating, in fact, that entire websites have been devoted to users who eschew modals:

Users are actively declaring that they abandon sites which present these distractions, but as the old user experience axiom goes “watch what people do, not what they say.”

SEO Impact

Google arguably has access to the most data on how technology and user experience affect user engagement; while no one but Google knows exactly how many websites use Google Analytics, sources estimate that Google Analytics is installed on 55–75% of all websites. Google has a vested interest in making the web more usable and less confusing for users, because the faster the web is and the more satisfied Google users are with the results, the more pages they will view and the more ads Google will get to display.

However Google does not allow their marketing efforts to obstruct the user’s goals, as Google Webmaster Matt Cutts explains, “we try to help people make the web a better experience, so people will be on the web longer, and people will be happier … for example, we never show popup ads on Google. Even though it might have meant a little bit more money up front, because we also knew it would also annoy users and make them less likely to come back.”

I’ve written about good UX being good SEO, and Google seems to agree with me.

Google has instituted changes to block technology similar to interstitials in the past, banning doorway pages in March 2015, and blocking “app interstitials” in September 2015 because they were “frustrating users”, and in August 2016 Google announced a search penalty for pages where content is not easily accessible to a user due to “intrusive interstitials”:

[C]ontent may be visually obscured by an interstitial. This can frustrate users because they are unable to easily access the content that they were expecting…

Pages that show intrusive interstitials provide a poorer experience to users than other pages where content is immediately accessible.

Here are some examples of techniques that make content less accessible to a user:

Showing a popup that covers the main content, either immediately after the user navigates to a page from the search results, or while they are looking through the page.

Displaying a standalone interstitial that the user has to dismiss before accessing the main content.

Using a layout where the above-the-fold portion of the page appears similar to a standalone interstitial, but the original content has been inlined underneath the fold.

– Helping users easily access content on mobile

The only true way to assess the impact of an interstitial (and results will vary) is to measure user interaction and conversion for users that have been presented with an interstitial.

An Empirical Study

If you plan on implementing an interstitial, I suggest using the following outline for an empirical study on the effectiveness of the modal:

Document how the modal functions, and include the following information:
- How and when the modal is presented to a user
- Methods in which focus is drawn to the modal
- Methods with which the modal may be dismissed (ESC key, clicking the background, an obvious escape/close button, etc.)
- Clarity of exit actions
- How and when the modal may be reintroduced
Gather metrics on the modal usage and performance
- Discover which “exit action” is used most frequently
- A/B test the modal for abandonment, sign-up rates
- Test all modal “conversions” for engagement and future conversions
Perform user testing and collect real user comments

Final Thoughts

Often these types of overlays are deliberate attempts to entice the user to take some action that benefits the organization (such as subscribing to a newsletter). Many web marketers seem to operate under the ‘ends justify the means’ principle, and believe that the benefit of increasing your subscriber base outweighs the negative consequences of annoying your users. If you attempt this strategy, make sure to follow up and measure whether those extra subscribers are actually qualified leads. Do they really end up buying your products? If not, you may be aggravating your true audience for no reason at all.

… Don’t use an overlay unless you have a clear, compelling case for why this content should not be presented within a regular page.

– Overuse of Overlays: How to Avoid Misusing Lightboxes

Every time you make a change, ask yourself, who will this change benefit? Does this change benefit you, as a business owner, or your users? If the answer doesn’t include the user, you should reconsider.

Forced Disclosure

In return for a free or low-cost action, the site requires the user to disclose extensive personal information – unnecessary to the transaction in-hand.

Privacy Zuckering

“The act of creating deliberately confusing jargon and user-interfaces which trick your users into sharing more info about themselves than they really want to.” (As defined by the EFF).

Road Blocking

When the user’s progress to task completion is restricted or stopped by something else on the screen.

↩

Designing & Optimizing Favicons

2016-08-13T14:08:06-04:00

Favicons, or favorites icons, provide users a visual way to differentiate between a browser’s tabs, bookmarks, favorites, and history items. Depending on browsing habits, a user may have more exposure to a website’s favicon than its logo. To craft a favicon that improves user experience and entices users to return to a site, authors should ensure that the favicon is unique, crisp and well optimized.

Browser Support

Modern browsers support a variety of file formats to be used as favicons. The most commonly used file types are PNG, GIF, and ICO, though JPG, APNG, and SVG may also be supported depending upon the browser. Each file type has it’s own distinct advantages and disadvantages.

	Transparency	Animation	Multiple Image	Compression
JPEG	✗	✗	✗	✓
GIF	✓	✓	✗	✓
PNG	✓	✗	✗	✓
ICO	✓	✗	✓	✗

The ICO format is ideal for modern favicons because it allows authors to embed multiple images in a single file, allowing different icon sizes to be used in different contexts, and supports full alpha transparency. Additionally, authors are not required to implement any code on a page to use an ICO formatted favicon.

While JPG favicons are supported in most browsers, this format should not be used because it does not support transparency, and its lossy compression is less than ideal for small icons.

Developers may choose to use the GIF format to encode favicons because the format supports animation, indexed transparency, and generally offers the smallest file size (at 16px). However, because index transparency is binary (either the pixel is fully transparent or fully opaque), the format often produces undesirable visual artifacts, unless the favicon can map directly to a pixel grid and does not require the use of semi-transparent pixels. Likewise, animation is rarely necessary or desirable in a favicon.

The PNG format overcomes the limitations of GIF by offering full alpha transparency and a superior compression algorithm that can produce much better visual results.

Until SVG favicons are widely supported, authors should use the ICO format and embed (at least) 16px and 32px sizes.

History of the `ICO` format

The ICO format, developed by Microsoft, acts as a wrapper for bitmapped images. In other words, ICO files are not intrinsically compressed. In 2007, with the release of Windows Vista, the ICO format was updated to support embedding of compressed PNG contents (though I’ve only ever had success embedding compressed content at 256px). As such, authors should ensure that ICO files are compressed and served with a compressed Content-Encoding.

Some limitations are built into the ICO format:

ICO files must be square.
ICO files must not exceed 256px.

Something for Nothing

As with any asset being served, a web site’s favicon must be requested by a browser before a response (the file) is returned; this almost always requires markup to be added to the page. A favicon may be requested by adding a link to the head of the page:

<link rel="icon" type="image/x-icon" href="/favicon.ico" />

It is possible, however, to serve a favicon to a browser without any markup at all. Browsers will automatically make a request to favicon.ico at the root domain (e.g. joeyhoer.com/favicon.ico); if an ICO encoded file is served at that location, it will be used by default. This behavior may be overridden by defining a link in the document head.

Creating Favicons

To convert images to ICO format, authors may use ImageMagick.

Because SVG favicons are not yet supported, it is recommended that individual icons are drawn specifically for each output size. This offers designers fine control over how the icon is displayed. The following command will generate an ICO file from two (or more) input PNG files.

convert 'source-16.png' 'source-32.png' 'favicon.ico'

Nevertheless, separate files may not always be available. Fortunately, ImageMagic supports many different image manipulation operations that can be used to perfect resizing. At a basic level, the following command may be used to automatically generate a favicon:

# Trim and resize image
convert 'source.png' -trim -resize 256x256 \
  -gravity center -background transparent -extent 256x256 \
  -define icon:auto-resize=16,32 -compress zip 'favicon.ico'

Alternatively, the command below offers additional control over how each size is generated. As is, the command is essentially equal to the auto-resize command above.

convert 'source.png' -trim -resize 256x256 \
  -gravity center -background transparent -extent 256x256 \
  \( -clone 0 -resize 16x16 \) \
  \( -clone 0 -resize 32x32 \) \
  -delete 0 -compress zip 'favicon.ico'

Note: The -compress option in the code snippets above is set to zip. As ImageMagick’s documentation on the -compress option states:

When writing an ICO file, you may request that the images be encoded in PNG format, by specifying Zip compression.

However, zip compression only appears to work when converting images to ICO files of 256px.

Extracting images from Favicons

# This will print the size, file type, calculated image type, calculated boolean for full opacity
identify -format "%wx%h %m %[type] %[opaque]\n"

Optimization

Quantization

Quantization, or reducing the overall colors in an image, allows favicons to be optimized. The following function will determine the total number of unique colors in an image, and quantize to that number:

quantize(){
  mogrify -colors `identify -format '%k' "$1"` +remap "$1"
}
quantize 'favicon.ico'

Icon Tools

Listed below are a number of tools that can be useful when producing icons for the web.

icoutils: a set of command-line programs for extracting and converting images in Microsoft Windows® icon and cursor files. The icotool program converts icon and cursor files into a set of PNG images.
iconutil: a utility to convert between .iconset and .icns files.
libicns: a library for manipulation of the Mac OS icns resource format
png2ico: converts a PNG to an ICO
ICOFormat: a Photoshop plugin that allows ICOs to be exported directly from Photoshop
ICOBundle: a Photoshop plugin that allows ICOs to be exported directly from Photoshop

Clearing favicon cache

When developing and testing favicons, it is useful to understand how to clear the favicon cache.

Clear Safari favicon cache

# Remove all favicons from Safari
sqlite3 ~/Library/Safari/WebpageIcons.db \
  "delete from IconData; delete from PageURL; delete from IconInfo;";

# Delete the favicon of a single domain from Safari
# delete from IconInfo where
sqlite3 ~/Library/Safari/WebpageIcons.db \
  "update IconInfo set stamp = 0 where url like '%url.com%';";

Clear Chrome favicon cache

# Remove all favicons from Chrome
sqlite3 ~/Library/Application\ Support/Google/Chrome/Default/Favicons \
  "delete from favicons;"

# Delete the favicon of a single domain from Chrome
sqlite3 ~/Library/Application\ Support/Google/Chrome/Default/Favicons \
  "delete * from favicons where url like '%url.com%';"

Ambient Video

2016-08-06T09:15:47-04:00

Ambient videos create engaging, “interactive” layouts¹ by setting a video as the background of a page or a section of a page. While this design technique can evoke a powerful emotional response, special care must be taken to ensure that implementation does not negatively impact user experience.

As a rule, ambient videos (also referred to as “environmental videos” or background videos) should not include audio. Generally, these videos do not require user interaction to begin playback and loop seamlessly. In many ways, ambient videos resemble animated GIFs; cinemagraphs are particularly well suited for ambient videos.

Challenges of ambient video include accessibility, playback compatibility, and performance.

Demo

Ambient Video

<div class="ambient-video">
  <div class="overlay">
    Overlay Content
  </div>
  <video autoplay loop muted playsinline poster="ambient-video.jpg">
    <source src="ambient-video.mp4" type="video/mp4">
  </video>
</div>

// Hide play button from inline autoplay videos on iOS
video[autoplay][playsinline] {
  &::-webkit-media-controls-play-button,
  &::-webkit-media-controls-start-playback-button {
    opacity: 0;
    pointer-events: none;
  }
}

Accessibility

Ambient videos frequently appear behind text, buttons and other elements. Ensure that all text and calls to action are clearly readable and distinguishable from the background content.

One way to improve contrast between foreground and background elements is to darken the background by applying a filter to the video. This can be done on the video asset directly (which may improve compression levels, making the file smaller and improving performance) or with CSS. It is also possible to add stokes, borders, and shadows to foreground elements to visually separate them from the background.

Increasing the size of foreground elements can also improve accessibility. This may particularly be useful if the background includes rapid motion.

Playback Compatibility

Perhaps the most difficult challenge faced when implementing ambient video is mobile compatibility. Some mobile browsers disable preloading², automatic video playback³ and inline playback⁴. However, new changes in iOS 10 have returned “more control over media playback to web developers” for silent <video> elements – ideal for ambient videos.

Until the features mentioned above are more widely supported, developers must circumvent the restrictions placed on mobile devices. Using JavaScript, developers may either seek a video to simulate inline playback, or load each frame onto a canvas.

Another option is to convert the video to an optimized GIF. However, this method results in extremely poor quality “videos” that are very large in file size. For this reason, I strongly recommend against using the GIF format for ambient videos.

Update (2017/12/06): Safari Tech Preview 43 resolved an issue with “movie-backed” <img> tags, allowing short, looped, muted MP4 files may now be loaded via <img> tags.

Performance

Video assets, particularly assets of sufficient quality to cover the viewport of large desktop browsers, are quite large in terms of file size.

To ensure that site visitors are not waiting for a large asset to download, generate a “poster image” for the video, which will display while the video is loading. For a seamless experience, it is best to use the first frame of the video for the poster image.

It is possible to easily generate a poster image from the first frame of a video using ffmpeg. In its simplest form:

ffmpeg -i "$infile" -vframes 1 -ss 0 -f image2 "$outfile"

I’ve written a script that allows your to quickly create poster images with ffmpeg, and will allow you to optionally scale and blur the poster image for additional performance improvements.

This can be used in combination with gifv – another wrapper I’ve written around ffmpeg, designed to generate high quality, low file size, ambient videos – built for the web.

I say “interactive” because ambient videos present the perception of interactivity without actually providing any interactive functionality. ↩
In Safari on iOS (for all devices, including iPad), where the user may be on a cellular network and be charged per data unit, preload and autoplay are disabled. No data is loaded until the user initiates it.

– Safari Device-Specific Considerations Documentation

↩
To prevent unsolicited downloads over cellular networks at the user’s expense, embedded media cannot be played automatically in Safari on iOS—the user always initiates playback

– Safari Audio and Video Tag Basics Documentation

↩
Safari optimizes video presentation for the smaller screen on iPhone or iPod touch by playing video using the full screen—video controls appear when the screen is touched, and the video is scaled to fit the screen in portrait or landscape mode. Video is not presented within the webpage.

– Safari Device-Specific Considerations Documentation

↩

Screen Recording

2016-04-24T11:59:06-04:00

Creating screen recordings on macOS doesn’t have to be difficult, time consuming or expensive. You may be surprised to learn that pre-installed applications and tools in macOS can be used to quickly create high quality screen recordings and animated screenshots of your Mac, iPhone and iPad.

QuickTime Player, which ships with macOS out of the box, includes built-in screen recording functionality. If you’re looking to create and share a short screen capture that does not require advanced video editing, QuickTime is an ideal solution.

To create a new screen recording, open QuickTime Player, and select File > New Screen Recording or use the ⌘ ⌥ N keyboard shortcut.

If you frequently need to create screen recordings, the following shell function may be used to quickly initiate a new recording:

##
# Activate new QuickTime screen recording
# Note: Assistive access must be enabled for "automated start" to work
##
rec() {
  # Enable Do Not Disturb
  dnd on

  # Activate new screen recording
  osascript <<'ASCRIPT' >/dev/null
    tell application "QuickTime Player"
      activate new screen recording
      -- Hit Spacebar
      tell application "System Events" to key code 49
      -- Alternative
      -- to tell window "Screen Recording" to click button 1
    end tell
ASCRIPT

  # Click-drag to match bounds of application window
  process="$1"
  if [ -n "$process" ]; then
    read x y dx dy <<<$(osascript <<ASCRIPT
      -- Bring target application to front
      tell application "${process}" to activate
      -- Bring QuickTime to front
      tell application "QuickTime Player" to activate
      -- Get window bounds
      set AppleScript's text item delimiters to " "
      tell application "System Events" to tell process "${process}"
        set window_bounds to {position, size} of first window
        get window_bounds as text
      end tell
ASCRIPT
    )

    clickdrag -x "$x" -y "$y" -dx "$dx" -dy "$dy"
  fi
}

This script uses two options scripts dnd (which controls macOS’s Do Not Disturb mode) and clickdrag (which controls the mouse, and allows the script to automatically record a single application window). You may install these scripts with the following command:

brew tap joeyhoer/homebrew-extras
brew install dnd --HEAD
brew install clickdrag --HEAD

Tip: An active screen recording may be stopped with the ⌘ ⌃ ⎋ keyboard shortcut.

The QuickTime screen recorder has options that can be set both through the GUI and on the command line.

# Set recording quality
# High:    MGCompressionPresetHighQuality
# Maximum: MGCompressionPresetMaximumQuality
defaults write com.apple.QuickTimePlayerX MGRecordingCompressionPresetIdentifier -string 'MGCompressionPresetMaximumQuality'

# Show mouse clicks in screen recordings
defaults write com.apple.QuickTimePlayerX MGScreenRecordingDocumentShowMouseClicksUserDefaultsKey -bool true

QuickTime provides some very basic, easy-to-use video editing functions like trimming and clipping. Unfortunately, more advanced functions like changing the video dimensions and speed are not available, and QuickTime can only export videos in predefined formats (which are not optimized for sharing).

Optimizing with Automator

It is possible, however, to automatically optimize screen recordings for sharing by using an Automator workflow. Automator, another application that ships with macOS by default, is a very powerful tool that allows you to easily automate actions on your machine.

To run a script automatically whenever a new file is added to a particular folder, use Automator to create a new Folder Action. Once you’ve opened a new Folder Action, select a folder where you will save your video files to (I use ~/Movies/). In the Library panel locate the Filter Finder Items action (found within the Files & Folders actions), and drag-and-drop the action into the workflow. You now have the ability to filter out files that you do not wish to optimize. I filter by files created today, of type movie, with a mov file extension. Next, locate the Run Shell Script action (found within the Utilities actions), ensure input is passed as arguments, and in the body of the action, call the automated script you wish to run on the files.

The following script uses gifv to create an optimized video file that is half the resolution and 2.5× faster than the original video. This can significantly reduce file size, and make the video file easier to share.

source $HOME/.exports
export PATH=$PATH:/usr/local/bin:/usr/local/var/rbenv/shims

# Optimize videos at double speed and half resolution
# Minimum resolution
min_res=480
gifv -p "trunc(max(min($min_res\,iw)\,iw/2)/2)*2:-2" -s 2.5 "$1"

`gifv`

gifv is a command-line wrapper around ffmpeg that converts video files and GIFs into videos optimized for GIF-like playback on the web.

The gifv tool was based on two “competing formats”, imgur’s GIFV and gfycat’s GFY (both pronounced “jiffy”). Essentially, these “formats” are wrappers around compressed video files (h.264 encoded MP4s or WebMs).

Whenever possible, video files should be preferred to GIFs, as video codecs offer superior compression, which results in smaller filesizes, better quality, and includes built-in playback controls.

Installation

Install gifv, add the script above to your ~/bin/ directory and install the Automator Folder Action workflow below by downloading it and moving it to ~/Library/Workflows/Applications/Folder Actions. Once you have the Folder Action in place, you must enable Folder Actions and assign the action to a folder. To do this, open the Folder Actions Setup app, check the box next to “Enable Folder Actions”, add a folder you wish to apply the action to (I chose the user’s Movies folder), and add the script as seen in the screenshot below.

All videos created by QuickTime, saved to ~/Movies/ will be automatically optimized.

Optimize Screen Recording

Other Tools

`gifify`

Some services do not allow video files to be used, but do accept animated GIFs. In these cases, you can use gifify to convert the video file to an animated GIF.

The version of gifify linked above if a fork of Jonathan Clem’s gifify which provides some additional options such as reversed and parametric (or patrol) playback, as well as a number of different options for quality and compression/generation speed.

Another option is to use the node package of the same name, which offers even more options.

Fluid Media for Responsive Designs

2016-02-26T22:59:59-05:00

Responsive images and other fluid media (like videos) are fundamental elements of modern web development. Despite the fact that responsive design has become ubiquitous in the web industry, a solution for high quality media that loads quickly and looks great on all viewports remains elusive.

There are two factors which determine how media is presented in the browser:

Applied Styles: the styling which is applied to a media element which defines it’s appearance (dimensions) at different viewport sizes.
Intrinsic Dimensions: the inherent dimensions encoded within the metadata of a media element.

Fluid Layout

See the Pen BWEjjQ by Joey Hoer (joeyhoer).

Layout Attributes

When thinking about how a media element’s dimensions affects page layout, it is important to consider how dimensions are calculated by the browser. Each image has an intrinsic width and height which is defined in the image metadata or described in the encoded image data and a layout width and height which is defined by the size at which the image is actually rendered on the page.

Before the browser can layout the appropriate amount of space on the page based on the intrinsic dimensions, it must download the image (or a sufficient amount of metadata from the image). This can lead to content “jumping” when a reflow occurs as the image dimensions are discovered by the browser during download. Fortunately, the HTML specification provides width and height attributes on image elements to allow authors to define the desired size of the image. I will refer to these attributes as “layout attributes”.

The height and width attributes give user agents an idea of the size of an image or object so that they may reserve space for it and continue rendering the document while waiting for the image data.

– HTML 4.01 specification

Old Standards

It was once best practice to define both the width and height attributes of image elements to ensure that a page rendered quickly and allowed images to load asynchronously without triggering layout reflows and repaints. As older versions of Google’s Page Speed recommendations explained:

When the browser lays out the page, it needs to be able to flow around replaceable elements such as images. It can begin to render a page even before images are downloaded, provided that it knows the dimensions to wrap non-replaceable elements around. If no dimensions are specified in the containing document, or if the dimensions specified don’t match those of the actual images, the browser will require a reflow and repaint once the images are downloaded. To prevent reflows, specify the width and height of all images, either in the HTML tag, or in CSS.

– Google Page Speed

Note: The Google Page Speed article linked above is from a cached version from March 2012. I was unable to find any equivalent recommendation in more recent versions of Google Page Speed.

Matching the layout attributes to the intrinsic dimensions of an image will ensure that the appropriate amount of space is reserved for the image. It is also possible to proportionally adjust these layout attributes (based on the intrinsic dimensions) without affecting the aspect ratio of the rendered image (which may be useful for displaying images on displays with high pixel density). For example, a 400px × 200px image could be scaled proportionally to a size of 200px × 100px on devices with 2x pixel density.

\frac{400px}{200px} = \frac{200px}{100px} = \frac{2}{1}

Images also have an intrinsic aspect ratio which is based on the intrinsic dimensions. When only one dimension is defined, and when the width and height properties are set to auto (the default value), the browser uses the image’s intrinsic aspect ratio to scale images proportionally to different sizes. In other words, if only one layout attribute is defined (e.g. width), the image will be scaled proportionally to match that dimension. This characteristic of images allows CSS to be used to smoothly scale images between different sizes in responsive design.

Fluid Width Images

To achieve fluid media elements, an image’s layout attributes must be overwritten with CSS. By setting the max-width of and image to 100%, developers can ensure that images will never be larger than their parent container. Defining a maximum width, and allowing the height to be calculated using the intrinsic aspect ratio, discussed earlier, effectively creates a fluid image solution.

However, an issue arises when max-width is set on an image that also has the height attribute defined. Because the image does not correctly maintain it’s aspect ratio as the parent container becomes smaller than the image width; instead the image becomes “smushed”. To work around this issue, the height property must always be set to auto—this can be accomplished with CSS.

img {
    max-width: 100%;
    height: auto;
}

See the Pen Visualizing Fluid Media by Joey Hoer (joeyhoer).

Note: Some developers may be confused here, knowing that inline styles have a higher specificity than other CSS rules. While this is true, “presentational attributes” (e.g. width and height attributes) are not the same as “inline styles” (i.e. the style attribute). The CSS specification outlines how browsers may implement specificity for “stylistic attributes”:

The UA may choose to honor presentational attributes in an HTML source document. If so, these attributes are translated to the corresponding CSS rules with specificity equal to 0, and are treated as if they were inserted at the start of the author style sheet. They may therefore be overridden by subsequent style sheet rules. In a transition phase, this policy will make it easier for stylistic attributes to coexist with style sheets.

– CSS 2.1 specification §6.4.4

Other Elements

The same styling may also be applied to other media elements which have an intrinsic height, allowing them to maintain their aspect ratio as they scale (when height is set to auto). These elements include:

audio
canvas
img
svg
video

Other elements may also benefit from having a max-width of 100% applied, but do not have an intrinsic height. These elements include:

select
textarea
input
iframe
embed
object

Reflows on Fluid Images

A layout issue occurs before the image has loaded; if height is set to auto and the intrinsic dimensions have not yet been determined, the calculated image height will be 0. Because the most common fluid image technique (seen above) relies on the height being set to auto, this approach causes reflows and repaints as images are downloaded.

An img element is initially unavailable.

When an img element is [partially or completely] available, it provides a paint source whose width is the image’s intrinsic width, whose height is the image’s intrinsic height, and whose appearance is the intrinsic appearance of the image.

[The] attributes width and height must return the rendered width and height of the image … if the image is being rendered … to a visual medium; or else the intrinsic width and height of the image … if the image is available but not being rendered to a visual medium; or else 0, if the image is not available.

– HTML5 Embedded Content Specification §4.7.5. The img element

Placeholder Solutions

One solution to prevent reflows when using fluid images is to utilize a placeholder element that reserves space for the image. This technique is common for lazy-loaded images, but provides benefits to all users, as devices with poor connections may require some time to download images, and content reflows always present a jarring user experience.

For a placeholder solutions to be effective, the author must know the desired aspect ratio of the image in advance.

Canvas Placeholder

For example, a blank canvas element could be used:

<div class="placeholder">
  <canvas width="{{ width }}" height="{{ height }}"></canvas>
  <img src="…" alt="…"/>
</div>

.placeholder {
  display: inline-block;
  position: relative;

  canvas {
    display: block;
    max-width: 100%;
  }

  img {
    position: absolute;
    top: 0;
    left: 0;
    width: 100%;
    height: 100%;
  }
}

In this example, a canvas element is defined with width and height attributes equal to that of the image which will be “replacing” it. The canvas will be “painted over” when the image loads due to absolute positioning of the image. This technique works because the canvas element scales proportionately when resized, even when constrained by a max-width. This is because:

The intrinsic dimensions of the canvas element when it represents embedded content are equal to the dimensions of the element’s bitmap.

– HTML5 Specification §4.11.4 The canvas element

Aspect-Ratio Placeholder

Alternatively, a placeholder can be created with a div rather than the canvas element. By using a pseudo element to preserve the aspect ratio of the container, a fluid placeholder can be created, allowing the area to be “painted over” by absolutely positioning the image.

<div class="placeholder" style="--aspect-ratio:calc(100%/({{ width }}/{{ height }}));max-width:{{ width }}px">
  <img src="…" alt="…"/>
</div>

.placeholder {
  display: inline-block;
  width: 100%;
  position: relative;

  &::before {
    content: '';
    display: block;
    padding-top: var(--aspect-ratio);
  }

  img {
    position: absolute;
    top: 0;
    left: 0;
    width: 100%;
    height: 100%;
  }
}

Sizes Attribute

In addition to the layout attributes (width and height) described above, the sizes attribute (introduced win HTML5) may also be used to specify varying images sizes for different pixel densities and viewport widths.

<img src="…" width="100" height="100" sizes="100vw" />

While these layout attributes are useful for preventing layout reflows, they do not directly make the images scale as their container scales.

Future Solutions

A proposal for a permanent solution to this problem is in development. The intrinsicsize attribute may one day allow authors to define the aspect ratio of a media element in advance without the need for additional styling hooks or placeholder elements.

Responsive Aspect Ratio & Fluid Text

2016-01-18T12:34:25-05:00

When designing, developing and optimizing a responsive website, you will often encounter elements which maintain their aspect ratio while scaling. Images do by default in HTML, but videos, background images, calls to action, ads, and other graphical elements do not.

A developer’s first inclination may be to set the width and height of an element to a percentage value. However, because percentages in CSS are calculated based on the dimensions of the equivalent property of the elements parent container (i.e. a percentage based height will be based on the parent’s height, width will be based on width), decreasing an element’s width will not directly affect its height.

Some may advocate the use of JavaScript to calculate and set the height of the element on resize, while others might suggest using images set to visibility: hidden or data URLs akin to a spacer.gif. These solutions are inelegant and difficult to implement.

There is, however, a simple elegant, pure CSS solution to the problem:

.aspect-ratio {
  position: relative;
  overflow: hidden;
}

.aspect-ratio::after {
  content: '';
  height: 0;
  padding-bottom: 33%; /* 1:3 Ratio - Change this value */
  display: block;
}

.aspect-ratio > * {
  position: absolute;
}

.aspect-ratio > .cover {
  top: 0;
  right: 0;
  bottom: 0;
  left: 0;
}

So, what’s the trick?

The proof is in the padding. According to the spec, when padding is defined as a percentage it is calculated based on the “width of the containing block” (i.e. parent container).

This may seem odd at first, but let’s examine the concept for a moment. If you define padding: 10px for an element, you would correctly assume that the padding would be a uniform 10 pixels around the element. Similarly, if you assign padding: 10% to an element, you would expect the padding to be uniform. If the top and bottom padding were calculated based on the height or the parent container rather than the width, the vertical padding would only be equal to the horizontal padding when the parent container was a perfect square.

Because pseudo elements are children of the element on which they are declared, setting a pseudo element to display: block will allow it to fill its parent horizontally and will allow it to expand vertically to the height of it’s contents. Assigning a percentage based padding value to either property on the horizontal axis (top or bottom) will create a box which maintains an aspect ratio based on the parent’s width.

Any content within the parent will also contribute to the height of the parent element. To prevent this, the direct descendants the containing block must be absolutely positioned. The .cover class in the example code above, can be used to allow a direct descendant to fill the container. Other content can be positioned manually, as needed.

Note: Some content – like videos – will require the use of width: 100% and height: 100% when covering an element.

Why not margins?

If you read the spec you will know that margins are also calculated based on the width of the containing block, so why shouldn’t we use margins? Margins collapse, padding doesn’t.

Adding Images & Content

Using this trick in combination with CSS viewport units (like vw) and background-size: cover allows developers to create completely fluid elements.

Attributes vs Categories

2016-01-18T12:25:00-05:00

Information architecture is one of the most important user experience elements of a website because it directly affects navigation, often affects URL structure, and it is important to both SEO and conversion optimization. On ecommerce websites, product “merchandising” may impact aspects of information architecture. To ensure that users can find the information they are looking for as quickly and as easily as possible, it is important to understand the technical differences between categories and attributes.

Attributes

Attributes are a form of description which outline the properties of an object (a product, a blog post, a user, etc.). An attribute defines an objective feature or quality regarded as a characteristic or inherent part of an object. Generally attributes are immutable, meaning they do not change over time. However, there are some attributes, such as a product’s rating or price, that may change over time. A single item may have a number of different attributes, but generally each item will have only one value for each attribute.

Tip: Attributes are generally adjectives.

Not all attributes need to be visible to a user, however attributes that are displayed may be combined with special markup and other technology to provide relevant information to search robots, crawlers, and assistive technology. This information can be used to enhance search results and provide a superior user experience. For example, attributes allow items to be easily compared, and because attributes may represent values on a range, attributes can be used to easily filter a large collection of items. This helps users quickly locate items that meet a specified criteria.

A t-shirt has a size, color, and style (and many other attributes like price, manufacturer, material, and rating). A consumer may want to compare price and rating for black, v-neck t-shirts that are either small or medium. Through the use of attributes, these types of complex filters and comparisons are made possible. The more accurately an object is described through attributes, the easier it will be to search, sort, or filter the object.

Crossover

On ecommerce sites, it is common for an important attribute value to also be used as a category; this may happen if an attribute is particularly popular, necessary in navigation or SEO, or is essential to marketing. Generally when attributes must be surfaced as categories, it is best to automate the categorization with rules so that the overhead of catalog maintenance remains low.

It may be helpful to think of categories as shelves, drawers or containers to place items inside of. Moving an item from one container to another does not change the qualities of the object itself (it’s attributes), but it does change it’s classification (category).

Going back to our previous t-shirt example, a v-neck t-shirt may be included in a “Featured” category which highlights items that are highly valued by customers or business objectives; this type of category would not need to rely on an attribute because being “featured” is not an objective quality of the t-shirt itself, instead it is a subjective grouping. The same v-neck t-shirt may also appear in a “V-necks” category that is populated automatically with all the available v-neck t-shirts. This “promotion” of the style attribute will allow visitors to locate the item, and other similar items more quickly particularly if the category is featured prominently in navigation, and by appearing in search results.

Flat vs Hierarchical URL Structure

2016-01-18T11:02:20-05:00

Some SEO consultants recommend “flat” URL structures. However, there is no evidence that a flat URL structure is better than a clearly hierarchical URL structure. While there is some debate in the SEO community, most reputable sources agree that there is no SEO/SERP benefit to flattening URL structure, and that hierarchical URLs are better for UX, crawling, and data analysis.

Since the flat site architecture concept appeared on the SEO horizon … many SEO consultants got it wrong. The flat site architecture concept is related to the click distance between pages in a site, and how relevancy is distributed according to internal links structure — yet has nothing to do with URLs.

The main misunderstanding was, and unfortunately still is, that you have to get rid of directories in URL structures. Although it is widely agreed that you may want to keep URLs short and locate keywords close to the root or left part of the URL, there are many reasons why you should keep a certain structure of folders or directories

– 3 Reasons to Always Have Structured URLs

What is a flat URL?

A flat URL refers to the path segment of a URL being not more than one “level” deep.

[The path contains data], usually organized in hierarchical form, that appears as a sequence of segments separated by slashes. Such a sequence may resemble or map exactly to a file system path, but does not always imply a relation to one.

– Uniform Resource Identifier Syntax, Wikipedia

In other words, a flat URL structure is one in which all URLs appear at the top level, with no hierarchy.

Let’s imagine, for example, that you have a website selling greeting cards; you may have an assortment of cards which you categorize as “musical” (the kinds of cards that play back a recording when opened). You want users to be able to locate these cards independently, but you also want users to be able to navigate, via subcategories, to musical cards for specific occasions and holidays. Let’s examine how this would be treated in a hierarchical URL structure and a flat URL structure:

Hierarchical URLs

example.com/holiday/new-year/musical/
example.com/holiday/musical/
example.com/occasion/birthday/musical/
example.com/occasion/musical/
example.com/musical/

Flat URLs

example.com/holiday/
example.com/new-year/
example.com/occasion/
example.com/birthday/
example.com/musical/

How flat is too flat?

As you may have noticed in the examples above, because there is no hierarchy in the flat URL structure, problems become apparent when multiple hierarchical categories use the same name (i.e. the delineation between “musical birthday cards” and “musical New Year’s cards”).

While it may be technically possible to present different content at a single URL for each user based on their flow through the website or entry-point, this will create a very confusing user experience, and make it difficult or impossible for search engines to index the site’s content correctly, in it’s entirety.

A solution to this problem is to maintain a pseudo hierarchical URL structure, replacing slashes (/) with dashes (-). For example:

Alternative Flat URLs

example.com/holiday-new-year-musical/
example.com/holiday-musical/
example.com/occasion-birthday-musical/
example.com/occasion-musical/
example.com/musical/

However, the question remains: why use a flat URL structure over a hierarchical one?

Arguments for Flat URLs

There are four arguments SEO consultants make when recommending flat URLs over hierarchical URLs:

Nested PageRank: The further a page resides from the root domain, the less PageRank is attributed to that page.
URL keyword density: The longer the URL, the less emphasis is placed on targeted keywords in the URL.
Usability: The shorter a URL is, the more user-friendly, sharable, and marketable it is, leading to increased conversion.
URL migration: Relying on directories negatively affects the ability to migrate URLs in the future with minimal SEO impact.

Note: The arguments above make incorrect assumptions.

URL Structure vs Site Architecture

While it is true that long, unreadable URLs do not lend themselves to good SEO or good UX, a site’s depth (i.e. architecture) is measured by how many clicks are required to navigate from one page to another (i.e. “click distance”) and by how pages are cross-linked. URL structure, “directory” (i.e. path segment) count, and keyword positioning is irrelevant to a site’s “flatness” as interpreted by search engines.

Google’s Matt Cutts debunks many of the incorrect assumptions about how URL structure affects search rankings:

Does the number of subdirectories in a URL affect its ranking?: [At Google,] there is virtually no difference [between flat URLs and hierarchical URLs] … It can be very helpful for users if you have subdirectories … [Organizing content within subdirectories] can be a great thing for usability, and I definitely recommend having your site categorized like that … Having one or two extra directories [in your URLs is] not a major factor in Google’s search engine rankings.
How does URL structure affect PageRank?: Google doesn’t worry about how deep a set of directories is.
Does the position of keywords in the URL affect ranking?: [While] it does help a little bit to have keywords in the URL … if there is a convenient way [to structure the URLs in a way] that’s good for users … [don’t worry about] how deep the [keywords] are in the path; take the first 2–5 words [of the content] and use that as the URL. Don’t use [too many] words [in the URL], because that looks spammy to users, and people will probably not click through as much. [Worry less about the position of keywords in the URL, and focus more on] having great content that people want to link to or that people want to find out about.

Google’s John Mueller also addressed URL length, stating the Google will “crawl and index URLs over 1000 characters long – but that doesn’t mean it’s a good practice”. The common SEO recommendation is to limit the full, indexed URL (including the scheme [http/https] and host [domain]) to between 75–115 characters in length to allow the full URL to be displayed in SERPs and preserve the readability and shareability of links.

Tip: While shorter URLs are preferred for readability, excessively short URLs, and supremely flat URLs are not recommended. Users should be able to predict content based on the URL; shorter, readable, memorable URLs are good for users, and hierarchy can help in that.

Benefits of Hierarchical URLs

Hierarchical URLs address the failings of flat URLs.

Search Engine Results Pages (SERPs)

Structured hierarchical URLs reinforce a search engine’s semantic understanding of content; search engines use URLs to calculate link value, and highlight keywords in URLs on SERPs. URLs containing keywords hold more value, rank higher in results, and are more likely to draw a user’s attention.

Because URLs appear in search engine results, hierarchical URLs with relevant keywords help localize a user in a site’s content, and provide hints at additional content that may be offered on a site – adding to a site’s authority.

Tip: Adding semantic markup and using schema (structured data) in combination with a well organized, hierarchical URL structure will increase the chances of breadcrumbs displaying on SERPs.

User Experience

The primary argument for a flat URL structure is that flat URLs are memorable. Recent moves by search engines and browser vendors have shown that the memorability of URLs is not a critical element of a user journey. Ease of navigation and good SERP results are more important factors to discovery and conversion.

“Intuitive URLs” are simple URLs that may be easily guessed by users, these types of URLs are easy to remember and can increase direct traffic. A hierarchical structure provides a better user experience & improves the SERP.

The URL, as it appears in the browser toolbar, serves as a virtual breadcrumb trail. It provides users a general context to content hierarchy, and it provides an opportunity for more advanced users to manually alter the URL to retrace their path in to broader sections of the site. Removing this data removes relevant context from users.

Crawling

Hierarchical structures are a fundamental element of the web. Search engines and analytics tools are designed to crawl and analyze content and user flow through hierarchical tree structures. Structured URLs help semantics by revealing how a site’s content is structured.

As mentioned above, flattening the URL structure removes important context from both users and search engines. In our example above, both /holiday/new-year/musical/ and /occasion/birthday/musical/ would be replaced in a flat URL structure with a single URL – /musical/. Does that address contain content from both categories? Neither? How are search engines (or users) to interpret that content?

Note: It is speculated that Google practices “URL stripping”, whereby “directories” are systematically removed from the path and crawled to discover new content. Therefor, it is important for category pages to contain a relevant description and links to its contents. “Virtual directories” which return no content or a 404 status code when queried directly may result in extra crawl errors.

Analytics

It is essential to analyze organic traffic by sections of a site through content drill down reports available in analytics tools; these tools rely on structured URLs to monitor user flow.

Gathering data on indexed pages is made easier with hierarchical URLs, enabling Google queries like site:example.com/category/. As noted below, Google has limits on how many results can be returned with a single query. For sites with many URLs, refining a indexing queries to subcategories can workaround these limitations.

Note: Google limits the maximum number of search results for a single query to 1000 results.

Migration

When migrating to a new URL structure, effective redirection is critical maintaining conversions across direct and organic traffic. Redirection is made easier with hierarchical URLs because regular expressions can be used to redirect entire sections of URLs to a new location.

In any case, migrating a site’s URL structure typically does have a temporary negative impact on organic traffic – usually a 10–20% drop in organic traffic can be expected during the first 1–3 months after a migration has occurred.

Product URLs

Product URLs are not inherently hierarchical, as a single product may appear in multiple categories. For optimal UX, the “virtual breadcrumb” URL structure should be preserved by using hierarchical URLs when navigating to products via categories and subcategories. However, product URLs should also appear at the root level (as a flat URL), and all hierarchical product URLs should be canonicalized to the flat URL, so the shorter URL appears in SERPs without creating duplicate content issues.

Crafting 301 Redirects

2016-01-15T09:51:08-05:00

When launching a website on a new platform or when changing the information architecture of a website, it is usually necessary to configure redirects to ensure users can locate the content they expect even when accessing the website via legacy URLs. These redirects also instruct search engines to reindex content at a new location and transfer page ranking.

Tip: If you are migrating a website to a new domain or subdomain, be sure to use Google’s Change of Address tool in addition to configuring the appropriate redirects.

Different Types of Redirects

Redirects are HTTP status codes which instruct the browser to perform a certain action. There are many different status codes which fall into different classes. Redirect statuses follow the pattern 3XX, with the primary status codes for redirects being 301 (permanent redirect) and 302 (temporary redirect), though there are other statuses – notably the 307 (temporary redirect) and 308 (permanent redirect) – that specify the user agent must not change the HTTP method used in the original request.

Permanent Redirects

The 301 (and 308) status code instructs the browser that the content of a resource has been moved permanently to a new location. In other words, the browser is told that it should no longer request the original location, instead using the new location for all future requests.

When a search engine robot encounters a permanent redirect it replaces the indexed URL with the new URL and transfers (a percentage) of PageRank to the new URL.

When a web browser encounters a permanent redirect it caches the redirect and will not attempt to request the original location again, instead pointing to the new location, until the cache is cleared.

It is possible to unset a cached permanent redirect by re-redirecting back to the original URL from the destination URL. For this reason, be cautious when performing permanent redirects to external targets, as it will be impossible to unset the redirect.

Temporary Redirects

The 302 (and 307) status code instructs the browser that the content of a resource has been moved temporarily to a new location. In other words, because this redirect may be altered occasionally, the browser should continue to request the original location for future requests.

Interestingly temporary redirects receive some perhaps unexpected treatment by search engines: “for on-domain [temporary] redirects… search engines will usually [index] the shorter URL” because “normal users usually like short, clean URLs.”

It is a common misconception that temporary redirects do not transfer PageRank, they do. Though it has also been suggested that 302 redirects transfer less PageRank than 301 redirects (which may be true).

Keep in mind that search engines are constantly updating their algorithms to deliver the best results. More than anything, they want webmasters to build websites that provide a great user experience and valuable content. Scheming to acquire more “link juice” is not a valid argument for using an improper redirect; in fact, Google has taken steps in the past to reduce visibility into PageRank to “avoid confusing users and webmasters about the significance of the metric.” As Matt Cutts says “use whatever is best for your purposes.”

Locating URLs

Before writing a redirect you must know three things:

The source URL
The destination (or target) URL
The redirect status

Acquiring source URLs

The easiest way to acquire a list of known URLs is to locate a website’s sitemap(s). The following script can be used to search for sitemap declarations in the robots.txt file and by testing for popular sitemap filenames.

##
# Locate sitemap path for a given domain
#
# @param [url] Domain name to search
##
function find_sitemap() {
  # Get sitemap entries from a `robots.txt` file
  sitemap=$(curl -sL "$1/robots.txt" | awk 'tolower($1) ~ /^sitemap/ {print $2}')
  if [[ -z $sitemap ]]; then
    # Try different combinations of capitalization and punctuation
    # sitemap site_map site-map
    output=$(curl -sLIw "%{http_code} %{url_effective}" "$1/sitemap.xml" -o /dev/null)
    if [[ $(echo $output | awk '{print $1}') == 200 ]]; then
      sitemap=$(echo $output | awk '{print $2}')
    else
      echo "No sitemap found"
      return 1
    fi
  fi
  echo "$sitemap"
}

find_sitemap $@

Alternatively, if the site has a Google Webmaster Tools account configured, you may check the Crawl Sitemap section to see if any sitemaps are known to Google.

If none of the above methods have located a valid sitemap, you may need to resort to searching Google for possible sitemaps. The search queries below may assist in locating a sitemap:

site:example.com filetype:xml inurl:sitemap

site:example.com filetype:xml

If you still cannot locate a valid sitemap file, you may need to build a list based on search engine indexes.

Extracting URLs from a Sitemap

In addition to a list of URLs, sitemaps include other information, unnecessary for the purposes of constructing redirects. To extract URLs from a sitemap, use the following script:

##
# Parse URLs out of a `sitemap.xml` file
# This will follow nested sitemap URLs
#
# @param [string] File or URL or sitemap
##
function parse_sitemap() {
  # Check is sitemap is local or remote
  net=
  if [[ "$1" =~ ^https?:// ]]; then
    http_code=$(curl -sLIw "%{http_code}" "$1" -o /dev/null)
    if [[ $http_code != 200 ]]; then
      return 1
    fi
    net='--net'
  fi
  # Must match the XML namespace to work correctly
  xmlns='http://www.sitemaps.org/schemas/sitemap/0.9'
  # Get all URLs - remove blank line if no URLs are found
  xml sel $net -N x=${xmlns} -t -v '//x:url/x:loc' -n "$1" | sed '/^\s*$/d'
  # Recursively parse embedded sitemaps
  sitemaps=$(xml sel $net -N x=${xmlns} -t -v '//x:sitemap/x:loc' -n "$1" | sed '/^\s*$/d')
  if [[ -n "$sitemaps" ]]; then
    while read -r sitemap; do
      parse_sitemap "$sitemap"
    done <<< "$sitemaps"
  fi
}

parse_sitemap $@

Note: This script requires XMLStarlet, and will descend recursively into nested sitemaps.

Writing Redirects

Once a list of source URLs is obtained, it’s time to begin mapping to the destination URL(s).

Note: This section describes the different modules and directives that may be used to author redirects in Apache; be aware of your server type and version.

There are two Apache modules that may be used to construct redirects:

These modules provide three directives that can be used for redirection; mod_alias provides Redirect and RedirectMatch while mod_rewrite provides RewriteRule. As noted by the Apache documentation for mod_alias:

mod_alias is designed to handle simple URL manipulation tasks. For more complicated tasks … use the tools provided by mod_rewrite.

The following rules may be applied when selecting the appropriate directive for a redirect:

Redirect: Simple one-to-one redirect mapping.
RedirectMatch: Intermediate redirect mapping supporting regular expressions.
RewriteRule: Advanced redirect mapping supporting regular expressions, conditionals and access to the query string.

Apache has also published helpful documentation on when not to use mod_rewrite, which echos the above:

[S]imple redirection of one URL, or a class of URLs, to somewhere else, should be accomplished using [the Redirect and RedirectMatch] directives rather than RewriteRule. RedirectMatch allows you to include a regular expression in your redirection criteria, providing many of the benefits of using RewriteRule.

The Redirect directive

The Redirect directive is extremely simplistic; it maps one URL to another:

Redirect 301 "/source" "/destination"

Note: In the example above (and with the other directives mentioned below), the redirect’s HTTP status code argument may be omitted. If no status code is defined, the redirect will be a 302 temporary redirect.

There are a few keywords that may be used in place of a numeric status code, which may improve readability. These keywords are limited however, and numeric status codes offer more flexibility and allow for greater consistency when a number of different redirect statuses are defined.

To improve clarity, always include a numeric status code with all redirect directives.

The RedirectMatch directive

The RedirectMatch directive expands the utility of Redirect while maintaining it’s elegant simplicity. Any valid Redirect directive is also a valid RedirectMatch.

RedirectMatch 301 "/source" "/destination"

The real power of RedirectMatch derives from its ability to interpret regular expressions (commonly referred to as regex). To master RedirectMatch (or redirection in general), learn regular expressions. Understanding even the basics of regex can be extremely valuable when redirecting a large number of URLs.

# Remove an extension from all URLs
RedirectMatch 301 "(.*)\.html" "$1"

# Move/rename a category
RedirectMatch 301 "/old-category/(.*)" "/new-category/$1"

# Redirect subdirectory to subdomain
RedirectMatch 301 "^/(blog)/(.*)$" "http://$1.example.com/$2"

This illustrates the power of using RedirectMatch with regular expressions.

Nevertheless, there is still much that RedirectMatch cannot accomplish.

The RedirectRule directive

The RedirectRule directive uses a more complex syntax to accomplish redirection. This added complexity affords much more flexibility and power.

RedirectRule /source /destination [R=301,L]

Tip: Some servers may not have mod_rewrite enabled. To avoid 500 errors wrap mod_rewrite directives in an IfModule section. Also, ensure that the RewriteEngine directive has been set to “On”, otherwise the RedirectRule directives will have no effect.

<IfModule mod_rewrite.c>
  RewriteEngine On
  # …
</IfModule>

The power of RewriteRule lies in its ability to be combined with conditional statements (RewriteCond) and access server variables, environment variables, HTTP headers, and time stamps.

# Redirect insecure, non-www to secure, www
RewriteCond %{HTTPS} !on [OR]
RewriteCond %{HTTP_HOST} !^www\. [NC]
RewriteCond %{HTTP_HOST} ^(?:www\.)?(.*) [NC]
RewriteRule (.*) https://www.%1/$1 [R=301,L]

# Collapse multiple slashes
# Note: The "$0" in the substitution string matches
# the entire contents of the matched pattern,
# regardless of grouping
RewriteCond %{THE_REQUEST} //
RewriteRule .* /$0 [R=301,NE]

# Parse data from a query string
# and remove the query string from the URL
# Note: In Apache 2.4 or later, the "QSD" option (qsdiscard)
# can be used to remove the query string opposed to
# appending a "?" the the substitution string
RewriteCond %{QUERY_STRING} page=([^&]+)
RewriteRule ^ /%1? [R=301,L]

# Redirect certain devices by user agent
RewriteCond %{HTTP_USER_AGENT} "i(phone|pad|pod)" [NC]
RewriteRule ^ http://m.example.com/ [R=302,L]

# Serve pre-compressed gzipped or bred (Brotli) assets
# Prefers Brotli compressed assets
RewriteCond %{HTTP:Accept-Encoding} \b(br)\b [OR]
RewriteCond %{HTTP:Accept-Encoding} \b(gz)ip\b
RewriteCond %{REQUEST_FILENAME} !.+\.%1$
RewriteCond %{REQUEST_FILENAME}.%1 -f [OR]
RewriteCond %{REQUEST_FILENAME}.%1 -l
RewriteRule (.*) $1.%1 [QSA,L]

Note: Be aware that not all directives are interpreted sequentially due to hooks.

As the Apache documentation notes:

The use of RewriteRule … may be appropriate if there are other RewriteRule directives in the same scope. [W]hen there are Redirect and RewriteRule directives in the same scope, the RewriteRule directives will run first, regardless of the order of appearance in the configuration file.

Placing a Redirect directive before a RewriteRule directive does not mean it will be executed first. mod_rewrite is loaded before mod_alias because of hooks, so RewriteRule directives are executed before Redirect and RedirectMatch directives.

Testing Redirects

While writing redirects, it is helpful to use a command line tool for testing. This avoids browser caching.

Because a percentage of PageRank “dissipates” with each redirect, and because each redirect contributes to a slower time to first byte, the total number of redirects between the source URL and destination URL (redirect “hops”) should be kept to a minimum. Whenever possible, each redirect should be accomplished in only one hop. The following script will display all the redirect hops of a single URL:

##
# Display all redirects (and their status code) between
# a given URL and the "effective URL"
#
# @param [url] The URL to crawl
##
function redirects() {
  url="$1";
  while data=$(curl -Iso /dev/null -w '%{http_code} %{redirect_url}' "$url"); do
    http_code=$(echo "$data" | awk '{print $1}')
    echo "$http_code $url"
    url=$(echo "$data" | awk '{print $2}')
  done
}

redirects $@

While the script above is helpful in a micro-sense – allowing developers to test individual URLs – the script below is more helpful in the macro-sense – allowing developers to test a large number of URLs at once.

Given a list of URLs to test, this script will display each URL in a file alongside the effective URL and it’s status code in CSV format.

##
# Test URLs from a list
# Prints HTTP status code, redirect hops, source URL, and effective URL
# in CSV format
#
# @param [file] List of URLs to test
# @param [string] Base URL to test URLs against
##
function test_urls() {
  base_url="$2"
  if [ -n "$base_url" ]; then
    # Remove protocol and domain name and
    # replace with new base URL
    sed -E "s%^https?://[^/]*%%g" "${1:-/dev/stdin}" | xargs -I {} \
      curl -o /dev/null --head -sLw \
      "%{http_code},%{num_redirects},${base_url}{},%{url_effective}\\n" "${base_url}{}"
  else
    cat "${1:-/dev/stdin}" | xargs -I {} \
      curl -o /dev/null --head -sLw \
      "%{http_code},%{num_redirects},{},%{url_effective}\\n" "{}"
  fi
}

test_urls $@

Tip: When analyzing the data generated by the script above, filter content by the status codes. Unsuccessful requests (≠ 200) – particularly 404 – indicate an error which may need to be addressed with redirects.

Text Resize Widgets

2015-04-08T13:25:00-04:00

Text resize widgets are redundant artifacts of the 20th century web, many being poorly designed an poorly implemented. While these widgets were intended to assist with accessibility, in reality, they not only fail to improve accessibility, but they also detract from site design and negatively affect site performance.

Antique Accessibility

In 1997, the W3C launched the Web Accessibility Initiative. Designed to educate designers and developers on how to improve accessibility on the web for users with disabilities, the WAI encouraged many designers and developers to implement their own solutions to solve accessibility needs. As the initiative grew, browsers began implementing their own solutions for users with disabilities. Opera 2.1, released in Dec. 1996, was the first browser to implement a “Zoom” feature, but it took 10 years for Microsoft to implement browser zoom, which was included in IE7. Today, all modern browsers support some variation of page zoom – with some browsers offering additional options like “Zoom Text Only”.

Accessibility Issues

The main problem with text resize widgets is their inherent inaccessibility. Let’s consider, for a moment, whom text resize widgets might benefit. The first group that comes to mind is users with poor eyesight. Now, let’s examine what happens when a user with this type of disability visits a page that utilizes a text resize widget. Where should they direct their attention if they are unable to read the content of the page? The answer obviously being the text resize widget. But where exactly is the text resize widget? This varies from site to site, occasionally even from page to page, and besides, the visitor has a disability – poor eyesight.

Depending on the size, contrast, prominence and clarity of purpose of the widget, it can be very difficult for someone with accessibility issues to identify. Because these widgets rely on the users ability to locate them on the page, they are themselves inaccessible.

Not only does the text resize widget make the site less accessible for users with poor vision, it also diminishes accessibility and usability for a number of other users. As noted by Jared Smith on Web Accessibility In Mind:

Users who are blind generally don’t care how big the text is, yet they must listen to and navigate through the widget controls on every page. Users with motor disabilities who use only the keyboard must also ‘tab’ through these controls while gaining no benefit from them. Users with auditory disabilities gain no benefit. Users with cognitive disabilities must handle the additional cognitive load of these controls (what does a stack of increasingly sized A’s mean anyway?). And most notably, users with adequate vision gain no utility from the widget (again, assuming your default font size is adequate), yet are still presented with the control on each and every page.

In fact, the number of users who potentially suffer from the negative side effects of poorly implemented text resize widgets far outnumbers the users who might potentially benefit. When implemented properly, these widgets rely on additional JavaScript, markup, styling, and often images in addition to cookies which are stored on a users machine to save preferences across the site and between visits. These aspects all contribute a negative impact to overall site performance, degrading the experience of every visitor to the site.

Intelligent Design

To minimize overhead and improve the quality of each visitor’s experience, the target audience should always be carefully considered during the design phase – text sizes should be selected accordingly. Smith argues that the addition of a text resize widget is an admission of error by the designer – “either they are forcing the user to account for a poor design or usability decision, or they are too indecisive to make the decision to begin with and thus place the burden to decide upon the user.”

By including a text resize widget on a site, a designer is making two assumptions:

a significant number of site visitors may require larger font sizes
visitors are not savvy enough to establish a larger default font setting

Under these conditions a designer could, hypothetically, create a text resize widget that takes prominence, is clearly identifiable and appropriately size, that when implemented correctly could improve accessibility for a very small percentage of users. However, the designer must weigh the sacrifices in design, accessibility, and usability for every other site visitor.

The Bottom Line

There are much better ways to improve accessibility than using text resize widgets, for example:

size text with relative units (like ems) instead of pixels to ensure that built-in zoom functions properly
leverage appropriate foreground-background contrast to improve readability
select a font size which suits the needs of the target audience to improve the quality of a visit
educate less-savvy users on how to use built-in functionality to make the web more progressive

CSS Font Units

2014-02-26T22:33:18-05:00

When Marshall McLuhan coined the phrase “the medium is the message”, he referred to a concept which lends itself well to modern day web design. He contended that the medium in which a message is delivered affects how that message is perceived. In the same way, today’s web developers must select the appropriate font units for the medium where their work is presented. The choice of font units has a literal impact on how the message is perceived. Therefore, it is important for designers and developers alike to be aware of what units are available and when to use them.

According to the W3 Consortium, lengths refer to horizontal or vertical measurements, and there are two types of length units: relative and absolute.

Relative units are:

em: the font-size of the relevant font
ex: the x-height of the relevant font
ch: width of the “0” (ZERO, U+0030) glyph in the element’s font
rem: font size of the root element
vw: 1% of viewport’s width
vh: 1% of viewport’s height
vmin: 1% of viewport’s smaller dimension
vmax: 1% of viewport’s larger dimension

Absolute units are fixed in relation to each other:

in: inches — 1in is equal to 96px
cm: centimeters – 1in/2.54 = 37.795px
mm: millimeters – 1/10th of 1cm = 3.779px
q: quarter-millimeters – 1/40th of 1in = 0.944px
pt: points — 1/72nd of 1in = 1.333px
pc: picas — 1/6th of 1in = 16px
px: pixels — one reference pixel

The CSS specification states, “absolute units are mainly useful when the output environment is known.” This is an updated paraphrasing of the old spec that stated, “absolute length units are only useful when the physical properties of the output medium are known.” With the advent of mobile devices the variety of screen sizes, resolutions, and orientations is more diverse than ever. A developer can seldom be certain of the physical properties of the output medium. To accommodate these discrepancies, the W3 Consortium recommends that developers rely on relative units to create flexible designs that render well on an assortment of displays.

The Incredible Em

By using ems, developers can improve accessibility, maintain flexible layouts and create code that is more readable. Ems are particularly well suited for font sizing on the web because an em is a measurement of typography and a relative unit. 1em is equal to the current font size; font size is an inherited property—i.e. it is passed from a parent element to its children.

The web is all about choice and accessibility. If an end user chooses to set their default font size to a value larger than 16px (which is most all browsers default), they should receive content that has been appropriately sized to fit their needs. This is particularly important for users with poor eyesight and small monitors. The following example demonstrates what occurs when absolute units are used:

div {font-size: 16px;}
h1  {font-size: 24px;}
p   {font-size: 12px;}

In the example above, if an end user decides to increase the default font size to 20px, the font-size will not change because pixels are absolute. However, if the units are changed to ems something very different happens:

div {font-size: 1em;}
h1  {font-size: 2em;}
p   {font-size: .75em;}

By default, the above example will look identical to the one before it, but when an end user opts for a larger default font size of 20px, the <h1> will receive a calculated font size of 40px, and the <p> will receive a font size of 15px. Admittedly, the example above is somewhat archaic. Most modern browsers can effectively scale pixels in the same way they scale ems, but he also points out their practical use in flexible designs.

Fluid, elastic and responsive designs rely heavily on ems to create scalable sites in addition to @media queries which can be used to target different devices. Developers who make use of ems can easily improve text readability on smaller screens by increasing the font size on the body and allowing it to cascade.

This makes code cleaner, because stylesheets can be smaller and more readable. A developer need not know what the actual display size is, so long as the ratio between elements is appropriate.

Note: There is a bug in older versions of Internet Explorer that “unacceptably exaggerate the smallness and largeness of the resized text,” according to A List Apart. The solution is simple, elegant, and is included in many CSS resets by default. Simply set body {font-size:100%;}.

Bonus Points

Because absolute units are particularly well suited to applications where the size of the medium is fairly consistent, points are best used in print design. Print design should not be neglected when designing for the web. It’s a good idea to target the printed page by making use of @media print queries to hide interactive elements such as site navigation, reset text and background colors, and redeclare font sizes in points to allow for more control over the printed page in an effort to conserve paper and ink while creating a better overall presentation.

Note: Because ems are relative to the current font size, they can also be effectively used in print style sheets. It is recommended that developers declare <body> size in points and allow the document to cascade.

Unitless Bliss

The CSS line-height property can accept unitless values, and should always be declared as such because it makes for cleaner, more readable style sheets. Unitless line height is relative and is calculated as a ratio of the current font size for each element.

ul {font-size: 15px; line-height: 1em;}
li {font-size: 10px;}

In the above example, the line height for each element is inherited from the <ul>, because ems are relative to the element on which they are defined the calculated line-height for the <ul> is 15px. This value is inherited by the children, so both the <li> and the <small> have a line-height of 15px.

Unitless values, on the other hand, are inherited as a “raw number”. By modifying the above CSS to use unitless values, the line-height of each child element is set relative to itself and it’s font-size, even if its font-size has changed.

ul {font-size: 15px; line-height: 1;}
li {font-size: 10px;}

In this example the <ul> still receives a line height of 15px, but now the <li> receives a line height of 10px, and the <small> gets a calculated line height 8px.

Live Demonstration

See the Pen Unitless line-height demo by Joey Hoer (joeyhoer).

This makes code cleaner, as it takes less CSS to achieve consistent styling. Additionally, readability is improved in the same way that ems improves readability of the font-size property; a developer doesn’t need to know anything about the child elements to modify their relative line-heights by making effective use of the cascade.

Joey Hoer's Blog

Creating Medium-style Permalinks in Jekyll

Duplicate Content

Always Accurate

Unfolding the Fold

What is “the fold”?

The Bottom Line

The Digital Fold

The Effective Use of Story Points

What is a story point?

What are story points used for?

Making timeline predictions with story points

Story Points vs. Man-Hours

Points to Actual Hours Conversion

Calibrating the Team

Communicating with Stakeholders

Thoughts on AMP

What is AMP?

Why does AMP exist?

Benefits of AMP

Criticisms of AMP

How does Google benefit from AMP?

Speed without AMP

Static Site URL Shortener

Faster Redirects with DBM files

Advanced nth-child Selectors

Modulo Selector

Middle Element Selector

Half of Elements Selector

The URL is Dead, Long Live the URL

The Long, Slow Death of the URL

Defending the URL

Conversational UI

Designing URLs for Humans

URL Length

URL Simplicity

Hyphens vs Underscores

Lowercase URLs

Remove File Extensions

Trailing Slash

Collapse Multiple Slashes

Stripping URL Parameters (Query String)

Redirect to secure www

URL Hierarchy

Namespaced URLs

URL Malleability

Query Strings for Facets

URL Persistence

Naming URLs

Migrating URLs via Redirection

One-Time Use URLs

Fixing Vertical Videos

History

Complaints

Biology

A Solution

The Curious Case of Route 53 and New Relic Compatibility

The Incident

The Clues

The Culprit

Robots Directives

Indexing Methods

Crawl Directives in robots.txt

Non-human Readable Files

Robots Tag

Noindex a Section of a Site

De-indexing Catch 22

Canonical Links

What is Canonicalization?

Defining a Canonical Link

Canonical Indexing

Content Discrepancies

Link Quantity

Link Placement

Broken and Malicious Links

Canonicalization vs Redirection

Canonicalized Pagination

Canonicalizing Facets

Combining with Robots Directives

The URL Structure of Faceted Navigation

Crawl Directives in `robots.txt`

History of the `ICO` format

`gifv`

`gifify`