Google search is no friend to the indie web:
Well-known brands often see most of their content indexed, while small or unknown bloggers face much stricter selectivity.
There was life before Google search. There will be life after Google search.
Information that you might search for may never appear in Google’s results. Not because it doesn’t exist, but because Google has chosen not to include it.
The whole point of the web is that we’re not supposed to be dependent on any one company or person or community to make it all work and the only reason why we trusted Google is because the analytics money flowed in our direction. Now that it doesn’t, the whole internet feels unstable. As if all these websites and publishers had set up shop perilously on the edge of an active volcano.
But that instability was always there.
There was life before Google search. There will be life after Google search.
Google is not a huge source of traffic and visibility. I get most of my visits from RSS readers, other people’s links including fellow bloggers, or websites like Hacker News. It’s hard to tell at this point since I don’t track anything, but that’s an educated guess.
Removing my website from Google would have very little impact, so I was wondering if I should just do it.
My phone rang today. I didn’t recognise the number so although I pressed the big button to answer the call, I didn’t say anything.
I didn’t say anything because usually when I get a call from a number I don’t know, it’s some automated spam. If I say nothing, the spam voice doesn’t activate.
But sometimes it’s not a spam call. Sometimes after a few seconds of silence a human at the other end of the call will say “Hello?” in an uncertain tone. That’s the point when I respond with a cheery “Hello!” of my own and feel bad for making this person endure those awkward seconds of silence.
Those spam calls have made me so suspicious that real people end up paying the price. False positives caught in my spam-detection filter.
Now it’s happening on the web.
I wrote about how Google search, Bing, and Mozilla Developer network are squandering trust:
Trust is a precious commodity. It takes a long time to build trust. It takes a short time to destroy it.
But it’s not just limited to specific companies. I’ve noticed more and more suspicion related to any online activity.
I’ve seen members of a community site jump to the conclusion that a new member’s pattern of behaviour was a sure sign that this was a spambot. But it could just as easily have been the behaviour of someone who isn’t neurotypical or who doesn’t speak English as their first language.
Jessica was looking at some pictures on an AirBnB listing recently and found herself examining some photos that seemed a little too good to be true, questioning whether they were in fact output by some generative tool.
Every email that lands in my inbox is like a little mini Turing test. Did a human write this?
Our guard is up. Our filters are activated. Our default mode is suspicion.
This is most apparent with web search. We’ve always needed to filter search results through our own personal lenses, but now it’s like playing whack-a-mole. First we have to find workarounds for avoiding slop, and then when we click through to a web page, we have to evaluate whether’s it’s been generated by some SEO spammer making full use of the new breed of content-production tools.
There’s been a lot of hand-wringing about how this could spell doom for the web. I don’t think that’s necessarily true. It might well spell doom for web search, but I’m okay with that.
Back before its enshittification—an enshittification that started even before all the recent AI slop—Google solved the problem of accurate web searching with its PageRank algorithm. Before that, the only way to get to trusted information was to rely on humans.
Humans made directories like Yahoo! or DMOZ where they categorised links. Humans wrote blog posts where they linked to something that they, a human, vouched for as being genuinely interesting.
There was life before Google search. There will be life after Google search.
Look, there’s even a new directory devoted to cataloging blogs: websites made by humans. Life finds a way.
All of the spam and slop that’s making us so suspicious may end up giving us a new appreciation for human curation.
It wouldn’t be a straightforward transition to move away from search. It would be uncomfortable. It would require behaviour change. People don’t like change. But when needs must, people adapt.
The first bit of behaviour change might be a rediscovery of bookmarks. It used to be that when you found a source you trusted, you bookmarked it. Browsers still have bookmarking functionality but most people rely on search. Maybe it’s time for a bookmarking revival.
A step up from that would be using a feed reader. In many ways, a feed reader is a collection of bookmarks, but all of the bookmarks get polled regularly to see if there are any updates. I love using my feed reader. Everything I’ve subscribed to in there is made by humans.
The ultimate bookmark is an icon on the homescreen of your phone or in the dock of your desktop device. A human source you trust so much that you want it to be as accessible as any app.
Right now the discovery mechanism for that is woeful. I really want that to change. I want a web that empowers people to connect with other people they trust, without any intermediary gatekeepers.
The evangelists of large language models (who may coincidentally have invested heavily in the technology) like to proclaim that a slop-filled future is inevitable, as though we have no choice, as though we must simply accept enshittification as though it were a force of nature.
But we can always walk away.
In their rush to cram in “AI” “features”, it seems to me that many companies don’t actually understand why people use their products.
Google is acting as though its greatest asset is its search engine. Same with Bing.
Mozilla Developer Network is acting as though its greatest asset is its documentation. Same with Stack Overflow.
But their greatest asset is actually trust.
If I use a search engine I need to be able to trust that the filtering is good. If I look up documentation I need to trust that the information is good. I don’t expect perfection, but I also don’t expect to have to constantly be thinking “was this generated by a large language model, and if so, how can I know it’s not hallucinating?”
“But”, the apologists will respond, “the results are mostly correct! The documentation is mostly true!”
Sure, but as Terence puts it:
The intern who files most things perfectly but has, more than once, tipped an entire cup of coffee into the filing cabinet is going to be remembered as “that klutzy intern we had to fire.”
Trust is a precious commodity. It takes a long time to build trust. It takes a short time to destroy it.
I am honestly astonished that so many companies don’t seem to realise what they’re destroying.
I don’t use Google Search myself—I use Duck Duck Go—but if you do, here’s how to avoid the slop.
UX London is just two months away!
The best way to enjoy the event is to go for all three days but if that’s not doable for you, each individual day is kind of like a mini-conference with its own theme.
The theme on day one, Tuesday, June 18th is design research.
In the morning there are four fantastic talks.
After lunch you’ll take part in one of four workshops. Choose wisely!
After your workshop there’s one final closing keynote to bring everyone back together. I’m keeping that secret for just a little longer, but trust me, it’s going to be inspiring—plenty to discuss at the drinks reception afterwards.
That’s quite a packed day. If design research is what you’re into, you won’t want to miss it. Get your ticket now.
Just to sweeten the deal and as a reward for reading all the way to the end, here’s a discount code you can use to get a whopping 20% off: JOINJEREMY.
My presumption that one should examine evidence before reaching a conclusion, rather than using it to support a conclusion, was not even an idea they could understand well enough to reject.
I am not against trying to be persuasive. That is a necessary art. But I was shaken they could not conceive of any use of information other than persuasion.
It’s a dream team of former Clearlefties: Harry and Cennydd joined forces to investigate and write an in-depth report looking into deceptive design practices used by Microsoft to stop people changing their default browser from Edge. They don’t pull their punches:
We judge that Microsoft cannot justify the use of these techniques, and should stop using them immediately. If they do not, we would welcome – where the law provides for it – regulatory intervention to protect against these harms.
No argument from me.
On the sad state of Google search today:
How did a site that captured the imagination of the internet and fundamentally changed the way we communicate turn into a burned-out Walmart at the edge of town?
I’m not down with Google swallowing everything posted on the internet to train their generative AI models.
This would mean a lot more if it happened before the wholesale harvesting of everyone’s work.
But I’m sure Google will put a mighty fine lock on that stable door that the horse bolted from.
Back when the web was young, it wasn’t yet clear what the rules were. Like, could you really just link to something without asking permission?
Then came some legal rulings to establish that, yes, on the web you can just link to anything without checking if it’s okay first.
What about search engines and directories? Technically they’re rifling through all the stuff we publish and reposting snippets of it. Is that okay?
Again, through some legal precedents—but mostly common agreement—everyone decided that on balance it was fine. After all, those snippets they publish are helping your site get traffic.
In short order, search came to rule the web. And Google came to rule search.
The mutually beneficial arrangement persisted uneasily. Despite Google’s search results pages getting worse and worse in recent years, the company’s huge market share of search means you generally want to be in their good books.
Google’s business model relies on us publishing web pages so that they can put ads around the search results linking to that content, and we rely on Google to send people to our websites by responding smartly to search queries.
That has now changed. Instead of responding to search queries by linking to the web pages we’ve made, Google is instead generating dodgy summaries rife with hallucina… lies (a psychic hotline, basically).
Google still benefits from us publishing web pages. We no longer benefit from Google slurping up those web pages.
With AI, tech has broken the web’s social contract:
Google has steadily been manoeuvring their search engine results to more and more replace the pages in the results.
Me, I just think it’s fuckin’ rude.
Google is a portal to the web. Google is an amazing tool for finding relevant websites to go to. That was useful when it was made, and it’s nothing but grown in usefulness. Google should be encouraging and fighting for the open web. But now they’re like, actually we’re just going to suck up your website, put it in a blender with all other websites, and spit out word smoothies for people instead of sending them to your website. Instead.
Ben proposes an update to robots.txt that would allow us to specify licensing information:
Robots.txt needs an update for the 2020s. Instead of just saying what content can be indexed, it should also grant rights.
Like crawl my site only to provide search results not train your LLM.
It’s a solid proposal. But Google has absolutely no incentive to implement it. They hold all the power.
Or do they?
There is still the nuclear option in robots.txt:
User-agent: Googlebot
Disallow: /
That’s what Vasilis is doing:
I have been looking for ways to not allow companies to use my stuff without asking, and so far I coulnd’t find any. But since this policy change I realised that there is a simple one: block google’s bots from visiting your website.
The general consensus is that this is nuts. “If you don’t appear in Google’s results, you might as well not be on the web!” is the common cry.
I’m not so sure. At least when it comes to personal websites, search isn’t how people get to your site. They get to your site from RSS, newsletters, links shared on social media or on Slack.
And isn’t it an uncomfortable feeling to think that there’s a third party service that you absolutely must appease? It’s the same kind of justification used by people who are still on Twitter even though it’s now a right-wing transphobic cesspit. “If I’m not on Twitter, I might as well not be on the web!”
The situation with Google reminds me of what Robin said about Twitter:
The speed with which Twitter recedes in your mind will shock you. Like a demon from a folktale, the kind that only gains power when you invite it into your home, the platform melts like mist when that invitation is rescinded.
We can rescind our invitation to Google.
Google has a serious AI problem. That problem isn’t “how to integrate AI into Google products?” That problem is “how to exclude AI-generated nonsense from Google products?”
AMP succeeded spectacularly. Then it failed. And to anyone looking for a reason not to trust the biggest company on the internet, AMP’s story contains all the evidence you’ll ever need.
This is a really good oral history of how AMP soured Google’s reputation.
Full disclosure: I’m briefly cited:
“When it suited them, it was open-source,” says Jeremy Keith, a web developer and a former member of AMP’s advisory council. “But whenever there were any questions about direction and control… it was Google’s.”
As an aside, this article contains a perfect description of the company cultures of Facebook, Apple, and Google:
“You meet with a Facebook person and you see in their eyes they’re psychotic,” says one media executive who’s dealt with all the major platforms. “The Apple person kind of listens but then does what it wants to do. The Google person honestly thinks what they’re doing is the best thing.”
Spot. On.
User research into deceptive design patterns:
Despite our technological advancement and understanding of human nature, it’s still common to see deception in digital products. To understand digital deception (and better define it), we surveyed 536 people to measure and discuss people’s understanding of this matter.
Google is a portal to the web. Google is an amazing tool for finding relevant websites to go to. That was useful when it was made, and it’s nothing but grown in usefulness. Google should be encouraging and fighting for the open web. But now they’re like, actually we’re just going to suck up your website, put it in a blender with all other websites, and spit out word smoothies for people instead of sending them to your website. Instead.
I concur with Chris’s assessment:
I just think it’s fuckin’ rude.