Techdirt removed from DuckDuckGo and Bing search results

Techdirt is one of the go-to places to get well informed about where technology and societal issues meet, about the often unforeseen repercussions of proposed legislation on tech issues and about tech policy in general. Founded by Mike Masnick in the 1990s, it has a frequently updated blog, a thriving community of people commenting every post and is also a podcast. I have been an avid reader and listener since the early years of the commercially available internet.

This morning I was trying to find an article on Techdirt, as I usually do to find references to texts I write on this blog. Being a privacy conscious person, I always use DuckDuckGo for my searches. To my surprise, I couldn’t find any search results at all from the techdirt.com domain name, even if I used the site: filter to restrict the search.

When searching for the site’s name, only results for pages about Techdirt on Wikipedia, Twitter, Patreon, Wired, etc. would show up, but still not any page on the techdirt.com domain name itself.

Strange, indeed. It’s not my first kind of experience like this with DDG, though. Months ago I had a really hard time finding an article my friend Yasodara Cordova had written in 2018 for The Intercept Brazil, related to her research in Berkman Klein Center for Internet & Society in Harvard, but which is also a very critical piece on Big Tech. That article now shows up easily in DDG, though.

Searching around a bit I found previous claims that DDG would remove results related to alleged piracy or alleged Russian propaganda, and they always do lay the blame on Bing. I couldn’t find, though, any case of DDG removing results related to critical thinking on tech policy. That seems indeed to be a first.

I tried to remember the names of other search engines that at least claim to be privacy oriented, so I could continue my search. Ecosia came to mind – they are featured as a search engine option in Firefox. Unfortunately, I found the very same problem there as DuckDuckGo had.

Then I remembered the fact that DuckDuckGo (and most likely Ecosia, too) gets its search results from Bing, and then tried searching there directly.

Still no results are displayed on Bing, but at least they do acknowledge the fact that “some” results have been removed. The link there does nothing to explain the reason why the whole site has been de-indexed, but is a rather generic Bing support page explaining several possible reasons that might cause Microsoft to remove pages from Bing search results.

Being busy and not able to write this blog post right away, I decided to inform Techdirt about the problem through the contact form on their site. Later in the day, I saw that Mike Masnick had written a blog post about the problem, mentioning having received a message from someone (yours truly) and basically speculating about the possible reasons for the site’s sudden disappearance from all those search engines. It even has a funny piece on asking Bing’s AI bot about the reasons behind the site’s removal.

Among the speculative reasons dreamed by the generative AI, the two most plausible seem to be either some kind of bug or technical issue, or censorship due to the political stance of Techdirt, which is very often critical of companies such as Microsoft, who many times have been accused of abusing their market dominance position to thwart the competition.

And the last one is of course possible: that Microsoft encountered “some legal or political pressure to remove or censor Techdirt.com, which is known for its critical and investigative reporting on various topics, such as technology, law, policy, and business,” but it would be nice if someone would just, you know, let me know?

Masnick explicitly claims to believe the former to be the case (i.e., a bug), and goes out of his way to remark the fact that Bing has every right to remove whatever they want for whatever reason (though it of course makes me and many people even less prone to use their search engine).

At no point did anyone at Bing let us know that we’ve been removed from the search index. And, of course, Bing has every right to kick us out of their index for whatever reason they want. But it does seem odd.

Someone commented at that post that other sites critical of the way Silicon Valley thinks of tech policy were supposedly being removed as well and cited as an example Tech Won’t Save Us (an excellent podcast by Paris Marx that I highly recommend, right beside others such as Your Undivided Attention, Team Human, The Cory Doctorow Podcast, EFF’s How to Fix the Internet, Reimagining the Internet and Techdirt itself). However, I searched for all of those on DDG and they were all in fact showing up as they normal would. So either they have somehow fixed really quick the rest of the websites, but not Techdirt, or this person is simply mistaken.

Later the user NoahVail pointed out in the comments that DDG’s CEO/Founder was looking into the problem and indeed, earlier in the evening I could see that at least a link to the main techdirt.com website was indeed back again appearing in DDG’s search results, but still not any links to other pages and articles.

I’m guessing that to show deeper links to Techdirt they would need to crawl the website, and that they’re entirely reliant on Bing for that. So it will probably take a while to fix.

In the meantime, I found out that another privacy oriented search engine is not having this problem, because they crawl the web themselves instead of relying on other larger search engines. It’s called Mojeek. For some reason, though, they have the annoying tendency of sometimes blocking my searches because they mistake my behavior for a bot, especially if I search for technical error messages, copying and pasting them (which is something extremely common that programmers do several times a day).

Update (2023-07-28): Bing, DuckDuckGo and Ecosia now show a few more results from Techdirt, mostly subdomains. But articles are still not indexed. For instance, a search for “site:techdirt.com drm” still comes up empty, even though DRM is a frequent topic in Techdirt articles.