YJL

I realized that Yahoo! Search BOSS API was gone last month, now Bing is going to shut down its Search API and asking developers to move on to Windows Azure Marketplace, which, unfortunately, is same as Yahoo's new API, not free. This is the email I just received:

Dear Bing API Developer:

For the past several years, the Bing Search API has made search data available for developers to innovate and build upon. Today we are announcing that the Bing Search API will transition to an offering made available on the Windows Azure Marketplace. The Windows Azure Marketplace is a one stop shop for cloud data, apps, and services, including the Microsoft Translator API. Through this platform, developers can access hundreds of data sets and APIs and distribute their applications through the marketplace.

A few important things to note regarding the upcoming transition:

• With the transition, Bing Search API developers will have access to fresher results, improved relevancy, and more opportunities to monetize their usage of the Search API. To offer these services at scale, we plan to move to a monthly subscription model. Developers can expect subscription pricing to start at approximately $40 (USD) per month for up to 20,000 queries each month.
• The transition will begin in several weeks and will take a few months to complete. Developers will be encouraged to try the Bing Search API for free on the Windows Azure Marketplace during the transition period, before we begin charging for the service.
• At this time, you can continue using Bing Search API 2.0 free of charge. After the transition period, Bing Search API 2.0 will no longer be available for free public use.

Details regarding the transition timeline, pricing structure, and other changes will be announced in upcoming weeks. In the meantime, we encourage you to explore the Windows Azure Marketplace and read the documentation. As a Bing Search API developer, you can expect the transition to involve targeting a new API end point, moderate changes to the request and response schemas, and a new security requirement to authenticate your application key. Developers using approximately 3 to 4 million queries and above can expect to transition through a separate process (details will be provided shortly).

We understand that many of you are using the API as an important element in your websites and applications, and we will continue to share details with you through the Bing Developer Blog as we approach the transition. We appreciate your patience during this time.

Bing Developer Team

If lucky, we will still have a few months of free Bing Search API.

Years ago, when you talked about an API, you just assumed that it's free for use. The only thing you need to care is not to violate the terms of use. To date, what you care are how much does it charge and what authentication model and library you should use.

APIs are getting more restrict, mind you it's not for security as the starting point, not entirely, but slightly rather for identifying the app which is accessing them.

One thing I really don't like is you have secret and/or key, then a hash. All is for the security. Even it's accessing public data, you may still be asked to sign your request. All I really want to do is

wget http://example.com/data

and process them in a shell script. But no, you have to go through a process in order to make sure everything is safe. It's harder and harder for shell scripts, even just to get a number via API.

I am sure in a near future, someone would ask "Anyone still remembers the HTTP Basic Authentication?" Maybe someday, OAuth will be supported by Wget or cURL.

Well, Bing Search isn't important to me, I only need a number from it and I can live without it.

Updated on 2012-05-20T06:26:05Z

I received another email and noticed this part:

For up to 5,000 queries per month, developers can access the API for free on the Windows Azure Marketplace. At this level, the large majority of our existing developers including non-profits, educational institutions, and smaller scale applications can continue using the service for free.

It sounds great for my usage of Bing Search since I only need a few API calls. However, after I signed up and got the new Account Key, I found out there is no data for total results from returned Atom (XML) or JSON format. In the end, I will still need to remove Bing Search, which will be gone on August 1, 2012.

Yesterday, I was looking into an old stuff I created using Google App Engine, which was evolved from a Bash script back in December, 2009. Here is a screenshot:

Yes, I shamelessly Google/Yahoo/Bing my username and yes, I unblushingly made a record for the number of search results with a chart. (Cough, two charts and I have this.)

When I opened that page, I noticed the records stopped on October 3, 2011. At first, I thought it might be the result fetching limit in my program, so I went to update the code, but I realized it was not like that.

Something went wrong when I saw a huge amount of unsuccessful tasks in task queue. Retry counts were around thousands. I looked into the logs and found out the problem was with Yahoo Search BOSS API. The domain of v1 API is gone.

So, I googled and found this announcement for v2 API, v1 was scheduled to be shut down on July 20, 2011, but it lasted until August 27. But then on September 20, it was back and lasted until October 3, then it is gone for sure since then.

Because my program requires three API returns successfully before it writes data into datastore. I should check tasks retry count, it is too high, then drop it and send me an email. But I didnt code it in that way, because I didnt think there is a chance of having the thousands of retries until yesterday. I lost about 6 months of data from Google and Bing. Will I add the email notification for something like this? Nah.

Yahoo Search BOSS v1 is not only one gone, soon Google Web Search API will be shut down around 2013. It was declared as deprecated before Yahoos, but it has longer transition time for developers, which is 3 years.

I think they both move to paid version of API, v2 and Custom Search. I dont use API for making money, so when they are gone, my program stops updating.

This is another not so useful script of mine. It is a Bash script and it gathers the search results counts via Google/Yahoo/Bing APIs to make an historical chart of specified keyword using the Annotated Timeline (with no annotations, :-))of Google Visualization API.

I made this script, search-result-count.sh, because I wanted to have a historical chart of this keyword livibetter. Yep, I searched my nickname regularly, I admit it! I like watching the number of result climbing up, which would make me feel better. :-D

I wasn't planning of using Yahoo and Bing because their APIs require AppIDs, the IDs will be out to apparently public if I use Bash to write this. I don't like it, but I couldn't resist to see the result counts from them.

Because it is still new, I could not have much data to show you. The following chart was collected about two weeks. (Bing results was not included)



The following is the screenshot of rendered HTML page:


Please aware of few things if you want to use this script:

• You can use cron to run it regularly, several times a day. Don't worry, it will only update the data file once a day.
• It will only update when three counts from three search engine are available. If any of them couldn't return the result, you may have a missing data. But it should be okay, the counts do not change much from day to day.