I want create a web crawler that can search the products from websites like amazon from their search engine, I mean I want to get the same benefit as the API have. As in India many websites don’t have APIs so I can’t rely on APIs.
You can use Python for web scraping.
Also possible to do this with other server-side scripting languages… C#, PHP, even the ancient Classic ASP can do this.
When I need to scrape a website I use php and curl, old school but it works.