Sometimes when web-scraping we care about persistent connection state. Here we created an instance of axios that will apply custom headers, timeout and proxy settings to every request! Tip: Automatic Cookie Tracking If you're ever unsure of a URL's structure, you can always fire up Node's interactive shell ( node in the terminal) and let it figure it out for you: $ node For example language=en would give us the English version of the resource. parameters - allows customizing of a resource.location - is the location of the resource we are requesting.host - is the address/domain of the server.Here, we can visualize each part of a URL: Though URLs can be quite complicated, let's take a look at how they are structured: URL (Universal Resource Location) is the most important part of our request - it tells where our nodejs scraper should look for the resources. DELETE requests are intended to delete a document. ![]() PUT requests are intended to either create a new document or update it.PATCH requests are intended to update a document.Other methods aren't as commonly encountered but it's good to be aware of them nevertheless: ![]() This is mostly used to check whether a web page has updated it's contents as these type of requests are super fast. For example, this is used in form submissions like login, search etc.
0 Comments
Leave a Reply. |