The internet can be regarded as a huge expanse of data that is interlinked together for the purpose of facilitating interactive access. It gives users an opportunity to seek information that is of interest to them by use of web addresses and following hyperlinks. Recent research reveals that over 1 billion web pages are growing daily. Since the information found on the internet continues to grow daily, the harvesting of such information becomes a tough task that consumes time and resources. This is so since it is difficult to regulate the unstructured and semi-structured web data. It is important to note that the internet is very different from print documents. This because the information contained on the internet is constantly evolving. This has made database management a complex process and therefore calls for web mining tool known as web scraping.
Web scraping is about the use of data mining tools in order to discover and extract data from the internet. It is important to note that web scraping can be divided into four sub tasks:
• Resource Finding. This is usually the first step in the web scraping process. The purpose of this process is to retrieve data that is contained both online and offline sources. The information can be resources that can be found on the internet such as newsletters, website content and HTML documents.
• Information Selection. Also known as pre-processing is an integral step in the web scraping process. After extracting the relevant data from the internal it is important the original data is transformed. The process involves the removal of stop words, stemming or anything else in order to obtain the targeted data like finding phrases in the training corpus, representing the text in the first order logic form and so on.
• Generalization. This is another important step that is very crucial in the web scraping process. It involves the identification of general patterns and trends on the individual web pages and other multiple web pages. It usually calls for a lot of data mining techniques and other relevant web oriented methodologies.
• Analysis. This is usually the last step in the web scraping process. In this step all the extracted data and information is laid across, validated and all the patterns that were identified are now interpreted. This is a very crucial step as it would not make sense if we extract data and fail to interpret it for purposes of decision making and learning the marketing performance.