Big Data Mining, Storing, and Processing

C#, MYSQL, PHP, Python, Selenium, Artificial Intelligence

Solo Project
Date: 4th March 2016 - 4th August 2016
Software: C#, MYSQL, PHP, Python, Selenium

For this big data mining project, I used a variety of programming languages and selenium to scrape and save over 100,000 real estate properties to multiple MYSQL databases. The big data mining program I created utilized selenium, JSON data, PHP curl, Python, XML data, and artificial intelligence.
Through the course of this project, I constructed an intelligent web scraping program and system. The goal of the project was to create a web scraper that could handle large amounts of data and have Artificial intelligence algorithms to automate complex data mining tasks. After months of working on the system, I was able to create a web scrapping program that had the ability to login to multiple web pages, check for elements on web pages, determine the amount of pages in pagination dynamically, and insert the data into different databases according to specific keywords.

Notable Accomplishments

  • Created a web scrapping system that was intelligent and could handle big data
  • Learned about Python, JSON data handling, PHP Curl, Selenium, and AI algorithms
  • Developed a system to stream line big data web scrapping