Webshop Product Analysis
Web scraper development
In this case study, I will guide you through the development of a web scraper and the accompanying challenges, solutions, and the successful outcome.
With this web scraper, I enable the client to automatically collect product data from various online stores. Previously, they did this manually.
Type
- Web scraper, data analysis
Role
- Software developer
- Maintenance
- CI/CD
Timeline
- 2024 - Present
Client
- External client
Challenges and goals
The client for this project is involved in data analysis of online products. They monitor specific products, particularly focusing on prices and review scores.
My goal for this project was to automate their manual work so that a new CSV feed with all the collected product data is available daily.
The most suitable programming language to use was Python. Since this was my first professional project where I had to apply Python, it became a bit more challenging for me. Despite that, I delivered a good product.
Architecture and technologies
As mentioned above, the web scraper was implemented in Python. This Python script takes an input CSV feed with products to be analyzed as an argument. The output CSV feed is available for download via a secured HTTP endpoint, using Caddy 2.
The entire system runs on an Ubuntu server. With the help of cron jobs on the server, the Python script is automatically executed at set times.
Development process
During this project, I worked iteratively. During interim meetings, I was able to incorporate changing requirements and feedback.
Since the deployment of the first version, I have continuously implemented updates and performed maintenance.
Testing
As always, I unit test all code. I used the built-in Python unittest & mock libraries for this.
