Overview
This project is a reusable Python-based web scraping toolkit originally developed to extract structured information from the Dwarf Fortress Wiki.
Unlike a one-off scraper designed for a single webpage, the system separates page discovery from content extraction, allowing different extraction strategies to be added without modifying the overall scraping workflow.
The scraper starts from a configurable MediaWiki page, discovers linked pages within a selected HTML container, filters and validates URLs, downloads each page, extracts required information, and exports the results into CSV format.
Current extraction strategies include sentence matching and extraction of embedded Dwarf Fortress RAW tokens. For example, the tool can collect creature preference descriptions or extract RAW metadata such as creature descriptions from hundreds of wiki pages automatically.
The project was designed with modularity in mind, separating link extraction, parsing logic, and command-line orchestration. This allows future extensions such as additional extraction modes, alternative export formats, retry mechanisms, and support for other MediaWiki-based websites.
The project demonstrates practical experience with Python automation, HTML parsing, regular expressions, command-line tools, and designing reusable data processing workflows.