Web Scraping and Analytics Solution for Automated Tenant Screening

Summary

A US tenant screening company manually checked public government databases to gather and update tenants' criminal-history data — a slow approach that couldn't guarantee timely detection of changes. The company wanted to automate this with web scraping but needed expert consulting on the technical and legal aspects. In two days, INNERLUXES delivered a high-level architecture for a web scraping solution with analytics and reporting modules, plus guidance on the legal and ethical considerations, helping the Client make an informed decision on implementation.

About the Client

The Client is a US tenant screening company that delivers applicant reliability reports to rental service providers.

Web scraping needed to automate tenant background checks

The Client manually screened publicly available government databases to gather and update data on tenants' criminal history. That approach was time-consuming and did not guarantee timely detection of changes in tenant background. The Client had an idea to automate the process with a web scraping system but needed expert consulting on both the technical aspects and the legalities of implementing such a solution. The company turned to INNERLUXES, trusting our deep experience in data management and analytics services and our knowledge of data protection regulations.

Designing a web scraping solution architecture with an analytics module

INNERLUXES held several interviews with the Client's tenant-data screening team to elicit their requirements for the web scraping solution. Our team studied the screened databases, the retrieved data, and how the Client's team uses it for reporting (e.g., preparing summaries of tenant criminal history, creating risk assessment reports).

We concluded that the Client needed not just a web scraping tool but an analytics solution that would cleanse and interpret the scraped data — automating both data gathering and reporting, the Client's main goal. The Client asked INNERLUXES to create a high-level architecture of the analytics solution to help stakeholders understand the overall functionality and benefits and make a final decision. In two days, our team delivered an architecture design supporting the following processes and data flows:

Data crawling and parsing

Each government website is screened by dedicated crawlers and parsers designed to navigate that specific database. The crawler maps out the database structure, and the parser extracts the relevant information (e.g., tenant name and demographics, committed offenses). If a page contains links to other useful pages, those pages are opened and parsed like the parent page.

Data cleansing and upload to raw data storage

The scraped data is cleansed and standardized (e.g., unifying date formats, removing duplicate entries, transforming offense types into numerical codes) and lands in raw data storage, which also features a metadata catalog that keeps additional information such as the source and date of scraping.

Data analytics and reporting

The cleaned data is further transformed to fit the chosen analytics model — for instance, grouping offenses into crime types or merging related fields (criminal sentence start and end dates merged into a sentence-duration column). The highly structured data is stored in a data warehouse (DWH) optimized for automated reporting and ad-hoc analytics queries.

Error reporting

The solution captures error events at every step of the data processing flow and creates error reports to enable timely identification and remediation of issues.

Data orchestration

All these processes are scheduled and automated with a data orchestration tool. INNERLUXES recommended Apache Airflow for its diverse functionality and cost-efficiency.

Consulting on legal and ethical aspects of web scraping

Our experts provided the following recommendations for ensuring the legality and ethical use of the solution:

• Review the terms of service of the screened websites to make sure they do not explicitly prohibit scraping.
• Review the robots.txt file of the government database websites to see which parts can be accessed by web crawlers.
• Implement rate limiting to avoid overloading the scraped websites' servers and prevent service disruption.
• Keep logs of data scraping activities to facilitate future compliance audits.
• Determine which federal and state personal data protection regulations the solution must comply with — the final list depends on the residency of the screened individuals (e.g., FCRA at the federal level, plus applicable state privacy laws).
• Ensure compliance with the defined regulations (e.g., providing data collection notices for rental companies to distribute to screened applicants; creating and publishing a comprehensive data collection and usage policy).

Web scraping and analytics solution architecture ready in 2 days

• In two days, the Client received an architecture for a web scraping solution with analytics and reporting modules that will let it automatically scrape tenant criminal-history data from government databases and report it to rental service providers.
• INNERLUXES also advised the Client on the legal and ethical concerns for the solution.
• Thanks to INNERLUXES's assistance, the Client could assess the feasibility of web scraping for its business and make an informed decision on implementation.

Technologies and Tools

Apache Airflow.