Clay (Free) to scrape and clean paginated data
Before we begin
Manually scraping data from large, paginated databases takes forever. And many web scrapers simply can’t navigate pagination easily to do it automatically.
Good news: this Claybook uses Zenrow and Clay’s custom formulas to scrape paginated databases in seconds.
How it works:
- Create thousands of paginated database URLs using Clay formulas, automatically
- Use Zenrows (natively integrated in Clay) to scrape each page, one by one
- Use scraped data for enrichment, personalization, outbound, and more
Steps
Copy this template
First, copy this template in Clay.
If you’re already a user, you’ll be directed to your workspace. If not, you’ll be prompted to create a free Clay account.
The interactive demos on the right will walk you through each step of this Claybook.
Step 2:
Add paginated page numbers
How many pages exist in the paginated database you want to scrape? Add them to the first column, giving each page its own row. We’ll use these page numbers in the next step to create dynamic URLs.
Step 3:
Create pagination URLs using Clay formulas
Next, so you don’t have to manually paste URLs into Clay, we’ll use a formula to add the page number to the database URL string automatically. Simply paste the URL of the database in the formula and insert the page number cell in the proper spot.
Once finished, run the Zenrow data scrape column.
Step 4:
Edit AI data cleaning prompt to fit your needs
Next, to ensure we’re pulling usable data from often complex databases, use ChatGPT to clean the dataset.
Step 5:
Write data to another table for further enrichment
Last, write your newly scraped database data to a new table. Why? We want to get all the datapoints into their own columns. The only way to do that is to write to a new table. In the new table, you can use this data for anything, like further enrichment or for outbound messaging.
Copy the template
Start scraping paginated databases in seconds.
Simply copy this template, follow the steps above, and let Clay work its magic.
Need help?
If you have any issue, you can ask for help in Slack or schedule time with a Clay expert.
Learn with our resources
Check out Clay University, read our GTM blog, or try out our top templates to transform your growth ideas into outreach in minutes.
Clay University
Master Clay in a few core lessons. Uplevel your data enrichment, craft better outbound messages, and book more meetings.
Go to market blog
Our best content on growing your GTM motion, from outbound sales automation to AI lead generation.
Integrations
Browse through all the integrations you can connect to Clay in seconds and discover what you can do!
Clay Experts
Hire a Clay Enterprise expert or find a Clay partner to help with your next project
Clay's Chrome extension
Clay’s Chrome Extension makes it wicked easy to find leads almost anywhere on the internet