Ndtyjky

I am doing data science for sociology research, and am trying to scrape a website by emulating their XHR requests using Python (it is the sort of website that loads more results as you scroll). The next url in the sequence is embedded in the current XHR object code. You just capture this url with regular expressions, decode it, and slap your access token on the end to get the data you want.

After a certain number of iterations, the XHR request fails to return the next url in the sequence, creating a dead end. The urls themselves, however, appear to follow a numerical sequence, which I haven't quite figured out yet. If I could figure out the pattern, I could go back further.

What I am wondering is, are there any good tools out there for comparing differences in url like strings, particularly in Python? The urls are about 1000 characters long, and changes are very hard to eyeball. I need the right tools.

Here is an example similar to what the urls looks like (I can't provide the actual url, because it has my username, access tokens, and other private information), except much shorter and with many fewer changeable sequences:

https://website.com/results/unit=9382729282&unixtime=1541174174&more_variables=x

https://website.com/results/unit=9382729790&unixtime=1541181234&more_variables=x

Note: Someone did suggest to me that I use Selenium to simply scroll infinitely. Aside from the fact infinitely scrolling websites have infinitely growing source code that eventually choke Selenium, the browser JavaScript is requesting the same XHR objects as my Python script. They both stop working at the same place, consequently.