d2jsp
Log InRegister
d2jsp Forums > Off-Topic > Computers & IT > Programming & Development > Webscraping With Spyder
Add Reply New Topic New Poll
Member
Posts: 23
Joined: Nov 7 2018
Gold: 5.00
Nov 10 2018 12:10pm
I generally have the idea to use selenium but I was told I could use another that starts with M, can't remember.
Anyhow, does anyone have good idea to do this?

Mainly to scrape market news from multiple websites.
Thank you.
Member
Posts: 10,942
Joined: Dec 25 2009
Gold: 7,540.14
Nov 10 2018 04:45pm
I always figured beautiful soup was very good for scraping. I tabbled with it using python but I have no real experience with it.
Member
Posts: 23
Joined: Nov 7 2018
Gold: 5.00
Nov 11 2018 12:49am
Quote (harleystcool77 @ Nov 11 2018 06:45am)
I always figured beautiful soup was very good for scraping. I tabbled with it using python but I have no real experience with it.



Thanks, that was the alternate to selenium I think?
Still, there is one more that starts with M, .. can't seem to recall that library.

Banned
Posts: 1,810
Joined: Jan 18 2009
Gold: 0.00
Warn: 20%
Nov 15 2018 05:49am
Quote (Draynich @ Nov 10 2018 11:49pm)
Thanks, that was the alternate to selenium I think?
Still, there is one more that starts with M, .. can't seem to recall that library.


https://www.digitalocean.com/community/tutorials/how-to-scrape-web-pages-with-beautiful-soup-and-python-3

Selenium is far better for javascript parsing. I don't think Beautiful soup can even follow javascript href links etc. But Selenium can. That goes without saying it's harder to get going perfectly as it does more.
Banned
Posts: 1,810
Joined: Jan 18 2009
Gold: 0.00
Warn: 20%
Nov 15 2018 05:51am
Code
var js = (IJavaScriptExecutor)driver;
js.ExecuteScript("$j(\"div[id='adHocAddDocDiv']\").click();");


apparently like this or similar
Member
Posts: 15,316
Joined: Aug 1 2008
Gold: 1.01
Warn: 100%
Nov 15 2018 05:56am
Quote (2Cents @ Nov 15 2018 11:51am)
Code
var js = (IJavaScriptExecutor)driver;
js.ExecuteScript("$j(\"div[id='adHocAddDocDiv']\").click();");


apparently like this or similar


please dont listen to 2cents, he dont know wtf he talking bout
Go Back To Programming & Development Topic List
Add Reply New Topic New Poll