So, I've got a solid background in data structures and overall programming languages due to school. However, the time has come that they no longer teach much about programming and it's time to work on some stuff on my own.

I was thinking of starting a program that would for example:
A. Log in to x website
B. (Some sort of sales website) scan for x word at x price.
C. Populate picture, price, and description on my desktop app.

From there, I can add more features but this seems like enough to atart.

I'm pretty sure .net is what I need to perform this task? I've never used it so if anyone can point me in the right direction that'd be cool.

School has taught me how to use languages but no Web ones so I'm a little lost.

you'd have a hard time finding a language that can't do this

I guess my question is: what would be easiest to do this? I get that most have libraries to do so, but there has to be a better choice than others?

why don't you start by listing what languages you're comfortable with? if you're not comfortable with .net, then don't do it. you can do it in python, java, c, c++, etc.

keep in mind that depending on the website, it can be easier or more difficult. there are a lot of websites that already do price comparisons. it might be easier to let them do the heavy lifting, and then you can steal data from them. that way you're less susceptible when any given website changes their layout.

I know c++, c, java, and vb.net enough to get by. Not excellent at any of them but proficient.

I guess having them do the searching might be easier and then populate the calculated data into my app.

In all honesty, I always want to do something with my skills but I always talk myself out of my ideas for whatever reason. Just trying to sharpen some skills since school hasn't required much programming lately.

if that's your goal instead of a specific program, then i suggest avoiding web scraping. try interacting with a simple API instead to make some sort of app. for example, i personally hate youtube's web page, so i used the API to create my own. instead of seeing all my subscriptions on one page, i can see subscriptions by category. it's also incredibly painful to look at a channel's playlists if there are a lot of them since you can't sort by # videos, # views, alphabetical order, etc and you can't filter by that either. i haven't yet, but one of these days i'm gonna make my own screen to see playlists the way i want to. it's MUCH easier to use an API than to try and read from an html page.

And if you want to use cURL so you can login through websites and extract data, copy your login cookie data to their headers.

Any language with regex support can do web scraping although ruby/python with the nokogiri add ons will do it much better.

Good & simple DOM parsing PHP script. Might not be exactly what you're looking for but its pretty useful. I try to avoid regular expressions when scraping.

http://simplehtmldom.sourceforge.net/manual.htm

This post was edited by SelfTaught on Aug 5 2014 04:42pm