Is my final year project feasible?

Nemesis379 · 21 Oct 2013 at 12:18

My final year project is to create a price comparison site for tons of products, using sites such as Amazon, Overclockers, etc.

Using HTML & CSS for the website, and either PHP or some kind of script for grabbing the products, I'm not entire sure how I'm going to do this, people have said that I could contact the companies and they could provide me with product info to be used on my website if it's created.

I was thinking of doing the website two ways:

Having a kind of Google search engine, whereby you search an item, and it searches the websites and provides the best price based on the price of the item inc VAT and shipping costs.

Or have a website whereby everything is displayed kind of like amazon. Perhaps even having the items stored?, I mean how else am I going to have multiple items from different categories unless I store items 1 by 1 which would take forever.

I only have 10 days to think of a final year project and write it up so any help would be greatly appreciated.

Nymins · 21 Oct 2013 at 12:43

I would have thought most sites stored some form of XML data that you could use (if they allow it to be available). IMDb for example can do this, people have built scrapers using their XML database.

This would allow you to grab all the item names, descriptions, prices etc, but its up to you to format it in the correct way.

AHarvey · 21 Oct 2013 at 12:46

That sounds like a massive undertaking, and no offence, if you're not sure how you'd do it already I wouldn't try it.

Better to do something small and well, than over reach and fail.

If you must do comparisons look for a smaller product market, something like computer games. Enter the games name and it comes back with places to buy them + reviews on the games. Maybe do this with sites like Quidco that offer cashback on purchases to see which one you should use.

Daft question, but what course are you doing? what language skills/experience have you already got? might help people with suggestions

Nemesis379 · 21 Oct 2013 at 13:15

Computer Science Networking, I have experience with HTML/CSS/PHP/C++ & some Java. I mean if there's something I don't know, I do have a willingness to learn for this project.

Hyburnate · 21 Oct 2013 at 13:19

It should be possible as there are sites out there that do this.

MrClark · 21 Oct 2013 at 16:03

Nemesis379 said:
Computer Science Networking, I have experience with HTML/CSS/PHP/C++ & some Java. I mean if there's something I don't know, I do have a willingness to learn for this project.

If you do decide to go with this look into using Perl or Python for the scarping.

GravyMonster · 21 Oct 2013 at 16:12

As said this is definitely not easy. The companies are very unlikely to give you open access to their APIs - even assuming they have one. This leaves you the option of scraping which is a PITA to write and maintain.

It's certainly do-able though.

uniQ · 21 Oct 2013 at 16:33

I would have thought your first port of call on whether or not your project is feasible is to talk to one or more of your tutors?

Price comparison sites that provide the data would imo be way to simple for a FYP. You could just sign up to something like affiliate window/trade doubler etc and download data feeds in various formats.

dfarrall · 21 Oct 2013 at 17:44

It sounds like a great project, the stumbling block will be based on which retailers you use and how the site is layed out!

Would definitely talk to your tutors and make sure they are OK with it and if you do come up with issues pulling down information that your not going to fail horrendously.

D.P. · 21 Oct 2013 at 18:10

Nemesis379 said:
Computer Science Networking, I have experience with HTML/CSS/PHP/C++ & some Java. I mean if there's something I don't know, I do have a willingness to learn for this project.

The first thing to do would be to ask potential supervisors what kind of project will be well regarded at your institute. On my course anything where UI was an important aspect was deeply frowned upon (except if it was somehow resolving a complex problem) and basic web pages would lead to failure ultimately. Not that UI or web interfaces were not possible, but something much deeper needs to be going on in the background an the UIside of things were Effectively ignored from a grading PoV. Obviously different if you studied we development or such like but your degree is computer science networking, I would be looking for some thing scientific!

Our projects were much more focused on algorithms and data structures and developing novel approaches to known problems. Your project sounds like a lot of work but ultimately is just a lot of bread and utter programming- connecting to sites, parsing different data formats, storing data and then making a basic website out of it.
Questions I ask y students are:
What are the algorithms or methods that you would need to develop to resolve the issues, and why hasn't anyone done that work before? What would be the novelty that you have added? What is the scientific question or hypothesis you are hoping to answer? What information do you wish to provide to the scientific community? Hw will you test your contribution is valid, how will you analyze the results, what will be your testing methodology, what statistical techniques would work for the data you expect to generate?

Obviously at an undergrad stage I was never expecting too much, but to get the student to start thinking a long the lines of suitable projects that are verifiable.

During my PhD I supervised over a dozen undergrad and masters projects, a few students worked supported later publications some of whom went on to do a PhD themselves. Where I supervised the project supervisors gave a big input into helping mould a sensible project with realistic timelines and an experimental hypothesis or scientific objective.

Nemesis379 · 21 Oct 2013 at 18:36

Yeah, I will be talking to my supervisor this week to see what he says.

edwardcasbon1 · 21 Oct 2013 at 18:49

If it's a Computer Science Networking course shouldn't the final year project be based more around networking than software programming?

Could you do something around cloud computing? Creating your own cloud service? How about combining the computers over a network and doing something with them? Write a chat client (that can extend lots of different chat protocols)?

Also, 10 days left to decide on final year project? You've probably had months to think about this right?

Good luck on whatever you decide to do.

gerico · 21 Oct 2013 at 19:10

Personally that's a massive undertaking, and usually teams of developers would be charged to build something like this. It can be massively complex, i wouldn't attempt it, but thats just my opinion

Nemesis379 · 21 Oct 2013 at 22:20

Something to do with cloud computing could be a good idea, hm...

Th3M8dH8tt3r · 22 Oct 2013 at 02:08

Guessing you've got the rest of the year to build this but even so, to build a properly functioning site that gets data from numerous sites, I wouldn't want to take it on for a "project", unless I was been paid.

It's the sort of project that will be very difficult to remain motivated throughout without the incentive of earning big bucks.

If you're thinking that it could be a good money maker after the year, sure but...do it when you've finished and focus on a project that you can excel in.

Just my opinion if I was in your shoes.

!bluetonic! · 23 Oct 2013 at 13:56

You could easily doing this project by having 'data-stubs' from different providers.

In other words a created-by-you-psuedo-XML file that 'they' would have generated from their DB, which gave you access to their DB and products.

Or a a variety of stubbed data sources such as RSS, XML, JSON, API calls etc that show a good range of skills collating that information to one website for comparison.

sunama · 29 Oct 2013 at 19:05

1. Get your tutor's opinion.
2. Find out how your project will be graded. When I did my project, my mark was based on my write-up, not the actual working code.

If your project is being marked based on the write-up, then you won't have to complete a working website/code. You will simply need to make a start. Perhaps you can show a working prototype, which crawls amazon and another popular website. If you can show that your website can compare products on 2 websites, then it can be done for X websites.

For sure, in 1 year, you will not be able to have a working product completed BUT you may not have to.

This is actually a good idea, simply because after your project is done and after Uni, you can create a real price comparison website, based on what you have learned and become a millionaire

PS. There will always be people who tell you that "you can't do this", "you're not good enough" (like someone in this thread already has). Don't ever listen to them. If you believe you are good enough and your tutor thinks you can do this, then do it. Set your goals high, work hard and there is good chance you will be successful.

Good luck dude.

Redrum · 29 Oct 2013 at 22:10

If you do this, can I suggest that instead of trying to scrape data off sites, you look into the available APIs. The rainforest has an API that lets you search for products then display that information on your site, and Google has an API that lets you search the data that e-commerce sites upload to them. I think the auction site also has an API. Those three sources between them probably cover three quarters of what's available online and will contain much much more than you'd be able to site scape in a one year project. You could then add some complexity by just selecting a couple of sites that are not in that API and writing something to scrape those.

The hardest part is probably matching the data up from all these sources because you won't necessarily have barcodes and different sellers will call the same items by slightly different names.

D.P. · 30 Oct 2013 at 12:52

The hardest part is not going to be the programming at all, but defining an experimental hypothesis and designing a scientific methodology to test the validity of that hypothesis.

The final year project of a good computer science degree should be a scientific project, the clue is in the name of the degree, "science". What will the hypothesis be, how will you test it, what will be the data or results, how will you analyse the results to know if you have invalided the hypothesis or not? Where is the innovation, what scientific question will you be helping to answer for the scientific community?

Those are the kinds of questions I would be concerned about at the start because if you don't define a realistic scientific project with defensible goals none of the programming will matter at all.

In fact, you may not have to program anything for your project if your hypothesis can be tested in other ways.a couple of more mathematically orientated computer science friends didn't write a single line of code for their final year project because they analysed existing new processes to understand their expected complexity on certain datasets with defined statistical properties.

AJK · 30 Oct 2013 at 13:01

D.P. said:
The hardest part is not going to be the programming at all, but defining an experimental hypothesis and designing a scientific methodology to test the validity of that hypothesis.

The final year project of a good computer science degree should be a scientific project, the clue is in the name of the degree, "science". What will the hypothesis be, how will you test it, what will be the data or results, how will you analyse the results to know if you have invalided the hypothesis or not? Where is the innovation, what scientific question will you be helping to answer for the scientific community?

Not that I'm necessarily disagreeing, but for many (most?) undergraduate CS degrees, the kind of scientific approach that you describe isn't a requirement; a properly documented development project is perfectly acceptable.