I'm interested to see if I can use machine learning/network analysis methods to automatically detect automatically generated (spam) webpages. I'm particularly interested in webpages that look structurally like a non-spam website, but on closer inspection are total rubbish.
If I want to test a method, I'd need some way of accessing (or generating) these webpages.
Question: Where can I access automatically generated (spam) webpages?
Here's a sample of the kind of webpage I'm thinking about:
Question: Where can I access automatically generated (spam) webpages?
You could use:
a parody generator program.
They're usually based on Markov chains (e.g. SubredditSimulator) or context-free grammars (e.g. the well known SCIGen... according to the authors "with Markov chains you might get something syntactically correct, but it is likely to be boring"!).
Also a cut-up technique could be enough.
a (web spam) dataset. The WEBSPAM-UK2007 collection is one of the largest and cleanest (and it's open). A description of the dataset can be found in A Reference Collection for Web Spam (the paper describes WEBSPAM-UK2006 but it's still valid).
You should check if it's too broad: the collections includes many different web spam aspects as possible (pages that are only advertising, pages with unrelated links, pages that automatically redirect...) and you seems only interested in pages that contain machine generated content.
the spam messages in the email: email spammers often include URLs in their spam messages. Following those links you can store the HTML content. You should add a few heuristics to identify some false positives (e.g. exclude Alexa's Top 500 list and use SiteAdvisor's rating system to compile a whitelist).
This is the method used to create another well known dataset: Webb Spam Corpus (the original corpus was collected by Steve Webb). Further details are contained in Introducing the Webb Spam Corpus: Using Email Spam to Identify Web Spam Automatically (Steve Webb, Jamesa Caverlee, Calton Pu).
