it's probably tricky to programmatically determine what a nasty comment is
I've actually been working on that problem with a bot that assists in moderating a subreddit using a text classifier. It's tricky, and needs more work, but it is not impossible.
it's not that hard. Those that work with classifiers, this kind of thing is pretty easy. Identifying sarcasm and irony are hard, but 'nasty comments' can be identified pretty simply using the well known text classifier algorithms. You find the training data and use it to train something like an SVM.
As you might expect, a subreddit about a politician with (in)famously devoted followers attracts its share of strife. It can be difficult to distinguish legitimate arguments from flamebait, and there's no shortage of people eager to take any bait offered. I should note that I'm not actively running the moderation bot at the moment.
I'm sure a bot that algorithmically classifies mean comments is possible. I'm not sure that the same can be said about trolling. Poe's law? Deep cover trolling?
When one of the criteria of trolling is the hidden intent of the person writing, then there's no physical process that can reliably find a trolling, short of looking inside their head.
A well-executed troll is, by definition difficult for humans to detect.I don't think there's much chance of reliably doing it with software. Fortunately, most political squabbling on reddit consists simply of people expressing scorn or outrage that someone would post something on the internet that disagrees with their deeply-held beliefs. That's a bit easier to detect.
I plan to. I've been doing a lot of work with text classification over the past couple years and would like to base a startup on it. I just need to come up with a product that's commercially viable and non-evil.
I've actually been working on that problem with a bot that assists in moderating a subreddit using a text classifier. It's tricky, and needs more work, but it is not impossible.