The Fiction Genome Project:

So I've stumbled upon a thing called the Music Genome Project, while reading a book on self-improvement that incidentally commented on artistic and literary creation:

This writer and businessman explains why is is very skeptical of the idea that only geniuses can come up with interesting ideas, suggests methodical approaches to help anyone at least contribute something worthwhile to the pool of whatever their chosen field is, and, using literature as an illutsrative example, writes this remarkably troper-worthy segment:

So what's this Genome Project then? I went to check it out. I almost had a heart attack of pure nerdasm:

Ooooh. Ooooooh. I... this brings tears to my eyes... Have we done this? I mean, we've been troping and troping and so on, but have we tried to statistically analyze the works, mathematize them? Make ponderated averages based on overall popularity, controversy, critical reception, and so on? Wouldn't that be a worthy project for this wiki? Could it be that academic dignity it literally at our doorstep? (the proper word would have been "finally" or "actually", but I'll keep it there because it works even better as a Take That Me... and because of the responses later on in the thread, also, "academic integrity is at our doorstep") Could it be that actual money could be made from such a project?

edited 27th Jun '12 6:57:20 AM by TheHandle

Since when do we have a literal doorstep?

Did you ever read Roald Dahl's "The Great Automatic Grammatizator"?

No, I haven't, but the title seems like an Author Tract all by itself. You seem to be under the impression that this is some sort of automatic fiction generator. You're wrong. That site (Pandora) is only a tool to help people find music that's similar to stuff they already like. Which this site already does. But it certainly could [be used to] do it better, faster, and more accurately.

As for a calculated, marketing-based approach to writing in general and to the selection, timing and dosing of tropes in particular, it's hardly unheard-of. Any editor worth his salt understands that. A mathematically-powered tool would only help make the process more efficient, mostly by being more comprehensive and playing with data in more sophisticated ways than "the polls fell this week, what did we do wrong?". A tool is a tool. It doesn't write for you, the same way "Virtual DJ" doesn't mix for you, and LATEX doesn't decide the layout for you. But it helps, and that's worth money.

Since when were you under the impression that I implied we had one?

edited 26th Jun '12 8:20:22 AM by TheHandle

"academic integrity it (sic) literally at our doorstep"

"You Keep Using That Word. I do not think it means what you think it means."

Ah, you caught me there. I misused the word. Now I get the other joke.

I'm depressed that all the responses I've gotten focused on my bad writing and on dismissing the project on the basis of fictional evidence from the last century. Maybe I should rewrite the introduction to make it look more serious? I really thought I'd get a much more enthusiastic response than this.

Anyway, I've actually gone and asked my teachers at college. It can be done, but it'll take lots of work, and free software would be the way to go if I make it a student project. The service might still be for profit, since it's one thing to write a software and another entirely to release a database for free, but that would depend on how complicated it is to build a database from TVT's contents. I'd also need to check what the "creative commons license" entails in detail.

There's about thirty thousand tropes, and about as many works pages. That's a huge amount of work to punch in. Perhaps there's a way of getting around that: can we have a tool that would build a vector from a works page, counting whether or not a trope occurs? Does such a thing already exist?

Luckily, when it comes to discovering new tropes, I think we've pretty much stabilized. The Music Genome Project only uses about four to five hundred "genes" in music. I'm not sure how one could select four hundred crucial tropes, as far as audience reactions go.

Have I made it clear that I am entirely serious about this, and that feasibility is almost certain? Have I done a good job at convincing you that what is of essence, now, is figuring out how to do it the optimal way?

There's, of course, ways to sophisticate this site even further. Such as giving the users the ability to give every trope used in a work a punctuation. Example: on a scale of one-to-ten, how well was the trope executed? How crucial was it to the plot? To the work's identity? You could break down the first further: how good was the timing? did it feel clichéd?

As a whole, you could get to grade a work on how sad, happy, angry or scared it made you feel, or how much it made you laugh. How challenging you found it, and how much you thought the payoff was worth it.

The main difficulty is gathering and organizing the data. From there, writing software that can turn the data into information is a lot more mundane.

I can't see a way this isn't win/win. Users get a new, shiny, powerful toy that will help them find fiction they like, writers and their editors will find useful data on what the public loves and hates (at least the bit of the public that will bother to use this app). Publishers get market polls done for them for free (or for very cheap), and perhaps even might take to using the site to shill their products.

And, yes, writing is a bit like cooking: it isn't enough to just blend the ingredients together to do something decent, and it isn't enough to follow even a detailed recipe if you want to do something great,. But it's still useful to know what ingredients people respond well to and what actual dishes are their favorites. Wouldn't you agree?

edited 27th Jun '12 6:56:23 AM by TheHandle

In theory, I do think this is a fascinating idea. That said, I think there's a huge practical issue with applying it across various entertainment genres.

Quite simply, it's much easier and quicker to pull this off with music (where you spend two to five minutes with a given piece to start getting opinions on different parts to it) than with a television show (at least 22 minutes) or anything longer.

Hmm. There are 30,000 or so tropes, and between cut tropes and new ones getting through via YKTTW, we can say that we're probably adding 1 trope per week at this point - in other words, pretty stable. But I think the crucial component would be to find out which tropes most closely tie in to plot and characterization, because tropes like... Teaser Equipment or Impact Silhouette may show up in works, but they're probably not the sort of thing you would consider when recommending a game or TV show or what have you.

It would also be worthwhile to separate tropes by medium. Some tropes are unique to video games, some are unique to anime, etc. Perhaps the best place to start are the Tropes of Legend, and work outward from there. Maybe check tropes that have at least... say, 1,000 wicks. It's a classic problem in signal/noise issues and image compression - you're going to lose some amount of signal by reducing the problem to something mathematically solvable, so the key is to lose as little signal as possible given a maximum size of the problem.

edited 27th Jun '12 10:08:08 AM by DarkConfidant

This is where the collective nature of the work shines. While the Music Genome Project must hire musicians to "trope" their songs for them, we've got masses of editors who do this for their own enjoyment. What worries me is the sample bias: the sort of people who would know this wiki, and edit it, aren't a very good focus group. If TVT can somehow be reshaped into an application like Spotify or Grooveshark or Encarta or Facebook, and make it fun and easy to interact with (starting with exempting prospective editors from absolutely awful text interface in favor of something more user-friendly), I think we could find ourselves attracting much broader audiences, and obtaining more valuable data.

I like how you phrased that. But genres are a blurry thing, so shouldn't we treat them as "clusters" of tropes that often happen together, rather than "sets" that fit a specific definition?

Just to make sure, everyone, I'll insist that I am completely and utterly serious about this. At this point, I only know some advanced calculus, algebra, and statistics, and some computer control, but I know jack shit about data processing, and I can barely code in C and Latex. What books should I start with? There's a lot of unknown unknowns for me here...

edited 27th Jun '12 10:28:27 AM by TheHandle

Well, yes. Sure. Some tropes, plot devices, and the like are going to go hand in hand. But even more broadly, you want to make sure that you're also differentiating on the basis of genre - I don't want a recommendation for StarTropics if I'm looking for a movie, for instance.

That's not a genre, that's a medium... And it should be customizable. After all, someone who loves courtroom dramas (a genre) but is unfamiliar with video games (a medium) might have never thought to ask whether there were any in that genre, and thus would have missed the complete gem that is the Ace Attorney series. Or maybe someone who really liked Lord of War and The Expendables would love Jormungard, and they have never watched an anime in their lives.

So... the AI so should be able to learn to recognize a user's preferences, offer suggestions, recalibrate its assessments from what the user accepted or rejected, from what the user has already consumed, and so that it also tries to explore a little by suggesting every now and then to the user things that are outside their comfort zone but have many traits that they like.

edited 27th Jun '12 10:37:27 AM by TheHandle

Sometimes, the correct word escapes me. But yes, medium would be the word I am looking for.

Medium has a much stricter definition than genre, that's for sure. Though, of course, some franchises sprawl across many media, but usually the quality material is limited to one of them.

We already have a tool that can return all tropes that two work pages have in common. If a tool could do that en masse, it might be able to return a short list of pages that share the most tropes with a given page.

This might be too server-intensive to be workable, though. I guess it also runs into problems with pages like The Dresden Files that are split for length.

Obviously we'd need to think this through carefully, especially in the contingency that we may actually meet success. Is the change economically sustainable? That is to say, would the extra revenue from the application justify the investment in a bigger server to support it? Would said bigger server be an opportunity for improving the wiki in general, by adding extra functions that the older one couldn't support? And, dare I be optimistic to the point of blasphemy, could the resulting virtuous circle generate enough revenue to have paid programmers and moderators?

This is basically how I saw the wiki to begin with. I find that it's a much more reliable system than getting good/bad opinions.

@The Handle: I wouldn't dismiss the value of medium preferences as lightly as it seems like you are.

Using your particular examples, for instance, I have never really watched - and will never watch - any anime, because the particular stylization it uses quite frankly gives me the creeps. A friend of mine, meanwhile, dislikes playing video games because he doesn't feel that he should have to work to get a story. And these are just two specific examples; in general I think it's safe to say that it should be entirely possible for people using this thing to create their own comfort zones and not be prodded outside them if they don't want to be.

edited 27th Jun '12 5:03:20 PM by nrjxll

Oh, that's easy, you just make a section as in "Try something different!" which is the only place the app will suggest to you stuff that's outside your comfort zone. I suppose, besides learning by trial and error, you could also set some of the preferences yourself, explicitly: "I definitely don't want any anime" "I don't want stories that feature guns" or "I don't want stories where the author uses a Red Herring". There's probably an easy way to tell the machine these things.

Yes, but it requires that you know what the tropes are in order to be effective. This app would spare you from the effort, though I'm certain many will eventually find themselves interested in those little tags.

I love the idea, but we'd need either an enormous amount of capital and hundreds of people at my estimated minimum who would be willing to work for minimum wage and/or unpaid overtime doing nothing but reading, watching, rereading, rewatching, and classifying media... Sorry for a moment there I had to wide the drool off my chin, and I'm only slightly exaggerating. But back to the serious note, this would take a much more dedicated worker-base than TV Tropes, though probably a smaller one, and the worker-base would need to be accurately trained to understand the definitions the system uses before ever being allowed to touch a genre or medium entry tab.

I would love doing boring stuff like that, because I don't find it boring (for frame of reference, I proofread published books for fun and literal profit), but we'd need:

• a heck of a lot more people like me (but preferably more social) (hereafter referred to as "scribers")
• a few additional people to handle the business, accounting, HR, and legal side of things (hereafter referred to as "Suits", plural, capital S, refered to as "Legal", capital L, if I feel like mentioning the lawyers specifically.)
• still more people (with more social skills than me, though I could do that early on and during any potential short-term emergencies) to work on the training, who would be willing to meet up with each other to have a meeting of seemingly absurd length every time even a minor change is made to the system (Hereafter referred to as "trainers")
• People dedicated to running the hardware side of things, such as servers and memory banks (hereafter referred to as "hardware monkeys", no stigma intended. I could edit that to, like, "binders" to fit with the "scribers" literary theme, or some other suggestion. It's only to make the things I type here more concisely clear.)
• More people dedicated to coding (most of whom would be covered under scribers, Suits, and maybe hardware monkeys, but some of whom would need to be full-time coders.)
  ...Each group of which would overlap with the other groups, since trainers would need to know the system as well as scribers or better, and hardware monkeys would need to be able to discuss with Suits the cost-benefit analysis of server upgrades, hard disk replacement, even fan replacement), just for a couple of examples.

Code could be run to make sure anything that is affected by a change gets put behind a curtain and flagged for updates, and requiring a manual deflag for the user-side system to notice it, but that would still require a lot of man-hours to run updates alone, so we would need to have the framework very planned out, and definitely with a level system by which we sorted scores within tropes, genre, and media, and then that level scoring system would need to be worked into an algorithm and then taught to our/the scribers before the first book or episode or even song is entered into the official (non-sandbox) system (speaking of songs, we'd need to either make a deal with Pandora and figure out how to integrate a copy of their archive system into ours, OR make a deal with Pandora and figure out if we're on any sort of shaky legal standpoint if we used a system inspired by them but using none of their code, and whether we would be on shakier ground if we listed them as "inspiration", "competitors", or both).

As for he money, I've been seeing good stats about Kickstarter's more popular projects (I've noticed that even people with effectively self-sustaining businesses like Rich Burlew have been using Kickstarter to... Well, success, though "great" success really depends on the sale of the project and how effectively they planned and will execute it). I'm going to go look into them in more depth, but even if we manage to get literal Scrooge-bins worth of hundred dollar bills, that doesn't take care of the human resources issue. I'd be willing to work on helping to design the algorithms, but I'd be no good as a major part of that team, and while I'm willing to put in a full-time-job worth of hours no matter how long it takes this thing to turn a profit, be it fifty weeks or fifty years, I have no money of my own to volunteer and cannot guarantee that I won't need to pull out if I need my hours to survive a financial crisis (say, if my entire family dies when the fleet of buses they chartered with the rest of our savings crashes into the still-mortgaged house).

Regarding whether you really, really want to do this or just think it would be a good idea, this post itself is my personal test for you. I'm not going to pass judgement for merely this reason, but if you skimmed a single part of this post, or skipped to the end, I want you to think long and hard about whether you really want to do this project, and for the sake of the project itself, not for any potential financial gain.

Edit: Oh, and it's been a few years since I read The Great Automatic Grammatizer, but I recall having loved it (though maybe that's just part of me being weird). A good idea, but terrible execution. Outside of BookWorld, it would only be good as a toy or a device for running tests across the board of subjects, though could sell quite well as a toy were it less expensive.

edited 28th Jun '12 6:16:05 AM by JET73L

I'll be a hundred percent honest here: my eyes did glaze over at some points, specifically in the "as for coding" paragraph. Not because I was bored, but because my imagination was running wild with people behind desks and roomfuls of computers and SCIENCE going on, and I'm overwhelmed with the vast swaths of knowledge and skills, technical, theoretical, and practical, that I feel I will need to acquire either before or during this project. As the outline becomes clearer and the soft blue moutains in the horizon become cliffs and glaciers and waterfalls and canyons, I only get more and more excited.

Greed is not my motivator. My motivator is accomplishment. The reason I want to do this project is, simply speaking, that I would love the shit out of doing it. That I would finally get the opportuinity to turn my vice into a source of professional success, and get the money that acknowledges that success before society, is simply the icing on the cake. It does a lot for the cake, but there still needs to be mostly cake. And there will be cake... won't it?

If I'm involved in the celebration, there is always cake.

One thing I'm concerned about is that we seldom trope actual structural tropes - like certain sentence structures, camera angles, and the like. Yes, we have tropes for those, but they're not often troped as often as they could be. One of the things that the Music Genome Project incorporates is how to include those structural elements in addition to the content of the songs themselves.

And you know, it does matter. We have tropes for the most outrageously descriptive and flowery textual creations one could possibly imagine, as well as minimalist text, but they're only used for the strongest examples of each. We don't necessarily have tropes for the middle grounds. We don't necessarily have tropes for certain grammatical structures that we might not immediately think of parsing but turn out to be extremely pleasant/awful to our ears.

TV Tropes as it stands is a great start to the idea. I just think there'd have to be a ton more work done, even on the wiki side, to be able to do this.

As a side note, though, I wonder how this would affect creators. We already see some who think we almost sterilize entertainment by dissecting it and dividing it into tropes. I think this kind of project would be seen by them as us effectively turning the process into a butcher's shop. In that case, I'd like to get a pound of Well-Intentioned Extremist, two pounds of Knight in Sour Armor, and a half pound of Xanatos Gambit for garnish, please.

This sounds like a great idea, but I know nothing about how applications actually work, so perhaps I'm a bit naive, however-

Am I missing something or shouldn't this be really simple? It just takes a spreadsheet with every trope and work listed in it. First step is doing a count on every trope listing the number of works they appear in. That way you find out which are the most common tropes. Let's say you create a new database using the 1000 most common tropes, and the works associated with them.

Next, you write a routine that for any given work first lists the tropes appearing in that work, then goes through every other work listed counting the number of times a trope associated with the first given work appears. Every work gets a score based on the number of common tropes, and the highest ranking works are fed back to the user with a comment "These works share the most tropes with the one you listed" along with a list of the common tropes, and links to them.

That would work exactly like a Pandora for tropes. It would also add a really cool new feature to the wiki.

I dont know how TVT structures its data, but assuming that it's in a relatively accessible format, what would stop this from working?

Or perhaps I've misunderstood what the OP is asking for. Is this making sense, or am I way over my head?

All that needs to be said about these people is that they are fighting a retreating battle.

I love those other points you raised. I'd like to hear what everyone thinks about those and about encouraging their implementation.

EDIT: In case anyone feels concerned that this will produce some sort of normative mathematical formula for the perfect piece of fiction... I can't guarrantee it won't happen, but I'm firmly confident that it won't happen. You see, people get bored of the same old formulas repeated over and over. Strictly Formula does work, otherwise people wouldn't try it, but it almost never results in brilliance. I don't think that Strictly Formula that relies on a better, more refined formula, will be so much better that it'll be qualitatively different.

edited 28th Jun '12 3:11:52 PM by TheHandle

Well, so far, Pandora doesn't seem to have ended music as we know it. My only concern is that I made it sound so easy. It just cant be as easy as I think it is, can it? What am I missing?

Not sure it matters - I don't see this as a great way to make new stories, but it would be an incredibly useful tool for finding existing ones.