Some definitions for the following paragraphs:

Text: any text. Can be a complete book, a word, a sentence, etc.

LR<Languages>: Language Recognizer for the languages.

Possible applications of the LR

Advanced Texthelper

Features

Can correct words without a dictionary: recognizes when something probably isn’t a word.

A word of length X has an average score of AVG. If there is a word with length X and its score is lower with significance S, then the word probably isn’t a word in the current language. I don’t know how to determine significance S.

Can correct sentences, since it also recognizes sentence structures.

Basically the same as the word recognition, but for sentences.

Can auto-complete sentences (and parts thereof).

Since the LR keeps tracks of how many times a word follows another word, it should be able to predict what word you are going to type, which could speed up writing a text. As you read, a lot of shoulda coulda woulda, but maybe we are clever enough to pull this off.

Language Recognizer

Features

Can tell if text A is more English than text B

If text A scores higher on LR<English> than B, A is more likely to be english than B.

Test results indicate very few false positives with a simple implementation.

Can help for auto-recognizing if something is human readable text.

If a text gets a higher score than previous texts, the text is more likely to be human readable text, which is useful for the “recovery” of encrypted text.

Language

I’ve been experimenting with some fun stuff lately, it involves language, programming, password-dictionaries, spell checkers, and more .

1. Get a file with a lot of words (i.e. a spell checker dictionary).
2. Count the occurrences of N-letter-groups in those words. (example with N = 3: “hello”, hel++, ell++, llo++)
3. Find high-scoring letter-groups which overlap (ie. “hel” + “ell” = “hell”) to form a word. (Obviously, repeat this to form bigger words)

Using an English dictionary, this gives output such as: “anteringlogratio”, “callinesthesional” and “prestionistering”. Altough those words are absolutely not English words (as far as I know), they look pretty English. Cool huh?

It can obviously also create ‘fake words’ in other languages, by using other dictionaries. For example, using a Dutch dictionary: “verderendelijker”, “heiderdelijkeren” and “eerderingelijkerigen”. (I’m Dutch, and I can tell you those words are easy to read and pronounce, altough they mean nothing to me)

I can hear you thinking: “Nice, but why’d you ever need to create ‘fake words’ ?”.
Well, for example, we could expand a password-dictionary by generating new passwords from the passwords already in there. This gives new passwords wich are likely to be passwords, instead of the 99% of garbage from a simple brute-forcer.

Also, if we can create words that seem to belong to some language, we can probably also recognize if something could be a real word.

We could create a program wich can tell random sequences of letters and real words apart by giving the words an ‘Englishness-score’, the more it uses high-scoring letter-groups, the higher the score. This might come in handy while brute-forcing a password for some text file. The program could detect if it succesfully decrypted the file, or just got some garbage-output. Or we could create a smart spell checker wich can check and correct words even when they’re not in the dictionary. If ‘hello’ has a much higher score than ‘helllo’, the user probably wanted to type ‘hello’. Or a language-recognizer, if all the words have a higher ‘Dutchness-score’ than ‘Englishness-score’, it’s probably not an English text.

When we take this a level higher, not using words and letter-groups, but sentences and word-groups, the awesomeness grows exponentially. A spell checker could simply correct “Ive is fun” to “Ice is fun”, but “Ive been running” to “I’ve been running”. An internet spider could tell real text apart from those random-keywords-pages. Lots of possibilities.

Anyway, I’ll try to code some of this stuff to see how well it all works, stay tuned.