The numbers of the episodes are stated to have nothing to do with their order, but rather with how original the game's concepts are.
This comes from a bad Google translation and is inaccurate—the episode numbers are the order in which the rough plans for each game were turned in.
Several worldbuilding and plot details from the game's canon bible can be found here.
Over the course of the series, Sting has explored character individuality by giving characters unusual dialects and using pronouns like "atai" that are rarely seen even in standard Japanese fiction. Recently there have also been a number of characters speaking with a light Kyoto-ben dialect (most notably Kylier and Vienya). One of the consistent complaints Atlus has received about the latter two games is the translation of all characters' dialogue to standard or formal English, thereby removing characters' accents.
They did, however, play up Lina's childish speech patterns in Riviera, filling her dialogue with a great deal of slang and contracted words.
The canon bible, Dept. Heaven Episodes World Guidance, has a section called the World Guide which details worldbuilding and backstory specifics. Feral Phoenix has translated it:
Thus far, the individual games have had three manga adaptations—a gag book called the Yggdra Union Anthology Comic was published by Enterbrain, featuring the work of various artists; serial comics previewing Yggdra Union and Knights in the Nightmare were published in Dengeki Maoh. Feral Phoenix translated the former (although those translations are not publicly available), and raw scans have recently become available through the Fan Dumb; however, scans of the serial comics do not seem to be available online.
Gulcasa and Nessiah were the first characters created for this series, and debuted in a game that series director Shinichi Ito created in the year 1996, a full two years before he joined Sting Entertainment. Said game also contained a prototype for Yggdra Union's systems. In his original designs, Nessiah had an explicit Good alignment; Characterization Marches On, indeed.
While Yggdra Union was in development, its beta system reused a number of images from Riviera while the visuals were still being designed. The Generality and Technique stats went by different names (LDS and INT respectively) in the beta stage, and Jaja's blog gives all aspects of the game three-letter codenames during the release.
Yggdra Union's codenames were "2nd" (before the title was decided) and "YGG". The beta title of Yggdra Unison was "Yggdra no Yabou: Seiken Buyuuden" (Yggdra's Ambition: Holy Sword Stories); the tagline was reused for the DS rebuild.
All complete Grim Angels (both true and artificial) other than Ein are easily identifiable by their unnaturally pale skin even without their black wings; this is actually a real condition known as leucism that involves lack of all types of pigmentation, not just melanin (the cause of albinism). It is not yet known what significance there is in Ein not being leucistic.
Ai Nonaka has had a role in every voiced game in the series.
There's also a tradition of killing off at least one childhood friend in every game. Our list so far is: Ledah; Kylier; Siskier or Jenon depending on route; Sacchito; Noah, Claude, and Theresa; Velgas.
Most battle themes in the Union games are written in Common Time. Three characters—Nessiah, Alanjame, and Elisha—have themes in waltz time, and Baretreenu's shifts time signature between 7/8 and common. Ordene's theme is written in 5/4.