The initials on Justice's gloves, "HPJ", stand for "High Prosecutor Justice". She held the title before passing the job on to Judgement, and the art book states that they were a gift from Lucifer, possibly to commemorate her being given the job.
Per the Old Testament Satan is characterized as a celestial prosecutor, whose job is to accuse humans to see if they're morally worthy. Christians take the more cynical approach that Satan is the accuser of the brethren, hoping to see Christians corrupted, and that Satan being a prosecutor is more of a formality rather than something God appointed. Either way, Satan serves the role of the prosecutor — this theme is most chiefly illustrated in the Book of Job. What makes this brilliant is that Lucifer in Helltaker has two prosecutors under her employment, which likely comes from her experience as a prosecutor in the Bible. The alternate take of loving to corrupt humans is certainly characteristic of Lucifer's demons, so the Bible backstory applies in both interpretations of Satan as a prosecutor.
Loremaster lost her arms during her battle to take over Hell and replaced them with Artificial Limbs, making her a "Cyberdemon".
Loremaster's seeming fondness for apple pies, and how Lucifer's new position as her maid means she's often getting served the dish, is likely a nod to how an apple was what corrupted Adam and Eve. Also, said apple was offered by Lucifer in snake form. Now, Lucifer's spending her days offering apples to another previously pure being whose corruption she was responsible for.
Loremaster says that Lucy tries to kill her twice a week. This is why Lucy ate all the apple pie in the last level of Examtaker. She wasn't hungry, but rather taking away Loremaster's only means of controlling Subject 67 in hopes that he would try to kill her like he promised.
In the Bible Beezelbub is one of the names of Satan, and is specifically used by the pharisees on more than one occasion when debating morality with Jesus. In the Helltaker continuity Beezelbub is her own separate entity different from the mainstream Lucifer. The brilliant subtext here is that Beezelbub probably became one of the names for Satan in the classical tradition because she claimed herself to be the ruler of hell, and in the eyes of many demons they started calling her Satan in a co-equal fashion to Lucifer herself. Eventually this came down the grapevine until by the time of Jesus even humans were aware that Beezelbub was an accepted alternative name for Satan, not truly knowing they were different people. Lucifer not wanting a danger to her rule and reputation to exist banished Beezelbub to the void, but by then her name was recorded in the Bible and the damage was partially done already.
The girls (and the order in which Helltaker meets them) do loosely correspond to the Divine Comedy.
The first circle is Limbo and houses those not bad enough to get into the other circles. Pandemonica, the first girl, is too tired to really do anything evil (until she gets her coffee).
The second circle is for the Lustful, matching Modeus's title of the Lustful Demon.
The third circle is for the Gluttonous, and Cerberus lives there.
I'm unsure how Malina and the 4th circle (greed) connect, but the fifth circle is for Wrath, and the fifth girl is Zdrada the Bitch Demon.
The Sixth Circle punishes Heresy. Your sixth girl is an angel, Azazel, who is committing heresy herself by living with and "researching" demons.
The Seventh Circle is the home of the Violent- and while Justice isn't that violent, she would have had to be in the past as the High prosecutor.
The 9th circle is for Traitors. The Final Boss, Judgement, is a Fallen Angel, and thus a "traitor" to Heaven, and because Lucifer can't control her despite being Queen of Hell, she's also a traitor to Hell.
In the Fur-Taker Comic, Lucifer attempts to get Helltaker to get off on her goat form. Helltaker refuses, adamantantly stating he isn't a furry, She responds that "Denial leads to enlightenment". At first it seems like it's an obvious case of her mocking him, suggesting he is a closet furry (The animated version has Helltaker's glasses reflect her tail and butt) with him getting visibly annoyed, But it may be also a play on the idea of "The Devil knows scripture" were she is mocking God's command of abstaining from sin and how being saved means denying indulging in sin.
