Elas's account of the conversation between Amyas and Caroline in the library - the one in which Caroline says "You and your women!" - was a lie. Are we to assume that Elsa was not called to recount that conversation at Caroline's trial? If she had, Caroline would have smelled a rat immediately.
I think the implication was that it was real- it was also when Caroline said: "I'll kill you one of these days!" The actual words did incriminate her, but only because Elsa drastically changed the manner in which they were spoken to sound furious and threatening, rather than affectionately exasperated.