I tend to play assassin/thief more so dual wielding is not a show stopper for me but annoying. Usually withing 5-10 seconds of a battle.
When you remove one, a 'stack' of two is removed the same applies to the second 'stack'. Like all followers who are capable of dual wielding, Rayya must be in actual combat before she will unsheathe the second weapon but will only unsheathe one weapon while out of combat.