The recusal is due to the justice's participation in an earlier version of the case, in 1985. (Yes, 1985.) For that, he gets a pass.
“It is an unfortunate fact of life that physical and mental capacity sometimes diminish with age," Justice O'Connor wrote in 1991.