The Mishna (52b) taught that having placed the coals and the incense in the Holy of Holies, the kohen gadol would back out of the kodesh kodashim the way he came, and, having completed his task, would say a short prayer.
The Gemara explains that it was necessary to walk out backwards out of respect to God’s presence in the Holy of Holies. Based on this, Rabbi Shmuel bar Nahmani quotes Rabbi Yonatan as teaching that anyone who left the Temple precincts did not turn his back on it, rather walked sideways as they left. Similarly a student who takes leave of his teacher should not turn his back on him, rather he should walk sideways. The Si’ah Yitzhak explains that it is difficult – and potentially dangerous – for someone to actually walk backwards, which is why the halakha is to walk away with one’s head still facing the Temple (or teacher) as they leave.
Apropos the obligation of a student to walk backward when taking leave of his teacher, the Gemara discuses a similar topic. Rabbi Alexandri said that Rabbi Yehoshua ben Levi said: One who prays must take three steps backward upon concluding his prayer [the amida] and then recite: Peace, in a manner befitting one who departs from before the Holy One, Blessed be He.
According to the ge’onim, a person who is praying is surrounded by four cubits of Divine presence. When he completes his prayers he must step out of that setting by walking back three steps (three steps from the middle of four amot (cubits) would remove him from that area). Rav Yosef Karo in his Bet Yosef quotes Rav Hai Ga’on as explaining that this relates to the three levels of stones that were on the eastern side on which the kohanim walked up and down. The Gra and the Maharsha relate this to the story that the Gemara tells about Nebuchadnezzar who honored God by taking three steps on his behalf, for which he received great reward.
According to the Gemara (Sanhedrin 96a) Nebuchadnezzar served as a secretary and scribe for a previous Babylonian monarch. Once, when Nebuchadnezzar was absent from work, another one of the royal scribes drafted a letter to be sent to the king of Yehudah, Hizkiyahu. The letter began: “Greetings to King Hizkiyahu! Greetings to the city of Jerusalem! Greetings to the great God!”
When Nebuchadnezzar returned to work and discovered how the letter was written, he objected, saying “you call Him ‘the great God,’ and then you mention Him last?!”
Nebuchadnezzar insisted that the letter be redone, writing: “Greetings to the great God! Greetings to the city of Jerusalem! Greetings to King Hizkiyahu!”
The problem was that the messenger had already been dispatched to Jerusalem with the first version of the letter in his hand. Nebuchadnezzar ran out to call the messenger back and redo the letter, running three steps to catch the messenger. The Talmud credits this behavior for his ascension to power.