I noticed that, when I quit an application that wasn't in the Dock, its icon remained in the Dock and I had to manually remove it. A quick look at the Dock pref pane revealed that "Show recent applications in Dock" was checked by default. I unchecked it and the issue was gone.
