The difference is: a microphone is an energy conversion device that converts sound signals into electrical signals, which is used to pick up the sound of your speech. The speaker is a kind of transducer that transforms electrical signal into sound signal, it's the electrical signal that converts the other side to produce into sound to play.
The function of the microphone is to collect sound, the function of the speaker is to play sound.
So when you turn off the speaker, you can't hear the other person, and when you turn off the microphone, other person can't hear your voice.