His comment was meant as a joke, not to be taken in a literal sense.
I literally forgot my passport at home.
They literally locked the door behind us.
She literally cried when she heard the news.
He literally fell asleep during the meeting.
I literally did not hear your last sentence.
We literally walked for hours without a break.
We literally waited at the gate all afternoon.
He literally answered every email the same day.
She literally cleaned the kitchen in ten minutes.
They literally moved the whole desk by themselves.