He is troubled by this atmosphere. He wants to put people at ease, but this Toraji, he's not a very good speaker. In addition to that, he knows that his stories are not that interesting, so he gets anxious even before he starts speaking, making his stories less funny.
In the end he stops speaking, and because he looks scary, no one dares speak to him.
And so the silence goes on forever.

Narrator ~ Tiger and Dragon SP