Help - Chapter 5 (Spoilers)

I'm at the bit where Mayu has just crossed the bridge between Kiryu and Tachibana house. I fought the ghost on the passageway and then grabbed Folk Scholar Writings V from the ghost inside. It says that, to get to Tachibana house, I need to leave using the underground Earth Walkway. I've spoken to Yao and he confirmed I need to unlcok a mechanism in Kiryu house to open the passageway.

Help! I've looked everywhere and can't for the life of me think what I've missed. I went around to the side entrance to Kiryu house and shot a photo of "Entering Man" however, if I try to enter that door, it has a padlock on it.

Any advice?