His scandal is insane. I too wouldn't know what kind of shrine that was if I was visiting.
I ended up doing some of my own research on it since I was suspicious that it was a targeted attack on him. (Not a fan or hater but more like an outsider who thought something didn't make sense when I looked into it). I actually asked my friend who lives in Tokyo how would you translate '齋館'. She showed me the jp web dictionary translation (NOT a jp to eng trans but the entire explanation is in JP) Based on the definition, it’s kind of a purifying/abstinence place for Shinto priests. Attached to that building is the shrine office. So likely the pic itself isn't the actual shrine but the open area where they got like 600 cherry blossoms that are used to determine cherry blossom season in Japan. (Info is from the yasukuni shrine site). My guess is that he went to Chidorigafuchi park that same day which is another big cherry blossom destination nearby. (It does make me miss traveling to Tokyo. Come on Covid)