Link doesn't reply - now that they've decided, he doesn't really see much need to keep looking at the conversation. Though really, he's not all that used to planning to meet up with people at all. Usually Link just shows up.

He's also wasn't exactly close to the market at the time, so it takes a while for him to make his way to the market. The fact that he makes several detours along the way - for mushrooms, for some moths, for some pretty yellow flowers - also doesn't help.

If Elma's already there when he arrives, he'll seek her out. If not, he'll find somewhere to sit and start work on her flower crown. Those yellow flowers should be nice for that...