He'll always notice you - but as long as you have bag space, you'll at least get away with the pollen.
After you completed the East Caves quests (where you end up helping Rose and Ixxy), he should no longer mind - but I realized that little detail was bugged, so I'll fix that. :3