Why is this different from a pit, where he chops it down and pays money? Rav Kahana said, a pot of partners, etc. - מאי שנא מבור דקוצץ ונותן דמים אמר רב כהנא קידרא דבי שותפי כולי
Summary:
We are not concerned for קידרי דבי שותפי, when he was סומך בהיתר; since he did nothing wrong, we are not concerned that they will leave the tree. If there would be a requirement to pay, they would not chop down the tree because the owner would demand to know who is paying him.