Who is responsible for disposing of tree branches growing over a property line?

A neighbor claimed that branches from a tree growing in my yard were getting too close to his house. He took a chain saw, cut off the branches and threw them in my yard. I confronted him and told him to dispose of the branches himself since he was the one that cut them. He said that since it's my tree, I'm the one that is responsible for their disposal. Who is correct?