Unfortunately for a project manager, insisting on tracking time can be perceived as overkill. And being a stickler about an employee's schedule is usually met with hostility. But, if you knew what obligations your project manager had to meet in order to keep the project and thus, your job, afloat, would you be more considerate of his or her requests?
Consider all the variables your project manager is tracking at one time. You are only one part of one section of the larger picture. Say, for instance, the project is a house - you write the contracts and your project manager wants you to track time spent on each trade contract. This could be because he or she needs to job cost each trade and your time is a component of costing out what was spent acquiring the contractor. Then, your project manager may go further and require you track phone calls discussing scopes of work with the contractor. If you have two different projects with two different contractors for the same trade, it is easy to see which contractor is the more efficient for future building and your project manager can get an idea where to shave the budget.
But, that is not all an effective project manager is tracking. At the same time, they are tracking the different trade contractors, fixtures and appliances, construction workers and the financing end. No matter what field the project is in, it will be multi-faceted for certain.
A good project manager will take time at the front end of a project to categorize what he or she wants to track and not get bogged down in unessential matters. Your project manager will also adjust the focus throughout the project and fine tune time tracking to get the best overall picture of the project.
You've got to give your project manager credit for knowing what's best. If he or she finds that your time can be better utilized, listen to his or her advice; chances are, your project manager has an accurate chart to back it up.