Under what regulatory authority do you possess and use the sources? I would start there.
If the regulations are silent on the issue the NRC may have added License Conditions to your license requiring a periodic inventory (and maybe even a leak test).
NUREG 1556 Volume 20 Appendix E has a list of standardized verbage that may be used by NRC licensing personnel to add license conditions to a materials license.
For example:
For Inventories: "The licensee shall conduct a physical inventory every 6 months, or at other intervals
approved by the U.S. Nuclear Regulatory Commission, to account for all sources and/or devices received and possessed under the license. Records of inventories shall be maintained for 5 years from the date of each inventory and shall include the radionuclides, quantities, manufacturer's name and model numbers, and the date of the inventory."
The exception for Leak Tests: "Sealed sources need not be tested if they contain only hydrogen-3; or they contain only a
radioactive gas; or the half-life of the isotope is 30 days or less; or they contain not more than 100 microcuries of beta- and/or gamma-emitting material or not more than 10 microcuries of alpha-emitting material."
It is possible that the licensee may have committed to a periodic inventory in its License Application and the NRC found that acceptable and it is included as a tie down thereby making it a requirement.
Again, it comes down to the question, "What's in your license (and tech specs and application and other tie downs)?"
I thought of another relevant standard license condition (it applies regardless of area classification, i.e., restricted, controlled, or unrestricted):
"Sealed sources need not be tested if they are in storage and are not being used; however,
when they are removed from storage for use or transferred to another person and have not been tested within the required leak test interval, they shall be tested before use or transfer. No sealed source shall be stored for a period of more than 10 years without being tested for leakage and/or contamination."