The question presumes, of course, that there is a library known to do this and the statistics can be examined. From the available facts, does it appear to be decreasing the number of holds? Is this acceptable? Does it raise enough funds (proportional to the need of the library and the costs of collecting the fee) to be worthwhile? How does this vary with different sized fees?
The Rancho Cucamonga Public Library in Southern California charges a fee to place holds. From their website..."Patrons may request items that are currently unavailable (i.e. checked out or not yet released) for a fee of $.50. The library will notify you via phone or email when the item is available for pick up. Items will be held at the Checkout Desk for 7 days. Requests may be placed using the ROC, or by calling the Information desk at either library. Hold fees are payable when the item is picked up. You may cancel your hold at any time, at no charge. If you don't cancel, you will still be charged for items not picked up." http://www.cityofrc.us/cityhall/lib/borrow/holds.asp
They've been doing this for years. You might contact them to see if they have any stats they could share. (909) 477-2720
Anecdotally, this was considered in my county when we were going through budget cuts. The hope was to save money by being more effective with the inter-county library holds. It costs a lot of money to drive books around to libraries throughout the county but then they aren't being picked up, so it's a waste of money to transport them around.
The idea was shot down, county wide at least. I heard rumors of some libraries putting them into place - though charging for not picked up holds rather than charging to place the hold. I don't have anymore information beyond that.
The county did change the interworking of holds, though. For example: if your home library owns the book then the ILS will wait something like 2 weeks to see if it is returned then it sends it out to the rest of the county.