Hi Larry,
Yes, it is permissible to use your putter.
In the Rules of Golf, rule 16-1a doesn't mention towels, hat etc. (maybe you're looking at a guide to the rules) and you may remove loose impediments by any means provided that you don't press anything down, except as provided in 16-1a-vi and 6-1c (re ; pitch marks and old hole plugs).
Look at the rule and decisions at
http://www.randa.org/en/Rules-and-Amateur-Status/Rules-of-Golf.aspx#/rules/?ruleNum=16&subRuleNum=1&decisionId=BAA43C68-5870-43E9-90AA-B1DEB131D5EE
Note also rule 8-2b which prohibits touching the line of putt (with anything) to indicate the line for putting.
Hope that helps