Totally agree -- Ballard Inn is lovely. Also, the Santa Ynez Inn. Here's a link to a travelzoo deal for it: http://www.travelzoo.com/local-deals/California-Central-Coast/Getaway/97041? cm=signedin&conf=1&source=_tzlock&placement=Overlay -TravelDeal-HomePage-HPLeftMenu-SignUpFP5&tz_adid=1658478&es=1&isFacebook=0&isg plus=0
Ballard Inn would be my first choice. In the evening, enjoy a glass of wine on the porch before having a delicious dinner in the restaurant. Fess Parker's Wine Country Inn in Los Olivos is nice; restaurant in the inn is Petros (same Greek restaurant as in SB). Both inns are not cheap, but if you're looking for a special stay in SYV, they are my recommendations.
Ballard Inn for capturing the essence of country life in places like Los Olivos and Ballard and far less "big city" compared to Fess Parker's/Winchester or the sleazy Chumash Hotel and gambling casino/dope den. I will be anticipating the full reconstruction of Mattei's Inn and Tavern in the future.