Will removing a closet to create a dedicated home office hurt my resale value?

We currently have a four-bedroom house, but we converted the smallest bedroom into a custom home office by removing the closet to build in a desk and shelving. We live in a family-heavy neighborhood in Texas where bedrooms are highly valued. Should we tear out the custom woodwork and frame the closet back in before we put it on the market? I am worried buyers will pass if it is officially listed as a three-bedroom.