Hi Filip, at this time it is not possible to customize the Date Range filter to remove granularities, even if these granularities are not enabled on the Dataset, as you noticed.
I checked internally, and our product team is already aware of this limitation and are considering adding this feature for future releases, although we do not have an ETA for that yet.