Chateau des Milandes was built as a gift for his wife Francois de Caumont. Caumont, who owns the Chateau de Castelnaud built the much newer chateau in the 15th century to please his wife who found the warlike nature of both Castelnaud and the surrounding region cold and unpleasant.

The Chateau didn't originally have much history. It fell out of use and slowly into disrepair during the French Revolution. It did not begin to develop any real history until it was purchased by singing star Josephine Baker in the 1930's.

The Chateau has the rare quality that its last 100 years is the reason most visitors come. It is maintained in the style that Baker decorated it in while she lived there in the 30's. It also houses a museum to her life and her rise from back alley New York clubs to worldwide fame.