A HIGHLAND castle owner who offered Prince Andrew a place to stay has suggested this "may not be workable" amid public outcry.