Preheat oven to 350°F. Place garlic cloves in a small ovenproof saucepan with duck fat, rosemary, and thyme. Cover and bake 40 minutes until garlic is soft and caramelized. Remove, strain fat into a heatproof jar, and keep aside. Set garlic and herbs aside. Increase oven to 425°F.
Bring a medium pot of salted water to boil. Add potatoes and boil for 10 minutes until half-cooked. Drain and shake potatoes to fluff edges. Set in a colander to dry about 10 minutes.
Toss potatoes in a bowl with duck fat, semolina, caraway seeds, harissa, and 2 teaspoon salt. Spread on parchment-lined baking sheet. Roast 45 minutes, turning potatoes once or twice, until golden brown.
Stir in the confit garlic and herbs. Roast another 10-15 minutes until dark golden and crispy.
Meanwhile, make garlic mayo by combining minced garlic clove and rice vinegar in a small bowl. Let sit 5 minutes, then stir in mayo.
When potatoes are done, sprinkle with flaky salt and serve with garlic mayo on the side.