Preheat the oven to 350°. Line a 9-inch pan with parchment; butter and flour the parchment.
Mix the all-purpose flour with the baking powder and salt in a small bowl. Set aside.
Pulse the almonds in the jar of a spice grinder until they become a powder, and then continue pulsing for a minute or so longer until the almond oils start to release slightly and the mixture becomes paste-like.
In a large bowl, or using a stand mixer, beat the sugar with the ground almond paste.
Add the butter, beating until fluffy.
Add the eggs one by one, beating well each time.
Add in the vanilla or brandy.
Gently fold in the flour mixture.
Pour the batter into the parchment-lined prepared pan. Arrange the plums over the top of the batter in a spiral.
Bake for 1 hour and 5 minutes, or until the cake top is well-browned and a toothpick inserted comes out clean.
Cool the cake in the pan for a few minutes. Then run a knife around the edge, and invert the cake onto a plate—peel off the parchment—and then invert it again onto a wire rack to cool completely. Serve warm or at room temperature topped with crème fraîche.