Another thing to consider is that, with the lack of inertia, you will need to slip the clutch quite a bit more when starting from a dead stop. This means the clutch will wear out faster, and you will be performing that maintenance a bit more often than with a heavier flywheel.
The general recommendation for spirited street driving and autocross is to go with a light steel flywheel. If the car is used strictly for track racing, then go aluminum.
Hope this helps.
.