I might be missing something but the old gasket looks swollen due to oil exposure - common issue for any gasket made from rubber. It probably pushed its way into the notches as it swelled.

As for the screw causing the cap to tilt away from the base - make sure the cap is correctly seated. I haven’t worked on a lot of these Honda’s, but the ones I have, I can recall seating the cap long enough to snug down at least one screw was a PITA.

If you’re sure the cap is seated correctly, the screws only need to go down far enough to compress the lock washer and then give them about 1/8 of an extra turn to snug them down. Any tighter is unnecessary and putting stress on the cap.

 

The notches in the groove are to hold the gasket in place when you install the cap. I agree with Gumby that the gasket may not be seated properly and could cause the cap to lift. If the gasket doesn't sit easily in the groove, I'd probably get a different gasket. Felpro seems to make good fitting aftermarket gaskets. You can always pick a Honda gasket at the dealership.

Since you had some grime inside the old distributor cap, wipe off the outside of the distributor and monitor for oil leaks especially where the distributor meets the cylinder head. There is an o-ring that can become brittle and seep oil. It is a pretty easy DIY. You just need draw a line from the top of the distributor to the cylinder head to reinstall the distributor at the correct angle. The distributor can be turned to adjust base timing, but reinstalling with the distributor at the same angle ensures you don't have to readjust.

 

Those notches look like a way for oil to drain out of the distributor in case the shaft seal leaks oil into the distributor. Clean up all that oil inside and outside the distributor. Keep an eye on it.

Oil leaking into the inside of the distributor is a more difficult fix. The easy-to-fix O-ring will allow oil to leak out where the distributor bolts to the head - where most of the black gunk is in your photo.

 

Thanks you all! I definitely tightened the screws a little too tight on the cap. I undid them and the cap is sitting fine and the gasket is in the groove.
Gumby, I did not realize oil could do that to the gasket. That must be what happened.
PA and Jim, that was my next task! I was going to try to track down that oil leak. Thanks for solving that for me, that is a real easy fix. Thankfully no oil inside the cap.

 

Upper left part of this page shows the O-ring, but I can't find a good picture of the shaft & O-ring groove. Chances are if you take off the distributor you'll find the O-ring is hard from age & heat.



You don't really have to worry about lining the engine up to #1 TDC; as long as you pay attention to the shaft and the socket where the shaft fits. The blade is off-center a little bit, so line up the distributor shaft to match the socket in the end of the camshaft. People have managed to get it installed backwards if they push it in place with enough force.

You have the F23A1 engine based on the photo of the "VTEC" valve cover.

Also be aware that oil can leak from the valve cover gasket just about the same place as the distributor O-ring. Or maybe from both places...