Like Chili said, the cartridges in the mag are putting more pressure on the bolt carrier when the mag is full...but as the mag gets to the last rounds, the pressure is less. The weaker pressure allows the rotating bolt to rotate faster, causing greater centrifugal force, which will make the extractor lose its grip on the case rim.
There are several ways to resolve this problem, but I will give you my personal recommendation.
First, on any shorty, you will want an improved extractor function. That can be done by adding a D-fender ring to a standard 4 coil extractor spring, or by adding the Colt black insert and a wolff extra power 5 coil extractor spring. That will help the extractor fight the powerful centrifugal force.
Also, a heavier buffer could be used, like an H, H2, H3, MGI etc.
With any buffer, I would highly recommend a David Tubb Premium Chrome Silicon buffer spring, which is rated as a "standard power" spring, but it improves the cycling function, because it remains CONSISTENT over the life of the firearm. Most buffer springs will shrink and get weaker over time, allowing the bolt to unlock faster and faster over time, which causes the problems that you mentioned. The Tubb Speedlock Spring will "lock" the rotating bolt speed at one consisent repeatable level, so that it will function the same on round # 500,000 as it did on round # 1.
In your case, it seems that the magazine spring pressure played a role in the problem. A solution to that is to use a better magazine spring...one with more consistent power throughout the entire range of the spring compression. ISMI Premium Magazine Springs are a great way to make your mags run better. Most mag springs have pressure that drops off dramatically when there are only a few rounds left...as you have seen even with an HK 416 mag.
If you check the "Parts & Accessories For Sale" section, you will see that I have a David Tubb Speedlock Chrome Silicon Premium Buffer Spring, and ISMI Chrome Silicon Premium 30 rnd magazine springs (which also come with the green anti tilt mag follower) for sale. These upgrades, along with a D-fender ring, should solve all of your problems in the cheapest way possible.
