Warning: lots of jargon below, on why a 100 lumen LED appears to be brighter than a 100 lumen incandescent bul
Lumens aren't actually a unit of brightness. They're actually a unit of something called flux, which is a somewhat dense mathematical concept, but it can be easily explained in this context without any complex math. It's important to distinguish between how bright a light is (jargon: total emissive intensity) measured in candelas and how much of the emitted light is visible (jargon: total luminous flux) measured in lumens. Flux is intensity times the solid angle subtended by the beam on the measurement surface. To explain without calculus by way of an example, let's say I have a spherical bulb. It has an intensity of 10 candelas. If I just hold it up, it radiates light equally in all directions, so the luminous flux is about 120 lumens (4*pi*10, because the surface area of a sphere is 4*pi*radius^2). Now let's say I have another bulb with intensity of 10 candelas, but it produces a small focused beam. Let's say that the area of the beam at distance x from the lamp is 1/20th of the area of a sphere which has radius x. Then the luminous flux of my 2nd bulb is only 6 lumens, even though it is, in the region that it shines, just as "bright" as the first one. Conversely then, if I picked two bulbs, one with a broad beam and one with a narrow beam, each having the same luminous flux, the narrow beam bulb would have a higher intensity.
Because LEDs have less dispersion and thus throw onto a smaller area than incandescent bulbs, an x lumen LED is more intense
where it shines than an x lumen incandescent lamp is on same area.