FOSS, learning curve, and niche field? I got you. VLSI layout software or SDR (software defined radio) software.

VLSI layout is kind of software allows you to design electronics hardware at the silicone/transistor level. IOW you have to draw the layers of materials that form transistors (cross poly silicon and p/n-diffused silicon to form) and connect them to metal layers. Learn how NAND and NOR gates are actually formed. Make a 1 bit data storage (flip flop), half adder, full adder. Go as far as you want.

Example software:

• Magic (what I used in school): https://opencircuitdesign.com/magic/
• klayout: https://www.klayout.de/

SDR. Less intense, but also niche and gives you lots of related things to learn/branch in to.

Can be used to receive and decode radio/satellite transmissions. Normally people would buy hardware and use software locally, but some people make their receivers accessible on the internet (websdr) which can give you a first intro to it. After you capture data you can run it through other software to decode (depending on what it is). For example, you can capture raw weather image data from satellites and decode it to get the images that the satellite is transmitting as it’s capturing the data.

A list of some SDRs you can access over the internet (doesn’t require hardware purchase to use): http://websdr.org/

It probably won’t be immediately obvious how to use either SDR or VLSI software but you asked for niche and learning curve, so enjoy. ;)

EDIT: Fixed some typos and grammar.