The main thing you need is the sony psp development kit that contains not only the library functions you need (the firmware has a lot of standard calls to do things like saving, reading/write memory sticks, reading the UMD, etc), as well as what registers are used for what. And since Sony is controlling all the UMD burners (the devkit PSP has a long cable coming out where the UMD drive is, which connects to the devkit box (which has a UMD drive), it's a bit more difficult to go that route. (The developers use the devkit box to "emulate" a UMD, then they send off the image to Sony to make some UMDs for them, which they can then test using regular PSPs or the UMD drive on the devkit box.)
Hello world has been done - it was the first demonstration app that used a flaw in the 1.0 firmware to load a duo program. And it all has source code too. But you're limited to those who have the original Japanese PSP and those who didn't update the firmware on the PSP (getting rarer as the originals will run only the games that could use the old firmware - new games require the new firmware).