Well, firt you should know that PS3 can play the following types of files under (Video).
•Memory Stick Video Format
- MPEG-4 SP (AAC LC)
- H.264/MPEG-4 AVC High Profile (AAC LC)
- MPEG-2 TS(H.264/MPEG-4 AVC, AAC LC)
•MP4 file format
- H.264/MPEG-4 AVC High Profile (AAC LC)
•MPEG-1 (MPEG Audio Layer 2)
•MPEG-2 PS (MPEG2 Audio Layer 2, AAC LC, AC3(Dolby Digital), LPCM)
•MPEG-2 TS(MPEG2 Audio Layer 2, AC3(Dolby Digital), AAC LC)
•MPEG-2 TS(H.264/MPEG-4 AVC, AAC LC)
•AVI
- Motion JPEG (Linear PCM)
- Motion JPEG (μ-Law)
•AVCHD (.m2ts / .mts)
•DivX
•WMV
- VC-1(WMA Standard V2)
Usually convert the other files to MP4 is good, which can be played in most cases. AS PSP is some similar to the PS3, I suppose this detailed guide can do some help for you:
http://www.ifunia.com/youtube-column/howto-put-youtube-videos-on-psp.html