Oh, if I knew the answer to all those questions I wouldn't be living on this side of the tracks, y'all. But having written a few hundred melodies over the years, I'll take a stab at offering my perspective on the question, and hope that it can help you in your own melodic quest.
As I may have mentioned in previous columns, I'm a total nerd. I love the strange little quirks and coincidences that can lead to cooler grooves and wilder chord progressions upon which to build our melodic fancies. One thing I try to look for is common tones–a single note that several chords share, or can be madeto share–that can pull the ears along on a particular journey. Don't you love how a first position Fmaj7 chord is just an Am with an F in the bass? And if you play an Am7 and then add the F, you get an Fmaj7add9, and isn't that just one of the best chords ever?
I also love how with the shift of one finger at a time, you can go from Asus4 to A to Am to A9. All those voicings around one single open chord.
One of the things I try to teach my students is that chords are not written in stone. We are free to add or take away any finger we want in any chord, or move one finger one fret lower or higher, or play an open shape somewhere on the fretboard other than first position. If a single note ties two or three chords together, even if it really doesn't belong diatonically in all of them, it makes your foundation stronger, like putting rebar into a concrete structure–you go from a strong foundation to one that a tornado can throw a Buick at and not even make a dent in. What this does for you is to allow you to keep your melody simple and versatile in places where your chords are more interesting, which means when you want to emphasize something important with a bigger, bolder or more intense melodic line, you are free to soar melodically without risking ear fatigue for your listeners. When you do that, switch over to simpler chords, like 5-chords (simple two-note chords missing the major or minor third) that can be anything, so your melody has some wiggle room.
Don't Follow the Damn Chords
Melody that simply restates the chord progression one note at a time is a major peeve for me. As previously stated, chords are there to create the structure and foundation for a song to build on. If your chord progression is linear going down, pick a common-tone out of all your chords and hang onto it for a while (consider the immortal line of John Hartford's: "It's knowing that your door is always open and your path is free to walk," which pretty much hangs onto that one note up to the last four words). Or, try making your melody skip up in a weaving fashion. One of the things I also don't like is if someone walks up from a I chord to a IV chord, and just follows the chords on the root or the third. I always have to suppress the urge to slap them and wail, "Oh, you can do better than that!" (My peeve with this has not prevented songs like "Me and Bobby McGee" from becoming wildly popular, however.)
A few years back I was playing with a chord progression that was kind of linear; the bass line didn't follow a linear progression down, but the voicings of the chords did.
Much to my chagrin, I was improvising a melody that sort of followed the chords, and I wasn't getting anywhere with either the composition or the words. A friend suggested that I start lower and bring the melody up instead, and within moments the entire song fell into place. I couldn't get engaged with the melody because it was, frankly,boring. Just one little shift was all it took.
Harmonize Yo' Bad Self
Another trick for riff-oriented guitar players is to try to harmonize your melody with the riff you've created, to continue the strong feel of the "hook" that you're working with but vary it a little to make it more interesting. In particularly guitar-oriented songs, you can use that trick to make the song as much about the guitar as it is about the melody or the words.
Have you ever listened to Thelonious Monk? He was a master of melodic and harmonic tension. In fact, to the uninitiated, some of his best stuff sounds like he's making mistakes. Listen intently to those angular melodies, that razor sharp stride, the rhythm and timing and phrasing, and you realize quickly that he not only knew exactly what he was doing, the cat was a frickin' genius. One of the things I like to do is land on a perfectly diatonic note and let it slide up or down by half a step or sometimes a whole step, and then hang on to it through a chord change or two, letting the tension build so the next phrase really gets youright here, dig?
Slips of the Tongue
Back in the early part of the twentieth century Czech composer Leoš Janáček used the rhythm, pitch contour and inflections of everyday speech as a melodic device for his music. He would be sitting in a café or walking on the street listening to the chatter around him, often writing down what he heard to be used for later melodic lines in his operas or instrumental works. It was a radical notion at the time, but it's a wonderful idea.
One of my "rules" of songwriting is, "If you wouldn't say it, why would you sing it?" I always prefer a very conversational songwriting form, and in doing so, have learned that songs with simple, natural, conversational melodic lines are highly effective. So the best trick in the book is to decide what you want to say, speak it, listen to the natural rhythm and cadence of the words as they leave your mouth, then sing it, gently or passionately or with whatever emotion is appropriate for your meaning.
I've taken to using this method for guitar solos, too. I think about what the song is about, how it makes me feel, what I want to say in addition to what the words are, and just play that like I'd say it. You can say things you'd never, ever put into words, simply by thinking the phrase and playing the rhythm, pitch contour and inflections of your own normal speech.