Why would Mahler use so many unusual instruments (like the off-stage cowbells or the celesta in Symphony No. 6) in his symphonies? Are they really helpful for expressing the themes?

Brian van der Spuy is quite right that the extensive orchestral palette is in line with Mahler’s desire to encompass the world in his symphonies.

Howard Levitsky’s answer, featuring Gershwin’s car horns, is ironic, given the contemporary reaction to the cowbells in Mahler’s 6th:

Gustav Mahler Pictures

The cartoon’s caption reads: “Herr Gott! Daß ich die Huppe vergessen habe! Jetzt kann ich noch eine Sinfonie schreiben.” (My God, I’ve forgotten the motor-horn! Now I shall have to write another symphony).

The cartoon was published in the Austrian/German magazine, Die Muskete (The Musket), on 10 January 1907. The cartoonist was Fritz Schönpflug (1873-1951).

Was it helpful? Yes, because Mahler (who after all had a day job as an opera conductor) had a keen sense of orchestral colour, and timbre is a major component of his musical meaning. If offstage bands and mistuned violins were part of the palette he used to communicate, then cowbells and harmoniums were too. And Mahler always used his colours with great delicacy, it wasn’t a wall of sound.

Was it effective? Personal opinion follows:

• The whip in the scherzo of the 5th: yes, it contributes to the manicness of that particular passage.
• The hammer in the 6th: most of the time the realisation of the hammer blow is underwhelming. Even when it isn’t, the “thud” is an extramusical, programmatic element; musically, I’d add a couple of bass drums would have done the same job. But watching a performance, and seeing the performer slowly walk up to the hammerbox, is powerful. (Opera conductor, remember.)
• The cowbells of the 5th and 6th: I know what Mahler was communicating, and why he was communicating it at those points. My opinion, he didn’t need it, it’s a little too real-world to be integrated into the fabric of what is going on—music on its own conveys pastoral solace just fine. I find it a little distracting. Mercifully in all the performances I’ve heard, it’s pretty damn quiet.
• The mandolin in the 7th: yes. There’s a lot of atmosphere-painting where it is used, and it fits.
• The celeste and the harmonium in the 8th. Sorry, I find the picture Mahler paints at that point, of the Mater Gloriosa, silly. But you know, that means that the celeste and the harmonium are all too effective: that was the picture Mahler meant to paint, and it’s what Goethe chose to write. The 2nd movement of the 8th is after all an oratorio, with a lot of tone painting going on. We just can’t buy into the Eternal feminine now.