You would only need to test your brother's Y-DNA. You can take the other tests yourself. I would get my feet wet with the Y-DNA test before deciding on anything else.
The mtDNA test can be used to trace a female ancestor to your mother through an all-female line. Both males and females can take the test however. They both would have the same mtDNA as their mother, although it is possible, but unlikely, that a mutation could have occurred in one or the other.
You would need to take the full-sequence mtDNA to use it in genealogy, although the lower resolution tests could be used to show that two people were not related through all-female lines. Because the surnames in these line change every generation, mtDNA testing does not have the same general use that Y-DNA testing has.