Sure, you can nest them, but the problem is that the pages they link to won't be filtered by month AND category.
CODE
<MTArchiveList archive_type="Monthly">
<$MTArchiveDate format="%B %Y"$><br />
<MTCategories>
<a href="<$MTCategoryArchiveLink$>"><$MTCategoryLabel$></a><br>
</MTCategories>
<br />
</MTArchiveList>
will output:
QUOTE
April 2004
Category 1
Category 2
Category 3
March 2004
Category 1
Category 2
Category 3
But the links will just lead to the category archives. If you want an archive of each category by month, you're going to have to do something fancy with PHP to display only the entries from that category and month.