All of Mr. McMurtry's antimythic groundwork -his refusal to glorify the West - works to reinforce the strength of the traditionally mythic parts of ''Lonesome Dove,'' by making it far more credible than the old familiar horse operas. These are real people, and they are still larger than life. The aspects of cowboying that we have found stirring for so long are, inevitably, the aspects that are stirring when given full-dress treatment by a first-rate novelist. Toward the end, through a complicated series of plot twists, Mr. McMurtry tries to show how pathetically inadequate the frontier ethos is when confronted with any facet of life but the frontier; but by that time the reader's emotional response is it does not matter - these men drove cattle to Montana!