Colin Donovan, Vice President for Theology at EWTN, has written a great article, summarizing how the Church evaluates reports of private revelation, such as Marian apparitions, and how prudent Catholics should react to such phenomena. You can also visit the Vatican website to read the Congregation for the Doctrine of the Faith's normative guidelines (1978) for evaluation claims of apparitions, as well as Jimmy Akin's helpful commentary here. Because I know that many of you are pressed for time, I've tried to summarize the relevant features:
First, we really need to keep straight the difference between Public Revelation and private revelations (CCC 66-67). The latter, such as the Blessed Mother's request to Catherine Labouré to create the "Miraculous Medal", can never be regarded as necessary for salvation. They are not elements of the Faith handed on by Christ to the Apostles (Public Revelation). Rather, authentic private revelations, are encouragements that God has sent, in particular times and places, to help us better live the Public Revelation.
We Catholics do not consider private revelations as matters of "divine and Catholic faith" the way we do Jesus' Real Presence in the Eucharist. The Church speaks of private revelations as "giving evidence of supernatural intervention." To use Pope Benedict's words, they are considered "worthy of pious credence" and are judged worthy of a prudent "human faith."
So what makes for an approved private revelation? First and foremost, it cannot contradict or claim to correct Public Revelation (Scripture & Tradition). It is investigated by the local bishop. What specific things does he consider?
a) the visionary - their psychological work-up, morality, and obedience to Church authority, whether he/she has sought to profit materially from the apparition
b) whether or not the revelations contradict Public Revelation
c) the presence of constant and abundant fruits - has the apparition led to increased prayer and displays of love of God and man
After looking at all of this the bishop, exercising prudent human judgment, issues one of three conclusions:
1) it is supernatural
2) it is definitely not of supernatural origin,
3) at the present time, although there are positive signs, there is insufficient evidence to declare it supernatural in origin.
In most cases the judgment is #2 - definitely not supernatural. In some rare cases the bishop deems #3 appropriate and gives approval for his flock to practice the devotions associated with the apparition as he and subsequent bishops continue to study the details of the apparition and watch for "constant and abundant" fruits. (A current example of this is devotion to Our Lady of America - see Jimmy Akin's blog.) In the case of the first approved apparition of Our Lady in the United States, Our Lady of Good Help (Champion, WI), devotions were approved for practice for well over a century before the Bishop of Green Bay issued his judgment that the apparition was of supernatural origin (#1 above).
The bottom line in all of these cases is that Mother Church expects us to abide by the local bishop's decision. When it has been positive the Pope will sometimes signal his own approval of the apparition by granting a liturgical feast to the apparition, canonizing the visionary, granting favors to the apparition site, or making a personal pilgrimage there.