[1]

Using NAEP to confirm state testing results in the No Child Left Behind Act. (2007). Practical Assessment, Research & Evaluation, 12(5). Available online: http://www.pareonline.net/pdf/v12n5.pdf

[a] There are two federal definitions for "proficient," one for NAEP and one for NCLB.  NAEP Proficient (i.e., the mastery of challenging subject matter that includes some above-grade-level content) is not synonymous with proficiency in a subject (which is the definition of NAEP Basic) nor with the attainment of grade-level expectations (which is the NCLB definition of proficient states must demonstrate to get federal funding).

[b] The percent of students At or Above Basic is the most appropriate NAEP statistic for confirming state reports of Adequate Yearly Progress expressed in terms of the state’s percentage of students scoring proficient or higher.  This finding is supported by rigorous statistical analysis as well as by NAEP and NCLB definitions.

[c] Confirmation of state testing results should not be conducted on a point-by-point basis.  Even point-by-point comparisons of NAEP Basic vs. state proficient can be misleading.

[d] NAEP policy denotes Basic as partial mastery of prerequisite knowledge and skills that are fundamental for proficient work at each grade. If classroom grades were applied to NAEP achievement levels, Basic would begin at C-/C and Proficient would begin at B+/A-.  Paraphrasing the NAEP policy definition of Basic, C-/C work denotes partial mastery of prerequisite knowledge and skills that are fundamental for B+/A- work at each grade.

[ps1]  "Using NAEP to confirm state testing results. . ." cites an evaluation of the NAEP achievement levels (i.e., Basic, Proficient and Advanced) conducted by the National Academy of Sciences. The Academy found the achievement levels "fundamentally flawed," but did suggest that they might be used to report trends in student achievement. It has been widely assumed that comparing achievement level trends from NAEP with state achievement level trends is a statistically appropriate practice. Research reported in recent literature, however, suggests that NAEP/state achievement level trend comparisons are statistically weak and should be avoided.

“Trend comparisons require both technical care and substantive consideration.  As useful as PAC [percent above cut-score] statistics have been in communicating test results to the public, their properties as trend statistics render them ill-suited for trend comparison.” 

Ho, A.D. (2007). Discrepancies between score trends from NAEP and state tests: A scale-invariant perspective. Educational Measurement: Issues and Practice, 26(4), pp. 11-20.

[ps2]  An explanation for the large differences between state and NAEP "proficiency" scores reported for reading in 2005. (2007). Paper presented at the Chief State School Officers 37th Annual National Conference on Large-Scale Assessment, Nashville, TN. Available online:
http://www.eric.ed.gov/ERICDocs/data/ericdocs2sql/content_storage_01/0000019b/80/2b/61/f5.pdf

[ps3]  Politicians are dumbing down tests, losing accountability.  Letter to the Editor, The Idaho Statesman, 2007.

[2]

Mapping 2005 Idaho Proficiency Cut Scores onto the NAEP Scales. (2007). Available online:  http://www.boardofed.idaho.gov/naep/misc/naepequivalentscore/nes_menu.htm

[a] There is no clear, consistent relationship or correlation between the rigor of a state’s performance standard for proficiency (i.e., cut score) and the overall achievement level of the state.

[b] Cause and effect statements about the rigor of performance standards and overall achievement are FALSE. For example, it is incorrect to claim that raising performance standards will increase student achievement.

[ps1] Few States Set World-Class Standards - Reaction. A reaction to the Peterson & Hess piece entitled "Check the Facts: Few States Set World-Class Standards" in Education Next, Summer 2008.

* It seems that "rigor" is a policy issue, but "achievement" is an operational issue. 

[3]

Please don't use NAEP scores to rank order the 50 states. (2005). Practical Assessment, Research & Evaluation, 10(9). Available online: http://pareonline.net/pdf/v10n9.pdf

[a] The uncertainty in NAEP data (i.e., the standard error) that is inherent in the sampling procedure is sufficiently large to render the use of rank order statistics inappropriate.

[b] The NAEP Data Explorer is an online data tool supported by the National Center for Education Statistics that is available for public use.  The tool uses an appropriate statistical procedure that lets the user identify a “focus jurisdiction” (i.e., nation, a particular state, D.C., etc.) and categorizes the remaining jurisdictions as either higher than, not different from, or lower than the focus jurisdiction.  The URL for the NAEP Data Explorer is: http://nces.ed.gov/nationsreportcard/nde/