Macquarie Uni’s MUSEC Briefing # 28: Academic vs Real World Evidence

Picture of David Stanley
David Stanley
Fast ForWord, For Principals

Question.jpgIf you are considering a product to help your brain or improve academic performance, what evidence would you rely on?

Many people don't want to read research and thus seek a trusted advisor. Sadly, they are often unaware of potential conflicts of interest.

A case in point is the Macquarie University's Special Education Centre (MUSEC) brief discussed herein, where Macquarie University crudely used one meta-analysis to support their commercial initiative.

Can you rely on MUSEC for an independent, impartial and unbiased opinion?

What about the practical, real world gold standard evidence: 20+ years of product validation by millions of users around the world:

In 1996 four world leaders in neuroscience, after 25 years of ground-breaking research, formed a company (Scientific Learning Corporation). Their core product - Fast ForWord® translates neuroplasticity-based training research into educational programs to develop learning capacity and reading skills. It has been continually revised and enhanced ever since.

  • Why is it still going strong 20 years later with primarily the same product, Fast ForWord, still selling?  With 45 years of research in total and 20 years of use to date, there must be some substance to the product.
  • Why have the various American state education systems spent hundreds of millions of dollars on Fast ForWord over the last decades, and then after reviewing the results continue to spend tens of millions of dollars buying it each year?

Nevada study.pngKeep in mind the state and local education agencies are constantly reviewing the results of where they spend taxpayer money. And they keep buying Fast ForWord (an example is the Nevada State Education system's review of 24 education products used in their schools. That review concluded that Fast ForWord was the “Highest-Gain product”).

Further, there are several hundred school-results studies available in a download at the end of this document. See also this comprehensive list of school results by (USA) state.

Let’s consider:

  • If multiple state and local education agencies spend millions of dollars on a product and repeat the purchases across many years, maybe it’s because they see the product working.
    The_Brain_That_Changes_Itself.jpgAre student brains different in New Zealand & Australia from any other first world English language speaking country?

    The evidence from New Zealand & Australian schools that use Fast ForWord would suggest the brains of our students have the same potential for making use of brain plasticity as students elsewhere in the world do. [for a commentary on Brain Plasticity read Chapter 3 of “The Brain That Changes Itself” by Dr Norman Doidge]. The practical results other education institutions get around the world are repeatable here – and 50 other countries’ education systems can’t all be wrong.
  • Schools, therapists and parents wanting to help their children have spent hundreds of millions of dollars because Fast ForWord works virtually every time the protocols are adhered to and the two Language & Cognitive enhancement modules are completed.

 

That’s our gold standard evidence – proven results people are prepared to pay for.

The Fast ForWord program is supported by hundreds of research studies, of various types, showing that, when used according to recommended-use protocols, it helps build the language & cognitive skills necessary for reading and learning capacity development. And it can ameliorate learning challenges experienced by many special needs students.

Sceptics, academics, journalists & businesses

As is usually the case in any environment regarding the interpretation of research results, there are a handful of sceptics, including some notable voices in Australia.  The aim of this document is to highlight some of the background around the sceptics’ views. And to encourage you to read the research papers fully, not just their conclusions. Then contrast opinions of the sceptics, including MUSEC, with the overwhelming and ongoing evidence from around the world of the efficacy of Fast ForWord .

Keep in mind the imperative for many academics is to get published so they can gain more funding and enhance their academic reputation. Similarly, the goal of journalists is to gain a broader readership and enhance their reputation. The goal of a business is to stay in business and over the long haul that is hard to do if your core product doesn’t “work”.

If a company makes a product which has survived 20 years in the market place, has been used by more than 2.5 million students in over 50 countries, and had hundreds of school studies and academic research papers written about effective results, do you think the product might work?

It is important to understand that if someone wants to find a reason to say “no" to any product, proposition or idea, they will “conveniently” ignore all the evidence that doesn’t support their argument. Academics trying to prove their point can be very good at this.

Macquarie University, MUSEC & MultiLit

For example, let's consider Macquarie University's MUSEC briefing # 28. 

In 1995, Professor Kevin Wheldall from Macquarie University's Special Education Centre (MUSEC) launched a commercial initiative, Making Up Lost Time In Literacy (or MultiLit) at MUSEC.

In 2011, Genevieve McArthur - now a professor at Macquarie University - authored a MUSEC briefing on Fast ForWord Language saying “Not recommended”.

In this MUSEC briefing Genevieve McArthur, under the heading " What is the evidence for its efficacy? “ only refers to Strong et al (2010), a meta-analysis - not the hundreds of other published articles and case studies on Fast ForWord. I will discuss "meta-analyses" in detail later.

To the best of our knowledge neither Genevieve McArthur nor the MUSEC has ever supervised a group of students completing the Fast ForWord programs.

The gold standard of evidence

There are two very salient points in considering academic vs "real world" evidence:

  • Firstly, in the real world the most important question is: “Is the product viable and sustainable i.e. the practical test - has it stood the test of time and user experience?”
  • Secondly, academic researchers are very good at setting up theoretical experiments to isolate one variable and their gold standard is a “randomised controlled trial”. But how practical is it to “isolate” a body of students from other brain stimulation over several weeks or months? Alternatively, when reviewing a bank of research, what are the “academic" considerations of a meta-analysis?

Meta-analysis: research about previous research

A meta-analysis is a statistical method for comparing the results of multiple research studies on the validity of that research. It does not aim to assess the efficacy of a product. As it says in Wikipedia: meta-analysis can be thought of as "conducting research about previous research”.

It is an “academic” exercise – the product is not used.

It is just a review of how the studies were constructed and an analysis and critique of them.

A meta-analysis doesn’t add to the knowledge base nor prove or disprove the validity of any product. The academic researcher has merely to conform to the reviewers (examiners) process checklist to determine: "did the researcher perform the research rituals adequately?”

How the academic researcher chooses his or her selection criteria and exclusion criteria (i.e. what papers and research will they include or exclude) can directly influence whether it is a positive or negative outcome.

Strong et al (2010) meta-analysis

Over recent years sceptics of Fast ForWord, including a few academics, point to the Strong et al (2010) meta-analysis as “proof” that Fast ForWord does not work.

So it might be helpful to discuss Strong et al and why relying on it, in isolation, is a flawed basis for an opinion about Fast ForWord’s effectiveness. Unless of course all you are looking for is a reason to say “no”.

Firstly, Strong et al did not use the product (Fast ForWord).

Secondly, their exclusion criteria is most interesting:

  • No school case studies were considered - ironically, what teachers like is hearing about “how does it work in a school setting?”
  • Independent education department result-reviews were excluded.
  • Merzenich Tallal.pngAny research involving any of the four founders of Scientific Learning was excluded (even though they are world leaders in their fields and each has been published in peer reviewed journals many more times than Strong et al). The credentials of these scientists are such that one of the founders (Dr Paula Tallal) won the Thomas Alvin Edison Patent Award for innovative research and has presented to the US House of Representatives several times as an expert on language and learning disorders. Another (Dr Michael Merzenich) was subsequently awarded the Kavli prize for neuroscience and is often referred to as the “father of brain plasticity”.
  • Further irony is that if a researcher survives peer review and gets published then it is generally considered that the research was valid. But Strong et al found a way to exclude it all.

Of the hundreds of published reports about Fast ForWord - disregarding the several hundred publically available case studies of school and school system results – using extremely restrictive inclusion criteria, Strong et al got it down to just 6 papers and only 5 of those were used for the meta-analysis.

Five of the six studies selected for Strong et al’s review had poor implementations as explained below. Fast ForWord is more than just the software. It is software which needs to be used in a precisely prescribed way (according to research based implementation protocols regarding how often and for how long it is used in each training session and the meeting of completion benchmarks). If you don't use the research proven protocol, what result would you expect to get?

This is similar to medicine and physical training.  If a doctor prescribes one pill every four hours for three weeks, and the patient just takes two a day for one week, what result would you expect?  If the neighbourhood gym subscription is purchased, but only used a handful of times each month, what result would you expect?

** What does the selection/exclusion criteria tell you about the intent of the researchers -
are they aiming for a positive or negative outcome? **


In two of the studies chosen (Borman et al, 2009; Rouse & Krueger, 2004), the researchers acknowledged their implementation problems and conducted additional analyses to examine the relationship between Fast ForWord product use and reading gains. Both groups subsequently found greater impacts on reading scores among students who had better product use. For example, Borman et al found a statistically significant effect of program completion on reading comprehension:

"Completing the program had a moderate to large impact on 
reading comprehension (effect size of d = 0.50)".


Strong et al did not include the above additional analysis. Why?

Of the studies included in the Strong et al meta-analysis, Gillam, et al’s 2008 study had the best implementation. In that study: 

“Students who used the Fast ForWord Language product achieved statistically significant improvements in language and reading skills – improvements comparable to receiving 50 hours of one-on-one intervention with a certified and licensed speech and language therapist.”

What a cost saving for a family or a school would it be to get a comparable result 
by using Fast ForWord rather than “50 hours of one-on-one intervention with 
a certified and licensed speech and language therapist”
 .


Discussing the results of the Gillam, et al (2008) study, the lead author noted:

“It is clear that a large majority of the children in our study who received treatment with Fast ForWord Language showed substantial improvements, reversing a long-time trend... 74% of the children in our study who received Fast ForWord Language had follow-up scores that were significantly greater than their pre-test scores six months after treatment ended. I judge that to be a substantial benefit.”

Overall, the studies reviewed by Strong et al illustrate that Fast ForWord programs positively impact students’ language and reading skills – but only if students actually use the programs as intended.

The irony is that the meta-analysis by Strong et al conclusively showed that if Fast ForWord is poorly implemented, ie protocol fidelity is poor, then as the authors concluded: “there was no significant effect of Fast ForWord on any outcome measure in comparison to active or untreated control groups.” Strong et al did not append “if poorly implemented” to their conclusion.

Hear Dr Martha Burns Speak on Interpreting Conflicting Research

The meta-analysis is the academic methodology, but what about the quality of the 6 reports chosen?  3 of the reports are mentioned above. Additionally: 

Cohen et al

In the Cohen et al study:

  • Students participated at home and there was considerable variability in their Fast ForWord usage – student use ranged from 7 to 42 days.
  • Students were supposed to be tested three times (before, after, and follow-up). Estimates were used in cases of missing scores.
Given the wide variability in program use, estimates for scores and the inclusion of students who used the programs for fewer than 10 days, it is not surprising that the average gains were not large enough to reach statistical significance.

Pokorni et al (2004)

Three interventions were included in this study:

  1. 20 students were assigned to the Fast ForWord Language product
  2. Some other students were assigned to use Lindamood Phoneme Sequencing (LiPS)
  3. The rest used Earobics.

Numerous prior studies have found improvements attributable to all three of these interventions. 

However, Pokorni et al found almost no effect for any of the three, suggesting that there were systemic implementation problems across all of the interventions.

In fact, the unorthodox protocol for product use created by the researchers called for students to complete three one - hour sessions each day for 20 days.

Product use information was not included in the report, but it seems likely that students were unable to stay motivated and complete this highly non-standard protocol.

** What does the selection/exclusion criteria tell you about the intent of the researchers -
are they aiming for a positive or negative outcome? **


An extensive list of published research and case studies from the last 20 years is available here.

Randomised controlled trials in schools

Separate to the above, in discussions with academic researchers about school results versus “randomised controlled trials”, the academic researchers often argue that the school results are not statistically valid because there could be any number of factors as to why the students had gains.

However, it's interesting to consider how could a “randomised controlled trial” of students doing Fast ForWord for 30-50 minutes a day, occur over several terms without the students going to school or having any other brain stimulus?

The thousands of sets of results, even with so many variables over 20 years, show one common element: the results of students who adhered to protocol AND completed the two Fast ForWord Language & Cognitive enhancement modules, improved irrespective of what else they were doing.

That volume of practical evidence is the gold standard you too can choose.

 Download "Fast ForWord - Does it work? What's the Evidence" -

Download button-1