Telehealth and the BOT-2

Computers and connectivity

Two computers with audio and video capability and stable Internet connectivity—one for the examiner and one for the examinee with facilitator—are required. A web camera, microphone, and speakers or headphones are required for both the examiner and the examinee. A second computer screen or split-screen format on a large computer monitor for the examiner is helpful to allow a view of the digital Administration Directions and Stimulus Book, but the examiner can also use a paper Administration Easel. The second computer or large screen also tends to make sharing test content more straightforward for the examiner.

Image/screen size

For best alignment with standardized administration, the paper Administration Easel used to teach the items should be at the examinee’s location to present the visual stimuli using a professional facilitator. However, at times the examiner might find it useful to share the examinee-facing portion of the digital Stimulus Book during a testing session when only an untrained facilitator is available (e.g., when using a BOT–2 item for clinical observation). When items with visual stimuli are presented, the digital image of the visual stimuli on the examinee’s screen should be at least 9.7” measured diagonally, similar to an iPad or iPad Air. Some teleconferencing platforms shrink the size of images, so the facilitator should verify the image size in the initial virtual meeting. It is recommended that computer screens used for teleconference assessment be at least 15” measured diagonally. Smaller screens, such as those of iPad minis, small tablet PCs, and smartphones, are not allowed for examinee-facing content, as these have not been examined empirically and may affect stimulus presentation, examinee response, and validity of the test results. Similarly, presenting stimuli on extremely large screens has not been examined, so the same precaution applies. At the beginning of the testing session, the examiner may ask the facilitator to aim a peripheral camera or device (as described further below) at the examinee’s screen to ensure that the examinee’s screen is displaying images in the correct aspect ratio and not stretching or obscuring the stimuli image.

Teleconference platform

A teleconference platform is required. Screensharing capability is required if anything other than items with verbal stimuli are administered.

Video

High-quality video (HD preferred) is required during the administration. Make sure the full faces of the examiner and the examinee are seen using each respective web camera during items administered at the table. In addition, alternative camera positions are needed when administering gross motor items so the examinee’s full body and movements can be seen by the examiner for scoring. The teleconference platform should allow all relevant visual stimuli to be fully visible to the examinee when providing instruction or completing items; the view of the examiner should not impede the examinee’s view of visual test stimuli.

Screensharing digital components

Digital components are shared within the teleconferencing software as specified in Table 1. There are two ways to view digital components in the Q-global Resource Library: through the PDF viewer in the browser window or full screen in presentation mode. Always use full screen (i.e., presentation) mode for digital components viewed by the examinee. This provides the cleanest presentation of test content without onscreen distractions (e.g., extra toolbars). Refer to Using Your Digital Assets on Q-global in the Q-global Resource Library for complete directions on how to enter presentation mode.

Test item security in the audiovisual environment

The examiner is responsible for ensuring test item security is maintained, as outlined in the Terms and Conditions for test use. The examiner should address test security requirements with the examinee (and facilitator, if applicable) during the informed consent process. The examiner should make it clear that the video should not be captured, photos should not be taken, and stimuli should not be copied or recorded, as this is a copyright violation. The examinee must agree that they will not record (audio or visual) or take photos or screenshots of any portion of the test materials or testing session, and not permit anyone to observe the testing session or be in the testing room (except for a facilitator, when necessary). Any Response Booklets used in the testing session must be returned to the examiner (see Assessment Procedures and Materials factor for more information).

Peripheral camera or device

A stand-alone peripheral camera that can be positioned to provide a view of the session from another angle or a live view of the examinee’s progress is helpful. Alternately, a separate device (e.g., a smartphone with a camera or another peripheral device) can be connected to the teleconference and set in a stable position to show the examinee’s responses whether seated at the table or during gross motor tasks. The examiner should guide positioning of the peripheral camera/device before administering subtests so that the examiner can see that the examinee’s real-time responses are within view. The repositioning of the peripheral camera or device may be required multiple times during the test administration, even within the same subtest, to ensure the examiner can score the performance accurately. The device’s audio should be silenced and the microphone should be muted to prevent feedback.

In a typical telehealth session, it is more feasible to make a document or moveable camera available in the examinee’s location. However, while social distancing is necessary, the only camera available may be a stationary camera integrated into the examinee’s laptop or computer screen. It is unrealistic to expect examinees to have document cameras within their homes. It may be necessary for examiners to think creatively about how to use a smartphone in the examinee’s location to gain a view of the examinee’s progress in a Response Booklet or gross motor actions. Prior to attempting this with an examinee, the examiner should work to become fluid and competent at directing examinees in these methods, which can require extensive practice with varied individuals and types of smartphones. In addition, this requires planning and practice in the initial virtual meeting to prevent technical difficulties, and so the examinee feels confident doing this when it is time.

Online instructional videos (e.g., here) demonstrate how a smartphone may be used with common household objects (e.g., a tower or stack of books, paper weight, ruler, rubber band, tape) to create an improvised document camera for use during tasks involving Response Booklets. Typically, devices provide the best view of the examinee’s screen and pointing responses when positioned in landscape format. While using a smartphone as the peripheral camera may not be an optimal solution for telehealth, it can be functional if executed well.

Gesturing

When gesturing to the Stimulus Books is necessary, the examiner should display them as digital assets onscreen and point using the mouse cursor. It may on occasion be necessary for the examiner to gesture to areas of a paper copy of a Response Booklet or to show how to respond to demonstration items (e.g., Folding Paper) on the examiner’s camera. Refer to Table 1 for specific instructions by subtest.

Capturing Response Booklet performance (if used)

It is important to use the original Response Booklets for telehealth administration as these materials are required for standardized administration and are a custom size. Any copies or modifications will void the data collected and be a copyright infringement. Examiners should plan in advance to ensure enough time for the Response Booklet to be delivered when scheduling a time for the evaluation. During a telehealth administration, the examiner may ask for the completed Response Booklet to be shown on camera immediately at the conclusion of a task, so that the examiner can confirm item completion before presenting the next item. One successful approach to protecting test security uses sealed envelopes. In the sealed envelope method, the examiner gathers Response Booklets and a self-addressed stamped envelope. The examiner places these materials in another envelope, signs it on the seal, and then mails or delivers it to the testing location. The examiner should emphasize that the sealed envelope containing the Response Booklets must not be opened until the examiner asks. After the completion of each subtest, the Response Booklets are placed in the provided self-addressed stamped envelope. At the conclusion of administration, the self-addressed stamped envelope is sealed and signed on the seal on camera, and then mailed or delivered with the test kit to the examiner immediately for scoring.

Audio check

The examiner should test the audio for both the examiner and examinee in the initial virtual meeting and at the beginning of the testing session to ensure a high-quality audio environment is present. Testing the audio should include an informal conversation prior to the administration where the examiner listens for any clicks, pops, or breaks in the audio signal that distorts or interrupts the voice of the examinee. The examiner should also ask if there are any interruptions or distortions in the audio signal on the examinee’s end. Any connectivity lapses, distractions, or intrusions that occur during testing should be reported.

Manage audiovisual distractions

As with any testing session, the examiner should do everything possible to make sure the examinee’s environment is free from audio and visual distractions. If the examiner is unfamiliar with the examinee’s planned physical location, a visual tour of the intended testing area should be given during the initial virtual meeting. The examiner can then provide a list of issues to address to transform the environment into one suitable for testing. For example, remove distracting items, silence all electronics, and close doors. The examiner should confirm that these issues have been addressed at the time of testing. If possible, the examinee should be positioned facing away from the door to ensure the examiner can verify through the examinee’s camera that the door remains shut and can monitor any interruptions. The examiner should confirm that all other applications on the computer, laptop, or peripheral device are closed; the keyboard is moved aside or covered after the session is connected; and alerts and notifications are silenced on the peripheral device. Radios, televisions, other cellular phones, fax machines, smart speakers, printers, and equipment that emit noise must be silenced and/or removed from the room.

Lighting

Good overhead and facial lighting should be established for the examiner and examinee. Blinds or shades should be closed to reduce sun glare on faces and the computer screens.

Disruptions

The examiner should record any and all atypical events that occur during the testing session. This may include delayed audio or video, disruptions to connectivity, the examinee being distracted by external stimuli, and any other anomalies. These can be noted on the Record Form and should be considered during interpretation and described in the written report.

Copyright

Permission must be obtained for access to copyrighted materials (e.g., Stimulus Books, Response Booklets) as appropriate. Pearson has provided a  letter of No Objection to permit use of copyrighted materials for telehealth via teleconferencing software and tools to assist in remote administration of assessment content during the COVID-19 pandemic.

Response booklets (if used)

The Response Booklets should be provided in advance of the testing session and the plan for securing and forwarding/returning materials, real-time and after testing, should be communicated. See the capturing Response Booklet performance portion of the Telehealth Environment and Equipment section for suggested procedures to facilitate secure handling of Response Booklets.

Standardized manipulatives

The standardized manipulatives from the BOT–2 kit must be used for any administration of the test, whether in-person or remote administration. Substitutions for the provided manipulatives are not permitted for standardized administration. An alternative manipulative may not allow for the same examinee performance on the item and would result in a nonstandardized administration for which normative scores would not be valid.

Digital assets

The examiner should practice using the digital assets until the use of the materials is as smooth as a face-to-face administration. It is not recommended that the examiner display items from paper Stimulus Books on a camera.

Considerations

Review Table 1 for the specific telehealth considerations for each subtest to be administered.

Input and output requirements and equivalence evidence

The examiner should consider the input and output requirements for each task, and the evidence available for telehealth equivalence for the specific task type.

Telehealth Versus Face-to-Face Administration

Preliminary research has compared results obtained in telehealth and face-to-face administration modes for some assessment tools to determine equivalence. To date, no studies have investigated equivalence between these modes of administration for BOT–2. Standardized, norm-referenced gross motor assessments have not been researched for the pediatric population, but some gross motor assessments for adult clients have shown feasibility, inter- and intrarater reliability, or equivalency between these modes of administration (Dorsey et al., 2010; Hoffman et al., 2008; Hwang et al., 2016; Palsbo et al., 2007). Some standardized measures with fine or visual motor tasks included have been researched and shown feasibility, inter- and intrarater reliability, or equivalency between telehealth and face-to-face administration, but most research has again focused on adult examinees (Abdolahi et al., 2014; Dorsey et al., 2010; Hoffman et al., 2008; Hwang et al., 2016; Stillerova et al., 2016).

While equivalence data on similar measures are relevant, practitioners should be mindful that more research is needed to establish equivalence in all ages and for all tasks on the BOT–2. Most telehealth-based studies were conducted with volunteer subjects in controlled environments and did not include motor performance tasks. When social distancing is key (such as during the COVID-19 pandemic) some evaluations may need to occur in patients’ homes, and it should be noted that very little research has been done about remote assessment in private homes.

It is important to consider the conditions under which equivalence studies of telehealth and face-to-face assessment modes were conducted and attempt to reproduce these as closely as possible if testing via telehealth. Typical telehealth studies that support telehealth and face-to-face equivalence involve the examiner becoming very familiar with the teleconference platform by using it for its intended purpose for several hours and administering tests (even those that are familiar in face-to-face mode) multiple times to practice examinees. Some studies that have established telehealth and face-to-face mode equivalence involve a professional facilitator. However, preliminary research conducted and described by Lana Harder (Stolwyk et al., 2020) with parents serving as in-home facilitators who managed audiovisual needs and Response Booklets found no significant differences across modes. During most telehealth studies, though, the examinee is typically in an office- or school-based setting. Therefore, if in-home assessment is taking place, it is advisable to prepare a similar environment as much as possible as described in the Telehealth Environment and Equipment section.

Digital Versus Traditional Format

Telehealth involves the use of technology in assessment as well as viewing onscreen stimuli. For these reasons, studies that investigate assessment in digital versus traditional formats are also relevant.

Evidence by Subtest

Table 2 lists each BOT–2 subtest, the input and output requirements, the direct evidence of subtest equivalence in telehealth–face-to-face and digital–traditional investigations, and the evidence for similar tasks. The abbreviations in the Input and Output column correspond to the various input and output requirements of each subtest, and a key appears at the bottom of the table. For example, brief spoken directions as an input requirement is abbreviated as BSD. The numbers in the evidence columns correspond to the studies in the reference list, which is organized alphabetically in telehealth and digital sections. For clarity, each study is denoted either T or D, with T indicating the study investigated telehealth–face-to-face mode, and D indicating the study addressed digital–traditional format.

Appropriateness

The examiner should first ensure that a telehealth administration is appropriate for the examinee and for the purpose of the assessment. Clinical judgment, best practice guidance for telehealth (AOTA, 2020; APA Services, 2020; ASPPB, 2013; IOPC, 2020), information from professional organizations and other professional entities (e.g., licensing boards, legal resources, professional liability insurance providers, payors), consultation with other knowledgeable psychologists, existing research, and any available federal or state regulations should be considered in the decision-making process. Consideration should be given to whether the necessary administrative and technological tasks involved in a telehealth session can be accomplished without influencing results.

Preparedness

Before initiating test administration, the examiner should ensure that the examinee is well-rested, able, prepared, and ready to appropriately and fully participate in the testing session.

Facilitator role

The role of the facilitator must be explained to the examinee so participation and actions are understood.

Practice

During the telehealth setup, and before administering to any actual examinee, the examiner should rehearse the mechanics and workflow of every item in the entire test using the selected teleconference platform so that the examiner is familiar with the administration procedures. For example, colleagues could be used as a practice examinee and facilitator.

Standardized procedures

The examiner must follow the administration procedures of face-to-face administration as much as possible and use the standardized manipulatives and print materials from the test kit.

Administrative and technological tasks

In order to conduct a smooth telehealth session, audiovisual needs and materials must be managed appropriately. The initial virtual meeting involves the examiner and the facilitator, and is the opportunity for the examiner to provide information about the audiovisual needs and materials. During the initial virtual meeting, the examiner should provide training in troubleshooting audiovisual needs that arise during the testing session, including camera angle, lighting, and audio checks. The examiner should provide verbal feedback to guide camera adjustment, checking the onscreen video shown by the peripheral camera/device to provide information about how to reposition it until the proper view is shown. The examiner should emphasize that no materials should be opened or made available to the examinee until the examiner provides instructions to do so, if applicable. The examiner should also expect to provide verbal guidance about these issues during the testing session. Refer to the Telehealth Environment and Equipment section and to Table 1 for specific subtest telehealth considerations.

The facilitator should assist with administrative and technological tasks and not manage rapport, engagement, or attention during the testing session. The examiner should direct them not to interfere with the examinee’s performance or responses. Any other roles and responsibilities for which an examiner needs support, such as behavior management, should be outlined and trained prior to the beginning of the testing session. The examiner is responsible for documenting all behaviors of the facilitator during test administration and taking these into consideration when reporting scores and performance.

There are special considerations for written reports describing testing that takes place via telehealth.

The professional completing the written report should state that the test was administered via telehealth, and briefly describe the method of telehealth used. For example, “The BOT–2 was administered via remote telehealth and a facilitator monitored the administration on-site during the live video connection using the [name of telehealth system, e.g., Zoom] platform. For appropriate subtests, the Response Booklet was provided in advance to the on-site facilitator and returned to the examiner for scoring.”

The professional should also make a clinical judgment, similar to a face-to-face session, about whether or not the examiner was able to obtain the examinee’s best performance. Clinical decisions should be explained in the report, including comments on the factors that led to the decision to conduct testing via telehealth and to report all (or not to report suspect) scores. In addition, include a record of any and all atypical events during the testing session (e.g., delayed video or audio, disruptions to connectivity, extraneous noises such as phone ringing or loud dog barking, person or animal unexpectedly walking into room, the examinee responding to other external stimuli). Notes may be recorded about these issues on the Record Form. List and describe these anomalies as behavioral observations in the written report, as well as any observed or perceived impact on the testing sessions and/or results, and consider these in the interpretation of results. For example, “The remote testing environment appeared free of distractions, adequate rapport was established with the examinee via video/audio, and the examinee appeared appropriately engaged in the task throughout the session. No significant technological problems or distractions were noted during administration. Modifications to the standardization procedure included: [list]. As BOT–2 subtests have not received initial validation for remote administration in telehealth, results should be interpreted with caution and additional measures (e.g., questionnaires, clinical observations) should be used to determine if results are considered a valid description of the examinee’s skills and abilities.”