This systematic literature review was conducted according to the preferred reporting items for systematic reviews and meta-analyses (PRISMA) guidelines (13).

Relevant medical databases, including PubMed, MEDLINE, EMBASE, Cochrane and Google Scholar were searched for publications related to the assessment of ultrasound skills up to May 2021, while keywords were predetermined by the authors (Figure 1). Titles and abstracts were analyzed for possible inclusion. In addition, reference lists of the identified articles were investigated for further potential inclusion. Agreement regarding potential relevance was reached by consensus and full text copies of relevant papers were obtained. The key questions of this systematic literature review were framed in the PICOS format (Population, Intervention, Comparator, Outcome, and Study design) as detailed in Table 1.

Articles meeting the following criteria were suitable for inclusion:

Exclusion criteria were:

The different assessment formats were structured in subclasses, which are the following: theoretical knowledge (written or online examination, multiple choice questions (MCQ) or essay questions), practical examination skills [e.g., US acquisition (US image rating), observed simulated clinical encounters (Objective Structured Clinical Examination (OSCEs), The Objective Structured Assessment of Ultrasound Skills (OSAUS), Direct Observation of Procedural Skills (DOPS))], and self-assessment (e.g., surveys regarding satisfaction and competence).

Data originated from full-text articles is presented in a structured table (Table 2) to illustrate the various assessment formats in ultrasound education and to give selected examples for the outlined evaluation methods.

In many cases evaluation is based on self-assessment using surveys or questionnaires, sometimes in a pre-/post course design. Self-assessment can help to evaluate the student's thoughts and manners when learning and it can identify tactics that enhance better understanding and improvement of skills.

Further, students can rate their own competence, which might motivate them to improve their skills as they detect incongruity between present and wanted performance. According to one study, self-evaluation can help students improve their critical thinking skills (52) further it might encourage reflection on personal performance (53).

Creating surveys is time- and cost-effective and the evaluator does not essentially have to be a specialist. Since the evaluation is based on subjective data when using self-assessment, there is the threat of discrepancy between actual performance and answers given in the survey. Students might not rate their actual performance competence but the effort they put into the course (54).

There is no acquisition of genuine knowledge and competence in examination as no objective data is generated. Additionally, there is a risk of decreased validity by response bias which are prevalent in research involving participant's self-report. For example, the study results can be influenced by acquiescence bias, which belongs to response bias and describes that participants in a survey have the tendency to agree with the asked questions.

The objective structured clinical examination (OSCE) was developed by Harden and colleagues in 1975 (55). The idea is rotating through multiple stations in a simulated clinical setting. Each of these stations challenges the student to solve a special task in a pre-specified time in order to test clinical skill performance. While the students carry out the examination they are observed by one or two assessors who rate the student's performance using checklists which have been developed in advance.

OSCE has been widely accepted as an objective form of assessing clinical competences (56) and is used in various specializations and clinical tasks.

It allows the assessment of scanning technique and image interpretation in real time and combines the evaluation of technical skills and theoretical knowledge. Further, the possibility for direct feedback on the student's performance is provided and it aims to prepare students for daily clinical practice. There are different assessors with every station and the students should rotate so that they should all have the same time and tasks to bring fairness.

Within an OSCE, different forms of assessment can be combined since e.g., at one station case- based US images could be diagnosed while the next task requires the students to examine a patient to obtain own images to observe the student's probe handling and scanning technique.

In general, the checklist for ultrasound can include various aspects, which have been defined prior to the assessment. Items that are often included are e.g., positioning of the patient as well as interaction with the patient, positioning/handling/orientation of the ultrasound probe, image adjustment, and interpretation. A different OSCE station is needed for each organ as the protocols are individually tailored.

Therefore, the large number of stations and the various protocols to test different organs as well as the required educated assessors may exceed the available resources (36). OSCEs require staff, equipment, clinical laboratories as well as long preparations. Furthermore, to avoid bias, two assessors each station would be preferable, resulting in even higher cost- and time investment. OSCEs might not display a realistic hospital setting (57).

As the time allotted to each station is predetermined, students may become stressed and be unable to complete the task to their own expectations owing to time constraints. However, limited time may compel a higher level of training motivation and motivate students to practice more in order to achieve satisfactory outcomes (7).

The concept of direct observation of procedural skills (DOPS) was developed by the Royal Medical College of England. The assessor observes the student during the clinical procedure on a real patient and gives feedback afterwards, it is a workplace-based assessment method.

Further, DOPS combines learning, supervision, rating and feedback (58) and can be both formative and summative (59).

Designing specific protocols for grading facilitates detailed feedback and fairness since the assessor has the same base to rate the different students.

The use of workplace-based assessment is useful since it determines not only the students' learning achievements but also their attempt to assume professional responsibilities (60). Scanning technique and image interpretation can be assessed in real time and therefore theoretical knowledge as well as examination technique can be rated. DOPS have been used for ultrasound assessment and seems to be a reliable and valid method (36) and requires less assessment stations and resources than an OSCE format. However, an educated assessor is necessary for the evaluation of the student and it would be even better to have two independent raters. Further, in comparison to OSCE, DOPS is not widely established in ultrasound assessment and there might be more studies required to show its efficacy and validity.

The Objective Structured Assessment of Ultrasound Skills (OSAUS) was developed as an approach to achieve international consensus across various specialities on an evaluation tool for ultrasound education (9). Based on a delphi-consensus seven key points have been identified and included in the protocol.

These key points are: (1) Indication for the examination, (2) Applied knowledge of ultrasound equipment, (3) Image optimization, (4) Systematic examination, (5) Interpretation of images, (6) Documentation of examination, and (7) Medical decision making.

OSAUS can be used to assess US competence in different clinical settings and disciplines. It is a time-effective method as the protocol should be used universally there is no need of developing a new protocol for various specializations/organs. However, OSAUS is measuring general aspects and is not procedure specific.

The student is asked, to state the indication for the examination, as well as how the examination could help in further decision making and treatment. Therefore, the student learns the importance of ultrasound and the practical applications of this imaging method.

Further studies are required to examine the value of the protocol to assess ultrasound competence in different fields and clinical settings.

Examinations in form of multiple choice (MCQ) or written questions are often used additionally to a direct observation of scanning technique (26, 27, 61).

Using MCQ is objective and the questions could be included in any existing examination.

Further, no educated assessor is required for the real-time assessment of skills.

Due to the absence of an ultrasound examination, evaluating scanning method and image optimization is not feasible. Whereas, ultrasound is a technical skill, the assessment with MCQ rather checks theoretical knowledge.

Including pictures and case-based assessments are more opportunities for the assessment of ultrasound knowledge. Here, students have to detect pathologic findings in US images and have to connect them to clinical cases and further clinical applications.

Short duration and case based presentations can increase the knowledge maintained after 2 weeks in learning (37).

Case based learning (CBL) is a teaching method which finds application in multiple medical fields using case vignettes to convey relevance and to connect theory to practice (62).

It is objective and does not require any special educated assessor. Just like MCQ, case-based questions can be incorporated into an already existing exam.

Since it is a cost- and time-effective tool it can be well-used when resources are limited.

However, probe handling, scanning technique and image adjustment cannot be evaluated with this method of assessment and would be necessary to further improve the students' competence.

Several studies have shown that simulation-based ultrasound training can lead to better clinical performance not only regarding diagnostic accuracy, but further students seem to need less supervision (63). Ultrasound simulators are important for training in anesthesia and gynecology and can be used for the assessment of competence (41). They are used especially for rather advanced exercises e.g., for the incorporation of central venous catheters or practicing regional anesthesia (64) as they provide the possibility to practice complex tasks prior to the performance with patients. Simulators can offer standardized and valid measurement of skills that can be compared not only nationally but globally (41). The clinical setting is missing and the interaction with the patient cannot be evaluated. On the other hand, the assessment in a clinical setting requires expenditure not every university can afford.

When the number of simulators is limited, the assessment can be affected if the students memorized the correct position. Since anatomy differs, the learning effect when scanning patients might be different.

In this assessment format students as well as experts were asked to perform the same examination and results were compared afterwards.

One study trained students to measure the liver size using ultrasound. Experienced physicians were asked to measure liver span with standard examination methods. Afterwards the results were compared to the student's findings to evaluate if the course was effective (44). Another study compared the precision of cardiovascular diagnoses by medical students using a mobile ultrasound device with the findings of cardiologists which were using standard physical examinations (43). This assessment form provides a clinical setting and students get to perform examinations with real patients. Theoretical knowledge cannot be assessed neither can scanning technique or probe handling since only the resulting image is rated. In this case the students do not get direct feedback on their performance.

In some studies US images of the students were rated by predetermined criteria. For example images from the pretraining scanning examination and the images from the post-training scanning examination were stored and then compared for improvement (46). In other studies the image quality was live-evaluated and students got a different score, depending on their ability to visualize the organ (47, 48).

There is no global accepted rating system of US images yet, even though the need for a standardized method to evaluate the quality of an US image is well-documented. The B-QUIET method for example represents such an approach to quantify the sonographer component of ultrasound images (65) (Table 3).