Audits

8.4 – Use of Statistical Sampling for Overpayment Estimation (Rev. 906, Issued: 09-26-19; Effective: 01-02-19; Implementation: 01-02-19) 

This section applies to UPICs, RACs, the SMRC, and MACs (including DME MACs), hereinafter referred to as “contractors.” With regard to RACs and the SMRC, RACs and SMRC are required to receive CMS COR approval through each step of the statistical sampling process as certain steps/requirements may not be applicable or used in all instances. 

8.4.1 – Introduction (Rev. 377, Issued: 05-27-11, Effective: 06-28-11, Implementation: 06-28-11) 

8.4.1.1 – General Purpose (Rev. 906, Issued: 09-26-19; Effective: 01-02-19; Implementation: 01-02-19) The purpose of this section is to provide instructions for contractors on the use of statistical sampling in their reviews and estimation of overpayments to be recovered by recoupment, offset, or otherwise. The intent of these instructions is to ensure that a probability sample drawn from the sampling frame of the target population yields a valid estimate of an overpayment in the target population. This means that every element in the sampling frame has a non-zero probability of being selected. It is important to note that this is consistent with methodologies such as stratification and cluster sampling as needed to warrant statistically sound inferences from the sample. Reviews conducted by the contractors to assist law enforcement with the identification, case development, and/or investigation of suspected fraud or other unlawful activities may use sampling methodologies that differ from those prescribed herein. However, in those cases, the methodologies used shall be well-accepted methodologies amongst statisticians, and complete explanation shall be provided for why the methodology used was the appropriate methodology in the situation. 

These instructions are provided so that a sufficient process is followed when conducting statistical sampling to project overpayments. Failure by a contractor to follow one or more of the requirements contained herein does not necessarily affect the validity of the statistical sampling that was conducted or the projection of the overpayment. An appeal challenging the validity of the sampling methodology must be predicated on the actual statistical validity of the sample as drawn and conducted and must demonstrate actual error in the methodology that affects the overpayment amount. Failure by the contractor to follow one or more of the requirements contained herein may result in review by CMS of their performance, but should not be construed as necessarily affecting the validity of the statistical sampling and/or the projection of the overpayment.

8.4.1.2 – The Purpose of Statistical Sampling (Rev. 906; Issued: 09-26-19; Effective: 01-02-19; Implementation: 01-02-19) A statistical sample is used to estimate the amount of overpayment(s) made on claims. The Medicare Prescription Drug, Improvement, and Modernization Act of 2003 (MMA), mandates that before using extrapolation (i.e., projection, extension, or expansion of known data) to determine overpayment amounts to be recovered by recoupment, offset, or otherwise, there must be a determination of sustained or high level of payment error, or documentation that educational intervention has failed to correct the payment error. By law, the determination that a sustained or high level of payment error exists is not subject to administrative or judicial review. In this chapter, details are provided on the use of statistical estimation (sometimes, especially in legal contexts, referred to as “extrapolation”) to determine overpayment amounts. 

8.4.1.3 – Steps for Conducting Statistical Sampling

(Rev. 11962; Issued: 04-21-23; Effective: 05-22-23; Implementation: 05-22-23) 

The major steps in conducting statistical sampling are — (1) Identifying the provider/supplier; (2) Identifying the period to be reviewed; (3) Defining the universe (target population) and the sampling unit, and constructing the sampling frame; (4) Assessing the distribution of the paid amounts in the sampling frame to determine the sampling design; it is very likely that the distribution of the overpayments will not be normal. However, there are many sampling methodologies (for example, use of the Central Limit Theorem) that may be used to accommodate non-normal distributions. The statistician should state the assumptions being made about the distribution and explain the sampling methodology selected as a result of that distribution. (5) Performing the appropriate assessment(s) to determine whether the sample size is appropriate for the statistical analyses used, and identifying, relative to the sample size used, the corresponding confidence interval; (6) Designing the sampling plan and selecting the sample from the sampling frame; (7) Examining each of the sampling units and determining if there was an overpayment or an underpayment; and (8) Estimating the overpayment. When an overpayment has been determined to exist, the contractor shall follow applicable instructions for notification and collection of the overpayment, unless otherwise directed by CMS. For each step, the contractor shall provide complete and clear documentation sufficient to explain the action(s) taken in the step and to replicate, if needed, the statistical sampling.

8.4.1.4 – Determining When Statistical Sampling May Be Used (Rev. 12127; Issued: 07-21-23; Effective: 08-21-23; Implementation: 08-21-23) 

The contractor shall use statistical sampling when it has been determined that a sustained or high level of payment error exists. The use of statistical sampling may be used after documented educational intervention has failed to correct the payment error. For purposes of extrapolation, a sustained or high level of payment error shall be determined to exist through a variety of means, including, but not limited to: 

• high error rate determinations by the contractor or by other medical reviews compared to similar service providers; 
• provider/supplier history (i.e., prior history of non-compliance for the same or similar billing issues, or historical pattern of non-compliant billing practices); 
• CMS approval provided in connection to a payment suspension;
• information from law enforcement investigations; 
• allegations of wrongdoing by current or former employees of a provider/supplier; and/or
• audits or evaluations conducted by the OIG. 

If the contractor believes that statistical sampling and/or extrapolation should be used for purposes of estimation, and it does not meet any of the criteria listed above, it shall consult with its BFL, with a copy to the COR, as defined in PIM Chapter 4, §4.7 – Investigations, prior to creating a statistical sample and issuing a request for medical records from the provider/supplier. Examples of this may include, but are not limited to: billing for non-covered services, billing for services not rendered, etc. Extrapolation should not be used when the above criteria is not met unless prior approval is given by the COR and BFL. 

When an overpayment is identified by data analysis alone, the contractor shall consult with its BFL, with a copy to the COR. In addition, if CMS approves the data driven overpayment, the contractor shall also consult with its BFL, with a copy to the COR, on whether statistical sampling and extrapolation are necessary to identify the overpayment.

Additionally, a UPIC shall consult with the appropriate MAC on whether an extrapolated overpayment is more efficient in processing a data-driven overpayment before requesting recoupment from the MAC. 

Once a decision has been made that statistical sampling may be used, factors also to be considered for determining when to undertake statistical sampling for overpayment estimation instead of a claim-by-claim review, include, but are not limited to: the number of claims in the universe and the dollar values associated with those claims; available resources; and the cost effectiveness of the expected sampling results.

8.4.1.5 – Consultation With a Statistical Expert (Rev. 906; Issued: 09-26-19; Effective: 01-02-19; Implementation: 01-02-19) 

The sampling methodology used in estimations of overpayments must be reviewed and approved by a statistician or by a person with equivalent expertise in probability sampling and estimation methods. This is done to ensure that a statistically appropriate sample is drawn, and that appropriate methods for estimating the overpayments are followed. The contractor shall obtain from the statistical expert a written approval of the methodology for the type of statistical sampling to be performed. Regardless of whether this sampling methodology is applied routinely and repeatedly, each time a sample size calculation or estimation is performed, a detailed methodology (See Section 8.4.7.1.) should be submitted by the statistical expert to the corresponding contractor.

 Prior to releasing a findings letter or overpayment demand letter, the contractor shall have the statistical expert review the results of the sampling and any other subsequent overpayment estimation or extrapolation. The contractor shall verify that the statistical findings have been reviewed and agreed to by the contractor. 

If questions or issues arise, the contractor shall also involve the statistical expert.

At a minimum, the statistical expert (either on-staff or consultant) shall meet one of the following criteria: 

• Have significant coursework in probability and estimation methodologies, and at least 10 years of experience applying methods of statistical sampling and interpreting the results.
• Possesses a Bachelor’s degree (e.g., B.A., B.S.) in statistics or in some related field (e.g., psychometrics, biostatistics, econometrics, mathematics) with significant coursework in probability and estimation methodologies, and at least 6 years of experience applying methods of statistical sampling and interpreting the results. 
• Possesses a Master’s degree (e.g., M.A., M.S.) in statistics or in some related field with significant coursework in probability and estimation methodologies, and at least 4 years of experience applying methods of statistical sampling and interpreting the results. 
• Possess a Doctoral degree in statistics or in some related field with significant coursework in probability and estimation methodologies, and at least 1 year of experience applying methods of statistical sampling and interpreting the results

If the contractor does not have staff with sufficient statistical experience as outlined here, it shall obtain such expert assistance prior to conducting statistical sampling.

8.4.1.6 – Use of Other Sampling Methodologies (Rev. 906; Issued: 09-26-19; Effective: 01-02-19; Implementation: 01-02-19) 

Once it is has been determined that statistical sampling may be used, nothing in these instructions precludes CMS or its contractor from relying on statistically sound sampling methodologies employed by other law enforcement agencies, including but not limited to the OIG, the DOJ, and/or the FBI. In these cases, a full explanation shall be provided explaining why the methodology was used and why it was statistically appropriate in the circumstances. 

Where it is foreseen that the results of a contractor’s review may be referred to law enforcement or another agency to support litigation or other law enforcement actions, the contractor shall discuss specific litigation and/or other requirements as they relate to statistical sampling with its statistical expert prior to undertaking the review to ensure that the review will meet their requirements and that such work will be funded accordingly. 

8.4.2 – Probability Sampling (Rev. 11962; Issued: 04-21-23; Effective: 05-22-23; Implementation: 05-22-23)

Regardless of the method of sample selection used, the contractor shall follow a procedure that results in a probability sample. For a procedure to be classified as probability sampling, the following two features must apply: 

• It must be possible, in principle, to enumerate a set of distinct samples that the procedure is capable of selecting if applied to the target universe. Although only one sample will be selected, each distinct sample of the set has a known probability of selection. It is not necessary to actually carry out the enumeration or calculate the probabilities. All that is required is that one could, in principle, write down the samples, the sampling units contained therein, and the relevant probabilities; and 
• Each sampling unit in each distinct possible sample must have a known probability of selection. In the case of statistical sampling for overpayment estimation, one of the possible samples is selected by a random process according to which each sampling unit in the target population receives its appropriate chance of selection. The selection probabilities do not have to be equal but they should all be greater than zero. In fact, some designs bring gains in efficiency by not assigning equal probabilities to all of the distinct sampling units. 

Once a procedure and design that satisfies these above properties has been selected, execution of the probability sampling may occur. If a particular probability sampling design is properly executed, i.e., defining the universe, the sampling frame, the sampling units, using proper randomization, accurately measuring the variables of interest, and using the correct formulas for estimation, then assertions that the sample or that the resulting estimates are “not statistically valid” cannot legitimately be made. In other words, a probability sample and its results are always “valid.” However, because of differences in the choice of a design, the level of available resources, and the method of estimation, some procedures lead to higher precision (smaller confidence intervals) than other methods. A feature of probability sampling is that the level of uncertainty can be incorporated into the estimate of overpayment as is discussed below. 

8.4.3 – Selection of Period to be Reviewed and Composition of Universe

(Rev. 377, Issued: 05-27-11, Effective: 06-28-11, Implementation: 06-28-11)

 8.4.3.1 – Selection of Period for Review (Rev. 906; Issued: 09-26-19; Effective: 01-02-19; Implementation: 01-02-19)

Following selection of the provider or supplier, the contractor in conjunction with a statistical expert shall determine the time period and the number of days, weeks, months, or years for which sampling units will be reviewed. For RACs and SMRC, CMS will approve the time period. The target universe shall be selected based on these criteria. The scope of the review is determined by considering several factors that include, but are not limited to: • How long the pattern of sustained or high level of payment error is believed to have existed,

• The volume of claims that are involved,
• The length of time that a national coverage decision or local coverage determination has been in effect, 
• The extent of prepayment review already conducted or currently being conducted, • The dollar value of the claims that are involved relative to the cost of the sample, 
• The applicable time periods for reopening claims (see Pub. 100-04, chapter 34, section 10.6), 
• A clear specification of the specific data elements to be used for defining the date (type of date, e.g., day, month, year).

As examples: If dates of service are used, specify, as needed, whether the dates are line service dates or claim service dates, the first day of the billing statement or the ending date of service, etc. If a span of dates is used, clarify the criteria used when the span overlaps the boundary of one of the other criterion date ranges. If paid dates or receipt dates are used, clarify the role of the adjusted claims. 

All case documentation, including the overpayment demand letter and methodology, should consistently reflect the same date range and specify the same type of date. 

NOTE: When sampling claims that are paid through cost report (as opposed to claims paid under a PPS reimbursement methodology), all claims reviewed must be drawn from within a provider’s/supplier’s defined cost reporting year. If the period under review is greater than one year, it is important to select a separate sample for each cost-reporting year.

8.4.3.2 – Defining the Universe, the Sampling Unit, and the Sampling Frame (Rev. 11962; Issued: 04-21-23; Effective: 05-22-23; Implementation: 05-22-23) 

The universe is the target population that contains all claims/claim lines potentially under review and is used to construct the sampling frame. 

The sampling frame lists all sampling units which may be selected by the statistical sampling software, and further refines the review criteria from the claims/claim lines listed in the universe. The sampling unit may be the claim line, or may be a higher-level unit such as: 

1. The claim/claim number, or 
2. A cluster of claims/claim lines associated with a patient, or 
3. A cluster of claims/claim lines associated with a treatment “day,” or 
4. Any other sampling unit appropriate for the issue under review. 

The auditor may refine the selection criteria during the construction of the sampling frame, for example: 

1. Excluding claims/claim lines that have been subject to a prior review, or 
2. Excluding claims/claim lines for which there was no payment, or 
3. Excluding claims/claim lines which cannot be assigned to a sampling unit due to missing information. 

The extrapolation estimate of total overpayments is an estimate of total overpayment for sampling units in the sampling frame. 

All information needed to recreate the sampling frame and sample shall be included in the case documentation. 

Other approaches to constructing the universe and sampling frame are possible depending on the specific circumstances. One possibility is that the sampling frame may be created first (for example, a list of beneficiaries) and then the universe corresponding to the sampling frame may be constructed by querying claims history for the matching claims. Regardless of the process that is followed, the documentation in the case file must include a list of all sampling units in the sampling frame, all the universe elements that are incorporated into those sampling units, and the elements in the universe. It must be possible to assemble the sampling units from the universe during the replication process.

8.4.3.2.1 – Composition of the Universe (Rev. 11962; Issued: 04-21-23; Effective: 05-22-23; Implementation: 05-22-23) A.

 Part A Claims: For providers/suppliers reimbursed through cost report, the universe of claims from which the sample is selected shall consist of fully and partially adjudicated claims obtained from the shared systems that meet the criteria in the definition of the universe. For such claims, use the service date to match findings to the cost report. 

For providers/suppliers reimbursed under PPS, the universe of claims from which the sample is selected will consist of all fully and partially paid claims submitted by the provider/supplier for the period under review. Sampling units with no final payment made at the time of sample selection should not be included in the sampling frame. Claims with no payment may be included in the universe from which the sampling frame is constructed and should be excluded when establishing the sampling frame. 

B. Part B Claims: The universe shall consist of all fully and partially paid claims submitted by the provider/supplier for the period selected for review and for the sampling units to be reviewed. For example, if the review is of Physician X for the period January 1, 2002 through March 31, 2002, and laboratory and other diagnostic tests have been selected for review, the universe would include all fully and partially paid claims for laboratory and diagnostic tests billed by that physician for the selected time period. For some reviews, the period of review may best be defined in terms of the date(s) of service because changes in coverage policy may have occurred. Sampling units with no final payment made at the time of sample selection should not be included in the sampling frame. Claims with no payment may be included in the universe from which the same frame is constructed. 

8.4.3.2.2 – The Sampling Unit (Rev. 11962; Issued: 04-21-23; Effective: 05-22-23; Implementation: 05-22-23)

Sampling units are the elements that are selected based on the chosen method of statistical sampling. They may be an individual line(s) within claims, individual claims, or clusters of claims (e.g., a beneficiary). For example, possible sampling units may include specific beneficiaries seen by a physician during the time period under review, or claims for a specific item or service. In certain circumstances (e.g., multi-stage sampling designs), other types of clusters of payments may be used. 

Certain sampling theorems require an assumption that sampled items are “identically and independently distributed” (iid). In sampling from a finite universe without replacement, there is always a certain amount of dependence because the probability of selection changes with each unit that is selected. However, correlations of characteristics in the target population do not imply dependence in sampling. Sampling units may be correlated because they come from the same location, the same provider/supplier, the same time period, or any number of other reasons. In this context, independence means the selection of one sampling unit does not influence, or gives no information about, the outcome of another selection. Overpayments are not random variables. They are fixed values, though unknown prior to sampling. Therefore, regardless of any correlation that may exist between sampling units, the outcome, or overpayment, of any particular unit does not change based on the outcomes of other units. 

Unlike procedures for suppliers, overpayment estimation and recovery procedures for providers/suppliers and non-physician practitioners who bill Part A MACs, in a non-PPS environment, must be designed so that overpayment amounts can be accurately reflected on the provider’s cost report. Therefore, sampling units must coincide with an estimation methodology designed specifically for that type of provider/supplier to ensure that the results can be placed at the appropriate points on the cost report. The sample may be either claim-based or composed of specific line items. For example, home health cost reports are determined in units of “visits” for disciplines 1 through 6 and “lower of costs or charges” for drugs, supplies, etc. If claims are paid under cost report, the services reviewed and how those units link to the provider/supplier’s cost report must be known. The contractor shall follow the instructions contained in section 8.4 et seq., but use the projection methodologies provided in Pub. 100-08, Exhibits 9 through 12, for the appropriate provider type. Pub. 100-08, Exhibits 9 through 12, are to be used only for claims not paid under 

PPS. 8.4.3.2.3 – The Sampling Frame (Rev. 11962; Issued: 04-21-23; Effective: 05-22-23; Implementation: 05-22-23)

The sampling frame is the set of all the possible sampling units from which the sample is selected. As examples, the frame may be a list of all beneficiaries receiving items from a selected supplier, a list of all claims for which fully or partially favorable determinations have been issued, or a list of all the line items for specific items or services for which fully or partially favorable determinations have been issued. . 

8.4.4 – Sample Selection (Rev. 377, Issued: 05-27-11, Effective: 06-28-11, Implementation: 06-28-11) 

8.4.4.1 – Sampling Methodology (Rev. 906; Issued: 09-26-19; Effective: 01-02-19; Implementation: 01-02-19) 

The contractor shall identify the sampling methodology to be followed. There are various ways a probability sample can be generated. The most appropriate method will depend on factors such as, the target universe and the resources available for sampling. (Refer to section 8.4.1.5 of this chapter regarding consultations with a statistical expert to determine the appropriate methodology.) 

8.4.4.2 – Random Number Selection (Rev. 906; Issued: 09-26-19; Effective: 01-02-19; Implementation: 01-02-19)

The contractor shall identify the source of the random numbers used to select the individual sampling units, if used. The contractor shall also document the program and its algorithm or table, when available, that is used; this documentation becomes part of the record of the sampling and must be available for review. The contractor shall document any starting point if using a random number table or drawing a systematic sample. In addition, the contractor shall document the known seed value if a computer algorithm is used. The contractor shall document all steps taken in the random selection process exactly as done to ensure that the necessary information is available for anyone attempting to replicate the sample selection. 

There are a number of well-known, reputable software statistical packages (SPSS, SAS, etc.) and tables that may be used for generating a sample. One such package is RATSTATS, available through the HHS-OIG website. It is emphasized that the different packages offer a variety of programs for sample generation and do not all contain the same program features or the same ease in operation. For any particular problem, the contractor’s statistician or systems programmer shall determine which package is best suited to the problem being reviewed. 

8.4.4.3 – Determining Sample Size (Rev. 11962; Issued: 04-21-23; Effective: 05-22-23; Implementation: 05-22-23)

The size of the sample (i.e., the number of sampling units constituting the sample) will have a direct bearing on the precision of the estimated overpayment, but it is not the only factor that influences precision. It is neither possible nor desirable to specify a minimum sample size that applies to all situations.

 In addition to the above considerations, real-world economic constraints shall be taken into account. As stated earlier, sampling is used when it is not administratively feasible to review the entire target population. In determining the sample size to be used, the contractor shall also consider its available resources. That does not mean, however, that the resulting estimate of overpayment is not valid, so long as proper procedures for the execution of probability sampling and overpayment estimation have been followed. Some challenges to the validity of the sample that are sometimes made include whether — (1) The probability sample was chosen and drawn in a statistically appropriate way from the target population or (2) The particular sample size is too small to yield meaningful results. Such challenges are without merit when presented in isolation from any reference to the actual sampling methodology used, and when presented without a complete account of the actual sampling methodology used. 

8.4.4.4 – Documentation of Sampling Methodology (Rev. 906; Issued: 09-26-19; Effective: 01-02-19; Implementation: 01-02-19) 

The contractor shall maintain complete documentation of the sampling methodology that was followed. 

8.4.4.4.1 – Documentation of Universe and Sampling Frame (Rev. 11962; Issued: 04-21-23; Effective: 05-22-23; Implementation: 05-22-23)

An explicit statement of how the universe is defined and elements included shall be made and maintained in writing. Further, the sampling frame and specific details as to the period covered, definition of the sampling unit(s), identifiers for the sampling units (e.g., claim numbers, carrier control numbers), and dates of service and source shall be specified and recorded in the contractor’s record of how the sampling was done. If the sampling frame does not contain the elements used to define the universe because the sampling unit does not permit it, then an electronic copy of the universe will be kept by the contractor. 

A record shall be kept of the random numbers used (if used) in the sample and how they were selected. Documentation shall be kept in sufficient detail so that the sampling frame can be re-created should the methodology be challenged. The contractor shall keep an electronic copy of the sampling frame.