Hundreds of people per day in Armenia are tested for the coronavirus. But are those locally produced test kits reliable? Hetq has examined the production process and our findings raise a few concerns.

On April 6, 2020, when the coronavirus had started to spread in Armenia, Prime Minister Nikol Pashinyan spoke about the need for Armenian-made coronavirus PCR test kits.

He said that the government had allocated $550,000 to the Institute of Molecular Biology of the National Academy of Sciences, which was engaged in the production of Armenian test kits.

On June 24, Minister of Health Arsen Torosyan announced that the Armenian-produced test kits tests were ready.

"This is very significant progress for our medical sciences," the minister said at the time.

Hetq has researched the Armenian-made test kits and found out that the devices used in the production of the tests have not been inspected for accuracy. That is, there was no objective data on whether the measuring instruments met the requirements set by law.

Moreover, the accuracy and specificity of the tests were determined by another agency, the National Center for Disease Control and Prevention of the Ministry of Health, whose devices have also never been tested. To produce the test kits, methodologies crafted in January 2020 were selected. These methodologies were designed "to be used only in emergencies”.

What is verification and why is it necessary?

Before touching upon the details of the production of Armenian tests, let us understand what metrology (the scientific study of measurement) and verification are and why they are necessary in the healthcare sector.

Device testing is part of the science of metrology.

All this is referred to in the RA Law on Ensuring the Consistency of Measurements adopted in 2012.

Article 13 of the law stipulates that one of the spheres of legislative metrology is healthcare. That is, devices used in the healthcare sector must be tested.

For example, if a scale is said to weigh 10 grams and testing shows that it weighs 9.5 grams, there is a difference of 0.5g. The same holds for a thermometer that shows 36.9C when, in fact, the temperature is 37.05. It’s matter of preciseness. Verification is performed to ensure the reliability of the measurement results.

The only authorized body in Armenia to carry out inspections of this type is the National Standards and Metrology Authority (SCAM), an agency of the Ministry of Economy.

The same law stipulates that production must follow an approved methodology. This is a document that defines the rules for preparing the measurements, the measurement method, the technical means, as well as the procedure for processing and calculating the measurement result.

The use of unverified, non-metrologically certified or unapproved devices is prohibited.

Government decision N-113-N (February 11, 2016), "On defining the list of measuring instruments subject to legislative metrological control" defines the list of devices subject to mandatory verification.

According to that list, scales used in the production of tests, dispensers, analyzers, optical-physical measuring devices, etc. are subject to mandatory testing. In addition, the production of tests in one way or another relates to the health sector, which already makes it mandatory to test all devices used in production.

The Institute of Molecular Biology has never tested its devices and lacks robotics 

During a press conference on July 10, Institute of Molecular Biology (IMB) Director Arsen Arakelyan said that they "received the equivalent of $250,000 from the government for 100,000 tests." The IMB did not receive additional funding.

On June 10, the IMB (attached to the National Academy of Sciences) informed Hetq that it had already purchased materials from the Chinese Biotech Biomedicine Group Ltd. to produce COVID-19 Armenian test kits. Twenty IMB researchers are involved in the production of tests.

Hetq asked the IMB to indicate whether, prior to the production of the tests, it had carried out a verification of the PCR analyzer (amplifier) ​​used in the production of the tests (included in the list of devices to be tested-author) in accordance with the law. 

In response, the IMB wrote that it uses the Rotor Gene Q MDx device of the German Qiagen company in the production of Armenian test kits, which "does not require" testing.

Hetq has reviewed page 268 of the Rotor Q MDx User Manual, which shows how the samples should be normalized, testing of the device, and the possible amplification deviations and errors due to it, options, etc. That is, the manufacturer does not rule out possible errors and inaccuracies in the device.

Excerpt from the "Rotor Q MDx" device user manual

"The calibration adjustment information is usually correct. If necessary, they can be changed by clicking the edit button.”

The IMB did not perform the verification of the automatic droppers used in the production of the tests. In response to Hetq's question as to why they were not checked, we were told that "the droppers are new" and have passed a “factory check". (There is no such concept. More details below-author).

"We ensure that the vial of each test kit contains 5-10% more volume than is needed to perform the tests, as the testing in the laboratories will be done mainly with ordinary, non-automated droppers, which have a certain allowable error," the IMB said.

The institute did not mention how much that "certain" error is, the percentage, because it has not been checked.

Hetq was also informed by the institute that "analytical scales" were not used in the production of tests, therefore, no verification was carried out.

At the time, we wanted to know if the devices used by the institute had ever been checked. The IMB did not give a clear answer. Instead, they expressed puzzlement as to how such a matter “is related to the production of test kits”.

It should be noted that the law does not require testing during scientific research, and in case of healthcare activities, as we have already mentioned, testing of devices is mandatory.

The IMB does not have any robotic devices used in the production of test kits. They are assembled by hand.

IMB Director Arsen Arakelyan confessed that it would be good to have a robotic device, as it would speed up the production process, but its existence is "not fundamental" and that manual assembly does not affect accuracy.

“In reality, such a problem (accuracy-author) cannot arise. Such views are meant to alter the context of the discussion,” says Arakelyan. “People who work in the field know that the work we do and as we have described. Thus, we do not need many automated devices. It is good to have them, but it is not fundamental. In fact, I personally, as a hands-on scientist, have done more than 1000 tests on some days.”

IMB Director Arsen Arakelyan

To evaluate their quality during the production of test kits, their accuracy, sensitivity, and specificity are calculated. They are clear, specific, and substantiated numbers. This is required by the Good Manufacturing Practices (GMP) and Good Laboratory Practice (GLP) rules.

Given that the equipment is not inspected for precision, and the tests are produced without robotics, we asked how the IMB calculated the accuracy of the tests, the percentage of uncertainty/error, and what those rates are.

We were informed that the IMB measures the "effectiveness" of the kits by comparing them with the results of already approved tests at the National Center for Disease Control and Prevention (NCDCP) laboratory. The NCDCP, which has no legal right to do so, and whose devices are also not verified, carries out the calculations for the IMB.

The IMB told Hetq that the accuracy of coronavirus tests depends on several factors, including the sample form and type, date, degree of infection, etc. and cited a scientific article entitled -  Detection of SARS-CoV-2 in Different Types of Clinical Specimens.

Hetq also asked the IMB to provide evidence of the accuracy of the Armenian PCR tests. The IMB provided us with one test result example, a graph, where "red is a positive test, dark green is a negative, green is a positive sample, orange is a negative." One example, however, is not complete proof of accuracy.

Graph IMB’s evidence of accuracy

The IMB does not have a legally established methodology for testing, but instead uses "internationally approved" methods (the wording of the IMB. There is no such concept; see below for details-author), citing the following articles by the U.S. Centers for Disease Control and Prevention (CDC) and the World Health Organization (WHO), which are merely guidelines developed in the early stages of coronavirus.

«CDC 2019-Novel Coronavirus Real-Time RT-PCR Diagnostic Panel»

An excerpt of the CDC guidelines noting that positive results do not always indicate the presence of the coronavirus.

January 17, 2020

WHO Molecular assays to diagnose COVID-19: Summary table for available protocols

Excerpt from WHO 2nd Guide.

“Being listed in this document does not imply any endorsement or validation by WHO. These protocol postings online are not being updated and may not reflect subsequent refinements of the assays.”

 January 24, 2020, WHO

WHO Diagnostic detection of 2019-nCov by real time RT-PCR

These, of course, can be suggestions and tips, but not an approved test production method, which, as we have already mentioned, is a document that defines the rules of preparation for the measurement, the method of measurement, the technical means, as well as the processing and calculation of the measurement result

Considering the dates of publication of these guidelines and the reservations mentioned by the sources, they cannot be completely reliable.

Physicist Vahan Sahakyan - There is no concept of "factory verification".  

Vahan Sahakyan, author of the RA Law on Ensuring the Consistency of Measurements

Hetq asked physicist Vahan Sahakyan, the author of the law "On Ensuring the Consistency of Measurements", to clarify the information we received from the IMB. Sahakyan reaffirmed that verification of the results for the accuracy of the results when carrying out activities in the healthcare sector and launching products is obligatory.

"Any device, no matter how accurate, must be verified, certified, if the law stipulates it. You see, the problem is that it is not possible to determine the real value of any physical quantity precisely, for example, the length and weight of this or that product, because if you weigh this with me, then weigh it in several other places, different results will be obtained. Second, if it is a measuring device, one way or another, it can change its characteristics over time. If there was no such thing, then these checks would not have taken place. If that were the case, they would have made it in the factory, put it there and sealed it. They would always work.”

• All devices used in the production of test kits must be verified

Vahan Sahakyan also participated in compiling the list of devices subject to mandatory verification of the decision N-113-N "On defining the list of measuring instruments subject to oversight by Legislative Metrology". According to him, the production of tests relates to accuracy and cannot be without "size and weight".

"If materials are mixed, then there must be appropriate droplets, their exact size must be maintained. Depending on the production technology, it will be clear what metrological security should be provided," says Sahakyan, adding that in order to get even the simplest mixtures, you have to weigh, therefore, you have to have a verified scale.

• There is no concept of “factory verification”

Sahakyan says that there is no such thing as "factory verification", the devices of well-known manufacturers need to be tested as well.

• The percentage of inaccuracy and error cannot be calculated with the equipment of another laboratory

As we have mentioned, the percentage of inaccuracy and error of Armenian-made coronavirus tests is calculated by the National Center for Disease Control and Prevention (NCDCP) of the Ministry of Health.

 Sahakyan says the inaccuracy and error must be gauged by the laboratory that produces the item.  

"It is nonsense in general. Suppose it is a scale. Its inaccuracy or error must be determined. Say you weigh something ten times and get different results. Then, you must get the average of them, divide them, and then look at its reference value. The percentage obtained is called an error. That is, your measurement error is so much. How does another laboratory calculate it,” says Sahakyan.

• There is no internationally approved methodology

"There is no notion of an “internationally approved methodology” At best, they can look at some recommendations," says Sahakyan.  

The National Institute of Standards has no “standard sampling” and has never inspected the devices used by the IMB and the NCDCP 

Albert Babayan, Acting Director of Armenia’s National Body of Standards and Metrology

Hetq sent a request to Armenia’s National Body of Standards and Metrology (NBSM), the state agency that’s supposed to test devices used in the healthcare sector and asked if the IMB had ever tested the devices it uses.

"We report that the Institute of Molecular Biology has not yet applied to the National Body of Standards and Metrology to have its measuring devices checked.

Hetq inquired whether the NBSM, in general, has a device to test the PCR analyzer in real time.

It turned out that their specialists can carry out the inspection, but the national body does not have a "standard sampling" for the inspections, as they are "expensive", have "strict conditions of transportation" and "short shelf life".

The NBSM has not tested its devices.

"During the last eleven years, there was only one application, in 2020, to confirm the type, and one application for verification purposes. As we report, the Company??? is ready to test the device in case of providing a standard sample by the customer," said the NBSM.

It should be noted, however, that the NBSM, as the inspection body, is required to have a standard sample for testing, as it is included in the list of mandatory testing devices by the government, and the manufacturer is not obliged to provide a standard sample.

Businesses with unverified equipment may be fined by the market inspection body in accordance with the law, while the authorized body does not have standard samples for examination.

Narek Zeynalyan: “If the law demands it, there must be testing. But just imagine how many medical devices there are.”

Narek Zeynalyan – Chair of the Armenian National Assembly’s Standing Committee on Health Care and Social Affairs

Hetq asked Narek Zeynalyan, Chairman of the NA Standing Committee on Healthcare and Social Affairs, an MP of the My Step faction, to clarify whether, according to him, not testing such devices is a violation of the law and whether it can affect the accuracy of the test kits

He said that he is not aware of the details regarding verification, but he knows that the specificity of the tests is quite high.

"We approved a bill regarding medical equipment in the law “On medical care and service for the population”. Yes, there is such a heading, where the standards must be developed so that any medical equipment available or imported in the Republic of Armenia meets those standards, is checked, and then starts operating. Naturally, the test kits are a product, and in that regard, I agree, they should be checked," Zeynalyan said.

According to him, verification should be based on standards which have yet to be crafted. Hetq mentioned that according to the Law on Ensuring the Consistency of Measurements, there are already standards set by the government, and the devices used in the production of tests have not been checked. We inquired whether this could affect the accuracy of the tests at the stage of their production.

"Naturally, I agree. “If the law demands it, there must be testing. But just imagine how many medical devices there are in Armenia and their variety. From cotton to the MRI machine, it takes a lot of resources to do all that in a complete, final examination. It takes a long time," said Zeynalyan.

As for the accuracy and quality of the tests, the deputy mentioned that the important thing is their specificity.

"If the specificity is high, if we get the same result with other tests or, say, with a deviation of 1-2%, this is more important, because the result is important. "Of course, verification is important, but if we make a comparative analysis of the result and get the same specificity, it means that this test can be used. That is, it will have the same effect with a Chinese test or German or Russian," he said. 

During the production of the test kits, Hetq had sent a request to the government’s Coronavirus Command Center for clarification on the verification procedure as defined by the law and various government decisions.

Our inquiry was redirected to the Ministry of Health and from there to the from the Ministry of Health to the National Center for Disease Control and Prevention (NCDCP)

One month later, the Ministry of Health responded that the inspection of IMB devices was beyond their purview. 

The NCDCP has to date ignored our inquiry and the issue of verification of their own equipment remains unanswered.