When the USAID Mission in Ukraine contracted for an evaluation of its judicial reform activity, everyone involved saw it as nothing more than another routine oversight and accountability exercise, much like the half dozen the Mission undertakes annually. That was not to be.

As with much of USAID’s work across the globe, the COVID-19 pandemic flipped those plans on their head and left the evaluators scrambling to deliver as they quickly worked to get out of a Ukraine that was quickly shutting down transportation services to stem the viral outbreak. After much confusion and uncertainty, perseverance and creativity followed frantic, in-person, and online meetings between USAID and the evaluation team to ensure that the team remained healthy and safe, and evaluation contingencies were developed. 

When evaluators from The Cloudburst Group, the U.S.-based research and consulting firm that won the evaluation contract, arrived in Ukraine in early March the COVID-19 outbreak was still centered in China, but countries were beginning to understand that it was a global threat.

The purpose of the evaluation was to assess the relevance and effectiveness of USAID/Ukraine’s New Justice activity, which is focused on advancing judicial reform and building public confidence in the judiciary, among other things.  Evaluations help to inform the Mission about the direction of its programming. By learning and adapting programming we are supporting Ukraine’s commitment and capacity to be self reliant.

Kate Marple-Cantrell, evaluation specialist for The Cloudburst Group, said the evaluation was to have primarily consisted of fieldwork during the month of March, with a focus on in-depth interviews and a survey. The agreement entailed filing a final report by early May.

The evaluators arrived in Ukraine on March 3, but the in-country environment changed just one day later, when Ukraine recorded its first COVID-19 case. With the ever-growing pandemic looming, the Cloudburst Group team and their USAID/Ukraine counterparts quickly initiated interviews in Kyiv and prepared to distribute online surveys to key stakeholders. 

The team planned to then conduct interviews and focus group discussions with practicing lawyers and judges at multiple venues in Kharkiv, before traveling to the smaller city of Chuhuyiv to meet with more judges. They also intended to visit Odesa to talk to key stakeholders about how USAID efforts to improve the judicial system were proceeding.

But their plans bumped up against the quickly developing COVID-19 outbreak. On March 12 as the evaluation team departed for Kharkiv, the government announced that it would close the country’s borders to foreigners. Then the Kharkiv event they had planned to attend was cancelled and the team was only able to interview a few judges in person. With no train service available due to the government-ordered shutdown, the evaluators cancelled their Odesa trip. 

“At that point it became evident that the fieldwork could not be completed in person in Ukraine,“ explained Andriy Kobzarenko, USAID monitoring and evaluation specialist. 

With an air transportation shut-down looming, on March 14 the team urgently discussed departing Ukraine on one of the last available flights. They returned to the U.S the next day.

The evaluation was only partially completed, and the challenge became how to remotely complete the necessary interviews and organize the survey. A central concern was that judges, primary beneficiaries of USAID’s support for judicial reform, who are reluctant in the best situations to meet with evaluators regarding how they work and how reforms are proceeding, would be even more hesitant in discussing these matters online or by phone.

USAID considered three options: postpone the fieldwork until transportation bans and quarantine restrictions were lifted, creating an indefinite delay that could inhibit the Mission’s future programming in justice sector reform; complete the fieldwork with currently completed interviews, which would significantly affect the data quality; or continue key informant interviews remotely, online, or by phone.

“In the end we decided to try to arrange as many remote key informant interviews as possible before making a final decision on how to proceed,” Mr. Kobzarenko said.

The original assumption, based on the initial fieldwork in Ukraine that many judicial system workers would refuse to be interviewed, proved wrong. The respondents seemed to change their anticipated behaviour in the face of the new reality and the evaluators arranged interviews with most of them. Consequently a decision was made to complete the fieldwork in remote mode.

“The breakthrough moment came a couple days after beginning to implement the virtual fieldwork plan, when we had a full schedule of interviews, including most of the key types of interlocutors,” Ms. Marple-Cantrell said. “That information showed us that our respondents had moved, like us, into the virtual work environment and were ready to adapt their expectations to the extraordinary circumstances.”

She added that the evaluation team succeeded in nimbly shifting fieldwork approaches in this evaluation and completed the remaining two weeks of field interviews through online technologies. In turn, the surveys received seven times more responses than initially expected, mirroring the success of the interviews. By quickly devising and executing a new approach and intensifying outreach efforts, the evaluation team met or exceeded all field work targets.

“I would say it was the evaluation team's ‘can do’  attitude that made USAID believe that we could finish the evaluation,” Mr. Kobzarenko explained. “The team had to adapt, that was their big contribution. Located on both the east and west coasts of the United States,  they had to organize early and late shifts to accommodate some interviews, but they overcame the obstacles to provide a professionally executed evaluation.”