Development Experience Clearinghouse DEC API

Speeches Shim

 

About the DEC

The Development Experience Clearinghouse (DEC) is the largest online resource for USAID funded technical and project materials for international development, with assets dating back to 1945. You are able to access information on over 190,000 USAID technical and program documents with more than 155,500 available for electronic download and growing everyday.

The purpose of the DEC is to strengthen USAID's development projects, activities, and programs by making these development experience documents available to USAID offices and mission staff, PVO's, NGO's, universities and research institutions, developing countries, and the public worldwide.

About the API

The DEC API can assist you in extracting meta-data and records from the DEC. The DEC website (https://dec.usaid.gov), with its rich user interface is optimized for search and discovery, but not the extraction of large amounts of data. Through the website, you can explore the DEC's offerings and become familiar with the content and search syntax of the DEC. After a certain amount of exploration, you may discover that you need to extract large amounts of data. That's where the DEC API comes in. It is designed to provide a mechanism to retrieve large amounts of data.

This page will provide basic instructions on how to call and interact with the inputs and outputs of the DEC API. In order to understand the offerings of the API, we expect that you have some experience with programming including constructing an URL, web methods calls, and http requests.

The API consist of three web points that provide distinct mechanisms to interface with the DEC. The points of access are qsearch.ashx, queries.svc, and request.axpx. Each contains their own methods to retrieve information from the DEC.

About the Dataset

The dataset includes the meta-data associated with each document record in the database. The meta-data is derived from the content of the medium and provides information such as the object's URL, title, publication date, authors, document types, and geographical regions. Please refer to the data dictionary for more information on what is returned by the API.

DEC Disclaimer

USAID provides data on the USAID Open Data Listing website ("this website") to the public to further USAID's values of transparency, participation, and collaboration in tangible ways that benefit the American people. The data may be reproduced or otherwise used freely, unless identified as being subject to copyright protection. If data is identified as being subject to a copyright, it is the obligation of the user to obtain permission from the original source to use or copy the data from the original source. USAID asks that the "USAID Open Data Listing" be cited as the source of the data used from this website.

USAID is providing this information as a public service and makes every effort to provide accurate and complete information. USAID makes no guarantee that there will be no errors, and makes no claims, promises or guarantees about the accuracy, completeness, or adequacy of the data on this website, and expressly disclaims liability arising from use, reference to, or reliance on any data contained on this website. This is a service that is continually under development, and USAID will make every effort to keep this site current and to correct errors brought to our attention. Errors and data requests are best reported via USAID’s Github account. USAID's open data team can be contacted at opendata@usaid.gov.

The data posted on this website may contain hypertext links or pointers to information created or maintained by other public and private organizations. These links and pointers are provided to enhance the public's information experience on the website. USAID does not control or guarantee the accuracy, relevance, timeliness, or completeness of any linked information. USAID make no claims, or guarantees about the accuracy and completeness of external data connected to this website, and expressly disclaims liability for errors or omissions.

The views and opinions of authors expressed herein do not necessarily represent the official views and opinions of USAID.

 

qsearch.ashx

Qsearch will immediately return the result set in the specified format as a downloadable feed for search results under 2,000 records. Any queries more than the record limit will return a "result set too large error."

Response Format:

CSV, JSON, XML

Response Type:

HTTPS GET

Authentication:

None

Required Variables:

q= String - Base64 URL Encoded Search Query for the DEC Database. If the field is ignored then qsearch will redirect back to help.html.
To form a valid Base64 URL encoded string, use these sites to assist in building your query.
"Search string" -> Base64 -> URLEncode -> Parameter -> Request -> Output
"Search string" -> U2VhcmNoIHN0cmluZw== -> U2VhcmNoIHN0cmluZw%3D%3D -> q=U2VhcmNoIHN0cmluZw%3D%3D -> https://dec.usaid.gov/api/qsearch.ashx?q=U2VhcmNoIHN0cmluZw%3D%3D

Optional Variables:

rtype= String Enumeration - Can be either CSV, JSON, or XML. Default is XML. If an invalid entry is supplied then the application will use default output type of XML.
Enumeration -> Parameter -> Output
CSV -> rtype=CSV -> https://dec.usaid.gov/api/qsearch.ashx?q=U2VhcmNoIHN0cmluZw%3D%3D&rtype=CSV

How to build your query:

Lets get Evaluations from the DEC using the search notation (documents.web_collection:("recent evaluations")) and output as a CSV file.

Take (documents.web_collection:("recent evaluations")) and convert it to Base64.

Take KGRvY3VtZW50cy53ZWJfY29sbGVjdGlvbjooInJlY2VudCBldmFsdWF0aW9ucyIpKQ== and URL encode it.

Take KGRvY3VtZW50cy53ZWJfY29sbGVjdGlvbjooInJlY2VudCBldmFsdWF0aW9ucyIpKQ%3D%3D and set as a 'q' parameter for the URL: https://dec.usaid.gov/api/qsearch.ashx

https://dec.usaid.gov/api/qsearch.ashx?q=KGRvY3VtZW50cy53ZWJfY29sbGVjdGlvbjooInJlY2VudCBldmFsdWF0aW9ucyIpKQ%3D%3D will return its results using its default output format as XML.

Append the string with the parameter and value notation of &rtype=CSV.
https://dec.usaid.gov/api/qsearch.ashx?q=KGRvY3VtZW50cy53ZWJfY29sbGVjdGlvbjooInJlY2VudCBldmFsdWF0aW9ucyIpKQ%3D%3D&rtype=CSV will now output as a CSV file.

Now you have a URL that will get all of the DEC's meta-data for the Evaluations as a CSV file.

 

Queries.svc

Queries.svc is a Mircosoft Windows Communication Foundation Service that is hosted by IIS. It provides an object to call methods that can return recourd counts, results below 2,000 records, and request larger sets in the same way Request.aspx provides. These methods can be referenced in an .NET application. Search the web for more information on WCF Services.

Response Type:

Mircosoft Windows Communication Foundation Service

Authentication:

None

Methods:

int CountResults(string email, string query, string version)

Counts the number of records in the search query. If the query does not parse or there was an internal error then zero(0) is returned.

email - requestor's email address
query - DEC search string
version - Reserved for future releases. Set to NULL the value is ignored.

Returns the number of records found in the DEC using the query string.

string Request(string email, string query, string version, string format)

Appends the request into the API's processing queue and sends a confirmation email on the success of the request.

email - requestor's email address
query - DEC search string
version - Reserved for future releases. Set to NULL the value is ignored.
format - Can be either CSV, JSON, or XML. If the value does not parse then XML is assumed.

Returns the confirmation or error text of the transaction.

string Search(string email, string query, string version, string format)

Searches the DEC for an immediate response. If the record count is larger than 2,000 records then a "result set too large error." is returned.

email - requestor's email address
query - DEC search string
version - Reserved for future releases. Set to NULL the value is ignored.
format - Can be either CSV, JSON, or XML. If the value does not parse then XML is assumed.

Returns the string representation of the result set in the output format provided. See the Output Samples for more information.

 

Request.aspx

Request.aspx will request and record the requestor's email address, search query, and output format and when the submit button is clicked, it appends the request into the API's processing queue. Immediately the site will send an email to the requestor's address to provide a confirmation of the request. Then during the site's off peek hours, the API will generate the result and send an additional email to the requestor with a link to the data in the requested format as a downloadable file from the site.

Links are stored on the server for 30 days. They are not renewed or refreshed automatically and its content stays the same. If you need to refresh the data then you should complete a new request. If you need a link to our site to remain pernment then contact the DEC webmaster at decwebmaster@usaid.gov and submit your request in the email.

Response Type:

Forms response

Authentication:

Requires email address to process the request.

Form fields:

Email *

Required - Requestor's email address to send the confirmation and result link to when the operation is completed.

Search Query *

Required - Search Query for the DEC Database

Check query

Optional function to evaluate your search query to a count of the records. Use this to make sure that your search query is going to return an expected number of results.

Output Format

Defaults to XML - A dropdown box that allows the user to choose a result format between XML, JSON, and CSV.

Submit

Checks the query for valid values and if they are valid then enters the values to the queue for processing after hours. Two emails will be generated afterward submittion. The first will be send immediatly to the requestor and the second will be send when the process is complete.

Reset

Reset the form fields for a new query. Note that Reset does not clear the confirmation or error text in the form.

Output

The page will return a confirmation text and a confirmation email to the user. You can use the confirmation number to track insidents with the API to the decwebmaster at decwebmaster@usaid.gov

Result:

The result depends upon the user specified "rtype" or output type of XML, JSON, or CSV. The schema contains the search term, the number of records found, and a listing of all matching records in the database.

 

Sample CSV Return

Sample Property,Another Sample Property\n
"Sample Value","Another Sample Value | And a multi-value"\n
Sample Property Another Sample Property
Sample Value Another Sample Value | And a multi-value

Note: Multivalues in the CSV format is delimited by space then pipe and then space again. (' | ').

 

Sample JSON Return

{

"SearchTerm": "Sample",
"SearchDate": "2014-01-01T11:59:59.0000000-04:00",
"RecordsFound": "68",
"Records": [

{

"Sample Property":
{

"name": "Sample Property",
"value": ["Sample Value"]

},

"Another Sample Property":
{

"name": "Another Sample Property",
"value": ["Another Sample Value", "And a multi-value"]

},
...

},
{

...

},
...

]

}

 

Sample XML Return

<Search>

<SearchTerm>Sample</SearchTerm>
<SearchDate>2014-01-01T11:59:59.0000000-04:00</SearchDate>
<RecordsFound>68</RecordsFound>
<Records>

<Record>

<Property name="Sample Property">

<Value>Sample Value</Value>

</Property>

<Property name="Another Sample Property">

<Value>Another Sample Value</Value>
<Value>And a multi-value</Value>

</Property>
...

</Record>
<Record>

...

</Record>
...

</Records>

</Search>

 

Data Dictionary

Header information

searchTerm
searchDate
RecordsFound

Property name Data type Sample value Description
searchTerm String Forward Evaluations Text value of the search query executed against the DEC.
searchDate Date 2014-01-01T11:59:59.0000000-04:00 Exact date and time the search query started its execution against the DEC.
RecordsFound Integer 68 Indicates the number of records found with the search query and the number of record nodes in the output.

Column ID's

Abstract
Ancillary_Data
Bibliographic_Type
ContentType
Contract_Grant_Number
Credit
Date_Resource_Created
Description
Descriptors_Topical
Descriptors_Geographic
File
File_Size
Inst_Author
Inst_Publisher
Inst_Sponsor
Language
Mime_Type
New_Thesaurus_Terms
Notes
Personal_Author
Primary_Subject
Publication_Date_Freeform
Related_Doc_Links
Report_Number
Series_Title
Title
Title_Translated
Unique_ID
URI
USAID_Geography
USAID_Project_Number

Property name Data type Sample value Description
Abstract String Summarizes mid-term evaluation of a project... The summary of the artifact.
Ancillary_Data String Period of report: August 2003-July 2004 Can be anything from the document that doesn't fit in a predefined field; also information that is not available in the document.
Bibliographic_Type String Reference Document Bibliographic Code and Name; Indicates the type of artifact.
ContentType String Documents Identifies the content type of the record. Can be Documents, Audio, Video, etc.
Contract_Grant_Number String PCE-I-00-00-00000-00 Number of the A.LD. contract, grant, PASA, or RSSA which funded the production of the artifact.
Credit String Director: James Cameron The people who are responsible for creating the object: i.e. photographer, director, writer, interviewer, interviewee, actor, etc.
Date_Resource_Created Date 1/1/1986 12:00:00 AM Displayed publication date on the document; Date the document was (is to be) published; submission of dissertation or thesis; defined format.
Description String Video is in color and contains sound Describes the artifact for its look in feel. Relates to Multimedia content types.
Descriptors_Topical String HIV/AIDS Field contains primary descriptors which reflect the subject content of the artifact.
Descriptors_Geographic String Latin America Geopolitical descriptors from the USAID thesaurus. This is the geographic topic of the asset.
File URL http://pdf.usaid.gov/pdf_docs/PNAAA000.pdf The actual artifact document.
File_Size String 300 KB The file size in KB.
Inst_Author String 000035 - U.S. Agency for International Development (USAID) Authoring organization name.
Inst_Publisher String 012781 - USAID. Knowledge Services Center (KSC) Publishing organization name.
Inst_Sponsor String 000035 - U.S. Agency for International Development (USAID) Sponsoring organization name.
Language String English Identifies the language of the multimedia file. Identifies the language of the text of the document being analyzed. If more than one language is used, enter the appropriate codes in alphabetical order, separated by a comma and space.
Mime_Type String application/pdf Mime type; the internet media type identifier.
New_Thesaurus_Terms String Interim evaluation Terms NOT included in the"A.I.D. Thesaurus" are entered separately here to describe concepts not adequately covered by existing descriptors. Field also contains proposed descriptors.
Notes String Additional information can be found at... Area for freeform notes.
Personal_Author String Shah, Rajiv Persons responsible for authorship, includes editors and compilers.
Primary_Subject String Development organizations Primary Subject; A major subject category is assigned to all technical document records entered after August 1985 and to all project documents entered after February 1987. The subject is derived from the "A.I.D. Thesaurus" list of Major Subjects.
Publication_Date_Freeform String Spring 1987 Displayed publication date on the document; Date the document was (is to be) published; submission of dissertation or thesis; unstructured format.
Related_Doc_Links String http://pdf.usaid.gov/pdf_docs/PNAAA001.pdf List of other documents that relate to this artifact.
Report_Number String AS-0-0 Alphanumeric number assigned to a document by the creators.
Series_Title String Population technical assistance project occasional paper, no. 6 Records the title of the publication series and the part.
Title String USAID HIV/AIDS segment : overview -- final The title and subtitle of an artifact.
Title_Translated String USAID HIV/AIDS segment : overview -- final English translation of the Document Title.
Unique_ID String PN-AAA-000 Unique identifier. Also used to indicate that the cataloguer has selected a particular document for cataloguing since they are the ones applying the DocID. PDs are for non-technical documents, PNs technical, and PCs are non-USAID documents, typically found on the web.
URI URL https://dec.usaid.gov/dec/content/Detail.aspx?ctID=ODVhZjk4NWQtM2YyMi00Y... URL to the details record page in the DEC.
USAID_Geography String Ecuador Geographic Area Name as defined by USAID; Related to code field. This is where the funding for the asset came from.
USAID_Project_Number String 1234567 Concatenation of the Project and Sub-project numbers. Seven or Nine digits.