从私有 CKAN 数据集下载资源 [英] Download resources from private CKAN datasets

问题描述

我的目标是使用 (a) CKAN API，或 (b) CKANAPI CLI，或 (c) 粘贴器(如果 (c) 可行)下载作为私有数据集中资源保存的文件.

My aim is to download files which are held as resources within private datasets using (a) the CKAN API, or (b) the CKANAPI CLI, or (c) paster (if (c) is possible).

我尝试使用 (a) 下载文件未成功.例如，使用资源 URL 和 urllib2 或 requests 下载文件但它已损坏 (.zip) 或 CKAN 登录页面存储在文件中(.xls).

I have attempted downloading the files using (a) unsuccessfully. For example using the resource URL and urllib2 or requests the file is downloaded but it is either corrupted (.zip) or the CKAN login page is stored within the file (.xls).

我尝试使用 (b) 太不成功.例如使用以下代码:

I have tried using (b) too unsuccessfully. For example using the following code:

ckanapi dump datasets dataset_name --datapackages=~/ckan_out -r http://localhost:5000 -a XXXXX-XXXX-XXXX-XXXX-XXXXXXXXX

URL xxxxxxxxxxxx refused connection. The resource will not be downloaded

我还没有找到任何具有 paster 下载资源功能的东西.

I haven't found anything that has the download resources functionality for paster yet.

是否可以使用 CKAN 工具自动化下载私有资源的过程?

Is it possible to automate the process of downloading private resources using CKAN tools?

我是否应该将数据集从私有更改为公开、下载资源，然后再次将它们设为私有?

Should I change datasets from private to public, download the resource, and then make them private again?

欢迎提供任何见解.

CKAN 2.5.2，UBUNTU 14.04

CKAN 2.5.2, UBUNTU 14.04

推荐答案

遗憾的是，CKAN API 不提供下载资源数据的功能(仅适用于元数据:resource_show).资源下载由 CKAN 的 Web UI 代码处理.这意味着您不能使用 API 提供的身份验证方法(即您的 API 密钥)，而必须使用您的普通凭据(用户名 + 密码):

Unfortunately, the CKAN API doesn't offer a function for downloading resource data (only for metadata: resource_show). Resource download is handled by CKAN's web UI code instead. This means that you cannot use the authentication methods provided by the API (i.e. your API-key) but have to use your normal credentials (username + password) instead:

import requests

CKAN_URL = 'http://localhost:5000'


def login(username, password):
    '''
    Login to CKAN.

    Returns a ``requests.Session`` instance with the CKAN
    session cookie.
    '''
    s = requests.Session()
    data = {'login': username, 'password': password}
    url = CKAN_URL + '/login_generic'
    r = s.post(url, data=data)
    if 'field-login' in r.text:
        # Response still contains login form
        raise RuntimeError('Login failed.')
    return s


def download_resource_data(session, pkg_id, res_id):
    url = '{ckan}/dataset/{pkg}/resource/{res}/download/'.format(
            ckan=CKAN_URL, pkg=pkg_id, res=res_id)
    return session.get(url).content


if __name__ == '__main__':
    session = login('my-user', 'my-password')
    data = download_resource_data(session, 'some-package', 'some-resource')
    print(data)

这篇关于从私有 CKAN 数据集下载资源的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！