Fiware Cygnus:CKAN中没有任何数据可持久 [英] Fiware Cygnus: no data have been persisted in CKAN
问题描述
我正在尝试将cygnus与CKAN一起使用，但是当属性为JSON类型时，CKAN中没有任何数据可持久化.首先，我向Orion发送信息:
I am trying to use cygnus with CKAN, but no data have been persisted in CKAN when the attribute is JSON type. First, I send information to Orion:
Accept: application/json
X-AUTH-TOKEN: <mytoken>
Fiware-Service: PapelClubDemo
Fiware-ServicePath: /events/leonliterario
{
"contextElements": [
{
"type": "Events",
"isPattern": "false",
"id": "thisweek",
"attributes": [
{
"name": "schedule",
"type": "json",
"value": [{"title": "Presentación Viva Mi Gente","date": "2015-11-30","location": "Salón de actos del ICAL","url": "http://www.papel.club"}]
}
]
}
],
"updateAction": "APPEND"
}
我在Cygnus中有该实体的订阅，这是我在Cygnus日志中收到的信息:
I have a suscription in Cygnus for this entity and this is the information I receive in Cygnus log:
01 Dec 2015 19:05:13,701 INFO [891993589@qtp-1988714671-0] (com.telefonica.iot.cygnus.handlers.OrionRestHandler.getEvents:232) - Received data ({"subscriptionId" : "565dd3497b72b7c7092d5a29", "originator" : "localhost", "contextResponses" : [ { "contextElement" : { "type" : "Events","isPattern" : "false", "id" : "thisweek004", "attributes" : [ { "name" : "schedule", "type" : "json", "value" : [ { "title" : "Presentación Viva MiGente", "date" : "2015-11-30", "location" : "Salón de actos del ICAL", "url" : "http://www.papel.club" }, { "title" : "Presentación Viva Mi Gente2","date" : "2015-11-30", "location" : "Salón de actos del ICAL", "url" : "http://www.papel.club" } ] } ] }, "statusCode" : { "code" : "200","reasonPhrase" : "OK" } } ]}) 01 Dec 2015 19:05:13,702 INFO [891993589@qtp-1988714671-0](com.telefonica.iot.cygnus.handlers.OrionRestHandler.getEvents:255) - Event put in the channel (id=2134043204, ttl=10) 01 Dec 2015 19:05:16,842 INFO[SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionCKANSink.persistOne:207) - [ckan-sink] Persisting data atOrionCKANSink (orgName=papelclubdemo, pkgName=papelclubdemo_events_leonliterario, resName=thisweek004_events,data=1448989513702,2015-12-01T17:05:13.702Z,thisweek004,Events,schedule,json,[{"title":"Presentación Viva MiGente","date":"2015-11-30","location":"Salón de actos del ICAL","url":"http://www.papel.club"},{"title":"Presentación Viva MiGente2","date":"2015-11-30","location":"Salón de actos del ICAL","url":"http://www.papel.club"}],[])
01 Dec 2015 19:05:19,479 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionSink.process:224) - Runtime error (Don't know how to treat response code 503)
01 Dec 2015 19:05:19,480 INFO [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionSink.process:233) - Finishing transaction (1448984542-601-0000000018)
这是我的天鹅座代理配置:
This is my cygnus agent config:
# Flume handler that will parse the notifications, must not be changed
cygnusagent.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.OrionRestHandler
# URL target
cygnusagent.sources.http-source.handler.notification_target = /notify
# Default service (service semantic depends on the persistence sink)
cygnusagent.sources.http-source.handler.default_service = def_serv
# Default service path (service path semantic depends on the persistence sink)
cygnusagent.sources.http-source.handler.default_service_path = def_servpath
# Number of channel re-injection retries before a Flume event is definitely discarded (-1 means infinite retries)
cygnusagent.sources.http-source.handler.events_ttl = 10
# Source interceptors, do not change
cygnusagent.sources.http-source.interceptors = ts gi
# TimestampInterceptor, do not change
cygnusagent.sources.http-source.interceptors.ts.type = timestamp
# GroupinInterceptor, do not change
cygnusagent.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.GroupingInterceptor$Builder
# Grouping rules for the GroupingInterceptor, put the right absolute path to the file if necessary
# See the doc/design/interceptors document for more details
cygnusagent.sources.http-source.interceptors.gi.grouping_rules_conf_file = /usr/cygnus/conf/grouping_rules.conf
# ============================================
# OrionCKANSink configuration
# channel name from where to read notification events
cygnusagent.sinks.ckan-sink.channel = ckan-channel
# sink class, must not be changed
cygnusagent.sinks.ckan-sink.type = com.telefonica.iot.cygnus.sinks.OrionCKANSink
# the CKAN API key to use
#cygnusagent.sinks.ckan-sink.api_key = <mykey>
# the FQDN/IP address for the CKAN API endpoint
cygnusagent.sinks.ckan-sink.ckan_host = demo.ckan.org
# the port for the CKAN API endpoint
cygnusagent.sinks.ckan-sink.ckan_port = 80
# Orion URL used to compose the resource URL with the convenience operation URL to query it
cygnusagent.sinks.ckan-sink.orion_url = http://127.0.0.1:1026
# how the attributes are stored, either per row either per column (row, column)
cygnusagent.sinks.ckan-sink.attr_persistence = row
# enable SSL for secure Http transportation; 'true' or 'false'
cygnusagent.sinks.ckan-sink.ssl = false
当天鹅将数据持久保存到demo.ckan.org中时，将正确创建组织，数据集和资源，但不会加载数据.
When cygnus persists data into demo.ckan.org, organization, dataset and resource are created correctyl but data are not loaded.
推荐答案
这是因为OrionCKANSink的row模式会自动创建资源及其关联的数据存储，其中所有字段的类型均为文本".为什么?因为orion发送的类型不是具有语义的实类型，而只是用户认为属性类型是什么的描述.我的意思是，猎户座类型可以是浮点数"，也可以是带精度4的浮点数".因此，无法获得数据的真实类型(在尝试尝试推断真实数据类型的启发式实验中不花费大量时间).因此，行模式的优点是可以自动创建资源(和数据存储)，但是约束是所有通知的数据必须是字符串.
That's because the row mode of OrionCKANSink automatically creates the resources and their associated datastore with all the fields having type "text". Why? Because the types sent by orion are not real types with semantic, but just a description of what the user consider the attribute type is. I mean, an Orion type can be "float" but "float number with precission 4" as well. Thus, the real type of the data cannot be got (without expending a lot of time in heuristics trying to infer the real data type). Thus, the row mode has the advantage of automatically creating the resources (and datastores), but the constraint is all the notified data must be a string.
如果您在CKAN数据存储区中需要真实的Json类型，则Cygnus的推荐模式是column一种.
If you need real Json types in your CKAN datastore, then the recommended mode for Cygnus is the column one.
您可以看到有关不同模式的更详细的讨论此处
You can see a more elaborated discussion about the different modes here
这篇关于Fiware Cygnus:CKAN中没有任何数据可持久的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
