Nebula如何导入datetime格式日期

Big_Feiyang · 2021 年4 月 15 日 03:39

有个日期格式为2011-08-17T14:26:59.961+0000这样的数据，我该如何将这个数据导入到DateTime类型的属性中去
导入配置文件：

name: creationDate
type: datetime
index: 1

nicole · 2021 年4 月 15 日 03:55

这种日期格式要预处理下，改成符合要求的格式：
https://docs.nebula-graph.com.cn/2.0/3.ngql-guide/3.data-types/4.date-and-time/#datetime

Big_Feiyang · 2021 年4 月 15 日 08:16

要如何进行预处理，我看文档中手动插入可以调用datetime(date)进行处理，导入配置文件是否存在参数format:yyyy这种格式，还是需要修改源文件

Big_Feiyang · 2021 年4 月 15 日 12:21

你好，有看到我的回复吗

HarrisChu · 2021 年4 月 16 日 02:27

你好，你是用什么工具导入？
目前 importer 不支持 DateTime 类型，有个 pr，添加数据导入对date、time、datetime类型的支持 by git-spower · Pull Request #114 · vesoft-inc/nebula-importer · GitHub
但是缺少 cla，还没合并。也比较简单，就是发 nebula-graph 的时候，转一下用 datetime(value)

Big_Feiyang · 2021 年4 月 16 日 02:55

好的，了解了，我改用String导入了
现在导入报错，报错日志：
2021/04/16 02:33:40 — START OF NEBULA IMPORTER —
2021/04/16 02:33:40 [INFO] connection_pool.go:74: [nebula-clients] connection pool is initialized successfully
2021/04/16 02:33:40 [INFO] clientmgr.go:28: Create 20 Nebula Graph clients
2021/04/16 02:33:40 [INFO] reader.go:26: The delimiter of /data12/cyl/nebula_home/data/comment_0_0.csv is U+007C ‘|’
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x524fdf]

goroutine 82 [running]:
github.com/vesoft-inc/nebula-importer/pkg/config.(*VID).String(0xc0001c4100, 0x7e87a1, 0x4, 0xc00016a1b0, 0x19)
/home/nebula-importer/pkg/config/config.go:437 +0x7f
github.com/vesoft-inc/nebula-importer/pkg/config.(*Vertex).String(0xc0001c40e0, 0x6, 0xc000818140)
/home/nebula-importer/pkg/config/config.go:678 +0x455
github.com/vesoft-inc/nebula-importer/pkg/config.(*Schema).String(0xc0001c40c0, 0x32, 0xc0000801c0)
/home/nebula-importer/pkg/config/config.go:365 +0x74
github.com/vesoft-inc/nebula-importer/pkg/reader.(*FileReader).startLog(0xc00042e080)
/home/nebula-importer/pkg/reader/reader.go:64 +0x83
github.com/vesoft-inc/nebula-importer/pkg/reader.(*FileReader).Read(0xc00042e080, 0x0, 0x0)
/home/nebula-importer/pkg/reader/reader.go:156 +0x626
github.com/vesoft-inc/nebula-importer/pkg/cmd.(*Runner).Run.func2(0xc000015d80, 0xc000015dc0, 0xc00042e080, 0xc0000801c0, 0x32)
/home/nebula-importer/pkg/cmd/runner.go:70 +0x40
created by github.com/vesoft-inc/nebula-importer/pkg/cmd.(*Runner).Run
/home/nebula-importer/pkg/cmd/runner.go:69 +0x705
请问这个是什么原因导致的

HarrisChu · 2021 年4 月 16 日 03:00

是 importer v2.0 的么？
请贴一下 config 文件的配置

Big_Feiyang · 2021 年4 月 16 日 03:06

version: v2
description: sf1
removeTempFiles: false
clientSettings:
retry: 3
concurrency: 20
channelBufferSize: 1024
space: SF1
connection:
user: root
password: nebula
address: localhost:9669
logPath: ./err/sf100.log
files:

path: data/comment_0_0.csv
failDataPath: ./err/ve.csv
batchSize: 1024
inOrder: false
type: csv
csv:
withHeader: true
withLabel: false
delimiter: “|”
schema:
type: vertex
vertex:
vid:
index: 0
type: string
tags:
- name: Comment
props:
- name: id
type: string
index: 0
- name: creationDate
type: string
index: 1
- name: locationIP
type: string
index: 2
- name: browserUsed
type: string
index: 3
- name: content
type: string
index: 4
- name: length
type: int
index: 5
path: data/post_0_0.csv
failDataPath: ./err/ve.csv
batchSize: 1024
inOrder: false
type: csv
csv:
withHeader: true
withLabel: false
delimiter: “|”
schema:
type: vertex
vertex:
vid:
index: 0
type: string
tags:
- name: Post
props:
- name: id
type: string
index: 0
- name: imageFile
type: string
index: 1
- name: creationDate
type: string
index: 2
- name: locationIP
type: string
index: 3
- name: browserUsed
type: string
index: 4
- name: lang
type: string
index: 5
- name: content
type: string
index: 6
- name: length
type: int
index: 7
path: data/company_0_0.csv
failDataPath: ./err/ve.csv
batchSize: 1024
inOrder: false
type: csv
csv:
withHeader: true
withLabel: false
delimiter: “|”
schema:
type: vertex
vertex:
vid:
index: 0
type: string
tags:
- name: Company
props:
- name: id
type: string
index: 0
- name: name
type: string
index: 1
- name: url
type: string
index: 2
path: data/university_0_0.csv
failDataPath: ./err/ve.csv
batchSize: 1024
inOrder: false
type: csv
csv:
withHeader: true
withLabel: false
delimiter: “|”
schema:
type: vertex
vertex:
vid:
index: 0
type: string
tags:
- name: University
props:
- name: id
type: string
index: 0
- name: name
type: string
index: 1
- name: url
type: string
index: 2
path: data/city_0_0.csv
failDataPath: ./err/ve.csv
batchSize: 1024
inOrder: false
type: csv
csv:
withHeader: true
withLabel: false
delimiter: “|”
schema:
type: vertex
vertex:
vid:
index: 0
type: string
tags:
- name: City
props:
- name: id
type: string
index: 0
- name: name
type: string
index: 1
- name: url
type: string
index: 2
path: data/country_0_0.csv
failDataPath: ./err/ve.csv
batchSize: 1024
inOrder: false
type: csv
csv:
withHeader: true
withLabel: false
delimiter: “|”
schema:
type: vertex
vertex:
vid:
index: 0
type: string
tags:
- name: Country
props:
- name: id
type: string
index: 0
- name: name
type: string
index: 1
- name: url
type: string
index: 2
path: data/continent_0_0.csv
failDataPath: ./err/ve.csv
batchSize: 1024
inOrder: false
type: csv
csv:
withHeader: true
withLabel: false
delimiter: “|”
schema:
type: vertex
vertex:
vid:
index: 0
type: string
tags:
- name: Continent
props:
- name: id
type: string
index: 0
- name: name
type: string
index: 1
- name: url
type: string
index: 2
path: data/tag_0_0.csv
failDataPath: ./err/ve.csv
batchSize: 1024
inOrder: false
type: csv
csv:
withHeader: true
withLabel: false
delimiter: “|”
schema:
type: vertex
vertex:
vid:
index: 0
type: string
tags:
- name: aTag
props:
- name: id
type: string
index: 0
- name: name
type: string
index: 1
- name: url
type: string
index: 2
path: data/tagclass_0_0.csv
failDataPath: ./err/ve.csv
batchSize: 1024
inOrder: false
type: csv
csv:
withHeader: true
withLabel: false
delimiter: “|”
schema:
type: vertex
vertex:
vid:
index: 0
type: string
tags:
- name: TagClass
props:
- name: id
type: string
index: 0
- name: name
type: string
index: 1
- name: url
type: string
index: 2
path: data/forum_0_0.csv
failDataPath: ./err/ve.csv
batchSize: 1024
inOrder: false
type: csv
csv:
withHeader: true
withLabel: false
delimiter: “|”
schema:
type: vertex
vertex:
vid:
index: 0
type: string
tags:
- name: Forum
props:
- name: id
type: string
index: 0
- name: title
type: string
index: 1
- name: creationDate
type: string
index: 2
path: data/person_0_0.csv
failDataPath: ./err/ve.csv
batchSize: 1024
inOrder: false
type: csv
csv:
withHeader: true
withLabel: false
delimiter: “|”
schema:
type: vertex
vertex:
vid:
index: 0
type: string
tags:
- name: Person
props:
- name: id
type: string
index: 0
- name: firstName
type: string
index: 1
- name: lastName
type: string
index: 2
- name: gender
type: string
index: 3
- name: birthday
type: string
index: 4
- name: creationDate
type: string
index: 5
- name: locationIP
type: string
index: 6
- name: browserUsed
type: string
index: 7
- name: speaks
type: string
index: 8
- name: email
type: string
index: 9

HarrisChu · 2021 年4 月 16 日 03:27

格式按 markdown 转换了，看上去你同时设置了 withHeader 和 index

如果 withHeader 是 true 的话，index 就不起作用了。建议设置成 false
具体参见：GitHub - vesoft-inc/nebula-importer: Nebula Graph Importer with Go

Big_Feiyang · 2021 年4 月 16 日 06:08

非常感谢，已经可以读取到文件的数据了，但是导入还是报错

我尝试手动插入数据，还是一样的错误：
(root@nebula) [SF100]> INSERT VERTEX University(id,name,url) VALUES “vid”: (“id”, “name”, “url”)
[ERROR (-8)]: Storage Error: part: 49, error: E_RPC_FAILURE(-3).

创建Tag的语句：CREATE TAG University(id STRING, name STRING, url STRING);
请问这是什么错误？

HarrisChu · 2021 年4 月 16 日 06:15

storage 是单副本还是多副本，console 里打一下 show hosts 看看是不是 storage 服务挂了。

如果重启 storage ，一导入就挂的话，可以适当调整 concurrency 和 channelBufferSize 调小一点

Big_Feiyang · 2021 年4 月 16 日 06:44

已经找到问题了，是我配置文件的local_ip写错了，抱歉给您带来麻烦了，非常感谢您的帮助