Databricks群集未初始化带有错误的Azure库:模块"lib"没有属性"SSL_ST_INIT" [英] Databricks cluster does not initialize Azure library with error: module 'lib' has no attribute 'SSL_ST_INIT'

问题描述

我正在将Azure DataBricks笔记本与Azure库一起使用，以获取Blob存储中的文件列表.该任务已安排好，完成工作后将终止集群，并以新的运行重新开始.

I am using Azure DataBricks notebook with Azure library to get list of files in Blob Storage. This task is scheduled and cluster is terminated after finishing the job and started again with new run.

我正在使用Azure 4.0.0库( https://pypi.org/project/azure/)

I am using Azure 4.0.0 library (https://pypi.org/project/azure/)

有时我会收到错误消息:

Sometimes I am getting error message:

• AttributeError:模块"lib"没有属性"SSL_ST_INIT"

而且很少:

• AttributeError:cffi库'_openssl'没有函数，常量或 名为"CRYPTOGRAPHY_PACKAGE_VERSION"的全局变量

• AttributeError: cffi library '_openssl' has no function, constant or global variable named 'CRYPTOGRAPHY_PACKAGE_VERSION'

我找到了一种解决方案，例如卸载openssl或azure库，重新启动群集并重新安装，但是由于可能需要处理更长的任务等，因此可能无法重新启动群集.

I have found a solution as uninstall openssl or azure library, restart cluster and install it again, but restarting cluster may not be possible because it may need to handle longer tasks, etc.

我还尝试在初始化脚本中安装/升级openSSL 16.2.0，但它无济于事，并开始与默认情况下位于Databricks集群中的另一个openSSL库发生冲突

I also tried to install/upgrade openSSL 16.2.0 in initialization script, but it does not help and start conflicting with some another openSSL library which is in Databricks cluster by default

有什么我可以做的选择吗?

Is there any option what I can do with it?

有用于从Blob存储中获取文件列表的代码:

There is the code for getting list of files from Blob Storage:

import pandas as pd
import re
import os
from pyspark.sql.types import *
import azure
from azure.storage.blob import BlockBlobService
import datetime
import time

r = []
marker = None
blobService = BlockBlobService(accountName,accountKey)
while True:
  result = blobService.list_blobs(sourceStorageContainer, prefix = inputFolder, marker=marker)
  for b in result.items:
    r.append(b.name)
  if result.next_marker:
    marker = result.next_marker
  else:
    break
print(r)

谢谢

推荐答案

此问题的解决方案是将Azure库降级为3.0.0.

Solution for this issue is downgrade Azure library to 3.0.0.

Azure v4似乎与Databricks中的某些初始库有冲突.

It looks like Azure v4 has conflicts with some initial libraries in Databricks.

这篇关于Databricks群集未初始化带有错误的Azure库:模块"lib"没有属性"SSL_ST_INIT"的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！