Ingesting Data With Onelake Part I
openai-chatgptlangchain-pythonchatgptgenaielasticsearchelasticopenaiAIingesting-data-with-onelake-part-ichatlogvectordatabasePythonsearchgenaistacksupporting-blog-contentvectorelasticsearch-labslangchainapplications
Export
Ingesting data with OneLake Part I
In this notebook we will learn how to index data from Microsoft OneLake OneLake to Elasticsearch. This demonstration is related to the article Ingesting data with OneLake Part I.
[ ]
Collecting elasticsearch==8.14
Downloading elasticsearch-8.14.0-py3-none-any.whl.metadata (7.2 kB)
Collecting azure-identity
Downloading azure_identity-1.19.0-py3-none-any.whl.metadata (80 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 80.6/80.6 kB 3.2 MB/s eta 0:00:00
Collecting azure-storage-file-datalake
Downloading azure_storage_file_datalake-12.17.0-py3-none-any.whl.metadata (16 kB)
Collecting azure-cli
Downloading azure_cli-2.65.0-py3-none-any.whl.metadata (8.3 kB)
Collecting python-docx
Downloading python_docx-1.1.2-py3-none-any.whl.metadata (2.0 kB)
Collecting elastic-transport<9,>=8.13 (from elasticsearch==8.14)
Downloading elastic_transport-8.15.1-py3-none-any.whl.metadata (3.7 kB)
Collecting azure-core>=1.31.0 (from azure-identity)
Downloading azure_core-1.32.0-py3-none-any.whl.metadata (39 kB)
Requirement already satisfied: cryptography>=2.5 in /usr/local/lib/python3.10/dist-packages (from azure-identity) (43.0.3)
Collecting msal>=1.30.0 (from azure-identity)
Downloading msal-1.31.0-py3-none-any.whl.metadata (11 kB)
Collecting msal-extensions>=1.2.0 (from azure-identity)
Downloading msal_extensions-1.2.0-py3-none-any.whl.metadata (7.6 kB)
Requirement already satisfied: typing-extensions>=4.0.0 in /usr/local/lib/python3.10/dist-packages (from azure-identity) (4.12.2)
Collecting azure-storage-blob>=12.23.0 (from azure-storage-file-datalake)
Downloading azure_storage_blob-12.23.1-py3-none-any.whl.metadata (26 kB)
Collecting isodate>=0.6.1 (from azure-storage-file-datalake)
Downloading isodate-0.7.2-py3-none-any.whl.metadata (11 kB)
Collecting antlr4-python3-runtime~=4.13.1 (from azure-cli)
Downloading antlr4_python3_runtime-4.13.2-py3-none-any.whl.metadata (304 bytes)
Collecting azure-appconfiguration~=1.7.0 (from azure-cli)
Downloading azure_appconfiguration-1.7.1-py3-none-any.whl.metadata (27 kB)
Collecting azure-batch~=14.2.0 (from azure-cli)
Downloading azure_batch-14.2.0-py3-none-any.whl.metadata (22 kB)
Collecting azure-cli-core==2.65.0 (from azure-cli)
Downloading azure_cli_core-2.65.0-py3-none-any.whl.metadata (1.8 kB)
Collecting azure-cosmos>=3.0.2,~=3.0 (from azure-cli)
Downloading azure_cosmos-3.2.0-py2.py3-none-any.whl.metadata (22 kB)
Collecting azure-data-tables==12.4.0 (from azure-cli)
Downloading azure_data_tables-12.4.0-py3-none-any.whl.metadata (31 kB)
Collecting azure-datalake-store~=0.0.53 (from azure-cli)
Downloading azure_datalake_store-0.0.53-py2.py3-none-any.whl.metadata (19 kB)
Collecting azure-graphrbac~=0.60.0 (from azure-cli)
Downloading azure_graphrbac-0.60.0-py2.py3-none-any.whl.metadata (10 kB)
Collecting azure-keyvault-administration==4.4.0b2 (from azure-cli)
Downloading azure_keyvault_administration-4.4.0b2-py3-none-any.whl.metadata (30 kB)
Collecting azure-keyvault-certificates==4.7.0 (from azure-cli)
Downloading azure_keyvault_certificates-4.7.0-py3-none-any.whl.metadata (34 kB)
Collecting azure-keyvault-keys==4.9.0b3 (from azure-cli)
Downloading azure_keyvault_keys-4.9.0b3-py3-none-any.whl.metadata (46 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 46.5/46.5 kB 2.6 MB/s eta 0:00:00
Collecting azure-keyvault-secrets==4.7.0 (from azure-cli)
Downloading azure_keyvault_secrets-4.7.0-py3-none-any.whl.metadata (28 kB)
Collecting azure-mgmt-advisor==9.0.0 (from azure-cli)
Downloading azure_mgmt_advisor-9.0.0-py2.py3-none-any.whl.metadata (8.6 kB)
Collecting azure-mgmt-apimanagement==4.0.0 (from azure-cli)
Downloading azure_mgmt_apimanagement-4.0.0-py3-none-any.whl.metadata (24 kB)
Collecting azure-mgmt-appconfiguration==3.0.0 (from azure-cli)
Downloading azure_mgmt_appconfiguration-3.0.0-py3-none-any.whl.metadata (9.3 kB)
Collecting azure-mgmt-appcontainers==2.0.0 (from azure-cli)
Downloading azure_mgmt_appcontainers-2.0.0-py3-none-any.whl.metadata (19 kB)
Collecting azure-mgmt-applicationinsights~=1.0.0 (from azure-cli)
Downloading azure_mgmt_applicationinsights-1.0.0-py2.py3-none-any.whl.metadata (11 kB)
Collecting azure-mgmt-authorization~=4.0.0 (from azure-cli)
Downloading azure_mgmt_authorization-4.0.0-py3-none-any.whl.metadata (18 kB)
Collecting azure-mgmt-batchai==7.0.0b1 (from azure-cli)
Downloading azure_mgmt_batchai-7.0.0b1-py2.py3-none-any.whl.metadata (8.9 kB)
Collecting azure-mgmt-batch~=17.3.0 (from azure-cli)
Downloading azure_mgmt_batch-17.3.0-py3-none-any.whl.metadata (19 kB)
Collecting azure-mgmt-billing==6.0.0 (from azure-cli)
Downloading azure_mgmt_billing-6.0.0-py2.py3-none-any.whl.metadata (7.5 kB)
Collecting azure-mgmt-botservice~=2.0.0b3 (from azure-cli)
Downloading azure_mgmt_botservice-2.0.0-py3-none-any.whl.metadata (18 kB)
Collecting azure-mgmt-cdn==12.0.0 (from azure-cli)
Downloading azure_mgmt_cdn-12.0.0-py3-none-any.whl.metadata (25 kB)
Collecting azure-mgmt-cognitiveservices~=13.5.0 (from azure-cli)
Downloading azure_mgmt_cognitiveservices-13.5.0-py3-none-any.whl.metadata (18 kB)
Collecting azure-mgmt-compute~=33.0.0 (from azure-cli)
Downloading azure_mgmt_compute-33.0.0-py3-none-any.whl.metadata (69 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 69.4/69.4 kB 4.6 MB/s eta 0:00:00
Collecting azure-mgmt-containerinstance==10.1.0 (from azure-cli)
Downloading azure_mgmt_containerinstance-10.1.0-py3-none-any.whl.metadata (13 kB)
Collecting azure-mgmt-containerregistry==10.3.0 (from azure-cli)
Downloading azure_mgmt_containerregistry-10.3.0-py3-none-any.whl.metadata (23 kB)
Collecting azure-mgmt-containerservice~=32.0.0 (from azure-cli)
Downloading azure_mgmt_containerservice-32.0.0-py3-none-any.whl.metadata (52 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 52.0/52.0 kB 2.9 MB/s eta 0:00:00
Collecting azure-mgmt-cosmosdb==9.6.0 (from azure-cli)
Downloading azure_mgmt_cosmosdb-9.6.0-py3-none-any.whl.metadata (108 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 108.2/108.2 kB 6.5 MB/s eta 0:00:00
Collecting azure-mgmt-databoxedge~=1.0.0 (from azure-cli)
Downloading azure_mgmt_databoxedge-1.0.0-py2.py3-none-any.whl.metadata (8.4 kB)
Collecting azure-mgmt-datamigration~=10.0.0 (from azure-cli)
Downloading azure_mgmt_datamigration-10.0.0-py2.py3-none-any.whl.metadata (13 kB)
Collecting azure-mgmt-devtestlabs~=4.0 (from azure-cli)
Downloading azure_mgmt_devtestlabs-4.0.0-py2.py3-none-any.whl.metadata (10 kB)
Collecting azure-mgmt-dns~=8.0.0 (from azure-cli)
Downloading azure_mgmt_dns-8.0.0-py2.py3-none-any.whl.metadata (8.9 kB)
Collecting azure-mgmt-eventgrid==10.2.0b2 (from azure-cli)
Downloading azure_mgmt_eventgrid-10.2.0b2-py3-none-any.whl.metadata (23 kB)
Collecting azure-mgmt-eventhub~=10.1.0 (from azure-cli)
Downloading azure_mgmt_eventhub-10.1.0-py3-none-any.whl.metadata (15 kB)
Collecting azure-mgmt-extendedlocation==1.0.0b2 (from azure-cli)
Downloading azure_mgmt_extendedlocation-1.0.0b2-py2.py3-none-any.whl.metadata (2.4 kB)
Collecting azure-mgmt-hdinsight==9.0.0b3 (from azure-cli)
Downloading azure_mgmt_hdinsight-9.0.0b3-py3-none-any.whl.metadata (15 kB)
Collecting azure-mgmt-imagebuilder~=1.3.0 (from azure-cli)
Downloading azure_mgmt_imagebuilder-1.3.0-py3-none-any.whl.metadata (9.4 kB)
Collecting azure-mgmt-iotcentral~=10.0.0b1 (from azure-cli)
Downloading azure_mgmt_iotcentral-10.0.0b2-py3-none-any.whl.metadata (8.3 kB)
Collecting azure-mgmt-iothub==3.0.0 (from azure-cli)
Downloading azure_mgmt_iothub-3.0.0-py3-none-any.whl.metadata (16 kB)
Collecting azure-mgmt-iothubprovisioningservices==1.1.0 (from azure-cli)
Downloading azure_mgmt_iothubprovisioningservices-1.1.0-py3-none-any.whl.metadata (7.3 kB)
Collecting azure-mgmt-keyvault==10.3.0 (from azure-cli)
Downloading azure_mgmt_keyvault-10.3.0-py3-none-any.whl.metadata (15 kB)
Collecting azure-mgmt-kusto~=0.3.0 (from azure-cli)
Downloading azure_mgmt_kusto-0.3.0-py2.py3-none-any.whl.metadata (4.7 kB)
Collecting azure-mgmt-loganalytics==13.0.0b4 (from azure-cli)
Downloading azure_mgmt_loganalytics-13.0.0b4-py3-none-any.whl.metadata (17 kB)
Collecting azure-mgmt-managementgroups~=1.0.0 (from azure-cli)
Downloading azure_mgmt_managementgroups-1.0.0-py2.py3-none-any.whl.metadata (5.6 kB)
Collecting azure-mgmt-maps~=2.0.0 (from azure-cli)
Downloading azure_mgmt_maps-2.0.0-py2.py3-none-any.whl.metadata (5.6 kB)
Collecting azure-mgmt-marketplaceordering==1.1.0 (from azure-cli)
Downloading azure_mgmt_marketplaceordering-1.1.0-py2.py3-none-any.whl.metadata (7.5 kB)
Collecting azure-mgmt-media~=9.0 (from azure-cli)
Downloading azure_mgmt_media-9.0.0-py3-none-any.whl.metadata (14 kB)
Collecting azure-mgmt-monitor~=5.0.0 (from azure-cli)
Downloading azure_mgmt_monitor-5.0.1-py3-none-any.whl.metadata (24 kB)
Collecting azure-mgmt-msi~=7.0.0 (from azure-cli)
Downloading azure_mgmt_msi-7.0.0-py3-none-any.whl.metadata (8.9 kB)
Collecting azure-mgmt-netapp~=10.1.0 (from azure-cli)
Downloading azure_mgmt_netapp-10.1.0-py3-none-any.whl.metadata (21 kB)
Collecting azure-mgmt-policyinsights==1.1.0b4 (from azure-cli)
Downloading azure_mgmt_policyinsights-1.1.0b4-py3-none-any.whl.metadata (11 kB)
Collecting azure-mgmt-privatedns~=1.0.0 (from azure-cli)
Downloading azure_mgmt_privatedns-1.0.0-py2.py3-none-any.whl.metadata (4.9 kB)
Collecting azure-mgmt-rdbms~=10.2.0b16 (from azure-cli)
Downloading azure_mgmt_rdbms-10.2.0b18-py3-none-any.whl.metadata (33 kB)
Collecting azure-mgmt-recoveryservicesbackup~=9.1.0 (from azure-cli)
Downloading azure_mgmt_recoveryservicesbackup-9.1.0-py3-none-any.whl.metadata (36 kB)
Collecting azure-mgmt-recoveryservices~=3.0.0 (from azure-cli)
Downloading azure_mgmt_recoveryservices-3.0.0-py3-none-any.whl.metadata (13 kB)
Collecting azure-mgmt-redhatopenshift~=1.5.0 (from azure-cli)
Downloading azure_mgmt_redhatopenshift-1.5.0-py3-none-any.whl.metadata (7.0 kB)
Collecting azure-mgmt-redis~=14.4.0 (from azure-cli)
Downloading azure_mgmt_redis-14.4.0-py3-none-any.whl.metadata (16 kB)
Collecting azure-mgmt-resource==23.1.1 (from azure-cli)
Downloading azure_mgmt_resource-23.1.1-py3-none-any.whl.metadata (37 kB)
Collecting azure-mgmt-search~=9.0 (from azure-cli)
Downloading azure_mgmt_search-9.1.0-py3-none-any.whl.metadata (10 kB)
Collecting azure-mgmt-security==6.0.0 (from azure-cli)
Downloading azure_mgmt_security-6.0.0-py3-none-any.whl.metadata (31 kB)
Collecting azure-mgmt-servicebus~=8.2.0 (from azure-cli)
Downloading azure_mgmt_servicebus-8.2.0-py3-none-any.whl.metadata (15 kB)
Collecting azure-mgmt-servicefabricmanagedclusters==2.0.0b6 (from azure-cli)
Downloading azure_mgmt_servicefabricmanagedclusters-2.0.0b6-py3-none-any.whl.metadata (9.1 kB)
Collecting azure-mgmt-servicelinker==1.2.0b2 (from azure-cli)
Downloading azure_mgmt_servicelinker-1.2.0b2-py3-none-any.whl.metadata (7.2 kB)
Collecting azure-mgmt-servicefabric~=2.1.0 (from azure-cli)
Downloading azure_mgmt_servicefabric-2.1.0-py3-none-any.whl.metadata (13 kB)
Collecting azure-mgmt-signalr==2.0.0b1 (from azure-cli)
Downloading azure_mgmt_signalr-2.0.0b1-py3-none-any.whl.metadata (9.9 kB)
Collecting azure-mgmt-sqlvirtualmachine==1.0.0b5 (from azure-cli)
Downloading azure_mgmt_sqlvirtualmachine-1.0.0b5-py3-none-any.whl.metadata (8.8 kB)
Collecting azure-mgmt-sql==4.0.0b19 (from azure-cli)
Downloading azure_mgmt_sql-4.0.0b19-py3-none-any.whl.metadata (70 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 70.7/70.7 kB 5.0 MB/s eta 0:00:00
Collecting azure-mgmt-storage==21.2.0 (from azure-cli)
Downloading azure_mgmt_storage-21.2.0-py3-none-any.whl.metadata (30 kB)
Collecting azure-mgmt-synapse==2.1.0b5 (from azure-cli)
Downloading azure_mgmt_synapse-2.1.0b5-py3-none-any.whl.metadata (15 kB)
Collecting azure-mgmt-trafficmanager~=1.0.0 (from azure-cli)
Downloading azure_mgmt_trafficmanager-1.0.0-py3-none-any.whl.metadata (7.8 kB)
Collecting azure-mgmt-web==7.2.0 (from azure-cli)
Downloading azure_mgmt_web-7.2.0-py3-none-any.whl.metadata (87 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 87.3/87.3 kB 6.7 MB/s eta 0:00:00
Collecting azure-monitor-query==1.2.0 (from azure-cli)
Downloading azure_monitor_query-1.2.0-py3-none-any.whl.metadata (32 kB)
Collecting azure-multiapi-storage~=1.3.0 (from azure-cli)
Downloading azure_multiapi_storage-1.3.0-py2.py3-none-any.whl.metadata (6.5 kB)
Collecting azure-storage-common~=1.4 (from azure-cli)
Downloading azure_storage_common-1.4.2-py2.py3-none-any.whl.metadata (8.1 kB)
Collecting azure-synapse-accesscontrol~=0.5.0 (from azure-cli)
Downloading azure_synapse_accesscontrol-0.5.0-py2.py3-none-any.whl.metadata (2.1 kB)
Collecting azure-synapse-artifacts~=0.19.0 (from azure-cli)
Downloading azure_synapse_artifacts-0.19.0-py3-none-any.whl.metadata (17 kB)
Collecting azure-synapse-managedprivateendpoints~=0.4.0 (from azure-cli)
Downloading azure_synapse_managedprivateendpoints-0.4.0-py2.py3-none-any.whl.metadata (2.3 kB)
Collecting azure-synapse-spark~=0.2.0 (from azure-cli)
Downloading azure_synapse_spark-0.2.0-py2.py3-none-any.whl.metadata (1.8 kB)
Requirement already satisfied: chardet~=5.2.0 in /usr/local/lib/python3.10/dist-packages (from azure-cli) (5.2.0)
Collecting colorama~=0.4.4 (from azure-cli)
Downloading colorama-0.4.6-py2.py3-none-any.whl.metadata (17 kB)
Collecting fabric~=3.2.2 (from azure-cli)
Downloading fabric-3.2.2-py3-none-any.whl.metadata (3.5 kB)
Collecting javaproperties~=0.5.1 (from azure-cli)
Downloading javaproperties-0.5.2-py2.py3-none-any.whl.metadata (5.8 kB)
Collecting jsondiff~=2.0.0 (from azure-cli)
Downloading jsondiff-2.0.0-py3-none-any.whl.metadata (562 bytes)
Requirement already satisfied: packaging>=20.9 in /usr/local/lib/python3.10/dist-packages (from azure-cli) (24.1)
Collecting pycomposefile>=0.0.29 (from azure-cli)
Downloading pycomposefile-0.0.32-py3-none-any.whl.metadata (356 bytes)
Collecting PyGithub~=1.38 (from azure-cli)
Downloading PyGithub-1.59.1-py3-none-any.whl.metadata (2.0 kB)
Collecting PyNaCl~=1.5.0 (from azure-cli)
Downloading PyNaCl-1.5.0-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl.metadata (8.6 kB)
Collecting scp~=0.13.2 (from azure-cli)
Downloading scp-0.13.6-py2.py3-none-any.whl.metadata (4.5 kB)
Collecting semver==2.13.0 (from azure-cli)
Downloading semver-2.13.0-py2.py3-none-any.whl.metadata (5.0 kB)
Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from azure-cli) (75.1.0)
Requirement already satisfied: six>=1.10.0 in /usr/local/lib/python3.10/dist-packages (from azure-cli) (1.16.0)
Collecting sshtunnel~=0.1.4 (from azure-cli)
Downloading sshtunnel-0.1.5-py2.py3-none-any.whl.metadata (17 kB)
Requirement already satisfied: tabulate in /usr/local/lib/python3.10/dist-packages (from azure-cli) (0.9.0)
Requirement already satisfied: urllib3 in /usr/local/lib/python3.10/dist-packages (from azure-cli) (2.2.3)
Collecting websocket-client~=1.3.1 (from azure-cli)
Downloading websocket_client-1.3.3-py3-none-any.whl.metadata (6.9 kB)
Collecting xmltodict~=0.12 (from azure-cli)
Downloading xmltodict-0.14.2-py2.py3-none-any.whl.metadata (8.0 kB)
Requirement already satisfied: distro in /usr/local/lib/python3.10/dist-packages (from azure-cli) (1.9.0)
Collecting argcomplete~=3.3.0 (from azure-cli-core==2.65.0->azure-cli)
Downloading argcomplete-3.3.0-py3-none-any.whl.metadata (16 kB)
Collecting azure-cli-telemetry==1.1.0.* (from azure-cli-core==2.65.0->azure-cli)
Downloading azure_cli_telemetry-1.1.0-py3-none-any.whl.metadata (1.8 kB)
Collecting azure-mgmt-core<2,>=1.2.0 (from azure-cli-core==2.65.0->azure-cli)
Downloading azure_mgmt_core-1.5.0-py3-none-any.whl.metadata (4.3 kB)
Collecting humanfriendly~=10.0 (from azure-cli-core==2.65.0->azure-cli)
Downloading humanfriendly-10.0-py2.py3-none-any.whl.metadata (9.2 kB)
Collecting jmespath (from azure-cli-core==2.65.0->azure-cli)
Downloading jmespath-1.0.1-py3-none-any.whl.metadata (7.6 kB)
Collecting knack~=0.11.0 (from azure-cli-core==2.65.0->azure-cli)
Downloading knack-0.11.0-py3-none-any.whl.metadata (5.2 kB)
Collecting msrestazure~=0.6.4 (from azure-cli-core==2.65.0->azure-cli)
Downloading msrestazure-0.6.4.post1-py2.py3-none-any.whl.metadata (15 kB)
Collecting paramiko<4.0.0,>=2.0.8 (from azure-cli-core==2.65.0->azure-cli)
Downloading paramiko-3.5.0-py3-none-any.whl.metadata (4.4 kB)
Collecting pkginfo>=1.5.0.1 (from azure-cli-core==2.65.0->azure-cli)
Downloading pkginfo-1.11.2-py3-none-any.whl.metadata (11 kB)
Requirement already satisfied: PyJWT>=2.1.0 in /usr/local/lib/python3.10/dist-packages (from azure-cli-core==2.65.0->azure-cli) (2.9.0)
Requirement already satisfied: pyopenssl>=17.1.0 in /usr/local/lib/python3.10/dist-packages (from azure-cli-core==2.65.0->azure-cli) (24.2.1)
Requirement already satisfied: requests[socks] in /usr/local/lib/python3.10/dist-packages (from azure-cli-core==2.65.0->azure-cli) (2.32.3)
Collecting microsoft-security-utilities-secret-masker~=1.0.0b2 (from azure-cli-core==2.65.0->azure-cli)
Downloading microsoft_security_utilities_secret_masker-1.0.0b3-py3-none-any.whl.metadata (3.2 kB)
Requirement already satisfied: psutil>=5.9 in /usr/local/lib/python3.10/dist-packages (from azure-cli-core==2.65.0->azure-cli) (5.9.5)
Collecting msrest>=0.6.21 (from azure-data-tables==12.4.0->azure-cli)
Downloading msrest-0.7.1-py3-none-any.whl.metadata (21 kB)
Collecting azure-common~=1.1 (from azure-keyvault-administration==4.4.0b2->azure-cli)
Downloading azure_common-1.1.28-py2.py3-none-any.whl.metadata (5.0 kB)
Collecting portalocker<3,>=1.4 (from msal-extensions>=1.2.0->azure-identity)
Downloading portalocker-2.10.1-py3-none-any.whl.metadata (8.5 kB)
Collecting applicationinsights<0.12,>=0.11.1 (from azure-cli-telemetry==1.1.0.*->azure-cli-core==2.65.0->azure-cli)
Downloading applicationinsights-0.11.10-py2.py3-none-any.whl.metadata (982 bytes)
Requirement already satisfied: lxml>=3.1.0 in /usr/local/lib/python3.10/dist-packages (from python-docx) (5.3.0)
Requirement already satisfied: cffi in /usr/local/lib/python3.10/dist-packages (from azure-datalake-store~=0.0.53->azure-cli) (1.17.1)
Requirement already satisfied: python-dateutil in /usr/local/lib/python3.10/dist-packages (from azure-multiapi-storage~=1.3.0->azure-cli) (2.8.2)
Requirement already satisfied: certifi in /usr/local/lib/python3.10/dist-packages (from elastic-transport<9,>=8.13->elasticsearch==8.14) (2024.8.30)
Collecting invoke>=2.0 (from fabric~=3.2.2->azure-cli)
Downloading invoke-2.2.0-py3-none-any.whl.metadata (3.3 kB)
Collecting decorator>=5 (from fabric~=3.2.2->azure-cli)
Downloading decorator-5.1.1-py3-none-any.whl.metadata (4.0 kB)
Requirement already satisfied: deprecated>=1.2 in /usr/local/lib/python3.10/dist-packages (from fabric~=3.2.2->azure-cli) (1.2.14)
Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from pycomposefile>=0.0.29->azure-cli) (6.0.2)
Requirement already satisfied: pycparser in /usr/local/lib/python3.10/dist-packages (from cffi->azure-datalake-store~=0.0.53->azure-cli) (2.22)
Requirement already satisfied: wrapt<2,>=1.10 in /usr/local/lib/python3.10/dist-packages (from deprecated>=1.2->fabric~=3.2.2->azure-cli) (1.16.0)
Requirement already satisfied: pygments in /usr/local/lib/python3.10/dist-packages (from knack~=0.11.0->azure-cli-core==2.65.0->azure-cli) (2.18.0)
Requirement already satisfied: requests-oauthlib>=0.5.0 in /usr/local/lib/python3.10/dist-packages (from msrest>=0.6.21->azure-data-tables==12.4.0->azure-cli) (1.3.1)
Collecting adal<2.0.0,>=0.6.0 (from msrestazure~=0.6.4->azure-cli-core==2.65.0->azure-cli)
Downloading adal-1.2.7-py2.py3-none-any.whl.metadata (6.9 kB)
Collecting bcrypt>=3.2 (from paramiko<4.0.0,>=2.0.8->azure-cli-core==2.65.0->azure-cli)
Downloading bcrypt-4.2.0-cp39-abi3-manylinux_2_28_x86_64.whl.metadata (9.6 kB)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests[socks]->azure-cli-core==2.65.0->azure-cli) (3.4.0)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests[socks]->azure-cli-core==2.65.0->azure-cli) (3.10)
Requirement already satisfied: PySocks!=1.5.7,>=1.5.6 in /usr/local/lib/python3.10/dist-packages (from requests[socks]->azure-cli-core==2.65.0->azure-cli) (1.7.1)
Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.10/dist-packages (from requests-oauthlib>=0.5.0->msrest>=0.6.21->azure-data-tables==12.4.0->azure-cli) (3.2.2)
Downloading elasticsearch-8.14.0-py3-none-any.whl (480 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 480.2/480.2 kB 26.4 MB/s eta 0:00:00
Downloading azure_identity-1.19.0-py3-none-any.whl (187 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 187.6/187.6 kB 13.4 MB/s eta 0:00:00
Downloading azure_storage_file_datalake-12.17.0-py3-none-any.whl (255 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 255.7/255.7 kB 20.3 MB/s eta 0:00:00
Downloading azure_cli-2.65.0-py3-none-any.whl (14.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.3/14.3 MB 60.4 MB/s eta 0:00:00
Downloading azure_cli_core-2.65.0-py3-none-any.whl (257 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 257.7/257.7 kB 15.9 MB/s eta 0:00:00
Downloading azure_data_tables-12.4.0-py3-none-any.whl (113 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 113.9/113.9 kB 7.9 MB/s eta 0:00:00
Downloading azure_keyvault_administration-4.4.0b2-py3-none-any.whl (105 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 105.1/105.1 kB 8.3 MB/s eta 0:00:00
Downloading azure_keyvault_certificates-4.7.0-py3-none-any.whl (428 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 428.1/428.1 kB 24.4 MB/s eta 0:00:00
Downloading azure_keyvault_keys-4.9.0b3-py3-none-any.whl (149 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 149.5/149.5 kB 9.4 MB/s eta 0:00:00
Downloading azure_keyvault_secrets-4.7.0-py3-none-any.whl (348 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 348.6/348.6 kB 20.7 MB/s eta 0:00:00
Downloading azure_mgmt_advisor-9.0.0-py2.py3-none-any.whl (46 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 46.9/46.9 kB 3.7 MB/s eta 0:00:00
Downloading azure_mgmt_apimanagement-4.0.0-py3-none-any.whl (804 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 804.5/804.5 kB 38.9 MB/s eta 0:00:00
Downloading azure_mgmt_appconfiguration-3.0.0-py3-none-any.whl (252 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 252.0/252.0 kB 17.2 MB/s eta 0:00:00
Downloading azure_mgmt_appcontainers-2.0.0-py3-none-any.whl (214 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 214.1/214.1 kB 13.7 MB/s eta 0:00:00
Downloading azure_mgmt_batchai-7.0.0b1-py2.py3-none-any.whl (99 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 99.1/99.1 kB 7.6 MB/s eta 0:00:00
Downloading azure_mgmt_billing-6.0.0-py2.py3-none-any.whl (166 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 167.0/167.0 kB 12.2 MB/s eta 0:00:00
Downloading azure_mgmt_cdn-12.0.0-py3-none-any.whl (239 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 239.4/239.4 kB 16.4 MB/s eta 0:00:00
Downloading azure_mgmt_containerinstance-10.1.0-py3-none-any.whl (87 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 87.3/87.3 kB 7.7 MB/s eta 0:00:00
Downloading azure_mgmt_containerregistry-10.3.0-py3-none-any.whl (2.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.3/2.3 MB 51.0 MB/s eta 0:00:00
Downloading azure_mgmt_cosmosdb-9.6.0-py3-none-any.whl (387 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 387.9/387.9 kB 18.2 MB/s eta 0:00:00
Downloading azure_mgmt_eventgrid-10.2.0b2-py3-none-any.whl (248 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 248.5/248.5 kB 12.8 MB/s eta 0:00:00
Downloading azure_mgmt_extendedlocation-1.0.0b2-py2.py3-none-any.whl (37 kB)
Downloading azure_mgmt_hdinsight-9.0.0b3-py3-none-any.whl (144 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 144.4/144.4 kB 10.8 MB/s eta 0:00:00
Downloading azure_mgmt_iothub-3.0.0-py3-none-any.whl (1.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 51.9 MB/s eta 0:00:00
Downloading azure_mgmt_iothubprovisioningservices-1.1.0-py3-none-any.whl (52 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 52.5/52.5 kB 4.4 MB/s eta 0:00:00
Downloading azure_mgmt_keyvault-10.3.0-py3-none-any.whl (933 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 933.0/933.0 kB 40.7 MB/s eta 0:00:00
Downloading azure_mgmt_loganalytics-13.0.0b4-py3-none-any.whl (162 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 162.6/162.6 kB 8.7 MB/s eta 0:00:00
Downloading azure_mgmt_marketplaceordering-1.1.0-py2.py3-none-any.whl (26 kB)
Downloading azure_mgmt_policyinsights-1.1.0b4-py3-none-any.whl (127 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 127.0/127.0 kB 8.8 MB/s eta 0:00:00
Downloading azure_mgmt_resource-23.1.1-py3-none-any.whl (2.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.6/2.6 MB 66.7 MB/s eta 0:00:00
Downloading azure_mgmt_security-6.0.0-py3-none-any.whl (1.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 49.6 MB/s eta 0:00:00
Downloading azure_mgmt_servicefabricmanagedclusters-2.0.0b6-py3-none-any.whl (204 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 204.4/204.4 kB 13.2 MB/s eta 0:00:00
Downloading azure_mgmt_servicelinker-1.2.0b2-py3-none-any.whl (95 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 95.1/95.1 kB 7.8 MB/s eta 0:00:00
Downloading azure_mgmt_signalr-2.0.0b1-py3-none-any.whl (123 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 123.6/123.6 kB 9.0 MB/s eta 0:00:00
Downloading azure_mgmt_sql-4.0.0b19-py3-none-any.whl (1.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 19.2 MB/s eta 0:00:00
Downloading azure_mgmt_sqlvirtualmachine-1.0.0b5-py3-none-any.whl (95 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 95.3/95.3 kB 4.7 MB/s eta 0:00:00
Downloading azure_mgmt_storage-21.2.0-py3-none-any.whl (3.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.2/3.2 MB 62.3 MB/s eta 0:00:00
Downloading azure_mgmt_synapse-2.1.0b5-py3-none-any.whl (547 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 547.1/547.1 kB 33.6 MB/s eta 0:00:00
Downloading azure_mgmt_web-7.2.0-py3-none-any.whl (6.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.0/6.0 MB 65.2 MB/s eta 0:00:00
Downloading azure_monitor_query-1.2.0-py3-none-any.whl (113 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 113.4/113.4 kB 10.7 MB/s eta 0:00:00
Downloading msal-1.31.0-py3-none-any.whl (113 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 113.1/113.1 kB 10.4 MB/s eta 0:00:00
Downloading msal_extensions-1.2.0-py3-none-any.whl (19 kB)
Downloading semver-2.13.0-py2.py3-none-any.whl (12 kB)
Downloading azure_cli_telemetry-1.1.0-py3-none-any.whl (11 kB)
Downloading python_docx-1.1.2-py3-none-any.whl (244 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 244.3/244.3 kB 20.9 MB/s eta 0:00:00
Downloading antlr4_python3_runtime-4.13.2-py3-none-any.whl (144 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 144.5/144.5 kB 12.2 MB/s eta 0:00:00
Downloading azure_appconfiguration-1.7.1-py3-none-any.whl (90 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 91.0/91.0 kB 6.2 MB/s eta 0:00:00
Downloading azure_batch-14.2.0-py3-none-any.whl (243 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 243.3/243.3 kB 18.0 MB/s eta 0:00:00
Downloading azure_core-1.32.0-py3-none-any.whl (198 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 198.9/198.9 kB 18.7 MB/s eta 0:00:00
Downloading azure_cosmos-3.2.0-py2.py3-none-any.whl (106 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 106.6/106.6 kB 9.4 MB/s eta 0:00:00
Downloading azure_datalake_store-0.0.53-py2.py3-none-any.whl (55 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 55.3/55.3 kB 4.1 MB/s eta 0:00:00
Downloading azure_graphrbac-0.60.0-py2.py3-none-any.whl (139 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 139.7/139.7 kB 9.7 MB/s eta 0:00:00
Downloading azure_mgmt_applicationinsights-1.0.0-py2.py3-none-any.whl (302 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 303.0/303.0 kB 20.5 MB/s eta 0:00:00
Downloading azure_mgmt_authorization-4.0.0-py3-none-any.whl (1.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 47.2 MB/s eta 0:00:00
Downloading azure_mgmt_batch-17.3.0-py3-none-any.whl (162 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 162.1/162.1 kB 11.3 MB/s eta 0:00:00
Downloading azure_mgmt_botservice-2.0.0-py3-none-any.whl (126 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 126.5/126.5 kB 10.0 MB/s eta 0:00:00
Downloading azure_mgmt_cognitiveservices-13.5.0-py3-none-any.whl (144 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 144.3/144.3 kB 11.7 MB/s eta 0:00:00
Downloading azure_mgmt_compute-33.0.0-py3-none-any.whl (5.9 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.9/5.9 MB 64.1 MB/s eta 0:00:00
Downloading azure_mgmt_containerservice-32.0.0-py3-none-any.whl (10.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.3/10.3 MB 60.8 MB/s eta 0:00:00
Downloading azure_mgmt_databoxedge-1.0.0-py2.py3-none-any.whl (1.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 42.0 MB/s eta 0:00:00
Downloading azure_mgmt_datamigration-10.0.0-py2.py3-none-any.whl (174 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 174.5/174.5 kB 12.4 MB/s eta 0:00:00
Downloading azure_mgmt_devtestlabs-4.0.0-py2.py3-none-any.whl (137 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 137.0/137.0 kB 10.6 MB/s eta 0:00:00
Downloading azure_mgmt_dns-8.0.0-py2.py3-none-any.whl (118 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 118.9/118.9 kB 9.4 MB/s eta 0:00:00
Downloading azure_mgmt_eventhub-10.1.0-py3-none-any.whl (598 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 598.9/598.9 kB 30.2 MB/s eta 0:00:00
Downloading azure_mgmt_imagebuilder-1.3.0-py3-none-any.whl (78 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.1/78.1 kB 5.9 MB/s eta 0:00:00
Downloading azure_mgmt_iotcentral-10.0.0b2-py3-none-any.whl (55 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 55.4/55.4 kB 3.8 MB/s eta 0:00:00
Downloading azure_mgmt_kusto-0.3.0-py2.py3-none-any.whl (73 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 73.9/73.9 kB 6.1 MB/s eta 0:00:00
Downloading azure_mgmt_managementgroups-1.0.0-py2.py3-none-any.whl (58 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 58.1/58.1 kB 3.8 MB/s eta 0:00:00
Downloading azure_mgmt_maps-2.0.0-py2.py3-none-any.whl (38 kB)
Downloading azure_mgmt_media-9.0.0-py3-none-any.whl (211 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 211.1/211.1 kB 14.6 MB/s eta 0:00:00
Downloading azure_mgmt_monitor-5.0.1-py3-none-any.whl (1.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 40.2 MB/s eta 0:00:00
Downloading azure_mgmt_msi-7.0.0-py3-none-any.whl (183 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 183.4/183.4 kB 12.2 MB/s eta 0:00:00
Downloading azure_mgmt_netapp-10.1.0-py3-none-any.whl (200 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 200.7/200.7 kB 16.0 MB/s eta 0:00:00
Downloading azure_mgmt_privatedns-1.0.0-py2.py3-none-any.whl (43 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 44.0/44.0 kB 2.9 MB/s eta 0:00:00
Downloading azure_mgmt_rdbms-10.2.0b18-py3-none-any.whl (556 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 556.6/556.6 kB 32.8 MB/s eta 0:00:00
Downloading azure_mgmt_recoveryservices-3.0.0-py3-none-any.whl (108 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 108.6/108.6 kB 9.4 MB/s eta 0:00:00
Downloading azure_mgmt_recoveryservicesbackup-9.1.0-py3-none-any.whl (570 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 570.9/570.9 kB 32.9 MB/s eta 0:00:00
Downloading azure_mgmt_redhatopenshift-1.5.0-py3-none-any.whl (408 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 408.4/408.4 kB 20.4 MB/s eta 0:00:00
Downloading azure_mgmt_redis-14.4.0-py3-none-any.whl (127 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 127.4/127.4 kB 8.9 MB/s eta 0:00:00
Downloading azure_mgmt_search-9.1.0-py3-none-any.whl (110 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 110.3/110.3 kB 10.1 MB/s eta 0:00:00
Downloading azure_mgmt_servicebus-8.2.0-py3-none-any.whl (940 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 940.1/940.1 kB 40.6 MB/s eta 0:00:00
Downloading azure_mgmt_servicefabric-2.1.0-py3-none-any.whl (124 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 124.3/124.3 kB 11.0 MB/s eta 0:00:00
Downloading azure_mgmt_trafficmanager-1.0.0-py3-none-any.whl (49 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 49.3/49.3 kB 3.9 MB/s eta 0:00:00
Downloading azure_multiapi_storage-1.3.0-py2.py3-none-any.whl (2.9 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.9/2.9 MB 50.5 MB/s eta 0:00:00
Downloading azure_storage_blob-12.23.1-py3-none-any.whl (405 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 405.6/405.6 kB 24.3 MB/s eta 0:00:00
Downloading azure_storage_common-1.4.2-py2.py3-none-any.whl (47 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 47.4/47.4 kB 3.3 MB/s eta 0:00:00
Downloading azure_synapse_accesscontrol-0.5.0-py2.py3-none-any.whl (30 kB)
Downloading azure_synapse_artifacts-0.19.0-py3-none-any.whl (495 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 495.8/495.8 kB 33.5 MB/s eta 0:00:00
Downloading azure_synapse_managedprivateendpoints-0.4.0-py2.py3-none-any.whl (45 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 46.0/46.0 kB 3.2 MB/s eta 0:00:00
Downloading azure_synapse_spark-0.2.0-py2.py3-none-any.whl (29 kB)
Downloading colorama-0.4.6-py2.py3-none-any.whl (25 kB)
Downloading elastic_transport-8.15.1-py3-none-any.whl (64 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 64.4/64.4 kB 3.4 MB/s eta 0:00:00
Downloading fabric-3.2.2-py3-none-any.whl (59 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 59.4/59.4 kB 4.9 MB/s eta 0:00:00
Downloading isodate-0.7.2-py3-none-any.whl (22 kB)
Downloading javaproperties-0.5.2-py2.py3-none-any.whl (19 kB)
Downloading jsondiff-2.0.0-py3-none-any.whl (6.6 kB)
Downloading pycomposefile-0.0.32-py3-none-any.whl (28 kB)
Downloading PyGithub-1.59.1-py3-none-any.whl (342 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 342.2/342.2 kB 27.3 MB/s eta 0:00:00
Downloading PyNaCl-1.5.0-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (856 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 856.7/856.7 kB 42.4 MB/s eta 0:00:00
Downloading scp-0.13.6-py2.py3-none-any.whl (8.2 kB)
Downloading sshtunnel-0.1.5-py2.py3-none-any.whl (23 kB)
Downloading websocket_client-1.3.3-py3-none-any.whl (54 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 54.3/54.3 kB 4.1 MB/s eta 0:00:00
Downloading xmltodict-0.14.2-py2.py3-none-any.whl (10.0 kB)
Downloading argcomplete-3.3.0-py3-none-any.whl (42 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 42.6/42.6 kB 3.7 MB/s eta 0:00:00
Downloading azure_common-1.1.28-py2.py3-none-any.whl (14 kB)
Downloading azure_mgmt_core-1.5.0-py3-none-any.whl (30 kB)
Downloading decorator-5.1.1-py3-none-any.whl (9.1 kB)
Downloading humanfriendly-10.0-py2.py3-none-any.whl (86 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 86.8/86.8 kB 6.9 MB/s eta 0:00:00
Downloading invoke-2.2.0-py3-none-any.whl (160 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 160.3/160.3 kB 13.3 MB/s eta 0:00:00
Downloading knack-0.11.0-py3-none-any.whl (60 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 60.8/60.8 kB 5.4 MB/s eta 0:00:00
Downloading microsoft_security_utilities_secret_masker-1.0.0b3-py3-none-any.whl (16 kB)
Downloading msrest-0.7.1-py3-none-any.whl (85 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 85.4/85.4 kB 7.3 MB/s eta 0:00:00
Downloading msrestazure-0.6.4.post1-py2.py3-none-any.whl (40 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.8/40.8 kB 2.9 MB/s eta 0:00:00
Downloading paramiko-3.5.0-py3-none-any.whl (227 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 227.1/227.1 kB 18.0 MB/s eta 0:00:00
Downloading pkginfo-1.11.2-py3-none-any.whl (31 kB)
Downloading portalocker-2.10.1-py3-none-any.whl (18 kB)
Downloading jmespath-1.0.1-py3-none-any.whl (20 kB)
Downloading adal-1.2.7-py2.py3-none-any.whl (55 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 55.5/55.5 kB 4.5 MB/s eta 0:00:00
Downloading applicationinsights-0.11.10-py2.py3-none-any.whl (55 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 55.1/55.1 kB 3.9 MB/s eta 0:00:00
Downloading bcrypt-4.2.0-cp39-abi3-manylinux_2_28_x86_64.whl (273 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 273.8/273.8 kB 20.5 MB/s eta 0:00:00
Installing collected packages: microsoft-security-utilities-secret-masker, jsondiff, azure-common, applicationinsights, antlr4-python3-runtime, xmltodict, websocket-client, semver, python-docx, pycomposefile, portalocker, pkginfo, jmespath, javaproperties, isodate, invoke, humanfriendly, elastic-transport, decorator, colorama, bcrypt, argcomplete, PyNaCl, knack, elasticsearch, azure-cosmos, azure-core, azure-cli-telemetry, paramiko, msrest, azure-storage-common, azure-storage-blob, azure-monitor-query, azure-mgmt-core, azure-keyvault-secrets, azure-keyvault-keys, azure-keyvault-certificates, azure-keyvault-administration, azure-appconfiguration, adal, sshtunnel, scp, PyGithub, msrestazure, msal, fabric, azure-synapse-spark, azure-synapse-managedprivateendpoints, azure-synapse-artifacts, azure-synapse-accesscontrol, azure-storage-file-datalake, azure-multiapi-storage, azure-mgmt-web, azure-mgmt-trafficmanager, azure-mgmt-synapse, azure-mgmt-storage, azure-mgmt-sqlvirtualmachine, azure-mgmt-sql, azure-mgmt-signalr, azure-mgmt-servicelinker, azure-mgmt-servicefabricmanagedclusters, azure-mgmt-servicefabric, azure-mgmt-servicebus, azure-mgmt-security, azure-mgmt-search, azure-mgmt-resource, azure-mgmt-redis, azure-mgmt-redhatopenshift, azure-mgmt-recoveryservicesbackup, azure-mgmt-recoveryservices, azure-mgmt-rdbms, azure-mgmt-privatedns, azure-mgmt-policyinsights, azure-mgmt-netapp, azure-mgmt-msi, azure-mgmt-monitor, azure-mgmt-media, azure-mgmt-marketplaceordering, azure-mgmt-maps, azure-mgmt-managementgroups, azure-mgmt-loganalytics, azure-mgmt-keyvault, azure-mgmt-iothubprovisioningservices, azure-mgmt-iothub, azure-mgmt-iotcentral, azure-mgmt-imagebuilder, azure-mgmt-hdinsight, azure-mgmt-extendedlocation, azure-mgmt-eventhub, azure-mgmt-eventgrid, azure-mgmt-dns, azure-mgmt-datamigration, azure-mgmt-databoxedge, azure-mgmt-cosmosdb, azure-mgmt-containerservice, azure-mgmt-containerregistry, azure-mgmt-containerinstance, azure-mgmt-compute, azure-mgmt-cognitiveservices, azure-mgmt-cdn, azure-mgmt-botservice, azure-mgmt-billing, azure-mgmt-batchai, azure-mgmt-batch, azure-mgmt-authorization, azure-mgmt-applicationinsights, azure-mgmt-appcontainers, azure-mgmt-appconfiguration, azure-mgmt-apimanagement, azure-mgmt-advisor, azure-data-tables, msal-extensions, azure-mgmt-kusto, azure-mgmt-devtestlabs, azure-graphrbac, azure-datalake-store, azure-batch, azure-identity, azure-cli-core, azure-cli
Attempting uninstall: websocket-client
Found existing installation: websocket-client 1.8.0
Uninstalling websocket-client-1.8.0:
Successfully uninstalled websocket-client-1.8.0
Attempting uninstall: decorator
Found existing installation: decorator 4.4.2
Uninstalling decorator-4.4.2:
Successfully uninstalled decorator-4.4.2
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
ipython 7.34.0 requires jedi>=0.16, which is not installed.
moviepy 1.0.3 requires decorator<5.0,>=4.0.2, but you have decorator 5.1.1 which is incompatible.
Successfully installed PyGithub-1.59.1 PyNaCl-1.5.0 adal-1.2.7 antlr4-python3-runtime-4.13.2 applicationinsights-0.11.10 argcomplete-3.3.0 azure-appconfiguration-1.7.1 azure-batch-14.2.0 azure-cli-2.65.0 azure-cli-core-2.65.0 azure-cli-telemetry-1.1.0 azure-common-1.1.28 azure-core-1.32.0 azure-cosmos-3.2.0 azure-data-tables-12.4.0 azure-datalake-store-0.0.53 azure-graphrbac-0.60.0 azure-identity-1.19.0 azure-keyvault-administration-4.4.0b2 azure-keyvault-certificates-4.7.0 azure-keyvault-keys-4.9.0b3 azure-keyvault-secrets-4.7.0 azure-mgmt-advisor-9.0.0 azure-mgmt-apimanagement-4.0.0 azure-mgmt-appconfiguration-3.0.0 azure-mgmt-appcontainers-2.0.0 azure-mgmt-applicationinsights-1.0.0 azure-mgmt-authorization-4.0.0 azure-mgmt-batch-17.3.0 azure-mgmt-batchai-7.0.0b1 azure-mgmt-billing-6.0.0 azure-mgmt-botservice-2.0.0 azure-mgmt-cdn-12.0.0 azure-mgmt-cognitiveservices-13.5.0 azure-mgmt-compute-33.0.0 azure-mgmt-containerinstance-10.1.0 azure-mgmt-containerregistry-10.3.0 azure-mgmt-containerservice-32.0.0 azure-mgmt-core-1.5.0 azure-mgmt-cosmosdb-9.6.0 azure-mgmt-databoxedge-1.0.0 azure-mgmt-datamigration-10.0.0 azure-mgmt-devtestlabs-4.0.0 azure-mgmt-dns-8.0.0 azure-mgmt-eventgrid-10.2.0b2 azure-mgmt-eventhub-10.1.0 azure-mgmt-extendedlocation-1.0.0b2 azure-mgmt-hdinsight-9.0.0b3 azure-mgmt-imagebuilder-1.3.0 azure-mgmt-iotcentral-10.0.0b2 azure-mgmt-iothub-3.0.0 azure-mgmt-iothubprovisioningservices-1.1.0 azure-mgmt-keyvault-10.3.0 azure-mgmt-kusto-0.3.0 azure-mgmt-loganalytics-13.0.0b4 azure-mgmt-managementgroups-1.0.0 azure-mgmt-maps-2.0.0 azure-mgmt-marketplaceordering-1.1.0 azure-mgmt-media-9.0.0 azure-mgmt-monitor-5.0.1 azure-mgmt-msi-7.0.0 azure-mgmt-netapp-10.1.0 azure-mgmt-policyinsights-1.1.0b4 azure-mgmt-privatedns-1.0.0 azure-mgmt-rdbms-10.2.0b18 azure-mgmt-recoveryservices-3.0.0 azure-mgmt-recoveryservicesbackup-9.1.0 azure-mgmt-redhatopenshift-1.5.0 azure-mgmt-redis-14.4.0 azure-mgmt-resource-23.1.1 azure-mgmt-search-9.1.0 azure-mgmt-security-6.0.0 azure-mgmt-servicebus-8.2.0 azure-mgmt-servicefabric-2.1.0 azure-mgmt-servicefabricmanagedclusters-2.0.0b6 azure-mgmt-servicelinker-1.2.0b2 azure-mgmt-signalr-2.0.0b1 azure-mgmt-sql-4.0.0b19 azure-mgmt-sqlvirtualmachine-1.0.0b5 azure-mgmt-storage-21.2.0 azure-mgmt-synapse-2.1.0b5 azure-mgmt-trafficmanager-1.0.0 azure-mgmt-web-7.2.0 azure-monitor-query-1.2.0 azure-multiapi-storage-1.3.0 azure-storage-blob-12.23.1 azure-storage-common-1.4.2 azure-storage-file-datalake-12.17.0 azure-synapse-accesscontrol-0.5.0 azure-synapse-artifacts-0.19.0 azure-synapse-managedprivateendpoints-0.4.0 azure-synapse-spark-0.2.0 bcrypt-4.2.0 colorama-0.4.6 decorator-5.1.1 elastic-transport-8.15.1 elasticsearch-8.14.0 fabric-3.2.2 humanfriendly-10.0 invoke-2.2.0 isodate-0.7.2 javaproperties-0.5.2 jmespath-1.0.1 jsondiff-2.0.0 knack-0.11.0 microsoft-security-utilities-secret-masker-1.0.0b3 msal-1.31.0 msal-extensions-1.2.0 msrest-0.7.1 msrestazure-0.6.4.post1 paramiko-3.5.0 pkginfo-1.11.2 portalocker-2.10.1 pycomposefile-0.0.32 python-docx-1.1.2 scp-0.13.6 semver-2.13.0 sshtunnel-0.1.5 websocket-client-1.3.3 xmltodict-0.14.2
Let's sign in with your Microsoft account. At the end of the process, the Google Colab console will display your available subscriptions. Select your subscription by entering the subscription number shown in the console.
[ ]
Error loading command module 'mysql': cannot import name 'mysql_flexibleservers' from 'azure.mgmt.rdbms' (/usr/local/lib/python3.10/dist-packages/azure/mgmt/rdbms/__init__.py) Error loading command module 'rdbms': No module named 'azure.mgmt.rdbms.mysql_flexibleservers' To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code AYGZ24EJG to authenticate. Retrieving tenants and subscriptions for the selection... The following tenants don't contain accessible subscriptions. Use `az login --allow-no-subscriptions` to have tenant level access. 56783f2b-514d-4ff9-b3ab-3fdc5e7ace5d 'Kibernum' [Tenant and subscription selection] No Subscription name Subscription ID Tenant ----- ------------------------- ------------------------------------ ------------------------------------ [1] * N/A(tenant level account) 56783f2b-514d-4ff9-b3ab-3fdc5e7ace5d 56783f2b-514d-4ff9-b3ab-3fdc5e7ace5d The default is marked with an *; the default tenant is '56783f2b-514d-4ff9-b3ab-3fdc5e7ace5d' and subscription is 'N/A(tenant level account)' (56783f2b-514d-4ff9-b3ab-3fdc5e7ace5d). Select a subscription and tenant (Type a number or Enter for no changes): 1 Tenant: 56783f2b-514d-4ff9-b3ab-3fdc5e7ace5d Subscription: N/A(tenant level account) (56783f2b-514d-4ff9-b3ab-3fdc5e7ace5d) [Announcements] With the new Azure CLI login experience, you can select the subscription you want to use more easily. Learn more about it and its configuration at https://go.microsoft.com/fwlink/?linkid=2271236 If you encounter any problem, please open an issue at https://aka.ms/azclibug [Warning] The login output has been updated. Please be aware that it no longer displays the full list of available subscriptions by default.
Variable Declaration
Necessary variables for the operation of the services we will use. Please, insert your Elasticsearch credentials below:
[ ]
Elastic Cloud ID: ·········· Elastic Api Key: ··········
Initialization of OneLake and Elasticsearch Services
[ ]
Functions declaration
[ ]
[ ]
[ ]
[ ]
[ ]
[ ]
Uploading data to OneLake
Before start to upload your files is higlhy recommended to clean the Google Colab directory.
[ ]
Upload the csv file:
[ ]
Saving products.csv to products.csv
Upload the 3 docx files:
[ ]
Saving sport-sneakers.docx to sport-sneakers.docx Saving classic-loafers.docx to classic-loafers.docx Saving beach-flip-flops.docx to beach-flip-flops.docx
[ ]
File: products.csv uploaded to the data lake. File: sport-sneakers.docx uploaded to the data lake. File: classic-loafers.docx uploaded to the data lake. File: beach-flip-flops.docx uploaded to the data lake.
Check if files are now available in your datalake.
[ ]
shoesticDatalake.Lakehouse/Files/ProductsData/beach-flip-flops.docx shoesticDatalake.Lakehouse/Files/ProductsData/classic-loafers.docx shoesticDatalake.Lakehouse/Files/ProductsData/products.csv shoesticDatalake.Lakehouse/Files/ProductsData/sport-sneakers.docx
Indexing data to Elasticsearch
Creating embeddings endpoint
[ ]
Embedding endpoint created successfully.
Creating Mappings
[ ]
[ ]
Indexing data
Retrieving documents from OneLake
[ ]
Extract content
[ ]
CSV FILE CONTENT: product_id,product_name,amount,tags P-115,Classic Loafers,19.69,"classic,formal" P-114,Sport Sneakers,156.18,"sport,casual" P-109,Ankle Boots,87.13,"boots,winter" P-118,Casual Sandals,128.22,"casual,summer" P-112,Running Shoes,132.27,"sport,running" P-116,Leather Sandals,107.63,"leather,casual" P-105,Canvas Sneakers,75.56,"casual,sneakers" P-107,Winter Boots,49.57,"winter,outdoor" P-111,Formal Oxfords,177.70,"formal,classic" P-119,Beach Flip-Flops,50.64,"beach,summer" DOCX FILE CONTENT: ['Crafted for an active lifestyle, this footwear offers excellent support and durability, featuring a non-slip sole that’s perfect for both workouts and casual use.', 'With a timeless design, this elegant choice pairs well with professional and social attire. Made from high-quality leather, it adds a touch of sophistication.', 'Ideal for warm, sunny days by the water, these lightweight essentials are water-resistant and come in bright colors, bringing a laid-back vibe to any outing in the sun.']
[ ]
['classic', 'formal'] ['sport', 'casual'] ['boots', 'winter'] ['casual', 'summer'] ['sport', 'running'] ['leather', 'casual'] ['casual', 'sneakers'] ['winter', 'outdoor'] ['formal', 'classic'] ['beach', 'summer']
Indexing data
[ ]
CSV data indexed successfully. DOCX data indexed successfully.
Retrieving data from Elasticsearch
Semantic search: let's search for the word 'summer' and see what results the search returns.
[ ]
Ideal for warm, sunny days by the water, these lightweight essentials are water-resistant and come in bright colors, bringing a laid-back vibe to any outing in the sun.
Now let's search for the same word in the product tags using a full-text search. Both responses are related.
[ ]
{'product_id': 'P-118', 'product_name': 'Casual Sandals', 'amount': '128.22', 'tags': ['casual', 'summer']}
Cleanup
Finally, we can delete the resources used to prevent them from consuming resources.
[ ]
<ipython-input-16-a48b0e63148f>:2: DeprecationWarning: Passing transport options in the API method is deprecated. Use 'Elasticsearch.options()' instead. es_client.indices.delete(index="shoestic-products", ignore=[400, 404]) <ipython-input-16-a48b0e63148f>:3: DeprecationWarning: Passing transport options in the API method is deprecated. Use 'Elasticsearch.options()' instead. es_client.indices.delete(index="shoestic-products-descriptions", ignore=[400, 404]) <ipython-input-16-a48b0e63148f>:6: DeprecationWarning: Passing transport options in the API method is deprecated. Use 'Elasticsearch.options()' instead. es_client.inference.delete_model(inference_id="onelake-inference-endpoint", ignore=[400, 404])
ObjectApiResponse({'acknowledged': True, 'pipelines': []})