Questions tagged [json]
37 questions
6
votes
1 answer
Database options for JSON storage, queried with Apache Drill
I am planning to set up a JSON storage system. It will store tens of millions of JSON records, all in the same format. I'd like to be able to query the data using Apache Drill.
It looks like there is Drill support for MongoDB and Postgres. However,…
sheldonkreger
- 1,169
- 8
- 20
5
votes
1 answer
Python & Pandas : TypeError: to_sql() got an unexpected keyword argument 'flavor'
I want to store JSON Data into MySQL Database using Python.
I used dataframe of pandas. I found that to_sql() can do this job easily.
Python Code:
jdata=json.loads(json_data)
df=pandas.DataFrame(jdata)
df.to_sql(con=con, name='crashTable',…
Dipankar Nalui
- 155
- 1
- 2
- 5
4
votes
1 answer
Load large .jsons file into Pandas dataframe
I'm trying to load a large jsons-file (2.5 GB) into a Pandas dataframe. Due to the large size of the file pandas.read_json() will result in a memory error.
Therefore I'm trying to read it in like this:
S_DIR = r'path-to-directory'
with…
sir_olf
- 99
- 1
- 2
- 3
2
votes
1 answer
Problem reading python code
Can someone explain the following python code.
value_geojson["features"][0]["properties"]["title"]
value_geojson is a geojson variable.
I like to think that I'm not a total python newbie but these are too many [] for me. I am a GeoJSON newbie…
DGIS
- 123
- 4
2
votes
1 answer
Keras save model FailedPreconditionError
Model works and fits. After adding model.save('model.h5'). I am receiving
tensorflow.python.framework.errors_impl.FailedPreconditionError: Attempting to use uninitialized value dense_11/bias[[Node: _retval_dense_11/bias_0_0 = _RetvalT=DT_FLOAT,…
Myron
- 123
- 1
- 3
- 14
2
votes
1 answer
AttributeError: 'str' object has no attribute 'keys'
While converting json file to csv,I got the above error
My code is shown below:
import json
import csv
data = open("SRInFlow.json",encoding="utf8")
rows = json.loads(data.read())
r=rows.values()
fieldnames = [b'dateTimeValues', b'timeSeries']
with…
Sheetal
- 53
- 2
- 3
- 7
2
votes
1 answer
Ingestion of periodic REST API Calls into Hadoop
We are planning to use REST API calls to ingest data from an endpoint and store the data to HDFS. The REST calls are done in a periodic fashion (daily or maybe hourly).
I've already done Twitter ingestion using Flume, but I don't think using Flume…
geek-tech
- 29
- 3
2
votes
3 answers
Unable to open .json file in pandas
I want to convert a json file into a dataframe in pandas (Python). I tried with read_json() but got the error:
UnicodeDecodeError:'charmap' codec can't decode byte 0x81 in position 21596351:character maps to
I think I have some…
Abhishek Pathak
- 45
- 1
- 1
- 6
2
votes
2 answers
Read big JSON file as a Dataframe using polars
I am pretty new to polars and have been working around it just to get acquainted.
import polars as pl
filename_data = 'endomondoHR.json'
pl.read_json(filename_data)
Data -> kaggle data
The error I am getting is…
shivanshu dhawan
- 188
- 1
- 2
- 9
2
votes
1 answer
How do I normalize json data into pandas (Covid-19 data)
I am trying to import all up-to-date datasets in JSON format on the covid-19 pandemic into a pandas dataframe. I believe it should be possible by using json_normalize but I can't make it work.
First, I initialize and import the json data:
import…
jontheil
- 21
- 6
1
vote
3 answers
Convert json to dataframe in python
I have a json array f below format
[{
'Address': 'xxx',
'Latitude': 28. xxx,
'Longitude': 77. xxx,
'reached': False
}, {
'Address': 'yyy',
'Latitude': 18. yyy,
'Longitude': 73. yyy,
'reached': False
}]
i want to…
phoenix
- 111
- 1
- 1
- 3
1
vote
0 answers
Python and data analysis how can we transform a json array into rows having json properties as columns titles?
I need to create a recommendation system for a small company using its 500 first orders.
I received a .json file having the following structure:
{
"data":
[
{
"order_id": 1,
"seller_id": 1,
…
alim1990
- 173
- 1
- 8
1
vote
0 answers
Tool/Technology/Framework to process file with 1GB
I'm new to datascience, but I have the following problem:
1) Read a file of 1GB, which each line is a json object
2) There are two more files, much smaller, which I need to JOIN some data
In this case, which kind of tools is the best?
I saw some…
Ramon Medeiros
- 113
- 5
1
vote
2 answers
How to save API data into CSV format?
Is there an easy way to convert API data with python commands and output the results in CSV format?
Here's what I'm doing now:
import requests
params = {
"api_key": "fec742c1-c846-4343-a9f1-91c729acd097",
"format": "jsonp"
}
r =…
Muhammad Ali
- 2,509
- 5
- 21
- 22
1
vote
1 answer
SQL and JSON database - R
I have a database (PostgresSQL) which looks like this:
""results"":
{""WEBS"":
{""WEBS_4"": [{""ip"": ""XXXXX"", ""value"": ""YYYY""}]}}
I would like to access the information of ip.
Then I write the following query in R, but it does…
MariaKK
- 11
- 1